\section{genetic Personalized Community Detection}

%
My proposed gPCD model contains an offline and an online step. Figure \ref{fig:pipeline} shows the pipeline of the whole framework. The offline step first encodes the user-independent binary community tree (\textit{Section \ref{sc:c3_offline}}), and subsequently learns embedding representations for both user need and nodes on the binary community tree (\textit{Section \ref{sc:c3_representation}}). The online step introduces the genetic personalized community detection approach (\textit{Section \ref{sc:c3_online}}). To accelerate the running speed, a distributed version of gPCD model is also deployed on HDFS (\textit{Section \ref{sc:c3_distributed}}).  To disambiguate the notations mentioned in this section to better explain the gPCD model, some commonly used notations can be found in Table \ref{tab:notation}.

\subsection{Offline Community Tree Index} \label{sc:c3_offline}

One challenge of solving personalized community detection problem is the computational cost due to the complexity of personalization and graph structure. In order to reduce the online workload, most of computation cost is put into the one-time offline step whose time cost can be excluded from the online personalized community detection step. Thus, I first convert the graph into a binary community tree offline to retain user-independent community information.

Infomap algorithm \cite{rosvall2011multilevel} is employed to generate the user-independent communities solely based on the graph \textit{G(V, E)}. Infomap algorithm simulates a random walker wandering on the graph and indexes the  description length of his / her random walk path via multilevel codebooks. By minimizing the description length based on the map equation below, community structures are formed for the graph.
%\begin{equation} \small
%\textit{$L(\mathcal{M})$} = q_{\curvearrowright}H(\mathcal{Q})+\sum_{i=1}^{m} \textit{$p_{\circlearrowright}^{i}$}H(\textit{$\mathcal{P}^{i}$})
%\end{equation}
%where $L(\mathcal{M})$ is the description length for a random walker in the current community $\mathcal{M}$. $q_{\curvearrowright}$ and $p_{\circlearrowright}^{i}$ are the jumping rates between communities and within the $i_{th}$ community. $H(\mathcal{Q})$ is the frequency-weighted average length of codewords in the global index codebook and $H(\mathcal{P}^{i})$ is frequency-weighted average length of codewords in the $i_{th}$ community codebook.

\begin{equation}
\textit{$L(\pi)$} = \sum_{i}^{m}q_{\curvearrowright}^{i}H(\mathcal{Q})+\sum_{i=1}^{m} \textit{$p_{\circlearrowright}^{i}$}H(\textit{$\mathcal{P}^{i}$})
\end{equation}
where \textit{$L(\pi)$} is the description length for a random walker under current community partition $\pi$. $q_{\curvearrowright}^{i}$ and $p_{\circlearrowright}^{i}$ are the jumping rates between communities and within the $i_{th}$ community in each step. $H(\mathcal{Q})$ is the frequency-weighted average length of codewords in the global index codebook and $H(\mathcal{P}^{i})$ is frequency-weighted average length of codewords in the $i_{th}$ community codebook. Followed by this equation to partition communities into sub-communities , a hierarchical community tree \textit{$T_{c}(N^{c},L^{c})$} is constructed from the original graph $G(V,E)$.
\begin{figure}
	% \advance\leftskip-1cm
	\center
	\includegraphics[width=1.0\columnwidth]{img/chapter3/4.pdf}
	%  \vspace{-3em}
	\caption{The framework of gPCD model. (a) refers to the offline construction step on the original graph and (b) refers to the online genetic pruning step to generate personalized communities.}
	\label{fig:pipeline}
	%  \vspace{-1em}
\end{figure}
In \textit{$T_{c}(N^{c},L^{c})$}, each parent node can have multiple child nodes which can be regarded as a community partition on the parent node. For instance, a node $N_{k}^{c} \in N^{c}$ from $T_{c}(N^{c},L^{c})$ represents a community of vertices.  Its $m$ child nodes $\{N_{k_{1}}^{c},N_{k_{2}}^{c},...,N_{k_{m}}^{c}\}$ represent $m$ sub-communities of vertices from $G(V,E)$ where we have $	\bigcap_{i=1}^{m} N_{k_{i}}^{c} = \varnothing$ and  $	\bigcup_{i=1}^{m} N_{k_{i}}^{c} =  N_{k}^{c}$.

In order to achieve an efficient personalized community detection in the following online step, I convert the hierarchical community tree \textit{$T_{c}(N^{c},L^{c})$} to a binary community tree \textit{$T_{b}(N^{b},L^{b})$} for index. Specifically, for $m$ child nodes of a parent node $N_{k}^{c}$, a bottom-up approach is proposed to merge a selected pair of sibling nodes as a new node in an iterative manner. The approach runs until all $m$ child nodes merged together to form the parent node $N_{k}^{c}$. To avoid an unbalanced tree where small communities are always left to merge with huge communities in the end, I first select the node with the smallest community size among all sibling nodes in each merging step. It is merged with its sibling node with the largest normalized linked weight (Please refer to Figure \ref{fig:pipeline}(a)). The normalized linked weight function $w(\cdot)$ between  two nodes $N_{i}^{c}$ and $N_{j}^{c}$ is defined as:
\begin{equation}
\textit{$w(N_{i}^{c},N_{j}^{c})$} =\frac{\textit{$N_{i}^{c}\odot N_{j}^{c}$}-\frac{\mathcal{D}(N_{i}^{c})\cdot \mathcal{D}(N_{j}^{c})}{2|E|} }{\textit{$| N_{i}^{c}||N_{j}^{c}|$}}
\end{equation}
where \textit{$N_{i}^{c}\odot N_{j}^{c}$} denotes the number of edges linked between vertices in node \textit{$N_{i}^{c}$} and \textit{$N_{j}^{c}$}, which can be interpreted as the linkage strength between them; \textit{$|N_{i}^{c}|$} is the number of vertices inside node \textit{$N_{i}^{c}$}; \textit{$\mathcal{D}(N_{i}^{c})$} is the out-degree of node \textit{$N_{i}^{c}$} (the total number of edges linked to other nodes) and \textit{$|E|$} is the total number of edges in the original graph $G(V,E)$ . $\frac{\mathcal{D}(N_{i}^{c})\cdot \mathcal{D}(N_{j}^{c})}{2|E|} $ denotes the random linkage strength between node \textit{$N_{i}^{c}$} and \textit{$N_{j}^{c}$}. The \textit{$w(\cdot)$} function calculates how much that two nodes are better connected beyond random connection and is normalized by node size. Given the node $N_{i}^{c}$ with the smallest community size and all its sibling node set $S$, The merging step can be formulated as:
\begin{equation}
\begin{aligned}
& N_{j}^{c} \Leftarrow \argmax_{N_{j}^{c} \in S}w(N_{i}^{c},N_{j}^{c})\\
& N_{*}^{c} = N_{i}^{c}\bigcup N_{j}^{c}
\end{aligned}
\end{equation}
The bottom-up process will stop until all child nodes are merged together to form the parent node. In the end, the hierarchical community tree \textit{$T_{c}(N^{c},L^{c})$} is fully converted to a binary community tree \textit{$T_{b}(N^{b},L^{b})$} with user-independent community information.  The node size $|N^{b}|$ as well as the link size $|L^{b}|$  in \textit{$T_{b}(N^{b},L^{b})$} is at most $2|V|$ which is smaller than the size of original graph $G(V,E)$. If I consider to form the binary community tree with only $k$ levels, the size of \textit{$T_{b}(N^{b},L^{b})$} can be even smaller.

\begin{table}[h]
	%\scriptsize
	%	\small
	\centering

	\begin{tabular}{|p{3cm}|p{11cm}|}
		\hline
		\textbf{Notations} & \textbf{Descriptions} \\ \hline
		$G(V,E)$ & Original graph $G$ with vertex set $V$ and edge set $E$ \\ \hline
		\textit{$T_{c}(N^{c},L^{c})$} & The hierarchical community tree generated from graph $G(V,E)$ with node set $N^{c}$ and link set $L^{c}$. Each node  $N^{c}_{k} \in N^{c}$ denotes a group of vertices belonging to $V$. \\\hline
		\textit{$T_{b}(N^{b},L^{b})$} & The binary community tree reconstructed from \textit{$T_{c}(N^{c},L^{c})$}. Each node $N^{b}_{k} \in N^{b}$ denotes a group of vertices belonging to $V$. \\ \hline
		$B$ & The binary codebook for \textit{$T_{b}(N^{b},L^{b})$}. Particularly, $B_{k} \in B$ denotes the binary code of both $N_{k}^{b} \in N^{b}$ and $L_{k}^{b}\in L^{b}$ where $L_{k}^{b}$ is the link points to node $N_{k}^{b}$.
		\\ \hline
	\end{tabular}
	\caption{Commonly used notations in gPCD model}
	\label{tab:notation}
	\vspace{-1em}
\end{table}

For running time analysis, calculating normalized linked weight takes constant time. In each merging step, node pair selection takes linear time. Therefore, in the worst case, the time complexity of binary community tree construction is $O(|V|^2)$ where the depth of the hierarchical community tree $T_{c}(N^{c},L^{c})$ is 1 and each vertex in $G(V,E)$ forms a single-vertex community.

To encode the nodes and links on $T_{b}(N^{b},L^{b})$ as binary code, the root node is encoded as `null' first. For a parent node \textit{$N_{k}^{b}$} with its left child node \textit{$N^{b}_{k_{l}}$} and right child node \textit{$N^{b}_{k_{r}}$}, the binary code of a child node and the related link defined in the Notation Table \ref{tab:notation} is calculated as:

\begin{eqnarray}\text{$B_{k_{i}}$}=
\begin{cases}
\text{$B_{k}$}+``0", & i = ``l"\cr
\text{$B_{k}$}+``1", & i = ``r"
\end{cases}
\end{eqnarray}

For instance, if the node \textit{$N_{k}^{b}$} is with binary code ``$00$," its left child node's binary code is ``$000$" while the right child node's binary code is ``$001$." The link \textit{$L_{k}^{b}$} that points to \textit{$N_{k}^{b}$} also has the binary code ``$00$".

\subsection{Community and User Need Representation} \label{sc:c3_representation}

Node2vec \cite{grover2016node2vec} helps to learn fixed-length embeddings for both user need and communities. It simulates random walks on the graph $G(V,E)$ and learns the vertex embedding by optimizing the sequential relationships from random walk paths. In the end, each vertex $V_{k}$ in graph $G(V,E)$ has a vector representation as $\vec{V_k}$. Each node $N_{k}^{b}$ on the binary community tree \textit{$T_{b}(N^{b},L^{b})$} refers to a vertex community $C_{k}$ in the graph $G(V,E)$. Its representation $\vec{C_{k}}$ is calculated as the averaged embedding of all vertices inside the community. In the end, the binary community tree \textit{$T_{b}(N^{b},L^{b})$} represents the hierarchical community partition of Graph $G(V,E)$. Each node \textit{$N_{k}^{b}$} on the tree is indexed with three attributes: a group of vertices from graph $G(V,E)$, a binary code \textit{$B_{k}$}, and an embedding representation \textit{$\vec{C}_{k}$}.

On the other hand, User need (query) $I$ can also be represented by a combination of $t$ different vertices $\{V_1, V_2... V_t\}$ in the graph $G(V,E)$. In this study, two different scenarios for user need representation are offered:

\textbf{Vertex-based Query}. User need can be directly represented by the vertices based on the generation probability $P(V_k|I)$ between them. Hence the user need representation $\vec{I}$ is calculated as:

\begin{equation}
\vec{I} = \sum_{k=1}^{t} P(V_k|I) \cdot \vec{V_k}
\end{equation}
For instance, in a music sharing network, each vertex $V_k$ denotes a music and a user listing history can be used to reflect the user need $I$. $P(V_k|I)$ therefore can be regarded as the probability that a music being listened by the user.

\textbf{Text-based Query}. Under this scenario, user need $I$ is represented as a text query, and each vertex $V_{k}$ in the graph $G(V,E)$ also contains textual content. From language model viewpoint, each vertex importance weight is the query likelihood $P(I|V_k)$, and the user need can is the weighted average of vertex embedding:

\begin{equation}
\vec{I} = \frac{\sum_{k=1}^{t} P(I|V_k) \cdot \vec{V_k}}{\sum_{k=1}^{t} P(I|V_k)}
\end{equation}

In either case, user need is conceptualized as an embedding with the same dimension as the node embeddings on the binary community tree. It enables very efficient online personalized community detection in later steps. And running Node2vec takes most of the time in this step.

\subsection{Online Genetic Pruning} \label{sc:c3_online}

The whole process, as the Figure \ref{fig:pipeline} shows, is to generate communities by pruning the constructed binary community tree. After each cut on a link, the original tree  will be separated into two sub-trees. After a specific number of cuts to the links on the tree, a fixed number of communities with different resolutions are detected.  By applying genetic selection, crossover, and mutation steps, the model converges to the optimized solution efficiently with a clear-defined fitness function. The details are shown in the following paragraphs.

\subsubsection{Genetic Representation}

A chromosome is formed by a set of genes \{$g_{1},g_{2},...,g_{K-1}$\}, and each gene $g_{i}$ holds a cut link $L_{i}^{b}$ in the binary community tree \text{$T_{b}(N^{b},L^{b})$}. Since communities can be created by cutting links on the offline tree, a chromosome can be represented as a generated community partition of the original graph $G(V,E)$ in this way.
To constrain a chromosome so that it can be decoded to a fixed number of communities, four \textbf{Cutting Rules} are necessarily to be applied:
\begin{itemize} %\itemsep0em
	\item \textbf{ Rule 1}: If a link $L_{i}^{b}$ is picked to cut on the binary community tree \text{$T_{b}(N^{b},L^{b})$}, its pointing node $N_{i}^{b}$ will be retrieved and all the vertices within it form a community.
	\item \textbf{ Rule 2}: If a link {$L_{i}^{b}$} and its ancestor link {$L_{j}^{b}$} are stored in the same chromosome, all vertices in {$L_{i}^{b}$}'s related node {$N_{i}^{b}$} are a subset of vertices in {$L_{j}^{b}$}'s related node {$N_{j}^{b}$}. In this case, the two cut links generate two communities where community $C_{i}$ is all vertices in $ N_{i}^{b}$ and community $C_{j}$ is the remaining vertices in $ N_{j}^{b}$ but not in $ N_{i}^{b}$. It can be formulated as   $C_{i} = \bigcup_{k}\{V_{k}|(V_{k}\in N_{i}^{b})\}$ and community $C_{j} = \bigcup_{k}\{V_{k}|(V_{k}\in N_{j}^{b}) \cap (V_{k}\notin N_{i}^{b})\}$.
	\item \textbf{ Rule 3}: Sibling links can't be stored in the same chromosome, and it is not allowed to store duplicated links in a chromosome.
	\item \textbf{ Rule 4}: The depth's upper bound is set to be $d$, which means all eligible cut links should be located in the first $d$ depth on the binary community tree. It avoids to generate super tiny communities and hugely reduces the genetic searching scope on cut links.
\end{itemize}

By applying the cutting rules to the online pruning process, I ensure a $K$ community partition can be retrieved from a chromosome with $K-1$ cut links.

\subsubsection{Initialization}

Initially, the model generates a given number $P$  chromosomes as the seed ``chromosome population". And each iteration in the genetic approach breeds a new ``generation'' of chromosome population. In order to ensure a chromosome is an encoder of a $K$ community partition, $K-1$ links will be randomly picked (on the binary community tree) following the cutting rules and stored in the related genes of a chromosome.

\subsubsection{Fitness Function}

As each chromosome can be decoded as a community partition, it is important to measure the quality of each generated chromosome (how well the generated communities can satisfy user need). The measurement is hosted in a fitness function.

In the proposed model, the fitness function simulates the user searching behavior on the graph given the community partition. For instance, a user can be more likely to pick the most relevant communities while avoiding the redundant information already selected. With the help of the offline step, the relevance score of node $N_{i}^{b}$ (community $C_{i}$) towards user need $I$ can be calculated with the cosine similarity $cos(\vec{I},\vec{C_{i}})$, and the information redundancy can be $\sum_{C_{j} \in S_c}cos(\vec{C_{j}},\vec{C_{i}})$ where $S_{c}$ is the set of communities that the user have already picked from the communities decoded from the target chromosome. Following this, I use a greedy selection approach to iteratively rank and pick communities given a chromosome (community partition) until all communities are picked:

\begin{equation}  \argmax_{C_{i}}\lambda \cdot cos(\vec{I},\vec{C_{i}})-(1-\lambda)\cdot \frac{\sum_{C_{j} \in S_c}^{ }cos( \vec{C_j},\vec{C_{i}})}{|S_c|}
\end{equation}

where $C_i$ is the candidate community to be picked and $|S_c|$ is the number of communities already been picked. $\lambda$ is a parameter controls  whether user prefers to obtain new useful information or to avoid redundant information.

For chromosome quality evaluation, a query-generated vertex ranking list $l_{q}$ is first created by retrieving top $n$ vertices relevant to the query (user need) with the largest cosine similarities on embeddings of graph $G(V,E)$. I store the top $n$ vertex ranking label $R(l_{q})=\{1,2,...,n\}$ as the pseudo ground truth. On the other hand, given the $k_{th}$ chromosome $ch_k$ in the current chromosome generation, I can also retrieve the community-generated ranking of each vertex $V_{k} \in l_{q}$ from the sequentially selected communities decoded by the chromosome. I assign the ranking label on each vertex $V_{k}$ based on the following formula:
\begin{equation}
\sum_{V_{j} \in l_{q}}\Phi(\delta(V_{j}) < \delta(V_{k})) + 1
\end{equation}

$V_{j}$ refers to all vertices in $l_{q}$. $\delta(V_{j})$ shows the ranking (selection sequence) of the community which $V_{j}$ belongs to. $\Phi$ is a binary operator to determine whether $V_{j}$ satisfy the condition $\delta(V_{j}) < \delta(V_{q})$. This formula helps to construct the community-generated ranking label $R(l_c)$. For instance, when $n=3$, I have a query-generated ranking list $l_q = \{V_1,V_2,V_3\}$ and its related ranking label $R(l_q) = \{1,2,3\}$. Given a chromosome where the community of $V_1$ and $V_2$ is the same and selected before $V_3$, I can generate the related community ranking label $R(l_c) = \{1,1,3\}$ with the same vertex sequence of $l_q$.

Then, I define the fitness function $f(\cdot)$ to evaluate the chromosome $ch_{k}$. As I have the query-generated ranking label $R(l_{q})$ (ground truth) and community-generated ranking label $R(l_{c})$ from $ch_{k}$, I calculate their Kendall's $\tau$  correlation coefficient  as the fitness score $f(ch_k)$ of chromosome $ch_k$ where higher score means the chromosome $ch_{k}$ can  generate better personalized communities to meet with user need.

\begin{equation}
\textit{$f(ch_{k})$} = 1-\frac{\sum_{i=1}^{n} R(l_{ci})\cdot R(l_{qi})}{\sum_{i=1}^{n} R(l_{ci})^2\cdot \sum_{i=1}^{n}R(l_{qi})^2}
\end{equation}
where $R(l_{ci})$ is the $i_{th}$ vertex ranking in community-generated ranking label $R(l_{c})$ and $R(l_{qi})$ is the $i_{th}$ vertex ranking in query-generated ranking label $R(l_{q})$. Kendall's $\tau$ is a widely used metric to evaluate the correlation between two lists where higher score means stronger correlation. Thus, higher fitness score reflects that the generated community ranking $R(l_{c})$ can better meet with user need $R(l_{q})$.

Moreover, it is clear that the fitness function aims to separate all top $n$ vertices in different communities to get the optimal case. It matches the research goal to generate high resolution communities on vertices which are more relevant to user need. As the number of community is a given number $K$, it also leads to a coarser manner partition on the remaining less relevant vertices. On the other hand, the binary community tree $T_{b}(N^{b},L^{b})$ and the Cutting Rule 4 naturally preserve the community structure and unite the most relevant vertices in the same community. Hence the whole genetic approach is a gambling process. The final chromosome result is the equilibrium case to detect communities both contain graph topological structure and meet with user need.

\subsubsection{Selection}

I select the superior chromosomes from current chromosome population based on their fitness scores. The probability that the $k_{th}$ chromosome $ch_k$ is picked can be calculated via the Softmax normalization function $p(ch_{i}) = \frac{exp(f(ch_{k}))}{\sum_{i=1}^{P}exp(f(ch_{i}))}$. Then, the Fitness Proportionate Selection method is applied to randomly select $P$ chromosomes into chromosome pairs based on probability distribution. In order to enhance optimization efficiency, I also use elitism selection to ensure the best chromosome in the current generation will always be selected to the next generation.

\subsubsection{Crossover}

To reach global optimum community partition efficiently, given a pair of chromosomes, the crossover operation can randomly exchange part of the genes in both chromosomes to produce a new pair of chromosomes with a certain crossover rate.

In order to make sure that the newly generated chromosomes meet the cutting rules, an \textbf{Exchange Rule} is defined to restrict gene exchange: If gene \textit{$g$} contains link $L^{b}_{g}$, \textit{$g$} can't do crossover process with genes that contain either link $L^{b}_{g}$ or its sibling link $L^{b'}_{g}$. This rule can help avoid having duplicated links or sibling links stored together in the newly generated chromosome (To satisfy Cutting rule 3).

After \textit{$m$} random numbers are selected from \{$1,2,...,K-1$\} as exchanged gene position indexes, genes located in the chosen positions of two chromosomes will exchange the stored link restricted by the Exchange Rule.
\subsubsection{Mutation}
Mutation operation is applied to avoid local optimization. If a chromosome is chosen to mutate, a gene within the chromosome will be randomly picked, and its stored link will be changed to another link restricted by the Exchange Rules. An example is illustrated in Figure \ref{fig:pipeline}(b) where the link stored in the second gene is changed from ``001'' to ``01''.

\subsubsection{Termination}

After $T$ iterations, the whole process stops and the current best chromosome is retrieved as the final result. Choosing the number of $T$ is dependent on the task. In order to decode the final chromosome to the related community partition, all genes in the chromosome are sorted in an ascending order based on the binary code of their stored cut link. Vertices whose binary codes start with the same cut link's binary code will be assigned to the same community label. And its later assigned community label can overwrite the previous assigned community label. For instance, if there are a vertex with binary code ``0011'' and two cut links with binary code ``00'' and ``001'', the vertex will be assigned to a community label ``00'' first, and its community label is overwritten by ``001'' afterwards. The Termination step in Figure \ref{fig:pipeline}(b) also illustrates a vivid example. In this way, the binary code of the binary community tree can help to decode the final chromosome into communities in an efficient way.

\subsection{Distributed gPCD} \label{sc:c3_distributed}

To enhance the online step efficiency, a MapReduce framework is utilized to enable the distributed genetic evolution. Figure \ref{fig:distributed} depicts the personalized community detection under a MapReduce framework. The chromosome collection is either originally initialized from binary community tree or obtained from the last generation. It contains the whole chromosome population in the central depository. In its first ``Splitter'' process, all chromosomes are split into $M$ groups based on their hash values and sent out to related $M$  Mappers to calculate the ``Fitness'' scores. In the same Mapper, after all chromosomes are assigned fitness scores, based on their scores, a Combiner groups all chromosomes together and random select equal number of chromosomes with duplicated as the ``Selection'' step. All the selected chromosomes are sent to $R$ reducers (I set $R = M$ arbitrarily in order to better represent time complexity) to form pairs for the ``Crossover'' and  ``Mutation'' step, calculate new chromosome offsprings for the next generation and store them back to the central repository.

\begin{figure}
	% \advance\leftskip-4cm
	\center
	\includegraphics[width=1\columnwidth]{img/chapter3/parallel.png}
	%  \vspace{-3em}
	\caption{Online parallel computing process on Hadoop Distributed File System (HDFS)}
	%  \vspace{-1em}
	\label{fig:distributed}
\end{figure}

The complexity of the proposed algorithm is $O(2^dKP)$ without parallel computing and $O(\frac{2^dKP}{M})$ with  parallel computing, where $d$ denotes the upper bound where the cut links are restricted in the top $d$ depth of the binary community tree $T_{b}(N^{b},L^{b})$; $K$ denotes the community number; $P$ denotes the initialized population size of the genetic algorithm and $M$ denotes the number of Mappers/Reducers in parallel environment. As all the parameters are considerably small (compared with the node/edge size in the original graph), the whole process runs very fast to retrieve the final community partition.


\documentclass{memoir}
\usepackage{notestemplate}

%\logo{~/School-Work/Auxiliary-Files/resources/png/logo.png}
%\institute{Rice University}
%\faculty{Faculty of Whatever Sciences}
%\department{Department of Mathematics}
%\title{Class Notes}
%\subtitle{Based on MATH xxx}
%\author{\textit{Author}\\Gabriel \textsc{Gress}}
%\supervisor{Linus \textsc{Torvalds}}
%\context{Well, I was bored...}
%\date{\today}

\begin{document}

% \maketitle

% Notes taken on 03/08/21

\section{Extension of Fields}
\label{sec:extension_of_fields}

\begin{defn}
	If \(K\) is a field containing a subfield \(F\), then \(K\) is said to be an \textbf{extension of \(F\)}, denoted by \(K / F\).\\

	The field \(F\) is sometimes called the \textbf{base field} of the extension.
\end{defn}
Note that if \(K\) is an extension of a field \(F\), then \(K\) is a \(F\)-vector space via the typical \(F\) action.

\begin{defn}
	The \textbf{degree} or \textbf{index} of a field extension \(K / F\), denoted \([K:F]\), is defined to be \(\textrm{dim}_F K\), the dimension of \(K\) as an \(F\)-vector space.
\end{defn}
For example, \([\Q(\sqrt{2} ) : \Q] = 2\) and \([\C: \R] = 2\). One can see the latter example by observing that \(\C\cong \R[x] / (x^2+1)\).

% Recall that we denote by \(F(\theta )\) the field \(F\) adjoined by the root \(\theta \), which is spanned by powers of \(\theta \) as an \(F\)-vector space.
\begin{thm}
	Let \(F\) be a field and \(p(x) \in F[x]\) be an irreducible polynomial. Then \(\exists \) a field extension \(K\) of \(F\) in which \(p(x)\) has a root.
\end{thm}
This field is given by \(K := F[x] / (p(x))\), but we will show this more formally later.

\begin{thm}
	Let \(p(x) \in F[x]\) be an irreducible polynomial of degree over \(F\), and let \(K\) be the field \(F[x] / (p(x))\). Take \(\theta := x + (p(x))\) (root of \(p(x)\) ). Then
	\begin{enumerate}
		\item The elements \(\left\{ 1_F, \theta , \theta ^2, \ldots, \theta ^{n-1} \right\} \) are an \(F\)-vector space basis of the \(F\)-vector space \(K\).
		\item \([K:F] = n\)
		\item \(K = \left\{a_0 + a_1\theta + a_2\theta^2 + \ldots + a_{n-1}\theta ^{n-1} \mid a_0,\ldots,a_{n-1} \in F \right\} \) as an \(F\)-vector space.
	\end{enumerate}
\end{thm}
Another nice example to be familiar with is \(K = \mathbb{F}_2[x] / (x^2+x+1)\). This is a field extension of \(\mathbb{F}_2\) as \(x^2+x+1\) is irreducible in \(\mathbb{F}_2\). We can see that \([\mathbb{F}_2[x] / (x^2+x+1) : \mathbb{F}_2[x]] = 2\) simply because the degree of the polynomial is \(2\), but we can also directly count elements in the set and see that it has twice the elements of \(\mathbb{F}_2[x]\).\\

Now let's define fields formed by adjoining roots more formally.

\begin{defn}
	Let \(K / F\) be a field extension, and let \(\alpha_1,\alpha_2,\ldots \in K\) be elements. The smallest subfield of \(K\) containing both \(F\) and the elements \(\alpha_1,\alpha_2,\ldots,\) denoted \(F(\alpha_1,\alpha_2,\ldots)\) is called the \textbf{field generated by \(\alpha_1, \alpha_2,\ldots\) over \(F\)}.
\end{defn}

\begin{defn}
	The field \(F(\alpha )\) generated by a single element \(\alpha \) over \(F\) is called a \textbf{simple extension of \(F\)}, and the element \(\alpha \) in this case is called \textbf{primitive}.
\end{defn}

\begin{thm}
	Let \(F\) be a field and let \(p(x) \in F[x]\) be an irreducible polynomial. Suppose \(K\) is an extension of \(F\) containing a root \(\alpha \) of \(p(x)\). Then \(F[x] / (p(x)) \cong F(\alpha )\).
\end{thm}
It is natural to view field extensions as the base field appended with roots, and as a result a few definitions arise.

\begin{defn}[Algebraic and Transcendental Elements]
	An element \(\alpha \in K\) is called \textbf{algebraic over \(F\)} if \(\alpha \) is a root of some nonzero polynomial \(f(x) \in F[x]\).\\

	If \(\alpha  \in K\) is not algebraic over \(F\), then we say that \(\alpha \) is \textbf{transcendental over \(F\)}.
\end{defn}
The extension \(K / F\) is \textbf{algebraic over \(F\)} if all elements of \(K\) are algebraic over \(F\).

\begin{exmp}[Examples of Algebraic and Transcendental Elements]
	\begin{itemize}
		\item \(\sqrt{2} \) is an algebraic element over \(\Q\) via the polynomial \(x^2-2\). This actually holds for all \(\sqrt[n]{2} \) with \(x^{n}-2\).
		\item \(i\) is algebraic over \(\R\) and \(\Q\) via the polynomial \(x^2+1\)
		\item Transcendental elements are much rarer-- examples include \(\pi \) and \(e\), but it is non-trivial to show an element is transcendental.
	\end{itemize}
\end{exmp}
\end{document}

Abstract Algebra - Introductory/Algebra II/Notes/source/Lecture15 - IntroFieldExtn.tex


% !TEX root = ../zeth-protocol-specification.tex

\section{Instantiating $\mkhash$}\label{instantiation:mkhash}

In this section we describe the instantiation of $\mkhash$ with a compression function based on \mimc{}~\cite{albrecht2016mimc}. We firstly show how the compression function is constructed, and prove that this instantiation complies with the security requirements mentioned in~\cref{zeth-protocol:sec-req}

\subsection{\mimc{} Encryption}\label{instantiation:mkhash:mimc-encryption}

\mimc{} is a block cipher with a simple design, consisting of a number of rounds (denoted $\rounds$). During the $i$-th round, the message $\msg$ is mixed with the encryption key $\key{}$ and a randomly chosen constant $c[i]$, and a permutation function is applied to generate a new value of $\msg$. The permutation function consists of exponentiation with a carefully chosen exponent $\exponent{}$ (see~\cref{instantiation:mkhash:mimc-encryption:security}). Note that \rounds{} depends on the desired security level $\secpar$. We denote the encryption function by \mimcEnc{} and illustrate it in~\cref{instantiation:fig:mimc}.

\begin{figure*}[ht]
    \centering
    \procedure[linenumbering]{$\mimcEnc(\key, \msg, c, \exponent{}, \rounds)$}{
        \pcforeach\ i \in [\rounds]:\\
        \t \msg \gets {(\key\ \mathsf{OP}\ c[i]\ \mathsf{OP}\ \msg)}^\exponent{}\\
        \pcreturn (\msg\ \mathsf{OP}\ \key)
    }
    \caption{\mimc{} Encryption function.}\label{instantiation:fig:mimc}
\end{figure*}

\mimcEnc{} can be defined on both binary and prime fields, and as such the $\mathsf{OP}$ operation corresponds to either $\oplus$ or $+ \pmod{p}$~\cite{albrecht2016mimc, grassi2016mpc}.
For general prime $p$ (resp.~positive integer $n$), we denote by $\mimcPrime{p}$ (resp.~$\mimc{}_{2^n}$) the \mimcEnc{} function defined over \FFx{p} (resp.~\FFx{2^n}).

\subsubsection{Security parameters and analysis}\label{instantiation:mkhash:mimc-encryption:security}

In this document, we only consider \mimc{} defined over prime fields (in particular, the field $\FFx{\rCURVE}$ over which \zksnark~operates).

Since block ciphers are usually defined over the product space of keys and messages, we consider the variables $c$, \rounds{} and \exponent{} as fixed. We thereby consider an instantiation of $\mimc{}$ with signature
\begin{align*}
    \mimcPrime{\rCURVE} &: \FFx{\rCURVE} \times \FFx{\rCURVE} \to \FFx{\rCURVE}
\end{align*}

In the sections below, and as in ~\cite{albrecht2016mimc}, we will consider exponents of the form $\exponent{} = 2^t-1$ and $\exponent{} = 2^t+1$ where $\gcd(\exponent{}, \rCURVE-1) = 1$. We note that the term cancellation happening with exponents of the form $\exponent{} = 2^t+1$ does not immediatelly translate to the context where $\mimc{}$ is carried out over prime fields of large odd characteristic. In fact, in the case of $\FFx{\rCURVE}$, where $\rCURVE > \binom{\exponent}{\lfloor \exponent / 2 \rfloor}$, polynomials $(x + y)^{\exponent}$ are not sparse.
This comes from the \emph{Binomial Theorem}
\[
  (x + y)^\exponent = \sum_{i = 0}^{\exponent} \binom{\exponent}{i} x^{i}y^{\exponent - i}
\]
and the observation that if $\binom{\exponent}{i} < \rCURVE$ then $\binom{\exponent}{i} \bmod \rCURVE = \binom{\exponent}{i}$, hence ensuring that all the polynomial coefficients are greater than $0$, and that the polynomial is dense.

To achieve a security of $\secpar$, we require that $\rounds \geq \secpar \log_{\exponent}(2)$.
Importantly, since we use $\mimc{}$ over prime fields $\FFx{\rCURVE}$ which are large (where $\rCURVE$ is the prime characteristic of the scalar field of an elliptic curve group, such that $\ceil{\log_2(\rCURVE)} > \secpar$\footnote{Longer elements are needed in Elliptic Curve Cryptography (\ecc) to resist algebraic attacks such as Number Field Sieve (\acrnfs)-based attacks on discrete logs~\cite{DBLP:journals/siamdm/Gordon93} for instance}), then, picking $\rounds = \left\lceil \frac{\log_2 \rCURVE}{\log_2 \exponent{}} \right\rceil > \secpar \log_{\exponent}(2)$ provides a margin of safety on the number of rounds selected to instantiate $\mimc{}$ with desired security level $\secpar$.

We refer to the $\mimc{}$ paper~\cite[Section 4.2 and 5.1]{albrecht2016mimc} and to~\cref{appendix:mimc-security} for more details on the security analysis and attacks on the scheme in the different settings. Note that $\mimcPrime{\rCURVE}$ does not suffer from \emph{inversion subfield attacks} as there are no proper subfields of $\FFx{\rCURVE}$.

\subsection{\mimc{}-based compression function}\label{instantiation:mkhash:mimc-compressionf}

There exist two main techniques to construct a hash function from a block-cipher (or permutation): sponge functions~\cite{bertoni2007sponge} and iterated compression functions~\cite{black2002black}.

A Merkle tree is a binary tree of values of fixed size, where the values in each ``layer'' are generated by hashing pairs of values from the previous ``layer''. That is, we require a compression function $\mkhash$, which we construct via the Miyaguchi-Preneel scheme. (Miyaguchi-Preneel is more secure~\cite[$f_5$ function]{black2002black} than the more flexible Davies-Meyer construct~\cite[Section 3]{gazzoni2006maelstrom}, but this flexibility is not required in our case).

\subsubsection{Miyaguchi-Preneel compression construct}

Miyaguchi-Preneel (MP)~\cite[$f_3$ function]{black2002black} is a general scheme for constructing compression functions from block ciphers (see~\cref{preliminaries:definitions:hashcomp}). Given a block cipher \Enc, the corresponding compression function by \fMP{} is given in~\cref{instantiation:fig:mp-constructions}. The original construction is defined over binary fields, however \zeth~operates over prime fields. Hence, in the general discussion here we replace the bitwise addition operator $\oplus$ by modular addition in $\FFx{\rCURVE}$ (see~\cite{mp-security-ethsnarks}).

We denote by \mimcMP{} the compression function defined by the application of the Miyaguchi-Preneel construct over \mimc{}. Similarly, for general prime $p$ we denote by $\mimcMPPrime{p}$ (see~\cref{instantiation:fig:mimc-mp-constructions}) the compression function defined by application of the Miyaguchi-Preneel construct over $\mimcPrime{p}$.

\begin{figure*}[ht]
    \begin{minipage}[t]{0.50\textwidth}
        \procedure[linenumbering]{$\fMP{} (\key, \msg)$}{
            res \gets \Enc_\key(\msg) \\
            \pcreturn (res + \msg + \key) \pmod{\rCURVE}
        }
        \caption{\MP{} construct in $\FFx{\rCURVE}$.}\label{instantiation:fig:mp-constructions}
    \end{minipage}%
    \begin{minipage}[t]{0.50\textwidth}
        \procedure[linenumbering]{$\mimcMPPrime{\rCURVE}(\key, \msg)$}{
            res \gets \mimcPrime{\rCURVE}(\key, \msg) \\
            \pcreturn (res + \key + \msg) \pmod{\rCURVE}
        }
    \caption{$\mimcMPPrime{\rCURVE}$ construction.}\label{instantiation:fig:mimc-mp-constructions}
    \end{minipage}%
\end{figure*}

\subsection{An efficient instantiation of \mimc{} primitives}\label{instantiation:mkhash:efficient-instance}

To select appropriate instances of $\mimcPrime{\rCURVE}$ and $\mimcMPPrime{\rCURVE}$, we consider the cost (in terms of gas consumption and prover efficiency). For given $\exponent{}$ and $\rounds{}$, the final definition of $\mimcMPPrime{\rCURVE}$ is given in \cref{instantiation:fig:mimcp-construction} and \cref{instantiation:fig:mimcp-mp-construction}.

\newcommand{\initRoundConstants}{\algostyle{InitRoundConstants}}

\begin{figure*}[ht]
    \begin{minipage}[t]{0.5\textwidth}
        \centering
        \procedure[linenumbering]{$\mimcPrime{\rCURVE}(\key, \msg)$}{
            c \gets \initRoundConstants() \\
            \pcforeach i \in [\rounds]:\\
            \t \msg \gets {(\key + c[i] + \msg)}^\exponent{} \pmod{\rCURVE}\\
            \pcreturn (\msg + \key) \pmod{\rCURVE}
        }
    \end{minipage}%
    \begin{minipage}[t]{0.5\textwidth}
        \centering
        \procedure{$\initRoundConstants()$}{
            iv \gets \keccak{256} (\text{``clearmatics\_mt\_seed''}) \\
            c[0] \gets 0 \\
            c[1] \gets \keccak{256} (iv) \\
            \pcforeach i \in \range{2}{\rounds}:\\
            \t  c[i] \gets \keccak{256} (c[i-1])\\
            \pcreturn c = (c[0], \ldots, c[\rounds-1])
        }
    \end{minipage}%
    \caption{$\mimcPrime{\rCURVE}$ full construction}\label{instantiation:fig:mimcp-construction}
\end{figure*}

\begin{figure*}[ht]
    \centering
    \begin{minipage}[t]{0.5\textwidth}
        \procedure{$\mimcMPPrime{\rCURVE}(\key, \msg)$}{
            \pcreturn \mimcPrime{\rCURVE}(\key, \msg) + \msg + \key \pmod{\rCURVE}
        }
    \caption{\mimcMPPrime{\rCURVE} full construction}\label{instantiation:fig:mimcp-mp-construction}
    \end{minipage}%
\end{figure*}

\begin{remark}
    Note that \keccak{256} is the $256$-bit digest instance of the \keccak{} family that won the NIST SHA-3 competition~\cite{keccak-submission}. It is supported by the \evm via an opcode (see~\cite[Appendix G]{wood2014ethereum}), making it convenient for use in smart contracts.
\end{remark}

\begin{remark}
    To increase the security of the $\mathsf{\mkhash}$, different round constants for each level of the Merkle tree could be used.
\end{remark}

We define $\mkhash$ to be $\mimcMPPrime{}$ over $\FFx{\rCURVE}$. Thereby, for input values $m_0$ and $m_1$, $\mkhash : \FFx{\rCURVE} \times \FFx{\rCURVE} \to \FFx{\rCURVE}$ is defined by
\begin{equation}\label{instantiation:eq:mkhash-instantiation}
    \mkhash(\msg_0, \msg_1) = \mimcMPPrime{\rCURVE}(m_0, m_1)
\end{equation}

\newcommand{\constraints}{\varstyle{constraints}}

For specific values of $\rCURVE$ (such as $\rBN$ for $\BNCurve$ or $\rBLS$ for $\BLSCurve$), it remains to select concrete values of $\exponent{}$ and rounds, where $\rounds = \lceil \frac{\log_2 \rCURVE}{\log_2 \exponent{}} \rceil$).  These values influence the number of constraints in the arithmetic circuit (see~\cref{zeth-protocol:statement} for details of the statement) and the gas cost of Merkle tree operations on the contract (see~\cref{zeth-protocol:process-tx} for details of the specific operations).

In the arithmetic circuit, an invocation of $\mimcMPPrime{}$ requires $\rounds \cdot \mults$ constraints, where $\mults$ is the number of multiplications required for exponentiation. For exponents of the form $\exponent = 2^t - 1$, we have $\mults = 2 \cdot t - 2$, (using the \emph{square-and-multiply} algorithm~\cite{menezes1996handbook}), and for $\exponent = 2^t + 1$ we have $mults = t + 1$. Thus we expect that exponents of the latter form are more optimal. The implementation in the contract performs a very similar set of arithmetic operations (exponentiation in the field through a series of multiplications and modulo reductions), and so the cost is dominated by the same number $\rounds \cdot \mults$ as for the circuit.  Hence, values of $\exponent$ and $\rounds$ that are optimal for the circuit will also result in gas-efficient implementations in the contract.

For several concrete values of $\exponent{}$, the number of $\rounds$ required to attain the desired security level, along with the number of constraints, are shown in~\cref{table:mimc-exp-analysis}.

\definecolor{Gray}{gray}{0.9}
\begin{table}
  \centering
    \begin{minipage}[t]{0.50\textwidth}
        \centering
        \begin{tabular}{r c c c c}
            \toprule
            \multirow{2}{*}{$\exponent{}$} & \multicolumn{2}{c}{\BNCurve} & \multicolumn{2}{c}{\BLSCurve} \\ [0.5ex]
            & $\rounds$ & $\constraints$ & $\rounds$ & $\constraints$ \\ [0.5ex]
            \midrule
            \rowcolor{Gray}
            5 & 110 & 331 & & \\
            7 & 91 & 365 & & \\
            \rowcolor{Gray}
            17 & 65 & 316 & 62 & 311 \\
            31 & 52 & 417 & 51 & 409 \\
            127 & 37 & 445 & 37 & 445 \\
            \rowcolor{Gray}
            257 & 32 & 289 & 32 & 289 \\
            511 & 29 & 465 & & \\
            2047 & 24 & 481 & 23 & 461 \\
            8191 & 20 & 481 & 20 & 481 \\
            32676 & 17 & 477 & & \\
            \rowcolor{Gray}
            65537 & 16 & 273 & 16 & 273 \\
            131071 & 15 & 481 & 15 & 481 \\
            524287 & 14 & 505 & 14 & 505 \\
            \rowcolor{Gray}
            1048577 & 13 & 274 & 13 & 274 \\
            2097151 & 13 & 521 & & \\
            \bottomrule
        \end{tabular}
    \end{minipage}%
    \caption{Arithmetic constraints required to represent \mimcMP{} as an R1CS program, for different exponents $\exponent{}$ and curves. Grey (resp.~white) lines represent exponents of shape $2^t + 1$ (resp.~$2^t - 1$). Missing entries where $\gcd(\exponent{},\rCURVE - 1) \neq 1$}\label{table:mimc-exp-analysis}
\end{table}

For the case of \BNCurve~we set $\exponent{} = 17$ with $\rounds{} = 65$, to achieve a $254$-bit security level. For \BLSCurve~we set $\exponent{} = 17$ with $\rounds{} = 62$, to achieve $253$-bit security. These values are chosen such that they satisfy the requirement that $\gcd(\exponent{},\rCURVE - 1) = 1$, and give a balance between the number of constraints in the arithmetic circuit and the gas cost of hashing on the contract.

\subsection{Security requirements satisfaction}\label{instantiation:mkhash:security}

After presenting the state of the art of MiMC cryptanalysis, we present the security proof of \mimcMPPrime{} collision resistance.

\subsubsection{Cryptanalysis of $\mimc{}$ block cipher and primitives}\label{instantiation:mkhash:security:cryptanalysis}

\mimc{}'s security is increasingly being analysed since the primitive has gained traction in zero-knowledge and cryptocurrency communities for its succinct algebraic constraint representation. As of today, we do not know of any attacks breaking \mimc{} on prime fields on full rounds.

The first attack on \mimc{} was an interpolation attack~\cite{li2019improved} which targets a reduced-round version for a scenario in which the attacker has only limited memory.
An attack on Feistel-based \mimc{}~\cite{bonnetain2019collisions} was discovered shortly after, by using generic properties of the used Feistel construction (instead of exploiting properties of the primitive itself).
Additionally,~\cite{albrecht2019algebraic} proposes an attack based on Gr\"{o}bner basis. The authors state that by introducing a new intermediate variable in each round, the resulting multivariate system of equations is a Gr\"{o}bner basis. As such, the first step of a Gr\"{o}bner basis attack can be obtained for free. However, the following steps of the attack are so computationally demanding that the attack becomes infeasible in practice.
A recent work~\cite{DBLP:conf/asiacrypt/Eichlseder0LORS20} targets \mimc{} on binary fields, and achieves a full-round break of the scheme. While, the attack presented does not apply to prime fields, the authors note that it ``can be generalized to include ciphers over $\FFx{p}$'', and that only the lack of efficient distinguishers over prime fields precludes this.
Another attack from Beyne et al~\cite{cryptoeprint:2020:188} uses a low complexity distinguisher against full \mimc{} permutation leading to a practical collision attack on reduced round sponge-based \mimc{} hash defined with security of 128 bits.

\subsubsection{Security proof of \mimcMPPrime{} collision resistance}\label{instantiation:mkhash:security:colres-proof}
We now prove that this compression scheme satisfies all the security requirements listed in~\cref{zeth-protocol:sec-req}. To do so, we first assume that the round constants are pseudo-random, i.e.~that $\keccak{256}$ is a \prf{}.

\begin{lemma}
	\keccak{256} is a $\prf$ with $\lambda=128$.
\end{lemma}

The security of \mimcMPPrime{} derives from a more general result, i.e.~from modelling \mimcPrime{} as an ideal cipher (see~\cref{preliminaries:def:ICM}). More specifically, we show a security result for the \MP{} construction on \FFx{\rCURVE} by proving that, in the Ideal Cipher Model, the collision resistance advantage of any adversary is bounded by $\frac{q(q+1)}{\rCURVE}$, where $q$ is the number of different queries that the attacker makes to the oracle. This means that, assuming a maximum $q$ number of possible encryption/decryption queries, parameter $\rCURVE$ can be chosen to make the advantage small as needed and $\fMP$ considered collision resistant. Similar result applies to the ${2^n}$ case.

The instance of \mimc{} we use is modelled as an ideal cipher defined on field elements, for this reason we consider a variant of the ICM model where the keys, inputs and outputs are field elements in $\FFx{\rCURVE}$ and the block cipher scheme, with key $\key$, correspond to a family of $\rCURVE$ independent random permutations $f_{\key}: \FFx{\rCURVE} \times \FFx{\rCURVE} \to \FFx{\rCURVE}$.

In the proof, without loss of generality, we assume the following conventions for an adversary \adv{}:
\begin{itemize}
    \item the adversary asks distinct queries: i.e.~if \adv{} asks a query $\oracleEnc(\key,\msg)$ and this returns $y$, then \adv{} does not ask a subsequent query of $\oracleEnc(\key,\msg)$ or $\oracleDec(\key,y)$, and inversely;
    \item the adversary necessarily obtained the candidate collision from the oracle. This property follows suite from modelling \mimc{} as an ideal cipher.
\end{itemize}

\begin{lemma}\label{lemma:colrescomp}
    Let \fMP{} be the \MP{} compression function built on an ideal block-cipher \Enc{} on \FFx{\rCURVE}, the probability for an adversary \adv{} to find a collision is not greater than $q(q+1)/\rCURVE$ where $q$ is a (positive) number of distinct oracle queries.
\end{lemma}

The following proof has been adapted from~\cite[Lemma 3.3]{black2002black}\footnote{It states the collision resistance of a set of compression functions $f_1,,\ldots,,f_{12}$, denoted as \emph{group-1 compression functions} and showed in~\cite[Figure 3]{black2002black}. As mentioned above, Miyaguchi-Preneel corresponds to $f_3$ of that group. Since the proof of~\cite[Lemma 3.3]{black2002black} shows collision resistance of $f_1$, we slightly modified it to work for $f_3$.}.

\begin{proof}
    Fix $h_0\in\FFx{\rCURVE}$. Let \adv{} be an adversary attacking the compression function \fMP{}.
    Assume that \adv{} asks the oracles \oracleEnc{} and \oracleDec{} a total of \emph{distinct} $q$ queries. Let us denote the result of the $q$ queries and output of the attacker (candidate collision) as $\left ( (\key_1, \msg_1, y_1), \ldots , (\key_q, \msg_q, y_q), \text{out} \right )$.
    If \adv{} is successful it means that it outputs $(\key, \msg)$, $(\key', \msg')$ such that either $(\key, \msg) \neq (\key', \msg')$ and $\fMP(\key, \msg) = \fMP(\key', \msg')$ or $\fMP(\key, \msg) = h_0$.
    By the definition of \fMP, we have that $\Enc_\key(\msg) + \msg + \key = \Enc_{\key'}(\msg') + \msg' + \key'$ for the first case, or $\Enc_\key(\msg) + \msg + \key = h_0$ for the second.
    So either there are distinct $r, s \in [1,\ldots, q]$ such that $(\key_r, \msg_r, y_r) = (\key, \msg, \Enc_\key(\msg))$ and $(\key_s,\msg_s, y_s) = (\key',\msg', \Enc_{\key'}(\msg'))$ and $\Enc_{\key_r}(\msg_r) + \msg_r + \key_r = \Enc_{\key_s}(\msg_s) + \msg_s + \key_s$ or else there is an $r \in [1,\ldots, q]\ \suchthat\ (\key_r, \msg_r, y_r) = (\key, \msg, h_0)$ and $\Enc_{\key_r}(\msg_r) + \msg_r + \key_r = h_0$. We show that this event is unlikely.

    In fact, for each $i \in [1,\ldots, q]$, let $C_i$ be the event that either $y_i + \msg_i + \key_i = h_0$ or does exist $j \in [1,\ldots, i-1]\ \suchthat\ y_i + \msg_i + \key_i = y_j + \msg_j + \key_j$. When carrying out the simulation $y_i$ or $\msg_i$ was randomly selected from a set of at least $\rCURVE - (i-1)$ elements, so $\prob{C_i}\leq i / (\rCURVE-i)$. This means that for the collision advantage of \adv{}, \advCollMP it holds that $\advCollMP \leq \prob{C_1 \lor \cdots \lor C_q} \leq \sum_{i=1}^{q} \prob{C_i}$. For $q \leq \frac{\rCURVE}{2}$ this probability is bounded by $l \cdot \frac{q(q+1)}{\rCURVE}$. However, we allow only a polynomial number of queries, thus for $q = \poly$ this probability becomes $\frac{\poly}{\rCURVE}$, where $\rCURVE \approx 2^\secpar$.
\end{proof}

\begin{notebox}
   \cref{lemma:colrescomp} is applicable to our case by the strong assumption of \mimcPrime{\rCURVE} being an ideal cipher. In other words, the proof does not take into account any structural weakness or knowledge that an attacker is aware of. Any such additional information could make~\cref{lemma:colrescomp} invalid, and consequently could be used to break the collision resistance.
\end{notebox}

\begin{remark}
    Note that from~\cref{lemma:colrescomp} follows that the collision resistance security of the \zeth{} Merkle tree is $\log_2(\rCURVE/2)$ (around $127$ bits for $\rCURVE = \rBN$ or $\rBLS$).
\end{remark}

\begin{notebox}
    $\mimc{}$ has \emph{not} received as much cryptanalytic scrutiny as other ``older'' and more established hash functions. This is important to note since, for these type of primitives which are not provably secure, the amount of attacks received by a scheme is a great indicator of its security and robustness.
    A natural alternative to $\mimc{}$ here consists in using Pedersen hash which is provably collision resistant under the discrete-logarithm assumption.
\end{notebox}


\documentclass[12pt,a4]{article}

\newcommand{\handoutdate}{Friday, 2020-04-17}
\newcommand{\firstduedate}{Friday, 2019-04-24}
\newcommand{\finalduedate}{Tuesday, 2020-05-01}

\input{preamble}

\setcounter{section}{3}

\section{Bottleneck Paths}

Let $G=(V,E)$ be a directed graph with an edge capacity function $c: E \rightarrow \R^+$. For a path
$p = u_0 u_1 \dots u_t$ define its {\em capacity} to be
\begin{align}
   c(p) := \min_{1 \leq i \leq t} c( \{u_{i-1}, u_i\}) \ .
\end{align}

\begin{quotation}
    \textbf{Maximum Capacity Path Problem (MCP).} Given a directed graph $G = (V,E)$, an edge capacity function
    $c: E \rightarrow \R^+$, and two vertices $s, t \in V$, compute the path $p^*$ maximizing $c(p)$. We
    denote by $p^*$ the optimal path and by $c^* := c(p^*)$ its cost.
\end{quotation}

\begin{exercise}
   Suppose the edges $e_1,\dots,e_m$ are sorted by their cost. Show how to solve MCP in time $O(n+m)$.
\end{exercise}

\begin{proof}
Design a algorithm following Pseudocode shows(Suppose the edges are sorted decreased, See in Algorithm 1):

And then we think about the correctness and complexity.

The algorithm means we can enum the answer. When we find a edge between the point visited and not visited, we can go through all the edges which's costs is higher than this edge. If now s is connected to t, means this edge is the largest edge while going through all edges higher than it from $s$ to $t$. That fits the answer we want.

Now, let's think about the complexity. For every node, it may be in queue at least once. And for every edge, it may be in $G'$ and used in bfs at least once. So the time complexity is $O(n+m)$.
\begin{algorithm}[H]
\caption{Solve MCP in time $\Theta(n+m)$ with all the edges sorted by their cost}
\begin{algorithmic}
\Procedure{MCP}{$G, s, t$}
	\State Initiate $visited \gets s$ and $G' \leftarrow \emptyset$
	\For {$e \in G$}
		\If{$e.from\in visited$ and $e.to\notin visited$}
			\State $bfs\_graph(e.to, G')$
		\Else
			\State $G'\gets G' \cup {e}$
		\EndIf
		\If{$t \in visited$}
			\State \Return $e.weight$
		\EndIf
	\EndFor
\EndProcedure
\Procedure{bfs\_graph}{$s, G$}
	\State $visited \gets visited \cup {s}$
	\State push s into the queue
	\While {queue is not empty}
		\State Let top be the head of the queue and pop the head.
		\For{$e \in {edges\ in\ G\ from\ top}$}
			\If {$e.to \notin visited$}
				\State $visited \gets visitied \cup{e.to}$
				\State push e.to into the queue
			\EndIf
		\EndFor
	\EndWhile
\EndProcedure
\end{algorithmic}
\end{algorithm}
\end{proof}

\begin{exercise}
   Give an algorithm for MCP of running time $O(m \log \log m)$. \textbf{Hint:} Using the median-of-medians algorithm,
   you can determine an edge $e$ such that at most $m/2$ edges are cheaper than $e$ and at most $m/2$ edges are
   more expensive than $e$. Can you determine, in time $O(n+m)$, whether $c^* < c(e)$, $c^* = c(e)$, or $c^* > c(e)$?
   Iterate to shrink the set of possible
    values for $c^*$ to $m/4$, $m/8$, and so on.
\end{exercise}
\begin{proof}
	As is shown by the hint, during each iteration, we can shrink the
	set of possible values of $c^{*}$, i.e.
	\[
E_1= \{e \mid c\left( e \right) \le M,e\in E'\},
		E_2 = \{e \mid c\left( e \right) > M, e\in E'\}
		\]
	We we can find a path in  $E_2$, then the lower bound $L$ of $c^{*}$ can
	be updated to $M$. Since if $c^{*}>L$, we must have a
	path $e$ with all the edges larger then $L$.

	Otherwise, the upper bound $U$ will be $M$ as there is no such path with
	all edges larger than $M$. However, if we take iterations until
	$L = U$, we will have $O\left( m\log m  \right)  $ running time. So we only do
	$ \log(s\left( m \right) )$
	iterations. Here $s$ is a place holder to decide later

	Consider the following algorithm
	\begin{algorithm}[H]
		\caption{Solve MCP in time $n\log\log n$ }
		\begin{algorithmic}
				\State $i=0, U = \infty, L = 0$
				\While {$i < \log s\left( m \right) $ }
				\State Determine the median of $\{e \mid e\in E',c\left( e \right) \le U\} $.
				\If{ $\left( V,E_2 \right) $ is $s$- $t$ connected }
				\State $E' \leftarrow E_2, L = M$,
				\Else
				\State $U = M$
				\EndIf
				\State $i=i+1$
				\EndWhile
				\State Number $t$ edges in set $\{e \in E' \mid c\left( e \right) \le U\} $ according to increasing order $e_1,e_2,\ldots.$
				\State Solve instance by \textbf{Algorithm 1} with the following
				ordering, i.e. the place of $e$ in the sorted array
				\[
						l\left( e \right) = \begin{cases}
								1, \quad c\left( e \right) \le L \\
								i, \quad  \exists i,e =e_{i} \\
								t, \quad c \left( e \right) > U
						\end{cases}
				.\]
		\end{algorithmic}
\end{algorithm}
Now we prove this algotihm is $n\log \log n$, just notice that
\[
		t = O\left( \frac{m}{s\left( m \right) } \right)
.\]
Since every iteration we make the set $\{ e  \mid  e\in E', c\left( e \right)
< U\} $ shrinks to $\frac{1}{2}$. That is
the size $t = \lvert  E'\rvert \le \frac{m}{2^{\log s\left( m \right) }}$.

Thus the running time is
\[
\log\left( s\left( m \right) \right)\cdot m + t \log t + t
\]
considering the sorting of the
$t$ edges plus the running time of \textbf{Algorithm 1} ,
now we choose $s\left( m \right) = \log m $ to minimize the time which is
\[
		m\left( \frac{\log m}{s} - \frac{\log s}{s} \right)
.\]
Hence
\[
		T =O\left( m \log\log m \right) + O \left( \frac{m}{\log m} \log \left( \frac{m}{\log m } \right) \right) +
		O \left( \log m \right)
		= O\left( m\log\log m  \right)
.\]
\end{proof}

\begin{exercise}
   Give an algorithm for MCP that runs in time $O(m \log \log \log m)$? How about $O(m \log \log \log \log m)$? How far can you get?
\end{exercise}
\begin{proof}
		We can have a better algorithm runs in $O\left( m \log\ldots\log m \right) $ (arbitrary number of log),

		Consider a set of edge $E$ with $\frac{m}{k}$ elements. We apply  \textbf{median of median} algorithm
		$k$ rounds. Each round we break one block into 2 parts. For example, in the first round, we find the median
		 $M$ of $E$, and break $E$ into
		\[
				E_1=\{e \mid c\left( e \right) < M, e\in E\} , E_2 = \{e  \mid c \left( e \right) \ge M, e\in E\}
		.\]
		And in the second round, we can obtain 4 blocks by apply MOM to $E_1,E_2$ respectively. Finally we have $2^{k}$ blocks $A_1,\ldots,A_{2^{k}}$ by using the above method
		recursively, while
		each block has $\frac{m}{k 2^{k}}$ elements. Plus by the property of median
		\[
				\forall i<j, \forall e_1 \in A_i, e_2\in A_j, c\left( e_1 \right) < c\left( e_2 \right)
		.\]
		Hence we can apply the idea we used in \textbf{Exercise 2}. That is,

	\begin{algorithm}[H]
		\caption{The key idea to solve MCP in time $n\log\ldots\log n$ }
		\begin{algorithmic}
				\State $E' = \bigcup_{i\le 2^{k}} A_{i}, L = 0, U = \max\left( E \right) $
				\For{$i$ in range$(2^{k})$}
				\If{ $\left( V,A_i \right) $ is $s$- $t$ connected}
				\State $E' \leftarrow A_i$, $L = \min \left( A_i \right) $
				\Else
				\State $U = \min \left( A_i \right) $
				\EndIf
				\EndFor
		\end{algorithmic}
\end{algorithm}
Hence our problem is reduced to a edge set $E'$ with whose size is $O\left( \frac{m}{k 2^{k}} \right) $, we
call this one  iteration. And this
costs
 \[
		 T = \sum_{i=1}^{k} \sum_{j=1}^{i} O\left( \frac{m}{k 2^{i}} \right) = k O\left( \frac{m}{k} \right) = O\left( m \right)
.\]
And let $f\left( k \right)  = k 2^{k}$, assume that after $r$ iterations, we can decide $c^{*}$ i.e. there is only one integer in the interval $E'$, we have
\[
		f^{r-1}\left( 1 \right) <  m \le  f^{r} \left( 1 \right)
.\]
Which leads that $m \ge g^{r-1}\left( 1 \right)  $, in which $g\left( x \right) = 2^{x}$. Hence $r$ is the
smallest integer such that
$1 \le  \log \log\ldots\log m$ with $r$ log, that means we only need
$r$ iterations to reduce the problem,  hence the overall running time is
$O\left( r m  \right)  $. Actually we can prove $\forall s$, $\exists m$
\[
		r < \log\log\ldots\log m \left( \text{with } s \log \right)
.\]
Otherwise $1 < \log\ldots\log r $ with $r-s$ log. Hence  $r >f^{r-s} \left( 1 \right) $, which
leads to a contradiction since $m$ and  $r$ can be arbitrarily large but $s$ is fixed.
Hence the running time could be written as $O\left( m \log \log \ldots\log m \right) $ (arbitrary number of log).
\end{proof}

\end{document}


\documentclass[11pt]{article}
\setlength{\textwidth}{7in} \setlength{\textheight}{9.8in}
\setlength{\topmargin}{-0.8in} \setlength{\oddsidemargin}{-0.25in}
\setlength{\evensidemargin}{-0.25in}

\usepackage{graphicx}
\usepackage{epstopdf}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}
\usepackage{color}
\usepackage{hyperref}
\usepackage{amssymb}
\usepackage{amsmath}
%\usepackage{algpseudocode}

\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\rhead{Homework2  Page \thepage}
\lhead{Shen Qu}
\chead{STAT 671}

\DeclareMathOperator{\trace}{trace}

\begin{document}

%\includegraphics{../../../psulogo_horiz_bw.eps}\hfill\includegraphics{../../../deptlogo}
\noindent
This homework is an exan that was given during a previous iteration of this course. I have voluntarily kept the presentation as it was for the exam.
\noindent
\noindent

\section{Multivariate Fisher Kernel}
The Fisher kernel is a kernel for data in $\mathbb{R}^d$ which is based on a probabilistic model. Specifically, let $x \in \mathbb{R}^d$, and let $p_{\theta}$ be a probability density function indexed by a parameter vector $\theta \in \mathbb{R}^k$.  First, fix some $\theta_0 \in \mathbb{R}^k$ and define the Fisher score of a data point $x$, by $\phi(x)$, the gradient of the logarithm of $p_{\theta}$ evaluated at $\theta=\theta_0$. That is,
\begin{equation}
\phi(x)=\nabla_\theta \ln p_{\theta}(x) |_{\theta=\theta_0}
\end{equation}
Thus $\phi: \mathbb{R}^d \to \mathbb{R}^k$.

http://yongyuan.name/blog/fim-fisher-kernel.html

https://wiseodd.github.io/techblog/2018/03/11/fisher-information/

Next, define the Fisher information matrix as the following expected value:
\begin{equation}
I = E_{p_{\theta_0}}[\phi(X)\phi^T(X)]
\end{equation}
Note that $I$ is a $(k,k)$ matrix. You can use the fact that this matrix is symmetric and positive definite.
Finally, define the Fisher kernel as
\begin{equation}
K(x,y)=\phi(x)^T I^{-1} \phi(y)
\end{equation}
for any couple of points $(x,y)$, both in $\mathbb{R}^d$
\begin{enumerate}
\item Verify that $K$ is symmetric

\begin{align*}
E_{p_{\theta_0}}[\phi(X)] &= E_{p_{\theta_0}}\left[\nabla_\theta \ln p_{\theta}(x) |_{\theta=\theta_0} \right]= \int \nabla \ln p_{\theta}(x) \, p_{\theta}(x) \, \text{d}x = \int \frac{\nabla p_{\theta}(x)}{p_{\theta}(x)} p_{\theta}(x) \, \text{d}x \\
&= \int \nabla p_{\theta}(x) \, \text{d}x = \nabla \int p_{\theta}(x) \, \text{d}x = \nabla 1 = 0
\end{align*}

$I = E_{p_{\theta_0}}[\phi(X)\phi^T(X)]=E[\phi(X)^2]=V[\phi(X)^2]-E[\phi(X)]^2=V[\phi(X)^2]-0\ge 0$

For $\underset{(k,k)}{I}$ is symmetric and positive definite, $I^{-1}=L^TL$ by Cholesky decomposition, where $L$ is a lower triangular matrix with real and positive diagonal entries.

\begin{align*}
K(x,y)&=\phi(x)^T I^{-1} \phi(y)=\phi(x)^T L^T L \phi(y)=(L\phi(x))^T (L \phi(y))\\
      &=\langle L\phi(x),L\phi(y)\rangle_{\mathcal{H}}=\langle L\phi(y),L\phi(x)\rangle_{\mathcal{H}}=K(y,x)
\end{align*}

where $\mathcal{H}$ is a Hilber Space. $L\phi(\cdot)$ is a function $\mathcal{X}\to \mathbb{R}$

\item Verify that $K$ is positive definite

Choose $x_1,..x_n\in\mathcal{X}$; $\alpha_1,..\alpha_n\in\mathbb{R}$

$$\sum_{i,j=1}^n\alpha_i\alpha_j k(x_i,x_j)=\sum_{i,j=1}^n\alpha_i\alpha_j(L\phi(x_i))^T (L \phi(x_j))=(\sum_{i=1}^n\alpha_i L\phi(x_i))^T (\sum_{j=1}^n\alpha_j L \phi(x_j))=\|\alpha L\phi(x)\|^2_{\mathcal{H}}\ge 0$$

\item
Consider the following multivariate Normal model with given invertible covariance matrix $\Lambda^{-1}$:
$p_\theta(x)=(2\pi)^{-d/2}(\det{\Lambda})^{1/2}\exp\left(-\frac{1}{2}(x-\theta)^T\Lambda (x-\theta)\right)$
Show that
$\phi(x)=\Lambda (x - \theta_0)$

\begin{align*}
\phi(x)&=\nabla_\theta \ln p_{\theta}(x) |_{\theta=\theta_0}=\nabla_\theta \ln(2\pi)^{-d/2}(\det{\Lambda})^{1/2}\exp\left(-\frac{1}{2}(x-\theta_0)^T\Lambda (x-\theta_0)\right)\\
&=\nabla_\theta -\frac{d}2\ln(2\pi)+\frac{1}2(\det{\Lambda})-\frac{1}2(x-\theta_0)^T\Lambda (x-\theta_0)\\
&=-\frac{2}2\Lambda (x-\theta_0)(-1)=\Lambda (x - \theta_0)
\end{align*}

\item Compute the Fisher information matrix $I$ for this model
\begin{align*}
I &= E_{p_{\theta_0}}[\phi(X)\phi^T(X)]=E_{p_{\theta}}[(\Lambda (x - \theta))^T\Lambda (x - \theta)]\\
  &=\Lambda E_{p_{\theta}}[(x - \theta)^T (x - \theta)]\Lambda=\Lambda^3
\end{align*}

\item Compute the Fisher kernel for this model

\begin{align*}
K(x,y)&=\phi(x)^T I^{-1} \phi(y)=(\Lambda (x - \theta_0))^T(\Lambda^3)^{-1}\Lambda (x - \theta_0)\\
  &=(x - \theta_0)^T\Lambda(\Lambda\Lambda\Lambda)^{-1}\Lambda (y - \theta_0)=(x-\theta_0)^T\Lambda^{-1}(y - \theta_0)
\end{align*}

\end{enumerate}

\section{Optimal ordering}
We consider a binary classification problem where the training set is $\mathcal{D}=\{(x_i,y_i), 1\leq i \leq n\}$, with $x_i \in \mathbb{R}^d$ and $y_i \in \{-1,+1\}$ is the class. For convenience, we consider the following subset of indices
\begin{eqnarray}
I_-&=&\{i, 1\leq i \leq n, y_i=-1\}\\
I_+&=& \{i, 1\leq i \leq n, y_i=+1\}
\end{eqnarray}
Moreover, notate $n_-$ the number of elements in $I_-$, and $n_+$ the number of elements in $I_+$. Thus
\begin{equation}
n_-+n_+=n
\end{equation}
 Our objective is to use $\mathcal{D}$ to construct a function $f:\mathbb{R}^d \to \mathbb{R}$ that assign larger values to the data points in the positive class than to the data points in the negative class while being smooth in some sense.

Thus, we propose to choose $f$ in a RKHS with a kernel K that minimizes the following functional:
\begin{equation}
J_0(f) = \frac{1}{n_- n_+}\sum_{i \in I_-}\sum_{j \in I_+} \mathbb{I}_{f(x_i)>f(x_j)} + \lambda ||f||_H^2
\end{equation}
 where $\mathbb{I}_u$ is the indicator function of the event $u$. It takes the value 1 if the event u occurs and 0 otherwise and $\lambda>0$.

Since $J_0$ is not convex, we propose instead to minimize
\begin{equation}
J(f) = \frac{1}{n_- n_+}\sum_{i \in I_-}\sum_{j \in I_+}\left(1-\left(f(x_j)-f(x_i)\right)\right) + \lambda ||f||_H^2
\end{equation}

\begin{enumerate}
\item Show that the minimum is achieved for a function $f$ index by a parameter $\alpha \in \mathbb{R}^d$ and of the form
\begin{equation}
f(x)=\sum_{i=1}^n \alpha_i K(x_i,x)
\end{equation}
Note: do not invoke the representer theorem. Instead, use the key elements in the proof of this theorem applied to this particular case.

Let $v=\text{span} [k(\cdot,x_i),..,k(\cdot,x_n)]$ $\mathcal{V}$ is closed linear subspace of $\mathcal{H}$. Then all minimizers of $J$ belong to  $\mathcal{V}$.
Thus, there is an unique decomposition $f=f_v+f_{\perp}$ with $f_v\in\mathcal{V}$

$\forall f\in\mathcal{V}$, $\langle f_{\perp},g\rangle=0$

$$\|f\|^2_{\mathcal{H}}=\|f_v+f_{\perp}\|^2_{\mathcal{H}}=\langle f_v+f_{\perp},f_v+f_{\perp}\rangle=\langle f_v,f_v\rangle+\langle f_{\perp},f_{\perp}\rangle+\underbrace{2\langle f_v,f_{\perp}\rangle}_{0}=\|f_v\|^2_{\mathcal{H}}+\|f_{\perp}\|^2_{\mathcal{H}}$$

$$f(x_i)=\langle f,k(\cdot,x_i)\rangle=\langle f_{v}+f_{\perp},k(\cdot,x_i)\rangle=\langle f_{v},k(\cdot,x_i)\rangle+\underbrace{\langle f_{\perp},k(\cdot,x_i)\rangle}_{0}=f_v(x_i)$$

For $f$ is strictly increasing

\begin{align*}
J(f)-J(f_v) &= \frac{1}{n_- n_+}\sum_{i \in I_-}\sum_{j \in I_+} [1-\left(f(x_j)-f(x_i)\right)]+ \lambda\|f\|_{\mathcal{H}}^2
              -\frac{1}{n_- n_+}\sum_{i \in I_-}\sum_{j \in I_+} [1-\left(f_v(x_j)-f_v(x_i)\right)]- \lambda\|f_v\|_{\mathcal{H}}^2\\
&=\lambda \|f\|_{\mathcal{H}}^2-\lambda \|f_v\|_{\mathcal{H}}^2=\lambda \|f_{\perp}\|_{\mathcal{H}}^2\ge 0
\end{align*}

The representer theorem allows us to reduce the optimization problem to a finite dimensional optimization problem.

Let $\alpha=(\alpha_1,..,\alpha_n)^T\in\mathbb{R}^n$ is the solution of $\min J(f)$, the function $f(x)=\sum_{i=1}^n \alpha_i k(x_i,x)$,$f \in\mathcal{H}$ that minimizes $J(f)$

\item Rewrite then $J(f)$ as a functional $J(\alpha)$ using the notation  $K$ for the $(n,n)$ matrix $K_{ij}=K(x_i,x_j)$ and $K_i$ for the $i^{th}$ column of $K$.

$$\|f\|^2_{\mathcal{H}}=\langle f,f\rangle_{\mathcal{H}}=\langle \sum_{i=1}^n\alpha_i k(\cdot,x_i),\sum_{j=1}^n\alpha_j k(\cdot,x_j)\rangle_{\mathcal{H}}=\sum_{i,j=1}^n\alpha_i\alpha_j\underbrace{\langle k(\cdot,x_i),k(\cdot,x_j)\rangle_{\mathcal{H}}}_{k(x_i,x_j)}=\underset{(1,n)}{\alpha^T}\underset{(n,n)}{K}\underset{(n,1)}{\alpha}$$

$$f(x_i)=\sum_{j=1}^n\alpha_j k(x_i,x_j)=\sum_{j=1}^n\alpha_j[\underset{(n,n)}{K}]_{i,j}=[K\alpha]_i$$

$$J(\sum_{i=1}^n\alpha_i k(x_i,\cdot))=\frac{1}{n_- n_+}\sum_{i \in I_-}\sum_{j \in I_+} [1-([K\alpha]_j-[K\alpha]_i)]+ \lambda \alpha^T{K}\alpha$$

\item Simplify further the expression $J(\alpha)$ using the notations:

$K_-=\frac{1}{n_-}\sum_{i \in I_-}K_i$;
$K_+=\frac{1}{n_+}\sum_{i \in I_+}K_i$;

$$J(\alpha)=1-[\frac{1}{n_+}\sum_{j \in I_+}[K\alpha]_j-\frac{1}{n_-}\sum_{i \in I_-}[K\alpha]_i)]+ \lambda \alpha^T{K}\alpha=1-[K_+-K_-]\alpha+ \lambda \alpha^T{K}\alpha$$

\item Compute $\nabla_\alpha J(\alpha)$, the gradient in $\alpha$ of $J(\alpha)$. Solve for $\alpha$, assuming that $K$
is invertible.

p.d. $K,X$ is symmetric, $K=K^T$, $X=X^T$; $K=P\Lambda P^T$; $I=PP^T$; $\Lambda$ is diagonal matrix with $\gamma_1,..,\gamma_n$.

$\lambda>0$,$\gamma_i>0$, $K+\lambda I=P(\Lambda+\lambda I) P^T$ is inversible.

\begin{align*}
\nabla_\alpha J&=\frac{\partial}{\partial\alpha}(1-[K_+-K_-]\alpha+ \lambda \alpha^T{K}\alpha)=-[K_+-K_-]+ \lambda \frac{\partial}{\partial\alpha}\langle\alpha,K\alpha\rangle\\
&=-[K_+-K_-]+ \lambda (IK\alpha+K^T\alpha)=-[K_+-K_-]+2\lambda K\alpha\overset{set}{=}0\\
\end{align*}

$$\alpha^\star=(2\lambda K)^{-1}[K_+-K_-]$$

\item Bonus question: Compute $f^*(x)$, where $f^*$ is the minimizer of $J(f)$ for the linear kernel $K(x,y)=x^Ty$. Use the notations
${x}_-=\frac{1}{n_-}\sum_{i \in I_-}x_i$;
${x}_+=\frac{1}{n_+}\sum_{i \in I_+}x_i$

\end{enumerate}

$$\frac{1}{n_+}\sum_{j \in I_+}f(x_l)=\frac{1}{n_+}\sum_{j \in I_+}\sum_{l=1}^n\alpha_l k(x_l,x_j)= \sum_{l=1}^n\alpha_l x_l\frac{1}{n_+}\sum_{j \in I_+}x_j=\alpha^T K x_+=f(.) x_+$$

$$\frac{1}{n_-}\sum_{i \in I_-}f(x_l)=\frac{1}{n_-}\sum_{i \in I_-}\sum_{l=1}^n\alpha_l k(x_l,x_i)= \sum_{l=1}^n\alpha_l x_l\frac{1}{n_-}\sum_{i \in I_-}x_j=\alpha^T K x_-=f(.) x_-$$

$$J(f)=1-(f(.) x_+-f(.) x_-)+ \lambda f(.)^T f(.)$$

\begin{align*}
\nabla_{f(.)} J&=\frac{\partial}{\partial f(.)}[1-(f(.) x_+-f(.) x_-+ \lambda f(.)^T f(.)]=-[x^T_+-x^T_-]+ \lambda \frac{\partial}{\partial\alpha}\langle f(.),f(.)\rangle\\
&=-[x^T_+-x^T_-]+ 2\lambda f(.)\overset{set}{=}0\\
\end{align*}

$$f^\star(x)=(2\lambda)^{-1} [x^T_+-x^T_-]$$

\end{document}


\chapter{M/EEG modelling and statistics}
\label{ch:eeg_stats}
After projection to 2D- or a 3D-space (source reconstruction), the
data is in voxel-space and ready to be analysed. There are several
ways how one can proceed. In this chapter, we will focus on analyzing
epoched time-series data. These can be event-related responses (ERPs),
event-related fields (ERFs) or single trials (M/EEG).

In the following, we will go through the various stages of modelling
using typical examples to illustrate the procedures.

\section{Preliminary remarks}
All analyses can be done using either the graphical user interface
(GUI) or a batch system (i.e. using scripts in the SPM2-fashion,
s.~below). The GUI has the advantage that one doesn't need
matlab-knowledge to analyse data. The batch system has the
advantage that it is a fast and efficient way of entering model and
data. Its disadvantage is that some Matlab-Knowledge is
required. However, with this distribution, we provide some template
scripts to analyse (typical) data in batch mode. We assume that with slight
modifications these scripts can be used for most analyses.

\section{How epoched time-series are analysed in SPM}
After preprocessing the data (i.e.~epoching, filtering, etc...) and
projection to voxel-space, we have discretely sampled versions of
continuous fields \cite{kiebel_spm_eeg1}. These data can be analysed
with a mass-univariate approach using results from Random Field theory
(RFT) to adjust p-values for multiple comparisons \cite{kjw_hbf2}. The
model used at each voxel is a general linear model
\cite{kiebel_spm_eeg2}. Typically one wants to analyse
multiple subjects' data acquired under multiple conditions. Given that
each evoked-response has up to hundreds of time points, this is an
awful lot of data at each voxel. The ideal way to analyse these data
would be to specify a single hierarchical model (1st level: within-subject,
2nd level: over subjects) and estimate its parameters. However, this
is computationally not feasible because of the length of the data
vector at each voxel. Fortunately, such a 2-level model can usually be
split up into two models: The 1st level and the 2nd level model. The
input data to the 2nd model are contrasts of the 1st level model
\cite{kiebel_spm_eeg2}. In all cases considered in this chapter, this
2-stage procedure
gives exactly the same results as the 2-level model. The reason for
this is that we are not really \textit{modelling} the data at the 1st
level, but simply forming weighted sums of the data, over time. For
example, if we are interested in the N170 component, one could average
the data from 150 to 190 milliseconds. This is exactly the approach
used in conventional ERP analysis. This approach is not a model,
because simply taking sums corresponds to using an identity matrix as
design matrix. This procedure leaves no degrees of freedom for error
estimation.\\

In summary, the SPM-approach is to form, at each voxel, weighted sums of
the data, over time, at the 1st level. We refer to these weighted sums
as contrast images. These form the input to the 2nd level, where
one usually tests for differences betwen conditions or between groups
(s.~below). The second level models are usually the same as the ones
one would use for functional magnetic resonance imaging (fMRI). Importantly,
these 2nd level models have enough degrees of freedom to estimate the
error, i.e.~statistics can be computed.\\

The output of such a 2nd level analysis is a
voxel-volume (or map), where each voxel contains one statistical
value. The associated p-value is adjusted for multiple
comparisons \cite{kjw_hbf2}. This adjustment is important, because there are many
other voxels or channels. One (disadvantageous) alternative to
adjustment is to consider only pre-selected channels or averages over
channels. This is why the adjustment is especially important for
high-density measurements, because there are many channels to
select from. We believe that it is generally too subjective
to select channels for analysis a-priori. We see the
GFT-adjustment as a good way of looking at the whole data without any
prior selection. This has been already demonstrated for EEG data (in
another context) by \cite{james_rft}.

\section{1st level}
At the 1st level, we select periods or time points in
peri-stimulus time that we would like to analyse. Critically, this
choice must be made a-priori by you. The alternative would be to not
treat peri-stimulus time as a factor, but as a dimension of a Random
Field approach. This alternative approach is often used in
time-frequency analysis of induced and evoked oscillations, where it
seems sometimes difficult to specifiy areas of interest on the
time-frequency plane a-priori \cite{james_rft}.

In the present approach, time is a factor, and
you have to form weighted-sums over peri-stimulus time to provide
input to the 2nd level. Of course, you don't need to constrain
yourself to a single contrast around a specific peri-stimulus time,
but you can compute as many as you like. For example, to analyse
multiple aspects of an ERP, it is not uncommon to form averages around
several time-points of an ERP. At the 2nd level, these can be either
analysed independently or within one model to make inferences about
interactions between conditions and peri-stimulus time.

In the follwing, we will go through model specification and
computation of contrast images. This guide is not written as a
tutorial (i.e.~detailed instructions for a given data set), but
describes each design option and hopefully provides deeper background
knowledge.

\subsection{The aim}
The aim of the 1st level is to compute contrast images that provide
the input to the 2nd level. We will describe this using the example of
2D-data, i.e.~data that has not been source reconstructed but, for
each peri-stimulus time point, has been projected to a 2D-plane
(s.~chapter \ref{ch:eeg_source}).

\subsection{Start}
Start SPM by the command \textit{'spm eeg'} from the matlab command
line. Press the \textit{EEG/MEG} button. Your first choice is to
either specify the model design or the input data. One always starts
with the design. Currently, there are two design options: (i)
\textit{all options} and (ii) \textit{ERP/ERF}. The latter option is a
shortcut to quickly input an evoked responses study. We will first
describe \textit{all options} and then treat the \textit{ERP/ERF}
option as a special case.

\subsection{All options}
You first have to answer the question whether
this is a 1st level design. This determines whether SPM expects to
model peri-stimulus time as a factor. Also, if one models first-level
data, SPM will ask next for \textbf{one} M/EEG-matfile before the
data was projected to voxel-space. The reason for this is that the voxel-images
lost important information during the conversion. For example, all timing
information were lost. With the nifti-images only, SPM doesn't
know the peri-stimulus time of each data point. However, this
information is critical as soon as you try to specify (later on)
linear weights in terms of peri-stimulus time. So, when you select an
M/EEG file, SPM will read timing information from this file. For an
ERP-study, the M/EEG-file of the average (ERP) is a good choice.

\subsection{How many factors?}
This question starts off the design specification proper. SPM needs to
know the number of factors which you want to model. At the 1st level,
there are typically only factors \textit{peri-stimulus time} and
\textit{condition}. If you like, you can further subdivide the
condition-factor in its components. For instance, if you have a 2x2
factorial design, you may want to specify 3 factors: \textit{factor1},
\textit{factor2} and  \textit{peri-stimulus time}.

\subsection{Factor names and \# of levels for factors}
For each factor, you now input its name, e.g.~condition, and enter the
number of levels. For instance, if you have 2 conditions, you enter
2. For peri-stimulus time, you enter the number of time points in your
evoked responses. Important: You should call the peri-stimulus time
factor  'time'. For the number of levels for this special factor, SPM
defaults to the correct number of peri-stimulus time points. (Note
that it is currently not possible to model only a subset of time points.)

\subsection{Select design component}
You have the choice between \textit{Identity} and
\textit{Constant}. Your selected design components are combined (by
Kronecker tensor product) to form the 1st level design matrix. This
has also been described in \cite{kiebel_spm_eeg2}. For the 1st level,
you simply choose for all factors \textit{identity}. This completes
model specification.

\subsection{Data}
For selecting data, press the \textit{EEG/MEG} button again. After
selecting the \textit{SPM.mat} file, you are asked to select data for
each factor. The order in which you input data  depends on the order
of how you named the individual factors. We recommend that you make
the \textit{peri-stimulus time} factor the last factor. After
projection to voxel-space, the data are stored as 4-dimensional files
with the third dimension $z = 1$. If you want to input all
peri-stimulus time points for a given file, you have to select all
volumes along the 4th dimension. This is done by setting the number
'1' in the SPM-file selector (below the 'Filt' line) to '1:101', where
'101' is the total number of peri-stimulus time points. Of course, you
have to replace '101' by the number of time points of your
data (or by any natural number bigger than that). This choice will make
all time points selectable. Then right-click over the file
names and \textit{Select all}. Press \textit{done} to confirm your
choice. This completes data selection.

\subsection{'Estimation'}
Although there is actually nothing to estimate, clicking the
\textit{Estimation} button will prepare some internal structure for
the results section. We kept this (otherwise redundant) estimation step to
provide for greater similiarity with other analyses using SPM.

\subsection{Results}
After clicking on \textit{Results}, choose the appropriate
\textit{SPM} and the contrast manager will pop up. In contrast to a usual
SPM study, we don't use the contrast manager to compute
statistics, but contrasts only!

Click \textit{Define new contrast...} and enter a name for your
contrast. Then note a (new) button called \textit{components} which is
only visible for M/EEG models. Clicking this button opens the contrast
components manager. This is simply a tool that exploits the knowledge
about the factors which you have specified earlier. Knowing the
factors and their levels makes it easy to split up a (long) contrast
weight vector into a few components. For each contrast weight vector,
each factor contributes one component. By using the Kronecker tensor
product, these components can be combined into the resulting contrast
weight vector. This is not only time-saving, but many people tend to
find this approach more intuitive than the usual approach of figuring
our the contrasts yourself. For instance, if you have specified
two conditions, you might be interested in their difference. Enter a
$[-1 \; \; 1]$ as contrast component. For the \textit{time} factor,
instead of entering one number for each time point, better click on
the \textit{Generate} button. Click on the 'Time' button and specify
a rectangular averaging window by providing the start and end of this
window (in milliseconds). Press \textit{Compute}. You can see now in
the contrast manager window that your contrast weights have been
computed and are displayed above the identity (design)
matrix. You can also specify the contrast weights as usual in the
contrast box, but this would require to enter several hundreds to
thousands of numbers. Press \textit{ok} to proceed and compute the
contrast.

\subsection{Display}
You can display the resulting contrast image by using the
\textit{Display} button.

\subsection{ERP/ERF}
You can shortcut some of the question and especially the data
selection by choosing the \textit{ERP/ERF} option (instead of
\textit{all options} when specifying a design. This options assumes
that you have two factors, \textit{condition} and \textit{time}. There
are less questions during design specification. When selecting data,
you don't need to select all time point, but only the first! SPM will
assume that you want to select all time points of the selected
file. Using this option will otherwise result in the same model
as described above.

\subsection{Multiple subjects}
For each of your subjects, you perform these operations in a separate
1st-level analysis. For each subject, you want to compute the same
contrasts and use them as input to a model, where subjects is the
repetition factor.

\section{2nd level models}
For 2nd level modelling, you can use different ways to specify a
model. There is \textit{Basic models} which was primarily developed for
PET/fMRI but is equally appropriate for EEG/MEG data. These are
suited best when the model is simple (like a 1-sample or 2-sample
t-test). In our experience, most EEG/MEG models fall into this
category of simple models. If models are more complicated, like, e.g.,
two groups with multiple subjects/conditions, we recommend using the
\textit{EEG/MEG} models.

\subsection{All options}
As above, go for \textit{All options}. This time, press 'no' for the
question 'Is this a first-level design'.

\subsection{Factors}
This includes all factors, even repetition factors. For example, at
the 2nd level a 2x2 factorial design has 3 factors: \textit{subject},
\textit{factor1} and \textit{factor2}.

\subsection{Design partitions and design components}
The way this modelling device constructs a design matrix is by using the
Kronecker tensor product on the hierarchy of specified design
components. However, some/many designs can't be constructed in this
way. For example, the design matrix of a paired two sample-test
consists of two merged partitions, each of which is a Kronecker tensor
product of design components. For each partition, the factors and the
levels are the same. The difference is in the choice of the design
components for each factor under each partition. For example, for a
paired two-sample-test, one has 2 factors (subjects and conditions)
and 2 design partitions. For the 1st partition, choose
\textit{Constant} for subjects and \textit{Identity} for
conditions. For the 2nd partition, it's the other way around,
i.e.~\textit{Identity} for subjects and \textit{Constant} for
conditions.

\subsection{Covariance components}
Specification of the covariance components determines the error
model \cite{daniel_hbf2}. For each factor, there are two questions:
(i) Identical variance for
factor $xxx$, and (ii) Independence for factor $xxx$. SPM constructs all the
variance components from your answers. The first question pertains to
the assumption whether each level of this factor has identical
variance. The second questions asks whether the different levels for a
given factor are correlated. Some examples: For a repetition factor
like subjects, you should always answer both questions with yes. For a
group factor, one would assume that the levels of this factor (the
groups) have unequal variance structures, but are uncorrelated (i.e.,
(i) no, (ii) yes). For a condition factor, the choice is up to
you. A very restrained model would follow from using (i) yes (ii)
yes, whereas the most liberal model is given by (i) no (ii) no.

\subsection{Data}
For each combination of factors, SPM asks you for the filenames of the
data. Sometimes, this process can be more convient for you, when you
have specified the factors in a specific order. For example, if you
have two factors \textit{subjects} and \textit{condition}, the order
(i) subjects, (ii) condition will ask for all images for each
subject. This is convienient if you have stored the contrast images
in their individual subject folder. This is the case, if you
have computed 1st level contrasts following the approach described
above. However, if, in an intermediate step, you have saved contrasts
in condition-specific folders, the alternative order ((i) condition, (ii)
subjects) is more appropriate.

\subsection{Estimation and Results}
The estimation follows the usual scheme, i.e.~for a classical
estimation procedure we use exactly the same routine as for PET/fMRI
data (i.e.~maximum-likelihood estimators for the parameters and Restricted
Maximum Likelihood for estimation of the variance parameters).

For specification of contrasts, you have the option to specify
contrasts component-wise. This can be useful for complex designs, when
it's no longer easy to work out the interaction contrasts.

For 2D data the statistical map is displayed instead of the usual
glass brain. You can invoke all the usual functions that are also
available for fMRI/PET data. An additional option is \textit{channels}
which let you visualise to which voxel each channel maps. You can
select this option by right-clicking the button on the statistical
map background. SPM asks you then for one of the original M/EEG-mat
files to read the channel mapping.


\label{sec:Res}
\section*{Results}

We partitioned variance due to the species effect, i.e. the part of the variance explained by the species of individuals (see~\autoref{fig:aov}). Depending on the trait, the species effect could explain between 27\% up to 75\% of the variability of our data. While species effect can explain over 75\% of the variability in wood density, it only explains less than 30\% of the variability in AGR.

To test how growth was affected by individual traits we made growth mixed-model for each one of them (see~\autoref{tab:seltraits} for traits and~\autoref{tab:growth_mod} for models) as predictors of the AGR. Based on the adapted R-squared for mixed-models~\citep{nakagawa_general_2013}, we selected the best model for each trait (see~\autoref{tab:growth_mod}).

For all traits models with distance measures were better than model with only species average term. But for wood density and SLA, hierarchical distances model performed as well as indvidual's trait models. By adding a distance term to the species average term model, we gained in predictive power, shown by the increase of condition R-squared between models.

For bark thickness the absolute distance model improved the conditional R-squared by 0.025 compared to the model with only species average trait. For wood density hierarchical distance model improved the R-squared by 0.056 (10\% of total $\text{R}^2$) compared to species average model. While for SLA hierarchical distance model only increased $\text{R}^2$ by 0.004. For laminar chlorophyll and toughness the hierarchical distance models gained 0.001 of R-squared compared to the species average term models.

Using the best models (in bold in~\autoref{tab:growth_mod}) that included a distance term we predicted AGR for a range of individual trait values and species average values. To underline the interplay between intra- and inter-specific variabilities we plotted the centered species average trait value vs. the hierarchical distance of indivdual's (see~\autoref{fig:simul}). We obtained AGR "landscapes", that show the pattern of variations of variation as a function of individual and species trait variation. Depending on the trait we had different patterns, for SLA and wood density we had similar patterns: the larger the species average trait value, the lower the AGR, and the higher the individual hierarchical distance the lower the AGR. For example, an increase of $10\text{cm}^2.\text{g}^{-1}$ in SLA species average decreases the growth by $0.035\text{mm}.\text{yr}^{-1}$ while the same increase in hierarchical distance decreases the AGR by $0.02\text{mm}.\text{yr}^{-1}$. For laminar chlorophyll content we observe that increasing species average as well as increasing hierarchical distance lead to increasing AGR. While for leaf toughness, species average and hierarchical AGR gradients are opposed: an increase in species average decreases AGR, and an increase individual hierarchical distance increases AGR. Because the best model for bark thickness was an absolute distance model, we observe the symmetrical patterns of AGR variations for change in hierarchical distance: globally, the higher the hierarchical distance the lower the AGR, while the higher the species average, the lower the growth.
\begin{table*}[!pt]
	\begin{center}
		\begin{tabular}{llcccc}
		  \hline
		 Trait Name & Model Type & Marginal $\text{R}^2$ & Cond. $\text{R}^2$ & AIC & logLikelihood \\
		  \hline
		Bark Thickness & Species Avg. & 0.104 & 0.472 & -2790 & 1407 \\
		            & Hierarchical Distance & 0.106 & 0.471 & -2794 & 1410 \\
		            & \textbf{Absolute Distance} & 0.100 & \textbf{0.497} & -2860 & \textbf{1443} \\
		            & Individual Trait & 0.102 & 0.470 & -2792 & 1408 \\
		  \cline{2-6}
		  Wood Density & Species Avg. & 0.142 & 0.474 & -1838 & 931 \\
		               & \textbf{Hierarchical Distance} & 0.139 & \textbf{0.530} & -1910 & \textbf{968} \\
		               & Absolute Distance & 0.140 & 0.529 & -1906 & 966 \\
		               & \textbf{Individual Trait} & 0.137 & \textbf{0.530} & -1911 & \textbf{968} \\
		  \cline{2-6}
		  SLA & Species Avg. & 0.094 & 0.485 & -3081 & 1553 \\
		      & \textbf{Hierarchical Distance} & 0.095 & \textbf{0.489} & -3092 & \textbf{1559} \\
		      & Absolute Distance & 0.096 & 0.459 & -3000 & 1513 \\
		      & \textbf{Individual Trait} & 0.096 & \textbf{0.490} & -3093 & \textbf{1559} \\
		  \cline{2-6}
		  Chloro. Content & Species Avg. & 0.092 & 0.487 & -3143 & 1583 \\
		                  & \textbf{Hierarchical Distance} & 0.093 & \textbf{0.488} & -3145 & \textbf{1586} \\
		                  & Absolute Distance & 0.092 & 0.486 & -3143 & 1584 \\
		                  & Individual Trait & 0.100 & 0.469 & -3075 & 1550 \\
		  \cline{2-6}
		  Toughness & Species Avg. & 0.088 & 0.478 & -3136 & 1580 \\
		            & \textbf{Hierarchical Distance} & 0.088 & \textbf{0.479} & -3135 & \textbf{1581} \\
		            & Absolute Distance & 0.088 & 0.478 & -3134 & 1580 \\
		            & Individual Trait & 0.088 & 0.479 & -3134 & 1579 \\
		  \cline{2-6}
		  All Traits & Species Avg. & 0.133 & 0.524 & -1739 & 885 \\
		             & \textbf{Hierarchical Distance} & 0.143 & \textbf{0.532} & -1761 & \textbf{901} \\
		             & Absolute Distance & 0.138 & 0.524 & -1743 & 892 \\
		             & Individual Trait & 0.138 & 0.528 & -1766 & 899 \\
		   \hline
		\end{tabular}
		\caption{\textbf{Summary table of tested trait-specific growth models.} We modeled radial growth using linear-mixed model, all models contained plot and species random effects, as well as DBH and $\log\text{DBH}$ terms to take growth curve shape into account~\citep{herault_functional_2011}. Then, for each trait, we added a fixed effect that contained different terms: \textbf{Species Avg.}, only the species average trait value; \textbf{Hierarchical Distance}, the species average plus the difference between individual trait and species average; \textbf{Absolute Distance}, the species average plus the absolute difference between individual trait and species average; \textbf{Individual Trait}, only the trait value of individuals. Models indicated in \textbf{bold} are those with the highest logarithmic likelihood per trait. Not all traits had been measured on each tree, giving a different number of comprised tree for each model, thus the likelihood of models for each trait are not comparable.}
		\label{tab:growth_mod}
	\end{center}
\end{table*}
\begin{figure*}[!tb]
	\centering
	\begin{subfigure}[c]{0.45\textwidth}
		\includegraphics[scale=0.75]{figures/Plots_Map_2015-05-26.pdf}
		\caption{}
		\label{fig:map}
	\end{subfigure}
	\begin{subfigure}[c]{0.5\textwidth}
		\includegraphics[scale=0.7]{figures/Aov_Var_Traits_2015-05-25.pdf}
		\caption{}
		\label{fig:aov}
	\end{subfigure}
	\caption{\textbf{(a) Plots map.} 9 1-ha plots were used, spread in French Guiana, two plots were surveyed both in Nouragues and in Paracou (see~\citealp{baraloto_decoupled_2010}) \textbf{(b) Explained variance by species effect in ANOVAs.} Dot-plot of explained variance in ANOVA by the species effect for traits and AGR. \textbf{Chloro. Content}: Laminar Chlorophyll Content, \textbf{AGR}: Annual Growth Rate (in diameter).}
	\label{fig:gen}
\end{figure*}

\begin{figure*}[!ptb]
	\centering
	\includegraphics{figures/Sel_Traits_Simul_AGR_2015-05-30.pdf}
	\caption{\textbf{Predictions of AGR depending on intra-specific and inter-specific variabilities in traits.} Surface plots of predicted AGR of simulated range of data: X-axis, centered species average trait (species average trait minus mean of all species average trait); Y-axis, individual distance to species average trait. Black lines are equal-AGR lines over the surface, i.e. on those line each point has the same AGR value, each line mark a $5e^{-3}\text{mm}.\text{yr}^{-1}$ break. For details on traits see~\autoref{tab:seltraits}.}
	\label{fig:simul}
\end{figure*}