\section{Approach}
We use fraction permissions to denote ownerhip. Each object has a field called \emph{ownership} denoting by a floating number between 0 and 1 inclusive. The permission 0 means that the variable is not accessible anymore. The permission 1 means that the variable is mutable. If \CODE{o} is greater than 0 and less than 1, then it is readable.

There is a partial function \CODE{globalOwnership} that maps from variables to their permissions.
There is a partial function \CODE{globalAliasing} that maps from variables to set of variables.

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
val x = new @mut Cell  [x -> 1][]
val y = @shr x            [x -> 0.5, y -> 0.5][x -> {y}]
val z = @shr x            [x - 0.25, z - 0.25, y - 0.5][x - {y,z}]
x.f = 5               [x -> 1, z -> 0, y -> 0][]

\end{lstlisting}

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
if (b){
val x = new @mut Cell  [x -> 1][]
val y = @shr x            [x -> 0.5, y -> 0.5][x -> {y}]
val z = @shr x            [x - 0.25, z - 0.25, y - 0.5][x - {y, z}]
x.f = 5               [x -> 1, z -> 0, y -> 0][]
}
val t = x             [x -> 0, z -> 0, y -> 0][]
\end{lstlisting}

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
val t = if (b){
val x = new @mut Cell  [x -> 1][]
val y = @shr x            [x -> 0.5, y -> 0.5][x -> {y}]
val z = @shr x            [x - 0.25, z - 0.25, y - 0.5][x ->{y, z}]
x.f = 5
x                     [x -> 1, z -> 0, y -> 0][]
}                     [t -> 1, x -> 0, z -> 0, y -> 0][]
\end{lstlisting}

\subsection{Methods}
There is a partial function \CODE{globalFun} that maps from method names to pairs of parameters' permissions and return value's permission.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[f1-> ([1], -1)]
def f1(c: @mut Cell) : Unit = {    [c -> 1][] // assumption
  val c0 = c;                      [c-> 0; c0 -> 1][]
}
def client() = {
  val cc = new @mut Cell;     [cc -> 1][]
  f1(@mut cc)                      [cc -> 0][] // assert cc -> 1
  cc.f                        // error
}
\end{lstlisting}

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[f2-> ([1], 1)]
def f2(c: Cell) : Cell = {    [c -> 1][]  // assumption
  val c0 = c;                 [c-> 0; c0 -> 1][]
  c0.f = 5;                   [c-> 0; c0 -> 1][]
  c0                                     // assert c0 -> 1
}
def client() = {
  val cc = new @mut Cell;      [cc -> 1][]
  val ct = f1(cc)             [cc -> 0, ct -> 1][] // assert cc == 1;
  ct.f = 5;                   // ok
}
\end{lstlisting}

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[f3-> ([0.5], -1)]
def f3(c: @shr Cell) : Unit = {  [c -> 0.5][] // assumption
  val c0 = c;                 [c -> 0.25, c0 -> 0.25][c -> {c0}]
  c.f = 5;   // error
}
def client() = {
  val cc = new @mut Cell;      [cc -> 1][]
  f3(&cc)                     [cc -> 1][] // assert cc > 0
  cc.f = 5;                   [cc -> 1][] // ok
}
\end{lstlisting}

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[f4-> ([0.5], 1)]
def f4(c: @shr Cell) : Unit = {  [c -> 0.5][] // assumption
  val t = c.f
  val c0 = new @mut Cell(t);   [c -> 0.5, c0 -> 1][]
  c0                                      // assert c0 -> 1
}
def client() = {
  val cc = new @mut Cell;      [cc -> 1][]
  val ct = f4(@shr cc)            [cc -> 1, ct -> 1] // assert cc > 0
  ct.f = 5;                   // ok
}
\end{lstlisting}

The following two cases may not be sensible. It needs to track aliasing in the method body.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[f5-> ([0.5], 0.5)]
def f5(c: @shr Cell) : @shr Cell = {  [c -> 0.5][] // assumption
  val cc = new @mut Cell;
  &cc                                      // assert cc > 0
}
def client() = {
  val cc = new @mut Cell;      [cc -> 1][]
  val ct = f5(@shr cc)            [cc -> 1, ct -> 0.5][?]  // assert cc > 0
  ct.f = 5;                    // error
  cc.f = 5;                   [cc -> 1, ct -> ?][?]
}
\end{lstlisting}

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[f6-> ([1], 0.5)]
def f6(c: Cell) : @shr Cell = {  [c -> 1][] // assumption
  @shr c                                      // assert cc > 0
}
def client() = {
  val cc = new @mut Cell;      [cc -> 1][]
  val ct = f6(@mut cc)             [cc -> 0, ct -> 0.5][?]  // assert cc == 1
  ct.f = 5;                   // error
  cc.f = 5;                   // error
}
\end{lstlisting}

\clearpage
\subsection{Higher-order functions}
%\CODE{f} cannot escape from \CODE{withFile}. So \CODE{fn} has to be \emph{local}:
%\begin{itemize}
%\item \CODE{f} cannot be returned from \CODE{fn}
%\item the return value of \CODE{fn} is readonly.
%\item \CODE{fn} can only be invoked by \emph{local} functions.
%\end{itemize}
\begin{itemize}
\item There is a partial function \CODE{globalOwnership} that maps from variables to their permissions.
\item There is a partial function \CODE{globalAliasing} that maps from variables to set of variables.
\item There is a partial function \CODE{globalFun} that maps from method names to a tuple of parameters' permissions,  return value's permission and aliasing information.
\end{itemize}

\paragraph{1st class values}

The following example shows that the method body is not type checked.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[withCell -> ([-1], 0.5, [?]), fn -> ([0.5], 0.5, [p1]), client -> ([-1], -1, [])]
def withCell(n: Int)(fn: & Cell =>  & Cell): & Cell = {
  val c = new @mut Cell(n);      [c -> 1][]
  try val ret = fn(@shr c)     [c -> 0.5, ret -> 0.5][c -> {ret}]
  finally c.free            [c -> 0, ret -> 0][]
  ret                       [c -> 0, ret -> 0][]    // error
}
def client(){
  val c1: Cell = withCell(5){ c => c};
}
\end{lstlisting}

Question marks mean that the return value is aliasing with somebody, but is definitely not with parameters.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[withCell -> ([-1], 0.5, [?]), fn -> ([0.5], 0.5, [?]), client -> ([-1], -1, [])]
def withCell(n: Int)(fn: @shr Cell =>  @shr Cell): @shr Cell = {
  val c = new @mut Cell(n);      [c -> 1][]
  try val ret = fn(@shr c)     [c -> 0.5, ret -> 0.5][]
  finally c.free            [c -> 0, ret -> 0.5][] // treated as mutation
  ret                       [c -> 0, ret -> 0.5]    // ok
}
def client(){
  val c1 : Cell = withCell(5){ c => val t = new Cell(c.f); @shr t }. // unsafe
}
\end{lstlisting}

\emph{Summary:}
\begin{itemize}
\item The apporach is \emph{not modular} because without knowing the implementation of \CODE{fn}, the aliasing information of \CODE{withCell} cannot be known.
\item Function parameters are like interfaces with one function in Java.
\end{itemize}

\paragraph{2nd class values}

The signature of \CODE{fn} means that the return value of \CODE{fn} is not aliased with the parameter.

Question marks mean that the return value is aliasing with somebody, but is definitely not with parameters.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[withCell -> ([-1], 0.5, []), fn -> ([0.5], 0.5, [?])]
def withCell(n: Int)(fn: @local @shr Cell =>  @shr Cell ): @shr Cell = {
  @local val c = new @mut Cell(n);    [c -> 1][]
  try val ret = fn(@shr c)          [c -> 0.5, ret -> 0.5][]
  finally c.free                 [c -> 0, ret -> 0.5][]
  ret                            [c -> 0, ret -> 0.5][]
}
def client(){
  val c1: Cell = withCell(5){
        c => c      // error
  }
  // first-class values are also second-class.
  val c2: Cell = withCell(5){ c => val t = new @shr Cell(c.f); & t}; [c2 -> 0.5][] //unsafe
  // don't know who c2 is aliased with.
}
\end{lstlisting}

Parameters and return values are both @local.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[withCell -> ([-1], -1, []), fn -> ([0.5], 0.5, [p1])]
def withCell(n: Int)(fn: @local @shr Cell =>  @local @shr Cell): Unit = {
  val c = new @local @mut Cell(n);    [c -> 1][]
  try val ret = fn(@shr c)          [c -> 0.5, ret -> 0.5][c -> {ret}]
  ret.f = 5                      [c -> 0, ret -> 1][]
  finally c.free                 [c -> 0, ret -> 0][]
}
def client(){
  withCell(5){ c => c }; [][]
}
\end{lstlisting}

Nested @local: need to disassemble \CODE{fn}. Is it allowed by @local type system?
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[withCell -> ([-1], -1, []), fn -> ([0.5], -1, [p1]), <fnn> -> ([0.5], 0.5, [p1])]
def withCell(n: Int)
            (fn: @local @shr Cell =>  (@local @shr Cell => @local @shr Cell)): Unit = {
  @local val c = new Cell(n)     [c -> 1][]
  try{
    val <ret_fn> = fn(@shr c)         [c -> 0.5][]
    val ret = ret_fn(c)          [c -> 0.5, ret -> 0.5][c -> {ret}]
    ret.f = 5                    [c -> 0, ret -> 1][]
  }
  finally ret.free               [c -> 0, ret -> 0][]
}
def client(){
  withCell(5){ c => (c => c) }; [][]
}
\end{lstlisting}

\clearpage

\subsection{Structs}

\cite{haller2010capabilities}
We use @rep to denote the object is owned by the enclosing structure.
We use @sep to denote the memory regions reachable from an object is disjoint with other objects.

\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
@sep              //
struct Pair{
  val x : Cell
  val y : Cell
}

[shift -> ([1, -1], -1, [])]
def shift(@mut c: Cell, i: Int) : Unit { // c is not used after the call to shift
   c.f = c.f + i
}

[shift -> ([1, -1], 1, [])]
def shift1(@mut c: Cell, i: Int) : @mut Cell { // @mut reference has to be returned
   c.f = c.f + i
   c
}

[incr -> ([0.5, -1], -1, [])]
def incr(@shr c: Cell, i: Int) : Int {       // c is not used after the call to shift
   c.f + i
}

[incr -> ([0.5, -1], -1, [])]
def incr1(@shr c: Cell, i: Int) : @shr Cell {  <not safe: dangling references>
   val ret = new Cell(c.f + i)
   @shr ret
}

[withPair -> ([1, 1], -1, []), fn -> ([0.5], -1, [])]
def withPair(@sep c1: Cell, @sep c2: Cell)(fn: @local @sep @shr Pair => Int) : Int {
 @local var p = new Pair(c1, c2)  [p -> 1][]
 fn(@shr p)                       [p -> 1][]
 shift(@mut p.x)             [p.x -> 1, p.y -> 1][]
 shift1(@mut p.x)                 [p.x -> 1, p.y -> 1][]
 val t = incr(@shr p.x)           [p.x -> 1, p.y -> 1][]
 free(p)                          [p -> 0][]  // assert(p.x -> 1, p.y -> 1)
}

def client1(){
  val c = new @mut Cell(1)           [c -> 1][]
  val c1 = @shr c                  [c -> 0.5][c1 -> 0.5][c -> {c1}]
  val p = withPair(c, c1){ ... }      // error
}

def client2(){
  val c = new @mut Cell(1)                         [c -> 1][]
  val c1 = new @mut Cell(2)                        [c -> 1, c1 -> 1][]
  val sum = withPair(c, c1){ p => p.x + p.y } [c -> 0, c1 -> 0][]
}

\end{lstlisting}

\emph{Array}
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
[add -> ([1, -1], 1, []), withArray -> ([1], 1, []), fn -> ([0.5], -1, [])]
def add(arr: Array[@sep Pair], idx : Int) : Array[@sep Pair]{
  val p = new @mut Pair(new Cell(1), new Cell(2))   [arr -> 1, p -> 1]
  arr[idx] = p                                 [arr#0 -> 1, $\cdots$, arr#arr.length - 1 -> 1]
  arr               // assert(arr#0 -> 1, $\cdots$, arr#arr.length - 1 -> 1)
}
def withArray(arr: Array[@sep Pair])(fn: @sep & Pair => Int) : Array[@sep Pair]{
                           [arr -> 1]
  arr.map(x => fn(x))      [arr#0 -> 1, $\cdots$, arr#arr.length - 1 -> 1]
  arr               // assert(arr#0 -> 1, $\cdots$, arr#arr.length - 1 -> 1)
}

def client(){
  val arr = Array[@sep Pair](5)                    [arr -> 1][]
  val arr1 = add(arr, 0)                           [arr-> 0, arr1 -> 1][]
  val arr2 = withArray(arr1){p => p.x + p.y } [arr-> 0, arr1 -> 0, arr2 -> 1][]
}
\end{lstlisting}

\emph{HashMap}
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
def client(){
  val m = mutable.HashMap[Int, @sep Pair]()
  val imm = Hash[Int, @sep Pari]()
}
\end{lstlisting}

\clearpage

\subsection{Using Ownership to Encode @local}
Can Rust's type system prevent \CODE{c} from escaping?
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
def withCell(n: Int)
    (fn:  @mut Cell => (@mut Cell => @mut Cell)): Unit = {
  val c = new Cell(n);
  val ret_fn = fn(@mut c)
  ret_fn.f = 5
  ret_fn.free
}
def client(){
  withCell(5){ c => (c => {c.f = 1; c})}
}

def client1() {
  var c1 : Cell = null;
  withCell(n){ c => {c1 = c; c } } // error: c does not have the permission
}
\end{lstlisting}
Use the modifier \emph{local} to denote that the object or the function value is owned by the enclosing method.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
def withCell(n: Int)
    (@local fn: @local @mut Cell => (@local @mut Cell => @local @mut Cell): Unit = {
  val c = @local new Cell(n);
  val @local ret_fn = fn(f)
  ret_fn.f = 5
  ret_fn.free
}
def client(){
  withCell(5){ c => (c => {c.f = 1}}
}

def client1() {
  var c1 : Cell = null;            // c1's owner is client1()'s owner
  withCell(n){ c => {c1 = c; c } } // error: c and c1 has different owner
}
\end{lstlisting}

Nested functions are \emph{owned} by their enclosing function.
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
def withCell(n: Int) = {
  val c = new @local Cell(n);
   def is_even(@mut c0: Cell){
     if (c0.f == 0)  true
     else {
       c0.f = c0.f -1; is_odd(c0)
     }
   }
   def is_odd(@mut c0: Cell){
     if (c0.f == 0)  false
     else {
       c0.f = c0.f - 1; is_odd(c0)
     }
   is_even(c);
   free(c)
}
\end{lstlisting}

In Rust, the following is not allowed:
\begin{lstlisting}[language=Scala,basicstyle=\footnotesize\ttfamily]
fn main() = {
  let x = 4;

  fn equal_to_x(z: i32) -> bool {
    z == x
  }

  let y = 4;

  assert!(equal_to_x(y));
}
\end{lstlisting}

\clearpage
\subsection{Properties}
\begin{itemize}
\item Data always has a \emph{unique owner}.
\item At any given time, you can have either one mutable reference or any number of immutable references.
\item References must always be valid.
\item The lifetime of passing references (@mut and @shr) spans the definition (scope) of the function.
\end{itemize}

\subsection{Questions}
%need a way to distinguish mutable variable and mutable heap locations
%
%\begin{lstlisting}[language=Scala]
%val x = new mut Cell // ok, the instance of Cell is mutable
%val x = new Cell     // ok, the instance of Cell is not mutable
%var y = new mut Cell // not ok?
%var y = new Cell     //
%
%\end{lstlisting}

%The \CODE{typeMap} does not have $\&$ information.
%
%I don't see the $\lambda$ node.

\clearpage

% Local Variables:
% word-wrap: 1
% auto-fill-function: nil
% End:


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\graphicspath{ {images/} }
\newcommand{\Mod}[1]{\ (\mathrm{mod}\ #1)}

\title{Sonny: A curve designed to perform high security circuit operations for Elliptic Curve protocols in Zero Knowledge }
\author{
  Carlos Perez\\
  Dusk Foundation\footnote{https://dusk.network/}\\
  \texttt{carlos@dusk.network}
  \and
  Luke Pearson\\
  Dusk Foundation\\
  \texttt{luke@dusk.network}
}
\date{October 2019}

\begin{document}

\maketitle
\thispagestyle{empty}
\pagestyle{empty}

\begin{abstract}

\end{abstract}

\newpage

\tableofcontents

\newpage

\section{Introduction}
The construction and use of elliptic curves is paramount to many cryptographic protocols. Elliptic curves are among the fastest performing primitives where the Discrete Logarithm Problem (DLP) is hard, which is why they are regarded as dominant in the field of cryptography. As the field of cryptography advances, elliptic curves have been proven to be unparalleled in their use for cryptographic systems which have speed and security as two of the most outstanding features. As they are held in such high regard, they can greatly steer the direction of new cryptographic protocols. When outlining new protocols which rely upon elliptic curves, there are a multitude of features which are affected depending on design; often regarded as factors of opportunity cost which leads to compromise on critical features within Elliptic Curve Cryptography (ECC). However, some contemporary techniques can be used to better facilitate systems that rely so heavily upon these primitives. Since many previous protocols are proven to be secure, it is $often$ far more efficient to add to these standards with compound technologies rather than seeking entire system replacements. \\\\ One of the most groundbreaking techniques used in state of the art cryptography is the constructing of Zero Knowledge  (ZK) proofs for nearly universal computation. As is with many cryptographic protocols, there is a choice of which proof system is best tailored to a system. Which includes a trade off, between proof sizes, verification times and other factors making up a ZK proof.  For particular proof systems, their expression relies upon an elliptic curve and an arithmetic circuit. The elliptic curve here is a function used to encode the public outputs which are represented as field elements, upon which a lot of operations rely. The operations for these proofs systems, however, are expressed in terms of a circuit which is determined by scalar curve arithmetic. This unfortunately restricts the operations which can be performed as they are dependent upon standard arithmetic circuit outputs - addition, multiplication, subtraction and division. Many protocols such as Elliptic Curve Digital Signature Algorithms (ECDSA), which are performed using field encodings, are in operable through the medium of generic ZK proofs which are expressed in terms of a circuit. \\\\
\boldsymbol{A} \boldsymbol{profound} \boldsymbol{solution}.
In section 3, we elaborate on how we use our derived curve, which is embedded into the scalar field of Curve25519, in order to extend the circuit operations to those which exceed the traditional bounds of constraint systems. The curve is constructed across a scalar field, as a base field, thus allowing the portability of ZK proofs onto non scalar protocols.\\\\
As with all elliptic curves, their construction will strongly influence the outcomes of the protocols in which they are implemented. In addition to this, there can be discrepancies in both the security and speed of cryptographic systems dependant upon on how they're implemented. For this reason, we wanted to make use of the the fastest complete formulas for elliptic curves, which are Twisted Edwards curves. In conjunction to their complete formulas, Twisted Edwards curves have been proven by Bernstein et al, to be Birationally equivalent to Montgomery curves. Which fit the purpose of the augmented construction, as Montgomery operations in arithmetic circuits were proven by the Z-cash team to provide a fast Montgomery ladder for in circuit multiplication. Whilst these curves models can provide some of the fastest and most simplistic operations, they do provide issues in security. Neither Montgomery, nor Edwards curves deliver prime order groups in their implementations. They provide curves which have a cofactor, $h$, which multiplies the prime of the subgroup to give the group order. Whereas curves like Weierstrass give prime order but their formulas are too inefficient for circuit operations.\\\\
For non prime order curve groups, the mismatch in desirability of prime order curves and inability to implement one directly from the curve can be patched with uniquely tailored modifications.  However, these fixes oft become perplexing to the non-implementors and with higher level protocols they are seldom straightforward. Using curves which provide prime order groups, such as Weierstrass Curves, have slower formulae and are very difficult to implement in constant time. Plentiful curve families allow for the encoding of different related curves for protocol specific purposes. For example, [] library uses Twisted Edwards forms for out of circuit operations like public key generation to exploit the high speed formula but uses Montgomery form for in circuit operations to benefit from the ladder multiplication. If an ad hoc fix needs to be given to each related curve model, then the implementation can become tedious and very complex.\\\\
\boldsymbol{A} \boldsymbol{centric} \boldsymbol{solution}.
In section 4, we explain how to use the Ristretto technique, which constructs prime order Edwards curves from non prime order groups, with our embedded curve, to compress the cofactor such that $h$ = 1. Ristretto makes use of the relationship between curves and provides a fix for the cofactor complication for all models in one place.\\\\

\subsection{$Compatibility$} (isogenies/facility) ristretto - sig - scalar

\section{Notation and Formulae}
\subsection{General notation}
$Finite Field$: $F_p$ is the finite field where the $char$ \neq {2}\vee{3}\\\\
$\varepsilon_{a,d}$ is a Edwards curve, given by the equation: $$ {a}x^2+y^2=1+{d}x^2y^2 $$ where {$d$} and {$ad$} are none square in $F$ and has no points at infinity. In this paper, the primary focus is upon Twisted Edwards curves, where $a = -1$. The identity point of an Edwards curve, $\varepsilon$, where (X,Y) $\epsilon$ in $F_p$, is given encoded to (0,1). When Edwards points are expressed in Extended Twisted coordinates, the identity encoding is given by (X : Y : Z : T) = (0 : 1 : 1 : 0).\\\\
${M}_{a,\frac{2-4d}{a}}$ is a Montgomery curve, given by the equation: $$ y^2=x^3+Ax^2+Bx $$ A Montgomery curve is Birationally equivalent to an Edwards curve - a definition used for algebraic substitution -  where its point at of infinity is the identity point, denoted as (0 : 1 : 0).\\\\
$\jmath_{a^{2},a−{2d}}$ is a Jacobi curve, given by the equation: $$y
^2 = {e}x^4 + 2Ax^2 + 1$$ A Jacobi curve, better known as a Jacobi quartic, is central to all curve models and to utilise this curve relationship we will only be using Jacobi curves where $e = {a}^2$, as such curves have a full 2-torsion point.\\\\
$Torsion\ points$:\ An element [P] in  $G$ is a torsion point if there is a mapping of $M$, by means of multiplication, where $M$ \cdot\ [P] = $0_{G}$. Torquing elements for a curve form a subgroup, $G$[$M$], where the order is divisible by ${M}^2$. The torsion subgroups for this curve family have order 1, 2 or 4.  \\\\
$Isogeny$: An isogeny, $\varphi$ , is a function which maps algebraic groups whilst preserving the group structure. This mapping must satisfy the properties of being surjective and having a finite kernel. The isogeny, in this paper, is used to transport an encoding between different curve models.\\\\
$Curve forms$: $\varepsilon_{a,d}$; ${M}_{a,\frac{2-4d}{a}}$; $\jmath_{a^{2},a-{2d}}$ \\\\ These curve models are all isogenous to one another. The Edwards, Montgomery and Twisted Edwards are independently 2-isogenous to the Jacobi quartic and are therefore 4-isogenous to one another.  \\\\
$Arithmetic$\ $circuits$: These are the computational models for computing circuits. They are universally bound to add and multiply, which are the functions performed at each node on all given inputs. \\\\
$Cofactor$\ $compression$: This refers a quasi-construction of cofactor 1 curves from cofactor 8 groups. Also known as cofactor division, it involves the process of point compression when points of order 4 or 8 are produced.\\

\subsection{Our contributions}
\subsubsection{Elliptic curve}
Here we present an elliptic curve, created for an safe and efficient Elliptic curve operations inside bulletproofs; called Sonny. Sonny is defined as an embedded curve which the gives the input for the proofs and the discrete log based proof is implemented using the outer curve, Curve25519. \\\\

\noindent\fbox{%
    \parbox{\textwidth}{%
    $$ Sonny  $$
        \begin{itemize}
    \item Curve equation in Twisted Edwards form: $$ ax^2+y^2=1-dx^2y^2 $$
    \item $a= -1$
    \item $d= -\frac{126296}{126297}$
    \item $Basepoint: Y = \frac{3}{5}$\\
    \item Montgomery form equivalent: $$ y^2=x^3+Ax^2+x $$
    \item $A = 505186 $
    \item $Basepoint: X = 4$\\
    \item The curve group order, G, is $$ 2^{252}+115924404605461509904689566245241897752 $$
    \item The order of the scalar field, $r$, is $$ 2^{249}+15114490550575682688738086195780655237219 $$       \item The order of the base field, $p$, is  $$ 2^{252} + 27742317777372353535851937790883648493 $$
    \item Cofactor: $$ h =\frac{G}{r} = 8$$
    \item Weierstrass form equivalent: \\  $$y^2=x^3+ax+b $$
    \item $a$ = 7237005577332262213973186563042994240857116359379907606001\\950828033483786813
    \item $b$ = 445582015604702849664
\end{itemize}
    }%
}\\\\

\section{Finite fields}
A finite field is a set of numbers on which arithmetic operations are performed and satisfies a specific set of rules. These operations include multiplication, addition, subtraction and division - notably, the most fundamental operations in ECC. Elliptic Curve arithmetic uses the finite field of integers reduced mod$p$, where $p$ is some prime. The use of finite fields as the extension of elliptic curves is to reinforce the hardness of the DLP, as well as make certain cryptographic assumptions about the order of the set. The choice of finite fields for elliptic curves, known as base curves, provide a cyclic group which gives precise knowledge to the amount of bits that need to be stored by point outputs. As stated, the base fields dictate the operations for the elliptic curves and thus selection of these fields affects the security, speed and simplicity of the implementation of the curve. Supplementary to standard operations performed mod$p$, there are many protocols can be performed in ECC which are implemented using finite sets but do not make use of the base field. They rely upon another prime order field, which is the curves scalar field, also known as the subgroup.  The existing systems like the verification systems are already deployed and because their operations are performed on on the base field, and not a scalar field, they are not directly updateable with a large range of contemporary techniques. This paper and findings focuses on Zero Knowledge  proofs as the 'add-on technology' for existing schemes. \\\\ Elliptic curves have both a base field, which is the finite field in which they are defined; and a scalar field, which is associated with the number of points on the curve. DL based proofs which use a curve and a circuit rely upon both of the finite fields. The base field here is a function used to encode the public coordinates which are represented as field elements. However, as the operations are performed mod$p$, where $p$ is prime,  the outputs are reduced to the prime scalar field. Thus operations which require the base field, cannot be performed inside proof systems which use arithmetic circuits as an expression. As a result the ZK outputs are limited to what circuit operations can be performed by the elliptic curves scalar field. The circuit, in this case, encodes relation between the input and outputs.
\subsection{Efficient ZK for higher operability }
To extend the range of ZK elliptic curves operations to those which employ the base field, we have built a curve which has a base field equal to the scalar field of Curve25519. This is defined in the following manner: Let $\varepsilon_{1}$ and $\varepsilon_{2}$ be elliptic curves. Where the prime subgroup order, or scalar field, of $\varepsilon_{1}$ is $r$; we define $\varepsilon_{2}$ over the base field $F_p$, where \#$F_p$ = $r$.
This will allow us to perform fast in circuit operations using $\varepsilon_{2}$ as the embedded curve within the scalar field of $\varepsilon_{1}$.One particular current issue that embedding curves helps to alleviate is the adding to, or updating of, existing software protocols with privacy techniques so that already deployed systems can benefit from high levels of privacy preservation. By constructing this, we are making a quasi representation of one finite field as both a scalar and a base field. We can therefore encode the field based protocols curve over the scalar field of existing systems - and protocols such as key signature verification, can be performed inside a Zero Knowledge  proof. We present a means of verifying only the scalar operation, in Zero Knowledge , so that Zero Knowledge  proof of statements derived for signature schemes can be proven rather than the signature itself. This is performed by expressing ZK proof of computations as the argument for computational models, such as arithmetic circuits.\\\\
In the case of Zerocaf protocol, we have the outer curve operations, using Curve25519, which implement the ZK proof system, where the operations are performed as integers mod the base field. Then there is Sonny, the inner curve, known as the embedded curve which is the curve we make the proofs about. For the case of signature schemes, like Elliptic Curve Digital Signature Algorithm (ECDSA), the operations for the signature generation are made using Sonny then the Zero Knowledge  arguments for these outputs are generated using Curve25519. By setting the scalar field of Curve25519 as the base field of Sonny, all the operations are efficient when expressed in terms of a circuit. The validation keys here are effectively turned into discrete log proofs, as the generation of ZK values is performed in one amalgamated protocol, even though it comes from two different curves. \\\\
Many software layers require information from the user - just like authentication certificates for websites, where the type of secret keys is known to the website for verification. The information given is often burnt into the hard memory of the website, which can be used by the software owners to discriminate against different keys and brands of keys hence the need to preserve privacy on these existing protocols.
\subsection{Circuits}
An circuit is combinational set of operations which are aligned in a set or series for the ultimate purpose of optimising otherwise standardised mathematical process. The operations, better known as the basic arithmetic operations (addition, subtraction, multiplication, and division), are theoretically performed in constant time. This statement is derived from the fact that the required RAM required is roughly equal for all operations. However, when computing these operations for some large integers, it is apparent that the magnitude greatly affects the costs of RAM. Thus giving a discrepancy for computational time between the theoretical arguments and the practical implementation. When the expression of these arithmetic operations is performed in a circuit, it is referred to as an arithmetic circuit. When expressed for computations within computer systems, any arithmetic circuit is constructed from various combinational elements, which are connected by wires. A combinational element is fixed element which performs a specific function from a constant number of inputs and outputs.
Circuits are used alongside the elliptic curves to construct discrete log based proof systems; when the circuit is defined over the scalar field.

\section{Prime order groups}
'A group of prime order' is always a cyclic group, that has a mapping - which respects the group structure - to the quotient of the group of integers by a subgroup. This subgroup is generated by a prime number. Groups of prime order are often a prerequisite to crytpographic prototcols, as they provide the basis for a hard DLP and thus increased security for implementation. For implementation, we have made efficiency the most paramount factor for curve selection, which led to us choosing a Twisted Edwards curve form. This is because the Edwards forms of the curves provide the fastest known formulas, which can be accredited to extended Twisted Edwards, introduced by Hysil et al, where auxiliary points are used with fewer field inversions. As elliptic curves are Abelian groups, they provide varying order for their respective groups. Edwards curves and their birational Montgomery equivalents, provide 'not quite' prime order groups over fields - the absence of prime groups can lead to timing variations when implementing protocols such as the signature schemes Sonny implements. Instead of certain Elliptic curve groups being prime, they have a cofactor $h$, meaning that $h \cdot q$ is the group order, where $h > 1$ and $q$ is a large prime. Having this property where $h > 1$ can lead to many implementation complexities.\\\\
There are cofactor relates attacks designed to extract information, in the form of bits, about a users private key. When generating a public key, it is ideally performed using a point operation on a given curve point, where a chosen scalar outputs a new point, modulo the base field. This provides a public key, from which the scalar, known as the private key, cannot be extracted. However, if points on the curve are selected by attackers to have order which divides $h$, then presented as valid curve points, they can be mistakenly used by a user. If an incognizant user generates a public key by inputting a secret scalar to a function which operates with points of order $h$, then the attacker can gain some bits about the input scalar. Whereas within a prime order group, there is no means of generating valid points which have order dividing $h$. The abstraction of having non prime order groups can be solved with specialised modifications towards individual protocols. One notorious method is to multiply points by the cofactor and check the result; if the resulting point is the identity point then it can be discarded. Many of the individual techniques produce continuous and substantial flaws, especially with regards to patchwork comprehension, which occurs when the protocols are being implemented by those who did not design them, i.e. Implementors not knowing at which step to multiply by $h$.
\subsection{Cofactor compression}
There are various advantages and disadvantages to having a cofactor larger than one, therefore a thorough analysis must be performed,  so that it is known whether or not cofactor manipulation is needed. For all curves, except for Hessian curves, the cofactor is divisible by 4. To become more useful to a broad spectrum of cryptography, Ristretto is apt for a large number of curves, which have a cofactor of 8 or 4. When the cofactor is greater than 1 multiple operations can be hindered. A quotient group can be constructed to allow for the implementation of prime order groups, thus effectively compressing the cofactor, by applying the Ristretto technique. This technique requires just one additional step to Mike Hamburgs decaf proposal for cofactor-4 curves. The technique works using following four functions:\\\\
${Equality}$ ${testing}$ This function checks the equality of group elements. \\\\
${Encode}$\ The encoding function is applied to an Edwards point and this becomes the internal representation for the new "Ristretto point', meaning the same Edwards point operations are performed on the Ristretto point, and with no overhead cost. The function encodes the elements as byte strings so that that the Ristretto elements can be encoded identically.\\\\
${Decode}$\ This function decodes the byte strings into the internal representations of Ristretto points. There is also a validity check which assesses the canonical representation of points, and only accepts those which are outputs of the encoding function. \\\\
${Curve}$ ${hashing}$\ For many protocols, mapping elements in a group to a curve is done by a hash function, as it provides standardised digests which can be encoded. Ristretto using an Elligator 2, which gives a 1:1 mapping of group elements to the curve. Elliga

\subsection{Isogenies}
By using the Ristretto technique, we are able to solve all cofactor related issues in one place and with one step. This is facilitated by its use in the relationship of the curves, and how this lets us transport the cofactor compression for curves, via the isogeny, to another curve in the same family. Which in turn means we work with prime order points in any operations of ECC. Otherwise, the implementation would have to deal with the issue at varying stages which is dependent upon a protocols ultimate design. An isogeny is a function which maps one algebraic group to another, whilst maintaining the structure of the group - which in terms of elliptic curves, means that a curve is allowed curve to take on the values of another and preserve the same point addition method. These functions are non-constant and are used as a tool for effective 'transportation' between curve models. Just as with all concrete mathematical formulae, these functions have a domain and co-domain, which means that given these two, it is possible to compute the function itself. In this document isogenies will be given the generalization as the multiplication by $m$ map, where they have a finite kernel and are restricted to rational mappings.
A deeper understanding of isogenies for elliptic curves has greatly advanced the field of ECC, as it is possible to deduce one mapping from the form of another. Additionally, if these relationships are well understood then they can be applied or integrated into other functions and broaden their domain of propriety to more use cases.\\\\
When there exists a non-constant function, $\varphi$, which gives a rational mapping from one group to another denoted as $\varphi$ : $\varepsilon$ $\rightarrow$ $\varepsilon\prime$. This mapping from $\varepsilon$ to $\varepsilon\prime$ has degree $n$. Where there is this separable isogeny, then exists a mapping from $\varepsilon$ to $\varepsilon\prime$, which is known as the dual isogeny, $\hat{\varphi}$, where both functions have degree $n$. The dual isogeny is conveyed as $\hat{\varphi}$ : $\varepsilon\prime$ $\rightarrow$ $\varepsilon$ of degree $n$. The isogeny $\hat{\varphi}$ here is known as the dual of $\varphi$, such that $\hat{\varphi}$ $\circ$ $\varphi$ is the multiplication by $n$, where $n$ = $\hat{\varphi}$ $\circ$ $\varphi$, from $\varepsilon$ to $\varepsilon\prime$. This dual isogeny has certain properties which allow for the two way transportation of functions between curves. These can be exploited to provide abstraction of protocols where it would otherwise be inapplicable.

\subsubsection {Curve mappings}
As curve models $\varepsilon_{a,d}$; ${M}_{a,\frac{2-4d}{a}}$; $\jmath_{a^{2},a-{2d}}$, have different implementation features we can utilise these relationships to achieve the implementation we desire, namely a prime order curve. It is possible to construct prime order curves using the Montgomery and Edwards curve forms by transporting encoding to and from the Jacobi quartic, via isogenies. The functions for cofactor compression would typically be used on the Jacobi quartic form, by means of canonically selecting outputs for curve points. Now this selection process can be applied to the Edwards and Montgomery form by integrating it to the function which maps between them. \\\\

\subsubsection{Encoding types}
\section{Zerocaf}
As is previously explained our curve has a base field which is the scalar field of Curve25519, to allow for the use of efficient Zero Knowledge  on operations within a circuit. the difficulty of breaking cryptographic systems stems solely from the hardness of the mathematical problems on which they are based. However, this proves not to be the case in practical implementations because of side channel attacks, which target the implementation as medium of encoding the cryptography - to circumvent these attacks, the operations need to be performed in constant time. The use of Edwards curves results in a uniform implementation which better facilitates these constant time operations.  The Edwards form of a curve is considered complete, as any two inputs, given as x and y, provide a correct result. Whereas in circuit operations can greater benefit from variable time implementations, as they perform faster. They can be applied when there is no secret data to protect, as they may lead to leakage of data. We therefore present an implementation which performs statement proofs in constant time with high security, and verification in variable time and high speed. \\

\section{Future work}
R1CS optimisation for constraints
Further isogeny use cases

\section{Conclusion}

\section{Acknowledgements}
We would like to give special thanks to Henry de Valence for his personalised help in understanding the Ristretto Protocol and being so responsive for questions regarding the implementation. We would also like to show our strong appreciation for Marta Bellés Muñoz for her contributions with the discrete log based theory used in the understanding of this project.

\newpage

\begin{thebibliography}{99}

\bibitem{c1} Stanford University, University College London and BlockStream, Benedikt Bünz, Johnathan Bootle, Dan Boneh, Andrew Polestra, Pieter Wuille and Greg Maxwell. Bulletproofs: Short Proofs for Confidential Transactions and More.\\ https://eprint.iacr.org/2017/1066.pdf
\bibitem{c2} Shafi Goldwasser, Silvio Micali, and Charles Rackoff. The knowledge complexity of interactive
proof-systems (extended abstract). In 17th Annual ACM Symposium on Theory of Computing
(STOC’85), pages 291–304, 1985."
\bibitem{c3} Pedersen T.P. (1992) Non-Interactive and Information-Theoretic Secure Verifiable Secret Sharing. In: Feigenbaum J. (eds) Advances in Cryptology — CRYPTO ’91. CRYPTO 1991. Lecture Notes in Computer Science, vol 576. Springer, Berlin, Heidelberg
\bibitem{c4} Isis Lovecruft and Henry de Valence. Ristretto. https://Ristretto.group/Ristretto.html
\bibitem{c5} Mike Hamburg : Deacaf. November, 2015. https://eprint.iacr.org/2015/673.pdf
\bibitem{c6} Feng Hao, Thales E-Security, Cambridge, UK https://eprint.iacr.org/2010/149.pdf
\bibitem{c7}Robert Dijkgraaf: Mirror Symmetry and Elliptic Curves, university of Amsterdam, November 15, 2002
\bibitem{c8} Tehcnological University of Visvesvaraya, Jnana Sangama https://www.academia.edu/8777556/

\end{thebibliography}

\end{document}


\section{The Trust Graph}
  We now engage in the formal description of the proposed system, accompanied by helpful examples.
  \subimport{common/definitions/}{gamegraph.tex}
  \noindent The nodes represent the players, the edges represent the existing direct trusts and the weights represent the
  amount of value attached to the corresponding direct trust. As we will see, the game evolves in turns. The subscript of the
  graph represents the corresponding turn.
  \subimport{common/definitions/}{players.tex}
  \noindent Each node has a corresponding non-negative number that represents its capital. A node's capital is the total value
  that the node possesses exclusively and nobody else can spend.
  \subimport{common/definitions/}{capital.tex}
  \noindent The capital is the value that exists in the game but is not shared with trusted parties. The capital of $A$ can be
  modified only during her turns, according to her actions. No capital can be added in the course of the game through external
  means.

  We also define a player's assets to be the sum of her capital and her total outgoing trust.

  \subimport{thesis/definitions/}{assets.tex}
  \noindent We consider the outgoing trust of a player as part of her assets. The formal definition of direct trust follows:
  \subimport{common/definitions/}{directtrust.tex}
  \noindent This definition agrees with the title of this paper and coincides with the intuition and sociological experimental
  results of \cite{kmrs} that the trust $Alice$ shows to $Bob$ in real-world social networks corresponds with the extent of
  danger in which $Alice$ is willing and able to expose herself to in order to help $Bob$. An example graph with its
  corresponding transactions in the UTXO can be seen below.

  \subimport{common/figures/}{utxo.tikz}

  \noindent Any algorithm that has access to the graph $\mathcal{G}_j$ has implicitly access to all direct trusts of this
  graph.  We use the notation $N^{+}(A)$ to refer to the nodes directly trusted by $A$ and $N^{-}(A)$ for the nodes that
  directly trust $A$. We also use the notation $in_{A, j}, out_{A, j}$ to refer to the total incoming and outgoing direct
  trust respectively. For a reference of common definitions, see the Appendix.


\section{Inter Rater Agreement}
\label{chp:fundamentals:sec:inter_rater_agreement}
Many \ac{ML} models require a dataset which is used to train them.
This data is often generated by multiple raters which assign a label to each datapoint.
The used dataset directly influences the \ac{ML} model \parencite{Gray:2011} and therefore, well designed research studies must include mechanisms to capture \ac{IRA} \parencite{McHugh:2012}.

This section gives an introduction to different \ac{IRA} metrics and their applications.

\input{chapters/02_fundamentals/sections/inter_rater_agreement/subsections/cohens_kappa}
\input{chapters/02_fundamentals/sections/inter_rater_agreement/subsections/scotts_pi}
\input{chapters/02_fundamentals/sections/inter_rater_agreement/subsections/fleiss_kappa}
\input{chapters/02_fundamentals/sections/inter_rater_agreement/subsections/free_marginal_multirater_kappa}


\subsection{VAR impulse response}


\section*{Materials and Methods}
\label{sec:M&M}

\subsection*{Data Provenance}

\subsubsection*{Growth Data}
The first data set is an inventory of all trees over 10cm in Diameter at Breast Height (DBH), i.e. measured at 1.3m high, in nine 1-ha plots in French Guiana (see map~\autoref{fig:map}). In each plot, tree diameters were measured every two or five years depending on the plot.

We selected a common measured period between 2001 and 2013 comprising a total of 3549 trees; we estimated annual growth rate (AGR) in diameter by fitting a linear regression of DBH over years. The slope of the regression gave us an average AGR for each followed tree on the comprised

\subsubsection*{Trait Data}

The second data set was a collection of five functional traits (see~\autoref{tab:seltraits}) extracted from a bigger database \citep{baraloto_decoupled_2010} on the same trees. Selected traits can be seen in two orthogonal axes of variations: leaf and stem economics spectra, contrasting different ecological strategies~\citep{westoby_leaf-height-seed_1998, baraloto_decoupled_2010}.

\textbf{Leaf economics spectrum.} Specific Leaf Area (SLA) is the photo-sensitive area per unit of dry mass of the leaf; high SLA underlines investment on high light-capturing leaves that have a short payback time per gram of dry matter invested; while low SLA reflects strategies with less light-capturing leaves and longer payback time that may appear competitive in some conditions. Total leaf chlorophyll content reflects the global strategy of the plant of having resource-expansive leaves with high payback or resource-cheap leaves with lower payback~\citep{coste_assessing_2010}. Laminar toughness measures the resistance of a leaf to pinching, high toughness values correlates with low herbivory rate, it correlates with defense strategy~\citep{westoby_leaf-height-seed_1998}.

\textbf{Stem economics spectrum.} Wood density underlines different ecological strategy for trees, a low wood density makes wood less stable and less better protected against herbivory but cheap volumetric construction cost because of low resource requirements; while a high wood density makes the tree more stable but with higher construction cost, meaning a lower growth~\citep{chave_towards_2009}. Trunk bark thickness associate with defense strategies in neotropical forests, thicker bark provides higher resistance to pathogens and herbivores~\citep{paine_functional_2010}.

\subsection*{Statistical analyses}

\subsubsection*{How are intra-specific variabilities structured compared to inter-specific variabilities?}

To understand how inter-specific variability contrasted with intra-specific variability, we partitioned the variance of each trait and performance using ANalyses Of VAriances (ANOVAs) with a species term as follow:
\begin{subequations}
	\begin{align}
	\label{eq:anovatrait}
	\text{Tr}_{s, i} &= \mu_s + \epsilon_i\\
	\label{eq:anovaagr}
	\text{AGR}_{s, i} &= \overline{\text{AGR}_{s}} + \epsilon'_i,
	\end{align}
\end{subequations}
with $\text{Tr}_{s, i}$, the trait of individual $i$ of species $s$; $\mu_s$ the mean trait of species $s$; $\epsilon_i$ the individual trait error term with a Gaussian distribution; $\text{AGR}_{s, i}$ the AGR of individual $i$ of species $s$; $\overline{\text{AGR}_{s}}$ the average AGR of species $s$; $\epsilon'_i$ the individual AGR error term and $\epsilon_i \sim \mathcal{N}(0, \sigma^2), \epsilon'_i \sim \mathcal{N}(0, \sigma'^2)$. The explained variance by the species effect can then be expressed by the proportion of group sum of squares over the total sum of squares. We considered the residual sum of squares as being the individual variance plus white noise. We partitioned the variance similarly for AGR.

\subsubsection*{How does trait intra-specific variability influence radial growth?}

In order to understand the influence of trait intra-specific variability on radial growth, we modeled AGR with a mixed linear model of traits and other factors. Based on~\citet{herault_functional_2011} ontogenical model, we used fixed terms $DBH$ and $\log(DBH)$, because they capture well the hump shape of growth during ontogeny for tropical trees. As they did, we modeled $\log(AGR + 1)$ because of data high heteroscedasticity. We added a random intercept for species effect to take inter-specific variability into account, this effect also influenced the slopes of $DBH$ and $\log(DBH)$ terms. We also added a random intercept plot effect to take inter-plot variability into account. Then depending on the hypothesis tested various fixed terms were added to the model:

\begin{align}
	\label{eq:growth_mod}
	\log(\text{AGR}_{p, s, i} + 1) =& \prescript{0}{}\theta + \prescript{0}{}\gamma_s + \gamma_p \nonumber \\
		&+ (\prescript{1}{}\theta + \prescript{1}{}\gamma_s) \times \text{DBH} \nonumber \\
		&+ (\prescript{2}{}\theta + \prescript{2}{}\gamma_s) \times \log(\text{DBH}) \nonumber \\
		&+ \delta + \epsilon_i,
\end{align}
with $\epsilon_i \sim \mathcal{N}(0, \prescript{3}{}\theta)$ the individual residual,
where $\text{AGR}_{p, s, i}$ is the AGR of tree $i$ of species $s$ in plot $p$; $\prescript{0}{}\theta \dots \prescript{3}{}\theta$ are parameters to be estimated; $\prescript{0}{}\gamma_{s} \dots \prescript{2}{}\gamma_s$ and $\gamma_p$ follow a zero-centered Gaussian distribution with unknown variances $\prescript{0}{}\sigma^2_s \dots \prescript{2}{}\sigma^2_s$ and $\sigma^2_p$. $\text{Tr}_s$ is the average trait value for species $s$.

We tested different hypotheses through the $\delta$ term in~\autoref{eq:growth_mod}. If we position an individual relative to its species mean, we can compare them on a hierarchy of traits, and the relevant variable is the hierarchical distance between individual trait and species average trait. We may also hypothesized as~\citet{kunstler_competitive_2012} did for competition, that the relevant relation for AGR would be the absolute distance. We obtain two models that have to be compared with models having only the specific trait or the individual trait.
\begin{equation}
\label{eq:delta}
\delta = \left\{
	\begin{array}{lll}
		\prescript{4}{}\theta\times \text{Tr}_s & \text{Species Average} &  \\
		\prescript{4}{}\theta \times \text{Tr}_s & + \prescript{5}{}\theta \times(\text{Tr}_i - \text{Tr}_s) & \text{Hierarchical Dist.} \\
		\prescript{4}{}\theta \times \text{Tr}_s & + \prescript{5}{}\theta' \times \vert \text{Tr}_i - \text{Tr}_s \vert & \text{Absolute Dist.} \\
		\prescript{4}{}\theta' \times \text{Tr}_i & \text{Individual Trait} & \\
	\end{array}
\right.
\end{equation}

 $\text{Tr}_s$ the species average value, with a parameter $\prescript{4}{}\theta$; $\text{Tr}_i - \text{Tr}_s$ the hierarchical distance of individual trait value $\text{Tr}_i$ to species average trait value $\text{Tr}_s$; or $\vert \text{Tr}_i - \text{Tr}_s \vert$ the absolute distance to species average trait; and $\text{Tr}_i$ the individual trait value.

\subsubsection*{Intra-specific variability vs. inter-specific variability effects on performance for each trait}

Intra-specific variability and inter-specific variaibilty may have different impacts on performance. For each trait we selected the growth model from~\autoref{eq:delta} that had the highest adapted R-squared for mixed models~\citep{nakagawa_general_2013}, i.e. the model that best described our data. Using them we predicted AGR based on a 5\%-95\% range of species average and hierarchical distance.

\subsection*{Data analysis}

All data analyses were made using \texttt{R} \citep{R_language} version 3.2.0 (2015-04-16), plots were made with \texttt{ggplot2} \citep{ggplot2_pkg}. We fit mixed-models with \texttt{lme4} \texttt{R} package \citep{lme4_pkg} 1.1-7 and computed adapted R-squared for mixed-models \citep{nakagawa_general_2013} implemented in \texttt{MuMIn} \texttt{R} package \citep{mumin_pkg} version 1.13.4.

\begin{table*}
	\begin{center}
		\begin{tabular}{lccc}
		\hline
		Trait Name & Units & Role \\
		\hline
		Trunk bark thickness & mm & Stem economics \\
		Xylem density (wood density) & $\text{g}.\text{cm}^{-3}$ & Stem economics \\
		Specific Leaf Area (SLA) & $\text{cm}^2.\text{g}^{-1}$ & Leaf economics \\
		Laminar total chlorophyll & $\text{µm}.\text{mm}^{-2}$  & Leaf economics \\
		Laminar toughness & N & Leaf economics \\
		\hline
		\end{tabular}
		\caption{\textbf{Selected functional traits.} Stem and Leaf Economics Spectrum are defined as in~\citep{baraloto_decoupled_2010}, the two axes unravel distinct ecological strategies. Leaf economics spectrum contrasts}
		\label{tab:seltraits}
	\end{center}
\end{table*}


%\begin{center}SECTION\end{center}

\chapter{CONCLUSIONS}

\section{CHARACTERIZATION OF TRANSPORT REGIME TRANSITIONS IN NON-CONSERVATIVE RANDOM MEDIA}

The purpose of the first portion of this dissertation was to characterize the effect of absorption and gain in ballistic, diffusive, and localized transport regimes. Particle and wave-based transport models were studied in one dimension (1D) and quasi-1D geometries.

An investigation of the ratio of transmission to energy in the system, $T/{\cal E}$, was performed in 1D using theoretical and numerical methods~\cite{2010_Payne_TE}. %~(c.f.~Section \ref{sec:TE2009}).
The numerical model uses the transfer matrix method~\cite{1981_MacKinnon_scaling} with self-embedding~\cite{1999_yamilov_selfembed} to simulate layers of dielectric material with random widths. Since diffusion cannot occur in 1D, the response of the parameter $T/{\cal E}$ in the regime of Anderson localization when gain is present was found. A decrease of $T/{\cal E}$ from the value given by the classical un-renormalized diffusion coefficient may be attributed to wave-interference localization effects. Although $T/{\cal E}$ does not diverge as the random lasing threshold is approached, there is a dependence on the position of the center of localization. This position dependence is closely related to the existance of necklace states~\cite{1987_Pendry}.

To investigate the transition from diffusion to Anderson localization, the previous numerical model was extended to quasi-1D geometry. To guide our efforts we developed a phase space diagram with 15 different transport regimes~\cite{2010_Yamilov_Regimes}. A way of characterizing which regime a given system is in was clearly needed. In the process of using the numerical model in this geometry, we showed that evanescant channels do not need to be included in simulations of passive media~\cite{2010_Payne_closed}. The effect of evanescant channels is renormalize the transport mean free path while conforming to single parameter scaling.

A parameter related to $T/{\cal E}$, the position dependent diffusion coefficient $D(z)$, was investigated for use in characterizing the multiple transport regimes in quasi-1D non-conservative random media~\cite{2010_Payne_PRL}. Our results indicate that $D(z)$ may serve as a useful criterion for the enumerated transport regimes.

\section{ANALYSIS OF DETERMINISTIC APERIODIC STRUCTURES}

Although random media produces novel features, these systems lack easy reproducibility. For applications such as photonic integrated circuits we are interested in novel features unavailable to periodic media while maintaining reproducibility. Thus media correlated disorder is a natural avenue of pursuit. The numerical model used in this dissertation can simulate any arrangement of scatters, making it amenable to aperiodic media. %The same simulation tool used for random media was applied to aperiodic media.

In the studies of deterministic aperiodic structures (DAS) we focused on the Thue-Morse pattern. This generation algorithm yeilds a singular continuous Fourier transform spectrum with self-similar features. We demonstrated the possibility of mapping the array of micro-cavities in the two dimensional (2D) Thue Morse DAS onto a periodic square lattice~\cite{2012_Payne_Mapping_2D_TM}. Such mapping allowed us to uniquely identify and enumerate the configurations of nearest and next-nearest neighbors. Thus the original aperiodic structure is reduced to the periodic structure with aperiodic arrangement of the limited set of pairings.

Once this step was completed, we demonstrated the applicability of the tight binding approach in a deterministic aperiodic array of photonic micro-cavities. Under realistic conditions, we observed hybridization of the modes of individual micro-cavities into the eigenstates of the entire array. Our work adds the tight binding approach to the arsenal of theoretical tools for studying of 2D~Thue-Morse structures as well as for design and analysis of experiments.

The tight binding model allows us to investigate the size scaling of the density of the optical states in large arrays of optical micro-cavities; monitor the evolution of the spectra; and to study spatial properties of the eigenstates via e.g. the inverse participation ratio. The inverse participation ratio shows coexistence of localized and extended states in the same spectral regions. Some of the extended states have nearly constant intensity across the entire sample. This property makes the considered system extremely promising for practical applications in optical control of light propagation via e.g. wave-front shaping.

%\section{TEST OF BORDERS}

%Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Ut purus elit, vestibulum ut, placerat ac, adipiscing vitae, felis. Curabitur dictum gravida mauris. Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna. Donec vehicula augue eu neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra metus rhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretium quis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Curabitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissim rutrum. Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbi auctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et, tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna, vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus et magnis dis par turient montes, nascetur ridiculus mus. Aliquam tincidunt urna. Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Ut purus elit, vestibulum ut, placerat ac, adipiscing vitae, felis. Curabitur dictum gravida mauris. Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna. Donec vehicula augue eu neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra metus rhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretium quis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Curabitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissim rutrum. Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbi auctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et, tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna, vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus et magnis dis par turient montes, nascetur ridiculus mus. Aliquam tincidunt urna. Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Ut purus elit, vestibulum ut, placerat ac, adipiscing vitae, felis. Curabitur dictum gravida mauris. Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna. Donec vehicula augue eu neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra metus rhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretium quis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Curabitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissim rutrum. Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbi auctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et, tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna, vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus et magnis dis par turient montes, nascetur ridiculus mus. Aliquam tincidunt urna. Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Ut purus elit, vestibulum ut, placerat ac, adipiscing vitae, felis. Curabitur dictum gravida mauris. Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna. Donec vehicula augue eu neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra metus rhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretium quis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Curabitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissim rutrum. Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbi auctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et, tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna, vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus et magnis dis par turient montes, nascetur ridiculus mus. Aliquam tincidunt urna. Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris.


\documentclass{article}

\setlength{\topmargin}{-0.75in}
\setlength{\textheight}{9in}
\setlength{\textwidth}{6.5in}
\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}

\begin{document}

\title{Energy Prediction and Adaptation in flux-e}
\author{Jacob Sorber\\Matt Brennan}
\maketitle

\section{Goal and Premise}
Given an energy buffer and source, our obvious goal is to never deplete our resources. Additionally, we want our energy state to remain as stable as possible, as this will give the user the most predictable system response. \\
We assume that there is periodicity in both the system load and the energy harvesting.  We divide this period in to epochs, and make our state adjustments on these boundaries.

\section{Definitions and Data Structures}
\begin{eqnarray*}
{paths\_trig} & \equiv & \mbox{set of triggered paths}\\
{paths\_timed} & \equiv & \mbox{set of timed paths}\\
{period} & \equiv & \mbox{length of time over which load and supply repeats}\\
{epoch} & \equiv & \mbox{division of a period for which statistics are separately}\\
{states} & \equiv & \mbox{set of possible energy states} \\
\delta_c & \equiv & \mbox{cost prediction adjustment dampening factor} \in(0,1)\\
\delta_r & \equiv & \mbox{revenue prediction adjustment dampening factor} \in(0,1)\\
\delta_b & \equiv & \mbox{buffer adjustment dampening factor} \in(0,1)\\
{cost\_actual} & \equiv  & \mbox{actual energy spent during previous epoch}\\
{cost\_pred}   & \equiv  & \mbox{predicted energy cost for previous epoch}\\
{rev\_actual} & \equiv  & \mbox{actual energy acquisition during previous epoch}\\
{rev\_pred}   & \equiv  & \mbox{energy acquisition predicted for previous epoch}\\
{timer\_level} & \equiv & \mbox{percentage of timer range selected} \\
{freq}[{path}][{state}] & \equiv  &\mbox{probability of taking a given path in a given state}\\
{load}[{epoch}] & \equiv & \mbox{average of loads during given epoch}\\
{energy\_in}[{epoch}] & \equiv & \mbox{average energy revenue during given epoch}\\
{cost\_static} & \equiv & \mbox{minimum energy cost of an epoch (ie, when idle)}\\
{cost}[{path}] & \equiv & \mbox{average energy cost of a given path, in addition to }{cost\_static}\\
{buffer\_level} & \equiv & \mbox{the percent of the buffer's capacity currently in use}\\
{buffer\_goal} & \equiv & \mbox{the percent of the buffer's capacity to aim for}\\
{scale\_cost} & \equiv & \mbox{a scaling factor to the stochastic energy consumption}\\
{scale\_rev} & \equiv & \mbox{a scaling factor to the energy revenue}\\
\end{eqnarray*}

\section{State Selection}

At the beginning of each ${epoch}$, we predict the base energy cost and revenue for the next ${period}$ (that is to say, the next ${NUM\_EPOCHS}$).  Base energy cost is calculated based on the probability of taking a triggered path, and assuming that timed paths are taken at the minimum frequency allowed for the state. A goal is set based on the expected revenue and the ${buffer\_level}$.  We then select the highest state which will have a cost lower than our goal.

Timer periods remain unselected.  To determine them, we calculate the cost for the same state with maximum frequency for all the active timers.  We find where our goal lies between the minimum and maximum costs, and scale the timers proportionally.

More precisely,

\begin{eqnarray*}
{scale\_rev} & = & 1 + \left(\delta_r\frac{{rev\_actual}-{rev\_pred}}{{rev\_pred}}\right) \\
{scale\_cost} & =&  1 + \left(\delta_c\frac{{cost\_actual}-{cost\_pred}}{{cost\_pred}}\right) \\
{predicted\_cost}_{min}[{state}]&  =&  \sum_{e\in{epoch}}\sum_{p\in{paths\_trig}}\left[ {freq}[p][{state}] \cdot {load}[e] \cdot {cost}[p] \right] \cdot {scale\_cost} +\\
                          &  &  \sum_{e\in{epoch}}\sum_{p\in{paths\_timed}}{cost}[p]\cdot1/{max\_period}_p \\\\
{predicted\_cost}_{max}[{state}]&  =&  \sum_{e\in{epoch}}\sum_{p\in{paths\_trig}}\left[ {freq}[p][{state}] \cdot {load}[e] \cdot {cost}[p] \right] \cdot {scale\_cost} +\\
                          &  &   \sum_{e\in{epoch}}\sum_{p\in{paths\_timed}}{cost}[p]\cdot1/{min\_period}_p \\\\
{predicted\_revenue}&  =&  \sum_{e\in{epoch}} {energy\_in}[e] \cdot {scale\_rev}\\
{goal} & = & {predicted\_revenue} \cdot \delta_b\left(({buffer\_level} - {buffer\_goal}\cdot{buffer\_capacity}\right)
\\
\end{eqnarray*}
\ttfamily
${state}$ = ${MAX\_STATE}$ ;\\
while (${predicted\_cost}_{min} < {goal}$) \\
\hspace{1cm}${state}$--;\\
\({timer\_level} = \frac{{goal} - {predicted\_cost}_{min}}{{predicted\_cost}_{min} - {goal}}\)

\rmfamily

\section{Path Cost Measurement}

Given the energy flow through the buffer (implemented as a hardware fuel gauge), and the total consumption of the system (implemented as a software fuel gauge), we are able to keep track of the power being consumed during execution of paths.  We allow for the possibility of parallel path execution by attributing equal parts of the total power consumption to each active path.  Acknowledging that this method may result in skewed values of ${cost}[{path}]$, we argue probabilistically that this will have nominal effect on our overall energy prediction.  If two paths with large energy consumption differences have a high enough probability of running in parallel enough to skew each other's results, that same high probability infers that it is unnecessary to distinguish between their values.

\end{document}


\documentclass[10pt,usepdftitle=false,aspectratio=169]{beamer}

\input{preamble_talk}
\renewcommand{\L}{\mathcal{L}}
\setbeameroption{show notes on second screen = right}
\renewcommand{\figurewidth}{\textwidth / 2}

\usetikzlibrary{external}
\tikzexternalize[mode=list and make]
% use  make -j 8 -f talk.makefile to compile with 8 parallel threads (this is what it takes to max out the machine, depsite it having 4 cores)
%\tikzset{external/force remake=true}
\tikzsetexternalprefix{fig/external/}

\begin{document}

	\tikzexternaldisable
	\begin{frame}
	\title{{\bf Bachelor Thesis}\newline Investigating Probabilistic Preconditioning on Deep Learning using DeepOBS
		\vspace*{-.7cm}}
	\author{Ludwig Bald\vspace{-1cm}} \date{T\"ubingen\\\today}

	\vspace{-1.5cm}
	\maketitle
	\vspace{-1.0cm}

	\thispagestyle{empty}
	\setcounter{framenumber}{0}

	\tikzifexternalizing{}{%
		\begin{tikzpicture}[remember picture,overlay]
		\node[anchor=south,yshift=-5mm] at (current page.south)
		{\includegraphics[width=0.9995\paperwidth]{assets/logo_TU_169_0.pdf}};
		\end{tikzpicture}%
	}%

\end{frame}
%\tikzexternalenable
\note[itemize]{
	\item Introduce myself, I study Cognitive Science
	\item I chose this topic because I wanted to get to know deep learning from the inside. I had never done Deep Learning before and wanted to learn it hands-on.
	\item In this talk I will talk about the science, but also about the process I used.
	\item If you have questions during the talk, ask them right away!
}

\setlength{\figwidth}{.9\textwidth}
\setlength{\figheight}{\textwidth /2 }

\section{Background}

\begin{frame}
\frametitle{Machine Learning}
\framesubtitle{Background}
	\begin{block}{Definition \footnote{Tom Mitchell, "Machine Learning", 1997}}
		"A computer program is said to learn from \emph{experience E} with respect to some class of \emph{tasks T} and \emph{performance measure P} if its performance at tasks in T, as measured by P, improves with experience E"
	\end{block}
	\begin{itemize}
		\item Special case in Deep Learning: Finding the \emph{optimal point} in \emph{parameter space}
		\item The \emph{Loss function $\L (w)$} tells us how bad a specific parametrization $w$ is. For example, the model's predictions for all possible inputs are compared to the desired outputs.
	\end{itemize}
\end{frame}
\note[itemize]{
	\item Most of you will have seen this definition before.
	\item As an example, have a look at this 2-parametrical problem
}

\begin{frame}[label=landscape]{}{ }
\begin{columns}
	\begin{column}{0.5 \linewidth}
		\include{images/original_prob}
	\end{column}
	\begin{column}{0.5 \linewidth}
		\include{images/rescaled_prob}
	\end{column}
\end{columns}
\end{frame}

\begin{frame}
\frametitle{Gradient Descent}
\framesubtitle{Optimization}
	Gradient Descent uses knowledge of the \emph{gradient} at a point in parameter space to take an update step:
		$$ w_{i+1} = w_i - \alpha \cdot \nabla \L(w_i) $$
	where $\alpha$ is the learning rate.

\end{frame}

\begin{frame}
\frametitle{Stochastic Gradient Descent}
\framesubtitle{Optimization}
\begin{itemize}
	\item In real life, we have Big Data. The true $\nabla \L(w)$ is expensive to compute.
	\item To speed things up, we compute the noisy estimate $\hat{\L}(w_i)$ on a minibatch of for example 128 data points.
\end{itemize}
The update rule still looks the same:
$$w_{i+1} = w_i - \alpha \cdot \nabla \hat{\L}(w_i) $$
where $\alpha$ is the learning rate.

\end{frame}

\againframe{landscape}

\begin{frame}
\frametitle{Preconditioning}
\framesubtitle{The condition number of the Hessian}
	\begin{itemize}
		\item The performance of (S)GD depends heavily on the shape of the loss landscape
		\item The \emph{condition number} is defined as $$\kappa = \frac{\lambda_n}{\lambda_1} > 1$$
			 where $\lambda_n, \lambda_1$ are the largest/smallest eigenvalues of the Hessian $\nabla \nabla \L(w)$
		\item For larger $\kappa$, (S)GD can converge slower.
		\item The condition number can be changed by carefully rescaling the gradient before taking the optimization step
		%TODO example image!
	\end{itemize}
\end{frame}

\againframe{landscape}

\begin{frame}{Probabilistic Preconditioning}{by Filip \& Philipp, 2019}
In the stochastic (minibatched) setting and while only having access to \emph{Hessian-vector products}, it isn't obvious how to construct the preconditioner. This is the method I'm testing:
\begin{enumerate}
	\item Empirically construct a prior for the multivariate Gaussian distribution and set the learning rate for SGD
	\item Gather observations and update the posterior estimate for the Hessian, using Bayes
	\item Create a rank-2 approximation of the Preconditioner
	\item apply the preconditioner at every step and do SGD
\end{enumerate}
\end{frame}
\note{If I'm grossly misrepresenting the algorithm, please correct me now! For an exact description check out the paper}

\begin{frame}{Deep learning}{Neural nets}
	For the purposes of this talk, a neural net is a model
	\begin{itemize}
		\item with many ( $>$ hundreds of thousands) parameters, weights $w$
		\item with an available noisy gradient $\nabla \hat\L(w_0)$, which was obtained by backpropagation
	\end{itemize}
\end{frame}

\section{Approach}
\begin{frame}{Evaluating an Optimizer}{Empirically}
	This is a hard problem in itself! How do you chose:
	\begin{itemize}
		\item Data Set
		\item Batch Size
		\item Model architecture
		\item Hyperparameter tuning method
		\item Measure of success
	\end{itemize}
	Comparing results between papers is very hard.
\end{frame}
\note{Now that we have roughly defined the algorithm, how do we test it?}

\begin{frame}{DeepOBS}{by Frank \& Aaron}
	\begin{itemize}
		\item A library for Tensorflow and Pytorch
		\item In order to test an optimizer, you have to specify only
			\begin{itemize}
			\item The optimizer class
			\item The hyperparameters of my optimizer
			\item One of the provided testproblems
			\end{itemize}
		\item DeepOBS then returns a json file
		\item And automatically generates figures
	\end{itemize}
\end{frame}

\begin{frame}[fragile]{Implementation Details}
	\framesubtitle{The class Preconditioner}
\begin{verbatim}
Preconditioner(params, est_rank=2, num_observations=5, prior_iterations=10,
               weight_decay=0, lr=None,
               optim_class=torch.optim.SGD, **optim_hyperparams)
start_estimate()
step()
get_log()
\end{verbatim}
\end{frame}

\begin{frame}{How to use the TCML Cluster}{Cloud Computing}
\begin{enumerate}
	\item Request an account by sending an email
	\item If you have any special code requirements, build a Singularity container (kind of like a virtual machine). Alternatively use a provided one.
	\item Create \& Submit a Slurm Batch job file
	\item Get an e-mail when your jobs start of finish
	\item Download the output files to your local machine. You can mount the cluster as a virtual drive.
\end{enumerate}
\end{frame}

\section{Experiments + Results + Discussion}
\begin{frame}{Experiments}{Overview}
\begin{itemize}
	\item Effectiveness of Preconditioning
	\item Computational Complexity
	\item Stability
	\item Learning Rate sensitivity
\end{itemize}
\end{frame}

\begin{frame}{Effectiveness of Preconditioning}{\tikz\draw[orange,fill=orange] (0,0) circle (.7ex); AdaptiveSGD \hspace{1cm} \tikz\draw[white,fill=blue] (0,0) circle (.7ex); PreconditionedSGD}
\vspace{3mm}
\input{images/exp_preconditioning.pgf}
\end{frame}
\note[itemize]{
\item - Show \& explain the figure exp\_preconditioning, It's created by DeepOBS
\item - Different Optimizers on two testproblems. Blue is the PreconditionedSGD and Orange is the AdaptiveSGD.
\item - Explain the Optimizers
\item - (Explain Testproblems) Only Convolutional nets!
\item - Both variants perform worse than other widely used algorithms.
\item - They converge much slower than the other algorithms, which seem to reach an optimum after 40 epochs, while the two don't converge even after 100 epochs
\item - Maybe mention the overfitting going on.
\item - Main conclusion: The algorithm does not perform better than others.
}

\begin{frame}{Effectiveness of Preconditioning}{\tikz\draw[orange,fill=orange] (0,0) circle (.7ex); AdaptiveSGD \hspace{1cm} \tikz\draw[white,fill=blue] (0,0) circle (.7ex); PreconditionedSGD}
\input{images/exp_preconditioning2.pgf}
\end{frame}

\begin{frame}{Discussion: Effectiveness}{Why might it be worse than plain SGD?}
	\begin{itemize}
		\item It's very noisy: Noise from the Hessian is amplified through the whole epoch.
		\item The constructed learning rate is not optimal
		\item PreconditionedSGD is worse than AdaptiveSGD, so the Preconditioning makes things worse
		\item The preconditioner has only rank 2, while there might be thousands large eigenvalues (usually ~10\%)
		\item The other optimizers are exhaustively tuned
		\item The other optimizers use more data for actual parameter updates: 1920/50.000 images per epoch are only used for the Hessian
	\end{itemize}
\end{frame}

\begin{frame}{Performance penalty}{Computational overhead}
\centering
\input{../thesis/images/exp_perf_prec.pgf}
\end{frame}
\note[itemize]{
\item - Running on mnist\_mlp with a batch size of 128.
\item - Shown Optimizers are
\item - SGD at 100% as a baseline.
\item - PreconditionedSGD, AdaptiveSGD, OnlyAdaptiveSGD
\item - They add a large performance penalty, but that would depend on the batch size.
}

%\begin{frame}{Stability depending on initialization}
%%TODO Do I want to keep this? & include figure
%\end{frame}

\begin{frame}{Learning Rate Sensitivity}{vs SGD}
\centering
\input{images/exp_lr_sens.pgf}
\end{frame}
\note[itemize]{
\item- Seeing that the algorithm is stable for an automatically constructed learning rate, I wanted to see if there are effects on success measures.
\item - Describing the plot
\item - Dropoff of SGD for small learning rates, plateau, cliff for large learning rates.
\item - Usually, on this testproblem the algorithm would estimate a learning rate of around 0.02 and then slowly decay it over epochs.
\item - This means we should let the algorithm chose the learning rate automatically
}

\begin{frame}{Conclusion/Final Remarks}{}

\end{frame}

\blackslidetext{end of presentation}

\end{document}


\chapter{Simulated Annealing}
\section{Introduction}
The Simulated Annealing search algorithm is examined in this chapter, looking  initially at the strategy options and configuration choices. The results section shows the performance of the algorithm in preliminary trials with a range of annealing schedules before presenting the results of the main trials.

\section{Search Strategy Options}
To be consistent with efforts of Random Search and Hill Climbing we again limit each trial to 25000 evaluations of the objective function. The operators used for Simulated Annealing are those that were used with Hill Climbing, however we have reduced the probability of occurrence of the shrink and grow operators based on our observations regarding duplication of solutions as discussed in Section~\ref{duplicate_solutions}.

The approach in general is similar to Hill Climbing, as the operators have been selected to allow fine-grained exploration of the search space while exploiting the capability of Simulated Annealing to avoid the pitfalls of local optima. The principle strategy choice for Simulated Annealing will be the selection of a suitable annealing schedule (see Section~\ref{schedule}), this is covered in detail in Section~\ref{schedule_selection}.

\section{Experimental Conditions}
Table~\ref{sa_param_table} shows the parameters used to configure Simulated Annealing. Symbolic Integration, Santa Fe and Blocks use initial Genome lengths of 100 while Symbolic Regression uses an initial length of 200. A temperature change rate of 0.95 is used to reduce the temperature on each iteration. A minimum temperature of \emph{0} is used to halt the annealing schedule. For a more complete explanation Section~\ref{sa_algorithm} provides a step-by-step guide to the algorithm.

\begin{table}[!hbp]
\begin{center}
\begin{tabular}{|l|l|l|l|l|}
\hline
Parameter &\multicolumn{4}{l|}{Problems}\\
\cline{2-1} \cline{3-1} \cline{4-1} \cline{5-1}
 & Sym Int & Santa Fe & Blocks & Sym Reg  \\
\hline
Number of Trials & 1000 & 1000 & 1000 & 1000  \\
Number of Objective & & & & \\
Function Evaluations  & 25000 & 25000 & 25000 & 25000 \\
Initial Genome Length & 100 & 100 & 100 & 200  \\
Variation Range  & 10\% & 10\% & 10\% & 10\%  \\
Zero Improvement Accept  & off & off & off & off  \\
Probability of Selecting & & & & \\
Shrink Operator  & .05 & .05 & .05 & .05  \\
Probability of Selecting & & & & \\
Grow Operator  & .05 & .05 & .05 & .05  \\
Probability of Selecting & & & & \\
Force-up Operator  & .45 & .45 & .45 & .45  \\
Probability of Selecting & & & & \\
Force-down Operator  & .45 & .45 & .45 & .45 \\
Temperature Change Rate & 0.95 & 0.95 & 0.95 & 0.95  \\
Minimum Temperature & 0 & 0 & 0 & 0  \\
\hline
\end{tabular}
\caption{\label{sa_param_table} Parameters used for the Simulated Annealing Search Algorithm.}
\end{center}
\end{table}

\section{Establishing an Annealing Schedule}
\label{schedule_selection}The establishment of an appropriate annealing schedule for each of the problems centres on selecting the desired level of probability for acceptance of a dis-improving solution.
At a temperature $t$, the probability $P$ for a change  in energy of magnitude $\delta E$ is given by

\large
\begin{displaymath}
P(\delta E) = e ^{ \frac{-\delta E}{kt}}
\end{displaymath}
\normalsize

Figure~\ref{santafe_accept_prob_score_delta} shows the change in the probability of accepting a dis-improving solution for a range of score deltas. Three different temperatures have been shown. If it was possible to predict the type of score deltas we were likely to see in the course of a trial then we could usefully employ this information to select a suitable starting temperature. We know from the analysis of the Hill Climbing results that solutions often emerge from areas of the solution landscape where scores are low score. This might suggest that it is reasonable to accept large dis-improving deltas late in a trial. However this policy would also have the consequence of having us accept even larger score deltas early in a trial, this could force the search to remain trapped in sub optimal regions of the search space.

This is better illustrated in Figure~\ref{santafe_accept_prob_score_temp} which again shows the change in the probability of accepting a dis-improving solution but this time it has been plotted against increasing temperature. Three plots for score deltas of 1, 5 and 10 have been included. Again we see some of the trade offs in this graph a high starting temperature exposes the search to higher probabilities of moving into low scoring regions of the search space while starting with a very low temperatures might leave the search unable to effectively explore the search space because of a overly captious approach.

\begin{figure}[]
\centerline{\hbox{
\psfig{file=Chapter7/graphs/accept_prob_score_delta.ps,width=4in}}}
\caption[Probability of accepting a dis-improving Solution for a Range of Score Deltas]{Probability of accepting a dis-improving Solution for a Range of Score Deltas at three different Temperatures on the Santafe Trail Problem.}
\label{santafe_accept_prob_score_delta}
\end{figure}

\begin{figure}[]
\centerline{\hbox{
\psfig{file=Chapter7/graphs/accept_prob_temp_change.ps,width=4in}}}
\caption[Probability of accepting a dis-improving Solution for a Range of Temperatures]{Probability of accepting a dis-improving Solution for a Range of Temperatures at three different Score Deltas on the Santafe Trail Problem.}
\label{santafe_accept_prob_score_temp}
\end{figure}

In order to get some indication of suitable  annealing schedules for the problems a limited number of runs were first performed for a range of starting temperatures. The number of iterations at each temperature accordingly  varied in order to maintain our limit of 25000 evaluations of the objective function in a single trial.
The results of these preliminary runs plus the results of the extended runs are presented in the next section.

\section{Results}

Tables~\ref{sa_symint_sched_table},~\ref{sa_santafe_sched_table} and~\ref{sa_blocks_sched_table} show the results from the preliminary trials used to determine the appropriate annealing schedule for Symbolic Integration, Santa Fe and Blocks respectively. Similar runs on the Symbolic Regression problem failed to find any correct solutions.

These preliminary trials consisted of 500 runs for each start temperature. The table shows the starting temperature and  the number of successes achieved.

For the case of Symbolic Integration the success rate appears to peak at 11\% for starting temperatures of 1 with the success rate trailing off either side of this. On the Santa Fe problem we see a plateau of similar scores for temperatures of 1 or less with a significant fall off in success rates for higher starting temperatures. The schedules for Blocks follows a  similar pattern to that of Santa Fe showing higher success rates at starting temperatures of one or less. The Symbolic Regression problem was not solved in any of the preliminary trials.

\begin{table}[h]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Starting Temperature & Successful Runs \\
\hline
0.3 & 10\% \\
0.1 & 6\% \\
1   &  11\% \\
5   &  5\% \\
10  & 2\% \\
\hline
\end{tabular}
\caption{\label{sa_symint_sched_table} Results from 500 trials of Simulated Annealing on the Symbolic Integration Problem for a Range of Starting Temperatures.}
\end{center}
\end{table}

\begin{table}[h]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Starting Temperature &  Successful Runs \\
\hline
0.3   &  17\%\\
0.1   &  20\%\\
1    &  19\% \\
5    &  4\% \\
10   &  5\% \\
\hline
\end{tabular}
\caption{\label{sa_santafe_sched_table} Results from 500 trials of Simulated Annealing on the Santa Fe Trail Problem for a Range of Starting Temperatures.}
\end{center}
\end{table}

\begin{table}[h]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Starting Temperature &  Successful Runs \\
\hline
0.1   &   11\%\\
0.3    &   13\%\\
1     &   12\%\\
5     &   6\%\\
10    &   9\%\\
\hline
\end{tabular}
\caption{\label{sa_blocks_sched_table} Results from 500 trials of Simulated Annealing on the Blocks Problem for a Range of Starting Temperatures.}
\end{center}
\end{table}

Table~\ref{sa_results_table} provides a summary of the final results for Simulated Annealing. 1000 runs were performed using most successful starting temperature of 1 from the preliminary trials. Symbolic Regression was not solved in any of the attempts, however Symbolic Integration and Blocks have similar success rates scoring 11\% and 13\% respectively, while Santa Fe scores 21\%.
While the algorithm is more successful on the Symbolic Integration and Santa Fe problems than Hill Climbing, the overall results are still poor relative to those of Random Search.

\begin{table}[h]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Problem & Successful Runs \\
\hline
Symbolic Integration & 11\% \\
Santa Fe Trail & 21\% \\
Blocks & 13\% \\
Symbolic Regression & 0\% \\
Spirals & 0\% \\
\hline
\end{tabular}
\caption{ \label{sa_results_table} Results from the Simulated Annealing Trials.}
\end{center}
\end{table}

\section{Characteristics of Solutions found by Simulated Annealing}

Table~\ref{sa_results_analysis_table} shows details of the solutions found by Simulated Annealing, wrapping features prominently in Santa Fe and Blocks. The average solution length for Santa Fe is 58 codons of which 48 are expressed. Symbolic Integration has solutions with an average genome length of 76 of which only 19 are expressed. An aspect of the results from the Blocks problem is that the number of expressed codons was 91, which is considerably greater than the average number of codons in a solution at 54. This aspect of the Block's results is caused by multiple wrap events where the codons are repeatedly re-used until a successful solution is found.

\begin{table}[h]
\begin{center}
\begin{tabular}{|l|l|l|l|l|}
\hline
Feature & Sym Int & Santa Fe & Blocks & Sym Reg  \\
\hline
Avg Number of Codons & & & &  \\
in Solution & 76 & 58 & 54 & n/a  \\
Avg Number of expressed & & & &  \\
Codons in Solution & 19 & 48 & 91 & n/a  \\
Avg Number of Solutions & & & &  \\
featuring Wrapping & 0\% & 50\% & 66\% & n/a  \\
\hline
\end{tabular}
\caption{\label{sa_results_analysis_table} Analysis of Characteristics from Solutions found by Simulated Annealing.}
\end{center}
\end{table}

\subsection{Analysis of search trajectory}

A typical search trajectory for a Simulated Annealing trial is shown in ~\ref{sa_search1}. In the early stages the algorithm explores areas of both low and high fitness but as the trials progresses we see a steady climb to areas of high fitness as the decreasing temperature reduces the probability of selecting dis-improving solutions.

\begin{figure}[hbp]
\centerline{\hbox{
\psfig{file=Chapter7/graphs/search_trajectory_sa.ps,width=5in}}}
\caption{\label{sa_search1}Search Trajectory for a Successful Simulated Annealing Trial}
\end{figure}

\section{Summary}
In this chapter we have looked at Simulated Annealing, a metaheuristic, which models itself on the way crystals form in solids during the cooling process. The principle strategy choice for the method was the selection of a suitable annealing schedule, to assist this selection a number of preliminary trials were performed using a range of starting temperatures. The final results showed unsuccessful attempts on two of the problem, Symbolic Regression and Spirals and low success rates on the other three.

The additional capability of Simulated Annealing to avoid sub-optima would appear to have added little or no improvement on the scores achieved by Hill Climbing, leaving it a considerable way behind Random Search.


\chapter{Analysis}
\label{Analysis}

Like any algorithm, we can not define "the best cryptography method;" rather, the cryptography algorithm that you use is
very case-dependant. In order to find a suitable candidate(s), one must examine the merits of a set of different algorithms.

When it comes to cryptography, security is at the root of the question. But how much do you \textit{really} value security?
For some, it may be a matter of "life or death." For others, security is a secondary feature of their program. Maybe you don't
even care about security? This can surely be the case, consider a puzzle game, one could employ the use of a substitution or
transposition cipher to create an interesting cryptography puzzle.

Let us consider a \textit{live chat application}. This application will send messages from one user to another (over the network).
We will assume that the messages sent are sensitive. In this case, security does seem to play a vital role in the function
of the application. Though we must also consider the \textit{comprising party}---the party which intends to infiltrate the
communication network.

Say we are designing a chat application for an intranet,\footnote{An intranet is a private network that is only accessible to
members of the network. An organization may use an intranet; only staff of the organization may access the network.} there may
no comprising party or the risk of a comprising party is low. Therefore we may not require any complex cryptography systems,
instead opting for a simple symmetric-key algorithm (such as Caesar Cipher or the slightly more complex DES scheme). On the
other hand, if we are developing a chat application for the internet where anyone can access the communication network. We may
place more importance on security as the network is public.

What if do we require a complex asymmetric-key system? We could use the discussed RSA scheme. All around, the RSA
scheme is an excellent cryptography method---it is extremely reliable---it is purely a mathematical solution---and it is secure.
But as always, there exists a caveat. The RSA scheme generates insanely large numbers. This is due to the fact that our plaintext
must be smaller than our modulus (the shared key). As discussed in
\hyperref[Asymmetric-key cryptography]{Asymmetric-key cryptography}, if our plaintext is
larger than our modulus, we must split our plaintext into appropriately sized blocks. We then encrypt each block in some mode
of operation (such as Feistel cipher). Unfortunately, this type of encryption will reveal redundancies in the ciphertext which
makes the cipher easier to break. Alternatively, we can expand the size of our modulus (and private and public keys)
to suit the size of our plaintext. This technique will work in theory but in practice there exists a hardware limitation---the
size of a data type; RSA will quickly exceed the max value of a $64$-bit integer. Instead we can use a \textit{Big Integer}, a
data structure which allows us to store \textit{very very} large numbers. This does however use a lot of memory. Nonetheless,
despite the existing alternatives, there exists the fact that RSA is a memory-hog. And despite advancements in computer
technology, there always exists an instantial upper-bound---whether it be storage or speed.