\section{Analysis}

The type of analyses we are interested in are \emph{intra-functional}, that is, we do not consider function calls. To root the design of the analysis infrastructure
we considered the four basic analyses in \cite{spa} which were also covered during the second part of the course, these are \emph{liveness}, \emph{available expressions},
\emph{very busy expressions} and \emph{reaching definitions}.

\newpar From a pure algorithmic point of view, these four analyses are striking similar: all of them operate on sets on a data structure that has a notion of successor and predecessor
and both use set union, intersection and difference as operations; this suggest an analysis framework with parametric analyses instead of a framework with \emph{hardcoded}
ones. This approach promotes code reuse, hides the details of the solver and underlying operations, e.g. set union, and it exposes an interface expressive enough to
write \emph{power set based} analyses.

%\subsection{Control Flow Graph}

\newpar The intermediate representation (of the program) upon which the analyses relay on is the Control Flow Graph with single-statement blocks (CFG). A \emph{Control Flow Graph} with \emph{single-statement blocks} is a digraph in which the control flow between non-control flow statements is modeled. This differ from a \emph{pure} control flow graph in the block (node) definition; in a typical control flow graph a block is a maximal sequence of linear statements. (\cite{cooper}).

\newpar CFG generation based on the parsed Abstract Syntax Tree (AST) has been implemented in Python, since existing AST-to-CFG parser libraries did not support generating single-statement blocks as required by this project. The CFG generation will generate a CFG with single-statement blocks given a generated TIP/C subset AST.

\newpar From each node in the CFG we require an interface to its successors (forward flow), predecessors (backward flow), left-hand side variable (if any), right-hand side expressions and type of statement.
An extra requirement for the CFG is the variables set, i.e. all the variables declared in the program and the expressions set.

The \emph{working unit} of an analysis is a set of monotone functions, and each monotone function take as an argument a node in the CFG.

To guide the construction and interfacing of the Analysis Engine (AE), we wrote \texttt{blue\_print.py} that focus on the design and inner workings of the analysis engine (represented by the class \texttt{Analysis}) while abstracting away all the implementation details about the components interfacing with it.

\newpar At the top level the AE requires a CFG and a non-empty list of monotone functions: \texttt{analysis = Analysis(cfg, monotone\_functions)}. To find the analysis' fix point to the given CFG, is just a matter of executing: \texttt{analysis.fix\_point()}. Also we designed the AE so the writing of monotone functions is as close as the mathematical formulation as possible.

\newpar To be able to find the fix point, the AE keeps a list of all user provided monotone functions (\texttt{self.monotone\_functions}) and a list \texttt{self.\_state} of size the number of nodes in the CFG. Each entry of the \texttt{self.\_state} list holds the current result of the monotone function that analysed the node. The signature of a monotone function looks like: \texttt{def join\_least\_upper\_bound (analysis, cfg\_node)} in which \texttt{analysis} is a reference to the AE and \texttt{cfg\_node} is a reference to the CFG node to be analysed. We send a reference to the AE to the monotone function so the user can have access the
\texttt{self.\_state} variable; this is useful when a function requires information from other nodes.
To find the fix point the AE iterates over all blocks in the CFG, per each block, each monotone function is applied in the same order as they were provided by the user when the AE was constructed, a block is considered to be analysed when a monotone function returns a non \texttt{None} value. Hence, a requirement is set to all monotone functions: if a monotone function does not apply to the given CFG node, then it must return \texttt{None}. This approach can be improved by caching, per CFG node, the function that returned a non \texttt{None} value.

\newpar The \texttt{Analysis} class exposes two functions to the monotone functions: \texttt{def least\_upper\_bound(self, left, right)} and \\\texttt{def greatest\_lower\_bound(self, left, right)} corresponding to the lattice functions \emph{least upper bound} and \emph{greatest lower bound}, respectively.

As an example of a monotone function implementation, we show the JOIN function w.r.t. the least upper bound:
\begin{minted}{python}
def join_least_upper_bound(analysis, cfg_node):
   club = frozenset()
   for successor in cfg_node.successors:
      club = analysis.least_upper_bound(club, analysis.state(successor))
   return club
\end{minted}

\newpar To find the least fix-point, we iterate over the nodes of the CFG, applying the corresponding monotone function depending on the \emph{type} of the node. A \texttt{\_state} variable keeps track of the latest monotone function result per CFG node. The least fix-point is found when the \texttt{\_state} variable does not change. The \texttt{\_state} variable is updated each time a monotone function is executed, as a consequence a small improvement on the time complexity might be achieved depending on the iteration order of the CFG nodes.

\subsection{Lattice based analyses}
To be able to handle general lattices and not only power set based analyses, certain changes must be made to the current implementation of \texttt{Analysis}. First, a new constructor must be added it will take the CFG and monotone functions as parameters but also will take as an additional parameter a list of pairs \texttt{lattice} that represent all the edges in the lattice. If $(x,y) \in$ \texttt{lattice} then $x < y$. From this list is possible to express the partial order completely as a matrix and use this matrix to calculate the least and greatest bounds. These changes are described in the \hyperref[futurework]{Future Work} section.


\documentclass[a4paper, 12pt]{article}
\usepackage{amsmath}
%\usepackage[utf16]{inputenc}
\usepackage{graphicx}
\usepackage[left=2.5cm, right=2.5cm, bottom=2.5cm, top=2.5cm]{geometry}
\usepackage{natbib}
\usepackage{microtype}
\usepackage{coloremoji}

\title{Measure --- or as the kids call it these days, addition}
\author{Brendon J. Brewer}
\date{}

\begin{document}
\maketitle

% Need this after the abstract
\setlength{\parindent}{0pt}
\setlength{\parskip}{8pt}

See \citet{knuth2016deeper}.

\section{Bigger and smaller}
Some things are bigger than others.

\section{How long is a piece of string?}

Consider a set, or collection, of objects, and suppose we can
unambiguously rank or sort the objects. For example, maybe we
have twenty apples, each of which has been weighed precisely.
We could put the apples in order by mass, from lowest to highest (or
the other way around).

More formally, think of a set $\mathcal{S} = \{s_1, s_2, ..., s_n\}$.
If we can take any two elements
(call them $x$ and $y$) and say that either $x > y$ or $y > x$ (where
we get to define what ``greater than'' means for a particular application),
then $\mathcal{S}$ is a {\em totally ordered set}. The definition of
$>$ that you use has to have the properties that $>$ usually does. For
example, if $x > y$ and $y > z$, then $x > z$.
Later, there will be a concept called a {\em chain} which is pretty much
the same as a partially ordered set.
If you are used to computer programming, then any
list/vector/array of items which can be unambiguously sorted
is a totally ordered set.

We can draw this situation using points on a line
(Figure~\ref{fig:totally_ordered_set}).
Each point
represents an element of the set, and they are arranged from highest
to lowest in terms of which elements are ``greater than'' other elements.
That is, if $x > y$ (element
$x$ is ranked above element $y$), then $x$ is drawn above $y$ in the
diagram.

\begin{figure}[!ht]
\centering
\includegraphics[scale=0.6]{figures/totally_ordered_set.pdf}
\caption{A set with five elements, each represented by a black dot.
When one dot is above another and they are connected with a line, it
means the higher element is greater than (using whatever definition of $>$ we
choose) the lower one.
This is a simple example of a {\em Hasse diagram}. More complex
Hasse diagrams will appear later.\label{fig:totally_ordered_set}}
\end{figure}

\section{The sum rule}

\section{Measure in boolean lattices}

\begin{figure}[!ht]
\centering
\includegraphics[width=0.5\textwidth]{figures/boolean_lattice.pdf}
\caption{\label{fig:boolean_lattice}}
\end{figure}

Apples.
One bag of apples combined with another bag of apples.
The exact same situation. The numbers of apples add, as
do the masses. Two different sum rules apply to the same
situation!

\section{Infidelity and ``signed measure''}

\subsection{A fate worse than death}

Infidelity allows for fates worse than death.
An absence of experience, denoted $\bot$,
has measure zero, so you can add arbitrary amounts
of non-existence to any experience without changing
its quality. So, for example:
\begin{align}
m(😃 \vee \bot)        &= m(😃)\\
m(😀 \vee 😪 \vee \bot) &= m(😀 \vee 😪)
\end{align}
and so on.
In other words, $\bot$ is the {\em identity element}
of the set of potential experiences.

\bibliographystyle{chicago}
\bibliography{references}

\end{document}


\documentclass[11pt,oneside]{article}    %use"amsart"insteadof"article"forAMSLaTeXformat
\usepackage{geometry}        %Seegeometry.pdftolearnthelayoutoptions.Therearelots.
\geometry{letterpaper}        %...ora4paperora5paperor...
%\geometry{landscape}        %Activateforforrotatedpagegeometry
%\usepackage[parfill]{parskip}        %Activatetobeginparagraphswithanemptylineratherthananindent
\usepackage{graphicx}                %Usepdf,png,jpg,orepsßwithpdflatex;useepsinDVImode
                                %TeXwillautomaticallyconverteps-->pdfinpdflatex
\usepackage{amssymb}
\usepackage[colorlinks]{hyperref}

%----macros begin---------------------------------------------------------------
\usepackage{color}
\usepackage{amsthm}

\def\conv{\mbox{\textrm{conv}\,}}
\def\aff{\mbox{\textrm{aff}\,}}
\def\E{\mathbb{E}}
\def\R{\mathbb{R}}
\def\Z{\mathbb{Z}}
\def\tex{\TeX}
\def\latex{\LaTeX}
\def\v#1{{\bf #1}}
\def\p#1{{\bf #1}}
\def\T#1{{\bf #1}}

\def\vet#1{{\left(\begin{array}{cccccccccccccccccccc}#1\end{array}\right)}}
\def\mat#1{{\left(\begin{array}{cccccccccccccccccccc}#1\end{array}\right)}}

\def\lin{\mbox{\rm lin}\,}
\def\aff{\mbox{\rm aff}\,}
\def\pos{\mbox{\rm pos}\,}
\def\cone{\mbox{\rm cone}\,}
\def\conv{\mbox{\rm conv}\,}
\newcommand{\homog}[0]{\mbox{\rm homog}\,}
\newcommand{\relint}[0]{\mbox{\rm relint}\,}

%----macros end-----------------------------------------------------------------

\title{Hierarchical structures with LAR
\footnote{This document is part of the \emph{Linear Algebraic Representation with CoChains} (LAR-CC) framework~\cite{cclar-proj:2013:00}. \today}
}
\author{Alberto Paoluzzi}
%\date{}                            %Activatetodisplayagivendateornodate

\begin{document}
\maketitle
\nonstopmode

\tableofcontents

%===============================================================================
\section{Affine transformations}
%===============================================================================
%-------------------------------------------------------------------------------
\subsection{Design decision}
%-------------------------------------------------------------------------------
First we state the general rules that will be satisfied by the matrices used in this module, mainly devoted to apply affine transformations to vertices of models in structure environments:
\begin{enumerate}
\item assume the scipy \texttt{ndarray} as the type of vertices, stored in row-major order;
\item use the last coordinate as the homogeneous coordinate of vertices, but do not store it explicitly;
\item store explicitly the homogeneous coordinate of transformation matrices.
\item use labels '\texttt{verts}' and '\texttt{mat}' to distinguish between vertices and transformation matrices.
\item transformation matrices are dimension-independent, and their dimension is computed as the length of the parameter vector passed to the generating function.
\end{enumerate}

%-------------------------------------------------------------------------------
\subsection{Affine mapping}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Apply an affine transformation to a LAR model
@{""" Apply an affine transformation to a LAR model  """
from scipy import array

def larApply(affineMatrix):
    def larApply0(model):
        if isinstance(model,Model):
            V = scipy.dot(array([v+[1.0] for v in model.verts]), affineMatrix.T).tolist()
            V = [v[:-1] for v in V]
            CV = copy.copy(model.cells)
            return Model((V,CV))
        elif isinstance(model,tuple) or isinstance(model,list):
            if len(model)==2: V,CV = model
            elif len(model)==3: V,CV,FV = model
            V = scipy.dot([list(v)+[1.0] for v in V], affineMatrix.T).tolist()
            if len(model)==2: return [v[:-1] for v in V],CV
            elif len(model)==3: return [v[:-1] for v in V],CV,FV
    return larApply0
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\subsection{Elementary matrices}
%-------------------------------------------------------------------------------
Elementary matrices for affine transformation of vectors in any dimensional vector space are defined here. They include translation, scaling, rotation and shearing.

\paragraph{Translation}
%-------------------------------------------------------------------------------
@D Translation matrices
@{def t(*args):
    d = len(args)
    mat = scipy.identity(d+1)
    for k in range(d):
        mat[k,d] = args[k]
    return mat.view(Mat)
@}
%-------------------------------------------------------------------------------
\paragraph{Scaling}
%-------------------------------------------------------------------------------
@D Scaling matrices
@{def s(*args):
    d = len(args)
    mat = scipy.identity(d+1)
    for k in range(d):
        mat[k,k] = args[k]
    return mat.view(Mat)
@}
%-------------------------------------------------------------------------------
\paragraph{Rotation}
%-------------------------------------------------------------------------------
@D Rotation matrices
@{def r(*args):
    args = list(args)
    n = len(args)
    @< plane rotation (in 2D) @>
    @< space rotation (in 3D) @>
    return mat.view(Mat)
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D plane rotation (in 2D)
@{if n == 1: # rotation in 2D
    angle = args[0]; cos = COS(angle); sin = SIN(angle)
    mat = scipy.identity(3)
    mat[0,0] = cos;    mat[0,1] = -sin;
    mat[1,0] = sin;    mat[1,1] = cos;
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D space rotation (in 3D)
@{if n == 3: # rotation in 3D
    mat = scipy.identity(4)
    angle = VECTNORM(args); axis = UNITVECT(args)
    cos = COS(angle); sin = SIN(angle)
    @< elementary rotations (in 3D) @>
    @< general rotations (in 3D) @>
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D elementary rotations (in 3D)
@{if axis[1]==axis[2]==0.0:    # rotation about x
    mat[1,1] = cos;    mat[1,2] = -sin;
    mat[2,1] = sin;    mat[2,2] = cos;
elif axis[0]==axis[2]==0.0:    # rotation about y
    mat[0,0] = cos;    mat[0,2] = sin;
    mat[2,0] = -sin;    mat[2,2] = cos;
elif axis[0]==axis[1]==0.0:    # rotation about z
    mat[0,0] = cos;    mat[0,1] = -sin;
    mat[1,0] = sin;    mat[1,1] = cos;
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D general rotations (in 3D)
@{else:        # general 3D rotation (Rodrigues' rotation formula)
    I = scipy.identity(3) ; u = axis
    Ux = scipy.array([
        [0,        -u[2],      u[1]],
        [u[2],        0,     -u[0]],
        [-u[1],     u[0],         0]])
    UU = scipy.array([
        [u[0]*u[0],    u[0]*u[1],    u[0]*u[2]],
        [u[1]*u[0],    u[1]*u[1],    u[1]*u[2]],
        [u[2]*u[0],    u[2]*u[1],    u[2]*u[2]]])
    mat[:3,:3] = cos*I + sin*Ux + (1.0-cos)*UU
@}

%===============================================================================
\section{Structure types handling}
%===============================================================================

In order to implement a structure as a list of models and transformations, we need to be able to distinguish between two different types of scipy arrays. The first type is the one of arrays of vertices, the second one is the matrix array used to represent the fine transformations.

\subsection{\texttt{Mat} and \texttt{Verts} classes}
%-------------------------------------------------------------------------------
@D types Mat and Verts
@{""" class definitions for LAR """
import scipy
class Mat(scipy.ndarray): pass
class Verts(scipy.ndarray): pass
@}
%-------------------------------------------------------------------------------

\subsection{\texttt{Model} class}
%-------------------------------------------------------------------------------
@D Model class
@{class Model:
    """ A pair (geometry, topology) of the LAR package """
    def __init__(self,(verts,cells)):
        self.n = len(verts[0])
        # self.verts = scipy.array(verts).view(Verts)
        self.verts = verts
        self.cells = cells
    def __getitem__(self,i):
        return list((self.verts,self.cells))[i]
@}
%-------------------------------------------------------------------------------

\subsection{\texttt{Struct} iterable class}
%-------------------------------------------------------------------------------
@D Struct class
@{""" Struct iterable class """
class Struct:
    """ The assembly type of the LAR package """
    def __init__(self,data=None,name=None,category=None):
        if data==None or data==[]:
            self.body = []
        else:
            self.body = [item for item in data if item != None]
            self.box = box(self)
            self.dim = len(self.box[0])
        if name != None:
            self.name = str(name)
        else:
            self.name = str(id(self))
        if category != None:
            self.category = str(category)
        else:
            self.category = "feature"
    def __name__(self):
        return self.name
    def __category__(self):
        return self.category
    def __iter__(self):
        return iter(self.body)
    def __len__(self):
        return len(list(self.body))
    def __getitem__(self,i):
        return list(self.body)[i]
    def __setitem__(self,i,value):
        self.body[i] = value
    def __print__(self):
        return "<Struct name: %s>" % self.__name__()
    def __repr__(self):
        return "<Struct name: %s>" % self.__name__()
        #return "'Struct(%s,%s)'" % (str(self.body),str(str(self.__name__())))
    def set_name(self,name):
        self.name = str(name)
    def clone(self,i=0):
        from copy import deepcopy
        newObj = deepcopy(self)
        if i != 0: newObj.name = self.name + "_" + str(i)
        return newObj
    def set_category(self,category):
        self.category = str(category)
    def larBoundary(self):
        data = struct2lar(self)
        if len(data) == 3:
            V,FV,EV = data
            #import pdb; pdb.set_trace()
            return V,FV,EV
        else:
            return "<Struct name: %s> boundary non computable" % self.__name__()
    def draw(self,color=WHITE,scaling=1,metric=ID):
        vmin,vmax = self.box
        delta = VECTDIFF([vmax,vmin])
        point = CCOMB(self.box)
        scalingFactor = scaling*delta[0]/20.
        text = TEXTWITHATTRIBUTES (TEXTALIGNMENT='centre', TEXTANGLE=0,
                    TEXTWIDTH=0.1*scalingFactor,
                    TEXTHEIGHT=0.2*scalingFactor,
                    TEXTSPACING=0.025*scalingFactor)
        point = metric([point])[0]
        return T([1,2,3])(point)(COLOR(color)(text(self.name)))
@}
%-------------------------------------------------------------------------------

\subsection{\texttt{Struct} containment box}

%-------------------------------------------------------------------------------
@D Computation of the containment box of a Lar Struct or Model
@{""" Computation of the containment box of a Lar Struct or Model """
import copy
def box(model):
    if isinstance(model,Mat): return []
    elif isinstance(model,Struct):
        dummyModel = copy.deepcopy(model)
        dummyModel.body = [term if (not isinstance(term,Struct)) else [term.box,[[0,1]]]  for term in model.body]
        listOfModels = evalStruct( dummyModel )
        #dim = checkStruct(listOfModels)
        theMin,theMax = box(listOfModels[0])
        for theModel in listOfModels[1:]:
            modelMin, modelMax = box(theModel)
            theMin = [val if val<theMin[k] else theMin[k] for k,val in enumerate(modelMin)]
            theMax = [val if val>theMax[k] else theMax[k] for k,val in enumerate(modelMax)]
        return [theMin,theMax]
    elif isinstance(model,Model):
        V = model.verts
    elif (isinstance(model,tuple) or isinstance(model,list)) and (len(model)==2 or len(model)==3):
        V = model[0]
    coords = TRANS(V)
    theMin = [min(coord) for coord in coords]
    theMax = [max(coord) for coord in coords]
    return [theMin,theMax]
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\section{Structure to LAR conversion}
%-------------------------------------------------------------------------------

\subsection{Remove duplicate faces}
%-------------------------------------------------------------------------------
@D Remove duplicate faces
@{""" Remove duplicate faces  """
from collections import defaultdict
def removeDups (CW):
    CW = list(set(AA(tuple)(CW)))
    CWs = list(set(AA(tuple)  (AA(sorted)(CW))  ))
    no_duplicates = defaultdict(list)
    for f in CWs: no_duplicates[f] = []
    for f in CW:
        no_duplicates[tuple(sorted(f))] += [f]
    CW = [f[0] for f in no_duplicates.values()]
    return CW
@}
%-------------------------------------------------------------------------------

\subsection{Structure to pair (Vertices,Cells) conversion}
%-------------------------------------------------------------------------------
@D Structure to pair (Vertices,Cells) conversion
@{""" Structure to pair (Vertices,Cells) conversion """

def struct2lar(structure,metric=ID):
    listOfModels = evalStruct(structure)
    vertDict = dict()
    index,defaultValue,CW,W,FW = -1,-1,[],[],[]

    for model in listOfModels:
        if isinstance(model,Model):
            V,FV = model.verts,model.cells
        elif (isinstance(model,tuple) or isinstance(model,list)):
            if len(model)==2: V,FV = model
            elif len(model)==3: V,FV,EV = model
        for k,incell in enumerate(FV):
            outcell = []
            for v in incell:
                key = vcode(4)(V[v])
                if vertDict.get(key,defaultValue) == defaultValue:
                    index += 1
                    vertDict[key] = index
                    outcell += [index]
                    W += [eval(key)]
                else:
                    outcell += [vertDict[key]]
            CW += [outcell]
        if len(model)==3:
            for k,incell in enumerate(EV):
                outcell = []
                for v in incell:
                    key = vcode(4)(V[v])
                    if vertDict.get(key,defaultValue) == defaultValue:
                        index += 1
                        vertDict[key] = index
                        outcell += [index]
                        W += [eval(key)]
                    else:
                        outcell += [vertDict[key]]
                FW += [outcell]

    if ((isinstance(model,tuple) or isinstance(model,list)) and len(model)==2) or (
        (isinstance(model,Model) and model.n==2)):
        if len(CW[0])==2:
            CW = list(set(AA(tuple)(AA(sorted)(CW))))
        else: CW = removeDups(CW)
        return metric(W),CW
    if ((isinstance(model,tuple) or isinstance(model,list)) and len(model)==3) or (
        (isinstance(model,Model) and model.n==3)):
        FW = list(set(AA(tuple)(AA(sorted)(FW))))
        CW = removeDups(CW)
        return metric(W),CW,FW
@}
%-------------------------------------------------------------------------------

\subsection{Model simplification}

\paragraph{Remove double instances of cells}

%-------------------------------------------------------------------------------
@O test/py/larstruct/test10.py
@{""" Remove double instances of cells (and the unused vertices) """
from larlib import *

@< Transform Struct object to LAR model pair @>
@< Remove the double instances of cells @>
VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS((W,FW))))

@< Remove the unused vertices @>
@}
%-------------------------------------------------------------------------------

The actual removal of double cells (useful in several applications, and in particular in the extraction of boundary models from 3D medical images) is performed by first generating a dictionary of cells, using as key the tuple given by the cells themselves, and then removing those discovered having a double instance.
The algorithm is extremely simple, and its implementation, given below, is straightforward.

%-------------------------------------------------------------------------------
@D Remove the double instances of cells
@{""" Remove the double instances of cells """
cellDict = defaultdict(list)
for k,cell in enumerate(FW):
    cellDict[tuple(cell)] += [k]
FW = [list(key) for key in cellDict.keys() if len(cellDict[key])==1]
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Remove the unused vertices
@{""" Remove the unused vertices """
print "len(W) =",len(W)
V,FV = larRemoveVertices(W,FW)
print "len(V) =",len(V)
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Remove the unused vertices from a LAR model pair
@{""" Remove the unused vertices """
def larRemoveVertices(V,FV):
    vertDict = dict()
    index,defaultValue,FW,W = -1,-1,[],[]

    for k,incell in enumerate(FV):
        outcell = []
        for v in incell:
            key = vcode(4)(V[v])
            if vertDict.get(key,defaultValue) == defaultValue:
                index += 1
                vertDict[key] = index
                outcell += [index]
                W += [eval(key)]
            else:
                outcell += [vertDict[key]]
        FW += [outcell]
    return W,FW
@}
%-------------------------------------------------------------------------------

\subsection{Embedding or projecting LAR models}
%-------------------------------------------------------------------------------

In order to apply 3D transformations to a two-dimensional LAR model, we must embed it in 3D space, by adding one more coordinate to its vertices.

\paragraph{Embedding or projecting a geometric model}

This task is performed by the function \texttt{larEmbed} with parameter $k$, that inserts its $d$-dimensional geometric argument in the $x_{d+1}, \ldots, x_{d+k}=0$ subspace of $\E^{d+k}$.
A projection transformation, that removes the last $k$ coordinate of vertices, without changing the object topology, is performed by the function \texttt{larEmbed} with \emph{negative} integer parameter.

%-------------------------------------------------------------------------------
@D Embedding and projecting a geometric model
@{def larEmbed(k):
    def larEmbed0(model):
        if len(model)==2: V,CV = model
        elif len(model)==3: V,CV,FV = model
        if k>0:
            V = [v+[0.]*k for v in V]
        elif k<0:
            V = [v[:-k] for v in V]
        if len(model)==2: return V,CV
        elif len(model)==3: return V,CV,FV
    return larEmbed0

def larEmbed(k):
    def larEmbed0(model):
        if k>0:
            model[0] = [v+[0.]*k for v in model[0]]
        elif k<0:
            model[0] = [v[:-k] for v in model[0]]
        return model
    return larEmbed0
@}
%-------------------------------------------------------------------------------

%===============================================================================
\section{Hierarchical complexes}
%===============================================================================
Hierarchical models of complex assemblies are generated by an aggregation
of subassemblies, each one defined in a local coordinate system, and
relocated by affine transformations of coordinates.  This operation
may be repeated hierarchically, with some subassemblies defined by
aggregation of simpler parts, and so on, until one obtains a set of
elementary components, which cannot be further decomposed.

Two main advantages can be found in a hierarchical modeling approach. Each elementary part and each assembly, at every hierarchical level, are defined independently from each other, using a local coordinate frame, suitably chosen to make its definition easier. Furthermore, only one copy of each component is stored in the memory, and may be instanced in different locations and orientations how many times it is needed.

%-------------------------------------------------------------------------------
\subsection{Traversal of hierarchical structures}
%-------------------------------------------------------------------------------

Of course, the main algorithm with hierarchical structures is the \emph{traversal} of the structure network, whose aim is to transform every encountered object from local to global coordinates, where the global coordinates are those of the network root (the only node with indegree zero).

A structure network can be modelled using a directed acyclic multigraph, i.e.~a triple $(N,A,f)$
made by a set $N$ of nodes, a set $A$ of arcs, and a function $f:A \to N^2$ from arcs to ordered pairs of nodes. Conversely that in standard oriented graphs, in this kind of structure more than one oriented arc is allowed between the same pair on nodes.

\begin{figure}[htbp] %  figure placement: here, top, bottom, or page
   \centering
   \includegraphics[width=0.8\linewidth]{images/traversal}
   \caption{Traversal algorithm of an acyclic multigraph.}
   \label{fig:traversal}
\end{figure}

A simple modification of a DFS (Depth First Search) visit of a graph can be used to traverse the structure network This algorithm is given in Figure~\ref{fig:traversal} from~\cite{Paoluzzi2003a}.

\subsubsection{Traversal of nested lists}

The representation chosen for structure networks with LAR is the serialised one, consisting in ordered sequences (lists) of either (a) LAR models, or (b) affine transformations, or (c) references to other structures, either directly nested within some given structure, or called by reference (name) from within the list.

The usual aim of a structure network traversal is, of course, to transform every component structure, usually defined in a local coordinate system, into the reference frame of the structure as a whole, normally corresponding with the reference system of the structure's root, called the \emph{world coordinate} system.

\paragraph{The pattern of calls and returned values}

In order to better understand the behaviour of the traversal algorithm, where every transformation is applied to all the following models, --- but only if included in the same structure  (i.e.~list) --- it may be very useful to start with an \emph{algorithm emulation}. In particular, the recursive script below discriminates between three different cases (number, string, or sequence), whereas the actual traversal must do with (a) Models, (b) Matrices, and (c) Structures, respectively.

%-------------------------------------------------------------------------------
@D Emulation of scene multigraph traversal
@{""" Emulation of scene multigraph traversal """
from larlib import *

def __traverse(CTM, stack, o):
    for i in range(len(o)):
        if ISNUM(o[i]): print o[i], REVERSE(CTM)
        elif ISSTRING(o[i]):
            CTM.append(o[i])
        elif ISSEQ(o[i]):
            stack.append(o[i])                # push the stack
            __traverse(CTM, stack, o[i])
            CTM = CTM[:-len(stack)]         # pop the stack

def algorithm(data):
    CTM,stack = ["I"],[]
    __traverse(CTM, stack, data)
@}
%-------------------------------------------------------------------------------

Some use example of the above algorithm are provided below. The printout produced at run time is shown from the \texttt{emulation of traversal algorithm} macro.
%-------------------------------------------------------------------------------
@D Examples of multigraph traversal
@{data = [1,"A", 2, 3, "B", [4, "C", 5], [6,"D", "E", 7, 8], 9]
print algorithm(data)
>>> 1 ['I']
    2 ['A', 'I']
    3 ['A', 'I']
    4 ['B', 'A', 'I']
    5 ['C', 'B', 'A', 'I']
    6 ['B', 'A', 'I']
    7 ['E', 'D', 'B', 'A', 'I']
    8 ['E', 'D', 'B', 'A', 'I']
    9 ['B', 'A', 'I']

data = [1,"A", [2, 3, "B", 4, "C", 5, 6,"D"], "E", 7, 8, 9]
print algorithm(data)
>>> 1 ['I']
    2 ['A', 'I']
    3 ['A', 'I']
    4 ['B', 'A', 'I']
    5 ['C', 'B', 'A', 'I']
    6 ['C', 'B', 'A', 'I']
    7 ['E', 'A', 'I']
    8 ['E', 'A', 'I']
    9 ['E', 'A', 'I']
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D Emulation of traversal algorithm
@{dat = [2, 3, "B", 4, "C", 5, 6,"D"]
print algorithm(dat)
>>> 2 ['I']
    3 ['I']
    4 ['B', 'I']
    5 ['C', 'B', 'I']
    6 ['C', 'B', 'I']
data = [1,"A", dat, "E", 7, 8, 9]
print algorithm(data)
>>> 1 ['I']
    2 ['A', 'I']
    3 ['A', 'I']
    4 ['B', 'A', 'I']
    5 ['C', 'B', 'A', 'I']
    6 ['C', 'B', 'A', 'I']
    7 ['E', 'A', 'I']
    8 ['E', 'A', 'I']
    9 ['E', 'A', 'I']
@}
%-------------------------------------------------------------------------------

\paragraph{Traversal of a scene multigraph}

The previous traversal algorithm is here customised for scene multigraph, where the objects are LAR models, i.e.~pairs of vertices of type '\texttt{Verts} and cells, and where the transformations are matrix transformations of type '\texttt{Mat}'.

\paragraph{Check models for common dimension}
The input list of a call to \texttt{larStruct} primitive is preliminary checked for uniform dimensionality of the enclosed LAR models and transformations. The common dimension \texttt{dim} of models and matrices is returned by the function \texttt{checkStruct}, within the class definition \texttt{Struct} in the module \texttt{lar2psm}. Otherwise, an exception is generated (TODO).

%-------------------------------------------------------------------------------
@D Check for dimension of a structure element (Verts or V)
@{@< Flatten a list @>
def checkStruct(lst):
    """ Return the common dimension of structure elements.

        TODO: aggiungere test sulla dimensione minima delle celle (legata a quella di immersione)
    """
    obj = lst[0]
    if (isinstance(obj,tuple) or isinstance(obj,list)):
        dim = len(obj[0][0])
    elif isinstance(obj,Model):
        dim = obj.n
    elif isinstance(obj,Mat):
        dim = obj.shape[0]-1
    elif isinstance(obj,Struct):
        dim = len(obj.box[0])
    return dim
@}
%-------------------------------------------------------------------------------

\paragraph{Flatten a list using Python generators}
The \texttt{flatten} is a generator that yields the non-list values of its input in order. In the example, the generator is converted back to a list before printing. Modified from \href{http://rosettacode.org/wiki/Flatten_a_list#Python}{\emph{Rosetta code}} project. It is used here to flatten a structure in order to check for common dimensionality of elements.

%-------------------------------------------------------------------------------
@D Flatten a list
@{""" Flatten a list using Python generators """
def flatten(lst):
    for x in lst:
        if (isinstance(x,tuple) or isinstance(x,list)) and len(x)==2:
            yield x
        elif (isinstance(x,tuple) or isinstance(x,list)):
            for x in flatten(x):
                yield x
        elif isinstance(x, Struct):
            for x in flatten(x.body):
                yield x
        else:
            yield x

#  lst = [[1], 2, [[3,4], 5], [[[]]], [[[6]]], 7, 8, []]
#  print list(flatten(lst))
#  [1, 2, 3, 4, 5, 6, 7, 8]

#  import itertools
#  chain = itertools.chain.from_iterable([[1,2],[3],[5,89],[],[6]])
#  print(list(chain))
#  [1, 2, 3, 5, 89, 6]    ###  TODO: Bug coi dati sopra?
@}
%-------------------------------------------------------------------------------

\paragraph{Initialization and call of the algorithm}

The function \texttt{evalStruct} is used to evaluate a structure network, i.e.~to return a \texttt{scene}
list of objects of type \texttt{Model}, all referenced in the world coordinate system. The input variable \texttt{struct} must contain an object of class \texttt{Struct}, i.e.~a reference to an unevaluated structure network. The variable \texttt{dim}  contains the embedding dimension of the structure, i.e.~the number of coordinates of its vertices (normally either 2 or 3), the CTM (Current Transformation Matrix) is initialised to the (homogeneous) identity matrix, and  the \texttt{scene} is returned by calling the \texttt{traverse} algorithm.

%-------------------------------------------------------------------------------
@D Traversal of a scene multigraph
@{""" Traversal of a scene multigraph """
@< Structure traversal algorithm @>
def evalStruct(struct):
    dim = checkStruct(struct.body)
    CTM, stack = scipy.identity(dim+1), []
    scene = traversal(CTM, stack, struct, [])
    return scene
@}
%-------------------------------------------------------------------------------

\paragraph{Structure traversal algorithm}

The \texttt{traversal} algorithm decides between three different cases, depending on the type of the currently inspected object. If the object is a \texttt{Model} instance, then applies to it the \texttt{CTM} matrix; else if the object is a \texttt{Mat} instance, then the \texttt{CTM} matrix is updated by (right) product with it; else if the object is a \texttt{Struct} instance, then the \texttt{CTM} is pushed on the stack, initially empty, then the \texttt{traversal} is called (recursion), and finally, at (each) return from recursion, the \texttt{CTM} is recovered by popping the stack.

%-------------------------------------------------------------------------------
@D Structure traversal algorithm
@{def traversal(CTM, stack, obj, scene=[]):
    for i in range(len(obj)):
        if isinstance(obj[i],Model):
            scene += [larApply(CTM)(obj[i])]
        elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and (
                len(obj[i])==2 or len(obj[i])==3):
            scene += [larApply(CTM)(obj[i])]
        elif isinstance(obj[i],Mat):
            CTM = scipy.dot(CTM, obj[i])
        elif isinstance(obj[i],Struct):
            stack.append(CTM)
            traversal(CTM, stack, obj[i], scene)
            CTM = stack.pop()
    return scene
@}
%-------------------------------------------------------------------------------

\paragraph{Structure embedding algorithm}
%-------------------------------------------------------------------------------
@D Embed a struct object
@{""" embed a struct object """
@< Structure embedding algorithm  @>

def embedStruct(n):
    def embedStruct0(struct,suffix="New"):
        if n==0:
            return struct, len(struct.box[0])
        cloned = Struct()
        cloned.box = hstack((struct.box, [n*[0],n*[0]])).tolist()
        cloned.name = str(id(cloned))  #struct.name+suffix
        cloned.category = struct.category
        cloned.dim = struct.dim + n
        cloned = embedTraversal(cloned,struct,n,suffix)
        return cloned
    return embedStruct0
@}
%-------------------------------------------------------------------------------

\paragraph{Structure embedding algorithm}
%-------------------------------------------------------------------------------
@D Structure embedding algorithm
@{""" Structure embedding algorithm """
def embedTraversal(cloned, obj,n,suffix):
    for i in range(len(obj)):
        if isinstance(obj[i],Model):
            cloned.body += [obj[i]]
        elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and (
                len(obj[i])==2):
            V,EV = obj[i]
            V = [v+n*[0.0] for v in V]
            cloned.body  += [(V,EV)]
        elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and (
                len(obj[i])==3):
            V,FV,EV = obj[i]
            V = [v+n*[0.0] for v in V]
            cloned.body  += [(V,FV,EV)]
        elif isinstance(obj[i],Mat):
            mat = obj[i]
            d,d = mat.shape

            newMat = scipy.identity(d+n*1)
            for h in range(d-1):
                for k in range(d-1):
                    newMat[h,k] = mat[h,k]
                newMat[h,d-1+n*1] = mat[h,d-1]
            cloned.body  +=  [newMat.view(Mat)]

        elif isinstance(obj[i],Struct):
            newObj = Struct()
            newObj.box = hstack((obj[i].box, [n*[0],n*[0]]))
            newObj.name = obj[i].name+suffix
            newObj.category = obj[i].category
            cloned.body  += [embedTraversal(newObj, obj[i], n, suffix)]
    return cloned
@}
%-------------------------------------------------------------------------------

%===============================================================================
\section{Larstruct exporting}
%===============================================================================
\label{sec:larstruct}
Here we assemble top-down the \texttt{lar2psm} module, by orderly listing the functional parts it is composed of. Of course, this one is the module version corresponding to the current state of the system, i.e.~to a very initial state. Other functions will be added when needed.
%------------------------------------------------------------------
@O larlib/larlib/larstruct.py
@{"""Module with functions needed to interface LAR with pyplasm via Struct"""
from larlib import *

@< Symbolic utility to represent points as strings @>
@< Translation matrices @>
@< Scaling matrices @>
@< Rotation matrices @>
@< Embedding and projecting a geometric model @>
@< Apply an affine transformation to a LAR model @>
@< Check for dimension of a structure element (Verts or V) @>
@< Remove duplicate faces @>
@< Remove the unused vertices from a LAR model pair @>
@< Traversal of a scene multigraph @>
@< types Mat and Verts @>
@< Model class @>
@< Struct class @>
@< Structure to pair (Vertices,Cells) conversion @>
@< Embedding and projecting a geometric model @>
@< Embed a struct object @>
@< Computation of the containment box of a Lar Struct or Model @>
@}
%------------------------------------------------------------------

%-------------------------------------------------------------------------------
%===============================================================================
\section{Examples}
%===============================================================================
%-------------------------------------------------------------------------------
Some examples of structures as combinations of LAR models and affine transformations are given in this section.

\paragraph{Global coordinates}
We start with a simple 2D example of a non-nested list of translated 2D object instances and rotation about the origin.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test04.py
@{""" Example of non-nested structure with translation and rotations """
from larlib import *

square = larCuboids([1,1])
table = larApply( t(-.5,-.5) )(square)
chair = larApply( s(.35,.35) )(table)
chair1 = larApply( t(.75, 0) )(chair)
chair2 = larApply( r(PI/2) )(chair1)
chair3 = larApply( r(PI/2) )(chair2)
chair4 = larApply( r(PI/2) )(chair3)
VIEW(SKEL_1(STRUCT(MKPOLS(table)+MKPOLS(chair1)+
                   MKPOLS(chair2)+MKPOLS(chair3)+MKPOLS(chair4))))
@}
%-------------------------------------------------------------------------------

\paragraph{Local coordinates}
A different composition of transformations, from local to global coordinate frames, is used in the following example.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test05.py
@{""" Example of non-nested structure with translation and rotations """
from larlib import *

square = larCuboids([1,1])
#square = Model(square)
table = larApply( t(-.5,-.5) )(square)
chair = larApply( s(.35,.35) )(table)
chair = larApply( t(.75, 0) )(chair)
struct = Struct([table] + 4*[chair, r(PI/2)])
scene = evalStruct(struct)
VIEW(SKEL_1(STRUCT(CAT(AA(MKPOLS)(scene)))))
@}
%-------------------------------------------------------------------------------

\paragraph{Call of nested structures by reference}
Finally, a similar 2D example is given, by nesting one (or more) structures via separate definition and call by reference from the interior. Of course, a cyclic set of calls must be avoided, since it would result in a \emph{non acyclic} multigraph of the structure network.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test06.py
@{""" Example of nested structures with translation and rotations """
from larlib import *

square = larCuboids([1,1])
#square = Model(square)
table = larApply( t(-.5,-.5) )(square)
chair = Struct([ t(.75, 0), s(.35,.35), table ])
struct = Struct( [t(2,1)] + [table] + 4*[r(PI/2), chair])
struct = Struct(10*[struct,t(0,2.5)])
struct = Struct(10*[struct,t(3,0)])
scene = evalStruct(struct)
VIEW(SKEL_1(STRUCT(CAT(AA(MKPOLS)(scene)))))
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@O test/py/larstruct/test08.py
@{""" LAR model input and handling """
from larlib import *

@< Input of LAR architectural plan @>
dwelling = larApply(t(3,0))(Model((V,FV)))
print "\n dwelling =",dwelling
VIEW(EXPLODE(1.2,1.2,1)(MKPOLS((dwelling.verts,dwelling.cells))))
VIEW(EXPLODE(1.2,1.2,1)(MKPOLS((dwelling.verts,EV))))
plan = Struct([dwelling,s(-1,1),dwelling])
VIEW(EXPLODE(1.2,1.2,1)(CAT(AA(MKPOLS)(evalStruct(plan)))))
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Input of LAR architectural plan
@{""" Input of LAR architectural plan """
from larlib import *

V = [[3,-3],
[9,-3],[0,0],[3,0],[9,0],[15,0],
[3,3],[6,3],[9,3],[15,3],[21,3],
[0,9],[6,9],[15,9],[18,9],[0,13],
[6,13],[9,13],[15,13],[18,10],[21,10],
[18,13],[6,16],[9,16],[9,17],[15,17],
[18,17],[-3,24],[6,24],[15,24],[-3,13]]
FV = [
[22,23,24,25,29,28], [15,16,22,28,27,30], [18,21,26,25],
[13,14,19,21,18], [16,17,23,22], [11,12,16,15],
[9,10,20,19,14,13], [2,3,6,7,12,11], [0,1,4,8,7,6,3],
[4,5,9,13,18,17,16,12,7,8],[17,18,25,24,23]]

polylines = lar2polylines((V,FV))
lines = CAT([zip(polyline[:-1],polyline[1:]) for polyline in polylines])
verts = dict(zip(AA(vcode(4))(V),range(len(V))))
edges = [tuple(sorted([verts[vcode(4)(v1)], verts[vcode(4)(v2)]])) for v1,v2 in lines]
EV = list(set(edges))
@}
%-------------------------------------------------------------------------------

\paragraph{Transformation of Struct object to LAR model pair}

The following test application first generates a grid $3\times 3$ of LAR cubes, extracts
its boundary cells as \texttt{BV}, then produces a \texttt{struct} object with 30 translated instances of it,
and finally transforms the \texttt{struct} object into a LAR pair \texttt{W,FW}.
Let us notice that due to the assembly process, some 2-cells in \texttt{FW} are doubled.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test09.py
@{""" Transformation of Struct object to LAR model pair """
from larlib import *

@< Transform Struct object to LAR model pair @>
@}
%-------------------------------------------------------------------------------

The actual generation of the structure and its transformation to a LAR model pair is actually performed in the following macro.

%-------------------------------------------------------------------------------
@D Transform Struct object to LAR model pair
@{""" Generation of Struct object and transform to LAR model pair """
cubes = larCuboids([10,10,10],True)
V = cubes[0]
FV = cubes[1][-2]
CV = cubes[1][-1]
bcells = boundaryCells(CV,FV)
BV = [FV[f] for f in bcells]
VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS((V,BV))))

block = Model((V,BV))
struct = Struct(10*[block, t(10,0,0)])
struct = Struct(10*[struct, t(0,10,0)])
struct = Struct(3*[struct, t(0,0,10)])
W,FW = struct2lar(struct)

VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS((W,FW))))
@}
%-------------------------------------------------------------------------------

%===============================================================================
\appendix
%===============================================================================
\subsection{Importing a generic module}
First we define a parametric macro to allow the importing of \texttt{larcc} modules from the project repository \texttt{lib/py/}. When the user needs to import some project's module, she may call this macro as done in Section~\ref{sec:lar2psm}.
%------------------------------------------------------------------
@d Import the module
@{import sys; sys.path.insert(0, 'lib/py/')
import @1
@}
%------------------------------------------------------------------

\paragraph{Importing a module} A function used to import a generic \texttt{lacccc} module within the current environment is also useful.
%------------------------------------------------------------------
@d Function to import a generic module
@{def importModule(moduleName):
    @< Import the module @(moduleName@) @>
@| importModule @}
%------------------------------------------------------------------

\subsection{Numeric utilities}

A small set of utility functions is used to transform a \emph{point} representation, given as array of coordinates, into a string of fixed format to be used as point key into python dictionaries.

%------------------------------------------------------------------
@D Symbolic utility to represent points as strings
@{""" TODO: use package Decimal (http://docs.python.org/2/library/decimal.html) """
#global PRECISION
#PRECISION = 4.

def verySmall(number): return abs(number) < 10**-(PRECISION)

def prepKey (args): return "["+", ".join(args)+"]"

def fixedPrec(PRECISION):
    def fixedPrec0(value):
        out = round(value*10**(PRECISION))/10**(PRECISION)
        if out == -0.0: out = 0.0
        return str(out)
    return fixedPrec0

def vcode (PRECISION=4):
    def vcode0 (vect):
        """
        To generate a string representation of a number array.
        Used to generate the vertex keys in PointSet dictionary, and other similar operations.
        """
        return prepKey(AA(fixedPrec(PRECISION))(vect))
    return vcode0
@}
%------------------------------------------------------------------

\bibliographystyle{amsalpha}
\bibliography{larstruct}

\end{document}


\chapter{Background}\label{sec:Background}
RL is a process that requires both interactive parts as well as algorithms that improve interactions. The following section \ref{reinforcement_learning} introduces the general concept of RL and its specifications. Afterwards, two popular learning algorithms for RL problems are presented: PPO and DQN.

\section{Reinforcement Learning}\label{reinforcement_learning}
Sutton and Barto wrote in ``Reinforcement learning: An introduction'' \cite{suba18} that RL is based on two components that interact with each other: an environment and an agent, see Figure \ref{fig:rl_cycle}. Those interactions take part during a time period with discrete time steps $t\in\mathbb{N}_0$ until a goal is reached or the ending condition applies. This process is called an episode. Formally, the journey of the agent to find the goal is described as a Markov Decision Process (MDP) \cite{suba18}. When multiple agents act in the same environment, the Markov decision process is called a stochastic game \cite{buba10}.

\begin{figure}[hpbt]
    \centering
    \includegraphics[width=0.6\textwidth]{pictures/RLInteractionSB}\\
    \caption[Reinforcement Learning Cycle]{The cycle of agent-environment interaction as
        shown in ``Reinforcement learning: An introduction'' \cite{suba18}}\label{fig:rl_cycle}
\end{figure}

One environment state $S_t$ is part of a set $S$ containing all possible states. In most cases, the environment state describes what the agent can see. An agent often only has a small field of view, which turns a MDP into a partially observable MDP \cite{suba18}. During each point in time $t$, the agent can interact with the environment by executing an action $A_t$, which changes the environment state. An example would be moving in the environment, which results in a new area that the agent can now see. In a multiagent environment, every agent chooses its action simultaneously and adds it into a joint action set, which is executed collectively during $t$ \cite{buba10}.

The reward $R_t$ is an element of a set of possible rewards $R \subset \mathbb{R}$ \cite{suba18}. Therefore, the reward can potentially be negative. Depending on the environment, that value can act as immediate feedback to the agents action. Other times, the reward is received as a result of a whole action sequence or the achievement of a certain state, for instance the goal or subgoals. The general concept of RL, as defined by Sutton and Barto \cite{suba18}, is for agents to maximize rewards. Unlike machine learning approaches, the agent starts with no knowledge about good or bad actions and enhances the decision-making over time.

Sutton and Barto defines the agents' action selection with respect to the current state as a policy $\pi$. They explain further that a policy could be as simple as a lookup table, mapping states to actions, or it could contain a complicated search process for the best decision. However, policies most of the time map action-state pairs to a selection probability, with all actions of a state adding up to 100\%.
During environment interactions, agents receive rewards which can be used to update the policy accordingly. As an example, the probability of policy $\pi(a \mid s)$ decreases when receiving a negative or low reward, reducing the chances of executing the same action in that specific state again.

While rewards only rate the immediate situation, a value function, i.e. the state-value function $V^\pi(s_t)$ for a policy $\pi$, can be used to estimate the long-term value of a state $s$ \cite{suba18}:
\begin{equation}\label{eq:value_func}
    v_\pi(s) \doteq \mathbb{E}_\pi \left[ G_t \mid S_t = s \right] = \mathbb{E}_\pi \left[ \sum^{\infty}_{k=0} \gamma^k R_{t+k+1 \mid S_t = s}  \right]
\end{equation}
The result is the estimated discounted cumulative reward an agent could get following that state and choosing actions based on the current policy. The discount factor is defined by Sutton and Barto as $0 \le \gamma < 1$ and provides a constant that reduces the importance of future rewards. A high $\gamma$ symbolizes a greater interest in rewards that are far away, whereas a discount of zero only takes the current reward into account. By setting $\gamma$ smaller than one, it is ensured that the infinite sum results in a value. Generally, states that offer immediate high rewards could end in a low reward streak. In the opposite case, a low reward state could subsequently yield high rewards. Therefore, value functions are of great use to achieve the maximum reward.

The last part to note about RL is that it entails the problem of balancing exploration and exploitation \cite{suba18}. On the one hand, an agent has to explore different options in order to learn and expand its knowledge. On the other hand, agents strive to maximize the reward, which can lead to greediness. An agent could start to exploit its knowledge too early, choosing actions of which it knows to result in positive rewards. However, if an agent does not explore enough, the best action sequence will stay hidden and the agents knowledge will not improve.

\section{Proximal Policy Optimization}
In 2017, Schulman et al. introduced the concept of PPO in the article ``Proximal Policy Optimization Algorithms'' \cite{scwo17}. Policy optimization is the improvement of the action selection strategy $\pi$ based on the current state $s_{t}$. This is achieved by rotating two steps \cite{scwo17}: \\
\begin{enumerate}
  \item Sampling data from the policy and
  \item Optimizing the objective with that data through several epochs.
\end{enumerate}

Using those steps results in the agent gathering a small batch of experiences while choosing actions with a policy $\pi$. Afterwards, this batch is used once to enhance the current policy. Then, the experiences are discarded and the agent uses the updated policy to gather a new batch. By repeating those two steps the agent can learn to choose better actions. PPO is applied to prevent drastic policy changes, which stabilizes the learning process.

The origin of PPO lies in a similar approach called Trust Region Policy Optimization (TRPO). TRPO also restricts policy updates by defining a trust region \cite{scle15}. This is achieved by maximizing the following function \cite{scwo17}:
\begin{equation}\label{eq:TRPO}
    \underset{\theta}{maximize}\,\hat{\mathbb{E}}_{t} \left[ \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{old}}(a_{t} \mid s_{t})}
        \hat{A}_{t}-\beta \, KL[\pi_{\theta_{old}}(\cdot \mid s_{t}),\pi_{\theta}(\cdot \mid s_{t})] \right]
\end{equation}
The expectation $\hat{\mathbb{E}}_{t}$ indicates, that an empirical average over a number $t$ of samples is used for estimation and the algorithm alternates between sampling and executing these calculations. The variable $\hat{A}_{t}$ describes an estimator of the advantage function. This function was defined in the paper ``Trust Region Policy Optimization'' \cite{scle15} with \\ $A_\pi(s,a) = Q_\pi(s,a)-V_\pi(s)$. The first part calculates the state-action value, estimating the upcoming rewards for an agent, starting at state s and initially selecting action a. Afterwards, the action selection is based on the current policy $\pi$.

The second part contains the state value function $V_\pi(s)$, which works very similarly by starting at state s and using $\pi$. However, the difference is that the agent always chooses actions according to the policy. The result of the advantage function $A_\pi(s,a)$ shows whether a profit could be gained when deviating from the policy by specifically choosing action a.

The fraction $\frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{old}}(a_{t} \mid s_{t})}$ in the Minuend of function \eqref{eq:TRPO} can be replaced by $r(\theta)$
and represents the probability ratio of an action in the current policy in comparison to the old policy \cite{scwo17}. $\theta$ represents a policy parameter. The result of $r(\theta)$ is greater than one, if an action is very probable in the current policy. Otherwise, the outcome lies between zero and one. Schulman et al. \cite{scwo17} further extract the first part of function \eqref{eq:TRPO} as the surrogate objective:
\begin{equation}\label{eq:TRPO_surrogate}
    L^{CPI}(\theta) = \hat{\mathbb{E}}_{t} \left[ \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{old}}(a_{t} \mid s_{t})} \hat{A}_{t} \right]
    = \hat{\mathbb{E}}_{t} \left[ r(\theta)\hat{A}_{t} \right]
\end{equation}
If only this part of function \eqref{eq:TRPO} is maximized on its own, it would result in large outcomes, which in turn leads to drastic policy updates. In order to stay in a trust region, as the name suggests, a penalty is subtracted from the surrogate function \eqref{eq:TRPO_surrogate}. The penalty is the subtrahend of equation \eqref{eq:TRPO} and contains the fixed coefficient $\beta$. Regardless of the function details and outcome of $KL$, the coefficient $\beta$ is hard to choose, since different problems require different penalty degrees \cite{scwo17}. Even during a training process it could be necessary to adapt the coefficient, due to changes.

Therefore, Schulman et al. introduced
\begin{equation}\label{eq:PPO}
    L^{CLIP}(\theta) = \hat{\mathbb{E}}_{t} \left[ \min \left( \; r(\theta)\hat{A}_{t}, \; clip(r(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_{t} \; \right) \right]
\end{equation}
which is very similar to equation \eqref{eq:TRPO} but does not require coefficients. The first $\min$ entry contains $L^{CPI}$ \eqref{eq:TRPO_surrogate}. The second part contains a $clip$ function, which narrows the space of policy mutation with the small hyperparameter $\epsilon$. After applying the clip function, $r(\theta)$ lies between $[1-\epsilon,1+\epsilon]$. Calculating the minimum of the clipped and unclipped probability ratio produces the lower bound of the unclipped $r(\theta)$, preventing the policy to change drastically.

Finally, the following equation is introduced
\begin{equation}\label{eq:PPO_algo}
    L_{t}^{CLIP+VF+S}(\theta) = \hat{\mathbb{E}}_{t} \left[ L_{t}^{CLIP}(\theta) - c_{1}L_{t}^{VF}(\theta) + c_{2}S[\pi_{\theta}](s_{t}) \right]
\end{equation}
with $c_{1}$ and $c_{2}$ as coefficients. The authors point out that the loss function \\
$L_{t}^{VF} = (V_{\theta}(s_{t})-V_{t}^{targ})^2$ combines the policy surrogate and the value function error term and is necessary once a neural network shares parameters between policy and value function. An entropy bonus $S$ is added to ensure exploration.

Furthermore, Schulman et al. point out that the policy is executed for $T$ time steps, with $T$ being a smaller value than the overall episode duration. Until now, the advantage function calculates values through an infinite loop, see the value function \eqref{eq:value_func} for example. Hence, the advantage function needs to be adjusted as well. It is necessary that the future estimations do not exceed that time step limit. In this context, the following advantage function is used \cite{scwo17}:
\begin{equation}\label{eq:advantage_func}
    \hat{A_t} = \delta_t+(\gamma \lambda)\delta_{t+1}+ \cdots + (\gamma \lambda)^{T-t+1}\delta_{T-1}
\end{equation}
\begin{equation}\label{eq:advantage_func_delta}
    \textrm{where} \qquad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)
\end{equation}

Schulman et al. also showed an example of the PPO algorithm, cf. algorithm \ref{algo:ppo_algo_code}. The example uses an actor-critic approach, which means that a critic is responsible to approximate the value function of the policy and the actor in turn improves the policy based on the approximation results of the critic \cite{kots03}. $N$ detonates actors collecting data in T time steps in each iteration. Meanwhile, the critic computes the estimations of the advantage values. Afterwards, the policy is replaced with a new one, in which the function $L_{t}^{CLIP+VF+S}(\theta)$ \eqref{eq:PPO_algo} is optimized during K epochs. For the optimization process, a small random batch of the previous time steps is used.

\begin{algorithm}[H]
    \DontPrintSemicolon
    \For(){\text{iteration=1,2,...}}{
        \For(){\text{actor=1,2,...,$N$}}{
            Run policy $\pi_{\theta_{old}}$ in environment for $T$ timesteps \;
            Compute advantage estimates $\hat{A}_{1},...\hat{A}_{T}$
        }
        Optimize surrogate $L$ wrt $\theta$, with K epochs and minibatch size $M \; \leq \; NT$ \;
        $\theta_{old} \leftarrow \theta$
    }
    \caption{PPO, Actor-Critic Style, as shown in ``Proximal Policy Optimization Algorithms'' \cite{scwo17}}\label{algo:ppo_algo_code}
\end{algorithm}

\section{Deep Q-Network}\label{dqn}
Another learning approach that is often compared with PPO is the training algorithm of a deep Q-Network with Q-learning and experience replay. Instead of improving a policy, agents improve by maximizing a value function. Hence, this algorithm relies on the action value function, that is formally defined as follows \cite{mnba16}:
\begin{equation}\label{eq:qvalue}
    Q^\pi(s,a) = \mathbb{E} \left[ R_t \mid s_t = s,a \right]
\end{equation}
$R_t$ represents the discounted cumulative reward $R_t=\sum^{\infty}_{k=0} \gamma^k r_{t+k}$. The estimated outcome is calculated by starting at a state $s$, executing a specific action $a$ and reaching the next states by using a policy $\pi$. Mnih et al. \cite{mnka15} state, that the optimal action-value can be approximated with a deep convolutional neural network and the following function:
\begin{equation}\label{eq:opt_qvalue}
    Q^*(s,a) =  \underset{\pi} \max \mathbb{E}\left[ r_{t} + \gamma r_{t+1} + \gamma^2 r_{t+2} + \ldots | s_t = s, a_t = a, \pi \right]
\end{equation}

The difference between function \eqref{eq:qvalue} and \eqref{eq:opt_qvalue} is, that in the second one, a policy is chosen, which optimizes the outcome. Mnih et al. continue by stating, that in a scenario where the sequence $s'$ of all actions $a'$ are known, the optimal $Q^*(s',a')$ of the next state can be calculated. Then, this Bellman equation could be applied \cite{mnba16}:
\begin{equation}\label{eq:bel_qvalue}
    Q^*(s,a) =  \mathbb{E}_{s'} \left[ r+ \gamma \underset{a'}\max \; Q^* (s', a') \mid s,a \right]
\end{equation}
Many RL algorithms estimate this function through iterative updates, by calculating $Q_{i+1}(s,a) =  \mathbb{E} \left[ r+ \gamma \underset{a'}\max \; Q_i (s', a') \mid s,a \right]$, $Q_{i+2}$, \dots \cite{mnka13}. Eventually the optimal Q value is reached with $i\rightarrow \infty$. Those calculations proved to be very impractical, since they require a lot of computational work, which is why Mnih et al. introduced the Q-network at this point. As a result, the parameters of the Q function are extended with $\theta$ as network weights ($Q(s,a;\theta)$).

However, the researchers argued that using a neural network in combination with the Q function proofed to be unstable. According to the authors, this is caused by correlating observations that are used to calculate the function. Additionally, small updates to the action value may lead to drastic changes of the policy. Such problems change the connection between Q values and their successive target values $r+\gamma \; \underset{a'} \max \; Q(s',a')$. To overcome these issues, Mnih et al. introduced two new concepts:
\begin{enumerate}
  \item An experience replay that enables random sampling of observations and
  \item An iterative update process of the action values approaching the target values.
\end{enumerate}
The target values are only updated periodically in their implementation.

In algorithm \ref{algo:dqn_algo_code}, a deep Q-learning approach with an experience replay is shown. The experience replay contains the acquired agent knowledge of each time step in form of a quadruple: (old state, action, reward, new state). The experience values are then stored into the replay memory $D$ across multiple episodes. The states are parameters of $\Phi_{t}$ in the example, since they are preprocessed, to match the network input conditions.

In addition to the action value function $Q$, the target action-value $\hat Q$ is initially defined with the same weights to enable iterative updates. In order to fill the memory, the agent first selects actions and acts in the environment. The action selection here is based on the $\epsilon$-greedy policy, meaning that with a probability of $\epsilon$ a random action is chosen \cite{mnka15}. Otherwise, the best option according to the Q-value is selected.

\begin{algorithm}[H]
    \DontPrintSemicolon
    Initialize replay memory $D$ to capacity $N$ \;
    Initialize action-value function $Q$ with random weights $\theta$ \;
    Initialize target action-value function $\hat{Q}$ with weights $\theta^{-} = \theta$ \;
    \For(){\text{episode=1,$M$}}{
        Initialize sequence $s_1 = \{x_1\}$ and preprocessed sequence $\phi_1 = \phi(s_1)$ \;
        \For(){\text{$t$=1,T}}{
            With probability $\epsilon$ select a random action $a_t$ \;
            otherwise select $a_t = \text{argmax}_a Q(\phi(s_t), a; \theta)$ \;
            Execute action $a_t$ in emulator and observe reward $r_t$ and image $x_{t+1}$ \;
            Set $s_{t+1} = s_{t}, a_{t}, x_{t+1}$ and preprocess $\phi_{t+1}=\phi(s_{t+1})$ \;
            Store transition ($\phi_{t}, a_{t}, r_{t}, \phi_{t+1}$) in $D$ \;
            Sample random minibatch of transitions ($\phi_{j}, a_{j}, r_{j}, \phi_{j+1}$) from $D$ \;
            Set $y_j = \begin{cases}
                r_j & \text{if episode terminates at step j + 1}\\
                r_j + \gamma \text{max}_{a'} \hat{Q}(\phi_{j+1}, a', \theta^{-}) & \text{otherwise}
                \end{cases}$  \;
            Perform a gradient descent step on $\left( y_j - Q(\phi, a_j ; \theta) \right) ^2$ with respect to the network parameters $\theta$ \;
            Every $C$ steps reset $\hat{Q} = Q$
        }
    }
    \caption{DQN with Experience Replay, as shown in ``Human-level control through deep reinforcement learning'' \cite{mnka15}}\label{algo:dqn_algo_code}
\end{algorithm}

Executing the selected action results in a memory entry in the form of the earlier described quadruple. Afterwards, a minibatch of the replay memory is randomly sampled to calculate the difference between the values. The action values with the current weights are subtracted from $y_i$. This variable calculates estimated Bellman equation by using the target action values with the old weights. The parameter $y_i$ contains just the reward value of the sample, if the sampled entry was the last step the episode of the entry. A gradient descent step is performed on the function, which means that the local minima of the function is searched by tweaking the parameter $\theta$.

Finally, every certain amount of steps C the target network is set to the current Q-Network. The suggested process offers several advantages \cite{mnka15}: the replay memory leads to a smaller deviation or fluctuation in the parameters. The random samples of minibatches can be efficient, since an experience might be used multiple times to update the network weights. Furthermore, through the randomness in the samples, the correlation of steps is interrupted. This leads to a decrease of variance in between updates. Lastly, updating the target network periodically improves the stability of the learning process.


\subsubsection{\stid{3.12} Sub-project: hypre}

\paragraph{Overview}
The {\sl hypre} software library \cite{hypre:homepage,hypre_design_impl_2006} provides high performance preconditioners and solvers for the solution of large sparse linear systems on massively parallel computers, with particular focus on algebraic multigrid solvers. One of {\sl hypre}’s unique features is the provision of a (semi)-structured interface, in addition to a traditional linear-algebra based interface. The semi-structured interface is appropriate for applications whose grids are mostly structured, but with some unstructured features. Examples include block-structured grids, composite grids in structured adaptive mesh refinement (AMR) applications, and overset grids. These interfaces give application users a more natural means for describing their linear systems, and provide access to methods such as structured multigrid solvers, which can take advantage of the additional information beyond just the matrix. Since current architecture trends are favoring regular compute patterns to achieve high performance, the ability to express structure has become much more important. The {\sl hypre} library provides both unstructured and structured multigrid solvers, which have shown excellent scalability on a variety of high performance computers, e.g Blue Gene systems (unstructured solver BoomerAMG has scaled up to 1.25 million MPI cores with a total of 4.5 million hardware threads). It is used by many ECP application teams, including ExaAM, Subsurface, ExaWind, CEED, and more. It requires a C compiler and an MPI implementation, but it also runs in an OpenMP environment. It also has GPU capabilities.

\paragraph{Key  Challenges}

While {\sl hypre}'s solvers contain much parallelism, their main focus is the solution of sparse linear systems, leading to  very large demands on memory bandwidth. In addition, the use of multiple levels, while greatly aiding convergence of the solvers, leads to decreasing systems sizes, number of operations and parallel efficiencies on coarser levels. Particularly the unstructured algebraic multigrid solver BoomerAMG\cite{HeYa2002}, which is {\sl hypre}'s most often used preconditioner, suffers from increasing communication complexities on coarser levels. Coarse grid operators are generated by multiplying three matrices leading to increasing numbers of nonzeroes per row in the resulting matrices and with it increasing numbers of neighbor processes. While BoomerAMG's solve phase mainly consists of matrix vector products and smoothing operations, which are fairly straight forward to parallelize, even on a GPU, its setup phase is highly complex, including many branches, a lot of integer operations as well as some sequential passages. Previous interpolation strategies that lead to best convergence and performance on distributed memory machines were not suitable for implementation on GPUs or similar architectures requiring extreme parallelism and required new algorithmic approaches. Since {\sl hypre} is a mature product with many solvers and interdependent features, any significant changes that affect the whole library, are tedious and require much testing to ensure that the library stays backward compatible and no features are broken.

\paragraph{Solution Strategy}

Since computer architectures continue to change rapidly, it was important to come up with strategies that will facilitate future porting of the software. Therefore we developed and implemented a new memory model that addresses the use of different memory locations.
Since the upcoming computer architectures are heterogeneous with accelerators, we focus on enabling {\sl hypre} for GPUs. We have looked into various options, such as the use of CUDA, OpenMP 4.5, as well as RAJA and Kokkos. We limited the latter three options to the structured interface and solvers which are more natural candidates for such an approach due to their use of macros, called BoxLoops, for loops. We adopted a modular approach for the unstructured interface, which relies on the restructuring the solver components to use smaller kernels that are and/or will be implemented in CUDA for Nvidia GPUs. Since hip is similar to CUDA, porting to AMD GPUs has been fairly straight forward. We are now investigating the use of a vendor conversion tool from CUDA to SYCL to port structured and unstructured solvers to upcoming exascale computers with Intel GPUs.

\paragraph{Recent Progress}

Previously we had enabled the structured interface and solvers, SMG and PFMG\cite{AsFa1996}, to completely run on GPUs, using CUDA, OpenMP4.5, RAJA and Kokkos, and have now also added hip to enable use on AMD GPUs.
For our unstructured AMG solver BoomerAMG, we had implemented suitable CUDA kernels for setup and solve phase, designed a new class of interpolation operators based on sparse matrix operations\cite{LiSY2020}, implemented it on GPUs and ported aggressive coarsening to the GPU. Recently, we added Umpire support for memory pooling on GPUs, which can significantly improve performance. We have enabled hypre's specialized solvers, including linear and eigensolvers for Maxwell problems and solvers for H-div problems, to run on Nvidia GPUs. Figure \ref{fig:AMS-PCG} shows some results for a Maxwell problem using finite elements of increasing order on 1 node of Lassen. We have ported many of the CUDA kernels and routines in the unstructured interface to hip to enable their use on AMD GPUs. We have also developed a new version of multipass interpolation based on matrix-matrix multiplications and implemented it in CUDA and hip. Multipass interpolation is a popular low-memory prolongation used with aggressive coarsening. Figure \ref{fig:Spock-AMG} shows a comparison of CPU and GPU runtimes on 2 nodes of Spock for a 3D diffusion problem with a 27-point stencil on a $n \times n \times n$ grid using AMG-PCG with aggressive coarsening on the first level for increasing $n$.

\begin{figure}[bth]
\centering
	\includegraphics[width=3.9in]{projects/2.3.3-MathLibs/2.3.3.12-SUNDIALS-hypre/AMS-PCG.png}
	%\includegraphics[width=3in]{AMS-PCG.png}
	\caption{\label{fig:AMS-PCG} Solution of a simple 3D electromagnetic diffusion problem corresponding to the 2nd order definite Maxwell equation  $\nabla \times \nabla \times E + E = f$ using AMS-PCG on 1 node of Lassen (4 GPUs vs. 40 CPU cores) using finite elements of increasing order on a Fichera mesh.}

\end{figure}

\begin{figure}[bth]
\centering
	\includegraphics[width=3in]{projects/2.3.3-MathLibs/2.3.3.12-SUNDIALS-hypre/spock-AMG-PCG.png}
	%\includegraphics[width=3in]{spock-AMG-PCG.png}
	\caption{\label{fig:Spock-AMG} Total times (setup plus solve times) on 2 nodes of Spock using AMG-PCG with aggressive coarsening and multi-pass interpolation for a 3D diffusion problem with a 27-point stencil on a $n \times n \times n$ grid. The CPU runs were performed with 8 MPI tasks with 16 OpenMP threads per MPI task, the GPU runs were performed on 8 AMD GPUs.}

\end{figure}

\paragraph{Next Steps}

We will pursue the following tasks:

\begin{itemize}
\item We will continue to add new GPU capabilities to {\sl hypre} and improve the performance of current capabilities. We will thoroughly investigate the performance on AMD GPUs and begin porting to Intel GPUs.
\item We will test and evaluate performance of important hypre solvers on AMD and Intel GPUs, improve their performance and port additional components if necessary.
\end{itemize}
In addition, we will work with ECP application teams who are using {\sl hypre}, such as ExaWind, ExaAM, AMReX,  to achieve best performance by tuning the solvers for them and potentially implementing suitable algorithmic changes.