\subsection{ARMAX}


\section{Analysis}

The type of analyses we are interested in are \emph{intra-functional}, that is, we do not consider function calls. To root the design of the analysis infrastructure
we considered the four basic analyses in \cite{spa} which were also covered during the second part of the course, these are \emph{liveness}, \emph{available expressions},
\emph{very busy expressions} and \emph{reaching definitions}.

\newpar From a pure algorithmic point of view, these four analyses are striking similar: all of them operate on sets on a data structure that has a notion of successor and predecessor
and both use set union, intersection and difference as operations; this suggest an analysis framework with parametric analyses instead of a framework with \emph{hardcoded}
ones. This approach promotes code reuse, hides the details of the solver and underlying operations, e.g. set union, and it exposes an interface expressive enough to
write \emph{power set based} analyses.

%\subsection{Control Flow Graph}

\newpar The intermediate representation (of the program) upon which the analyses relay on is the Control Flow Graph with single-statement blocks (CFG). A \emph{Control Flow Graph} with \emph{single-statement blocks} is a digraph in which the control flow between non-control flow statements is modeled. This differ from a \emph{pure} control flow graph in the block (node) definition; in a typical control flow graph a block is a maximal sequence of linear statements. (\cite{cooper}).

\newpar CFG generation based on the parsed Abstract Syntax Tree (AST) has been implemented in Python, since existing AST-to-CFG parser libraries did not support generating single-statement blocks as required by this project. The CFG generation will generate a CFG with single-statement blocks given a generated TIP/C subset AST.

\newpar From each node in the CFG we require an interface to its successors (forward flow), predecessors (backward flow), left-hand side variable (if any), right-hand side expressions and type of statement.
An extra requirement for the CFG is the variables set, i.e. all the variables declared in the program and the expressions set.

The \emph{working unit} of an analysis is a set of monotone functions, and each monotone function take as an argument a node in the CFG.

To guide the construction and interfacing of the Analysis Engine (AE), we wrote \texttt{blue\_print.py} that focus on the design and inner workings of the analysis engine (represented by the class \texttt{Analysis}) while abstracting away all the implementation details about the components interfacing with it.

\newpar At the top level the AE requires a CFG and a non-empty list of monotone functions: \texttt{analysis = Analysis(cfg, monotone\_functions)}. To find the analysis' fix point to the given CFG, is just a matter of executing: \texttt{analysis.fix\_point()}. Also we designed the AE so the writing of monotone functions is as close as the mathematical formulation as possible.

\newpar To be able to find the fix point, the AE keeps a list of all user provided monotone functions (\texttt{self.monotone\_functions}) and a list \texttt{self.\_state} of size the number of nodes in the CFG. Each entry of the \texttt{self.\_state} list holds the current result of the monotone function that analysed the node. The signature of a monotone function looks like: \texttt{def join\_least\_upper\_bound (analysis, cfg\_node)} in which \texttt{analysis} is a reference to the AE and \texttt{cfg\_node} is a reference to the CFG node to be analysed. We send a reference to the AE to the monotone function so the user can have access the
\texttt{self.\_state} variable; this is useful when a function requires information from other nodes.
To find the fix point the AE iterates over all blocks in the CFG, per each block, each monotone function is applied in the same order as they were provided by the user when the AE was constructed, a block is considered to be analysed when a monotone function returns a non \texttt{None} value. Hence, a requirement is set to all monotone functions: if a monotone function does not apply to the given CFG node, then it must return \texttt{None}. This approach can be improved by caching, per CFG node, the function that returned a non \texttt{None} value.

\newpar The \texttt{Analysis} class exposes two functions to the monotone functions: \texttt{def least\_upper\_bound(self, left, right)} and \\\texttt{def greatest\_lower\_bound(self, left, right)} corresponding to the lattice functions \emph{least upper bound} and \emph{greatest lower bound}, respectively.

As an example of a monotone function implementation, we show the JOIN function w.r.t. the least upper bound:
\begin{minted}{python}
def join_least_upper_bound(analysis, cfg_node):
   club = frozenset()
   for successor in cfg_node.successors:
      club = analysis.least_upper_bound(club, analysis.state(successor))
   return club
\end{minted}

\newpar To find the least fix-point, we iterate over the nodes of the CFG, applying the corresponding monotone function depending on the \emph{type} of the node. A \texttt{\_state} variable keeps track of the latest monotone function result per CFG node. The least fix-point is found when the \texttt{\_state} variable does not change. The \texttt{\_state} variable is updated each time a monotone function is executed, as a consequence a small improvement on the time complexity might be achieved depending on the iteration order of the CFG nodes.

\subsection{Lattice based analyses}
To be able to handle general lattices and not only power set based analyses, certain changes must be made to the current implementation of \texttt{Analysis}. First, a new constructor must be added it will take the CFG and monotone functions as parameters but also will take as an additional parameter a list of pairs \texttt{lattice} that represent all the edges in the lattice. If $(x,y) \in$ \texttt{lattice} then $x < y$. From this list is possible to express the partial order completely as a matrix and use this matrix to calculate the least and greatest bounds. These changes are described in the \hyperref[futurework]{Future Work} section.


\documentclass[a4paper, 12pt]{article}
\usepackage{amsmath}
%\usepackage[utf16]{inputenc}
\usepackage{graphicx}
\usepackage[left=2.5cm, right=2.5cm, bottom=2.5cm, top=2.5cm]{geometry}
\usepackage{natbib}
\usepackage{microtype}
\usepackage{coloremoji}

\title{Measure --- or as the kids call it these days, addition}
\author{Brendon J. Brewer}
\date{}

\begin{document}
\maketitle

% Need this after the abstract
\setlength{\parindent}{0pt}
\setlength{\parskip}{8pt}

See \citet{knuth2016deeper}.

\section{Bigger and smaller}
Some things are bigger than others.

\section{How long is a piece of string?}

Consider a set, or collection, of objects, and suppose we can
unambiguously rank or sort the objects. For example, maybe we
have twenty apples, each of which has been weighed precisely.
We could put the apples in order by mass, from lowest to highest (or
the other way around).

More formally, think of a set $\mathcal{S} = \{s_1, s_2, ..., s_n\}$.
If we can take any two elements
(call them $x$ and $y$) and say that either $x > y$ or $y > x$ (where
we get to define what ``greater than'' means for a particular application),
then $\mathcal{S}$ is a {\em totally ordered set}. The definition of
$>$ that you use has to have the properties that $>$ usually does. For
example, if $x > y$ and $y > z$, then $x > z$.
Later, there will be a concept called a {\em chain} which is pretty much
the same as a partially ordered set.
If you are used to computer programming, then any
list/vector/array of items which can be unambiguously sorted
is a totally ordered set.

We can draw this situation using points on a line
(Figure~\ref{fig:totally_ordered_set}).
Each point
represents an element of the set, and they are arranged from highest
to lowest in terms of which elements are ``greater than'' other elements.
That is, if $x > y$ (element
$x$ is ranked above element $y$), then $x$ is drawn above $y$ in the
diagram.

\begin{figure}[!ht]
\centering
\includegraphics[scale=0.6]{figures/totally_ordered_set.pdf}
\caption{A set with five elements, each represented by a black dot.
When one dot is above another and they are connected with a line, it
means the higher element is greater than (using whatever definition of $>$ we
choose) the lower one.
This is a simple example of a {\em Hasse diagram}. More complex
Hasse diagrams will appear later.\label{fig:totally_ordered_set}}
\end{figure}

\section{The sum rule}

\section{Measure in boolean lattices}

\begin{figure}[!ht]
\centering
\includegraphics[width=0.5\textwidth]{figures/boolean_lattice.pdf}
\caption{\label{fig:boolean_lattice}}
\end{figure}

Apples.
One bag of apples combined with another bag of apples.
The exact same situation. The numbers of apples add, as
do the masses. Two different sum rules apply to the same
situation!

\section{Infidelity and ``signed measure''}

\subsection{A fate worse than death}

Infidelity allows for fates worse than death.
An absence of experience, denoted $\bot$,
has measure zero, so you can add arbitrary amounts
of non-existence to any experience without changing
its quality. So, for example:
\begin{align}
m(😃 \vee \bot)        &= m(😃)\\
m(😀 \vee 😪 \vee \bot) &= m(😀 \vee 😪)
\end{align}
and so on.
In other words, $\bot$ is the {\em identity element}
of the set of potential experiences.

\bibliographystyle{chicago}
\bibliography{references}

\end{document}


\documentclass[11pt,oneside]{article}    %use"amsart"insteadof"article"forAMSLaTeXformat
\usepackage{geometry}        %Seegeometry.pdftolearnthelayoutoptions.Therearelots.
\geometry{letterpaper}        %...ora4paperora5paperor...
%\geometry{landscape}        %Activateforforrotatedpagegeometry
%\usepackage[parfill]{parskip}        %Activatetobeginparagraphswithanemptylineratherthananindent
\usepackage{graphicx}                %Usepdf,png,jpg,orepsßwithpdflatex;useepsinDVImode
                                %TeXwillautomaticallyconverteps-->pdfinpdflatex
\usepackage{amssymb}
\usepackage[colorlinks]{hyperref}

%----macros begin---------------------------------------------------------------
\usepackage{color}
\usepackage{amsthm}

\def\conv{\mbox{\textrm{conv}\,}}
\def\aff{\mbox{\textrm{aff}\,}}
\def\E{\mathbb{E}}
\def\R{\mathbb{R}}
\def\Z{\mathbb{Z}}
\def\tex{\TeX}
\def\latex{\LaTeX}
\def\v#1{{\bf #1}}
\def\p#1{{\bf #1}}
\def\T#1{{\bf #1}}

\def\vet#1{{\left(\begin{array}{cccccccccccccccccccc}#1\end{array}\right)}}
\def\mat#1{{\left(\begin{array}{cccccccccccccccccccc}#1\end{array}\right)}}

\def\lin{\mbox{\rm lin}\,}
\def\aff{\mbox{\rm aff}\,}
\def\pos{\mbox{\rm pos}\,}
\def\cone{\mbox{\rm cone}\,}
\def\conv{\mbox{\rm conv}\,}
\newcommand{\homog}[0]{\mbox{\rm homog}\,}
\newcommand{\relint}[0]{\mbox{\rm relint}\,}

%----macros end-----------------------------------------------------------------

\title{Hierarchical structures with LAR
\footnote{This document is part of the \emph{Linear Algebraic Representation with CoChains} (LAR-CC) framework~\cite{cclar-proj:2013:00}. \today}
}
\author{Alberto Paoluzzi}
%\date{}                            %Activatetodisplayagivendateornodate

\begin{document}
\maketitle
\nonstopmode

\tableofcontents

%===============================================================================
\section{Affine transformations}
%===============================================================================
%-------------------------------------------------------------------------------
\subsection{Design decision}
%-------------------------------------------------------------------------------
First we state the general rules that will be satisfied by the matrices used in this module, mainly devoted to apply affine transformations to vertices of models in structure environments:
\begin{enumerate}
\item assume the scipy \texttt{ndarray} as the type of vertices, stored in row-major order;
\item use the last coordinate as the homogeneous coordinate of vertices, but do not store it explicitly;
\item store explicitly the homogeneous coordinate of transformation matrices.
\item use labels '\texttt{verts}' and '\texttt{mat}' to distinguish between vertices and transformation matrices.
\item transformation matrices are dimension-independent, and their dimension is computed as the length of the parameter vector passed to the generating function.
\end{enumerate}

%-------------------------------------------------------------------------------
\subsection{Affine mapping}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Apply an affine transformation to a LAR model
@{""" Apply an affine transformation to a LAR model  """
from scipy import array

def larApply(affineMatrix):
    def larApply0(model):
        if isinstance(model,Model):
            V = scipy.dot(array([v+[1.0] for v in model.verts]), affineMatrix.T).tolist()
            V = [v[:-1] for v in V]
            CV = copy.copy(model.cells)
            return Model((V,CV))
        elif isinstance(model,tuple) or isinstance(model,list):
            if len(model)==2: V,CV = model
            elif len(model)==3: V,CV,FV = model
            V = scipy.dot([list(v)+[1.0] for v in V], affineMatrix.T).tolist()
            if len(model)==2: return [v[:-1] for v in V],CV
            elif len(model)==3: return [v[:-1] for v in V],CV,FV
    return larApply0
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\subsection{Elementary matrices}
%-------------------------------------------------------------------------------
Elementary matrices for affine transformation of vectors in any dimensional vector space are defined here. They include translation, scaling, rotation and shearing.

\paragraph{Translation}
%-------------------------------------------------------------------------------
@D Translation matrices
@{def t(*args):
    d = len(args)
    mat = scipy.identity(d+1)
    for k in range(d):
        mat[k,d] = args[k]
    return mat.view(Mat)
@}
%-------------------------------------------------------------------------------
\paragraph{Scaling}
%-------------------------------------------------------------------------------
@D Scaling matrices
@{def s(*args):
    d = len(args)
    mat = scipy.identity(d+1)
    for k in range(d):
        mat[k,k] = args[k]
    return mat.view(Mat)
@}
%-------------------------------------------------------------------------------
\paragraph{Rotation}
%-------------------------------------------------------------------------------
@D Rotation matrices
@{def r(*args):
    args = list(args)
    n = len(args)
    @< plane rotation (in 2D) @>
    @< space rotation (in 3D) @>
    return mat.view(Mat)
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D plane rotation (in 2D)
@{if n == 1: # rotation in 2D
    angle = args[0]; cos = COS(angle); sin = SIN(angle)
    mat = scipy.identity(3)
    mat[0,0] = cos;    mat[0,1] = -sin;
    mat[1,0] = sin;    mat[1,1] = cos;
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D space rotation (in 3D)
@{if n == 3: # rotation in 3D
    mat = scipy.identity(4)
    angle = VECTNORM(args); axis = UNITVECT(args)
    cos = COS(angle); sin = SIN(angle)
    @< elementary rotations (in 3D) @>
    @< general rotations (in 3D) @>
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D elementary rotations (in 3D)
@{if axis[1]==axis[2]==0.0:    # rotation about x
    mat[1,1] = cos;    mat[1,2] = -sin;
    mat[2,1] = sin;    mat[2,2] = cos;
elif axis[0]==axis[2]==0.0:    # rotation about y
    mat[0,0] = cos;    mat[0,2] = sin;
    mat[2,0] = -sin;    mat[2,2] = cos;
elif axis[0]==axis[1]==0.0:    # rotation about z
    mat[0,0] = cos;    mat[0,1] = -sin;
    mat[1,0] = sin;    mat[1,1] = cos;
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D general rotations (in 3D)
@{else:        # general 3D rotation (Rodrigues' rotation formula)
    I = scipy.identity(3) ; u = axis
    Ux = scipy.array([
        [0,        -u[2],      u[1]],
        [u[2],        0,     -u[0]],
        [-u[1],     u[0],         0]])
    UU = scipy.array([
        [u[0]*u[0],    u[0]*u[1],    u[0]*u[2]],
        [u[1]*u[0],    u[1]*u[1],    u[1]*u[2]],
        [u[2]*u[0],    u[2]*u[1],    u[2]*u[2]]])
    mat[:3,:3] = cos*I + sin*Ux + (1.0-cos)*UU
@}

%===============================================================================
\section{Structure types handling}
%===============================================================================

In order to implement a structure as a list of models and transformations, we need to be able to distinguish between two different types of scipy arrays. The first type is the one of arrays of vertices, the second one is the matrix array used to represent the fine transformations.

\subsection{\texttt{Mat} and \texttt{Verts} classes}
%-------------------------------------------------------------------------------
@D types Mat and Verts
@{""" class definitions for LAR """
import scipy
class Mat(scipy.ndarray): pass
class Verts(scipy.ndarray): pass
@}
%-------------------------------------------------------------------------------

\subsection{\texttt{Model} class}
%-------------------------------------------------------------------------------
@D Model class
@{class Model:
    """ A pair (geometry, topology) of the LAR package """
    def __init__(self,(verts,cells)):
        self.n = len(verts[0])
        # self.verts = scipy.array(verts).view(Verts)
        self.verts = verts
        self.cells = cells
    def __getitem__(self,i):
        return list((self.verts,self.cells))[i]
@}
%-------------------------------------------------------------------------------

\subsection{\texttt{Struct} iterable class}
%-------------------------------------------------------------------------------
@D Struct class
@{""" Struct iterable class """
class Struct:
    """ The assembly type of the LAR package """
    def __init__(self,data=None,name=None,category=None):
        if data==None or data==[]:
            self.body = []
        else:
            self.body = [item for item in data if item != None]
            self.box = box(self)
            self.dim = len(self.box[0])
        if name != None:
            self.name = str(name)
        else:
            self.name = str(id(self))
        if category != None:
            self.category = str(category)
        else:
            self.category = "feature"
    def __name__(self):
        return self.name
    def __category__(self):
        return self.category
    def __iter__(self):
        return iter(self.body)
    def __len__(self):
        return len(list(self.body))
    def __getitem__(self,i):
        return list(self.body)[i]
    def __setitem__(self,i,value):
        self.body[i] = value
    def __print__(self):
        return "<Struct name: %s>" % self.__name__()
    def __repr__(self):
        return "<Struct name: %s>" % self.__name__()
        #return "'Struct(%s,%s)'" % (str(self.body),str(str(self.__name__())))
    def set_name(self,name):
        self.name = str(name)
    def clone(self,i=0):
        from copy import deepcopy
        newObj = deepcopy(self)
        if i != 0: newObj.name = self.name + "_" + str(i)
        return newObj
    def set_category(self,category):
        self.category = str(category)
    def larBoundary(self):
        data = struct2lar(self)
        if len(data) == 3:
            V,FV,EV = data
            #import pdb; pdb.set_trace()
            return V,FV,EV
        else:
            return "<Struct name: %s> boundary non computable" % self.__name__()
    def draw(self,color=WHITE,scaling=1,metric=ID):
        vmin,vmax = self.box
        delta = VECTDIFF([vmax,vmin])
        point = CCOMB(self.box)
        scalingFactor = scaling*delta[0]/20.
        text = TEXTWITHATTRIBUTES (TEXTALIGNMENT='centre', TEXTANGLE=0,
                    TEXTWIDTH=0.1*scalingFactor,
                    TEXTHEIGHT=0.2*scalingFactor,
                    TEXTSPACING=0.025*scalingFactor)
        point = metric([point])[0]
        return T([1,2,3])(point)(COLOR(color)(text(self.name)))
@}
%-------------------------------------------------------------------------------

\subsection{\texttt{Struct} containment box}

%-------------------------------------------------------------------------------
@D Computation of the containment box of a Lar Struct or Model
@{""" Computation of the containment box of a Lar Struct or Model """
import copy
def box(model):
    if isinstance(model,Mat): return []
    elif isinstance(model,Struct):
        dummyModel = copy.deepcopy(model)
        dummyModel.body = [term if (not isinstance(term,Struct)) else [term.box,[[0,1]]]  for term in model.body]
        listOfModels = evalStruct( dummyModel )
        #dim = checkStruct(listOfModels)
        theMin,theMax = box(listOfModels[0])
        for theModel in listOfModels[1:]:
            modelMin, modelMax = box(theModel)
            theMin = [val if val<theMin[k] else theMin[k] for k,val in enumerate(modelMin)]
            theMax = [val if val>theMax[k] else theMax[k] for k,val in enumerate(modelMax)]
        return [theMin,theMax]
    elif isinstance(model,Model):
        V = model.verts
    elif (isinstance(model,tuple) or isinstance(model,list)) and (len(model)==2 or len(model)==3):
        V = model[0]
    coords = TRANS(V)
    theMin = [min(coord) for coord in coords]
    theMax = [max(coord) for coord in coords]
    return [theMin,theMax]
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
\section{Structure to LAR conversion}
%-------------------------------------------------------------------------------

\subsection{Remove duplicate faces}
%-------------------------------------------------------------------------------
@D Remove duplicate faces
@{""" Remove duplicate faces  """
from collections import defaultdict
def removeDups (CW):
    CW = list(set(AA(tuple)(CW)))
    CWs = list(set(AA(tuple)  (AA(sorted)(CW))  ))
    no_duplicates = defaultdict(list)
    for f in CWs: no_duplicates[f] = []
    for f in CW:
        no_duplicates[tuple(sorted(f))] += [f]
    CW = [f[0] for f in no_duplicates.values()]
    return CW
@}
%-------------------------------------------------------------------------------

\subsection{Structure to pair (Vertices,Cells) conversion}
%-------------------------------------------------------------------------------
@D Structure to pair (Vertices,Cells) conversion
@{""" Structure to pair (Vertices,Cells) conversion """

def struct2lar(structure,metric=ID):
    listOfModels = evalStruct(structure)
    vertDict = dict()
    index,defaultValue,CW,W,FW = -1,-1,[],[],[]

    for model in listOfModels:
        if isinstance(model,Model):
            V,FV = model.verts,model.cells
        elif (isinstance(model,tuple) or isinstance(model,list)):
            if len(model)==2: V,FV = model
            elif len(model)==3: V,FV,EV = model
        for k,incell in enumerate(FV):
            outcell = []
            for v in incell:
                key = vcode(4)(V[v])
                if vertDict.get(key,defaultValue) == defaultValue:
                    index += 1
                    vertDict[key] = index
                    outcell += [index]
                    W += [eval(key)]
                else:
                    outcell += [vertDict[key]]
            CW += [outcell]
        if len(model)==3:
            for k,incell in enumerate(EV):
                outcell = []
                for v in incell:
                    key = vcode(4)(V[v])
                    if vertDict.get(key,defaultValue) == defaultValue:
                        index += 1
                        vertDict[key] = index
                        outcell += [index]
                        W += [eval(key)]
                    else:
                        outcell += [vertDict[key]]
                FW += [outcell]

    if ((isinstance(model,tuple) or isinstance(model,list)) and len(model)==2) or (
        (isinstance(model,Model) and model.n==2)):
        if len(CW[0])==2:
            CW = list(set(AA(tuple)(AA(sorted)(CW))))
        else: CW = removeDups(CW)
        return metric(W),CW
    if ((isinstance(model,tuple) or isinstance(model,list)) and len(model)==3) or (
        (isinstance(model,Model) and model.n==3)):
        FW = list(set(AA(tuple)(AA(sorted)(FW))))
        CW = removeDups(CW)
        return metric(W),CW,FW
@}
%-------------------------------------------------------------------------------

\subsection{Model simplification}

\paragraph{Remove double instances of cells}

%-------------------------------------------------------------------------------
@O test/py/larstruct/test10.py
@{""" Remove double instances of cells (and the unused vertices) """
from larlib import *

@< Transform Struct object to LAR model pair @>
@< Remove the double instances of cells @>
VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS((W,FW))))

@< Remove the unused vertices @>
@}
%-------------------------------------------------------------------------------

The actual removal of double cells (useful in several applications, and in particular in the extraction of boundary models from 3D medical images) is performed by first generating a dictionary of cells, using as key the tuple given by the cells themselves, and then removing those discovered having a double instance.
The algorithm is extremely simple, and its implementation, given below, is straightforward.

%-------------------------------------------------------------------------------
@D Remove the double instances of cells
@{""" Remove the double instances of cells """
cellDict = defaultdict(list)
for k,cell in enumerate(FW):
    cellDict[tuple(cell)] += [k]
FW = [list(key) for key in cellDict.keys() if len(cellDict[key])==1]
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Remove the unused vertices
@{""" Remove the unused vertices """
print "len(W) =",len(W)
V,FV = larRemoveVertices(W,FW)
print "len(V) =",len(V)
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Remove the unused vertices from a LAR model pair
@{""" Remove the unused vertices """
def larRemoveVertices(V,FV):
    vertDict = dict()
    index,defaultValue,FW,W = -1,-1,[],[]

    for k,incell in enumerate(FV):
        outcell = []
        for v in incell:
            key = vcode(4)(V[v])
            if vertDict.get(key,defaultValue) == defaultValue:
                index += 1
                vertDict[key] = index
                outcell += [index]
                W += [eval(key)]
            else:
                outcell += [vertDict[key]]
        FW += [outcell]
    return W,FW
@}
%-------------------------------------------------------------------------------

\subsection{Embedding or projecting LAR models}
%-------------------------------------------------------------------------------

In order to apply 3D transformations to a two-dimensional LAR model, we must embed it in 3D space, by adding one more coordinate to its vertices.

\paragraph{Embedding or projecting a geometric model}

This task is performed by the function \texttt{larEmbed} with parameter $k$, that inserts its $d$-dimensional geometric argument in the $x_{d+1}, \ldots, x_{d+k}=0$ subspace of $\E^{d+k}$.
A projection transformation, that removes the last $k$ coordinate of vertices, without changing the object topology, is performed by the function \texttt{larEmbed} with \emph{negative} integer parameter.

%-------------------------------------------------------------------------------
@D Embedding and projecting a geometric model
@{def larEmbed(k):
    def larEmbed0(model):
        if len(model)==2: V,CV = model
        elif len(model)==3: V,CV,FV = model
        if k>0:
            V = [v+[0.]*k for v in V]
        elif k<0:
            V = [v[:-k] for v in V]
        if len(model)==2: return V,CV
        elif len(model)==3: return V,CV,FV
    return larEmbed0

def larEmbed(k):
    def larEmbed0(model):
        if k>0:
            model[0] = [v+[0.]*k for v in model[0]]
        elif k<0:
            model[0] = [v[:-k] for v in model[0]]
        return model
    return larEmbed0
@}
%-------------------------------------------------------------------------------

%===============================================================================
\section{Hierarchical complexes}
%===============================================================================
Hierarchical models of complex assemblies are generated by an aggregation
of subassemblies, each one defined in a local coordinate system, and
relocated by affine transformations of coordinates.  This operation
may be repeated hierarchically, with some subassemblies defined by
aggregation of simpler parts, and so on, until one obtains a set of
elementary components, which cannot be further decomposed.

Two main advantages can be found in a hierarchical modeling approach. Each elementary part and each assembly, at every hierarchical level, are defined independently from each other, using a local coordinate frame, suitably chosen to make its definition easier. Furthermore, only one copy of each component is stored in the memory, and may be instanced in different locations and orientations how many times it is needed.

%-------------------------------------------------------------------------------
\subsection{Traversal of hierarchical structures}
%-------------------------------------------------------------------------------

Of course, the main algorithm with hierarchical structures is the \emph{traversal} of the structure network, whose aim is to transform every encountered object from local to global coordinates, where the global coordinates are those of the network root (the only node with indegree zero).

A structure network can be modelled using a directed acyclic multigraph, i.e.~a triple $(N,A,f)$
made by a set $N$ of nodes, a set $A$ of arcs, and a function $f:A \to N^2$ from arcs to ordered pairs of nodes. Conversely that in standard oriented graphs, in this kind of structure more than one oriented arc is allowed between the same pair on nodes.

\begin{figure}[htbp] %  figure placement: here, top, bottom, or page
   \centering
   \includegraphics[width=0.8\linewidth]{images/traversal}
   \caption{Traversal algorithm of an acyclic multigraph.}
   \label{fig:traversal}
\end{figure}

A simple modification of a DFS (Depth First Search) visit of a graph can be used to traverse the structure network This algorithm is given in Figure~\ref{fig:traversal} from~\cite{Paoluzzi2003a}.

\subsubsection{Traversal of nested lists}

The representation chosen for structure networks with LAR is the serialised one, consisting in ordered sequences (lists) of either (a) LAR models, or (b) affine transformations, or (c) references to other structures, either directly nested within some given structure, or called by reference (name) from within the list.

The usual aim of a structure network traversal is, of course, to transform every component structure, usually defined in a local coordinate system, into the reference frame of the structure as a whole, normally corresponding with the reference system of the structure's root, called the \emph{world coordinate} system.

\paragraph{The pattern of calls and returned values}

In order to better understand the behaviour of the traversal algorithm, where every transformation is applied to all the following models, --- but only if included in the same structure  (i.e.~list) --- it may be very useful to start with an \emph{algorithm emulation}. In particular, the recursive script below discriminates between three different cases (number, string, or sequence), whereas the actual traversal must do with (a) Models, (b) Matrices, and (c) Structures, respectively.

%-------------------------------------------------------------------------------
@D Emulation of scene multigraph traversal
@{""" Emulation of scene multigraph traversal """
from larlib import *

def __traverse(CTM, stack, o):
    for i in range(len(o)):
        if ISNUM(o[i]): print o[i], REVERSE(CTM)
        elif ISSTRING(o[i]):
            CTM.append(o[i])
        elif ISSEQ(o[i]):
            stack.append(o[i])                # push the stack
            __traverse(CTM, stack, o[i])
            CTM = CTM[:-len(stack)]         # pop the stack

def algorithm(data):
    CTM,stack = ["I"],[]
    __traverse(CTM, stack, data)
@}
%-------------------------------------------------------------------------------

Some use example of the above algorithm are provided below. The printout produced at run time is shown from the \texttt{emulation of traversal algorithm} macro.
%-------------------------------------------------------------------------------
@D Examples of multigraph traversal
@{data = [1,"A", 2, 3, "B", [4, "C", 5], [6,"D", "E", 7, 8], 9]
print algorithm(data)
>>> 1 ['I']
    2 ['A', 'I']
    3 ['A', 'I']
    4 ['B', 'A', 'I']
    5 ['C', 'B', 'A', 'I']
    6 ['B', 'A', 'I']
    7 ['E', 'D', 'B', 'A', 'I']
    8 ['E', 'D', 'B', 'A', 'I']
    9 ['B', 'A', 'I']

data = [1,"A", [2, 3, "B", 4, "C", 5, 6,"D"], "E", 7, 8, 9]
print algorithm(data)
>>> 1 ['I']
    2 ['A', 'I']
    3 ['A', 'I']
    4 ['B', 'A', 'I']
    5 ['C', 'B', 'A', 'I']
    6 ['C', 'B', 'A', 'I']
    7 ['E', 'A', 'I']
    8 ['E', 'A', 'I']
    9 ['E', 'A', 'I']
@}
%-------------------------------------------------------------------------------
%-------------------------------------------------------------------------------
@D Emulation of traversal algorithm
@{dat = [2, 3, "B", 4, "C", 5, 6,"D"]
print algorithm(dat)
>>> 2 ['I']
    3 ['I']
    4 ['B', 'I']
    5 ['C', 'B', 'I']
    6 ['C', 'B', 'I']
data = [1,"A", dat, "E", 7, 8, 9]
print algorithm(data)
>>> 1 ['I']
    2 ['A', 'I']
    3 ['A', 'I']
    4 ['B', 'A', 'I']
    5 ['C', 'B', 'A', 'I']
    6 ['C', 'B', 'A', 'I']
    7 ['E', 'A', 'I']
    8 ['E', 'A', 'I']
    9 ['E', 'A', 'I']
@}
%-------------------------------------------------------------------------------

\paragraph{Traversal of a scene multigraph}

The previous traversal algorithm is here customised for scene multigraph, where the objects are LAR models, i.e.~pairs of vertices of type '\texttt{Verts} and cells, and where the transformations are matrix transformations of type '\texttt{Mat}'.

\paragraph{Check models for common dimension}
The input list of a call to \texttt{larStruct} primitive is preliminary checked for uniform dimensionality of the enclosed LAR models and transformations. The common dimension \texttt{dim} of models and matrices is returned by the function \texttt{checkStruct}, within the class definition \texttt{Struct} in the module \texttt{lar2psm}. Otherwise, an exception is generated (TODO).

%-------------------------------------------------------------------------------
@D Check for dimension of a structure element (Verts or V)
@{@< Flatten a list @>
def checkStruct(lst):
    """ Return the common dimension of structure elements.

        TODO: aggiungere test sulla dimensione minima delle celle (legata a quella di immersione)
    """
    obj = lst[0]
    if (isinstance(obj,tuple) or isinstance(obj,list)):
        dim = len(obj[0][0])
    elif isinstance(obj,Model):
        dim = obj.n
    elif isinstance(obj,Mat):
        dim = obj.shape[0]-1
    elif isinstance(obj,Struct):
        dim = len(obj.box[0])
    return dim
@}
%-------------------------------------------------------------------------------

\paragraph{Flatten a list using Python generators}
The \texttt{flatten} is a generator that yields the non-list values of its input in order. In the example, the generator is converted back to a list before printing. Modified from \href{http://rosettacode.org/wiki/Flatten_a_list#Python}{\emph{Rosetta code}} project. It is used here to flatten a structure in order to check for common dimensionality of elements.

%-------------------------------------------------------------------------------
@D Flatten a list
@{""" Flatten a list using Python generators """
def flatten(lst):
    for x in lst:
        if (isinstance(x,tuple) or isinstance(x,list)) and len(x)==2:
            yield x
        elif (isinstance(x,tuple) or isinstance(x,list)):
            for x in flatten(x):
                yield x
        elif isinstance(x, Struct):
            for x in flatten(x.body):
                yield x
        else:
            yield x

#  lst = [[1], 2, [[3,4], 5], [[[]]], [[[6]]], 7, 8, []]
#  print list(flatten(lst))
#  [1, 2, 3, 4, 5, 6, 7, 8]

#  import itertools
#  chain = itertools.chain.from_iterable([[1,2],[3],[5,89],[],[6]])
#  print(list(chain))
#  [1, 2, 3, 5, 89, 6]    ###  TODO: Bug coi dati sopra?
@}
%-------------------------------------------------------------------------------

\paragraph{Initialization and call of the algorithm}

The function \texttt{evalStruct} is used to evaluate a structure network, i.e.~to return a \texttt{scene}
list of objects of type \texttt{Model}, all referenced in the world coordinate system. The input variable \texttt{struct} must contain an object of class \texttt{Struct}, i.e.~a reference to an unevaluated structure network. The variable \texttt{dim}  contains the embedding dimension of the structure, i.e.~the number of coordinates of its vertices (normally either 2 or 3), the CTM (Current Transformation Matrix) is initialised to the (homogeneous) identity matrix, and  the \texttt{scene} is returned by calling the \texttt{traverse} algorithm.

%-------------------------------------------------------------------------------
@D Traversal of a scene multigraph
@{""" Traversal of a scene multigraph """
@< Structure traversal algorithm @>
def evalStruct(struct):
    dim = checkStruct(struct.body)
    CTM, stack = scipy.identity(dim+1), []
    scene = traversal(CTM, stack, struct, [])
    return scene
@}
%-------------------------------------------------------------------------------

\paragraph{Structure traversal algorithm}

The \texttt{traversal} algorithm decides between three different cases, depending on the type of the currently inspected object. If the object is a \texttt{Model} instance, then applies to it the \texttt{CTM} matrix; else if the object is a \texttt{Mat} instance, then the \texttt{CTM} matrix is updated by (right) product with it; else if the object is a \texttt{Struct} instance, then the \texttt{CTM} is pushed on the stack, initially empty, then the \texttt{traversal} is called (recursion), and finally, at (each) return from recursion, the \texttt{CTM} is recovered by popping the stack.

%-------------------------------------------------------------------------------
@D Structure traversal algorithm
@{def traversal(CTM, stack, obj, scene=[]):
    for i in range(len(obj)):
        if isinstance(obj[i],Model):
            scene += [larApply(CTM)(obj[i])]
        elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and (
                len(obj[i])==2 or len(obj[i])==3):
            scene += [larApply(CTM)(obj[i])]
        elif isinstance(obj[i],Mat):
            CTM = scipy.dot(CTM, obj[i])
        elif isinstance(obj[i],Struct):
            stack.append(CTM)
            traversal(CTM, stack, obj[i], scene)
            CTM = stack.pop()
    return scene
@}
%-------------------------------------------------------------------------------

\paragraph{Structure embedding algorithm}
%-------------------------------------------------------------------------------
@D Embed a struct object
@{""" embed a struct object """
@< Structure embedding algorithm  @>

def embedStruct(n):
    def embedStruct0(struct,suffix="New"):
        if n==0:
            return struct, len(struct.box[0])
        cloned = Struct()
        cloned.box = hstack((struct.box, [n*[0],n*[0]])).tolist()
        cloned.name = str(id(cloned))  #struct.name+suffix
        cloned.category = struct.category
        cloned.dim = struct.dim + n
        cloned = embedTraversal(cloned,struct,n,suffix)
        return cloned
    return embedStruct0
@}
%-------------------------------------------------------------------------------

\paragraph{Structure embedding algorithm}
%-------------------------------------------------------------------------------
@D Structure embedding algorithm
@{""" Structure embedding algorithm """
def embedTraversal(cloned, obj,n,suffix):
    for i in range(len(obj)):
        if isinstance(obj[i],Model):
            cloned.body += [obj[i]]
        elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and (
                len(obj[i])==2):
            V,EV = obj[i]
            V = [v+n*[0.0] for v in V]
            cloned.body  += [(V,EV)]
        elif (isinstance(obj[i],tuple) or isinstance(obj[i],list)) and (
                len(obj[i])==3):
            V,FV,EV = obj[i]
            V = [v+n*[0.0] for v in V]
            cloned.body  += [(V,FV,EV)]
        elif isinstance(obj[i],Mat):
            mat = obj[i]
            d,d = mat.shape

            newMat = scipy.identity(d+n*1)
            for h in range(d-1):
                for k in range(d-1):
                    newMat[h,k] = mat[h,k]
                newMat[h,d-1+n*1] = mat[h,d-1]
            cloned.body  +=  [newMat.view(Mat)]

        elif isinstance(obj[i],Struct):
            newObj = Struct()
            newObj.box = hstack((obj[i].box, [n*[0],n*[0]]))
            newObj.name = obj[i].name+suffix
            newObj.category = obj[i].category
            cloned.body  += [embedTraversal(newObj, obj[i], n, suffix)]
    return cloned
@}
%-------------------------------------------------------------------------------

%===============================================================================
\section{Larstruct exporting}
%===============================================================================
\label{sec:larstruct}
Here we assemble top-down the \texttt{lar2psm} module, by orderly listing the functional parts it is composed of. Of course, this one is the module version corresponding to the current state of the system, i.e.~to a very initial state. Other functions will be added when needed.
%------------------------------------------------------------------
@O larlib/larlib/larstruct.py
@{"""Module with functions needed to interface LAR with pyplasm via Struct"""
from larlib import *

@< Symbolic utility to represent points as strings @>
@< Translation matrices @>
@< Scaling matrices @>
@< Rotation matrices @>
@< Embedding and projecting a geometric model @>
@< Apply an affine transformation to a LAR model @>
@< Check for dimension of a structure element (Verts or V) @>
@< Remove duplicate faces @>
@< Remove the unused vertices from a LAR model pair @>
@< Traversal of a scene multigraph @>
@< types Mat and Verts @>
@< Model class @>
@< Struct class @>
@< Structure to pair (Vertices,Cells) conversion @>
@< Embedding and projecting a geometric model @>
@< Embed a struct object @>
@< Computation of the containment box of a Lar Struct or Model @>
@}
%------------------------------------------------------------------

%-------------------------------------------------------------------------------
%===============================================================================
\section{Examples}
%===============================================================================
%-------------------------------------------------------------------------------
Some examples of structures as combinations of LAR models and affine transformations are given in this section.

\paragraph{Global coordinates}
We start with a simple 2D example of a non-nested list of translated 2D object instances and rotation about the origin.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test04.py
@{""" Example of non-nested structure with translation and rotations """
from larlib import *

square = larCuboids([1,1])
table = larApply( t(-.5,-.5) )(square)
chair = larApply( s(.35,.35) )(table)
chair1 = larApply( t(.75, 0) )(chair)
chair2 = larApply( r(PI/2) )(chair1)
chair3 = larApply( r(PI/2) )(chair2)
chair4 = larApply( r(PI/2) )(chair3)
VIEW(SKEL_1(STRUCT(MKPOLS(table)+MKPOLS(chair1)+
                   MKPOLS(chair2)+MKPOLS(chair3)+MKPOLS(chair4))))
@}
%-------------------------------------------------------------------------------

\paragraph{Local coordinates}
A different composition of transformations, from local to global coordinate frames, is used in the following example.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test05.py
@{""" Example of non-nested structure with translation and rotations """
from larlib import *

square = larCuboids([1,1])
#square = Model(square)
table = larApply( t(-.5,-.5) )(square)
chair = larApply( s(.35,.35) )(table)
chair = larApply( t(.75, 0) )(chair)
struct = Struct([table] + 4*[chair, r(PI/2)])
scene = evalStruct(struct)
VIEW(SKEL_1(STRUCT(CAT(AA(MKPOLS)(scene)))))
@}
%-------------------------------------------------------------------------------

\paragraph{Call of nested structures by reference}
Finally, a similar 2D example is given, by nesting one (or more) structures via separate definition and call by reference from the interior. Of course, a cyclic set of calls must be avoided, since it would result in a \emph{non acyclic} multigraph of the structure network.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test06.py
@{""" Example of nested structures with translation and rotations """
from larlib import *

square = larCuboids([1,1])
#square = Model(square)
table = larApply( t(-.5,-.5) )(square)
chair = Struct([ t(.75, 0), s(.35,.35), table ])
struct = Struct( [t(2,1)] + [table] + 4*[r(PI/2), chair])
struct = Struct(10*[struct,t(0,2.5)])
struct = Struct(10*[struct,t(3,0)])
scene = evalStruct(struct)
VIEW(SKEL_1(STRUCT(CAT(AA(MKPOLS)(scene)))))
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@O test/py/larstruct/test08.py
@{""" LAR model input and handling """
from larlib import *

@< Input of LAR architectural plan @>
dwelling = larApply(t(3,0))(Model((V,FV)))
print "\n dwelling =",dwelling
VIEW(EXPLODE(1.2,1.2,1)(MKPOLS((dwelling.verts,dwelling.cells))))
VIEW(EXPLODE(1.2,1.2,1)(MKPOLS((dwelling.verts,EV))))
plan = Struct([dwelling,s(-1,1),dwelling])
VIEW(EXPLODE(1.2,1.2,1)(CAT(AA(MKPOLS)(evalStruct(plan)))))
@}
%-------------------------------------------------------------------------------

%-------------------------------------------------------------------------------
@D Input of LAR architectural plan
@{""" Input of LAR architectural plan """
from larlib import *

V = [[3,-3],
[9,-3],[0,0],[3,0],[9,0],[15,0],
[3,3],[6,3],[9,3],[15,3],[21,3],
[0,9],[6,9],[15,9],[18,9],[0,13],
[6,13],[9,13],[15,13],[18,10],[21,10],
[18,13],[6,16],[9,16],[9,17],[15,17],
[18,17],[-3,24],[6,24],[15,24],[-3,13]]
FV = [
[22,23,24,25,29,28], [15,16,22,28,27,30], [18,21,26,25],
[13,14,19,21,18], [16,17,23,22], [11,12,16,15],
[9,10,20,19,14,13], [2,3,6,7,12,11], [0,1,4,8,7,6,3],
[4,5,9,13,18,17,16,12,7,8],[17,18,25,24,23]]

polylines = lar2polylines((V,FV))
lines = CAT([zip(polyline[:-1],polyline[1:]) for polyline in polylines])
verts = dict(zip(AA(vcode(4))(V),range(len(V))))
edges = [tuple(sorted([verts[vcode(4)(v1)], verts[vcode(4)(v2)]])) for v1,v2 in lines]
EV = list(set(edges))
@}
%-------------------------------------------------------------------------------

\paragraph{Transformation of Struct object to LAR model pair}

The following test application first generates a grid $3\times 3$ of LAR cubes, extracts
its boundary cells as \texttt{BV}, then produces a \texttt{struct} object with 30 translated instances of it,
and finally transforms the \texttt{struct} object into a LAR pair \texttt{W,FW}.
Let us notice that due to the assembly process, some 2-cells in \texttt{FW} are doubled.

%-------------------------------------------------------------------------------
@O test/py/larstruct/test09.py
@{""" Transformation of Struct object to LAR model pair """
from larlib import *

@< Transform Struct object to LAR model pair @>
@}
%-------------------------------------------------------------------------------

The actual generation of the structure and its transformation to a LAR model pair is actually performed in the following macro.

%-------------------------------------------------------------------------------
@D Transform Struct object to LAR model pair
@{""" Generation of Struct object and transform to LAR model pair """
cubes = larCuboids([10,10,10],True)
V = cubes[0]
FV = cubes[1][-2]
CV = cubes[1][-1]
bcells = boundaryCells(CV,FV)
BV = [FV[f] for f in bcells]
VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS((V,BV))))

block = Model((V,BV))
struct = Struct(10*[block, t(10,0,0)])
struct = Struct(10*[struct, t(0,10,0)])
struct = Struct(3*[struct, t(0,0,10)])
W,FW = struct2lar(struct)

VIEW(EXPLODE(1.2,1.2,1.2)(MKPOLS((W,FW))))
@}
%-------------------------------------------------------------------------------

%===============================================================================
\appendix
%===============================================================================
\subsection{Importing a generic module}
First we define a parametric macro to allow the importing of \texttt{larcc} modules from the project repository \texttt{lib/py/}. When the user needs to import some project's module, she may call this macro as done in Section~\ref{sec:lar2psm}.
%------------------------------------------------------------------
@d Import the module
@{import sys; sys.path.insert(0, 'lib/py/')
import @1
@}
%------------------------------------------------------------------

\paragraph{Importing a module} A function used to import a generic \texttt{lacccc} module within the current environment is also useful.
%------------------------------------------------------------------
@d Function to import a generic module
@{def importModule(moduleName):
    @< Import the module @(moduleName@) @>
@| importModule @}
%------------------------------------------------------------------

\subsection{Numeric utilities}

A small set of utility functions is used to transform a \emph{point} representation, given as array of coordinates, into a string of fixed format to be used as point key into python dictionaries.

%------------------------------------------------------------------
@D Symbolic utility to represent points as strings
@{""" TODO: use package Decimal (http://docs.python.org/2/library/decimal.html) """
#global PRECISION
#PRECISION = 4.

def verySmall(number): return abs(number) < 10**-(PRECISION)

def prepKey (args): return "["+", ".join(args)+"]"

def fixedPrec(PRECISION):
    def fixedPrec0(value):
        out = round(value*10**(PRECISION))/10**(PRECISION)
        if out == -0.0: out = 0.0
        return str(out)
    return fixedPrec0

def vcode (PRECISION=4):
    def vcode0 (vect):
        """
        To generate a string representation of a number array.
        Used to generate the vertex keys in PointSet dictionary, and other similar operations.
        """
        return prepKey(AA(fixedPrec(PRECISION))(vect))
    return vcode0
@}
%------------------------------------------------------------------

\bibliographystyle{amsalpha}
\bibliography{larstruct}

\end{document}


\chapter{Related Work}

Much work has been done in the area of alias analysis, both in proposing new alias analysis techniques, and for evaluating such techniques. Because alias analysis is an undecidable problem \cite{undecidable}, many potential avenues exist for developing more precise or efficient approximations.

\section{Current Alias Analyses}
Several existing alias analyses are currently used as part of contemporary compiler optimizations. These analyses take different approaches in identifying aliases, and thus vary in terms of efficiency and effectiveness. We are interested in the alias analysis implementations that target LLVM instructions. The following example C program is used to demonstrate some of the differences between the following alias analyses.

\begin{figure} [h]
   \lstinputlisting[language=C, basicstyle=\ttfamily\scriptsize, tabsize=8, numbers=left] {figures/codex.c}
   \caption{Example Program for Evaluating Alias Analyses}
   \label{fig:codex}
\end{figure}

\subsection{Andersen Analysis}
Andersen's Alias Analysis \cite{Andersen} is an analysis technique for determining pointer aliases within functions without considering program flow. Andersen analysis provides set notation and type inference rules meant for the C programming language. Pointers are initially stated to be part of specific types of pointers, such as global variables, dynamically allocated memory, and function parameters. Additional type inference rules are used to represent different operations performed with pointers, such as dereferencing, assignment, and type casting. These rules are used to generate set constraints for the pointer values within a function. For alias analysis across function calls, Andersen analysis uses static call graphs and additional inference rules to generate context-sensitive constraints. The generated constraints for the aliases are solved by using a set of rewriting rules for type normalization and propagation; the constraint solving algorithm's time complexity is polynomial in terms of program size.

In the example program in Figure \ref{fig:codex}, constraints are generated for the statements on lines 5, 6, 7, 12, 15, and 18, which all involve assignment of pointer values. After resolving these constraints, \texttt{x} is found to reference \texttt{a}, \texttt{y} is found to reference \texttt{b} and an unknown pointer, and \texttt{z} is found to reference \texttt{a} and \texttt{b}. The conditional statement only collects constraints within its then and else clauses, and as statements are processed, possible aliases for pointers are only added; this can result in some inaccuracies, such as with analyzing \texttt{y} and \texttt{z}. Additionally, expressions with binary operators, such as those in lines 12 and 15, are dependent on functions that generate constraints based on the operator and the operands; for the first statement, the constraint refers to an unknown pointer due to the operator subtracting an integer from a pointer to a single variable, leading into an unknown area, and for the second statement, the constraint refers to an unknown pointer because the difference of two pointers is considered an integer.

\subsection{Steensgaard Analysis}
Steensgaard Alias Analysis \cite{Steensgaard} is another alias analysis technique that works across function calls without considering program flow. Steensgaard analysis is based off an abstract pointer-based language that includes pointer operations, n-ary operators, dynamic memory, and functions. Type inference rules are used to generate alias sets for the pointer variables in the program. Each statement is initially processed once to generate the initial set of pointers; these values are stored in a union-find data structure, and are combined in alias sets in subsequent join operations. The resulting algorithm was found to be linear in space complexity and almost linear in terms of time complexity.

In the example program in Figure \ref{fig:codex}, the types for \texttt{a}, \texttt{b}, \texttt{c}, \texttt{x}, \texttt{y}, and \texttt{z} are discovered as distinct values in the first pass. After processing each statement once in the second pass of the algorithm, \texttt{x} refers to \texttt{a}, \texttt{y} refers to \texttt{a} and \texttt{b}, and \texttt{z} refers to \texttt{a} and \texttt{b}. Thus, all three pointer variables are found to alias under this analysis. Program flow is not captured well in this analysis, as possible aliases are always added for pointers instead of changed; in this example, \texttt{z} should only refer to \texttt{a} at the end of the program, and \texttt{y} only has a defined alias if the statically constant value for \texttt{c} is accounted for, which is usually not the case for an alias analysis. Additionally, for this analysis, the operands of a primitive operation, such as addition and subtraction, have the same type as the destination variable. This also results in inaccuracies, as shown in lines 12 and 15.

\subsection{LLVM Basic Alias Analysis}
The LLVM infrastructure features a basic alias analysis implementation available for use with compiler implementations \cite{llvmaa}. This alias analysis is local per function and depends on a series of heuristics to determine which pointers alias. For this analysis, distinct global variables, local variable declarations, and heap memory can never alias. Additionally, such values never alias the null pointer \cite{llvmaa}. Similarly, differing structure fields and array references that are statically different do not alias. Some C standard library functions are assumed to either never access program memory, or only access read-only memory. Pointers that refer to constant global values, such as strings, are said to point to constant memory. Finally, function calls cannot access local variables that never escape from the function that allocates them.

\subsection{Automatic Reference Counting}
Originally developed for the Objective-C programming language, Automatic Reference Counting (ARC) is a system of keeping track of allocated objects within a program \cite{ARC}. Dynamically allocated objects are given a reference count and a class based on its ownership, such as strong or weak ownership. To prevent memory leaks or accidental deallocations, objects are retained to add owners, and released to remove owners; objects with no owners are deallocated, and their pointers are set to null. Operations that refer to object pointers, such as reads, writes, initialization, destruction, and moving, are given different rules depending on the object's ownership type. While objects do not exist in C, ARC-based mechanisms can be applied to track pointer references, and are used in the LLVM infrastructure as part of an alias analysis that can be used for program optimizations.

\section{Evaluating Alias Analyses}
The nearest analogue to this thesis's work can be found in Michael Hind and Anthony Pioli's research report \cite{Hind_old}, which attempts to measure several different alias analysis techniques under the same conditions and performance metrics. Specifically, Hind and Pioli explore three different techniques with varying degrees of precision and efficiency: Flow Insensitive Analysis, Flow Sensitive Analysis, and Flow Insensitive Analysis with Kill Information. These techniques are performed on input programs that are broken down into Control Flow Graphs (CFGs), and sets of pointer aliases are calculated at varying degrees of granularity. The Flow Insensitive analysis calculates possible aliases for variables across the entire function, with the Flow Insensitive Analysis with Kill including additional information about pointer definition and usage intended to improve the Flow Insensitive Analysis. On the other hand, the Flow Sensitive Analysis creates two alias sets for each CFG node, reflecting possible changes in the program due to control flow constructs. All three alias analyses are run on fourteen benchmark C programs, and precision is defined as the number of possible objects, or values, that a given pointer could refer to. Additional statistics are also collected from running these benchmarks, including the execution time of each analysis technique, distinctions between pointers used for reading or writing, and the type of pointer within the context of the program, such as local variables, global variables, formal parameters, and heap variables.

After running the benchmarks, the authors found that additional kill information did not improve the precision of Flow Insensitive analysis. The authors also found that the Flow Insensitive analysis was at least as precise, if not more so, than the Flow Sensitive analysis in half of the benchmarks used. The authors attribute this discrepancy to three possible causes: the first is that Flow Sensitive analysis becomes less precise as the size of a CFG increases, the second is that the consideration of formal and actual parameters is the same for Flow Sensitive and Flow Insenstive analyses, and the third is that pointers are often not modified in ways that would require the additional overhead needed for Flow Sensitive analysis. The authors also propose efficiency improvements for alias analysis techniques, namely sharing alias sets between CFG nodes using Sparse Evaluation Graph (SEG) nodes to save space and reduce overhead in traversing the CFG, using sorted worklists to traverse CFG nodes, and only propagating alias relations that can be reached from a given function call. All of these improvements were shown to speed up the alias analyses in varying degrees. While this report does provide a detailed method of evaluating alias analysis techniques, its definition of precision is limited by the static nature of alias analysis techniques. Thus, there is no additional confirmation on whether a given alias is accurate with respect to the actual program.

Hind and Pioli produced another report evaluating various alias analysis techniques that expands on their previous work \cite{Hind}. This time, they examined six different context-insensitive analysis techniques. Four of these are flow-insensitive, one is flow-sensitive, and one is flow-insensitive but with additional kill information. The first technique, Address Taken (AT) analysis, is flow insensitive and computes a single global alias set for all objects in the program that were assigned to another variable. With its linear time complexity and limited precision, AT served as a baseline technique for comparison with the other techniques. The next technique, Steensgaard (ST) analysis, is a flow-insensitive analysis that computes a single union/find alias set in a single pass in almost linear time. The next flow-insensitive technique, Andersen (AN) analysis, implements Andersen's algorithm; normally, this algorithm uses constraint solving, but in the interest of efficiency, the analysis technique used in this report uses an iterative dataflow. Two flow-insensitive techniques proposed by Burke et al. (B1 and B2) calculate local alias sets for each function call, with B2 including additional kill information for variable definition and usage. The final flow-sensitive algorithm proposed by Choi et al. (CH) operates similarly to the B1 technique, but at the level of SEG nodes instead of at the function level.

For this report, Hind and Pioli used twenty-four benchmark C programs, varying from under 1000 to almost 30,000 lines of code. These benchmarks themselves are compared according to the resulting CFG's that are created for the pointer analyses by measuring the number of CFG nodes, the number of function calls, and the number of heap allocations. As with the previous report, precision for each analysis technique is defined by the number of possible aliases for each given pointer. In addition to precision, execution time, memory usage, and the number of pointer reads and writes are measured for each analysis technique.

After running the benchmarks, both AT and ST were found to be efficient in terms of speed and memory usage. ST was significantly more precise than AT, especially as programs increased in size, with only minimal increases in overhead. AN and B1 varied in comparison to each other, with one significantly outperforming the other, and vice versa, in different benchmarks. The B2 analysis was consistently slower than B1, and the CH analysis was generally significantly slower, save for some benchmarks. The AN, B1, and B2 analyses had the same level of precision as one another, and were comparable with the CH analysis for many of the benchmarks. Thus, the additional kill information in B2 was again, not found to provide any significant benefit in increasing precision. As with the previous report, the benchmarks did not encounter the types of statements that would benefit from flow-sensitive analysis. Additionally, the CH analysis's memory usage was several times higher than that of its flow-insensitive counterparts, even after additional optimizations were implemented to reduce its memory footprint. The speed of pointer analysis was found to be dependent on both program size and the number of propagated alias relations throughout a program's graph. Because this report is an expansion on a previous experiment, it also possesses the same limitations as the previous experiments, namely the definition of precision being limited by purely static analysis techniques.

\section{Proposed Analysis Techniques}

Much of the overhead behind inclusion-based pointer analysis is due to the size of the generated constraint graph describing the relationship between pointer aliases, thus Hardekopf and Lin \cite{Hardekopf} proposed two new algorithms to detect cycles within constraint graphs to reduce the size of the graph. The first method, Lazy Cycle Detection (LCD), occurs when an alias set is propagated across nodes in the constraint graph; LCD checks the constraint graph for cycles based on two conditions; the first is whether or not two alias sets are identical, and the second is whether or not the graph edge related to the current pointer relation was searched previously. The second method, Hybrid Cycle Detection (HCD) performs a static analysis of the program before the actual pointer analysis to create a constraint graph and collapse any possible cycles; this preprocessing reduces the number of traverals performed by the actual pointer analysis, and provides additional information about which pointers might be part of a cycle, even if its alias sets are incomplete after performing HCD. LCD and HCD are evaluated with other comparative optimization algorithms for inclusion-based pointer analysis in five C benchmarks. In addition to the overall reduced number of constraints for each benchmark, the execution time and memory usage is measured for each algorithm. For the benchmarks, HCD is measured both by itself and in combination with the other algorithms. As individual algorithms, LCD and HCD had execution times comparable to the other algorithms. However, though LCD's memory usage was on par with the other algorithms, due to its preprocessing nature, HCD by itself could not complete all of the benchmarks before running out of memory. When used in tandem, LCD and HCD significantly outperformed the other algorithms in terms of speed, with minimal decreases in memory usage. HCD also provided similar performance improvements when used in conjunction with the other algorithms.

One of the alias analysis techniques used in Hardekopf and Lin's experiments to compare against their proposed algorithms was a context-insensitive pointer analysis method developed by Pearce et al. \cite{Pearce} to account for fields and function pointers in an efficient, precise manner. Previously, context-insensitive pointer analyses lacked the constraint types necessary to accurately reflect references to fields within user-defined structures; aggregate types were generally treated as a single variable, or treated as a distinct set of fields for either a unique instance of an aggregate or for all aggregates of the same type. Additionally, function pointers lacked any particular notation that could be used in an elegant or efficient manner. To account for these shortcomings, the authors introduced pointer constraints that included integer offsets, along with inference rules that utilize these constraints. These offsets can be used to model aggregate fields, and functions based on their addresses and parameters, and are treated as edge weights in a constraint graph. After running both field-sensitive and field-insensitive versions of their new analysis technique on seven benchmarks, Pearce et al. found that field-sensitive analysis offered more precision, but at the cost of increased execution time. However, the increase in precision was also found to decrease with larger programs.


\title{Example List}
%\author{}
\date{}

\newcommand{\LinkText}{[info]}

\begin{document}

Example files are given in three directories: (1) \href{\examplespec}{examplespec} contains problem specifications, (2) \href{\examplesoln}{examplesoln} contains solutions, and (3) \href{\examplesynt}{examplesynt} contains solutions which retain artifacts from the specification.
For each file in \href{\examplesynt}{examplesynt}, there is usually a corresponding file in \href{\examplesoln}{examplesoln} which is more presentable and simpler to verify.

\tableofcontents

\section{Sum-Not-2}

Aly Farahat's \href{http://digitalcommons.mtu.edu/etds/178}{dissertation} includes a simple yet nontrivial unidirectional ring protocol.
It has been extremely helpful for reasoning about the nature of \href{http://dx.doi.org/10.1007/978-3-319-03089-0\_12}{livelocks in unidirectional rings} and has a simple enough transition system to actually show in a presentation.
\begin{itemize}
\item SumNotTwo (\href{\examplespec/SumNotTwo.prot}{spec})
\item SumNotTarget (\href{\examplespec/SumNotTarget.prot}{spec}, \href{\examplesoln/SumNotTarget.prot}{soln})
- The general case sum-not-$(l-1)$ protocol given by Farahat.
\end{itemize}

\section{Coloring}

\href{http://en.wikipedia.org/wiki/Graph_coloring}{Graph coloring} is a well-known problem with many applications.
Each node in the graph is assigned a color.
For this assignment to be called a \textit{coloring}, each node must have a different color than the nodes adjacent to it.
In a computer network, a coloring applies to processes that communicate directly with each other rather than nodes connected by edges.
\begin{itemize}
\item ColorRing \href{Coloring.html#sec:ColorRing}{\LinkText}
- 3-coloring on a ring.
\item ColorRingDizzy (\href{\examplespec/ColorRingDizzy.prot}{spec}, \href{\examplesoln/ColorRingDizzy.prot}{soln})
- Unoriented ring.
\item ColorUniRing (\href{\examplespec/ColorUniRing.prot}{spec}, \href{\examplesoln/ColorUniRing.prot}{soln})
- Randomized 3-coloring on a unidirectional ring.
\item ColorRingLocal \href{Coloring.html#sec:ColorRingLocal}{\LinkText}
- Randomized distance-2 coloring on a unidirectional ring using $5$ colors.
Neither a process or its neighbors can have the same color as another neighbor.
\item ColorRingDistrib \href{Coloring.html#sec:ColorRingDistrib}{\LinkText}
- Randomized 3-coloring on a ring where processes have communication delay.
\item ColorChain (\href{\examplespec/ColorChain.prot}{spec})
- 2-coloring on a chain.
\item ColorTree (\href{\examplespec/ColorTree.prot}{spec}, \href{\examplesoln/ColorTree.prot}{soln})
- Tree.
\item ColorTorus (\href{\examplespec/ColorTorus.prot}{spec})
- Torus.
\item ColorMobius (\href{\examplespec/ColorMobius.prot}{spec})
- Mobius ladder.
\item ColorKautz (\href{\examplespec/ColorKautz.prot}{spec})
- 4-coloring on generalized Kautz graph of degree 2.
We can find a protocol that stabilizes for up to $13$ processes, but not $14$.
Tweaking the topology by
\href{\examplespec/ColorKautzReverse.prot}{reversing edges}
or
\href{\examplespec/ColorKautzDizzy.prot}{removing orientation}
(even with \href{\examplespec/ColorKautzBi.prot}{bidirectional edges})
gives definite impossibility results at around $8$ processes.
\end{itemize}

\section{Maximal Matching}

\href{http://en.wikipedia.org/wiki/Matching_(graph_theory)}{Matching} is well-known problem from graph theory.
A matching is a set of edges which do not share any common vertices.
For a matching to be \textit{maximal}, it must be impossible to add another edge to the set without breaking the matching property.
\begin{itemize}
\item MatchRing \href{Matching.html#sec:MatchRing}{\LinkText}
- Natural specification for matching using the edges in the ring.
This gives the 1-bit solution.
\item MatchRingThreeState \href{Matching.html#sec:MatchRingThreeState}{\LinkText}
- Maximal matching on a ring where processes point in certain directions.
\item MatchRingOneBit \href{Matching.html#sec:MatchRingOneBit}{\LinkText}
- Using the 3-state specification, find a matching using 1 bit per process.
\item MatchRingDizzy (\href{\examplespec/MatchRingDizzy.prot}{spec}, \href{\examplesett/MatchRingDizzy.args}{args})
- Maximal matching on an unoriented ring.
\item SegmentRing \href{Matching.html#sec:SegmentRing}{\LinkText}
- A problem similar to matching where a ring is segmented into chains.
\end{itemize}

\section{Sorting on Chains and Rings}

\begin{itemize}
\item SortChain (\href{\examplespec/SortChain.prot}{spec}, \href{\examplesoln/SortChain.prot}{soln})
- Sorting a chain of values.
Easy to synthesize, but solution is manually simplified.
\item SortRing (\href{\examplespec/SortRing.prot}{spec}, \href{\examplesoln/SortRing.prot}{soln})
- Sorting a ring of values using a unique zero value to mark the beginning of the sequence.
Easy to synthesize, but solution is manually simplified.
\end{itemize}

\section{Orientation}

\begin{itemize}
\item OrientDaisy \href{Orientation.html#sec:OrientDaisy}{\LinkText}
- Silent orientation for daisy chains (either ring and chain), verified for $2$ to $15$ processes.
The version in \href{\examplesoln}{examplesoln} behaves similarly, is simpler, but is slightly less optimal.
\item OrientRing \href{Orientation.html#sec:OrientRing}{\LinkText}
- Manually designed silent ring orientation, verified for $2$ to $26$ processes.
Also works under synchronous scheduler.
\item OrientRingOdd \href{Orientation.html#sec:OrientRingOdd}{\LinkText}
- From \href{http://dx.doi.org/10.1007/BFb0020439}{Hoepman}.
\item OrientRingViaToken (\href{\examplespec/OrientRingViaToken.prot}{spec}, \href{\examplesoln/OrientRingViaToken.prot}{soln})
- From \href{http://dx.doi.org/10.1006/inco.1993.1029}{Israeli and Jalfon}.
\end{itemize}

\section{Token Passing}

\begin{itemize}
\item TokenRingFiveState \href{TokenPassing.html#sec:TokenRingFiveState}{\LinkText}
- 5-state token ring whose behavior is specified with shadow variables.
\item TokenRingSixState \href{TokenPassing.html#sec:TokenRingSixState}{\LinkText}
- 6-state token ring specified with variable superposition.
\item TokenRingThreeBit \href{TokenPassing.html#sec:TokenRingThreeBit}{\LinkText}
- 3-bit token ring from \href{http://dx.doi.org/10.1006/jpdc.1996.0066}{Gouda and Haddix}.
\item TokenRingFourState (\href{\examplesynt/TokenRingFourState.prot}{synt})
- 4-state token ring specified with variable superposition.
This version is only stabilizing for $2$ to $7$ processes.
It operates much like the 3-bit token ring.
\item TokenRingDijkstra \href{TokenPassing.html#sec:TokenRingDijkstra}{\LinkText}
- Dijkstra's stabilizing token ring.
\item TokenChainThreeState \href{TokenPassing.html#sec:TokenChainThreeState}{\LinkText}
- 3-state token passing on a chain topology.
\item TokenChainDijkstra \href{TokenPassing.html#sec:TokenChainDijkstra}{\LinkText}
- Dijkstra's 4-state token passing on a chain topology.
\item TokenRingThreeState \href{TokenPassing.html#sec:TokenRingThreeState}{\LinkText}
- 3-state token passing on a bidirectional ring.
One solution is from Dijkstra, the other is from \href{http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.153.6017}{Chernoy, Shalom, and Zaks}.
\item TokenRingOdd (\href{\examplespec/TokenRingOdd.prot}{spec}, \href{\examplesoln/TokenRingOdd.prot}{soln})
- Randomized token passing protocol on odd-sized rings.
\end{itemize}

%\section{Leader Election}
%\section{Reduction from 3-SAT}

\section{Other}

\begin{itemize}
\item ByzantineGenerals (\href{\examplespec/ByzantineGenerals.prot}{spec}, \href{\examplesoln/ByzantineGenerals.prot}{soln})
- The Byzantine generals problem formulated as an instance of self-stabilization.
\item DiningCrypto (\href{\examplespec/DiningCrypto.prot}{spec}, \href{\examplesoln/DiningCrypto.prot}{soln})
- The dining cryptographers problem as an instance of self-stabilization, where the initial state is assumed to randomize the coins.
We can't model anonymity, only determination of whether a cryptographer or the NSA pays for dinner.
\item DiningPhilo (\href{\examplespec/DiningPhilo.prot}{spec}, \href{\examplesoln/DiningPhilo.prot}{soln})
- The dining philosophers problem. This version assumes a coloring in order to break symmetry.
\item DiningPhiloRand (\href{\examplespec/DiningPhiloRand.prot}{spec}, \href{\examplesoln/DiningPhiloRand.prot}{soln})
- The dining philosophers problem. This version uses randomization to break symmetry.
\item LeaderRingHuang (\href{\examplespec/LeaderRingHuang.prot}{spec}, \href{\examplesoln/LeaderRingHuang.prot}{soln})
- Leader election protocol on prime-sized rings from \href{http://dx.doi.org/10.1145/169683.174161}{Huang}.
\item Sat (\href{\examplespec/Sat.prot}{spec})
- Example reduction from 3-SAT to adding stabilization from our \href{http://dx.doi.org/10.1007/978-3-642-40213-5_2}{paper showing NP-completeness} of adding convergence.
\item StopAndWait (\href{\examplespec/StopAndWait.prot}{spec}, \href{\examplesoln/StopAndWait.prot}{soln})
- The Stop-and-Wait protocol, otherwise known as the Alternating Bit protocol when the sequence number is binary.
\end{itemize}

\end{document}


\chapter{Speech Recognition with Deep Neural Networks}

\section{Combining time-and frequency-domain convolution in convolutional neural network-based phone recognition}

A paper by L{\'a}szl{\'o} T{\'o}th that develops a convolutional network for phone recognition
on the TIMIT database.

\section{Convolutional Deep Maxout Networks for Phone Recognition}
In this section we are writing up an explanation of
\cite{toth2014convolutional}.  The paper use mel features
(as described in Chapter~\ref{sec:mel-spectral-features})
as the basis for the system.

\subsection{Convolutional Neural Networks}

The convolutional network uses as its base input a 17-frame by
7 mel-channel time-mel block: i.e. 7 rows by 17 columns in the
time frequency representation.  Pooling and convolution are central
to convolutional neural networks and they are usually performed
along the frequency axis.  An $r$-pooling network will move the $7\times 17$
along filter along $7$ adjacent mel feature blocks and takes the maximum over
them.


\subsection{Twisted bundles and the Klein bottle}

\subsubsection{Twisted bundles}

\subsubsection{Klein bottles}

\(S_1 \times S_1\), but twisted


\chapter{Background}\label{sec:Background}
RL is a process that requires both interactive parts as well as algorithms that improve interactions. The following section \ref{reinforcement_learning} introduces the general concept of RL and its specifications. Afterwards, two popular learning algorithms for RL problems are presented: PPO and DQN.

\section{Reinforcement Learning}\label{reinforcement_learning}
Sutton and Barto wrote in ``Reinforcement learning: An introduction'' \cite{suba18} that RL is based on two components that interact with each other: an environment and an agent, see Figure \ref{fig:rl_cycle}. Those interactions take part during a time period with discrete time steps $t\in\mathbb{N}_0$ until a goal is reached or the ending condition applies. This process is called an episode. Formally, the journey of the agent to find the goal is described as a Markov Decision Process (MDP) \cite{suba18}. When multiple agents act in the same environment, the Markov decision process is called a stochastic game \cite{buba10}.

\begin{figure}[hpbt]
    \centering
    \includegraphics[width=0.6\textwidth]{pictures/RLInteractionSB}\\
    \caption[Reinforcement Learning Cycle]{The cycle of agent-environment interaction as
        shown in ``Reinforcement learning: An introduction'' \cite{suba18}}\label{fig:rl_cycle}
\end{figure}

One environment state $S_t$ is part of a set $S$ containing all possible states. In most cases, the environment state describes what the agent can see. An agent often only has a small field of view, which turns a MDP into a partially observable MDP \cite{suba18}. During each point in time $t$, the agent can interact with the environment by executing an action $A_t$, which changes the environment state. An example would be moving in the environment, which results in a new area that the agent can now see. In a multiagent environment, every agent chooses its action simultaneously and adds it into a joint action set, which is executed collectively during $t$ \cite{buba10}.

The reward $R_t$ is an element of a set of possible rewards $R \subset \mathbb{R}$ \cite{suba18}. Therefore, the reward can potentially be negative. Depending on the environment, that value can act as immediate feedback to the agents action. Other times, the reward is received as a result of a whole action sequence or the achievement of a certain state, for instance the goal or subgoals. The general concept of RL, as defined by Sutton and Barto \cite{suba18}, is for agents to maximize rewards. Unlike machine learning approaches, the agent starts with no knowledge about good or bad actions and enhances the decision-making over time.

Sutton and Barto defines the agents' action selection with respect to the current state as a policy $\pi$. They explain further that a policy could be as simple as a lookup table, mapping states to actions, or it could contain a complicated search process for the best decision. However, policies most of the time map action-state pairs to a selection probability, with all actions of a state adding up to 100\%.
During environment interactions, agents receive rewards which can be used to update the policy accordingly. As an example, the probability of policy $\pi(a \mid s)$ decreases when receiving a negative or low reward, reducing the chances of executing the same action in that specific state again.

While rewards only rate the immediate situation, a value function, i.e. the state-value function $V^\pi(s_t)$ for a policy $\pi$, can be used to estimate the long-term value of a state $s$ \cite{suba18}:
\begin{equation}\label{eq:value_func}
    v_\pi(s) \doteq \mathbb{E}_\pi \left[ G_t \mid S_t = s \right] = \mathbb{E}_\pi \left[ \sum^{\infty}_{k=0} \gamma^k R_{t+k+1 \mid S_t = s}  \right]
\end{equation}
The result is the estimated discounted cumulative reward an agent could get following that state and choosing actions based on the current policy. The discount factor is defined by Sutton and Barto as $0 \le \gamma < 1$ and provides a constant that reduces the importance of future rewards. A high $\gamma$ symbolizes a greater interest in rewards that are far away, whereas a discount of zero only takes the current reward into account. By setting $\gamma$ smaller than one, it is ensured that the infinite sum results in a value. Generally, states that offer immediate high rewards could end in a low reward streak. In the opposite case, a low reward state could subsequently yield high rewards. Therefore, value functions are of great use to achieve the maximum reward.

The last part to note about RL is that it entails the problem of balancing exploration and exploitation \cite{suba18}. On the one hand, an agent has to explore different options in order to learn and expand its knowledge. On the other hand, agents strive to maximize the reward, which can lead to greediness. An agent could start to exploit its knowledge too early, choosing actions of which it knows to result in positive rewards. However, if an agent does not explore enough, the best action sequence will stay hidden and the agents knowledge will not improve.

\section{Proximal Policy Optimization}
In 2017, Schulman et al. introduced the concept of PPO in the article ``Proximal Policy Optimization Algorithms'' \cite{scwo17}. Policy optimization is the improvement of the action selection strategy $\pi$ based on the current state $s_{t}$. This is achieved by rotating two steps \cite{scwo17}: \\
\begin{enumerate}
  \item Sampling data from the policy and
  \item Optimizing the objective with that data through several epochs.
\end{enumerate}

Using those steps results in the agent gathering a small batch of experiences while choosing actions with a policy $\pi$. Afterwards, this batch is used once to enhance the current policy. Then, the experiences are discarded and the agent uses the updated policy to gather a new batch. By repeating those two steps the agent can learn to choose better actions. PPO is applied to prevent drastic policy changes, which stabilizes the learning process.

The origin of PPO lies in a similar approach called Trust Region Policy Optimization (TRPO). TRPO also restricts policy updates by defining a trust region \cite{scle15}. This is achieved by maximizing the following function \cite{scwo17}:
\begin{equation}\label{eq:TRPO}
    \underset{\theta}{maximize}\,\hat{\mathbb{E}}_{t} \left[ \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{old}}(a_{t} \mid s_{t})}
        \hat{A}_{t}-\beta \, KL[\pi_{\theta_{old}}(\cdot \mid s_{t}),\pi_{\theta}(\cdot \mid s_{t})] \right]
\end{equation}
The expectation $\hat{\mathbb{E}}_{t}$ indicates, that an empirical average over a number $t$ of samples is used for estimation and the algorithm alternates between sampling and executing these calculations. The variable $\hat{A}_{t}$ describes an estimator of the advantage function. This function was defined in the paper ``Trust Region Policy Optimization'' \cite{scle15} with \\ $A_\pi(s,a) = Q_\pi(s,a)-V_\pi(s)$. The first part calculates the state-action value, estimating the upcoming rewards for an agent, starting at state s and initially selecting action a. Afterwards, the action selection is based on the current policy $\pi$.

The second part contains the state value function $V_\pi(s)$, which works very similarly by starting at state s and using $\pi$. However, the difference is that the agent always chooses actions according to the policy. The result of the advantage function $A_\pi(s,a)$ shows whether a profit could be gained when deviating from the policy by specifically choosing action a.

The fraction $\frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{old}}(a_{t} \mid s_{t})}$ in the Minuend of function \eqref{eq:TRPO} can be replaced by $r(\theta)$
and represents the probability ratio of an action in the current policy in comparison to the old policy \cite{scwo17}. $\theta$ represents a policy parameter. The result of $r(\theta)$ is greater than one, if an action is very probable in the current policy. Otherwise, the outcome lies between zero and one. Schulman et al. \cite{scwo17} further extract the first part of function \eqref{eq:TRPO} as the surrogate objective:
\begin{equation}\label{eq:TRPO_surrogate}
    L^{CPI}(\theta) = \hat{\mathbb{E}}_{t} \left[ \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{old}}(a_{t} \mid s_{t})} \hat{A}_{t} \right]
    = \hat{\mathbb{E}}_{t} \left[ r(\theta)\hat{A}_{t} \right]
\end{equation}
If only this part of function \eqref{eq:TRPO} is maximized on its own, it would result in large outcomes, which in turn leads to drastic policy updates. In order to stay in a trust region, as the name suggests, a penalty is subtracted from the surrogate function \eqref{eq:TRPO_surrogate}. The penalty is the subtrahend of equation \eqref{eq:TRPO} and contains the fixed coefficient $\beta$. Regardless of the function details and outcome of $KL$, the coefficient $\beta$ is hard to choose, since different problems require different penalty degrees \cite{scwo17}. Even during a training process it could be necessary to adapt the coefficient, due to changes.

Therefore, Schulman et al. introduced
\begin{equation}\label{eq:PPO}
    L^{CLIP}(\theta) = \hat{\mathbb{E}}_{t} \left[ \min \left( \; r(\theta)\hat{A}_{t}, \; clip(r(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_{t} \; \right) \right]
\end{equation}
which is very similar to equation \eqref{eq:TRPO} but does not require coefficients. The first $\min$ entry contains $L^{CPI}$ \eqref{eq:TRPO_surrogate}. The second part contains a $clip$ function, which narrows the space of policy mutation with the small hyperparameter $\epsilon$. After applying the clip function, $r(\theta)$ lies between $[1-\epsilon,1+\epsilon]$. Calculating the minimum of the clipped and unclipped probability ratio produces the lower bound of the unclipped $r(\theta)$, preventing the policy to change drastically.

Finally, the following equation is introduced
\begin{equation}\label{eq:PPO_algo}
    L_{t}^{CLIP+VF+S}(\theta) = \hat{\mathbb{E}}_{t} \left[ L_{t}^{CLIP}(\theta) - c_{1}L_{t}^{VF}(\theta) + c_{2}S[\pi_{\theta}](s_{t}) \right]
\end{equation}
with $c_{1}$ and $c_{2}$ as coefficients. The authors point out that the loss function \\
$L_{t}^{VF} = (V_{\theta}(s_{t})-V_{t}^{targ})^2$ combines the policy surrogate and the value function error term and is necessary once a neural network shares parameters between policy and value function. An entropy bonus $S$ is added to ensure exploration.

Furthermore, Schulman et al. point out that the policy is executed for $T$ time steps, with $T$ being a smaller value than the overall episode duration. Until now, the advantage function calculates values through an infinite loop, see the value function \eqref{eq:value_func} for example. Hence, the advantage function needs to be adjusted as well. It is necessary that the future estimations do not exceed that time step limit. In this context, the following advantage function is used \cite{scwo17}:
\begin{equation}\label{eq:advantage_func}
    \hat{A_t} = \delta_t+(\gamma \lambda)\delta_{t+1}+ \cdots + (\gamma \lambda)^{T-t+1}\delta_{T-1}
\end{equation}
\begin{equation}\label{eq:advantage_func_delta}
    \textrm{where} \qquad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)
\end{equation}

Schulman et al. also showed an example of the PPO algorithm, cf. algorithm \ref{algo:ppo_algo_code}. The example uses an actor-critic approach, which means that a critic is responsible to approximate the value function of the policy and the actor in turn improves the policy based on the approximation results of the critic \cite{kots03}. $N$ detonates actors collecting data in T time steps in each iteration. Meanwhile, the critic computes the estimations of the advantage values. Afterwards, the policy is replaced with a new one, in which the function $L_{t}^{CLIP+VF+S}(\theta)$ \eqref{eq:PPO_algo} is optimized during K epochs. For the optimization process, a small random batch of the previous time steps is used.

\begin{algorithm}[H]
    \DontPrintSemicolon
    \For(){\text{iteration=1,2,...}}{
        \For(){\text{actor=1,2,...,$N$}}{
            Run policy $\pi_{\theta_{old}}$ in environment for $T$ timesteps \;
            Compute advantage estimates $\hat{A}_{1},...\hat{A}_{T}$
        }
        Optimize surrogate $L$ wrt $\theta$, with K epochs and minibatch size $M \; \leq \; NT$ \;
        $\theta_{old} \leftarrow \theta$
    }
    \caption{PPO, Actor-Critic Style, as shown in ``Proximal Policy Optimization Algorithms'' \cite{scwo17}}\label{algo:ppo_algo_code}
\end{algorithm}

\section{Deep Q-Network}\label{dqn}
Another learning approach that is often compared with PPO is the training algorithm of a deep Q-Network with Q-learning and experience replay. Instead of improving a policy, agents improve by maximizing a value function. Hence, this algorithm relies on the action value function, that is formally defined as follows \cite{mnba16}:
\begin{equation}\label{eq:qvalue}
    Q^\pi(s,a) = \mathbb{E} \left[ R_t \mid s_t = s,a \right]
\end{equation}
$R_t$ represents the discounted cumulative reward $R_t=\sum^{\infty}_{k=0} \gamma^k r_{t+k}$. The estimated outcome is calculated by starting at a state $s$, executing a specific action $a$ and reaching the next states by using a policy $\pi$. Mnih et al. \cite{mnka15} state, that the optimal action-value can be approximated with a deep convolutional neural network and the following function:
\begin{equation}\label{eq:opt_qvalue}
    Q^*(s,a) =  \underset{\pi} \max \mathbb{E}\left[ r_{t} + \gamma r_{t+1} + \gamma^2 r_{t+2} + \ldots | s_t = s, a_t = a, \pi \right]
\end{equation}

The difference between function \eqref{eq:qvalue} and \eqref{eq:opt_qvalue} is, that in the second one, a policy is chosen, which optimizes the outcome. Mnih et al. continue by stating, that in a scenario where the sequence $s'$ of all actions $a'$ are known, the optimal $Q^*(s',a')$ of the next state can be calculated. Then, this Bellman equation could be applied \cite{mnba16}:
\begin{equation}\label{eq:bel_qvalue}
    Q^*(s,a) =  \mathbb{E}_{s'} \left[ r+ \gamma \underset{a'}\max \; Q^* (s', a') \mid s,a \right]
\end{equation}
Many RL algorithms estimate this function through iterative updates, by calculating $Q_{i+1}(s,a) =  \mathbb{E} \left[ r+ \gamma \underset{a'}\max \; Q_i (s', a') \mid s,a \right]$, $Q_{i+2}$, \dots \cite{mnka13}. Eventually the optimal Q value is reached with $i\rightarrow \infty$. Those calculations proved to be very impractical, since they require a lot of computational work, which is why Mnih et al. introduced the Q-network at this point. As a result, the parameters of the Q function are extended with $\theta$ as network weights ($Q(s,a;\theta)$).

However, the researchers argued that using a neural network in combination with the Q function proofed to be unstable. According to the authors, this is caused by correlating observations that are used to calculate the function. Additionally, small updates to the action value may lead to drastic changes of the policy. Such problems change the connection between Q values and their successive target values $r+\gamma \; \underset{a'} \max \; Q(s',a')$. To overcome these issues, Mnih et al. introduced two new concepts:
\begin{enumerate}
  \item An experience replay that enables random sampling of observations and
  \item An iterative update process of the action values approaching the target values.
\end{enumerate}
The target values are only updated periodically in their implementation.

In algorithm \ref{algo:dqn_algo_code}, a deep Q-learning approach with an experience replay is shown. The experience replay contains the acquired agent knowledge of each time step in form of a quadruple: (old state, action, reward, new state). The experience values are then stored into the replay memory $D$ across multiple episodes. The states are parameters of $\Phi_{t}$ in the example, since they are preprocessed, to match the network input conditions.

In addition to the action value function $Q$, the target action-value $\hat Q$ is initially defined with the same weights to enable iterative updates. In order to fill the memory, the agent first selects actions and acts in the environment. The action selection here is based on the $\epsilon$-greedy policy, meaning that with a probability of $\epsilon$ a random action is chosen \cite{mnka15}. Otherwise, the best option according to the Q-value is selected.

\begin{algorithm}[H]
    \DontPrintSemicolon
    Initialize replay memory $D$ to capacity $N$ \;
    Initialize action-value function $Q$ with random weights $\theta$ \;
    Initialize target action-value function $\hat{Q}$ with weights $\theta^{-} = \theta$ \;
    \For(){\text{episode=1,$M$}}{
        Initialize sequence $s_1 = \{x_1\}$ and preprocessed sequence $\phi_1 = \phi(s_1)$ \;
        \For(){\text{$t$=1,T}}{
            With probability $\epsilon$ select a random action $a_t$ \;
            otherwise select $a_t = \text{argmax}_a Q(\phi(s_t), a; \theta)$ \;
            Execute action $a_t$ in emulator and observe reward $r_t$ and image $x_{t+1}$ \;
            Set $s_{t+1} = s_{t}, a_{t}, x_{t+1}$ and preprocess $\phi_{t+1}=\phi(s_{t+1})$ \;
            Store transition ($\phi_{t}, a_{t}, r_{t}, \phi_{t+1}$) in $D$ \;
            Sample random minibatch of transitions ($\phi_{j}, a_{j}, r_{j}, \phi_{j+1}$) from $D$ \;
            Set $y_j = \begin{cases}
                r_j & \text{if episode terminates at step j + 1}\\
                r_j + \gamma \text{max}_{a'} \hat{Q}(\phi_{j+1}, a', \theta^{-}) & \text{otherwise}
                \end{cases}$  \;
            Perform a gradient descent step on $\left( y_j - Q(\phi, a_j ; \theta) \right) ^2$ with respect to the network parameters $\theta$ \;
            Every $C$ steps reset $\hat{Q} = Q$
        }
    }
    \caption{DQN with Experience Replay, as shown in ``Human-level control through deep reinforcement learning'' \cite{mnka15}}\label{algo:dqn_algo_code}
\end{algorithm}

Executing the selected action results in a memory entry in the form of the earlier described quadruple. Afterwards, a minibatch of the replay memory is randomly sampled to calculate the difference between the values. The action values with the current weights are subtracted from $y_i$. This variable calculates estimated Bellman equation by using the target action values with the old weights. The parameter $y_i$ contains just the reward value of the sample, if the sampled entry was the last step the episode of the entry. A gradient descent step is performed on the function, which means that the local minima of the function is searched by tweaking the parameter $\theta$.

Finally, every certain amount of steps C the target network is set to the current Q-Network. The suggested process offers several advantages \cite{mnka15}: the replay memory leads to a smaller deviation or fluctuation in the parameters. The random samples of minibatches can be efficient, since an experience might be used multiple times to update the network weights. Furthermore, through the randomness in the samples, the correlation of steps is interrupted. This leads to a decrease of variance in between updates. Lastly, updating the target network periodically improves the stability of the learning process.