\section{Overview}
\label{sec:refinementreflection:overview}
\label{sec:examples}

We begin with an overview of refinement reflection and
how it allows us to write proofs \emph{of} and \emph{by}
Haskell functions.

\subsection{Refinement Types}

First, we recall some preliminaries about refinement types
and how they enable shallow specification and verification.

\mypara{Refinement types} are the source program's (here
Haskell's) types decorated with logical predicates drawn
from a(n SMT decidable) logic~\citep{ConstableS87,Rushby98}.
%
For example, we can define the @Nat@ type by refining
Haskell's @Int@ type with a predicate @0 <= v@:
%
\begin{code}
  type Nat = { v:Int | 0 <= v }
\end{code}
%
Here, @v@ names the value described by the type:
the above can be read as the
``set of @Int@ values @v@ that are not less than 0".
The refinement is drawn from the logic of quantifier
free linear arithmetic and uninterpreted functions
(QF-UFLIA~\cite{SMTLIB2}).

\mypara{Specification \& Verification}
%
We can use refinements to define and type the
textbook Fibonacci function as:
%
\begin{code}
  fib :: Nat -> Nat
  fib 0 = 0
  fib 1 = 1
  fib n = fib (n-1) + fib (n-2)
\end{code}
%
Here, the input type's refinement specifies a
\emph{pre-condition} that the parameters must
be @Nat@, which is needed to ensure termination,
and the output types's refinement specifies a
\emph{post-condition} that the result is also a @Nat@.
%
Refinement type checking lets us specify
and (automatically) verify the shallow property
that if @fib@ is invoked with a non-negative
@Int@, then it terminates and yields
a non-negative @Int@.

\mypara{Propositions}
%
We can use refinements to define a data type
representing propositions simply as an alias
for unit, a data type that carries no useful
runtime information:
%
\begin{mcode}
  type $\typp$ = ()
\end{mcode}
%
which can be \emph{refined} with
propositions about the code.
%
For example, the following states the proposition
$2 + 2$ equals $4$.
%
\begin{mcode}
  type Plus_2_2_eq_4 = { v: $\typp$ | 2 + 2 = 4 }
\end{mcode}
%
For clarity, we abbreviate the above type by omitting
the irrelevant basic type $\typp$ and variable @v@:
%
\begin{mcode}
  type Plus_2_2_eq_4 = { 2 + 2 = 4 }
\end{mcode}
%
Function types encode universally quantified propositions:
%
\begin{mcode}
  type Plus_com = x:Int -> y:Int -> { x + y = y + x }
\end{mcode}
%
The parameters @x@ and @y@ refer to input
values. Any inhabitant of the above type is a
proof that @Int@ addition is commutative.

\mypara{Proofs}
%
We \emph{prove} the above theorems by providing inhabitants to type specifications
in forms of Haskell programs. To ease this task \toolname
provides primitives to construct proof terms by
``casting'' expressions to \typp.
%
\begin{mcode}
  data QED = QED

  (**) :: a -> QED -> $\typp$
  _ ** _  = ()
\end{mcode}
%
To resemble mathematical proofs, we make this casting post-fix.
Thus, we write @e ** QED@ to cast @e@ to a value of \typp.
%
For example, we can prove the above propositions by writing
%
\begin{code}
  pf_plus_2_2 :: Plus_2_2_eq_4
  pf_plus_2_2 = trivial ** QED

  pf_plus_comm :: Plus_comm
  pf_plus_comm = \x y -> trivial ** QED

  trivial = ()
\end{code}
%
Via standard refinement type checking, the above code yields
the respective verification conditions (VCs),
%
\begin{align*}
                      2 + 2 & = 4 \\
  \forall \ x,\ y\ .\ x + y & = y + x
\end{align*}
%
which are easily proved valid by the SMT solver, allowing us
to prove the respective propositions.

\mypara{A Note on Bottom:} Readers familiar with Haskell's
semantics may be feeling anxious about whether the
dreaded ``bottom", which inhabits all types, makes our
proofs suspect.
%
Fortunately, as described in \cite{Vazou14}, \toolname
ensures that all terms with non-trivial refinements
provably evaluate to (non-bottom) values, thereby making
our proofs sound.

\subsection{Refinement Reflection}

Suppose we wish to prove properties about the @fib@
function, \eg @fib 2@ equals @1@.
%
\begin{code}
  type fib2_eq_1 = { fib 2 = 1 }
\end{code}
%
%% \NV{By Standard refinement type checking, you mean liquid types, not FStar}
Standard refinement type checking runs into two problems.
%
First, for decidability and soundness, arbitrary user-defined
functions do not belong the refinement logic, \ie we cannot
\emph{refer} to @fib@ in a refinement.
%
Second, the only information that a refinement type checker
has about the behavior of @fib@ is its shallow type
specification @Nat -> Nat@ which is far too weak to verify
@fib2_eq_1@.
%
To address both problems, we use the following annotation,
which sets in motion the three steps of refinement reflection:
%
\begin{code}
  reflect fib
\end{code}

\mypara{Step 1: Definition}
%
The annotation tells \toolname to declare an
\emph{uninterpreted function} @fib :: Int -> Int@
in the refinement logic.
%
By uninterpreted, we mean that the logical @fib@
is \emph{not} connected to the program function
@fib@; in the logic, @fib@
only satisfies the \emph{congruence axiom}
%
$$\forall n, m.\ n = m\ \Rightarrow\ \fib{n} = \fib{m}$$
%
On its own, the uninterpreted function is not
terribly useful, as it does not let us prove
% It lets us prove theorems like
% $$\forall m,\ n.\ m = n \Rightarrow \fib{m} = \fib{n}$$
%
%% \begin{code}
  %% fib_cong :: n:Nat -> m:Nat -> {m=n => fib m = fib n}
  %% fib_cong = trivial ** QED
%% \end{code}
%% %
%but not
@fib2_eq_1@ which requires reasoning about the
\emph{definition} of @fib@.

\mypara{Step 2: Reflection}
%
In the next key step, \toolname reflects the
definition into the refinement type of @fib@
by automatically strengthening the user defined
type for @fib@ to:
%
\begin{code}
  fib :: n:Nat -> { v:Nat | fibP v n }
\end{code}
%
where @fibP@ is an alias for a refinement
\emph{automatically derived} from the
function's definition:
%
\begin{mcode}
  fibP v n = v = if n = 0 then 0 else
                 if n = 1 then 1 else
                 fib(n-1) + fib(n-2)
\end{mcode}

\mypara{Step 3: Application}
%
With the reflected refinement type,
each application of @fib@ in the code
automatically unfolds the @fib@ definition
\textit{once} in the logic.
%
We prove @fib2_eq_1@ by:
%
\begin{code}
  pf_fib2 :: { fib 2 = 1 }
  pf_fib2 = let t0 = #fib# 0
                t1 = #fib# 1
                t2 = #fib# 2
            in  ()
\end{code}
%
We write @#f#@ to denote places where the
unfolding of @f@'s definition is important.
%
Via refinement typing, the above proof yields the
following verification condition that is
discharged by the SMT solver, even though @fib@
is uninterpreted:
%
\begin{align*}
   (\fibdef\ (\fib\ 0)\ 0) \ \wedge\ (\fibdef\ (\fib\ 1)\ 1) \ \wedge\
   (\fibdef\ (\fib\ 2)\ 2) \  \Rightarrow\ (\fib{2} = 1)
\end{align*}
%
Note that the verification of @pf_fib2@ relies
merely on the fact that @fib@ was applied
to (\ie unfolded at) @0@, @1@ and @2@.
%
The SMT solver automatically \emph{combines}
the facts, once they are in the antecedent.
The following is also verified:
%
\begin{code}
  pf_fib2' :: { fib 2 = 1 }
  pf_fib2' = [ #fib# 0, #fib# 1, #fib# 2 ] ** QED
\end{code}
%
%
Thus, unlike classical dependent typing, refinement
reflection \emph{does not} perform any type-level
computation.

\mypara{Reflection vs. Axiomatization}
%
An alternative \emph{axiomatic} approach,
used by Dafny~\citep{dafny} and
\fstar~\citep{fstar},
is to encode @fib@ using a universally
quantified SMT formula (or axiom):
$$\forall n.\ \fibdef\ (\fib\ n)\ n$$
%
Axiomatization offers greater automation than
reflection. Unlike \toolname, Dafny
%and \fstar
will verify the following by
\emph{automatically instantiating} the above
axiom at @2@, @1@ and @0@:
%
\begin{code}
  axPf_fib2 :: { fib 2 = 1 }
  axPf_fib2 = trivial ** QED
\end{code}

The automation offered by axioms is a bit of a
devil's bargain, as axioms render checking of
the VCs \emph{undecidable}.
%
In practice, automatic axiom instantation can
easily lead to infinite ``matching loops''.
%
For example, the existence of a term \fib{n} in a VC
can trigger the above axiom, which may then produce
the terms \fib{(n-1)} and \fib{(n-2)}, which may then
recursively give rise to further instantiations
\emph{ad infinitum}.
%
To prevent matching loops an expert must carefully
craft ``triggers'' and provide a ``fuel''
parameter~\citep{Amin2014ComputingWA} that can be
used to restrict the numbers of the SMT unfoldings,
which ensure termination, but can cause the axiom
to not be instantiated at the right places.
%
In short, per the authors of Dafny, the
undecidability of the VC checking and its
attendant heuristics makes verification
unpredictable~\citep{Leino16}.

\subsection{Structuring Proofs}

In contrast to the axiomatic approach,
with refinement reflection, the VCs are
deliberately designed to always fall in
an SMT-decidable logic, as function symbols
are uninterpreted.
%
It is up to the programmer to unfold the
definitions at the appropriate places,
which we have found, with careful design
of proof combinators, to be quite
a natural and pleasant experience.
%
To this end, we have developed a library
of proof combinators that permits reasoning
about equalities and linear arithmetic,
inspired by Agda~\citep{agdaequational}.

\mypara{``Equation'' Combinators}
%
We equip \toolname with a family of
equation combinators @op.@ for each
logical operator @op@ in
$\{=, \not =, \leq, <, \geq, > \}$,
the operators in the theory QF-UFLIA.
%
The refinement type of @op.@  \emph{requires}
that $x \odot y$ holds and then \emph{ensures}
that the returned value is equal to @x@.
%
For example, we define @=.@ as:
%
\begin{code}
  (=.) :: x:a -> y:{a| x=y} -> {v:a| v=x}
  x =. _ = x
\end{code}
%
and use it to write the following ``equational" proof:
%
\begin{code}
  eqPf_fib2 :: { fib 2 = 1 }
  eqPf_fib2 =  #fib# 2
            =. #fib# 1 + #fib# 0
            =. 1
            ** QED
\end{code} %$

\mypara{``Because'' Combinators}
%
Often, we need to compose ``lemmata'' into larger
theorems. For example, to prove @fib 3 = 2@ we
may wish to reuse @eqPf_fib2@ as a lemma.
%
To this end, \toolname has a ``because'' combinator:
%
\begin{mcode}
  ($\because$) :: ($\typp$ -> a) -> $\typp$ -> a
  f $\because$ y = f y
\end{mcode}
%
The operator is simply an alias for function
application that lets us write
%
@ x op. y $\because$ p@ (instead of @(op.) x y p@)
where @(op.)@ is extended to accept an \textit{optional} third proof
argument via Haskell's typeclass mechanisms.
%
We use the because combinator to
prove that @fib 3 = 2@ with a Haskell function:
%
\begin{mcode}
  eqPf_fib3 :: { fib 3 = 2 }
  eqPf_fib3 =  #fib# 3
            =. fib 2 + #fib# 1
            =. 2              $\because$ eqPf_fib2
            ** QED
\end{mcode}

\mypara{Arithmetic and Ordering}
%
SMT based refinements let us go well beyond just equational
reasoning. Next, lets see how we can use arithmetic and
ordering to prove that @fib@ is (locally) increasing,
%
\ie for all $n$, $\fib{n} \leq \fib{(n+1)}$
%
\begin{mcode}
  fibUp :: n:Nat -> { fib n <= fib (n+1) }
  fibUp n
    | n == 0
    =  #fib# 0 <. #fib# 1
    ** QED

    | n == 1
    =  fib 1 <=. fib 1 + fib 0 <=. #fib# 2
    ** QED

    | otherwise
    =  #fib# n
    =. fib (n-1) + fib (n-2)
    <=. fib n     + fib (n-2) $\because$ fibUp (n-1)
    <=. fib n     + fib (n-1) $\because$ fibUp (n-2)
    <=. #fib# (n+1)
    ** QED
\end{mcode} %$

\mypara{Case Splitting and Induction}
%
The proof @fibUp@ works by induction on @n@.
%
In the \emph{base} cases @0@ and @1@, we simply assert
the relevant inequalities. These are verified as the
reflected refinement unfolds the definition of
@fib@ at those inputs.
%
The derived VCs are (automatically) proved
as the SMT solver concludes $0 < 1$ and $1 + 0 \leq 1$
respectively.
%
In the \emph{inductive} case, @fib n@ is unfolded
to  @fib (n-1) + fib (n-2)@, which, because of the
induction hypothesis (applied by invoking @fibUp@
at @n-1@ and @n-2@) and the SMT solver's arithmetic
reasoning, completes the proof.

\mypara{Higher Order Theorems}
%
Refinements smoothly accomodate higher-order reasoning.
%
For example, lets prove that every locally increasing
function is monotonic, \ie
if @f z <= f (z+1)@ for all @z@,
then @f x <= f y@ for all @x < y@.
%
\begin{mcode}
  fMono :: f:(Nat -> Int)
        -> fUp:(z:Nat -> {f z <= f (z+1)})
        -> x:Nat
        -> y:{x < y}
        -> {f x <= f y} / [y]
  fMono f inc x y
    | x + 1 == y
    =  f x <=. f (x+1) $\because$ fUp x
           <=. f y
           ** QED

    | x + 1 < y
    =  f x <=. f (y-1) $\because$ fMono f fUp x (y-1)
           <=. f y     $\because$ fUp (y-1)
           ** QED
\end{mcode}
%
We prove the theorem by induction
on @y@, which is specified by the
annotation @/ [y]@ which states
that @y@ is a well-founded
termination metric that decreases
at each recursive call~\citep{Vazou14}.
%
% All reflected functions are proved terminating.
% When the annotation metric is not explicit Liquid Haskell
% successfully uses heuristics to automatically prove termination.
%
If @x+1 == y@, then we use @fUp x@.
%
Otherwise, @x+1 < y@, and we use
the induction hypothesis \ie apply
@fMono@ at @y-1@, after which
transitivity of the less-than
ordering finishes the proof.
%
We can use the general @fMono@
theorem to prove that @fib@
increases monotonically:
%
\begin{code}
  fibMono :: n:Nat -> m:{n<m} -> {fib n <= fib m}
  fibMono = fMono fib fibUp
\end{code}

\subsection{Case Study: Deterministic Parallelism}
\label{sec:detpar}

%% The natural integration of deep verification with a language like Haskell makes
%% it possible to engage in lightweight, incremental verification of program
%% properties.

One benefit of an in-language prover is that it lowers the barrier to {\em
  small} verification efforts that touch only a fraction of the program, and yet
ensure critical invariants that Haskell's type system cannot.  Here we consider
parallel programming, which is commonly considered error prone and entails
proof obligations on the user that typically go unchecked.

The situation is especially precarious with parallel programming frameworks that
claim to be {\em determinstic} and thus usable within purely functional
programs.  These include Deterministic Parallel Java (DPJ \cite{DPJ}), Concurrent
Revisions for .NET~\cite{concurrent-revisions-oopsla}, and Haskell's
LVish~\cite{kuper2014freeze}, Accelerate~\cite{accelerate-icfp13}, and
REPA~\cite{repa-icfp10}.
%
Accelerate's parallel fold function, for instance, claims to be
deterministic---and its purely functional type means the Haskell optimizer will
{\em assume} its referential transparency---but its determinism depends on an
associativity guarantee which must be assured {\em by the programmer} rather than the
type system.
%
Thus simply folding the minus function, @fold (-) 0 arr@, is sufficient to
violate determinism and Haskell's pure semantics.

Likewise, DPJ goes to pains to develop a new type system for parallel
programming, but then provides a ``commutes'' annotation for methods updating
shared state, compromising the {\em guarantee} and going back to trusting the
user. LVish has the same Achilles heel. Consider set insertion:

\begin{code}
  insert :: Ord a => a -> Set s a -> Par s ()
\end{code}

Here @insert@ returns an (effectful) @Par@ computation, which can be run within a
pure function to produce a pure result.  At first glance it would seem that
trusting the implementation of the concurrent set is sufficient to assure a
deterministic outcome.  Yet the interface has an @Ord@ constraint. This
 polymorphic function works with user-defined data types, and thus
user-defined orderings.  What if the user fails to implement a total order?
Then, even a correct implementation of, e.g. a concurrent
skiplist~\cite{concurrent-skiplist}, can reveal
different insertion orders due to concurrency.

%% verifiedInsert :: HasPut e => VerifiedOrd a
%%                => a -> ISet s a -> Par e s ()

% \mypara{LVish}
%% We demonstrate the use of \toolname{} to ensure guarantees of
%% deterministic parallel programming. We choose this case study, because, to the
%% best of our knowledge, there exists no practical deterministic parallel
%% programming system, including user-defined parallel folds, which does not have
%% {\em soundness holes}---due to trusted assumptions of user code.

%% {\em LVish}\cite{kuper2014freeze} is a programming library for Haskell, which
%% exposes effectful parallel programming against lattice-variables (LVars) whose
%% states change monotonically during parallel regions of program execution. LVish
%% programs operate on Haskell data types, and LVish requires the operations on
%% these datatypes to satisfy some first order laws, which cannot be expressed in
%% Haskell. However, we can leverage \toolname to verify these properties for
%% arbitrary user-defined datatypes.

%% LVish provides two implementations of concurrent sets, @PureSet@ and @SLSet@,
%% where the underlying data structure is a size-balanced binary tree and
%% concurrent skiplist respectively. The @insert@ operation on a set requires a
%% total ordering on the elements, we can express that in the type signature by
%% \new{The implementation doesn't change, in fact, the}
%% @VerifiedOrd@ \new{methods do not even need to exist at runtime. A sufficiently
%%   smart compiler could optimize away these proof obligations during code
%%   generation.}
% \RN{Let's save the issue of runtime impact for the eval.}

In summary, parallel programs naturally need to communicate, but the mechanisms
of that communication---such as folds or inserts into a shared
structure---typically carry additional proof obligations.  This in turn makes
parallelism a liability.  But we can remove the risk with verification.

% But what if we could use verification to remove the risk?

% through contributions to shared structures (otherwise they are really separate
% programs)

\mypara{Verified typeclasses}
%
Our solution is simply to change the @Ord@ constraint above to
@VerifiedOrd@.
\begin{mcode}
  insert :: VerifiedOrd a => a -> Set s a -> Par s ()
\end{mcode}
%
This constraint changes the interface but not the implementation of @insert@.
%
% \NV{Why does insert now requires Verified Ord? Is it using the extra methods
% in the implementation?}
%% \note{VerifiedSemigroup story + lifting + isomorphism ("bootstrapping
%%   instances") + detpar propaganda}
%
%% It is an informal requirement when using
%% typeclasses in GHC that some typeclass laws be satisfied. For example, the @Ord@
%% typeclass in GHC requires that the $\leq$ operation be a total order. Using
%% \toolname, we can extend it to include the required properties of a total order,
%% which we call a @VerifiedOrd@.
The additional methods of the verified type class don't add operational
capabilities, but rather impose additional proof obligations:

\begin{code}
  class Ord a => VerifiedOrd a where
   antisym :: x:a -> y:a -> { x <= y && y <= x => x = y }
   trans   :: x:a -> y:a -> z:a -> { x <= y && y <= z => x <= z }
   total   :: x:a -> y:a -> { x <= y || y <= x }
\end{code}

% ---------------------------------------------------------------
% \mypara{Verified Monoids}

Similarly, we can extend
the @Monoid@ typeclass to a @VerifiedMonoid@, with refinements
expressing @Monoid@ laws.
%
\begin{code}
  class Monoid a => VerifiedMonoid a where
   lident :: x:a -> { mempty <> x = x }
   rident :: x:a -> { x <> mempty = x }
   assoc  :: x:a -> y:a -> z:a -> { x <> (y <> z) = (x <> y) <> z }
\end{code}
The @VerifiedMonoid@ typeclass constraint requires the binary operation
to be associative, thus can be safely used to fold on
an unknown number of processors.
%% A parallel fold requires the underlying binary operation to be associative and
%% have a well-behaved identity element, or a @Monoid@.

%% We can then extend the @ParFoldable@ typeclass to a @VerifiedParFoldable@ which
%% enforces a @VerifiedMonoid@ constraint.

%% \NV{Not sure if the below code adds any information: too difficult to follow,
%%   especially for non Haskell people} \NV{I suggest to say similarly to Verified
%%   Ord and add a link to appendix}
%%\RN{I concur with Niki -- we often whitewash away details of the library for
%%  presentation purposes.  E.g. we are not going to explain effect signatures in
%%  this paper.}

%% \begin{code}
%% class ParFoldable c
%%    => VerifiedParFoldable c where
%%   verifiedPmapFold :: forall m e s a .
%%   ( ParFuture m, HasGet e
%%   , HasPut e, FutContents m a,
%%   , VerifiedMonoid a )
%%   => (ElemOf c -> m e s a) -- compute one
%%                            -- result
%%   -> c                     -- element generator
%%                            -- to consume
%%   -> m e s a
%% \end{code}

%%  -------------------------------------------------------------------------

\mypara{Verified instances for primitive types}
@VerifiedOrd@ instances for primitive types like @Int@, @Double@ are trivial to
write; they just appeal to the SMT solver's built-in theories.
%
For example, the following is a valid totality proof on @Int@.
\begin{code}
  totInt :: x:Int -> y:Int -> {x <= y || y <= x}
  totInt _ _ = trivial ** QED
\end{code}

\mypara{Verified instances for algebraic datatypes}
%
To prove the class laws for user defined algebraic datatypes,
refinement reflection allows for structurally inductive proof terms.
%
For example, we can inductively define Peano numerals
%
\begin{code}
  data Peano = Z | S Peano
\end{code}
%
We can compare two @Peano@ numbers via
\begin{code}
  reflect leq :: Peano -> Peano -> Bool
  leq Z _         = True
  leq (S n) Z     = False
  leq (S n) (S m) = leq n m
\end{code}
%
In \S~\ref{sec:refinementreflection:theory} we will describe
exactly how the reflection mechanism (illustrated
via @fibP@) is extended to account for ADTs like @Peano@.
%
\toolname automatically checks
that @leq@ is total~\citep{Vazou14}, which
lets us safely @reflect@ it into the logic.

Next, we prove that @leq@ is total on @Peano@ numbers
%
\begin{mcode}
  totalPeano :: n:Peano -> m:Peano -> {leq n m || leq m n} / [toInt n + toInt m]
  totalPeano Z m = leq Z m ** QED
  totalPeano n Z = leq Z n ** QED
  totalPeano (S n) (S m)
   =  leq (S n) (S m) || leq (S m) (S n)
   =. leq n m || leq m n
   =. True $\because$ totalPeano m n
   ** QED
\end{mcode}
The proof goes by induction, splitting cases on
whether the number is zero or non-zero. Consequently,
we pattern match on the parameters @n@ and @m@, and furnish
separate proofs for each case.
%
In the ``zero" cases, we simply unfold the definition
of @leq@.
%
In the ``successor" case, after unfolding we (literally)
apply the induction hypothesis by using the because operator.
%
The termination hint @[toInt n + toInt m]@,
where @toInt@ maps @Peano@ numbers to integers,
is used to verify well-formedness of the @totalPeano@
proof term.
%
\toolname's termination and totality checker
use the hint to
verify that we are in fact doing induction
properly~(\S~\ref{sec:types-reflection}).

Similarly to @totalPeano@, we can define the rest of the @VerifiedOrd@
proof methods and use them to create the verified instance.
%
\begin{code}
  instance Ord Peano where
    (<=) = leq

  instance VerifiedOrd Peano where
    total = totalPeano
\end{code}
%
Proving all the four @VerifiedOrd@ laws
is a burden on the programmer.
%becomes a burden as the datatype grows more complicated.
%
Since @Peano@ is isomorphic to @Nat@s,
next we present how
to reduce the @Peano@ proofs into the
SMT automated integer proofs.

\mypara{Isomorphisms}
%
In order to reuse proofs for a custom datatype,
we provide a way to translate verified instances between isomorphic types~\cite{barthe2001type}.
%% If our datatype is isomorphic to a nesting of binary sums and products, we
%% should be able to reusing existing proofs.
%% To verify operations on custom data types efficiently, we
%% need to be able lift verified instances on one type to another.
%
We design a typeclass @Iso@ which witnesses the fact that
two types are isomorphic.
%, with respect to the built-in equality in \toolname{}
%which is a congruence.

\begin{mcode}
  class Iso a b where
    to      :: a -> b
    from    :: b -> a
    to$\circ$from :: x:a -> {to (from x) = x}
    from$\circ$to :: x:a -> {from (to x) = x}
\end{mcode}
%
For two isomorphic types @a@ and @b@
we compare instances of @b@ using @a@'s
comparison method.
%
\begin{mcode}
  instance (Ord a, Iso a b) => Ord b where
    x <= y = from x <= from y
\end{mcode}
%
Then, we prove that @VerifiedOrd@ laws are closed under isomorphisms.
%
For example, we prove totality of comparison on @b@s
using the @VerifiedOrd@ totality on @a@s

\begin{mcode}
  isoTotal :: (VerifiedOrd a, Iso a b) => x:b -> y:b -> {x <= y || y <= x}
  isoTotal x y
   =  x <= y || y <= x
   =. (from x) <= (from y) || (from y) <= (from x)
      $\because$ total (from x) (from y)
   ** QED
\end{mcode}
%
We use @isoTotal@ to create a verified instance on @b@s.
\begin{mcode}
  instance (VerifiedOrd a, Iso a b) => VerifiedOrd b where
    total   = isoTotal
\end{mcode}
%
With the above technique,
and using Haskell's instances,
getting a @VerifiedOrd@ instance for @Peano@
reduces to definition of an @Iso Nat Peano@.
%\VC{Iso (Either () Peano) Peano}

\mypara{Proof Composition via Products}
Finally, we present a mechanism to automatically
reduce proofs on product types to proofs of the product components.
%
For example, lexicographic ordering preserves the ordering laws.
%
First, we use class instances to define lexicographic ordering.
%
\begin{mcode}
  instance (VerifiedOrd a, VerifiedOrd b) => Ord (a, b) where
    (x1, y1) <= (x2, y2) = if x1 == x2 then y1 <= y2 else x1 <= x2
\end{mcode}
%
Then, we prove that lexicographic ordering
preserves the ordering laws.
%
For example, it preserves totality.
%
\begin{mcode}
  prodTotal :: (VerifiedOrd a, VerifiedOrd b)
            => p:(a, b) -> q:(a, b) -> {p <= q || q <= p}
  prodTotal p@(x1, y1) q@(x2, y2)
   =  p <= q || q <= p
   =. if x1 == x2 then (y1 <= y2 || y2 <= y1) else True
      $\because$ total x1 x2
   =. if x1 == x2 then True                   else True
      $\because$ total y1 y2
   ** QED
\end{mcode}
%
Finally, using the @prodTotal@ proof method,
we conclude that each instance defined via the lexicographic
ordering is indeed a verified instance.
%
\begin{mcode}
  instance (VerifiedOrd a, VerifiedOrd b) => VerifiedOrd (a, b) where
    total   = prodTotal
\end{mcode}
%
For example the type @(Peano, Peano)@ is derived to be a @VerifiedOrd@ instance.

In short, we can decompose an algebraic datatype into an isomorphic type using sums and
products to generate verified instances for arbitrary Haskell
datatypes. This could be combined with the Glasgow Haskell Compiler's (GHC) support
for generics~\cite{ghc-generics} to automate the derivation of verified instances
for user datatypes.
In \S\ref{sec:eval-parallelism}, we use these ideas to develop fully safe
interfaces to LVish modules, as well as verifying programming patterns from DPJ.


\chapter{Interlude: Cauchy's functional equation and Zorn's lemma}
\label{ch:zorn}
\emph{This is an informal chapter on Zorn's lemma,
which will give an overview of what's going to come in the last parts of the Napkin.
It can be omitted without loss of continuity.}

\medskip

In the world of olympiad math, there's a famous functional equation that goes as follows:
\[ f : \RR \to \RR \qquad f(x+y) = f(x) + f(y). \]
Everyone knows what its solutions are!
There's an obvious family of solutions $f(x) = cx$.
Then there's also this family of\dots\ uh\dots\ noncontinuous solutions (mumble grumble) pathological
(mumble mumble) Axiom of Choice (grumble).

There's also this thing called Zorn's lemma. It sounds terrifying,
because it's equivalent to the Axiom of Choice, which is also terrifying because why not.

In this post I will try to de-terrify these things,
because they're really as terrifying as they sound.

\section{Let's construct a monster}
Let us just see if we can try and construct a ``bad'' $f$ and see what happens.

By scaling, let's assume WLOG that $f(1) = 1$.
Thus $f(n) = n$ for every integer $n$, and you can easily show from here that
\[ f\left( \frac mn \right) = \frac mn. \]
So $f$ is determined for all rationals. And then you get stuck.

None of this is useful for determining, say, $f(\sqrt 2)$.
You could add and subtract rational numbers all day
and, say, $\sqrt 2$ isn't going to show up at all.

Well, we're trying to set things on fire anyways, so let's set
\[ f(\sqrt 2) = 2015 \]
because why not?
By the same induction, we get $f(n\sqrt2) = 2015n$, and then that
\[ f\left( a + b \sqrt 2 \right) = a + 2015b. \]
Here $a$ and $b$ are rationals.
Well, so far so good -- as written, this is a perfectly good solution,
other than the fact that we've only defined $f$ on a tiny portion of the real numbers.

Well, we can do this all day:
\[ f\left( a + b \sqrt 2 + c \sqrt 3 + d \pi \right) = a + 2015b + 1337c - 999d. \]
Perfectly consistent.

You can kind of see how we should keep going now.
Just keep throwing in new real numbers which are ``independent''
to the previous few, assigning them to whatever junk we want.
It feels like it \emph{should} be workable. . .

In a moment I'll explain what ``independent'' means (though you
might be able to guess already), but at the moment there's a bigger issue:
no matter how many numbers we throw, it seems like we'll never finish.
Let's address the second issue first.

\section{Review of finite induction}
When you do induction, you get to count off $1$, $2$, $3$, \dots and so on.
So for example, suppose we had a ``problem'' such as:
\begin{quote}
	Prove that the intersection of $n$ open intervals is either $\varnothing$
	or an open interval.
\end{quote}
You can do this by induction easily: it's true for $n = 2$, and
for the larger cases it's similarly easy.

But you can't conclude from this that \emph{infinitely} many open intervals intersect
at some open interval. Indeed, this is false: consider the intervals
\[
	\left( -1, 1 \right), \quad
	\left( -\frac12, \frac12 \right), \quad
	\left( -\frac13, \frac13 \right), \quad
	\left( -\frac14, \frac14 \right), \quad
	\dots
\]
This \emph{infinite} set of intervals intersects at a single point $\{0\}$!

The moral of the story is that induction doesn't let us reach infinity.
Too bad, because we'd have loved to use induction to help us construct a monster.
That's what we're doing, after all -- adding things in one by one.

\section{Transfinite induction}
Well, it turns out we can, but we need a new notion of number,
the so-called \emph{ordinal number}.
I define these in their full glory in the first two sections of \Cref{ch:ordinal}
(and curious readers are even invited to jump ahead to those two sections),
but for this chapter I won't need that full definition yet.

Here's what I want to say: after all the natural numbers
\[ 0, \; 1, \; \dots, \]
I'll put a \emph{new number} called $\omega$,
the first ordinal greater than all the natural numbers.
After that there's more numbers called
\[\omega+1, \; \omega+2, \; \dots \]
and eventually a number called $2\omega$.

The list goes on:
\[
\begin{aligned}
	0, & 1, 2, 3, \dots, \omega \\
	& \omega+1, \omega+2, \dots, \omega+\omega \\
	& 2\omega+1, 2\omega+2, \dots, 3\omega \\
	& \vdots \\
	& \omega^2 + 1, \omega^2+2, \dots \\
	& \vdots \\
	& \omega^3, \dots, \omega^4, \dots, \omega^\omega
	\dots, \omega^{\omega^{\omega^{\dots}}} \\
\end{aligned}
\]
Pictorially, it kind of looks like this:
\begin{center}
	\includegraphics[scale=0.70]{media/500px-Omega-exp-omega-labeled.png}
	\\ \scriptsize Image from \cite{img:omega500}
\end{center}
(Note that the diagram only shows an initial segment;
there are still larger ordinals like $\omega^{\omega^{\omega}}+1000$ and so on).

Anyways, in the same way that natural numbers ``dominate'' all finite sets,
the ordinals dominate \emph{all the sets}, in the following sense.
Essentially, assuming the Axiom of Choice,
it follows that for every set $S$ there's some ordinal $\alpha$
which is larger than $S$ (in a sense I won't make precise until later chapters).

But it turns out (and you can intuitively see) that as large as the ordinals grow,
there is no \emph{infinite descending chain}.
Meaning: if I start at an ordinal (like $2 \omega + 4$) and jump down, I can only
take finitely many jumps before I hit $0$.
(To see this, try writing down a chain starting at $2 \omega + 4$ yourself.)
Hence, induction and recursion still work verbatim:
\begin{theorem}[Transfinite induction]
	Given a statement $P(-)$, suppose that
	\begin{itemize}
		\ii $P(0)$ is true, and
		\ii If $P(\alpha)$ is true for all $\alpha < \beta$, then $P(\beta)$ is true.
	\end{itemize}
	Then $P(\beta)$ is true.
\end{theorem}
Similarly, you're allowed to do recursion to define $x_\beta$ if you know the
value of $x_\alpha$ for all $\alpha < \beta$.

The difference from normal induction or recursion is that we'll often
only do things like ``define $x_{n+1} = \dots$''.
But this is not enough to define $x_\alpha$ for all $\alpha$.
To see this, try using our normal induction and see how far we can climb up the ladder.

Answer: you can't get $\omega$!
It's not of the form $n+1$ for any of our natural numbers $n$ -- our finite induction only lets us
get up to the ordinals less than $\omega$.
Similarly, the simple $+1$ doesn't let us hit the ordinal $2\omega$,
even if we already have $\omega+n$ for all $n$.
Such ordinals are called \vocab{limit ordinals}.
The ordinal that \emph{are} of the form $\alpha+1$ are called \vocab{successor ordinals}.

So a transfinite induction or recursion is very often broken up into three cases.
In the induction phrasing, it looks like
\begin{itemize}
	\ii (Zero Case) First, resolve $P(0)$.
	\ii (Successor Case) Show that from $P(\alpha)$ we can get $P(\alpha+1)$.
	\ii (Limit Case) Show that $P(\lambda)$ holds given $P(\alpha)$ for all $\alpha < \lambda$,
	where $\lambda$ is a limit ordinal.
\end{itemize}
Similarly, transfinite recursion often is split into cases too.
\begin{itemize}
	\ii (Zero Case) First, define $x_0$.
	\ii (Successor Case) Define $x_{\alpha+1}$ from $x_\alpha$.
	\ii (Limit Case) Define $x_\lambda$ from $x_\alpha$ for all $\alpha < \lambda$,
	where $\lambda$ is a limit ordinal.
\end{itemize}
In both situations, finite induction only does the first two cases,
but if we're able to do the third case we can climb far above the barrier $\omega$.

\section{Wrapping up functional equations}
Let's return to solving our problem.

Let $S_n$ denote the set of ``base'' numbers we have at the $n$th step.
In our example, we might have
\[
	S_1 = \left\{ 1 \right\}, \quad
	S_2 = \left\{ 1, \sqrt 2 \right\}, \quad
	S_3 = \left\{ 1, \sqrt 2, \sqrt 3 \right\}, \quad
	S_4 = \left\{ 1, \sqrt 2, \sqrt 3, \pi \right\}, \quad
	\dots
\]
and we'd like to keep building up $S_i$ until we can express all real numbers.
For completeness, let me declare $S_0 = \varnothing$.

First, I need to be more precise about ``independent''.
Intuitively, this construction is working because
\[ a + b \sqrt 2 + c \sqrt 3 + d \pi \]
is never going to equal zero for rational numbers $a$, $b$, $c$, $d$ (other than all zeros).
In general, a set $X$ of numbers is ``independent'' if the combination
\[ c_1 x_1 + c_2 x_2 + \dots + c_m x_m = 0 \]
never occurs for rational numbers $\QQ$ unless $c_1 = c_2 = \dots = c_m = 0$.
Here $x_i \in X$ are distinct. Note that even if $X$ is infinite,
I can only take finite sums!
(This notion has a name: we want $X$ to be \textbf{linearly independent} over $\QQ$;
see the chapter on vector spaces for more on this!)

When do we stop?
We'd like to stop when we have a set $S_{\text{something}}$ that's so big,
every real number can be written in terms of the independent numbers.
(This notion also has a name: it's called a $\QQ$-basis.)
Let's call such a set \textbf{spanning};
we stop once we hit a spanning set.

The idea that we can induct still seems okay:
suppose $S_\alpha$ isn't spanning.
Then there's some number that is independent of $S_\alpha$, say $\sqrt{2015}\pi$ or something.
Then we just add it to get $S_{\alpha+1}$.
And we keep going.

Unfortunately, as I said before it's not enough to be able to go from $S_\alpha$ to $S_{\alpha+1}$
(successor case); we need to handle the limit case as well.
But it turns out there's a trick we can do.
Suppose we've constructed \emph{all} the sets $S_0$, $S_1$, $S_2$, \dots, one for each positive integer $n$,
and none of them are spanning.
The next thing I want to construct is $S_\omega$; somehow I have to ``jump''.
To do this, I now take the infinite union
\[ S_\omega \overset{\text{def}}{=} S_0 \cup S_1 \cup S_2 \cup \dots. \]
The elements of this set are also independent (why?).

Ta-da!
With the simple trick of ``union all the existing sets'',
we've just jumped the hurdle to the first limit ordinal $\omega$.
Then we can construct $S_{\omega+1}$, $S_{\omega+2}$, \dots, once again --
just keep throwing in elements.
Then when we need to jump the next hurdle to $S_{2 \omega}$,
we just do the same trick of ``union-ing'' all the previous sets.

So we can formalize the process as follows:
\begin{enumerate}
	\ii Let $S_0 = \varnothing$.
	\ii For a successor stage $S_{\alpha+1}$, add any element to $S_\alpha$ to obtain $S_{\alpha+1}$.
	\ii For a limit stage $S_{\lambda}$, take the union $\bigcup_{\gamma < \lambda} S_\gamma$.
\end{enumerate}
How do we know that we'll stop eventually?
Well, the thing is that this process consumes a lot of real numbers.
In particular, the ordinals get larger than the size of $\RR$ (assuming Choice).
Hence if we don't stop we will quite literally reach a point where we have used up every single real number.
Clearly that's impossible, because by then the elements can't possibly be independent!

So by transfinite recursion, we eventually hit some $S_\gamma$ which is spanning:
the elements are all independent, but every real number can be expressed using it.
Done!

\section{Zorn's lemma}
Now I can tell you what Zorn's lemma is:
it lets us do the same thing in any poset.

We can think of the above example as follows:
consider all sets of independent elements.
These form a partially ordered set by inclusion, and what we did
was quite literally climb up a chain
\[ S_0 \subsetneq S_1 \subsetneq S_2 \subsetneq \dots. \]
It's not quite climbing since we weren't just going one step at a time:
we had to do ``jumps'' to get up to $S_\omega$ and resume climbing.
But the main idea is to climb up a poset until we're at the very top;
in the previous case, when we reached the spanning set.

The same thing works verbatim with any \href{http://en.wikipedia.org/wiki/Partially_ordered_set}{partially ordered set}
$\mathcal P$.
Let's define some terminology.
A \vocab{local maximum} of the entire poset $\mathcal P$ is an element
which has no other elements strictly greater than it.
(Most authors refer to this as ``maximal element'', but I think
``local maximum'' is a more accurate term.)

Now a \vocab{chain of length $\gamma$} is a set of elements $p_\alpha$ for every $\alpha < \gamma$
such that $p_0 < p_1 < p_2 < \dots$.
(Observe that a chain has a last element if and only if $\gamma$ is a successor ordinal, like $\omega+3$.)
An \vocab{upper bound} to a chain is an element $\tilde p$ which is greater than or equal
to all elements of the chain;
In particular, if $\gamma$ is a successor ordinal, then just taking the last element of the chain works.

In this language, Zorn's lemma states that
\begin{theorem}
	[Zorn's lemma]
	Let $\mathcal P$ be a nonempty partially ordered set.
	If every chain has an upper bound,
	then $\mathcal P$ has a local maximum.
\end{theorem}

Chains with length equal to a successor ordinal always have upper bounds,
but this is not true in the limit case.
So the hypothesis of Zorn's lemma is exactly what
lets us ``jump'' up to define $p_\omega$ and other limit ordinals.
And the proof of Zorn's lemma is straightforward: keep climbing up the poset at successor stages,
using Zorn's condition to jump up at limit stages, and thus building a really long chain.
But we have to eventually stop, or we literally run out of elements of $\mathcal P$.
And the only possible stopping point is a local maximum.

If we want to phrase our previous solution in terms of Zorn's lemma, we'd say:
\begin{proof}
	Look at the poset whose elements are sets of independent real numbers.
	Every chain $S_0 \subsetneq S_1 \subsetneq \dots$ has an upper bound $\bigcup S_\alpha$
	(which you have to check is actually an element of the poset).
	Thus by Zorn, there is a local maximum $S$.
	Then $S$ must be spanning, because otherwise we could add an element to it.
\end{proof}
So really, Zorn's lemma is encoding all of the work of climbing that I argued earlier.
It's a neat little package that captures all the boilerplate, and tells
you exactly what you need to check.

\begin{center}
	\includegraphics[scale=0.5]{media/zornaholic.png}
	\\ \scriptsize Image from \cite{img:zornaholic}
\end{center}

One last thing you might ask:
where is the Axiom of Choice used?
Well, the idea is that for any chain there could be lots of $\tilde p$'s,
and you need to pick one of them.
Since you are making arbitrary choices infinitely many times, you need the Axiom of Choice.
(Actually, you also need choice to talk about cardinalities as in theorem 1.)
But really, it's nothing special.

\section{\problemhead}
\begin{problem}
	[Tukey's lemma]
	Let $\mathcal F$ be a nonempty family of sets.
	Assume that for any set $A$,
	the set $A$ is in $\mathcal F$
	if and only if all its finite subsets are in $\mathcal F$.
	Prove that there exists $Y \in \mathcal F$
	such that $X \subseteq Y$ for every $X \in \mathcal F$.
\end{problem}


\chapter{Universal Functions}
It is known that we may write a program that gets another program as an argument
and run it. (Such programs are known as interpreters.) To use this observation
we give the following definition and theorem.
\begin{definition}
  We say that a function $U$ is a \emph{universal function} (for the set of
  univariate computable functions) iff for each $n \in \N$,
  \[
    U_n : x \mapsto U(n, x)
  \]
  (we say that $U_n$ is a section of $U$)
  is computable and any univariate computable function is among $U_n$'s.
\end{definition}

\begin{theorem}
\label{theorem:universal-function-computable}
  There is a computable universal function $U$.
\end{theorem}

\begin{exercise}
  Assume that every section of a function $U$ is computable.
  Is it necessarily true that $U$ is computable?
\end{exercise}

Similarly to the notion of universal function we may define the notion
of universal sets.
\begin{definition}
  Let $F \subseteq 2^\N$. We say that $W \subseteq \N^2$ is universal for $F$
  if $W_n = \set[(n, x) \in F]{x \in \N}$ is an element of $F$ for all
  $n \in \N$ and any set $S \in F$ is among $W_n$'s.
\end{definition}

\begin{theorem}
\label{theorem:universal-set-enumerable}
  There is an enumerable set $W$ such that it is universal for the set of
  all enumerable subsets of $\N$.
\end{theorem}

\section{Enumerable but Not Decidable Set}
\begin{theorem}
\label{theorem:enumerable-not-decidable}
  There is a set $S \subseteq \N$ such that $S$ is enumerable but it is not
  decidable.
\end{theorem}
\begin{proof}
  Let $U$ be be a universal computable function, it exists by
  \Cref{theorem:universal-function-computable}.
  To prove the statement we are going to use the diagonalization method.
  Let us consider $S = \set[U(n, n) = 0]{n \in \N}$.

  It is easy to see that $S$ is enumerable. Assume for the sake of
  contradiction that $S$ is decidable. Let $\Algorithm{A}$ be the algorithm
  deciding $S$. There is $n \in \N$ such that $\Algorithm{A}$ computes $U_n$
  since $U$ is universal. Let us now consider two following cases.
  \begin{enumerate}
    \item Assume that $n \in S$. In this case $\Algorithm{A}(n) = 1$ since
      $\Algorithm{A}$ decides $S$. However, $U_n(n) \neq 1$ by the definition of
      $S$. These two equalities together leads us to a contradiction since
      $\Algorithm{A}$ computes $U_n$.
    \item Assume that $n \notin S$. In this case $\Algorithm{A}(n) = 0$ since
      $\Algorithm{A}$ decides $S$. However, $U_n(n) = 1$ by the definition of
      $S$. These two equalities together leads us to a contradiction since
      $\Algorithm{A}$ computes $U_n$.
  \end{enumerate}
\end{proof}

Using the proof of this result we can prove the following surprising
observation.
\begin{theorem}
\label{theorem:halting}
  Let $U$ be a universal function.
  Let $\Halting : \N^2 \to \set{0, 1}$ be the function such that
  $\Halting(n, x) = 1$ iff $U_n(x)$ is defined. Then $\Halting$ is not
  computable.
\end{theorem}
Informally, this theorem says that it is impossible to check whether a given
algorithm terminates or not on some input.

\section{Diagonalization Method}
This section will give several other applications of the diagonalization method.

In the previous section we constructed a computable universal function for
the set of computable functions of one variable. Now we can prove that
it is impossible for total functions.
\begin{theorem}
    There is no computable universal total function for
    the set of computable total functions of one variable.
\end{theorem}
\begin{proof}
  Assume that such a function $U$ exists. Let us consider the total computable
  function $d : \N \to \N$ such that $d(n) = U(n, n) + 1$. Since $U$ is a
  computable universal total function for the set of computable total functions
  of one variable, there is $m \in \N$ such that $d(n) = U(m, n)$ for any
  $n \in \N$. Note that this implies that $U(m, m) = d(m) = U(m, m) + 1$,
  which is a contradiction.
\end{proof}
Note that the crucial point in this argument is that $U(m, m)$ is different
from $U(m, m) + 1$; however, if the functions are not total it is possible
that $U(m, m)$ is simply not defined. However, a part of the argument can be
used, nonetheless.
\begin{theorem}
\label{theorem:intersecting-function}
    There is a computable function $f : \N \to \N$ such that
    no computable function $g : \N \to \N$ can differ from $f$
    everywhere; i.e., for any computable function $g : \N \to \N$
    there is $n \in \N$ such that $f(n) = g(n)$.
    (The last equality says that the values are either equal or
    both values are not defined for $n$.)
\end{theorem}
\begin{proof}
  Let $U$ be a universal function for computable functions, and
  let $d : \N \to \N$ be a partial function such that $d(n) = U(n, n)$.
  It is clear that $d$ satisfies the statement of the theorem. Indeed,
  any computable function $f$ is equal to $U_n$  for some $n$. Hence,
  $d(n) = U(n, n) = f(n)$.
\end{proof}

\begin{theorem}
    There is a computable function that does not admit a total computable
    extension.
\end{theorem}
\begin{proof}
  Let $d$ be the function from the previous theorem. Let us consider the
  partial function $e : \N \to \N$ such that $e(n) = d(n) + 1$.
  We wish to prove that $e$ does not have a total extension. Let us assume
  the opposite, let $e'$ be a computable total extension of $e$. Then $e'$
  differs from $d$ everywhere, therefore $e'$ is not computable.
\end{proof}

Note that the last theorem gives another proof of
\Cref{theorem:enumerable-not-decidable}. Indeed, let $f$ be the function
without total computable extension. Let $S = \Im{f}$.
By \Cref{theorem:enumerable-via-computable-funcitons},
$S$ is enumerable. Assume for the sake of contradiction, that $S$ is decidable.
Then the total function $g : \N \to \N$ such that
\[
    g(x) =
    \begin{cases}
        f(x) & \text{if } x \in S \\
        0 & \text{if } x \notin S
    \end{cases}
\]
is computable. Moreover, $g$ is an extension of $f$ which leads to a
contradiction.

\begin{chapterendexercises}
    \exercise Let $U \subseteq \N^2$ be any enumerable set of pairs of
        natural numbers that is universal for the set of all
        enumerable sets of natural numbers.
        Prove that its ``diagonal section''
        $K = \set[(x, x) \in U]{x}$ is an enumerable undecidable set.
    \exercise Let $S \subseteq \N$ be decidable and let
        \[
            D =
            \set[
                p \text{ is prime and }
                p \text{ divides some } n \in S
            ]{p \in \N}.
        \]
        Is the set $D$ always decidable?
    \exercise Show that there exist countably many disjoint
        enumerable sets such that any two of them are inseparable
        (cannot be separated by a decidable set).
\end{chapterendexercises}


\documentclass[%
 %twocolumn,
 %preprint,
 onecolumn,
 amsmath, amssymb, aps, pra, 10pt
]{revtex4-2}
\usepackage{amsmath}
\usepackage{appendix}
\usepackage[colorlinks,citecolor=blue,urlcolor=black,bookmarks=false,hypertexnames=true]{hyperref}
\begin{document}
\title{Unifying Variable Order Fractional Derivatives}% Force line breaks with \
\author{Luke A. Siemens}
\email{luke.siemens@lsiemens.com}
\noaffiliation
\date{\today}
\maketitle

I have found a generalized definition of variable order fractional derivatives. I will show that this definition applies to any variable order fractional derivative provided it maps analytic functions to analytic functions and that they are analytic with respect to the order of differentiation. I will make the simplifying assumptions that the functions are analytic, that the domain of the functions is either $\mathbb{C}$ or $\mathbb{C} \times \mathbb{C}$, that they are analytic in the neighborhood of the origin and that they are analytic almost everywhere (in a measure theory sense). Denote the set of all analytic functions on the domain $\mathbb{C} \times \mathbb{C}$ as $\mathcal{O}(\mathbb{C}^2)$. Let us define a function space $\mathbb{S}$ as the set,
\begin{equation}
\mathbb{S} = \left\lbrace f(x, a) \in \mathcal{O}(\mathbb{C}^2) \middle| \partial_x f(x, a) = f(x, a - 1) \right\rbrace
\label{differentiable_set}
\end{equation}
and define the fractional integral operator $J^{\alpha}$ acting on elements of the set $\mathbb{S}$ as,
\begin{equation}
J^{\alpha} f(x, a) = T_{a}^{\alpha} f(x, a) = f(x, a + \alpha)
\label{fractional_derivative}
\end{equation}
where $T_{a}^{\alpha}$ is an operator shifting the variable $a$ by the amount $\alpha$. Note that if $f(x, a) \in \mathbb{S}$ then $J^{\alpha} f(x, a) \in \mathbb{S}$. I will now show that this operator, when acting on functions in the function space $\mathbb{S}$, satisfies the necessary properties to be considered a variable order fractional derivative, and it is equivalent to any sufficiently analytic variable order fractional derivative on some subspace of $\mathbb{S}$.
I will use the criterion, $\,^3P$, given in your paper, \textit{What is a fractional derivative} \cite{ORTIGUEIRA20154}, to determine if the operator \eqref{fractional_derivative} is a fractional derivative when acting on functions in the set \eqref{differentiable_set}. In the following arguments let us assume that $f(x, a), g(x, a) \in \mathbb{S}$ and $C_1, C_2, \alpha, \beta \in \mathbb{C}$.

\subsection*{$\,^3P1$ : Linearity}
Since the shift operator $T_{a}^{\alpha}$ is a linear operator, then the operator $J^{\alpha}$ is also a linear operator.

\subsection*{$\,^3P2$ : Identity}
Using the definition of $J^{\alpha}$ for $\alpha=0$ acting on $f(x, a)$ results in, $J^{0}f(x, a) = T_{a}^{0}f(x, a) = f(x, a)$. So the property $\,^3P2$ is satisfied.

\subsection*{$\,^3P3$ : Backwards compatibility}
Taking the fractional integral, $J^{\alpha} f(x, a)$, for $\alpha \in\ \mathbb{Z}$. In the case where $\alpha$ is a negative integer we can apply $\partial_x f(x, a) = f(x, a - 1)$ repeatedly, let $\alpha = -k$ with $k \in \mathbb{Z}^+$,
$$J^{-k} f(x, a) = T_{a}^{-k} f(x, a)=T_{a}^{-k + 1}T_{a}^{-1} f(x, a) = T_{a}^{-k + 1}f(x, a - 1) = T_{a}^{-k + 1}\partial_x f(x, a)$$
Since $\partial_x f(x, a) \in \mathbb{S}$ we can repeat this argument, resulting in
$$J^{-k} f(x, a) = \partial_{x}^{k} f(x, a)$$
For the positive case first let us try $k=1$. In this case $J^{1} f(x, a) = f(x, a + 1)$, so $\partial_x f(x, a + 1) = f(x, a)$ leading to the solution $J^{1} f(x, a) = \int_{x_0}^{x} f(t, a)dt + f(x_0, a + 1)$. Provided that $f(x, a)$ is an entire function for all $a$, then applying this repeatedly produces
$$J^{k} f(x, a) = \frac{1}{\Gamma(k)}\int_{x_0}^{x} (x - t)^{k - 1}f(t, a)dt + \sum_{i = 0}^{k - 1}f(x_0, a + k - i)\frac{(x - x_0)^i}{i!}$$
So if $\alpha \in \mathbb{Z}$, then $J^{\alpha}$ represents either repeated integration or differentiation.

\subsection*{$\,^3P4$ : Index law}
Applying the fractional integral twice and simplifying yields
$$J^{\beta}J^{\alpha} f(x, a) = T{a}^{\beta}T_{a}^{\alpha} f(x, a) = T_{a}^{\beta} f(x, a + \alpha) = f(x, a + \alpha + \beta) = J^{\beta + \alpha} f(x, a)$$
So the index law is satisfied.

\section*{Implications of $\,^3P1 - \,^3P4$ for $J^{\alpha}$}
The criterion $\,^3P1 - \,^3P4$ are satisfied, but before addressing criterion $\,^3P5$ I show that any sufficiently analytic variable order fractional derivative is equivalent to the operator \eqref{fractional_derivative} acting on some subspace of the set $\mathbb{S}$. Note I will use the fractional integral derived from the variable order fractional derivative. Now let $J'^{\alpha}$ be an arbitrary variable order fractional derivative that satisfies $\,^3P1 - \,^3P4$, and that there exists a set $\mathbb{S}''$ of analytic functions on which the operator $J'^{\alpha}$ satisfies $\,^3P1 - \,^3P4$ and where $\forall F(x) \in \mathbb{S}'', J'^{\alpha}F(x) \in \mathcal{O}(\mathbb{C}^2)$. The action of $J'^{\alpha}$ acting on $F(x) \in \mathbb{S}''$ can be expressed as $f(x, a) = J'^{\alpha} F(x)$. Using this let
$$\mathbb{S}' = \left\lbrace f(x, a) \in \mathcal{O}(\mathbb{C}^2) \middle| \exists F(x) \in \mathbb{S}'', f(x, a) = J'^{\alpha} F(x) \right\rbrace$$
Given $f(x, a) \in \mathbb{S}'$ then $J'^{-1}f(x, a) = \partial_x f(x, a)$ by $\,^3P3$ and $J'^{-1} f(x, a) = J'^{-1}J'^{\alpha} F(x) = J'^{\alpha - 1} F(x) = f(x, \alpha - 1)$ by $\,^3P4$, so
$$\forall f(x, a) \in \mathbb{S}', \partial_x f(x, a) = f(x, a - 1) \therefore \mathbb{S}' \subseteq \mathbb{S}$$
Given $f(x, a) \in \mathbb{S}'$ then taking a fractional integral and using $\,^3P4$ results in $J'^{\alpha} f(x, a) = J'^{\alpha}J'^{a} F(x) = J'^{\alpha + a} F(x) = f(x, a + \alpha) = T_{a}^{\alpha} f(x, a)$, so
$$J'^{\alpha} f(x,a) = T_{a}^{\alpha} f(x, a) = J^{\alpha} f(x, a)$$
We have shown that $\mathbb{S}' \subseteq \mathbb{S}$ and that $\forall f(x, a) \in \mathbb{S}', J'^{\alpha} f(x, a) = J^{\alpha} f(x, a)$. So every variable order fractional derivative which is sufficiently analytic is equivalent to $J^{\alpha}$ acting on some subspace of $\mathbb{S}$. The fractional integral defined in \eqref{fractional_derivative} when acting on functions in the space \eqref{differentiable_set} provides a general description of all sufficiently analytic variable order fractional derivatives subject to $\,^3P1 - \,^3P4$.

Up until this point I have used the equation $\partial_x f(x, a) = f(x, a - 1)$ to determine if a function is in the set $\mathbb{S}$ but have not considered whether or not this equation holds on the entire domain of $f(x, a)$ or only on some subset. Given $f(x, a) \in \mathcal{O}(\mathbb{C}^2)$ we can define $g(x, a) = \partial_x f(x, a) - f(x, a - 1)$. Note that if $f(x, a) \in \mathcal{O}(\mathbb{C}^2)$ then $g(x, a) \in \mathcal{O}(\mathbb{C}^2)$ and that at every point in the domain $\partial_x f(x, a) = f(x, a - 1) \iff g(x, a) = 0$. Since $g(x, a)$ is a complex analytic function it can be analytically continued and its continuation is either zero everywhere, or it is zero almost nowhere on the domain of the analytic continuation. Therefore $\partial_x f(x, a) = f(x, a - 1)$ must either be satisfied on the entire domain of $f(x, a)$, or satisfied almost nowhere since $f(x, a)$ is complex analytic.

\subsection*{$\,^3P5$ : Generalized Leibniz rule}
So far I have not found a consistent way to define multiplication in general. If $f(x, a), g(x, a) \in \mathbb{S}$ and $h(x, a) = f(x, a) \cdot g(x, a)$ then $\partial_x h(x, a) = f(x, a - 1) \cdot g(x, a) + f(x, a) \cdot g(x, a - 1) \neq h(x, a - 1)$, unless either $f(x, a)$ or $g(x, a)$ is a constant function. So if multiplication by non-constant functions is possible then the product operator needs to be modified, and due to $\,^3P3$ it needs to be compatible with the General Leibniz rule. In the simpler case of multiplication by an analytic function of one variable $g(x) \in \mathcal{O}(\mathbb{C})$, then a solution is to use a modification of the equation given in $\,^3P5$. Given the functions $f(x, a) \in \mathbb{S}$ and $g(x) \in \mathcal{O}(\mathbb{C})$ define multiplication, denoted by the symbol $*$, as
\begin{equation}
f(x, a) * g(x) = (f * g)(x, a) = \sum_{k=0}^{\infty} \binom{-a}{k}f(x, a + k) \cdot \frac{d^k}{dx^k} g(x)
\label{multiplication}
\end{equation}
Taking the partial derivative of $(f * g)(x, a)$ with respect to $x$, then
$$\partial_x (f * g)(x, a) = \sum_{k=0}^{\infty} \binom{-a}{k}f(x, a + k - 1) \cdot \frac{d^k}{dx^k} g(x) + \binom{-a}{k}f(x, a + k) \cdot \frac{d^{k + 1}}{dx^{k + 1}} g(x)$$
Using $k = k' - 1$ in the second term and $k = k'$ in the first, and since $\binom{-a}{k-1}$ is zero if $k=0$, we can rearrange to find
$$\partial_x (f * g)(x, a) = \sum_{k'=0}^{\infty} \left( \binom{-a}{k'} + \binom{-a}{k' - 1} \right)f(x, (a - 1) + k') \cdot \frac{d^{k'}}{dx^{k'}} g(x)$$
Now using the fact that $\binom{-a}{k} + \binom{-a}{k - 1} = \binom{-a + 1}{k}$ and the definition of $(f * g)(x, a)$, then
$$\partial_x (f * g)(x, a) = \sum_{k'=0}^{\infty} \binom{-(a - 1)}{k'}f(x, (a - 1) + k') \cdot \frac{d^{k'}}{dx^{k'}} g(x) = (f * g)(x, a - 1)$$
So if $f(x, a) \in \mathbb{S}$ and $g(x) \in \mathcal{O}(\mathbb{C})$, then $(f * g)(x, a) \in \mathbb{S}$, if it exists, and clearly $(f * g)(x, a)$ satisfies $\,^3P5$. Currently I am working on finding an operator that naturally generalizes multiplication that works when both functions are in the set $\mathbb{S}$.

\section*{PDE representation}
We can construct an alternative for $\partial_x f(x, a) = f(x, a - 1)$ by using the definition of the shift operator $T_{x}^{t} = e^{t \partial_x}$. Expanding the exponential function in this definition $T_{x}^t = \sum_{k=0}^{\infty} \frac{t^k}{k!} \partial_{x}^{k}$, allows us to express $\partial_x f(x, a) = f(x, a - 1)$ as the PDE $\partial_x f(x, a) - \sum_{k=0}^{\infty} \frac{(-1)^k}{k!}\partial_{a}^{k} f(x, a) = 0$. So we can solve for elements of $\mathbb{S}$ by solving the system of PDEs
\begin{align*}
&\partial_x f(x, a) - \sum_{k=0}^{\infty} \frac{(-1)^k}{k!} \partial_{a}^{k} f(x, a) = 0 \\
&\partial_{\bar{x}} f(x, a) = 0 \\
&\partial_{\bar{a}} f(x, a) = 0
\end{align*}
where the last two PDEs ensure that $f(x, a)$ is complex analytic.

\section*{Series solution}
Given $f(x, a) \in \mathcal{O}(\mathbb{C}^2)$ it can be described by the power series
$$f(x, a) = \sum_{j=0}^{\infty} \sum_{k=0}^{\infty} C_{j, k} \frac{x^j}{j!} \frac{a^k}{k!}$$
We can rearrange the series to the form $f(x, a) = \sum_{j=0}^{\infty} \frac{x^j}{j!} \sum_{k=0}^{\infty} C_{j, k} \frac{a^k}{k!}$. Then define $g_n(a) = \sum_{k=0}^{\infty} C_{n, k} \frac{a^k}{k!}$, so that $f(x, a) = \sum_{j=0}^{\infty} \frac{x^j}{j!} g_j(a)$, if $g_n(a)$ converges for all $n \in \mathbb{Z}^+$. If $f(x, a) \in \mathbb{S}$, then applying the equation $\partial_x f(x, a) = f(x, a - 1)$ to the power series representation produces,
$$\partial_x f(x, a) = \sum_{j=0}^{\infty} \sum_{k=0}^{\infty} C_{j + 1, k} \frac{x^j}{j!} \frac{a^k}{k!} = \sum_{j=0}^{\infty} \sum_{k=0}^{\infty} C_{j, k} \frac{x^j}{j!} \frac{(a - 1)^k}{k!}$$
Taking the $n$th partial derivative of $x$ on both sides and letting $x=0$ yields,
$$\sum_{k=0}^{\infty} C_{n + 1, k} \frac{a^k}{k!} = \sum_{k=0}^{\infty} C_{n, k} \frac{(a -1)^k}{k!}$$
Using the definition of $g_n(a)$ results in,
$$g_{n+1}(a) = g_{n}(a - 1)$$
Applying this equation repeatedly yields the equation,
$$g_{n}(a) = g_{0}(a - n)$$
So if $f(x, a) \in \mathbb{S}$, then
$$f(x, a) = \sum_{j=0}^{\infty} \frac{x^j}{j!} g_{j}(a) = \sum_{j=0}^{\infty} \frac{x^j}{j!} g_{0}(a - j)$$
Thus for any $g(a) \in \mathcal{O}(\mathbb{C})$, if the series $f(x, a) = \sum_{j=0}^{\infty} \frac{x^j}{j!} g(a - j)$ converges, then $f(x, a) \in \mathbb{S}$. Note that $g_{0}(a) = f(0, a)$. So if $f(x, a) \in \mathbb{S}$, then
\begin{equation}
f(x, a) = \sum_{j=0}^{\infty} \frac{x^j}{j!} f(0, a - j)
\label{series_solution}
\end{equation}
and if $f(x, a)$ is an entire function, then the series solution converges for all finite $x$ and $a$. Currently I am looking into using this idea to construct an alternative representation of the fractional calculus I have described in this document, as it looks like it might be a more natural description.

\bibliographystyle{plain}
\bibliography{scv_fractional_calculus.bib}
\end{document}


\documentclass[../../main.tex]{subfiles}

\begin{document}

\subsection{Motivation}

%Each report must be created in a separate subfolder. Take the "example" folder as an example of a report (please do not change).
%
%Here it is required to describe the problematic of the problem. Basic premises and results. It is important to indicate a list of references on this topic~\cite{AuthorYear}. All refernces put to the main.bib file.
%
%After writing your report, include the report to main.tex similar to the sample report.

%All changes are made through GitHub\footnote{\url{https://github.com/Intelligent-Systems-Phystech/GeometricDeepLearning}}.

The work investigates the problem of time series forecasting. The goal is to get a continuous forecast given discrete data points.

\subsection{Problem statement}

Given a time series $\mathbf{X} = (t_i, a_i)^{\ell}_{i=1}, ~t_1 \leq \ldots \leq t_\ell$. The number $a_i$ is the accelerometer reading. Given also unobservable velocities $v_i \in \mathbb{R}^1$ at each timestamp $t_i$. The current velocity $v_i$ can be computed through the previous one as follows: $v_i = f(v_{i-1},a_{i}, \boldsymbol\theta_\text{rnn})$. It is assumed that at each interval $[t_{i-1}, t_i]$ the pendulum equation is fulfilled:
\[
\begin{cases}
\frac{d}{dt}a(t) = v(t),\\
\frac{d}{dt}v(t) = -\theta\sin a(t), \quad \theta > 0,\\
a(t_{i-1}) = a_{i-1} + \xi_{i-1}, ~v(t_{i-1}) = v_{i-1}.
\end{cases} \quad 2 \leq i \leq \ell
\]
Let $\boldsymbol\theta = [\theta, \boldsymbol\theta_\text{rnn}]$. Given a loss function $\mathcal{L}(\boldsymbol{\theta}) = \sum_{i=2}^\ell\xi_i^2(\boldsymbol\theta)$. Optimal parameters $\boldsymbol\theta$ are solution of the following optimization problem:
\[
\boldsymbol{\theta}^* = \arg\min_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta}).
\]

\subsection{Problem solution}
Consider basic algorithm ODE-RNN \cite{NEURIPS2019_42a6845a}:
\begin{algorithm}[H]
\begin{algorithmic}
\caption{ODE-RNN}
\REQUIRE Data points $\{(x_i, t_i)\}_{i=1}^N$.
\STATE Initialize $\mathbf{h}_0$.
\FOR{$i = 1, \ldots, N$}
\STATE $\mathbf{h}_i' = \mathrm{ODESolve}(f_\theta, \mathbf{h}_{i-1}, (t_{i-1}, t_i))$.
\STATE $\mathbf{h}_i = \mathrm{RNNCell}(\mathbf{h}_i', x_i)$.
\ENDFOR
\STATE For each $i$ compute outputs $o_i = \mathrm{OutputNN}(\mathbf{h}_i)$.
\RETURN $\{o_i\}_{i=1}^N$
\end{algorithmic}
\end{algorithm}

We introduce a modification of ODE-RNN. The difference is that we map hidden state to the space (from $\mathbb{R}^\text{hidden}$) of the ODE solution ($\mathbb{R}^2$) and back.
\begin{algorithm}[H]
\begin{algorithmic}
\caption{ODE-RNN with modifications}
\REQUIRE Data points $\{(x_i, t_i)\}_{i=1}^N$.
\STATE Initialize $\mathbf{h}_0$.
\FOR{$i = 1, \ldots, N$}
\STATE $\mathbf{h}_i' = \mathrm{ODESolve}(f_\theta, \textcolor{red}{\mathrm{To2d}}(\mathbf{h}_{i-1}), (t_{i-1}, t_i))$.
\STATE $\mathbf{h}_i = \mathrm{RNNCell}(\textcolor{red}{\mathrm{ToHidden}}(\mathbf{h}_i'), x_i)$.
\ENDFOR
\STATE For each $i$ compute outputs $o_i = \mathrm{OutputNN}(\mathbf{h}_i)$.
\RETURN $\{o_i\}_{i=1}^N$
\end{algorithmic}
\end{algorithm}

We compute gradients of the loss function $\mathcal{L}(\boldsymbol{\theta})$ w.r.t. $\boldsymbol\theta$ using backpropagation. Then we perform optimizing step. After some iterations we get a solution.

\subsection{Code analysis}

The computational experiment can be found on GitHub repository\footnote{https://github.com/Konstantin-Iakovlev/MathMethodsOfForecasting}.

\subsection{Computational experiment}

The goal of the computational experiment is to compare the prediction quality of the proposed method with other methods. The data was taken from WISDM dataset \cite{kwapisz2011activity}. We considered an accelerometer measurements. The number of train timestamps is 60. The number of validation timestamps is 20. Hidden size of LSTM is 20. Output network, networks that map to hidden space and back are one layer dense networks.

First, we ran ODE-RNN with modifications with $f_\theta$ as a right hand side of a pendulum equation. We used Adam with learning rate $5\cdot 10^{-3}$ for optimization. We ran 500 epochs of optimization. Validation loss is 0.6090.

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/pendulum.png}
\caption{Forecast of ODE-RNN with modifications, pendulum equation.}
\end{figure}

After that we ran ODE-RNN with modifications with $f_\theta$ as a two fully connected layers with Tanh activation with hidden size 2. We also ran 500 epochs with Adam optimizer. The validation loss is 1.3758.

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/dense.png}
\caption{Forecast of ODE-RNN with modifications,  dense network.}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/pendulum_train.png}
\caption{Train loss, pendulum equation.}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/dense_train.png}
\caption{Train loss, dense network.}
\end{figure}

During our experiments, we noticed that the learning process of the network in the case of a pendulum is more stable and reach better minimum. It seems to be that the using of the pendulum equation's right hand side as a dynamic of an ODE is a powerful regularizer.
%\begin{table}
%\caption{Dependence of $Q^2$ on $R$ on validation dataset}
%\label{table}
%\begin{tabular}{c|c|c|c|c|c|c|c|}
%Method & $R= 1$ & $R= 4$ & $R = 10$ & $R = 13$ & $R = 16$ & $R = 19$\\ \hline
%HOPLS  &0.41 $\pm$ 0.12 & 0.51 $\pm$ 0.19&  0.41 $\pm$ 0.12& 0.04 $\pm$ 0.47 & -0.22 $\pm$ 0.68&  -0.32 $\pm$ 0.68 \\
%N-PLS & 0.42 $\pm$ 0.15 & 0.45 $\pm$ 0.11 & 0.35 $\pm$ 0.18 & 0.35 $\pm$ 0.19 & 0.33 $\pm$ 0.20 & 0.30 $\pm$ 0.20
%\end{tabular}
%\end{table}

\end{document}


\documentclass[11pt]{article}

\usepackage[headings]{fullpage}
\usepackage[utopia]{mathdesign}
\usepackage{color}
\usepackage{graphicx}

\pagestyle{myheadings}
\markboth{Magic}{Magic}

\input{../../fncextra}

\begin{document}

\begin{center}
  \bf Do you believe in magic?
\end{center}

Magic squares have fascinated humanity for centuries. In our
terminology, a magic square is an $n\times n$ matrix whose entries are
the integers from 1 to $n^2$, arranged so that the sum of every row,
every column, and the two long diagonals is the magic number
$n(n^2+1)/2$.

A famous example is Durer's magic square,
\[
\begin{bmatrix}
  16 & 3 & 2 & 13 \\ 5 & 10 & 11 & 8 \\ 9 & 6 & 7 & 12 \\ 4 & 15 & 14 & 1
\end{bmatrix}
\]
One of the coolest facts about this magic square is that it contains the numbers 15 and 14 next to each other, and in 1514 he made an engraving about it:
\begin{center}
  \includegraphics[height=2in]{durer}
\end{center}
It has a couple more tricks up its sleeve as well.

We can check the magic square properties with a little linear algebra. First consider the inner product
\begin{equation}
	\label{ones}
\begin{bmatrix}
a & b & c & d
\end{bmatrix} \begin{bmatrix}
1 \\ 1 \\ 1 \\ 1
\end{bmatrix}
= a + b + c + d,
\end{equation}
allowing us to sum the entries of a vector. We can do something similar with the square matrix $M$, summing all the rows simultaneously:
\begin{equation}
	\label{Mones}
	M \begin{bmatrix}
		1 \\ 1 \\ 1 \\ 1
	\end{bmatrix}
	= 1 \bm{m}_1 + 1 \bm{m}_2 + 1 \bm{m}_3 + 1 \bm{m}_4.
\end{equation}
The result is a column vector, each entry of which is the sum of elements in its row. To sum columns instead, we can first transpose $M$ so that the columns become rows. Finally, note that if $\mA$ is $n\times n$ and $\bfx$ is a column vector of $n$ ones, then we can do two multiplications and get
\begin{equation}
	\label{sumall}
	\bfx^T \mA \bfx = \bfx^T ( \bfa_1 + \cdots + \bfa_n) = \sum_{i=1}^n \sum_{j=1}^n A_{ij}.
\end{equation}

\subsection*{Preparation}

Read section 2.2, and the online help for the \texttt{diag} command.

\subsection*{Goals}

You will use MATLAB commands to manipulate Durer's magic square and check its special properties. While you could do all the operations by summing in loops, we'll use linear algebra techniques that make vectors and matrices the central focus.

\subsection*{Required submission elements}

Download the template script and complete it to do the following. You should use the techniques described in the introduction.

\begin{enumerate}
\item Let \texttt{M} be Durer's magic square as given above.
\item Use the matrix-vector multiplication in~\eqref{Mones} to compute the row sums of $M$.
\item Use a transpose and~\eqref{Mones} to compute the column sums of $M$.
\item There are four $2\times 2$ submatrices made by drawing lines
  horizontally and vertically through the center of the matrix. Using the identity~\eqref{sumall}, verify that the sum of the elements of each of these submatrices is also the magic number.
\item There is another $2\times 2$ submatrix made by the four interior elements. Verify that the sum of these elements is---wait for it---the magic number.
\item Find the sum of the main diagonal of \texttt{M} using \texttt{diag} and~\eqref{ones}. Then do the same for the sum on the ``antidiagonal.'' (Hint: How can you rearrange $M$ so that the antidiagonal becomes the diagonal?)
\end{enumerate}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\section{Monte Carlo Methods}
Monte Carlo methods learn state and action values by sampling and averaging returns (i.e. not from dynamics like DP). These methods learn from experience (real or simulated) and require no prior knowledge of the environments dynamics.\\

Monte Carlo methods thus require well defined returns, so we will consider them only for episodic tasks. Only on completion of an episode do values and policies change.\\

We still use the generalised policy iteration framework, but we adapt it so that we learn the value function from experience rather than compute it \emph{a priori}.

\subsection{Monte Carlo Prediction}
The idea is to average the returns following each state to get an estimate of the state value
\[
    v_\pi(s) = \Epi{} [G_{t+1} | S_t =s].
\]
Given enough observations, the sample average converges to the true state value under the policy $\pi$.\\

Given a policy $\pi$ and a set of episodes, here are two ways in which we might estimate state values
\begin{itemize}
    \item \emph{First Visit MC} average returns from first visit to state $s$ in order to estimate $v_\pi(s)$
    \item \emph{Every Visit MC} average returns following every visit to state $s$.
\end{itemize}

First visit MC generates iid estimates of $v_\pi(s)$ with finite variance, so the sequence of estimates converges to the expected value by the law of large numbers as visits to $s$ tend to $\infty$. Every visit MC does not generate independent estimates, but still converges.\\

An algorithm for first visit MS (what we will focus on) is below. Every visit is the same, just without the check for $S_k$ occurring earlier in the episode.\\

\includegraphics[width=\textwidth]{\ProjectDir/data/notes_images/first_visit_mc_algo.png} \\

Monte Carlo methods are often used even when the dynamics of the environment are knowable, e.g. in Blackjack. It is often much easier to create sample games than it is to calculate environment dynamics directly.\\

MC estimates for different states are independent (unlike bootstrapping in DP). This means that we can use MC to calculate the value function for a subset of the states, rather than the whole state space as with DP. Along with the ability to learn from experience and simulation, this is the another advantage that MC has over DP.

\subsection{Monte Carlo Estimation of Action Values}
If we don't have a model for the environment, then it is more useful to estimate action-values. With a model we can use state values to find a policy by searching possible actions, as with DP (value iteration, etc.). We can't do this without knowledge of the dynamics, so one of the primary goals of MC is to estimate $q_*$. We start with policy evaluation for action-values.

\subsubsection*{Policy Evaluation for Action-Values}
The policy evaluation problem for action-values is to estimate $q_\pi(s, a)$ for some $\pi$. This is essentially the same as for state values, only we now talk about state-action pairs being visited, i.e. taking action $a$ in state $s$, rather than just states being visited.\\

If $\pi$ is deterministic, then we will only estimate the values of actions that $\pi$ dictates. We therefore need to incorporate some exploration in order to have useful action-values (since, after all, we want to use them to make informed decisions).\\

One consideration is to make $\pi$ stochastic, e.g. $\varepsilon$-soft. Another is the assumption of \emph{exploring starts}, which specifies that ever state-action pair has non-zero probability of being selected as the starting state. Of course, this is not always posible in practice.\\

For now we assume exploring start. Later we will come back to the issue of \emph{maintaining exploration}

\subsection{Monte Carlo Control}
We make use of the GPI framework for action-values. Policy evaluation is done as described. Policy improvement is done by making the policy greedy with respect to the action-value function, so no model is needed for this step
\[
    \pi(s) \doteq \argmax_a q(s, a).
\]

We generate a sequence of policies $\pi_k$ each greedy with respect to $q_{\pi_{k-1}}(s, a)$. The policy improvement theorem applies: for all $s \in \mathcal{S}$
\begin{align*}
    q_{\pi_k}(s, a=\pi_{k+1}(s)) &= q_{\pi_k}(s, \argmax_a q_\pi(s, a)) \\
                    &= \max_a q_{\pi_k}(s, a) \\
                    &\geq q_{\pi_k}(s, \pi_k(s))\\
                    &= v_{\pi_k}(s)
\end{align*}

So $\pi_{k+1}$ uniformly better than $\pi_k$ or it is optimal. \\

The above procedure's convergence depends on assumptions of exploring starts and infinitely many episodes. We will relax the first later, but we will address the second now.\\

Two approaches to avoid infinitely many episodes:
\begin{enumerate}
    \item Stop the algorithm once the $q_{\pi_k}$ stop moving within a certain error. (In practice this is only useful on the smallest problems.)
    \item Stop policy evaluation after a certain number of episodes, moving the action value towards $q_{\pi_k}$, then go to policy improvement.
\end{enumerate}

For MC policy evaluation, it is natural to alternate policy evaluation and improvement on a episode by episode basis. We give such an algorithm below (with the assumption of exploring starts).\\

\includegraphics[width=\textwidth]{\ProjectDir/data/notes_images/mc_policy_iteration_exploring_starts.png}\\

It is easy to see that optimal policies are a fixed point of this algorithm. Whether this algorithm converges in general is still, however, an open question.

\subsection{Monte Carlo Control without Exploring Starts}
\subsubsection*{On Policy vs. Off Policy}
On-policy methods evaluate or improve the policy that is used to make decisions, whereas off-policy methods evaluate or improve one that is different than the one used to generate the data.

\subsubsection*{On-Policy Techniques without Exploring Starts}
We consider $\varepsilon$-greedy policies that put probability $1 - \varepsilon + \frac{\varepsilon}{|\mathcal{A}(s)|}$ on the maximal action and $\frac{\varepsilon}{|\mathcal{A}(s)|}$ on each of the others. These are examples of $\varepsilon$-soft policies in which $\pi(a|s) \geq \frac{\varepsilon}{|\mathcal{A}(s)|}$.\\

We use this idea in the GPI framework:\\

\includegraphics[width=\textwidth]{\ProjectDir/data/notes_images/on_policy_mc_algo.png}\\

We now show that an $\varepsilon$-greedy policy with respect to $q_\pi$, $\pi'$, is an improvement over any $\varepsilon$-soft policy $\pi$. For any $s \in \mathcal{S}$
\begin{align}
    q_\pi(s, \pi'(s)) &= \sum_a \pi'(a|s) q_\pi(s, a) \\
                      &= \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a q_\pi(s, a) + (1 - \varepsilon)\max_a q_\pi(s, a) \\
                      &\geq \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a q_\pi(s, a) + (1 - \varepsilon)\sum_a\frac{\pi(a|s) - \frac{\varepsilon}{|\mathcal{A}(s)|}}{1 - \varepsilon} q_\pi(s, a) \\
                      &= \sum_a \pi(a|s) q_\pi(s, a)\\
                      &= v_\pi(s)
\end{align}
(where line 3 follows because a weighted average with weights $w_i \geq 0$ and $\sum_i w_i = 1$ is $\leq$ the max term).\\

This satisfies the condition of the policy improvement theorem so we now know that $\pi' \geq \pi$.\\

Previously, with deterministic greedy policies, we would get automatically that fixed points of policy iteration are optimal policies since
\[
    v_*(s) \doteq \max_\pi v_\pi(s) \quad \forall s \in \mathcal{S}.
\]
Now our policies are not deterministically greedy, our value updates do not take this form. We note, however, that we can consider an equivalent problem where we change the environment to select state and reward transitions at random with probability $\varepsilon$ and do what our agent asks with probability $1 - \varepsilon$. We have moved the stochasticity of the policy into the environment, creating an equivalent problem. The optimal value function in the new problem satisfies its Bellman equation
\begin{align}
    \tilde{v}_\pi(s) &= (1 - \varepsilon) \max_a \tilde{q}_\pi(s, a) + \frac{\varepsilon}{|\mathcal{A}(s)|}\sum_a \tilde{q}_\pi(s, a) \\
                     &= (1 - \varepsilon) \max_a \sum_{s', r} p(s', r|s, a)[r + \gamma \tilde{v}_\pi(s')] + \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a \sum_{s', r} p(s', r|s, a)[r + \gamma \tilde{v}_\pi(s')].
\end{align}
We also know that at fixed points of our algorithm
\begin{align}
    v_\pi(s) &= (1 - \varepsilon) \max_a q_\pi(s, a) + \frac{\varepsilon}{|\mathcal{A}(s)|}\sum_a q_\pi(s, a) \\
                     &= (1 - \varepsilon) \max_a \sum_{s', r} p(s', r|s, a)[r + \gamma v_\pi(s')] + \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a \sum_{s', r} p(s', r|s, a)[r + \gamma v_\pi(s')].
\end{align}
This is the same equation as above, so by uniqueness of solutions to the Bellman equation we have that $v_\pi = \tilde{v}_\pi$ and so $\pi$ is optimal.

\subsection{Off-Policy Prediction via Importance Sampling}
Off-policy learning uses information gained by sampling the \emph{behaviour policy} $b$ to learn the \emph{target policy} $\pi$. The behaviour policy explores the environment for us during training and we update the target policy accordingly.\\

In this section we consider the prediction problem: estimating $v_\pi$ or $q_\pi$ for a fixed and known $\pi$ using returns from $b$. In order to do this we need the assumption of coverage:
\begin{equation}
    \pi(a|s) \geq 0 \implies b(a|s) \geq 0.
\end{equation}
This implies that $b$ must be stochastic wherever it is not identical to $\pi$. The target policy $\pi$ may itself be deterministic, e.g. greedy with respect to action-value estimates.

\subsubsection*{Importance Sampling}
We use \emph{importance sampling} to evaluate expected returns from $\pi$ given returns from $b$.\\

Define the importance sampling ratio as the relative probability of a certain trajectory from $S_t$
\begin{align}
    \rho_{t:T-1} &= \frac{\P{}(A_t, S_{t+1}, A_{t+1}, \dots)| S_t, A_{t:T-1} \sim \pi}{\P{}(A_t, S_{t+1}, A_{t+1}, \dots)| S_t, A_{t:T-1} \sim b} \\
                 &= \frac{\prod_{k=t}^{T-1}\pi(A_k|S_k)\P{}(S_{k+1}|S_k, A_k)}{\prod_{k=t}^{T-1}b(A_k|S_k)\P{}(S_{k+1}|S_k, A_k)}\\
                 &=\prod_{k=t}^{T-1}\frac{\pi(A_k|S_k)}{b(A_k|S_k)}
\end{align}
where the state transition dynamics $\P{}$ cancel out.\\

If we have returns $G_t$ from evaluating policy $b$, so $v_b(s) = \E[G_t|S_t=s]$, then we can calculate
\[
    v_\pi(s) = \E[\rho_{t:T-1}G_t|S_t=s]
\]

\subsubsection*{Estimation}
Introduce new notation:
\begin{itemize}
    \item Label all time steps in a single scheme. So maybe episode 1 is $t=1, \dots, 100$ and episode 2 is $t = 101, \dots, 200$, etc.
    \item Denote the set times of first/every visit to $s$ by $\mathcal{T}(s)$ (spanning episodes).
    \item Let $T(t)$ be the first termination after $t$
    \item Let $G_t$ be the returns from $t$ to $T(t)$
\end{itemize}

We can now give two methods of values for $\pi$ from returns from $b$:\\
{\bfseries{Ordinary Importance Sampling}}
\begin{equation}
    V(s) \doteq \frac{\sum_{t\in \mathcal{T}(s)} \rho_{t:T-1}G_t}{|\mathcal{T}(s)|}
\end{equation}
{\bfseries{Weighted Importance Sampling}}
\begin{equation}
    V(s) \doteq \frac{\sum_{t\in \mathcal{T}(s)} \rho_{t:T-1}G_t}{\sum_{t\in \mathcal{T}(s)} \rho_{t:T-1}}
\end{equation}
or 0 if the denominator is 0.\\

Weighted importance sampling is biased (e.g. it's expectation is $v_b(s)$ after 1 episode) but has bounded variance. The ordinary importance sampling ratio is unbiased, but has possibly infinite variance, because the variance of the importance sampling ratios themselves is unbounded.\\

Assuming bounded returns, the variance of the weighted importance sampling estimator converges to 0 even if the variance of the importance sampling ratios is infinite. In practice, this estimator usually has dramatically lower variance and is strongly preferred.

\subsection{Incremental Implementation}
We look for incremental calculations of the averages that make up the estimates, as in Chapter 2.\\

For on-policy methods the incremental averaging is the same as in Chapter 2. For off-policy methods, but with ordinary importance sampling, we only need to multiply the returns by the importance sampling ratio and then we can average as before.\\

We will now consider weighted importance sampling. We have a sequence of returns $G_i$, all starting in the same state $s$ and each with a random weight $W_i$ (e.g. $W_i = \rho_{i:T(i)-1}$). We want to iteratively calculate (for $n \geq 2$)
\[
    V_n = \frac{\sum_{k=1}^{n-1}W_kG_k}{\sum_{k=1}^{n-1}W_k}.
\]
We can do this with the following update rules
\begin{align}
    V_{n+1} &= V_n + \frac{W_n}{C_n}[G_n - V_n]\\
    C_{n+1} &= C_n + W_{n+1}
\end{align}
where $C_0 = 0$ and $V_1$ is arbitrary (notice that it cancels out as $V_2 = G_1$).\\

Below is an algorithm for off-policy weighted importance sampling (set $b=\pi$ for on policy). The estimator $Q$ converges to $q_\pi$ for all encountered state-action pairs.\\

\includegraphics[width=\textwidth]{\ProjectDir/data/notes_images/off_policy_mc_prediction_algo.png}\\

\subsection{Off-Policy Monte Carlo Control}
Below is an algorithm for estimating $\pi_*$ and $q_*$ in the GPI framework. The target policy $\pi$ is the greedy policy with respect to $Q$, which is an estimate of $q_\pi$. This algorithm converges to $q_\pi$ as long as an infinite number of returns are observed for each state-action pair. This can be achieved by making $b$ $\varepsilon$-soft. The policy $\pi$ converges to $\pi_*$ at all encountered states even if $b$ changes (to another $\varepsilon$-soft policy) between or within episodes.\\

\includegraphics[width=\textwidth]{\ProjectDir/data/notes_images/off_policy_mc_control_algo.png}\\

Notice that this policy only learns from episodes in which $b$ selects only greedy actions after some timestep. This can greatly slow learning.

\subsection{*Discounting Aware Importance Sampling}
We present a method of importance sampling that recognises the return as a discounted sum of rewards. This can help in estimation, since if an episode is of length 100 and $\gamma = 0$ then the final 99 terms of the importance sampling ration contribute nothing to the expected value of our estimator (they have expected value of 1) but can greatly increase its variance. We therefore construct a method of importance sampling that takes into account discounting.\\

Introduce the \emph{flat partial returns}
\[
    \bar{G}_{t:h} \doteq \sum_{i=t+1}^h R_{i} \quad 0 \leq t \leq h \leq T
\]
then it can be shown (by rearranging) that
\begin{align}
    G_t &\doteq \gamma^{i-t}R_{i+1}\\
        &= (1 - \gamma)\sum_{h=t+1}^{T-1}\gamma^{h-t-1}\bar{G}_{t:h} + \gamma^{T-t-1}\bar{G}_{t:T}.
\end{align}
Now we can scale each flat partial return by a truncated importance sampling ratio (hence reducing variance).\\

{\bfseries{Ordinary Importance Sampling Ratio}}
\begin{equation}
    V(s) \doteq \frac{\sum_{t\in \mathcal{T}(s)} \left[ (1 - \gamma) \sum_{h=t+1}^{T(t-1)} \gamma^{h - t - 1}\rho_{t:h-1}\bar{G}_{t:h} + \gamma^{T(t) - t - 1}\rho_{t:T(t) - 1} \bar{G}_{t:T(t)} \right]}{|\mathcal{T}(s)|}
\end{equation}

{\bfseries{Weighted Importance Sampling Ratio}}
\begin{equation}
    V(s) \doteq \frac{\sum_{t\in \mathcal{T}(s)} \left[ (1 - \gamma) \sum_{h=t+1}^{T(t-1)} \gamma^{h - t - 1}\rho_{t:h-1}\bar{G}_{t:h} + \gamma^{T(t) - t - 1}\rho_{t:T(t) - 1} \bar{G}_{t:T(t)} \right]}{\sum_{t\in \mathcal{T}(s)} \left[ (1 - \gamma) \sum_{h=t+1}^{T(t-1)} \gamma^{h - t - 1}\rho_{t:h-1} + \gamma^{T(t) - t - 1}\rho_{t:T(t) - 1} \right]}
\end{equation}

\subsection{*Per-Decision Importance Sampling}
There is another way in which we may be able to reduce variance in off-policy importance sapling, even in the absence of discounting ($\gamma = 1$). Notice that the off-policy estimators are made up of terms like
\[
    \rho_{t:T-1}G_t = \rho_{t:T-1} (R_{t+1} + \gamma R_{t+2} + \dots+ \gamma^{T-t-1}R_{T})
\]
and that each of these terms is of the form
\[
    \rho_{t:T-1}R_{t+1} = \frac{\pi(A_t|S_t)}{b(A_t|S_t)}\dots\frac{\pi(A_{T-1}|S_{T-1})}{b(A_{T-1}|S_{T-1})}R_{t+1}.
\]
Now notice that only the first and last terms here are correlated, while all the others have expected value 1 (taken with respect to $b$). Clearly this is also the case at each $t$. This means that
\[
    \E{}[\rho_{t:T-1}R_{t+k}] = \E{}[\rho_{t:t+k-1}R_{t+k}]
\]
therefore
\[
    \E{}[\rho{t:T-1}G_t] = \E{}[\tilde{G}_t]
\]
where
\[
    \tilde{G}_t \doteq \sum_{i=t}^{T-1}\gamma^{i-t}\rho_{t:i}R_{i+1}.
\]
Now we can write the ordinary importance sampling estimator as
\[
    V(s) \doteq \frac{\sum_{t\in\mathcal{T}(s)} \tilde{G}_t}{|\mathcal{T}(s)|}
\]
possibly reducing variance in the estimator. \\

The weighted importance sampling estimators of this form that have so far been found have been shown to not be consistent (in the statistical sense). We don't know if a consistent weighted average form of this exists.


\appendix
\appendixpage
\section{Terms That Need More Explanation Then A Footnote}
\subsection{Potential} \label{sec:potential}
Potential is the energy change that occurs when the position of an object changes \cite{potential}. There are many potentials, like electric potential, gravitational potential and elastic
potential. Let me explain the concept with an example. Say you are walking on a set of stairs in the upwards direction. As your muscles move to bring you one step upwards, energy that is used
by your muscles is converted into gravitational potential. Now imagine you turn around and go downwards instead. Notice how that is easier? That is due to the gravitational potential being
converted back into energy so your muscles have to deliver less energy to get you down. The potential is usually tied to a force, like the gravitational force.

\subsection{Asymptotic Runtime} \label{sec:runtime}
Asymptotic runtime is what we use in computer science to indicate how fast an algorithm works. We do it this way because concrete time indications (seconds, minutes, hours) are very machine
dependent. It matters a lot if your CPU, RAM and GPU are fast or not for the runtime. Therefore, we needed something to compare algorithms by which is machine independent. That is what asymptotic
runtime is. We have 3 notations for asymptotic runtime, $\Omega$ which is the lower bound of the runtime: not faster than; $O$ which is the upperbound of the runtime: not slower than; and we
have $\Theta$ which is the tight bound: not slower but also not faster than. After these 3 notations we usually denote the runtime in algebraic letters which stand for the input size. $O(n)$ for
instance means that for an input of size $n$ the algorithm will not run slower than $n$ operations. Whereas $\Omega(n^3)$ means that the algorithm needs for an input of size $n$ at least $n^3$
operations. Now this is not an exact match, as there are constants and other terms in the real runtime, but for asymptotic runtime we look at the most dominant factor, as that outgrows all the
other factors if the input size increases. You can compare this by plotting the functions $y = x$ and $z = x^2$ on a graphical calculator. No matter which constant $a$ you put in front of the $x$,
$z = x^2$ will at some point (note we don't specify when or where) be larger than $y = ax$. What you need to remember for all this is that polynomials are faster than exponentials ($n^2 < 2^n$)
and logarithms are faster than polynomials ($\log(n) < n$). $n!$ is very slow, $\log(n)$ is very fast.

\subsection{Complex Numbers} \label{sec:complex}
As you all know in the real numbers ($\mathbb{R}$) negative roots are not allowed as they do not exist. But what would happen if we would allow them to exist? Then we move into the area of
complex numbers. A complex number consists out of two parts, a real part and an imaginary part in the form $a + bi$ where $i = \sqrt{-1}$. Complex numbers have all kinds of properties, but what
we need them for are rotations. This is captured in Euler's formula $e^{it} = \cos(x) + i\sin(x)$ \cite{eulerFormula}. Which means that for time $t$ we rotate around the origin (of the complex
plane) forming a circle with radius one (the unit circle). Now if you would set $t = \pi$ then the result is $0$. What this means is that we have come full circle (hah) when $t = 2\pi$.


\section{The Argument Principle, Local Degree and Rouch\'e's Theorem}
\begin{definition}
    Let $D$ be a domain, we say a closed curve $\gamma:[0,1]\to\mathbb C$ bounds $D$ if $I(\gamma;w)=1$ for any $w\in D$ and $I(\gamma;w)=0$ for any $w\notin D\cup\gamma([0,1])$.
\end{definition}
Note that the orientation matters here.
$D$ is indeed bounded since $\gamma([0,1])$ is contained in $D_R(0)$ for a large enough $R$, but $I(\gamma;w)=0$ for any $w\notin D_R(0)$, so $D\subset D_R(0)$, hence is bounded.
\begin{theorem}[The Argument Principle]
    Let $\gamma$ be a closed curve bounding a domain $D$ and suppose that $f$ is meromorphic in some open set $U\subset \bar{D}\cup\gamma([0,1])$ such that $f$ has no pole or zero on $\gamma([0,1])$.
    If $f$ has precisely $N$ zeros and $P$ poles in $D$ (both counting with multiplicity), then
    $$N-P=\frac{1}{2\pi i}\int_\gamma\frac{f^\prime(z)}{f(z)}\,\mathrm dz=I(\Gamma;0)$$
    where $\Gamma=f\circ\gamma$.
\end{theorem}
\begin{proof}
    $N$,$P$ are finite since $D$ is bounded and zeros and poles are isolated points.
    Also note that $0\notin\Gamma([0,1])$ since $f\circ\gamma$ is never $0$.
    So
    $$I(\Gamma,0)=\frac{1}{2\pi i}\int_\Gamma\frac{1}{z}\,\mathrm dz=\frac{1}{2\pi i}\int_0^1\frac{1}{f(\gamma(t))}f^\prime(\gamma(t))\gamma^\prime(t)\,\mathrm dt=\frac{1}{2\pi i}\int_\gamma\frac{f^\prime(z)}{f(z)}\,\mathrm dz$$
    which is the second equality.\\
    As for the other part, note that if $a$ is neither a pole or a zero at $f$, then $f^\prime/f$ is holomorphic at $a$.
    Also, if $a$ is a zero (or a pole) of order $k$ at $z=a$, then $f^\prime/f$ has a simple pole at $z=a$ with $\operatorname{Res}_{f^\prime/f}(a)=k$ (or $-k$ respectively).
    Now $I(\gamma,w)=0$ for any $w\in D$ by hypothesis, so applying the residue theorem to $f^\prime/f$ shall give the result.
\end{proof}
\begin{definition}
    Let $f:D_R(a)\to\mathbb C$ be holomorphic and nonconstant, then the local degree of $f$ at $a$, denoted $\deg_f(a)$, is the order of zero of $f(z)-f(a)$ at $0$.
\end{definition}
Note that the local degree has to be positive.
\begin{theorem}[Local Degree Theorem]
    Consider $f:D_R(a)\to\mathbb C$ nonconstant and holomorphic.
    Suppose $\deg_f(a)=d>0$, then for every sufficiently small $r>0$, there is $\epsilon>0$ such that for any $w$ with $0<|w-f(a)|<\epsilon$, the equation $f(z)=w$ has precisely $d$ roots in $D(a,r)\setminus\{a\}$ that are all distinct.
\end{theorem}
\begin{proof}
    Since $f$ is nonconstant, by the principle of isolated zeros, we can choose $r>0$ such that $f(z)-f(a)\neq 0$ and $f^\prime(z)\neq 0$ for any $z\in B_r(a)\setminus\{a\}$.
    Now take $\gamma(t)=a+re^{it}$ for $t\in [0,2\pi]$, then $f(\gamma(t))\neq f(a)$ for any $t$, so $\Gamma=f\circ\gamma$ misses $f(a)$, so we can choose $\epsilon$ such that $\Gamma$ never enters $D_\epsilon(f(a))$.
    Then for $w\in D_\epsilon(f(a))\setminus\{f(a)\}$, by the argument principle, the number of zeros counting multiplicity of $f(z)-w$ in $D_r(a)$ is $I(\Gamma;w)=I(\Gamma;f(a))=d$.
    These zeros must all be in $D_r(a)\setminus\{a\}$ and none of them has multiplicity more than one as $f^\prime(a)\neq 0$.
\end{proof}
\begin{corollary}
    A nonconstant holomorphic function on a domain is an open map.
\end{corollary}
\begin{proof}
    Follows directly.
\end{proof}
\begin{theorem}[Rouch\'e's Theorem]
    Let $\gamma:[0,1]\to\mathbb C$ be a curve that bounds a domain $D$ and $f,g$ holomorphic on a open set $U$ containing $\bar D\cup\gamma([0,1])$.
    If $|f|>|g|$, then $f,f+g$ have the same number of zeros on $D$.
\end{theorem}
\begin{proof}
    $|f|>|g|$ hence $f,f+g$ are nowhere zero on $\gamma$, so we apply the argument principle on $h=(f+g)/f=1+g/f$.
    We have $I(h\circ\gamma;0)=0$, so $h$ has $N=P$, so $f+g,f$ has the same number of zeros.
\end{proof}