\chapter{Applications and Case Studies}
This chapter contains a number of case studies designed to deepen our understanding of \textsl{Python}.

\section{Solving Equations via Fixed-Point Algorithms}
\blue{Fixed-Point iterations} are very important, both in computer science and in mathematics.  As a first
example, we show how to solve an equation via a fixed point iteration.  Suppose we want to solve the equation
\\[0.2cm]
\hspace*{1.3cm} $x = \cos(x)$. \\[0.2cm]
Here, $x$ is a real number that we seek to compute.  Figure \ref{fig:xEqualsCosX.pdf} on page
\pageref{fig:xEqualsCosX.pdf} shows the graphs of the two functions
\\[0.2cm]
\hspace*{1.3cm}
$y = x$  \quad and \quad $y = \cos(x)$.
\\[0.2cm]
Since the graphs of these functions intersect, it is obvious that there exists a value $x$ such that $x = \cos(x)$.
Furthermore, from Figure \ref{fig:xEqualsCosX.pdf} it is obvious that this value of $x$ is bigger than $0.6$
and less than $0.8$.

\begin{figure}[!ht]
  \hspace*{-3.0cm}
  \epsfig{file=Figures/xEqualsCosX.pdf,scale=0.6}

  \caption{The functions $y = x$ and $y = cos(x)$.}
  \label{fig:xEqualsCosX.pdf}
\end{figure}

A simple approach that lets us compute the exact value of $x$ is to use a
\href{https://en.wikipedia.org/wiki/Fixed-point_iteration}{fixed-point iteration}.  To this end, we
define the sequence $\bigl(x_n\bigr)_{n\in\mathbb{N}}$ inductively as follows:
\\[0.2cm]
\hspace*{1.3cm}
$x_0 = 0$ \quad and \quad $x_{n+1} = \mathtt{cos}(x_n)$ \quad for all $n \in \mathbb{N}$.
\\[0.2cm]
With the help of the
\href{https://en.wikipedia.org/wiki/Banach_fixed-point_theorem}{Banach fixed-point theorem}\footnote{
  The Banach fixed-point theorem is discussed in the lecture on
  \href{https://en.wikipedia.org/wiki/Differential_calculus}{differential calculus}.  This lecture is part of the
  second semester.
}
it can be shown that this sequence converges to a solution of the equation $x = \cos(x)$, i.e.~if we define
\\[0.2cm]
\hspace*{1.3cm}
$\bar{x} = \lim\limits_{n\rightarrow\infty} x_n$,
\\[0.2cm]
then we have
\\[0.2cm]
\hspace*{1.3cm}
$\cos\bigl(\bar{x}\bigr) = \bar{x}$.
\\[0.2cm]
Figure \ref{fig:solve.py} on page \pageref{fig:solve.py} shows the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/solve.py}{\texttt{solve.py}}
that uses this approach to solve the equation $x = \cos(x)$.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    import math

    x     = 1.0
    old_x = 0.0
    i     = 1
    while abs(x - old_x) >= 4.0E-16:
        old_x = x
        x = math.cos(x)
        print(f'{i} : {x}')
        i += 1
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Solving the equation $x = \cos(x)$ via fixed-point iteration.}  \label{fig:solve.py}
\end{figure} %\$

In this program, the iteration stops as soon as the difference between the variables \texttt{x} and
\texttt{old\_x} is less that $4 \cdot 10^{-16}$.  Here, \texttt{x} corresponds to $x_{n+1}$, while \texttt{old\_x}
corresponds to $x_n$.  Once the values of $x_{n+1}$ and $x_n$ are sufficiently close, the execution of the \texttt{while} loop
terminates.
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/Fixed-Point-Iteration.ipynb}{Fixed-Point-Iteration.ipynb}
shows a \textsl{Jupyter} notebook that implements fixed point iteration.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    from math import cos

    def solve(f, x0):
        """
        Solve the equation f(x) = x using a fixed point iteration.
        x0 is the start value.
        """
        x = x0
        for n in range(10000):  # at most 10000 iterations
            oldX = x;
            x    = f(x);
            if abs(x - oldX) < 1.0e-15:
                return x;

    print("solution to x = cos(x): ", solve(cos, 0));
    print("solution to x = 1/(1+x):", solve(lambda x: 1/(1+x), 0));
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A generic implementation of the fixed-point algorithm.}
\label{fig:fixpoint.py}
\end{figure}

Figure \ref{fig:fixpoint.py} on page \pageref{fig:fixpoint.py} shows the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/fixpoint.py}{\texttt{fixpoint.py}}.
In this program we have implemented a function \texttt{solve} that takes two arguments.
\begin{enumerate}
\item \texttt{f} is a unary function.  The purpose of the \texttt{solve} is to compute the solution of the equation
      \\[0.2cm]
      \hspace*{1.3cm}
      $f(x) = x$.
      \\[0.2cm]
      This equation is solved with the help of a fixed-point algorithm.
\item \texttt{x0} is used as the initial value for the fixed-point iteration.
\end{enumerate}
Line 11 calls \texttt{solve} to compute the solution of the equation $x = \cos(x)$.
Line 12 solves the equation
\\[0.2cm]
\hspace*{1.3cm}
$\ds x = \bruch{1}{1+x}$.
\\[0.2cm]
This equation is equivalent to the quadratic equation $x^2 + x = 1$.  Note that we have defined the function
 $\ds x \mapsto \frac{1}{1+x}$ via the expression
 \\[0.2cm]
\hspace*{1.3cm}
\texttt{lambda x: 1/(1+x)}.
\\[0.2cm]
This expression is called an \blue{anonymous function} since we haven't given a name to the function.

\remarkEng
The function \texttt{solve} is only able to solve the equation $f(x) = x$ if the function $f$ is a
\href{https://en.wikipedia.org/wiki/Contraction_mapping}{contraction mapping}.  A function
$f:\mathbb{R} \rightarrow \mathbb{R}$
is called a \blue{contraction mapping} iff
\\[0.2cm]
\hspace*{1.3cm}
$|f(x) - f(y)| < |x - y|$ \quad for all $x,y \in \mathbb{R}$.
\\[0.2cm]
This notion will be discussed in more detail in the lecture on
\href{https://github.com/karlstroetmann/Analysis/blob/master/Script/analysis.pdf}{analysis} in the second
semester. \eox

\section{Case Study: Computation of Poker Probabilities}
In this short section we are going to show how to compute probabilities for the
\href{https://en.wikipedia.org/wiki/Texas_hold_%27em}{\textsl{Texas Hold'em}} variation of
\href{https://en.wikipedia.org/wiki/Poker}{poker}.   Texas Hold'em poker is played with a deck of 52
cards.  Every card has a \blue{value}.  This value is an element of the set
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{Values} = \{ 2, 3, 4, 5, 6, 7, 8, 9, 10, \textsl{Jack}, \textsl{Queen}, \textsl{King}, \textsl{Ace} \}$.
\\[0.2cm]
Furthermore, every card has a \blue{suit}.  This suit is an element of the set
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{Suits} = \{ \club, \mbox{$\color{red}{\heart}$}, \mbox{$\color{red}{\diamondsuit}$}, \spade \}$.
\\[0.2cm]
These suits are pronounced \blue{club}, \blue{heart}, \blue{diamond}, and \blue{spade}.
As a card is determined by its value and its suit, a card can be represented as a pair $\pair(v,s)$, where $v$
denotes the value while $s$ is the suit of the card.  Hence, the set of all cards can be represented as the set
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{Deck} = \bigl\{ \pair(v,s) \mid v \in \textsl{Values} \wedge \textsl{s} \in \textsl{Suits} \bigr\}$.
\\[0.2cm]
At the start of a game of Texas Hold'em, every player receives two cards.  These two cards are known
as the \blue{preflop} or the \blue{hole}.  Next, there is a \blue{bidding phase} where players can bet on their
cards.   After this bidding phase, the dealer puts three cards open on the table.  These three cards are
known as \blue{flop}.  Let us assume that a player has been dealt the set of cards
\\[0.2cm]
\hspace*{1.3cm}
$\{ \pair(3, \club), \pair(3, \spade) \}$.
\\[0.2cm]
This set of cards is known as a \blue{pocket pair}.  Then the player would like to know the probability
that the flop will contain another card with value $3$, as this would greatly increase her chance of
winning the game.  In order to compute this probability we have to compute the number of possible
flops that contain a card with the value $3$ and we have to divide this number by the number of all
possible flops:
\\[0.2cm]
\hspace*{1.3cm}
$\ds \frac{\;\mbox{number of flops containing a card with value $3$}\;}{\mbox{number of all possible flops}}$
\\[0.2cm]
The program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/poker-triple.py}{poker-triple.py}
shown in Figure \ref{fig:poker-triple.py} performs this computation.  We proceed to discuss this
program line by line.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.0cm,
                  xrightmargin  = 0.0cm,
                ]
    Values = { "2", "3", "4", "5", "6", "7", "8", "9", "T", "J", "Q", "K", "A" }
    Suits  = { "c", "h", "d", "s" }
    Deck   = { (v, s) for v in Values for s in Suits }
    Hole   = { ("3", "c"), ("3", "s") }
    Rest   = Deck - Hole
    Flops  = { (k1, k2, k3) for k1 in Rest for k2 in Rest for k3 in Rest
                            if  len({ k1, k2, k3 }) == 3
             }
    Trips  = { f for f in Flops if ("3", "d") in f or ("3", "h") in f }
    print(len(Trips) / len(Flops))
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing a probability in poker.}
\label{fig:poker-triple.py}
\end{figure}

\begin{enumerate}
\item In line 1 the set \texttt{Values} is defined to be the set of all possible values that a card
      can take.  In defining this set we have made use of the following abbreviations:
      \begin{enumerate}
      \item ``\texttt{T}'' is short for ``\blue{Ten}'',
      \item ``\texttt{J}'' is short for ``\blue{Jack}'',
      \item ``\texttt{Q}'' is short for ``\blue{Queen}'',
      \item ``\texttt{K}'' is short for ``\blue{King}'', and
      \item ``\texttt{A}'' is short for ``\blue{Ace}''.
      \end{enumerate}
\item In line 2 the set \texttt{Suits} represents the possible suits of a card.  Here, we have used
      the following abbreviations:
      \begin{enumerate}
      \item ``\texttt{c}'' is short for $\club$, which is pronounced as \blue{club},
      \item ``\texttt{h}'' is short for \mbox{\color{red}{$\heart$}}, which is pronounced as \blue{heart},
      \item ``\texttt{d}'' is short for \mbox{\color{red}{$\diamondsuit$}}, which is pronounced as \blue{diamond}, and
      \item ``\texttt{s}'' is short for $\spade$, which is pronounced as \blue{spade}.
      \end{enumerate}
\item Line 3 defines the set of all cards.  This set is stored as the variable \texttt{Deck}.  Every
      card is represented as a pair of the form $[v,s]$. Here, $v$ is the value of the card, while $s$ is its suit.
\item Line 4 defines the set \texttt{Hole}.  This set represents the two cards that have been given to our player.
\item The remaining cards are defined as the variable  \texttt{Rest} in line 5.
\item Line 6 computes the set of all possible flops.  Since the order of the cards in the flop does
      not matter, we use sets to represent these flops.  However, we have to take care that the flop
      does contain three \colorbox{amethyst}{different} cards.  Hence, we have to ensure that the three
      cards \texttt{k1}, \texttt{k2}, and \texttt{k3} that make up the flop satisfy the inequalities
      \\[0.2cm]
      \hspace*{1.3cm}
      $\mathtt{k1} \not= \mathtt{k2}$, \quad $\mathtt{k1} \not= \mathtt{k3}$,  \quad and \quad $\mathtt{k2} \not= \mathtt{k3}$.
      \\[0.2cm]
      These inequalities are satisfied if and only if the set
      $\{ \mathtt{k1}, \mathtt{k2}, \mathtt{k3} \}$ contains exactly three elements.  Hence, when
      choosing \texttt{k1}, \texttt{k2}, and \texttt{k3} we have to make sure that the condition
      \\[0.2cm]
      \hspace*{1.3cm}
      $\texttt{len}\bigl({\{ \mathtt{k1}, \mathtt{k2}, \mathtt{k3} \} \;\mathtt{==}\; 3 }\bigr)$
      \\[0.2cm]
      holds.
\item Line 9 computes the subset \texttt{Trips} of those flops that contain at least one card with a value of 3.
      As the 3 of clubs and the 3 of spades have already been dealt to our player, the only cards
      with value 3 that are left in the deck are the 3 of diamonds and the 3 of hearts.  Therefore, we are looking for
      those flops that contain one of these two cards.
\item Finally, the probability for obtaining another card with a value of 3 in the flop is computed as
      the ratio of the number of flops containing a card with a value of 3 to the number of all possible flops.
\end{enumerate}
When we run the program we see that the probability of improving a \blue{pocket pair} on the flop to \blue{trips} or better
is about  $11.8\%$.  A \textsl{Jupyter} notebook showcasing this computation outlined above can be fount at
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/Poker.ipynb}{Poker.ipynb}.

\remarkEng
The method to compute probabilities that has been sketched above only works if the sets that have to
be computed are small enough to be retained in memory.  If this condition is
not satisfied we can use the \href{https://en.wikipedia.org/wiki/Monte_Carlo_method}{\emph{Monte Carlo method}}
to compute the probabilities instead.  This method will be discussed in the lecture on
\href{https://github.com/karlstroetmann/Algorithms/blob/master/Lecture-Notes/algorithms.pdf}{algorithms}.

\section{Finding a Path in a Graph}
In the following section, I will present an application that is more interesting since it is practically
relevant.  In order to prepare for this, we will now discuss the problem of finding a \blue{path} in a
\href{https://en.wikipedia.org/wiki/Directed_graph}{directed graph}.
Abstractly, a graph consists of \blue{vertices} and \blue{edges} that connect these vertices.  In an application, the
vertices could be towns and villages, while the edges would be interpreted as streets connecting these
villages.  To simplify matters, let us assume for now that the vertices are given as natural numbers, while the
edges are represented as pairs of natural numbers.  Then, the graph can be represented as the set of its edges,
as the set of vertices is implicitly given once the edges are known.  To make things concrete, let us consider
an example.  In this case, the set of edges is called \texttt{R} and is defined as follows:
\\[0.2cm]
\hspace*{1.3cm}
$\texttt{R}\; \mathtt{=}\; \bigl\{ \pair(1,2), \pair(2,3), \pair(1,3), \pair(2,4), \pair(4,5) \bigr\}$.
\\[0.2cm]
In this graph, the set of vertices is given as
\\[0.2cm]
\hspace*{1.3cm}
$\{ 1, 2, 3, 4, 5 \}$.
\\[0.2cm]
This graph is shown in Figure \ref{fig:graph0} on page \pageref{fig:graph0}.  You should note that the
connections between vertices that are given in this graph are \blue{unidirectional}:  While there is a connection from
vertex $1$ to vertex $2$, there is no connection from vertex $2$ to vertex $1$.

\begin{figure}[!ht]
  \centering
  \epsfig{file=Figures/graph0,scale=0.6}

  \caption{A simple graph.}
  \label{fig:graph0}
\end{figure}

\noindent
The graph given by the relation \texttt{R} contains only the direct connections of vertices.  For example, in
the graph shown in Figure \ref{fig:graph0}, there is a direct connection from vertex $1$ to vertex $2$ and
another direct connection from vertex $2$ to vertex $4$.  Intuitively, vertex $4$ is reachable from vertex $1$,
since from vertex $1$ we can first reach vertex $2$ and from vertex $2$ we can then reach vertex $4$.  However,
there is is no direct connection between the vertices $1$ and $4$.  To make this more formal, define
a \blue{path} of a graph $R$ as a list of vertices
\\[0.2cm]
\hspace*{1.3cm}
$[x_1, x_2, \cdots, x_n]$ \quad such that \quad $\pair(x_i,x_{i+1}) \in R$ \quad for all $i=1,\cdots,n-1$.
\\[0.2cm]
In this case, the path $[x_1, x_2, \cdots, x_n]$ is written as
\\[0.2cm]
\hspace*{1.3cm}
$x_1 \mapsto x_2 \mapsto \cdots \mapsto x_n$
\\[0.2cm]
and has the \blue{length} $n-1$.  It is important to note that the length of a path
$[x_1,x_2,\cdots,x_n]$ is defined as the number of edges connecting the vertices and not as the
number of vertices appearing in the path.

Furthermore,  two vertices $a$ and $b$ are said to be \blue{connected} iff there exists a path
\\[0.2cm]
\hspace*{1.3cm}
$[x_1,\cdots,x_n]$ \quad such that \quad $a = x_1$ \quad and \quad $b = x_n$.
\\[0.2cm]
The goal of this section is to develop an algorithm that checks whether two vertices $a$ and $b$ are connected.
Furthermore, we want to be able to compute the corresponding path connecting the vertices $a$ and $b$.

\subsection{Computing the Transitive Closure of a Relation}
We have already noted that a graph can be represented as the set of its edges and hence as a \blue{binary relation}.
A \blue{binary relation} is defined as a set of pairs.  We also need the notion of a \blue{relational product}:
If $Q$ and $R$ are binary relations, then the \blue{relational product} $Q \circ R$ of $Q$ and $R$ is defined as
\\[0.2cm]
\hspace*{1.3cm}
$Q \circ R := \bigl\{ \pair(x, z) \bigm| \exists y:(\pair(x,y) \in Q \wedge \pair(y,z) \in R) \bigr\}$.
\\[0.2cm]
Furthermore, for any $n \in \mathbb{N}^*$ we can define the $n$-th power of the relation $R$ by induction.
\begin{enumerate}
\item[B.C.:] $n = 1$.

             $R^1 := R$
\item[I.S.:] $n \mapsto n+1$

             $R^{n+1} := R^n \circ R$.
\end{enumerate}
In order to decide whether there is a path connecting two vertices we have to compute the
\href{https://en.wikipedia.org/wiki/Transitive_closure}{transitive closure} $R^+$ of a relation $R$.
To understand this notion, we first need to define the concept of \blue{transitivity}:  A relation $R$ is
transitive if and only if the following holds:
\\[0.2cm]
\hspace*{1.3cm}
$\pair(x,y) \in T \wedge \pair(y, z) \in T \rightarrow \pair(x, z) \in T$ \quad for all $x,y,z$.
\\[0.2cm]
Now the \blue{transitive closure} $R^+$ of a binary relation $R$ is the smallest relation $T$ such that the following
condition holds:
\begin{itemize}
\item $R$ is a subset of $T$, i.e. we have $R \subseteq T$.
\item $T$ is transitive.
\end{itemize}
The lecture on
\href{https://github.com/karlstroetmann/Lineare-Algebra/blob/master/Script/lineare-algebra.pdf}{Lineare Algebra}
gives a prove that the transitive closure $R^+$ of a binary relation can be computed as follows:
\\[0.2cm]
\hspace*{1.3cm}
$R^+ = \bigcup\limits_{n=1}^{\infty} R^n = R^1 \cup R^2 \cup R^3 \cup \cdots$
\\[0.2cm]
Initially, this formula might look intimidating as it suggests an infinite computation.
Fortunately, it turns out that we do not have to compute all powers of the form $R^n$.  Let me
explain the reason that allows us to cut the computation short.
\begin{enumerate}
\item $R$ is the set of direct connections between two vertices.
\item $R^2$ is the same as $R \circ R$ and this relational product is defined as
      \\[0.2cm]
      \hspace*{1.3cm}
       $R \circ R = \bigl\{ \pair(x,z) \bigm| \exists y \colon(\pair(x,y) \in R \wedge \pair(y,z)) \in R \bigr\}$.
      \\[0.2cm]
      Hence, $R \circ R$ contains those pairs $\pair(x,z)$ that are connected via one intermediate vertex $y$,
      i.e.~there is a path of the form $x \mapsto y \mapsto z$ that connects $x$ and $z$.  This path
      has length 2.  In general, we can show by induction that $R^n$ connect those pairs that are
      connected by a path of length $n$.  The induction step of this proof runs as follows:
\item $R^{n+1}$ is defined as $R^n \circ R$ and therefore we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $R^n \circ R = \{ \pair(x,z) \mid \exists y \colon \pair(x,y) \in R^n \wedge \pair(y,z) \in R \}$.
      \\[0.2cm]
      As $\pair(x,y) \in R^n$, the induction hypothesis guarantees that the vertices $x$ and $y$ are
      connected by a path of length $n$.  Hence, this
      path has the form
      \\[0.2cm]
      \hspace*{1.3cm}
      $\underbrace{x \mapsto \cdots \mapsto y}_{\mbox{\scriptsize path of length $n$.}}$
      \\[0.2cm]
      Adding $x$ at the end of this path will produce the path
      \\[0.2cm]
      \hspace*{1.3cm}
      $x \mapsto \cdots \mapsto y \mapsto z$.
      \\[0.2cm]
      This path has a length of $n + 1$ and, furthermore, connects $x$ and $z$.  Hence $R^{n+1}$
      contains those pairs $\pair(x, z)$ that are connected by a path of length $n+1$.
\end{enumerate}
Now the important observation is the following. The set of all vertices is finite.  For the arguments sake, let
us assume there are $k$ different vertices.  But then every path that has a length of $k$ or greater must
contain at least one vertex that is visited more than once and hence this path is longer than necessary,
i.e.~there is a shorter path that connects the same vertices.  Therefore, for a finite graph with $k$ vertices,
the formula to compute the transitive closure can be simplified as follows:
\\[0.2cm]
\hspace*{1.3cm}
$\ds R^+ = \bigcup\limits_{i=1}^{k-1} R^i$.
\\[0.2cm]
While we could use this formula as its stands, it is more efficient to use a \blue{fixed-point iteration} instead.
To this end, we prove that the transitive closure $R^+$ satisfies the following equation:
\begin{equation}
  \label{fixpunkt}
  R^+ = R \cup R^+ \circ R.
\end{equation}
Let me remind you that the precedence of the operator $\circ$
is higher than the precedence of the operator $\cup$.  Therefore, the expression $R \cup R^+ \circ R$ is parenthesized
as $R \cup (R^+ \circ R)$.  Equation \ref{fixpunkt} can be proven algebraically.  We have:
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}{cll}
    & R \cup R^+ \circ R \\[0.2cm]
  = & R \cup \Bigl(\bigcup\limits_{i=1}^{\infty} R^i \Bigr) \circ R \\[0.4cm]
  = & R \cup \bigl(R^1 \cup R^2 \cup R^3 \cup \cdots \bigr) \circ R \\[0.2cm]
  = & R \cup \bigl(R^1 \circ R \cup R^2 \circ R \cup R^3 \circ R \cup \cdots \bigr) \\[0.2cm]
  = & R \cup \bigl(R^2 \cup R^3 \cup  R^4 \cup \cdots \bigr)  \\[0.2cm]
  = & R^1 \cup \bigl(R^2 \cup R^3 \cup  R^4 \cup \cdots \bigr) \\[0.2cm]
  = & \bigcup\limits_{i=1}^{\infty} R^i \\[0.4cm]
  = & R^+.
\end{array}
$
\\[0.2cm]
Equation  \ref{fixpunkt} can now be used to compute $R^+$ via a fixed-point iteration.
To this end, let us define a sequence of relations $(T_n)_{n \in \mathbb{N}}$ by induction on $n$:
\begin{enumerate}
\item[I.A.] $n = 0$:

            $T_0 = R$
\item[I.S.] $n \mapsto n+1$:

            $T_{n+1} = R \cup T_n \circ R $.
\end{enumerate}
The relation  $T_n$ can be expressed via the relation $R$, we have
\begin{enumerate}
\item $T_0 = R$.
\item $T_1 = R \cup T_0 \circ R = R \cup R \circ R = R^1 \cup R^2$.
\item$\begin{array}[t]{lcl}
       T_2  & = & R \cup T_1 \circ R \\
            & = & R \cup (R^1 \cup R^2) \circ R \\
            & = & R^1 \cup R^2 \cup R^3. \\
       \end{array}
      $
\end{enumerate}
In general, we can show by induction that
\\[0.2cm]
\hspace*{1.3cm}
$T_n = \bigcup\limits_{i=1}^{n+1} R^i$
\\[0.2cm]
holds for all $n \in \mathbb{N}$.  The base case of this proof is immediate from the definition of $T_0$.
In the induction step we observe the following:
\\[0.2cm]
\hspace*{1.3cm}
$
 \begin{array}{lcll}
   T_{n+1} & = & \ds R \cup T_n \circ R & \mbox{(by definition)} \\[0.2cm]
           & = & \ds R \cup \biggl(\bigcup\limits_{i=1}^{n+1} R^i\biggr) \circ R &
                 \mbox{(by induction hypothesis)} \\[0.4cm]
           & = & \ds R \cup \left(R \cup \cdots \cup R^{n+1}\right) \circ R \\[0.2cm]
           & = & \ds R^1 \cup R^2 \cup \cdots \cup R^{n+2}  &
                 \mbox{(by the distributivity of $\circ$ over $\cup$)} \\[0.2cm]
           & = & \ds \bigcup\limits_{i=1}^{n+2} R^i & \Box
   \end{array}
$
\\[0.2cm]
The sequence $(T_n)_{n\in\mathbb{N}}$ has another useful property:  It is
\blue{monotonically increasing}.  In general, a sequence of sets $(X_n)_{n\in\mathbb{N}}$ is called
\blue{monotonically increasing} iff we have
\\[0.2cm]
\hspace*{1.3cm}
$\forall n \in \mathbb{N}: X_n \subseteq X_{n+1}$,
\\[0.2cm]
i.e.~the sets $X_n$ get bigger with growing index $n$.
The monotonicity of the sequence  $(T_n)_{n \in \mathbb{N}}$ is an immediate consequence of the equation
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_n = \bigcup\limits_{i=1}^{n+1} R^i$
\\[0.2cm]
because we have:
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}[t]{llcl}
                & \ds T_n \subseteq T_{n+1} \\[0.2cm]
\Leftrightarrow & \ds \bigcup\limits_{i=1}^{n+1} R^i \subseteq \bigcup\limits_{i=1}^{n+2} R^i \\[0.5cm]
\Leftrightarrow & \ds \bigcup\limits_{i=1}^{n+1} R^i \subseteq \bigcup\limits_{i=1}^{n+1} R^i \cup R^{n+2} \\
\end{array}
$
\\[0.2cm]
If the relation  $R$ is finite, then the transitive closure $R^+$ is finite, too.  The sets $T_n$
are all subsets of $R^+$ because we have
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_n = \bigcup\limits_{i=1}^{n+1} R^i \subseteq \bigcup\limits_{i=1}^{\infty} R^i = R^+$ \quad for all $n \in \mathbb{N}$.
\\[0.2cm]
Hence the sets $T_n$ can not grow indefinitely.  Because of the monotonicity of the sequence
$(T_n)_{n\in\mathbb{N}}$ it follows that there exists an index  $k \in \mathbb{N}$ such that the sets $T_n$ do
not grow any further once $n$ has reached $k$, i.e.~we have
\\[0.2cm]
\hspace*{1.3cm}
$\ds \forall n \in \mathbb{N}:( n \geq k \rightarrow T_n = T_k)$.
\\[0.2cm]
But this implies that
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_n = \bigcup\limits_{i=1}^{n+1} R^i = \bigcup\limits_{i=1}^{\infty} R^i = R^+$
\quad holds for all $n \geq k$.
\\[0.2cm]
Therefore, the algorithm for computing  $R^+$ iterates the equation
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_{n+1} = R \cup T_n \circ R$
\\[0.2cm]
until the equation  $T_{n+1} = T_n$ is satisfied, since this implies that $T_n = R^+$.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def product(R1, R2):
        "Compute the relational product of R1 and R2."
        return { (x, z) for (x, y1) in R1 for (y2, z) in R2 if y1 == y2 }

    def transClosure(R):
        "Compute the transitive closure of the binary relation R."
        T = R
        while True:
            oldT = T
            T    = product(R,T).union(R)
            if T == oldT:
                return T

    R = { (1,2), (2,3), (1,3), (2,4), (4,5) }
    print( "R  = ", R );
    print( "Computing the transitive closure of R:" );
    T = transClosure(R);
    print( "R+ = ", T );
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing the transitive closure.}
\label{fig:transitive-closure.py}
\end{figure} %\$

\noindent
The program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/transitive-closure.py}{\texttt{transitive-closure.py}}
that is shown in Figure
\ref{fig:transitive-closure.py} on page \pageref{fig:transitive-closure.py} shows an implementation of this idea.
The program produces the following output:
\begin{verbatim}
    R  = {(1, 2), (1, 3), (4, 5), (2, 3), (2, 4)}
    Computing the transitive closure of R:
    R+ = {(1, 2), (1, 3), (4, 5), (1, 4), (1, 5), (2, 3), (2, 5), (2, 4)}
\end{verbatim}
The transitive closure $R^+$ of a relation $R$ has a very intuitive interpretation:
It contains all pairs $\pair(x,y)$ such that there is a path leading from
$x$ to $y$.
The function $\texttt{product}(R_1, R_2)$ computes the relational product $R_1\circ R_2$
according to the formula
\\[0.2cm]
\hspace*{1.3cm}
$R_1 \circ R_2 = \{ \langle x, z \rangle \mid \exists y:\bigl(\pair(x,y) \in R_1 \wedge \pair(y,z) \in R_2\bigr) \}$.

\subsection{Computing the Paths}
So far, given a graph represented by a relation $R$ and two vertices $x$ and $y$, we can only check
whether there is a path leading from $x$ to $y$, but we cannot compute this path.  In this
subsection we will extend the procedure \texttt{transClosure} so that it will also compute the
corresponding path.  The main idea is to extend the notion of a relational product to the notion of
a \blue{path product}, where a \blue{path product} is defined on sets of paths.  In order to do so,
we introduce three functions for tuples.
\begin{enumerate}
\item Given a tuple $T$, the function $\texttt{first}(T)$ returns the first element of $T$:
      \\[0.2cm]
      \hspace*{1.3cm}
      $\texttt{first}\bigl(\langle x_1,\cdots,x_m\rangle\bigr) = x_1$.
\item Given a tuple $T$, the function $\texttt{last}(T)$ returns the last element of $T$:
      \\[0.2cm]
      \hspace*{1.3cm}
      $\texttt{last}\bigl(\langle x_1,\cdots,x_m\rangle\bigl) = x_m$.
\item If $S = \langle x_1, \cdots, x_m\rangle$ and $T = \langle y_1, \cdots, y_n \rangle$ are two tuples  such that
      $\texttt{first}(S) = \texttt{last}(S)$, we define the \blue{join} $S \oplus T$ of $S$ and $T$ as \\[0.2cm]
      \hspace*{1.3cm}
      $S \oplus T = \langle x_1, \cdots, x_m, y_2, \cdots, y_n \rangle$.
\end{enumerate}
If $\mathcal{P}_1$ and $\mathcal{P}_2$ are sets of tuples representing paths, we define the \blue{path product} of
$\mathcal{P}_1$ and $\mathcal{P}_2$ as follows: \\[0.2cm]
\hspace*{1.3cm}
$\mathcal{P}_1 \bullet \mathcal{P}_2 =
\bigl\{\; T_1 \oplus T_2 \mid T_1 \in \mathcal{P}_1 \wedge T_2 \in \mathcal{P}_2 \wedge \texttt{last}(T_1) = \texttt{first}(T_2) \;\bigr\}
$.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def findPaths(R):
        P = R;
        while True:
            oldP = P
            P    = R.union(pathProduct(P, R))
            print(P)
            if P == oldP:
                return P

    def pathProduct(P, Q):
        return { join(S, T) for S in P for T in Q if S[-1] == T[0] }

    def join(S, T):
        return S + T[1:]

    R = { (1,2), (2,3), (1,3), (2,4), (4,5) }
    print("R = ", R)
    print("Computing all paths:" )
    P = findPaths(R)
    print("P = ", P)
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing all connections.}  \label{fig:path.py}
\end{figure} %\$

\begin{figure}[!ht]
  \centering
  \vspace*{-9cm}

  \epsfig{file=Figures/graph-zykl,scale=0.5}
  \vspace*{-1cm}

  \caption{A graph with a cycle.}
  \label{fig:graph-zykl}
\end{figure}

Using the notion of a \blue{path product} we are able to extend the program shown in Figure
\ref{fig:transitive-closure.py} such that it computes all paths between two vertices.
The resulting program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/path.py}{\texttt{path.py}}
is shown in Figure \ref{fig:path.py} on page \pageref{fig:path.py}.
Unfortunately, the program does not work any more if the graph is \blue{cyclic}.  A graph is defined
to be \blue{cyclic} if there is a path of length greater than $1$ that starts and ends at the same
vertex.  This path is then called a \blue{cycle}.
Figure \ref{fig:graph-zykl} on page \pageref{fig:graph-zykl} shows a cyclic graph.  This graph is
cyclic because it contains the path
\\[0.2cm]
\hspace*{1.3cm}
$\langle 1, 2, 4, 1 \rangle$
\\[0.2cm]
and this path is a cycle.
The problem with this graph is that it contains an infinite number of paths that connect the vertex
1 with the vertex 2: \\[0.2cm]
\hspace*{1.3cm}
$\langle 1, 2 \rangle$, $\langle 1, 2, 4, 1, 2 \rangle$,
$\langle 1, 2, 4, 1, 2, 4, 1, 2 \rangle$,
$\langle 1, 2, 4, 1, 2, 4, 1, 2, 4, 1, 2 \rangle$, $\cdots$
\\[0.2cm]
Of course, there is no point in computing a path that visits a vertex more than once as these paths
contain cycles.  Our goal is to eliminate all those paths that contain cycles.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ numbers       = left,
                  numbersep     = -0.2cm,
                  frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  xleftmargin   = 0.0cm,
                  xrightmargin  = 0.0cm,
                ]
    def pathProduct(P, Q):
        return { join(S, T) for S in P for T in Q
                            if S[-1] == T[0] and noCycle(S, T)
               }

    def noCycle(T1, T2):
        return len({ x for x in T1 }.intersection({ x for x in T2 })) == 1
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing the connections in a cyclic graph.}
\label{fig:path-cyclic.py}
\end{figure} %\$

Figure \ref{fig:path-cyclic.py} on page shows how the implementation of the function
\texttt{pathProduct} has to be changed so that the resulting program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/path-cyclic.py}{\texttt{path-cyclic.py}}
works also for cyclic graphs.
\begin{enumerate}
\item In line 2 and 3, we compute only those paths that are not cyclic.
\item Line 6 defines a function \texttt{noCycle} that tests, whether the join  $\texttt{T1} \oplus \texttt{T2}$ is cyclic.  The join
      of \texttt{T1} and \texttt{T2} is cyclic iff the tuples \texttt{T1} and \texttt{T2} have more
      than one common element.  The tuples \texttt{T1} and \texttt{T2} will always have at least one common element, as we join
      these tuples only if the last element of \texttt{T1} is equal to the first element of  \texttt{T2}.
      If there would be an another vertex common to both \texttt{T1} and \texttt{T2}, then the path
      $\texttt{T1} \oplus \texttt{T2}$ would be cyclic.
\end{enumerate}

In general, we are not really interested to compute all possible paths between two given vertices
\texttt{x} and \texttt{y}.  Instead, we just want to compute the shortest path leading from \texttt{x} to \texttt{y}.
Figure \ref{fig:find_path.py} on page \pageref{fig:find_path.py} shows the procedure \texttt{reachable}.
This procedure takes three arguments:
\begin{enumerate}
\item \texttt{start} and \texttt{goal} are vertices of a graph.
\item \texttt{R} is a binary relation representing a directed graph.
\end{enumerate}
The call  \texttt{reachable(start, goal, R)} checks whether \texttt{start} and \texttt{goal} are connected and, furthermore,
computes the shortest path from \texttt{start} to \texttt{goal}, provided such a path exists.
The complete program can be found in the file
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/find\_path.py}{\texttt{find\_path.py}}.
Next, we discuss the implementation of the procedure  \texttt{reachable}.
\begin{enumerate}
\item Line 2 initializes the set \texttt{P}.  After $n$ iterations, this set will contain all paths
      that start with the vertex \texttt{start} and that have a length of at most $n$.

      Initially, there is just the trivial path $\langle\texttt{start}\rangle$ that starts with vertex
      \texttt{start} and has length $0$.
\item Line 5 tries to extend all previously computed paths by one step.
      If we are lucky, the set \texttt{P} is increased in this step.
\item Line 6 selects all those paths from the set \texttt{P} that lead to the vertex \texttt{goal}.
      These paths are stored in the set \texttt{Found}.
\item Line 7 checks whether we have indeed found a path ending at \texttt{goal}.  This is the case if
      the set \texttt{Found} is not empty.
      In this case, we return any of these paths.
\item If we have not yet found the vertex \texttt{goal} and, furthermore, we have not been able to find
      any new paths during this iteration,  the procedure returns in line 10.
      As the \texttt{return} statement in line 11 does not return a value, the procedure will
      instead return the value \texttt{None}.
\end{enumerate}
The procedure call \texttt{reachable(start,goal R)} will compute the \textbf{shortest} path connecting
\texttt{start} and \texttt{goal} because it computes path with increasing length.  The first iteration
computes all paths starting in \texttt{start} that have a length of at most 1, the second iteration
computes all paths starting in \texttt{start} that have a length of at most 2, and in general the $n$-th
iteration computes all paths starting in \texttt{start} that have a length of at most $n$.  Hence, if
there is a path of length $n$, then this path will be found in the $n$-iteration unless a shorter path has
already been found in a previous iteration.

\remarkEng
The algorithm described above is known as
\href{https://en.wikipedia.org/wiki/Breadth-first_search}{breadth first search}. \eox

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def reachable(start, goal, R):
        P = { (start,) }
        while True:
            oldP  = P
            P     = P.union(path_product(P, R))
            Found = { T for T in P if T[-1] == goal }
            if Found != set({}):
                return Found.pop()
            if P == oldP:
                return

    def path_product(P, R):
        return set( add(T1, T2) for T1 in P for T2 in R
                             if T1[-1] == T2[0] and noCycle(T1, T2)
                  )

    def noCycle(T1, T2):
        return len(set(T1).intersection(set(T2))) == 1

    def add(T, P):
        return T + (P[-1],)
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Finding the shortest path between two vertices.}
\label{fig:find_path.py}
\end{figure}

\subsection{The Wolf, the Goat, and the Cabbage}
Next, we present an application of the theory developed so far.  We solve a problem that has puzzled
the greatest agricultural economists for centuries.  The puzzle we want to solve is known as the
\href{http://jeux.lulu.pagesperso-orange.fr/html/anglais/loupChe/loupChe1.htm}{wolf-goat-cabbage puzzle}:
\vspace*{0.3cm}

\begin{minipage}[c]{14cm}
{\sl
An agricultural economist has to sell a wolf, a goat, and a cabbage on a market place.  In order to
reach the market place, she has to cross a river.  The boat that she can use is so small that it can
only accommodate either the goat, the wolf, or the cabbage in addition to the agricultural economist.
Now if the agricultural economist leaves the wolf alone with the goat, the wolf will eat the goat.
If, instead, the agricultural economist leaves the goat with the cabbage, the goat will eat the cabbage.
Is it possible for the agricultural economist to develop a schedule that allows her to cross the river
without either the goat or the cabbage being eaten?
}
\end{minipage}
\vspace*{0.3cm}

\noindent
In order to compute a schedule, we first have to model the problem.  The various \blue{states} of the problem will
be regarded as \blue{vertices} of a graph and this graph will be represented as a binary relation.
To this end we define the set
\begin{verbatim}
  All = {'farmer', 'wolf, 'goat', 'cabbage'}
\end{verbatim}
Every node will be represented as a subset \texttt{S} of the set \texttt{All}.  The idea is that the set \texttt{S}
specifies those objects that are on the left side of the river.  We assume that initially the farmer
is on the left side of the river.
Therefore, the set of all possible states can be defined as the set
\begin{verbatim}
  States = {S for S in power(All) if not problem(S) and not problem(All-S)}
\end{verbatim}
Here, we have used the procedure \texttt{problem} to check whether a given set \texttt{S} has a problem.
Note that since \texttt{S} is the set of objects on the left side, the expression $\texttt{All-S}$
computes the set of objects on the right side of the river.

Next, a set \texttt{S} of objects has a problem if both of the following conditions
are satisfied:
\begin{enumerate}
\item The farmer is not an element of \texttt{S} and
\item either \texttt{S} contains both the goat and the cabbage or \texttt{S} contains both the wolf and the goat.
\end{enumerate}
Therefore, we can implement the function \texttt{problem} as follows:
\begin{verbatim}
  def problem(S):
      return ("farmer" not in S) and             \
             (("goat" in S and "cabbage" in S) or   # goat eats cabbage
              ("wolf" in S and "goat"    in S)   )  # wolf eats goat
\end{verbatim}
We proceed to compute the relation \texttt{R} that contains all possible transitions between
different states.  We will compute \texttt{R} using the formula:
\\[0.2cm]
\hspace*{0.75cm}
\texttt{R = R1 + R2;}
\\[0.2cm]
Here \texttt{R1} describes the transitions that result from the farmer crossing the river from left
to right, while \texttt{R2} describes the transitions that result from the farmer crossing the river
from right to left.  We can define the relation \texttt{R1} as follows:
\begin{verbatim}
  R1 = { (S, S-B) for S in States for B in power(S)
                  if S-B in States and 'farmer' in B and len(B) <= 2
       }
\end{verbatim}
Let us explain this definition in detail:
\begin{enumerate}
\item Initially, \texttt{S} is the set of objects on the left side of the river.  Hence, \texttt{S}
      is an element of the set of all states that we have defined as \texttt{P}.
\item \texttt{B} is the set of objects that are put into the boat and that do cross the river.  Of
      course, for an object to go into the boat is has to be on the left side of the river to begin
      with.  Therefore, \texttt{B} is a subset of \texttt{S} and hence an element of the power set
      of \texttt{S}.
\item Then  \texttt{S-B} is the set of objects that are left on the left side of the river after
      the boat has crossed.  Of course, the new state \texttt{S-B} has to be a state that does not
      have a problem.  Therefore, we check that \texttt{S-B} is an element of \texttt{States}.
\item Furthermore, the farmer has to be inside the boat.  This explains the condition
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{\symbol{39}farmer\symbol{39} in B}.
\item Finally, the boat can only have two passengers.  Therefore, we have added the condition
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{len(B) <= 2}.
\end{enumerate}
Next, we have to define the relation \texttt{R2}.  However, as crossing the river from right to left
is just the reverse of crossing the river from left to right, \texttt{R2} is just the \blue{inverse} of
\texttt{R1}.   Hence we define:
\begin{verbatim}
  R2 = { (S2, S1) for (S1, S2) in R1 }
\end{verbatim}
Next, the relation \texttt{R} is the union of \texttt{R1} and \texttt{R2}:
\begin{verbatim}
  R = R1.union(R2)
\end{verbatim}
Finally, the start state has all objects on the left side.  Therefore, we have
\begin{verbatim}
  start = All
\end{verbatim}
In the end, all objects have to be on the right side of the river.  That means that nothing is left
on the left side.  Therefore, we define
\begin{verbatim}
  goal = {}
\end{verbatim}

\begin{figure}[h]
  \centering

  \epsfig{file=Figures/wolf-goat-cabbage, scale=0.4}

  \caption{The relation \texttt{R} shown as a directed graph.}
  \label{fig:wolf-goat-cabbage.pdf}
\end{figure}

Figure \ref{fig:wolf-ziege} on page \pageref{fig:wolf-ziege} shows the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/wolf-goat-cabbage.py}{\texttt{wolf-goat-cabbage.py}}
that combines the statements shown so far.  The solution computed by this program is shown in Figure
 \ref{fig:wolf-ziege-solution}.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ codes         = {\catcode`$=3\catcode`_=8\catcode`^=7},
                  frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.3cm,
                  xrightmargin  = 0.3cm,
                ]
    def problem(S):
        return ('farmer' not in S) and             \
               (('goat' in S and 'cabbage' in S) or   # goat eats cabbage
                ('wolf' in S and 'goat'    in S)   )  # wolf eats goat

    All   = frozenset( ['farmer', 'wolf', 'goat', 'cabbage'] )
    R1    = { (S, S - B) for S in States for B in power(S)
                         if S - B in States and 'farmer' in B and len(B) <= 2
            }
    R2    = { (S2, S1) for (S1, S2) in R1 }
    R     = R1.union(R2)
    start = All
    goal  = frozenset()
    Path  = findPath(start, goal, R)
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Solving the wolf-goat-cabbage problem.}
\label{fig:wolf-ziege}
\end{figure}

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ codes         = {\catcode`$=3\catcode`_=8\catcode`^=7},
                  frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    {"cabbage", "farmer", "goat", "wolf"}                                 {}
                             >>>> {"farmer", "goat"} >>>>
    {"cabbage", "wolf"}                                   {"farmer", "goat"}
                             <<<< {"farmer"} <<<<
    {"cabbage", "farmer", "wolf"}                                   {"goat"}
                             >>>> {"farmer", "wolf"} >>>>
    {"cabbage"}                                   {"farmer", "goat", "wolf"}
                             <<<< {"farmer", "goat"} <<<<
    {"cabbage", "farmer", "goat"}                                   {"wolf"}
                             >>>> {"cabbage", "farmer"} >>>>
    {"goat"}                                   {"cabbage", "farmer", "wolf"}
                             <<<< {"farmer"} <<<<
    {"farmer", "goat"}                                   {"cabbage", "wolf"}
                             >>>> {"farmer", "goat"} >>>>
    {}                                 {"cabbage", "farmer", "goat", "wolf"}
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A schedule for the agricultural economist.}
\label{fig:wolf-ziege-solution}
\end{figure}
\pagebreak
\vspace*{\fill}

\section{Symbolic Differentiation}
In this section we will develop a program that reads an arithmetic expression like
\\[0.2cm]
\hspace*{1.3cm}
\texttt{"x * exp(x)"},
\\[0.2cm]
interprets this string as describing the real valued function
\\[0.2cm]
\hspace*{1.3cm}
$x \mapsto x \cdot \exp(x)$,
\\[0.2cm]
and then takes the derivative of this function with respect to the variable $x$.  In order to specify the input
of this program more clearly, we first define the notion of an \blue{arithmetic expression} inductively.
\begin{enumerate}
\item Every number $c \in \mathbb{R}$ is an arithmetic expression.
\item Every variable $v$ is an arithmetic expression.
\item If $s$ and $t$ are arithmetic expressions, then
      \\[0.2cm]
      \hspace*{1.3cm}
      $s + t$, \quad $s - t$, \quad $s * t$, \quad $s / t$, \quad and \quad $s \,\mathtt{**}\, t$
      \\[0.2cm]
      are arithmetic expressions.  Here $s \,\mathtt{**}\, t$ is interpreted as $s^t$.

\item If $e$ is an arithmetic expression, then both
      \\[0.2cm]
      \hspace*{1.3cm}
      $\exp(e)$ \quad and \quad $\ln(e)$
      \\[0.2cm]
      are arithmetic expressions.
\end{enumerate}
We want do implement a function \texttt{diff} that takes two arguments:
\begin{enumerate}
\item The first argument \texttt{expr} represents an arithmetic expression.
\item The second argument \texttt{var} is the name of a variable.
\end{enumerate}
The function call \texttt{diff(expr, var)} will then compute the derivative of \texttt{expr} with respect to the variable \texttt{var}.  For example, the function call \texttt{diff("x*exp(x)", "x")} will compute the output
\\[0.2cm]
\hspace*{1.3cm}
"\texttt{1*exp(x) + x*exp(x)}"
\\[0.2cm]
because we have:
$$ \frac{\mathrm{d}\;}{\mathrm{d}x} \bigl( x \cdot \mathrm{e}^x \bigr) = 1 \cdot x + x \cdot \mathrm{e}^x $$
It would be very tedious to \blue{represent} arithmetic expressions as strings.  Instead, we will represent
arithmetic expressions as \blue{nested tuples}.  The notion of a \emph{nested tuple} is defined inductively:
\begin{itemize}
\item $\langle x_1, x_2, \cdots, x_n \rangle$ is a nested tuple if each of the components $x_i$ is either a
      number, a string, or a nested tuple.
\end{itemize}
For example, the arithmetic expression "\texttt{x*exp(x)}" is represented as the nested tuple
\\[0.2cm]
\hspace*{1.3cm}
$\langle\texttt{"}*\texttt{"}, \texttt{"}x\texttt{"}, \langle \texttt{"}\mathtt{exp}\texttt{"}, \texttt{"}x\texttt{"} \rangle\rangle$.
\\[0.2cm]
In order to be able to convert string into nested tuples, we need a \blue{parser}.  A parser is a program that
takes a string as input and transforms this string into a nested tuple, which is then returned as a result.
I have implemented a parser in the file "\texttt{exprParser.py}".  The details of the implementation of this
parser will be discussed in the lecture on algorithms.

\noindent
We close this section by showing an example that demonstrates the power of matching.
The function \texttt{diff} that is shown in Figure \ref{fig:diff.py} on page \pageref{fig:diff.py} is part
of the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/diff.py}{\texttt{diff.py}}.
This function is called with one argument:
The argument \texttt{e} is an arithmetic expression.
The function \texttt{diff} interprets its argument \texttt{e} as a function of the variable
\texttt{x}.  We take the \href{https://en.wikipedia.org/wiki/Derivative}{derivative} of this
function with respect to the variable \texttt{x}.  For example, in order to compute the derivative of
the function
\\[0.2cm]
\hspace*{1.3cm}
$x \mapsto x^x$,
\\[0.2cm]
we can call the function  \texttt{diff} as follows:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{diff("x ** x")}.
\\[0.2cm]
Let us now discuss the implementation of the function \texttt{diff} in more detail.
\begin{enumerate}
\item The lines 3 - 6 implement the rule:
      $$\frac{\mathrm{d}\;}{\mathrm{d}x}\bigl(f(x) + g(x)\bigr) = \frac{\mathrm{d}\;}{\mathrm{d}x} f(x) + \frac{\mathrm{d}\;}{\mathrm{d}x} g(x)$$
\item Line 7 - 10 implement the rule:
      $$\frac{\mathrm{d}\;}{\mathrm{d}x}\bigl(f(x) - g(x)\bigr) = \frac{\mathrm{d}\;}{\mathrm{d}x} f(x) - \frac{\mathrm{d}\;}{\mathrm{d}x} g(x)$$
\item Line 11 - 14 deals with the case where \texttt{e} is a product.  The
      \href{https://en.wikipedia.org/wiki/Product\_rule}{product rule} is
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x}\bigl(f(x) \cdot g(x)\bigr) = \left(\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)\right)\cdot g(x) + f(x) \cdot \left(\frac{\mathrm{d}\;}{\mathrm{d}x} g(x)\right)
      $$
\item Line 15 - 17 deals with the case where \texttt{e} is a quotient.  The
      \href{https://en.wikipedia.org/wiki/Quotient\_rule}{quotient rule} is
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x}\left(\frac{f(x)}{g(x)}\right) =
         \frac{\displaystyle\left(\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)\right)\cdot g(x) -
         f(x) \cdot \left(\frac{\mathrm{d}\;}{\mathrm{d}x} g(x)\right)}{g(x) \cdot g(x)}
      $$
\item Line 19 - 21 deals with the case where \texttt{e} is a power.  Now in order to take the derivative of an
      expression of the form
      $$  f(x)^{g(x)} $$
      we first need to rewrite this expression using the following trick:
      $$ f(x)^{g(x)} = \exp\bigl(\ln\bigl(f(x)^{g(x)}\bigr)\bigr) = \exp\bigl(g(x) \cdot \ln(f(x))\bigr) $$
      Then, we can recursively call \texttt{diff} for this expression.  This works, because the function
      \texttt{diff} can deal with both the exponential function $x \mapsto \exp(x)$ and with the natural
      logarithm $x \mapsto \ln(x)$.  This rewriting is done in line 21.
\item Line 22-25 deals with the case where \texttt{e} has the form
      $$\ln\bigl(f(x)\bigr)$$
      In order to take the derivative of this expression, we first need to know the derivative of the natural
      logarithm.  This derivative is given as
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x} \ln(x) = \frac{1}{x}$$
      Then, using the \href{https://en.wikipedia.org/wiki/Chain\_rule}{chain rule} we have that
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x} \ln\bigl(f(x)\bigr) = \frac{\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)}{f(x)}$$
\item Line 26 - 29 deals with the case where \texttt{e} has the form $\exp\bigl(f(x)\bigr)$.
      In order to take the derivative of this expression, we first need to know the derivative of the
      \href{https://en.wikipedia.org/wiki/Exponential\_function}{exponential function}.
      This derivative is given as
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x} \exp(x) = \exp(x)$$
      Then, using the \href{https://en.wikipedia.org/wiki/Chain\_rule}{chain rule} we have that
      $$\frac{\mathrm{d}\;}{\mathrm{d}x} \exp\bigl(f(x)\bigr) = \left(\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)\right) \cdot \exp\bigl(f(x)\bigr) $$
\item Line 30-31 deals with the case where \texttt{e} is a variable and happens to be the same variable as
      \texttt{x}.  This is checked using the condition
      \texttt{e == x}.  As we have
      $$\frac{\mathrm{d}x}{\mathrm{d}x} = 1,$$
      the function \texttt{diff} returns \texttt{1} in this case.
\item Otherwise, the expression is assumed to be a constant and hence we return 0.
\end{enumerate}

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def diff(e):
        "differentiate the expressions e with respect to the variable x"
        if e[0] == '+':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('+', fs, gs)
        if e[0] == '-':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('-', fs, gs)
        if e[0] == '*':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('+', ('*', fs, g), ('*', f, gs))
        if e[0] == '/':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('/', ('-', ('*', fs, g), ('*', f, gs)), ('*', g, g))
        if e[0] == '**':
            f , g  = e[1:]
            return diff(('exp', ('*', g, ('ln', f))))
        if e[0] == 'ln':
            f  = e[1]
            fs = diff(f)
            return ('/', fs, f)
        if e[0] == 'exp':
            f  = e[1]
            fs = diff(f)
            return ('*', fs, e)
        if e == 'x':
            return '1'
        return 0
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A function for symbolic differentiation}
\label{fig:diff.py}
\end{figure}

In order to test this function we can implement a function \texttt{test} as shown in Figure \ref{fig:test-diff.py}.
Then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{diff("x ** x")}
\\[0.2cm]
yields the result:
\\[0.2cm]
\hspace*{1.3cm}
d/dx x ** x = (1*ln(x) + x*1/x)*exp(x*ln(x))
\\[0.2cm]
This shows that
\\[0.2cm]
\hspace*{1.3cm}
$\ds \frac{\mathrm{d};}{\mathrm{d}x} x^x = \bigl(\ln(x) + 1\bigr) \cdot \exp\bigl(x \cdot \ln(x)\bigr) =
 \bigl(\ln(x) + 1\bigr) \cdot x^x
$.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    import exprParser as ep

    def test(s):
        t = ep.ExprParser(s).parse()
        d = diff(t)
        print(f"d/dx {s} = {ep.toString(d)}")
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Testing symbolic differentiation.}
\label{fig:test-diff.py}
\end{figure}

\pagebreak
\vspace*{\fill}

\section{Reflection}
After having completed this chapter, you should be able to answer the following questions.
\begin{enumerate}
\item Which data types are supported in \textsl{Python}?
\item What are the different methods to define a set in \textsl{Python}?
\item Do you understand how to construct lists via iterator?
\item How can lists be defined in \textsl{Python}?
\item How does \textsl{Python} support binary relations?
\item How does list slicing and list indexing work?
\item What are nested tuples?
\item How does a fixed-point algorithm work?
\item What type of control structures are supported in \textsl{Python}?
\end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "logic"
%%% End:


\section{Regular expressions}

In \Pascal, an identifier is a letter followed by zero or more letters
or digits, that is, and identifier is a member of the set defined by
\(L(L \cup D)^{*}\). The notation we introduced so far is comfortable
for mathematics but not for computers. Let us introduce another
notation, called \emph{regular expressions}, for describing the same
languages and define its meaning in terms of the mathematical
notation. With this notation, we might define \Pascal identifiers as
\begin{center}
\term{letter} \lparen\term{letter} \disj \term{digit}\rparen\kleene
\end{center}
where the vertical bar means `or', the parentheses group
subexpressions, the star means `zero or more instances of' the
previous expression and juxtaposition means concatenation. A regular
expression~\(r\) is built up out of simpler regular expressions using
a set of rules, as follows. Let~\(\Sigma\) be an alphabet and \(L(r)\)
the language denoted by~\(r\). Then
\begin{enumerate*}

   \item \(\epsilon\) is a regular expression that denotes
     \(\{\varepsilon\}\);\label{regexp_empty}

   \item if \(a \in \Sigma\), then \(a\)~is a regular expression that
     denotes~\(\{a\}\). This is ambiguous: \(a\)~can denote a
     language, a word or a letter --~it depends on the
     context; \label{regexp_sym}

   \item assume \(r\)~and~\(s\) denote the languages \(L(r)\)~and
     \(L(s)\); \(a\)~denotes a letter. Then \label{regexp_rec}
   \begin{enumerate*}

     \item \(r\) \disj \(s\) is a regular expression
     denoting \(L(r) \cup L(s)\);

     \item \(r s\) is a regular expression denoting \(L(r) L(s)\);

     \item \(r\)\kleene{} is a regular expression
     denoting \((L(r))^{*}\);

     \item \lparen\(r\)\rparen{} is a regular expression
     denoting \(L(r)\);

     \item \(\overline{a}\) is a regular expression denoting
       \(\Sigma\backslash \{a\}\).

   \end{enumerate*}

\end{enumerate*}
A language described by a regular expression is a \emph{regular
  language}. Rules~\ref{regexp_empty} and~\ref{regexp_sym} form the
base of the definition. Rule~\ref{regexp_rec} provides the inductive
step. Unnecessary parentheses can be avoided in regular expressions if
\begin{itemize*}

  \item the unary operator \kleene{} has the highest precedence and
  is left associative,

  \item concatenation has the second highest precedence and is left
  associative,

  \item \disj{} has the lowest precedence and is left associative.

\end{itemize*}
Under those conventions, \lparen\(a\)\rparen{} \disj
\lparen\lparen\(b\)\rparen\kleene\lparen\(c\)\rparen\rparen{} is
equivalent to \(a\) \disj \(b\)\kleene\(c\). Both expressions denote
the language containing either the string \(a\) or zero or more
\(b\)'s followed by one \(c\): \(\{a, c, bc, bbc, bbbc, \dots\}\). For
example,
\begin{itemize*}

  \item the regular expression \(a\) \disj \(b\) denotes the set
    \(\{a, b\}\);

  \item the regular expression \lparen\(a\) \disj
    \(b\)\rparen\lparen\(a\) \disj \(b\)\rparen{} denotes \(\{aa, ab,
    ba, bb\}\), the set of all strings of \(a\)'s and \(b\)'s of
    length two. Another regular expression for the set is \(aa\) \disj
    \(ab\) \disj \(ba\) \disj \(bb\);

  \item the regular expression \(a\)\kleene{} denotes the set of all
    strings of zero or more \(a\)'s, i.e. \(\{\varepsilon, a, aa, aaa,
    \dots\}\);

  \item the regular expression \lparen\(a\) \disj
    \(b\)\rparen\kleene{} denotes the set of all strings containing
    zero of more instances of an \(a\) or \(b\), that is the language
    of all words made of \(a\)'s and \(b\)'s. Another expression is
    \lparen\(a\)\kleene\(b\)\kleene\rparen\kleene.

\end{itemize*}
If two regular expressions \(r\)~and~\(s\) denote the same language,
we say that \(r\)~and~\(s\) are \emph{equivalent} and write \(r =
s\). In \fig~\vref{fig:laws},
\begin{figure}
\centering
\begin{tabular}{c|l}
\toprule
  \multicolumn{1}{c}{\textsc{Law}}
& \multicolumn{1}{c}{\textsc{Description}}\\
\midrule
  \(r\) \disj \(s\) = \(s\) \disj \(r\)
& \disj is commutative\\
\hline
  \(r\) \disj \lparen\(s\) \disj \(t\)\rparen{}
  = \lparen\(r\) \disj \(s\)\rparen{} \disj \(t\)
& \disj is associative\\
\hline
  \lparen\(rs\)\rparen \(t\) = \(r\)\lparen\(st\)\rparen
& concatenation is associative\\
\hline
  \(r\)\lparen\(s\) \disj \(t\)\rparen{} = \(rs\) \disj \(rt\)
& concatenation distributes over \disj\\
  \lparen\(s\) \disj \(t\)\rparen \(r\) = \(sr\) \disj \(tr\)
&\\
\hline
  \(\epsilon r = r\)
& \(\epsilon\) is the identity element\\
  \(r \epsilon = r\)
& for the concatenation\\
\hline
  \(r\)\kleene\kleene = \(r\)\kleene
& Kleene closure is idempotent\\
\hline
  \(r\)\kleene = \(r\)\plus \disj \(\epsilon\)
& Kleene closure and positive closure\\
  \(r\)\plus = \(r r\)\kleene
& are closely linked\\
\bottomrule
\end{tabular}
\caption{Algebraic laws on regular languages\label{fig:laws}}
\end{figure}
we show useful algebraic laws on regular languages.

\subsection*{Regular definitions}

It is convenient to give names to regular expressions and define new
regular expressions using these names as if they were symbols. If
\(\Sigma\) is an alphabet, then a \emph{regular definition} is a
series of definitions of the form
\begin{align*}
    d_1 &\rightarrow r_1\\
    d_2 &\rightarrow r_2\\
    &\cdots\\
    d_n &\rightarrow r_n
\end{align*}
where each \(d_i\) is a distinct name and each \(r_i\) is a regular
expression over the alphabet \(\Sigma \cup \{d_1, d_2, \dots,
d_{i-1}\}\), \emph{i.e.,} the basic symbols and the previously defined
names. The restriction to \(d_j\) such \(j < i\) allows to construct a
regular expression over \(\Sigma\) only by repeatedly replacing all
the names in it. For instance, as we have stated, the set of \Pascal
identifiers can be defined by the regular definitions
\begin{align*}
\term{letter} & \rightarrow \text{\exc{A} \disj \exc{B} \disj
  \ldots \disj \exc{Z} \disj \exc{a} \disj \exc{b} \disj \ldots \disj
  \exc{z}} \\
\term{digit} & \rightarrow \text{\exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{id} & \rightarrow \text{\term{letter} \lparen\term{letter}
  \disj \term{digit}\rparen\kleene}
\intertext{Unsigned numbers in \Pascal are strings like
\texttt{5280}, \texttt{39.37}, \texttt{6.336E4}
or \texttt{1.894E-4}.}
\term{digit} & \rightarrow \text{ \exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{digits} & \rightarrow \text{\term{digit} \term{digit}\kleene}\\
\term{optional\_fraction} & \rightarrow \text{\exc{.} \term{digits}
  \disj} \, \epsilon\\
\term{optional\_exponent} & \rightarrow \text{\lparen\exc{E} \lparen
  \exc{+} \disj \exc{-} \disj} \, \epsilon \, \text{\rparen{}
  \term{digits}\rparen{} \disj} \, \epsilon\\
\term{num} & \rightarrow \text{\term{digits} \term{optional\_fraction}
  \term{optional\_exponent}}
\end{align*}
Certain constructs occur so frequently in regular expressions that it
is convenient to introduce notational shorthands for them:
\begin{itemize}

  \item \emph{Zero or one instance.} The unary operator `\opt{}' means
    `zero or one instance of.' Formally, by definition, if~\(r\) is a
    regular expression then \(r\)\opt = \(r\) \disj \(\epsilon\). In
    other words, \lparen\(r\)\rparen\opt{} denotes the language \(L(r)
    \cup \{\varepsilon\}\).
\begin{align*}
\term{digit} & \rightarrow \text{\exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{digits} & \rightarrow \text{\term{digit}\plus}\\
\term{optional\_fraction} & \rightarrow \text{\lparen\exc{.}
  \term{digits}\rparen\opt}\\
\term{optional\_exponent} & \rightarrow \text{\lparen\exc{E} \lparen
  \exc{+} \disj \exc{-}\rparen\opt{} \term{digits}\rparen\opt}\\
\term{num} & \rightarrow \text{\term{digits} \term{optional\_fraction}
  \term{optional\_exponent}}
\end{align*}

 \item It is also possible to write:
\begin{align*}
\term{digit} & \rightarrow \text{\exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{digits} & \rightarrow \text{\term{digit}\plus}\\
\term{fraction} & \rightarrow \text{\exc{.} \term{digits}}\\
\term{exponent} & \rightarrow \text{\exc{E} \lparen \exc{+} \disj
  \exc{-}\rparen\opt{} \term{digits}}\\
\term{num} & \rightarrow \text{\term{digits} \term{fraction}\opt{}
  \term{exponent}\opt}
\end{align*}

\end{itemize}
If we want to specify the characters `\texttt{?}', `\texttt{*}',
`\texttt{+}', `\texttt{|}', we write them with a preceding backslash,
\emph{e.g.,} `\verb+\?+', or between double-quotes, \emph{e.g.,}
\verb+"?"+. Then, of course, the character double-quote must have a
backslash: \verb+\"+. It is also sometimes useful to match against end
of lines and end of files: \verb+\n+ stands for the control character
`end of line' and \term{\$} is for `end of file'.

\subsection*{Non-regular languages}

Some languages cannot be described by any regular expression. For
example, the language of balanced parentheses cannot be recognised by
any regular expression: \lparen\rparen, \lparen\lparen\rparen\rparen,
\lparen\rparen\lparen\rparen,
\lparen\lparen\lparen\rparen\rparen\lparen\rparen\rparen{}
etc. Another example is the C programming language: it is not a
regular language because it contains embedded blocs between `\verb+{+'
and `\verb+}+'. Therefore, a lexer cannot recognise valid C programs:
we need a parser.

\paragraph{Exercises}

\input{regexp_question_01}
\input{regexp_answer_01}
\input{regexp_question_02}
\input{regexp_answer_02}
\input{regexp_question_03}
\input{regexp_answer_03}


\subsubsection{Snapshots and Transformations to a Frequency Domain}

While it is often useful to know the exact state of the system at any time step $q$, this actually leads to problems in practical implementations:

\begin{description}
  \item[Memory Consumption] Assuming that each value is a complex value with double precision leads to a space consumption of $2\cdot\SI{64}{bit}$ = \SI{128}{bit} or \SI{16}{byte} for every single point of $\mathbf{E}$ and $\mathbf{H}$. A cubic system with a side length of \SI{200}{nm} and a grid size of \SI{0.5}{nm} already has 64 million points which corresponds to about \SI{976}{mb} of memory consumption just for the $\mathbf{E}$ field for a single point in time.
  \item[Mixed $\mathbf{E}$ and $\mathbf{H}$ Fields] If the discretisation $\Delta_t$ is not corresponding to the period of the electromagnetic wave there will be a mixture of $\mathbf{E}$ and $\mathbf{H}$ at any given timestep $q$ which is undesired in most cases because only the $\mathbf{E}$ field might be of interest. The same problem arises for the spatial propagation of the wave.
\end{description}

To solve these problems most implementations of FDTD transform the electric field $\mathbf{E}(\mathbf{x},t)$ with a Fourier transformation to $\mathbf{E}(\mathbf{x}, \nu)$ with $\nu$ being the frequency of the electric field. In case only a specific frequency $\nu$ is of interest all other values can already be discarded while the simulation is running which is greatly reducing the memory needed to perform a calculation.

This also solves the problem of mixed fields: Over many time steps $q$ the calculated $\mathbf{E}$ values will average out the error that would be present by just using a single iteration.


% !TEX root = index.tex

\section{Vector spaces}

\subsection{Motivation}
\label{section:VectorSpaces}
A \emph{scalar} is a real number.
A (column) \emph{vector} $\vec{v}$ of size $n$ is a column of $n$ scalars
\begin{align*}
  \vec{v} =
  \begin{bmatrix}
    v_1 \\
    v_2 \\
    \vdots \\
    v_n
  \end{bmatrix}
\end{align*}
The scalars $v_i$ are called the \textit{coordinates} of $\vec{v}$.
The set of all (column) vectors is called the \emph{Euclidean space} of dimension $n$ and is denoted by $\bbr^n$.
Denote by $\vec{0}$ the vector with all coordinates 0.
$\vec{e}_i$ is the vector whose $i^{th}$ coordinate is 1 and all other coordinates are 0.
The vector $\vec{e}_i$ is called the \emph{$i^{th}$ standard basis vector} and the collection $\calb= \set{\vec{e}_1, \dots, \vec{e}_n}$ is called the \emph{standard basis}.

The spaces $\bbr^1$, $\bbr^2$, $\bbr^3$ are very commonly studied objects, but unfortunately, most of us are unable to visualize the higher dimensions.
Linear algebra is the language that lets us do exactly this.

\begin{figure}[H]
  \centering
  \begin{subfigure}[b]{0.45\textwidth}
    \input{images/r1.tex}
    \caption*{$\bbr^1$ is the real line,}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.45\textwidth}
    \input{images/r2.tex}
    \caption*{$\bbr^2$ is the Euclidean plane,}
  \end{subfigure}
\end{figure}
\begin{figure}[H]
  \centering
  \begin{subfigure}[b]{0.45\textwidth}
    \input{images/r3.tex}
    \caption*{$\bbr^3$ is the three dimensional space,}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.45\textwidth}
    {\Huge \begin{align*}
      ?
    \end{align*}}
    \caption*{What does $\bbr^n$ look like?}
  \end{subfigure}
\end{figure}

There are two operations we can perform on vectors in $\bbr^n$:
\begin{enumerate}
  \item\textbf{Scalar multiplication:} Given a scalar $c$ and a vector $\vec{v}$, the vector $c \vec{v}$ is the vector obtained by multiplying every coordinate of $\vec{v}$ by $c$.

  Geometrically, this is scaling the vector $\vec{v}$ by a factor of $c$.\footnote{If $c$ is negative, we ``flip'' $\vec{v}$ across the origin.}

  \item\textbf{Addition:} Given two vectors $\vec{v}$, $\vec{w}$, the sum $\vec{v} + \vec{w}$ is the vector obtained by adding the respective coordinates.

  Geometrically, vector addition is given by the Parallelogram Law.
\end{enumerate}
\begin{qbox}
  \label{q:parallelogramLaw}
    Let $\vec{v} = \begin{bmatrix}v_1 \\ v_2\end{bmatrix}$ and $\vec{w} = \begin{bmatrix}w_1 \\ w_2\end{bmatrix}$ be vectors in $\bbr^2$.
    Verify that
    \begin{align*}
      \vec{0}, \: \vec{v}, \: \vec{v} + \vec{w}, \: \vec{w}
    \end{align*}
    form the vertices of a parallelogram.\tablefootnote{Here we are thinking of vectors as points. It can be a bit confusing at first, to switch between points and arrows, but this flexibility is very useful once you get used to it.}
    This is called the \textit{parallelogram law of vector addition}. The parallelogram law holds even for $\bbr^n$.
    \begin{figure}[H]
      \input{images/ParallelogramLaw.tex}
      \caption{Parallelogram law of vector addition.}
    \end{figure}
\end{qbox}

\subsubsection{Lines and planes in $\bbr^3$}
A line in $\bbr^2$ is given by an equation $y = mx + c$. What about a line in $\bbr^3$?
One way to decribe a line in $\bbr^3$ is using vector notation.
\begin{ex}
  \label{example:lineAsSpan}
  Every line $L$ in $\bbr^3$ passing through the origin can be written as
  \begin{align*}
    L = \set{c \vec{v} : c \in \bbr}
  \end{align*}
  where $\vec{v}$ is a vector in $\bbr^3$.
  We can then \emph{define} a line passing through the origin in $\bbr^n$ as $\set{c \vec{v} : c \in \bbr}$ for some $\vec{v} \in \bbr^n$.
\end{ex}

What about a plane? We needed one vector to describe a line. We will need two to describe a plane.

\begin{qbox}
  \label{q:planeAsSpan}
    Let $\vec{v}_1$ and $\vec{v}_2$ be non-zero vectors in $\bbr^2$ which are not scalar multiples of each other.
    Using the Parallelogram Law from Q. \ref{q:parallelogramLaw} , argue that for every vector $\vec{v}$ in $\bbr^2$ there exist scalars $c_1, c_2$ such that \begin{equation*}
      \vec{v} = c_1 \vec{v}_1 + c_2 \vec{v}_2.
  \end{equation*}
    \begin{figure}[H]
      \input{images/planeBasis.tex}
    \end{figure}
    So that we can write $\bbr^2$ as
    \begin{align*}
      \bbr^2 = \set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}
    \end{align*}
\end{qbox}
\begin{qbox}
  What is the set $\set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}$, if $\vec{v}_1$ is a scalar multiple of $\vec{v}_2$, for $\vec{v}_1$, $\vec{v}_2 \in \bbr^2$.
\end{qbox}

One can similarly check that if $\vec{v}_1$ and $\vec{v}_2$ are non-zero vectors in $\bbr^3$ which are not scalar multiples of each other, the set $\set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}$ describes planes passing through the origin.
We then \emph{define} a plane in $\bbr^n$ passing through the origin, to be a set of the form $\set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}$.

\begin{qbox}
  How would you define lines and planes in $\bbr^n$ not necessarily passing through the origin (using vector notation)?
\end{qbox}

Our goal then is to generalize this entire story to higher dimensions, by defining ``\emph{$m$ dimensional subspaces of $\bbr^n$}''.
We'll first define a subspace, then slowly move toward defining dimension\footnote{Which is a surprisingly difficult thing to define!}.

\subsection{Subspaces}
\begin{definition}
  A \emph{subspace} of $\bbr^n$ is a \emph{non-empty} subset $V \subseteq \bbr^n$ satisfying the following conditions.
  \begin{enumerate}
    \item (closed under scalar multiplication) For every real number $c$ and vector $\vec{v}$ in $V$, the vector $c \vec{v}$ is in $V$.
    \item (closed under addition) For every $\vec{v}$ and $\vec{w}$ in $V$, the vector $\vec{v} + \vec{w}$ is in $V$.
  \end{enumerate}
\end{definition}
Subspaces, and more generally abstract vector spaces, are the primary objects of study in linear algebra.

\begin{qbox}
  \begin{enumerate}
    \item Show that $\bbr^n$ is a subspace of $\bbr^n$.
    \item Show that the set $V = \{\vec{0}\}$ is a subspace of $\bbr^n$.
    \item For any subspace $V \subseteq \bbr^n$, show that the vector $\vec{0}$ is in $V$.
  \end{enumerate}
\end{qbox}

\begin{qbox}
  Determine, with proof, all the subspaces of $\bbr^1$.
\end{qbox}

We already know two subspaces of $\bbr^2$: $\set{0}$ and $\bbr^2$.
$\bbr^2$ has one other family of subspaces given by lines.
\begin{qbox}
  When is the line in $\bbr^2$ a subspace of $\bbr^2$? What about $\bbr^3$?
\end{qbox}

\begin{qbox}
  When is a plane in $\bbr^3$ a subspace of $\bbr^3$?
\end{qbox}

\begin{qbox}
  Make a guess as to what all the subspaces of $\bbr^2$ and $\bbr^3$ are. How would you prove this?\tablefootnote{We will give a rigorous proof of this tomorrow.}
\end{qbox}

\begin{qbox}
  Let $V$ and $W$ be subspaces of $\bbr^n$.
  \begin{enumerate}
    \item Show that the intersection $V \cap W$ is also a subspace.
    \item What can you say about the union $V \cup W$?
  \end{enumerate}
\end{qbox}

\subsection{Span, Linear Independence, and Basis}
$\spn$ is a general procedure for constructing subspaces of $\bbr^n$. It generalizes the concepts in Example \ref{example:lineAsSpan} and Q. \ref{q:planeAsSpan}.

\begin{definition}
  For a subset $S$ of  $\bbr^n$, the span of $S$ is defined to be the set of finite \emph{linear combinations} of elements of $S$.
  \begin{align*}
    \spn(S) := \{ c_1 \vec{v}_1 + \dots + c_k \vec{v}_k \quad : \quad
    & c_1, \dots, c_k \mbox{ are real numbers}, \\
    & \vec{v}_1, \dots, \vec{v}_k \mbox{ are vectors in } S \}
  \end{align*}
  For the empty set $ \varnothing \subseteq \bbr^n$ we define $\spn(\varnothing):=\{\vec{0}\}$.
  We think of $S$ as being a \emph{generating set} of the subspace $\spn(S)$.
\end{definition}

\begin{qbox}
  Let $S$ be a subset of $\bbr^n$. Show that $\spn(S)$ is a subspace of $\bbr^n$.
\end{qbox}

\begin{qbox}[Practice problems]
  \label{q:span}
  Find simple descriptions of $\spn$s of the following subsets of $\bbr^3$.
  \begin{multicols}{2}
    \begin{enumerate}
      \item $\set{ \vec{e_1}}$
      \item $\set{\vec{e}_1, \vec{e}_{2}}$
      \item $\set{\vec{e}_1, \vec{e}_2 , \vec{e}_3}$
      \item $\bbr^3$
      % \item $\set{\begin{bmatrix} 1 \\ 0 \\ 0\end{bmatrix}, \begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}}$
      \item $\set{\vec{e}_1,\vec{e}_1 + \vec{e}_2, \vec{e}_3}$
      \item $\set{\vec{e}_1-\vec{e}_2, \vec{e}_2-\vec{e}_3, \vec{e}_3-\vec{e}_1}$
      \item The plane $z = 0$
      \item The plane $z = 1$
      % \item $\set{\begin{bmatrix} 1 \\ 0 \\ 0\end{bmatrix}, \begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 2 \\ 0\end{bmatrix}}$
    \end{enumerate}
  \end{multicols}
\end{qbox}

\begin{qbox}
  Let $S$ be a subset of $\bbr^n$ and $V$ be a subspace of $\bbr^n$.
  Show that
  \begin{equation*}
    \mbox{if } S \subseteq V \mbox{ then }\spn(S) \subseteq V.
  \end{equation*}
  Thus $\spn(S)$ is the \emph{smallest vector space} containing $S$.
\end{qbox}

The notion of $\spn$ allows for redundancies. For example, the set $\set{\vec{e}_1, \vec{e}_2 , \vec{e}_3}$ has 3 elements and the set $\bbr^3$ has infinitely many, but both of these sets have the same $\spn$.
This redundancy is precisely captured by \emph{linear dependence}.

\begin{definition}
  \label{def:linearIndependence}
  A finite set of vectors $\cals = \set{\vec {v}_1, \dots, \vec{v}_k} \subseteq \bbr^n$ is \emph{linearly dependent} if there exist real numbers $c_1, \dots, c_k$, not all 0, satisfying
  \begin{equation*}
    c_1 \vec{v}_1 + \dots + c_k \vec{v}_k = 0.
  \end{equation*}
  $\cals$ is called \emph{linearly independent} otherwise i.e. $\cals$ is \emph{linearly independent} if the only real numbers $c_1, \dots, c_k$ for which
  \begin{equation*}
    c_1 \vec{v}_1 + \dots + c_k \vec{v}_k = 0.
  \end{equation*}
  are $c_1 = \dots = c_k = 0$.

  An infinite set $\cals \subseteq \bbr^n$ is said to be \emph{linearly dependent} if it contains a finite subset which is linearly dependent, it is called \emph{linearly independent} otherwise.
\end{definition}

\begin{qbox}
  Let $\vec{v}_1$, $\vec{v}_2$ be vectors in $\bbr^n$.
  \begin{enumerate}
    \item When is the set $\cals = \set{\vec{v}_1}$ linearly independent?
    \item When is the set $\cals' = \set{\vec{v}_1, \vec{v}_2}$ linearly independent?
  \end{enumerate}
\end{qbox}

\begin{qbox}
  \sout{Is the empty set $\varnothing \subset \bbr^n$ linearly dependent or independent?}
\end{qbox}

\begin{qbox}
  \begin{enumerate}
    \item Show that a set $\cals$ is linearly independent if and only if every subset of $\cals$ is linearly independent.
    \item Is the statement still true if we replace linear independence with linear dependence?
  \end{enumerate}
\end{qbox}

\begin{definition}
  For a subspace $V$ of $\bbr^n$, a set $\calb \subseteq V$ is said to be a \emph{basis} of $V$ if
  \begin{enumerate}
    \item $\spn(\calb) = V$,
    \item ${\calb}$ is linearly independent.
  \end{enumerate}
\end{definition}

\begin{qbox}
  For each of the sets $\cals$ in Q. \ref{q:span}, find a basis of $\spn(\cals)$.
\end{qbox}

The following theorem is what makes a basis extremely useful.

\begin{theorem}
  \label{theorem:generalizedCoordinates}
  Let $V$ be a subspace of $\bbr^n$ with a basis $\calb = \set{\vec{v}_1, \dots, \vec{v}_k}$.
  For every vector $\vec{v}$ in $V$ there exist unique scalars $c_1, \dots, c_k$ such that
  \begin{align*}
    \vec{v} = c_1 \vec{v}_1 + \dots + c_k \vec{v}_k.
  \end{align*}
\end{theorem}

\begin{qbox}
  Prove Theorem \ref{theorem:generalizedCoordinates}.\hint{First show that the scalars $c_i$ exist. Then argue that they are unique using a proof by contradiction.}
\end{qbox}

\begin{definition}
  Let $V$ be a subspace of $\bbr^n$ with a basis $\calb = \set{\vec{v}_1, \dots, \vec{v}_k}$.
  If $\vec{v} = c_1 \vec{v}_1 + \dots + c_k \vec{v}_k$ as in Theorem \ref{theorem:generalizedCoordinates}, we denote
  \begin{align*}
    \coords{v}{\calb}
    = \begin{bmatrix}
      c_1 \\
      \vdots \\
      c_k
      \end{bmatrix}.
  \end{align*}
  The $c_i$'s are the \emph{coordinates} of $\vec{v}$ in the basis $\calb$.
\end{definition}
Thus if the subspace $V$ has a basis $\calb$ of size $k$ then $V$ ``behaves'' like the Euclidean space $\bbr^k$.
\begin{ex}
  If $\vec{v}$
  is a vector in $\bbr^n$ and $\calb$ is the standard basis, then $\coords{v}{\calb} = \vec{v}$, hence the name \emph{standard basis}.
  The standard basis is a special basis that exists only for $\bbr^n$, for other vector spaces there is usually no natural choice of a basis.
  But we will show that \emph{a} basis always exists.
\end{ex}

\begin{qbox}
    Let $V$ be a subspace of $\bbr^n$ with a basis $\calb = \set{\vec{v}_1, \dots, \vec{v}_k}$.
    Let $\vec{v}$, $\vec{w}$ be vectors in $V$ and let $\alpha$ be a scalar.
    Show that
    \begin{enumerate}
      \item $[\alpha \vec{v}]_\calb = \alpha [\vec{v}]_\calb$
      \item $[\vec{v} + \vec{w}]_\calb = [\vec{v}]_\calb + [\vec{w}]_\calb$
      \item $\vec{v} = \vec{w}$ if and only if $[\vec{v}]_\calb = [\vec{w}]_\calb$.
    \end{enumerate}
\end{qbox}

\begin{qbox}[Practice problems]
  Find $\coords{v}{\calb}$ of the vector $\vec{v} = \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix} \in V$ for $V$ and $\calb$ as below.
    \begin{enumerate}
      \item $V = \bbr^3$, $\calb = \set{\vec{e}_1, \vec{e}_2, \vec{e}_3}$
      \item $V = \bbr^3$, $\calb = \set{\vec{e}_1, \vec{e}_3, \vec{e}_2}$
      \item $V = \bbr^3$, $\calb = \set{\vec{e}_1 + \vec{e}_2, \vec{e}_2, \vec{e}_3}$
      \item $V = \set{z = 0}$, $\calb = \set{\vec{e}_1, \vec{e}_2}$
      \item $V = \set{x + y + z = 0}$, $\calb = \set{\vec{e}_1 - \vec{e}_3, \vec{e}_2 - \vec{e}_3}$
      \item $V = \spn\set{\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}}$, $\calb = \set{\begin{bmatrix} 2 \\ -2 \\ 0 \end{bmatrix}}$
      \end{enumerate}
\end{qbox}

	Generalized coordinates are \emph{intrinsic} to the subspace and do not depend on the ambient vector space.
  However, they are not unique and depend upon the choice of a basis.

  We will show in the next section that the \emph{number} of generalized coordinates, on the other hand, does not depend on the basis. The dimension is then defined to equal this number.

\subsection{Optional: Abstract vector spaces}
We repeatedly used scalar multiplication and addition of vectors in the previous sections.
Everything we did naturally carries over to any structure that has scalar multiplication and addition.

\begin{definition}
  A \emph{vector space} over the real numbers is any set $V$ which has addition and scalar multiplication.
\end{definition}
\begin{ex}
  A subspace of $\bbr^n$ is an example of a vector space.
\end{ex}

\begin{qbox}
  Verify that the following are vector spaces.
  \begin{enumerate}
    \item Set of polynomials in a single variable $x$.
    \begin{align*}
      \mathrm{Poly} = \set{a_0 + a_1 x + \dots + a_n x^n \mid n \in \bbz_{\ge 0}, a_i \in \bbr}
    \end{align*}
    \item Set of polynomials $\mathrm{Poly}_n$ in a single variable $x$ of degree $\le n$.
    \begin{equation*}
      \mathrm{Poly}_n = \set{ a_0 + \dots + a_k x^k \mid k \le n, a_i \in \bbr}.
    \end{equation*}
    \item Set of functions $f: \bbr \rightarrow \bbr$.
    \begin{align*}
      \Maps(\bbr,\bbr) = \set{f:\bbr \rightarrow \bbr}
    \end{align*}
    \item Set of functions $f: \bbr \rightarrow \bbr$ with $f(0) = 0$.
    \begin{align*}
      \Maps_0(\bbr,\bbr) = \set{f:\bbr \rightarrow \bbr}
    \end{align*}
  \end{enumerate}
\end{qbox}

\begin{qbox}
  The set of polynomials in a single variable $x$ of degree $= n$ is \emph{not} a vector space. Why?
\end{qbox}

The notion of a subspace naturally extends to abstract vector spaces: $\mathrm{Poly}_n$ is a subspace of $\mathrm{Poly}$ and $\Maps_0(\bbr,\bbr)$ is a subspace of $\Maps(\bbr,\bbr)$.
The notions of spans, linear independence, and basis also naturally generalize.

\begin{qbox}
  Find bases for $\mathrm{Poly}_n$ and $\mathrm{Poly}$.
\end{qbox}

\begin{qbox}
  Can you find a basis for $\Maps(\bbr,\bbr)$? Are you sure?
\end{qbox}


\section{Subspaces}
\label{sec:vector-space-subspaces}

\begin{outcome}
  \begin{enumerate}
  \item Determine whether a set of vectors is a subspace of a given
    vector space.
  \item Determine whether two sets of vectors span the same subspace.
  \end{enumerate}
\end{outcome}

In this section we will consider subspaces of general vector spaces.

\begin{definition}{Subspace}{subspace-vector-space}
  Let $V$ be a vector space over a field $K$. A subset $W\subseteq V$
  is said to be a \textbf{subspace}%
  \index{subspace}%
  \index{subspace!of vector space} of\/ $V$ if the following conditions hold:
  \begin{enumerate}
  \item $\vect{0}\in W$, where $\vect{0}$ is the additive unit of\/ $V$.
  \item $W$ is \textbf{closed under addition}%
    \index{closed under!addition}%
    \index{addition!closed under}: Whenever $\vect{u}, \vect{v}\in W$,
    then $\vect{u} + \vect{v}\in W$.
  \item $W$ is \textbf{closed under scalar multiplication}%
    \index{closed under!scalar multiplication}%
    \index{scalar multiplication!closed under}: Whenever $k\in K$ and
    $\vect{u}\in W$, then $k\vect{u}\in W$.
  \end{enumerate}
\end{definition}

\begin{example}{Subspaces of $\R^3$}{subspaces-r3}
  As we have seen in Section~\ref{sec:subspaces-rn}, the subspaces of
  $\R^3$ are:
  \begin{itemize}
  \item the zero subspace%
    \index{zero subspace} $\set{\vect{0}}$;
  \item lines through the origin;
  \item planes through the origin;
  \item $\R^3$ itself.
  \end{itemize}
\end{example}

\begin{example}{Space of continuous functions}{subspace-continuous}
  Let $V = \Func_{\R,\R}$, the vector space of functions from real numbers to
  real numbers. Let $W\subseteq V$ be the subset of {\em continuous}
  functions%
  \index{continuous function}%
  \index{function!continuous}. Then $W$ is a subspace of\/ $V$.
\end{example}

\begin{proof}
  We know from calculus that:
  \begin{enumerate}
  \item the zero function, defined by $f(x)=0$ for all $x$, is
    continuous;
  \item if $f,g$ are continuous functions, then $f+g$ is
    continuous;
  \item if $f$ is a continuous function and $k$ a constant, then $kf$
    is continuous.
  \end{enumerate}
  It follows that $W$ contains $0$, and is closed under addition and
  scalar multiplication. Therefore, $W$ is a subspace of\/ $V$.
\end{proof}

\begin{example}{Space of differentiable functions}{subspace-differentiable}
  Let $V = \Func_{\R,\R}$. Recall from calculus that a function
  $f:\R\to\R$ is called \textbf{differentiable}%
  \index{differentiable function}%
  \index{function!differentiable} if the derivative $f'(x)$ exists for
  all $x\in\R$. Let $W\subseteq V$ be the subset of differentiable
  functions. Then $W$ is a subspace of\/ $V$.
\end{example}

\begin{proof}
  We know from calculus that:
  \begin{enumerate}
  \item The zero function, defined by $f(x)=0$ for all $x$, is
    differentiable. In fact, its derivative is $f'(x) = 0$.
  \item If $f,g$ are differentiable functions, then $h=f+g$ is
    differentiable. In fact, $h'(x) = f'(x) + g'(x)$.
  \item if $f$ is a differentiable function and $k$ a constant, then
    $h=kf$ is differentiable. In fact, $h' = kf'$.
  \end{enumerate}
  It follows that $W$ contains $0$, and is closed under addition and
  scalar multiplication. Therefore, $W$ is a subspace of\/ $V$.
\end{proof}

\begin{example}{Space of sequences satisfying a linear recurrence}{subspace-recurrence}
  Let $V=\Seq_{\R}$, the vector space of sequences of real
  numbers. Let
  \begin{equation*}
    W = \set{a \in \Seq_{\R} \mid \mbox{for all $n\geq 0$, $a_{n+2}=a_n+a_{n+1}$}}.
  \end{equation*}
  In other words, $W$ is the set of all sequences satisfying the
  recurrence%
  \index{recurrence!as a subspace} relation
  $a_{n+2}=a_n+a_{n+1}$. Then $W$ is a subspace of\/ $V$.
\end{example}

\begin{proof}
  \def\x#1{\makebox[1.2em][r]{$#1$}}
  \def\y#1{\makebox[3em][r]{$#1$}}
  \def\z#1{\makebox[0em][l]{$#1$}}
  Before we prove that $W$ is a subspace, let us first consider an
  example. The following sequences are elements of\/ $W$, because they
  both satisfy the recurrence:
  \begin{equation*}
    \begin{array}{rc@{}l}
      \y{a} &=& \x{(1},\x{1},\x{2},\x{3},\x{5},\x{8},\x{13},\x{21},\x{\ldots})\z{\,,} \\
      \y{b} &=& \x{(1},\x{3},\x{4},\x{7},\x{11},\x{18},\x{29},\x{47},\x{\ldots})\z{\,.}
    \end{array}
  \end{equation*}
  Note that if we add these sequences, we get
  \begin{equation*}
    \begin{array}{rc@{}l}
      \y{a+b} &=& \x{(2},\x{4},\x{6},\x{10},\x{16},\x{26},\x{42},\x{68},\x{\ldots})\z{\,,} \\
    \end{array}
  \end{equation*}
  which again satisfies the recurrence. Therefore, the set $W$ is
  closed under the addition of these particular sequences $a$ and
  $b$. We now prove the properties in general.
  \begin{enumerate}
  \item Let $z$ be the zero sequence%
    \index{zero sequence}%
    \index{sequence!zero sequence}, defined by $z_n=0$ for all $n$.
    Then $z$ satisfies the recurrence relation, since for all $n\geq 0$,
    $z_{n+2}=0=z_n+z_{n+1}$. Therefore $z\in W$.
  \item To show that $W$ is closed under addition, consider any two
    sequences $a,b\in W$, and let $c=a+b$. Then for all $n\geq 0$,
    \begin{equation*}
      c_{n+2}
      = a_{n+2} + b_{n+2}
      = (a_n + a_{n+1}) + (b_n + b_{n+1})
      = (a_n + b_n) + (a_{n+1} + b_{n+1})
      = c_n + c_{n+1},
    \end{equation*}
    so $c$ satisfies the recurrence. It follows that $c\in W$, and
    therefore $W$ is closed under addition.
  \item To show that $W$ is closed under scalar multiplication,
    consider any $k\in\R$ and $a\in W$, and let $c=ka$. Then for all
    $n\geq 0$,
    \begin{equation*}
      c_{n+2}
      = ka_{n+2}
      = k(a_n + a_{n+1})
      = ka_n + ka_{n+1}
      = c_n + c_{n+1},
    \end{equation*}
    so $c$ satisfies the recurrence. It follows that $c\in W$, and
    therefore $W$ is closed under scalar multiplication.
  \end{enumerate}
\end{proof}

\begin{example}{Solution space of a linear differential equation}{subspace-differential-equation}
  Let $V=\Func_{\R,\R}$. Recall from calculus that a
  \textbf{differential equation}%
  \index{differential equation} is an equation about an unknown
  function and its derivatives. For example
  \begin{equation*}
    f'' = -f
  \end{equation*}
  is a differential equation. The functions $f(x)=\sin x$,
  $f(x)=\cos x$, and $f(x)=0$ are examples of solutions of this
  differential equation.  Let $W$ be the set of all functions that are
  solutions of the differential equation $f'' = -f$. Then $W$ is a
  subspace of\/ $V$.
\end{example}

\begin{proof}
  \begin{enumerate}
  \item The zero function $f(x)=0$ is a solution of the differential
    equation, and therefore an element of\/ $W$.
  \item To show that $W$ is closed under addition, let $f,g\in W$ and
    consider $h=f+g$. Then $f''=-f$ and $g''=-g$, and therefore
    $h'' = f'' + g'' = -f+(-g) = -h$. Therefore, $h\in W$, and $W$ is
    closed under addition.
  \item To show that $W$ is closed under scalar multiplication, let
    $k\in\R$ and $f\in W$, and consider $h=kf$. Then $f''=-f$, and
    therefore $h'' = kf'' = k(-f) = -h$. It follows that $h\in W$, and
    therefore $W$ is closed under scalar multiplication.
  \end{enumerate}
\end{proof}

\begin{example}{Subspace of polynomials}{polynomial-subspace}
  Consider $\Poly_2$, the vector space of polynomials of degree at
  most 2, with coefficients in a field $K$. Fix some element $r\in K$,
  and let $W \subseteq \Poly_2$ be the subset of polynomials that have
  $r$ as a root. Then $W$ is a subspace of $\Poly_2$.
\end{example}

\begin{proof}
  We can express $W$ as follows:
  \begin{equation*}
    W = \set{p\in\Poly_2 \mid p(r)  = 0}.
  \end{equation*}
  We need to show that $W$ is a subspace.
  \begin{enumerate}
  \item The zero polynomial, given by $0(x)=0$, satisfies $0(r)=0$, so
    $0\in W$.
  \item To show that $W$ is closed under addition, assume
    $p, q\in W$, and let $s=p+q$. Then $p(r)=0$ and $q(r)=0$,
    therefore $s(r) = p(r)+q(r) = 0$. It follows that $s\in W$.
  \item To show that $W$ is closed under scalar multiplication, assume
    $p(x)\in W$ and $k$ be a scalar. Then $(kp)(r)=k(p(r))=k0=0$, and
    therefore $kp\in W$.
  \end{enumerate}
\end{proof}

\begin{example}{Trivial subspaces of\/ $V$}{trivial-subspaces}
  Let $V$ be an arbitrary vector space over a field $K$. Then
  $\set{\vect{0}}$ is a subspace of\/ $V$, called the \textbf{zero
    subspace}. Also, $V$ is a subspace of itself.
\end{example}

\begin{proof}
  Clearly $\set{\vect{0}}$ contains $\vect{0}$, and is closed under
  addition and scalar multiplication because
  $\vect{0}+\vect{0}=\vect{0}$ and $k\vect{0}=\vect{0}$ for all $k$.
  Similarly, $V$ contains $\vect{0}$ and is closed under addition and
  scalar multiplication, because addition and scalar multiplication
  are operations on $V$.  Therefore, both $\set{\vect{0}}$ and $V$ are
  subspaces of\/ $V$.
\end{proof}

The interest of subspaces lies in the fact that they are vector spaces
in their own right, as stated in the following proposition.

\begin{proposition}{Subspaces are vector spaces}{subspaces-are-vector-spaces}
  Let $W$ be a subspace of a vector space $V$. Then $W$ satisfies the
  vector space axioms (A1)--(A4) and (SM1)--(SM4), with respect to the
  same operations (addition and scalar multiplication) as those
  defined on $V$.
\end{proposition}

\begin{proof}
  Since $W$ is a subspace, it is closed under addition and scalar
  multiplication. This ensures that addition and scalar multiplication
  are well-defined operations on $W$. The axioms (A1), (A2), (A4), and
  (SM1)--(SM4) all obviously hold in $W$, because they hold in $V$
  (two elements of\/ $W$ are equal in $W$ if and only if they are equal
  in $V$). The axiom (A3) holds because $\vect{0}\in W$.
\end{proof}

We end this section with an observation about spans.

\begin{proposition}{Span is smallest subspace containing given vectors}{span-smallest-subspace}
  Let $V$ be a vector space over some field $K$, and consider a set of
  vectors $S\subseteq V$. Then $\sspan S$ is the smallest subspace of\/
  $V$ containing $S$. More explicitly, we have:
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item The set $\sspan S$ is a subspace of\/ $V$, and
      $S\subseteq \sspan S$.
    \item If $W$ is any other subspace of\/ $V$ such that
      $S\subseteq W$, then $\sspan S\subseteq W$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proposition}

\begin{proof}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item To show that $\sspan S$ is a subspace, first note that
      $\vect{0}\in\sspan S$, because $\vect{0}$ is the empty linear
      combination. Also, if $\vect{v},\vect{u}\in\sspan S$, then by
      definition of span, there exist
      $\vect{v}_1,\ldots,\vect{v}_k,\vect{u}_1,\ldots,\vect{u}_{\ell}\in
      S$ and $a_1,\ldots,a_k,b_1,\ldots,b_{\ell}\in K$ such that
      \begin{eqnarray*}
        \vect{v} &=& a_1\vect{v}_1 + \ldots + a_k\vect{v}_k, \\
        \vect{u} &=& b_1\vect{u}_1 + \ldots + b_{\ell}\vect{u}_{\ell}.
      \end{eqnarray*}
      Then
      \begin{eqnarray*}
        \vect{v}+\vect{u}
        &=& a_1\vect{v}_1 + \ldots + a_k\vect{v}_k
            + b_1\vect{u}_1 + \ldots + b_{\ell}\vect{u},
      \end{eqnarray*}
      and therefore $\vect{v}+\vect{u}\in\sspan S$. It follows that
      $\sspan S$ is closed under addition. The proof for scalar
      multiplication is similar. Finally, every $\vect{v}\in S$ is
      trivially a linear combination of itself, $\vect{v}=1\vect{v}$,
      and therefore $S\subseteq \sspan S$.
    \item Consider any other subspace $W$ of $V$ such that
      $S\subseteq W$. To show that $\sspan S\subseteq W$, consider an
      arbitrary element $\vect{v}\in\sspan S$. By definition of span,
      there exist $\vect{v}_1,\ldots,\vect{v}_k\in S$ and
      $a_1,\ldots,a_k\in K$ such that
      $\vect{v}=a_1\vect{v}_1 + \ldots + a_k\vect{v}_k$. By
      assumption, $\vect{v}_1,\ldots,\vect{v}_k\in W$. Since $W$ is
      closed under addition and scalar multiplication, it follows that
      $\vect{v}\in W$. Since $\vect{v}$ was an arbitrary element of
      $\sspan S$, it follows that $\sspan S\subseteq W$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proof}

While the last proposition looks technical, it can actually be useful
for proving that two sets of vectors span the same subspace. The
following in an example of this.

\begin{example}{Equal spans}{equal-span}
  Show that the sets $S=\set{x^2-2x+1,~ x-1}$ and
  $T=\set{x^2-1,~ x^2-x}$ span the same subspace of $\Poly_2$. In other
  words, show that
  \begin{equation*}
    \sspan S = \sspan T.
  \end{equation*}
\end{example}

\begin{solution}
  To show that two sets are equal, we must show that each is a subset
  of the other. So we will show $\sspan S \subseteq \sspan T$ and
  $\sspan T \subseteq \sspan S$. By
  Proposition~\ref{prop:span-smallest-subspace}, it is sufficient to show that
  $S\subseteq \sspan T$ and $T\subseteq \sspan S$, i.e., we must show
  that every element of $S$ is a linear combination of elements of $T$
  and vice versa.
  \begin{enumerate}
  \item $S \subseteq \sspan T$. We have
    \begin{equation*}
      \begin{array}{rclcl}
      x^2-2x+1 &=& (-1)(x^2-1) + 2(x^2-x) &\in& \sspan T, \\
      x-1 &=& 1(x^2-1) - 1(x^2-x) &\in& \sspan T.
      \end{array}
    \end{equation*}
    Since each element of $S$ is an element of $\sspan T$, it follows
    that $S\subseteq\sspan T$. By
    Proposition~\ref{prop:span-smallest-subspace}, this implies that
    $\sspan S\subseteq \sspan T$.
  \item $T \subseteq \sspan S$. We have
    \begin{equation*}
      \begin{array}{rclcl}
        x^2-1 &=& 1(x^2-2x+1) - 2(x-1) &\in& \sspan S, \\
        x^2-x &=& 1(x^2-2x+1) - 1(x-1) &\in& \sspan S.
      \end{array}
    \end{equation*}
    Since each element of $T$ is an element of $\sspan S$, it follows
    that $T\subseteq\sspan S$. By
    Proposition~\ref{prop:span-smallest-subspace}, this implies that
    $\sspan T\subseteq \sspan S$.
  \end{enumerate}
\end{solution}