\subsection{From Langevin dynamics to Fokker-Planck equation}

Now that we built a description for the probability distribution over the
ensemble of trajectories that a Markovian stochastic process can follow we are
left with the task of how to define the Fokker-Plank equation coefficients from
the stochastic dynamics defined by the Langevin equation. Specifically in its
more general form we derived \eref{eq_fokker_planck} to depend on three things:
the probability distribution of allele frequencies $P(f, t)$, and the two jump
moments $a^{(1)}(f, t)$ and $a^{(2)}(f, t)$, where these moments are given by
\eref{eq_jump_mom}. A lot of the population genetics literature use a
non-mathematical argument to directly assign the values of these coefficients
given the either deterministic or stochastic nature of the terms. For example,
given the dynamics that we defined in \secref{seq_determ_mut} for the
deterministic form of mutation and selection (\eref{eq_sel_mut}) most texts
would directly assign these dynamics to be the first jump moment
$a^{(1)}(f,t)$. There is nothing intrinsically wrong with this approach, since
this is actually the correct answer, but it is not necessarily obvious that
this should be the case especially since \eref{eq_fokker_planck} requires us to
take a derivative with respect to $x$ of this coefficient multiplied by the
allele frequency distribution.

We will take a more formal approach and find how the terms in the Langevin
dynamics that we defined in \secref{sec_langevin_intro} connect to the
coefficients of the Fokker-Planck equation. To do so first we introduce a
quantity defined as
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) \equiv \int_{-\infty}^\infty dx\;
    (x - x')^m P(x, t + \tau \mid x', t), \; \text{for } m \geq 1.
    \label{eq_compute_jump_mom}
\end{equation}
This is nothing else than the average of $[x(t + \tau) - x(t)]^m$ with a sharp
initial condition $x(t) = x'$ for a Markov process that obeys the transition
probability distribution $P(x, t + \tau \mid x', t)$. Given that our Langevin
dynamics are Markov processes, the ensemble of trajectories must obey a
continuous master equation with such transition probability. Therefore
\eref{eq_compute_jump_mom} can be thought as the average displacement between
two time points $t$ and $t + \tau$ for our Langevin dynamics. In other words
$\mathcal{A}^{(m)}(x; \tau, t)$ can be expressed as
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) \equiv
    \ee{\left[x(t + \tau) - x(t)\right]^m \mid x(t) = x'}.
    \label{eq_average_A}
\end{equation}
Just as we did to derive the master equation, for small $\tau$ we can rewrite
the conditional distribution $P(x, t + \tau \mid x', t)$ using
\eref{eq_transition_short_time}. Using this we can rewrite
\eref{eq_compute_jump_mom} as
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) = \int_{-\infty}^\infty dx\; (x - x')^m
    \underbrace{
    \left[ \delta(x - x') \left( 1 - a^{(0)}(x', t) \tau \right) +
    \phi_t(x \mid x')\tau \right]
    }_{P(x, t + \tau \mid x', t)}.
\end{equation}
Distributing the integrals results in
\begin{equation}
    \begin{aligned}
    \mathcal{A}^{(m)}(x; \tau, t) &=
    \int_{-\infty}^\infty dx\; (x - x')^m \delta(x - x') \\
    &- \int_{-\infty}^\infty dx\; (x - x')^m \delta(x - x') a^{(0)}(x', t)
    \tau \\
    &+ \int_{-\infty}^\infty dx\; (x - x')^m \phi_t(x \mid x') \tau.
    \end{aligned}
\end{equation}
For the first two terms we have that the $\delta$-function is one only when $x
= x'$ while the term $(x - x')^m$ is zero for this particular case. Therefore
both of these terms cancel and we are only left with
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) = \int_{-\infty}^\infty dx\; (x - x')^m
    \phi_t(x \mid x') \tau.
\end{equation}
This is non other than the definition of the jump moments times $\tau$ given by
\eref{eq_jump_mom} since we defined $r \equiv x - x'$ and $\phi_t(x'; r) \equiv
\phi_t(x \mid x')$. Now we can see the connection; we started defining
$\mathcal{A}^{(m)}(x; \tau, t)$ to be the average displacement defined by our
Langevin dynamics over a small time interval $\tau$, and we found that this is
connected to the jump moments that define the coefficients of the Fokker-Planck
equation. So we have that
\begin{equation}
    \mathcal{A}^{(m)}(x; \tau, t) = \tau \cdot a^{(m)}(x, t).
\end{equation}
Given this result we can compute the jump moment by taking the derivative of
$\mathcal{A}^{(m)}$ with respect to $\tau$ and evaluating this derivative at
$\tau = 0$. This is
\begin{equation}
    a^{(m)}(x, t) = \left. {\partial \over \partial \tau}
    \mathcal{A}^{(m)}(x; \tau, t) \right\vert_{\tau = 0}.
    \label{eq_A_tau_deriv}
\end{equation}
The reason we evaluate $\tau = 0$ is for convenience. In principle we could
evaluate the derivative for any value of $\tau$ since $a^{(m)}$ does not depend
on $\tau$, but zero is the one that simplifies things the most. Using the
definition of derivatives we then write \eref{eq_A_tau_deriv} as
\begin{equation}
    a^{(m)}(x, t) = \lim_{\Dt \rightarrow 0}
    \left.{\mathcal{A}^{(m)}(x, \tau + \Dt, t) -
    \mathcal{A}^{(m)}(x, \tau, t) \over \Dt} \right\vert_{\tau = 0}.
\end{equation}
If we now substitute the equivalence of $\mathcal{A}^{(m)}$ with the average
displacement on our Langevin dynamics, i.e. \eref{eq_average_A} we obtain
\footnotesize
\begin{equation}
    a^{(m)}(x, t) = \lim_{\Dt \rightarrow 0} {1 \over \Dt}
    \left[
    \ee{\left[x(t + \tau + \Dt) - x(t)\right]^m \mid x(t) = x'} -
    \ee{\left[x(t + \tau) - x(t)\right]^m \mid x(t) = x'}
    \right]_{\tau = 0}.
    \label{eq_deriv_average}
\end{equation}
\normalsize
Now we use the condition $\tau = 0$ that we chose for simplification since the
second term in \eref{eq_deriv_average} cancels. We are then left with a simple
expression of the form
\begin{equation}
    a^{(m)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{\left[x(t + \Dt) - x(t)\right]^m \mid x(t) = x'} \over \Dt}.
\end{equation}
This result shows that given that our Langevin dynamics are a continuous Markov
process, we can compute the displacement dynamics for a small time interval,
elevate it to the $m\tth$ power, and then average it, to then compute the $m$
coefficient for the Fokker-Planck equation. Let's do this first for the first
coefficient $a^{(m)}(x, t)$.

\subsubsection{Directional coefficient of the Fokker-Planck equation}

For the first coefficient of the Fokker-Planck equation that defines the
directional terms of the dynamics we have
\begin{equation}
    a^{(1)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{x(t + \Dt) - x(t) \mid x(t) = x'} \over \Dt}.
\end{equation}
Recall that we defined the Langevin equations as a stochastic differential
equation given in \eref{eq_langevin}. Since $x(t)$ is the solution to this
differential equation we can use the fundamental theorem of calculus to rewrite
the average displacement as
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \ee{\left. \int_t^{t + \Dt} dt'\; {dx \over dt'} \right\vert x(t) = x'}.
\end{equation}
We then substitute \eref{eq_langevin} on the right-hand side to obtain
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \ee{\left.\int_t^{t + \Dt} dt'\; A(x, t') + B(x, t') \xi(t')\right\vert
    x(t) = x'}.
\end{equation}
If we distribute the integral and use the linearity of expected values this
results in
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \int_t^{t + \Dt} dt\; A(x, t') +
    \int_t^{t + \Dt} dt\; B(x, t') \ee{ \left.\xi(t') \right\vert x(t) = x'},
\end{equation}
where the only thing we needed to average over was the noise term $\xi(t)$. For
the Langevin equation we defined the average of this noise term to be zero
regardless of the initial position; that implies that the average displacement
is then simply given by
\begin{equation}
    \ee{x(t + \Dt) - x(t) \mid x(t) = x'} =
    \int_t^{t + \Dt} A(x, t').
\end{equation}
As a consequence the directional coefficient of the Langevin equation is then
of the form
\begin{equation}
    a^{(1)}(x, t) = \lim_{\Dt \rightarrow 0} {1 \over \Dt}
    \int_t^{t + \Dt} A(x, t').
\end{equation}
Since we are taking the limit when $\Dt \rightarrow 0$ we can approximate the
integral simply as
\begin{equation}
    a^{(1)}(x, t) \approx \lim_{\Dt \rightarrow 0}{1 \over \Dt} [A(x, t) \Dt] =
    A(x, t).
    \label{eq_approx_int_directional}
\end{equation}
We therefore mathematically justify the empirical argument commonly found in
the literature that the directional term associated with the Langevin equation
can be directly associated with the directional term of the Fokker-Planck
equation. In other words we showed that
\begin{equation}
    a^{(1)}(x, t) = A(x, t).
\end{equation}

\subsubsection{Diffusive coefficient of the Fokker-Planck equation}

To compute the second coefficient of the Fokker-Planck equation associated with
the diffusive random displacements we can follow the exact same recipe. First
we know that this coefficient is given by
\begin{equation}
    a^{(2)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{\left[x(t + \Dt) - x(t)\right]^2 \mid x(t) = x'} \over \Dt}.
\end{equation}
Again using the fundamental theorem of calculus gives us
\begin{equation}
    a^{(2)}(x, t) = \lim_{\Dt \rightarrow 0}
    {\ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; {dx \over dt'}
    \right]^2 \right\vert x(t) = x'} \over \Dt}.
    \label{eq_diffusive_fp_lim}
\end{equation}
Substituting \eref{eq_langevin} on the expected value on right-hand side gives
\begin{equation}
    \ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; {dx \over dt'}
    \right]^2 \right\vert x(t) = x'} =
    \ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; A(x, t') +
    \int_t^{t + \Dt} dt'\; B(x, t') \xi(t').
    \right]^2 \right\vert x(t) = x'},
\end{equation}
where we already distributed the integral. We now expand the binomial on the
right-hand side and use the linearity of expected values to obtain
\begin{equation}
    \begin{aligned}
    \ee{ \left. \left[
    \int_t^{t + \Dt} dt'\; A(x, t') +
    \int_t^{t + \Dt} dt'\; B(x, t') \xi(t')
    \right]^2 \right\vert x(t) = x'} =
    \left(
    \int_t^{t + \Dt} dt'\; A(x, t')
    \right)^2\\
    + 2
    \int_t^{t + \Dt} dt'\; A(x, t')
    \int_t^{t + \Dt} dt'\; B(x, t') \ee{ \left.\xi(t')
    \right\vert x(t) = x'}\\
    \left(
    \int_t^{t + \Dt} dt'\; B(x, t') \ee{\left.\xi(t')
    \right\vert x(t) = x'}
    \right)^2,
    \end{aligned}
    \label{eq_int_binom}
\end{equation}
where again we used the fact that the only variable we can average over is the
white noise term $\xi(t)$. We can see that the second term on the right-hand
side cancels since the average of our stochastic variable is equal to zero. For
the two terms left we can rewrite the squared integrals as double integrals.
Let's take a look at each of these terms one at the time. First the integral
involving the directional term takes the form
\begin{equation}
    \left(
        \int_t^{t + \Dt} dt'\; A(x, t')
    \right)^2 =
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    A(x, t') A(x, t'').
\end{equation}
Just as we did for \eref{eq_approx_int_directional} we can use the fact that we
are working at the limit where $\Dt \rightarrow 0$ to approximate the integral
as
\begin{equation}
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    A(x, t') A(x, t'') \approx \left[ A(x, t)  \Dt\right]^2.
    \label{eq_directional_int_approx}
\end{equation}
For the integral on \eref{eq_int_binom} involving the diffusive term we again
rewrite it as a double integral of the form
\begin{equation}
    \left(
    \int_t^{t + \Dt} dt'\; B(x, t') \ee{\left.\xi(t')
    \right\vert x(t) = x'}
    \right)^2 =
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    B(x, t') B(x, t'') \ee{\left. \xi(t') \xi(t'') \right\vert x(t) = x}.
\end{equation}
In \secref{sec_langevin_intro} where we introduced the Langevin equation we
defined the second moment of our noise term as given in \eref{eq_noise_delta}.
Using this we evaluate one of the integrals first obtaining
\begin{equation}
    \int_t^{t + \Dt} dt'\; \int_t^{t + \Dt} dt''\;
    B(x, t') B(x, t'') \ee{\left. \xi(t') \xi(t'') \right\vert x(t) = x} =
    2D \int_t^{t + \Dt} dt'\; B(x, t')^2.
\end{equation}
Finally we can again approximate the integral as
\begin{equation}
    2D \int_t^{t + \Dt} dt'\; B(x, t')^2 \approx 2D B(x, t)^2 \Dt.
    \label{eq_diffusive_int_approx}
\end{equation}
Putting back \eref{eq_directional_int_approx} and
\eref{eq_diffusive_int_approx} into our computation of the second coefficient
of the Fokker-Planck equation in \eref{eq_diffusive_fp_lim} results in
\begin{equation}
    a^{(2)}(x, t) \approx \lim_{\Dt \rightarrow 0} {1 \over \Dt}
    \left[ A(x, t)^2\Dt^2 + 2D B(x, t)^2\Dt \right].
\end{equation}
Once we simplify the corresponding $\Dt$ terms and take the limit, the
directional term goes to zero and we are left with a Fokker-Planck diffusive
coefficient of the form
\begin{equation}
    a^{(2)}(x, t) = 2D B(x, t)^2.
\end{equation}


\section{Exercises in Transformations of Random Variables}\label{S:xsTransformationsOfRVs}
\begin{ExerciseList}
%transformation of RVs
\Exercise
Let $X$ be the outcome of a fair die roll with probability mass function given by
\[
f_X(x) =
\begin{cases}
\frac{1}{6} & \text{ if } x \in \{1,2,3,4,5,6\}\\
0 & \text{ otherwise} \, .
\end{cases}
\]
If $Y = (X-3)^2$ then find the probability mass function of $Y$,  $f_Y (y)$.
\Answer
Using Equation~\eqref{E:PMFOfgOfX}, we can tabulate as follows:

\begin{tabular}{|c|c|c|c|c|}
\hline
$y$ & $0$ & $1$ & $4$ & $9$\\\hline
$f_Y(y)$ & $f_X(3)=\frac{1}{6}$ & $f_X(2)+f_X(4)=\frac{2}{6}$ & $f_X(1)+f_X(5)=\frac{2}{6}$ & $f_X(6)=\frac{1}{6}$\\\hline
\end{tabular}

\Exercise
Given a natural number $n$ as a parameter, i.e., given a parameter $n \in \{1,2,3,\ldots\}$, let $X$ be a discrete uniform random variable on the finite set
$$\mathbb{X}=\{-n,-n+1,\ldots,-1,0,1,\ldots,n-1,n\}$$
i.e.~the probability mass function of $X$ is:
\[
f_X(x;n) =
\begin{cases}
\frac{1}{2n+1} & \text{ if } x \in \mathbb{X} \\
0 & \text{ otherwise}\, .
\end{cases}
\]
Find the probability mass function $f_Y(y;n)$ for $Y=|X|$, the absolute value of $X$.
\Answer
The probability mass function $f_Y(y;n)$ for $Y=|X|$, the absolute value of $X$, comes from applying the formula:
\[
f_Y(y;n) = \sum_{x \in \{ x: g(x)=y\}} f_X(x;n) \enspace ,
\]
as follows:
\[
f_Y(y) =
\begin{cases}
\sum_{x \in \{ x: |x|=0\}} f_X(x;n) = f_X(0;n) = \frac{1}{2n+1} & \text{ if } y =0 \\
 \sum_{x \in \{ x: |x|=y\}} f_X(x;n) = \left( f_X(y;n)+f_X(-y;n) \right) = \frac{2}{2n+1} & \text{ if } y \in \{1,2,\ldots,n\} \\
0 & \text{otherwise}
\end{cases}
\]

\Exercise
If $X$ is a $\geometric(\theta)$ random variable and $Y=\left(\frac{1}{2}\right)^X$ then find an expression for $f_Y(y)$.
\Answer
We are given that $Y=2^{-X}$. Define the function
$$g: \{1,2,3,\ldots\} \to \{2^{-1},2^{-2},2^{-3},\ldots\}$$
by $y=g(x)=2^{-x}$.  Then $g$ is one-to-one and onto and so by Equation~\eqref{E:PMFOfgOfX},
\[
f_Y(y) = \sum_{x \in g^{[-1]}(y)} f_X(x) = \sum_{x \in g^{-1}(y)} f_X(x) = \sum_{x \in \{ -\log_2(y) \} } f_X(x) = f_X(-\log_2(y))\, .
\]
Note that the second equality above is emphasizing that the inverse image $g{[-1]}(y)$ is indeed the inverse function $g^{-1}(y)$ for this $g:\{1,2,3,\ldots\} \to \{2^{-1},2^{-2},2^{-3},\ldots\}$.  Therefore,
\[
f_Y(y) =
\begin{cases}
f_X(-\log_2(y)) = \theta (1-\theta)^{-\log_2(y)-1} & \text{ if } y \in \{2^{-1},2^{-2},2^{-3},\ldots\}\\
0 & \text{ otherwise} \, .
\end{cases}
\]

\Exercise
If $X$ is a $\poisson(\lambda)$ random variable find the probability mass function,  $f_Y (y)$, of
\[
Y\,=\,\frac{1}{(X+1)^{2}} \enspace .
\]
\Answer
Since $X$ is a $\poisson(\lambda)$ random variable (by suppressing the `$;\lambda$' in the argument to $f_X(\cdot)$ for notational ease), we get
\[ f_X(x) \,=\, \P(X=x)\,=\,
\begin{cases}
\frac{\lambda^x \, e^{-\lambda} }{ x!} & \text{ for }  x= 0, 1, 2, \dots \\
0 & \text{ otherwise} \,.
\end{cases}
\]

If $Y=(X+1)^{-2}=1/(X+1)^2$ then  \[ \{\ldots,2,1,0\} \ni x \xmapsto{(x+1)^{-2}} y \,\in\, \left\{1, \frac{1}{4}, \frac{1}{9}, \dots \right\} \]
and since $y=g(x)=(x+1)^{-2}$ {\scriptsize as it maps or associates each $y \in \left\{1, \frac{1}{4}, \frac{1}{9}, \dots \right\}$ to exactly one $x \in \{0,1,2,\ldots\}$ given by $g^{-1}(y)=y^{-1/2}-1 = \frac{1}{\sqrt{y}}-1= x$, its inverse function, this is because $g$ is {\em injective} or {\em one-to-one} as explained here if you want to recall quickly \url{https://en.wikipedia.org/wiki/Injective_function} again}, so we get:
\[
f_Y(y)\;=\; \P(Y=y)\;=\; \sum_{\{ x: g(x)=y\}}f_X(x)  \;=\; f_X\left( \frac{1}{\sqrt y}-1\right) \;=\; \frac{\lambda^{(\frac{1}{\sqrt y} -1)} \, e^{-\lambda}}{(\frac{1}{\sqrt y} -1)!}
\]
for $y = 1, \frac{1}{4}, \frac{1}{9}, \dots $, and $0$ otherwise.\\[4pt]
{\scriptsize CAUTION: This is a discrete RV and so don't just blindly apply the change of variable formula that only applies to continuous RV with a monotone and one-to-one function
$g$ with inverse $g^{-1}$; it's just that in this discrete RV setting the inverse image also happens to satisfy these properties. But Poisson is discrete and `change of variable formula'' is hence inapplicable.}

\Exercise
If $X$ is a continuous random variable with probability density function
\[
f_X(x)\;=\;\begin{cases} x e^{- x} &  x\geqslant 0\\ 0 & x< 0\end{cases},
\]
find the probability density function of $Y\,=\, e^X$.
\Answer
Since  $y=g(x) = e^x$ is a monotone increasing function for $x \geqslant 0$,  we can apply the change of variable formula.

Now $x= g^{-1}(y) = \log_e(y)$ is a monotone increasing function for $y$ in $[1,\infty)$
\[\left|  \frac{d}{dy}\left(g^{-1}(y)\right)\right|\;=\; \left|  \frac{d}{dy}\left(\log_e(y)\right)\right|\;=\; \frac{1}{y}\,.\]

Therefore
\[
  f_Y(y) \;=\; f_X\left( \log_e(y)\right) \times  \left| \frac{1}{y} \right|\;=\;  \log_e(y) \,e^{-\log_e(y)} \times  \frac{1}{y} \;=\; \log_e(y) \, \frac{1}{y^2}
\]
since $e^{-\log_e(y)} = e^{\log_e(y^{-1})}  = y^{-1}$.

So the  probability density
  function  of $Y$ is given by
\[f_Y(y)\;=\;\begin{cases} \displaystyle  \frac{\log_e(y)}{y^2} &   \text{ if } x \geqslant 1 \\ 0 &
  \text{otherwise}
\end{cases} \,.
\]

\Exercise
If $X$, the received power at an antenna is an $\exponential(\lambda)$ random variable then find the probability density function of the amplitude $Y\,=\, \sqrt X$.
\Answer
Since $y=g(x) = \sqrt{x}$ is a monotone increasing function for $x \geqslant 0$,  we can apply the change of variable formula.

Now $x= g^{-1}(y) = y^2$ is a monotone increasing function for $y\geqslant 0$ so on this interval
\[
\left|  \frac{d}{dy}\left(g^{-1}(y)\right)\right|\;=\; \left|  \frac{d}{dy}\left( y^2\right)\right|\;=\; 2y\,.\]
Therefore
\[
f_Y(y)
\;=\; f_X\left(  y^2\right) \times  \left| 2y \right|
\;=\;  \lambda e ^{-\lambda y^2} \times  2y  \;=\;   2 \lambda y e ^{-\lambda y^2}\,.
\]
So the  probability density function  of $Y$ is given by
\[
f_Y(y)\;=\;\begin{cases} \displaystyle  2 \lambda y e ^{-\lambda y^2}   &    y\geqslant 0 \\ 0 & y <0
\end{cases} \,.
\]

\Exercise
If $X$ is a $\uniform(a,b)$ random variable where $0 < a < b$, find  the probability density function, $f_Y (y)$, of \[ Y\,=\, \log_e(X) \, .\]
\Answer
First note that $y=g(x) = \log_e(x)$ is a monotone increasing function over  $a \leqslant x \leqslant b$, so we can apply the change of variable formula.

 $x= g^{-1}(y) = e^y$ is a monotone increasing function over $\log_e(a) \leqslant   \log_e(x) \leqslant \log_e(b)$, that is, over $\log_e(a) \leqslant   y \leqslant   \log_e(b)$.

For $\log_e(a) \leqslant   y \leqslant   \log_e(b)$,

\[\left|  \frac{d}{dy}\left(g^{-1}(y)\right)\right|\;=\; \left|  \frac{d}{dy}\left(e^y)\right)\right|\;=\; e^y\,.\]

Therefore
\[
  f_Y(y) \;=\; f_X\left(g^{-1}(y)\right) \times  \left|  \frac{d}{dy} \left(g^{-1}(y)\right)\right|\;=\; \frac{1}{b-a} \times e^y\,.
\]
So the  probability density
  function  of $Y$ is given by
\[f_Y(y)\;=\;\begin{cases} \displaystyle  \frac{e^y}{b-a}  &   \log_e(a)\leqslant x \leqslant \log_e(b)\\ 0 &
  \text{otherwise}
\end{cases} \,.
\]

\end{ExerciseList}


\subsubsection{Second Order Variation of Parameters}
\noindent
We'll modify our second order equation of have a 1 as the coefficient of the $y''$ term by dividing to get an equation of the form
\begin{equation*}
	y'' + py' + qy = g(x).
\end{equation*}
Just like for undetermined coefficients, we'll find homogeneous and particular solutions $y = y_h + y_p$.
Since the equation is second-order, the solution to the homogeneous equation will yield two fundamental solutions $y_1$ and $y_2$ where $y_h = C_1y_1 + C_2y_2$.\\

\noindent
So, we can write $y$ as
\begin{equation*}
	y(x) = A(x)y_1 + B(x)y_2,
\end{equation*}
where
\begin{equation*}
	\begin{cases}
		A'y_1  + B'y_2  = 0 \\
		A'y_1' + B'y_2' = g(x)
	\end{cases}.
\end{equation*}
We will then solve this system to solve for $A'$ and $B'$ and integrate.

\begin{example}
	Find the general solution to the following equation
	\begin{equation*}
		y'' + y = \csc{x}
	\end{equation*}
	given that $y_h = C_1\cos{x} + C_2\sin{x}$.
\end{example}
\noindent
$y_h$ gives us our two fundamental solutions
\begin{equation*}
	\begin{cases}
		y_1 = \cos{x} \\
		y_2 = \sin{x}
	\end{cases}.
\end{equation*}
So, our system of equations is
\begin{equation*}
	\begin{cases}
		A'\cos{x} + B'\sin{x} = 0 \\
		-A'\sin{x} + B'\cos{x} = \csc{x}
	\end{cases} \to \begin{cases}
		A'\cos{x} + B'\sin{x} = 0 \\
		-A'\cos{x} + B'\frac{\cos^{2}{x}}{\sin{x}} = \frac{\cos{x}}{\sin^{2}{x}}
	\end{cases}.
\end{equation*}
So,
\begin{equation*}
	B'\frac{1}{\sin{x}} = \frac{\cos{x}}{\sin^{2}{x}} \implies B' = \frac{\cos{x}}{\sin{x}} = \cot{x} \implies B = \ln{\abs{\sin{x}}} + C_2,
\end{equation*}
and
\begin{equation*}
	A'\cos{x} + \cos{x} = 0 \implies A' = -1 \implies A = -x + C_1.
\end{equation*}
So, our general solution is
\begin{equation*}
	y = \left(C_1 - x\right)\cos{x} + \left(\ln{\abs{\sin{x}}} + C_2\right)\sin{x} = C_1\cos{x} + C_2\sin{x} - x\cos{x} + \sin{x}\ln{\abs{\sin{x}}}.
\end{equation*}
Note how $y_h$ and $y_p$ appear together.


\subsection{Symbols}

As we saw earlier, symbols are defined using an equals sign.

\begin{Verbatim}[formatcom=\color{blue}]
N = 212^17
\end{Verbatim}

\noindent
No result is printed when a symbol is defined.
To see the value of a symbol, just evaluate it.

\begin{Verbatim}[formatcom=\color{blue}]
N
\end{Verbatim}

\noindent
$\displaystyle N=3529471145760275132301897342055866171392$

\bigskip
\noindent
Symbols can have more that one letter.
Everything after the first letter is displayed as a subscript.

\begin{Verbatim}[formatcom=\color{blue}]
NA = 6.02214 10^23
NA
\end{Verbatim}

\noindent
$\displaystyle N_A=6.02214\times10^{23}$

\bigskip
\noindent
A symbol can be the name of a Greek letter.

\begin{Verbatim}[formatcom=\color{blue}]
xi = 1/2
xi
\end{Verbatim}

\noindent
$\displaystyle \xi=\tfrac{1}{2}$

\bigskip
\noindent
Greek letters can appear in subscripts.

\begin{Verbatim}[formatcom=\color{blue}]
Amu = 2.0
Amu
\end{Verbatim}

\noindent
$\displaystyle A_\mu=2.0$

\bigskip
\noindent
The following example shows how
Eigenmath scans the entire symbol to find Greek letters.

\begin{Verbatim}[formatcom=\color{blue}]
alphamunu = 1
alphamunu
\end{Verbatim}

\noindent
$\displaystyle \alpha_{\mu\nu}=1$

\bigskip
\noindent
When a symbolic chain is defined,
Eigenmath follows the chain as far as possible.
The following example sets $A=B$ followed by $B=C$.
Then when $A$ is evaluated, the result is $C$.

\begin{Verbatim}[formatcom=\color{blue}]
A = B
B = C
A
\end{Verbatim}

\noindent
$\displaystyle A=C$

\bigskip
\noindent
Although $A=C$ is printed,
inside the program the binding of $A$ is still $B$, as can be seen with
the $binding$ function.

\begin{Verbatim}[formatcom=\color{blue}]
binding(A)
\end{Verbatim}

\noindent
$\displaystyle B$

\bigskip
\noindent
The {\it quote} function returns its argument unevaluated
and can be used to clear a symbol.
The following example clears $A$ so that its evaluation goes back to
being $A$ instead of $C$.

\begin{Verbatim}[formatcom=\color{blue}]
A = quote(A)
A
\end{Verbatim}

\noindent
$\displaystyle A$

\subsection{User defined functions}

Most of the functions commonly used in math and physics are included in Eigenmath.
See the Reference section at the end of the manual for a complete list.
There is also a facility for the user to define additional functions.

\bigskip
\noindent
User functions are defined using the syntax {\it function-name} ( {\it arg-list} ) = {\it expr}
where {\it arg-list} is a comma separated list of zero to nine symbols that receive arguments.
Unlike symbol definitions, {\it expr} is not evaluated when {\it function-name} is defined.
Instead, {\it expr} is evaluated when {\it function-name} is used in a subsequent computation.

\bigskip
\noindent
The following example defines a sinc function and evaluates it at $\pi/2$.

\begin{Verbatim}[formatcom=\color{blue}]
f(x) = sin(x)/x
f(pi/2)
\end{Verbatim}

\noindent
$\displaystyle \frac{2}{\pi}$

\bigskip
\noindent
After a user function is defined, {\it expr} can be recalled using the {\it binding} function.

\begin{Verbatim}[formatcom=\color{blue}]
binding(f)
\end{Verbatim}

\noindent
$\displaystyle \frac{\sin(x)}{x}$

\bigskip
\noindent
If local symbols are needed in a function, they can be appended to {\it arg-list}.
(The caller does not have to supply all the arguments.)
The following example uses Rodrigues's formula to
compute an associated Legendre function of $\cos\theta$.
\begin{equation*}
P_n^m(x)=\frac{1}{2^n\,n!}(1-x^2)^{m/2}\frac{d^{n+m}}{dx^{n+m}}(x^2-1)^n
\end{equation*}
Function $P$ below first computes $P_n^m(x)$ for local variable
$x$ and then uses {\it eval} to replace $x$ with $f$.
In this case, $f=\cos\theta$.

\begin{Verbatim}[formatcom=\color{blue}]
x = 123 -- global x in use, need local x in P
P(f,n,m,x) = eval(1/(2^n n!) (1 - x^2)^(m/2) d((x^2 - 1)^n,x,n + m),x,f)
P(cos(theta),2,0) -- arguments f, n, m, but not x
\end{Verbatim}

\noindent
$\displaystyle \tfrac{3}{2} \cos(\theta)^2-\tfrac{1}{2}$

\bigskip
\noindent
The scope of function arguments is limited to the function definition.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

The operation \textsc{Heap-Delete}$(A,i)$ deletes the item in node $i$ from heap $A$.
Give an implementation of \textsc{Heap-Delete} that runs in $\mathcal{O}(\log n)$ time for an $n$-element max-heap.
You are as well expected to show that your proposed algorithm run in $\mathcal{O}(\log n)$ time.

\subsection*{Solution}

To remove node $i$ from array $A$ where $A$ is a \textit{Heap} data structure with $n$ elements, we first replace $A[i]$ with $A[n]$; last node in height $H$ where $H$ is height of the tree.
This is done to maintain \textit{completeness} of the binary tree.
Obviously this step has a constant runtime cost which is negligible.
As it is likely that such modification violates the \textit{heap} structure of the tree, we should \textsc{Heapify} the modified binary tree to maintain the heap structure after deletion.

Now the following two cases might occur:

\begin{enumerate}[label=(\alph*)]
\item $A[n] < A[\lceil \frac{i}{2} \rceil ] $\\
If node $n$ is within the subtree with root at node $i$, we can rest assured that the value of node $n$ is less than the value of parent of node $i$, in which case \textsc{Heapify} can only be called for the subtree whose root is node $\lceil \frac{i}{2} \rceil$, to move node $i$ one step lower.

\item $A[n] > A[\lceil \frac{i}{2} \rceil ] $
If there is no subtree that includes nodes $i$ and $n$, the value of node $n$ may be higher than the value of parent of node $i$, in which case each call of \textsc{Heapify} is likely to move node $i$ one step higher.
\end{enumerate}

In either case, the number of calls to \textsc{Heapify} is surely less than or equal to the the height $H$ of the tree.
Since $ 2^{H} - 1 < n $, we can derive $ h < \log {n+1} $ and thus the runtime of proposed \textsc{Heap-Delete} algorithm is $\mathcal{O}(\log n)$.


\section{Divergence Theorem}
\begin{theorem}
	Let $V$ be a compact solid, and let $S$ be its boundary surface. For any differentiable vector field $\vec{F}(x,y,z)$,
	\begin{equation*}
		\oint\limits_{S}{\oint{\vec{F} \cdot \mathrm{d}\vec{s}}} = \iiint\limits_{V}{\nabla \cdot \vec{F}\mathrm{d}V}
	\end{equation*}
	where $\mathrm{d}\vec{s} = \hat{n}\mathrm{d}s = (\vec{r_u}\times\vec{r_v})\mathrm{d}u\mathrm{d}v$ and $\mathrm{d}V$ is the volume differential.
\end{theorem}

[INSERT IMAGE]

\noindent
This says that the flux through a closed surface is equal to the sum of the divergence inside that surface.\\
Intuitively, divergence describes how much a vector field is going in or out at a point, so summing it up inside some solid would tell us the amount the vector field is going in or out on the solid’s boundary, which is flux.\\

\noindent
For example, let’s find the flux through the unit sphere centered at the origin from the vector field $\vec{F}(x,y,z) = \langle x, y, z^2 \rangle$.
\begin{equation*}
	S = \left\{(\rho, \theta, \phi) \mid \rho=1, 0 \leq \theta \leq 2\pi, 0 \leq \phi \leq \pi \right\}
\end{equation*}
\begin{equation*}
	V = \left\{(\rho, \theta, \phi) \mid 0 \leq \rho \leq 1, 0 \leq\ theta \leq 2\pi, 0 \leq \phi \leq \pi \right\}
\end{equation*}
\begin{align*}
	\text{Flux} &= \oint\limits_{S}{\oint{\vec{F} \cdot \mathrm{d}\vec{s}}}	\\
	&= \int_{0}^{1}{\int_{0}^{\pi}{\int_{0}^{2\pi}{\nabla \cdot \langle x, y, z^2 \rangle\rho^2\sin{\phi}\mathrm{d}\theta}\mathrm{d}\phi}\mathrm{d}\rho} \\
	&= \int_{0}^{1}{\int_{0}^{\pi}{\int_{0}^{2\pi}{(2 + 2z)\rho^2\sin{\phi}\mathrm{d}\theta}\mathrm{d}\phi}\mathrm{d}\rho} \\
	&= \int_{0}^{1}{\int_{0}^{\pi}{\int_{0}^{2\pi}{(2 + 2\rho\sin{\phi})\rho^2\sin{\phi}\mathrm{d}}\mathrm{d}}\mathrm{d}} \\
	&= 2\pi\int_{0}^{1}{\int_{0}^{\pi}{2\rho^2\sin{\phi} + 2\rho^3\sin{\phi}\cos{\phi}\mathrm{d}\phi}\mathrm{d}\rho} \\
	&= 4\pi\int_{0}^{\pi}{\frac{1}{3}\sin{\phi} + \frac{1}{4}\sin{\phi}\cos{\phi}\mathrm{d}\phi} \\
	&= \frac{8\pi}{3}
\end{align*}

\input{./vectorAnalysis/gausssLaws}


\lab{Application}{Newton's Method, Julia Sets and Basins of Attraction}{Newton's Method}
\label{lab:NewtonsMethod}
\objective{Understand Newton's Method. Understand definition of basin of attraction.  Basic understanding of Julia Sets.}

One important technique in technical computing is Newton's method.
The goal of Newton's method is to find $\overline{x}$ such that $f\left(\overline{x}\right) = 0$.
This method is especially important in optimization, where our goal is to find minima and maxima of functions.
Newton's method is an iterative method.
In one dimension, it is defined as follows:
\[
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
\]

Essentially Newton's method approximates a function by its tangent line, and then uses the zero of the tangent line as the next guess for $x_n$.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{newton_iters}
\caption{An illustration of how one iteration of Newton's method works}
\end{figure}

Newton's method is powerful because of the speed of convergence.
In many cases Newton's method converges to the actual root quadratically, meaning that the error term is squared at every iteration.
This fast convergence makes it a very powerful algorithm.

Newton's method does suffer from the flaw that its convergence is dependent upon an initial guess.
If the initial guess is not sufficiently close the convergence can be much slower, or may never occur.
There are even certain pathological functions for which newton's method will never converge.

\begin{problem}
\label{prob:newton_arr}
Write a Newton's method function that runs whether or not the user inputs a derivative function.
In python this can be done by defining the derivative function as a keyword argument that defaults to \li{None}.
Also accept a maximum of how many iterations it will run and a tolerance used for the stopping condition.
Return a tuple with the computed value and boolean value telling whether it converged or not.

Compare the performance of Newton's method when you input the derivative and when you don't.
How well does each converge?
Which runs faster?
Try the following functions:

\begin{itemize}
\item $cos(x)$
\item $x^2sin(\frac{1}{x})$
\item $(\frac{sin(x)}{x})-x$
\item $x^2-1$
\item $x^3 - x$
\end{itemize}

For the last two, look at the results of the iteration on equispaced points between -2 and 2.
Which points converge to which roots?

There are also examples of when Newton's Method does not converge.
Test your newton's method function on $x^{1/3}$.
Have it test at eqispaced starting points on a given interval and return the corresponding values after a given number of iterations.
What do you see?
\end{problem}

\begin{comment}
\begin{problem}
Extend your Newton's method even further so that it will work on systems of equations.
Suppose that $F: \mathbb{R}^n \rightarrow \mathbb{R}^n $.
The relevant equation is
\[
x_{i+1} = x_i - J^{-1}F(x_i)
\]
Note that you should not calculate the inverse Jacobian.
\li{scipy.linalg.solve(A,b)} gives you the solution $x$ to the equation $Ax=b$.
Use this fact to calculate $J^{-1}F$ from $J$ and $F$.
You should be able to make this function work whether or not the user inputs a Jacobian.
This also means that you will have to use your own \li{jacobian} function.
\end{problem}
\end{comment}

\section*{Basins of Attraction and Julia Sets}
Thus far we have only noted the fact that Newton's method converges under good circumstances.
Another topic of interest is determining which starting values converge to which roots.
An plot showing which points converge to which roots for the function $x^2 + x - 2$ is shown in Figure \ref{Fig:basins1}.

\begin{figure}
\begin{center}
\includegraphics[scale=0.5]{basins1}
\caption{The plot of $f(x) = x^2 + x - 2$ along with 20 seed values for Newton's Method.
The green values all converge to the root at -2, and the black values will converge to the root at 1.}
\label{Fig:basins1}
\end{center}
\end{figure}

It turns out that for $f(x) = x^2 + x - 2$, any seed value will converge to one root or the other.
We call the set of points that converge to a single value through an iterative process a basin of attraction.
We can see in Figure \ref{Fig:basins1} a set of seed values that are color coded to indicate which root they converge to with Newton's method.

We now extend these ideas to complex functions.
There is an entire field of mathematics devoted to the study of such functions, but here we only examine some basic properties as they pertain to the generation of Julia sets.
Like other functions we have thus far been exposed to, a complex function maps an input to a unique output.
However, in this case both the input and the output come from the space of complex numbers.

Given a sufficiently ``nice'' complex function, we can apply Newton's method in a similar way to how we applied it in the real case.
However due to the nature of the space we are mapping from and to, we no longer have access to the intuitive visual representations of functions that we saw in the real case.
We can, however, graph the basins of attraction for Newton's method on the complex plane.
For example, let:
\[
f(z) = z^2 - 1
\]
Derivatives for nice complex functions behave much the same way as in the real case:
\[
f'(z) = 2z
\]
It is straightforward to verify that $f$ has two roots at 1 and -1.

In order to better visualize basins of attraction in the complex plane we will use the function li{pcolormesh} from the pyplot library in matplotlib.
The following is an example of how to use this function.
It allows us to color different portions of a plane according to the value of a function at each point.

\begin{lstlisting}
import scipy as sp
from matplotlib import pyplot as plt
n=401
x=sp.linspace(-2,2,n)
y=sp.linspace(-2,2,n)
X,Y=sp.meshgrid(x,y)
def func(x):
    return sp.real(x**3-2*x**2-x+2)
C=func(X+complex(0,1)*Y)
plt.pcolormesh(X,Y,C)
plt.show()
\end{lstlisting}

\begin{problem}
Modify your code from Problem \ref{prob:newton_arr} to graph the basins of attraction for a polynomial in the complex plane.
Examine the basins of attraction for the following complex functions on the following domains.
\begin{itemize}

\item $z^3 - 2 z^2 - 2 z + 2$ on the set of complex numbers $a + b i$ such that $a \in \left[-\frac{1}{2}, 0\right]$ and $b \in \left[-\frac{1}{4}, \frac{1}{4}\right]$.

\item $3 z^3 - 2 z^2 - 2 z + 2$ on the set of complex numbers $a + b i$ such that $a \in \left[-1, 1\right]$ and $b \in \left[-1, 1\right]$.

\item $z^4 + 3 z^3 - 2 z^2 - 2 z + 2$ on the set of complex numbers $a + b i$ such that $a \in \left[-1, 1\right]$ and $b \in \left[-1, 1\right]$.

\item $z^3 - 1$ on the set of complex numbers $a + b i$ such that $a \in \left[-1, 1\right]$ and $b \in \left[-1, 1\right]$.

\end{itemize}
Be aware that Newton's method may not converge for every point in these intervals, so you will have to manually set the pixels that result in a value of \li{nan} to something else.
One good approach is to set the roots of the polynomial to numbers $0, 1, \dots, k$ and then set the values of \li{nan} to $k+1$.

Make sure that when plotting you start with low resolution.
Even if you generate a plot using an array of $100$ by $100$ points, you will be running Newton's method on your test function $10,000$ times, so be careful.
\end{problem}

Another well-studied fractal in the complex plane is the Mandelbrot set.
It is defined as the points $c \in \mathbb{C}$ for which the sequence
\[z_n = z_{n-1}^2 + c\]
is bounded.

\begin{problem}
Generate another grid of complex numbers on $[-1.5,.5]\times[-i,i]$.
Consider the recurrence relation $x_n = x_{n-1}^2 + c$.
Run this iteration 30 times with a resolution of $200 \times 200$ and display the plot.
\end{problem}


\chapter{Applications and Case Studies}
This chapter contains a number of case studies designed to deepen our understanding of \textsl{Python}.

\section{Solving Equations via Fixed-Point Algorithms}
\blue{Fixed-Point iterations} are very important, both in computer science and in mathematics.  As a first
example, we show how to solve an equation via a fixed point iteration.  Suppose we want to solve the equation
\\[0.2cm]
\hspace*{1.3cm} $x = \cos(x)$. \\[0.2cm]
Here, $x$ is a real number that we seek to compute.  Figure \ref{fig:xEqualsCosX.pdf} on page
\pageref{fig:xEqualsCosX.pdf} shows the graphs of the two functions
\\[0.2cm]
\hspace*{1.3cm}
$y = x$  \quad and \quad $y = \cos(x)$.
\\[0.2cm]
Since the graphs of these functions intersect, it is obvious that there exists a value $x$ such that $x = \cos(x)$.
Furthermore, from Figure \ref{fig:xEqualsCosX.pdf} it is obvious that this value of $x$ is bigger than $0.6$
and less than $0.8$.

\begin{figure}[!ht]
  \hspace*{-3.0cm}
  \epsfig{file=Figures/xEqualsCosX.pdf,scale=0.6}

  \caption{The functions $y = x$ and $y = cos(x)$.}
  \label{fig:xEqualsCosX.pdf}
\end{figure}

A simple approach that lets us compute the exact value of $x$ is to use a
\href{https://en.wikipedia.org/wiki/Fixed-point_iteration}{fixed-point iteration}.  To this end, we
define the sequence $\bigl(x_n\bigr)_{n\in\mathbb{N}}$ inductively as follows:
\\[0.2cm]
\hspace*{1.3cm}
$x_0 = 0$ \quad and \quad $x_{n+1} = \mathtt{cos}(x_n)$ \quad for all $n \in \mathbb{N}$.
\\[0.2cm]
With the help of the
\href{https://en.wikipedia.org/wiki/Banach_fixed-point_theorem}{Banach fixed-point theorem}\footnote{
  The Banach fixed-point theorem is discussed in the lecture on
  \href{https://en.wikipedia.org/wiki/Differential_calculus}{differential calculus}.  This lecture is part of the
  second semester.
}
it can be shown that this sequence converges to a solution of the equation $x = \cos(x)$, i.e.~if we define
\\[0.2cm]
\hspace*{1.3cm}
$\bar{x} = \lim\limits_{n\rightarrow\infty} x_n$,
\\[0.2cm]
then we have
\\[0.2cm]
\hspace*{1.3cm}
$\cos\bigl(\bar{x}\bigr) = \bar{x}$.
\\[0.2cm]
Figure \ref{fig:solve.py} on page \pageref{fig:solve.py} shows the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/solve.py}{\texttt{solve.py}}
that uses this approach to solve the equation $x = \cos(x)$.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    import math

    x     = 1.0
    old_x = 0.0
    i     = 1
    while abs(x - old_x) >= 4.0E-16:
        old_x = x
        x = math.cos(x)
        print(f'{i} : {x}')
        i += 1
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Solving the equation $x = \cos(x)$ via fixed-point iteration.}  \label{fig:solve.py}
\end{figure} %\$

In this program, the iteration stops as soon as the difference between the variables \texttt{x} and
\texttt{old\_x} is less that $4 \cdot 10^{-16}$.  Here, \texttt{x} corresponds to $x_{n+1}$, while \texttt{old\_x}
corresponds to $x_n$.  Once the values of $x_{n+1}$ and $x_n$ are sufficiently close, the execution of the \texttt{while} loop
terminates.
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/Fixed-Point-Iteration.ipynb}{Fixed-Point-Iteration.ipynb}
shows a \textsl{Jupyter} notebook that implements fixed point iteration.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    from math import cos

    def solve(f, x0):
        """
        Solve the equation f(x) = x using a fixed point iteration.
        x0 is the start value.
        """
        x = x0
        for n in range(10000):  # at most 10000 iterations
            oldX = x;
            x    = f(x);
            if abs(x - oldX) < 1.0e-15:
                return x;

    print("solution to x = cos(x): ", solve(cos, 0));
    print("solution to x = 1/(1+x):", solve(lambda x: 1/(1+x), 0));
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A generic implementation of the fixed-point algorithm.}
\label{fig:fixpoint.py}
\end{figure}

Figure \ref{fig:fixpoint.py} on page \pageref{fig:fixpoint.py} shows the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/fixpoint.py}{\texttt{fixpoint.py}}.
In this program we have implemented a function \texttt{solve} that takes two arguments.
\begin{enumerate}
\item \texttt{f} is a unary function.  The purpose of the \texttt{solve} is to compute the solution of the equation
      \\[0.2cm]
      \hspace*{1.3cm}
      $f(x) = x$.
      \\[0.2cm]
      This equation is solved with the help of a fixed-point algorithm.
\item \texttt{x0} is used as the initial value for the fixed-point iteration.
\end{enumerate}
Line 11 calls \texttt{solve} to compute the solution of the equation $x = \cos(x)$.
Line 12 solves the equation
\\[0.2cm]
\hspace*{1.3cm}
$\ds x = \bruch{1}{1+x}$.
\\[0.2cm]
This equation is equivalent to the quadratic equation $x^2 + x = 1$.  Note that we have defined the function
 $\ds x \mapsto \frac{1}{1+x}$ via the expression
 \\[0.2cm]
\hspace*{1.3cm}
\texttt{lambda x: 1/(1+x)}.
\\[0.2cm]
This expression is called an \blue{anonymous function} since we haven't given a name to the function.

\remarkEng
The function \texttt{solve} is only able to solve the equation $f(x) = x$ if the function $f$ is a
\href{https://en.wikipedia.org/wiki/Contraction_mapping}{contraction mapping}.  A function
$f:\mathbb{R} \rightarrow \mathbb{R}$
is called a \blue{contraction mapping} iff
\\[0.2cm]
\hspace*{1.3cm}
$|f(x) - f(y)| < |x - y|$ \quad for all $x,y \in \mathbb{R}$.
\\[0.2cm]
This notion will be discussed in more detail in the lecture on
\href{https://github.com/karlstroetmann/Analysis/blob/master/Script/analysis.pdf}{analysis} in the second
semester. \eox

\section{Case Study: Computation of Poker Probabilities}
In this short section we are going to show how to compute probabilities for the
\href{https://en.wikipedia.org/wiki/Texas_hold_%27em}{\textsl{Texas Hold'em}} variation of
\href{https://en.wikipedia.org/wiki/Poker}{poker}.   Texas Hold'em poker is played with a deck of 52
cards.  Every card has a \blue{value}.  This value is an element of the set
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{Values} = \{ 2, 3, 4, 5, 6, 7, 8, 9, 10, \textsl{Jack}, \textsl{Queen}, \textsl{King}, \textsl{Ace} \}$.
\\[0.2cm]
Furthermore, every card has a \blue{suit}.  This suit is an element of the set
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{Suits} = \{ \club, \mbox{$\color{red}{\heart}$}, \mbox{$\color{red}{\diamondsuit}$}, \spade \}$.
\\[0.2cm]
These suits are pronounced \blue{club}, \blue{heart}, \blue{diamond}, and \blue{spade}.
As a card is determined by its value and its suit, a card can be represented as a pair $\pair(v,s)$, where $v$
denotes the value while $s$ is the suit of the card.  Hence, the set of all cards can be represented as the set
\\[0.2cm]
\hspace*{1.3cm}
$\textsl{Deck} = \bigl\{ \pair(v,s) \mid v \in \textsl{Values} \wedge \textsl{s} \in \textsl{Suits} \bigr\}$.
\\[0.2cm]
At the start of a game of Texas Hold'em, every player receives two cards.  These two cards are known
as the \blue{preflop} or the \blue{hole}.  Next, there is a \blue{bidding phase} where players can bet on their
cards.   After this bidding phase, the dealer puts three cards open on the table.  These three cards are
known as \blue{flop}.  Let us assume that a player has been dealt the set of cards
\\[0.2cm]
\hspace*{1.3cm}
$\{ \pair(3, \club), \pair(3, \spade) \}$.
\\[0.2cm]
This set of cards is known as a \blue{pocket pair}.  Then the player would like to know the probability
that the flop will contain another card with value $3$, as this would greatly increase her chance of
winning the game.  In order to compute this probability we have to compute the number of possible
flops that contain a card with the value $3$ and we have to divide this number by the number of all
possible flops:
\\[0.2cm]
\hspace*{1.3cm}
$\ds \frac{\;\mbox{number of flops containing a card with value $3$}\;}{\mbox{number of all possible flops}}$
\\[0.2cm]
The program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/poker-triple.py}{poker-triple.py}
shown in Figure \ref{fig:poker-triple.py} performs this computation.  We proceed to discuss this
program line by line.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.0cm,
                  xrightmargin  = 0.0cm,
                ]
    Values = { "2", "3", "4", "5", "6", "7", "8", "9", "T", "J", "Q", "K", "A" }
    Suits  = { "c", "h", "d", "s" }
    Deck   = { (v, s) for v in Values for s in Suits }
    Hole   = { ("3", "c"), ("3", "s") }
    Rest   = Deck - Hole
    Flops  = { (k1, k2, k3) for k1 in Rest for k2 in Rest for k3 in Rest
                            if  len({ k1, k2, k3 }) == 3
             }
    Trips  = { f for f in Flops if ("3", "d") in f or ("3", "h") in f }
    print(len(Trips) / len(Flops))
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing a probability in poker.}
\label{fig:poker-triple.py}
\end{figure}

\begin{enumerate}
\item In line 1 the set \texttt{Values} is defined to be the set of all possible values that a card
      can take.  In defining this set we have made use of the following abbreviations:
      \begin{enumerate}
      \item ``\texttt{T}'' is short for ``\blue{Ten}'',
      \item ``\texttt{J}'' is short for ``\blue{Jack}'',
      \item ``\texttt{Q}'' is short for ``\blue{Queen}'',
      \item ``\texttt{K}'' is short for ``\blue{King}'', and
      \item ``\texttt{A}'' is short for ``\blue{Ace}''.
      \end{enumerate}
\item In line 2 the set \texttt{Suits} represents the possible suits of a card.  Here, we have used
      the following abbreviations:
      \begin{enumerate}
      \item ``\texttt{c}'' is short for $\club$, which is pronounced as \blue{club},
      \item ``\texttt{h}'' is short for \mbox{\color{red}{$\heart$}}, which is pronounced as \blue{heart},
      \item ``\texttt{d}'' is short for \mbox{\color{red}{$\diamondsuit$}}, which is pronounced as \blue{diamond}, and
      \item ``\texttt{s}'' is short for $\spade$, which is pronounced as \blue{spade}.
      \end{enumerate}
\item Line 3 defines the set of all cards.  This set is stored as the variable \texttt{Deck}.  Every
      card is represented as a pair of the form $[v,s]$. Here, $v$ is the value of the card, while $s$ is its suit.
\item Line 4 defines the set \texttt{Hole}.  This set represents the two cards that have been given to our player.
\item The remaining cards are defined as the variable  \texttt{Rest} in line 5.
\item Line 6 computes the set of all possible flops.  Since the order of the cards in the flop does
      not matter, we use sets to represent these flops.  However, we have to take care that the flop
      does contain three \colorbox{amethyst}{different} cards.  Hence, we have to ensure that the three
      cards \texttt{k1}, \texttt{k2}, and \texttt{k3} that make up the flop satisfy the inequalities
      \\[0.2cm]
      \hspace*{1.3cm}
      $\mathtt{k1} \not= \mathtt{k2}$, \quad $\mathtt{k1} \not= \mathtt{k3}$,  \quad and \quad $\mathtt{k2} \not= \mathtt{k3}$.
      \\[0.2cm]
      These inequalities are satisfied if and only if the set
      $\{ \mathtt{k1}, \mathtt{k2}, \mathtt{k3} \}$ contains exactly three elements.  Hence, when
      choosing \texttt{k1}, \texttt{k2}, and \texttt{k3} we have to make sure that the condition
      \\[0.2cm]
      \hspace*{1.3cm}
      $\texttt{len}\bigl({\{ \mathtt{k1}, \mathtt{k2}, \mathtt{k3} \} \;\mathtt{==}\; 3 }\bigr)$
      \\[0.2cm]
      holds.
\item Line 9 computes the subset \texttt{Trips} of those flops that contain at least one card with a value of 3.
      As the 3 of clubs and the 3 of spades have already been dealt to our player, the only cards
      with value 3 that are left in the deck are the 3 of diamonds and the 3 of hearts.  Therefore, we are looking for
      those flops that contain one of these two cards.
\item Finally, the probability for obtaining another card with a value of 3 in the flop is computed as
      the ratio of the number of flops containing a card with a value of 3 to the number of all possible flops.
\end{enumerate}
When we run the program we see that the probability of improving a \blue{pocket pair} on the flop to \blue{trips} or better
is about  $11.8\%$.  A \textsl{Jupyter} notebook showcasing this computation outlined above can be fount at
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/Poker.ipynb}{Poker.ipynb}.

\remarkEng
The method to compute probabilities that has been sketched above only works if the sets that have to
be computed are small enough to be retained in memory.  If this condition is
not satisfied we can use the \href{https://en.wikipedia.org/wiki/Monte_Carlo_method}{\emph{Monte Carlo method}}
to compute the probabilities instead.  This method will be discussed in the lecture on
\href{https://github.com/karlstroetmann/Algorithms/blob/master/Lecture-Notes/algorithms.pdf}{algorithms}.

\section{Finding a Path in a Graph}
In the following section, I will present an application that is more interesting since it is practically
relevant.  In order to prepare for this, we will now discuss the problem of finding a \blue{path} in a
\href{https://en.wikipedia.org/wiki/Directed_graph}{directed graph}.
Abstractly, a graph consists of \blue{vertices} and \blue{edges} that connect these vertices.  In an application, the
vertices could be towns and villages, while the edges would be interpreted as streets connecting these
villages.  To simplify matters, let us assume for now that the vertices are given as natural numbers, while the
edges are represented as pairs of natural numbers.  Then, the graph can be represented as the set of its edges,
as the set of vertices is implicitly given once the edges are known.  To make things concrete, let us consider
an example.  In this case, the set of edges is called \texttt{R} and is defined as follows:
\\[0.2cm]
\hspace*{1.3cm}
$\texttt{R}\; \mathtt{=}\; \bigl\{ \pair(1,2), \pair(2,3), \pair(1,3), \pair(2,4), \pair(4,5) \bigr\}$.
\\[0.2cm]
In this graph, the set of vertices is given as
\\[0.2cm]
\hspace*{1.3cm}
$\{ 1, 2, 3, 4, 5 \}$.
\\[0.2cm]
This graph is shown in Figure \ref{fig:graph0} on page \pageref{fig:graph0}.  You should note that the
connections between vertices that are given in this graph are \blue{unidirectional}:  While there is a connection from
vertex $1$ to vertex $2$, there is no connection from vertex $2$ to vertex $1$.

\begin{figure}[!ht]
  \centering
  \epsfig{file=Figures/graph0,scale=0.6}

  \caption{A simple graph.}
  \label{fig:graph0}
\end{figure}

\noindent
The graph given by the relation \texttt{R} contains only the direct connections of vertices.  For example, in
the graph shown in Figure \ref{fig:graph0}, there is a direct connection from vertex $1$ to vertex $2$ and
another direct connection from vertex $2$ to vertex $4$.  Intuitively, vertex $4$ is reachable from vertex $1$,
since from vertex $1$ we can first reach vertex $2$ and from vertex $2$ we can then reach vertex $4$.  However,
there is is no direct connection between the vertices $1$ and $4$.  To make this more formal, define
a \blue{path} of a graph $R$ as a list of vertices
\\[0.2cm]
\hspace*{1.3cm}
$[x_1, x_2, \cdots, x_n]$ \quad such that \quad $\pair(x_i,x_{i+1}) \in R$ \quad for all $i=1,\cdots,n-1$.
\\[0.2cm]
In this case, the path $[x_1, x_2, \cdots, x_n]$ is written as
\\[0.2cm]
\hspace*{1.3cm}
$x_1 \mapsto x_2 \mapsto \cdots \mapsto x_n$
\\[0.2cm]
and has the \blue{length} $n-1$.  It is important to note that the length of a path
$[x_1,x_2,\cdots,x_n]$ is defined as the number of edges connecting the vertices and not as the
number of vertices appearing in the path.

Furthermore,  two vertices $a$ and $b$ are said to be \blue{connected} iff there exists a path
\\[0.2cm]
\hspace*{1.3cm}
$[x_1,\cdots,x_n]$ \quad such that \quad $a = x_1$ \quad and \quad $b = x_n$.
\\[0.2cm]
The goal of this section is to develop an algorithm that checks whether two vertices $a$ and $b$ are connected.
Furthermore, we want to be able to compute the corresponding path connecting the vertices $a$ and $b$.

\subsection{Computing the Transitive Closure of a Relation}
We have already noted that a graph can be represented as the set of its edges and hence as a \blue{binary relation}.
A \blue{binary relation} is defined as a set of pairs.  We also need the notion of a \blue{relational product}:
If $Q$ and $R$ are binary relations, then the \blue{relational product} $Q \circ R$ of $Q$ and $R$ is defined as
\\[0.2cm]
\hspace*{1.3cm}
$Q \circ R := \bigl\{ \pair(x, z) \bigm| \exists y:(\pair(x,y) \in Q \wedge \pair(y,z) \in R) \bigr\}$.
\\[0.2cm]
Furthermore, for any $n \in \mathbb{N}^*$ we can define the $n$-th power of the relation $R$ by induction.
\begin{enumerate}
\item[B.C.:] $n = 1$.

             $R^1 := R$
\item[I.S.:] $n \mapsto n+1$

             $R^{n+1} := R^n \circ R$.
\end{enumerate}
In order to decide whether there is a path connecting two vertices we have to compute the
\href{https://en.wikipedia.org/wiki/Transitive_closure}{transitive closure} $R^+$ of a relation $R$.
To understand this notion, we first need to define the concept of \blue{transitivity}:  A relation $R$ is
transitive if and only if the following holds:
\\[0.2cm]
\hspace*{1.3cm}
$\pair(x,y) \in T \wedge \pair(y, z) \in T \rightarrow \pair(x, z) \in T$ \quad for all $x,y,z$.
\\[0.2cm]
Now the \blue{transitive closure} $R^+$ of a binary relation $R$ is the smallest relation $T$ such that the following
condition holds:
\begin{itemize}
\item $R$ is a subset of $T$, i.e. we have $R \subseteq T$.
\item $T$ is transitive.
\end{itemize}
The lecture on
\href{https://github.com/karlstroetmann/Lineare-Algebra/blob/master/Script/lineare-algebra.pdf}{Lineare Algebra}
gives a prove that the transitive closure $R^+$ of a binary relation can be computed as follows:
\\[0.2cm]
\hspace*{1.3cm}
$R^+ = \bigcup\limits_{n=1}^{\infty} R^n = R^1 \cup R^2 \cup R^3 \cup \cdots$
\\[0.2cm]
Initially, this formula might look intimidating as it suggests an infinite computation.
Fortunately, it turns out that we do not have to compute all powers of the form $R^n$.  Let me
explain the reason that allows us to cut the computation short.
\begin{enumerate}
\item $R$ is the set of direct connections between two vertices.
\item $R^2$ is the same as $R \circ R$ and this relational product is defined as
      \\[0.2cm]
      \hspace*{1.3cm}
       $R \circ R = \bigl\{ \pair(x,z) \bigm| \exists y \colon(\pair(x,y) \in R \wedge \pair(y,z)) \in R \bigr\}$.
      \\[0.2cm]
      Hence, $R \circ R$ contains those pairs $\pair(x,z)$ that are connected via one intermediate vertex $y$,
      i.e.~there is a path of the form $x \mapsto y \mapsto z$ that connects $x$ and $z$.  This path
      has length 2.  In general, we can show by induction that $R^n$ connect those pairs that are
      connected by a path of length $n$.  The induction step of this proof runs as follows:
\item $R^{n+1}$ is defined as $R^n \circ R$ and therefore we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $R^n \circ R = \{ \pair(x,z) \mid \exists y \colon \pair(x,y) \in R^n \wedge \pair(y,z) \in R \}$.
      \\[0.2cm]
      As $\pair(x,y) \in R^n$, the induction hypothesis guarantees that the vertices $x$ and $y$ are
      connected by a path of length $n$.  Hence, this
      path has the form
      \\[0.2cm]
      \hspace*{1.3cm}
      $\underbrace{x \mapsto \cdots \mapsto y}_{\mbox{\scriptsize path of length $n$.}}$
      \\[0.2cm]
      Adding $x$ at the end of this path will produce the path
      \\[0.2cm]
      \hspace*{1.3cm}
      $x \mapsto \cdots \mapsto y \mapsto z$.
      \\[0.2cm]
      This path has a length of $n + 1$ and, furthermore, connects $x$ and $z$.  Hence $R^{n+1}$
      contains those pairs $\pair(x, z)$ that are connected by a path of length $n+1$.
\end{enumerate}
Now the important observation is the following. The set of all vertices is finite.  For the arguments sake, let
us assume there are $k$ different vertices.  But then every path that has a length of $k$ or greater must
contain at least one vertex that is visited more than once and hence this path is longer than necessary,
i.e.~there is a shorter path that connects the same vertices.  Therefore, for a finite graph with $k$ vertices,
the formula to compute the transitive closure can be simplified as follows:
\\[0.2cm]
\hspace*{1.3cm}
$\ds R^+ = \bigcup\limits_{i=1}^{k-1} R^i$.
\\[0.2cm]
While we could use this formula as its stands, it is more efficient to use a \blue{fixed-point iteration} instead.
To this end, we prove that the transitive closure $R^+$ satisfies the following equation:
\begin{equation}
  \label{fixpunkt}
  R^+ = R \cup R^+ \circ R.
\end{equation}
Let me remind you that the precedence of the operator $\circ$
is higher than the precedence of the operator $\cup$.  Therefore, the expression $R \cup R^+ \circ R$ is parenthesized
as $R \cup (R^+ \circ R)$.  Equation \ref{fixpunkt} can be proven algebraically.  We have:
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}{cll}
    & R \cup R^+ \circ R \\[0.2cm]
  = & R \cup \Bigl(\bigcup\limits_{i=1}^{\infty} R^i \Bigr) \circ R \\[0.4cm]
  = & R \cup \bigl(R^1 \cup R^2 \cup R^3 \cup \cdots \bigr) \circ R \\[0.2cm]
  = & R \cup \bigl(R^1 \circ R \cup R^2 \circ R \cup R^3 \circ R \cup \cdots \bigr) \\[0.2cm]
  = & R \cup \bigl(R^2 \cup R^3 \cup  R^4 \cup \cdots \bigr)  \\[0.2cm]
  = & R^1 \cup \bigl(R^2 \cup R^3 \cup  R^4 \cup \cdots \bigr) \\[0.2cm]
  = & \bigcup\limits_{i=1}^{\infty} R^i \\[0.4cm]
  = & R^+.
\end{array}
$
\\[0.2cm]
Equation  \ref{fixpunkt} can now be used to compute $R^+$ via a fixed-point iteration.
To this end, let us define a sequence of relations $(T_n)_{n \in \mathbb{N}}$ by induction on $n$:
\begin{enumerate}
\item[I.A.] $n = 0$:

            $T_0 = R$
\item[I.S.] $n \mapsto n+1$:

            $T_{n+1} = R \cup T_n \circ R $.
\end{enumerate}
The relation  $T_n$ can be expressed via the relation $R$, we have
\begin{enumerate}
\item $T_0 = R$.
\item $T_1 = R \cup T_0 \circ R = R \cup R \circ R = R^1 \cup R^2$.
\item$\begin{array}[t]{lcl}
       T_2  & = & R \cup T_1 \circ R \\
            & = & R \cup (R^1 \cup R^2) \circ R \\
            & = & R^1 \cup R^2 \cup R^3. \\
       \end{array}
      $
\end{enumerate}
In general, we can show by induction that
\\[0.2cm]
\hspace*{1.3cm}
$T_n = \bigcup\limits_{i=1}^{n+1} R^i$
\\[0.2cm]
holds for all $n \in \mathbb{N}$.  The base case of this proof is immediate from the definition of $T_0$.
In the induction step we observe the following:
\\[0.2cm]
\hspace*{1.3cm}
$
 \begin{array}{lcll}
   T_{n+1} & = & \ds R \cup T_n \circ R & \mbox{(by definition)} \\[0.2cm]
           & = & \ds R \cup \biggl(\bigcup\limits_{i=1}^{n+1} R^i\biggr) \circ R &
                 \mbox{(by induction hypothesis)} \\[0.4cm]
           & = & \ds R \cup \left(R \cup \cdots \cup R^{n+1}\right) \circ R \\[0.2cm]
           & = & \ds R^1 \cup R^2 \cup \cdots \cup R^{n+2}  &
                 \mbox{(by the distributivity of $\circ$ over $\cup$)} \\[0.2cm]
           & = & \ds \bigcup\limits_{i=1}^{n+2} R^i & \Box
   \end{array}
$
\\[0.2cm]
The sequence $(T_n)_{n\in\mathbb{N}}$ has another useful property:  It is
\blue{monotonically increasing}.  In general, a sequence of sets $(X_n)_{n\in\mathbb{N}}$ is called
\blue{monotonically increasing} iff we have
\\[0.2cm]
\hspace*{1.3cm}
$\forall n \in \mathbb{N}: X_n \subseteq X_{n+1}$,
\\[0.2cm]
i.e.~the sets $X_n$ get bigger with growing index $n$.
The monotonicity of the sequence  $(T_n)_{n \in \mathbb{N}}$ is an immediate consequence of the equation
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_n = \bigcup\limits_{i=1}^{n+1} R^i$
\\[0.2cm]
because we have:
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}[t]{llcl}
                & \ds T_n \subseteq T_{n+1} \\[0.2cm]
\Leftrightarrow & \ds \bigcup\limits_{i=1}^{n+1} R^i \subseteq \bigcup\limits_{i=1}^{n+2} R^i \\[0.5cm]
\Leftrightarrow & \ds \bigcup\limits_{i=1}^{n+1} R^i \subseteq \bigcup\limits_{i=1}^{n+1} R^i \cup R^{n+2} \\
\end{array}
$
\\[0.2cm]
If the relation  $R$ is finite, then the transitive closure $R^+$ is finite, too.  The sets $T_n$
are all subsets of $R^+$ because we have
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_n = \bigcup\limits_{i=1}^{n+1} R^i \subseteq \bigcup\limits_{i=1}^{\infty} R^i = R^+$ \quad for all $n \in \mathbb{N}$.
\\[0.2cm]
Hence the sets $T_n$ can not grow indefinitely.  Because of the monotonicity of the sequence
$(T_n)_{n\in\mathbb{N}}$ it follows that there exists an index  $k \in \mathbb{N}$ such that the sets $T_n$ do
not grow any further once $n$ has reached $k$, i.e.~we have
\\[0.2cm]
\hspace*{1.3cm}
$\ds \forall n \in \mathbb{N}:( n \geq k \rightarrow T_n = T_k)$.
\\[0.2cm]
But this implies that
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_n = \bigcup\limits_{i=1}^{n+1} R^i = \bigcup\limits_{i=1}^{\infty} R^i = R^+$
\quad holds for all $n \geq k$.
\\[0.2cm]
Therefore, the algorithm for computing  $R^+$ iterates the equation
\\[0.2cm]
\hspace*{1.3cm}
$\ds T_{n+1} = R \cup T_n \circ R$
\\[0.2cm]
until the equation  $T_{n+1} = T_n$ is satisfied, since this implies that $T_n = R^+$.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def product(R1, R2):
        "Compute the relational product of R1 and R2."
        return { (x, z) for (x, y1) in R1 for (y2, z) in R2 if y1 == y2 }

    def transClosure(R):
        "Compute the transitive closure of the binary relation R."
        T = R
        while True:
            oldT = T
            T    = product(R,T).union(R)
            if T == oldT:
                return T

    R = { (1,2), (2,3), (1,3), (2,4), (4,5) }
    print( "R  = ", R );
    print( "Computing the transitive closure of R:" );
    T = transClosure(R);
    print( "R+ = ", T );
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing the transitive closure.}
\label{fig:transitive-closure.py}
\end{figure} %\$

\noindent
The program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/transitive-closure.py}{\texttt{transitive-closure.py}}
that is shown in Figure
\ref{fig:transitive-closure.py} on page \pageref{fig:transitive-closure.py} shows an implementation of this idea.
The program produces the following output:
\begin{verbatim}
    R  = {(1, 2), (1, 3), (4, 5), (2, 3), (2, 4)}
    Computing the transitive closure of R:
    R+ = {(1, 2), (1, 3), (4, 5), (1, 4), (1, 5), (2, 3), (2, 5), (2, 4)}
\end{verbatim}
The transitive closure $R^+$ of a relation $R$ has a very intuitive interpretation:
It contains all pairs $\pair(x,y)$ such that there is a path leading from
$x$ to $y$.
The function $\texttt{product}(R_1, R_2)$ computes the relational product $R_1\circ R_2$
according to the formula
\\[0.2cm]
\hspace*{1.3cm}
$R_1 \circ R_2 = \{ \langle x, z \rangle \mid \exists y:\bigl(\pair(x,y) \in R_1 \wedge \pair(y,z) \in R_2\bigr) \}$.

\subsection{Computing the Paths}
So far, given a graph represented by a relation $R$ and two vertices $x$ and $y$, we can only check
whether there is a path leading from $x$ to $y$, but we cannot compute this path.  In this
subsection we will extend the procedure \texttt{transClosure} so that it will also compute the
corresponding path.  The main idea is to extend the notion of a relational product to the notion of
a \blue{path product}, where a \blue{path product} is defined on sets of paths.  In order to do so,
we introduce three functions for tuples.
\begin{enumerate}
\item Given a tuple $T$, the function $\texttt{first}(T)$ returns the first element of $T$:
      \\[0.2cm]
      \hspace*{1.3cm}
      $\texttt{first}\bigl(\langle x_1,\cdots,x_m\rangle\bigr) = x_1$.
\item Given a tuple $T$, the function $\texttt{last}(T)$ returns the last element of $T$:
      \\[0.2cm]
      \hspace*{1.3cm}
      $\texttt{last}\bigl(\langle x_1,\cdots,x_m\rangle\bigl) = x_m$.
\item If $S = \langle x_1, \cdots, x_m\rangle$ and $T = \langle y_1, \cdots, y_n \rangle$ are two tuples  such that
      $\texttt{first}(S) = \texttt{last}(S)$, we define the \blue{join} $S \oplus T$ of $S$ and $T$ as \\[0.2cm]
      \hspace*{1.3cm}
      $S \oplus T = \langle x_1, \cdots, x_m, y_2, \cdots, y_n \rangle$.
\end{enumerate}
If $\mathcal{P}_1$ and $\mathcal{P}_2$ are sets of tuples representing paths, we define the \blue{path product} of
$\mathcal{P}_1$ and $\mathcal{P}_2$ as follows: \\[0.2cm]
\hspace*{1.3cm}
$\mathcal{P}_1 \bullet \mathcal{P}_2 =
\bigl\{\; T_1 \oplus T_2 \mid T_1 \in \mathcal{P}_1 \wedge T_2 \in \mathcal{P}_2 \wedge \texttt{last}(T_1) = \texttt{first}(T_2) \;\bigr\}
$.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def findPaths(R):
        P = R;
        while True:
            oldP = P
            P    = R.union(pathProduct(P, R))
            print(P)
            if P == oldP:
                return P

    def pathProduct(P, Q):
        return { join(S, T) for S in P for T in Q if S[-1] == T[0] }

    def join(S, T):
        return S + T[1:]

    R = { (1,2), (2,3), (1,3), (2,4), (4,5) }
    print("R = ", R)
    print("Computing all paths:" )
    P = findPaths(R)
    print("P = ", P)
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing all connections.}  \label{fig:path.py}
\end{figure} %\$

\begin{figure}[!ht]
  \centering
  \vspace*{-9cm}

  \epsfig{file=Figures/graph-zykl,scale=0.5}
  \vspace*{-1cm}

  \caption{A graph with a cycle.}
  \label{fig:graph-zykl}
\end{figure}

Using the notion of a \blue{path product} we are able to extend the program shown in Figure
\ref{fig:transitive-closure.py} such that it computes all paths between two vertices.
The resulting program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/path.py}{\texttt{path.py}}
is shown in Figure \ref{fig:path.py} on page \pageref{fig:path.py}.
Unfortunately, the program does not work any more if the graph is \blue{cyclic}.  A graph is defined
to be \blue{cyclic} if there is a path of length greater than $1$ that starts and ends at the same
vertex.  This path is then called a \blue{cycle}.
Figure \ref{fig:graph-zykl} on page \pageref{fig:graph-zykl} shows a cyclic graph.  This graph is
cyclic because it contains the path
\\[0.2cm]
\hspace*{1.3cm}
$\langle 1, 2, 4, 1 \rangle$
\\[0.2cm]
and this path is a cycle.
The problem with this graph is that it contains an infinite number of paths that connect the vertex
1 with the vertex 2: \\[0.2cm]
\hspace*{1.3cm}
$\langle 1, 2 \rangle$, $\langle 1, 2, 4, 1, 2 \rangle$,
$\langle 1, 2, 4, 1, 2, 4, 1, 2 \rangle$,
$\langle 1, 2, 4, 1, 2, 4, 1, 2, 4, 1, 2 \rangle$, $\cdots$
\\[0.2cm]
Of course, there is no point in computing a path that visits a vertex more than once as these paths
contain cycles.  Our goal is to eliminate all those paths that contain cycles.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ numbers       = left,
                  numbersep     = -0.2cm,
                  frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  xleftmargin   = 0.0cm,
                  xrightmargin  = 0.0cm,
                ]
    def pathProduct(P, Q):
        return { join(S, T) for S in P for T in Q
                            if S[-1] == T[0] and noCycle(S, T)
               }

    def noCycle(T1, T2):
        return len({ x for x in T1 }.intersection({ x for x in T2 })) == 1
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing the connections in a cyclic graph.}
\label{fig:path-cyclic.py}
\end{figure} %\$

Figure \ref{fig:path-cyclic.py} on page shows how the implementation of the function
\texttt{pathProduct} has to be changed so that the resulting program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/path-cyclic.py}{\texttt{path-cyclic.py}}
works also for cyclic graphs.
\begin{enumerate}
\item In line 2 and 3, we compute only those paths that are not cyclic.
\item Line 6 defines a function \texttt{noCycle} that tests, whether the join  $\texttt{T1} \oplus \texttt{T2}$ is cyclic.  The join
      of \texttt{T1} and \texttt{T2} is cyclic iff the tuples \texttt{T1} and \texttt{T2} have more
      than one common element.  The tuples \texttt{T1} and \texttt{T2} will always have at least one common element, as we join
      these tuples only if the last element of \texttt{T1} is equal to the first element of  \texttt{T2}.
      If there would be an another vertex common to both \texttt{T1} and \texttt{T2}, then the path
      $\texttt{T1} \oplus \texttt{T2}$ would be cyclic.
\end{enumerate}

In general, we are not really interested to compute all possible paths between two given vertices
\texttt{x} and \texttt{y}.  Instead, we just want to compute the shortest path leading from \texttt{x} to \texttt{y}.
Figure \ref{fig:find_path.py} on page \pageref{fig:find_path.py} shows the procedure \texttt{reachable}.
This procedure takes three arguments:
\begin{enumerate}
\item \texttt{start} and \texttt{goal} are vertices of a graph.
\item \texttt{R} is a binary relation representing a directed graph.
\end{enumerate}
The call  \texttt{reachable(start, goal, R)} checks whether \texttt{start} and \texttt{goal} are connected and, furthermore,
computes the shortest path from \texttt{start} to \texttt{goal}, provided such a path exists.
The complete program can be found in the file
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/find\_path.py}{\texttt{find\_path.py}}.
Next, we discuss the implementation of the procedure  \texttt{reachable}.
\begin{enumerate}
\item Line 2 initializes the set \texttt{P}.  After $n$ iterations, this set will contain all paths
      that start with the vertex \texttt{start} and that have a length of at most $n$.

      Initially, there is just the trivial path $\langle\texttt{start}\rangle$ that starts with vertex
      \texttt{start} and has length $0$.
\item Line 5 tries to extend all previously computed paths by one step.
      If we are lucky, the set \texttt{P} is increased in this step.
\item Line 6 selects all those paths from the set \texttt{P} that lead to the vertex \texttt{goal}.
      These paths are stored in the set \texttt{Found}.
\item Line 7 checks whether we have indeed found a path ending at \texttt{goal}.  This is the case if
      the set \texttt{Found} is not empty.
      In this case, we return any of these paths.
\item If we have not yet found the vertex \texttt{goal} and, furthermore, we have not been able to find
      any new paths during this iteration,  the procedure returns in line 10.
      As the \texttt{return} statement in line 11 does not return a value, the procedure will
      instead return the value \texttt{None}.
\end{enumerate}
The procedure call \texttt{reachable(start,goal R)} will compute the \textbf{shortest} path connecting
\texttt{start} and \texttt{goal} because it computes path with increasing length.  The first iteration
computes all paths starting in \texttt{start} that have a length of at most 1, the second iteration
computes all paths starting in \texttt{start} that have a length of at most 2, and in general the $n$-th
iteration computes all paths starting in \texttt{start} that have a length of at most $n$.  Hence, if
there is a path of length $n$, then this path will be found in the $n$-iteration unless a shorter path has
already been found in a previous iteration.

\remarkEng
The algorithm described above is known as
\href{https://en.wikipedia.org/wiki/Breadth-first_search}{breadth first search}. \eox

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def reachable(start, goal, R):
        P = { (start,) }
        while True:
            oldP  = P
            P     = P.union(path_product(P, R))
            Found = { T for T in P if T[-1] == goal }
            if Found != set({}):
                return Found.pop()
            if P == oldP:
                return

    def path_product(P, R):
        return set( add(T1, T2) for T1 in P for T2 in R
                             if T1[-1] == T2[0] and noCycle(T1, T2)
                  )

    def noCycle(T1, T2):
        return len(set(T1).intersection(set(T2))) == 1

    def add(T, P):
        return T + (P[-1],)
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Finding the shortest path between two vertices.}
\label{fig:find_path.py}
\end{figure}

\subsection{The Wolf, the Goat, and the Cabbage}
Next, we present an application of the theory developed so far.  We solve a problem that has puzzled
the greatest agricultural economists for centuries.  The puzzle we want to solve is known as the
\href{http://jeux.lulu.pagesperso-orange.fr/html/anglais/loupChe/loupChe1.htm}{wolf-goat-cabbage puzzle}:
\vspace*{0.3cm}

\begin{minipage}[c]{14cm}
{\sl
An agricultural economist has to sell a wolf, a goat, and a cabbage on a market place.  In order to
reach the market place, she has to cross a river.  The boat that she can use is so small that it can
only accommodate either the goat, the wolf, or the cabbage in addition to the agricultural economist.
Now if the agricultural economist leaves the wolf alone with the goat, the wolf will eat the goat.
If, instead, the agricultural economist leaves the goat with the cabbage, the goat will eat the cabbage.
Is it possible for the agricultural economist to develop a schedule that allows her to cross the river
without either the goat or the cabbage being eaten?
}
\end{minipage}
\vspace*{0.3cm}

\noindent
In order to compute a schedule, we first have to model the problem.  The various \blue{states} of the problem will
be regarded as \blue{vertices} of a graph and this graph will be represented as a binary relation.
To this end we define the set
\begin{verbatim}
  All = {'farmer', 'wolf, 'goat', 'cabbage'}
\end{verbatim}
Every node will be represented as a subset \texttt{S} of the set \texttt{All}.  The idea is that the set \texttt{S}
specifies those objects that are on the left side of the river.  We assume that initially the farmer
is on the left side of the river.
Therefore, the set of all possible states can be defined as the set
\begin{verbatim}
  States = {S for S in power(All) if not problem(S) and not problem(All-S)}
\end{verbatim}
Here, we have used the procedure \texttt{problem} to check whether a given set \texttt{S} has a problem.
Note that since \texttt{S} is the set of objects on the left side, the expression $\texttt{All-S}$
computes the set of objects on the right side of the river.

Next, a set \texttt{S} of objects has a problem if both of the following conditions
are satisfied:
\begin{enumerate}
\item The farmer is not an element of \texttt{S} and
\item either \texttt{S} contains both the goat and the cabbage or \texttt{S} contains both the wolf and the goat.
\end{enumerate}
Therefore, we can implement the function \texttt{problem} as follows:
\begin{verbatim}
  def problem(S):
      return ("farmer" not in S) and             \
             (("goat" in S and "cabbage" in S) or   # goat eats cabbage
              ("wolf" in S and "goat"    in S)   )  # wolf eats goat
\end{verbatim}
We proceed to compute the relation \texttt{R} that contains all possible transitions between
different states.  We will compute \texttt{R} using the formula:
\\[0.2cm]
\hspace*{0.75cm}
\texttt{R = R1 + R2;}
\\[0.2cm]
Here \texttt{R1} describes the transitions that result from the farmer crossing the river from left
to right, while \texttt{R2} describes the transitions that result from the farmer crossing the river
from right to left.  We can define the relation \texttt{R1} as follows:
\begin{verbatim}
  R1 = { (S, S-B) for S in States for B in power(S)
                  if S-B in States and 'farmer' in B and len(B) <= 2
       }
\end{verbatim}
Let us explain this definition in detail:
\begin{enumerate}
\item Initially, \texttt{S} is the set of objects on the left side of the river.  Hence, \texttt{S}
      is an element of the set of all states that we have defined as \texttt{P}.
\item \texttt{B} is the set of objects that are put into the boat and that do cross the river.  Of
      course, for an object to go into the boat is has to be on the left side of the river to begin
      with.  Therefore, \texttt{B} is a subset of \texttt{S} and hence an element of the power set
      of \texttt{S}.
\item Then  \texttt{S-B} is the set of objects that are left on the left side of the river after
      the boat has crossed.  Of course, the new state \texttt{S-B} has to be a state that does not
      have a problem.  Therefore, we check that \texttt{S-B} is an element of \texttt{States}.
\item Furthermore, the farmer has to be inside the boat.  This explains the condition
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{\symbol{39}farmer\symbol{39} in B}.
\item Finally, the boat can only have two passengers.  Therefore, we have added the condition
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{len(B) <= 2}.
\end{enumerate}
Next, we have to define the relation \texttt{R2}.  However, as crossing the river from right to left
is just the reverse of crossing the river from left to right, \texttt{R2} is just the \blue{inverse} of
\texttt{R1}.   Hence we define:
\begin{verbatim}
  R2 = { (S2, S1) for (S1, S2) in R1 }
\end{verbatim}
Next, the relation \texttt{R} is the union of \texttt{R1} and \texttt{R2}:
\begin{verbatim}
  R = R1.union(R2)
\end{verbatim}
Finally, the start state has all objects on the left side.  Therefore, we have
\begin{verbatim}
  start = All
\end{verbatim}
In the end, all objects have to be on the right side of the river.  That means that nothing is left
on the left side.  Therefore, we define
\begin{verbatim}
  goal = {}
\end{verbatim}

\begin{figure}[h]
  \centering

  \epsfig{file=Figures/wolf-goat-cabbage, scale=0.4}

  \caption{The relation \texttt{R} shown as a directed graph.}
  \label{fig:wolf-goat-cabbage.pdf}
\end{figure}

Figure \ref{fig:wolf-ziege} on page \pageref{fig:wolf-ziege} shows the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/wolf-goat-cabbage.py}{\texttt{wolf-goat-cabbage.py}}
that combines the statements shown so far.  The solution computed by this program is shown in Figure
 \ref{fig:wolf-ziege-solution}.

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ codes         = {\catcode`$=3\catcode`_=8\catcode`^=7},
                  frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.3cm,
                  xrightmargin  = 0.3cm,
                ]
    def problem(S):
        return ('farmer' not in S) and             \
               (('goat' in S and 'cabbage' in S) or   # goat eats cabbage
                ('wolf' in S and 'goat'    in S)   )  # wolf eats goat

    All   = frozenset( ['farmer', 'wolf', 'goat', 'cabbage'] )
    R1    = { (S, S - B) for S in States for B in power(S)
                         if S - B in States and 'farmer' in B and len(B) <= 2
            }
    R2    = { (S2, S1) for (S1, S2) in R1 }
    R     = R1.union(R2)
    start = All
    goal  = frozenset()
    Path  = findPath(start, goal, R)
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Solving the wolf-goat-cabbage problem.}
\label{fig:wolf-ziege}
\end{figure}

\begin{figure}[!ht]
  \centering
\begin{Verbatim}[ codes         = {\catcode`$=3\catcode`_=8\catcode`^=7},
                  frame         = lines,
                  framesep      = 0.3cm,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    {"cabbage", "farmer", "goat", "wolf"}                                 {}
                             >>>> {"farmer", "goat"} >>>>
    {"cabbage", "wolf"}                                   {"farmer", "goat"}
                             <<<< {"farmer"} <<<<
    {"cabbage", "farmer", "wolf"}                                   {"goat"}
                             >>>> {"farmer", "wolf"} >>>>
    {"cabbage"}                                   {"farmer", "goat", "wolf"}
                             <<<< {"farmer", "goat"} <<<<
    {"cabbage", "farmer", "goat"}                                   {"wolf"}
                             >>>> {"cabbage", "farmer"} >>>>
    {"goat"}                                   {"cabbage", "farmer", "wolf"}
                             <<<< {"farmer"} <<<<
    {"farmer", "goat"}                                   {"cabbage", "wolf"}
                             >>>> {"farmer", "goat"} >>>>
    {}                                 {"cabbage", "farmer", "goat", "wolf"}
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A schedule for the agricultural economist.}
\label{fig:wolf-ziege-solution}
\end{figure}
\pagebreak
\vspace*{\fill}

\section{Symbolic Differentiation}
In this section we will develop a program that reads an arithmetic expression like
\\[0.2cm]
\hspace*{1.3cm}
\texttt{"x * exp(x)"},
\\[0.2cm]
interprets this string as describing the real valued function
\\[0.2cm]
\hspace*{1.3cm}
$x \mapsto x \cdot \exp(x)$,
\\[0.2cm]
and then takes the derivative of this function with respect to the variable $x$.  In order to specify the input
of this program more clearly, we first define the notion of an \blue{arithmetic expression} inductively.
\begin{enumerate}
\item Every number $c \in \mathbb{R}$ is an arithmetic expression.
\item Every variable $v$ is an arithmetic expression.
\item If $s$ and $t$ are arithmetic expressions, then
      \\[0.2cm]
      \hspace*{1.3cm}
      $s + t$, \quad $s - t$, \quad $s * t$, \quad $s / t$, \quad and \quad $s \,\mathtt{**}\, t$
      \\[0.2cm]
      are arithmetic expressions.  Here $s \,\mathtt{**}\, t$ is interpreted as $s^t$.

\item If $e$ is an arithmetic expression, then both
      \\[0.2cm]
      \hspace*{1.3cm}
      $\exp(e)$ \quad and \quad $\ln(e)$
      \\[0.2cm]
      are arithmetic expressions.
\end{enumerate}
We want do implement a function \texttt{diff} that takes two arguments:
\begin{enumerate}
\item The first argument \texttt{expr} represents an arithmetic expression.
\item The second argument \texttt{var} is the name of a variable.
\end{enumerate}
The function call \texttt{diff(expr, var)} will then compute the derivative of \texttt{expr} with respect to the variable \texttt{var}.  For example, the function call \texttt{diff("x*exp(x)", "x")} will compute the output
\\[0.2cm]
\hspace*{1.3cm}
"\texttt{1*exp(x) + x*exp(x)}"
\\[0.2cm]
because we have:
$$ \frac{\mathrm{d}\;}{\mathrm{d}x} \bigl( x \cdot \mathrm{e}^x \bigr) = 1 \cdot x + x \cdot \mathrm{e}^x $$
It would be very tedious to \blue{represent} arithmetic expressions as strings.  Instead, we will represent
arithmetic expressions as \blue{nested tuples}.  The notion of a \emph{nested tuple} is defined inductively:
\begin{itemize}
\item $\langle x_1, x_2, \cdots, x_n \rangle$ is a nested tuple if each of the components $x_i$ is either a
      number, a string, or a nested tuple.
\end{itemize}
For example, the arithmetic expression "\texttt{x*exp(x)}" is represented as the nested tuple
\\[0.2cm]
\hspace*{1.3cm}
$\langle\texttt{"}*\texttt{"}, \texttt{"}x\texttt{"}, \langle \texttt{"}\mathtt{exp}\texttt{"}, \texttt{"}x\texttt{"} \rangle\rangle$.
\\[0.2cm]
In order to be able to convert string into nested tuples, we need a \blue{parser}.  A parser is a program that
takes a string as input and transforms this string into a nested tuple, which is then returned as a result.
I have implemented a parser in the file "\texttt{exprParser.py}".  The details of the implementation of this
parser will be discussed in the lecture on algorithms.

\noindent
We close this section by showing an example that demonstrates the power of matching.
The function \texttt{diff} that is shown in Figure \ref{fig:diff.py} on page \pageref{fig:diff.py} is part
of the program
\href{https://github.com/karlstroetmann/Logik/blob/master/Python/diff.py}{\texttt{diff.py}}.
This function is called with one argument:
The argument \texttt{e} is an arithmetic expression.
The function \texttt{diff} interprets its argument \texttt{e} as a function of the variable
\texttt{x}.  We take the \href{https://en.wikipedia.org/wiki/Derivative}{derivative} of this
function with respect to the variable \texttt{x}.  For example, in order to compute the derivative of
the function
\\[0.2cm]
\hspace*{1.3cm}
$x \mapsto x^x$,
\\[0.2cm]
we can call the function  \texttt{diff} as follows:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{diff("x ** x")}.
\\[0.2cm]
Let us now discuss the implementation of the function \texttt{diff} in more detail.
\begin{enumerate}
\item The lines 3 - 6 implement the rule:
      $$\frac{\mathrm{d}\;}{\mathrm{d}x}\bigl(f(x) + g(x)\bigr) = \frac{\mathrm{d}\;}{\mathrm{d}x} f(x) + \frac{\mathrm{d}\;}{\mathrm{d}x} g(x)$$
\item Line 7 - 10 implement the rule:
      $$\frac{\mathrm{d}\;}{\mathrm{d}x}\bigl(f(x) - g(x)\bigr) = \frac{\mathrm{d}\;}{\mathrm{d}x} f(x) - \frac{\mathrm{d}\;}{\mathrm{d}x} g(x)$$
\item Line 11 - 14 deals with the case where \texttt{e} is a product.  The
      \href{https://en.wikipedia.org/wiki/Product\_rule}{product rule} is
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x}\bigl(f(x) \cdot g(x)\bigr) = \left(\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)\right)\cdot g(x) + f(x) \cdot \left(\frac{\mathrm{d}\;}{\mathrm{d}x} g(x)\right)
      $$
\item Line 15 - 17 deals with the case where \texttt{e} is a quotient.  The
      \href{https://en.wikipedia.org/wiki/Quotient\_rule}{quotient rule} is
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x}\left(\frac{f(x)}{g(x)}\right) =
         \frac{\displaystyle\left(\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)\right)\cdot g(x) -
         f(x) \cdot \left(\frac{\mathrm{d}\;}{\mathrm{d}x} g(x)\right)}{g(x) \cdot g(x)}
      $$
\item Line 19 - 21 deals with the case where \texttt{e} is a power.  Now in order to take the derivative of an
      expression of the form
      $$  f(x)^{g(x)} $$
      we first need to rewrite this expression using the following trick:
      $$ f(x)^{g(x)} = \exp\bigl(\ln\bigl(f(x)^{g(x)}\bigr)\bigr) = \exp\bigl(g(x) \cdot \ln(f(x))\bigr) $$
      Then, we can recursively call \texttt{diff} for this expression.  This works, because the function
      \texttt{diff} can deal with both the exponential function $x \mapsto \exp(x)$ and with the natural
      logarithm $x \mapsto \ln(x)$.  This rewriting is done in line 21.
\item Line 22-25 deals with the case where \texttt{e} has the form
      $$\ln\bigl(f(x)\bigr)$$
      In order to take the derivative of this expression, we first need to know the derivative of the natural
      logarithm.  This derivative is given as
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x} \ln(x) = \frac{1}{x}$$
      Then, using the \href{https://en.wikipedia.org/wiki/Chain\_rule}{chain rule} we have that
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x} \ln\bigl(f(x)\bigr) = \frac{\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)}{f(x)}$$
\item Line 26 - 29 deals with the case where \texttt{e} has the form $\exp\bigl(f(x)\bigr)$.
      In order to take the derivative of this expression, we first need to know the derivative of the
      \href{https://en.wikipedia.org/wiki/Exponential\_function}{exponential function}.
      This derivative is given as
      $$ \frac{\mathrm{d}\;}{\mathrm{d}x} \exp(x) = \exp(x)$$
      Then, using the \href{https://en.wikipedia.org/wiki/Chain\_rule}{chain rule} we have that
      $$\frac{\mathrm{d}\;}{\mathrm{d}x} \exp\bigl(f(x)\bigr) = \left(\frac{\mathrm{d}\;}{\mathrm{d}x} f(x)\right) \cdot \exp\bigl(f(x)\bigr) $$
\item Line 30-31 deals with the case where \texttt{e} is a variable and happens to be the same variable as
      \texttt{x}.  This is checked using the condition
      \texttt{e == x}.  As we have
      $$\frac{\mathrm{d}x}{\mathrm{d}x} = 1,$$
      the function \texttt{diff} returns \texttt{1} in this case.
\item Otherwise, the expression is assumed to be a constant and hence we return 0.
\end{enumerate}

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    def diff(e):
        "differentiate the expressions e with respect to the variable x"
        if e[0] == '+':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('+', fs, gs)
        if e[0] == '-':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('-', fs, gs)
        if e[0] == '*':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('+', ('*', fs, g), ('*', f, gs))
        if e[0] == '/':
            f , g  = e[1:]
            fs, gs = diff(f), diff(g)
            return ('/', ('-', ('*', fs, g), ('*', f, gs)), ('*', g, g))
        if e[0] == '**':
            f , g  = e[1:]
            return diff(('exp', ('*', g, ('ln', f))))
        if e[0] == 'ln':
            f  = e[1]
            fs = diff(f)
            return ('/', fs, f)
        if e[0] == 'exp':
            f  = e[1]
            fs = diff(f)
            return ('*', fs, e)
        if e == 'x':
            return '1'
        return 0
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A function for symbolic differentiation}
\label{fig:diff.py}
\end{figure}

In order to test this function we can implement a function \texttt{test} as shown in Figure \ref{fig:test-diff.py}.
Then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{diff("x ** x")}
\\[0.2cm]
yields the result:
\\[0.2cm]
\hspace*{1.3cm}
d/dx x ** x = (1*ln(x) + x*1/x)*exp(x*ln(x))
\\[0.2cm]
This shows that
\\[0.2cm]
\hspace*{1.3cm}
$\ds \frac{\mathrm{d};}{\mathrm{d}x} x^x = \bigl(\ln(x) + 1\bigr) \cdot \exp\bigl(x \cdot \ln(x)\bigr) =
 \bigl(\ln(x) + 1\bigr) \cdot x^x
$.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    import exprParser as ep

    def test(s):
        t = ep.ExprParser(s).parse()
        d = diff(t)
        print(f"d/dx {s} = {ep.toString(d)}")
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Testing symbolic differentiation.}
\label{fig:test-diff.py}
\end{figure}

\pagebreak
\vspace*{\fill}

\section{Reflection}
After having completed this chapter, you should be able to answer the following questions.
\begin{enumerate}
\item Which data types are supported in \textsl{Python}?
\item What are the different methods to define a set in \textsl{Python}?
\item Do you understand how to construct lists via iterator?
\item How can lists be defined in \textsl{Python}?
\item How does \textsl{Python} support binary relations?
\item How does list slicing and list indexing work?
\item What are nested tuples?
\item How does a fixed-point algorithm work?
\item What type of control structures are supported in \textsl{Python}?
\end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "logic"
%%% End: