\chapter{Finite Markov Chains}\label{C:FiniteMarkovChains}

{\bf NOTE: No materials from this Chapter will be in Probability Theory I exam!}\\[6pt]

{\em This topic is only introduced briefly in Probability Theory I to give concrete instances of dependent sequence of random variables.
We will revisit these ideas in the sequel. You only need to understand the ideas behind:
\bit
\item Example~\ref{EX:FlippantFreddy} and
\item Example~\ref{EX:DryWetChain}.
\eit
as done in lectures.}

{
\section{Stochastic Processes}\label{S:StochProc}

\begin{definition}[Stochastic Process]
A collection of RVs  \[
\left(X_{\alpha} \right)_{\alpha \in N} := \left( \  X_{\alpha} : \alpha \in \Az \  \right)
\]
is called a {\bf stochastic process}.  Thus, for every $\alpha \in  \Az$, the index set of the stochastic process, $X_{\alpha}$ is a RV.  If the index set $ \Az  \subset \Zz$ then we have  a {\bf discrete time stochastic process}, typically denoted by
\[
\left(X_i\right)_{i \in \Zz} := \ldots, X_{-2},X_{-1},X_0, X_1,X_2,\ldots , \  \text{or}
\]
\[
\left( X_i \right)_{i \in \Nz} := X_1,X_2,\ldots , \  \text{or}
\]
\[
\left( X_i \right)_{i \in [n]} := X_1,X_2,\ldots , X_n , \ \text{where, } [n]:= \{1,2,\ldots,n\} \ .
\]
If $\Az \subset \Rz$ then we have a {\bf continuous time stochastic process}, typically denoted by $\{X_t\}_{t \in \Rz}$, etc.
\end{definition}

Of course the above process is quite general and can allow for arbitrary dependence among the RVs.
Generally, we cannot produce useful models without making simplifying assumptions.
The absolutely simplest but extremely useful assumption is that of the {\bf Independent and Identically Distributed or IID Process} or merely {\bf IID Sequence of RVs} when the index set is a subset of $\Nz$.

This is exactly the sequence of RVs associated with our product experiment $\mathcal{E}^{\otimes \infty} := (\Omega, \mathcal{F}_{\mathcal{X}}, P_{\theta})^{\otimes \infty}$.

\begin{definition}[Independent and Identically Distributed (IID) Process]
The finite or infinite sequence of RVs or the stochastic process $X_1, X_2,\ldots$ is said to be independent and identically distributed or IID if :
\begin{itemize}
\item they are an idependently distributed according to \hyperref[D:IndRVs]{Definition \ref*{D:IndRVs}}, and
\item $F(X_1) = F(X_2) = \cdots $, ie.~all the $X_i$'s have the same DF $F(X_1)$.
\end{itemize}
This is perhaps the most elementary class of stochastic processes and we succinctly denote it by
\[
\left(X_i\right)_{i \in [n]} := X_1, X_2,\ldots, X_n \overset{\IID}{\sim} F, \quad \text{or} \quad \left(X_i\right)_{i \in \Nz} := X_1, X_2,\ldots  \overset{\IID}{\sim} F \ .
\]
We sometimes replace the DF $F$ above by the name of the RV.
 \end{definition}

\begin{definition}[Independently Distributed]
The sequence of RVs or the stochastic process $\left(X_i\right)_{i \in \Nz} := X_1, X_2,\ldots$ is said to be independently distributed if :
\begin{itemize}
\item $X_1, X_2,\ldots$ is independently distributed according to \hyperref[D:IndRVs]{Definition \ref*{D:IndRVs}}.
\end{itemize}
This is a class of stochastic processes that is more general than the IID class.
\end{definition}

As an example of such a class consider the sequence of RVs that are independent but non-identically distributed with each $X_i \sim \bernoulli(\theta_i)$.

When a stochastic process $\left(X_{\alpha} \right)_{\alpha \in \Az}$ is not independent it is said to be dependent.
So far we have mostly concerned ourselves with independent processes.
In this chapter we introduce finite Markov chains and their  simulation methods.
Finit Markov chains are among the simplest stochastic processes with a `first-order' dependence called Markov dependence.

\section{Introduction}\label{S:FiniteMCIntro}
A finite Markov chain is a stochastic process that moves among elements in a finite set $\Xz$ as follows: when at $x \in \Xz$ the next position is chosen at random according to a fixed probability distribution $P(\cdot | x)$.  We define such a process more formally below.

\begin{definition}[Finite Markov Chain]\label{D:TimeHomFiniteMC}
A stochastic sequence, $$\left(X_n\right)_{n \in \Zz_+} := (X_0,X_1,\ldots),$$ is a homogeneous {\bf Markov chain} with {\bf state space} $\Xz$ and {\bf transition matrix} $P:=\left(P(x,y)\right)_{(x,y)\in \Xz^2}$ if for all pair of {\bf states} ${(x,y)\in \Xz^2 := \Xz \times \Xz}$, all integers $t \geq 1$, and all probable historical events $H_{t-1} := \bigcap_{n=0}^{t-1} \{ X_n = x_n \}$ with $\P \left(H_{t-1} \cap \{X_t = x\} \right) > 0$, the following {\bf Markov property} is satisfied:
\begin{equation}\label{E:FiniteMarkovProperty}
\P\left(X_{t+1} = y | H_{t-1} \cap \{X_t = x\} \right)=\P\left(X_{t+1} = y | X_t = x \right) =: P(x,y) \enspace .
\end{equation}
\end{definition}
The Markov property means that the conditional probability of going to state $y$ at time $t+1$ from state $x$ at current time $t$ is always given by the $(x,y)$-th entry $P(x,y)$ of the transition matrix $P$, no matter what sequence of states $(x_0,x_1,\ldots,x_{t-1})$ preceded the current state $x$.  Thus, the $|\Xz| \times |\Xz|$ matrix $P$ is enough to obtain the state transitions since the $x$-th row of $P$ is the probability distribution $P(x,\cdot) := \left( P(x,y) \right)_{y \in \Xz}$.  For this reason $P$ is called a {\bf stochastic matrix}, i.e.,
\begin{equation}\label{E:StochasticMatrixConds}
P(x,y) \geq 0 \quad \text{for all } (x,y) \in \Xz^2 \qquad \text{and} \quad \sum_{y \in \Xz} P(x,y) = 1 \quad \text{for all } x \in \Xz \enspace .
\end{equation}
Thus, for a Markov chain $\left(X_n\right)_{n \in \Zz_+}$, the distribution of $X_{t+1}$ given $X_0,\ldots,X_t$  depends on $X_t$ alone. Because of this dependence on the previous state, the stochastic sequence, $(X_0,X_1,\ldots)$, are {\it not} independent.  We introduce the most important concepts using a simple example.

\begin{example}[Flippant Freddy]\label{EX:FlippantFreddy}
Freddy the flippant frog lives in an enchanted pond with only two lily pads, {\em rollopia} and {\em flipopia}.  A wizard gave a  die and a silver coin to help flippant Freddy decide where to jump next.  Freddy left the die on rollopia and the coin on flipopia.  When Freddy got restless in rollopia he would roll the die and if the die landed odd he would leave the die behind and jump to flipopia, otherwise he would stay put.  When Freddy got restless in flipopia he would flip the coin and if it landed Heads he would leave the coin behind and jump to rollopia, otherwise he would stay put.

Let the state space $\Xz=\{r,f\}$, and let $(X_0, X_1,\ldots)$ be the sequence of lily pads occupied by Freddy after his restless moments.  Say the die on rollopia $r$ has probability $p$ of turning up odd and the coin on flipopia $f$ has probability $q$ of turning up heads.  We can visualise the rules of Freddy's jumps by the following {\bf transition diagram}:
\begin{figure}[htpb]
\caption{Transition Diagram of Flippant Freddy's Jumps.\label{F:FlippantFreddyTransDiag}}
\centering   \makebox{\includegraphics[width=6.5in]{figures/FlippantFreddyTransDiag}}
\end{figure}

Then Freddy's sequence of jumps $(X_0, X_1,\ldots)$ is a Markov chain on $\Xz$ with transition matrix:
\begin{equation}\label{E:FlippantFreddyP}
P
= \bordermatrix{~ & r & f \cr
r & P(r,r) & P(r,f)\cr
f & P(f,r) & P(f,f) }
= \bordermatrix{~ & r & f \cr
r & 1-p & p \cr
f & q & 1-q } \enspace .
\end{equation}
Suppose we first see Freddy in rollopia, i.e., $X_0=r$.  When he gets restless for the first time we know from the first row of $P$ that he will leave to flippopia with probability $p$ and stay with probability $1-p$, i.e.,
\begin{equation}\label{E:Freddy1Step}
\P(X_1=f | X_0=r) = p, \quad \P(X_1=r | X_0=r) = 1-p \enspace .
\end{equation}
What happens when he is restless for the second time?  By considering the two possibilities for $X_1$, Definition of conditional probability and the Markov property, we see that,
\begin{eqnarray}
\P(X_2 = f | X_0 = r)
&=& \P(X_2=f, X_1=f | X_0=r) + \P(X_2=f, X_1=r | X_0=r) \notag \\
&=& \frac{\P(X_2=f, X_1=f , X_0=r)}{\P(X_0=r)} + \frac{\P(X_2=f, X_1=r , X_0=r)}{\P(X_0=r)} \notag \\
&=& \P(X_2=f | X_1=f , X_0=r)\frac{\P(X_1=f , X_0=r)}{\P(X_0=r)} \notag \\
&& \qquad + \P(X_2=f | X_1=r , X_0=r) \frac{\P(X_1=r , X_0=r)}{\P(X_0=r)} \notag \\
%&=& \frac{\P(X_2=f | X_1=f , X_0=r)\P(X_1=f | X_0=r)\P(X_0=r)}{\P(X_0=r)} \\
%&& \qquad + \frac{\P(X_2=f | X_1=r , X_0=r) \P(X_1=r | X_0=r)\P(X_0=r)}{\P(X_0=r)} \\
&=&{\P(X_2=f | X_1=f , X_0=r)\P(X_1=f | X_0=r)} \notag \\
&& \qquad +{\P(X_2=f | X_1=r , X_0=r) \P(X_1=r | X_0=r)} \notag \\
&=& \P(X_2=f | X_1=f) \P(X_1=f | X_0=r) \notag \\
&& \qquad+ \P(X_2=f | X_1=r) \P(X_1=r | X_0=r) \notag \\
&=& P(f,f) P(r,f) + P(r,f) P(r,r) \notag \\
&=& (1-q) p + p (1-p) \label{E:Freddy2Stepa}
\end{eqnarray}
Similarly,
\begin{equation} \label{E:Freddy2Stepb}
\P(X_2=r | X_0 =r) = P(f,r) P(r,f) + P(r,r) P(r,r) = q p + (1-p)(1-p)
\end{equation}
Instead of elaborate computations of the probabilities of being in a given state after Freddy's $t$-th restless moment, we can store the state probabilities at time $t$ in a row vector:
\[
\mu_t := \left(  \P(X_t = r | X_0 = r), \P(X_t=f | X_0=r) \right) \enspace ,
\]
Now, we can conveniently represent Freddy starting in rollopia by the {\bf initial distribution} $\mu_0 = (1,0)$ and obtain the 1-step {\bf state probability vector} in \eqref{E:Freddy1Step} from $\mu_1 = \mu_0 P$ and the 2-step state probabilities in \eqref{E:Freddy2Stepa} and \eqref{E:Freddy2Stepb} by $\mu_2 = \mu_1 P = \mu_0 P P = \mu_0 P^2$.  In general, multiplying $\mu_t$, the state probability vector at time $t$, by the transition matrix $P$ on the right updates the state probabilities by another step:
\[
\mu_{t} = \mu_{t-1} P \qquad \text{for all } t \geq 1 \enspace .
\]
And for any initial distribution $\mu_0$,
\[
\mu_{t} = \mu_0 P^t  \qquad \text{for all } t \geq 0 \enspace .
\]
This can be easily implemented in \Matlab as follows:
%\begin{VrbM}
%>> p=0.5; q=0.5; P = [1-p p; q 1-q] % assume a fair coin and a fair die
%P =
%    0.5000    0.5000
%    0.5000    0.5000
%
%>> mu0 = [1, 0] % inital state vector since Freddy started in rollopia
%mu0 =     1     0
%
%>> mu0*P^0    % intial state distribution at t=0 is just mu0
%ans =     1     0
%
%>> mu0*P^1    % state distribution at t=1
%ans =    0.5000    0.5000
%
%>> mu0*P^2    % state distribution at t=2
%ans =    0.5000    0.5000
%\end{VrbM}
%Thus for a fair coin and die we get equal probabilities of being in the two states right after the first jump.  Let us see what happens for an unfair coin and die:
\begin{VrbM}
>> p=0.85; q=0.35; P = [1-p p; q 1-q] % assume an unfair coin and an unfair die
P =
    0.1500    0.8500
    0.3500    0.6500
>> mu0 = [1, 0] % inital state vector since Freddy started in rollopia
mu0 =     1     0
>> mu0*P^0    % intial state distribution at t=0 is just mu0
ans =     1     0
>> mu0*P^1    % state distribution at t=1
ans =    0.1500    0.8500
>> mu0*P^2    % state distribution at t=2
ans =    0.3200    0.6800
>> mu0*P^3    % state distribution at t=3
ans =    0.2860    0.7140
\end{VrbM}
Now, let us compute and look at the probability of being in rollopia after having started there for three values of $p$ and $q$ according to the following script:
\VrbMf[label=FlippantFreddyRollopiaProbs.m]{scripts/FlippantFreddyRollopiaProbs.m}

\begin{figure}[htpb]
\caption{The probability of being back in rollopia in $t$ time steps after having started there under transition matrix $P$ with (i) $p=q=0.5$ (blue line with asterisks), (ii) $p=0.85$, $q=0.35$ (black line with dots) and (iii) $p=0.15$, $q=0.95$ (red line with pluses).\label{F:FlippantFreddyRollopiaProbs}}
\centering   \makebox{\includegraphics[width=6.5in]{figures/FlippantFreddyRollopiaProbs}}
\end{figure}

It is evident from \hyperref[F:FlippantFreddyRollopiaProbs]{Figure~\ref*{F:FlippantFreddyRollopiaProbs}} that as $t \to \infty$, $\mu_t$ approaches a distribution, say $\pi$, that depends on $p$ and $q$ in $P$.  Such a limit distribution is called the {\bf stationary distribution} and must satisfy the fixed point condition:
\[
\pi P = \pi \enspace ,
\]
that gives the solution:
\[
\pi(r) = \frac{q}{p+q}, \qquad \pi(f) = \frac{p}{p+q} \enspace .
\]
In \hyperref[F:FlippantFreddyRollopiaProbs]{Figure~\ref*{F:FlippantFreddyRollopiaProbs}} we see that $\P(X_t=r)$ approaches $\pi(r) = \frac{q}{p+q}$ for the three cases of $p$ and $q$:
\begin{align*}
\text{(i)}& \ p=0.50, q=0.50, & \P(X_t=r) & \to \pi(r) = \frac{q}{p+q} =  \frac{0.50}{0.50+0.50} = 0.5000,\\
\text{(ii)}& \  p=0.85, q=0.35, & \P(X_t=r) & \to \pi(r) = \frac{q}{p+q} =  \frac{0.35}{0.85+0.35} = 0.2917, \\
\text{(iii)}& \ p=0.15, q=0.95, & \P(X_t=r) & \to \pi(r) = \frac{q}{p+q} = \frac{0.95}{0.15+0.95} = 0.8636.
\end{align*}
\end{example}

Now let us generalise the lessons learned from \hyperref[EX:FlippantFreddy]{Example~\ref*{EX:FlippantFreddy}}.

\begin{prop}\label{P:FiniteMCProbsAtTimet}
For a finite Markov chain $\left(X_t\right)_{t \in \Zz_+}$ with state space $\Xz=\{s_1,s_2,\ldots,s_k\}$, initial distribution $$\mu_0 := \left( \mu_0(s_1), \mu_0(s_2), \ldots, \mu_0(s_k) \right),$$ where $\mu_0(s_i) = \P(X_0=s_i)$, and transition matrix $$P := \left(P(s_i,s_j)\right)_{(s_i,s_j)\in \Xz^2},$$ we have for any $t \in \Zz_+$ that the distribution at time $t$ given by:
$$\mu_t := \left( \mu_t(s_1), \mu_t(s_2), \ldots, \mu_t(s_k) \right),$$
where $\mu_t(s_i) = \P(X_t=s_i)$, satisfies:
\begin{equation}\label{E:mutismu0Pt}
\mu_t = \mu_0 P^t \enspace .
\end{equation}
\begin{proof}
We will prove this by induction on $\Z_+:=\{0,1,2,\ldots\}$.  First consider the case when $t=0$.  Since $P^0$ is the identity matrix $I$, we get the desired equality:
\[
\mu_0 P^0 = \mu_0 I = \mu_0 \enspace .
\]
Next consider the case when $t=1$.  We get for each $j \in \{1,2,\ldots,k\}$, that
\begin{align*}
\mu_1(s_j) &= \P(X_1 = s_j) = \sum_{i=1}^k \P(X_1=s_j,X_0=s_i)\\
&= \sum_{i=1}^k \P(X_1=s_j | X_0=s_i) \P(X_0=s_i) \\
&= \sum_{i=1}^k P(s_i,s_j) \mu_0(s_i)\\
&= (\mu_0 P)(s_j), \quad \text{the $j$-th entry of the row vector $(\mu_0 P)$} \enspace .
\end{align*}
Hence, $\mu_1=\mu_0 P$.  Now, we will fix $m$ and suppose that \eqref{E:mutismu0Pt} holds for $t=m$ and prove that \eqref{E:mutismu0Pt} also holds for $t=m+1$.
For each $j \in \{1,2,\ldots,k\}$, we get
\begin{align*}
\mu_{m+1}(s_j) &= \P(X_{m+1} = s_j) = \sum_{i=1}^k \P(X_{m+1}=s_j,X_m=s_i)\\
&= \sum_{i=1}^k \P(X_{m+1}=s_j | X_m=s_i) \P(X_m=s_i) \\
&= \sum_{i=1}^k P(s_i,s_j) \mu_m(s_i)\\
&= (\mu_m P)(s_j), \quad \text{the $j$-th entry of the row vector $(\mu_m P)$} \enspace .
\end{align*}
Hence, $\mu_{m+1}=\mu_m P$.  But $\mu_{m}=\mu_0 P^m$ by the induction hypothesis, and therefore:
\[
\mu_{m+1} = \mu_m P = \mu_0 P^m P = \mu_0 P^{m+1} \enspace .
\]
Thus by the principle of mathematical induction we have proved the proposition.
\end{proof}
\end{prop}
Thus, multiplying a row vector $\mu_0$ by $P^t$ on the right takes you from current distribution over the state space to the distribution in $t$ steps of the chain.

Since we will be interested in Markov chains on $\Xz =\{s_1,s_2,\ldots,s_k\}$ with the same transition matrix $P$ but different initial distributions, we introduce $\P_{\mu}$ and $\E_{\mu}$ for probabilities and expectations  given that the initial distribution is $\mu$, respectively.  When the initial distribution is concentrated at a single initial state $x$ given by:
$$\BB{1}_{\{x\}}(y) := \begin{cases} 1 & \text{if } y=x\\ 0 & \text{if } y \neq x \end{cases}$$
we represent it by $e_x$, the $1 \times k$ ortho-normal basis row vector with a $1$ in the $x$-th entry and a $0$ elsewhere.
We simply write $\P_x$ for $\P_{\BB{1}_{\{x\}}}$ or $\P_{e_x}$ and $\E_x$ for $\E_{\BB{1}_{\{x\}}}$ or $\E_{e_x}$.  Thus, \hyperref[P:FiniteMCProbsAtTimet]{Proposition~\ref*{P:FiniteMCProbsAtTimet}} along with our new notations means that:
\[
\P_x (X_t = y)  = (e_x P^t)(y) = P^t(x,y) \enspace .
\]
In words, the probability of going to $y$ from $x$ in $t$ steps is given by the $(x,y)$-th entry of $P^t$, the {\bf $t$-step transition matrix}.  We refer to the $x$-th row and the $x$-th column of $P$ by $P(x,\cdot)$ and $P(\cdot,x)$, respectively.

Let the function $f(x): \Xz\to \Rz$ be represented by the column vector $f := (f(s_1),f(s_2),\ldots,f(s_k)) \in \Rz^{k \times 1}$.  Then the $x$-th entry of $P^t f$ is:
\[
P^t f(x) = \sum_{y} P^t(x,y) f(y) = \sum_{y} f(y) \P_x (X_t = y) = \E_x (f(X_t)) \enspace .
\]
This is the expected value of $f$ under the distribution of states in $t$ steps given that we start at state $x$.
Thus multiplying a column vector $f$ by $P^t$ from the left takes you from a function on the state space to its expected value in $t$ steps of the chain.

%Let us look at some more examples of Markov chains.
\begin{example}[Dry-Wet Christchurch Weather]\label{EX:DryWetChain}
Consider a toy weather model for dry or wet days in Christchurch using a Markov chain with state space $\{d,w\}$.  Let the transition diagram in \hyperref[F:DryWetTransDiag]{Figure~\ref*{F:DryWetTransDiag}} give the transition matrix $P$ for our dry-wet Markov chain.
\begin{figure}[htpb]
\caption{Transition Diagram of Dry and Wet Days in Christchurch.\label{F:DryWetTransDiag}}
\centering   \makebox{\includegraphics[width=3.5in]{figures/DryWetTransDiag}}
\end{figure}
Using \eqref{E:mutismu0Pt} we can find that the probability of being dry on the day after tomorrow is $0.625$ given that it is wet today as follows:
\begin{VrbM}
>> P=[0.75 0.25; 0.5 0.5] % Transition Probability Matrix
P =
    0.7500    0.2500
    0.5000    0.5000
>> mu0=[0 1] % it is wet today gives the initial distribution
mu0 =     0     1
>> mu0 * P^2 % the distribution in 2 days from today
ans =    0.6250    0.3750
\end{VrbM}
Suppose you sell \$100 of lemonade at a road-side stand on a hot day but only \$50 on a cold day.  Then we can compute your expected sales tomorrow if today is dry as follows:
\begin{VrbM}
>> P=[0.75 0.25; 0.5 0.5] % Transition Probability Matrix
P =
    0.7500    0.2500
    0.5000    0.5000
>> f = [100; 50] % sales of lemonade in dollars on a dry and wet day
f =
   100
    50
>> P*f % expected sales tomorrow
ans =
   87.5000
   75.0000
>> mu0 = [1 0] % today is dry
mu0 =     1     0
>> mu0*P*f % expected sales tomorrow if today is dry
ans =   87.5000
\end{VrbM}
\end{example}

\begin{exercise}[Freddy discovers a gold coin]\label{EXR:FreddyGoldCoin}
Flippant Freddy of \hyperref[EX:FlippantFreddy]{Example~\ref*{EX:FlippantFreddy}} found a gold coin at the bottom of the pond.  Since this discovery he  jumps around differently in the enchanted pond.  He can be found now in one of three states: flipopia, rollopia and hydropia (when he dives into the pond). His state space is $\Xz=\{r,f,h\}$ now and his transition mechanism is as follows: If he rolls an odd number with his fair die in rollopia he will jump to flipopia but if he rolls an even number then he will stay in rollopia only if the outcome is $2$ otherwise he will dive into hydropia.  If the fair gold coin toss at the bottom of hydropia is Heads then Freddy will swim to flipopia otherwise he will remain in hydropia. Finally, if he is in flipopia he will remain there if the silver coin lands Heads otherwise he will jump to rollopia.

Make a Markov chain model of the new jumping mechanism adopted by Freddy.  Draw the transition diagram, produce the transition matrix $P$ and compute using \Matlab the probability that Freddy will be in hydropia after one, two, three, four and five jumps given that he starts in hydropia.
\end{exercise}

\begin{exercise}\label{Exr:NonMarkovProjection1}
Let $(X_t)_{t\in\Zz_+}$ be a Markov chain with state space $\{a,b,c\}$, initial distribution $\mu_0=(1/3,1/3,1/3)$ and transition matrix
$$P =
\bordermatrix{~ & a & b & c \cr
a & 0 & 1 & 0\cr
b & 0 & 0 & 1\cr
c & 1 & 0 & 0} \enspace .
$$
For each $t$, define $Y_t = \BB{1}_{\{b,c\}}(X_t)$.  Show that $(Y_t)_{t\in\Zz_+}$ is not a Markov chain.
\end{exercise}

\begin{exercise}\label{Exr:RegularSampledChainIsMarkov}
Let $(X_t)_{t\in\Zz_+}$ be a (homogeneous) Markov chain on $\Xz=\{s_1,s_2,\ldots,s_k\}$ with transition matrix $P$ and initial distribution $\mu_0$.  For a given $m \in \Nz$, let $(Y_t)_{t\in\Zz_+}$ be a stochastic sequence with $Y_t = X_{mt}$.  Show that $(Y_t)_{t\in\Zz_+}$ is a Markov chain with transition matrix $P^m$.  This establishes that Markov chains that are sampled at regular time steps are also Markov chains.
\end{exercise}

Until now our Markov chains have been {\bf  homogeneous} in time according to \hyperref[D:TimeHomFiniteMC]{Definition~\ref*{D:TimeHomFiniteMC}}, i.e., the transition matrix $P$ does not change with time.  We define inhomogeneous Markov chains that allow their transition matrices to possibly change with time.  Such Markov chains are more realistic as models in some situations and more flexible as algorithms in the sequel.

\begin{definition}[Inhomogeneous finite Markov chain]\label{D:TimeInhomFiniteMC}
Let $P_1,P_2,\ldots$ be a sequence of $k \times k$ stochastic matrices satisfying the conditions in \hyperref[E:StochasticMatrixConds]{Equation~\ref*{E:StochasticMatrixConds}}.  Then, the stochastic sequence $\left( X_t \right)_{t \in \Z_+} := (X_0,X_1,\ldots)$ with finite state space $\Xz := \{s_1,s_2,\ldots,s_k\}$ is called an inhomogeneous Markov chain with transition matrices $P_1,P_2,\ldots$, if for all pairs of states $(x,y) \in \Xz \times \Xz$, all integers $t \geq 1$, and all probable historical events $H_{t-1} := \bigcap_{n=0}^{t-1} \{ X_n = x_n \}$ with $\P\left(H_{t-1} \cap \{X_t = x\} \right) > 0$, the following {\bf Markov property} is satisfied:
\begin{equation}\label{E:InHomFiniteMarkovProperty}
\P\left(X_{t+1} = y | H_{t-1} \cap \{X_t = x\} \right)=\P\left(X_{t+1} = y | X_t = x \right) =: P_{t+1}(x,y) \enspace .
\end{equation}
\end{definition}

\begin{prop}\label{P:FiniteInHomMCProbsAtTimet}
For a finite inhomogeneous Markov chain $\left(X_t\right)_{t \in \Zz_+}$ with state space $\Xz=\{s_1,s_2,\ldots,s_k\}$, initial distribution $$\mu_0 := \left( \mu_0(s_1), \mu_0(s_2), \ldots, \mu_0(s_k) \right),$$ where $\mu_0(s_i) = \P(X_0=s_i)$, and transition matrices
$$\left(P_1,P_2,\ldots\right), \quad P_t := \left(P_t(s_i,s_j)\right)_{(s_i,s_j)\in \Xz\times \Xz}, \ t \in \{1,2,\ldots\}$$ we have for any $t \in \Zz_+$ that the distribution at time $t$ given by:
$$\mu_t := \left( \mu_t(s_1), \mu_t(s_2), \ldots, \mu_t(s_k) \right),$$
where $\mu_t(s_i) = \P(X_t=s_i)$, satisfies:
\begin{equation}\label{E:InhomMutismu0Pt}
\mu_t = \mu_0 P_1 P_2 \cdots P_t \enspace .
\end{equation}
\begin{proof}
Left as \hyperref[Exr:ProveInHomMultismu0Pt]{Exercise~\ref*{Exr:ProveInHomMultismu0Pt}}.
\end{proof}
\end{prop}

\begin{exercise}\label{Exr:ProveInHomMultismu0Pt}
Prove \hyperref[P:FiniteInHomMCProbsAtTimet]{Proposition~\ref*{P:FiniteInHomMCProbsAtTimet}} using induction as done for \hyperref[P:FiniteMCProbsAtTimet]{Proposition~\ref*{P:FiniteMCProbsAtTimet}}.
\end{exercise}

\begin{example}[a more sophisticated dry-wet chain]\label{EX:DryWetChainHotCold}
Let us make a more sophisticated version of the dry-wet chain of \hyperref[EX:DryWetChain]{Example~\ref*{EX:DryWetChain}} with state space $\{d,w\}$ .  In order to take some seasonality into account in our weather model for dry and wet days in Christchurch, let us have two transition matrices for hot and cold days:
\[
P_{\text{hot}} =
\bordermatrix{~ & d & w \cr
d & 0.95 & 0.05 \cr
w & 0.75 & 0.25},
\qquad
P_{\text{cold}} =
\bordermatrix{~ & d & w \cr
d & 0.65 & 0.35 \cr
w & 0.45 & 0.55} \enspace .
\]
We say that a day is hot if its  maximum temperature is more than $20^{\circ}$ Celsius, otherwise it is cold.  We use the transition matrix for today to obtain the state probabilities for tomorrow.  If today is dry and hot and tomorrow is supposed to be cold then what is the probability that the day after tomorrow will be wet?  We can use \eqref{E:InhomMutismu0Pt} to obtain the answer as $0.36$:
\begin{VrbM}
>> Phot = [0.95 0.05; 0.75 0.25] % Transition Probability Matrix for hot day
Phot =
    0.9500    0.0500
    0.7500    0.2500
>> Pcold = [0.65 0.35; 0.45 0.55] % Transition Probability Matrix for cold day
Pcold =
    0.6500    0.3500
    0.4500    0.5500
>> mu0 = [1 0] % today is dry
mu0 =     1     0
>> mu1 = mu0 * Phot % distribution for tomorrow since today is hot
mu1 =    0.9500    0.0500
>> mu2 = mu1 * Pcold % distribution for day after tomorrow since tomorrow is supposed to be cold
mu2 =    0.6400    0.3600
>> mu2 = mu0 * Phot * Pcold % we can also get the distribution for day after tomorrow directly
mu2 =    0.6400    0.3600
\end{VrbM}
\end{example}

\begin{exercise}\label{Exr:DryWetChainHotCold}
For the Markov chain in \hyperref[EX:DryWetChainHotCold]{Example~\ref*{EX:DryWetChainHotCold}}  compute the probability that the day after tomorrow is wet if today is dry and hot but tomorrow is supposed to be cold.
\end{exercise}

\section{Irreducibility and Aperiodicity}\label{S:IrredAperiod}
The utility of our mathematical constructions with Markov chains depends on a delicate balance between generality and specificity.  We introduce two specific conditions called irreducibility and aperiodicity that make Markov chains more useful to model real-word phenomena.

\begin{definition}[Communication between states]\label{D:Communication} Let $(X_t)_{t\in\Zz_+}$ be a homogeneous Markov chain with transition matrix $P$ on state space $\Xz:=\{s_1,s_2,\ldots,s_k\}$.
We say that a state $s_i$ {\bf communicates} with a state $s_j$ and write $s_i \rightarrow s_j$ or $s_j \leftarrow s_i$ if there exists an $\eta(s_i,s_j) \in \Nz$ such that:
\[
\P \left( X_{t+\eta(s_i,s_j)} = s_j | X_t = s_i \right) = P^{\eta(s_i,s_j)} (s_i, s_j) > 0 \enspace .
\]
In words, $s_i$ communicates with $s_j$ if you can eventually reach $s_j$ from $s_i$.  If $P^{\eta} (s_i, s_j)=0$ for every $\eta \in \Nz$ then we say that $s_i$ {\bf does not communicate} with $s_j$ and write $s_i  \nrightarrow s_j$ or $s_j  \nleftarrow s_i$.

We say that two states $s_i$ and $s_j$ {\bf intercommunicate} and write $s_i \leftrightarrow s_j$ if $s_i \rightarrow s_j$ and $s_j \rightarrow s_i$.  In words, two states intercommunicate if you can eventually reach one from another and vice versa.  When $s_i$ and $s_j$ do not intercommunicate we write $s_i \nleftrightarrow s_j$.
\end{definition}

\begin{definition}[Irreducible]\label{D:Irreducible}
A homogeneous Markov chain $(X_t)_{t\in\Zz_+}$ with transition matrix $P$ on state space $\Xz:=\{s_1,s_2,\ldots,s_k\}$ is said to be {\bf irreducible} if $s_i \leftrightarrow s_j$ for each $(s_i,s_j) \in \Xz^2$.  Otherwise the chain is said to be {\bf reducible}.
\end{definition}

We have already seen examples of reducible and irreducible Markov chains.  For example, Flippant Freddy's family of Markov chains with the $(p,q)$-parametric family of transition matrices, $\{P_{(p,q)} : (p,q) \in [0,1]^2\}$, where each $P_{(p,q)}$ is given by \hyperref[E:FlippantFreddyP]{Equation~\ref*{E:FlippantFreddyP}}.  If $(p,q) \in (0,1)^2$, then the corresponding Markov chain is irreducible because we can go from rollopia to flippopia or vice versa in just one step with a positive probability.  Thus, the Markov chains with transition matrices in $\{P_{(p,q)} : (p,q) \in (0,1)^2\}$ are irreducible.  But if $p$ or $q$ take probability values at the boundary of $[0,1]$, i.e., $p \in \{0,1\}$ or $q \in \{0,1\}$ then we have to be more careful because we may  never get from at least one state to the other and the corresponding Markov chains may be reducible.   For instance, if $p=0$ or $q=0$ then we will be stuck in either rollopia or flippopia, respectively.  However, if $p=1$ and $q \neq 0$ or $q=1$ and $p \neq 0$ then we can get from each  state to the other.  Therefore,  only the transition matrices in $\left\{P_{(p,q)} : p \in \{0\} \text{ or } q \in \{0\}\right\}$ are reducible.

The simplest way to verify whether a Markov chain is irreducible is by looking at its transition diagram (without the positive edge probabilities) and checking that from each state there is a sequence of arrows leading to any other state.  %For instance, from the transition diagram in \hyperref[F:SixLounges]{Figure~\ref*{F:SixLounges}} of the lounge-hopping Markov chain of \hyperref[EX:SixLounges]{Example~\ref*{EX:SixLounges}}, it is clear that if you start at state $6$ you cannot find any arrow going to any other state.  Therefore, the chain is reducible since $6 \nrightarrow i$ for any $i \in \{1,2,3,4,5\}$.

\begin{exercise}\label{EXR:ExsIrreducibleOrNot}
Revisit all the Markov chains we have considered up to now and determine whether they are reducible or irreducible by checking that from each state there is a sequence of arrows leading to any other state in their transition graphs.
\end{exercise}

\begin{definition}[Return times and period]
Let $\Tz(x) := \{t \in \Nz : P^t(x,x)>0\}$ be the set of {\bf possible return times} to the starting state $x$.  The {\bf period} of state $x$ is defined to be $\gcd(\Tz(x))$, the greatest common divisor of $\Tz(x)$.  When the period of a state $x$ is $1$, i.e., $\gcd(\Tz(x))=1$, then $x$ is said to be an {\bf aperiodic state}.
\end{definition}

\begin{prop}
If the Markov chain $(X_t)_{t\in\Zz_+}$ with transition matrix $P$ on state space $\Xz$ is irreducible then $\gcd(\Tz(x)) = \gcd(\Tz(y))$ for any $(x,y) \in \Xz^2$.
\begin{proof}
Fix any pair of states $(x,y) \in \Xz^2$.  Since, $P$ is irreducible, $x \leftrightarrow y$ and therefore there exists natural numbers $\eta(x,y)$ and $\eta(y,x)$ such that $P^{\eta(x,y)}(x,y)>0$ and $P^{\eta(y,x)}(y,x)>0$.  Let $\eta' = \eta(x,y)+\eta(y,x)$ and observe that $\eta' \in \Tz(x) \cap \Tz(y)$, $\Tz(x) \subset \Tz(y) - \eta' := \{t-\eta' : t \in \Tz(y)\}$ and $\gcd(\Tz(y))$ divides all elements in $\Tz(x)$.  Thus, $\gcd(\Tz(y)) \leq \gcd(\Tz(x))$.  By a similar argument we can also conclude that $\gcd(\Tz(x)) \leq \gcd(\Tz(y))$.  Therefore $\gcd(\Tz(x))=\gcd(\Tz(y))$.
\end{proof}
\end{prop}

\begin{definition}[Aperiodic]
A Markov chain $(X_t)_{t\in\Zz_+}$ with transition matrix $P$ on state space $\Xz$ is said to be {aperiodic} if all of its states are aperiodic, i.e., $\gcd(\Tz(x))=1$ for every $x \in \Xz$.  If a chain is not aperiodic, we call it {\bf periodic}.
\end{definition}

We have already seen example of irreducible Markov chains that  were either periodic or aperiodic.  For instance, Freddy's Markov chain with $(p,q) \in (0,1)^2$ is aperiodic since the period of either of its two states is given by $\gcd(\{1,2,3,\ldots\})=1$.  However, the Markov chain model for a drunkard's walk around a block over the state space $\{0,1,2,3\}$ %(\hyperref[SIM:DrunkardsWalkBlock]{Simulation~\ref*{SIM:DrunkardsWalkBlock}})
is periodic because you can only return to the starting state in an even number of time steps and
$$
\gcd(\Tz(0))=\gcd(\Tz(1))=\gcd(\Tz(2))=\gcd(\Tz(3))= \gcd \left( \{2,4,6,\ldots\} \right) =2 \neq 1 \enspace .
$$

\begin{exercise}\label{EXR:DrunkardsWalkOnKGonIrredAperiod}
Show that the Markov chain corresponding to a drunkard's walk around a polygonal block with $k$ corners is irreducible for any integer $k>1$.  Show that it is aperiodic only when $k$ is odd and has period $2$ when $k$ is even.
\end{exercise}

\begin{prop}\label{P:AdditionNonlattice} Let $A=\{a_1,a_2,\ldots\} \subset \Nz$ that satisfies the following two conditions:
\begin{enumerate}
\item $A$ is a {\bf nonlattice}, meaning that $\gcd(A)=1$ and
\item $A$ is closed undur addition, meaning that if $(a,a') \in A^2$ then $a+a' \in A$.
\end{enumerate}
Then there exists a positive integer $\eta < \infty$ such that $n \in A$ for all $n \geq \eta$.
\begin{proof}
See Proofs of Lemma 1.1, Lemma 1.2 and Theorem 1.1 in Appendix of {\em Pierre Br\'emaud, Markov Chains, Gibbs Fields, Monte Carlo Simulation, and Queues, Springer, 1999}.
\end{proof}
\end{prop}

\begin{prop}
If the Markov chain $(X_t)_{t\in\Zz_+}$ with transition matrix $P$ on state space $\Xz$ is irreducible and aperiodic then there is an integer $\tau$ such that $P^t(x,x)>0$ for all $t \geq \tau$ and all $x \in \Xz$.
\begin{proof}
TBD
\end{proof}
\end{prop}

\begin{prop}
If the Markov chain $(X_t)_{t\in\Zz_+}$ with transition matrix $P$ on state space $\Xz$ is irreducible and aperiodic then there is an integer $\tau$ such that $P^t(x,y)>0$ for all $t \geq \tau$ and all $(x,y) \in \Xz^2$.
\begin{proof}
TBD
\end{proof}
\end{prop}

\begin{exercise}[King's random walk on a chessboard]\label{EXR:KingRWChessBoard}
Consider the squares in the chessboard as the state space $\Xz = \{ 0,1,2,\ldots,7\}^2$ with a randomly walking black king, i.e., for each move from current state $(u,v) \in \Xz$ the king chooses one of his $k(u,v)$ possible moves uniformly at random.
Is the Markov chain corresponding to the randomly walking black king on the chessboard irredicible and/or aperiodic?
%Write a \Matlab script to simulate a sequence of $n$ states visited by the king if he started from $(0,0)$, the most south-west state on the chessboard.
\end{exercise}

\begin{exercise}[King's random walk on a chesstorus]\label{EXR:KingRWChessTorus}
We can obtain a chesstorus from a pliable chessboard by identifying the eastern edge with the western edge (roll the chessboard into a cylinder) and then identifying the northern edge with the southern edge (gluing the top and bottom end of the cylinder together by turning into a doughnut or torus).  Consider the squares in the chesstorus as the state space $\Xz = \{ 0,1,2,\ldots,7\}^2$ with a randomly walking black king, i.e., for each move from current state $(x,y) \in \Xz$ the king chooses one of his $8$ possible moves uniformly at random according to the scheme: $X_t \gets X_{t-1}+ W_t$, where $W_t$ is independent and identically distributed as follows:
\[
\P(W_t = w) =
\begin{cases}
\frac{1}{8} & \text{if } w \in \left\{ (1,1), (1,0), (1,-1), (0,-1), (-1,-1), (-1,0), (-1,1), (0,1) \right\} , \\
0 & \text{ otherwise}.
\end{cases}
\]
Is the Markov chain corresponding to the randomly walking black king on the chesstorus irredicible and/or aperiodic?  Write a \Matlab script to simulate a sequence of $n$ states visited by the king if he started from $(0,0)$ on the chesstorus.
\end{exercise}

\section{Stationarity}\label{S:Stationarity}

We are interested in statements about a Markov chain that has been running for a long time.
For any nontrivial Markov chain $(X_0,X_1,\ldots)$ the value of $X_t$ will keep fluctuating in the state space $\Xz$ as $t \to \infty$ and we cannot hope for convergence to a fixed point state $x^* \in \Xz$ or to a $k$-cycle of states $\{x_1,x_2,\ldots,x_k\} \subset \Xz$.  However, we can look one level up into the space of probability distributions over $\Xz$ that give the probability of the Markov chain visiting each state $x \in \Xz$ at time $t$, and hope that the distribution of $X_t$ over $\Xz$ settles down as $t \to \infty$.  The Markov chain convergence theorem indeed sattes that the distribution of $X_t$ over $\Xz$ settles down as $t \to \infty$, provided the Markov chain is irreducible and aperiodic.

\begin{definition}[Stationary distribution]
Let $\left( X_t \right)_{t \in \Zz_+}$ be a Markov chain with state space $\Xz=\{s_1,s_2,\ldots,s_k\}$ and transition matrix $P = \left( P(x,y) \right)_{(x,y) \in \Xz^2}$.  A row vector $$\pi = \left( \pi(s_1), \pi(s_2), \ldots, \pi(s_k) \right) \in \Rz^{1\times k}$$ is said to be a {\bf stationary distribution} for the Markov chain, if it satisfies the conditions of being:
\begin{enumerate}
\item {\em a probability distribution}: $\pi(x) \geq 0$ for each $x \in \Xz$ and $\sum_{x \in \Xz} \pi(x) = 1$, and
\item {\em a fixed point}: $\pi P = \pi$, i.e., $\sum_{x \in \Xz} \pi(x) P(x,y) = \pi(y)$ for each $y \in \Xz$.
\end{enumerate}
\end{definition}

\begin{definition}[Hitting times]
If a Markov chain $\left( X_t \right)_{t \in \Zz_+}$ with state space $\Xz=\{s_1,s_2,\ldots,s_k\}$ and transition matrix $P = \left( P(x,y) \right)_{(x,y) \in \Xz^2}$ starts at state $x$, then we can define the {\bf hitting time}
\[
T(x,y) = \min \{ t \geq 1: X_t = y \} \enspace .
\]
and let $T(x,y) = \min \{\} = \infty$ if the Markov chain never visits $y$ after having started from $x$.  Let the {\bf  mean hitting time}
\[
\tau(x,y) := \E(T(x,y)) ,
\]
be the expected time taken to reach $y$ after having started at $x$.  Note that $\tau(x,x)$ is the {\bf mean return time} to state $x$.
\end{definition}

\begin{prop}[Hitting times of irreducible aperiodic Markov chains]
If  $\left( X_t \right)_{t \in \Zz_+}$ is an irreducible aperiodic Markov chain with state space $\Xz=\{s_1,s_2,\ldots,s_k\}$, transition matrix $P = \left( P(x,y) \right)_{(x,y) \in \Xz^2}$ then for any pair of states ${(x,y) \in \Xz^2}$,
\[
\P\left( T(x,y) < \infty \right) = 1 \enspace ,
\]
and the mean hitting time is finite, i.e.,
\[
\tau(x,y) < \infty \enspace .
\]
\end{prop}

\begin{prop}[Existence of Stationary distribution]
For any irreducible and aperiodic Markov chain there exists at least one stationary distribution.
\begin{proof}
TBD
\end{proof}
\end{prop}

\begin{definition}[Total variation distance]\label{D:TotVarDist}
If $\nu_1:=\left(\nu_1(x)\right)_{x\in \Xz}$ and $\nu_2 := \left(\nu_2(x)\right)_{x\in \Xz}$ are elements of $\mathcal{P}(\Xz)$, the set of all probability distributions on $\Xz:=\{s_1,s_2,\ldots,s_k\}$, then we define the {\bf total variation distance} between $\nu_1$ and $\nu_2$ as
\begin{equation}\label{E:TotVarDist}
\dtv \left( \nu_1, \nu_2 \right) := \frac{1}{2} \sum_{x \Xz} \abs \left( \nu_1(x) - \nu_2(x) \right), \quad \dtv : \mathcal{P}(\Xz) \times \mathcal{P}(\Xz)  \to [0,1] \enspace .
\end{equation}
If $\nu_1, \nu_2, \ldots$ and $\nu$ are probability distributions on $\Xz$, then we say that $\nu_t$ {\bf converges in total variation} to $\nu$ as $n \to \infty$ and write $\nu_t  \overset{\mathsf{TV}}{\longrightarrow} \nu$, if
\[
\lim_{t \to \infty} \dtv \left( \nu_t, \nu \right) = 0 \enspace .
\]
Observe that if $\dtv(\nu_1,\nu_2)=0$ then $\nu_1=\nu_2$. The constant $1/2$ in \hyperref[E:TotVarDist]{Equation~\ref*{E:TotVarDist}} ensures that the range of $\dtv$ is in $[0,1]$.  If $\dtv(\nu_1,\nu_2)=1$ then $\nu_1$ and $\nu_2$ have disjoint  supports, i.e., we can partition $\Xz$ into $\Xz_1$ and $\Xz_2$, i.e., $\Xz=\Xz_1 \cup \Xz_2$ and $\Xz_1 \cap \Xz_2 = \emptyset$, such that $\sum_{x \in \Xz_1} \nu_1(x)=1$ and  $\sum_{x \in \Xz_2} \nu_2(x)=1$.  The total variation distance gets its name from the following natural interpretation:
\[
\dtv \left( \nu_1, \nu_2 \right) = \max_{A \subset \Xz} \abs \left( \nu_1(A) - \nu_2(A) \right) \enspace .
\]
This interpretation means that the total variation distance between $\nu_1$ and $\nu_2$ is the maximal difference in probabilities that the two distributions assign to any one event $A \in \sigma(\Xz) = 2^{\Xz}$.
\end{definition}

In words, \hyperref[P:MCConvergence]{Proposition~\ref*{P:MCConvergence}} says that if you run the chain for a sufficiently long enough time $t$, then, regardless of the initial distribution $\mu_0$, the distribution at time $t$ will be close to the stationary distribution $\pi$.  This is referred to as the Markov chain {\bf approaching equilibrium} or {\bf stationarity} as $t \to \infty$.

\begin{prop}[Markov chain convergence theorem]\label{P:MCConvergence}
Let $\left( X_t \right)_{t \in \Zz_+}$ be an irreducible aperiodic Markov chain with state space $\Xz=\{s_1,s_2,\ldots,s_k\}$, transition matrix $P = \left( P(x,y) \right)_{(x,y) \in \Xz^2}$ and initial distribution $\mu_0$.  Then for any distribution $\pi$ which is stationary for the transition matrix $P$, we have
\begin{equation}
\mu_t \overset{\mathsf{TV}}{\longrightarrow} \pi  \enspace .
\end{equation}
\begin{proof}
TBD
\end{proof}
\end{prop}

\begin{prop}[Uniqueness of stationary distribution]\label{P:UniqueStationaryDistrn}
Any irreducible aperiodic Markov chain has a unique stationary distribution.
\begin{proof}
TBD
\end{proof}
\end{prop}

\begin{exercise}\label{EXR:SixStatesWith3Blockof2}
Consider the Markov chain on $\{1,2,3,4,5,6\}$ with the following transition matrix:
\[
P =
\bordermatrix{ ~ & 1 & 2 & 3 &  4 & 5 & 6 \cr
1 & \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 & 0 \cr
2 & \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 & 0 \cr
3 & 0 & 0 &  \frac{1}{4} & \frac{3}{4} & 0 & 0  \cr
4 & 0 & 0 &  \frac{3}{4} & \frac{1}{4} & 0 & 0  \cr
5 & 0 & 0 &  0 & 0 & \frac{3}{4} & \frac{1}{4}  \cr
6 & 0 & 0 &  0 & 0 & \frac{1}{4} & \frac{3}{4}  } \enspace .
\]
Show that this chain is reducible and it has three stationary distributions:
\[
(1/2,1/2,0,0,0,0), \quad (0,0,1/2,1/2,0,0), \quad (0,0,0,0,1/2,1/2) \enspace .
\]
\end{exercise}

\begin{exercise}\label{EXR:ConvexCombof2StationaryDistrns}
If there are two stationary distributions $\pi$ and $\pi'$ then show that there is a infinite family of stationary distributions $\{\pi_p : p \in [0,1] \}$, called the convex combinations of $\pi$ and $\pi'$.
\end{exercise}

\begin{exercise}\label{EXR:ConvergengeinTVFailsforPeriodicDrunkardWalk}
Show that for a drunkard's walk chain started at state $0$ around a polygonal block with $k$ corners labelled $\{0,1,2,\ldots,k-1\}$, the state probability vector at time step $t$
\[
\mu_t \overset{\mathsf{TV}}{\longrightarrow} \pi
\]
if and only if $k$ is odd.  Explain what happens to $\mu_t$ when $k$ is even.
\end{exercise}

\section{Reversibility}

We introduce another specific property called reversibility.
This property will assist in conjuring Markov chains with a desired stationary distibution.

\begin{definition}[Reversible]\label{D:Reversible}
A probability distribution $\pi$ on $\Xz = \{s_1,s_2,\ldots,s_k\}$ is said to be a {\bf reversible distribution} for a Markov chain $\left(X_t\right)_{t\in \Zz}$ on $\Xz$ with transition matrix $P$ if for every pair of states $(x,y) \in \Xz^2$:
\begin{equation}\label{E:ReversibilityCondition}
\pi(x) P(x,y) = \pi(y) P(y,x) \enspace .
\end{equation}
A Markov chain that has a reversible distribution is said to be a reversible Markov chain.
\end{definition}

In words, $\pi(x) P(x,y) = \pi(y) P(y,x)$ says that if you start the chain at the reversible distribution $\pi$, i.e., $\mu_0 = \pi$, then the probability of going from $x$ to $y$ is the same as that of going from $y$ to $x$.

\begin{prop}[A reversible $\pi$ is a stationary $\pi$]\label{P:ReversibleIsStationary}
Let $\left(X_t\right)_{t \in \Zz_+}$ be a Markov chain on $\Xz = \{s_1,s_2,\ldots,s_k\}$ with transition matrix $P$.
If $\pi$ is a reversible distribution for $\left(X_t\right)_{t \in \Zz_+}$ then $\pi$ is a stationary distribution for $\left(X_t\right)_{t \in \Zz_+}$.
\begin{proof}
Suppose $\pi$ is a reversible distribution for $\left(X_t\right)_{t \in \Zz_+}$ then $\pi$ is a probability distribution on $\Xz$ and $\pi(x) P(x,y) = \pi(y) P(y,x)$ for each $(x,y)\in \Xz^2$.
We need to show that for any $y \in \Xz$ we have $$\pi(y)=\sum_{x\in\Xz}\pi(y) P(y,x) \enspace .$$
Fix a $y \in \Xz$,
\begin{eqnarray*}
LHS
&=& \pi(y)= \pi(y) \, 1 = \pi(y) \, \sum_{x \in \Xz} P(y,x) \text{, since $P$ is a stochastic matrix} \\\\
&=& \sum_{x \in \Xz} \pi(y) P(y,x) = \sum_{x \in \Xz} \pi(x) P(x,y) \text{,  by reversibility} \\
&=& RHS \enspace .
\end{eqnarray*}
\end{proof}
\end{prop}

%\section{Classical Examples}
%\work
%\subsection{Random Walks on Graphs}

\begin{definition}[Graph]\label{D:Graph}
A {\bf Graph} $\Gz := (\Vz,\Ez)$ consists of a {\bf vertex set} $\Vz := \{v_1,v_2,\ldots,v_k\}$ together with an {\bf edge set} $\Ez := \{e_1,e_2,\ldots,e_l\}$.  Each edge connects two of the vertices in $\Vz$.  An edge $e_h$ connecting vertices $v_i$ and $v_j$ is denoted by $\langle v_i, v_j \rangle$.  Two vertices are {\bf neighbours} if they share an edge.
The {\bf negihbourhood} of a vertex $v_i$ denoted by $\nbhd(v_i):=\left\{v_j : \langle v_i,v_j\rangle \in \Ez \right\}$ is the set of neighbouring vertices of $v_i$.
The number of neighbours of a vertex $v_i$ in an undirected graph is called its {\bf degree} and is denoted by $\deg(v_i)$.
Note that $\deg(v_i) = \# \nbhd(v_i)$.
In a graph we only allow one edge per pair of vertices but in a {\bf multigraph} we allow more than one edge per pair of vertices.
An edge can be {\bf directed} to preserve  the order of the pair of vertices they connect or they can be {\bf undirected}.
An edge can be {\bf weighted} by being associated with a real number called its weight.
We can represent a directed graph by its {\bf adjacency matrix} given by:
\[
A := \left( A(v_i,v_j) \right)_{(v_i,v_j) \in \Vz \times \Vz}, \quad
A(v_i,v_j) =
\begin{cases}
1 & \text{if } \  \langle v_i, v_j \rangle \in \Ez \\
0 & \text{otherwise} \enspace .
\end{cases}
\]
Thus the adjacency matrix of an undirected graph is symmetric.
In a directed graph, each vertex $v_i$ has {\bf in-edges} that come into it and {\bf out-edges} that go out of it.
The number of in-edges and out-edges of $v_i$ is denoted by $\ideg(v_i)$ and $\odeg(v_i)$ respectively.
Note that a transition diagram of a Markov chain is a weighted directed graph and is represented by the transition probability matrix.
\end{definition}

\begin{model}[Random Walk on an Undirected Graph]\label{M:RWGraph}
A random walk on an undirected graph $\Gz=(\Vz,\Ez)$ is a Markov chain with state space $\Vz:= \{v_1,v_2,\ldots,v_k\}$ and the following transition rules: if the chain is at vertex $v_i$ at time $t$ then it moves uniformly at random to one of the neighbours of $v_i$ at time $t+1$.  If $\deg(v_i)$ is the degree of $v_i$ then the transition probabilities of this Markov chain is
\[
P(v_i,v_j) =
\begin{cases}
\frac{1}{\deg(v_i)} & \text{if $\langle v_i, v_j \rangle \in \Ez$}\\
0 & \text{otherwise} ,
\end{cases}
\]
\end{model}

\begin{prop}\label{P:RWUGpi}
The random walk on an undirected graph $\Gz=(\Vz,\Ez)$, with vertex set $\Vz:= \{v_1,v_2,\ldots,v_k\}$ and degree sum $d = \sum_{i=1}^k{\deg(v_i)}$ is a reversible Markov chain with the reversible distribution $\pi$ given by:
\[
\pi = \left( \frac{\deg(v_1)}{d}, \frac{\deg(v_2)}{d}, \ldots, \frac{\deg(v_k)}{d}  \right) \enspace .
\]
\begin{proof}
First note that $\pi$ is a probability distribution provided that $d > 0$.
To show that $\pi$ is reversible we need to verify \hyperref[E:ReversibilityCondition]{Equation~\ref*{E:ReversibilityCondition}} for each $(v_i,v_j) \in \Vz^2$.
Fix a pair of states $(v_i,v_j) \in \Vz^2$, then
\begin{eqnarray*}
\pi(v_i) P(v_i,v_j) =
\begin{cases}
\frac{\deg(v_i)}{d}\frac{1}{\deg(v_i)}=\frac{1}{d}=\frac{\deg(v_j)}{d}\frac{1}{\deg(v_j)}=\pi(v_j) P(v_j,v_i) & \text{ if } \langle v_i, v_j \rangle \in \Ez\\
0 = \pi(v_j) P(v_j,v_i) & \text{ otherwise}.
\end{cases}
\end{eqnarray*}
By \hyperref[P:ReversibleIsStationary]{Proposition~\ref*{P:ReversibleIsStationary}} $\pi$ is also the stationary distribution.
\end{proof}
\end{prop}

\begin{exercise}\label{EXR:DirectlyProveRWUGpi}
Prove \hyperref[P:RWUGpi]{Proposition~\ref*{P:RWUGpi}} by directly showing that $\pi P = \pi$, i.e., for each $v_i \in \Vz$, $\sum_{i=1}^k \pi(v_i) P(v_i, v_j) = \pi(v_j)$.
\end{exercise}

\begin{example}[Random Walk on a regular graph]\label{EX:RWRegGraph}
A graph $\Gz=(\Vz,\Ez)$ is called regular if every vertex in $\Vz=\{v_1,v_2,\ldots,v_k\}$ has the same degree $\delta$, i.e., $\deg(v_i)=\delta$ for every $v_i \in \Vz$.
Consider the random walk on a regular graph with symmetric transition matrix
\[
Q(v_i,v_j) =
\begin{cases}
\frac{1}{\delta} & \text{ if } \langle v_i,v_j \rangle \in \Ez \\
0 & \text{ otherwise}
\end{cases} \enspace .
\]
By \hyperref[P:RWUGpi]{Proposition~\ref*{P:RWUGpi}}, the stationary distribution of the random walk on $\Gz$ is the uniform distribution on $\Vz$ given by
\[
\pi
= \left( \frac{\delta}{\delta \#\Vz }, \ldots , \frac{\delta}{\delta \#\Vz}  \right)
= \left( \frac{1}{\#\Vz}, \ldots , \frac{1}{\#\Vz} \right)
\enspace .
\]
\end{example}

\begin{example}[Triangulated Quadrangle]\label{EX:TriangulatedQuadrangle}
The random walk on the undirected graph
$$\Gz=(\{1,2,3,4\}, \{\langle 1,2 \rangle, \langle 3,1 \rangle, \langle 2,3 \rangle, \langle 2,4 \rangle, \langle 4,3\rangle\})$$ depicted below with adjacency matrix $A$ is a Markov chain on $\{1,2,3,4\}$ with transition matrix $P$:
$$
A =
\bordermatrix{~ & 1 & 2 & 3 & 4 \cr
1 & 0 & 1 & 1 & 0 \cr
2 & 1 & 0 & 1 & 1\cr
3 & 1 & 1 & 0 & 1\cr
4 & 0 & 1 & 1 & 0} ,
\quad
P =
\bordermatrix{~ & 1 & 2 & 3 & 4 \cr
1 & 0 & \frac{1}{2} &  \frac{1}{2} & 0 \cr
2 & \frac{1}{3} & 0 &  \frac{1}{3} &  \frac{1}{3} \cr
3 & \frac{1}{3} &  \frac{1}{3} & 0 &  \frac{1}{3} \cr
4 & 0 &  \frac{1}{2} &  \frac{1}{2} & 0 } ,
\quad
\makebox{\includegraphics[width=2.5in]{figures/TriQuadrangle}}
\enspace .$$
By \hyperref[P:RWUGpi]{Proposition~\ref*{P:RWUGpi}}, the stationary distribution of the random walk on $\Gz$ is
\[
\pi = \left( \frac{\deg(v_1)}{d}, \frac{\deg(v_2)}{d}, \frac{\deg(v_3)}{d}, \frac{\deg(v_4)}{d} \right)
= \left( \frac{2}{10}, \frac{3}{10}, \frac{3}{10}, \frac{2}{10} \right) \enspace .
\]
\end{example}

%\begin{exercise}\label{EXR:DrunkardAroundBlockFairReversible}
%Show that the Drunkard's walk around the block from \hyperref[SIM:DrunkardsWalkBlock]{Simulation~\ref*{SIM:DrunkardsWalkBlock}} is a random walk on the undirected graph $\Gz=(\Vz,\Ez)$ with $\Vz=\{0,1,2,3\}$ and $\Ez=\{\langle 0,1 \rangle,\langle 1,2 \rangle,\langle 2,3 \rangle,\langle 0,3 \rangle \}$.  What is its reversible distribution?
%\end{exercise}

\begin{example}[Drunkard's biased walk around the block]\label{SIM:DrunkardsBiasedWalkBlock}
Consider the Markov chain $\left(X_t\right)_{t \in \Zz_+}$ on $\Xz=\{0,1,2,3\}$ with initial distribution $\BB{1}_{\{3\}}(x)$ and transition matrix
$$P =
\bordermatrix{~ & 0 & 1 & 2 & 3 \cr
0 & 0 & 1/3 & 0 & 2/3\cr
1 & 1/3 & 0 & 2/3 & 0\cr
2 & 0 & 1/3 & 0 & 2/3\cr
3 & 1/3 & 0 & 2/3 & 0 } \enspace .
$$
Draw the transition diagram for this Markov chain that corresponds to a drunkard who flips a biased coin to make his next move at each corner.  The stationary distribution is $\pi = (1/4,1/4,1/4,1/4)$ (verify $\pi P= \pi$).

We will show that $\left(X_t\right)_{t \in \Zz_+}$ is not a reversible Markov chain.
Sine $\left(X_t\right)_{t \in \Zz_+}$ is irreducible (aperiodicity is not necessary for uniqueness of $\pi$) $\pi$ is the unique stationary distribution.
Due to \hyperref[P:ReversibleIsStationary]{Proposition~\ref*{P:ReversibleIsStationary}}, $\pi$ has to be a reversible distribution in order for $\left(X_t\right)_{t \in \Zz_+}$ to be a reversible Markov chain.
But reversibility fails for $\pi$ since,
\[
\pi(0) P(0,1) = \frac{1}{4} \times \frac{1}{3} = \frac{1}{12} < \frac{1}{6} = \frac{1}{4} \times \frac{2}{3} = \pi(1)P(1,0) \enspace .
\]
\end{example}

\begin{exercise}\label{EXR:PiKingRWChessTorus}
Find the stationary distribution of the Markov chain in \hyperref[EXR:KingRWChessTorus]{Exercise~\ref*{EXR:KingRWChessTorus}}.
\end{exercise}

\begin{model}[Random Walk on a Directed Graph]\label{M:RWDGraph}
A random walk on a directed graph $\Gz=(\Vz,\Ez)$ is a Markov chain with state space $\Vz:= \{v_1,v_2,\ldots,v_k\}$ and transition matrix given by:
\[
P(v_i,v_j) =
\begin{cases}
\frac{1}{\odeg(v_i)} & \text{if $\langle v_i, v_j \rangle \in \Ez$}\\
0 & \text{otherwise} ,
\end{cases}
\]
\end{model}

\begin{example}[Directed Triangulated Quadrangle]\label{EX:DirectedTriangulatedQuadrangle}
The random walk on the directed graph
$$\Gz=(\{1,2,3,4\}, \{\langle 1,2 \rangle, \langle 3,1 \rangle, \langle 2,3 \rangle, \langle 2,4 \rangle, \langle 4,3\rangle\})$$ depicted below with adjacency matrix $A$ is a Markov chain on $\{1,2,3,4\}$ with transition matrix $P$:
$$
A =
\bordermatrix{~ & 1 & 2 & 3 & 4 \cr
1 & 0 & 1 & 0 & 0 \cr
2 & 0 & 0 & 1 & 1 \cr
3 & 1 & 0 & 0 & 0 \cr
4 & 0 & 0 & 1 & 0},
\quad
P =
\bordermatrix{~ & 1 & 2 & 3 & 4 \cr
1 & 0 & 1 & 0 & 0 \cr
2 & 0 & 0 & \frac{1}{2} & \frac{1}{2} \cr
3 & 1 & 0 & 0 & 0 \cr
4 & 0 & 0 & 1 & 0},
\quad
\makebox{\includegraphics[width=2.5in]{figures/DirTriQuadrangle}}
\enspace .$$
\end{example}

\begin{exercise}\label{EXR:DirectedTriangulatedQuadrangleNoReversiblePi}
Show that the there is no reversible distibution for the Markov chain in \hyperref[EX:DirectedTriangulatedQuadrangle]{Example~\ref*{EX:DirectedTriangulatedQuadrangle}}.
\end{exercise}

\begin{example}[Random surf on the word wide web]\label{EX:RandomSurferwww}
Consider the huge graph with vertices as webpages and hyper-links as undirected edges.
Then \hyperref[M:RWGraph]{Model~\ref*{M:RWGraph}} gives a random walk on this graph.
However if a page has no links to other pages, it becomes a sink and therefore terminates the random walk.
Let us modify this random walk into a {\bf random surf} to avoid getting stuck.
If the random surfer arrives at a sink page, she picks another page at random and continues surfing at random again.
Google's PageRank formula uses a random surfer model who gets bored after several clicks and switches to a random page.
The PageRank value of a page reflects the chance that the random surfer will land on that page by clicking on a link.
The stationary distribution of the random surfer on the world wide web is a very successful model for ranking pages.
\end{example}


\chapterimage{head1} % Chapter heading image

\chapter{Voltage on Neuron Morphology}

\section{Introduction}

The standard algorithm used to compute the Voltage on neurons' morphology
 is the Hines algorithm [8]. This algorithm is based on the Thomas
 algorithm [4], which solves tridiagonal systems. Although the use of
  GPUs to compute the Thomas algorithm has been deeply studied
  [5], the differences among these two algorithms,
   Hines and Thomas, makes us impossible to use the last one as this
   can not deal with the sparsity of the Hines matrix.

Previous works [13] have explored the use of other algorithms based on
 the Stone's method [10]. Unlike Thomas algorithm, this method is
  parallel. However, it is in need of a higher number of operations
  ($20 n \log n$) with respect to the ($8n$) operations of the Thomas
   algorithm to solve one single system of size $n$. Also, the use of
   parallel methods present some additional drawbacks to be dealt with.
    For instance, it would be difficult to compute those neurons that
     compromise a size bigger than the maximum number of threads per CUDA block (1024) or shared memory (48KB). Other problems are the computationally expensive operations such as atomic accesses and synchronizations necessary to compute this method. Each neuron presents a particular morphology and so a different scheduling (preprocessing) must be applied to each of them which makes even more difficult its implementation.

Unlike the work presented in [13], where a relatively low number of
neurons (128) is computed using single precision operations, in \cite{cuHines}
 the authors are able to execute a very high number of neurons
 (up to hundreds of thousands) using double precision operations.
 They have used the Hines algorithm, which is the optimum method in
 terms of number of operations, avoiding high expensive computational
  operations, such as synchronizations and atomic accesses. The code
   is able to compute a high number of systems (neurons) of any size in
    one call (CUDA kernel), using one thread per Hines system instead of
     one CUDA block per system. Although multiple works have explore the
      use of GPUs to compute multiple independent problems in parallel
       without transforming the data layout [10], the
       particular characteristics of the sparsity of the Hines matrices
        forces us to modify the data layout to efficiently exploit the
        memory hierarchy of the GPUs (coalescing accesses to GPU memory).
         These modifications have not been explored previously, which are
          deeply described and analyzed in the present work.

\section{GPU and cuSPARSE}

Although GPUs are traditionally associated to interactive applications involving high
rasterization performance, they are also widely used to accelerate much more general
applications (now called General Purpose Computing on GPU (GPGPU)) which
require an intense computational load and present parallel characteristics. The main
feature of these devices is a large number of processing elements integrated into a single chip, which reduces significantly the cache memory. These processing elements
can access to a local high-speed external DRAM memory, connected to the computer
through a high-speed I/O interface (PCI-Express). Overall, these devices can offer a
higher main memory bandwidth and can use data parallelism to achieve a higher floating point throughput than CPUs

The \cite{cuSPARSE} library contains a set of basic linear algebra subroutines used for
handling sparse matrices. It is implemented on top of the NVIDIA CUDA runtime (which is part of the CUDA Toolkit) and is designed to be called from C and C++.
The library routines can be classified into four categories:

\begin{itemize}
    \item Level 1 : operations between a vector in sparse format and a vector in dense
    format
    \item Level 2 : operations between a matrix in sparse format and a vector in dense
    format
    \item Level 3 : operations between a matrix in sparse format and a set of vectors in
    dense format (which can also usually be viewed as a dense tall matrix)
    \item Conversions: operations that allow conversion between different matrix formats,
    and compression of csr matrices.
\end{itemize}

The cuSPARSE library allows developers to access the computational resources of the
NVIDIA graphics processing unit (GPU), although it does not auto-parallelize across
multiple GPUs. As we will see in subsequent sections this library contains the reference
solver for tridiagonal linear systems in GPU, gtsvStridedBatch which is going to be a
great reference in order to evaluate our implementations.

\vspace{10ex}
\section{Tridiagonal Linear Systems}
The state-of-the-art method to solve tridiagonal systems is the called Thomas algorithm [28]. Thomas algorithm is a specialized application of the Gaussian elimination
that takes into account the tridiagonal structure of the system. Thomas algorithm
consists of two stages, commonly denoted as forward elimination and backward substitution.

Given a linear $Au = y$ system, where $A$ is a tridiagonal matrix:

\begin{equation}
    A=\left[\begin{array}{cccccc}
        b_{1} & c_{1} & & & & 0 \\
        a_{2} & b_{2} & c_{2} & & & \\
        & & \cdot & \cdot & & \\
        & & \cdot & \cdot & \cdot & \\
        & & & a_{n-1} & b_{n-1} & c_{n-1} \\
        & & & & a_{n} & b_{n}
        \end{array}\right]
\end{equation}

The forward stage eliminates the lower diagonal as follows:

\begin{equation}
    \begin{aligned}
        c_{1}^{\prime}=\frac{c_{1}}{b_{1}}, \quad c_{i}^{\prime}=\frac{c_{i}}{b_{i}-c_{i-1}^{\prime} a_{i}} & \text { for } i=2,3, \ldots, n-1 \\
        y_{1}^{\prime}=\frac{y_{1}}{b_{1}}, \quad y_{i}^{\prime}=\frac{y_{i}-y_{i-1}^{\prime} a_{i}}{b_{i}-c_{i-1}^{\prime} a_{i}} & \text { for } i=2,3, \ldots, n-1
    \end{aligned}
\end{equation}

and then the backward stage recursively solve each row in reverse order:

\begin{equation}
    u_{n}=y_{n}^{\prime}, u_{i}=y_{i}^{\prime}-c_{i}^{\prime} u_{i+1} \text { for } i=n-1, n-2, \ldots, 1
\end{equation}

Overall, the complexity of Thomas algorithm is optimal: $8n$ operations
in $2n - 1$ steps.

Cyclic Reduction (CR) is a parallel alternative to Thomas algorithm.
It also consists of two phases (reduction and substitution). In each intermediate step
of the reduction phase, all even-indexed (i) equations
$a_{i} x_{i-1}+b_{i} x_{i}+c_{i} x_{i+1}=d_{i}$ are
reduced. The values of $a_i$, $b_i$, $c_i$ and $d_i$ are updated in
each step according to:

\begin{equation}
    \begin{array}{c}
        a_{i}^{\prime}=-a_{i-1} k_{1}, b_{i}^{\prime}=b_{i}-c_{i-1} k_{1}-a_{i+1} k_{2} c_{i}^{\prime}=-c_{i+1} k_{2}, y_{i}^{\prime}=y_{i}-y_{i-1} k_{1}-y_{i+1} k_{2} \\
        k_{1}=\frac{a_{i}}{b_{i-1}}, k_{2}=\frac{c_{i}}{b_{i+1}}
    \end{array}
\end{equation}

After $log_2 n$ steps, the system is reduced to a single equation that is
 solved directly. All odd-indexed unknowns $x_i$ are then solved in the
 substitution phase by introducing the already computed $u_{i−1}$ and
 $u_{i+1}$ values:

\begin{equation}
    u_{i}=\frac{y_{i}^{\prime}-a_{i}^{\prime} x_{i-1}-c_{i}^{\prime} x_{i+1}}{b_{i}^{\prime}}
\end{equation}

Overall, the CR algorithm needs $17n$ operations and $2 \log_2 n - 1$
steps. Figure \ref{fig:22} graphically illustrates its access pattern.

\vspace{5ex}
\begin{figure}[htbp]
    \centering
    \includegraphics[width = 0.35\textwidth]{fig-22}
    \label{fig:22}
    \caption{Access pattern of the CR algorithm}
\end{figure}

Parallel Cyclic Reduction (PCR) is a variant of CR, which only has
substitution phase. For convenience, we consider cases where $n = 2^s$
, that involve $s = \log_2 n$ steps. Similarly to CR $a$, $b$, $c$ and $y$
are updated as follows, for $j = 1, 2, \cdots, s$ and $k = 2^{j-1}$:

\begin{equation}
    \begin{array}{c}
        a_{i}^{\prime}=\alpha_{i} a_{i}, b_{i}^{\prime}=b_{i}+\alpha_{i} c_{i-k}+\beta_{i} a_{i+k} \\
        c_{i}^{\prime}=\beta_{i} c_{i+1}, y_{i}^{\prime}=b_{i}+\alpha_{i} y_{i-k}+\beta_{i} y_{i+k} \\
        \alpha_{i}=\frac{-a_{i}}{b_{i-1}}, \beta_{i}=\frac{-c_{i}}{b_{i}}
    \end{array}
\end{equation}

finally the solution is achieved as:

\begin{equation}
    u_{i}=\frac{y_{i}^{\prime}}{b_{i}}
\end{equation}

Essentially, at each reduction stage, the current system is transformed into two smaller
systems and after $\log_2 n$ steps the original system is reduced to $n$
independent equations. Overall, the operation count of PCR is $12n log_2 n$.
Figure \ref{fig:23} sketches the corresponding access pattern.

\vspace{5ex}
\begin{figure}[htbp]
    \centering
    \includegraphics[width = 0.35\textwidth]{fig-23}
    \label{fig:23}
    \caption{Access pattern of the PCR algorithm}
\end{figure}

We should highlight that apart from their computational complexity these algorithms differ in their data access and synchronization patterns, which also have a strong
influence on their actual performance. For instance, in the CR algorithm synchronizations are introduced at the end of each step and its corresponding memory access
pattern may cause bank conflicts. PCR needs less steps and its memory access pattern
is more regular.

In fact, hybrid combinations that try to exploit the best of each algorithm have
been explored. Figure \ref{fig:24} illustrates the access pattern of the
CR-PCR combination proposed in [13]. CR-PCR reduces the system to a certain size
using the forward reduction phase of CR and then solves the reduced (intermediate)
system with the PCR algorithm. Finally, it substitutes the solved unknowns back into
the original system using the backward substitution phase of CR. Indeed, this is the
method implemented by the gtsvStridedBatch routine into the cuSPARSE package.

\vspace{5ex}
\begin{figure}[htbp]
    \centering
    \includegraphics[width = 0.38\textwidth]{fig-24}
    \label{fig:24}
    \caption{Access pattern of the CR-PCR algorithm}
\end{figure}

\vspace{10ex}
\section{Hines Algorithm}
In this section, we describe the numerical framework behind the computation of the
Voltage on neurons morphology. It follows the next general form:

\begin{equation}
    C \frac{\partial V}{\partial t}+I=f \frac{\partial}{\partial x}\left(g \frac{\partial V}{\partial x}\right)
\end{equation}

where $f$ and $g$ are functions on $x$-dimension and the current $I$ and
capacitance $C$ depend on the voltage $V$. Discretizing the previous
equation on a given morphology we obtain a system that has to be solved
every time-step. This system must be solved at each point:

\begin{equation}
    a_{i} V_{i+1}^{n+1}+d_{i} V_{i}^{n+1}+b_{i} V_{i-1}^{n+1}=r_{i}
\end{equation}

where the coefficients of the matrix are defined as follow:

upper diagonal:

\begin{equation}
    a_{i}=-\frac{f_{i} g_{i+\frac{1}{2}}}{2 \Delta_{x}^{2}}
\end{equation}

lower diagonal:

\begin{equation}
    b_{i}=-\frac{f_{i} g_{i+\frac{1}{2}}}{2 \Delta_{x}^{2}}
\end{equation}

diagonal:

\begin{equation}
    d_{i}=\frac{C_{i}}{\Delta_{t}}-\left(a_{i}+b_{i}\right)
\end{equation}

rhs:

\begin{equation}
    r_{i}=\frac{C_{i}}{\Delta_{t}} V_{i}^{n}-I-a_{i}\left(V_{i-1}^{n}-V_{i}^{n}\right)-b_{i}\left(V_{i+1}^{n}-V_{i}^{n}\right)
\end{equation}

The $a_i$ and $b_i$ are constant in the time, and they are computed once at start up.
Otherwise, the diagonal (d) and right-side-hand (rhs) coefficients are updated every
time-step when solving the system.

The discretization above explained is extended to include branching, where the
spatial domain (neuron morphology) is composed of a series of one-dimension sections
that are joined at branch points according to the neuron morphology.

For sake of clarity, we illustrate a simple example of a neuron morphology in Figure \ref{fig:25}. It is important to note that the graph formed by the neuron morphology is an
acyclic graph, i.e. it has no loops. The nodes are numbered using a scheme that gives
the matrix sparsity structure that allows to solve the system in linear time.

\vspace{5ex}
\begin{figure}[htbp]
    \centering
    \includegraphics[width = 0.6\textwidth]{fig-25}
    \label{fig:25}
    \caption{Example of a neuron morphology and its numbering (left-top and bottom)
    and sparsity pattern corresponding to the numbering followed (top-right)}
\end{figure}

The Hines matrices feature the following properties: they are symmetric, the diagonal
coefficients are all nonzero and per each off-diagonal element, there is one off-diagonal
element in the corresponding row and column (see row/column 7, 12, 17 and 22 in
Figure \ref{fig:25}).

Given the aforementioned properties, the Hines systems ($Ax = b$) can be efficiently
solved by using an algorithm similar to Thomas algorithm for solving tri-diagonal systems. This algorithm, called Hines algorithm, is almost identical to the Thomas algorithm except by the sparsity pattern given by the morphology of the neurons whose
pattern is stored by the $p$ vector. An example of the sequential code used to implement
the Hines algorithm is illustrated in pseudo-code in Algorithm 1.

\vspace{5ex}
\begin{figure}[htbp]
    \centering
    \includegraphics[width = 1.0\textwidth]{hines}
    \label{fig:hines}
\end{figure}


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{MAT257 Notes}
\author{Jad Elkhaleq Ghalayini}
\date{February 8 2019}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{mathtools}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{cancel}

\usepackage[margin=1in]{geometry}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem*{corollary}{Corollary}
\newtheorem{exercise}{Exercise}
\newtheorem{claim}{Claim}

\DeclareMathOperator{\Int}{Int}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Ker}{Ker}
\DeclareMathOperator{\Ima}{Im}
\DeclareMathOperator{\Vol}{vol}

\newcommand{\reals}[0]{\mathbb{R}}
\newcommand{\nats}[0]{\mathbb{N}}
\newcommand{\ints}[0]{\mathbb{Z}}
\newcommand{\rationals}[0]{\mathbb{Q}}
\newcommand{\brac}[1]{\left(#1\right)}
\newcommand{\sbrac}[1]{\left[#1\right]}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\eval}[3]{\left.#3\right|_{#1}^{#2}}
\newcommand{\ip}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\prt}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\mb}[1]{\mathbf{#1}}
\newcommand{\hlfspc}[0]{\mathbb{H}}
\newcommand{\loint}[0]{\operatorname{L}\int}
\newcommand{\hiint}[0]{\operatorname{U}\int}
\newcommand{\indic}[1]{\chi_{#1}}

\begin{document}

\maketitle

Recall that if \(\dim V = n\), then \(\dim \Omega^n(V) = 1\). Furthermore, every \(\omega \in \Omega^n(\reals^n)\) is a multiple of \(\det\).

More generally, we have the following lemma:
\begin{lemma}
  If \(\omega \in \Omega^n(V)\), and \(v_1,...,v_n \in V\), let
  \begin{equation}
    w_i = \sum_{j = 1}^na_{ij}v_j
  \end{equation}
  Then, if \(A = (a_{ij}) \in \reals^{n \times n}\),
  \begin{equation}
    \omega(w_1,...,w_n) = \det(A)\omega(v_1,...,v_n)
    \label{thisthing}
  \end{equation}
\end{lemma}
\begin{proof}
  Let's express things in terms of an \(n\)-form on \(\reals^n\). Define \(\eta \in \Omega^n(\reals^n)\) by equation \ref{thisthing} as on the columns of \(A\) (being a function taking in \(n\) vectors) as
  \begin{equation}
    \eta((a_{11},...,a_{1n}),(a_{21},...,a_{2n}),...,(a_{n_1},...,a_{nn})) = \omega\left(\sum a_{1j}v_i, ... \sum a_{nj}v_j\right)
    \label{star}
  \end{equation}
  We have that, for some \(\lambda \in \reals\)
  \begin{equation}
    \eta = \lambda\det \implies \eta(e_1,...,e_n) = \lambda \det(e_1,...,e_n) = \lambda = \omega(v_1,...,v_n)
  \end{equation}
  But this tells us that
  \begin{equation}
    \omega(w_1,...,w_n) = w(v_1,...,v_n)\det(a_j)
    \label{eq2}
  \end{equation}
  since the left hand side of equation \ref{eq2} is equal to equation \ref{star}. One thing that's good to point out is the following: suppose \(v_1,...,v_n\) were linearly dependent. This would tell us that \(\omega(v_1,...,v_n)\) and hence \(\omega(w_1,...,w_n)\) would be zero.
\end{proof}
In general, this shows us that if \(\dim V = n\), any nonzero \(\omega \in \Omega^n(V)\) divides the set of all bases of \(V\) into two groups:
\begin{itemize}
  \item Those where \(\omega(v_1,...,v_n) > 0\)
  \item Those where \(\omega(v_1,...,v_n) < 0\)
\end{itemize}
Another way of saying the same thing without using differential forms is saying that \((v_1,...,v_n)\) and \((\omega_1,...,\omega_n)\) are in the same group if one can be transformed into the other with a matrix \(A\) having positive determinant.

\section{Orientations of \(V\)}

\begin{definition}
Define an \underline{orientation of \(V\)} to be a choice of one of these two groups.
We can consider this as providing an equivalence relation: two bases are equivalent if they belong to the same group. We will write these as
\begin{itemize}
  \item \([v_1,...,v_n]\) to denote the equivalence class of \(v_1,...,v_n\)
  \item \(-[v_1,...,v_n]\) to denote the equivalence class of the opposite group
\end{itemize}
\end{definition}
\begin{definition}
  The \underline{standard orientation} of \(\reals^n\) is \([e_1,...,e_n]\)
\end{definition}
For example, in \(\reals^3\), the standard orientation is given by \([e_1, e_2, e_3]\), giving the physics right hand rule: if you put your index and middle fingers in the direction of \(e_1, e_2\), your thumb points in the direction of \(e_3\).

Suppose \(V\) has an inner product \(T\). In this case, we can consider orthonormal bases with respect to \(T\). So consider two different orthonormal bases \(v_1,...,v_n\) and \(w_1,...,w_n\) with respect to \(T\). That is,
\begin{equation}
  T(v_i, v_j) = T(w_i, w_j) = \delta_{ij}
\end{equation}
So we can write one in terms of the other, i.e. find \(A = (a_{ij})\) such that
\begin{equation}
  w_i = \sum_{a_{ij}}v_j
\end{equation}
where \(\det A = \pm 1\). How do we see that? Well, let's compute:
\begin{equation}
  T(w_i, w_j) = T\left(\sum_ka_{ik}v_k, \sum_\ell a_{j\ell}v_\ell\right) = \sum_{k, \ell}a_{ik}a_{j\ell}T(v_k, v_\ell) = \sum_{k, \ell}a_{ik}a_{j\ell}\delta_{k\ell} = \sum_ka_{ik}a_{jk} = \delta_{ij}
\end{equation}
So everything is zero here unless \(i = j, k = \ell\), in which this becomes
\begin{equation}
  (AA^T)_{ij} = \delta_{ij} \implies AA^T = I \implies \det A^2 = 1 \implies \det A = \pm 1
\end{equation}
as desired. We can now obtain the following result
\begin{lemma}
  Given inner product \(T\) and orientation \(\mu\) for \(V\), there is a unique \(\omega \in \Omega^n(V)\) such that \(\omega(v_1,...,v_n) = 1\) whenever \(v_1,...,v_n\) are an orthonormal basis and \([v_1,...,v_n] = \mu\).
\end{lemma}
The point is, if have an orthonromal basis \(v_1,...,v_n\) where \(\omega(v_1,...,v_n) = 1\) and take \textit{another} orthonormal basis \(w_1,...,w_n\), then
\begin{equation}
  \omega(w_1,...,w_n) = \pm\omega(v_1,...,v_n) = \pm 1
\end{equation}
which, of course, will be \(1\) if \((w_1,...,w_n) \in \mu\).
\textit{This} is what we're going to call volume. More precisely, we call \(\omega\) the \underline{volume form} on \(V\). Of course, this is determined by an inner product and an orientation.

For example, the volume element (form) of \(\reals^n\) determined by the standard inner product and the standard orientation is \(\det\).
We're usually going to use the word ``form'' to mean an alternating tensor at every point, but it doesn't matter, as alternating tensor and form can be used interchangeably.

One thing to mention before we leave this stuff is
\subsection{The Cross Product}
We have that
\begin{equation}
  \det\begin{pmatrix}
    v_{11} & v_{12} & v_{13} \\
    v_{21} & v_{22} & v_{23} \\
    w_{1} & w_{2} & w_{3}
  \end{pmatrix} = \langle v_1 \times v_2, w \rangle
\end{equation}
where \(v_1 \times v_2\) denotes the cross product. It turns out you can do the same thing with \(n - 1\) vectors in \(\reals^n\). Define \(\varphi \in \Omega^1(V) = \mc{T}^1(V) = V^*\), i.e. let \(\phi\) be a linear function from \(\reals^n \to \reals\), where
\begin{equation}
  \varphi(w) = \det\begin{pmatrix} v_1 \\ ... \\ v_{n - 1} \\ w \end{pmatrix}
\end{equation}
Then there is a unique \(z \in V\) such that \(\varphi(w) = \langle z, w \rangle\). This is something you proved last year in linear algebra... I hope...
So we'll call \(z\) the \underline{cross product} of \(v_1,...,v_{n - 1}\), \(v_1 \times v_2 \times ... \times v_{n - 1}\).
Note that the cross product is an alternating multilinear form. It also satisfies the distributive property.

Wh do we want to use this? Well, we want to start proving the following:
\begin{equation}
  \int_{\text{cube}}f(x_1,...,x_n)dx_1 \wedge ... \wedge dx_n = \int_{\text{cube}}f dx_1...dx_n
\end{equation}
And we'll get to that next time...

\end{document}