\chapter{Statistics - Bayesian}

Spectroscopy is a great starting point for Bayesian statistics to those who're interested in astronomy. This is because it has plenty of data points along the $ x $-axis (e.g., wavelength) with error-bars, and we usually fit a simple analytic function to the dataset.

In this chapter, I will show an example from a previously published data (\href{https://ui.adsabs.harvard.edu/abs/2018ApJ...866..112G/abstract}{Greco+2018ApJ}). Another good example of this kind, with a comprehensive discussions on the prior selection, is given in Gregory P. (2005) ``Bayesian Logical Data Analysis for the Physical Sciences'' section 3.6 to 3.8.

Consider you have the 1-D spectrum as in \cref{fig:greco2018f2-rep}. The big question is this: \textbf{How likely is that the peak is an actual line(s), not due to random noise?}
\begin{figure}[ht!]
  \centering
  \includegraphics[width=0.6\linewidth]{figs/Greco2018F2-rep}
  \caption{The data extracted from \href{https://ui.adsabs.harvard.edu/abs/2018ApJ...866..112G/abstract}{Greco+2018ApJ}, without Gaussian fit. Note that $ 1 \,\mathrm{erg/s/cm^2/\AA} = 1 \, \mathrm{mW/m^2/\AA} $. See the notebook \texttt{Spectroscopy\_Simulation} for the codes I used in this chapter.}
  \label{fig:greco2018f2-rep}
\end{figure}

To test this, let's set up hypotheses: The null hypothesis ($H_0$) is that there is no line, while the alternative hypothesis ($H_1$) is that there \textit{is} a line\footnote{Reminder: \textit{the null hypothesis is what we want to reject}.}. Note that we haven't specified the properties of the line (amplitude, width, etc) yet. Then our strategy must be
\begin{enumerate}
\item We have to check which hypothesis is more likely (assuming there're only two possibilities, $ H_0 $ and $ H_1 $).
\item If $H_0$ is more likely, we accept the null hypothesis, i.e., ``we argue the non-existence of line in this data with certainty of xxx''.
\item If $H_1$ is more likely, we have to find the line properties (amplitude, width, etc) in the form of $ x \pm dx $.
\end{enumerate}
The first is called the \textbf{Model Selection} and the third is called the \textbf{Parameter Estimation}. If the second is the case, things are simple: no line! That's all. Model selection is \textit{discrete} (finite number of different hypotheses), while parameter estimation is usually \textit{continuous}.

\section{Bayes Theorem}
\begin{thm}[Bayes' Theorem]
If $H$, $D$, and $I$ are the hypothesis, data, and prior information, respectively, Bayes' theorem states
\begin{equation}\label{eq: bayes thm}
  P(H | D, I) = \frac{P(H|I) \times P(D|H, I)}{P(D|I)} ~.
\end{equation}
Usually it is simply put as
\begin{equation}\label{eq: bayes thm propto}
  P(H | D, I) \propto P(H|I) \times P(D|H, I) ~.
\end{equation}
\end{thm}
The \textbf{posterior} $P(H|D, I)$ is understood as ``the probability that the hypothesis is true, given the data equals to $D$ and we have prior information $I$.'' The \textbf{prior} $P(H|I)$ can be quite subjective, since it means ``the probability that the hypothesis is true.'' In most cases, we take either uniform prior or Jeffrey's prior (see below). The denominator, $P(D | I)  := \int P(D | H, I) P(H|I) dH$ is understood as the normalization constant for the posterior.

The term $P(D|H, I)$ is called the \textbf{likelihood}, and this is what we have calculated during our high school course works:

\begin{ex}[High school exam problem: calculating liklihood]
Given the hypothesis that a person has possibility 0.5 to win a game, what is the probability for her to win 3 games in a row?

In this case, the hypothesis is $H = (p=0.5)$, and the data $D = (\mathrm{win~3~in~a~row})$. The prior knowledge is that ``the game rule does not change'', ``the person doesn't use faul measures to make the possibility to change over time'', etc, which are just trivial assumptions. Most importantly, we have to assume that the results of each game should be \textit{independent}, so that we can use the multiplication law. Therefore,
\begin{equation*}
  P(D|H, I) = (1/2)^3 = 1/8 ~.
\end{equation*}

In real scientific problems, $I$ can be something like ``the fundamental physical laws do not change over time'', ``this galaxy is spiral for sure (if you're working on rotational curves of S galaxies, you may assume your samples are definitely S, not E or Irr)'', etc.
\end{ex}

When you are finding the mean of the data, it is a single parameter case, and therefore, $ H $ just means the mean value, so you can write something like this: $ P(H|D, I) = P(m|D, I) $. When you have multiple parameters, usually people denote the set of paramers as $ \vec{\theta} $, so $ P(H|D, I) = P(\vec{\theta}|D, I) $. If we're fitting the data with, say, a 3rd order polynomial with 4 parameters, we can understand that $ I $ includes ``the data is described by a 3rd order polynomial with 4 parameters''.

\section{Towards the Model Selection}
\subsection{Model Selection Concept}
The term ``odd'' is used for $ \frac{\mathrm{probability}}{1 - \mathrm{probability}} $. In this case the $ P(H_1|I) / P(H_0|I) $ is called the prior odds because $ P(H_1|I) + P(H_0|I) = 1$, as there is no other possibility other than $ H_0 $ and $ H_1 $. Similarly, since $ P(D|H_0, I) + P(D|H_1, I) = 1$, the LHS is also called the posterior odds.

Now come back to the original question: model selection. The model selection is done based on the \textbf{odds ratio}:
\begin{equation}\label{eq: odds ratio}
  R := \frac{P(H_1 | D, I)}{P(H_0 | D, I)}
    = \frac{P(H_1|I)}{P(H_0|I)} \times \frac{P(D|H_1, I)}{P(D|H_0, I)}
    = \mathrm{odds}(\mathrm{prior}) \times B_{10}~.
\end{equation}
Here $ B_{10} $ is called the \textbf{Bayes' factor}.

If $ R > 1 $, we select $H_1$ over $H_0$ and vice versa. If $R=1$, we can't give any conclusion. Equivalently, we can use $\ln R $ and compare it with 0. This is because we frequently get extremely large or small $R$ values (e.g., $10^{-40}$), which is inappropriate for some computer programming.

\subsection{Prior Selection}
There are two major priors: uniform and Jeffreys' prior.
\begin{itemize}
\item \textbf{Uniform prior} assumes uniform probability for the model within reasonable range.
\item \textbf{Jeffreys' prior} assumes the probability proportional to the determinant of the Fisher information matrix. Simply put, for the parameters like standard deviation, if the value is small (has more information), we give more weight to it.
\end{itemize}
Another possibility is that iteratively updated prior based on the accumulated data, as sometimes used in artificial intelligence.

\begin{ex} [Uniform prior]
Consider an emission line fitting to the 1-D spectrum. If you are sure that the line must be Gaussian with center $ \lambda_c \in [650, 655]\,\mathrm{nm} $, the uniform prior of the line center will be $ \lambda_c \sim \mathcal{U}(650, 655) $, which has the pdf of $ p(\lambda_c) = 1/5 $ for $ \lambda_c \in [650, 655]\,\mathrm{nm} $ and 0 otherwise. That is, uniform prior regards it is equally likely to have any value within the bound specified by the user.
\end{ex}

%\begin{ex} [Jeffereys' prior]
%Jeffereys' prior is suitable for parameters like scatter of the data. For instance, for a set of $ N $ samples which show a Gaussian distribution of the single-valued population. An example is like this: You make multiple measurements of the standard star's magnitude, which must be a constant over the time, to understand how accurately your instrument can measure the stellar magnitude, i.e., the stability of the instrument.
%\end{ex}
%That is, if the parameter is small, we give more weight to it; this is similar to the information theory which gives higher weight if a data is "unlikely". For this reason, we usually use Jeffrey's prior for standard deviation estimation. Quantitatively, it is $ P(A|I) = \frac{1}{A \log_e(75/25)}$. This is understood as the uniform prior of the log of $A$: $P(\log_e A| I) d(\log_e A) = \frac{d(\log_e A)}{\log_e A_{max} - \log_e A_{min}} $ $\rightarrow P(\log_e A| I) = \frac{1}{A \log_e (A_{max}/A_{min})}$ .

%Since Jeffrey's prior is non-sensical when we include model of $A=0$, a **modified Jeffrey's prior** is also used: $ P(A|I) = \frac{1}{A+A_0} \times \frac{1}{\log_e \frac{A_0+A_{max}}{A_0+A_{min}}} $. This is nothing but a parallel shift of parameters by a certain constant $A_0$. $A_0$ can be taken as a noise level, such as readout noise in our case.

\subsection{Likelihood Calculation}
Assume all data values, e.g., the $ f_\lambda $ values, are independent\footnote{A strange thing happens on CCD sometimes maybe? If this is true, then nothing is independent. See \href{https://ui.adsabs.harvard.edu/abs/2018PASP..130f4504B/abstract}{BooneK+18 PASP} (\texttt{2018PASP..130f4504B}) ``A Binary Offset Effect in CCD Readout and Its Impact on Astronomical Data
''.}. Denoting the $ i $-th pixel's value as $ D_i $, %. Applying the product rule to independent measurements,
\begin{equation*}
  P(D|H, I) = P(D_1, \cdots , D_N | H, I) = P(D_1 | H, I) \times \cdots \times P(D_N | H, I) ~.
\end{equation*}

Under $H_0$ (no emission line), $P(D_i | H_0, I) $ is the probability to measure $D_i$ electrons when there is only the pixel noise $\sigma_i$ (no actual line). Each pixel we have Poissonian noise which is nearly Gaussian, and sky estimation noise term, which is difficult to quantify but we usually assume it is Gaussian, and the Gaussian readout noise in standard CCD. Therefore, each pixel is assumed to have a Gaussian noise. Thus,
\begin{equation*}
  P(D_i | H_0, I) = \frac{1}{\sqrt{2 \pi} \sigma_i} \exp{ \left ( -\frac{(D_i - 0)^2}{2 \sigma_r^2} \right )}
  \quad\rightarrow\quad
  P(D|H_0, I) =  \prod_{i=1}^{N} P(D_i | H_0, I)
\end{equation*}
Then the \textbf{log-liklihood} is
\begin{equation}\label{eq: log-l H0}
  \ln P(D|H_0, I)
    = C_\sigma
      - \sum_{i=1}^{N} \frac{D_i^2}{2 \sigma_i^2} ~,
\end{equation}
where $ C_\sigma := \sum \ln (\sqrt{2 \pi} \sigma_i)^{-1} $ is a constant. Note that even the second term is a calculable constant once the data is given.

For $H_1$, we can just change the zero mean to the Gaussian line profile. Consider the best-fit line profile is described as $ g(x| \vec{\theta}_0) $ for $ \vec{\theta}_0 = (A, w, \lambda_c) $ (amplitude, width sigma, central wavelength). Then
\begin{equation}\label{eq: log-l H1}
  \ln P(D|H_1, I)
    = C_\sigma
      - \sum_{i=1}^{N} \frac{(D_i - g(x_i|\vec{\theta}_0))^2}{2 \sigma_i^2}
    \equiv C_\sigma - \frac{1}{2}\chi^2
      ~.
\end{equation}
Here, $ \chi^2 $ is the usual chi-square statistic from the data and model, because in our case, the error-bars are independent and normally distributed. Finding the best-fit parameter set, $ \vec{\theta}_0 $, is done by the least-square fitting (also called the $ \chi^2 $-minimization).

Although it is mathematically too trivial, let me put another theorem to emphasize.
\begin{thm}
Maximizing (log-)likelihood is identical to minimizing $ \chi^2 $, when the error-bars are independent and follows Gaussian.
\end{thm}

\subsection{Model Selection Calculation}
Now let's do the real calculation to compare $H_0$ and $H_1$. Because I want to use amplitude rather than flux as a free paramter, the gaussian function will be $ g(x|\vec{\theta}_0) = A e^{-(\lambda - \lambda_c)^2/2w^2} $. The best fit Gaussian function to the data shown in \cref{fig:greco2018f2-rep} is found to have the following parameters:
\begin{equation}
  \mathrm{amplitude} = 3.213 \times 10^{-20}
  \quad;\quad
  \lambda_c = 5039.4 \,\mathrm{\AA}
  \quad;\quad
  w = 2.33 \,\mathrm{\AA}
\end{equation}
with the integrated flux $ \lg (F_\mathrm{O_{III}} / \mathrm{mW\, m^{-2}}) = -15.73 $, because $ F = A \sqrt{2\pi w^2} $. This matches well with the original publication $ -15.7 \pm 0.1 $.

From the data, $ C_\sigma = 2739.234 $. The log-likelihood of $ H_0 $ and $ H_1 $ are
\begin{align*}
  \ln P(D|H_0, I)
    &= C_\sigma
      - \sum_{i=1}^{N} \frac{D_i^2}{2 \sigma_i^2}
    &&= 2720.810 \\
  \ln P(D|H_1, I)
    &= C_\sigma
      - \sum_{i=1}^{N} \frac{(D_i - g(x_i|\mathrm{amplitude}, \lambda_c, w))^2}{2 \sigma_i^2}
    &&= 2730.136 ~.
\end{align*}

Assume the probability of $ H_0 $ being true is the same as that of $ H_1 $ being true. The  Bayes' ratio in \cref{eq: odds ratio} therefore becomes just a Bayes' factor:
\begin{equation}
  R = B_{10} = \frac{e^{2730.136}}{e^{2720.810}} = 1.12 \times 10^{4} \gg 1 ~,
\end{equation}
which means $ H_1 $ is extremely more likely. Thus, we now believe there must be an emission line, and have to find the CI of the parameters (e.g., flux value).

\subsection{Model Selection with AIC, BIC}
But wait, is this all? No.

The pitfall of this na\"{i}ve approach using Bayes' ratio is that, you can minimize $ \chi^2 $ to 0, by increasing the number of fitting parameters. When the number of free parameters is equal to the number of data points, you must be able to make a function $ f $ such that $ D_i - f(\lambda_i) $ is always 0. Does the $ N $-parameter model a better choice than 1-parameter case, for example, if just the ratio is large? It can't be.

The ``number of paramter'' problem is not an easy thing to solve, but we have simple rule-of-thumbs: The Akaike Information Criteria, AIC, and the Bayesian Information Criteria, BIC. The derivations are not shown here, because it can become a bit lengthy while that derivation itself is not at the heart of the understanding.

\begin{thm}[Bayesian Information Criterion; BIC]
For the $ N (\rightarrow \infty) $ data points, the BIC for the model with $ n $ free parameters denoted as $ \vec{\theta} $ is given as
\begin{equation}\label{eq: bic}
  \mathrm{BIC} := n \ln N - 2 \ln P(D|\vec{\theta}_0, I) = n \ln N + \chi^2_\mathrm{min} - 2C_\sigma ~,
\end{equation}
where $ \vec{\theta}_0 $ is the best-fit parameter which results in the minimum $ \chi^2 $, or maximum likelihood ($ P(D|H, I) $).
For the same data, comparing with two different models, the difference in the BICs is used:
\begin{equation}\label{eq: dbic}
  \Delta \mathrm{BIC}
    = (n_1 - n_2) \ln N
      - 2 \ln \frac{P(D|\vec{\theta}_{0, 1}, I)}{P(D|\vec{\theta}_{0, 2}, I)}
    = (n_1 - n_2) \ln N
      + (\chi^2_\mathrm{min, 1} - \chi^2_\mathrm{min, 2}) ~.
\end{equation}
The second equalities above including $ \chi^2 $ hold only if the error-bars are independent and normally distributed.

Some people prefer to define in the opposite sign and/or half of this value to remove the factor 2 in front of the log-likelihood.
\end{thm}
Note that BIC is usable only if $ N \gg n $. Also the prior distribution only affects when you find $ \vec{\theta} $, not when calculating the BIC.

Since RafteryAE's work\footnote{RafteryAE (1995) ``Bayesian Model Selection in Social Research'', Sociological Methodology, 25, 111}, the following criteria for choosing models are widely used:

\begin{table}[ht!]
\centering
\label{tab: bic}
\begin{tabular}{|c|c|c|c|c|}
  \hline
  $ |\Delta\mathrm{BIC}| \in $ & [0, 2] & [2, 6] & [6, 10] & 10+ \\
  \hline
  Evidence & Not worth more than a bare mention & Positive & Strong & Very Strong \\
  \hline
\end{tabular}
\end{table}

\begin{thm}[Akaike Information Criterion; AIC]
For the $ N (\rightarrow \infty) $ data points, the BIC for the model with $ n $ free parameters denoted as $ \vec{\theta} $ is given as
\begin{equation}\label{eq: aic}
  \mathrm{AIC} := 2n - 2 \ln P(D|\vec{\theta}_0, I) = 2n + \chi^2_\mathrm{min} - 2C_\sigma ~,
\end{equation}
where $ \vec{\theta}_0 $ is the best-fit parameter which results in the minimum $ \chi^2 $, or maximum likelihood ($ P(D|H, I) $).
For the same data, comparing with two different models, the difference in AICs is used:
\begin{equation}\label{eq: daic}
  \Delta \mathrm{AIC}
  = 2(n_1 - n_2) - 2 \ln \frac{P(D|\vec{\theta}_{0, 1}, I)}{P(D|\vec{\theta}_{0, 2}, I)}
  = 2(n_1 - n_2) + (\chi^2_\mathrm{min, 1} - \chi^2_\mathrm{min, 2}) ~.
\end{equation}
The second equalities above including $ \chi^2 $ hold only if the error-bars are independent and normally distributed.
\end{thm}

Although I am not so familiar with hard-core statistics, it seems like there are debates about which should be preferred (BIC or AIC). BIC is known to prefer the ``true model'', if it is in our set of alternative hypotheses, with probability 1 when $ N \rightarrow \infty $, while AIC doesn't. On the other hand, if the data is too few (note that the difference in BIC and AIC is the $ \ln N $ term), BIC tend to seek for the model which explains that small dataset, so it can prefer worse model which tries to explain the bad data points than AIC.

Now let's calculate $ \Delta \mathrm{BIC} $ and $ \Delta \mathrm{AIC} $ for our sample data to test $ H_0 $ VS $ H_1 $. As before, $ C_\sigma = 2739.234 $, and log-likelihoods are 2720.810 and 2730.136. Then because $ H_0 $ has no parameter and $ H_1 $ has three paramters, and $ N = 61 $,
\begin{align*}
  \Delta \mathrm{BIC}
    &= (0 - 3) \ln 61 - 2 (2720.810 - 2730.136)
    &&= 6.32 \\
  \Delta \mathrm{AIC}
    &= 2 (0 - 3) - 2 (2720.810 - 2730.136)
    &&= 12.65
\end{align*}
The fact that these are positive means we should prefer the $ H_1 $, but not as strong as what we've seen from the odd's ratio.

\section{Towards the Parameter Estimation}

\subsection{Brute-Force}
From the last section, we learned we have a clear evidence that there is a line. Then how can we estimate the line properties with uncertainties? This is the same as we did in the statistics chapter. First is the \textbf{brute-force} fixed-grid search scheme

In the chi-square sense, what we have to do are
\begin{enumerate}
\item Calculate the chi-square statistic at each parameter space position.
\item Keep only those with $\chi^2 < \chi^2_\mathrm{min} + \Delta(n_\theta, \alpha)$
  \begin{enumerate}
  \item [-] $\Delta$: inverse cdf (cumulative distribution function) of $\chi^2$ distribution.
  \item [-] $\alpha$: significance level ($\alpha = 0.6827$ for 1-sigma)
  \item [-] $ n_\theta $: number of free parameters.
  \item [-] In python, you can do \pyth{delta = scipy.stats.chi2.ppf(0.6827, n_param)}
  \end{enumerate}
\item These are the models ``within 1-sigma level confidence interval.''
\item Get the min/max of each of the parameters and set these as lower/upper limit of the parameters.
\item The \textit{center} of the parameters can be obtained by simple maximum likelihood estimation, such as least-square fitting.
\end{enumerate}

For the 1-sigma contour of 2 parameters, $ \chi^2 < \chi^2_\mathrm{min} + 2.30 $. The marginalized pdfs is usually drawn together to grasp the distribution of the parameters. If you have used chi-square statistic, you can use the fact that $ P \propto e^{-\chi^2/2} $. Define $ \bar{P} =  e^{-\chi^2/2}$. Then the normalization constant will be $ A = \sum_{parameters} \bar{P} $, i.e., you can use $ P = \bar{P}/A $ as the normalized probability values.

In this example, because the authors mentioned that the $ w $ parameter is obtained from the $ \mathrm{H\alpha} $ fitting, I just fixed the $ w $ value as the best fit value. From zooming in the oringinal paper, I couldn't find any difference from theirs to ours.

\begin{figure}[ht!]
  \centering
  \includegraphics[width=0.7\linewidth]{figs/Greco2018-fit}
  \caption{Our fitting of Gaussian curve to the emission line. The 2-D contour shows the $ \chi^2 $ map in the 2-D parameter space. The 1-D plots show marginalized pdf from $ P = e^{-\chi^2/2} / \bar{P} $.}
  \label{fig:greco2018-fit}
\end{figure}

In the figure, the green vertical lines:
\begin{itemize}
  \item dashed = our best fit central wavelength
  \item dotted = the uncertainty range from the paper's redshift uncertainty measured from $ \mathrm{H\alpha} $ line ($ 1742 \pm 19 \,\mathrm{km/s} $)
  \item Because their uncertainty is from $ \mathrm{H\alpha} $, which has signal much better than $ \mathrm{O^{2+}} $, the error-bar is much smaller than ours.
\end{itemize}
and the blue horizontal lines:
\begin{itemize}
\item dashed = paper's flux ($ 10^{-15.7} \,\mathrm{mW/m^2} $) converted to amplitude
\item dotted = paper's uncertainty around the paper's value.
\item The difference between our fit and paper's value is only 0.03 dex (maybe the authors obtained the identical value but just dropped the significance numbers)
\item Error bars seem slightly understimated, but in reality if we use error bar of 0.14 dex in log scale, it's similar to ours. Maybe the authors just did not care about such detailed numbers, which is understandable.
\end{itemize}

\subsubsection*{Notes}
The original paper argue that they used gaussian with flat spectrum\footnote{From the paper: ``When fitting the [O III] $ \lambda $5007 line, we assume a flat continuum plus a single Gaussian line profile with standard deviation given by the $ \mathrm{H\alpha} $ fit.''}. When I tried this, however, the best fit constant is \texttt{-1.36e-21} , which must be a marginally visible negative shift to the fitted Gaussian, but I cannot find this by zooming in the paper's plot. Thus, I guess they may have just ignored this flat offset or set a bound that this constant must be positive, so that the best fit value is just 0.

\subsection{Markov Chain Monte Carlo (MCMC)}
Markov chain is a fancy name of ``memoryless'':
\begin{equation}\label{eq: markov chain}
  p(x_5|x_4, x_3, x_2, x_1) = p(x_5|x_4)
\end{equation}
The Monte Carlo (MC)\footnote{Historical background adopted from \href{https://events.mpifr-bonn.mpg.de/indico/event/30/material/slides/12.pdf}{KalinovaV's lecture note 2017 Feb}.} came from the casino ``Monte Carlo'' in Monaco. Stanislaw Ulam and John von Neumann, in 1940, wanted to find best-fit parameters to make a better nuclear weapon at the Los Alamos National Laboratory. They arrived at the idea of (now-called) Monte Carlo, but wanted it to be secret to enemies, so they chose the secret name MC, where Ulam's uncle used to play gamble.

When you have, e.g., 7 parameters to fit 100 data points, and if the model is too complicated, the usual grid searching in the previous section is computationally impossible. Therefore, the ``hopping'' in the N-dimensional parameter space is suggested and that's MC. Currently I don't have complete plan to cover MCMC in AO class, but you may refer to many available packages from websites\footnote{A great compilation is available \href{https://gabriel-p.github.io/pythonMCMC/?fbclid=IwAR2hxgATmm1w-QFAjsjcrTbpOHeGV3aJKCCpSnnSuimEXLk9xtC3lpzXgo0}{here}}. I recommend you to try \texttt{emcee} (although I used \texttt{pymc3}, but I feel \texttt{emcee} is more standard and safe as it's classical. \texttt{pymc3} is too much a black box to my eyes, and my friends kind of agreed).

% Let me assume uniform prior to all three paramters with domains $A \in [0.1, 10]$, $\lambda_c \in [5030, 5050]$, and $w \in [0.1, 5]$, and thus $P(A|I) = (1/9.9) (^\forall A \in [0.1, 10])$, etc. Assuming all paramters are independent, we can use product rule and
%\begin{equation}
%  P(D|H_1, I) = \int_{0.1}^{5} \int_{5030}^{5050} \int_{0.1}^{10}
%                P(D | A, \lambda_0, w, I) P(A|I) P(\lambda_0|I) P(w|I)  dA d\lambda_0 dw ~,
%\end{equation}
%
%or
%$$ \log_e P(D|H_1, I) = - \log_e (10000) + \log_e \left [ \int_{5}^{15} \int_{590}^{610} \int_{25}^{75}
 %               P(D | A, \lambda_0, w, I) ~dA~d\lambda_0~dw \right ] ~, $$
%with
%$$ P(D|A, \lambda_0, w, I) = \left ( \frac{1}{\sqrt{2 \pi} \sigma_r} \right )^N
%                \times \exp \left [ -\sum_{i=1}^{N} \frac{(D_i - E_i(A, \lambda_0, w))^2}{2 \sigma_r^2} \right ] ~. $$
%Since $H_0$ has no more parameters, we just use the previous $\log_e P(D|H_0, I)$ formula.


\pagenumbering{arabic}
\setcounter{page}{1}

\chapter{Introduction}

The primary topic of this thesis is to develop and numerically test a large scale implementation of an incompressible magnetohydrodynamics model. In this introductory chapter, we will first  present a description of the model problem studied. We then give a brief overview of finite element methods and Krylov subspace methods for this problem. Finally, we outline the objectives, contributions and structure of the thesis.

\section{A model problem in incompressible magnetohydrodynamics}

The area of incompressible magnetohydrodynamics (MHD)  describes the behaviour of electrically conductive incompressible fluids (liquid metals, plasma, salt water, etc) in an electromagnetic field \cite{davidson2001introduction,le2006mathematical,muller2001magnetofluiddynamics}. MHD models couple electromagnetism and fluid dynamics. The coupling effects are due to two fundamental physical properties. Firstly, through the movement of the conductive material that induces a magnetic field which then modifies any existing electromagnetic field. Secondly, the magnetic and electric fields generate a mechanical force on the fluid known as the Lorentz force. The Lorentz force accelerates the fluid particles in the direction normal to both the electric and magnetic fields.

Incompressible MHD has a number of important applications within technology and industry as well as Geophysical and Astrophysical applications. Some such applications are: electromagnetic pumping, aluminium electrolysis, the Earth's molten core and solar flares. For more applications see \cite{muller2001magnetofluiddynamics}.

In this thesis, we are principally interested in an incompressible MHD model. This means that the electrically conductive fluid is incompressible, i.e., the mass of the fluid is conserved, and the electric resistivity of the fluid cannot be ignored. The MHD model we consider consist of two coupled fundamental equations: the incompressible Navier-Stokes equations and Maxwell's equations. We will outline the derivation of a formulation of an incompressible MHD model for a homogeneous and isotropic medium to ensure that all material parameters are constant; for full details see \cite{armero1996long}.

The transient incompressible Navier-Stokes equations that govern incompressible fluid flow are given by:
\begin{subequations}
\label{eq:ns}
\begin{alignat}2
\rho_f\bigg(\frac{\partial \uu{u}}{\partial t}+ (\uu{u} \cdot \nabla)\uu{u}\bigg)- \mu  \, \Delta\uu{u} +\nabla p &= \uu{f}+\uu{f}_L & \qquad &\mbox{in $\Omega\times(0,T)$},\\[.1cm]
\nabla\cdot\uu{u} &= 0 & \qquad &\mbox{in $\Omega\times(0,T)$}.
\end{alignat}
\end{subequations}
Here $\uu{u}$ and $p$ are the velocity and pressure of the fluid, $\uu{f}$ denotes the body forces acting on the fluid and $\uu{f}_L$ is the Lorentz force, which will be specified later. The parameters $\mu>0$  and $\rho_f>0$ denote the dynamic viscosity and density of the fluid, respectively. The spatial domain is given by $\Omega$ and the end time is denoted by $T$. Mass conservation is given by (\ref{eq:ns}b), see  \cite[Chapter 0]{elman2005finite} for the derivation of the incompressible Navier-Stokes equations.

Maxwell's equations that govern electromagnetic effects are given by:
\begin{subequations}
\label{eq:maxwell2}
    \begin{alignat}2
        &\mbox{Faraday's law:}\quad \quad & \partialt{\uu{b}} +\curl \uu{e} &= \uu{0}, \\
        &\mbox{Coulomb's law:}\quad \quad & \div \uu{d} &= \hat{\rho_{e}}, \\
        &\mbox{Amp\`{e}re's law:}\quad \quad & -\partialt{\uu{d}} + \curl \uu{h} &= {\uu{j}},\\
        &\mbox{Gauss's law:}\quad \quad & \div \uu{b} &= 0.
    \end{alignat}
\end{subequations}
The fields in \eqref{eq:maxwell2} are given by: $\uu{h}$ the magnetic field, $\uu{e}$ the electric field, $\uu{d}$ the electric displacement, $\uu{b}$ the magnetic induction and ${\uu{j}}$ the electric current density. The parameter $\hat{\rho_{e}}$ \RE{is} the electric charge density. In a homogeneous and isotropic medium, the following linear relations hold:
\begin{equation} \label{eq:assumpt}
    \uu{d} = \delta \uu{e}, \quad \uu{b} = \mu \uu{h},
\end{equation}
where the constant $\delta>0$ denotes the electric permittivity and the constant $\mu>0$ the magnetic permeability. Using \eqref{eq:maxwell2} together with \eqref{eq:assumpt} yields the form of Maxwell's equations considered in this thesis:
\begin{subequations}
\label{eq:maxwell}
    \begin{alignat}2
        &\mbox{Faraday's law:}\quad \quad & \partialt{\uu{b}} +\curl \uu{e} &= \uu{0}, \\
        &\mbox{Coulomb's law:}\quad \quad & \div \uu{e} &= \rho_{e}, \\
        &\mbox{Amp\`{e}re's law:}\quad \quad & -\partialt{(\delta\uu{ e})} + \curl (\frac{1}{\mu}\uu{b}) &= \uu{j},\\
        &\mbox{Gauss's law:}\quad \quad & \div \uu{b} &= 0,
    \end{alignat}
\end{subequations}
where $\rho_{e}=\frac{\hat{\rho_{e}}}{\delta}$; see \cite[Chapter 1]{monk2003finite} for more details on Maxwell's equations.

The physical assumptions we consider to form a MHD model are the same as in \cite[Section 2.1]{armero1996long}. More precisely we assume:
\begin{itemize}
    \item Non-relativistic motion: The characteristic fluid velocity is assumed to be orders of magnitude smaller than the speed of light.
    \item Low-frequency approximation: Phenomena involving high frequencies are omitted. That is, the term $\partialt{(\delta\uu{ e})}$ involving the  displacement current is neglected in Maxwell's equations. Therefore, Amp\`{e}re's law~(\ref{eq:maxwell}c) simplifies to
    \begin{equation} \label{eq:AmpereModified}
        \curl \left(\frac{1}{\mu}\uu{b}\right) = \uu{j}.
    \end{equation}
    \item Quasi-neutrality assumption:  Positive and negative charges are equal in any given region. The convection current is omitted and Ohm's law now reads
    \begin{equation} \label{eq:QNassumpt}
        \uu{j} = \theta (\uu{e}+\uu{u}\times\uu{b}),
    \end{equation}
    where positive parameter $\theta$ defines the electric conductivity of the fluid and $\uu{u}\times\uu{b}$ corresponds to the charge density induced by the fluid motion. The electrical resistivity is given by $\nicefrac{1}{\theta}$ and causes dissipative effects in Maxwell's equations but is not to be neglected in this model.
    \item Non-magnetisation and non-polarisation: The assumptions of homogeneity and isotropy also imply that the medium is non-magnetisable and non-polarisable.
\end{itemize}
The Lorentz force $\uu{f}_L$ in (\ref{eq:ns}a) can now be expressed as:
\begin{equation} \label{eq:lorentz}
    \uu{f}_L = \frac{1}{\rho_{f}\,\mu} (\curl \uu{b})\times \uu{b}.
\end{equation}
The electromotive term in \eqref{eq:QNassumpt} now enters Faraday's law (\ref{eq:maxwell}a) by combining \eqref{eq:QNassumpt} and \eqref{eq:AmpereModified} into the following expression for $\uu{e}$:
\begin{equation} \nonumber
    \uu{e} = \frac{1}{\theta}\left(\curl \left(\frac{1}{\mu}\uu{b}\right)-\uu{u}\times\uu{b}\right).
\end{equation}

% \begin{itemize}
%     \item Non-relativistic motion:~\\ In general,  Maxwell's equations are invariant under transformations of the Lorentz group. In this thesis, we assume non-relativistic motion, i.e., the fluid velocity is orders of magnitude smaller than the speed of light. Hence, we assume the Galilean invariant approximation of the electromagnetic field transformations. Defining $\uu{e}'$ and $\uu{b}'$ as the electric field and  magnetic induction at rest in the reference frame  moving at velocity $\uu{u}$ with respect to spatial system, then we suppose that
%     $$\uu{e}'= \uu{e}+\uu{u}\times \uu{b}, \quad \uu{b}'=\uu{b},$$
%     where $\uu{e}$ and $\uu{b}$ are the spatial electric field and spatial magnetic induction.
%     \item High frequencies: ~\\ Phenomena involving high frequencies are not considered, hence, displacement current $\partialt{(\delta\uu{ e})}$ can be neglected. Therefore, Amp\`{e}re's law~(\ref{eq:maxwell}c) simplifies to
%     \begin{equation} \label{eq:AmpereModified}
%         \curl (\frac{1}{\mu}\uu{b}) = \uu{j}.
%     \end{equation}
%     \item Quasi-neutrality: ~\\ Positive and negative charges are equal in any given region. Hence, the convection current can be omitted and Ohm's law becomes
%     $$\uu{j} = \theta \uu{e}'=  \theta (\uu{e}+\uu{u}\times\uu{b}).$$
%     The positive parameter $\theta$ defines the electric conductivity of the fluid and $\uu{u}\times\uu{b}$ describes the flow of the fluid that has been induced by the electric field.  The electrical resistivity is given by $\nicefrac{1}{\theta}$ and causes dissipative effects in Maxwell's equations but is not neglected.
%     \item Non-magnetisable and non-polarised medium: ~\\ This assumption implies that the relations \eqref{eq:assumpt} hold for all values of the electric permeability and magnetic permittivity. Therefore, the Lorentz force $\uu{f}_L$ in (\ref{eq:ns}a) is now given by:
%     $$\uu{f}_L = \frac{1}{\rho_{f}} \uu{j}\times \uu{b}.$$
%     Substituting Amp\`{e}re's law \eqref{eq:AmpereModified} (under the low frequency assumption) into the Lorentz force gives the following relationship:
%     $$\uu{f}_L = \frac{1}{\rho_{f}\,\mu} (\curl \uu{b})\times \uu{b}.$$
% \end{itemize}

{Using the assumptions above, elimination of the electric field $\uu{e}$ and non-dimensionalisation, the systems in \eqref{eq:ns} and \eqref{eq:maxwell} are coupled into the following set of partial differential equations:}
\begin{subequations}
\label{eq:mhdnon}
\begin{alignat}2
 \frac{\partial \uu{u}}{\partial t}- \nu  \, \Delta \uu{u} + (\uu{u} \cdot \nabla) \uu{u}+\nabla p - \kappa\, (\nabla\times\uu{b})\times\uu{b} &= \uu{f}, \\
\nabla\cdot\uu{u} &= 0, \\
\frac{\partial \uu{b}}{\partial t}+\nu_m  \, \nabla\times( \nabla\times \uu{b})
-  \, \nabla\times(\uu{u}\times \uu{b}) &= \uu{0}, \\
\nabla\cdot\uu{b} &= 0,
\end{alignat}
\end{subequations}
 with suitable initial conditions and boundary conditions; see \cite[Section 2]{armero1996long}. The unknowns $\uu{u}$, $p$ and $\uu{b}$ are the fluid velocity, the fluid pressure and the magnetic field, respectively.
 % The first coupling term $\kappa\, (\nabla\times\uu{b})\times\uu{b}$ in (\ref{eq:mhdnon}a) represents to the Lorentz force $f_L$ in \eqref{eq:lorentz}, whereas the second coupling term $\nabla\times(\uu{u}\times \uu{b})$ in (\ref{eq:mhdnon}c) corresponds to the electromotive force modifying the magnetic field, due to the motion of the conductive fluid.

The solution to \eqref{eq:mhdnon} depends on three non-dimensional parameters $\nu = \nicefrac{1}{\rm Re}$, $\nu_m = \nicefrac{1}{\rm Rm}$ and $\kappa$. The first parameter Re is the hydrodynamic Reynolds number, which indicates the balance between the inertial forces and the viscous forces. The parameter Rm is the magnetic Reynolds number, which measures the effect by which the magnetic field induces flow motion. The final parameter, the coupling number, $\kappa$, represents the influence of the electromagnetic field on the flow. It is sometimes defined in terms of the Hartmann number denoted by Ha, as
\begin{equation}
    \nonumber
    \mbox{Ha} = \sqrt{\kappa \, \mbox{Re\,Rm}}.
    % \kappa = \frac{\mbox{Ha}^2}{\mbox{Re\,Rm}}.
\end{equation}
To find typical physical values for these parameters, we refer to \cite{armero1996long,le2006mathematical,roberts1967introduction}. We refer to $\nu$ as the viscosity for the rest of this thesis.

In this thesis, we are interested in the steady-state ($\frac{\partial}{\partial t} = 0$) version of~\eqref{eq:mhdnon}:
\begin{subequations}
\label{eq:steadystate}
\begin{alignat}2
 - \nu  \, \Delta \uu{u} + (\uu{u} \cdot \nabla) \uu{u}+\nabla p - \kappa\, (\nabla\times\uu{b})\times\uu{b} &= \uu{f}, \\
\nabla\cdot\uu{u} &= 0, \\
\label{eq:curlcurl}
\kappa\nu_m  \, \nabla\times( \nabla\times \uu{b})
- \kappa \, \nabla\times(\uu{u}\times \uu{b}) &= \uu{0}, \\
\nabla\cdot\uu{b} &= 0.
\end{alignat}
\end{subequations}
Note we have multiplied (\ref{eq:steadystate}c) by $\kappa$ to enforce skew-symmetry of the coupling terms. We will consider both two- and three-dimensional solutions to \eqref{eq:steadystate}. \RE{See Appendix~\ref{Curl} for curl definitions.}

% The curl operator is well defined in three-dimensions and the two-dimensional curl may be defined as follows: given 2D vector fields $\uu{b}(x,y) = (b_1,b_2)$, $\uu{u}(x,y) = (u_1,u_2)$ and the scalar function $r(x,y)$ then the curl and cross products are
% \begin{subequations}
% \nonumber
% \begin{alignat}2
% \curl \uu{b} &=& \frac{\partial b_2}{\partial x} - \frac{\partial b_1}{\partial y}, \\
% \curl r &=& \Big(\frac{\partial r}{\partial y}, -\frac{\partial r}{\partial x}\Big),\\
% \uu{u} \times \uu{b} &=& u_1b_2-u_2b_1.
% \end{alignat}
% \end{subequations}
% Note that taking the curl of a 2D vector field results in a scalar function which is the component in the normal direction to the 2D field ($z$-component).

\section{Numerical solution}

The partial differential equation (PDE) system given in \eqref{eq:steadystate} requires a numerical approximation as in general an analytical solution is not possible. There are two main components in computing a numerical solution of a PDE:
\begin{itemize}
    \item[1.] Discretisation: take a continuous model and transfer it into a discrete model;
    \item[2.] Solve: take the discretised model and solve for the unknowns.
\end{itemize}
In this thesis, we use mixed finite element methods for discretising an MHD model problem, and solve it using preconditioned Krylov subspace methods.
In the sequel, we briefly describe these components in the context of the approach we take.

\subsection{Finite element methods for incompressible MHD problems}

There are several finite element methods for discretising MHD problems as in \eqref{eq:steadystate}. A common approach in the literature is to approximate the magnetic field using standard nodal $H^1$-conforming elements \cite{phillips2014block,armero1996long,gerbeau2000stabilized,gunzburger1991existence}. Such a formulation enables one to use the following vector calculus identity
\begin{equation} \label{eq:CurlIdentity}
-\Delta \uu{b} = \curl (\curl \uu{b}) - \nabla(\div \uu{b}).
\end{equation}
Since $\uu{b}$ is divergences-free it is then possible to apply an augmentation technique to replace the curl-curl operator with a vector Laplacian. This then reduces one of the principal  computational difficulties, namely the large null-space of the curl-curl operator in \eqref{eq:curlcurl}. However, one of the main problems using $H^1$-conforming elements for the magnetic field is that for non-convex domains (such as the 2D L-shaped domain with a reentrant corner, or the 3D Fichera corner domain with reentrant edges and corners) the magnetic field will converge to a solution that is not correct around the singular point, see \cite{codina2006stabilized,costabel2000singularities}. Therefore, we consider a mixed discretisation that captures singular solutions correctly. One such family of elements are $H({\rm curl})$ conforming \nedelec elements \cite{nedelec1980mixed}.

To enable the use of \nedelec elements for the magnetic field we use the mixed formulation in \cite{schotzau2004mixed,GreifLiSchotzauWei2010}. This leads to the following governing equations in a domain $\Omega$:
\begin{subequations}
\label{eq:mhd}
\begin{alignat}2
\label{eq:mhd1} - \nu  \, \Delta\uu{u} + (\uu{u} \cdot \nabla)
\uu{u}+\nabla p - \kappa\,
(\nabla\times\uu{b})\times\uu{b} &= \uu{f} & \qquad &\mbox{in $\Omega$},\\[.1cm]
\label{eq:mhd2}
\nabla\cdot\uu{u} &= 0 & \qquad &\mbox{in $\Omega$},\\[.1cm]
\label{eq:mhd3}
\kappa\nu_m  \, \nabla\times( \nabla\times \uu{b})
+ \nabla r
- \kappa \, \nabla\times(\uu{u}\times \uu{b}) &= \uu{g} & \qquad &\mbox{in $\Omega$},\\[.1cm]
\label{eq:mhd4} \nabla\cdot\uu{b} &= 0 & \qquad &\mbox{in $\Omega$},
\end{alignat}
\end{subequations}
where we have introduced the Lagrange multiplier, $r$, in the form of $\nabla r$ in \eqref{eq:mhd3}. Again, $\uu{u}$ and $p$ are the velocity and pressure of the fluids and $\uu{b}$ is the magnetic field. The introduction of the Lagrange multiplier, $r$, corresponds to the divergence-free constraint \eqref{eq:mhd4} of the magnetic field. With the addition of the Lagrange multiplier, $r$, we may also introduce a generic forcing term $\uu{g}$ associated with the Maxwell part of \eqref{eq:mhd}.

The numerical tests that we will consider will have inhomogeneous Dirichlet boundary conditions for the fluid and magnetic fields and homogeneous Dirichlet boundary condition for the multiplier, $r$, of the form:
\begin{subequations}
\label{eq:bc}
\begin{alignat}2
\label{eq:bc1} \uu{u} &= \uu{u_D} & \qquad &\mbox{on $\partial\Omega$},\\[.1cm]
\label{eq:bc2}
   \uu{n}\times\uu{b} &= \uu{n} \times \uu{b_D} & \qquad &\mbox{on $\partial\Omega$},\\[.1cm]
\label{eq:bc3}      r &=0 &\qquad &\mbox{on $\partial\Omega$},
\end{alignat}
\end{subequations}
where $\uu{u_D}$ and $\uu{b_D}$ are given functions, and $\uu{n}$ is the unit outward normal to the boundary $\partial \Omega$. Notice, by taking the divergence of equation \eqref{eq:mhd3} we obtain Poisson's equation (as $\div \curl \uu{b} = 0$):
$$\Delta r = \div \uu{g} \quad \mbox{in } \Omega, \quad r = 0 \quad \mbox{on } \partial \Omega.$$
In many physical applications $\uu{g}$ is divergence-free, which implies that the multiplier, $r$, is zero. In general, the main purpose of the magnetic multiplier is to provide stability, see~\cite{demkowicz1998modeling}.

\subsection{Preconditioning incompressible MHD problems}

Incompressible  MHD problems have been extensively studied in the context of various discretisation and formulations. However, the development of preconditioned iterative solutions to MHD problems is limited. \RE{We refer the reader to the Appendix for a review of Krylov subspace solvers for linear systems. In this thesis our focus will be on the implementation of preconditioners which are tailored to MHD model problem.}

% there does not seem to have been much study of preconditioned iterative solutions.

In the literature there have not been too many approaches to precoditioning the MHD equations given in either the non-multiplier form \eqref{eq:steadystate} or with the multiplier form \eqref{eq:mhd}. In the very recent work \cite{phillips2014block}, an operator-based preconditioner for the non-multiplier MHD equations \eqref{eq:steadystate} has been proposed in the context of $H^1$-conforming elements for the magnetic field. To form their preconditioner the authors use the identity \eqref{eq:CurlIdentity} and a discrete commutator idea to form approximations to the Schur complements. This is based on an approach for a preconditioner to the  Navier-Stokes system as in \cite[Chapter 8]{elman2005finite}. The preconditioners  we employ are based on similar Navier-Stokes preconditioners but rely on the Maxwell preconditioner in \cite{greif2007preconditioners} for $H({\rm curl})$ elements.

\section{Objectives and contributions}

The aim of this thesis is to develop and test fully scalable iterative solution methods for the incompressible MHD model \eqref{eq:mhd}, \eqref{eq:bc} using natural Taylor-Hood elements \cite{taylor1973numerical} for the fluid variables and \nedelec mixed element  \cite{nedelec1980mixed} pair for the magnetic variables.  Our numerical results show good scalability with respect to the mesh size. We provide  several tests  to study the performance of the preconditioners with respect to the relevant non-dimensional parameters. We also present two- and three-dimensional results.

To enable large scale preconditioned tests of the MHD model we use the finite element software \fenics \cite{wells2012automated} together with the linear algebra software from {\tt PETSc} \cite{petsc-web-page,petsc-user-ref}. Using these two principal software packages, experiments were run in excess of 20 million degrees of freedom. As well, it provides an example of how significant physical problems described by partial differential equations can be solved by combining state of the art numerical software packages. The aim is to release the code for public use.

As stated above,  little has been done in terms of preconditioned iterations for the MHD model other than the works of  \cite{phillips2014block} for the non-multiplier form \eqref{eq:steadystate}. Our approach is based on $H({\rm curl})$ elements for the magnetic field \cite{schotzau2004mixed}, and motivated by the preliminary results of \cite{li2010numerical} in the context of exactly divergence-free elements for the velocity field. It combines preconditioners for the incompressible Navier-Stokes and Maxwell's equations in \cite{elman2005finite,greif2007preconditioners,MR2911387}.

% The scalable solvers considered in this thesis may now enable more investigation of the physical problems they describe.
The availability of our large-scale solvers and code will hopefully allow more development and research into such MHD models.

\section{Outline}

This thesis is made up of five chapters and is structured as follows. In Chapter 2, we introduce a mixed finite element approximation to the MHD system \eqref{eq:mhd}-\eqref{eq:bc}. The mixed approximation is based on a standard nodal Taylor-Hood finite element approximation for the velocity field and the pressure, together with a mixed \nedelec element approximation for the magnetic field and the multiplier. Using this approximation, we introduce three possible non-linear iteration schemes.

In Chapter 3, we present an overview of the preconditioning approaches for the individual subproblems separately, namely the incompressible Navier-Stokes and Maxwell's equations. We then apply these preconditioning techniques to propose preconditoning strategies for the linearisations which arise from the three non-linear iteration schemes from Chapter~2. In particular, for the coupled linearised Picard scheme we propose an inner-outer preconditioning approach.

In Chapter 4, to perform numerical experiments in both two and three spatial dimensions we use the following two main software packages \fenics \cite{wells2012automated} and {\tt PETSc} \cite{petsc-web-page,petsc-user-ref}. We show convergence results for the linearised MHD system along with the incompressible Navier-Stokes and Maxwell subproblems in isolation. Along with the convergence results we numerically test the preconditioning approaches for the three non-linear iteration schemes, providing heuristic tests with respect to the dimensionless parameters ($\nu$,~$\nu_m$~and~$\kappa$;~see~\eqref{eq:mhd}) and mesh size. These tests examine the robustness of both the preconditioners and the iteration schemes.

Chapter 5 provides conclusions and outlines possible extensions for future work.


%% LyX 2.3.6.2 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[12pt,a4paper,fleqn,american,parskip=half-,svgnames]{scrartcl}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}

\makeatletter

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
\pdfpageheight\paperheight
\pdfpagewidth\paperwidth

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
\newcommand{\velo}{\upsilon}
\newcommand{\cheat}{C}

\usepackage{lastpage}
\usepackage{textgreek}
\usepackage{fixltx2e}
\usepackage{hyperref} %ermöglicht \href{}
\usepackage{tabularx}
\usepackage{url}
\usepackage{microtype}
\usepackage[nomessages]{fp}
\usepackage{pdfpages}

%colors
\definecolor{link_color}{HTML}{00406E} % dark blue
\definecolor{cite_color}{HTML}{590000} % dark red
\definecolor{dark-violet}{HTML}{9400d3}
\definecolor{forest-green}{HTML}{228b22}
\definecolor{dark-red}{HTML}{8b0000}
\definecolor{dark-blue}{HTML}{00008b}
\definecolor{dark-pink}{HTML}{ff1493}
\definecolor{dark-salmon}{HTML}{e9967a}
\definecolor{midnight-blue}{HTML}{191970}

\hypersetup{hidelinks = true}
\hypersetup{
    colorlinks   = true, %Colours links instead of ugly boxes
    urlcolor     = link_color, %Colour for external hyperlinks
    linkcolor    = black, %Colour of internal links
    frenchlinks  = true, %small caps
    citecolor    = cite_color %Colour of citations
}

\makeatother

\usepackage{babel}
\begin{document}

\section{Phase field modeling}

Given is the following energy functional
\begin{align*}
F\left[\phi(x,y,t)\right] & =\int_{-\infty}^{\infty}\underbrace{\left(\frac{U}{2}\left[a^{2}\left((\partial_{x}\phi)^{2}+(\partial_{y}\phi)^{2}\right)+g(\phi)\right]+\mu_{0}h(\phi)\right)}_{f(\phi,\partial_{x}\phi,\partial_{y}\phi)}dx,
\end{align*}
where $a$ and $U$ are constants of the dimension length and energy
respectively. $f(\phi,\partial_{x}\phi,\partial_{y}\phi)$\footnote{$\partial_{x}$ is an abbreviation for the partial derivative with
respect to $x$, i.e.~$\partial_{x}\equiv\partial/\partial x$} denotes the local free energy density and $\mu$ the bulk free energy
density difference between the two phases. Depending on the sign of
$\mu$, this can either favor the growth of the one or the other phase.
$g(\phi)=\phi^{2}(1-\phi)^{2}$ is the double well potential and $h(\phi)=\phi^{2}(3-2\phi)$
is the interpolation function. Please note, that for the reason of
phase-stability we have to demand $|\mu|<U/6$.

Variational principles provide the phase field equation
\begin{align}
\frac{1}{M_{\phi}}\frac{\partial\phi}{\partial t} & =-\frac{\delta F}{\delta\phi}\nonumber \\
 & =\partial_{x}\frac{\partial f}{\partial\left(\partial_{x}\phi\right)}+\partial_{y}\frac{\partial f}{\partial\left(\partial_{y}\phi\right)}-\frac{\partial f}{\partial\phi}\nonumber \\
 & =U\left(a^{2}(\partial_{x}^{2}\phi+\partial_{y}^{2}\phi)-\frac{1}{2}\frac{\partial g(\phi)}{\partial\phi}\right)-\mu\frac{\partial h(\phi)}{\partial\phi}.\label{eq:Kinetische-Gl}
\end{align}

\subsection{Stability of the homogenous and time independent solutions}

We look for constant solutions of Eq.~\ref{eq:Kinetische-Gl}
\begin{align*}
0= & \frac{U}{2}\frac{\partial g(\phi)}{\partial\phi}+\mu\frac{\partial h(\phi)}{\partial\phi}
\end{align*}
Calculation of the partial derivatives of the polynomial functions
leads to
\begin{align*}
\frac{\partial g(\phi)}{\partial\phi} & =2\phi(1-\phi)(1-2\phi)\\
\frac{\partial h(\phi)}{\partial\phi} & =6\phi(1-\phi)
\end{align*}
 Inserting in the equation above yields
\begin{align*}
0 & =U\phi(1-\phi)(1-2\phi)+\mu6\phi(1-\phi)\\
0 & =\phi(1-\phi)(1-2\phi)+\frac{6\mu}{U}\phi(1-\phi)\\
0 & =\phi(1-\phi)\left(1-2\phi+\frac{6\mu}{U}\right)\\
\Rightarrow & \phi_{1}=0;\phi_{2}=1;\phi_{3}=\frac{1}{2}+\frac{3\mu}{U};
\end{align*}
Stability of the solutions:
\begin{itemize}
\item $\phi_{1}=0$ is a global (local) minimum if $\mu_{0}>0$ ($\mu_{0}<0$),
i.e.~stabile (meta stabile)
\item $\phi_{2}=1$ is local (global) minimum if $\mu_{0}>0$ ($\mu_{0}<0$),
d.h.~meta stabile (stabile)
\item $\phi_{3}=\frac{1}{2}+3\mu_{0}/U$ is for positive and negative $\mu$
unstable ( $|\mu|<U/6$)
\end{itemize}

\subsection{Phase-field profile function}

We show that
\begin{align}
\phi_{0}(x,t)= & \frac{1}{2}\left(1+\tanh\frac{(x-vt)}{2a}\right)\label{eq:1DPhasenfeldloesung}
\end{align}
is a heterogeneous solution of the phase field equation (\ref{eq:Kinetische-Gl})
if $v=6M_{\phi}a\mu$. Note that from $\partial_{x}\left(\tanh(x)\right)=1-\tanh^{2}(x)$
we can deduct the following property of this solution $\partial_{x}\phi_{0}=\phi_{0}\left(1-\phi_{0}\right)/a$.

Calculation of the derivatives:
\begin{align}
\frac{\partial\phi_{0}}{\partial x} & =\frac{1}{2}\frac{\partial}{\partial x}\left(1+\tanh\frac{(x-vt)}{2a}\right)=\frac{1}{4a}\left(1-\tanh^{2}\frac{(x-vt)}{2a}\right)\nonumber \\
 & =\frac{1}{4a}\left(1+\tanh\frac{(x-vt)}{2a}\right)\left(1+1-1-\tanh\frac{(x-vt)}{2a}\right)\nonumber \\
 & =\frac{1}{4a}\left(1+\tanh\frac{(x-vt)}{2a}\right)\left(2-\left(1+\tanh\frac{(x-vt)}{2a}\right)\right)\nonumber \\
 & =\frac{1}{a}\frac{1}{2}\left(1+\tanh\frac{(x-vt)}{2a}\right)\left(1-\frac{1}{2}\left(1+\tanh\frac{(x-vt)}{2a}\right)\right)\nonumber \\
 & =\frac{1}{a}\phi_{0}\left(1-\phi_{0}\right)\label{eq:Intro-Phasefield-stat-1d-profil-first-deriv}\\
\frac{\partial^{2}\phi_{0}}{\partial x^{2}} & =\frac{1}{a}\frac{\partial}{\partial x}\left[\phi_{0}\left(1-\phi_{0}\right)\right]=\frac{1}{a}\frac{\partial}{\partial\phi_{0}}\left[\phi_{0}\left(1-\phi_{0}\right)\right]\frac{\partial\phi_{0}}{\partial x}\nonumber \\
 & =\frac{1}{a^{2}}\phi_{0}\left(1-\phi_{0}\right)\left(1-2\phi_{0}\right),\label{eq:Intro-Phasefield-stat-1d-profil-sec-deriv}\\
\frac{\partial\phi_{0}}{\partial t} & =-v\frac{\partial\phi_{0}}{\partial x}=-\frac{v}{a}\phi_{0}\left(1-\phi_{0}\right).
\end{align}
Where the second derivative has been calculated using the chain rule
$\frac{\partial f(\varphi(x))}{\partial x}=\frac{\partial f}{\partial\varphi}\frac{\partial\varphi}{\partial x}$.
Inserting in the nonlinear partial differential equation
\begin{align*}
-\frac{v}{a}\phi_{0}\left(1-\phi_{0}\right) & =M_{\phi}U\underbrace{\left[a^{2}\frac{1}{a^{2}}\phi_{0}\left(1-\phi_{0}\right)\left(1-2\phi_{0}\right)-\phi(1-\phi)(1-2\phi)\right]}_{=0}\\
 & \hspace*{1em}-M_{\phi}\mu6\phi_{0}\left(1-\phi_{0}\right)\\
\Leftrightarrow\quad-v & =-6M_{\phi}a\mu.
\end{align*}

\subsection{Interface energy density}

The interface energy density $\gamma$ in the phase-field model corresponds
to the total free energy of the heterogeneous solution $\gamma=F[\phi_{0}(x,t)]$:

\begin{align*}
F & =\frac{U}{2}\int_{-\infty}^{\infty}\left(a^{2}\left(\frac{\partial\phi_{0}}{\partial x}\right)^{2}+\phi_{0}^{2}(1-\phi_{0})^{2}\right)dx\\
 & =\frac{U}{2}\int_{-\infty}^{\infty}\left(a^{2}\left(\frac{1}{a}\phi_{0}\left(1-\phi_{0}\right)\right)^{2}+\phi_{0}^{2}(1-\phi_{0})^{2}\right)dx\\
\left[dx=\frac{a}{\phi_{0}(1-\phi_{0})}d\phi_{0}\right] & =aU\int_{0}^{1}\frac{\phi_{0}^{2}(1-\phi_{0})^{2}}{\phi_{0}(1-\phi_{0})}d\phi_{0}\\
 & =aU\int_{0}^{1}\phi_{0}(1-\phi_{0})d\phi_{0}\\
 & =aU\left(\frac{1}{2}\phi_{0}^{2}-\frac{1}{3}\phi_{0}^{3}\right)|_{0}^{1}=\frac{aU}{6}
\end{align*}

\subsection{Calibration of the field model}

We calibrate the phase field model according to the 1D considerations
above, i.e. we switch from the parameters $a,U,M_{\phi}$ to the parameters
$\xi=2a$ for the phase-field width, $\Gamma=aU/6$ for interface
energy density and $M=M_{\phi}a^{2}U$ for the kinetic coefficient
$M[\mathrm{m}^{2}/\mathrm{s}]$. The calibrated phase-field model
provide the following relation between the driving force $\mu$ and
the resulting stationary interface velocity $v$
\begin{align*}
v & =\frac{M}{\Gamma}\mu_{0}=K\mu_{0}.
\end{align*}
With these parameters we obtain the following phase-field equation

\begin{align*}
\frac{1}{M}\partial_{t}\phi & =\underbrace{\partial_{x}^{2}\phi+\partial_{y}^{2}\phi}_{\mathrm{Laplace-Operator}}-\frac{2}{\xi^{2}}\partial_{\phi}g(\phi)-\frac{\mu}{3\Gamma\xi}\partial_{\phi}h(\phi).
\end{align*}

\end{document}


% -*-LaTeX-*-

% $Log: convolution.tex,v $
% Revision 1.6  2007/12/11 02:32:54  stiber
% Small edits for start of Winter 2008.
%
% Revision 1.5  2007/03/20 23:53:12  stiber
% Updated LaTeX.
%
% Revision 1.4  2007/03/20 01:25:56  stiber
% Modifications made to make this a standalone text.
%
% Revision 1.3  2006/03/27 23:36:42  stiber
% Fixed error in formula.
%
% Revision 1.2  2004/03/29 19:51:48  stiber
% Updated for Spring 2004 and new textbook (DSP First).
%
% Revision 1.1  2004/02/19 00:26:01  stiber
% Initial revision
%

\chapter{The Z-Transform and Convolution}
\label{ch:convolution}

Two signal processing tools --- the \emph{z transform} and
\emph{convolution} --- are introduced in this chapter.  These
operations play important roles in the analysis of discrete-time
signals (which is what we do in the computer). We shall see that they
are related --- the convolution of two time-domain signals (which is
what we do when we filter a signal) is equivalent to multiplication of
their corresponding z-transforms. This is one example of how these
representations can greatly simplify computation.

After studying this chapter, you should be able to understand what the
z-transform and convolution are, and how to implement them. You should
understand the differences between them and the other transforms:
Fourier series, Fourier transform, and discrete Fourier transform. You
will enrich your knowledge of filter transfer functions with its time
domain representation: its \emph{impulse response}.

\section{Domains}

Up to this point, we have covered the Fourier series representation of
a signal as a weighted sum of sinusoids. In effect, the Fourier series
\emph{transforms} a finite and periodic, continuous signal in the time
\emph{domain} into an infinite, discrete spectrum in the frequency
\emph{domain}.
\index{domains}
\index{time domain}
\index{frequency domain}
\index{domain!time}
\index{domain!frequency}
When we use the term \emph{domain}, we merely mean a particular way of
looking at a signal. In this case, we have two different ways of
thinking about our signals: as functions of time or as functions of
frequency. These are equivalent, in the sense that we can convert the
signal's representation back and forth between the two domains without
loss of information (neglecting matters such as roundoff error).

In future chapters, we will cover two other transforms:
\begin{description}
\item[Fourier transform] transforms an infinite, continuous signal in
the time domain into an infinite, continuous spectrum in the frequency
domain.

\item[Discrete Fourier transform] transforms a finite, discrete signal
in the time domain into a finite, discrete spectrum in the frequency
domain.
\end{description}

Here, however, we will learn about the
\emph{z-transform}, which converts an infinite, discrete signal in the
time domain into a finite, continuous spectrum in the frequency
domain. The z-transform fills the last combination among ``finite vs.
infinite; continuous vs. discrete''. Table~\ref{tb:zt-transforms}
summarizes all four transforms.  As you can see, continuous versus
discrete in the time domain transforms to infinite versus finite in
the frequency domain, while finite versus infinite in the time domain
transforms to discrete versus continuous in the frequency domain.

\begin{table}
\caption{Summary of frequency transforms.
\label{tb:zt-transforms}}
\begin{center}
\begin{tabular}{|l|l|l|} \hline
Transform & Time Domain & Frequency Domain\\ \hline\hline
Fourier Series & Finite, Continuous & Infinite, Discrete \\
Fourier Transform & Infinite, Continuous & Infinite, Continuous \\
Discrete Fourier Transform & Finite, Discrete & Finite, Discrete \\
Z-Transform & Infinite, Discrete & Finite, Continuous \\ \hline
\end{tabular}
\end{center}
\end{table}

\section{The z-transform}
The \emph{z-transform} of a discrete time signal $x[n]$, $n=0, \pm 1,
\pm 2, \ldots, \pm\infty$ is defined as the power series
\begin{equation}
X(z) \equiv \sum_{k=-\infty}^{\infty} x[k] z^{-k}
\label{eq:zt}
\end{equation}
where $z$ is a continuous complex variable. It transforms the time
domain, infinite, discrete sequence into its complex plane
representation $X(z)$. Since the z-transform is an infinite power
series, it exists only for those values of $z$ for which this series
converges. The \emph{region of convergence} (ROC) of $X(z)$ is the set
of all values of $z$ for which $X(z)$ has a finite value. We consider
the z-transform to be a transform between the time domain and the
frequency domain because we can substitute $z=e^{j\hat{\omega}}$ into
equation~(\ref{eq:zt}) to get the signal's (finite and continuous)
frequency content $\mathcal{X}(\hat{\omega})$ --- just as we
previously made the same substitution to derive a feedforward filter's
frequency response from its transfer function.

The relationship between $x[n]$ and $X(z)$ can be indicated
by the \emph{transform pair}
\begin{equation}
\underbrace{x[n]}_{\stackrel{\text{function of}}{_\text{sample \#}}}
   \stackrel{\mathbf{Z}}{\longleftrightarrow}
   \underbrace{X(z)}_{\stackrel{\text{function of}}{_\text{complex $z$}}}
\end{equation}

\problemset{
\subsubsection{Self-Test Exercises}

See~\ref{sc:ch6ex} \#\ref{it:ch6ex1}--\ref{it:ch6ex2} for answers.

\begin{enumerate}
\item Determine the z-transform for the sequence
  $x[n]=\{1,2,5,7,0,1\}$, $n=0,1,2,3,4,5$

\item Determine the z-transform of the sequence
  $x[n]=\{1,2,5,7,0,1\}$, $n=-2,-1,0,1,2,3$
\end{enumerate}}

\subsection{Example: z-transform of an impulse}
\label{sc:zx-impulse}

\index{z-transform!of an impulse}
\index{unit impulse}
The \emph{unit impulse} or \emph{unit sample} signal is the $\delta$
function,
\begin{equation}
\delta[n] = \left\{\begin{array}{ll}
                        1 & n=0 \\
                        0 & n \neq 0
          \end{array}\right.
\end{equation}
It has value of zero for every sample except $n=0$, for which it has a
value of one. Substituting this signal into~(\ref{eq:zt}) to get its
z-transform, we have
\begin{align}
\Delta(z) &= \sum_{k=-\infty}^{\infty} \delta[k] z^{-k} \notag\\
     &= 1z^{-0}=1
\end{align}
that is
\begin{equation}
\delta[n]\stackrel{\mathbf{Z}}{\longleftrightarrow} 1
\end{equation}

Since the frequency content of the signal is the magnitude of its
z-transform on the unit circle in the z-plane,
\begin{equation}
|\mathcal{D}(\hat{\omega})|=|\Delta(e^{j\hat{\omega}})|=1
\label{eq:zt-delta}
\end{equation}
This tells us that the frequency content is the same for all
frequencies: an impulse has a \emph{flat spectrum}.

What about time shifted impulses,
\begin{align}
\delta[n-n_0] &= \left\{\begin{array}{ll}
                        1 & n=n_0 \\
                        0 & n \neq n_0
          \end{array}\right., \quad n_0>0 \\
\delta[n+n_0] &= \left\{\begin{array}{ll}
                        1 & n=-n_0 \\
                        0 & n \neq -n_0
          \end{array}\right., \quad n_0>0
\end{align}
In these cases the nonzero value is not at sample zero, but at samples
$n_0$ or $-n_0$.  We can compute the z-transform as before,
\begin{align}
\Delta(z) &= \sum_{k=-\infty}^{\infty} \delta[k-n_0] z^{-k} \notag\\
          &= 1z^{-n_0}=z^{-n_0}=\frac{1}{z^{n_0}}
\end{align}
for $\delta[n-n_0]$.  The z-transform for this shifted unit impulse
has one value, $z^{-n_0}$, for any $z \neq 0$.
\begin{equation}
\delta[n-n_0] \stackrel{\mathbf{Z}}{\longleftrightarrow} \frac{1}{z^{n_0}},
\quad n_0>0
\end{equation}

Its frequency content is also one, just like $\delta[n]$ (remember
that we compute the spectrum for values of $z$ on the unit circle),
\begin{equation}
|\mathcal{D}(\hat{\omega})|=|\Delta(e^{j\hat{\omega}})|=|e^{-jn_0\hat{\omega}}|=1
\end{equation}
This is not surprising at all, because it is, after all, just a
time-shifted version of $\delta[n]$. The signal $\delta[n+n_0]$ is
left as a self-test exercise.

\problemset{
\subsubsection{Self-Test Exercises}

See~\ref{sc:ch6ex} \#\ref{it:ch6ex3}--\ref{it:ch6ex4} for answers.

\begin{enumerate}
\item Sketch equation~(\ref{eq:zt-delta}).
\item Compute the z-transform and frequency content for the signal
  $\delta[n+n_0]$.
\end{enumerate}}

\subsection{Example: z-transform of exponential signal}
\label{sc:zx-exp}

\index{z-transform!of an exponential}
An exponential signal is defined as
\begin{equation}
x[n] = \left\{\begin{array}{ll}
                        \alpha^n & \quad  n \ge 0 \\
                        0        & \quad n < 0
          \end{array}\right.
\label{eq:zt-expof}
\end{equation}
where $\alpha$ can be any real or complex value less than one. The
signal consists of an infinite number of samples. The z-transform of
this signal is
\begin{align}
X(z) &= \sum_{k=-\infty}^{\infty} x[k] z^{-k} \notag\\
     &= \sum_{k=0}^{\infty}\alpha^k  z^{-k} \notag\\
     &= \sum_{k=0}^{\infty}(\alpha z^{-1})^k
\label{eq:exp-zt}
\end{align}

This is an infinite \emph{geometric series}: a sum in which each
successive term is the previous term times some (unchanging)
expression (i.e., the ratio of successive terms is constant).
\index{geometric series} The ratio of two successive terms in a
geometric series like \index{geometric series!common ratio}
equation~(\ref{eq:exp-zt}) is called its \emph{common ratio}, which in
this case is $\alpha z^{-1}$. We can show this by rewriting
equation~(\ref{eq:exp-zt}) as $X(z) = 1 + \alpha z^{-1} + \alpha^{2}
z^{-2} + \cdots$. Rewriting the \spscript{$i+1$}{st} element of this
series as a recurrence relation (in terms of the \spscript{$i$}{th}),
\index{recurrence relation}
we get $X(z)_{i+1} = X(z)_i \alpha z^{-1}$, which shows the common ratio.

If we have a geometric series in which successive terms $b_i$ and
$b_{i+1}$ have the common ratio $r$ (i.e., $b_{i+1}/b_i = r$), then
any term in the series can be expressed in terms of the first term as
\index{geometric series!as function of first term}
\begin{equation*}
b_i = b_0 r^i
\end{equation*}
So, a geometric series can be expressed as a sum of these, $b_0 + b_0r
+ b_0r^2 + \cdots$. We can factor out the zeroth term, leaving us with
the task of simplifying $1 + r + r^2 + \cdots$. Multiplying this sum
by $(1-r)/(1-r)$, we obtain
\begin{align}
(1 + r + r^2 + \cdots)\frac{1-r}{1-r}
  & = \frac{1 + r + \cdots - r - r^2 - \cdots}{1-r} \notag\\
  & = \frac{1-r^N}{1-r}
\end{align}

In this case, $|r|<1$ so $r^N \rightarrow 0$ as $N \rightarrow \infty$:
\begin{equation}
1+r+r^2+r^3+\ldots = \frac{1}{1-r}, \quad\text{if } |r|<1
\end{equation}
Consequently, for $|r|=|\alpha z^{-1}|<1$ or $|z|>|\alpha|$,
$X(z)$ converges to
\begin{equation}
X(z)=\frac{1}{1-\alpha z^{-1}}, \quad |z|>|\alpha|
\label{eq:zt-expoF}
\end{equation}

In the z-plane, $|z|>|\alpha|$ refers to any $z$ that is outside of
the radius $|\alpha|$ circle. We see that in this case, the
z-transform provides a compact alternative representation of the
signal $x[n]$.

Let's check out some special cases:

\begin{figure}
\centerline{\includegraphics[width=3.5in]{ch-conv/zt_expo_rf}}
\caption{The exponential signal $x[n]=(1/2)^n, n=0,1,2,\ldots$.
\label{fig:zt-expo-r0.5f}}
\end{figure}

\paragraph*{When $\alpha$ is a real number, say $\alpha=1/2$:}
The discrete signal in this case is
\begin{equation}
x[n] = \left\{\begin{array}{ll}
                        \left(\frac{1}{2}\right)^n & \quad  n \ge 0 \\
                        0             & \quad n < 0
          \end{array}\right.
\end{equation}
or
\begin{equation}
x[n] = \left\{1, \frac{1}{2}, \left(\frac{1}{2}\right)^2,
        \left(\frac{1}{2}\right)^3, \ldots, \right\}
\end{equation}
Figure~\ref{fig:zt-expo-r0.5f} shows the graph of the signal $x[n]$.

Replacing $\alpha$ with $1/2$ in~(\ref{eq:zt-expoF}), its
z-transform is expressed as
\begin{equation}
X(z)=\frac{1}{1-\frac{1}{2}z^{-1}}, \quad |z|>\frac{1}{2}
\end{equation}

As you should be familiar with now, the frequency content of this
signal is the magnitude of its z-transform on the unit circle
$z=e^{j\hat{\omega}}$ in the z-plane, which is
\begin{equation}
|\mathcal{X}(\hat{\omega})|=|X(e^{j\hat{\omega}})|
           =\left|\frac{1}{1-\frac{1}{2}e^{-j\hat{\omega}}}\right|
\end{equation}

\begin{figure}
\centerline{\includegraphics[width=2.5in]{ch-conv/zt_expo_r0-5F}}
\caption{Frequency content of the signal shown
in figure~\protect\ref{fig:zt-expo-r0.5f}.
\label{fig:zt-expo-r0.5F}}
\end{figure}

Figure~\ref{fig:zt-expo-r0.5F} is the plot of
$|\mathcal{X}(\hat{\omega})|$ versus frequency $\omega$.  The
amplitude decreases along increasing frequency. Its peak is at zero
frequency, which is called \emph{DC} (which literally means ``direct
current,'' implying the constant --- actually mean --- component of
the signal).

\paragraph*{When $\alpha=1$:}

For $\alpha=1$, the sequence becomes
\begin{equation}
x[n] = u[n] = \left\{\begin{array}{ll}
                        1 & n \ge 0 \\
                        0 & n < 0
          \end{array}\right.
\end{equation}
or
\begin{equation}
u[n] = \{1, 1, 1, \ldots\}, \quad n \ge 0
\end{equation}

\index{unit step}
This is a discrete time, infinite duration \emph{unit step}
signal. Notice the difference between the unit step signal and the
unit impulse signal. The latter only has one nonzero value at one
particular time, the former has value one for all time after some
particular time. By analogy with $\delta[n]$ , a unit step occurring at
sample $k$ is called $u[n-k]$.  Substituting $\alpha=1$
into~(\ref{eq:zt-expoF}) we have
\index{z-transform!of a unit step}
\begin{equation}
U(z)=\frac{1}{1-z^{-1}}, \quad |z|>1
\end{equation}
We can see that the pole (a zero in the denominator; you'll learn more
about this in Chapter~\ref{ch:fb-filters}) is at $z=1$, where the
z-transform has an infinite value.

Let's evaluate the frequency content of the unit step signal. If we
evaluate $\mathcal{U}(\hat{\omega})$ on the unit circle (except at
$z=1$), we obtain
\begin{align}
\mathcal{U}(\hat{\omega})
&= U(e^{j\hat{\omega}})
  =\frac{1}{1-e^{-j\hat{\omega}}}\frac{e^{j\hat{\omega}/2}}{e^{j\hat{\omega}/2}}
\notag\\
&= \frac{e^{j\hat{\omega}/2}}{e^{j\hat{\omega}/2}-e^{-j\hat{\omega}/2}} \notag\\
&= \frac{e^{j\hat{\omega}/2}}{2j\sin\hat{\omega}/2} \notag\\
&= \frac{e^{j(\hat{\omega}/2-\pi/2)}}{2\sin\hat{\omega}/2}, \quad \hat{\omega}\ne
2\pi k, k=0,1,\ldots
\end{align}
because $-j=e^{-j\pi/2}$ (see the self-test exercises).  Hence, the
presence of a pole (a zero in the denominator) at $z=1$ (that is, at
$\hat{\omega}=0$) creates a problem only when we want to compute
$|\mathcal{U}(\hat{\omega})|$ at $\hat{\omega}=0$, because
$|\mathcal{U}(\hat{\omega})|\rightarrow \infty$ as
$\hat{\omega}\rightarrow 0$. For any other value of $\hat{\omega}$,
$|\mathcal{U}(\hat{\omega})|$ is finite.

\begin{figure}
\centerline{\includegraphics[width=3.5in]{ch-conv/zt_expo_r1F}}
\caption{Frequency content of the unit step signal.
\label{fig:zt-expo-r1F}}
\end{figure}

Figure~\ref{fig:zt-expo-r1F} shows a plot of
$|\mathcal{U}(\hat{\omega})|$ vs.  $\hat{\omega}$.  Since the signal
is a unit step, and so has a constant value from zero onwards, we
might expect the signal to have zero frequency components at all
frequencies except at $\hat{\omega}=0$, but that is not the case. The
reason is that the signal is not a constant for all $-\infty <n<
\infty$. Instead, it is turned on at $n=0$. This \emph{abrupt jump}
creates all the frequency components existing in the range
$0<\hat{\omega}\le\pi$. Generally, \emph{all} signals which start at a
finite time will have nonzero frequency components everywhere in the
frequency axis from zero up to the Nyquist frequency. All such signals
can be considered to be the product of some infinite signal with a
unit step; we will see the effect of this on their spectrum when we
explore convolution in section~\ref{sc:convolution}.

\paragraph*{\textsc{Optional:} When $\alpha$ is a complex number, $\alpha=Re^{j\theta}$:}

When $\alpha=Re^{j\theta}$, equations~(\ref{eq:zt-expof})
and~(\ref{eq:zt-expoF}) become
\begin{equation}
x[n] = \left\{\begin{array}{ll}
                        R^ke^{jn\theta} & \quad  n \ge 0 \\
                        0             & \quad n < 0
          \end{array}\right.
\end{equation}

\begin{equation}
X(z)=\frac{1}{1-Re^{j\theta}z^{-1}}, \quad |z|>|R|
\label{eq:zt-expo-cF}
\end{equation}

When $z=Re^{j\theta}$ we have what we call a \emph{pole} (a zero in the
denominator; you'll learn more about this in
Chapter~\ref{ch:fb-filters}). Equation~(\ref{eq:zt-expo-cF}) can be
broken into real and imaginary parts using Euler's formula:
\begin{align}
X(z) &= \frac{1}{1-Re^{j\theta}z^{-1}} \notag\\
     &= \frac{1}{1-R(\cos\theta+j \sin\theta )z^{-1}} \notag\\
     &= \frac{1}{1-R\cos\theta z^{-1}-j R\sin\theta z^{-1}} \notag\\
     &= \frac{1-R\cos\theta z^{-1}+j R\sin\theta z^{-1}}
     {[1-R\cos\theta z^{-1}]^2 - [jR\sin\theta z^{-1}]^2} \notag\\
     &= \frac{1-R\cos\theta z^{-1}+j R\sin\theta z^{-1}}
     {1-2R\cos\theta z^{-1} + R^2 z^{-2}} \notag\\
     &= \frac{1-R\cos\theta z^{-1}}
     {1-2R\cos\theta z^{-1} + R^2 z^{-2}}
     +j\frac{R\sin\theta z^{-1}}
     {1-2R\cos\theta z^{-1} + R^2 z^{-2}}
\end{align}

We break the signal into two parts, too:
\begin{align}
\Real\{x[n]\} &= R^n\cos(k\theta), \quad n\ge 0\\
\Imag\{x[n]\} &= R^n\sin(k\theta), \quad n\ge 0
\end{align}
From real part we get
\begin{equation}
R^n\cos(n\theta)\stackrel{\mathbf{Z}}{\longleftrightarrow} \
\frac{1-R\cos\theta z^{-1}}
{1-2R\cos\theta z^{-1} + R^2 z^{-2}}
\end{equation}
and from imaginary parts we have
\begin{equation}
R^n\sin(n\theta)\stackrel{\mathbf{Z}}{\longleftrightarrow} \
\frac{R\sin\theta z^{-1}}
{1-2R\cos\theta z^{-1} + R^2 z^{-2}}
\end{equation}

When $R<1$, $R^n\cos(n\theta)$ and $R^n\sin(n\theta)$ are damped
cosine and sine waves.

\problemset{
\subsubsection{Self-Test Exercises}

See~\ref{sc:ch6ex} \#\ref{it:ch6ex5}--\ref{it:ch6ex7} for answers.

\begin{enumerate}
\item What is the derivative of $u[n-k]$ (the unit step at time step
  $k$)?
\item Show that $e^{j\hat{\omega}/2}-e^{-j\hat{\omega}/2} =
  2j\sin\hat{\omega}/2$.
\item Prove that $e^{-j\pi/2}=-j$.
\end{enumerate}}

\section{Convolution}
\label{sc:convolution}

\index{convolution!discrete!definition} Convolution is an important
operation for implementing digital filters.  Let's first define what
convolution is. For two infinite, discrete signals $x[n]$ and $h[n]$,
the \emph{convolution} $y[n]$ of them at sample $n$ is defined as
\begin{equation}
y[n] = \sum_{k=-\infty}^{\infty} h[k] x[n-k]
\label{eq:convolution}
\end{equation}
The notation for convolution is ``$\ast$'', so this can be written as
\begin{equation}
Y = X \ast H
\end{equation}
where $Y$, $X$, and $H$ are the signals with samples at $n$ being
$y[n]$, $x[n]$, and $h[n]$.

Let's consider the case when both $x$ and $h$ both start at zero. This
is equivalent to saying that both have values of zero before that
sample. So, $h[k] = 0$ when $k<0$ and $x[n-k]=0$ when
$n-k<0$. Equation~(\ref{eq:convolution}) becomes
\begin{equation}
y[n] = \sum_{k=0}^{n} h[k] x[n-k]
\label{eq:zt-conv}
\end{equation}
Actually, this is a more realistic situation than $k$'s summation
from $-\infty$ to $\infty$.

We expand the summation in~(\ref{eq:zt-conv}) as
\begin{multline}
y[n] = h[0] x[n] + h[1] x[n-1] + h[2] x[n-2] + \ldots + h[k] x[n-k]
       + \ldots \\
       {}+ h[n-2] x[2] + h[n-1] x[1] + h[n] x[0]
\label{eq:zt-conv-exp}
\end{multline}
Using this formulation, you may show that the convolution $X \ast H$ has the
following properties:
\begin{enumerate}
\item Commutative: $X \ast H=H \ast X$
\index{convolution!commutative property}
\item Distributive :  $X \ast (H_1 + H_2)=X \ast H_1 + X \ast H_2$
\index{convolution!distributive property}
\item Associative : $(X \ast H) \ast G = X \ast (H \ast G)$
\index{convolution!associative property}
\end{enumerate}
You should be able to convince yourself that $x \ast \vec{0} = \vec{0}
\ast x = 0$ (where $\vec{0}$ is a vector of all zeros). How about
$\vec{1} \ast \vec{1}$ (where $\vec{1}$ is a vector of all ones, in
this case $\vec{1}=u[n]$; see the self-test exercise)?

\subsection{Example of Convolution}

Determine the convolution $e^n \ast e^n$, $n=0,1,2,\ldots,
$. Using~(\ref{eq:zt-conv}),
\begin{align}
e^n \ast e^n &= \sum_{k=0}^{n}e^ke^{n-k} \notag\\
             &= \sum_{k=0}^{n}e^n=e^n\sum_{k=0}^{n}1=ne^n
\end{align}

\problemset{
\subsubsection{Self-Test Exercises}

See~\ref{sc:ch6ex} \#\ref{it:ch6ex8}--\ref{it:ch6ex9} for answers.

\begin{enumerate}
\item Determine if $u[n] \ast H \ne H$ is true, where $h[n] = n$,
$n=0,1,2,\ldots$ (a \emph{ramp}).
\item Compute $u[n] \ast u[n]$.
\end{enumerate}}

\subsection{Implementing Convolution}

\index{convolution!discrete!implementation}
From observation of equation~(\ref{eq:zt-conv-exp}), we know that for
a fixed sample $n$, $y[n]$ can be computed by the term-by-term
multiplication of the sequence
\begin{equation}
\{h[0], h[1], h[2], \ldots, h[k], \ldots, h[n-2], h[n-1], h[n]\}
\end{equation}
and the time-reversed sequence
\begin{equation}
\{x[n], x[n-1], x[n-2], \ldots, x[n-k] , \ldots, x[2], x[1], x[0]\}
\end{equation}
We just multiply the corresponding terms (for example, $h[k]x[n-k]$),
then add these products. This produces the convolution for one sample
$n$.  Remember, however, that the output $y[n]$ is also a sequence,
$n=0,1,2,\ldots$. We need to repeat this process for all $\{n\}$ to
get the full sequence, as shown in algorithm~\ref{alg:convolution}.

\begin{algorithm}
\caption{Discrete convolution.\label{alg:convolution}}
\begin{algorithmic}
\REQUIRE $h[n]$ is a finite, discrete signal, $n = 0, 1, 2, \ldots$
\REQUIRE $x[n]$ is a finite, discrete signal, $n = 0, 1, 2, \ldots$
\ENSURE $y[n]$ is the convolution $X * H$, $n = 0, 1, 2, \ldots$
\FOR{$n=0, 1, 2, \ldots$}
   \STATE Reverse $x[k]$ to produce $x'[k] = x[n-k]$, $k = 0, 1, 2,
          \ldots, n$
   \STATE $s[k] = h[k] x'[k]$
   \STATE $y[n] = \sum_{k=0}^n s[k]$
\ENDFOR
\end{algorithmic}
\end{algorithm}

A more-or-less direct implementation of this algorithm in C is:

\index{C code!convolution|(}
\begin{small}
\begin{verbatim}
/* Convolution of two vectors

   Input: vectors x and h of lengths nx and nh (nh < nx)
   Output: vector y of length nx + nh - 1 (storage already allocated)
 */
void convolve(int x[], unsigned nx, int h[], unsigned nh, int y[])
{
   for (unsigned n=0; n<nx+nh-1; n++) {
      y[n] = 0;
      for (unsigned k=max(0,n-nh+1); k<=min(nx-1,n); k++)
         y[n] += x[k] * h[n-k];
   }
}
\end{verbatim}
\end{small}
\index{C code!convolution|)}

\begin{figure}
\centerline{\includegraphics[width=\textwidth]{ch-conv/fig6-5}}
\caption{Example of convolution function execution for $n_x=5$ and
$n_h=2$.\label{fg:convex}}
\end{figure}

You'll notice something nasty was done here: the identities of
$x$ and $h$ were reversed! Of course, this is perfectly OK, given the commutative
property of convolution. This was done because we generally assume that $h$
(the shorter vector) is a property of the filter (we will see about
this later) and it is easier to think about reversing
it than reversing the signal $x$. If for no other reason, this makes
sense because $h$ has a shorter length.

The implementation is for finite-length signals, rather than the
infinite ones we've been discussing. It assumes that $h$ is shorter
than $x$ ($n_h < n_x$). This $h$ input is often called the convolution
\emph{kernel} because, as we shall see, $h$ represents the
\index{convolution!kernel}
action of our signal processing system while $x$ is the actual input
signal.  Notice that, for small $n<n_h-1$, not all of $h$ is
used. This is equivalent to multiplying the unused elements of $h$
against the zero values of $x[k]$, $k<0$. Similarly, for large
$n>nx-1$, part of $h$ is also unused --- multiplied against the zero
values of $x[k]$, $k>n_x-1$. Figure~\ref{fg:convex} illustrates
function execution for a signal $X$ of length 5 and a kernel $H$ of
length 2. This is one way to deal with the \emph{boundary conditions}
\index{convolution!boundary conditions}
associated with the convolution: what to do at the ends of the
signal $X$.  In general, there are three ways of dealing with these
boundary conditions:
\begin{enumerate}
\item ``Pad'' $X$ with zeros past its ends. In effect, this is what
was done in the code above.
\item ``Reflect'' $X$ by copying element $k$ to index $-k$. This is
sometimes done in image processing operations.
\item ``Truncate'' the convolution at the ends of $X$. This means that
$n$ would cover the range $n_h-1 \leq n \leq n_x$ and $Y$ would be
shorter than $X$.
\end{enumerate}

\index{MATLAB code!convolution}
MATLAB has the built-in convolution function \verb|conv|, which takes
two vectors as inputs and outputs the convolution result with length
equal to one less than the sum of the two input vector lengths. You
can use ``\verb|help conv|'' to get more information (how does
\verb|conv| deal with boundary conditions [answer in~\ref{sc:ch6ex}
\#\ref{it:ch6ex10}]?)

\begin{figure}
\centerline{\includegraphics[width=4in]{ch-conv/zt_conv_etet}}
\caption[Convolution of $e^n \ast e^n$]{Convolution of $e^n \ast
  e^n$. Top blue line is $e^n$, top red line is time reversed version
  of $e^n$, the bottom plot is the result of convolution.
  \label{fig:zt-conv-etet}}
\end{figure}

Let's look at the convolution of $X \ast H =e^n \ast e^n$ again. In
this case, $X$ and $H$ are the same function. In
figure~\ref{fig:zt-conv-etet} (top), $X$ is shown as a blue curve and
the time-reversed $H$ is shown as a red curve. The convolution result
is in the bottom graph.

\problemset{
\subsubsection{Self-Test Exercises}

See~\ref{sc:ch6ex} \#\ref{it:ch6ex11} for the answer.

\begin{enumerate}
\item Use MATLAB to compute the convolution $e^{-n}*e^{-n}$ and plot
the result.
\end{enumerate}}

\section{Properties of the Z-Transform}

\index{z-transform!properties of}
\begin{table}
\caption{Some properties of the z-transform.\label{tb:zt-property}}
\begin{center}
\begin{tabular}{|l|c|c|} \hline
Property      & Time Domain, $Z^{-1}\{\cdot\}$ & z-Domain, $Z\{\cdot\}$ \\ \hline\hline
Linearity     & $a_1x[n]+a_2y[n]$ & $a_1X(z)+a_2Y(z)$\\
Time shift    & $x[n-k]$       & $z^{-k}X(z)$\\
Scaling in the z-domain
              & $a^nx[n]$        & $X(a^{-1}z)$ \\
Time reversal & $x[-n]$        & $X(z^{-1})$\\
Differentiation in the z-domain
              & $nx[n]$          & $-z \deriv{X(z)}{z}$ \\
Convolution   & $x[n] \ast y[n]$       & $X(z)Y(z)$ \\ \hline
\end{tabular}
\end{center}
\end{table}

The z-transform is a very powerful signal processing tool because it
has some very important properties. Some of these properties are
listed in
table~\ref{tb:zt-property}, where the time-domain signals $x[k]$
and $y[k]$ have z-transforms of $X(z)$ and $Y(z)$.

Knowing these properties can be very convenient. For example, the
z-transform of a signal shifted (delayed) by $k$ samples, $x[n-k]$,
is $z^{-k}X(z)$; this is our familiar $z$ (delay) operator. You can
\index{z-transform!relation to $z$ operator}
\index{z-transform!and delay}
also see that the convolution property of the z-transform means that
convolution in the time domain is multiplication in the z-domain. So,
if we have the z-transform of two signals, it is much easier to
perform convolution.  Later on, you will find out that this is very
important in filtering. Let's prove this property:

From~(\ref{eq:zt-conv}) a convolution of $x[n]$ and $h[n]$ is defined as:
\begin{equation}
y[n] = x[n] \ast h[n] = \sum_{k=-\infty}^{\infty} h[k] x[n-k]
\end{equation}
The z-transform of $y[n]$ is
\begin{align}
Y(z) &= \sum_{n=-\infty}^{\infty} y[n]z^{-n} \notag\\
     &= \sum_{n=-\infty}^{\infty}
     \left( \sum_{k=-\infty}^{\infty} h[k] x[n-k] \right)z^{-n}
\end{align}
Interchanging the order of the summations (which is equivalent to
factoring out the $h[k]$ and distributing the $z^{-n}$ over the inner
summation),
\begin{equation}
Y(z) = \sum_{k=-\infty}^{\infty} h[k]
             \left( \sum_{n=-\infty}^{\infty} x[n-k] z^{-n} \right)
\end{equation}
The inner summation is merely the z-transform of $x[n]$ shifted by $k$
samples.  Applying the time shift property of the z-transform, we
obtain
\begin{align}
Y(z) &= \sum_{k=-\infty}^{\infty} h[k] X(z) z^{-k} \notag\\
     &= X(z)\sum_{k=-\infty}^{\infty} h[k] z^{-k}
     = X(z) H(z)
\end{align}
Which is the product of the two z-transforms. I'll present a couple
examples of using these properties.

\subsection{Example: Time Shifting}

\index{z-transform!of a time-shifted impulse}
Remember that the z-transform of the unit impulse
$\delta[n]$ was discussed in section~\ref{sc:zx-impulse}
\begin{equation}
\delta[n] = \left\{\begin{array}{ll}
                        1 & n=0 \\
                        0 & n \neq 0
          \end{array}\right.
\end{equation}
as being 1 and that the z-transform of the shifted unit impulse
$\delta[n-k]$ is $z^{-k}$? Using the time-shifting property of the
z-transform, this is easy to determine:
\begin{equation}
\delta[n]\stackrel{\mathbf{Z}}{\longleftrightarrow} 1
\end{equation}
then
\begin{equation}
\delta[n-k]\stackrel{\mathbf{Z}}{\longleftrightarrow} 1 \times z^{-k}=z^{-k}
\end{equation}

\subsection{Example: Convolution}

\index{z-transform!computing convolution with}
Given the signals
\begin{equation}
x[n] = \{1, -3, 2, 1\}, \quad n = 0,1,2,3
\end{equation}
and
\begin{equation}
h[n] = \left\{\begin{array}{ll}
                        1 &  n=0,1\\
                        0 &  n=2,3
          \end{array}\right.
\end{equation}
use the z-transform to compute their convolution $Y = X \ast H$.

According to~(\ref{eq:zt}),
\begin{align}
X(z) &= 1-3z^{-1}+2z^{-2}+z^{-3}\\
H(z) &= 1+z^{-1}
\end{align}
Then using the convolution property of the z-transform, we have
\begin{align}
Y(z)=X(z)H(z)
&= (1-3z^{-1}+2z^{-2}+z^{-3})(1+z^{-1}) \notag\\
&= 1-2z^{-1}-z^{-2}+3z^{-3}+z^{-4}
\end{align}
Now we can easily get the inverse z-transform $y[n]=x[n] \ast h[n]$ from the
result $Y(z)$. Again from the z-transform
definition~(\ref{eq:zt}),
\begin{equation}
y[n] = x[n] \ast h[n] = \{1,-2, -1, 3, 1\}
\end{equation}

We can also compute the convolution directly, according to the
convolution definition~(\ref{eq:zt-conv}). There are only 4 nonzero
terms in $x[n]$ and 2 in $h[n]$. For a fixed sample $n$, the
convolution is given by
\begin{align}
y[n] &= \sum_{k=0}^{n} x[k] h[n-k] \notag\\
     &= x[0] h[n] + x[1] h[n-1] + x[2] h[n-2] + x[3] h[n-3]
\end{align}
Changing $n$ gives the sequence of the convolution output as a
function of time. In the following computation, $h[n-k]=0$ is used,
if $n-k<0$ and $x[k]=0$ and $h[k]=0$ if $k>3$:
\begin{align*}
y[0] &= x[0] h[0] = 1\\
y[1] &= x[0] h[1] + x[1] h[0] = 1-3 = -2\\
y[2] &= x[0] h[2] + x[1] h[1] + x[2] h[0] = 0-3+2 = -1\\
y[3] &= x[0] h[3] + x[1] h[2] + x[2] h[1] +x[3] h[0] = 0+0+2+1 = 3\\
y[4] &= x[1] h[3] + x[2] h[2] + x[3] h[1] = 0+0+1 = 1\\
y[n] &= 0, \quad n>4
\end{align*}
Therefore, this also gives the result
\begin{equation}
y[n] = x[n] \ast h[n] = \{1,-2, -1, 3, 1\}
\end{equation}

\problemset{
\subsubsection{Self-Test Exercises}

See~\ref{sc:ch6ex} \#\ref{it:ch6ex12} for the answer.

\begin{enumerate}
\item Prove the scaling property of the z-transform; that is, if
  \begin{equation*}
    x[n]\stackrel{\mathbf{Z}}{\longleftrightarrow} X(z)
  \end{equation*}
  then
  \begin{equation*}
    a^n x[n]\stackrel{\mathbf{Z}}{\longleftrightarrow} X(a^{-1}z)
  \end{equation*}
\end{enumerate}}

\section{Impulse Response and the Transfer Function}

Recall the a filter's input/output relationship is summarized by the
transfer function discussed in chapter~\ref{ch:filt-intro} (and which
you'll see again in chapter~\ref{ch:fb-filters}). Let's denote the
input signal as $x[n]$ and output as $y[n]$ in the time domain. You
learned that the the transfer function in the z domain is $H(z)$. What
is its time domain representation? We will answer this shortly; first
let's give a name to the time domain transfer function,
$h[n]$: the filter's
\emph{impulse response}. The filter's input/output relationship can be
\index{filter!impulse response}
written using $h[n]$ as:
\begin{equation}
y[n] = x[n] \ast h[n]
\label{eq:zt-hx}
\end{equation}
This says that the output of filter results from the convolution
between input signal $x[n]$ and filter's impulse response $h[n]$. From
earlier in this chapter, you now know that convolution of two signals
in the time domain is equivalent to multiplying their z-transforms in
the $z$ domain. So, we obtain the input/output relationship via the
transfer function in the $z$ domain as
\begin{equation}
Y(z) =X(z)H(z)
\label{eq:zt-HX}
\end{equation}
where $X(z)$ and $Y(z)$ are the z-transforms of $x[n]$ and $y[n]$ and
$H(z)$ is the z-transform of $h[n]$.  Actually, this $H(z)$ is just
the transfer function we talked about in chapter~\ref{ch:filt-intro}.
In other words, a filter's transfer function is the z-transform of its
impulse response!
\index{z-transform!relationship to filter transfer function}
\index{filter!transfer function!relationship to its z-transform}

\index{filter!transfer function!determining with z-transform}
As a simple example of how to use the z-transform to determine a
filter's transfer function from its defining equation, consider the
feedforward filter:
\begin{equation}
y[n] = b_0x[n] + b_1x[n-k]
\end{equation}

Applying the z-transform to both sides,
\begin{equation}
Y(z) = b_0X(z) + b_1z^{-k}X(z)
\end{equation}
because of the time shift property of the z-transform. This can be
rearranged to be
\begin{equation}
Y(z) = (b_0 + b_1z^{-k})X(z)
\end{equation}
So the transfer function is
\begin{equation}
H(z) = b_0 + b_1z^{-k}
\end{equation}
and therefore, we have
\begin{equation}
Y(z)=H(z)X(z)
\end{equation}

Applying the inverse z-transform, we can get the time domain
representation of $H(z)$, or the impulse response $h[n]$. In a similar
manner, we can get the output $y[n]$ from $Y(z)$.  In fact,
using~(\ref{eq:zt-HX}), we found an easy way to compute a filter's
response to a signal if we have already know the signal's z-transform
and the filter's transfer function.

Let's see why $h[n]$ is called the impulse response.  Remember the unit
impulse, which is a signal that has the value one at $n=0$ and zero
otherwise, and which can be expressed as
\begin{equation}
\delta[n] = \left\{\begin{array}{ll}
                        1 & n=0 \\
                        0 & n \neq 0
          \end{array}\right.
\end{equation}

The impulse response of a filter is its output when a unit impulse is
applied. Now we compute these two functions' convolution,
\begin{equation}
y[n]=\sum_{k=0}^{n}\delta[k] h[n-k] = h[n]
\end{equation}
because all of the terms except $k=0$ drop out (since $\delta[k]=0$
for all $k \ne 0$).  This tell us that the filter response $y[n]$ to a
unit impulse is $h[n]$. This is why $h[n]$ is called the impulse
response.

Applying the z-transform to both side of the above equation, we get
\begin{equation}
Y(z)= H(z)
\end{equation}
Therefore, the \emph{transfer function} can be viewed as the
z-transform of the filter's \emph{impulse response}, or its impulse
response in the $z$ domain.

If we restrict $z$ to lie on the unit circle, $z=e^{j\hat{\omega}}$,
from~(\ref{eq:zt-HX}) we obtain
\begin{equation}
\mathcal{Y}(\hat{\omega})=\mathcal{H}(\hat{\omega})\mathcal{X}(\hat{\omega})
\end{equation}
where $\mathcal{X}(\hat{\omega})$ is the signal's frequency content,
$\mathcal{Y}(\hat{\omega})$ is the frequency content of the filter
output and $\mathcal{H}(\hat{\omega})$ is the filter's frequency
response.

\section{Problems}

\begin{enumerate}
\item Compute the z-transform of
\begin{equation}
x[n] = \left\{\begin{array}{ll}
                        (-1)^n & n \ge 0 \\
                        0 & n < 0
          \end{array}\right.
\end{equation}
and determine its frequency content.

\item Determine the z-transform of the signal
\begin{equation}
x[n] = \left\{\begin{array}{ll}
                        \cos\hat{\omega}_0 n & n \ge 0 \\
                        0 & n < 0
          \end{array}\right.
\end{equation}

\item Find the z-transform of the signal
\[x[n] = \left\{\begin{array}{ll} 1/n! & n \geq 0 \\
                0 & n < 0 \end{array} \right. \]
Recall that $0!=1$.

\item The impulse response of a system is $h[n]=\{1,2,1,-1\}$, $n=-1,
0,1,2$. Determine the response of the system to the input signal
$x[n]=\{1,2,3,1\}$, $n=0, 1,2,3$.

\end{enumerate}

\section{Further Reading}

\begin{itemize}
\item James H McClellan, Ronald W. Schafer, and Mark A. Yoder,
  \textit{DSP First: A Multimedia Approach}, Prentice Hall, 1998,
  chapter 7 (\S 7.1--7.6.3).
\end{itemize}

% LocalWords:  DTFT signal's feedforward