\chapter{Inference for the mean}
\label{meaninf}

We introduced the Mahalanobis distance earlier in \ref{standarddist}.   Consideration of the \emph{squared} Mahalabnobis distance leads us to consider the so called T$^{2}$ statistic (the nomenclature reflects that this relates to a $t$-statistic for one variable). This can be found as:

\begin{equation}
\label{t2}
T^{2} = n (\boldsymbol{\mu}_{0} - \bar{\boldsymbol{x}})^{T} \boldsymbol{S}  (\boldsymbol{\mu}_{0} - \bar{\boldsymbol{x}})
\end{equation}
where $n$ is the sample size, $\boldsymbol{\mu}_{0}$ is the hypothesised mean, $\bar{\boldsymbol{x}}$ and $\boldsymbol{S}$ are the sample mean and covariance matrices respectively.    It turns out that this statistic follows a T$^{2}$ distribution, however, given that there is a simple relationship between the T$^{2}$ and $F$ distribution it is often easier to work with the latter.

%turn this into a theorm ala flury 379, proof in Anderson and Seber (and muirhead
If $\boldsymbol{x}_{i}$, $i = 1, \ldots n$ represent a sample from a $p$ variate normal distribution with mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$, provided $\boldsymbol{\Sigma}$ is positive definite and $n > p$, given sample estimators for mean and covariance $\bar{\boldsymbol{x}}$ and $\boldsymbol{S}$ respectively, then:
\begin{equation}
F = \left(\frac{n}{n-1}\right) \left(\frac{n-p}{p}\right)  (\boldsymbol{\mu}_{0} - \bar{\boldsymbol{x}})^{T} \boldsymbol{S}  (\boldsymbol{\mu}_{0} - \bar{\boldsymbol{x}})
\end{equation}
follows an $F$-distribution with $p$ and $(n-p)$ degrees of freedom.   Note the requirement that $n > p$, i.e. that $\boldsymbol{S}$ is non-singular.   This clearly limits the use of this test in bio-informatic applications and we may examine a few proposals to deal with this.   To carry out a test on $\boldsymbol{\mu}$, we determine whether $F \leq F_{(1-\alpha),p,n-p}$, the $(1-\alpha)$ quantile of the $F$ distribution on $p$ and $n-p$ degrees of freedom.   We reject the null hypothesis if our test statistic exceeds this value.   We will not consider the one sample T$^{2}$ test any further, but will now examine the two-sampled test.

\section{Two sample Hotelling's T$^{2}$  test}
\label{t2}

Analagous to the univariate context, we wish to determine whether the mean vectors are comparable, more formally:
\begin{equation}
H_{0}: \boldsymbol{\mu}_{1} = \boldsymbol{\mu}_{2}
\end{equation}

The T$^{2}$ statistic proposed by \cite{Hotelling:1931}, will be based this time on the distance between two mean vectors.   It can be calculated as:

\begin{equation}
\label{hotelling}
T^{2} = \left(\frac{n_{1}n_{2}}{n_{1}+n_{2}}\right)(\boldsymbol{\bar{x}_{1}} - \boldsymbol{\bar{x}_{2}})^{T}\boldsymbol{S}^{-1}(\boldsymbol{\bar{x}_{1}} - \boldsymbol{\bar{x}_{2}})
\end{equation}
where $S^{-1}$ is the inverse of the pooled correlation matrix given by:
\begin{displaymath}
\label{poolcov}
\boldsymbol{S} = \frac{(n_{1} - 1) \boldsymbol{S_{1}} + (n_{2} - 1) \boldsymbol{S_{2}}}{n_{1} + n_{2} - 2}
\end{displaymath}
given the sample estimates for covariance, $\boldsymbol{S_{1}}$ and $\boldsymbol{S_{2}}$ in the two samples.   As before, there is a simple relationship between the test statistic, $T^2$, and the $F$ distribution.

%make theorm flury 404, proof seber, anderson
If $\boldsymbol{x}_{1i}$, $i = 1, \ldots n_{1}$ and $\boldsymbol{x}_{2i}$, $i = 1, \ldots n_{2}$ represent independent samples from two $p$ variate normal distribution with mean vectors $\boldsymbol{\mu}_{1}$ and  $\boldsymbol{\mu}_{2}$ but with common covariance matrix $\boldsymbol{\Sigma}$, provided $\boldsymbol{\Sigma}$ is positive definite and $n > p$, given sample estimators for mean and covariance $\bar{\boldsymbol{x}}$ and $\boldsymbol{S}$ respectively, then:
\begin{displaymath}
F = \frac{(n_{1} + n_{2} - p - 1) T^{2}}{(n_{1} + n_{2} - 2)p}
\end{displaymath}
has an $F$ distribution on $p$ and $(n_{1}+n_{2}-p-1)$ degrees of freedom.   Essentially, we compute the test statistic, and see whether it falls within the $(1-\alpha)$ quantile of the F distribution on those degrees of freedom.   note again that to ensure non-singularity of $\boldsymbol{S}$, we require that $n_{1}+n_{2} > p$.

Whilst we won't use the next formula for computation, it may clarify understanding of this test if we consider the sample estimate of the mean difference $\boldsymbol{d} = \bar{\boldsymbol{x}}_{1} - \bar{\boldsymbol{x}}_{2}$, and the corresponding population distance $\boldsymbol{\delta} = \boldsymbol{\mu}_{1} - \boldsymbol{\mu}_{2}$ we can use the formula:
\begin{displaymath}
F = \left( \frac{n_{1} + n_{2} - p - 1}{p(n_{1} + n_{2} - 2)} \right) \left(\frac{n_{1}n_{2}}{n_{1}+n_{2}} \right) (\boldsymbol{d} - \boldsymbol{\delta})^{T}\boldsymbol{S}^{-1}(\boldsymbol{d} - \boldsymbol{\delta})
\end{displaymath}
to calculate the test statistic.

We are going to consider an example using data from Flea Beetles reported by \cite{Lubischew:1962} and used in [page 307] \cite{Flury:1997}.   It should be noted that in terms of practical computation, methods are based on the QR decomposition will be used, details are given in \cite{Seber:1984}.   However, for the purposes of understanding the principles behind the test, we follow the formula directly.

\singlespacing
\begin{verbatim}
> library(Flury)
> ?flea.beetles
> data(flea.beetles)
\end{verbatim}
\onehalfspacing

It can be seen that there is a factor ``Species'' denoting whether the beetles are from 'oleracea' or 'carduorum'.   There are four numeric variables as follows: 'TG'; Distange of the Transverse Groove to the posterior border of
          the prothorax (microns), 'Elytra'; Length of the Elytra (in units of 0.01mm), 'Second.Antenna'; Length of the second antennal joint (microns) and 'Third.Antenna'; Length of the third antennal joint (microns).   We need to estimate the mean for each sample, and calculate the difference between the two vectors:
\singlespacing
\begin{verbatim}
mu <- by(flea.beetles[,-1], flea.beetles$Species, colMeans)
mudiff <- mu[[1]] - mu[[2]]
p <- dim(flea.beetles)[2] - 1 ## how many variables are we using
\end{verbatim}
\onehalfspacing

The next step is to extract the two covariance matrices:

\singlespacing
\begin{verbatim}
> covmats <- by(flea.beetles[,-1], flea.beetles$Species, cov)
> covmats
\end{verbatim}
\onehalfspacing
and then to estimate the pooled covariance matrix $\boldsymbol{S}$ for the flea beetle data (where N[1] gives $n_{1}$,  N[2] gives $n_{2}$), can be calculated as:

\singlespacing
\begin{verbatim}
> N <- xtabs(~flea.beetles[,1])
> pooledS <- ((N[1]-1) * covmats[[1]] + (N[2]-1) * covmats[[2]]) / (N[1] + N[2] -2)
> pooledS
> Sinv <- solve(pooledS)
> Sinv
                         TG        Elytra Second.Antenna Third.Antenna
TG              0.013257964 -0.0053492256   0.0015134494 -0.0021617878
Elytra         -0.005349226  0.0066679441  -0.0047337699 -0.0005969439
Second.Antenna  0.001513449 -0.0047337699   0.0130490933 -0.0007445297
Third.Antenna  -0.002161788 -0.0005969439  -0.0007445297  0.0060093005
\end{verbatim}
\onehalfspacing

Having calculated the inverse of the pooled correlation matrix we also need the scaling factor $\frac{n_{1} n_{2}}{n_{1} + n_{2}}$.   Hotellings T$^{2}$ is then quite straightforward to calculate:

\singlespacing
\begin{verbatim}
> scaleFact <- (N[1]*N[2]) / (N[1]+N[2])
> Hotellings <-  t(mudiff) %*% Sinv %*% mudiff * scaleFact
> Hotellings
         [,1]
[1,] 133.4873
\end{verbatim}
\onehalfspacing
which is the value of the T$^{2}$ statistic.   We could work with this value directly, but it is more convenient to transform it into something we can compare with the $F$ distribution.
\singlespacing
\begin{verbatim}
test <- ((N[1] + N[2] - p - 1) * Hotellings )/ ((N[1] + N[2] - 2) * p)
test
       [,1]
[1,] 30.666
\end{verbatim}
\onehalfspacing

and we compare this with an $F$ distribution having $p$ and $(n_{1} + n_{2} - p - 1)$ d.f.

And we can check this as follows:
\singlespacing
\begin{verbatim}
> pf(test, p, N[1]+N[2]-p-1,lower.tail = FALSE )
             [,1]
[1,] 3.215324e-11
\end{verbatim}
\onehalfspacing
which gives us the area under the curve from our test statistic ($30.666$) to $\infty$.   Clearly in this case, we have reject H$_{0}$, i.e. there is evidence that the mean vectors, $\bar{\boldsymbol{x}}_{oleracea} = (194.4737, 267.0526, 137.3684, 185.9474), \bar{\boldsymbol{x}}_{carduorum} = (179.55, 290.80, 157.20, 209.25)$,
 for the two species differ.   This is perhaps no surprise if you consider the data.   Figure \ref{lubishew} contains a scatterplot where different symbols have been used in the lower panels for the two species.   However, we do need to consider this in a little more detail.

\begin{figure}
\begin{center}
\includegraphics[width = 0.6\textwidth]{images/lubishew}
\caption{Scatterplot for Lubischew flea data, lower panel has symbols denoting the two species}
\label{lubishew}
\end{center}
\end{figure}

\section{Constant Density Ellipses}
\label{cdellipse}

\cite{Flury:1997} gives an interpretation of constant density ellipses in terms of the Mahalanobis distance which is worth reading.   Essentially, we wish to find a region of squared Mahlanobis distance such that:
\begin{displaymath}
Pr \left( (\boldsymbol{\bar{x}} - \boldsymbol{\mu})^{T} \boldsymbol{S}^{-1} (\boldsymbol{\bar{x}} - \boldsymbol{\mu}) \right) c^{2})
\end{displaymath}
and we can find $c^{2}$ as follows:
\begin{displaymath}
c^{2} = \left( \frac{n-1}{n} \right) \left( \frac{p}{n-p} \right) F_{(1-\alpha), p, (n-p)}
\end{displaymath}
where $F_{(1-\alpha), p, (n-p)}$ is the $(1-\alpha)$ quantile of the $F$ distribution with $p$ and $n-p$ degrees of freedom, $p$ represents the number of variables and $n$ the sample size.

This illustration is based on, but differs from code provided by Marco Bee to accompany and will make much more sense if used in conjunction with that book.   It is worth checking how and why this code differs!   Firstly, we need a function to draw ellipses:

\singlespacing
\begin{verbatim}
ellipse <- function(covmat, centroid, csquare, resolution, plot = TRUE) {
angles <- seq(0, by = (2 * pi)/resolution, length = resolution)
  sd <- covmat[1,2] / sqrt(covmat[1,1] * covmat[2,2])
    projmat <- matrix(0,2,2)
    projmat[1,1] <- sqrt(covmat[1,1] %*% (1+sd)/2)
    projmat[1,2] <- -sqrt(covmat[1,1] %*% (1-sd)/2)
    projmat[2,1] <- sqrt(covmat[2,2] %*% (1+sd)/2)
    projmat[2,2] <- sqrt(covmat[2,2] %*% (1-sd)/2)
circle <- cbind(cos(angles), sin(angles))
ellipse <- t(centroid + sqrt(csquare) * projmat %*% t(circle))
if (plot == TRUE) {lines(ellipse)}
return(ellipse)
}
\end{verbatim}
\onehalfspacing

It is possible to define a function which calculates $c^{2}$ and calls the ellipse routine (I'm not completely convinced this is doing the calculation correctly yet, in particular I'm not sure I'm using the correct tail).

\singlespacing
\begin{verbatim}
function (data, alpha=0.05, resolution=500)
{
xbar <- colMeans(data)
n <- dim(data)[1]
p <- dim(data)[2]
f <- qf(1-alpha, p, n-p)
csquare <- ((n-1)/n) * (p / (n-p)) * f
cat(csquare)
ellipse <- ellipse(cov(data), xbar, csquare, resolution)
}
\end{verbatim}
\onehalfspacing

%# call procedure ellips

%X <- ellips(A, m, const, k)

%# graph the results

For illustrative purposes, we'll create a $n \times 2$ data object from our flea beetles, and plot the confidence ellipse for these.

\singlespacing
\begin{verbatim}
X <- cbind(flea.beetles[,2], flea.beetles[,3])
plot(X)
cdellipse(X, alpha = 0.01)
cdellipse(X, alpha = 0.05)
\end{verbatim}
\onehalfspacing

These can be contrasted with the univariate confidence intervals:

\singlespacing
\begin{verbatim}
abline(v = confint(lm(X[,1]~1)))
abline(h = confint(lm(X[,2]~1)))
\end{verbatim}
\onehalfspacing

This exercise should be repeated with the turtles data!   However, it is possible to illustrate the basic idea with the sibling heads data, where we construct two derivative variables indicating the difference in head breadth and the difference in head width.   These are plotted in figure \ref{cdellipse}, it can be seen that the univariate confidence intervals and the constant density ellipse support different areas of parameter space.   Ignoring the correlation structure in these data could lead to flaws in inference when assessesing parameter uncertainty.

\begin{figure}
\begin{center}
\includegraphics[width = 0.5\textwidth]{images/cdellipse}
\caption{Constant density ellipse for mean vector for difference in head width and breadth, and univariate confidence intervals for the mean of each variable}
\label{cdellipse}
\end{center}
\end{figure}

% X <- sibling.heads
%Y <- cbind((X[, 1] - X[, 3]),(X[, 2] - X[, 4]))
%plot(Y, main = "Contrasts in sibling head measurements", xlab = "Difference in length", ylab = "Difference in breadth", pch = 16)
%X <- Y
%cdellipse(X)
%abline(v = confint(lm(X[,1]~1)))
%abline(h = confint(lm(X[,2]~1)))

%\section{Likelihood ratio tests}
%\label{lrt}

%\section{Profile analysis}
%\label{profile}

%\section{Confidence regions}
%\label{confR}

%\section{Multivariate analysis of variance}
%\label{manova}

%\section{Multivariate regression}
%\label{mvreg}

%\section{Shrinkage of parameters in multivariate regression}
%\label{shrinkage}

%\section{Introduction to the forward search}
%\label{forward}

\section{Multivariate Analysis of Variance}
\label{manova}

As with the univariate situation, t-tests are fine for comparing the means of two groups, but we would have ``multiple comparison'' problems if we tried to compare more than two.   In an analagous way, in the multivariate context we have MANOVA.

First of all we'll enter some data relating to the production of plastic film reported in \cite{Krzanowski:2000}.   Tear, gloss and opacity are measures of the manufactured films.

\singlespacing
\begin{verbatim}
> tear <- c(6.5, 6.2, 5.8, 6.5, 6.5, 6.9, 7.2, 6.9, 6.1, 6.3,
            6.7, 6.6, 7.2, 7.1, 6.8, 7.1, 7.0, 7.2, 7.5, 7.6)
> gloss <- c(9.5, 9.9, 9.6, 9.6, 9.2, 9.1, 10.0, 9.9, 9.5, 9.4,
             9.1, 9.3, 8.3, 8.4, 8.5, 9.2, 8.8, 9.7, 10.1, 9.2)
> opacity <- c(4.4, 6.4, 3.0, 4.1, 0.8, 5.7, 2.0, 3.9, 1.9, 5.7,
               2.8, 4.1, 3.8, 1.6, 3.4, 8.4, 5.2, 6.9, 2.7, 1.9)
Y <- cbind(tear, gloss, opacity)
\end{verbatim}
\onehalfspacing

We now need to put in information on the rate of extrusion, and the amount of additive used (\texttt{gl()} is a command which specifically creates these kind of experimental factors).

\singlespacing
\begin{verbatim}
> rate <- factor(gl(2,10), labels=c("Low", "High"))
> additive <- factor(gl(2, 5, len=20), labels=c("Low", "High"))
\end{verbatim}
\onehalfspacing

There are three conventional ANOVA that could be considered here, but to consider the three responses together we may wish to conduct a MANOVA.   However, we can use \texttt{manova()} to fit the multivariate ANOVA, and use \texttt{summary.aov()} to extract the results of the unvariate analyses.

There are three matrices of interest in MANOVA:

\begin{itemize}
\item Total SSP ($\boldsymbol{T}$)
\item Between-group SSP ($\boldsymbol{B} = \boldsymbol{T} - \boldsymbol{W}$)
\item Within-group SSP ($\boldsymbol{W}$)
\end{itemize}

Wilk's Lambda is the ratio $\frac{|\boldsymbol{W}|}{|\boldsymbol{T}|}$

\singlespacing
\begin{verbatim}
>      fit <- manova(Y ~ rate * additive)
>      summary.aov(fit)           # univariate ANOVA tables
 Response tear :
              Df  Sum Sq Mean Sq F value   Pr(>F)
rate           1 1.74050 1.74050 15.7868 0.001092 **
additive       1 0.76050 0.76050  6.8980 0.018330 *
rate:additive  1 0.00050 0.00050  0.0045 0.947143
Residuals     16 1.76400 0.11025
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

 Response gloss :
              Df  Sum Sq Mean Sq F value  Pr(>F)
rate           1 1.30050 1.30050  7.9178 0.01248 *
additive       1 0.61250 0.61250  3.7291 0.07139 .
rate:additive  1 0.54450 0.54450  3.3151 0.08740 .
Residuals     16 2.62800 0.16425
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

 Response opacity :
              Df Sum Sq Mean Sq F value Pr(>F)
rate           1  0.421   0.421  0.1036 0.7517
additive       1  4.901   4.901  1.2077 0.2881
rate:additive  1  3.961   3.961  0.9760 0.3379
Residuals     16 64.924   4.058
\end{verbatim}
\onehalfspacing

A call to \texttt{summary()} will give the MANOVA table.   If you leave out the \texttt{text = \"Wilks\"} call you will get a default Pillai-Bartlett statistic.

\singlespacing
\begin{verbatim}
>      summary(fit, test="Wilks") # ANOVA table of Wilks' lambda
              Df  Wilks approx F num Df den Df   Pr(>F)
rate           1 0.3819   7.5543      3     14 0.003034 **
additive       1 0.5230   4.2556      3     14 0.024745 *
rate:additive  1 0.7771   1.3385      3     14 0.301782
Residuals     16
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}
\onehalfspacing

As with Hotellings T$^{2}$, Wilk's Lambda has to be converted into an F statistic, a calculation that has been done by the software.   It is possible to approximate this by a $\chi^{2}_{pb df}$ with $W = -\left( w - \frac{p - b + 1}{2}\right)log \Lambda$ where p = number of variables, w = residual degrees of freedom (16), b = number of hypotheses degrees of freedom (1).

In any case, the interaction term is not significant.   We can fit the model without interactions, which as anticipated suggests that both additive and extrusion rate have an effect on the outcome measures.   We will use \texttt{by()} to examine the various group means.

\singlespacing
\begin{verbatim}
> fit <- manova(Y ~ rate + additive)
> summary(fit, test = "Wilks")
          Df  Wilks approx F num Df den Df   Pr(>F)
rate       1 0.3868   7.9253      3     15 0.002120 **
additive   1 0.5538   4.0279      3     15 0.027533 *
Residuals 17
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
> by(Y, rate, mean) ## group means according to extrusion rate
INDICES: Low
   tear   gloss opacity
   6.49    9.57    3.79
------------------------------------------------------------
INDICES: High
   tear   gloss opacity
   7.08    9.06    4.08
> by(Y, additive, mean) ## group means according to additive
INDICES: Low
   tear   gloss opacity
   6.59    9.14    3.44
------------------------------------------------------------
INDICES: High
   tear   gloss opacity
   6.98    9.49    4.43
>
> by(Y, list(rate,additive), mean) ## group means by both.
: Low
: Low
   tear   gloss opacity
   6.30    9.56    3.74
------------------------------------------------------------
: High
: Low
   tear   gloss opacity
   6.88    8.72    3.14
------------------------------------------------------------
: Low
: High
   tear   gloss opacity
   6.68    9.58    3.84
------------------------------------------------------------
: High
: High
   tear   gloss opacity
   7.28    9.40    5.02
\end{verbatim}
\onehalfspacing

High levels of extrusion rate lead to higher levels of tear and opacity but lower levels of gloss.   High levels of addtive lead to higher levels of tear, gloss and opacity.

%%% Local Variables: ***
%%% mode:latex ***
%%% TeX-master: "../book.tex"  ***
%%% End: ***


%% Added this section, to ha ve a list of algorithms in lwarp
\chapter{CENG242Cheat}
\todo{Remember to add more algorithms later}{Future Work}

\begin{algorithm}
\caption{Euclid’s algorithm}\label{euclid}
\begin{algorithmic}[1]
\Procedure{Euclid}{$a,b$}\Comment{The g.c.d. of a and b}
\State $r\gets a\bmod b$
\While{$r\not=0$}\Comment{We have the answer if r is 0}
\State $a\gets b$
\State $b\gets r$
\State $r\gets a\bmod b$
\EndWhile\label{euclidendwhile}
\State \textbf{return} $b$\Comment{The gcd is b}
\EndProcedure
\end{algorithmic}
\end{algorithm}

 \begin{algorithm}
	\caption{Merge Sort}
	\begin{algorithmic}[1]
		\Function{Merge}{$A,p,q,r$}\Comment{Where A - array, p - left, q - middle, r - right}

		\State ${n_1} = q - p + 1$
		\State ${n_2} = r - q$
		\State Let $L[1 \ldots {n_1} + 1]$ and $R[1 \ldots {n_2} + 1]$ be new arrays
		\For{$i = 1$ to ${n_1}$}
		\State $L[i] = A[p + i - 1]$
		\EndFor
		\For{$j = 1$ to ${n_2}$}
		\State $R[i] = A[q + j]$
		\EndFor
		\State $L[{n_1} + 1] =  \infty $
		\State $R[{n_2} + 1] =  \infty $
		\State $i = 1$
		\State $j = 1$
		\For{$k = p$ to $r$}
		\If {$L[i] < R[j]$}
		\State $A[k] = L[i]$
		\State $i = i + 1$
		\ElsIf {$L[i] > R[j]$}
		\State $A[k] = R[j]$
		\State $j = j + 1$
		\Else
		\State $A[k] = - \infty$ \Comment{We mark the duplicates with the largest negative integer}
		\State $j = j + 1$
		\EndIf
		\EndFor
		\EndFunction

	\end{algorithmic}
\end{algorithm}

\section{Basic Graph Definitions}
\begin{itemize}
	\item A graph $G = (V,E)$ consists of a finite set of {\em vertices} $V$
	and a finite set of {\em edges} $E$.
	\begin{itemize}
		\item {\em Directed graphs}: $E$ is a set of ordered pairs of vertices
		$(u,v)$ where $u,v \in V$ \\
		% \centerline{\epsfig{file=directed_example.eps,height=3.5cm}}
		\item {\em Undirected graph}: $E$ is a set of unordered pairs of
		vertices $\{u,v\}$ where $u,v \in V$ \\ \\
		% \centerline{\epsfig{file=undirected_example.eps,height=3cm}}
	\end{itemize}
	\item Edge $(u,v)$ is {\em incident} to $u$ and $v$
	\item {\em Degree} of vertex in undirected graph is the number of edges
	incident to it.
	\item {\em In (out) degree} of a vertex in directed graph is the number of
	edges entering (leaving) it.
	\item A {\em path} from $u_1$ to $u_2$ is a sequence of vertices
	$<u_1$=$v_0,v_1,v_2, \cdots , v_k$=$u_2>$ such that $(v_i,v_{i+1}) \in E$
	(or $\{v_i,v_{i+1}\} \in E$)
	\begin{itemize}
		\item We say that $u_2$ is {\em reachable} from $u_1$
		\item The {\em length} of the path is $k$
		\item It is a {\em cycle } if $v_0 = v_k$
	\end{itemize}

	\item An undirected graph is {\em connected /} if every pair of vertices are
	connected by a path
	\begin{itemize}
		\item The {\em connected components } are the equivalence classes of
		the vertices under the ``reachability'' relation. (All connected
		pair of vertices are in the same connected component).
	\end{itemize}
	\item A directed graph is {\em strongly connected } if every pair of
	vertices are reachable from each other
	\begin{itemize}
		\item The {\em strongly connected components } are the equivalence
		classes of the vertices under the ``mutual reachability'' relation.
	\end{itemize}

	\vspace*{\baselineskip}

	\item Graphs appear all over the place in all kinds of applications, e.g:
	\begin{itemize}
		\item Trees $(\vert E \vert = \vert V \vert - 1)$
		\item Connectivity/dependencies (house building plans, WWW-page
		connections = internet graph)
	\end{itemize}
	\item Often the edges $(u,v)$ in a graph have weights $w(u,v)$, e.g.
	\begin{itemize}
		\item Road networks (distances)
		\item Cable networks (capacity)
	\end{itemize}
\end{itemize}

\subsection{Representation}
\begin{itemize}
	\item {\em Adjacency-list} representation:
	\begin{itemize}
		\item Array of $\vert V \vert$ list of edges incident to each
		vertex. \\

		Examples: \\
		\item Note: For undirected graphs, every edge is stored twice.
		\item If graph is weighted, a weight is stored with each edge.
	\end{itemize}

	\item {\em Adjacency-matrix} representation:
	\begin{itemize}
		\item $\vert V \vert \times \vert V \vert$ matrix $A$ where
		\[ a_{ij} = \left\{ \begin{array}{ll}
		1        & \mbox{ if $(i,j) \in E$} \\
		0        & \mbox{ otherwise } \\
		\end{array}
		\right. \]

		Examples: \\
		\item Note: For undirected graphs, the adjacency matrix is
		symmetric along the main diagonal ($A^T = A$).
		\item If graph is weighted, weights are stored instead of one's.
	\end{itemize}
	\item Comparison of matrix and list representation: \\
	\begin{itemize}
		\item[]
		\begin{tabular}{l|l}
			Adjacency list & Adjacency matrix \\ \hline
			$O(\vert V \vert + \vert E \vert)$ space & $O(\vert V \vert ^2)$
			space \\
			Good if graph {\em sparse} $(\vert E \vert << \vert V
			\vert ^2)$ & Good if graph {\em dense} $(\vert E \vert \approx
			\vert V \vert ^2)$ \\
			No quick access to $(u,v)$ & $O(1)$ access to $(u,v)$ \\
		\end{tabular}
	\end{itemize}
	\item We will use adjacency list representation unless stated otherwise
	($O(|V|+|E|)$ space).
\end{itemize}

\section{Graph traversal}
\begin{itemize}
	\item There are two standard (and simple) ways of traversing all
	vertices/edges in a graph in a systematic way
	\begin{itemize}
		\item Breadth-first
		\item Depth-first
	\end{itemize}
	\item We can use them in many fundamental algorithms, e.g finding cycles,
	connected components, $\dots$
\end{itemize}

\subsection{Breadth-first search (BFS)}
\begin{itemize}
	\item Main idea:
	\begin{itemize}
		\item Start at some source vertex $s$ and visit,
		\item All vertices at distance 1,
		\item Followed by all vertices at distance 2,
		\item Followed by all vertices at distance 3,

		$\vdots$
	\end{itemize}
	\item BFS corresponds to computing {\em shortest path} distance (number of
	edges) from $s$ to all other vertices.
	\item To control progress of our BFS algorithm, we think about {\em coloring}
	each vertex
	\begin{itemize}
		\item {\em White } before we start,
		\item {\em Gray } after we visit the vertex but before we have
		visited all its adjacent vertices,
		\item {\em Black } after we have visited the vertex and all its
		adjacent vertices (all adjacent vertices are gray).
	\end{itemize}
	\item We use a queue $Q$ to hold all gray vertices---vertices we have seen
	but  are still not done with.
	\item We remember from which vertex a given vertex $v$ is colored gray
	-- i.e. the node that discovered $v$ first; this is called
	parent[$v$].
	\item Algorithm: \\ \\
	\fbox{
		\parbox{10cm}{
			{\sc BFS($s$) }
			\begin{itemize}
				\item[] color[$s$] = gray
				\item[] $d[s] = 0$
				\item[] ENQUEUE($Q,s$)
				\item[] WHILE $Q$ not empty DO
				\begin{itemize}
					\item[] DEQUEUE($Q,u$)
					\item[] FOR $(u,v)\in E$ DO
					\begin{itemize}
						\item[] IF color[$v$] = white THEN
						\item[] ~~~~color[$v$] = gray
						\item[] ~~~~$d[v] = d[u] + 1$
						\item[] ~~~~parent[$v$] = u
						\item[] ~~~~ENQUEUE($Q,v$)
						\item[] FI
						\item[] color[$u$] = black
					\end{itemize}
				\end{itemize}
				\item[] OD
			\end{itemize}
	}}
	\vspace*{\baselineskip}

	\item Algorithm runs in $O(\vert V \vert + \vert E \vert)$ time

	\item Example (for directed graph): \\
	\item Note:
	\begin{itemize}
		\item parent[$v$] forms a tree; {\em BFS-tree}.
		\item $d[v]$ contains length of shortest path from $s$ to
		$v$. (Prove by induction)
		\item We can use parent[$v$] to find the shortest path from $s$ to a
		given vertex.
		%       \item We can use algorithm to detect cycles; just check if gray
		%       node is ever met.
	\end{itemize}
	\item If graph is not connected we have to try to start the traversal
	at all nodes. \\

	\fbox{
		\parbox{8cm}{
			FOR each vertex $u \in V $ DO
			\begin{itemize}
				\item[] IF color[$u$] = white THEN BFS($u$)
			\end{itemize}
			OD
	}}

	\begin{itemize}
		\item Note: We can use algorithm to compute connected components in
		$O(|V|+|E|)$ time.
	\end{itemize}
\end{itemize}

\subsection{Depth-first search (DFS)}
\begin{itemize}
	\item If we use stack instead of queue $Q$ we get another traversal order;
	depth-first
	\begin{itemize}
		\item We go ``as deep as possible'',
		\item Go back until we find unexplored adjacent vertex,
		\item Go as deep as possible,

		$\vdots$
	\end{itemize}
	\item Often we are interested in ``start time'' and ``finish time'' of
	vertex $u$
	\begin{itemize}
		\item {\em Start time\/} (d[$u$]): indicates at what ``time'' vertex
		is first visited.
		\item {\em Finish time\/} (f[$u$]): indicates at what ``time'' all
		adjacent vertices have been visited.
	\end{itemize}
	\item We can write DFS iteratively using the same algorithm as for BFS
	but with a STACK instead of a QUEUE, or, we can write a recursive DFS
	procedure
	\begin{itemize}
		\item We will color a vertex gray when we first meet it and black
		when we finish processing all adjacent vertices.
	\end{itemize}
	\item Algorithm: \\

	\fbox{
		\parbox{8cm}{
			{\sc DFS($u$) }
			\begin{itemize}
				\item[] color[$u$] = gray
				\item[] $d[u]$ = time
				\item[] time = time + 1
				\item[] FOR $(u,v)\in E$ DO
				\begin{itemize}
					\item[] IF color[$v$] = white THEN
					\begin{itemize}
						\item[] parent[$v$] = $u$
						\item[] {\sc DFS(v)}
					\end{itemize}
					\item[] FI
				\end{itemize}
				\item[] OD
				\item[] color[$u$] = black
				\item[] $f[u]$ = time
				\item[] time = time + 1
			\end{itemize}
	}}

	\item Algorithm runs in $O(\vert V \vert + \vert E \vert)$ time
	\begin{itemize}
		\item As before we can extend algorithm to unconnected graphs and
		we can use it to detect cycles in $O(|V|+|E|)$ time.
	\end{itemize}

	\item As previously parent[$v$] forms a tree; {\em DFS-tree}
	\begin{itemize}
		\item Note: If $u$ is descendent of $v$ in DFS-tree  then $d[v] <
		d[u] < f[u] < f[v]$
	\end{itemize}
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Topological sorting}
\begin{itemize}
	\item Definition: Topological sorting of {\em directed acyclic graph}
	$G=(V,E)$ is a linear ordering of vertices $V$ such that $(u,v) \in E
	\Rightarrow u$ appear before $v$ in ordering.
	\item Topological ordering can be used in scheduling:
	\begin{itemize}
		\item Example: Dressing (arrow implies ``must come before'') \\ \\
		% \centerline{\epsfig{file=top_sort_example.eps,height=5.5cm}} \\

		We want to compute order in which to get dressed. One possibility: \\

		\vspace*{\baselineskip}

		The given order is one possible topological order.
	\end{itemize}
	\item Algorithm: Topological order just reverse DFS finish time ($
	\Rightarrow O(\vert V \vert + \vert E \vert)$ running time).
	\item Correctness: $(u,v) \in E \Leftrightarrow f(v) < f(u)$
	\begin{itemize}
		\item Proof: When $(u,v)$ is explored by DFS algorithm, $v$ must be
		white or black (gray $\Rightarrow$ cycle).
		\begin{itemize}
			\item $v$ white: $v$ visited and finished before $u$ is
			finished $\Rightarrow f(v) < f(u)$
			\item $v$ black: $v$ already finished $\Rightarrow f(v) <
			f(u)$
		\end{itemize}
	\end{itemize}
	\item Alternative algorithm: Count in-degree of each vertex and repeatedly
	number and remove in-degree 0 vertex and its outgoing edges: Homework.

\end{itemize}


%---------------------------Maximum Edge Ratio---------------------------
\section{Maximum Edge Ratio}

Given principal axes with lengths $L_f$ and $L_g$,
the aspect ratio is defined as the largest ratio of those lengths
\[
   A_{fg} = \max\left\{ \frac{L_f}{L_g}, \frac{L_g}{L_f} \right\}.
\]
Since a hexahedron has 3 principal axes, we take the largest of all pairwise combinations of axes.
\[
  q  = \max\left\{
    A_{\normvec{ X_1 }\normvec{ X_2 }},
    A_{\normvec{ X_1 }\normvec{ X_3 }},
    A_{\normvec{ X_2 }\normvec{ X_3 }}
  \right\}
\]

Note that if $\normvec{X_1}$ or $\normvec{X_2}$ or $\normvec{X_3} < DBL\_MIN$, we set $q = DBL\_MAX$.

\hexmetrictable{maximum edge ratio}%
{$1$}%                                      Dimension
{$[1,1.3]$}%                                Acceptable range
{$[1,DBL\_MAX]$}%                           Normal range
{$[1,DBL\_MAX]$}%                           Full range
{$1$}%                                      Unit square
{Adapted from \cite{tf:89}}%                Citation
{v\_hex\_max\_edge\_ratio}%                      Verdict function name


In this section, we will closely follow\ \cite{pitfalls}

\subsubsection{Introduction}

Normally, samples in machine learning are of huge sample sizes, for which most MCMC algorithms are not designed to process.
As a result of the computational cost, several new approaches were proposed recently, Stochastic Gradient Langevin Dynamics (SGLD) is a popular one.
SGLD is based on the Langevin Monte Carlo (LMC)
LMC – a discretization of a continuous-time process, it requires to compute the gradient of the log-posterior at the current fit of the parameter and avoid the accept/reject step.
SGLD – use unbiased estimator of the gradient of log-posterior based on subsampling, suitable for samples of huge size.

\subsubsection{Governing Equation}
Recall the following equations:
Euler discretization of the Langevin SDE:
$$\theta_{n+1} = \theta_n - h \nabla U(\theta_n)+\sqrt{2h} Z_{n+1}$$ where $h > 0$ is a constant step size and $(Z_n)_{n\geq1}$ is a sequence of i.i.d standard d - dimensional Guassian vectors.
To reduce the costs of the algorithms, we will switch to SGLD, for which we will replace $\nabla U$ with an unbiased estimate $\nabla U_0+(\frac{N}{p}) \sum_{i\in S}\nabla U_i$, where S is a minibatch of \{1,...., N\} with replacement of size p. Our iterations were then updated as
$$\theta_{n+1} = \theta_{n}-h \Bigg (\nabla U_0(\theta_n)+\frac{N}{p}\sum_{i\in S_{n+1}}\nabla U_i(\theta_n)\Bigg )+\sqrt{2h}Z_{n+1}$$
Stochastic Gradient Descent (SGD) is characterised by the same recursion as SGLD without the Gaussian noise, (the last term):

$$\theta_{n+1} = \theta_{n}-h \Bigg (\nabla U_0(\theta_n)+\frac{N}{p}\sum_{i\in S_{n+1}}\nabla U_i(\theta_n)\Bigg)$$
\textbf{Analysis in Wasserstein distance}
\subsubsection{Definitions and Notations in Markov Chain theory}
Recall the following definitions:
$\mathcal{P}_2(\mathbb{R}^d)$ the set of probablity measures with finite second momet.\\
$\mathcal{B}(\mathbb{R}^d)$ the Borel $\sigma$ - algebra of $\mathbb{R}^d$.\\
For $\lambda, \nu \in \mathcal{P}_2(\mathbb{R}^d)$, we define the Wasserstein distance by

$$W_2(\lambda, \nu) =\inf_{\xi \in \Pi(\lambda, \nu)}(\int_{\mathbb{\mathbb{R}^d \times \mathbb{R}^d}}||\theta-\vartheta)||^2 \xi(d\theta, d\vartheta))^{\frac{1}{2}}$$
where, $\Pi(\lambda, \nu)$ is the set of probablity measures $\xi$ on $\mathcal{B}(\mathbb{R}^d)\otimes\mathcal{B}(\mathbb{R}^d)$ satisfying for all $A \in \mathcal{B}(\mathbb{R}^d), \xi(A \times \mathbb{R}^d)= \lambda(A)$ and $\xi (\mathbb{R}^d \times A) = \nu(A)$.\\
For any probablity measure $\lambda$ on $\mathcal{B}(\mathbb{R}^d)$, we define $\lambda R$ for all $A \in \mathcal{B}(\mathbb{R}^d)$ by $\lambda R(A) = \int_{\mathbb{R}^d}\lambda(d\theta)R(\theta, A)$.\\
For all $k\in \mathbb{N}*$, we define the Markov kernel $R^k$ recursively by $R^1 = R$ and for all $\theta \in \mathbb{R}^d$ and $A \in \mathcal{B}(\mathbb{R}^d)$, $R^{k+1}(\theta, A) = \int_{\mathbb{R}^d}  R^k(\theta, d\vartheta)R(\vartheta, A).$\\
A probablity measure $\bar{\pi}$ is invariant for R if $\bar{\pi}R = \bar{\pi}$.\\
Our algorithms LMC, SGLD, SGD and SGLDFP algorithms are homogeneous Markov chains with Markov kernels denoted $R_{LMC}, R_{SGLD}, R_{SGD}$ and $R_{FP}$.

\subsubsection{Results}
Recall our assumptions:
\begin{enumerate}[label={\bf H\arabic*}]
    \item For all $i \in \{0,...,N\}, \ U_i$ is four times continuously differentiable and for all $j \in \{2, 3, 4\},$$sup_{\theta \in \mathbb{R}^d}||D^j U_i(\theta)||\leq \Tilde{L}.$ In particular for all $i\in \{0, ..., N\}$, $U_i$ $\Tilde{L}$-gradient Lipschitz, i.e. for all $\theta_1, \theta_2\in \mathbb{R}^d$, $||\nabla U_i(\theta_1) - \nabla U_i (\theta_2)|| \leq \Tilde{L}||\theta_1 - \theta_2||.$
    \item U is m-strongly convex, i.e. for all $\theta_1, \theta_2 \in \mathbb{R}^d$,$\left< \nabla U(\theta_1) - \nabla U(\theta_2), \theta_1 - \theta_2\right>\geq m ||\theta_1 - \theta_2||^2.$
    \item For all $i\in\{0,...,N\}$, $U_i$ is convex.
\end{enumerate}
For the below Lemma, Theorem and Corollary, we assume H1, H2 and H3.
\begin{lemma}
For any step size $h \in (0, \frac{2}{L})$, $R_{SGLD}$(respectively $R_{LMC}, R_{SGD}, R_{FP}$) has a unique invariant measure $\pi_{SGLD}\in \mathcal{P}_2(\mathbb{R}^d)$(respectively $\pi_{LMC}, \pi_{SGD}, \pi_{FP}$). In addition, for all $h \in (0, \frac{1}{L}], \theta\in \mathbb{R}^d and k\in\mathbb{N}$,
$$W_2^2(R_{SGLD}^k(\theta, \cdot), \pi_{SGLD})\leq(1-mh)^k\int_{\mathbb{R}^d}||\theta-\vartheta||^2\pi_{SGLD}(d\vartheta)$$
same inequality holds for LMC, SGD and SGLDFP.
\end{lemma}

\begin{theorem}
For all $h\in(0,\frac{1}{L}], \lambda, \nu\in \mathcal{P}_2(\mathbb{R}^d)\ and \ n\in\mathbb{N}$, we have the following upper- bounds in Wasserstein distance between
\begin{enumerate}
	\item
	LMC and SGLDFP,

\begin{dmath}
W_2^2(\lambda R_{LMC}^n, \nu R_{FP}^n)\leq(1-mh)^nW_2^2(\lambda, \nu) + \frac{2L^2h d}{pm^2}+\frac{L^2h^2}{p}n(1-mh)^{n-1}\int_{\mathbb{R}^d}||\vartheta-\theta*||^2 \mu(d\vartheta)
\end{dmath},
	\item
	the Langevin diffusion and LMC,
\begin{dmath}
W_2^2(\lambda R_{LMC}^n, \mu P_{nh})\leq2(1-\frac{mLh}{m+L})^nW_2^2(\lambda, \mu)+dh\frac{m+L}{2m}(3+\frac{L}{m})(\frac{13}{6}+\frac{L}{m})\\+ne^{-(\frac{m}{2})h(n-1)}L^3h^3(1+\frac{m+L}{2m})\int_{\mathbb{R}^d}||\vartheta - \theta*||^2 \mu(d\vartheta)
\end{dmath},
	\item
	SGLD and SGD
	\begin{dmath}
	W_2^2(\lambda R_{SGLD}^n, \mu R_{SGD}^n)\leq (1-mh)^n W_2^2(\lambda, \mu)+\frac{(2d)}{m}.
	\end{dmath}
\end{enumerate}
\end{theorem}
Proof omitted.
\begin{cor}
Set $h - \frac{\eta}{N} \ with \ \eta \in (0, \frac{1}{(2L)}]$ and assume that $lim \inf_{N \to \infty}mN^{-1}>0$. Then
\begin{enumerate}
\item
for all $n \in N$, we get $W_2(R_{LMC}^n(\theta*, \cdot), R_{FP}^{n}(\theta*, \cdot)) = \sqrt{d\eta}\mathcal{O}(N^{-\frac{1}{2}})$ and $W_2(\pi_{LMC}, \pi_{FP}) = \sqrt{d\eta}\mathcal{O}(N^{-\frac{1}{2}})$.
\item
for all $n\in \mathbb{N}, we get W_2(R_{SGLD}^{n}(\theta*, \cdot), R^n_{SGD}(\theta*, \cdot)) = \sqrt{d}\mathcal{O}(N^{-\frac{1}{2}})$, and $W_2(\pi_{SGLD}, \pi_{SGD}) = \sqrt{d}\mathcal{O}(N^{-\frac{1}{2}})$.
\end{enumerate}
\end{cor}


\chapter{COORDINATE SYSTEMS}
A co-ordinate system is a mathematical tool to describe the position of an object. It  is used extensively in describing a physical problem or a situation . The information about problem of interest is depicted in 'co-ordinate', it is the set of two or more numbers that specifies the position of a point, line, or other geometric figure in relation to some reference system.\newline There are mainly four kinds of coordinate systems. Cartesian (Rectangular) coordinate system, Plane polar coordinate system, Cylindrical polar coordinate system and Spherical polar coordinate system.The choice of the coordinate system is based on the symmetry problem we are interested in.
\section{Cartesian Coordinate System}
The most common coordinate systems that we are likely to encounter are Cartesian coordinate systems. These are used where the plane, surface or space can be described in flat, rectangular dimensions (like a box, or a square ).
\\
\begin{minipage}{0.50\textwidth}
	A 3-D Cartesian coordinate consist of a set of 3 mutually perpendicular axes, which intersect at a common point, the origin O. In a three dimensional Cartesian coordinate system, the position of an object is specified by using the three coordinates, ${x}, {y},$ and ${z}$. The range of the coordinates are,
	\begin{align*}
	-\infty \leqslant x \leqslant \infty \\
	-\infty \leqslant y \leqslant \infty \\
	-\infty \leqslant z \leqslant \infty
	\end{align*}
\end{minipage}\hfill
\begin{minipage}{0.35\textwidth}
	\begin{figure}[H]
		\begin{center}
			\includegraphics[width=6cm,height=5cm]{cartesian }
		\end{center}
		\caption{3-D Cartesian coordinate system}
	\end{figure}
\end{minipage}
\subsection{Position Vector and Unit Vectors}
\textbf{Position vector :} A position vector defines the position of point with respect to origin in a plane or space.
\\\\\textbf{Unit vector \hspace{0.55cm}:} A vector of  unit length is called as  a unit vector. It's also called the direction vector since it specifies the direction of a vector. In general  the unit vector in the direction of $ x $ is represented as $\hat{e_{x}}$, in the direction of y,  $\hat{e_{y}}$ and in the direction of z,  $\hat{e_{z}}$.
\begin{align*}
\intertext{Then the position vector in cartesian coordinate system in general is represented by,}
\vec{r}&=x\hat{e_{x}}+y\hat{e_{y}}+z\hat{e_{z}}
\intertext{In conventional form the unit vectors are defined as $\hat{e_{x}}$ ,$\hat{e_{y}}$ and $\hat{e_{z}}$ in $ x, y \ \text{and } z$ directions respectively. Then the position vector becomes,}
\vec{r}&=x\hat{e_{x}}+y\hat{e_{y}}+z\hat{e_{z}}
\end{align*}

\subsection{Line, Area and Volume Elements}
\begin{figure}[h]
	\begin{minipage}{0.30\textwidth}
		\includegraphics[scale=0.4]{cs-05-crop.pdf}
		\caption{line element}
		\label{line element}
	\end{minipage}
	\begin{minipage}{0.35\textwidth}
		\includegraphics[scale=0.4]{cs-03-crop.pdf}
		\caption{Area element}
	\end{minipage}
	\begin{minipage}{0.30\textwidth}
		\includegraphics[scale=0.4]{cs-06-crop.pdf}
		\caption{Volume element}
	\end{minipage}
\end{figure}
\begin{itemize}
	\item{\textbf{Line element}}\\
	Consider a small infinitesimal displacement $dl$  between two points $p_{1}$ and $p_{2}$ as in the fig.\ref{line element} then it  is defined as, \begin{align*}
	d {\vec{l}}&=d x \hat{e_{x}}+d y \hat{e_{y}}+d z \hat{e_{z}}
	\end{align*}
	\item{\textbf{Area element}}\\ The infinitesimal area element in a cartesian coordinate system is defined as ,
	\begin{align*}
	d\vec{A}&=dx dy \hat{e_{z}}\qquad \text{Area element in the x-y plane.}\\
	\intertext{Similiarly, }\
	d\vec{A}&=dx dz \hat{e_{y}} \qquad \text{Area element in the x-z plane.}\\
	\text{And}\quad  d\vec{A}&=dy dz  \hat{e_{x}} \qquad \text{Area element in the y-z plane.}
	\end{align*}
	\item{\textbf{Volume element}}\\ The infinitesimal volume element in a cartesian coordinate system is defined as,
	\begin{equation*}
	d{V}=dx dy dz
	\end{equation*}
	Remember that the infinitesimal volume element is a scalar quantity.

\end{itemize}
\subsection{Kinematic Vectors in Cartesian Coordinate System}
\begin{itemize}
	\item \textbf{Velocity}
	\begin{align*}
	\intertext{We know that velocity is the rate of change of displacement. So, differentiating the displacement vector we get,}
	\text { Velocity }, \vec{v}&=\frac{d \vec{r}}{d t}\\
	\text{Velocity in cartesian coordinate system:} \\
	\vec{v}&=\frac{d \vec{r}}{d t}=\frac{d}{d t}(x \hat{e_{x}}+y \hat{e_{y}}+z \hat{e_{z}})\\&=\dot{x} \hat{e_{x}}+\dot{y} \hat{e_{y}}+\dot{z} \hat{e_{z}}\\
	\text{Where,} \ \dot{x}&=\frac{dx}{dt}\quad ;\quad \dot{y}=\frac{dy}{dt}\quad ;\quad\dot{z}=\frac{dz}{dt}
	\end{align*}

	\item \textbf{Acceleration}
	\begin{align*}
	\intertext{Acceleration is the rate of change of velocity. differentiating velocity vector with respect to time we get acceleration .}
	\text{Accelaration} ,\ \vec{a}&=\frac{d \vec{v}}{d t}\\
	\text{Accelaration in cartesian coordinate system:}\\ \vec{a}&=\frac{d \vec{v}}{d t}=\frac{d \dot{\vec{x}}}{d t}+\frac{d \dot{\vec{y}}}{d t}+\frac{d \dot{\vec{z}}}{d t}\\&=\ddot{x} \hat{e_{x}}+\ddot{y} \hat{e_{y}}+\ddot{z} \hat{e_{z}}
	\end{align*}

\end{itemize}

%.........................
\section{Polar Co-Ordinate System}
The polar coordinate system is just an another way of describing the locations of points in a plane. We need to use polar coordinates , where there is circular  symmetry in the form of a physical object, or some kind of circular or orbital (oscillatory) motion. A polar equation describes a relationship between $\rho$ and $\phi$ on a polar grid. One of the main  application of polar coordinates is in orbital motions of celestial objects.  Instead of giving x and y coordinates, we’ll describe the location of a point by,\\
\begin{minipage}{0.45\textwidth}
	\begin{itemize}
		\item {$\rho$= Distance  from origin}
		\item{$\phi$= Angle between the ray from the origin to the point and the horizontal
			axis.}
		\begin{align*}
		x&=\rho \cos \phi
		\quad;\quad y=\rho\sin \phi
		\intertext{ $\rho$ and $\phi$ varies as,}
		0 &\leq \rho \leq \infty\\ 0 &\leq \phi\leq 2 \pi
		\end{align*}
	\end{itemize}
\end{minipage}\hfill
\begin{minipage}{0.45\textwidth}
	\begin{figure}[H]
		\centering
		\includegraphics[width=6cm,height=5cm]{polar}
		\caption{ Polar Coordinate system}
		\label{ Polar Coordinate system}
	\end{figure}
\end{minipage}\\\\
In the plane polar coordinates the condition $\rho=\text{Constant}$ , provides circles, where as $\phi=\text{Constant }$ provides radial line.
Since we are dealing with free vectors, we can translate the polar reference frame for a given point $(\rho, \phi),$ to cartesian, by  a standard change of basis procedure.
\\So that the inverse relationships to convert rectangular coordinates to polar coordinates, we can  use,

\begin{align*}
x&= \rho \cos \phi\\
y&= \rho \sin \phi\\
\rho &=\sqrt{x^{2}+y^{2}} \\
\phi &=\tan ^{-1} \frac{y}{x}
\end{align*}

\subsection{Position Vector and Unit Vector}
\begin{align*}
\intertext{The position vector $\vec{r}$ in polar coordinate is given by :}
\vec{r}&=\rho\hat{{e_{\rho}}}\\
\intertext{In the polar coordinate system we have $\hat{e_{\rho}}$ in the increasing direction of $\rho$ and  $\hat{e_{\phi}}$ in the increasing direction of $\phi$. Since the direction of the unit vectors depend on the position of the point both $\hat{e_{\rho}}$ and $\hat{e_{\phi}}$ are a function of $\phi$,}
\intertext{The unit vectors are defined as ,}  \hat{e_{\rho}}&=\frac{\partial \vec{r} / \partial \rho}{|\partial \vec{r} / \partial \rho|}=\cos \phi \hat{e_{x}}+\sin \phi \hat{e_{y}}\\ \hat{e_{\phi}}&=\frac{\partial \vec{r} / \partial \phi}{|\partial \vec{r} / \partial \phi|}=-\sin \phi \hat{e_{x}}+\cos \phi \hat{e_{y}}
\intertext{In matrix notation,}
\left(\begin{array}{c}
\hat{e_{\rho}} \\
\hat{e_{\phi}}
\end{array}\right)&=\left(\begin{array}{cc}
\cos \phi & \sin \phi \\
-\sin \phi & \cos \phi
\end{array}\right)\left(\begin{array}{c}
\hat{e_{x}} \\
\hat{e_{y}}
\end{array}\right)\\\\
\hat{e_{\rho}} \times \hat{e_{\phi}}&=\hat{e_{z}}
\end{align*}
Since the unit vectors transforms this way components of any general vector $A$ should transform in the same way. And we have the transformation matrix,
\begin{align*}
\left(\begin{array}{c}
A_{\rho} \\
A_{\phi}
\end{array}\right)&=\left(\begin{array}{cc}
\cos \phi & \sin \phi \\
-\sin \phi & \cos \phi
\end{array}\right)\left(\begin{array}{c}
A_{x} \\
A_{y}
\end{array}\right) \\\intertext { And } \quad\left(\begin{array}{c}
A_{x} \\
A_{y}
\end{array}\right)&=\left(\begin{array}{cc}
\cos \phi & -\sin \phi \\
\sin \phi & \cos \phi
\end{array}\right)\left(\begin{array}{c}
A_{\rho} \\
A_{\phi}
\end{array}\right)
\end{align*}

\subsection{Line, Area and Volume Elements}
\begin{itemize}
	\item \textbf{Line element}
	\begin{align*}
	\vec{dl}=d \rho \hat{e_{\rho}}+\rho d \hat{e_{\phi}}
	\end{align*}
	\item \textbf{Area element} \begin{align*}
	d \vec{A}&=d \rho \hat{e_{\rho}} \times \rho d \phi \hat{e_{\phi}}\\&=\rho d \rho d \phi \hat{e_{z}}
	\end{align*}
\end{itemize}
	\begin{figure}[h]
	\begin{minipage}{0.45\textwidth}
		\includegraphics[width=6cm,height=5.5cm]{diagram-20211126(3)}
		\caption{Circular symmetry of Polar coordinate system}

	\end{minipage}\hfill
	\begin{minipage}{0.45\textwidth}
		\includegraphics[width=6cm,height=5.5cm]{diagram-20211126(2)}
		\caption{Area element}
	\end{minipage}
\end{figure}
\subsection{Kinematic in Polar Coordinate System}
\begin{itemize}
	\item {\textbf{Velocity vector}}
	\begin{equation*}
	\text{Velocity }=\vec{v}=\frac{d \vec{r}}{d t}
	\end{equation*}
	But here the displacement vector is given by, $\vec{r}=\rho \hat{e_{\rho}}$
	\\\\Since the unit vectors  changes with time, they should have finite time derivatives:
	\begin{align*}
	\frac{d }{dt} e_{\rho}&=\dot{\hat{e_{\rho}}}\\&=\dot{\phi}(-\sin \phi \hat{e_{x}}+\cos \phi \hat{e_{y}})\\&=\dot{\phi} \hat{e_{\phi}}\\
	\text{And}\quad	\frac{d }{dt} e_{\rho}&=  \dot{\hat{e_{\phi}}}\\&=\dot{\phi}(-\cos \phi \hat{e_{x}}-\sin \phi \hat{e_{y}})\\&=-\dot{\phi} \hat{e_{\rho}}\\
	\intertext{Therefore the velocity is given by:}
	\vec{v}&=\frac{d \vec{\rho}}{d t}=\dot{\rho} \hat{e_{\rho}}+\rho \dot{\hat{e_{\rho}}}\\&=\dot{\rho} \hat{e_{\rho}}+\rho \dot{\phi} \hat{e_{\phi}}\\
	\text{Where,}\quad v_{\rho}&=\dot{\rho} \hat{e_{\rho}} \hspace{0.62cm} -\text{Radial velocity component.}\\
	v_{\phi}&=\rho \dot{\phi} \hat{e_{\phi}}\quad -\text{Tangential / Azimuthal velocity component.}
	\end{align*}

	\item{\textbf{Acceleration vector}}\\
	Differentiating again with respect to time, we obtain the acceleration
	\begin{align*}
	\boldsymbol{a}&=\dot{\boldsymbol{v}}=\ddot{\rho} \hat{e_{\rho}}+\dot{\rho} \dot\hat{e_{\rho}}+\dot{\rho} \dot{\phi} \hat{e_{\phi}}+\rho \ddot{\phi} \hat{e_{\phi}}+\dot{\rho} \dot{\boldsymbol{\phi}} \hat{\dot{\phi}}\\
	\boldsymbol{a}&=\left(\ddot{\rho}-\rho \dot{\phi}^{2}\right) \hat{e_{\rho}}+(\rho \ddot{\phi}+2 \dot{\rho} \dot{\phi}) \hat{e_{\phi}}\\
	a_{\rho}&=\left(\ddot{\rho}-\rho \dot{\phi}^{2}\right) \hspace{0.62cm}-\text{ Radial acceleration component.}\\
	a_{\phi}&=(\rho \ddot{\phi}+2 \dot{\rho} \dot{\phi})\quad -\text{ Azimuthal acceleration component.}\\
	|a|&=\sqrt{a_{\rho}^{2}+a_{\phi}^{2}}
	\end{align*}

\end{itemize}
%--------------------------------------------
The Cylindrical and spherical polar systems are the three-dimensional relatives of the two-dimensional polar coordinate system. Let us look into those systems.
\section{ Cylindrical Polar Co-Ordinate System}

\begin{wrapfigure}{r}{0.32\textwidth}
	\begin{center}
		\includegraphics[width=6cm,height=5cm]{cylindrical 1}
	\end{center}
	\caption{Cylindrical Polar Coordinate system}
	\label{Cylindrical Polar Coordinate system}
\end{wrapfigure}
We make the use of Cylindrical polar coordinate system when there is a \textbf{cylindrical symmetry} in the form of a physical object.  Polar coordinates can be extended to three dimensions in a very straightforward manner. We simply add
the z coordinate, which is then treated in a cartesian like manner. Every point in space is determined by
the $\rho$ and $\theta$ coordinates of its projection in the xy plane, and its z coordinate.
\\ In the figure  \ref{Cylindrical Polar Coordinate system} ,$\rho$\ is the distance of the foot of the perpendicular drawn from the point to the $x-y(\rho, \theta)$\ plane.
Note that $\rho$ here is not the distance of the point $\mathrm{P}$ from the origin, as is the case in polar coordinate systems.\\The range of the coordinate varies as,\begin{align*}
0 &\leq \rho \leq \infty\\
0 &\leq \phi \leq 2 \pi\\
-\infty &\leq z \leq+\infty
\intertext{In terms of cartesian coordinates  }
x&=\rho \cos \phi \\
y&=\rho \sin \phi \\
z&=z
\end{align*}
We can change the basis of a
vector in cylindrical coordinate system to cartesian. Any vector $\vec{A}$ can be expressed in terms of them in the usual way,
$$
\vec{A}=A_{\rho} \hat{e_{\rho}}+A_{\phi} \hat{\phi}+A_{z} \hat{z}
$$
In matrix form we can write,
$$
\begin{aligned}
&\left[\begin{array}{c}
A_{\rho} \\
A_{\phi} \\
A_{z}
\end{array}\right]=\left[\begin{array}{ccc}
\cos (\phi) & \sin (\phi) & 0 \\
-\sin (\phi) & \cos (\phi) & 0 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{l}
A_{x} \\
A_{y} \\
A_{z}
\end{array}\right]
\end{aligned}
$$

\subsection{ Unit Vectors}
The unit vectors $\hat{e_{\rho}}, \hat{\phi}$ and $\hat{e_{z}},$ expressed in cartesian coordinates, are,
$$
\begin{array}{l}
\hat{e_{\rho}}=\cos \phi \hat{e_{x}}+\sin \phi \hat{e_{y}} \\
\hat{e_{\phi}}=-\sin \phi \hat{e_{x}}+\cos \phi \hat{e_{y}}\\
\hat{e_{z}}=\hat{e_{z}}
\end{array}
$$

\subsection{Line, Area and Volume Elements}
\begin{itemize}
	\item  \textbf{Line element}
	$$
	\overrightarrow{d l}= d \rho \hat{e_{\rho}}+\rho d \phi \hat{\phi}+d \hat{e_{z}}
	$$

	\item \textbf{Area element }
	\\ Orthogonal surfaces in cylindrical coordinate system can be generated in three ways.\\\\ i.e., $\rho=$ constant,  $\phi=$ constant,
	$\mathrm{z}=$ constant.
	\begin{align*}
	\rho&= \text{constant : A circular cylinder} \Rightarrow d A=\rho d \phi d z\\
	\phi&=\text{constant : A semi infinite plane with its edge along $\mathrm{z}$ axis} \Rightarrow d A= \phi d\rho   d z \\
	\mathrm{z}&= \text{constant : An infinite plane as in the rectangular system.}\Rightarrow d A= zd\phi d\rho
	\end{align*}

	\begin{figure}[h]
		\begin{minipage}{0.30\textwidth}
			\includegraphics[width=6cm,height=6cm]{cylindrical 3}
			\caption{Constant $\rho$ surface}

		\end{minipage}\hspace{0.5cm}
		\begin{minipage}{0.35\textwidth}
			\includegraphics[width=6cm,height=6cm]{cylindrical 2}
			\caption{Constant $\phi$ surface}
		\end{minipage}
		\begin{minipage}{0.30\textwidth}
			\includegraphics[width=6cm,height=6cm]{cylindrical 4}
			\caption{Constant $z$ surface}
		\end{minipage}
	\end{figure}
	\item   \textbf{Volume element}
	$$
	d V=\rho d \phi d \rho d z
	$$
	\end {itemize}

	\subsection{Kinematic Vectors in  Cylindrical Polar Coordinate System}
	\begin{itemize}
		\item \textbf{Velocity Vector}
		\begin{align*}
		\boldsymbol{v}&=\dot{\rho} \hat{e_{\rho}}+{\rho} \dot{\phi} \hat{\phi}+\dot{z} \boldsymbol{k}\\
		\text{Where,}\quad  v_{\rho}&=\dot{\rho}\quad;\quad v_{\phi}=r \dot{\phi}\quad;\quad v_{z}=\dot{z}\\ \text{And} \quad
		v&=\sqrt{v_{\rho}^{2}+v_{\phi}^{2}+v_{z}^{2}} \\\intertext{Where $ \dot{\rho}$  is radial velocity in $ \hat{e_{\rho}} $ direction and $ r \dot{\phi}$  is tangential velocity in $ \hat{\phi} $ direction}
		\end{align*}

		\item \textbf{Acceleration Vector}
		\begin{align*}
		\boldsymbol{a}&=\left(\ddot{\rho}-\rho \dot{\phi}^{2}\right) \hat{e_{\rho}}+(\rho \ddot{\phi}+2 \dot{\rho} \dot{\phi}) \hat{\phi}+\ddot{z} \boldsymbol{k}\\
		\text{Where}  \quad a_{\rho}&=\ddot{\rho}-\rho \dot{\phi}^{2}\quad;\quad a_{\phi}=\rho \ddot{\phi}+2 \dot{\rho} \dot{\phi}\quad;\quad a_{z}=\ddot{z}\\ \text{And}\quad a&=\sqrt{a_{\rho}^{2}+a_{\phi}^{2}+a_{z}^{2}}
		\end{align*}

		\end {itemize}

		%------------------------------------------------------
		\section{Spherical Polar Co-Ordinate System}

		We make  use of Spherical polar coordinate system when there is a \textbf{Spherical symmetry} in the form of a physical object.
		In spherical polar coordinates, we utilize two angles and a distance to specify the position of a particle. Every point in space is determined by
		the $r$ , $\theta$  and $\phi $ coordinates.
		The range of coordinates varies as, \\
		\begin{minipage}{0.45\textwidth}
			\begin{align*}
			0 &\leq r \leq \infty\\
			0 &\leq \theta \leq \pi\\
			0&\leq \phi \leq 2\pi
			\end{align*}
			If you are given spherical coordinates $(r, \theta, \phi)$ of a point in the plane.The Cartesian coordinates $(x, y, z)$ can be determined from the coordinate transformations.

			\begin{align*}
			x&=r \sin \theta \cos \phi \\
			y&=r \sin \theta \sin \phi \\
			z&=r \cos \theta
			\end{align*}
		\end{minipage}
	\begin{minipage}{0.45\textwidth}
	\begin{figure}[H]
		\centering
	\includegraphics[width=7cm,height=6.5cm]{spherical 1}
		\caption{Spherical Polar Coordinate system}
		\label{Spherical Polar Coordinate system}
	\end{figure}
	\end{minipage}

		\subsection{Unit Vectors}
		Curresponding to radial distance(r), polar distance ($\theta$) and azimuthal angle ($\phi$) We have three orthogonal unit vectors in SPC and they constitute an orthonormal basis $\left\lbrace \hat{e_{r}},\hat{e_{\theta}},\hat{e_{\phi}} \right\rbrace $. The unit vectors also are related by the coordinate transformations
		$$
		\begin{array}{l}
		\hat{e_{r}}=\sin \theta \cos \phi \hat{e_{x}}+\sin \theta \sin \phi \hat{e_{y}}+\cos \theta \hat{e_{z}} \\
		\hat{e_{\theta}}=\cos \theta \cos \phi \hat{e_{x}}+\cos \theta \sin \phi \hat{e_{y}}-\sin \theta \hat{e_{z}} \\
		\hat{e_{\phi}}=-\sin \phi \hat{e_{x}}+\cos \phi \hat{e_{y}}
		\end{array}
		$$
		 In matrix representation,
		$$\left[\begin{array}{c}
			\hat{e_{r}} \\
			\hat{e_{\theta}} \\
			\hat{e_{\phi}}
		\end{array}\right]=\left[\begin{array}{ccc}
			\sin \theta \cos \phi & \sin \theta \sin \phi & \cos \theta \\
			\cos \theta \cos \phi & \cos \theta \sin \phi & -\sin \theta \\
			-\sin \phi & \cos \phi & 0
		\end{array}\right]\left[\begin{array}{c}
			\hat{e_{x}} \\
			\hat{e_{y}} \\
			\hat{e_{z}}
		\end{array}\right]$$
		The inverse transformation will be
			$$\left[\begin{array}{c}
			\hat{e_{x}} \\
			\hat{e_{y}} \\
			\hat{e_{z}}
		\end{array}\right]=\left[\begin{array}{ccc}
		\sin \theta \cos \phi & \cos \theta \cos \phi & -\sin \phi \\
		\sin \theta \sin \phi & \cos \theta \sin \phi & \cos \phi \\
		\cos \theta & -\sin \theta & 0
		\end{array}\right]\left[\begin{array}{c}
		\hat{e_{r}} \\
		\hat{e_{\theta}} \\
		\hat{e_{\phi}}
		\end{array}\right]$$
			Then any vector $\vec{A}$ can be expressed in terms of them in the usual way:
		\begin{align*}
		\vec{A}&=A_{r} \hat{e_{r}}+A_{\theta} \hat{e_{\theta}}+A _{\phi} \hat{e_{\phi}}\\
		A_{r}&=A_{x} \sin \theta \cos \phi+A_{y} \sin \theta \sin \phi+A_{z} \cos \theta\\
		A_{\theta}&=A_{x} \cos \theta \cos \phi+A_{y} \cos \theta \sin \phi-A_{z} \sin \theta\\
		A_{\phi}&=-A_{x} \sin \phi+A_{y} \cos \phi
		\intertext{	In matrix form we can write}
		\left[\begin{array}{c}
		A_{r} \\
		A_{\theta} \\
		A_{\phi}
		\end{array}\right]&=\left[\begin{array}{ccc}
		\sin \theta \cos \phi & \sin \theta \sin \phi & \cos \theta \\
		\cos \theta \cos \phi & \cos \theta \sin \phi & -\sin \theta \\
		-\sin \phi & \cos \phi & 0
		\end{array}\right]\left[\begin{array}{c}
		A_{x} \\
		A_{y} \\
		A_{z}
		\end{array}\right]
		\end{align*}
		\subsection{Line, Area ,Volume Elements}
		\begin{itemize}
			\item\textbf{ Line element}$$d \vec{{l}}=d r \hat{e_{r}}+r d \theta \hat{e_{\theta}}+r \sin \theta d \phi \hat{e_{\phi}}$$
			\item \textbf{Area element}\\
			\\ Orthogonal surfaces in Spherical polar coordinate system can be generated in three ways \\ i.e., $r=$ constant $\phi=$ constant and
			$\mathrm{z}=$ constant.
			\begin{align*}
			r&= \text{constant : A sphere.}\Rightarrow d A_{r}=r^{2} \sin \theta d \theta d \phi  \\
			\theta&=\text{constant : A cone.} \Rightarrow d A_{\theta}=r \sin \theta d r d \phi \\
			\phi&= \text{constant : An semi infinite plane as in the rectangular system.} \Rightarrow d A_{\phi}=r d r d \theta
			\end{align*}

			\begin{figure}[h]
				\begin{minipage}{0.30\textwidth}
					\includegraphics[width=6.5cm,height=6cm]{spherical 2}
					\caption{Constant $r$ surface}
				\end{minipage}\hspace{0.5cm}
				\begin{minipage}{0.35\textwidth}
					\includegraphics[width=6cm,height=6cm]{spherical 5}
					\caption{Constant $\theta$ surface}
				\end{minipage}
				\begin{minipage}{0.30\textwidth}
					\includegraphics[width=6.5cm,height=6cm]{spherical 4}
					\caption{Constant $\phi$ surface}
				\end{minipage}
			\end{figure}

			\item \textbf{Volume elment}
			$$
			d V=r^{2}d r \sin \theta d \theta d \phi
			$$
		\end{itemize}

		\subsection{Kinematic Vectors in Spherical Polar Coordinate System}
		\begin{itemize}
			\item	\textbf{Velocity Vector}\begin{align*}
			\vec{v}&=\dot{\vec{r}}\\&=\dot{\hat{e_{r}}} r+\hat{e_{r}} \dot{r} \\
			\vec{v}&=\hat{e_{r}} \dot{r}+\hat{e_{\theta}} r \dot{\theta}+\hat{e_{\phi}} r \dot{\phi} \sin \theta
			\end{align*}
			\item \textbf{Acceleration Vector}
			\begin{align*}
			\vec{a}=&\hat{e_{r}}\left(\ddot{r}-r \dot{\theta}^{2}-r \dot{\phi}^{2} \sin ^{2} \theta\right)+\hat{e_{\theta}}\left(r \ddot{\theta}+2 \dot{r} \dot{\theta}-r \dot{\phi}^{2} \sin \theta \cos \theta\right)\\&+\hat{e_{\phi}}(r \ddot{\phi} \sin \theta+2 r \dot{\theta} \dot{\phi} \cos \theta+{2 \dot{r} \dot{\phi} \sin \theta})\\
			\intertext{Where,}
			a_{r}&=\left(\ddot{r}-r \dot{\theta}^{2}-r \dot{\phi}^{2} \sin ^{2} \theta\right)\\
			a_{\theta}&=\left(r \ddot{\theta}+2 \dot{r} \dot{\theta}-r \dot{\phi}^{2} \sin \theta \cos \theta\right)\\
			a_{\phi}&=(r \ddot{\phi} \sin \theta+2 r \dot{\theta} \dot{\phi} \cos \theta+2 \dot{r} \dot{\phi} \sin \theta)
			\end{align*}

			\end {itemize}

			%----------------------------------------------
			\begin{table}[H]
				\centering
				\renewcommand*{\arraystretch}{2}
				\arrayrulecolor{ocre}

				\begin{tabular}{|p{3cm}|p{3.5cm}|p{2.5cm}|p{3.5cm}|}
					\hline
					\multicolumn{4}{|c|}{\textbf{Co-ordinate systems}}\\\hline
					\textbf{System}&\textbf{Symmetry}&\textbf{Co-ordinates}
					&\textbf{Length elements}\\\hline
					Cartesian &Rectangular &$x,y,z$&$dx,dy,dz$\\\hline
					Polar&Circular symmetry &$r ,\theta $&$dr ,r d\theta $ \\\hline
					Cylindrical polar&Cylindrical symmetry &$r ,\theta,z $&$dr ,r d\theta, dz $ \\\hline
					Spherical polar&Spherical symmetry &$r ,\theta ,\phi$&$dr ,r d\theta, r \sin \theta d\phi $\\\hline

				\end{tabular}
				\caption{Basic summary of Coordinate system.}
			\end{table}


\section[Bayesian Inference]{Lecture 2: Bayesian Inference}

'Bayesian Inference' refers to computation of the posterior distribution over parameters given the data.

\begin{itemize}
\item Data $\mathcal{D}=\{t_1, t_2, \ldots, t_N\}$
\item Supervised learning: $\mathcal{D}=\{(x_1,t_1), (x_2,t_2), \ldots, (x_N,t_N)\}$
\item Likelihood function $P(t|w)$ \hspace{1cm}(parameterized by $w$)
\item $P(D|w)= \prod_{n=1}^N P(t_n|w)$ Independence assumption
\item Prior distribution $\pi(w)$
\item Bayes rule: $\mbox{Posterior} \propto \mbox{Likelihood}  \times \mbox{Prior}$
\begin{align}
P(w | D) =\frac{1}{Z} P(D|w) P(w)
\end{align}
\item Probabilities must normalize to $1$:
\begin{align}
Z= P(D)= \int P(D|w) P(w) dw
\end{align}
Marginalization is the difficult part in Bayesian inference, how we can get arund this step will be shown in a later section
\end{itemize}

\subsection{Usage of the posterior distribution}

	\begin{itemize}
		\item Predictive Distribution
		\begin{align}
			P(t^*|x^*, D)= \int_w P(t^*| x^*, w) P(w| D)   dw
		\end{align}
		where $(t^*,x^*)$ is a new observation

		\item Decision making
		Suppose we have calculated $P(t^*|x^*, D)$, and someone asks us to give a guess $\hat t$ of $t^*$. While we could just take the \emph{most likely value} of $t^*$, it really depends on the cost function. Are mistakes in one direction as costly as mistakes in the other direction? For example, what is the cost of a false positive or false negative? Given a {cost function} $C(\hat t, t)$, one can calculate the 'Bayes-optimal' decision from the posterior distribution.
		\item Scientific statements
		e.g. 'After observing 100 data points, we were 90\% sure that the parameter $\theta$ is between -.1 and .3. Now that we have observed another 200, we are 97\% sure.'
	\end{itemize}

\subsection{Alternatives to a full Bayesian treatment}
In most cases, the posterior distribution can not be calculated exactly, and approximations have to be used.
\begin{itemize}
\item Ignore prior, maximize likelihood $P(D|w)$: {Maximum likelihood learning}
\item Only search for mode of posterior ({Maximum a posteriori, MAP}), i.e. $\mbox{argmax}_w P(w |D) $. In practice, maximize log-posterior.\\
\end{itemize}

\begin{bbbox}{Maximum a posteriori}
\textcolor{gray}{Q: Why is finding the mode of the posterior so much easier than finding the full posterior?}\\
It is a point estimate similar to ML, this makes the approach similarly expensive in computational terms.
We can escape the intractability of the normalization constant.

\textcolor{gray}{Q: When is MAP a really bad idea?}
\begin{itemize}
	\item multiple local maxima
	\item Very skewed posterior distribution
\end{itemize}
\end{bbbox}

\begin{itemize}
\item Use simplified model to approximate posterior: Find parameters of model $q(w,\Phi)$ such that
$q(w) \approx P(w|D)$. \\

\item Examples: Varational Inference, Expectation Propagation, Laplace Approximation
\item Very often, a Normal approximation is used: $q(w) \approx \mathcal{N}(w|\mu,\Sigma)$.
\item Use MCMC sampling to generate samples from posterior distribution
\end{itemize}

\subsection{Coin flip example}
\begin{itemize}
\item Suppose we have $N$ throws of a coin, $D=\{t_1, t_2, \ldots, t_N\}$
\item We write $T_n= 1$ if the n-th throw was head, and $t_n=0$ if it was tail.
\item One parameter:  $q\in [0, 1]$, the probability of obtaining heads
\item Likelihood of one throw:
\begin{align}
 P(T_n=1|q)&= q\\
 P(T_n=0|q)&= (1-q)
\end{align}
Multiple throws: Bernoulli process
\item Likelihood of data $D$: [on board]
\end{itemize}
\begin{bbbox}{Bernoulli likelihood}
\begin{flalign*}
	P(T_n = t | q) &= q^t (1-q)^{1-t}  \\
	P(D | q) &= \prod_{n_1}^N q^{t_n} (1-q)^{1-t_n}
\end{flalign*}
\end{bbbox}

\subsubsection{The beta distribution as a conjugate prior for the Bernoulli distribution}
The shape of the distribution is determined by two parameters.

\begin{figure}
\includegraphics[width=\textwidth]{./lecture2/BetaZoo}
\caption{The beta distribution for different parameter configurations. Notice the symmetricity for $\alpha_1 = \alpha_2$. A prior distribution reflects our beliefs about the parameters. If we have no beliefs we will stick to a flat prior in the upper left corner. Whereas if we believe $q$ to be close to $1$ we might prefer to use the prior in the lower left corner.}
\end{figure}

\begin{itemize}
\item
Beta distribution:
\begin{align}
\pi(q| \alpha, \alpha_{2})= \frac{1}{Z}q^{\alpha_1-1} (1-q)^{\alpha_{2}-1}
\end{align}
\item Normalizing constant: the 'beta function'
\begin{align}
Z= \int_0^1 q^{\alpha_1-1} (1-q)^{\alpha_{2}-1} dq=: B(\alpha_1, \alpha_{2})
\end{align}
However there is no need to calculate this constant due to the conjugacy. We will see more on that in a later section.
\item Mean and Variance:
\begin{align}
\mbox{E}(q|\alpha_1,\alpha_{2})&= \frac{\alpha_1}{\alpha_1+\alpha_{2}}\\
\mbox{Var}(q|\alpha_1,\alpha_{2})&= \frac{\alpha_1\alpha_{2}}{(\alpha_1+\alpha_{2})^2(\alpha_1+\alpha_{2}+1)}
\end{align}
\item Symmetric case and heuristics:  [on board]
\end{itemize}

\begin{bbbox}{Symmetric case and heuristics}
\begin{align*}
	Say: \; \alpha_1 = \alpha_2 \\
	then \mbox{E}(q|\alpha_1,\alpha_{2})&= \frac{\alpha_1}{\alpha_1+\alpha_{2}}\\
	&= \frac{\alpha_1}{2 \alpha_1} = 0.5\\
	\mbox{Var}(q|\alpha_1,\alpha_{2})&= \frac{\alpha_1^2}{(2\alpha_1)^2(2\alpha_1 + 1)} \\
									 &= \frac{1}{4(2\alpha_1 + 1)}
\end{align*}
That means for $\alpha_1 = \alpha_2 \rightarrow \infty : \mbox{Var}(q|\alpha_1,\alpha_{2}) \rightarrow 0$
\end{bbbox}

\subsubsection{Posterior inference}

\begin{bbbox}{Posterior inference}
We use $S_n$ to denote the number of heads on the first $n$ trials.
$S_N = \sum_{n_1}^N t_n$
\vspace{.5cm}
Maximum likelihood estimation:
$L(D,q) = log P(D,q) = \sum_{n_1}^N t_n log(q) + (1-t_n) log(1-q) = log(q) S_N + log(1-q)(N-S_N)$\\
Find the q for which the log likelihood is maximized, such that
$\frac{\partial L(D,q)}{\partial q} = 0$\\
$\frac{\partial L(D,q)}{\partial q} = \frac{S_N}{q} - \frac{N-S_N}{1-q} = 0$\\
$\frac{q}{S_N} = \frac{1-q}{N-S_N}$ \\
$q = \frac{S_N}{N}$
\vspace{.5cm}
[on board]

\vspace{.5cm}
Posterior distribution:
\begin{align}
P(q | D) &= \frac{1}{Z}P(D|q)P(q) \\
&= \frac{1}{Z} \prod_{n=1}^N q^{t_n} (1-q)^{1-t_n} \frac{1}{B(\alpha_1,\alpha_2)} q^{\alpha_1 -1} (1-q)^{\alpha_2 - 1} \nonumber \\
&= \frac{1}{Z_2} q^{S_N} (1-q)^{N - S_N} q^{\alpha_1 -1} (1-q)^{\alpha_2 - 1} \nonumber \\
&= \frac{1}{Z_2} q^{S_N + \alpha_1 -1} (1-q)^{N - S_N + \alpha_2 - 1} \nonumber \\
where \; Z_2 &= B(S_N + \alpha_1, N - S_n + \alpha_2) \nonumber
\end{align}

$q | D \backsim Beta(S_N + \alpha_1, N - S_n + \alpha_2)$ \\
\vspace{.5cm}
[on board]
\end{bbbox}

\vspace{.5cm}
We can either take all the data and calculate the posterior at once, or do it sequentially as new data comes in:
\vspace{.5cm}
[on board]

Data: $D=\{1     1     0     1     1     1     1     1     0\}$

\includegraphics[width=\textwidth]{./lecture2/BetaPosterior}

\subsubsection{Predictive distribution}
\vspace{.5cm}
After observing $N$ coin-flips, what is our prediction for the next coin flip?
\begin{align}
P(T^*=1| D)&= \int_0^1 P(T^*=1|q) P(q|D) dq\\
& = \int_0^1 q P(q|D) dq\\
&= \mbox{E}(q|D) \\
&=\frac{\alpha_1+S_N}{(\alpha_1+ S_N)+(\alpha_{2}+N-S_N)}\\
&=\frac{\alpha_1+S_N}{(\alpha_1+\alpha_2+N)}
\end{align}

\vspace{.5cm}
In our example, $\mbox{E}(q|D)=0.69$, and $\mbox{MLE}=0.78$.

\begin{bbbox}{What happens if $N$ gets very large?}

For $ \alpha_1,\alpha_2 \rightarrow 0$ or $ S_N , N \rightarrow \infty $ the prediction will approach the maximum likelihood solution.
\end{bbbox}

\vspace{.5cm}
Note: This is a bit of a special case-- in general, the predictive distribution is not simply the likelihood evaluated at the mean!!!

\subsubsection{Statistical Reasoning}
\includegraphics[width=\textwidth]{./lecture2/BetaPosteriorConfidence}

\subsubsection{Decision making}
Someone offers you the bet that you get 1 euros if you predict the next coin toss correctly, but you have to pay 2 euros if if you are wrong.

Should you take the bet? What should you predict?

\begin{align}
C(t^*,\hat t)=
\begin{cases}
-1 &\mbox{if~} t^*= \hat t\\
2 &\mbox{if~} t^*\neq \hat t\\
\end{cases}
\end{align}

Expected cost:
\begin{align}
E(C)=\sum_{t^*=0}^1 C(t^*,\hat t) P(t^*| D)
\end{align}

If we predict tail ($t^*=0$):
\begin{align}
E(C)&= P(t^*=1|D) C(0,1)+P(t^*=0|D) C(0,0)  \\
&=0.69 (2)  + 0.31 (-1)= 1.07
\end{align}

If we predict head ($t^*=1$):
\begin{align}
E(C)=0.69 (-1)  + 0.31 (2)= -0.07;
\end{align}

\subsection{The exponential family and conjugate priors}
Why was inference so easy here?

\begin{itemize}
\item Posterior distribution had a closed form solution.
\item In fact, the posterior had the same functional form as the prior, just different parameters.
\item Parameters of posterior could be calculated by simply adding observations to prior parameters.
\item We used a likelihood from the {exponential family} and its {conjugate prior}. In this case,  Bayesian inference is always easy.
\item For this reason, exponential families and conjugate priors are used extensively in Bayesian modelling, often as 'building blocks' of more complicated models.
\end{itemize}

Inference is easy whenever the likelihood is in the exponential family and the prior is its conjugate.
\begin{itemize}
\item Exponential family distributions  have the form
\begin{align}
P(\xx|\theta)&= g(\theta) f(\xx) \exp\left(\phi(\theta)^\top S(\xx)\right)
\end{align}

\item The conjugate prior is
\begin{align}
\pi(\theta)&= F(\tau,\nu) g(\theta)^\nu \exp(\phi(\theta)^\top \tau)
\end{align}
\item Calculating the posterior:
\begin{align}
 [\mbox{on board}]
\end{align}
\end{itemize}

The posterior given an exponential family likelihood and conjugate prior is
\begin{align}
P(\theta|D)= F(\tau+ \sum_i S(x_i), \nu+N) g(\theta)^{\nu+N} \exp\left(\phi(\theta)^\top(\tau +\sum_i S(x_i)    )\right)
\end{align}

\begin{itemize}
\item $\phi(\theta)$ is the vector of {natural parameters}
\item $\sum_i S(x_i)$ is the vector of {sufficient statistics}
\item $\tau$ are {pseudo-observations}
\item $\nu$ is the scale of the prior\\
\end{itemize}

The exponential family includes most common distributions, including the Normal, Exponential, Gamma, Chi-square, Beta, Dirichlet, Bernoulli, Poisson, Wishart and the Inverse Wishart.

\begin{bbbox}{How can we put our coin-example into this framework?}
\begin{align*}
	P(x | \theta) &= \theta^x (1-\theta)^{1-x} \\
	              &= \theta^x (1-\theta)(1-\theta)^{-x} \\
	              &= (1-\theta) \exp\left(\log\left(\theta^x (1-\theta)^{-x}\right)\right) \\
	              &= (1-\theta) \exp\left(x \log\left(\frac{\theta}{1-\theta}\right)\right) \\
        g(\theta) &= 1 - \theta \\
             f(x) &= 1 \\
     \phi(\theta) &= log\left(\frac{\theta}{1 - \theta}\right) : log odds \\
             S(x) &= x
\end{align*}
\end{bbbox}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS671: Machine Learning
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 3}

Consider a learning problem where each instance is described by a conjunction of $n$ boolean attributes $A_1$, ..., $A_n$.
Here, a boolean attribute is an attribute whose domain consists of two values, \textbf{t} and \textbf{f}.
Thus, a typical instance would be as given in Expression \ref{eq31}.

\begin{equation}\label{eq31}
(A_1 = \textbf{t}) \wedge (A_2 = \textbf{f}) \wedge ... \wedge (A_n = \textbf{t})
\end{equation}

Consider a hypothesis space \textit{H} in which each hypothesis is a disjunction of constraints over these attributes.
For example, a typical hypothesis would be as given in Expression \ref{eq32}.

\begin{equation}\label{eq32}
(A_1 = \textbf{t}) \vee (A_5 = \textbf{f}) \vee (A_7 = \textbf{t})
\end{equation}

Design an algorithm that accepts a series of training examples and outputs a consistent hypothesis \textit{if one exists}.
Your algorithm should run in time that is polynomial in \textit{n} (the number of attributes) and in \textit{m}, the number of training examples.

\subsection*{Solution}

Suppose $\mathcal{C}_i$ is the concept class containing set of conjunctions of Boolean literals $x_1$, $x_2$, ..., $x_n$, $\bar{x_1}$, $\bar{x_2}$, ..., $\bar{x_n}$.
The series of training examples can therefore be defined as shown in Equation \ref{eq33}.

\begin{equation}\label{eq33}
\mathcal{C} = \{\prod_{j=1}^{n} a_i | 1 \leq i \leq m, a_i \in \{x_i, \bar{x_i}, 1\}\}
\end{equation}

Using fundamental \textit{De Morgan}'s laws in boolean algebra, the proposed algorithm would output the consistent hypothesis $\mathcal{H}$ by taking the complement of the inconsistent hypothesis as shown in Equation \ref{eq34}.

\begin{equation}\label{eq34}
\mathcal{H} = \overline{(\overline{C_1}\wedge \overline{C_2} \wedge ... \wedge \overline{C_m})}
\end{equation}

As $C_i$ is based on conjunction of Boolean literals corresponding to attributes of training instances, we will define $D_i$ as $\bar{C_i}$ and call it disjunctions of Boolean literals, in conformance to Expression \ref{eq32}.
We can thus rewrite Equation \ref{eq34} as given below.

\begin{equation}\label{eq35}
\mathcal{H} = \overline{(D_1 \wedge D_2 \wedge ... \wedge D_m)}
\end{equation}

Therefore, to obtain the concept target it suffices to find disjunctions of $n$ boolean literals for each $D_i$ when $1 \leq i \leq m$.
To achieve this objective \textsc{Disjunct} is proposed in Algorithm 2 that outputs disjunctions of Boolean Literals for both positive or negative concepts $C$.

\begin{algorithm}[H]
\caption{\textsc{Disjunct}($C$, $i$)}
\begin{algorithmic}[1]
\If {(example is positive \textbf{and} $C(a_i) = 0$) \textbf{or} (example negative \textbf{and} $C(a_i) = 1$)}
\State $term \leftarrow X_i$
\Else
\State $term \leftarrow \overline{X_i}$
\EndIf

\If {i = 1}
\State \textbf{return} term
\Else
\State \textbf{return} \textsc{Disjunct}($C$, $i-1$) $\vee$ term
\EndIf
\end{algorithmic}
\end{algorithm}

By obtaining disjunctions of boolean literals of concept $C$ using \textsc{Disjunct}($C$, $i$), the main proposed algorithm can be given in Algorithm 3.

\begin{algorithm}[H]
\caption{\textsc{Find-Target-Concept}}
\begin{algorithmic}[1]
\State hypothesis $\leftarrow 1$
\For {$i \leftarrow 1$ to $m$}
\State hypothesis $\leftarrow$ hypothesis $\wedge$ Disjunct($C_i$, $m$)
\EndFor
\State \textbf{return} compliment of hypothesis.
\end{algorithmic}
\end{algorithm}

As \textsc{Disjunct} is a recursive algorithm that calls itself $n$ times for a concept of $n$ attributes, its runtime is $\mathcal{O}(n)$.
Since \textsc{Disjunct} is called once for each $m$ examples, the total runtime of the proposed algorithm would be $\mathcal{O}(n) + \mathcal{O}(m) = \mathcal{O}(nm)$.


\section{Summation notation, product notation}

One should read
\[\sum_{k = m}^na_m\] as the value of \verb|result| after executing this code:
\begin{verbatim}
    result = 0;
    for (k = m; k <= n; k = k + 1) {
        result = result + a_m.
    }
\end{verbatim}

Ideally, most of the time you should see the above sum as just a compactified way to write
\[a_m + a_{m + 1} + \cdots + a_n\]

Notice that this code does not make sense if $m > n$, because the index $k$ is \textbf{always} incremented. In this case we always take the sum to be $0$ by default. So a sum like
\[\sum_{k = 1}^0k\] is equal to $0$, not $1$.

Similarly for products. one should read \[\prod_{k = m}^na_m\] as the value of \verb|result| after executing this code:

\begin{verbatim}
    result = 1;
    for (k = m; k <= n; k = k + 1) {
        result = result * a_m.
    }
\end{verbatim}

Again this code does not make sense if $m > n$. In this case we always take the product to be $1$ by default.

The following equations are some useful properties related to summations.
\[\sum_{i = m}^n(a_i + b_i) = \sum_{i = m}^na_i + \sum_{i = m}^nb_i\]
The above property is often colloquially referred to as ``splitting a sum''.
\[\sum_{i = m}^nca_i = c\sum_{i = 1}^ma_i\] if $c$ is a constant that does not depend on the index $i$.
\[\sum_{i = m}^na_i = a_m + \sum_{i = m + 1}^na_i\]
\[\sum_{i = m}^na_i = a_n + \sum_{i = n}^{n-1}a_i\]
Note that the above two equations make sense if $n \leq m$. (What happens when $n = m$?)

By yourself come up with some analogous formulas for products.

\section{The Principle of Mathematical Induction}

\begin{definition}
    A proposition $P(n)$ is a logical statement involving some value $n$.
\end{definition}

It is very helpful to consider an example or two. For example, define $P(n)$ to be the logical statement that $n = 1$. In other words, $P(n) \equiv (n = 1)$. Then $P(n) \equiv T$ if and only if $n = 1$, which is true but is silly to say. Another example: define $P(n) \equiv (2 \mid n)$. Then
\[P(n) \equiv \begin{cases}T, & 2 \mid n \\ F, & 2 \nmid n\end{cases}\]

But as of now we aren't really interested in examples like this. We'd like to know if there is a way to show that $P(n) \equiv T$ for $n \geq k$ for some integer $k$, for example.

For example, we'd like to figure out whether or not
\[\sum_{i = 1}^n(2i - 1) = n^2\] for all $n$. It seems true! But how might we prove it? This leads us to a technique called \textit{mathematical induction}, a powerful technique which lets us prove classes of statements indexed by the natural numbers. The technique of mathematical induction essentially relies on the following theorem. We will prove this theorem is true in a future section using a property of the natural numbers called the \textbf{Well-Ordering Principle}.

\begin{theorem}
Let $a$ be any natural number. And let $P(k)$ be a proposition about $k \in \mathbb{N}$. Assume the following claims are true:
\begin{itemize}
    \item $P(a) \equiv T$
    \item Assuming $P(k)$ is true for any natural number $k \geq a$, we can prove that $P(k + 1)$ is true.
\end{itemize}

Then $P(k)$ is true for all $k \geq a$.
\end{theorem}

Essentially, suppose you have an indexed family of propositions $P(n)$ which you want to prove true for $n \geq a$. A proof by mathematical induction is essentially the same as satisfying the hypotheses of the above theorem. Essentially, in a proof by induction you show the following:

\begin{enumerate}
    \item First we show that $P(a) \equiv T$.
    \item Next, assuming that $P(k)$ is true for $k \geq a$, prove that $P(k + 1)$ is true.
\end{enumerate}

Here is an example of this in action. We will prove the first equality stated at the beginning of the chapter.
\begin{proposition}
\[\sum_{i = 1}^n(2i - 1) = n^2\] for all $n$ for all natural numbers $n \geq 1$.
\end{proposition}
\begin{proof}
We will prove this equality using mathematical induction. First, we observe for $n = 1$ that
\[\sum_{i = 1}^1(2i - 1) = 1 = 1^2.\]
Now assume that
\[\sum_{i = 1}^k(2i - 1) = k^2.\]
Then
\[\sum_{i = 1}^{k + 1}(2i - 1) = \sum_{i = 1}^k(2i - 1) + 2(k + 1) - 1 = k^2 + 2k + 1 = (k + 1)^2,\]
so we are done.
\end{proof}

Let's break down this proof a little bit further. Here the statement $P(n)$ is the statement that the equality
\[\sum_{i = 1}^n(2i - 1) = n^2\] is true for that specific $n$. In the first part of the proof we show that $P(1)$ is true. In the second part of the proof we show that the truth of $P(k)$ implies the proof of $P(k + 1)$. To do this, we split the sum
\[\sum_{i = 1}^{k + 1}(2i - 1) = \sum_{i = 1}^k(2i - 1) + 2(k + 1) - 1\]
into two parts. By our hypothesis, the left part is exactly $k^2$. The rest of the prooof simplifies the sum to achieve the desired conclusion: that is to show
\[\sum_{i = 1}^{k + 1}(2i - 1) = (k + 1)^2.\]

\section{Exercises}
\begin{enumerate}
    \item \input{Ch5/1_ex/problem_2}
    \item \input{Ch5/1_ex/problem_1}
    \item \input{Ch5/1_ex/problem_3}
    \item \input{Ch5/1_ex/problem_4}
    \item \input{Ch5/1_ex/problem_5}
    \item \input{Ch5/1_ex/problem_6}
\end{enumerate}


\section{Linear Algebra}

\subsection{Distance}

Consider a line  $L$  in  $\mathbb{R}^2$  given by the equation $L:\theta \cdot x + \theta_0=$ where $\theta$ is a vector normal to the line $L$. Let the point $P$ be the endpoint of a vector $x_0$(so the coordinates of $P$  equal the components of $x_0$).\\

The shortest distance $d$ between the line $L$ and the point $P$ is:
\begin{align*}
d= \displaystyle \frac{\mid \theta \cdot x_0+ \theta _0\mid }{\mid \mid \theta \mid \mid }
\end{align*}


\subsection{Discrete uniform distribution}

There is a set \(s\) such that:

\(P(x\in s)=p\)

\(P(x\not\in s)=0\)

\subsubsection{Moments of the uniform distribution}

The mean is the mean of the set \(s\).

If the set is all numbers of the real line between two values, \(a\) and \(b\), then:

The mean is \(\dfrac{1}{2}(a+b)\).

The variance is \(\dfrac{(b-a)^2}{12}\) in the continuous case.

The variance is \(\dfrac{(b-a+1)^2-1}{12}\) in the discrete case.


%\section{Numerical examples}
In this section, we will list some numerical results using different methods on the MNIST dataset.

%The MNIST data set contains the pictures of single digits. The size of each digit picture is 28*28 and the label is the digit from 0 to 9. There are 60000 training data and 10000 test data in all.
\subsection{Least square}

We first present the details of the experiments using the least square for classification.

The most straightforward model for prediction is the linear model
$$
y=Wx+b,
$$
where $x\in \mathbb{R}^{784}$ is the input data, $y\in \mathbb{R}^{10}$ is the prediction and $W \in \mathbb{R}^{10 \times 784}, b \in \mathbb{R}^{10} $ are the parameters of the model.
For each data in the MNIST dataset, $x_i\in \mathbb{R}^{784}, i=1,...,60000$, denote the corresponding label as $y_i\in \mathbb{R}^{10}$.
We denote the training set and labels in matrix form,
$$A=\begin{pmatrix}
    x_{1} &x_{2}  &...  &x_{60000}     \\
    1&1&... &1\\
\end{pmatrix}^T \in \mathbb{R}^{60000\times 785},$$
$$Y=\begin{pmatrix}
    y_{1}        ,
    y_{2}       ,
    ... ,
    y_{60000}
\end{pmatrix}^T\in \mathbb{R}^{60000\times 10}.
$$
Our goal is to find the optimal parameter
$\theta=\left[ W ,b  \right]^T\in \mathbb{R}^{785\times 10}$ to minimize the total sum of square loss
$$
\sum_{i=1}^{60000} \|y_i-(W x_i+b)\|^2_2=\|Y- A\theta\|^2_2.
$$
The empirical formulae gives the optimal parameter as
\begin{align}
\hat{\theta}=(A^TA)^{-1}A^TY.
\end{align}

For the MNIST dataset, after $\hat{\theta}$ is calculated, we can classify $x_i$ based on the linear model
\begin{align}
\hat{y_i}=\hat{W}x_i+\hat{b} \in \mathbb{R}^{10}.
\end{align}
We choose the entry in $\hat{y_i}$ with the maximum value as the predicted label for $x_i$. Finally, the training accuracy is 85.77$\%$ and the test accuracy is 86.03$\%$.

\subsection{Logistic regression }
The model of logistic regression is slightly different from least square in the prediction and the loss function. With similar notations, we use $\hat{y}$=softmax($Wx+b$) as the predicted label of the input.
Here there is one more operation of softmax in the predicted label of an input $x$. It is used for normalizing the output to a probability distribution. Moreover, the loss function here is the cross entropy loss with an $\ell^2$ regularization term
\begin{align}
L_{\lambda}(\theta)=L(\theta)+\lambda R(\|\theta\|)=
-(y \log(\hat{y})+(1-y)\log(1-\hat{y}))
+\lambda \|\theta\|^2.
\end{align}
We use the {\color{red}Adam} optimization algorithm in the numerical experiment. We choose the learning rate to be 0.001,  batch size to be 100
and regularization coefficient to be 0.0001 for 50 training epochs. Finally, the training accuracy is 93.46$\%$ and the test accuracy is 92.56$\%$.

\subsection{Support vector machine (SVM) }
Finally, we use SVM for classification which we discussed in Section \ref{sec:SVMintro}. The main hypercoefficient is the kernel to be used in the algorithm. There are many kernels available such as linear kernel, Gaussian kernel, polynomial kernel and so on. We recall these kernels here.
\begin{itemize}
	\item Polynomial kernel: $k(x,y) = (a\langle x,y\rangle+ b)^n, a > 0, b\geq 0, n\in \mathbb{N}^+$.
	\item Gaussian kernel: $k(x,y) = e^{-\gamma\|x-y\|^2}, \gamma > 0$.
\end{itemize}
We choose the regularization parameter $\lambda=1$ in \eqref{SVM_Quad_soft} and use polynomial kernel
and Gaussian kernel as well as no kernel
for numerical experiments.
With no kernel, the training accuracy is 87.91$\%$ and the test accuracy is 87.73$\%$.
With polynomial kernel, the training accuracy is 99.16$\%$ and the test accuracy is 97.71$\%$.
With Gaussian kernel, the training accuracy is 98.99$\%$ and the test accuracy is 97.92$\%$.

%The training accuracy is 99.16$\%$ and the test accuracy is 97.71$\%$.

\subsection{Summary }
For the dataset of MNIST, the traditional linear machine learning models can achieve good classification accuracy.

  If we apply standard least square to the  MNIST dataset, we only get 86.03\% test accuracy. If we apply logistic regression to the  MNIST dataset, we get 92.56\% test accuracy. If we apply support vector machine (SVM) with polynomial kernel to the  MNIST dataset, we get 97.71\% test accuracy. If we apply support vector machine (SVM) with Gaussian kernel to the  MNIST dataset, we get 97.92\% test accuracy.

\begin{table}[!htbp]
	\begin{center}
		\begin{tabular}{|c|c|c|}
			\hline
			Model &  Training accuracy(\%)   &   Test accuracy(\%) \\
			\hline
			Least square & 85.77	           &   86.03 \\ \hline
			Logistic regression & 93.46   &   92.56	     \\ \hline
			SVM without kernel& 87.91           &   87.73         \\	\hline
			SVM with kernel& 99.16            &   97.71         \\	\hline
			CNN & 99.99            &   99.79         \\	\hline
		\end{tabular}
	\end{center}
	\caption{Training and test accuracy of different models on MNIST.}
	\label{tab:MNIST-LR-Results}
\end{table}

\begin{table}[!htbp]
	\begin{center}
		\begin{tabular}{|c|c|c|}
			\hline
			Model &  Training accuracy(\%)   &   Test accuracy(\%) \\
			\hline
			Least square & 45.10           &   32.86 \\ \hline
			Logistic regression & 39.25   &   35.23	     \\ \hline
			SVM without kernel& 33.02            &   23.76         \\	\hline
			SVM with kernel& 70.29            &   54.37         \\	\hline
			CNN & 99.97            &   95.05         \\	\hline
		\end{tabular}
	\end{center}
	\caption{Training and test accuracy of different models on CIFAR-10.}
	\label{tab:MNIST-LR-Results2}
\end{table}
In the meanwhile, the accuracy of the state-of-art CNN model (see more details in Section \ref{sec:CNNs})applied on the MNIST dataset is 99.79\%. We summarize the accuracy of these models on MNIST in the following Table \ref{tab:MNIST-LR-Results}.

In the following Table \ref{tab:MNIST-LR-Results2}, we summarize the accuracy of different models on CIFAR-10.

Based on the previous results, CNN models outperform all the other models significantly.
Thus we want to investigate the convolutional neural networks for better performance rather than the traditional machine learning models.


\subsubsection{Defective Matrix}
If $A$ has an eigenvalue $\lambda$ with multiplicity $k$ that does not generate $k$ corresponding linearly independent eigenvectors, the the matrix $A$ is defective. To generate enough vectors, we need to extend eigenvectors.
\begin{definition}
	Let $A$ be a square matrix. A nonzero vector $\vec{v}$ satisfying
	\begin{equation*}
		(A-\lambda I)^n \vec{v} = \vec{0}
	\end{equation*}
	for some eigenvalue $\lambda$ and some positive integer $n$ is a generalized eigenvector or rank $n$.
\end{definition}
These generalized eigenvectors also make up solutions.
\begin{theorem}
	Let $\vec{v}$ be an generalized eigenvector of a square matrix $A$ corresponding to an eigenvalue $\lambda$ with multiplicity $k$. Then
	\begin{equation*}
		e^{At}\vec{v} = e^{\lambda t}\left(\vec{v} + t(A-\lambda I)\vec{v} + \frac{t^2}{2!}(A-\lambda I)^2\vec{v} + \ldots\right)
	\end{equation*}
	is the corresponding fundamental solution. Further, the above sequence will terminate after $k$ or fewer terms (all subsequent terms are 0).
\end{theorem}

\ifodd\includeLinearSystemsExamples\input{./linearSystems/homogeneousSystems/defectiveMatrices_example.tex}\fi


\documentclass[
12pt, %font size
letterpaper, %for printing on American sized paper
fleqn, % for ligning up our equations
notitlepage % we don't want a title page
]{article}

\usepackage{amsmath,amsfonts,amssymb} % Math packages
\usepackage[letterpaper,margin=1in]{geometry} % Margins
\usepackage{setspace} % let's us choose between single or double spacing

\usepackage{sectsty} % Allows customizing section commands
\allsectionsfont{\normalfont\scshape} % Make all sections centered, the default font and small caps
\numberwithin{equation}{section} % Number equations within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{figure}{section} % Number figures within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{table}{section} % Number tables within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)

\setlength\parindent{0pt} % Removes all indentation from paragraphs - comment this line for an assignment with lots of text

\singlespacing
\frenchspacing % No extra spaces after sentances

\begin{document}

%------------------------------------------------
% title information
\title{\vspace{-2cm}Precalculus: Chapter 5 Notes}
\author{Amin Mesbah}
\date{}
\maketitle

%------------------------------------------------

\section{Exponents}

%------------------------------------------------

\subsection*{Laws of Exponents} % the asterisk ommits the section number
If $s$, $t$, $a$, and $b$ are real numbers with $a>0$ and $b>0$, then

% the align environment aligns things on multiple lines based on where I put an '&'
\begin{align}
	a^s \cdot a^t &= a^{s+t} &(a^s)^t &= a^{st} & (ab)^s &= a^s \cdot b^s \\
	1^s &= 1 & a^{-s} &= \frac{1}{a^s} = \left(\frac{1}{a}\right)^s & a^0 &= 1
\end{align}
% fractions are a bit tricky. You use them like this:
% \frac{numerator}{denominator}

%------------------------------------------------

\subsection*{Exponential Functions}
An \textbf{exponential function} is a function of the form

\begin{align}
	f(x) = Ca^x
\end{align}

where $a \in \mathbb{R}$, $a>0$, $a\neq1$, and $C\neq0$ is a real number. The domain of $f$ is $\mathbb{R}$. The base $a$ is the \textbf{growth factor}, and because $f(0) = Ca^0 = C$, we call $C$ the \textbf{initial value}.\\

For $f(x)=Ca^x$, where $a>0$ and $a\neq1$, if $x\in\mathbb{R}$, then

\begin{align}
	\frac{f(x+1)}{f(x)} &= a & \text{or} && f(x+1)&=af(x)
\end{align}

%------------------------------------------------

\subsection*{The number $e$}

\begin{align}
	e&=\lim_{n\to\infty}\left(1+\frac{1}{n}\right)^n
\end{align}

%------------------------------------------------

\subsection*{Solving Exponential Equations}

Use this property, expressing each side of the equation using the same base:

\begin{align}
	\text{If }a^u=a^v\text{, then }u=v
\end{align}

%------------------------------------------------

\section{Logarithms}

%------------------------------------------------

$\log_{a}x$ represents the exponent to which $a$ must be raised to obtain $x$.

\begin{align}
	y &=\log_{a}x\text{ if and only if }x=a^y \\
	y &=\ln{x}\text{ if and only if }x=e^y \\
	y &=\log{x}\text{ if and only if }x=10^y
\end{align}

The domain of $y=\log_{a}x$ is $\{x|x>0\}$.

%------------------------------------------------

\subsection*{Logarithmic Functions}

The logarithmic function is the inverse of the exponential function.

\begin{align}
	\text{If }f(x)=a^x\text{, then }f^{-1}(x)=\log_{a}x
\end{align}

The domain of $f^{-1}$ is the range of $f$. The range of $f^{-1}$ is the domain of $f$.

%------------------------------------------------

\subsection*{Properties of Logarithms}

In the following properties, $M$, $N$, and $a$ are positive real numbers, $a\neq1$, and $r \in \mathbb{R}$.

\begin{align}
	\log_{a}1 &= 0 & log_{a}a &= 1 \\
	a^{\log_{a}M} &= M & \log_{a}a^r &= r \\
	\log_{a}M^r &= r\log_{a}M & a^x &= e^{x\ln{a}}
\end{align}

\begin{align}
	\log_{a}(MN) &= \log_{a}M + \log_{a}N &
	\log_{a}\left(\frac{M}{N}\right) &= \log_{a}M - \log_{a}N
\end{align}

\begin{align}
	\text{If }M &= N\text{, then }\log_{a}M = \log_{a}N &
	\text{If }\log_{a}M &= \log_{a}N\text{, then }M = N
\end{align}

%------------------------------------------------

\subsection*{Change of Base Formula}

If $a\neq1$, $b\neq1$, and $M$ are positive real numbers, then

\begin{align}
	\log_{a}M &= \frac{\log_{b}M}{\log_{b}a} & \text{so,} &&
	\log_{a}M &= \frac{\log{M}}{\log{a}} & \text{and} && \log_{a}M &= \frac{\ln{M}}{\ln{a}}
\end{align}

%------------------------------------------------

\end{document}


\section*{Variables and constraints}
\subsection*{Input data}
$ |E| $ is the total number of exams.
\newline
$ |S| $ is the total number of enrolled students.
\newline
$ |T| $ is the total number of available timeslots.
\newline
$ \vec{ns} $ is a $ n \times n $ matrix, being $ n = |E| $, containing the number of students in conflit among two exams. More precisely, $ ns_{ij} $ contains the number of students in conflit among the exam $i$ and the exam $j$.
\newline
$ ns_{ij} \in \mathbb{N} \quad i = 1, \dots , |E| \quad j = 1, \dots , |E| $.
\newline
$ \vec{ns} $ is built directly from the input data.

\subsection*{Variables}
$ \vec{p} $ is a vector containing the penalty value associated to exams. More precisely, $ p_{k} $ contains the penalty weight associated to exams which are $k$ timeslots far, being $ k \in \mathbb{N}, 1 \le k \le 5 $. Hence $p_{k} = 2^{5-k}$.
\newline
$ \vec{y} $ is a 3d matrix containing the information about conflicts at a certain distance. More precisely,
\[
y_{ijk} =
\begin{cases}
1	&	\text{if exams $ i $ and $ j $ are $k$ timeslot far}	\\
0	&	\text{otherwise}
\end{cases}
\]

\subsection*{Output data}
$ \vec{x} $ is a vector containing the number of the timeslot assigned to an exam. More precisely, $ x_{i} $ contains the number of the timeslot assigned to the exam $i$. Hence $ 1 \le x_{i} \le |T| $.

\subsection*{Constraints}
$ ns_{ij} > 0 $ means that there are some students that can not substain the same exam in the same timeslot. Hence, it is needed to force $ x_{i} \ne x_{j} $.
In order to express this condition, it is necessary to introduce two new variables
\[
l_{ij} =
\begin{cases}
1	&	ns_{ij} > 0	\\
0	&	\text{otherwise}
\end{cases}
\]
\[
q_{ij} =
\begin{cases}
1	&	x_{i} \ne x_{j}	\\
0	&	\text{otherwise}
\end{cases}
\]
and to generate the following constraints:
\begin{gather*}
ns_{ij} \le M l_{ij} \qquad ns_{ij} \ge m l_{ij} \\
x_{i} - x_{j} \ge m q_{ij} \qquad x_{i} - x_{j} \le M q_{ij} \\
l_{ij} \le q_{ij}
\end{gather*}
\newline
It is needed to activate the proper $ y_{ijk} $ in order to represent the correct couple of exams in conflict at a certain distance.
\newline
$ x_{i} - x_{j} = k $ means that exam $i$ and exam $j$ are $k$ slots far. Hence, it is needed to force $ y_{ijk} = 1 $ being $ k \in \mathbb{N}, 1 \le k \le 5 $.
In order to express this condition, it is necessary to introduce two different blocks of constraints.

The first one forces $ y_{ijk} = 0 $ when $ x_{i} - x_{j} \ne k $ in order to "facilitate" the minimization of the objective function.
\begin{gather*}
k - (x_{i} - x_{j}) \le M (1-y_{ijk}) \\
(x_{i} - x_{j}) - k \le M (1-y_{ijk})
\end{gather*}

The second one forces $ y_{ijk} = 1 $ when $ x_{i} - x_{j} = k $.
In order to express this constrains it is needed to introduce two more boolean variables, $ y_{ijk, I} $ and $ y_{ijk, II} $.
\begin{gather}
\label{eqn:1}
(x_{i} - x_{j}) - k - \epsilon \ge (m - \epsilon) y_{ijk, I} \\
\label{eqn:2}
(x_{i} - x_{j}) - k + \epsilon \le (M + \epsilon) y_{ijk, II} \\
\label{eqn:3}
y_{ijk, I} + y_{ijk, II} - 1 \le y_{ijk}
\end{gather}
being $ \epsilon < 1 $ an arbitrarily small positive quantity.

With the relation \ref{eqn:1} the following constraint is expressed:
\[
y_{ijk, I} =
\begin{cases}
1	&	\text{if $ (x_{i} - x_{j}) \ge k $}	\\
0	&	\text{otherwise}
\end{cases}
\]
Analagously, with the relation \ref{eqn:2} the following constraint is expressed:
\[
y_{ijk, II} =
\begin{cases}
1	&	\text{if $ (x_{i} - x_{j}) \le k $}	\\
0	&	\text{otherwise}
\end{cases}
\]
Finally, the relation \ref{eqn:3} forces $ y_{ijk} = 1 $ only if $ y_{ijk, I} = 1 $ and $ y_{ijk, II} = 1 $, that represents the situation in which $ x_{i} - x_{j} = k $.