\newpage
%!TEX root = 497Notes-Temple.tex
\section{Batch Normalization in DNN and CNN}
Recall the classical (fully connected)  artificial deep neural network (DNN) $f^L$ in \eqref{compress-dnn}. We can define
%\begin{equation}\label{key}
%\mathcal{F}_\text{DNN}:= \{ f(x;\Theta) = \text{softmax} \circ f^J(x;\Theta) ~|~ x\in\mathbb{R}^n,\Theta \in \mathbb{R}^{\mathcal N} \}.
%\end{equation}
\begin{equation}\label{key}
\rm{DNN}_L:= \{  f^L(x;\Theta) ~|~ \Theta \in \mathbb{R}^{\mathcal N} \}.
\end{equation}
Or, we have a more comprehensive notation for classical DNN models.
\begin{equation}\label{eq:DNNdef_J}
\begin{aligned}
{\rm{DNN}_L} :=\{& f:f=
\theta^L \circ \sigma \circ \theta^{L-1} \cdots \sigma \circ \theta^1(x), \\
&\theta^\ell \in \mathbb{R}^{n^{\ell+1} \times (n^\ell+1)}, \quad n^0 = d, \quad n^{L} = 1, \quad n^\ell \in \mathbb{N}^+\}.
\end{aligned}
\end{equation}
Generally speaking, a deep learning problem consists of the next few components:
\begin{equation}\label{DL-process}
\text{Data}  \rightarrow \text{Model} \rightarrow \text{Training} \rightarrow \text{Testing}.
\end{equation}
We will try to give a different explanation for batch normalization based on the above outline.

\subsection{Ideas Behind the BN for DNN: Internal \mbox{Covariate} Shift in Training}
The { Internal Covariate Shift} is defined in \cite{ioffe2015batch} as
the change in the distribution of network activations
due to the change in network parameters during training.

For example, considering the training data with
$X=\{(x^i,y^i)~|~i=1,..,N\}\subset \mathcal{X} : =\{(x,y)~|~x,\in\mathbb{R}^n, y \in \mathbb{R}^c\}$,
then output of $\nu$-th layer (the input of activation function in $\nu+1$-the layer) $\{f^\nu ({x^i}) \}_{i=1}^N$
will satisfy different discrete distributions if you have different parameters $\Theta$ which
always happened during training process.

More intuitively, fixed distribution of inputs to a sub-network would have positive
consequences for the layers {\em outside} the sub-network, as
well. Consider a layer with a sigmoid activation function $f =
\sigma(W x+b)$ where $x$ is the layer input, the weight matrix $W$ and
bias vector $b$ are the layer parameters to be learned, and $\sigma(x) =
\frac{1}{1+e^{-x}}$.  It is easy to see
$$
\sigma'(x) \to 0 \quad \text{if} \quad |x| \to \infty.
$$
This means that for all dimensions of $y = Wx+b$ except those with
small absolute values, the gradient flowing down to $x$ will vanish
and the model will train slowly. However, since $y$ is affected by
$W, b$ and the parameters of all the layers below, changes to those
parameters during training will likely move many dimensions of $y$
into the saturated regime of the nonlinearity and slow down the
convergence. This effect is amplified as the network depth
increases. In practice, the saturation problem and the resulting
vanishing gradients are usually addressed by using ${\rm ReLU}(x)=\max(x,0)$, careful initialization and small learning rates.
However, if we could ensure that the distribution of nonlinearity inputs
remains more stable as the network trains, then the optimizer would be
less likely to get stuck in the saturated regime, and the training
would accelerate.

In summary, the gradient vanishes in the following two situations:
\begin{itemize}
	\item Saturated problem:
	$$
	\sigma'(x) \to 0 \quad \text{for $x$ located in some saturated domain}.
	$$
	\item Instability from multiplication:
	$$
	\nabla_{W^1}  \ell(f^L(x^i;\Theta), y^i) \approx \Pi_{\ell=1}^L W^{\ell} {\rm Diag}(\sigma'(f^{\ell-1})) \to 0~ , \quad \text{ if $L$ is big}.
	$$
\end{itemize}

Under this principle,
it has been long known \cite{lecun1998neural, wiesler2011convergence}
that the network training converges faster if its inputs are whitened, i.e.,
linearly transformed to have zero means and unit variances, and  decorrelated.

Thus to say, to improve the training,  one may seek to reduce the internal covariate shift.
By fixing the distribution of the layer inputs $f^\ell$ as the training progresses,
it is expected to improve the training speed. As each layer
observes the inputs produced by the layers below, it would be advantageous to
achieve the same whitening of the inputs of each layer.  By whitening the
inputs to each layer, BN would take a step towards achieving the fixed
distributions of inputs that would remove the ill effects of the internal covariate shift.

Within this framework, whitening the layer inputs is expensive, as it requires
computing the covariance matrix
\begin{equation}
{\rm Cov}[f^\ell] = \mathbb E_{ x \in X} [f^\ell(x) [f^\ell(x)]^T]-\mathbb E_{ x \in X} [f^{\ell}(x)]\mathbb E_{ x \in X} [f^\ell(x)]^T
\end{equation}
and its inverse square root, to produce the whitened activations
\begin{equation}
\tilde  f^\ell = \left({\rm Cov}[f^\ell \right)^{-1/2}\left(f^\ell-\mathbb E[f^\ell]\right),
\end{equation}
as well as the derivatives of these transforms for backpropagation.
However, this is impossible to take gradient for $\left({\rm Cov}[f^\ell] \right)^{-1/2}$
w.r.t $\Theta$ as $ f^\ell = f^\ell(x; \Theta)$.

The original version is to do whitening in Batch i.e. $\mathbb E_{ x \in X} [\cdot ]$, that. This
is why this is called Batch Normalization not mini-batch normalization which
is the practical version of batch normalization.

\subsection{Practical batch normalization: assume i.i.d and add scale and shift }
Since the full whitening of each layer's inputs is costly and not
everywhere differentiable, BN makes two necessary simplifications.

\paragraph{Take batch normalization for each scalar feature (neuron).}

The first is that instead of whitening the features in layer
inputs and outputs {\bf jointly}, batch normalization will normalize each {\bf scalar feature (neuron)}
independently, by making it have the mean of zero and the variance of
1.   Assume the distribution of
scalar features to be i.i.d.

For a layer with $n^\ell$-dimensional input $f^\ell = (f^\ell_1, \ldots,  f^\ell_{n^\ell})$, we
will normalize each dimension
$$ \hat{f}^\ell_k = \frac{f^\ell_k-\mathbb E [ f^\ell_k]}{
	\sqrt{{\rm Var}[f^\ell_k]}}$$
where the expectation and variance are
computed over the training data set. As shown in
\cite{lecun1998neural}, such normalization speeds up convergence,
even when the  features are not decorrelated.

\paragraph{Add scale and shift into batch normalization.}
Note that simply normalizing each input of a layer may change what the
layer can represent. For instance, normalizing the inputs of a
sigmoid would constrain them to the linear
regime of the nonlinearity. To address this, we make sure that {\em the transformation inserted in
	the network can represent the identity transform}.  To
accomplish this, we introduce, for each activation $\hat f^\ell_k$, a pair of
parameters $\gamma^\ell_k, \beta^\ell_k$, which scale and shift the
normalized value:
\begin{equation}
\tilde{f^\ell}_k  = \gamma^\ell_k \hat{f^\ell}_k +\beta^\ell_k.
\end{equation}
These parameters are learned along with the original model
parameters, and restore the representation power of the
network. Indeed, by setting $\gamma^\ell_k = \sqrt{{\rm Var}[f^\ell_k]}$ and
$\beta^\ell_k = \mathbb E [f^\ell_k]$, we could recover the original activations, if that were the optimal thing to do.

\subsection{Batch normalization for DNN}
\paragraph{Definition of batch normalization operation based on the batch}
Following the idea in \eqref{DL-process}, we consider that we have the all
training data as
\begin{equation}\label{eq:trainingdata}
(X,Y) := \{x^i, y^i\}_{i=1}^N.
\end{equation}
Since the normalization is applied to
each activation independently, let us focus on a particular activation $ f^\ell_k$ and omit $k$ as $f^\ell$ for clarity.
We have $N$ values of this activation
in the batch,
$$ X=\{x_1, \cdots, x_N\}.$$
Let the normalized values be
$\hat f^\ell$, and their linear transformations be $\tilde f^\ell$.
\begin{equation}\label{def:BNeq}
\begin{aligned}
\mu^\ell_{ X} & \leftarrow \mathbb{E}_{x \sim X} [f^\ell(x)] =  \frac{1}{N}\sum_{i=1}^N f^\ell(x^i) & \text{ batch mean}& \\
\sigma^\ell_{ X} & \leftarrow \mathbb{E}_{x \sim X}  \left[(f^\ell(x)-\mathbb{E}_{x \sim X}[ f^\ell(x)])^2 \right] = \frac{1}{N}\sum_{i=1}^N (f^\ell(x^i)-\mu_{ X}^\ell)^2 &  \text{ batch variance}&\\
\hat f^\ell (x) & \leftarrow \frac{f^\ell(x)-\mu^\ell_{ X}}{\sqrt{\sigma^\ell_{ X}+\epsilon}}   &\text{ normalize}&\\
\tilde f^\ell(x)  & \leftarrow \gamma^\ell \hat f^\ell (x) + \beta^\ell
&\text{ scale and shift}&
\end{aligned}
\end{equation}
Here we note that all these operations in the previous equation are defined  element-wise.
Then at last, we define the batch normalization operation based on the batch set as
\begin{equation}\label{eq:BNop}
{\rm BN}_{X}({f^\ell(x)}) = \tilde f^\ell(x) := \gamma^\ell  \frac{f^\ell(x)-\mu^\ell_{ X}}{\sqrt{\sigma^\ell_{ X}+\epsilon}}  + \beta^\ell  ,
\end{equation}
where $\tilde f^\ell(x)$, $\mu^\ell_{ X}$ and $\sigma^\ell_{ X}$  are given above.

\paragraph{Model with batch normalization}
Then we have the new DNN model with batch normalization as:
\begin{equation}\label{nn-BN0}
\begin{cases}
\tilde f^1(x^i)&= (\theta^1 (x^i) ),\\
\tilde f^\ell &= \theta^{\ell} \circ \sigma \circ {\rm BN}_{ X}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
%Here we would like to notice that all these input data
%$x_i \in X$, it is already normalized over all the original data set with
%the same fashion.
%Thai is to say
%$$
%x_i = {\rm BN}_{\bar X}(\bar x_i),
%$$
%for the original data $\bar x_i \in \bar X$.
For a more comprehensive notation,
we can use the next notation
\begin{equation}
\sigma_{\rm BN} := \sigma \circ {\rm BN}_{ X}.
\end{equation}
We can remove the basis $b^\ell$ in $\theta^\ell$, thus to say the real model we use should be
\begin{equation}\label{nn-BN}
\begin{cases}
\tilde f^1(x^i)&= W^1 x^i ,\\
\tilde f^\ell &= W^{\ell}  \sigma_{\rm BN}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
Combine the two definitions, we note
\begin{equation}
\tilde \Theta := \{W, \gamma, \beta\},
\end{equation}
where
$W = \{W^1, \cdots, W^l \}$, $\gamma := \{\gamma^2, \cdots, \gamma^L\}$ and $\beta := \{\beta^2, \cdots, \beta^L\}$.
Finally, we have the loss function
\begin{equation}\label{eq:loss-BN}
\mathcal L(\tilde \Theta) = \mathbb{E}_{(x,y)\sim (X,Y)} \approx \frac{1}{N}\sum_{i=1}^N \ell(\tilde f^L(x^i; \tilde \Theta), y^i).
\end{equation}
A key observation in \eqref{eq:loss-BN} and the new BN model \eqref{nn-BN} is that
\begin{equation}\label{eq:threeExpectation}
\begin{aligned}
\mu^\ell_{ X}
%& \leftarrow \frac{1}{N}\sum_{i=1}^N f^\ell(x_i)
&= \mathbb{E}_{x \sim X} [f^\ell(x)],\\
\sigma^\ell_{ X}
%& \leftarrow \frac{1}{N}\sum_{i=1}^N (f^\ell(x_i)-\mu_{ X})^2
&=  \mathbb{E}_{x \sim X}  \left[(f^\ell(x)-\mathbb{E}_{x \sim X}[ f^\ell(x)])^2 \right], \\
\mathcal L(\tilde \Theta)
%&= \sum_{i=1}^N L(\tilde f^J(x_i; \tilde \Theta), y_i)
&=  \mathbb{E}_{(x,y)\sim (X,Y)}  \left[\ell(\tilde f^L(x^i; \tilde \Theta), y^i) \right].
\end{aligned}
\end{equation}
Here we need to mention that
$$
x \sim X
$$
means $x$ subjects to the discrete distribution of all data $X$.
%For example:
%$$
%{p}_{X}(x = x_i) = \frac{1}{N}.
%$$
%or more mathematically we can use the Dirac distribution as:
%$$
%p_{X}(x) = \frac{1}{N} \sum_{i=1}^N\delta(x - x_i).
%$$

%Similar with the case from (batch) gradient descent to stochastic gradient descent.
%Therefore, BN makes the second simplification: since we use mini-batches
%in stochastic gradient training, {\em each mini-batch produces estimates of the mean and variance} of each
%activation for whole batch.

\subsection{Batch normalization: some ``modified" SGD training algorithm}
Following the key observation in \eqref{eq:threeExpectation}, and recall the
similar case in SGD, we do the the sampling trick in \eqref{eq:loss-BN} and
obtain the mini-batch SGD:
\begin{equation}\label{eq:mini-batch-sample}
x \sim X \approx x \sim \mathcal B,
\end{equation}
here $\mathcal B$ is a mini-batch of batch $X$ with $\mathcal B \subset X$.

However, for problem in \eqref{eq:loss-BN}, it is very difficult to find some
subtle sampling method because of the composition of $\mu^\ell_{\mathcal X}$
and $[\sigma^\ell_{\mathcal X}]$. However, one simple way for sampling
\eqref{eq:loss-BN} can be chosen as taking \eqref{eq:mini-batch-sample} for
all the expectation case in \eqref{eq:loss-BN} and \eqref{eq:threeExpectation}.
This is to say, in training process ($t$-th step for example), once we choose $B_t \subset X$
as the mini-batch, then the model becomes
\begin{equation}\label{nn-BN-training}
\begin{cases}
\tilde f^1(x^i)&= W^1 x^i ,\\
\tilde f^\ell &= W^{\ell}  \sigma_{\rm BN}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
where
\begin{equation}
\sigma_{\rm BN} := \sigma \circ {\rm BN}_{\mathcal B_t},
\end{equation}
or we can say that $X$ is replaced by $\mathcal B_t$ in this case.
Here ${\rm BN}_{\mathcal B_t}$ is defined by
\begin{equation}\label{def:BNeq-traning}
\begin{aligned}
\mu^\ell_{\mathcal B_t} & \leftarrow \frac{1}{m}\sum_{i=1}^m f^\ell(x^i) & \text{ mini-batch mean}& \\
\sigma^\ell_{\mathcal B_t} & \leftarrow \frac{1}{m}\sum_{i=1}^m (f^\ell(x^i)-\mu_{\mathcal B_t})^2 &  \text{ mini-batch variance}&\\
\hat f^\ell (x) & \leftarrow \frac{f^\ell(x)-\mu^\ell_{\mathcal B_t}}{\sqrt{\sigma^\ell_{\mathcal B_t}+\epsilon}}   &\text{ normalize}&\\
{\rm BN}_{\mathcal B_t}(\tilde f^{\ell}) := \tilde f^\ell(x) & \leftarrow \gamma^\ell \hat f^\ell (x) + \beta^\ell
&\text{ scale and shift}&
\end{aligned}
\end{equation}
Here batch normalization operation introduces some new parameters such as $\gamma$ and $\beta$.
Thus to say, for training phase, if we choose mini-batch as $\mathcal B_t$ in $t$-th training
step, we need to take gradient
\begin{equation}
\frac{1}{m}\nabla_{\tilde \Theta} \sum_{i \in \mathcal B_t} \ell(\tilde f^L(x^i; \tilde \Theta), y^i),
\end{equation}
which requires the gradient of $\mu_{B_t}^\ell$ or $[\sigma_{B_t}^\ell]$
w.r.t $W^i$ for $i \le \ell$.
To derive the new gradient formula for batch normalization step,
$$
\mu^\ell_{\mathcal B_t}, \quad \text{and} \quad \sigma^\ell_{\mathcal B_t},
$$
contain the output of $\tilde f^{\ell-1}$.
This is exact the batch normalization method described in \cite{ioffe2015batch}.

\subsection{Final model  with BN in DNN after training}
One key problem is that, in the batch normalization operator, we need to compute the mean and variance
in a data set (batch or mini-batch). However, in the inference step, we just input one data
into this DNN, how to compute the batch normalization operator in this situation.

Actually,  the $\gamma$ and $\beta$ parameter is fixed after training, the only problem is
to compute the mean $\mu$ and variance $\sigma$.
All the mean $ \mu_{\mathcal B_t}  $ and variance $\sigma_{\mathcal B_t} $ during the training phase
are just the approximation of the mean and variance of whole batch i.e.  $ \mu_{X}  $ and $\sigma_{X}$
as shown in \eqref{eq:threeExpectation}.

One natural idea might be just use the batch normalization operator w.r.t to the whole training data set, thus to say
just compute $\mu_X$ and $\sigma_X$ by definition in \eqref{def:BNeq}.

However, there are at least the next few problems:
\begin{itemize}
\item computation cost,
\item ignoring the statistical approximation (don't make use of the $ \mu_{\mathcal B_t}  $ and  $\sigma_{\mathcal B_t} $ in training phase).
\end{itemize}

Considering that we have the statistical approximation for $ \mu_{X}  $ and $\sigma_{X}$
during each SGD step, moving average might be a more straightforward way.
Thus to say, we define the $\mu^\ell$ and $[\sigma^\ell]$ for the inference (test) phase
as
\begin{equation}
\mu^\ell = \frac{1}{T}\sum_{t=1}^T \mu^\ell_{\mathcal B_t},
\quad \sigma^\ell =\frac{1}{T} \frac{m}{m-1}\sum_{t=1}^T\sigma^\ell_{\mathcal B_t}.
\end{equation}
Here we take Bessel's correction for unbiased variance.
The above moving average step is found in the original paper of batch normalization in \cite{ioffe2015batch}.

Another way to do this is to call the similar idea in momentum.
At each time step we update the running averages for mean and variance using an exponential decay based on the momentum parameter:
\begin{equation}
\begin{aligned}
\mu^\ell_{\mathcal B_t} &=\alpha \mu^\ell_{\mathcal B_{t-1}} + (1-\alpha)\mu^\ell_{\mathcal B_t} , \\
\sigma^\ell_{\mathcal B_t} &=\alpha \sigma^\ell_{\mathcal B_{t-1}} + (1-\alpha)\sigma^\ell_{\mathcal B_t} .
\end{aligned}
\end{equation}
$\alpha$ is close to $1$, we can take it as $0.9$ generally. Then we all take bath mean and variance as
$\mu^\ell_X  \approx \mu^\ell_{\mathcal B_T}$  and $\sigma^\ell_X  \approx \sigma^\ell_{\mathcal B_T}$.

Many people argue that the variance here should also use Bessel's correction.

\subsection{Batch Normalization for CNN}
One key idea in batch normalization is to do normalization with each scalar features (neurons)
separately along a mini-batch.
Thus to say, we need to identify what is neuron in CNN.
This is a historical problem, some people think neuron in CNN
should be the pixel in each channel, some other people think that each channel is just
one neuron. Batch normalization chooses the later one.
One important reason for this choice is the fact of computation cost.

For convolutional layers, batch normalization additionally requires the normalization
to  obey the convolutional property so that different elements
of the same feature map, at different locations, are normalized in the
same way.
To compute $\mu^\ell_{\mathcal B_t}$, we take mean of the set of all values in a feature map across both the
elements of a mini-batch and spatial locations, so for a mini-batch
of size $m$ and feature maps of size $m_\ell \times n_\ell$ (image geometrical size),
we use the effective mini-batch of size $ m m_\ell n_\ell$.
We learn a pair of parameters $\gamma_k$ and $\beta_k$ per feature map (k-th channel), rather than per activation.

For simplicity, then have the following batch normalization scheme for CNN
\begin{equation}\label{def:BNeq-traningCNN}
\begin{aligned}
\mu^{\ell, j}_{\mathcal B_t} & \leftarrow \frac{1}{m \times m_\ell \times n_\ell }\sum_{i=1}^m \sum_{1\le s\le m_\ell, 1\le t \le n_\ell} f^{\ell, j}_{s,t}(x^i)
 &&\text{ mean on channel }j \\
\sigma^{\ell, j}_{\mathcal B_t} & \leftarrow \frac{1}{m \times m_\ell \times n_\ell }\sum_{i=1}^m \sum_{1\le s\le m_\ell, 1\le t \le n_\ell}
 (f^{\ell, j}_{s,t}(x^i) - \mu^{\ell, j}_{\mathcal B_t})^2   &&\text{ variance on channel }j\\
\hat f^{\ell, j}_{s,t} (x) & \leftarrow \frac{f^{\ell, j}_{s, t}(x)] - \mu^{\ell, j}_{\mathcal B_t}}{\sqrt{\sigma^{\ell, j}_{\mathcal B_t} +\epsilon}}   &&\text{ normalize }\\
[{\rm BN}_{\mathcal B_t}(\tilde f^{\ell})]_{j;st} &:= \tilde f^{\ell, j}_{s,t}(x)  \leftarrow \gamma^{\ell,j} \hat f^{\ell, j}_{s,t} (x)  + \beta^{\ell,j}
&&\text{ scale and shift on channel}
\end{aligned}
\end{equation}

\subsection{Batch normalization for MgNet}
When we train MgNet, we also adopt the batch normalization mechanism.
This means that, when we do training for MgNet, we
apply the following basic block in MgNet
\begin{equation}\label{key}
u^{\ell,\nu} \leftarrow u^{\ell,\nu-1} + {\rm ReLU} \circ {\rm BN}_{\mathcal B_t} \circ B^{\ell,\nu}\ast {\rm ReLU} \circ {\rm BN}_{\mathcal B_t} ({f^\ell -  A^{\ell} \ast u^{\ell,\nu-1}}),
\end{equation}
or simply
\begin{equation}\label{key}
u^{\ell,\nu} \leftarrow u^{\ell,\nu-1} + {\rm ReLU}_{\rm BN} \circ B^{\ell,\nu}\ast {\rm ReLU}_{\rm BN} ({f^\ell -  A^{\ell} \ast u^{\ell,\nu-1}}),
\end{equation}
where
\begin{equation}\label{key}
{\rm ReLU}_{\rm BN} =  {\rm ReLU} \circ {\rm BN}_{\mathcal B_t}.
\end{equation}


\section*{Bayesian decision theory}
Given:\\
- Conditional distribution over labels $P(y|x)$\\
- Set of actions $\mathcal{A}$\\
- Cost function $C:Y\times \mathcal{A} \rightarrow \mathbb{R}$\\
Pick action that minimizes the expected cost:
$a^* = \underset{a \in \mathcal{A}}{\operatorname{argmin}} \mathbb{E}_y[C(y,a)|x] = \sum_y P(y|x) \cdot C(y,a)$
\subsection*{Optimal decision for logistic regression}
$a^* = \underset{y}{argmax} \hat{P}(y|x) = sign(w^T x)$

% \subsection*{Example: logistic regression}
% \begin{itemize}
% 	 Est. cond. dist: $P(y|x,w) = Ber(\sigma(w^Tx))$
% 	 Action set: $\mathcal{A} = \{ +1, -1\}$
% 	 Cost function: $C(y,a) = [y \not = a]$ \\
% 	$= \left \{
% 		\begin{array}{lr}
% 			1 \text{ , if } y \not = a\\
% 			0 \text{ , otherwise}
% 		\end{array}
% 		$
% \end{itemize}
% $a^* = \underset{a \in \mathcal{A}}{\operatorname{argmin}} \mathbb{E}_y[C(y,a)|x]$\\
% $= \underset{a \in \mathcal{A}}{\operatorname{argmin}} P(y=1|x,w)[a=1] + P(Y=-1|x,w)[a=+1]$\\
% $= \underset{a \in \mathcal{A}}{\operatorname{argmin}} P(y \not = a | x,w) = \underset{a \in \mathcal{A}}{\operatorname{argmin}} \frac{1}{1+exp(aw^Tx)}$\\
% $=\underset{a \in \mathcal{A}}{\operatorname{argmax}} (1 + exp(aw^Tx))$
% $=\underset{a \in \mathcal{A}}{\operatorname{argmax}} (aw^Tx)$\\
% $= sign (w^Tx)$

%\subsection*{Example: Asymmetric costs}
%	 Est. cond. dist: $\hat{P}(y|x,w) = Ber(\sigma(w^Tx))$
%	 Action set: $\mathcal{A} = \{ +1, -1\}$
%	 Cost function: $C(y,a) =
%	 \begin{cases}
%	 	c_{FP} \text{ , if $y=-1$ and $y=+1$}\\
%			c_{FN} \text{ , if $y=+1$ and $y=-1$}\\
%			0 \text{ , otherwise}
%	 \end{cases}
%		$
%The action that minimizes the expected cost is:\\
%$C_+ = \mathbb{E}_y[C(y,+1)|x] = P(y=+1|x) \cdot 0 + (P(y=-1)|x) \cdot c_{FP}$\\
%$C_- = \mathbb{E}_y[C(y,-1)|x] = P(y=+1|x) \cdot c_{FN} + P(y=-1|x) \cdot 0$\\
%Predict +1 if $C_+ \leq C_- \Leftrightarrow P(y=+1|x) \geq \frac{c_{FP}}{c_{FP} + c_{FN}}$

\subsection*{Doubtful logistic regression}
	 Est. cond. distr.: $\hat{P}(y|x) = Ber(y;\sigma(\hat{w}^Tx))$\\
	 Action set: $\mathcal{A} = \{ +1, -1, D\}$;  Cost function:\\
	 $C(y,a) = \begin{cases}
			[y \neq a] &\text{if } a \in \{+1,-1\}\\
			c &\text{if } a = D
       \end{cases}$\\
The action that minimizes the expected cost\\
$a^* = y \text{ if } \hat{P}(y|x) \geq 1-c\text{, D otherwise}$
%$a^* = \begin{cases}
%		y &\hat{P}(y|x) \geq 1-c\\
%		D &\text{otherwise}
%	   \end{cases}
%$

\subsection*{Linear regression}
	 Est. cond. distr.: $\hat{P}(y|x,w) = \mathcal{N}(y;w^Tx, \sigma^2)$\\
 $\mathcal{A} = \mathbb{R}$; $C(y,a) = (y-a)^2$\\
The action that minimizes the expected cost\\
$a^* = \mathbb{E}_y[y|x] = \int \hat{P}(y | x) \partial y = \hat{w}^Tx$

\subsection*{Asymmetric cost for regression}
	 Est. cond. distr.: $\hat{P}(y|x) = \mathcal{N}(\hat{y};\hat{w}^Tx, \sigma^2)$\\
	 $\mathcal{A} = \mathbb{R}$; $C(y,a) = c_1 \max(y-a,0) + c_2 \max(a-y,0)$
Action that minimizes the expected cost:\\
$a^* = \hat{w}^Tx + \sigma \Phi^{-1} (\frac{c_1}{c_1 + c_2})$, $\Phi$: Gaussian CDF\\
%$\frac{\partial}{\partial a}$\\
%$\frac{\partial}{\partial a} \mathbb{E}_y[C(y,a)|x] = \int_{-\infty}^{\infty} C(y,a) P(y | x) dy \overset{!}{=} 0$\\
%$= -c_1 \int_{a}^{\infty} P(y | x) dy + c_2 \int_{-\infty}^{a} P(y | x) dy$\\
%$= -c_1 [1-\phi(a; w^Tx, \sigma^2)] + c_2 \phi(a; w^Tx, \sigma^2)$\\
%$\phi(a; w^Tx, \sigma^2) = \frac{c_1}{c_1 + c_2}$\\
%using $\phi(u;v,w) = \phi((u-v)/\sqrt{w};0,1)$ and applying inverse CDF of std. ND $\phi^{-1}$ we get\\
%$a^* = w^Tx + \sigma\phi^{-1} (\frac{c_1}{c_1 + c_2})$


\subsection{Intervals of Increase/Decrease, and the First Derivative Test}\label{sec:FirstDer}
The method of Section~\ref{subsec:LocalExtremaSubsection} for deciding whether there is a
local maximum or minimum at a critical value is not always
convenient. We can instead use information about the derivative
$f'(x)$ to decide; since we have already had to compute the derivative
to find the critical values, there is often relatively little extra
work involved in this method.

How can the derivative tell us whether there is a maximum, minimum, or
neither at a point? Suppose that $f$ is differentiable at and around $x=a$, and suppose further that $a$ is a critical point of $f$. Then we have several possibilities:
\begin{enumerate}
\item	There is a local maximum at $x=a$. This happens if $f'(x)>0$ as we approach $x=a$ from the left (i.e. when $x$ is in the vicinity of $a$, and $x<a$) and $f'(x)<0$ as we move to the right of $x=a$ (i.e. when $x$ is in the vicinity of $a$, and $x>a$).
\item	There is a local minimum at $x=a$. This happens if $f'(x)<0$ as we approach $x=a$ from the left (i.e. when $x$ is in the vicinity of $a$, and $x<a$) and $f'(x)>0$ as we move to the right of $x=a$ (i.e. when $x$ is in the vicinity of $a$, and $x>a$).
\item	There is neither a local maximum or local minimum at $x=a$. If $f'(x)$ does not change from negative to positive, or from positive to negative, as we move from the left of $x=a$ to the right of $x=a$ (that is, $f'(x)$ is positive on both sides of $x=a$, or negative on both sides of $x=a$) then there is neither a maximum nor minimum when $x=a$.
\end{enumerate}
See the first graph in Figure~\ref{fig:max and min points}
and the graph in Figure~\ref{fig:non extremum}
for examples.

\begin{example}{Local Maximum and Minimum}{localmaxandmin}
Find all local maximum and minimum points for $f(x)=\sin x+\cos
x$ using the first derivative test.
\end{example}

\begin{solution}
The derivative is $f'(x)=\cos
x-\sin x$ and from Example~\ref{max and min} the critical
values we need to consider are $\pi/4$ and $5\pi/4$.

We analyze the graphs of $\sin x$ and $\cos x$.
Just to the left of $\pi/4$ the cosine is larger than the
  sine, so $f'(x)$ is positive; just to the right the cosine is
  smaller than the sine, so $f'(x)$ is negative. This means there is a
  local maximum at $\pi/4$. Just to the left of $5\pi/4$ the cosine is
  smaller than the sine, and to the right the cosine is larger than
  the sine. This means that the derivative $f'(x)$ is negative to the
  left and positive to the right, so $f$ has a local minimum at
  $5\pi/4$.
\end{solution}

The above observations have obvious intuitive appeal as
you examine the graphs in Figures~\ref{fig:max and min points} and \ref{fig:non extremum}. We can extend these ideas
further and then formulate and prove a theorem: If the graph of $f$ is
increasing before (i.e., to the left of) $x=a$ and decreasing after (i.e.,
to the right of) $x=a,$ then there is a local maximum at $x=a.$ If the graph
of $f$ is decreasing before $x=a$ and increasing after $x=a,$ then there is
a local minimum at $x=a.$ If the graph of $f$ is consistently increasing on
either side of $x=a$ or consistently decreasing on either side of $x=a,$
then there is neither a local maximum nor a local minimum at $x=a.$ We can
prove the following theorem using the Mean Value Theorem.

\begin{theorem}{Intervals of Increase and Decrease}{IntervalsIncDecTheorem}
If $f^{\prime }\left( x\right) >0$ for every $x$ in an interval, then $f$ is
increasing on that interval.

If $f^{\prime }\left( x\right) <0$ for every $x$ in an interval, then $f$ is
decreasing on that interval.
\end{theorem}
\begin{proof}
We will prove the increasing case. The proof of the decreasing case
is similar. Suppose that $f^{\prime }\left( x\right) >0$ on an interval $I.$
Then $f$ is differentiable, and hence also, continuous on $I.$ If $x_{1}$
and $x_{2}$ are any two numbers in $I$ and $x_{1}<x_{2},$ then $f$ is
continuous on $\left[ x_{1},x_{2}\right] $ and differentiable on $\left(
x_{1},x_{2}\right) .$ By the Mean Value Theorem, there is some $c$ in $%
\left( x_{1},x_{2}\right) $ such that
\begin{equation*}
f^{\prime }\left( c\right) =\frac{f\left( x_{2}\right) -f\left( x_{1}\right)
}{x_{2}-x_{1}}.
\end{equation*}%
But $c$ must be in $I,$ and thus, since $f^{\prime }\left( x\right) >0$ for
every $x$ in $I,$ $f^{\prime }\left( c\right) >0$. Also, since $x_{1}<x_{2},$
we have $x_{2}-x_{1}>0.$ Therefore, both the left hand side and the
denominator of the right hand side are positive. It follows that the
numerator of the right hand must be positive. That is, $f\left( x_{2}\right)
-f\left( x_{1}\right) >0,$ or in other words, $f\left( x_{1}\right) <f\left(
x_{2}\right) .$ This shows that between $x_{1}$ and $x_{2}$ in $I,$ the
larger one, $x_{2},$ necessarily has the larger function value, $f\left(
x_{2}\right) ,$ and the smaller one, $x_{1},$ necessarily have the smaller
function value, $f\left( x_{1}\right) .$ This means that $f$ is increasing
on $I$.
\end{proof}

\begin{example}{Local Minimum and Maximum}{LocalMaxMinExample}
Consider the function $f\left( x\right) =x^{4}-2x^{2}.$ Find
where $f$ is increasing and where $f$ is decreasing. Use this information to
find the local maximum and minimum points of $f$.
\end{example}
\begin{solution}
We compute $f^{\prime }\left( x\right) $ and analyze its sign.
\begin{equation*}
f^{\prime }\left( x\right) =4x^{3}-4x=4x\left( x^{2}-1\right) =4x\left(
x-1\right) \left( x+1\right) .
\end{equation*}%
The solution of the inequality $f^{\prime }\left( x\right) >0$ is $\left(
-1,0\right) \cup \left( 1,\infty \right) $. So, $f$ is increasing on the
interval $\left( -1,0\right) $ and on the interval $\left( 1,\infty \right)
. $ The solution of the inequality $f^{\prime }\left( x\right) <0$ is $%
\left( -\infty ,-1\right) \cup \left( 0,1\right) .$ So, $f$ is decreasing on
the interval $\left( -\infty ,-1\right) $ and on the interval $\left(
0,1\right) .$ Therefore, at the critical points $-1,$ $0$ and $1,$
respectively, $f$ has a local minimum, a local maximum and a local minimum.
\end{solution}

%\figure[!ht]
%\vbox{\beginpicture
%\normalgraphs
%%\ninepoint
%\setcoordinatesystem units <1.5truecm,1.5truecm>
%\setplotarea x from 0 to 6.28, y from -1 to 1
%\axis left shiftedto x=0 /
%\axis bottom shiftedto y=0 ticks withvalues {${\pi\over4}$}
%      {$5\pi\over4$} / at 0.7853981635 3.926990818 / /
%\setquadratic
%\plot
%0.000 0.000
%0.063 0.063 0.126 0.125 0.188 0.187 0.251 0.249
%0.314 0.309 0.377 0.368 0.440 0.426 0.503 0.482 0.565 0.536
%0.628 0.588 0.691 0.637 0.754 0.685 0.817 0.729 0.880 0.771
%0.942 0.809 1.005 0.844 1.068 0.876 1.131 0.905 1.194 0.930
%1.257 0.951 1.319 0.969 1.382 0.982 1.445 0.992 1.508 0.998
%1.571 1.000 1.634 0.998 1.696 0.992 1.759 0.982 1.822 0.969
%1.885 0.951 1.948 0.930 2.011 0.905 2.073 0.876 2.136 0.844
%2.199 0.809 2.262 0.771 2.325 0.729 2.388 0.685 2.450 0.637
%2.513 0.588 2.576 0.536 2.639 0.482 2.702 0.426 2.765 0.368
%2.827 0.309 2.890 0.249 2.953 0.187 3.016 0.125 3.079 0.063
%3.142 0.000 3.204 -0.063 3.267 -0.125 3.330 -0.187 3.393 -0.249
%3.456 -0.309 3.519 -0.368 3.581 -0.426 3.644 -0.482 3.707 -0.536
%3.770 -0.588 3.833 -0.637 3.896 -0.685 3.958 -0.729 4.021 -0.771
%4.084 -0.809 4.147 -0.844 4.210 -0.876 4.273 -0.905 4.335 -0.930
%4.398 -0.951 4.461 -0.969 4.524 -0.982 4.587 -0.992 4.650 -0.998
%4.712 -1.000 4.775 -0.998 4.838 -0.992 4.901 -0.982 4.964 -0.969
%5.027 -0.951 5.089 -0.930 5.152 -0.905 5.215 -0.876 5.278 -0.844
%5.341 -0.809 5.404 -0.771 5.466 -0.729 5.529 -0.685 5.592 -0.637
%5.655 -0.588 5.718 -0.536 5.781 -0.482 5.843 -0.426 5.906 -0.368
%5.969 -0.309 6.032 -0.249 6.095 -0.187 6.158 -0.125 6.220 -0.063
%6.283 0.000 /
%\plot
%0.000 1.000 0.063 0.998 0.126 0.992 0.188 0.982 0.251 0.969
%0.314 0.951 0.377 0.930 0.440 0.905 0.503 0.876 0.565 0.844
%0.628 0.809 0.691 0.771 0.754 0.729 0.817 0.685 0.880 0.637
%0.942 0.588 1.005 0.536 1.068 0.482 1.131 0.426 1.194 0.368
%1.257 0.309 1.319 0.249 1.382 0.187 1.445 0.125 1.508 0.063
%1.571 0.000 1.634 -0.063 1.696 -0.125 1.759 -0.187 1.822 -0.249
%1.885 -0.309 1.948 -0.368 2.011 -0.426 2.073 -0.482 2.136 -0.536
%2.199 -0.588 2.262 -0.637 2.325 -0.685 2.388 -0.729 2.450 -0.771
%2.513 -0.809 2.576 -0.844 2.639 -0.876 2.702 -0.905 2.765 -0.930
%2.827 -0.951 2.890 -0.969 2.953 -0.982 3.016 -0.992 3.079 -0.998
%3.142 -1.000 3.204 -0.998 3.267 -0.992 3.330 -0.982 3.393 -0.969
%3.456 -0.951 3.519 -0.930 3.581 -0.905 3.644 -0.876 3.707 -0.844
%3.770 -0.809 3.833 -0.771 3.896 -0.729 3.958 -0.685 4.021 -0.637
%4.084 -0.588 4.147 -0.536 4.210 -0.482 4.273 -0.426 4.335 -0.368
%4.398 -0.309 4.461 -0.249 4.524 -0.187 4.587 -0.125 4.650 -0.063
%4.712 0.000 4.775 0.063 4.838 0.125 4.901 0.187 4.964 0.249
%5.027 0.309 5.089 0.368 5.152 0.426 5.215 0.482 5.278 0.536
%5.341 0.588 5.404 0.637 5.466 0.685 5.529 0.729 5.592 0.771
%5.655 0.809 5.718 0.844 5.781 0.876 5.843 0.905 5.906 0.930
%5.969 0.951 6.032 0.969 6.095 0.982 6.158 0.992 6.220 0.998
%6.283 1.000 /
%\endpicture}
%\label{fig:sin and cos}
%\caption{The sine and cosine.}
%\endfigure

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:FirstDer}}

\begin{enumialphparenastyle}

Find all critical points and identify them as local maximum points, local minimum points, or neither.

%%%%%%%%%%
\begin{ex}
 $\ds y=x^2-x$
\begin{sol}
 min at $x=1/2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=2+3x-x^3$
\begin{sol}
 min at $x=-1$, max at $x=1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^3-9x^2+24x$
\begin{sol}
 max at $x=2$, min at $x=4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^4-2x^2+3$
\begin{sol}
 min at $x=\pm 1$, max at $x=0$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^4-4x^3$
\begin{sol}
 min at $x=1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=(x^2-1)/x$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^2-(1/x^2)$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $y=\cos(2x)-x$
\begin{sol}
 min at $x=7\pi/12+k\pi$, max at $x=-\pi/12+k\pi$, for integer $k$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
$\ds f(x) = (5-x)/(x+2)$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds f(x) = |x^2 - 121|$
\begin{sol}
 max at $x=0$, min at $x=\pm 11$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds f(x) = x^3/(x+1)$
\begin{sol}
 min at $x=-3/2$, neither at $x=0$
\end{sol}
\end{ex}

%%%%%%%%%%%
%\begin{ex}
% $\ds f(x)= \cases{x^2 \sin(1/x)  & $x\neq 0$ \cr
% 0  & $x=0$\cr}$

%%%%%%%%%%
\begin{ex}
 $\ds f(x) = \sin ^2 x$
\begin{sol}
 min at $n\pi$, max at $\pi/2+n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the maxima and minima of $f(x)=\sec x$.
\begin{sol}
 min at $2n\pi$, max at $(2n+1)\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
  Let $\ds f(\theta) = \cos^2(\theta) -
 2\sin(\theta)$.  Find the intervals where $f$ is increasing and the
 intervals where $f$ is decreasing in $[0,2\pi]$.  Use this
 information to classify the critical points of $f$ as either local
 maximums, local minimums, or neither.
\begin{sol}
 min at $\pi/2+2n\pi$, max at $3\pi/2+2n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Let $r>0$. Find the local
maxima and minima of the function $\ds f(x)
=\sqrt{r^2 -x^2 }$ on its domain $[-r,r]$.
\end{ex}

%%%%%%%%%%
\begin{ex}
 Let $\ds f(x) =a x^2 + bx + c$ with $a\neq 0$. Show that $f$
has exactly one critical point using the first derivative test. Give
conditions on $a$ and $b$ which guarantee that the critical point will
be a maximum. It is possible to see this without using calculus at
all; explain.
\end{ex}

\end{enumialphparenastyle}


\subsection{Fibonacci sequence}

\subsubsection{The generating function}

Let's use a generating function to create a function for the Fibonacci sequence's \(c\)th digit.
\(F(c)=\sum_{i=c} x^is_i\)

Let's look at it for other starts:

\(F(c+k)=\sum_{i=c} x^{i+k}s_{i+k}\)

\(F(c+k)=\sum_{i=c+k} x^is_i\)

\(F(c+1)=\sum_{i=c} x^{i+1}s_{i+1}\)

\(F(c+2)=\sum_{i=c} x^{i+2}s_{i+2}\)

This means

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^i s_i x^2 +\sum_{i=c} x^{i+1} s_{i+1} x\)

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^{i+2}s_i+\sum_{i=c} x^{i+2}s_{i+1}\)

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^{i+2}(s_i+s_{i+1})\)
\subsubsection{Using the definiton of the Fibonacci sequence}

From the definition of the fibonacci sequence, \(s_{i}+s_{i+1}=s_{i+2}\).

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^{i+2}(s_{i+2})\)

\(F(c)x^2+F(c+1)x=F(c+2)\)

\subsubsection{Reducing the functions}

Next, we expand out \(F(c+1)\) and \(F(c+2)\).

\(F(c)-F(c+k)=\sum_{i=c} x^i s_i -\sum_{i=c+k} x^i s_i\)

\(F(c)-F(c+k)=\sum^{c+k}_{i=c} x^i s_i\)

\(F(c+k)=F(c)-\sum^{c+k}_{i=c} x^i s_i\)

So:

\(F(c+1)=F(c)-\sum^{c+1}_{i=c} x^i s_i\)

\(F(c+1)=F(c)-x^c s_c\)

\(F(c+2)=F(c)-\sum^{c+2}_{i=c} x^i s_i\)

\(F(c+2)=F(c)-x^{c+1}s_{c+1}-x^c s_c\)

Let's take our previous equation

\(F(c)x^2+F(c+1)x=F(c+2)\)

\(F(c)x^2+[F(c)-x^c s_c]x=F(c)-x^{c+1}s_{c+1}-x^c s_c\)

\(F(c)x^2+F(c)x-x^{c+1} s_c=F(c)-x^{c+1}s_{c+1}-x^c s_c\)

\(F(c)[x^2+x-1]=x^{c+1}s_c-x^{c+1}s_{c+1}-x^c s_c\)

\(F(c)=\dfrac{x^c s_c + x^{c+1}s_{c+1}-x^{c+1}s_c}{1-x-x^2}\)

\subsubsection{Using the first element in the sequence}

For the start of the sequence, \(c=0\), \(s_0=s_1=1\).

\(F(0)=\dfrac{x^0 1 + x - x}{1-x-x^2}\)

\(F(0)=\dfrac{1}{1-x-x^2}\)

Let's factorise this:

\(F(0)=\dfrac{-1}{(x+\dfrac{1}{2}+\dfrac{\sqrt 5}{2})(x+\dfrac{1}{2}-\dfrac{\sqrt 5}{2})}\)

We can then use partial fraction decomposition

\(\dfrac{1}{(M+\delta)(M-\delta)}=\dfrac{1}{2\delta}[\dfrac{1}{M-\delta}-\dfrac{1}{M+\delta}]\)

To show that

\(F(0)=\dfrac{-1}{\sqrt 5}[\dfrac{1}{x+\dfrac{1}{2}-\dfrac{\sqrt 5}{2}}-\dfrac{1}{x+\dfrac{1}{2}+\dfrac{\sqrt 5}{2}}]\)

\(F(0)=\dfrac{-1}{\sqrt 5}[\dfrac{\dfrac{1}{2}+\dfrac{\sqrt 5}{2}}{(x+\dfrac{1}{2}-\dfrac{\sqrt 5}{2})(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})}-\dfrac{\dfrac{1}{2}-\dfrac{\sqrt 5}{2}}{(x+\dfrac{1}{2}+\dfrac{\sqrt 5}{2})(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})}]\)

\(F(0)=\dfrac{-1}{\sqrt 5}[\dfrac{\dfrac{1}{2}+\dfrac{\sqrt 5}{2}}{x(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})-1}-\dfrac{\dfrac{1}{2}-\dfrac{\sqrt 5}{2}}{x(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})-1}]\)

\(F(0)=\dfrac{1}{\sqrt 5}[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})\dfrac{1}{1-x(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})}-(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})\dfrac{1}{1-x(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})}]\)

\subsubsection{Finishing off}

As we know

\(\dfrac{1}{1-x}=\sum_{i=0}x^i \)

So

\(F(0)=\dfrac{1}{\sqrt 5}[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})\sum_{i=0} x^i(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^i -(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})\sum_{i=0}x^i(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^i]\)

\(F(0)=\dfrac{1}{\sqrt 5}[\sum_{i=0} x^i(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^{i+1} -\sum_{i=0}x^i(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^{i+1}]\)

\(F(0)=\dfrac{1}{\sqrt 5}\sum_{i=0} x^i[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^{i+1} -(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^{i+1}]\)

So the \(n\)th number in the sequence (treating \(n=1\) as the first number) is:

\(\dfrac{1}{\sqrt 5}[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^{n} -(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^{n}]\)


\subsection{Open Delta Neighborhoods}
\noindent
An open delta neighborhood of a point $x_0$ is defined as the set
\begin{equation*}
	N\left(x_0, \delta\right) = \left\{x \in \mathbb{R}^n \mid \norm{x-x_0} < \delta\right\}.
\end{equation*}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{./differentialMultivariableCalculus/open_delta.png}
	\caption{An open delta neighborhood centered at $(x_0, y_0)$}
\end{figure}

\noindent
This simply means all points less than a distance $\delta$ away from point $x_0$.\\
For example, $N( (1,2) , 7) = \left\{ (x,y) \mid \sqrt{(x-1)^2 + (y-2)^2}<7 \right\}$, which is a ball (filled-in circle) of radius 7 centered at $(1, 2)$.


\subsection{Spherical coordinate system}
A sphere with center $O=(x_0,y_0,z_0)$ and radius $r$ is the set of points at distance exactly $r$ from $O$.
We can describe it this by equation
\[(x-x_0)^2+(y-y_0)^2+(z-z_0)^2 = r^2\]

\centerFig{sph-0}

To describe a point on a sphere, we can either directly give its coordinates $x,y,z$ or use the spherical coordinate system: this is the system used to position locations on Earth. We use an angle $\varphi$, the latitude, which tells us how far North the point is (or South, if $\varphi<0$), and an angle $\lambda$, the longitude, which tells us how far East the point is (or West, if $\lambda<0)$. We usually take $\varphi \in [-\frac{\pi}{2},\frac{\pi}{2}]$ and $\lambda \in (-\pi,\pi]$.

\centerFig{sph-1}

If the sphere is centered at the origin, the position represented by coordinates $(\varphi,\lambda)$ is
\[(r\cos\varphi\cos\lambda, r\cos\varphi\sin\lambda, r\sin\varphi)\]
where the $z$-axis points North and the $x$-axis points towards meridian $\lambda=0$ (on Earth, the Greenwich meridian).
%Note that this also gives a parametric representation of the sphere.

The function below finds the position given angles in degrees:
\begin{lstlisting}
p3 sph(double r, double lat, double lon) {
    lat *= M_PI/180, lon *= M_PI/180;
    return {r*cos(lat)*cos(lon), r*cos(lat)*sin(lon), r*sin(lat)};
}
\end{lstlisting}

\subsection{Sphere-line intersection}
A sphere $(O,r)$ and a line $l$ have either 0, 1, or 2 intersection points.

\centerFig{sph-2}

Finding them is exactly like circle-line intersection: first compute the projection $P$ of the center onto the line, then find the intersections by moving the appropriate distance forward or backward along $l$.

\centerFig{sph-3}

This function returns the number of intersection points and places them in pair \lstinline|out| if they exist.
\begin{lstlisting}
int sphereLine(p3 o, double r, line3d l, pair<p3,p3> &out) {
    double h2 = r*r - l.sqDist(o);
    if (h2 < 0) return 0; // the line doesn't touch the sphere
    p3 p = l.proj(o); // point P
    p3 h = l.d*sqrt(h2)/abs(l.d); // vector parallel to l, of length h
    out = {p-h, p+h};
    return 1 + (h2 > 0);
}
\end{lstlisting}


\subsection{Maximum A-Priori (MAP) estimation}

Mode estimate

\(Arg max_\theta p(\theta | X)\)

Using Bayes theorem:

\(P(\theta | X)= \dfrac{P(X|\theta )P(\theta)}{P(X)}\)

So:

\(P(\theta | X)= \dfrac{P(X|\theta )P(\theta)}{P(X)}\)

\(Argmax_\theta p(\theta | X)=Argmax_\theta \dfrac{p(X|\theta )P(\theta)}{P(X)}\)

The denominator isn't affected so:

\(Arg max_\theta p(\theta | X)=Arg max_\theta p(X|\theta )P(\theta)\)

If \(P(\theta )\) is a constant then this is the same as the MLE estimator.

\subsubsection{Other}

\(Argmax_\theta p(\theta|X)\)

Mode estimate

\(p(\theta|X)= \dfrac{p(X| \theta)p(\theta )}{p(X)}\)

\(Argmax_\theta \dfrac{p(X| \theta)p(\theta )}{p(X)}\)

\(\theta \) doesn't change denominator so can instead use:

\(Argmax_\theta p(X| \theta)p(\theta )\)

It is the same as maximum likelihood estimator if \(p(\theta )\) is a constant.


\lab{Application}{Nearest Neighbor Search}{Nearest Neighbor Search}
\label{Ch:NNS}

\objective{Teach about branch and bound and the curse of dimensionality using the nearest neighbor search problem.}

\section*{The Nearest Neighbor Search Problem}

You move into a city that has several post offices.
You want to know which one is the closest.
This problem is  known as the nearest neighbor search problem or the post-office problem.
The general problem is to find the closest of a set of points to any new point.

This has many applications which include computer vision, pattern recognition, internet marketing and data compression.

The naive way to solve this problem is to check the distance of all the data against the point.

\begin{problem}
Write a function that solves the nearest neighbor search problem by exhaustively checking all the distances.
The function should take in the set of points that is the data and a single point.
The output should be the distance to the closest data point and the index of that point.
Your function should be able to take in data in an arbitrary dimension.
\end{problem}

The complexity of this algorithm is $O(kn)$.
Where  is the $k$ number of dimensions and $n$ is the number of data points.

\section*{K-D Trees}

A faster way to solve this problem is to build a k-d tree and search the k-d tree for the nearest neighbor.

A k-d tree is a binary tree where the nodes to the left of parent node have a lower value in the i-th dimension and the nodes to the right of the parent node have a greater value in the i-th dimension.
Which dimension you split the nodes alternates at different levels.
In the $3$ dimensional case the root node is divided in the $x$ dimension, children in the $y$ dimension, grandchildren in the $z$ dimension, and the great-grandchildren in the $x$ dimension and so on.
Each node stores its location, left child and right child.
This requires sorting at each level so the complexity is $O(n log^2(n))$, but we only need to build the k-d tree once and after that we can query it as many times as we want.

Included is a function that takes in a set of data and builds k-d tree.
The leaf nodes' children are  python's ``None" object.

TODO: revise algorithm description

The search of the tree is done recursively.
We will call the node that we are on the parent node.
We first see if the distance between the parent node at the less is current best.
If so, we update it.
We then compare the values in the i-th dimension of the point and the parent.
If it is point's value is less than the parent we recursively go down the left child, greater, the right child.
Then we have to check if the hypersphere around the point with radius being the current best distance crosses the dividing hyperplane created by the parent.
If the point's value is less than the parent, we do this by checking to see if the value in the i-th dimension plus the current best distance is greater than the parent's value in the i-th dimension.
If the point's value is greater than the parent, we do this by checking to see if the value n the i-th dimension minus the current best distance is less than the parent's value in the i-th dimension.
If this be the case, we recursively go down the other child as well.

\begin{problem}
Write a function that solves the nearest neighbor search problem by searching through a k-d tree.
The function should take in k-d tree and a single point.
The output should be the distance to the closest data point and the coordinates of that point.
\end{problem}

\begin{problem}
Time both the functions you have created with the number of data points being $10,000-100,000$ every multiple of $10,000$ with $4$ dimensions.
Time only the searching of the k-d tree not the building of it.
Plot both times on the same plot.
How do the two algorithms compare?
\end{problem}

\begin{figure}[H]
\includegraphics[scale = .5]{4dTime.pdf}
\caption{Your graph should look like this.
The green line is the naive version and blue line is using a kd-tree.
As you can see the kd-tree is much faster.}
\end{figure}

The complexity of this algorithm is $O(klog(n))$ in optimal time.
Its worst case is $O(k*n^{1-\frac{1}{k}})$ where $k$ is the number of dimensions and $n$ is the number of points in the tree.
The reasons for this are discussed in the next section.

\section*{Curse of Dimensionality}

As you increase the number of dimensions the number of times that you have to go down both branches increases.
You get to the point where you eliminate very few points by using a k-d tree.

\begin{problem}
Time both algorithms for the number of data points being $10,000-100,000$ every multiple of $10,000$ with $20$ dimensions.
Plot both times on the same plot. Now how do the two algorithms compare?
\end{problem}

\begin{figure}[H]
\includegraphics[scale = .5]{20dTime.pdf}
\caption{
Your graph should look like this.
The green line is the naive version and blue line is using a kd-tree.
As you can see the kd-tree is slightly slower than the naive version.}
\end{figure}

\begin{problem}
Time the SciPy built in function for searching a k-d tree (do not time the building of the kd-tree) for the number of dimensions points being $2-50$ with $20,000$ data points.
Plot the time.
What do you notice?
\li{from scipy.spatial import KDTree} will import the built in k-d tree.
Create the tree by \li{tree = KDTree(data)} and search it by \li{tree.query(point)}.

\li{scipy.spatial} includes a \li{cKDTree} object which is implemented in C.
Is it any better?

\end{problem}
\begin{figure}[h!]
\includegraphics[scale = .5]{curseD.pdf}
\caption{
Your graph should look similar to this.
Around 15 dimensions the time jumps.
At that point using a kd-tree is not more effective than checking all the combinations.}
\end{figure}


\subsection{k-medoids}

\subsubsection{Introduction}

k-mediods is similar to k-means clustering, with two key differences:

\begin{itemize}
\item Centroids are now always located on data points, rather than floating freely.
\item We mimimise \(l_1\) distance, rathern than \(l_2\).
\end{itemize}

\subsubsection{Partitioning Around Medoids (PAM) algorithm}

This is the most common approach for k-medoids.

We initialise randomly, as we do for k-means.

We then iterate the following:

\begin{itemize}
\item Calculate the loss for the current allocation
\item For each medoid, see if swapping allocation with another (non-medoid) data point decreases the cost.
\item If it does, make the swap.
\end{itemize}


% design and analysis
%
% DESIGN AND ANALYSIS: Objective of the study, data source, statistical model/tools/methodology,
% validity of the assumptions if any, results of the study (graphs, tables will go here),
% results discussion, (interpretations/consclusions/inferences)
%
%----------------------------------------------------------------------------------------
%	PACKAGES AND OTHER DOCUMENT CONFIGURATIONS
%----------------------------------------------------------------------------------------
% none

\section{Design and Analysis}
To best model the dichotomous response variable, Y\textunderscore HighGradeCancer, in the Prostate Cancer case study, I will employ a multiple logistic regression model, where 1 indicates high grade cancer and 0 indicates not high grade cancer. \par
In statistics, if \(\pi = f(x)\) is a probability then \(\frac{\pi}{1-\pi}\) is the corresponding \textit{odds}, and the \textbf{logit} of the probability is the logarithm of the odds:
\begin{equation}
	logit(\pi) = log(\frac{\pi}{1-\pi})
\end{equation}

Now, simple logistic regression means assuming that \(\pi(x)\) is related to \(\beta_0 + \beta_1x\) (the \textit{logit response function}) by the logit function. By equating \(logit(\pi)\) to the logit response function, we understand that the logarithm of the odds is a linear function of the predictor. In particular, the slope parameter \(\beta_1\) is the change in the log odds associated with a one-unit increase in \textit{x}. This implies that the odds itself changes by the multiplicative factor \(e^{\beta_1}\) when \textit{x} increases by 1 unit.

\begin{equation}
	log(\frac{\pi}{1-\pi}) = \beta_0 + \beta_1x
\end{equation}

From here, straightforward algebra will then show the Simple Linear Regression Model:

\begin{equation}
	E[Y] = \pi(x) = \frac{e^{\beta_0 + \beta_1x}}{1+e^{\beta_0 + \beta_1x}}
\end{equation}

Next, this simple logistic regression model is easily extended to more than one predictor variable by inclusion of the following two vectors, in matrix notation:

\[
	\boldsymbol{\beta} =
	\begin{bmatrix}
		\beta_0 \\ \beta_1 \\ \vdots \\ \beta_{p-1}
	\end{bmatrix} \quad
	\textbf{X} =
	\begin{bmatrix}
		1 \\ X_1 \\ X_2 \\ \vdots \\ X_{p-1}
	\end{bmatrix}
\]

With this notation, the simple logistic response function (Eqn. 3) extends to the multiple logistic response function as follows:

\begin{equation}
	E[Y] = \pi(\textbf{X}) = \frac{exp(\textbf{X}'\boldsymbol{\beta})}{1+exp(\textbf{X}'\boldsymbol{\beta})}
\end{equation}

Fitting the logistic regression to the sample data requires that the parameters \(\beta_0\), \(\beta_1\),\(\cdots\), \(\beta_{p-1}\) be estimated. This will be done using the maximum likelihood technique provided within the statistical packages of both \textbf{R} and \textit{Python}.

\subsection{Data Transformations and Standardization}
Variable transformation is an important technique to create robust models using logistic regression, and the appropriate transformations on continuous variables are necessary to optimize the model predictiveness. Because the predictors are linear in the log of the odds, it is often helpful to transform the continuous variables to create a more linear relationship. \par
The raw data collected contained several predictors with high skewness values. A few concerning features were determined to be PSA Level (skewness = 4.39), Cancer Volume (skewness = 2.18), and Weight (skewness = 7.46). As a prepossessing step to reduce skewness, I elected to transform these continuous predictor variables using the log-transformation, and standardize \textit{all} the data on top of that. The standardization step was used to normalize the data, and did not affect any underlying distributions among the predictor variables. \par
The finalized data skewness is summarized directly below in Figure 1.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{final_skewness}
	\caption{Finalized Skewness Values of Transformed Predictor Variables.}
\end{figure}

Additionally, I've included the histogram of PSA Level vs. Cancer Volume in Figure 2 - a helpful visual for the two predictors which carried the most significance through much of my analysis, as we soon shall see. Notice how the distributions exhibit no notable skewness, are quite symmetrical, and are centered on zero.

\begin{figure}[H]
	\centering
	\includegraphics{psalevel_cancervol_skewness}
	\caption{Finalized PSALevel vs. CancerVol Histogram.}
\end{figure}

\subsection{Model Selection}
The data of 97 individual men in the Prostate Cancer sample was split at 80\% for train (model-building) and test (validation) sets. The training set is a random 76 observations and was used for fitting the model, and the remaining 21 cases were saved to serve as a validation data set. Figure 3 in columns 1-8 contains the variables, and shows a portion of the finalized and processed training data. \par

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.7]{train_data_python}
	\caption{Portion of Processed Model-Building Data Set - \textit{Python} Dataframe.}
\end{figure}

\pagebreak
\subsubsection{Best Subsets Procedure}
The procedure outlined here will help identify a group of subset models that give the best values of a specified criterion. This technique has been developed by time-saving algorithms which can find the most promising models, without having to evaluate all \(2^{p-1}\) candidates. The use of the best subset procedure is based on the \textit{AIC\textsubscript{p}} criteria, where promising models will yield a relatively small value. \par
The minimized \textit{AIC\textsubscript{p}} stepwise output given by \textbf{R} is provided in Figure 4 below.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{best_subset}
	\caption{Full Linear Model  \textit{AIC\textsubscript{p}} Best Subset Results - \textbf{R} Output.}
\end{figure}

In this procedure, I instructed \textbf{R} to iterate "backwards" through all 7 predictor variables and it was determined \textit{AIC\textsubscript{p}} was minimized for \(p=3\). In particular, the results reveal that the best two-predictor model for this criteria is based on \textit{PSA Level} and \textit{Cancer Volume}. The \textit{AIC\textsubscript{p}} was minimized to 50.63, with a Null Deviance equal to 72.61 and Residual Deviance equal to 44.63.

\subsubsection{Model Fitting}
A first-order multiple logistic regression model with two predictor variables was considered to be reasonable by \S4.2.1:

\begin{equation}
\pi(\textbf{X}) = \frac{exp(\textbf{X}'\boldsymbol{\beta})}{1+exp(\textbf{X}'\boldsymbol{\beta})} = [1+exp(-\textbf{X}'\boldsymbol{\beta})]^{-1}
\end{equation}

where:

\begin{equation}
\textbf{X}'\boldsymbol{\beta} = \beta_0+\beta_1X_1+\beta_2X_2
\end{equation}

This model was then fit by the method of maximum likelihood to the data from the 76 random training cases. Results are summarized in the Figure 5 \textit{Python} output below. Provided in the output are the estimated coefficients, their standard errors, \textit{z}-scores, \textit{p}-values, and the accompanying 95\% confidence intervals. \par

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{model_fit_python}
	\caption{Maximum Likelihood Estimates of Logistic Regression Function - \textit{Python} Output.}
\end{figure}

Thus, the estimated logistic response function is:

\begin{equation}
\hat{\pi}=[ 1+ exp(-2.6867 + 1.0577X_1 + 1.5502X_2)]^{-1}
\end{equation}

\textbf{Note}: Although the PSALevel predictor is not of 5\% significance (\textit{p}-value=0.0879), I did find it necessary to maintain it within the model. When removed, the Residual Deviance score of 44.628 from Figure 4 rose to a value of 48.123. Therefore, I've deemed it significant, and a valuable and impactful variable to achieve high model accuracy, and have not removed it from this subset of predictors. \par
With the estimated logistic regression equation now developed, it is left to consider second-order options and make adjustments if required, analyze the residuals and influential observations, test goodness of fit, apply a prediction rule for new observations, and finally apply the final model to the validation data and evaluate the results.

\subsubsection{Geometric Interpretation}
When fitting a standard multiple logistic regression model with two predictors, the estimated regression shape is an S-shaped surface in three-dimensional space. Figure 6 displays a three-dimensional plot of the estimated logistic response function that depicts the relationship between the diagnosis of high grade prostate cancer (\textit{Y}, the binary outcome) and two continuous predictors, PSA Level (\textit{X}\textsubscript{1}) and Cancer Volume (\textit{X}\textsubscript{2}). \par
This surface increases in an approximately linear fashion with increasing values of PSA Level and Cancer Volume, but levels off and is nearly horizontal for very small and large values of these predictors.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.6]{3d_plot}
	\caption{Three-Dimensional Fitted Logistic Response Surface.}
\end{figure}

\subsubsection{Second-Order Predictors}
Occasionally, the first-order logistic model may not provide a sufficient fit to the data, and the inclusion of higher-order predictors may be considered. I'll conclude my model development stage by attempting to fit the Prostate Cancer data to a \textit{polynomial logistic} regression model of the second order, and analyze the results. \par
For simplicity, a 2\textsuperscript{nd}-order polynomial model in \textit{two} predictors has a logit response function as:

\begin{equation}
logit(\pi) = \beta_0 + \beta_1x_1 + \beta_2x_2 + \beta_{11}x_1^2 + \beta_{22}x_2^2 + \beta_{12}x_1x_2
\end{equation}

\noindent and can be extended to more predictors by the inclusion of additional variables, their coefficients, and accompanying cross terms. Please recall, the Prostate Cancer data set considers 7 predictors. \par
In many situations the true regression function has one or more peaks or valleys, and in such cases a polynomial function can provide a satisfactory approximation. However, a polynomial fit was not successful here, as indicated by non-significant \textit{p}-values across all predictors, at 5\% significance (Figure 7).

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.9]{poly_output}
	\caption{Logistic Regression Fit for Second-Order Model - \textbf{R} Output.}
\end{figure}

Additionally, my preliminary scatter plot analysis did not indicate any reason to believe a polynomial fit would be suitable in this study. For example, PSALevel was a major focus of this study and I've provided the scatter plot below in Figure 8. Additional scatter plots are provided in the Appendix, \S7.1.

\begin{figure}[H]
	\centering
	\includegraphics{psalevel_scatterplot}
	\caption{PSALevel vs Y\_HighGradeCancer Scatterplot - Train Data.}
\end{figure}

Without evidence to be concerned of successfully fitting a model with second-order predictors, I will move forward with my analysis of the previously developed multiple logistic linear regression model.

\subsection{Analysis of Residuals}
In this section I will discuss the analysis of residuals and the identification of any influential observations for logistic regression. Due to the nature of logistic regression, and the fact that non-constant variance is always present in this setting, I will focus only on the detection of model inadequacy.

\pagebreak
\subsubsection{Logistic Regression Residuals}
If the logistic regression model is correct, then \(E[Y_i]=\pi_i\) and it follows that:

\begin{equation}
E[Y_i-\hat{\pi}_i]=E[e_i]=0
\end{equation}

This suggests that if the model is correct, a lowess smooth of the plot of residuals against the linear predictor \(\hat{\pi}^{'}_i\) should result in approximately a horizontal line with zero intercept. Any significant departure from this suggests that the model may be inadequate. Shown in Figure 9 are the Pearson residuals plotted against the linear predictor, with the lowess smooth superimposed.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.70]{residual_plot}
	\caption{Pearson Residual Plot with Lowess Smooth.}
\end{figure}

Looking at the plot, the lowess smooth adequately approximates a line having zero slope and zero intercept, and I conclude that no significant model inadequacy is apparent.

\subsubsection{Influential Observations}
To aid in the identification of influential observations, I will use the \textbf{Cook's Distance} statistic, \(D_i\), which measures the standardized change in the linear predictor \(\hat{\pi}_i\) when the \textit{i}th case is deleted. Cook's distances are listed in the \textbf{R} Appendix \S7.2 for a portion of the Prostate Cancer testing data. \par
The plot of distances in Figure 10 identifies observation 90 as being the most outlying in the \textit{X} space, and therefore potentially influential - observations 37 and 91 also read relatively high values. Observation 90 was temporarily deleted and the logistic regression fit was obtained. The results were not particularly different from those obtained from the full test set, and the observation was retained. \textbf{Note}: I additionally and temporarily removed observations 37 and 91 and obtained a fit to the updated final model. The results were not particularly different, and those records were also retained. Thus, no changes to the model are yet necessary.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.55]{cooks_distance}
	\caption{Index Plot of Cook's Distances.}
\end{figure}

\subsection{Goodness Of Fit Evaluation}
The appropriateness of the fitted logistic regression model needs to be examined before it is accepted for use. In particular, we need to examine whether the estimated response function for the data is monotonic and sigmoidal in shape, as are logistic response functions. Here I will employ the Hosmer-Lemeshow test, which is useful for unreplicated data sets, as is the Prostate Cancer data. The test can detect major departures from a logistic response function, and the alternatives of interest are as follows:

\begin{align}
\begin{split}
	H_0: E[Y]=  [1+exp(-\textbf{X}'\boldsymbol{\beta})]^{-1} \\
	H_1: E[Y] \neq  [1+exp(-\textbf{X}'\boldsymbol{\beta})]^{-1}
\end{split}
\end{align}

\subsubsection{Hosmer-Lemeshow}
The Hosmer-Lemeshow Goodness of Fit procedure consists of grouping that data into classes with similar fitted values \(\hat{\pi}_i\), with approximately the same number of cases in each class. Once the groups are formed, the Hosmer-Lemeshow goodness of fit statistic is calculated by using the Pearson chi-square test statistic of observed and expected frequencies. The test statistic is known to be well approximated by the chi-square distribution with \(c-2\) degrees of freedom.

\begin{equation}
	\chi^2 = \sum_{j=1}^{c} \sum_{k=0}^{1} \frac{(O_{jk}-E_{jk})^2}{E_{jk}}
\end{equation}

The output from \textbf{R} using 5 groups is shown in Figure 11 below.

\begin{figure}[H]
	\centering
	\includegraphics{gof_detail}
	% \caption{insert caption here}
	\includegraphics{gof_results}
	\caption{Hosmer-Lemshow Goodness of Fit Test for Logistic Regression Function.}
\end{figure}

Large values of the test statistic X\textsuperscript{2} indicate that the logistic response function is not appropriate. The decision rule for testing the alternatives (Eqn. 10) when controlling the level of significance at \(\alpha\) therefore is:

\begin{align}
\begin{split}
	\textrm{If X\textsuperscript{2}} \leq \chi^2(1-\alpha; c-p)\textrm{, conclude } H_0 \\
	\textrm{If X\textsuperscript{2}} > \chi^2(1-\alpha; c-p)\textrm{, conclude } H_1
\end{split}
\end{align}

Thus, for \(\alpha=0.05\) and \(c-2=5-2=3\), we require \(\chi^2(0.95; 3)=7.81\). Since \(X^2=0.838\leq7.81\), we conclude \textit{H}\textsubscript{0}, that the logistic response function is appropriate. The \textit{p}-value of the test is 0.8403.

\subsection{Development of ROC Curve}
Multiple logistic regression is often employed for making predictions for new observations.
The \textit{receiver operating characteristic} (ROC) \textit{curve} plots \(P(\hat{Y}=1 | Y=1)\) as a function of \(1-P(\hat{Y}=0 | Y=0)\) and is an effective way to graphically display prediction rule information, and possible cutoff points. \par
The "True Positive" \textit{y}-axis on an ROC curve is also known as \textit{sensitivity}, and the "False Positive" \textit{x}-axis is 1-\textit{specificity}. Figure 12 below exhibits the ROC curve for my model (Eqn. 7) for all possible cut points between 0 and 1.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.45]{roc}
	\caption{ROC Curve.}
\end{figure}

\subsubsection{Prediction Rule}
In the training data set (which represented a random 80\% of the 97 provided observations), there were 14 men who were observed as high grade cancer patients; hence the estimated proportion of persons who had high grade cancer is \(14/76=0.184\). This proportion can be used as the starting point in the search for the best cutoff in the prediction rule. \par
Thus, if \(\hat{\pi}_h\) represents a newly fitted observation, my first prediction rule investigated is:

\begin{equation}
	\textrm{Predict 1 if } \hat{\pi}_h \geq 0.184\textrm{; predict 0 if } \hat{\pi}_h < 0.184
\end{equation}

The Confusion Matrix of Table 1 below provides a summary of the number of correct and incorrect classifications based on the initial prediction rule (Eqn. 13). Of the 62 men without high grade cancer, 13 would be incorrectly predicted to have high grade cancer, or an error rate of 21.0\%. Furthermore, of the 14 persons with high grade cancer, 1 would be incorrectly predicted to not have high grade cancer, or 7.1\%. Altogether, \(13+1=14\) of the 76 predictions would be incorrect, so that the prediction error rate for the rule is \(14/76=0.184\) or 18.4\%. Coincidentally, the model exactly matches our training set proportions with the current prediction rule. \par

\begin{table}[H]
	\centering
	\begin{tabular}{ |c||c|c||c|  }
 	\hline
 	\multicolumn{4}{|c|}{Prediction Rule Eqn. 13} \\
 	\hline\hline
 	True Classification&\(\hat{Y}=0\)&\(\hat{Y}=1\)&Total\\
 	\hline
 	\(Y=0\)&49&13&62\\
 	\(Y=1\)&1&13&14\\
 	\hline\hline
 	Total&50&26&76\\
 	\hline
	\end{tabular}
 	\caption{Classification based on Logistic Response Function Eqn. 7 and Prediction Rule Eqn. 13.}
\end{table}

\pagebreak
With this baseline understood, it is straightforward to choose a stronger cutoff point in utilizing the ROC curve of Figure 12. As detailed above, the false-positive rate is not ideal at 21.0\% - there are too many cases where a man may opt for additional screening and treatment, even invasive actions, because he believes he has high grade prostate cancer. It will be wise to now reference the ROC curve to better choose a prediction cutoff, while also not significantly disturbing the false-negative accuracy for the worse. \par
Looking at Figure 12, a step occurs at 0.20 and I use this value for my new cutoff candidate. Thus, my updated prediction rule is stated as follows:

\begin{equation}
	\textrm{Predict 1 if } \hat{\pi}_h \geq 0.20\textrm{; predict 0 if } \hat{\pi}_h < 0.20
\end{equation}

and the effects of this change can be summarized by the Confusion Matrix in Table 2 below.

\begin{table}[H]
	\centering
	\begin{tabular}{ |c||c|c||c| }
 	\hline
 	\multicolumn{4}{|c|}{Prediction Rule Eqn. 14} \\
 	\hline\hline
 	True Classification&\(\hat{Y}=0\)&\(\hat{Y}=1\)&Total\\
 	\hline
 	\(Y=0\)&52&10&62\\
 	\(Y=1\)&2&12&14\\
 	\hline\hline
 	Total&54&226&76\\
 	\hline
	\end{tabular}
 	\caption{Classification based on Logistic Response Function Eqn. 7 and Prediction Rule Eqn. 14.}
\end{table}

Here, of the 62 men without high grade cancer, 10 are incorrectly predicted, or an error rate of 16.1\%. Continuing, of the 14 men with high grade cancer, 2 would be incorrectly predicted, or an error rate of 14.3\%. Altogether, updated prediction rule (Eqn. 14) now provides a total error rate of \(12/76=0.158\) or 15.8\%. Thus, the model accuracy has now increased with a significantly better false-positive rate, which is intended to reduce unnecessary financial stress across the healthcare economy.

\subsection{Model: Strengths and Weaknesses}
\subsubsection{Strengths}
-The two predictors which build the final logistic model are PSA Level and Cancer Volume, and they both are adequately correlated with the dependent (outcome) variable Y\_HighGradeCancer - their correlation values are 0.489 and 0.493, respectively. In fact, they are more correlated to the dependent variable than any other predictors of the data set. A consumable heat-map version of a correlation matrix is provided by Figure 13 below, with a color legend given in the upper-left corner.

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.7]{corr_map}
	\caption{Correlation Heatmap - Train Data}
\end{figure}

-The final model has an excellent accuracy score of 84.2\% against training data. If this model also performs well in the validation step, then deploying such a model in use could possibly help non high grade cancer men be categorized as such, and thus not pursue unnecessary invasive testing and not inflate costs within the healthcare system. Also, by properly identifying those men who are high grade cancer patients, treatment and a plan can be devised sooner, as well as doing so by only use of PSA Level and Cancer Volume information, and not invasive testing. \\

\subsubsection{Weaknesses}
-An initial concern while building this model occurs at the level of the provided raw data, namely the existence of multicollinearity. Figure 14 below is the Correlation Matrix of the two final predictors which built the final logistic model: PSA Level and Cancer Volume.

\begin{figure}[H]
	\centering
	\includegraphics[scale=1.0]{corr_matrix}
	\caption{PSALevel vs. CancerVol Correlation Matrix - Train Data}
\end{figure}

\pagebreak
As shown, PSA Level and Cancer Volume have a mild correlation value of 0.738 in the full data set. One primary danger in designing models with multicollinearity is that small changes to the input data can lead to large changes in the model, which can further lead to over-fitting. Therefore, this logistic model may be considered mildly "noisy", sensitive, and not particularly robust. \\

-The Goodness of Fit Evaluation of \S4.4 deserves some concern regarding the Pearson chi-square test. As described previously, the Hosmer-Lemeshow procedure was utilized to determine a goodness of fit, and the test statistic is known to be well approximated by the chi-square distribution with \(c-2\) degrees of freedom (Eqn. 11). However, in view of the \textbf{R} output (Figure 11) with 5 groupings, the expected values (\textit{y\textsubscript{1}}) returned were: 0, 0, 1, 4, 9. Because many values are less than 5, and two of the expected values equal 0, the conditions for a chi-square test may be voided, and it may not be an appropriate test procedure here. At the very lest, the results of the Hosmer-Lemeshow test should be accepted carefully. \\

-The final model may produce high false-negative rates. In view of Figure 15-A we see that the first prediction cutoff of 0.184 produced 1 count of false-negatives (\(7.1\%\) error rate), and an overall model accuracy of \(81.6\%\). After ROC analysis the final prediction cutoff I've employed is 0.20. By Figure 15-B this rule produces 2 counts of false-negatives (\(14.3\%\) error rate), and an overall model accuracy of 84.2\%. Thus the overall model accuracy has improved 2.6 percentage points by correctly predicting more non high grade cancer cases, but has concurrently doubled the false-negative rate. Because correctly identifying high grade cancer patients may be considered most important, this arrangement may be a downfall of the final model.

\begin{figure}[H]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{confusion_matrix_PR1}
  \caption{Confusion Matrix - 0.184 Cutoff Rule.}
  \label{fig:sub1}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.8\linewidth]{confusion_matrix_PR2}
  \caption{Confusion Matrix - 0.20 Cutoff Rule}
  \label{fig:sub2}
\end{subfigure}
\caption{Classification Based on Logistic Response Function (Eqn. 7) and Prediction Rules (Eqn. 13) and (Eqn. 14).}
\label{fig:test}
\end{figure}