\subsection{The Harrod-Domar model}

\subsubsection{Introduction to growth models}

We have output as a function of capital.

\(Y=f(K)\)

We also have capital dynamics.

\(\dot K=I-\delta K\)

\(I=S=sY\)

This gives us:

\(\dot K = sY-\delta K\)

\subsubsection{Introduction}

The production function is:

\(Y=cK\)

This gives us:

\(\dot K=(sc-\delta )K\)

\subsubsection{Growth}

\(\dot Y=c\dot K \)

\(\dfrac{\dot Y}{Y}=c\dfrac{\dot K}{Y}\)

\(\dfrac{\dot Y}{Y}=c\dfrac{(sc-\delta )K}{cK}\)

\(\dfrac{\dot Y}{Y}=sc-\delta \)

\subsubsection{Per-capita growth}

Per capita income is:

\(y=\dfrac{Y}{L}\)

\(k=\dfrac{K}{L}\)


\chapter{Summary of Babu, Krishnan and Paleri}
\label{chap:chapter5}

One of the problems with other former approaches to Herbrand
equivalence is that most of the alogrithms were based on fix point
computations. But the classical definition of Herbrand equivalence is
not a fix point based definition making it difficult to prove their
precision or completeness. Babu, Krishnan and Paleri \cite{Babu} gave
a new lattice theoretic formulation of Herbrand equivalences and
proved its equivalence to the classical version.

The paper defines a congruence relation on the set of all possible
expressions and shows that the set of all congruences for a complete
lattice. Then for a given dataflow framework with $n$ program points,
a continuous composite transfer function is defined over the n-fold
product of the above lattice such that the maximum fix point of the
function yields the set of Herbrand equivalence classes at various
program points. Finally, equivalence of this approach to the
classical meet over all path definition of Herbrand Equivalence is
established.

Below is a brief summary of the developments in the paper, for more
detailed approach and proofs and for equivalence to MOP
characterization refer to \cite{Babu}.

\section{Program Expressions}

Let $C$ and $X$ be the set of constants and variables occurring in
the program respectively. The program expressions (terms) can be
described as
$$t\; ::=\; c\; |\; x\; |\; t_1 + t_2$$
where $c \in C$ and $x \in X$.

\section{Congruence Relation}
Let $T$ be the set of all program terms. A partition $P$ of terms in $T$ is said to be a congruence (of terms) if
\begin{itemize}
    \item For $t$, $t'$, $s$, $s'$ $\in$ $T$, $t' \cong t$ and $s' \cong s$ iff $t' + s' \cong t + s$.
    \item For $c \in C$, $t \in T$, if $t \cong c$ then either $t = c$ or $t \in X$.
\end{itemize}
Let $G(T)$ be the set of all congruences over $T$. We say $P_1
\preceq P_2$ for $P_1, P_2 \in G(T)$, if
$\forall A_1 \in P_1, \exists A_2 \in P_2$ such that
$A_1 \subseteq A_2$. We define \textit{confluence} operation as
$$P_1 \land P_2\; =\; \{A_i \cap B_j\: |\: A_i \in P_1 \text{ and } B_j \in P_2\}$$
Now, we extend $G(T)$ to $\overline{G(T)}$ by introducing abstract
congruence $\top$ satisfying
$P \land \top = \top, \forall P \in \overline{G(T)}$.
Also, we denote the congruence in which every element is in a
separate class as $\bot$.

$(\overline{G(T)}, \preceq, \bot, \top)$
forms a complete lattice, with $\land$ as meet operator.

\section{Transfer function}
An assignment $y := \beta$ transforms a congruence $P$ to another
congruence $P'$. This can be described in the form of transfer
function $f_{y = \beta}:G(T) \to G(T)$, given by
\begin{itemize}
    \item $B_i = \{t \in T\ |\ t[y \leftarrow \beta] \in A_i\}, \text{ for each } A_i \in P$
    \item $f_{y = \beta}(P) = \{B_i\ | \ B_i \neq \phi\}$
\end{itemize}
We extend this definition to form extended transfer function,
$\overline{f}_{y=\beta} : \overline{G(T)} \to \overline{G(T)}$
by defining $\overline{f}_{y=\beta}(\top)\ =\ \top$, otherwise $\overline{f}_{y=\beta}(P) = f_{y=\beta}(P)$.
The extended transfer function is distributive, monotonic and continuous.

\section{Non deterministic assignment}
An assignment $y := *$ transforms a congruence $P$ to another
congruence $P'$. This can be described in the form of another
transfer function $f_{y = *}:G(T) \to G(T)$, given by:
for every $t$, $t' \in T$, $t \cong_{f(P)} t'$, (here $f(P) = f_{y = *}(P)$ for simplicity) iff
\begin{itemize}
    \item $t \cong_P t'$
    \item $\forall \beta \in (T \setminus T(y)),\ t[y \leftarrow \beta] \cong_p t'[y \leftarrow \beta]$
\end{itemize}
As before we extend this transfer function to
$\overline{f}_{y=*} : \overline{G(T)} \to \overline{G(T)}$ by defining
$\overline{f}_{y=*}(\top)\ =\ \top$, otherwise
$\overline{f}_{y=*}(P) = f_{y=*}(P)$.
The function $\overline{f}_{y=*}$ is also continuous.

\section{Dataflow analysis Framework}
A dataflow framework over $T$ is $D = (G, F)$ where $G(V, E)$ is the
control flow graph associated with the program and $F$ is a
collection of transfer function associated with program points.

\section{Herbrand Equivalence}
The Herbrand Congruence function $H_D : V(G) \to \overline{G(T)}$
gives the Herbrand Congruence associated with each program point and
is defined to be the maximum fix point of the \textit{continuous
composite transfer function}
$f_D : \overline{G(T)}^n \to \overline{G(T)}^n$, where
$\overline{G(T)}^n$ is the product lattice, $f_D$ is a function satisfying $\pi_k \circ f_D = f_k$. Here $\pi_k$ is the projection map
and $f_k : \overline{G(T)}^n \to \overline{G(T)}$ is defined as follows
\begin{itemize}
    \item   If k = 1, the entry point of the program $f_k = \bot$.
    \item   If k is a function point with $Pred(k) = \{j\}, \text{ then } f_k = h_k \circ \pi_j$ where
    $h_k$ is the extended transfer function corresponding to function point k.
    \item   If k is a confluence point with $Pred(k) = {i, j}, \text{ then } f_k = \pi_{i, j}, \text{ where }
    \pi_{i, j}:\overline{G(T)}^n \to \overline{G(T)}$ is given by $\pi_{i,j}(P_1,\ \dots,\ P_n) = P_i \land P_j$.
\end{itemize}.


\begin{solution}
In this problem we interpolate the different functions using polynomial interpolation on Chebishev nodes. The rate of convergence is given in the next figure. Observe that the rate of convergence is the highest for the function of part d,since it is infinitely smooth, followed by the function of part c,a and b. The order of this convergence is because of the smoothness of the different functions. In real numbers the function of part c is infinitely smooth as well, whereas the other two functions arent.
\begin{figure}[H]
\centering     %%% not \center
\hspace*{\fill}
\subfigure[$f(x)=|x|^3$.]{\includegraphics[scale=0.5]{IMAGES/problem4a.eps}}
\hfill
\subfigure[$f(x)=exp\left(\frac{-1}{\sin{(2x^2)}}\right)$.]{\includegraphics[scale=0.5]{IMAGES/problem4b.eps}}
\hspace*{\fill}

\hspace*{\fill}
\subfigure[$f(x)=\frac{1}{\sin{(1+x^2)}}$.]{\includegraphics[scale=0.5]{IMAGES/problem4c.eps}}
\hfill
\subfigure[$f(x)=\sinh{x}$.]{\includegraphics[scale=0.5]{IMAGES/problem4d.eps}}
\hspace*{\fill}
\caption{Error of the polynomial interpolation on Chebyshev nodes for the different functions.}
\end{figure}
\subsection*{Matlab code for this problem}
\begin{verbatim}
% Part a
N = 1:100;
err = 0*N;
ff = @(x) abs(x).^3;
f = chebfun(ff,'splitting','on');
for k = 1:length(N)
    n = N(k);
    g = chebfun(ff,n);
    err(k) = norm(f-g,inf);
end
figure
loglog(N,err,'*',N,N.^-1,'--',N,N.^-3,'--')
set(gca,'fontsize',14)
grid on
xlabel('$N$ (log scale)','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
saveas(gcf,'IMAGES/problem4a','epsc')
% Part b
ff = @(x) exp(-1/sin(2*x^2));
f = chebfun(ff,'splitting','on');
for k = 1:length(N)
    n = N(k);
    g = chebfun(ff,n);
    err(k) = norm(f-g,inf);
end
figure
semilogy(N,err,'*',N,N.^-1,'--',N,N.^-3,'--')
set(gca,'fontsize',14)
grid on
xlabel('$N$','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
saveas(gcf,'IMAGES/problem4b','epsc')

% Part c
ff = @(x) 1/sin(1+x^2);
f = chebfun(ff,'splitting','on');
for k = 1:length(N)
    n = N(k);
    g = chebfun(ff,n);
    err(k) = norm(f-g,inf);
end
figure
semilogy(N,err,'*',N,N.^-1,'--',N,N.^-3,'--')
set(gca,'fontsize',14)
grid on
xlabel('$N$ ','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
saveas(gcf,'IMAGES/problem4c','epsc')

% Part d
ff = @(x) sinh(x)^2;
f = chebfun(ff,'splitting','on');
for k = 1:length(N)
    n = N(k);
    g = chebfun(ff,n);
    err(k) = norm(f-g,inf);
end
figure
semilogy(N,err,'*',N,N.^-1,'--',N,N.^-3,'--')
set(gca,'fontsize',14)
grid on
xlabel('$N$ (log scale)','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
saveas(gcf,'IMAGES/problem4d','epsc')
\end{verbatim}

\end{solution}


\chapter{Generalized Linear Models \label{chapter:glms}}

Linear and logistic regression, which we have seen already in Chapters~\ref{chapter:classification}, \ref{chapter:regression}, \ref{chapter:linreg}, and \ref{chapter:logreg}, are members of a broader class of supervised learning models called \textbf{generalized linear models (GLMs)}. In GLMs, the outcome variable, $y$, is assumed to follow a probability distribution of a particular type. For example, in linear regression, $y$ follows a normal distribution. In logistic regression, $y$ is binary $\left( y \in \{0, 1\} \right)$ and follows a Bernoulli distribution\footnote{In \textbf{grouped} logistic regression, it follows a binomial distribution.}. The expected value, or mean, of the outcome distribution is related to a \textbf{linear combination} of the predictors, $\beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$, via a model-specific \textbf{link function}.

GLMs, like maximum likelihood (Chapter~\ref{chapter:mlebasics}), are normally considered an advanced topic. However, they provide a nice example of how the same formalism -- modeling the response variable using a probability distribution, assuming a certain form for the predictors, optimizing the whole thing using maximum likelihood -- can be applied to solve different-looking problems. They are also a good entryway into the sorts of optimization tasks performed by graphical models and deep learning algorithms.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Model Assumptions}

GLMs require us to make several assumptions which affect both our choice of model and our interpretation of model output:

\begin{enumerate}
\item We assume that the outcome follows a certain type of distribution (e.g. Bernoulli distribution for a logistic regression model, normal for linear, etc.) conditional on the predictors. This assumption is baked into the model structure. It is, therefore, important to consider whether the outcome distribution you chose actually makes sense for your particular problem. It is generally not advisable to use a linear regression model, for example, when your outcome is a count.
\item We assume that the predictors are fixed and known, and thus have no error associated with their measurements\footnote{Bayesian versions of these models relax this assumption.}.
\item We assume that the predictors enter the model as a linear combination, $\beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$. This is why GLMs are referred to as ``linear models''.
\item We assume that the $n$ samples in our dataset are collected independently, so that the errors of the $n$ sample outcomes are uncorrelated\footnote{Think back to our formulation of the likelihood in Chapter~\ref{chapter:mlebasics} and how it depended on the samples' being independent and identically distributed, or iid.}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Notation for the Predictors}

As mentioned above, GLMs assume that the predictors enter the model as a linear combination. A linear combination is an expression constructed from a set of terms by multiplying each term by a constant and adding the results. We denote the number of predictors in the model by $p$ and the vector of predictors by $x$, where
$$ x = \begin{bmatrix}
1 \\
           x_{1} \\
           x_{2} \\
           \vdots \\
           x_{p}
         \end{bmatrix} $$
and we have included a ``1'' as the first element to allow for an \textbf{intercept}. We write $x^{(i)}$ to denote the vector of predictors associated with the $i$th training example. The coefficients of the linear combination (i.e. the model parameters we are hoping to learn) are denoted by:
$$ \beta = \begin{bmatrix}
\beta_0 \\
           \beta_{1} \\
           \beta_{2} \\
           \vdots \\
           \beta_{p}
         \end{bmatrix} $$
and we often express the linear combination as an \textbf{inner product}, or \textbf{dot product}, of the two vectors, written as
$$ \beta^T x = \beta_0 + \sum_{j=1}^p \beta_j x_j. $$
This is just notational shorthand.
\vspace{3mm}

\begin{question}{}
We saw the details of linear and logistic regression models in Chapters~\ref{chapter:linreg} and \ref{chapter:logreg} and discussed the limitations of predictors' entering as a linear combination. What are some of those limitations?
\end{question}

\begin{question}{}
Just to confirm that you understand this notation, write out the form of $\beta^Tx$ for a model with (a) one predictor, (b) three predictors. Write both the general form and the form for one training example, $x^{(i)}$.
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Modeling the Outcome}

Generalized linear models model the expected value of the outcome, $E[y]$, as a function of this linear combination of predictors.

\subsection{Linear Regression}

In linear regression, we assume that the outcome, $y$, follows a normal distribution (see Section~\ref{sect:normal}), whose mean is controlled by the values of the predictors. Recall that the normal distribution is a continuous probability distribution with the following properties:
$$ p(y) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(y-\mu)^2}{2 \sigma^2}} \qquad  E[y] = \mu \qquad \text{var}(y) = \sigma^2 $$
where $y \in \mathbb{R}$. Its mean, $\mu$, can be any real number. To link $\mu$ to the predictors, therefore, we simply set it equal to $\beta^Tx$, like so:
\begin{equation} E[y] = \mu = \beta^T x \label{eqn:meanlinear} \end{equation}
This is called using the \textbf{identity link}. The relationship between $E[y] = \mu$ and $\beta^T x$ is shown below.

\begin{center}
\includegraphics[width=0.5\textwidth]{img/l02-figure1-linreg.png}
\end{center}

\vspace{3mm}

\begin{question}{}
In this model, how much does the mean of the outcome distribution, $\mu$, change as you vary each predictor? For example, if you have $p=3$ predictors, by how much does $\mu$ change as the value of $x_2$ changes by one unit (for example, from $1$ to $2$)? How much does $\mu$ change as the value of $x_2$ changes from $3$ to $4$? What about $x_1$ and $x_3$?
\end{question}

\subsection{Logistic Regression}

In logistic regression the outcome, $y$, is either $0$ or $1$. We model it using the Bernoulli distribution (see Section~\ref{sect:bernoulli}), which is a discrete probability distribution with the following properties:
$$ p(y) = \mu^y (1 - \mu) ^ {1-y} \qquad E[y] = \mu \qquad \text{var}(y) = \mu (1 - \mu) $$
where $y \in \{0, 1\}$. Because $\mu$ is a probability, it must be a real number between 0 and 1. No matter how large or small $\beta^T x$ gets, the value of $E[y] = \mu$ cannot be outside this range. We therefore apply the \textbf{logistic function}, $f(x) = 1/(1 + \exp(-x))$, which has the range $(0, 1)$, to $\beta^T x$ to squash it:
\begin{equation} E[y] = \mu = \frac{1}{1 + \exp{(-\beta^Tx)}} \label{eqn:meanlogistic} \end{equation}
The relationship between $E[y]$ and $\beta^T x$ is shown below. We typically invert the model to write
$$ \log{\frac{\mu}{1-\mu}} = \beta^T x. $$
The function $\log \left( \mu/(1-\mu) \right)$ is called the logit, and we say we use the \textbf{logit link}.

\begin{center}
\includegraphics[width=0.5\textwidth]{img/l02-figure2-logistic.png}
\end{center}

\vspace{3mm}

\begin{question}{}
Let's revisit Question~\ref{question:logreglink}. Now that we've described logistic regression in the framework of GLMs, what more can you say about why the model is not of the form
$$ \mu = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p \text{?}$$
\end{question}

\subsection{Poisson Regression}

In Poisson regression, the outcome is a count. We model the outcome using a Poisson distribution, which is a discrete probability distribution with the following properties (Section~\ref{sect:poisson}):
$$ p(y) = \frac{e^{-\lambda} \lambda^y}{y!} \qquad E[y] = \lambda \qquad \text{var}(y) = \lambda $$
where $y \in 0, 1, 2, \dots$. Because $\lambda$, the mean of the outcome distribution, is the expected value of a count, it must be a real number greater than or equal to zero. In particular, no matter how small $\beta^T x$ gets, the value of $E[y] = \lambda$ cannot be negative. We therefore exponentiate $\beta^T x$ to ensure that $\lambda$ is greater than zero:
\begin{equation} E[y] = \lambda = \exp(\beta^T x) \label{eqn:meanpoisson} \end{equation}
The relationship between $E[y]$ and $\beta^T x$ is shown below. We typically invert the model to write
$$ \log(\lambda) = \beta^T x $$
which is the standard form of the Poisson regression model. We say these models use the \textbf{log link}.

\vspace{3mm}

\begin{question}{}
There are many other generalized linear models. In each case, the mean (expected value) of a probability distribution is related, via a link function, to a linear combination of the predictors.

Knowing this, how would you create a GLM where the outcome follows an exponential distribution (Section~\ref{sect:exponential})? Which link would you use?
\end{question}

\begin{center}
\includegraphics[width=0.5\textwidth]{img/l02-figure3-poisson.png}
\end{center}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Maximum Likelihood for GLMs \label{section:mleglms}}

GLMs are fit using maximum likelihood estimation (see Chapter~\ref{chapter:mlebasics}). A full treatment of MLE for GLMs is outside the scope of these notes, but I've put the start of the calculations for each type of model below. The only difference between these calculations and those in Chapter~\ref{chapter:mlebasics} is that now our parameters of interest, the means of our outcome distributions, are functions of our predictors $x_1, \dots, x_p$. Our job is to find the coefficients on those predictors, $\beta_0, \dots, \beta_p$, that provide the best fit between our model and our training data.

\subsection{Linear Regression}

The likelihood for the linear regression model is:
$$ \mathcal{L}(\mu^{(1)}, \dots, \mu^{(n)}, \sigma) = \prod_{i=1}^n \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left[ - \frac{(y^{(i)} - \mu^{(i)})^2}{2 \sigma^2} \right] $$
where we use $\mu^{(i)}$ to represent the model's estimate of the mean of the outcome at the position of training example $i$. We can use Equation~\ref{eqn:meanlinear} to rewrite this as a function of the predictors:
$$ \mathcal{L}(\beta, \sigma) = \prod_{i=1}^n \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left[ - \frac{(y^{(i)} - \beta^T x^{(i)})^2}{2 \sigma^2} \right] $$
Taking the log, we obtain the log-likelihood:
$$ \log \mathcal{L}(\beta, \sigma) = -\frac{n}{2} \log (2 \pi) - \frac{n}{2} \log(\sigma^2) - \frac{1}{2 \sigma^2} \sum_{i=1}^n \left( y^{(i)} - \beta^T x^{(i)} \right)^2 $$
Taking derivatives of the log-likelihood with respect to the $\beta$s, we find that we can maximize the likelihood by minimizing the sum-squares: $\sum_{i=1}^n \left( y^{(i)} - \beta^T x^{(i)} \right)^2$.

\vspace{3mm}

\begin{question}{}
Take a minute to stare at this result. When most people learn linear regression, they learn that these models are fitted by minimizing the sum of squared residuals (see Chapter~\ref{chapter:linreg}). Indeed, linear regression models predate GLMs and are typically fit using ordinary least squares, not maximum likelihood. If you fit a linear regression model in R using the \texttt{lm} package, you're using OLS. If you use the \texttt{glm} package with the argument \texttt{family = "gaussian"}, you're using maximum likelihood. However, both methods will produce the same fitted models. Do you see why this is?
\end{question}

\subsection{Logistic Regression}

The likelihood for the logistic regression model is:
$$ \mathcal{L}(\mu^{(1)}, \dots, \mu^{(n)}) = \prod_{i=1}^n {\mu^{(i)}}^{y^{(i)}} (1-\mu^{(i)})^{1 - y^{(i)}} $$
Rewriting this as a function of the predictors, we get:
$$ \mathcal{L}(\beta) = \prod_{i=1}^n \left( \frac{1}{1 + \exp(-\beta^T x^{(i)})} \right)^{y^{(i)}} \left( \frac{\exp(-\beta^T x^{(i)})}{1 + \exp(-\beta^T x^{(i)})} \right)^{1 - y^{(i)}} $$
Taking the log, we obtain the log-likelihood:
$$ \log \mathcal{L}(\beta) = \sum_{i=1}^n \left[ y^{(i)} \beta^T x^{(i)} - \log \left(1 + \exp(\beta^T x^{(i)}) \right) \right] $$
Again, we will take derivatives of the log-likelihood with respect to the $\beta$s to maximize it. However, we cannot solve for the optimal $\beta$s analytically in this case. Numerical optimization methods are used to find the maximum likelihood estimates, $\hat{\beta}_0, \hat{\beta}_1,$ etc.

\subsection{Loglinear (Poisson) Regression}

The likelihood for the Poisson regression model is:
$$ \mathcal{L}(\lambda^{(1)}, \dots, \lambda^{(n)}) = \prod_{i=1}^n \frac{{\lambda^{(i)}}^{y^{(i)}} e^{-\lambda^{(i)}}}{y^{(i)}!} $$
Rewriting this as a function of the predictors, we get:
$$ \mathcal{L}(\beta) = \prod_{i=1}^n \frac{\exp{(y^{(i)} \beta^T x^{(i)})} e^{-\exp{(\beta^T x^{(i)})}}}{y^{(i)}!} $$
Taking the log, we obtain the log-likelihood:
$$ \log \mathcal{L}(\beta) = \sum_{i=1}^n \left[ y^{(i)} \beta^T x^{(i)} - \exp(\beta^T x^{(i)}) - \log (y^{(i)}!) \right] $$
As with logistic regression, we cannot solve for the optimal $\beta$s analytically; numerical optimization methods are used.
\vspace{1mm}

\begin{question}{}
Think of the log-likelihood as measuring the height of a hill. Your data,
$$\left\{x^{(1)},\dots,x^{(n)}\right\}$$
don't change, so we don't care about their effect on the height. What we care about are the parameters, $\beta_0, \dots, \beta_p$. For each combination of those $p+1$ parameters, the height changes. We want to find the combination of parameters that puts us at the top of the hill.

The first derivative of the log-likelihood with respect to one of the parameters, $\beta_j$, is
$$ \frac{\partial \log \mathcal{L}}{\partial \beta_j} $$
and the vector of all of these first derivatives for $\beta_0, \dots, \beta_j$ is called the \textbf{gradient}. Evaluated at a particular set of parameters, the gradient tells you how steep your hill is in the direction of each of your $p+1$ parameters. How could you use this information to maximize the likelihood? You don't need to do any math. Just say how you would do it.
\end{question}

\begin{question}{}
There are many different numerical optimization algorithms that one can use to maximize the likelihood (i.e., find the top of the hill). One of them is called \textbf{Fisher scoring}. Examine the output of the logistic regression models in Chapter~\ref{chapter:logreg} and the Poisson regression model shown below in Section~\ref{sect:poisreg}. Where do you see the term ``Fisher scoring''? What do you think the term ``Fisher scoring iterations'' refers to?
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Standard Errors and Hypothesis Tests \label{section:sehyp}}

Here, once again, is the summary output from a logistic regression model of the ER readmissions example from Chapter~\ref{chapter:classification}, reprinted again in Section~\ref{sect:eragain}:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/glm-binomial-example.png}
\end{center}
As we discussed in Chapter~\ref{chapter:logreg}, the magnitudes of the coefficients in these models matter, but they are only important in relation to:
\begin{enumerate}
\item The scale on which the predictors are measured.
\item The amount of uncertainty the model has about their values.
\end{enumerate}
For example, if a predictor varies only across a tiny range of values, its model coefficient may be large, since it quantifies the change in the link-function-transformed outcome when the predictor changes by 1.0. However, that doesn't mean that the predictor itself is important to the outcome\footnote{This is one reason many advocate \textbf{scaling} and \textbf{centering} predictors before fitting a model. Centering means subtracting the mean value of a predictor from all of its individual measurements so that the mean of each centered predictor is zero. Scaling means dividing the values of each predictor by their standard deviation, so that the standard deviation of each predictor is 1.0. This enables the relative magnitudes of the model coefficients to be compared directly.}.

Similarly, the model may be highly uncertain about a coefficient's value, owing to factors like a small dataset (small $n$) or collinearity (correlations) among the predictors. Mathematically, high uncertainty means that the value of the likelihood doesn't change very rapidly as you move away from the maximum likelihood estimate of a coefficient. For example, here is how the log-likelihood for the logistic regression example above changes when we vary $\beta_1$ (the coefficient of $x_1$), keeping $\beta_0$ (the intercept) and $\beta_2$ (the coefficient of $x_2$) fixed at their MLEs:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/esl-logistic-beta1.png}
\end{center}
The gray vertical lines are related to the \textbf{standard error} of the model coefficient, which is in turn related to the ``flatness'' of the likelihood surface around the MLE. The gray lines are situated at 1 and 2 standard errors away from the MLE in either direction. You can see that in the case of $\beta_1$, the gray lines overlap zero. The value zero (no effect) is a plausible estimate of the impact of $x_1$ on the outcome.

Contrast this with how the log-likelihood varies around the MLE for $\beta_2$:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/esl-logistic-beta2.png}
\end{center}
Here the standard error is larger, but the magnitude of the coefficient is also larger, so the range of the gray lines does not overlap zero.

\begin{question}{}
These findings are reflected in the relative values of the Z-statistic (\texttt{z value}) and P-value (\verb|Pr(>|z|)|) in the model output for the two coefficients. With that in mind, let's reconsider Question~\ref{question:nulllogregexample}. How do these likelihood plots and the null distributions shown in Question~\ref{question:nulllogregexample} convey the same information?
\end{question}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Example: Nesting Horseshoe Crabs Dataset \label{sect:poisreg}}

Let's examine some output from a Poisson regression model, which is a type of GLM with which you may not already be familiar.

These data come from a study of nesting horseshoe crabs. Each of the 173 observed female horseshoe crabs had a male crab resident in her nest. The study investigated factors affecting whether the female crab had any other males, called \emph{satellites}, residing nearby. (Source: Agresti, \emph{Categorical Data Analysis}, Table 4.3. Data courtesy of Jane Brockmann, Zoology Department, University of Florida; study described in \emph{Ethology} \textbf{102}: 1-21, 1996.)

\begin{center}
\texttt{\small
\begin{tabular}{ll}
\toprule
SATELL & Number of satellites \\
COLOR & Color of the female crab \\
& (1 = light medium, 2 = medium, 3 = dark medium, \\
& 4 = dark) \\
SPINE & Spine condition \\
& (1 = both good, 2 = one work or broken, \\
& 3 = both worn or broken) \\
WIDTH & Carapace width of the female crab (cm) \\
WEIGHT & Weight of the female crab (g) \\
\bottomrule
\end{tabular}
}
\end{center}

\noindent The GLM output of this model is:

\begin{center}
\includegraphics[width=0.8\textwidth]{img/poisson-horseshoe-model.png}
\end{center}

\begin{question}{}
Comment on how the variables \texttt{color} and \texttt{spine} are coded here. Does this make sense in light of what those variables mean?
\end{question}

\begin{question}{}
Interpret the values of each of these coefficients. Based on the coefficient values and their standard errors, which predictor(s) do you think have the greatest impact on the number of male satellites around a nesting female horseshoe crab?
\end{question}

\begin{question}{}
How could you use a decision tree to model the horseshoe crabs data? What are its advantages and disadvantages relative to Poisson regression (a type of GLM)?
\end{question}


\lab{Python}{The Standard Library}{Standard Library}
\objective{Become familiar with the Python standard library}

One of the reasons Python is so useful as a scientific computing platform is because
it is not limited to only scientific computing.  Python has a very large and comprehensive standard library that is available with almost every Python environment.
In this lab, we will look at some of the useful parts of Python's standard library.  The standard library is comprised of over a hundred different modules that provide extra functionality to Python.
Modules can be imported via the \li{import} statement.

\section*{\texttt{math} Module}
The \li{math} module and its companion, \li{cmath} (for complex numbers), are very useful modules.
Common mathematical functions are defined in this module such as: \li{cos}, \li{sin}, \li{log}, \li{sqrt}, etc.  These functions wrap around the functionality of the C math library.

\begin{problem}
NumPy also implements these special mathematical functions.  However, there is a difference.
The NumPy variants are designed to work with NumPy arrays.  For scalar inputs, the functions
in Python's \li{math} or \li{numpy.math} are much faster.

Import NumPy with \li{import numpy as np}.  Time how long it takes to execute the following statements.
\begin{lstlisting}
np.sin(.5)
math.sin(.5)
\end{lstlisting}
\end{problem}

\section*{\texttt{random} Module}
Python includes an implementation of the Mersenne Twister pseudorandom number generator.
The \li{random} module contains many helpful functions for obtaining random numbers.  It also contains functions that operate on sequences.
The \li{randint()} function will return a random integer in a desired interval.
\li{randrange()} will randomly select a random number from a range of numbers (similar to a range generated by \li{range()}, except it doesn't actually construct the range object).

For sequences, we can randomly choose elements using the \li{choice()} function.  If wish to randomly sample a sequence we can use \li{sample()}, which will return a random sampling of unique elements of desired length from the given sequence.  A sequence can be randomly shuffled in place by the \li{shuffle()} function.

The \li{random} module can sample from several distributions such as the uniform, normal, beta, gamma, exponential, and many others.  NumPy's \li{random} module is even more fully featured than the standard Python \li{random} module.

\section*{\texttt{csv} Module}
CSV files are common for exchanging data from databases and tables.
Python has a very useful module for reading and writing data as comma separated values.
The \li{csv} module provides \li{reader} and \li{writer} objects.  There are also analogous \li{DictReader} and \li{DictWriter} objects that use dictionaries for handling data.
\begin{lstlisting}
import csv

#print the contents of a csv_file
with open('test.csv', 'r') as csv_file:
    csv_reader = csv.reader(csv_file)
    for line in csv_reader:
        print line
\end{lstlisting}

Writing with a CSV \li{writer} object is very similar to writing with a regular file object.
\begin{lstlisting}
contents = [["Column 1", "Column 2", "Column 3"],
            [0,1,2], [3, 2, 1], [4,5,2], [68, 38, 99]]
with open('test_out.csv', 'w') as csv_file:
    csv_writer = csv.writer(csv_file):
    for record in contents:
        csv_writer.writerow(record)
\end{lstlisting}

\begin{problem}
Practice reading and writing a CSV file.  Read \texttt{test.csv} and write it to \texttt{test\_out.csv}.
When writing the CSV file, you must use a delimiter other than a comma.  Read the Python documentation for
instructions of how to do this.
\end{problem}

\section*{\texttt{sys} Module}
The \li{sys} module allow you to access information specific to the system running Python.
One of the most commonly used functions is \li{sys.argv}.  This returns a list of arguments passed to the current environment.  Accessing these arguments is important.  Many programs are written to execute differently based on the various arguments and options specified at execution.  For example, if we execute our script from the command line as follows
\begin{verbatim}
python myscript.py 5 no yes yes
\end{verbatim}
\li{sys.argv} would return the list
\begin{verbatim}
[`myscript.py', `5', `no', `yes', `yes']
\end{verbatim}
We can use \li{sys.argv} in combination with \li{argparse} to obtain a full featured argument processing system.

\section*{\texttt{pickle} Module}
The pickle module turns a Python object into a bytestream and saves it to a file.
It can also take a bytestream and turn it back into a Python object.
Pickle can be used to store any builtin Python data type such as lists, tuple, integers, etc.
Not all Python objects can be pickled.
Dumping a bytestream to a file is easy
\begin{lstlisting}
import pickle
a = range(10)
pickle.dump(a, open(`out.pkl', `w'))
\end{lstlisting}
What pickle does is write a small program that will rebuild your data structures when you read it.
When unpickling a file, pickle executes this file in the interpreter.  Because of this,
pickle is meant to be used as a temporary storage and data persistence mechanism.  It is not designed
to be used for long term storage of data.  The pickle documentation includes this warning
\begin{quote}
\textbf{The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.}
\end{quote}
To unpickle an object on simple use \li{pickle.load()}.  It accepts a file handle and returns the Python object it creates.

\begin{problem}
Create a list of numbers and strings.  Pickle the object to a file.  Inspect the contents of the file you created when pickling.  Unpickle the object.
\end{problem}

The \li{pickle} module only allows you to store one object per file.
If storing many objects is desired, you may want to use the \li{shelve} module.
This stores objects in a dictionary-like data structure with keys and values.
The values are pickled objects.  Since \li{shelve} relies on \li{pickle}, the
same warning against untrusted sources applies to \li{shelve} as well.

\section*{\texttt{timeit} Module}
This module is used to time the execution of small bits of Python code.
It is recommended to time lines of Python using this module because it avoids a number of common pitfalls in measuring execution time.  IPython's \li{\%timeit} magic function is a wrapper around this module.

\begin{problem}
For most timing situations, we rely on IPython's \li{\%timeit} magic function.
One major drawback is it only works in IPython.
The solution to this problem will be useful in other labs where you will be
asked to time the performance of your coded solutions.
Write a function that will
time the execution of another function.  You will need to use the \li{timeit} module.
Your function, should accept as arguments, a function, $f$, and any arguments that
should be passed to $f$.  Your function should return the minimum runtime.

Because of the way that Python's \li{timeit} module functions, we must use a \emph{callable}
function.  This essentially means we have to wrap the function we are timing and all of
its arguments into a function object that can be called by \li{timeit}.
This can be done by declaring a Python \li{lambda} function which takes no arguments.
\begin{lstlisting}
pfunc = lambda: f(*args, **kargs)
\end{lstlisting}
where \li{args} is a tuple and \li{kargs} is a dictionary.
This syntax is explained in chapter 4 of the Official Python Tutorial).
\end{problem}

\section*{\texttt{os.path} Module}
The \li{os.path} module contains several methods for interfacing with the local file system in a cross-platform manner.
\begin{table}[h]
\begin{tabular}{|l|p{9cm}|}
\hline
\li{os.path.abspath} & Return a normalized absolute version given path. \\
\li{os.path.exists} & Return \li{True} if given path exists on the file system. \\
\li{os.path.isfile} & Return \li{True} if given path refers to an existing file. \\
\li{os.path.isdir} & Return \li{True} if given path refers to an existing directory. \\
\li{os.path.join} & Join one or more elements of a path intelligently, depending on the current platform. \\
\li{os.realpath} & Return the system's canonical form of a given path. \\
\li{os.split} & Split a given path into a tuple where the second entry is the final path element and the first element is all elements up to the final element. \\
\li{os.splitext} & Split a given path into a tuple of two elements where the second entry is the extension of the file referred to by the path.  The first entry contains everything up to the extension separator. \\
\hline
\end{tabular}
\caption{Some useful functions provided by \li{os.path}}
\end{table}

\section*{\texttt{collections} Module}
This module defines several specialized data structures to use in addition to the builtin
Python data structures.  Some of these useful data structures are named tuples, deques, and a Counter object.

Named tuples are designed to help improve code readability in some cases.
Standard tuples in Python are accessed by index.
Named tuples allow access via index or by a field name.
Compare the following
\begin{lstlisting}
from collections import namedtuple
pt = (32.1, 63.2)

Pt = namedtuple(`Point', `y x')
npt = Pt(32.1, 63.2)
\end{lstlisting}
The tuple \li{pt} is a standard tuple, which we can surmise represents the coordinates
of a pt in 2D space.  We must assume that \li{pt[0]} is the x-coordinate and \li{pt[1]}
is the y-coordinate.  When declaring a named tuple, we clearly defined what each index
represents.  We know for certain that \li{npt.x} is the x-coordinate and that \li{npt.y}
is the y-coordinate.

Ordered dictionaries are exactly like standard dictionaries except for one important
difference.  Ordered dictionaries remember the order in which key-value pairs were added
to the data structure.  This data structure is useful if we want to iterate over key-value pairs
of a dictionary in a specific order.

Default dictionaries are a very convenient way to set a default value for all new keys in a dictionary.  While this can be done with standard dictionaries using the \li{setdefault()} method,
using a default dictionary is simpler and faster.

\begin{problem}
A double-ended queue, or deque, can be thought of as a deck of cards.
Inserting and removing elements from either end is very efficient.
Python's deque implementation does not allow inserting into any other place
in the data structure except for the left and right ends.
A list, however, is very inefficient when adding elements to the front.
Write two functions that will rotate the elements of a deque and a list respectively.
To rotate, remove elements from the right end one by one and insert them on the left end.
Compare the timings you obtain from a deque and a list of 10000 elements.
\end{problem}


Lists are intrinsically monodimensional, without any additional information besides sequentialisation. The implication of this is that any operation on lists will require to potentially go through the whole list, and we will be sure that we can stop only after having processed all elements. For this reason lists are expensive as containers: as the number of elements of the list grows, so grows the number of steps needed for processing operations.

A solution to this issue is to build data structures where the position of elements within the data structures has some correlation with the value of the elements themselves, and stronger yet with the relationship between the value of each element and the values of other elements. One of the most used such data structures is the binary search tree. A binary search tree is a tree where each node contains an element and two subtrees, which are usually called \texttt{left} and \texttt{right}. The \textit{fundamental search property} of binary trees states that, for any node within the tree, each element is bigger than all elements within the left tree and smaller than all elements within the right tree. Thanks to this fundamental property, when searching a tree for an element we will always be able to determine with a single comparison whether all elements of the left or right subtree may be immediately discarded.

Of course a tree may also be empty. We encode such a tree as follows, with the \texttt{nil} keyword:

\begin{lstlisting}
Data [] "nil" [] Priority 0 Type BinTreeInt
\end{lstlisting}

A node of a tree, encoded with the \texttt{node} keyword, takes three parameters as input: the left sub-tree, the element of the node, and the right sub-tree:

\begin{lstlisting}
Data [] "node" [BinTreeInt <<int>> BinTreeInt] Priority 1010 Type BinTreeInt
\end{lstlisting}

According to the definitions above, a tree such as:

\begin{lstlisting}
    5
   / \
  2   7
 / \
1   3
\end{lstlisting}

would be encoded as: \texttt{node (node (node nil 1 nil) 2 (node nil 3 nil)) 5 (node nil 7 nil)}.

\subsection{Insertion}
Adding an element to a binary tree is not trivial as it is for lists. Whereas in a list we simply use \texttt{;} to push yet an element on top of the list, for binary trees we must make sure to return a tree which still satisfies the fundamental search property. The \texttt{add} keyword, which takes a left parameter which is the initial tree, and a right parameter which is the integer value to add, will return a new tree which contains the elements of the initial tree, plus the new element to add, all the while respecting the fundamental search property:

\begin{lstlisting}
Func [BinTreeInt] "add" [<<int>>] Priority 100 Class Expr => BinTreeInt
\end{lstlisting}

Let us begin with the simplest case of adding an element \texttt{k} to an empty tree. There is not much to do in this case, since we can simply create a new node with element \texttt{k} and empty left and right subtrees:

\begin{lstlisting}
----------------------------
nil add k => node nil k nil
\end{lstlisting}

If the current node of the binary tree contains an element which is identical to the one we are inserting, then there is not much to do. In this case we simply return the original tree, as it already satisfied our requirement that the returned tree contains the desired element:

\begin{lstlisting}
x == k
---------------------------------
(node l x r) add k => node l k r
\end{lstlisting}

If the current node of the binary tree contains an element which is bigger than the element we are adding, then we add the element to the left subtree and we use the resulting left subtree (which contains the elements of \texttt{l} plus \texttt{k}) as the left subtree of the final result:

\begin{lstlisting}
k < x
l add k => l'
----------------------------------
(node l x r) add k => node l' x r
\end{lstlisting}

If the current node of the binary tree contains an element which is smaller than the element we are adding, then we add the element to the right subtree and we use the resulting right subtree (which contains the elements of \texttt{r} plus \texttt{k}) as the right subtree of the final result:

\begin{lstlisting}
k > x
r add k => r'
------------------------------------
(node l x r) add k => (node l x r')
\end{lstlisting}

For example, consider adding element \texttt{3} to tree \texttt{node nil 5 (node nil 7 nil)}:

\begin{lstlisting}
5
 \
  7
\end{lstlisting}

We start with the following program:

\begin{lstlisting}
-----------------------------------------
(node nil 5 (node nil 7 nil)) add 3 => ?
\end{lstlisting}

The first rule which is applied is the rule for recursion in the left subtree, thus:

\begin{lstlisting}
3 < 5
nil add 3 => l'
-----------------------------------------------------------------
(node nil 5 (node nil 7 nil)) add 3 => node l' 5 (node nil 7 nil)
\end{lstlisting}

Addition within an empty sub-tree is trivially resolved, thus we obtain:

\begin{lstlisting}
l' := node nil 3 nil
---------------------
nil add 3 => l'
-----------------------------------------------------------------
(node nil 5 (node nil 7 nil)) add 3 => node l' 5 (node nil 7 nil)
\end{lstlisting}

We can now unwind the stack, which after just one unwinding step yields the final result:

\begin{lstlisting}
-------------------------------------------------------------------------------
(node nil 5 (node nil 7 nil)) add 3 => node (node nil 3 nil) 5 (node nil 7 nil)
\end{lstlisting}

which (as expected) is the tree:

\begin{lstlisting}
  5
 / \
3   7
\end{lstlisting}

\subsection{Search}
Just like we did for lists, we can perform a search within a binary search tree. Search determines whether or not a given element is contained within the tree, and we encode it with the operator \texttt{contains} which has a tree as its left parameter and an integer element as right parameter:

\begin{lstlisting}
Func [BinTreeInt] "contains" [<<int>>] Priority 100 Type Expr => YesNo
\end{lstlisting}

An empty tree never contains an element:

\begin{lstlisting}
---------------------
nil contains k => no
\end{lstlisting}

On the other hand, a tree that begins with the element we are looking for trivially contains the element:

\begin{lstlisting}
x == k
-------------------------------
(node l k r) contains x => yes
\end{lstlisting}

If the tree begins with an element \texttt{x} that is bigger than the element \texttt{k} we are looking for, than we know that only the left subtree is of interest: all elements of the right subtree are bigger than \texttt{x}, and thus also of \texttt{k}:

\begin{lstlisting}
k < x
l contains k  => res
-------------------------------
(node l x r) contains k => res
\end{lstlisting}

Symmetrically for a tree that begins with an element \texttt{x} that is bigger than the element \texttt{k} we are looking for, than we know that only the right subtree is of interest: all elements of the left subtree are smaller than \texttt{x}, and thus also of \texttt{k}:

\begin{lstlisting}
k > x
r contains k  => res
-------------------------------
(node l x r) contains k => res
\end{lstlisting}

Consider now searching element \texttt{10} within tree:

\begin{lstlisting}
  5
 / \
3   7
\end{lstlisting}

This means resolving:

\begin{lstlisting}
-----------------------------------------------
(node nil 5 (node nil 7 nil)) contains 10 => ?
\end{lstlisting}

Since \texttt{10} is bigger than \texttt{5}, we proceed recursively into the right sub-tree:

\begin{lstlisting}
5 < 10
(node nil 7 nil) contains 10 => res
-------------------------------------------------
(node nil 5 (node nil 7 nil)) contains 10 => res
\end{lstlisting}

Again, since \texttt{10} is bigger than \texttt{7}, we proceed recursively into the right-subtree:

\begin{lstlisting}
10 < 7
nil contains 10 => res'
res := res'
------------------------------------
(node nil 7 nil) contains 10 => res
-------------------------------------------------
(node nil 5 (node nil 7 nil)) contains 10 => res
\end{lstlisting}

Searching within the empty tree always fails, therefore:

\begin{lstlisting}
res' := no
------------------------
nil contains 10 => res'
res := res'
------------------------------------
(node nil 7 nil) contains 10 => res
-------------------------------------------------
(node nil 5 (node nil 7 nil)) contains 10 => res
\end{lstlisting}

At this point we can begin unwinding. The first unwinding step yields:

\begin{lstlisting}
nil contains 10 => no
res := no
------------------------------------
(node nil 7 nil) contains 10 => res
-------------------------------------------------
(node nil 5 (node nil 7 nil)) contains 10 => res
\end{lstlisting}

after just another unwinding step we end up with the final result:

\begin{lstlisting}
(node nil 5 (node nil 7 nil)) contains 10 => no
\end{lstlisting}

\subsection{Performance}
The more the tree is balanced, the more effective search and insertion into a binary search tree will be. Let us intuitively analyse the number of steps we will need to perform during a binary search. Assume that we have a perfectly balanced tree with \texttt{n+1} elements; since the tree is balanced, then both the left and the right sub-trees contain exactly \texttt{n/2} elements each.

In the beginning we are searching among the whole \texttt{n+1} elements, that is our tree is:

\begin{lstlisting}
  x    = 1 element
 / \
l   r  = n/2 + n/2 elements
\end{lstlisting}

After comparing the searched value and the current element of the tree, we choose one between the two sub-trees and fully discard the other. This means that after one step we focus on a smaller sub-tree:

\begin{lstlisting}[mathescape=true]
  x'    = 1 element
 / \
l'  r'  $\approx$ n/4 + n/4 elements
\end{lstlisting}

After yet another step we get:

\begin{lstlisting}[mathescape=true]
  x''    = 1 element
 / \
l'' r''  $\approx$ n/8 + n/8 elements
\end{lstlisting}

It should be clear that every step roughly divides the number of active elements (the set within which we are searching) by two, therefore after \texttt{k} steps we have only

$$\frac{n}{2^k}$$

elements left to consider. We are sure that at worst we will stop when this set of elements left has at most one element, thus we stop when:

$$\frac{n}{2^k} = 1$$

By multiplying both sides by $2^k$, we get:

$$n = 2^k$$

And by taking the logarithm of both sides, we obtain the final result that:

$$\log_2 n = k$$

that is we will stop searching after (at most) \texttt{k} steps, which is the logarithm in base $2$ of \texttt{n}.\footnote{One might go back to the definition of logarithm for a moment to clarify this last step: \textit{the logarithm is the exponent to which a fixed value, the base, must be raised to produce the desired number}. This means literally that if $\log_b(x)=y$ then $b^y=x$.}

Logarithms grow quite slowly. This means that we will take roughly:
\begin{itemize}
\item $10$ search steps if the tree has $10^3$ elements
\item $20$ search steps if the tree has $10^6$ elements
\item $40$ search steps if the tree has $10^{12}$ elements
\end{itemize}

This makes binary search trees one of the most efficient data structures for handling large data sets, but one word of warning is necessary. The original assumption that binary search trees need to be balanced for such good performance properties to be verified is quite a strong hypothesis, which needs further exploration.


\section{Presentation and Discussion of Results}

As we briefly explained in section \ref{data-visualization}, the number of initial face examples is slightly greater than the number of non-face examples, so, we decided not to use accuracy as performance metric, because it becomes misleading. Instead, we used the combination of \textbf{F1 Score}, \textbf{Accuracy}, \textbf{Precision} and \textbf{Recall} to compare both models.

To clarify these concepts:

\begin{itemize}
\item \textbf{Precision} - the fraction of correctly classified positive examples from all classified as positive.
\item \textbf{Recall} - actual positive rate of all positive examples, that is, the fraction of correctly classified examples.
\item \textbf{F1 Score} - weighted average of Precision and Recall.
\end{itemize}

Furthermore, these concepts have mathematical representations, as follows:

\begin{itemize}
\item  \(Precision = \frac{TP}{TP+FP}\)  \\
\item  \(Recall = \frac{TP}{TP+FN}\) \\
\item  \(F1 Score = 2*\frac{Recall * Precision}{Recall + Precision}\) \\
\end{itemize}

\subsection{SVM}

After optimizing our SVM classifier, we tested it with our testing dataset. As we can see in figure \ref{fig:svm-cross-validation} previously showed, the results of both train and cross-validation phases were good, as we could find a combination of parameters where the accuracy of the model in both phases had the value of 1.

Hence, in the table \ref{table:svm-results} we can clearly see what values of \textbf{F1 Score}, \textbf{Accuracy}, \textbf{Precision} and \textbf{Recall} our model was able to achieve.

\begin{table}[H]
\centering
\caption{SVM Results}
\begin{tabular}{ccccc}
\cline{2-3}
\multicolumn{1}{l|}{}                & \multicolumn{1}{l|}{\textbf{Train \& Cross-Validation}} & \multicolumn{1}{l|}{\textbf{Test}} &  &  \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{F1 Score}} & \multicolumn{1}{c|}{1}                     & \multicolumn{1}{c|}{0.985}                              &  &  \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{Accuracy}} & \multicolumn{1}{c|}{1}                     & \multicolumn{1}{c|}{0.985}                              &  &  \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{Precision}} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{0.985} & & \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{Recall}} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{0.986} & & \\\cline{1-3} & & & &
\end{tabular}
\label{table:svm-results}
\end{table}

As we can see, and reiterating the behaviour of our SVM classifier on both testing and cross-validation phases, we can see that all four metrics have the value of \(1\). In the testing phase, the results were not perfect, that is, the values are a little bit lower than \(1\), however, were quite good, as we were able to achieve a success rate approximately close to \(0.985\). It is necessary to remember that these results are not related to just one execution, but to \(50\) executions of the entire flow. This means that the values presented in table \ref{table:svm-results}, represent the average of each metric for the \(50\) repetitions of the entire flow (dataset splitting, training, cross-validation and testing).

Furthermore, to help us understand if the classifier was systematically incorrectly classifying one of the classes, we decided to produce the confusion matrix that can be seen in figure \ref{fig:svm-confusion-matrix}.

\begin{figure}[htbp]
\centerline{\includegraphics[width=0.65\linewidth]{images/svm_conf_matrix.png}}
\caption{SVM - Confusion Matrix}
\label{fig:svm-confusion-matrix}
\end{figure}

As we analyse the confusion matrix related to the results on the testing dataset, in figure \ref{fig:svm-confusion-matrix}, we can see that the model it's not systematically miss-classifying some of the classes, which means that our data augmentation of the dataset was able to regularize the learning process of our model. We can also see that in each class we have less than 5 examples miss-classified, which we see as positive outcome.

\subsection{NN}

To the Neural Network classifier, the adopted flow was similar to the one detailed above, that is, we optimized our NN model and then tested it on the testing dataset.

The table \ref{table:nn-results} summarizes the result values the NN model was able to achieve.

\clearpage

\begin{table}[htbp]
\centering
\caption{NN Results}
\begin{tabular}{ccccc}
\cline{2-3}
\multicolumn{1}{l|}{}                & \multicolumn{1}{l|}{\textbf{Train \& Cross-Validation}} & \multicolumn{1}{l|}{\textbf{Test}} &  &  \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{F1 Score}} & \multicolumn{1}{c|}{1}                     & \multicolumn{1}{c|}{0.973}                              &  &  \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{Accuracy}} & \multicolumn{1}{c|}{1}                     & \multicolumn{1}{c|}{0.973}                              &  &  \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{Precision}} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{0.974} & & \\ \cline{1-3}
\multicolumn{1}{|l|}{\textbf{Recall}} & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{0.974} & & \\\cline{1-3} & & & &
\end{tabular}
\label{table:nn-results}
\end{table}

As can be seen in table \ref{table:nn-results}, as in our SVM model, for training and cross-validation phases, the model achieve the value of \(1\) to every metric, but for the testing phase, our NN model performed worse than our SVM model, achieving the average values of \(0.97\) to each evaluation metric.

Moreover, we also produced the confusion matrix to this model, as can be seen in figure \ref{fig:nn-confusion-matrix}. As in the SVM model, our NN classifier it's not systematically miss-classifying the examples, which confirms the conclusion we stated above about the data augmentation process. As in our SVM model, we can see that the miss-classified examples were also on a reduced number, although a little higher than in the SVM model. With this, we were able to conclude that neither of the models had systematic miss-classification problems and that our NN model was not able to generalize its predictions as much as the SVM classifier.

\begin{figure}[htbp]
\centerline{\includegraphics[width=0.65\linewidth]{images/nn_conf_matrix.png}}
\caption{NN - Confusion Matrix}
\label{fig:nn-confusion-matrix}
\end{figure}

The purpose of this paper was always to compare the two models against each other. However, we always thought the NN model would be better than the SVM model, even if by a little range. After these results, we came to the conclusion that this could be related with a number of causes:

\begin{itemize}
\item The dataset size that was not that big, and can harm the learning process of the NN model, that ends up needing more data than the SVM model.
\item The feature extractor function: both models used the same feature extraction function (HOG) as a mean to achieve common ground and a true state of comparison, and that could indeed be one of the reasons, since other feature extractors, such as \textit{Fast Gabor Filtering}, produce better results on Neural Network models.
\end{itemize}

These possibilities will be further detailed in section \ref{novelty}.

\subsection{External Test Examples}

Finally, after testing our two models with the testing dataset, we decided to test both models with external images that weren't part of the original dataset. Thus, we slightly modified how we processed the images and its predictions so we could test an example with multiple faces and a more complex context to assert how our models would handle that test case.

As we can see in figures \ref{fig:svm-family-result} and \ref{fig:nn-family-result}, both models successfully identified all the existent faces in the images and both models produced instances of false positive results. We can also see that the NN model produced more false positives (it produced three false positives in the testing image) than the SVM model, that only produced one. This does not come as a surprise, because we already saw that our SVM model produces better results than our NN model.

\begin{figure}[htbp]
\centerline{\includegraphics[width=1\linewidth]{images/svm_persons.png}}
\caption{SVM - Family External Image Test Result}
\label{fig:svm-family-result}
\end{figure}

\begin{figure}[htbp]
\centerline{\includegraphics[width=1\linewidth]{images/nn_persons.png}}
\caption{NN - Family External Image Test Result}
\label{fig:nn-family-result}
\end{figure}

With this external test image, we were asserted that both models were able to correctly detect all existent faces, and that both had problems with false positives, which was expected as none of our models produced result scores of approximately \(1\) in the testing phase.


\mychapter{4}{Lesson 4} %181005

\section{Negligible function}

What is exactly a negligible function? Below here there is a possible interpretation of this notion, taken from an answer to a question in the Cryptography Stack Exchange website:
\begin{quotation}
    ``[...] in modern cryptographic schemes, we generally do not try to achieve perfect secrecy [...]. Instead, we define security against a specific set of adversaries whose computational power is bounded. Generally, we assume an adversary that is bounded to run in time polynomial to $n$, where $n$ is the security parameter given to the key generation algorithm [...].

    So consider a scheme $\Pi$ where the only attack against it is the brute-force attack. We consider $\Pi$ to be secure if it cannot be broken by a brute-force attack in polynomial time.

    The idea of \emph{negligible probability} encompasses this exact notion. In $\Pi$, let's say that we have a polynomial-bounded adversary. Brute force attack is not an option. But instead of brute force, the adversary can try (a polynomial number of) random values and hope to guess the right one. In this case, we define security using negligible functions: The probability of success has to be smaller than the reciprocal of any polynomial function.

    And this makes a lot of sense: if the success probability for an individual guess is a reciprocal of a polynomial function, then the adversary can try a polynomial amount of guesses and succeed with high probability. If the overall success rate is $\oneover{\poly(n)}$ then we consider this attempt a feasible attack to the scheme, which makes the latter insecure.

    So, we require that the success probability must be less than the reciprocal of every polynomial function. This way, even if the adversary tries $\poly(n)$ guesses, it will not be significant since it will only have tried:
    \[
        \frac{\poly(n)}{superpoly(n)}
    \]
    As $n$ grows, the denominator grows far faster than the numerator and the success probability will not be significant.''\footnote{\linkicon \href
        {https://crypto.stackexchange.com/questions/5832/what-exactly-is-a-negligible-and-non-negligible-function}
        {\textsf{``What exactly is a negligible (and non-negligible) function?'' --- Cryptography Stack Exchange}}}
\end{quotation}

\begin{definition}
    Let $f : \nonneg \to \nonneg$ be a function. Then it is deemed \emph{polynomial}, and denoted as $f \in \poly(\lambda)$, iff:
    \[
        \exists c \in \nonneg : f(\lambda) \in O(\lambda^c) \qedhere
    \]
\end{definition}

% AP190904: Big O or small O?!?
% ^^^^^^^^: Big O, as specified here: https://en.wikipedia.org/wiki/Randomness_extractor#Randomness_extractors_in_cryptography
\begin{definition}
    Let $\nu : \nonneg \to \real$ be a function. Then it is deemed \emph{negligible}, and denoted as $\nu \in \negl(\lambda)$, iff:
    \[
        \forall f \in \poly(\lambda)  \implies \nu(\lambda) \in O\left(\oneover{f(\lambda)}\right) \qedhere
    \]
\end{definition}

Note that these actually represent upper bounds for functions: a negligible function adheres to the polynomial function definition, whereas the opposite isn't true. To sum it up: $\negl \subset \poly$.

\begin{exercise}
    Let $p(\lambda), p'(\lambda) \in \poly(\lambda)$ and $\nu(\lambda), \nu'(\lambda) \in \negl(\lambda)$. Then prove the following:

    \begin{enumerate}
        \item $p(\lambda) \cdot p'( \lambda) \in  \poly(\lambda)$
        \item \label{ex:negl} $\nu(\lambda) + \nu'(\lambda) \in \negl(\lambda)$
    \end{enumerate}
\end{exercise}

\begin{solution}[\ref{ex:negl}]

    \todo{Questa soluzione usa disuguaglianze deboli; per essere negligibile una funzione dev'essere strettamente minore di un polinomiale inverso. Da approfondire}

    We need to show that for any $c \in \nonneg $, then there is $n_0$ such that $\forall n > n_{0} \implies \nu(n) + \nu'(n) < \oneover{n^c}$.

    Consider an arbitrary $c \in \nonneg$. Then, since $c + 1 \in \nonneg$, and both $\nu$ and $\nu'$ are negligible, there exist $n_\nu$ and $n_{\nu'}$ such that:

    \begin{align*}
        \forall n \geq n_{\nu} \implies& \nu(n) \leq n^{-(c+1)} \\
        \forall n \geq n_{\nu'} \implies& \nu'(n) \leq n^{-(c+1)}
    \end{align*}

    Fix $n_{0} = \max(n_{\nu}, n_{\nu'})$. Then, since $n_0 \geq 2$, $\forall n \geq n_0$ we have:
    \begin{align*}
        & \nu(n) + \nu'(n) \\
        \leq& n^{-(c+1)} + n^{-(c+1)} \\
        =& 2n^{-(c+1)} \\
        \leq& n^{-c}
    \end{align*}
    Therefore, we conclude that $\nu(n) + \nu'(n) \in \negl(\lambda)$.
\end{solution}

\section{One-way functions}

From here, we start defining an object that is fundamental to everyday cryptography: the \emph{one-way} function, or \owf{} in short. Colloquially, a one-way function is a function that is ``easy to compute'', while being ``hard to invert'' at the same time, the concept of hardness being borrowed by complexity theory.

\begin{definition}
    Let $f : \binary^{n(\lambda)} \to \binary^{n(\lambda)}$ be a function. Then it is a \owf{} iff:
    \[
        \forall \adversary \in \ppt\, \exists \nu(\lambda) \in \negl(\lambda) : \Pr\left[\cryptog{owf}[f](\lambda) = 1 \right] \leq \nu(\lambda) \qedhere
    \]
\end{definition}

\begin{cryptogame}
    {owfdef}
    {One-Way Function hardness}
    {owf}

    \receive{\shortstack[l]{
        $x \pickUAR \binary^n$ \\
        $y = f(x)$
    }}{$y$}{}

    \cseqdelay

    \send{}{$x'$}{\textsc{Output 1 iff} $f(x') = y$}

\end{cryptogame}

The structure of the ``game'' appearing in the definition is depicted in figure \ref{cryptogame:owfdef}. Do note that the game does not check for $x = x'$, but rather for $f(x) = f(x')$; in a sense, the adversary is not trying to guess what the original $x$ was: its goal is to find any value such that its image is $y$ according to $f$, and such value may very well not be unique.

\begin{exercise} \label{ex:owf}
    Prove the following claims:
    \begin{enumerate}
        \item There exists an inefficient adversary that wins $\cryptog{owf}[f]$ with probability $1$
        \item There exists an efficient adversary that wins $\cryptog{owf}[f]$ with probability $2^{-n}$
    \end{enumerate}
\end{exercise}

\begin{solution}[\ref{ex:owf}]\
    \begin{enumerate}
        \item Adversary uses a brute-force attack.
        \item Adversary makes a random guess.
    \end{enumerate}
\end{solution}

A one-way function can be thought as a function which is very efficient in generating ``puzzles'' that are very hard to solve from scratch. Furthermore, given a candidate solution, one can efficiently verify its validity. In a twist of perspective, for a given couple $(\mathcal{P}_\textsc{gen},\mathcal{P}_\textsc{ver})$ of a puzzle generator and a puzzle verifier, another ``game'' can be drawn as in figure \ref{cryptogame:owpuzzle}.

\begin{cryptogame}
    {owpuzzle}
    {The puzzle game}
    {puzzle}

    \receive{$(x, y) \pickUAR \mathcal{P}_\textsc{gen}$}{$y$}{}

    \cseqdelay

    \send{}{$x'$}{\textsc{Output} $\mathcal{P}_\textsc{ver}(x', y)$}

\end{cryptogame}

It can also be said that the one-way puzzle problem is in \textsc{np}, because witness checking is easy, but not in \textsc{p} because finding a solution to begin with is hard.

\subsubsection{Impagliazzo's Worlds}

Suppose to have Gauss, a genius child, and his professor. The professor gives to Gauss some mathematical problems, and Gauss wants to solve them all.

Imagine now that, if using one-way functions, the problem is $f(x)$, and its solution is $x$. According to Impagliazzo, we live in one of these possible worlds:
\begin{itemize}
    \item \textit{Algorithmica}: $\textsc{P} = \textsc{NP}$, meaning all efficiently verifiable problems are also efficiently solvable.

    The professor can try as hard as possible to create a hard scheme, but he won't succeed because Gauss will always be able to efficiently break it using the verification procedure to compute the solution

    \item \textit{Heuristica}: \textsc{NP} problems are hard to solve in the worst case but easy on average.

    The professor, with some effort, can create a game difficult enough, but Gauss will solve it anyway; here there are some problems that the professor cannot find a solution to

    \item \textit{Pessiland}: \textsc{NP} problems are hard on average but no one-way functions exist

    \item \textit{Minicrypt}: One-way functions exist but public-key cryptography is impractical

    \item \textit{Cryptomania}: Public-key cryptography is possible: two parties can exchange secret messages over open channels
\end{itemize}

\section{Computational Indistinguishability}

Distribution ensembles $X = \{X_{\lambda \in \mathbb{N}}\}$ and $Y = \{Y_{\lambda \in \mathbb{N}}\}$ are distribution sequences.

\begin{definition}[\emph{Comp. indist.}]
    Let $X$ and $Y$ be two distribution sequences; they are deemed \emph{computationally indistinguishable}, written as ``$X \compindist Y$'' iff:
    \[
        \forall \adversary \in \ppt \exists \nu(\lambda) \in \negl(\lambda) : \left|\Pr[\adversary(X_\lambda) = 1] - \Pr[\adversary(Y_\lambda) = 1]\right| \leq \nu(\lambda) \qedhere
    \]
\end{definition}

In words: any \emph{efficient} adversary attempting to distinguish outputs between the two ensembles will succeed with a probability that is negligibly different than randomly guessing. Note the emphasis on ``efficient'', which makes this relationship between ensembles weaker than what would be a purely statistical one.

With the purpose of making these new concepts clearer, it is presented this mental game.

\todo{AP181129-2344: There may be room for improvement, but I like how it's worded: it puts some unusual perspective into the cryptographic game, and it could be a good thing since it closely precedes our first reduction, and the whole hybrid argument mish-mash.}

A \emph{challenger} \challenger{} chooses a value $z$ among $X_\lambda$ and $Y_\lambda$, and gives it to a \emph{distinguisher} \distinguisher{}. In turn, \distinguisher{} has to correctly guess which was the source of $z$: either $X_\lambda$ or $Y_\lambda$.

If we let $X_\lambda$ and $Y_\lambda$ to be \emph{computationally indistinguishable}, then, fixed 1 as one of the sources, the probability that \distinguisher{} says ``1!'' when \challenger{} picks $z$ from $X_\lambda$ is \emph{not so far} from the probability that \distinguisher{} says ``1!'' when \challenger{} picks $z$ from $Y_\lambda$.

So, this means that, when this property is verified by two random variables, there isn't too much \textit{difference} between the two variables in terms of information available to \distinguisher{}, otherwise the distance between the two probabilities should be much more than a negligible quantity.

\begin{lemma} \label{lem:compmall}
    Let $f$ be a function that has polynomial time-complexity. Then, for any two ensembles $X$ and $Y$:
    \[
        X \compindist Y \implies f(x) \compindist f(y) \qedhere
    \]
\end{lemma}

\begin{proof}
    This proof is by contradiction and uses a reduction. Let $X \compindist Y$ be two indistinguishable ensembles, and $f \in \ppt$ an arbitrary poly-time complex function. Assume there exists an adversary \adversary{} to the challenge of distinguishing the ensembles' images $f(X)$ from $f(Y)$ that does efficiently succeed, as shown in figure \ref{cryptogame:fdistin}.

    \begin{cryptogame}
        {fdistin}
        {A distinguisher for $f$}
        {$f$-\textsc{dist}}

        % Adversary asks for the challenge. Challenger chooses evenly between X and Y, and samples a value; then it sends that value's image by f to the adversary
        \receive{\shortstack[l]{
            $z_0 \pickUAR X$ \\
            $z_1 \pickUAR Y$ \\
            $b \pickUAR \binary$
        }}{$f(z_b)$}{}

        \cseqdelay

        % Adversary attempts to guess from which distribution the value's image came from
        \send{}{$b'$}{\textsc{Output 1 iff} $b' = b$}

    \end{cryptogame}

    Fix this adversary to be the distinguisher $\distinguisher_f$. From here, another adversary $\adversary \in \ppt$ can use $\distinguisher_f$ to effectively distinguish the original ensembles, as depicted in figure \ref{cryptoredux:fdistin}:
    \begin{enumerate}
        \item \adversary{} asks for the original sample from the challenger
        \item \adversary{} applies $f$ on the sample
        \item \adversary{} relays the resulting image to $\distinguisher_f$
        \item $\distinguisher_f$ replies with his outcome
        \item \adversary{} relays the outcome to the challenger
    \end{enumerate}
    All of this is done in polynomial time, since all functions and machines involved in the process operate in \ppt. This contradicts the computational indistinguishability of $X$ and $Y$.

    \begin{cryptoredux}
        {fdistin}
        {Distinguisher reduction}
        {dist}
        {f-\textsc{dist}}

        % The distinguisher asks for the challenge
        \receive{\shortstack[l]{
            $z_0 \pickUAR X$ \\
            $z_1 \pickUAR Y$ \\
            $b \pickUAR \binary$
        }}{$z_b$}{}

        % The distinguisher applies f to z and relays the image to A
        \invoke{}{$f(z_b)$}{}

        % Adversary distinguishes between the two distributions
        \return{}{$b'$}{}

        \send{}{$b'$}{\textsc{Output 1 iff} $b' = b$}

    \end{cryptoredux}

\end{proof}

\section{Pseudo-random generators}

A deterministic function $G \in \binary^\lambda \to \binary^{\lambda + l(\lambda)}$ is called a \emph{pseudo-random generator}, or \prg{} in short, iff:

\begin{itemize}
    \item $G \in \ppt(\lambda)$
    \item $|G(s)| = \lambda + l(\lambda)$ % Well duh, we can see it by def...
    \item Given $U_n$ to be a distribution ensemble of $n$ uniform random variables:
    \[
        G(U_{\lambda}) \compindist U_{\lambda + l(\lambda)}
    \]
\end{itemize}

\begin{cryptogame}
    {prg}
    {The pseudorandom game}
    {prg}

    \receive{\shortstack[l]{
        $x_0 \pickUAR G(U_\lambda)$ \\
        $x_1 \pickUAR U_{\lambda+l(\lambda)}$ \\
        $b \pickUAR \binary$
    }}{$x_b$}{}

    \cseqdelay

    \send{}{$b'$}{\textsc{Output 1 iff} $b = b'$}

\end{cryptogame}

So, if we take $s \pickUAR U_\lambda$, the output of $G$ will be indistinguishable from a random pick from $U_{\lambda + l(\lambda)}$.


\chapter{Week 0: The function and the field}

% ================================================================

\section{The function and other preliminaries}

\subsection{Set terminology and notation}
\begin{definition}[Set]
  A set is an unordered collection of objects.
\end{definition}
\begin{itemize}
\item $\in$: indicates that an object belongs to a set. (e.g. $a\in\{a,b,\ldots\}$)
\item $\sA \subseteq \sB$: ``$\sA$ is a \textbf{subset} of $\sB$''. Every element of $\sA$ is also an element of $\sB$
\item $\sA = \sB$: two sets are equal if they contain exactly the same elements.
\end{itemize}

\subsubsection{Set expressions}
$\{ x\in\R : x \ge 0 \}$ is the set of nonnegative numbers. First part specifies where the elements of the set comes from and introduces variables. The second part gives a rule that restricts which elements specified in the first part actually get to make it into the set.

\begin{definition}[Cardinality]
  If a set $\sS$ is not infinite, we use $|\sS|$ to denote the number of elements or \emph{cardinality} of the set.
\end{definition}

\begin{definition}
  $\sA \times \sB$ is the set of all pairs $(a, b)$ where $a\in\sA$ and $b\in\sB$
\end{definition}

\subsection{The function}
Informally, for each input element in a set $\sA$, a function assigns a single output element from another set $\sB$
\begin{itemize}
\item $\sA$ is called the \textbf{domain} of the function
\item $\sB$ is called the \textbf{co-domain}
\end{itemize}

\begin{definition}[Function]
  A function is a set of pairs $(a, b)$ no two of which have the same first element.
\end{definition}

\begin{definition}[Image]
  The output of a given input is called the \emph{image} of that input. The image of $q$ under a function $f$ is denoted $f(q)$
\end{definition}

If $f(q) = r$, we say $q$ maps to $r$ under $f$. In Mathese, we write this as $q \mapsto r$.

The set from which all the outputs are chosen is called the co-domain.
We write:
\begin{equation*}
  f : \sD \rightarrow \sF
\end{equation*}
when we want to say that $f$ is a function with domain $\sD$ and co-domain $\sF$.

\begin{definition}[Image of a function]
  The image of a function is the set of all images of inputs. Mathese: $\Ima f$
\end{definition}

\begin{example}
  $\cos : \R \rightarrow \R$, which means the domain is $\R$, and the co-domain is $\R$. The image of $\cos(x)$, $\Ima \cos$ is $\{ x\in\R : -1 \le x \le 1 \}$.
\end{example}

\begin{definition}
  For sets $\sF$ and $\sD$, $\sF^{\sD}$ denotes all functions from $\sD$ to $\sF$.
\end{definition}

\begin{proposition}
  For finite sets, $|\sF^{\sD}| = |\sF|^{|\sD|}$.
\end{proposition}

\begin{definition}[Identity function]
  For any domain $\sD$. $\mathrm{id}_{\sD} : \sD \rightarrow \sD$ maps each domain element $d$ to itself.
\end{definition}

\begin{definition}[Functional composition]
  For functions $f : \sA \rightarrow \sB$ and $g : \sB \rightarrow \sC$, the functional composition of $f$ and $g$ is the function $(g \circ f) : \sA \rightarrow \sC$ defined by $(g \circ f)(x) = g(f(x))$.
\end{definition}

\begin{proposition}
  $h \circ (g \circ f) = (h \circ g) \circ f$
\end{proposition}

\begin{definition}[Functional inverses]
  Functions $f$ and $g$ are functional inverses if $f \circ g$ and $g \circ f$ are defined and are identity functions. A function that has an inverse is invertible.
\end{definition}

\begin{definition}
  $f : \sD \rightarrow \sF$ is \textbf{one-to-one} if $f(x) = f(y)$ implies $x = y$.
\end{definition}

\begin{definition}
  $f : \sD \rightarrow \sF$ is \textbf{ontox} if for every $z \in \sF$ there exists an $a$ such that $f(a) = z$.
\end{definition}

\begin{proposition}
  Invertible functions are one-to-one.
\end{proposition}

\begin{theorem}[Function Invertibility Theorem]
  A function $f$ is invertible if and only if it is one-to-one and onto.
\end{theorem}

% ================================================================

\section{The Field: Introduction to complex numbers}

$i = \sqrt{-1}$ is an imaginary number, this is a solution to an equation such as $x^2 = -1$. For $(x-1)^2 = 9$, the solution is $x = 1 + 3i$.

A \textbf{complex number} has a real part and an imaginary part.

\subsection{Field notation}
When we want to refer to a field without specifying which field we will use the notation $\fF$.

We study three fields;
\begin{itemize}
\item The field $\R$ of real numbers.
\item The field $\sC$ of complex numbers.
\item The finite field $GF(2)$, which consists of $0$ and $1$ under $\mod 2$ arithmetic.
\end{itemize}

\section{The Field of playing with $\sC$}
We can interpret real and imaginary parts of a complex number as $x$ and $y$ coordinates. Assume that $z\in\sC$.

\begin{itemize}
\item \textbf{Translation} $f(z) = z + z_0, z_0 \in \sC$. A translation can ``move'' a picture anywhere in the complex plane.
\item \textbf{Scaling} $f(z) = mz, m \in \R$.
\item \textbf{Invert} $f(z) = (-1)z$.
\item \textbf{Rotate counterclockwise by 90 degreesx} $f(z) = iz$.
\item \textbf{Rotating by an angle} $f(z) = z \cdot e^{\tau i}$, does rotation by angle $\tau$.
\end{itemize}

\section{The Field of playing with $GF(2)$}
$GF(2) = \text{Galois Field 2}$, has just two elements: $0$ and $1$.
\begin{itemize}
\item Addition is like exclusive-or. (e.g. $\mathrm{XOR}(a, b) = a \not\equiv b ; a, b \in GF(2)$)
\item Multiplication is just like normal multiplication.
\end{itemize}