\lab{Algorithms}{Value Function Iteration}{Value Function Iteration}
\newcommand\ve{\varepsilon}
\objective{This section teaches the fundamentals of Dynamic Programming using value function iteration.}

Often it is of interest to optimize decision making in some sequential process.  For example, an oil company may need to decide
how much oil to excavate and sell each month as prices change, a person entering retirement may need to decide how much of their
savings to spend each year, or a model of economic growth may require a decision about how much to invest in capital versus how
much to spend each year.  In this lab we will formulate a general dynamic optimization problem.  We will explore techniques for
solving such a problem with both finite and infinite time horizons.

\section*{The Sequential Problem, Finite Horizon}
Suppose there are time periods $t=0,1,\ldots, T$ and at each time period we take an action $c_t$. Furthermore, at the beginning
of each time period $t$ we are in some state $W_t$.  In many cases $W_t$ might represent an available resource, such as money.
At each time we receive some reward, $u(W_t,c_t)$, for taking action $c_t$ given state $W_t$.  We assume that rewards are worth
more now than later. We let $\beta\in (0,1)$ represent what is called the discount factor and gives the ratio of preference for
rewards today versus rewards tomorrow.  For example, receiving a dollar today is preferable to receiving a dollar in a year
because taking a dollar today and putting it in a  savings account results in having more than a dollar in a year.  Lastly, over
time our state variable $W_t$ changes according to some rule depending on the previous state and our actions,
\begin{equation}
\label{motion}
W_{t+1} = g(W_t,c_t).
\end{equation}
Equation \eqref{motion} is sometimes referred to as the law of motion, as it describes how we move from state to state.
Mathematically such a problem can be represented as follows:
\begin{equation*}
\max \sum_{t=0}^T \beta^t u(W_t,c_t) \quad \text{s.t.} \quad W_{t+1} = g(W_t,c_t)
\end{equation*}
where our initial state, $W_0$ is given.  There may also be restrictions on our choices $c_t$.  For example, in many applications
the state $W_t$ represents the amount of some resource available, and $c_t$ represents the amount we use up in time period $t$.
 In this case we would require $c_t \in [0,W_t]$.

For simplicity, lets assume that $u$ is a function of $c_t$ only (this is often, though not always, the case in practice).
 First let's consider the case that $T=0$.  So we maximize
\begin{equation}\label{1perprob}
u(c_0)
\end{equation}
over $c_0 \in [0,W_0]$.  In most cases $u$ is increasing, which we will assume here.  In this case it will be optimal to choose
 the largest value of $c_0$ possible, that is, $c_0 = W_0$.  Thinking of $W_0$ as our available resources, this simply means,
 that if we don't have future periods to consider, we will use all of it.

In fact this is always true in the last period.  In the problem with $T$-periods we know that we will use all of our resources
remaining in period $T$.  Consider the two period problem:
\begin{equation}
\label{2perprob}
\max \, \{u(c_0) + \beta u(c_1)\}
\end{equation}
where $c_0 \in [0,W_0]$, $c_1 \in [0,W_1]$ and $W_1 = g(W_0,c_0)$.  We know that in the last period we will use all of our
remaining resources so that $c_1 = W_1$.  Substituting gives
\begin{equation*}
\max \, \{u(c_0) + \beta u(g(W_0,c_0))\}.
\end{equation*}
Then we need only determine $c_0$.  Taking the derivative of \eqref{2perprob} with respect to $c_0$ and setting equal to zero
gives the first order condition
\begin{equation}
\label{FOC}
u'(c_0) = -\beta u'(g(W_0,c_0))g_c(W_0,c_0)
\end{equation}
where $g_c$ is the partial derivative of $g$ with respect to $c$.

Given a specific form for $u$ we could solve for $W_1$ and obtain the optimal solution.  In fact, we can solve a problem of
any length $T$ in this manner by starting at the last time period and working backward.  We know that $W_{T+1} = 0$.  Working
backward in time we obtain an equation at each time step $t<T$ by taking the derivative with respect to $c_t$ and setting equal
 to zero.  This process is called backward induction.  The equations at each time step, such as equation \eqref{FOC} are
 sometimes called the inter-temporal Euler equations.  These equations, along with $c_T = W_T$ make $T+1$ equations to go with
 our $T+1$ unknowns $c_0,c_1,c_2,\ldots,c_T$ where we can use the law of motion to relate the $c_t$ and $W_t$.

\section*{The Recursive Problem, Finite Horizon}
Approaching the problem sequentially like this can be somewhat messy.  The dynamic programming approach we consider now is more
easily adaptable to many situations.  The key to the dynamic programming approach is to define our optimization problem in terms
 of subproblems.  Notice that if we are in time period $t$, we face a problem of exactly the same form as the problem at time
 $0$.  We are in some state $W_t$, and want to maximize the sum from $t$ to $T$.  With this idea in mind, we define a function
  $V_t(W_t)$ called the value function.  The function $V_t$ gives the value of entering time $t$ in state $W_t$ and making
  optimal decisions moving forward.  So
\begin{equation*}
V_{t-1}(W_{t-1}) = \max_{c_t} \left\{u(c_{t-1}) + \beta V_t(g(W_{t-1},c_{t-1}))\right\}.
\end{equation*}
This is called the Bellman Equation.  The key to this formulation is that we decide what to do in period $t-1$ with the
assumption that our actions in the remaining periods will be optimal.  This is called the principal of optimality.

Let us consider a specific example from economics called The Cake Eating Problem.  Suppose $W_t$ represents the amount of
cake available at time $t$.  At each time period we can choose how much to consume.  What we eat, $c_t$, gives us a reward.
What we save, $W_t-c_t$, does not give us a reward (until it is eaten in a later period).  The law of motion \eqref{motion}
becomes
\begin{equation}\label{LOM_EX}
W_{t+1} = g(W_t,c_t) = W_t-c_t
\end{equation}

Now we have completely defined the problem.  The Bellman Equation is
\begin{equation*}
V_{t-1}(W_{t-1}) = \max_{c_t} \left\{u(c_{t-1}) + \beta V_t(W_{t-1}-c_{t-1})\right\}.
\end{equation*}
Notice that by the law of motion, each $c_t$ is determined by $W_t$ and $W_{t+1}$.
In fact, rearranging \eqref{LOM_EX}, we have
\begin{equation*}
c_t = W_t - W_{t+1}.
\end{equation*}
We can therefore rewrite the value function as
\begin{equation}
V_{t-1}(W_{t-1}) = \max_{W_t} \left\{u(W_{t-1} - W_{t}) + \beta V_t(W_t)\right\}.
\label{cake_valfn}
\end{equation}

We see that determining the optimal actions $c_t$ is equivalent to determining the optimal states $W_{t}$ in the
above formulation. The solution to this
problem is often called a \emph{policy function}.  A policy function determines an action based on the current
state.  Denoting the policy function by $\psi$, this can be written as
\begin{equation*}
W_{t+1}=\psi_t \left(W_t\right).
\end{equation*}
The policy function gives the optimal amount of cake to leave for next period (equivalent to the amount of consumption) given
the amount of cake at the start of the period.  In other words, it determines the choice of $W_t$ that satisfies the $\max$
condition in \eqref{cake_valfn}.

As before, we know that in the last time period we should not save anything.  So $V_{T+1}(W_{T+1}) = 0$, i.e. there is
no value in leaving wealth for period $T+1$. Stated in another way, our action at time $T$ should be to eat all of the
remaining cake $W_T$, so $W_{T+1} = \psi_T(W_T) = 0$.
Plugging this result into the Bellman Equation gives us $V_T(W_T) = u(W_T)$.
Now consider the value function equation for period $T-1$:
\begin{align*}
V_{T-1}(W_{T-1}) &= \max_{W_T} \left\{u(W_{T-1} - W_T) + \beta V_T(W_T)\right\} \\
                 &= \max_{W_T} \left\{u(W_{T-1} - W_T) + \beta u(W_T)\right\}.
\end{align*}
We can determine this value by optimizing over $W_T$, where $0 \leq W_T \leq W_{T-1}$.
Continuing backwards in this manner leads us to the solution of the original problem.

\begin{problem}
\label{prob:cake_prob}
Follow the steps below to solve the problem described above.  Take $u(c_t) = \sqrt{c_t}$.
You will write a function called \li{eatCake} that takes parameters $\beta$ (the discount factor),
$N$ (the number of discrete cake values to consider), $W_{max}$ (the original size of the cake,
set to the default value of $1$), a keyword argument \li{finite} (set to default value \li{True}),
a keyword argument  $T$ (the number of time periods, set to
default value \li{None}), and a keyword argument \li{plot}, which indicates whether
to plot the computed results. The function should return arrays representing the value function and the
policy function (we describe how to compute these in the following steps).
\begin{enumerate}
\item Approximate the continuum of possible cake sizes by creating an array
of evenly-spaced values that range from to 0 to $W_{max}$ inclusive.
Let the number of possible cake values be given by $N$. In Python, this can be accomplished easily by
using the \li{linspace} function in NumPy. You should obtain an array (call it $w$) of the form
\[
w = (w_1, w_2, \ldots, w_N),
\]
where $w_1 = 0$ and $w_N = W_{max}$.

\item Note that in order to compute the value function, we need $u(W_{t-1}-W_t)$.
We will pre-compute all possible such values and store them in an array, as follows.
Create an $N$ by $N$ matrix that contains
all possible values of $W_{t-1} - W_t$ (where $W_{t-1}$ corresponds to rows and $W_{t}$ to columns).  Make sure that
$c_t \geq 0$ is satisfied by replacing negative entries in the matrix with zero.  Then take the square root to get a matrix
of $u(W_{t-1}-W_t)$.  To make sure we do not choose $W_{t-1} - W_t < 0$ when maximizing, replace the corresponding entries of
the $u(W_{t-1}-W_t)$ matrix with a large negative number (e.g. $-10^{10}$). You should end up with a matrix whose
$(i,j)$-th entry is equal to $\sqrt{w_i - w_j}$ when $i \geq j$, and is equal to $-10^{10}$ when $i < j$.

\item Next, create an $N$ by $T+2$ (corresponding to $t=0,1,\ldots, T+1$) matrix representing the value function for a given
time $t$ and state $W_t$.  We can initialize it to zeros and begin filling in the columns starting with the last (which we
know is zeros).

\item Now we are ready to iterate backward and compute the value function for each time period.  To find $V_T$, we first compute
$u(W_T - W_{T+1}) + \beta V_{T+1}(W_{T+1})$ for all values of $W_{T}$ and $W_{T+1}$.  This will result in an $N$ by $N$ matrix
where the rows correspond to values of $W_{T}$ and the columns correspond to values of $W_{T+1}$.
%Note that to compute this
%we need a matrix representing $\beta V_{T+1}(W_{T+1})$.  Because this quantity does not depend on $W_T$, its rows should be
%equal.  To do this, we want to take $\beta V_{T+1}(W_{T+1})$ as a row vector and stack this vector to create a matrix with
%equal rows.  There are multiple ways to do this.  One is the \li{np.repeat} function.  For example, if \li{b} is a row vector
%it could be used like the following.
%\begin{lstlisting}
%b = [[1, 2, 3]]
%np.repeat(b, 3, axis = 0)
%array([[1, 2, 3],
%[1, 2, 3],
%[1, 2, 3]])
%\end{lstlisting}
%In general, be careful about having the correct rows, columns, transposes, etc throughout your code.

Now we maximize over choices of $W_{T+1}$ (choosing how much to save for next period).  Then we will have a row vector
representing the value function for period $T$ across all possible $W_{T+1}$.  Iterate this procedure to fill in the value
function for all $t=0,1,\ldots, T+1$.

\item In each iteration, you maximize to find the value function at time $t$.  Save the values of $W_{t+1}$ that achieve the
maximum.  The result is an $N$ by $T+1$ matrix whose $(n,t)$ entry gives the optimal amount of cake to leave for period
$t+1$ given that we start period $t$ with the the $n$-th value of our vector of cake.  This is the policy function.

\item If the keyword argument \li{plot} is set to \li{True}, plot the surface of the Value and Policy functions.
This can be done by including the following import lines
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> from matplotlib import cm
>>> from mpl_toolkits.mplot3d import Axes3D
\end{lstlisting}
and using the following code:
\begin{lstlisting}
>>> W = np.linspace(0, Wmax, N)
>>> x = np.arange(0, N)
>>> y = np.arange(0, T+2)
>>> X, Y = np.meshgrid(x, y)
>>> fig1 = plt.figure()
>>> ax1 = Axes3D(fig1)
>>> ax1.plot_surface(W[X], Y, np.transpose(V), cmap=cm.coolwarm)
>>> plt.show()

>>> fig2 = plt.figure()
>>> ax2 = Axes3D(fig2)
>>> y = np.arange(0,T+1)
>>> X, Y = np.meshgrid(x, y)
>>> ax2.plot_surface(W[X], Y, np.transpose(psi), cmap=cm.coolwarm)
>>> plt.show()
\end{lstlisting}
where \li{W} is the vector of cake amounts, \li{V} is the value function, and \li{psi} is the policy function.

\item Return the arrays giving the value function and the policy function.
\end{enumerate}

Solve the problem using cake size $1$, discount factor $\beta = .9$, number of time periods $T = 10$, and number of
discrete cake values $N = 100$. You should also try plotting the value and policy functions for fixed time periods across
$W_t$, or for fixed $W_t$
across time and make sure that these plots fit your intuition. See Figure \ref{fig:valueslices}. Your output should
agree with the figure.
\end{problem}

\begin{figure}
\begin{subfigure}{.5\textwidth}
    \includegraphics[width=\textwidth]{fixed_time.pdf}
\end{subfigure}
\begin{subfigure}{.5\textwidth}
    \includegraphics[width=\textwidth]{fixed_w.pdf}
\end{subfigure}
\caption{Slices of the finite horizon value function for fixed values of $t$ and $W$.}
\label{fig:valueslices}
\end{figure}

\section*{The Recursive Problem, Infinite Horizon}\label{SecRecProbInFHor}
Next we consider an infinite horizon problem.  For simplicity, we continue with the example from the previous section.
Suppose that rather than optimizing over $t = 0,1,\ldots,T$, we wish to optimize over an infinite time horizon:
\begin{equation*}
\max \sum_{t=0}^\infty \beta^t u(W_t,c_t) \quad \text{s.t.} \quad W_{t+1} = g(W_t,c_t).
\end{equation*}
Since at any time $t$, there are an infinite number of periods remaining, one might suspect that the optimal policy will
not depend on the current time $t$.

\begin{problem}
\label{prob:cake_prob2}
Compute the solution to Problem \ref{prob:cake_prob} with $T = 1000$, and the rest of the inputs the same.
Plot the policy function across time for fixed $W_t = 1$.
Notice that it is the same for all time periods, except those near the end time $T$.
\end{problem}

As suggested by the results of Problem \ref{prob:cake_prob2},  the policy function for the infinite horizon problem does not
depend on the time $t$ (this can be proved).  That is, at any time $t$, the optimal decision depends only on the amount of cake
at the beginning of the period, not the value of $t$.  So everything can now be written in terms of variables today and variables
tomorrow. We will denote variables tomorrow with a ``$\:'\:$".
\begin{equation}
\label{EqBellman}
V\left(W\right) = \max_{W'\in[0,W]}\:\: \left\{u\left(W - W'\right) + \beta V\left(W'\right)\right\}
\end{equation}
Note that the value function $V$ on the left-hand-side of \eqref{EqBellman} and on the right-hand-side are the same function.

Because the problem now has an infinite horizon, the nature of the solution is a little different. The solution to \eqref{EqBellman}
is a policy function $W'=\psi(W)$ that creates a fixed point in $V$. In other words, the solution is a policy function $\psi(W)$
that makes the function $V$ on the left-hand-side of \eqref{EqBellman} equal the function $V$ on the right-hand-side.

Define $C$ as an operator on any value function $V_k\left(W\right)$. Let $C$ perform the following operation.
\begin{equation}
\label{EqContraction}
C\Bigl(V_k\left(W\right)\Bigr) \equiv \max_{W'\in[0,W]}\:\: \left\{u\left(W-W'\right) + \beta V_k\left(W'\right)\right\}.
\end{equation}
Note that the value function on the right-hand-side of \eqref{EqContraction} and on the left-hand-side are the same function $V_k$,
but have a different inputs--$W$ versus $W'$. The operator $C$ takes in a function $V_k$, and gives a new
function which we will call $V_{k+1}$:
\begin{equation*}
V_{k+1}\left(W\right) \equiv C\Bigl(V_k\left(W\right)\Bigr).
\end{equation*}
The value function $V_{k+1}$ that results from the operation $C$ is not necessarily the same as the value function that the system
began with $V_k$. However, according to equation \eqref{EqBellman} we seek a $V$ such that $C(V) = V$.  The solution, then, is the
fixed point in $V$.
\begin{equation*}
C\Bigl(V_k\left(W\right)\Bigr) = V_{k+1}\left(W\right) = V_k\left(W\right) = V\left(W\right)
\end{equation*}

When trying to solve a fixed point equation, it is often very helpful to utilize the Contraction Mapping Principle, which
guarantees the existence of a fixed point of a mapping, provided that the map sends any two distinct inputs to outputs
that are strictly closer to each other than the inputs, in a controlled way. This principle also provides a constructive
way to obtain the fixed point, namely by iterating the map.
Fortunately, it can be shown that if $u(\cdot)$ is real-valued, continuous, and bounded, $\beta\in(0,1)$, and that the constraint
set $W'\in[0,W]$ is nonempty, compact-valued, and continuous, then the operator $C$ is a contraction and thus we can obtain
a solution $V$ by iteration:
\begin{equation*}
\lim_{k\rightarrow\infty}\: C^k\Bigl(V_0\left(W\right)\Bigr) = V_k(W) =  V\left(W\right)
\end{equation*}
for any $V_0$.

Remember, in the infinite horizon problem both the value and policy functions do not depend on time.  Computationally, this means that
 the value and policy functions in the infinite horizon problem are one dimensional.

\begin{problem}
Expand your \li{eatCake} function to solve the Cake Eating Problem with an infinite time horizon. If the keyword argument
\li{finite} has the value \li{True}, then your function should behave as in Problem \ref{prob:cake_prob}, solving the finite
time horizon problem. However, if \li{finite = False}, solve the infinite time horizon problem through the following steps.
Both problems will require you to pre-compute the values $u(W - W')$, where $W$ and $W'$ range over the set of discrete cake
amounts. Be sure to avoid replicating code by factoring it out.
As in Problem \ref{prob:cake_prob}, take $u(c_t) = \sqrt{c_t}$.
\begin{enumerate}
\item As in Problem \ref{prob:cake_prob}, approximate the continuum of possible
cake sizes by a column vector called $W$ that ranges from 0 to $W_{max}$ in $N$ steps.

\item \label{item:step2} Initialize the value function, V to a vector of zeros of length $N$.  This is $V_0$.  Perform one iteration
of the contraction operation given in equation \eqref{EqContraction} to get a new value function $V_1$ (this should be very similar
to Problem 1).  Determine the resulting policy function $W' = \psi_1\left(W\right)$.  [HINT: The policy function should be a vector
of length $N$ of optimal future values of the cake $W'$ given the current value of the cake $W$, and $V_T$ should be an $N$-length
vector representing the value of entering a period with cake size $W$.]

\item \label{item:step3} Measure the distance between the two value functions as the sum of the
squared differences,
\begin{equation}
\label{EqDist}
\delta_1\equiv \norm{V_1\left(W\right) - V_0\left(W'\right)}_2^2 = \left(V_1 - V_0\right)^T\left(V_1 - V_0\right).
\end{equation}
Defined in this way, $\delta_1\in [0,\infty)$.

%\item \label{item:step4} Take the resulting $V_1$ from \ref{item:step2}, and perform the same contraction on it to generate $V_2$
%and $\psi_2$. That is, generate,
%\begin{equation*}
%  V_2\left(W\right) = C\Bigl(V_1\left(W\right)\Bigr) = \max_{W'\in[0,W]}\: u\left(W - W'\right) + \beta V_1\left(W'\right)
%\end{equation*}
%and the accompanying policy function $W'=\psi_2\left(W\right)$. Calculate the accompanying distance measure for $\delta_2$ using
% the formula from \eqref{EqDist} with the updated period subscripts. Compare $\delta_2$ with $\delta_1$ from \ref{item:step3}.
%
%\item \label{item:step5} Repeat \ref{item:step4} and generate $V_3$ and $\psi_2$ by performing the contraction on $V_2$. Compare
%$\delta_3$ to $\delta_2$ and $\delta_1$.

\item Write a loop that performs the contraction operation from steps \ref{item:step2} and \ref{item:step3} iteratively
until the distance measure is very small ($\delta_k < 10^{-9}$).  The distance measure $\delta_k$ being arbitrarily close to zero means
 you have converged to the fixed point $V_k = V_{k+1} = V$. (For fun, you can show that the policy function converges to the same
 function regardless of what you put in for your initial policy function value.)

\item If \li{plot = True}, plot the converged policy function vector
($y$-axis) as a function of the cake amounts ($x$-axis).

\item Return the value function and policy function arrays.

Compute the value function and policy function for the infinite time horizon problem with
cake size $1$, discount factor $\beta = .9$, and number of
discrete cake values $N = 100$. The plot you generate should agree with Figure \ref{fig:infinitePolicy}.
\end{enumerate}
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{infiniteHorizon.pdf}
\caption{Policy function for infinite time horizon.}
\label{fig:infinitePolicy}
\end{figure}

\begin{figure}
\includegraphics[width=\textwidth]{convergence.pdf}
\caption{Due to the contraction mapping principle, $\delta_k$ decreases as we perform iterations
until it is small enough to meet our convergence tolerance.}
\end{figure}

\section*{Infinite Horizon, Stochastic, i.i.d.}\label{SecRecProbInfinHorStochiid}

In practice, dynamic programming problems often involve some level of uncertainty.
For example as time progresses prices may fluctuate, resources may vary, or preferences themselves may change.
 In this lab, we reexamine the cake eating problem, this time allowing for uncertainty.

We consider again the problem of optimizing a sequence of decisions over an infinite time horizon.
We assume that the individual's preferences deviate each period according to some ``shock" $\ve$,
where $\ve$ is a random variable.  We assume that the shock terms $\ve$ for each time period are
independent and identically
distributed (i.i.d.).  In effect, this means the probabilities associated with the $\ve$ are the
same for any time $t$ and do not depend on each other.  We assume for now that the $\ve$ are
distributed normally with mean $\mu$ and variance $\sigma^2$.  The Bellman equation can be easily
rewritten in the following way to incorporate the uncertainty,
\begin{equation}\label{stoch_Bellman}
   V\left(W,\ve\right) = \max_{W'\in[0,W]}\: \{\ve u\left(W - W'\right) +
   \beta E_{\ve'}\left[V\left(W',\ve'\right)\right]\},
\end{equation}
where $\ve \sim N(\mu,\sigma^2)$ and
$E$ is the unconditional expectation operator over $\ve$.  Note that now the value function depends
on two variables.  It represents the value of entering the period with $W$, the amount of cake,
and a preference shock of $\ve$.  For example, in a period where the realization of $\ve$ is higher,
we will get more value from the cake eaten in the current period.  Because we do not know the value of
the shock in the next period $\ve'$, we consider only the expected value for future time.

It turns out, we can solve this problem in a manner similar to the infinite horizon deterministic cake-eating
problem considered in the Value Function Iteration lab.  It is worth noting that in this case,
 the value and policy functions will be two dimensional as they will depend on both $W$ and $\ve$.

In order to deal with $\ve$ computationally, we would like to represent it as a vector of possible values
it could take along with the corresponding probabilities that it takes each of those values.  However,
$N(\mu,\sigma^2)$ is a continuous distribution, so we cannot represent every value $\ve$ could take.
We need a discrete distribution that approximates $N(\mu,\sigma^2)$.

To do this, we choose $N$ equally spaced points centered about the mean, at which to approximate the
distribution. Call these values $\ve_1,\ldots,\ve_N$, and let the spacing between adjacent points be
given by $\delta$.
We can then break up the support of the distribution into $N$ bins, where adjacent bins share a common
endpoint. Call the endpoints of these bins $v_1,\ldots,v_{N+1}$. By choosing the endpoints to be
halfway between each $\ve_k$, we have the formula
\[
v_k = \ve_k - \frac{1}{2}\delta, \qquad k=1,\ldots,N
\]
and
\[
v_{N+1} = \ve_N + \frac{1}{2}\delta.
\]

\begin{figure}[h!]
\label{stoch1_fig1}
\begin{center}
\includegraphics[width = \textwidth]{discnorm.pdf}
\end{center}
\caption{Discretization of $N(\mu,\sigma^2)$.  We approximate $P(\ve = \ve_k)$ by the area of the shaded region.}
\end{figure}

We can then associate $\ve_k$ with the area under the curve from $v_k$ to $v_{k+1}$.
In Python, we can find the area using the function \li{norm.cdf} found in the \li{stats} package.
The cdf (cumulative distribution function) gives the area under the curve from $-\infty$ to a specified value.
For example, in the following code, \li{eps} is the area under the curve from 0 to 1.

\begin{lstlisting}
>>> from scipy import stats as st
>>> mu = 0
>>> sigma = 1
>>> eps = st.norm.cdf(1,loc=mu,scale=sigma) - st.norm.cdf(0,loc=mu,scale=sigma)
\end{lstlisting}

In general, it is sufficient to take our points $\ve_k$ ranging from $\mu - 3\sigma$ to $\mu + 3\sigma$, as this
range contains about 99.7\% of the probability mass.

\begin{problem}
Write a function called \li{discretenorm} that accepts an integer $K$ representing the number of discrete points
desired, a mean $\mu$, and a standard deviation $\sigma$. It should return a length-$K$ vector of equally-spaced
values ranging from $\mu - 3\sigma$ to $\mu + 3\sigma$ inclusive,
and a length-$K$ vector containing the associated probabilities.
Plot the approximation of $N(0,1)$ using different values of $K$ to check that your results are plausible.
\end{problem}

Now that we have a discrete distribution for $\ve$, we can solve for the value and policy functions
determined by \eqref{stoch_Bellman}.

\begin{problem}
Complete the following steps to solve the problem described above.
Assume that the period utility function is $u(c)=\sqrt{c}$.
Write a function \li{stochEatCake}
that accepts parameters $\beta$ (discount factor), $N$ (number of discrete cake values),
a tuple of values \li{e_params}, $W_{max}$ (the original size of the cake, set to default value 1),
a keyword argument \li{iid} (set to default value \li{True}), and
a keyword argument \li{plot} (set to default value of \li{False}). Inside the function, carry out the steps
outlined below.

The argument \li{e_params} is a tuple consisting of the values needed to generate
the discrete approximation to $\ve$. In the present case, this tuple consists (in order) of
$K$ (the number of discrete approximations of $\ve$), $\mu$ (the
mean of the shock term $\ve$), and $\sigma$ (the standard deviation of the shock term $\ve$),
since these are the arguments we need to pass to our \li{discretenorm} function.

\begin{enumerate}
\item First, compute an approximation of $\ve$ using the \li{discretenorm} function created in Problem 1.
Use $K$ equally spaced points to approximate $N(\mu,\sigma^2)$. Denote the resulting $K$-length
vector of equally-spaced values by
\[e =(e_1,\ldots,e_K),
\]
and denote the $K$-length vector of the associated probabilities
by
\[\Gamma = (\Gamma_1,\ldots,\Gamma_K).
\]
Note that $\Gamma_k$ give the probability $P(\ve = e_k)$.

Since the values needed for the \li{discretenorm} function are contained in the \li{e_params} input,
we can feed these values directly into the function in the following way:
\begin{lstlisting}
>>> e, gamma = discretenorm(*e_params)
\end{lstlisting}
The \li{*} operator essentially unpacks the values of a tuple or list.

\item As in the Value Function Iteration lab, create a vector
\[w = (w_1,\ldots,w_N)
\]
of possible cake sizes. This should be
a length-$N$ vector of equally spaced values from 0 to $W_{max}$, inclusive.

\item Represent the value function as a $N \times K$ matrix $v$, satisfying
\[
v_{i,j} = V(w_i, e_j).
\]
(The rows correspond to different values of $W$ and the columns correspond to different values of $\ve$.)
Initialize each entry of the matrix to 0.

Likewise, represent the policy function as a $N \times K$ matrix $p$, satisfying
\[
p_{i,j} = \psi(w_i,e_j).
\]
Initialize all entries to 0.

\item In order to evaluate the value function equation, we need to pre-compute $\ve u(W-W')$ for all values of
$\ve,W,W'$.
Begin by computing all possible values of $u(W-W')$, and storing these values in a $N \times N$ array,
as in the Value Function Iteration lab. Call this array $u$. Make sure that the upper triangular
entries of this array are equal to zero, as these entries correspond to consuming more cake than is
available, which is impossible.

The values $\ve u(W-W')$ will be represented by a three-dimensional array $\hat{u}$ of size
$N\times N\times K$, satisfying
\[
\hat{u}_{i,j,k} = v_{i,j}e_k.
\]
We can compute this array easily as follows:

\begin{lstlisting}
>>> import numpy as np
>>> u_hat = np.repeat(u, K).reshape((N,N,K))*e
\end{lstlisting}

\item We also need to compute $E_{\ve'}\Bigl[V\left(W',\ve'\right)\Bigr]$ for each value of $W'$.
The expected value is simply
\begin{equation*}
E_{\ve'}\Bigl[V\left(W',\ve'\right)\Bigr] = \sum_{k=1}^K \Gamma_kV(W',e_k').
\end{equation*}
The result is length $N$ vector, call it $E$, satisfying
\[
E_i = E_{\ve'}\Bigl[V\left(w_i,\ve'\right)\Bigr] = \sum_{k=1}^K \Gamma_kv_{i,k}
\]
This calculation can be done by multiplying $\Gamma$ element-wise to each row of the
value function matrix $v$, and then summing along the rows. Something like the following
line of code should do the trick:
\begin{lstlisting}
>>> E = (v*gamma).sum(axis=1)
\end{lstlisting}

\item We can now compute the value function contraction
\begin{equation*}\label{EqContractStochiid}
C\Bigl(V\left(W,\ve\right)\Bigr) \equiv \max_{W'\in[0,W]}\:
\Bigl\{\ve u\left(W-W'\right) + \beta E_{\ve'}\Bigl[V\left(W',\ve'\right)\Bigr]\Bigr\}.
\end{equation*}
The first task is to create a $N \times N \times K$ array $c$ satisfying
\[
c_{i,j,k} = \hat{u}_{i,j,k} + \beta E_j.
\]
This can be done in any manner of ways. Below is a one-liner that does the job.
\begin{lstlisting}
>>> c = np.swapaxes(np.swapaxes(u_hat, 1, 2) + beta*E, 1, 2)
\end{lstlisting}

Now, for any $k$ and $i < j$, set $c_{i,j,k}$ to a large negative number, say $-10^{10}$,
so that when maximizing over this array, we do not choose to consume more cake than is available.
Again, this can be done in a variety of different ways, but the following does the job concisely:
\begin{lstlisting}
>>> c[np.triu_indices(N, k=1)] = -1e10
\end{lstlisting}

Finally, maximize over the second axis of $c$ (which corresponds to different values of $W'$)
to obtain the updated value function matrix:
\begin{lstlisting}
>>> v_new = np.max(c, axis=1)
\end{lstlisting}
You can likewise update your policy function matrix as follows:
\begin{lstlisting}
>>> max_indices = np.argmax(c, axis=1)
>>> p = w[max_indices]
\end{lstlisting}

\item We now have our updated value function matrix $v_{new}$ as well as the
previous $v$, which we refer to here as $v_{old}$. As we iterate on the value function equation, we need a norm
\begin{equation*}
\delta = \|v_{new} - v_{old}\|_2
\end{equation*}
that measures the distance between these two value functions to determine convergence.
You may compute the norm using the SciPy function \li{scipy.linalg.norm}, or by direct calculation.
At the end of each iteration, make sure to set $v$ to $v_{new}$, so that the updates carry through the
loop.
Iterate on the contraction until $\delta < 10^{-9}$.

\item If \li{plot = True}, make a 3-D surface plot of the policy function for the converged problem
$W' = \psi\left(W,\ve\right)$ which gives the value of the cake tomorrow as a
function of the cake today  and the taste shock today.  Do the same for the value function.
Example code to create the value function plot is provided below.
\begin{lstlisting}
>>> x = np.arange(0,N)
>>> y = np.arange(0,K)
>>> X,Y = np.meshgrid(x,y)
>>> fig1 = plt.figure()
>>> ax1 = Axes3D(fig1)
>>> ax1.plot_surface(w[X], Y, v.T, cmap=cm.coolwarm)
>>> plt.show ()
\end{lstlisting}
Creating the policy function plot is similar.

\item Return the converged value function matrix $v$ and policy function matrix $p$.

\end{enumerate}
Test your function using values $\beta = .9$, $N = 100$, $K = 7$, $\sigma = .5$, $\mu = 4\sigma$,
and \li{plot = True}.
The proper way to set this up and call the function is as follows:
\begin{lstlisting}
>>> e_params = (7, 4*.5, .5)
>>> stuff = stochEatCake(.9, 100, e_params, plot=True)
\end{lstlisting}
\end{problem}

\begin{figure}
    \centering
    \includegraphics[width = \textwidth]{stoch_value.pdf}
    \caption{3D surface representing the value function for the Stochastic Cake-Eating problem.}
\end{figure}

\section*{Infinite Horizon, Stochastic, AR(1)}\label{SecRecProbInfinHorStochAR1}

In the previous example, we assumed that the shocks at time $t$ were independent of what happened in
previous periods.  Often a shock may depend on recent events.  We will assume now that the shocks are persistent,
meaning preferences in the current period are more likely to be close to what they were in the previous period.
We can characterize the persistence by what is called an autoregressive process of order one, denoted AR(1).
Such a process is defined as follows.
\begin{equation}\label{EqAR1shock}
\ve' = (1-\rho)\mu + \rho\ve + \nu' \quad\text{where}\quad \rho\in(0,1) \quad\text{and}\quad \nu\sim N(0,\sigma^2).
\end{equation}

Essentially, instead of allowing the shocks to have a mean which is independent of the past, the mean is now a
weighted average (weighted by $\rho$) of some $\mu$ and the previous realization of the shock, $\ve$.  As it turns
 out, we can approximate this process by thinking of it as a Markov Chain. This means we need to determine a
 discrete set of points representing possible values of $\ve$ and a Markov transition matrix that gives the
 probabilities of moving from one value of $\ve$ to another.  There are methods for determining the discrete
 approximation of $\ve$ with a Markov transition matrix.  These methods are beyond the scope of this section,
 but you can use the file \li{tauchenhussey.py} to implement them in the next problem.

The Bellman equation becomes the following, in which the only change from the i.i.d. shock case is that the
expectations operator is now conditional on the current shock $\ve$:

\begin{equation*}
   V\left(W,\ve\right) = \max_{W'\in[0,W]}\: \{\ve u\left(W - W'\right) +
   \beta E_{\ve'|\ve}\left[V\left(W',\ve'\right)\right]\},
\end{equation*}
where $\ve'$ is distributed according to \eqref{EqAR1shock}.
Let $\Gamma_{i,j}=P\left(\ve_j'|\ve_i\right)$ where $\ve_j'$ is the value of the shock in
the next period and $\ve_i$ is the value of the shock in the current period.
In other words, $\Gamma$ is the Markov transition matrix.

The solution to this problem is of the same type as that in the i.i.d. case, since the only difference is
the probability distributions of the $\ve$.

\begin{problem}
Expand your \li{stochEatCake} function to handle the case of AR(1) shock terms. The function should
handle this case for the parameter value \li{iid = False}, and should handle the previous case of
normally distributed i.i.d. shock terms for the parameter value \li{iid = True}. You will need to
add a few ``if ... else" statements, as well as implement the steps outlined below, but most of the
code will remain unchanged.

\begin{enumerate}
\item In the AR(1) case, the \li{e_params} argument should be a tuple of values needed to
generate the arrays $e$ and $\Gamma$ that approximate the values and distribution of $\ve$
as a Markov chain.
Use the file \li{tauchenhussey.py} to calculate these arrays.
The provided Python function \li{tauchenhussey} produces the vector $e$ of length $M$
and an $M\times M$ transition matrix $\Gamma$.
Thus, you simply need the following lines of code, similar to the previous case.
\begin{lstlisting}
>>> from tauchenhussey import tauchenhussey
>>> e, gamma = tauchenhussey(*e_params)
\end{lstlisting}

\item Because our values for $e$ and $\Gamma$ are different in the AR(1) case than
in the i.i.d. case, we must compute the expectation in a different manner.
In particular, we need to compute the conditional expectation
\begin{equation*}
E_{\ve'|\ve}\Bigl[V\left(W',\ve'\right)\Bigr].
\end{equation*}
We obtain a two-dimensional array, since the expectation depends on both $W'$ and on $\ve$.
The expectation can be computed by the matrix multiplication $v\Gamma^T$.
Your code should match the following.
\begin{lstlisting}
>>> E = v.dot(gamma.T)
\end{lstlisting}

\item The last difference comes in computing the array $c$. Fortunately, it is easier in this case.
Recall that $c$ gives the values for
\[
\ve u\left(W-W'\right) + \beta E_{\ve'|\ve}\Bigl[V\left(W',\ve'\right)\Bigr].
\]
The array $\hat{u}$ contains the values for the first term in the expression, and the array $E$
contains the values for the expectation term.
Hence, we obtain $c$ by simple addition. Array broadcasting makes this work without problems.
\begin{lstlisting}
>>> c = u_hat + beta*E
\end{lstlisting}
You will still need to set the upper triangular entries of $c$ to a large negative number, just as in the
previous case.
\end{enumerate}

Those are the only differences. Let the following code snippet be a guideline for how to implement
these differences.
\begin{lstlisting}
>>> if iid:
>>>     # compute E as outlined in problem 2
>>> else:
>>>     # compute E as outlined in the current problem
\end{lstlisting}

Now test your function with $\beta = .9$, $N = 100$, \li{iid = False}, and \li{plot = True}.
As inputs to \li{tauchenhussey}, let $K=7$, the mean of the process
$\mu=4\sigma$, $\rho = 1/2$, $\sigma=1/2$, and
\[baseSigma=(0.5+\frac{\rho}{4})\sigma +
(0.5 - \frac{\rho}{4})\frac{\sigma}{\sqrt{1-\rho^2}}.
\]
Your \li{e_params} parameter will therefore be a tuple of values containing (in order)
$K$, $\mu$, $\rho$, $\sigma$, and $baseSigma$.
\end{problem}


\chapter{Dynamic Programming}
\section{Summary}
Dynamic programming is a \textbf{collection of algorithms} that can be used to find the \textbf{optimal policy}. It assumes a perfect model of the system (MDP) and uses a lot of computational power.

\begin{equation}
v_*(s) = \max_a \sum_{s',r} p(s', r | s, a)[r + \gamma v_*(s')]
\label{eq:bellman optimality equation state-value function}
\end{equation}

\begin{equation}
q_*(s, a) = \sum_{s', r} p(s', r | s, a) [r + \gamma \max_{a'} q_*(s', a')]
\label{eq:bellman optimality equation action-value function}
\end{equation}

\subsection{Policy evaluation}
The bellman equation from \ref{eq:bellman equation value function derivation} can be converted into an iterative method called \textbf{iterative policy evaluation} to find the value function. It takes the expected value over all the same next states. All updates in dynamic programming are called \textbf{expected updates}, because they are based on expectation over all possible next states rather then the sample next states.

\begin{equation}
\begin{split}
v(s)_{k+1}
& = \EX_\pi\left[ R_{t+!} + \gamma v_k(S_{k+1}) | S_t = s \right] \\
& = \sum_a \pi(a | s) \sum_{s', r} p(s', r | s, a) \left[r + \gamma v(s')\right] \\
\end{split}
\label{eq:iterative policy evaluation update rule}
\end{equation}

\subsection{Policy improvement}
\begin{equation}
\begin{split}
q_\pi(a, s) & = \EX\left[R_{t+1} + \gamma v_\pi(S_{t+1}) | S_t = s, A_t = a \right]\\
& = \sum_{s', r} p(s', r | s, a)\big[r + \gamma v_\pi(s')\big]
\end{split}
\label{eq:policy improvement, select the next action}
\end{equation}

Given a policy $\pi$ and value function $v_\pi(s)$, one action $a$ can be selected that maximizes equation~\ref{eq:policy improvement, select the next action} and all sequential actions follow the policy $\pi$. The \textbf{policy improvement theorem}  say's that if a new policy $\pi'$ satisfies equation~\ref{eq:policy improvement theorem condition}, the the new policy will satisfy equation~\ref{eq:policy improvement theorem result}. And be as good or better then the original policy.(proof on page 78-79 of the book)

\begin{equation}
q_\pi(s, \pi'(s)) \geq v_\pi(s)
\label{eq:policy improvement theorem condition}
\end{equation}

\begin{equation}
v_{\pi}(s) \leq v_{\pi'}(s)
\label{eq:policy improvement theorem result}
\end{equation}

The new improved policy $\pi'$ is formally written down in equation~\ref{eq:greedy policy action-value}. The corresponding value function is formally written down in equation~\ref{eq:greedy policy value function to bellman equation}. Where we \textbf{end up with the bellman optimality equation}. Indicating that the policy can improve until it's the optimal policy.

\begin{equation}
\begin{split}
\pi'(s) & = \argmax_a q_\pi (s, a) \\
& = \argmax_a \EX\left[ R_{t+1} + \gamma v_\pi(S_{t+1} | S_t = s, A_t = a) \right] \\
& = \argmax_a \sum_{s',r} p(s', r|s, a)\left[r + \gamma v_{\pi}(s')\right]
\end{split}
\label{eq:greedy policy action-value}
\end{equation}

\begin{equation}
\begin{split}
v_{\pi'}(s)
& = \max_a \EX\left[ R_{t+1} + \gamma v_\pi'(S_{t+1}) | S_t = s, A_t=a \right] \\
& = \max_a \sum_{s', r}p(s', r|s, a)\left[r + \gamma v_{\pi'}(s')\right]
\end{split}
\label{eq:greedy policy value function to bellman equation}
\end{equation}

\subsection{Policy iteration}

The iterative process of evaluating a policy, and then creating a new policy that is greedy towards the old one is called \textbf{policy iteration}.

\subsection{Value iteration}

Instead of evaluating the complete policy first, and then improving the policy. The policy can be improved after every state evaluation. Effective \textbf{turning the bellman optimality equation into the iterative update} of equation~\ref{eq:value iteration}.

\begin{equation}
v_{k+1} = \max_a = \sum_{s',r} p(s', r| s, a)\big[ r + \gamma v_k(s')\big]
\label{eq:value iteration}
\end{equation}

\subsection{Generalized policy iteration}
The iterative process of repeatedly evaluating a policy and using it to create an improved version of that policy, is referred to as \textbf{generalized policy iteration} or short GPI. Both policy iteration and value iteration are GPI, as do many stochastic methods.

\section{Exercises}

\subsection{Exercise 4.8}
The reward is only obtained when the capital is above 99. When the capital is at 50, there is a 50\% chance you can win the game. So this obviously is the optimal policy. When you reach 51: it would be rather odd to bet the entire capital, as you don't need to risk it all to reach 100. Bigger downside, but same upside. So the best course of action is to bet with 1, see if you can grow this above 50. If you lose it, you still have a 50\% chance to win by betting it all.


\newpage
\subsection{A new training algorithm}
%\begin{figure}[H]
%	\centering
%	\includegraphics[width=3in]{part_logGDfullL.png}
%	\caption{part of log(Train Loss) for full dataset}
%\end{figure}

If we print the first several steps, we will notice that the loss of Net0 descends the fastest among those networks. After 5-10 steps, it suddenly become the slowest. If at this point we can switch to another curve, then the training process might speed up.\\

\subsection{Regression training}
Actually, there is two ways to achieve this. We may use a very simple regression example to discribe those two methods. We denote our training data as $X\in \mathbb{R}^{d_x\times N}$, the label is $Y\in \mathbb{R}^{d_y\times N}$. Basically, our model is the simplest linear model
\begin{equation}
	f(x) = Wx,
\end{equation}
and our loss function is
\begin{equation}
	L(W, D) = \|WX - Y\|_F^2.
\end{equation}
First, we use
\begin{equation}
	W = W - lr \frac{\partial L}{\partial W}
\end{equation}
to do several iterations to obtain $\overline{W}$. Then we have three ways to continue.\\
\begin{itemize}
	\item Reformulate the problem as
	\begin{equation}
	\min_{U,V} \|(\overline{W}+ UV)X - Y\|_F^2
	\end{equation}
	where $U\in \mathbb{R}^{d_y\times d_1}$, $V\in\times \mathbb{R}^{d_1\times d_x}$. If the solution of above problem is $\overline{U}, \overline{V}$, then our final solution is $W = \overline{W}+\overline{U}\overline{V}$. Notice that if we use a zero initialization in the above problem, the begining loss is $\|\overline{W} X - Y\|_F^2$.\\
	\item Factorize $\overline{W} = U_0 V_0$, where $\overline{U}_0\in \mathbb{R}^{d_y\times d_1}$, $\overline{V}_0\in\times \mathbb{R}^{d_1\times d_x}$. And we use $(U_0,V_0)$ as the initial value for the following problem:
	\begin{equation}
	\min_{U,V} \|UVX - Y\|_F^2.
	\end{equation}
	If the solution of above problem is $\overline{U}, \overline{V}$, then our final solution is $W = {\overline{U}}{\overline{V}}$.
\end{itemize}

\subsection{Classification Training}
For classification problem, we have three possible ways to speed up the training algorithms. Define function $l: \mathbb{R}^{d_x}\times \{1,2,\cdots,d_y\} \rightarrow \mathbb{R}$ as
\begin{equation}
	l(\bm s, i) = \log ({\bm 1}^T e^{\bm s}) - s_i,
\end{equation}
where $\bm s = (s_1,s_2,\cdots,s_{d_y})$. So our final cross-entropy loss can be written as
\begin{equation}
	L(\bm\theta,D) = \sum_{i = 1}^{d_y} \sum_{x\in D_i} l(f(x), i)
\end{equation}
where $f$ is the mapping of our model.
We use single-layer and 2-layer linear NN as our examples to state our possible ways to speed up training. For single-layer linear NN we use $W,b$ to represents the weight matrix and bias vector, so $f(x) = Wx + b$. For 2-layer linear NN we use $W_1,W_2,b$ to represents the 2 weight matrices and the bias vector , and $f(x) = W_2 W_1 x + b$ (which implies the first layer is without bias).\\

There are 3 main ideas of speed-up training algorithms.
\begin{itemize}
	\item  Use single-layer linear NN and cross-entropy loss to do several iterations to get parameter $\overline{W}, \overline{b}$. Then we modify the optimization problem as
	\begin{equation}
		\min_{W_1,W_2,b}\sum_{i = 1}^{d_y} \sum_{x\in D_i} l((\overline{W}+W_2 W_1)x + (\overline{b}+b), i)
	\end{equation}
	If $\tilde{W}_1,\tilde{W}_2,\tilde{b}$ is one of the solution of the above problem, then our final solution is $W^* = \overline{W} + \tilde{W}_2\tilde{W}_1, b^* = \overline{b} + \tilde{b}$.\\

	\item In the second training algorithm, we use a 2-layer linear NN all the time. The loss is
	\begin{equation}
	\sum_{i = 1}^{d_y} \sum_{x\in D_i} l(W_2 W_1x + b, i)
	\end{equation}
	 But in first several steps, we fix the initial $W_1$, only update $W_2$ and $b$. If you regard $W_1 x$ as a whole, we can see we are actually training a single-layer linearNN at the begining. After several steps, we start to update $W_1,W_2,b$ together just as the regular GD.\\

	 \item The third idea is, just like the first idea, we use single-layer linear NN and cross-entropy loss to do several iterations to get parameter $\overline{W}, \overline{b}$. But after that, we factorize $\overline{W} = U_0 V_0$, where $\overline{U}_0\in \mathbb{R}^{d_y\times d_1}$, $\overline{V}_0\in\times \mathbb{R}^{d_1\times d_x}$. And we use $W_2 = U_0, W_1 = V_0, b = \overline{b}$ as the initial value for the following problem:
	 \begin{equation}
	 \min_{W_1,W_2,b} \sum_{i = 1}^{d_y} \sum_{x\in D_i} l(W_2 W_1x + b, i) .
	 \end{equation}

\end{itemize}

\subsubsection{Idea I}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train1_hidden100.png}
	\caption{New training algorithm 1: $d_1 = 100, lr = 0.1$}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train1_hidden500.png}
	\caption{New training algorithm 1: $d_1 = 500, lr = 0.1$}
\end{figure}
Here we found that when we switch to the 2-layer model, the descent speed becomes slower.

\subsubsection{Idea II}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train2_hidden10.png}
	\caption{New training algorithm 2: $d_1 = 10, lr = 0.1$}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train2_hidden100.png}
	\caption{New training algorithm 2: $d_1 = 100, lr = 0.1$}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train2_hidden500.png}
	\caption{New training algorithm 2: $d_1 = 500, lr = 0.1$}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train2_hidden1000.png}
	\caption{New training algorithm 2: $d_1 = 1000, lr = 0.1$}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train2_hidden2000_lr1.png}
	\caption{New training algorithm 2: $d_1 = 2000, lr = 0.1$}
\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=2.5in]{train2_hidden2000_lr05.png}
	\caption{New training algorithm 2: $d_1 = 2000, lr = 0.05$}
\end{figure}

Here are some interesting things we can observe:
\begin{itemize}
	\item It seems that idea II do have an acceleration at 20th iteration (bacause we start to update all parameters that step), but the descending speed is slower than Net0 in the first 20 steps when the hidden size is less than 500.
	\item When we increase the hidden size to 1000 or 2000, our idea do descend faster than Net0 at the beginning, but slower than Net1. So our idea II seems useless when size of hidden-layer is very large.

\end{itemize}


\chapter{Duals, adjoint, and transposes}
This chapter is dedicated to the basis-free interpretation
of the transpose and conjugate transpose of a matrix.

Poster corollary: we will see that
symmetric matrices with real coefficients
are diagonalizable and have real eigenvalues.

\section{Dual of a map}
\prototype{The example below.}
We go ahead and now define a notion
that will grow up to be the transpose of a matrix.

\begin{definition}
	Let $V$ and $W$ be vector spaces.
	Suppose $T \colon V \to W$ is a linear map.
	Then we actually get a map
	\begin{align*}
		T^\vee \colon W^\vee &\to V^\vee \\
		f &\mapsto f \circ T.
	\end{align*}
	This map is called the \vocab{dual map}.
\end{definition}

\begin{example}
	[Example of a dual map]
	Work over $\RR$.
	Let's consider $V$ with basis $e_1$, $e_2$, $e_3$
	and $W$ with basis $f_1$, $f_2$.
	Suppose that
	\begin{align*}
		T(e_1) &= f_1 + 2f_2 \\
		T(e_2) &= 3f_1 + 4f_2 \\
		T(e_3) &= 5f_1 + 6f_2.
	\end{align*}
	Now consider $V^\vee$ with its dual basis $e_1^\vee$,
	$e_2^\vee$, $e_3^\vee$
	and $W^\vee$ with its dual basis $f_1^\vee$, $f_2^\vee$.
	Let's compute $T^\vee(f_1^\vee) = f_1^\vee \circ T$:
	it is given by
	\begin{align*}
		f_1^\vee \left( T(ae_1 + be_2 + ce_3) \right)
		&= f_1^\vee\left( (a+3b+5c) f_1 + (2a+4b+6c) f_2 \right) \\
		&= a + 3b + 5c.
	\end{align*}
	So accordingly we can write
	\[ T^\vee(f_1^\vee) = e_1^\vee + 3e_2^\vee + 5e_3^\vee \]
	Similarly,
	\[ T^\vee(f_2^\vee) = 2e_1^\vee + 4e_2^\vee + 6e_3^\vee. \]
	This determines $T^\vee$ completely.
\end{example}
If we write the matrices for $T$ and $T^\vee$ in terms of our basis,
we now see that
\[ T = \begin{bmatrix}
		1 & 3 & 5 \\
		2 & 4 & 6
	\end{bmatrix}
	\quad\text{and}\quad
	T^\vee = \begin{bmatrix}
		1 & 2 \\
		3 & 4 \\
		5 & 6
	\end{bmatrix}.
\]
So in our selected basis,
we find that the matrices are \vocab{transposes}:
mirror images of each other over the diagonal.

Of course, this should work in general.
\begin{theorem}
	[Transpose interpretation of $T^\vee$]
	Let $V$ and $W$ be finite-dimensional $k$-vector spaces.
	Then, for any $T \colon V \to W$,
	the following two matrices are transposes:
	\begin{itemize}
		\ii The matrix for $T \colon V \to W$
		expressed in the basis $(e_i)$, $(f_j)$.
		\ii The matrix for $T^\vee \colon W^\vee \to V^\vee$
		expressed in the basis $(f_j^\vee)$, $(e_i^\vee)$.
	\end{itemize}
\end{theorem}
\begin{proof}
	The $(i,j)$th entry of the matrix $T$
	corresponds to the coefficient of $f_j$ in $T(e_i)$,
	which corresponds to the coefficient of $e_i^\vee$
	in $f_j^\vee \circ T$.
\end{proof}
The nice part of this is that the definition of $T^\vee$ is basis-free.
So it means that if we start with any linear map $T$,
and then pick whichever basis we feel like,
then $T$ and $T^\vee$ will still be transposes.

\section{Identifying with the dual space}
For the rest of this chapter, though,
we'll now bring inner products into the picture.

Earlier I complained that there was no natural isomorphism $V \cong V^\vee$.
But in fact, given an inner form
we can actually make such an identification:
that is we can naturally associate every linear map
$\xi \colon V \to k$ with a vector $v \in V$.

To see how we might do this, suppose $V = \RR^3$
for now with an orthonormal basis $e_1$, $e_2$, $e_3$.
How might we use the inner product to
represent a map from $V \to \RR$?
For example, take $\xi \in V^\vee$ by
$\xi(e_1) = 3$, $\xi(e_2) = 4$ and $\xi(e_3) = 5$.
Actually, I claim that
\[ \xi(v) = \left< v, 3e_1 + 4e_2 + 5e_3 \right> \]
for every $v$.
\begin{ques}
	Check this.
\end{ques}

And this works beautifully in the real case.
\begin{theorem}[$V \cong V^\vee$ for real inner form]
	\label{thm:real_dual_isomorphic}
	Let $V$ be a finite-dimensional \emph{real}
	inner product space and $V^\vee$ its dual.
	Then the map $V \to V^\vee$ by
	\[ v \mapsto \left<  -, v \right> \in V^\vee \]
	is an isomorphism of real vector spaces.
\end{theorem}
\begin{proof}
	It suffices to show that the map is injective and surjective.
	\begin{itemize}
		\ii Injective: suppose $\left< v_1, v \right> = \left< v_2, v \right>$
		for every vector $v \in V$.
		This means $\left< v_1 - v_2, v \right> = 0$ for every vector $v \in V$.
		This can only happen if $v_1 - v_2 = 0$; for example, take $v = v_1 - v_2$
		and use positive definiteness.
		\ii Surjective: take an orthonormal basis $e_1$, \dots $e_n$
		and let $e_1^\vee$, \dots, $e_n^\vee$ be the dual basis on $V^\vee$.
		Then $e_1$ maps to $e_1^\vee$, et cetera.
		\qedhere
	\end{itemize}
\end{proof}
Actually, since we already know $\dim V = \dim V^\vee$
we only had to prove one of the above.
As a matter of personal taste, I find the proof of injectivity more elegant,
and the proof of surjectivity more enlightening,
so I included both.
Thus
\begin{moral}
	If a real inner product space $V$ is given an inner form,
	then $V$ and $V^\vee$ are canonically isomorphic.
\end{moral}

Unfortunately, things go awry if $V$ is complex.
Here is the results:
\begin{theorem}[$V$ versus $V^\vee$ for complex inner forms]
	Let $V$ be a finite-dimensional \emph{complex}
	inner product space and $V^\vee$ its dual.
	Then the map $V \to V^\vee$ by
	\[ v \mapsto \left<  -, v \right> \in V^\vee \]
	is a bijection of sets.
\end{theorem}
Wait, what? Well, the proof above shows that it is both injective
and surjective, but why is it not an isomorphism?
The answer is that it is not a linear map:
since the form is sesquilinear we have for example
\[ iv \mapsto \left< -, iv\right> = -i \left< -, v\right> \]
which has introduced a minus sign!
In fact, it is an \emph{anti-linear} map, in the sense we defined before.

Eager readers might try to fix this by defining
the isomorphism $v \mapsto \left< v, - \right>$ instead.
However, this also fails, because the right-hand side
is not even an element of $V^\vee$:
it is an ``anti-linear'', not linear.

And so we are stuck.
Fortunately, we will only need the ``bijection'' result
for what follows, so we can continue on anyways.
(If you want to fix this, \Cref{prob:complex_conj_space}
gives a way to do so.)

\section{The adjoint (conjugate transpose)}
We will see that, as a result of the flipping above,
the \emph{conjugate transpose} is actually the better concept
for inner product spaces: since it can be defined using only the inner product
with making mention to dual spaces at all.
\begin{definition}
	Let $V$ and $W$ be finite-dimensional inner product spaces,
	and let $T \colon V \to W$.
	The \vocab{adjoint} (or \vocab{conjugate transpose})
	of $T$, denoted $T^\dagger \colon W \to V$,
	is defined as follows: for every vector $w \in W$,
	we let $T^\dagger(w) \in V$ be the unique vector with
	\[ \left< v, T^\dagger(w) \right>_V = \left< T(v), w \right>_W \]
	for every $v \in V$.
\end{definition}

Some immediate remarks about this definition:
\begin{itemize}
\ii Our $T^\dagger$ is well-defined,
because $\left< T(-), w \right>_W$ is some function in $V^\vee$,
and hence by the bijection earlier
it should be uniquely of the form $\left< -, v \right>$ for some $v \in V$.
\ii This map $T^\dagger$ is indeed a linear map (why?).
\ii The niceness of this definition is that it doesn't
make reference to any basis or even $V^\vee$,
so it is the ``right'' definition for a inner product space.
\ii By symmetry, of course, we also have
$\left< T^\dagger(v), w \right> = \left< v, T(w)\right>$.
\end{itemize}

\begin{example}
	[Example of an adjoint map]
	We'll work over $\CC$, so the conjugates are more visible.
	Let's consider $V$ with orthonormal basis $e_1$, $e_2$, $e_3$
	and $W$ with orthonormal basis $f_1$, $f_2$.
	We put
	\begin{align*}
		T(e_1) &= if_1 + 2f_2 \\
		T(e_2) &= 3f_1 + 4f_2 \\
		T(e_3) &= 5f_1 + 6if_2.
	\end{align*}
	We compute $T^\dagger(f_1)$.
	It is the unique vector $x \in V$ such that
	\[ \left< v, x \right>_V = \left< T(v), f_1 \right>_W \]
	for any $v \in V$.
	If we expand $v = ae_1 + be_2 + ce_3$ the above equality becomes
	\begin{align*}
		\left< ae_1 + be_2 + ce_3, x \right>_V
		&= \left< T(ae_1 + be_2 + ce_3), f_1 \right>_W \\
		&= ia + 3b + 5c.
	\end{align*}
	However, since $x$ is in the second argument,
	this means we actually want to take
	\[ T^\dagger(f_1) = -ie_1 + 3e_2 + 5e_3 \]
	so that the sesquilinearity will conjugate the $i$.
\end{example}

The pattern continues, though we remind the reader that
we need the basis to be orthonormal to proceed.
\begin{theorem}
	[Adjoints are conjugate transposes]
	Fix an \emph{orthonormal} basis of a finite-dimensional inner product space $V$.
	Let $T \colon V \to V$ be a linear map.
	If we write $T$ as a matrix in this basis,
	then the matrix $T^\dagger$ (in the same basis)
	is the \emph{conjugate transpose} of the matrix of $T$;
	that is, the $(i,j)$th entry of $T^\vee$
	is the complex conjugate of the $(j,i)$th entry of $T$.
\end{theorem}
\begin{proof}
	One-line version: take $v$ and $w$ to be basis elements,
	and this falls right out.

	Full proof: let
	\[ T = \begin{bmatrix}
			a_{11} & \dots & a_{1n} \\
			\vdots & \ddots & \vdots \\
			a_{n1} & \dots & a_{nn}
		\end{bmatrix} \]
	in this basis $e_1$, \dots, $e_n$.
	Then, letting $w = e_i$ and $v = e_j$ we deduce that
	\[ \left< e_i, T^\dagger(e_j) \right> = \left< T(e_i) , e_j\right> = a_{ji}
		\implies
		\left< T^\dagger(e_j), e_i \right> = \ol{a_{ji}} \]
	for any $i$, which is enough to deduce the result.
\end{proof}

\section{Eigenvalues of normal maps}
We now come to the advertised theorem.
Restrict to the situation where $T \colon V \to V$.
You see, the world would be a very beautiful place if it turned out
that we could pick a basis of eigenvectors that was also \emph{orthonormal}.
This is of course far too much to hope for;
even without the orthonormal condition,
we saw that Jordan form could still have $1$'s off the diagonal.

However, it turns out that there is
a complete characterization of exactly when our overzealous dream is true.

\begin{definition}
	We say a linear map $T$
	(from a finite-dimensional inner product space to itself)
	is \vocab{normal} if $TT^\dagger = T^\dagger T$.

	We say a complex $T$ is \vocab{self-adjoint} or \vocab{Hermitian} if $T = T^\dagger$;
	i.e.\ as a matrix in any orthonormal basis, $T$ is its own conjugate transpose.
	For real $T$ we say ``self-adjoint'', ``Hermitian'' or \vocab{symmetric}.
\end{definition}
\begin{theorem}
	[Normal $\iff$ diagonalizable with orthonormal basis]
	Let $V$ be a finite-dimensional complex inner product space.
	A linear map $T \colon V \to V$ is normal
	if and only if one can pick an orthonormal basis of eigenvectors.
\end{theorem}
\begin{exercise}
	Show that if there exists such an orthonormal basis
	then $T \colon V \to V$ is normal,
	by writing $T$ as a diagonal matrix in that basis.
\end{exercise}
\begin{proof}
	This is long, and maybe should be omitted on a first reading.
	If $T$ has an orthonormal basis of eigenvectors,
	this result is immediate.

	Now assume $T$ is normal.
	We first prove $T$ is diagonalizable; this is the hard part.
	\begin{claim}
		If $T$ is normal, then $\ker T = \ker T^r = \ker T^\dagger$ for $r \ge 1$.
		(Here $T^r$ is $T$ applied $r$ times.)
	\end{claim}
	\begin{subproof}
		[Proof of Claim]
		Let $S = T^\dagger \circ T$, which is self-adjoint.
		We first note that $S$ is Hermitian and $\ker S = \ker T$.
		To see it's Hermitian, note
		$\left<Sv, w \right> = \left<Tv, Tw \right> = \left<v, S w \right>$.
		Taking $v = w$ also implies $\ker S \subseteq \ker T$
		(and hence equality since obviously $\ker T \subseteq \ker S$).

		First, since we have $\left< S^r(v), S^{r-2}(v) \right>
		= \left< S^{r-1}(v), S^{r-1}(v)\right>$,
		an induction shows that $\ker S = \ker S^r$ for $r \ge 1$.
		Now, since $T$ is normal, we have $S^r = (T^\dagger)^r \circ T^r$,
		and thus we have the inclusion
		\[ \ker T \subseteq \ker T^r \subseteq \ker S^r = \ker S = \ker T \]
		where the last equality follows from the first claim.
		Thus in fact $\ker T = \ker T^r$.

		Finally, to show equality with $\ker T^\dagger$ we
		\begin{align*}
			\left< Tv, Tv\right> &= \left< v, T^\dagger T v\right> \\
			&= \left< v, T T^\dagger v \right> \\
			&= \left< T^\dagger v, T^\dagger v \right>. \qedhere
		\end{align*}
	\end{subproof}

	Now consider the given $T$, and any $\lambda$.
	\begin{ques}
		Show that $(T - \lambda\id)^\dagger = T^\dagger - \ol\lambda \id$.
		Thus if $T$ is normal, so is $T - \lambda \id$.
	\end{ques}
	In particular, for any eigenvalue $\lambda$ of $T$,
	we find that $\ker (T-\lambda\id) = \ker(T-\lambda\id)^r$.
	This implies that all the Jordan blocks of $T$ have size $1$;
	i.e.\ that $T$ is in fact diagonalizable.
	Finally, we conclude that the eigenvectors of $T$ and $T^\dagger$ match,
	and the eigenvalues are complex conjugates.

	So, diagonalize $T$.
	We just need to show that if $v$ and $w$ are eigenvectors of $T$
	with distinct eigenvalues, then they are orthogonal.
	(We can use Gram-Schmidt on any eigenvalue that appears multiple times.)
	To do this, suppose $T(v) = \lambda v$ and $T(w) = \mu w$
	(thus $T^\dagger(w) = \ol\mu w $).
	Then
	\[ \lambda \left< v,w\right>
		= \left< \lambda v, w\right>
		= \left< Tv, w\right>
		= \left< v, T^\dagger(w)\right>
		= \left< v, \ol \mu w \right>
		= \mu \left< v, w\right>. \]
	Since $\lambda \neq \mu$, we conclude $\left< v,w \right> = 0$.
\end{proof}
This means that not only can we write
\[ T = \begin{bmatrix}
		\lambda_1 & \dots & \dots & 0 \\
		0 & \lambda_2 & \dots & 0 \\
		\vdots & \vdots & \ddots & \vdots \\
		0 & 0 & \dots & \lambda_n
	\end{bmatrix}
\]
but moreover that the basis associated with this matrix happens to be orthonormal vectors.

As a corollary:
\begin{theorem}[Hermitian matrices have real eigenvalues]
	A Hermitian matrix $T$ is diagonalizable,
	and all its eigenvalues are real.
\end{theorem}
\begin{proof}
	Obviously Hermitian $\implies$ normal,
	so write it in the orthonormal basis of eigenvectors.
	To see that the eigenvalues are real, note that $T = T^\dagger$
	means $\lambda_i = \ol{\lambda_i}$ for every $i$.
\end{proof}

\section{\problemhead}
\begin{sproblem}
	[Double dual]
	\gim
	\label{prob:double_dual}
	Let $V$ be a finite-dimensional vector space.
	Prove that
	\begin{align*}
		V &\to (V^\vee)^\vee \\
		v &\mapsto \left( \xi \mapsto \xi(v) \right)
	\end{align*}
	gives an isomorphism.
	(This is significant because the isomorphism is \emph{canonical},
	and in particular does not depend on the choice of basis.
	So this is more impressive.)
	\begin{hint}
		You can \emph{prove} the result just by taking a basis
		$e_1$, \dots, $e_n$ of $V$
		and showing that it is a linear map sending $e_1$
		to the basis $(e_1^\vee)^\vee$.
	\end{hint}
\end{sproblem}

\begin{problem}
	[Fundamental theorem of linear algebra]
	Let $T \colon V \to W$ be a map of finite-dimensional $k$-vector spaces.
	Prove that
	\[ \dim \img T = \dim \img T^\vee
		= \dim V - \dim \ker T = \dim W - \dim \ker T^\vee. \]
	\begin{hint}
		Use \Cref{thm:linear_map_basis} and it will be immediate
		(the four quantities equal the $k$ in the theorem).
	\end{hint}
	\begin{sol}
		By \Cref{thm:linear_map_basis},
		we may select $e_1$, \dots, $e_n$ a basis of $V$
		and $f_1$, \dots, $f_m$ a basis of $W$
		such that $T(e_i) = f_i$ for $i \le k$
		and $T(e_i) = 0$ for $i > k$.
		Then $T^\vee(f_i^\vee) = e_i^\vee$ for $i \le k$
		and $T^\vee(f_i^\vee) = 0$ for $i > k$.
		All four quantities are above are then equal to $k$.
	\end{sol}
\end{problem}

\begin{dproblem}
	[Row rank is column rank]
	A $m \times n$ matrix $M$ of real numbers is given.
	The \emph{column rank} of $M$ is the dimension of the span in $\RR^m$
	of its $n$ column vectors.
	The \emph{row rank} of $M$ is the dimension of the span in $\RR^n$
	of its $m$ row vectors.
	Prove that the row rank and column rank are equal.
	\begin{hint}
		This actually is just the previous problem in disguise!
		The row rank is $\dim \img T^\vee$
		and the column rank is $\dim \ker T$.
	\end{hint}
\end{dproblem}

\begin{problem}
	[The complex conjugate spaces]
	\label{prob:complex_conj_space}
	Let $V = (V, +, \cdot)$ be a complex vector space.
	Define the \vocab{complex conjugate vector space},
	denoted $\ol V = (V, +, \ast)$
	by changing just the multiplication:
	\[ c \ast v = \ol c \cdot V. \]
	Show that for any sesquilinear form on $V$,
	if $V$ is finite-dimensional, then
	\begin{align*}
		\ol V & \to V^\vee \\
		v &\mapsto \left< -, v \right>
	\end{align*}
	is an isomorphism of complex vector spaces.
\end{problem}

\begin{problem}
	[$T^\dagger$ vs $T^\vee$]
	Let $V$ and $W$ be real inner product spaces
	and let $T \colon V \to W$ be an inner product.
	Show that the following diagram commutes:
	\begin{center}
	\begin{tikzcd}
		W \ar[r, "T^\dagger"] \ar[d, "\cong"']
			& V \ar[d, "\cong"] \\
		W^\vee \ar[r, "T^\vee"']  & V^\vee
	\end{tikzcd}
	\end{center}
	Here the isomorphisms are $v \mapsto \left< -, v\right>$.
	Thus, for real inner product spaces,
	$T^\dagger$ is just $T^\vee$ with the duals eliminated
	(by \Cref{thm:real_dual_isomorphic}).
\end{problem}

\begin{problem}
	[Polynomial criteria for normality]
	Let $V$ be a complex inner product space
	and let $T \colon V \to V$ be a linear map.
	Show that $T$ is normal if and only if
	there is a polynomial\footnote{Here,
		$p(T)$ is meant in the same composition
		sense as in Cayley-Hamilton.}
	$p \in \CC[t]$ such that \[ T^\dagger = p(T). \]
	\begin{hint}
		If there is a polynomial, check $TT^\dagger = T^\dagger T$ directly.
		If $T$ is normal, diagonalize it.
	\end{hint}
	\begin{sol}
		First, suppose $T^\ast = p(T)$.
		Then $T^\ast T = p(T) \cdot T = T \cdot p(T) = T T^\ast$ and we're done.

		Conversely, suppose $T$ is diagonalizable
		in a way compatible with the inner form
		(OK since $V$ is finite dimensional).
		Consider the orthonormal basis.
		Then $T$ consists of eigenvalues on the main diagonals
		and zeros elsewhere, say
		\[ T = \left(
			\begin{array}{cccc}
				\lambda_1 & 0 & \dots & 0 \\
				0 & \lambda_2 & \dots & 0 \\
				\vdots & \vdots & \ddots &  \vdots \\
				0 & 0 & \dots & \lambda_n
			\end{array}
			\right). \]
		In that case, we find that for any polynomial $q$ we have
		\[ q(T) = \left(
			\begin{array}{cccc}
				q(\lambda_1) & 0 & \dots & 0 \\
				0 & q(\lambda_2) & \dots & 0 \\
				\vdots & \vdots & \ddots &  \vdots \\
				0 & 0 & \dots & q(\lambda_n)
			\end{array}
			\right). \]
		and
		\[ T^\ast = \left(
			\begin{array}{cccc}
				\ol{\lambda_1} & 0 & \dots & 0 \\
				0 & \ol{\lambda_2} & \dots & 0 \\
				\vdots & \vdots & \ddots &  \vdots \\
				0 & 0 & \dots & \ol{\lambda_n}
			\end{array}
			\right). \]
		So we simply require a polynomial $q$
		such that $q(\lambda_i) = \ol{\lambda_i}$ for every $i$.
		Since there are finitely many $\lambda_i$,
		we can construct such a polynomial using Lagrange interpolation.
	\end{sol}
\end{problem}


\section{Polygons}
In this section we will discuss basic tasks on polygons: how to find their area and two ways to detect if a point is inside or outside them.

\subsection{Polygon area}\label{ss:polygon-area}
To compute the area of a polygon, it is useful to first consider the area of a triangle $ABC$.

\centerFig{polygon0}

We know that the area of this triangle is $\frac{1}{2} |AB| |AC| \sin\theta$, because $|AC| \sin\theta$ is the length of the height coming down from $C$. This looks a lot like the definition of cross product: in fact,
\[\frac{1}{2} |AB| |AC| \sin\theta = \frac{1}{2} \left|\crossv{AC}{AC}\right|\]

Since $O$ is the origin, it can be implemented simply like this:
\begin{lstlisting}
double areaTriangle(pt a, pt b, pt c) {
    return abs(cross(b-a, c-a)) / 2.0;
}
\end{lstlisting}

Now that we can compute the area of a triangle, the intuitive way to find the area of a polygon would be to
\begin{enumerate}
\item divide the polygon into triangles;
\item add up all the areas.
\end{enumerate}

However, it turns out that reliably dividing a polygon into triangles is a difficult problem in itself. So instead we'll add and subtract triangle areas in a clever way. Let's take this quadrilateral as an example:
\centerFig{polygon1}

Let's take an arbitrary reference point $O$. Let's consider the vertices of $ABCD$ in order, and for every pair of consecutive points $P_1,P_2$, we'll add the area of $OP_1P_2$ to the total if $\vv{P_1P_2}$ goes counter-clockwise around $O$, and subtract it otherwise. Additions are marked in blue and subtractions in red.
\centerFig{polygon2}

We can see that this will indeed compute the area of quadrilateral $ABCD$. In fact, it works for any polygon (draw a few more examples to convince yourself).

Note that the sign (add or subtract) that we take for the area of $OP_1P_2$ is exactly the sign that the cross product takes. If we take the origin as reference point $O$, it gives this simple implementation:
\begin{lstlisting}
double areaPolygon(vector<pt> p) {
    double area = 0.0;
    for (int i = 0, n = p.size(); i < n; i++) {
        area += cross(p[i], p[(i+1)%n]); // wrap back to 0 if i == n-1
    }
    return abs(area) / 2.0;
}
\end{lstlisting}
We have to take the absolute value in case the vertices are given in clockwise order. In fact, testing the sign of \lstinline|area| is a good way to know whether the vertices are in counter-clockwise (positive) or clockwise (negative) order. It is good practice to always put your polygons in counter-clockwise order, by reversing the array of vertices if necessary, because some algorithms on polygons use this property.

\subsection{Cutting-ray test}
Let's say we want to test if a point $A$ is inside a polygon $P_1 \cdots P_n$. Then one way to do it is to draw an imaginary ray from $A$ that extends to infinity, and check how many times this ray intersects $P_1 \cdots P_n$. If the number of intersections is odd, $A$ is inside, and if it is even, $A$ is outside.

\centerFig{polygon3}
%For example, in the figure above, the ray from $A_1$ has 3 intersections with the polygon so $A_1$ is inside, while the ray from $A_2$ has 2 intersections with the polygon so $A_2$ is outside.

However, sometimes this can go wrong if the ray touches a vertex of the polygon, as below. The ray from $A_3$ intersects the polygon twice, but $A_3$ is inside. We can try to solve the issue by counting one intersection per segment touched, which would give three intersections for $A_3$, but then the ray from $A_4$ will intersect the polygon twice even though $A_4$ is inside.

\centerFig{polygon4}

So we need to be more careful in defining what counts as an intersection. We will split the plane into two halves along the ray: the points lower than $A$, and the points at least as high (blue region). We then say that a segment $[P_iP_{i+1}]$ crosses the ray right of $A$ if it touches it \emph{and} $P_i$ and $P_{i+1}$ are on opposite halves.

\centerFig{polygon5}

Below we show for some segments whether they are considered to cross the ray or not. We can see in the last two examples that the behavior is different if the segment touches the ray from below or from above.

\begin{center}
\includeFig{polygon6}
\includeFig{polygon11}
\end{center}

\exo{Verify that, with this new definition of crossing, $A_3$ and $A_4$ are correctly detected to be inside the polygon.}

Checking the halves to which the points belong is easy, but checking that the segment touches the ray is a bit more tricky. We could check whether the segments $[P_i,P_{i+1}]$ and $[AB]$ intersect for $B$ very far on the ray, but it actually we can do it more simply using $\orient$: if $P_i$ is below and $P_{i+1}$ above, then $\orient(A,P_i,P_{i+1})$ should be positive, and otherwise it should be negative.
We can then implement this with the code below:
\begin{lstlisting}
// true if P at least as high as A (blue part)
bool above(pt a, pt p) {
    return p.y >= a.y;
}
// check if [PQ] crosses ray from A
bool crossesRay(pt a, pt p, pt q) {
    return (above(a,q) - above(a,p)) * orient(a,p,q) > 0;
}
\end{lstlisting}

If we now return to the original problem, we still have to check whether $A$ is on the boundary of the polygon. We can do that by using \lstinline|onSegment()| defined in \ref{onsegment}.
\begin{lstlisting}
// if strict, returns false when A is on the boundary
bool inPolygon(vector<pt> p, pt a, bool strict = true) {
    int numCrossings = 0;
    for (int i = 0, n = p.size(); i < n; i++) {
        if (onSegment(p[i], p[(i+1)%n], a))
            return !strict;
        numCrossings += crossesRay(a, p[i], p[(i+1)%n]);
    }
    return numCrossings & 1; // inside if odd number of crossings
}
\end{lstlisting}

\subsection{Winding number}\label{ss:wind-2d}
Another way to test if $A$ is inside polygon $P_1 \cdots P_n$ is to think of a string with one end attached at $A$ and the other following the boundary of the polygon, doing one turn. If between the start position and the end position the string has done a full turn, then we are inside the polygon. If however the direction string has simply oscillated around the same position, then we are outside the polygon. Another way to test it is to place one finger on point $A$ while another one follows the boundary of the polygon, and see if the fingers are twisted at the end.

This idea can be generalized to the \emph{winding number}. The winding number of a closed curve around a point is the number of times this curve turns counterclockwise around the point. Here is an example.

\centerFig{polygon7}

Points $A_1$ and $A_2$ are completely out of the curve so the winding number around them is $0$ (no turn). Points $A_3$ and $A_4$ are inside the main loop, which goes counterclockwise, so the winding number around them is $1$. The curve turns twice counterclockwise around $A_5$, so the winding number is $2$. Finally the curve goes clockwise around $A_6$, for a winding number of $-1$.

\begin{mathy}
In fact, we can move the curve continuously without changing the winding number as long as we don't touch the reference point. Therefore we can ``untie'' loops which don't contain the point. That's why, when looking at $A_3$ or $A_4$, we can completely ignore the loops that contain $A_5$ and $A_6$.
\end{mathy}

\begin{mathy}
If we move the reference point while keeping the curve unchanged, the value of the winding number will only change when it crosses the curve. If it crosses the curve from the right (according to its orientation), the winding number increases by 1, and if it crosses it from the left, the winding number decreases by 1.

\centerFig{polygon8}
\end{mathy}

\exoWithSolution{
    What value will \lstinline|areaPolygon()| (section~\ref{ss:polygon-area}) give when applied to a closed polyline that crosses itself, like the curve above, instead of a simple polygon? Assume we don't take the absolute value.
}{
    It will give the sum of the areas of the parts delimited by the curve, multiplied by their corresponding winding numbers. For the curve below, it will give the sum of:
    \protect\begin{itemize}
    \protect\item $1\ \times$ the area of the part containing $A_3$ and $A_4$;
    \protect\item $2\ \times$ the area of the part containing $A_5$;
    \protect\item $-1\ \times$ the area of the part containing $A_6$.
    \protect\end{itemize}

    \protect\centerFig{polygon7}
}{area-crossing}

To compute the winding number, we need to keep track of the amplitude travelled, positive if counterclockwise, and negative if clockwise. We can use \lstinline|angle()| from section~\ref{ss:dot} to help us.
\begin{lstlisting}
// amplitude travelled around point A, from P to Q
double angleTravelled(pt a, pt p, pt q) {
    double ampli = angle(p-a, q-a);
    if (orient(a,p,q) > 0) return ampli;
    else return -ampli;
}
\end{lstlisting}

Another way to implement it uses the arguments of points:
\begin{lstlisting}
double angleTravelled(pt a, pt p, pt q) {
    // remainder ensures the value is in [-pi,pi]
    return remainder(arg(q-a) - arg(p-a), 2*M_PI);
}
\end{lstlisting}

Then we simply sum it all up and figure out how many turns were made:
\begin{lstlisting}
int windingNumber(vector<pt> p, pt a) {
    double ampli = 0;
    for (int i = 0, n = p.size(); i < n; i++)
        ampli += angleTravelled(a, p[i], p[(i+1)%n]);
    return round(ampli / (2*M_PI));
}
\end{lstlisting}

\begin{warning}
The winding number is not defined if the reference point is on the curve/polyline. If it is the case, this code will give arbitrary results, and potentially \lstinline|(int)NAN|.
\end{warning}

\subsubsection{Angles of integer points}
While the code above works, its use of floating-point numbers makes it non ideal, and when coordinates are integers, we can do better. We will define a new way to work with angles, as a type \lstinline|angle|. This type will also be useful for other tasks, such as for sweep angle algorithms.

Instead of working with amplitudes directly, we will represent angles by a point and a certain number of full turns.\footnote{This approach is based on an original idea in \cite{kactl}, see ``Angle.h''.} More precisely, in this case, we will use point $(x,y)$ and number of turns $t$ to represent angle $\atanTwo(y,x) + \turn t$.

We start by defining the new type \lstinline|angle|. We also define a utility function \lstinline|t360()| which turns an angle by a full turn.
\begin{lstlisting}
struct angle {
    pt d; int t = 0; // direction and number of full turns
    angle t180(); // to be defined later
    angle t360() {return {d, t+1};}
};
\end{lstlisting}
The range of angles which have the same value for $t$ is $(-\half + \turn t, \half + \turn t]$.

We will now define a comparator between angles. The approach is the same as what we did for the polar sort in section~\ref{polar-sort}, so we will reuse the function \lstinline|half()| which separates the plane into two halves so that angles within one half are easily comparable:
\begin{lstlisting}
bool half(pt p) {
    return p.y > 0 || (p.y == 0 && p.x < 0);
}
\end{lstlisting}

It returns \lstinline|true| for the part highlighted in blue and \lstinline|false| otherwise. Thus, in practice, it allows us to separate each range $(-\half + \turn t, \half + \turn t]$ into the subranges $(-\half + \turn t, \turn t]$, for which \lstinline|half()| returns \lstinline|false|, and $(\turn t, \half + \turn t]$, for which \lstinline|half()| returns \lstinline|true|.
\centerFig{polygon9}

We can now write the comparator between angles, which is nearly identical to the one we used for polar sort, except that we first check the number of full turns \lstinline|t|.
\begin{lstlisting}
bool operator<(angle a, angle b) {
    return make_tuple(a.t, half(a.d), 0) <
           make_tuple(b.t, half(b.d), cross(a.d,b.d));
}
\end{lstlisting}

We also define the function \lstinline|t180()| which turns an angle by half a turn counterclockwise. The resulting angle has an opposite direction. To find the number of full turns $t$, there are two cases:
\begin{itemize}
\item if \lstinline|half(d)| is \lstinline|false|, we are in the lower half $(-\half + \turn t, \turn t]$, and we will move to the upper half $(\turn t, \half + \turn t]$, without changing $t$;
\item if \lstinline|half(d)| is \lstinline|true|, we are in the upper half $(\turn t, \half + \turn t]$, and we will move to $(-\half + \turn (t+1), \turn (t+1)]$, the lower half for $t+1$.
\end{itemize}
\begin{lstlisting}
angle t180() {return {d*(-1), t + half(d)};}
\end{lstlisting}

We will now implement the function that will allow us to compute the winding number. Consider an angle with direction point $D$. Given a new direction $D'$, we would like to move the angle in such a way that if direction $D'$ is to the left of $D$, the angle increases, and if $D'$ is to the right of $D$, the angle decreases.

\centerFig{polygon10}

In other words, we want the new angle to be an angle with direction $D'$, and such that the difference between it and the old angle is at most $180\degree$. We will use this formulation to implement the function:
\begin{lstlisting}
angle moveTo(angle a, pt newD) {
    // check that segment [DD'] doesn't go through the origin
    assert(!onSegment(a.d, newD, {0,0}));

    angle b{newD, a.t};
    if (a.t180() < b) // if b more than half a turn bigger
        b.t--;        //     decrease b by a full turn
    if (b.t180() < a) // if b more than half a turn smaller
        b.t++;        //     increase b by a full turn
    return b;
}
\end{lstlisting}
We know that \lstinline|b| as it is first defined is less than a full turn away from \lstinline|a|, so the two conditions are enough to bring it within half a turn of \lstinline|a|.

We can use this to implement a new version of \lstinline|windingNumber()| very simply. We start at some vertex of the polygon, move vertex to vertex while maintaining the angle, then read the number of full turns once we come back to it.
\begin{lstlisting}
int windingNumber(vector<pt> p, pt a) {
    angle a{p.back()}; // start at last vertex
    for (pt d : p)
        a = moveTo(a, d); // move to first vertex, second, etc.
    return a.t;
}
\end{lstlisting}