\section{Surface Integrals}\label{sec:SurfaceIntegrals}

In the integral for surface area, from Equation \ref{eq:surfaceareaintegral},
$$\int_a^b\int_c^d |\vect{r}_u\times\vect{r}_v|\,du\,dv,$$
the integrand $|\vect{r}_u\times\vect{r}_v|\,du\,dv$
is the area of a tiny parallelogram, that is, a very small surface
area, so it is reasonable to abbreviate it $dS$; then a shortened
version of the integral is
$$\iint_{D} 1\cdot dS.$$
We have already seen that if $D$ is a region in the plane, the area of
$D$ may be computed with
$$\iint_{D} 1\cdot dA,$$
so this is really quite familiar, but the $dS$ hides a little more
detail than does $dA$.

Just as we can integrate functions $f(x,y)$ over regions in the plane,
using
$$\iint_{D} f(x,y)\, dA,$$
so we can compute integrals over surfaces in space, using
$$\iint_{D} f(x,y,z)\, dS.$$
In practice this means that we have a vector function
$\vect{r}(u,v)=\langle x(u,v),y(u,v),z(u,v)\rangle$ for the surface,
and the integral we compute is
$$\int_a^b\int_c^d f(x(u,v),y(u,v),z(u,v))|\vect{r}_u\times\vect{r}_v|\,du\,dv.$$
That is, we express everything in terms of $u$ and $v$, and then we
can do an ordinary double integral.

\begin{example}{Mass and Center of Mass}{}
Suppose a thin object occupies the upper hemisphere of
$x^2+y^2+z^2=1$ and has density $\sigma(x,y,z)=z$. Find the mass
and center of mass of the
object. (Note that the object is just a thin shell; it does not occupy
the interior of the hemisphere.)
\end{example}
\begin{solution}
We write the hemisphere as $\vect{r}(\phi,\theta)=
\langle \cos\theta\sin\phi, \sin\theta\sin\phi, \cos\phi\rangle$,
$0\le\phi\le \pi/2$ and $0\le\theta\le 2\pi$. So
$\vect{r}_\theta = \langle -\sin\theta\sin\phi, \cos\theta\sin\phi, 0\rangle$
and
$\vect{r}_\phi =\langle \cos\theta\cos\phi, \sin\theta\cos\phi, -\sin\phi\rangle$.
Then
$$\vect{r}_\theta\times\vect{r}_\phi =
\langle -\cos\theta\sin^2\phi,-\sin\theta\sin^2\phi,-\cos\phi\sin\phi\rangle$$
and
$$ |\vect{r}_\theta\times\vect{r}_\phi| = |\sin\phi| = \sin\phi,$$
since we are interested only in $0\le\phi\le \pi/2$.
Finally, the density is $z=\cos\phi$ and the integral for mass is
$$\int_0^{2\pi}\int_0^{\pi/2} \cos\phi\sin\phi\,d\phi\,d\theta=\pi.$$

By symmetry, the center of mass is clearly on the $z$-axis, so we only
need to find the $z$-coordinate of the center of mass. The moment
around the $x$-$y$ plane is
$$\int_0^{2\pi}\int_0^{\pi/2} z\cos\phi\sin\phi\,d\phi\,d\theta
=\int_0^{2\pi}\int_0^{\pi/2} \cos^2\phi\sin\phi\,d\phi\,d\theta
={2\pi\over 3},$$
so the center of mass is at $(0,0,2/3)$.
\end{solution}

Now suppose that $\vect{f}$ is a vector field; imagine that it
represents the velocity of some fluid at each point in space. We would
like to measure how much fluid is passing through a surface $D$, the
\dfont{flux}\index{flux} across $D$. As usual, we imagine computing
the flux across a very small section of the surface, with area $dS$,
and then adding up all such small fluxes over $D$ with an
integral. Suppose that vector $\vect{N}$ is a unit normal to the surface
at a point; $\vect{f}\cdot\vect{N}$ is the scalar projection of $\vect{f}$
onto the direction of $\vect{N}$, so it measures how fast the fluid is
moving across the surface. In one unit of time the fluid moving across
the surface will fill a volume of $\vect{f}\cdot\vect{N}\,dS$, which is
therefore the rate at which the fluid is moving across a small patch
of the surface. Thus, the total flux across $D$ is
$$\iint_{D} \vect{f}\cdot\vect{N}\,dS=\iint_{D} \vect{f}\cdot\,d\vect{S},$$
defining $d\vect{S}=\vect{N}\,dS$.

As usual, certain conditions must be met for this to work out; chief
among them is the nature of the surface. As we integrate over the
surface, we must choose the normal vectors $\vect{N}$ in such a way that
they point ``the same way'' through the surface. For example, if the
surface is roughly horizontal in orientation, we might want to measure
the flux in the ``upwards'' direction, or if the surface is closed,
like a sphere, we might want to measure the flux ``outwards'' across
the surface. In the first case we would choose $\vect{N}$ to have
positive $z$ component, in the second we would make sure that $\vect{N}$
points away from the origin. Unfortunately, there are surfaces that
are not \dfont{orientable}\index{orientable surface}: they have
only one side, so that it is not possible to choose the normal vectors
to point in the ``same way'' through the surface. The most famous such
surface is the M\"obius strip shown in Figure~\ref{fig:moebius}. It
is quite easy to make such a strip with a piece of paper and some
tape. If you have never done this, it is quite instructive; in
particular, you should draw a line down the center of the strip until
you return to your starting point. No matter how unit normal vectors
are assigned to the points of the M\"obius strip, there will be normal
vectors very close to each other pointing in opposite directions.

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <2.5truecm,2.5truecm>
\setplotarea x from -1 to 1, y from 0 to 1
\put {\hbox{\epsfxsize7cm\epsfbox{images/moebius.eps}}} at 0 0
\endpicture}}
\caption{A M\"obius strip. \label{fig:moebius}}
\end{figure}

Assuming that the quantities involved are well behaved, however, the
flux of the vector field across the surface $\vect{r}(u,v)$ is
$$\iint_{D} \vect{f}\cdot\vect{N}\,dS
=\iint_{D}\vect{f}\cdot
 {\vect{r}_u\times\vect{r}_v\over|\vect{r}_u\times\vect{r}_v|}
 |\vect{r}_u\times\vect{r}_v|\,dA
=\iint_{D}\vect{f}\cdot (\vect{r}_u\times\vect{r}_v)\,dA.$$
In practice, we may have to use $\vect{r}_v\times\vect{r}_u$
or even something a bit more complicated to make sure that the normal
vector points in the desired direction.

\begin{example}{Flux}{}
Compute the flux of $\vect{f}=\langle x,y,z^2\rangle$ across the
cone $z=\sqrt{x^2+y^2}$, $0\le z\le 2$, in the downward direction.
\end{example}
\begin{solution}
We write the cone as a vector function: $\vect{r}=\langle v\cos u, v\sin u,
v\rangle$, $0\le u\le 2\pi$ and $0\le v\le 2$.
Then $\vect{r}_u=\langle -v\sin u, v\cos u,0\rangle$ and
$\vect{r}_v=\langle \cos u, \sin u, 1\rangle$ and
$\vect{r}_u\times\vect{r}_v=\langle v\cos u,v\sin u,-v\rangle$.
The third coordinate $-v$ is negative, which is exactly what we
desire, that is, the normal vector points down through the
surface. Then
\begin{align*}
\int_0^{2\pi}\int_0^2 \langle x,y,z^2\rangle\cdot\langle v\cos u,v\sin u,-v\rangle \,dv\,du
&=\int_0^{2\pi}\int_0^2 xv\cos u+yv\sin u-z^2v\,dv\,du	\\
&=\int_0^{2\pi}\int_0^2 v^2\cos^2 u+ v^2\sin^2 u-v^3\,dv\,du	\\
&=\int_0^{2\pi}\int_0^2 v^2-v^3\,dv\,du=-\frac{8\pi}{3}.
\end{align*}
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:SurfaceIntegrals}}

\begin{enumialphparenastyle}

\begin{ex}
Find the center of mass of an object that occupies the upper
hemisphere of $x^2+y^2+z^2=1$ and has density $x^2+y^2$.
\begin{sol}
	$(0,0,3/8)$
\end{sol}
\end{ex}

\begin{ex}
Find the center of mass of an object that occupies the
surface $z=xy$, $0\le x\le1$, $0\le y\le 1$ and has density $\sqrt{1+x^2+y^2}$.
\begin{sol}
	$(11/20,11/20,3/10)$
\end{sol}
\end{ex}

\begin{ex}
Find the center of mass of an object that occupies the
surface $\ds z=\sqrt{x^2+y^2}$, $1\le z\le4$ and has density $x^2z$.
\begin{sol}
	$(0,0,1364/425)$
\end{sol}
\end{ex}

\begin{ex}
Find the centroid of the surface of a right circular cone of
height $h$ and base radius $r$, not including the base.
\begin{sol}
	on center axis, $h/3$ above the base
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds \iint_{D} \langle 2,-3,4\rangle\cdot \vect{N}\,dS$, where $D$ is given by $z=x^2+y^2$, $-1\le x\le 1$, $-1\le
y\le 1$, oriented up.
\begin{sol}
	$16$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds \iint_{D} \langle x,y,3\rangle\cdot\vect{N}\,dS$, where $D$ is given by $z=3x-5y$, $1\le x\le 2$, $0\le
y\le 2$, oriented up.
\begin{sol}
	$7$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds \iint_{D} \langle x,y,-2\rangle\cdot \vect{N}\,dS$, where $D$ is given by $z=1-x^2-y^2$, $x^2+y^2\le1$,
oriented up.
\begin{sol}
	$-\pi$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds \iint_{D} \langle e^x, e^y,z\rangle\cdot\vect{N}\,dS$, where $D$ is given by $z=xy$, $0\le x\le 1$, $-x\le
y\le x$, oriented up.
\begin{sol}
	$-2/e$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds \iint_{D} \langle xz,yz,z\rangle\cdot\vect{N}\,dS$, where $D$ is given by $z=a^2-x^2-y^2$, $x^2+y^2\le b^2$,
oriented up.
\begin{sol}
	$\pi b^2(-4b^4-3b^2+6a^2b^2+6a^2)/6$
\end{sol}
\end{ex}

\begin{ex}
A fluid has density 870 kg/m$^3$ and flows with velocity $\vect{v} =
 \langle z,y^2,x^2\rangle$, where distances are in meters and the
 components of $\vect{v}$ are in meters per second.  Find the rate of flow
 outward through the portion of the cylinder $x^2+y^2 = 4$, $0\leq
 z\leq 1$ for which $y>0$.
\begin{sol}
	$9280$ kg/s
\end{sol}
\end{ex}

\begin{ex}
Gauss's Law says that the net charge, $Q$,
enclosed by a closed surface, $S$, is
$$Q=\epsilon_0 \iint \vect{e}\cdot \vect{N}\,dS$$
where $\vect{e}$ is an electric field and $\epsilon_0$ (the
permittivity of free space) is a known constant; $\vect{N}$ is oriented
outward.
Use Gauss's Law to find the charge contained in the cube with vertices
$(\pm 1, \pm 1, \pm 1)$ if the electric field is
$\vect{e} = \langle x,y,z\rangle$.
\begin{sol}
	$24\epsilon_0$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\subsection{Cardinality of the natural numbers}

Consider the infinite set, that is the set of all natural numbers which is defined in ZFC. Clearly there isn’t a natural number cardinality of this – we instead write \(\aleph_0\).

We call sets with this cardinality, countably infinite.

So:

\(|\mathbb{N} |=\aleph_0\)

\subsubsection{Cardinality of natural  numbers}

We define:

\(|\emptyset |=0\)

That, the empty set has a cardinality of \(0\).

As we define \(0\) as the empty set, \(|0|=0\).

What is \(1\)? using the definition above we know \(|1|>|0|\), so let's say \(|1|=1\), and more generally:

\(\forall n \in \mathbb{N} |n|=n\)


\chapter{DT Frequency Response}

In this lecture we are going to focus on the frequency response of discrete-time systems and highlight it's importance in linear systems theory.

\section{Determining the frequency response (FR) of a DT system}

The frequency response of a DT LTI system can be thought of as arising in several equivalent ways. What follows is a common, but not exhaustive, list of ways the frequency response can be derived from other representations.

\subsection*{Using the Eigenvalues / Transfer Function}

Recall if we apply the Eigenfunction $z^{n}$ for $z \in \mathbb{C}$ as the input to a LTI system, the output is the Eigenfunction scaled by the Eigenvalue (transfer function) $H(z)$ for values of $z$ in the region of convergence, where
\[
H(z) = \sum\limits_{-\infty}^{\infty} h[n] z^{-n}\; dt \; .
\]
is the bilateral Z transform of the impulse response.

\begin{center}
  \includegraphics[scale=1]{graphics/19-dt-tf.pdf}
\end{center}

If a system is stable, then the region of convergence includes the unit circle $z = e^{j\omega}$. In that case, evaluating the Eigenvalues on the unit circle gives the DT frequency response $H\left(e^{j\omega}\right)$. This converts from a function of a complex variable, $z$, to one of a real variable $\omega$.

\begin{example} Consider a system with Eigenvalues (transfer function)
  \[
  H(z) = \frac{z}{z+\tfrac{1}{2}}\mbox{ for } |z| > \frac{1}{2}
  \]
  Determine the frequency response of the system, if possible.\\

  Solution: We first need to check of the system is stable using the region-of-convergence. Since the region of convergence includes the unit circle, the system is stable. To find the frequency response we substitute $s = e^{j\omega}$ to give
  \[
  H\left(e^{j\omega}\right) = \frac{e^{j\omega}}{e^{j\omega} + \tfrac{1}{2}}
  \]
  \\$\blacksquare$
\end{example}

\begin{example} Consider an apparently similar system with Eigenvalues
  \[
  H(z) = \frac{z}{z+2}\mbox{ for } |z| > 2
  \]
  Determine the frequency response of the system, if possible.\\

  Solution: Again, we first need to check of the system is stable using the region-of-convergence. Since the region of convergence does not include the unit circle, the system is unstable. Thus, the frequency response does not exist.
\\$\blacksquare$
\end{example}

\subsection*{Using the DTFT}

Another way we can view the frequency response is as the DT Fourier Transform of the impulse response. If the system is stable, then the impulse response is absolutely integrable, and the Fourier transform exists giving $H\left(e^{j\omega}\right) = \mathcal{F}\left\{h[n]\right\}$. This is connected to the transfer function by noting the bilateral Z transform and the DT Fourier Transform are identical under the substitution $z = e^{j\omega}$, which is allowed if the system is stable.

\begin{example} Suppose the impulse response of a DT LTI system is given by
  \[
  h[n] = \left(\frac{1}{4}\right)^n u[n] + 5\left(\frac{2}{3}\right)^n u[n]
  \]
  Determine the frequency response of the system, if possible.\\

  Solution: If the system is stable, the Fourier transform of the impulse response exists. Since $\left(\frac{1}{4}\right) < 1$ and $\left(\frac{2}{3}\right) < 1$
  \[
H\left(e^{j\omega}\right) = \mathcal{F}\left\{ \left(\frac{1}{4}\right)^n u[n] + 5\left(\frac{2}{3}\right)^n u[n] \right\} = \mathcal{F}\left\{ \left(\frac{1}{4}\right)^n u[n]\right\} + 5 \mathcal{F}\left\{ \left(\frac{2}{3}\right)^n u[n] \right\} = \frac{e^{j\omega}}{e^{j\omega} - \left(\frac{1}{4}\right)} + \frac{5e^{j\omega}}{e^{j\omega} - \left(\frac{2}{3}\right)}
\]\\
$\blacksquare$
\end{example}

\subsection*{Directly from a LCCDE}

By the convolution theorem of the DTFT, the frequency response is the ratio of the output to input in the frequency domain, i.e.
\[
H\left(e^{j\omega}\right) = \frac{Y\left(e^{j\omega}\right)}{X\left(e^{j\omega}\right)}
\]
We can easily determine this ratio from the LCCDE representation of the system using the shifting property of the DT Fourier Transform. Recall this property states if $\mathcal{F}\{x[n]\} = X\left(e^{j\omega}\right)$ then
\[
\mathcal{F}\left\{x[n-m] \right\} =  e^{-j\omega m} X\left(e^{j\omega}\right)\; .
\]
for index shift $m\in\mathbb{Z}$.

If the system is stable (and thus the frequency response exists) then \textbf{all} roots of the characteristic equation $Q(E)$ have magnitude that are less than one. If the system is stable we can take the Fourier transform of each term of the LCCDE using the shift property, then algebraically solve for the ratio of output to input. Note this provides a significant savings in analysis effort since we do not have to first find the impulse response, then take it's Fourier transform to arrive at the frequency response (although that approach is still valid).

\begin{example} Consider a system described by the LCCDE
  \[
  3y[n+1]-y[n] = x[n+1]
  \]
  Determine the frequency response of the system, if possible.

  Solution: We first need to check for stability. The characteristic equation is $Q(E) = 3E - 1$ which has a single root of $\frac{1}{3}$. Since it is less than one, the system is stable. Next we take the Fourier transform of both sides and apply the derivative property
  \[
  3e^{j\omega} Y\left(e^{j\omega}\right) - Y\left(e^{j\omega}\right) = e^{j\omega} X\left(e^{j\omega}\right)
  \]
  and rearrange to get the frequency response
  \[
  H\left(e^{j\omega}\right) = \frac{e^{j\omega}}{3e^{j\omega}-1}
  \]\\
  $\blacksquare$
\end{example}

\section{Magnitude-phase representation of the DTFR}

Note that any complex valued function can be expressed in polar form using the magnitude and phase. Specifically the input and output can be put into this form
\[
X\left(e^{j\omega}\right) = |X\left(e^{j\omega}\right)|e^{\angle X\left(e^{j\omega}\right)}
\]
\[
Y\left(e^{j\omega}\right) = |Y\left(e^{j\omega}\right)|e^{\angle Y\left(e^{j\omega}\right)}
\]

By the convolution theorem then
  \[
  H\left(e^{j\omega}\right) = \frac{Y\left(e^{j\omega}\right)}{X\left(e^{j\omega}\right)} = \frac{|Y\left(e^{j\omega}\right)|e^{\angle Y\left(e^{j\omega}\right)}}{ |X\left(e^{j\omega}\right)|e^{\angle X\left(e^{j\omega}\right)}} = \frac{|Y\left(e^{j\omega}\right)|}{|X\left(e^{j\omega}\right)|}e^{\angle Y\left(e^{j\omega}\right) - \angle X\left(e^{j\omega}\right)} = |H\left(e^{j\omega}\right)|e^{\angle H\left(e^{j\omega}\right)}
  \]
  Thus we see that
  \[
  |H\left(e^{j\omega}\right)| = \frac{|Y\left(e^{j\omega}\right)|}{|X\left(e^{j\omega}\right)|}
  \]
  and
  \[
  \angle H\left(e^{j\omega}\right) = \angle Y\left(e^{j\omega}\right) - \angle X\left(e^{j\omega}\right)
  \]
  This is the magnitude and phase representation of the frequency response.

\section{DTFR acting on sinusoids}

The advantage of the magnitude and phase representation of the frequency response, is the ease with which we can find the output due to a sinusoidal input. If we apply a sinusoidal input $x[n] = A e^{j\omega n}$, the output is a the same sinusoid scaled by the frequency response $y[n] = H\left(e^{j\omega}\right) A e^{j\omega n}$.

\begin{center}
  \includegraphics[scale=1]{graphics/19-dt-fr.pdf}
\end{center}

Now using the magnitude and phase representation
\[
y[n] = H\left(e^{j\omega}\right) A e^{j\omega n} = |H\left(e^{j\omega}\right)|e^{\angle H\left(e^{j\omega}\right)} A e^{j\omega n} = A |H\left(e^{j\omega}\right)| e^{j\omega n + \angle H\left(e^{j\omega}\right)}
\]
Thus we can interpret the frequency response as telling us how the input sinusoids are scaled in magnitude and phase shifted as they pass through the system.

By the linearity property this extends to real sinusoidal inputs since
\begin{align*}
  x[n] &\longrightarrow y[n]\\
  \sin(\omega n) &\longrightarrow \frac{1}{2j}|H\left(e^{j\omega}\right)| e^{j\omega n + \angle H\left(e^{j\omega}\right)} - \frac{1}{2j}|H\left(e^{j\omega}\right)| e^{-j\omega n + \angle H\left(e^{j\omega}\right)}\\
  \sin(\omega n) &\longrightarrow |H\left(e^{j\omega}\right)|\sin(\omega n + \angle H\left(e^{j\omega}\right))
\end{align*}
and
\begin{align*}
  x[n] &\longrightarrow y[n]\\
  \cos(\omega n) &\longrightarrow \frac{1}{2}|H\left(e^{j\omega}\right)| e^{j\omega n + \angle H\left(e^{j\omega}\right)} + \frac{1}{2}|H\left(e^{j\omega}\right)| e^{-j\omega n + \angle H\left(e^{j\omega}\right)}\\
  \cos(\omega n) &\longrightarrow |H\left(e^{j\omega}\right)|\cos(\omega n + \angle H\left(e^{j\omega}\right))
\end{align*}

Also by the linearity property this analysis extends to the DT Fourier representation of a signal (an infinite sum of sinusoids):
\[
x[n] = \frac{1}{2\pi}\int\limits_{2\pi} X\left(e^{j \omega}\right) \, e^{j \omega n}\; d\omega \;\longrightarrow\; y[n] = \frac{1}{2\pi}\int\limits_{2\pi} H\left(e^{j \omega}\right) X\left(e^{j \omega}\right) \, e^{j \omega n}\; d\omega = \frac{1}{2\pi}\int\limits_{2\pi} \left| H\left(e^{j\omega}\right)\right| X\left(e^{j\omega}\right) \, e^{j \omega n + \angle H\left(e^{j\omega}\right)}\; d\omega
\]

Thus we arrive at the reason for the name DT \textit{Frequency Response} -- it specifies the the response of a stable system to any linear combination of DT sinusoidal inputs, i.e. any signal with a Fourier Transform.

\section{Plotting the DT frequency response}

As in CT, we can visualize the frequency response as a plot of the real and imaginary part, or, of the magnitude and phase. Since the magnitude and phase allow us to directly see the system behavior at a given frequency, those plots are much more useful.

In contrast to CT, where the unique Bode plot format is used, for the DTFR it is most common to plot the magnitude spectrum in dB and the phase spectrum in rad over just $\omega = [0, \pi]$. Since the DTFR is periodic there is no need to compress the information using a logarithmic frequency scale. Further, if $x[n]$ is real the DTFR magnitude spectrum is even, so that the magnitude from $\omega = [\pi, 2\pi]$ is the same as from $\omega = [-\pi, 0]$. Also if $x[n]$ is real the DTFR phase spectrum is odd, so that the phase from $\omega = [\pi, 2\pi]$ is the same as the negative from $\omega = [-\pi, 0]$. Note we would not call this kind of plot a Bode plot as that is typically reserved for the CTFR.

As with the CTFR it is important to understand these plots well enough to create them on your own using software and read them.

\begin{example} Consider a frequency response given by
  \[
  H\left(e^{j\omega}\right) = \frac{4e^{j\omega}}{4e^{j\omega} - 1}
  \]
  The following Matlab code shows you how to plot the spectrum (with some extra code to make it look nicer).

\begin{verbatim}
% compute the FTFR
w = 0:0.001:pi;
H = 4.*exp(j*2*w)./(4*exp(j*2*w) - 1);

% Create a nice DTFR plot
hFig = figure();
hold on;

subplot(2,1,1);
hm = plot(w,20*log10(abs(H)));
axis tight;
grid on;
hTitle  = title ('Frequency Response');
hYLabel1 = ylabel('Magnitude (dB)');
set(gca, 'FontSize', 14, ...
    'Box', 'off', 'LineWidth', 2);

subplot(2,1,2);
hp = plot(w,angle(H));
axis tight;
grid on;
hYLabel2 = ylabel('Phase (radians)');
hXLabel = xlabel('Frequency (rad/sample)');
set(gca, 'FontSize', 14, 'Box', 'off', 'LineWidth', 2);

set(hm, 'linewidth', 2);
set(hp, 'linewidth', 2);
set([hXLabel, hYLabel1, hYLabel2]  , ...
     'FontSize'   , 14          );
set( hTitle                    , ...
     'FontSize'   , 14          , ...
     'FontWeight' , 'bold'      );
\end{verbatim}
This gives the following plot
\begin{center}
\includegraphics[scale=0.5]{graphics/lecture21_1.png}
\end{center}

$\blacksquare$
\end{example}

To read a Bode plot to see the behavior of the system at a given frequency, one need only read the values off the plot and convert from dB to a unit-less gain.

\begin{example}
  Suppose you are given the DTFR plot (only) from the previous example and are asked: what the output of the system is when the input is $x[n] = \cos\left(\frac{\pi}{4} n\right)$?\\
  \textbf{Solution:} We go to the frequency $\frac{\pi}{4} \approx 0.78$ on the plot and read off a value of about $-0.1$ dB for the magnitude and about $-0.25$ rad for the phase. To convert back from dB
  \[
  \left|H\left(e^{j\frac{\pi}{4}}\right)\right| = 10^{\frac{-0.1}{20}} \approx 0.988
  \]
  so the output would be
  \[
  y[n] \approx 0.988\cos\left(\frac{\pi}{4} n  - 0.25\right)
  \]
  $\blacksquare$
\end{example}


\section*{Problem 3 Solution}

\begin{enumerate}[a)]

\item

\begin{align*}
\frac{dy}{dx}	&= -3y \\
\frac{dy}{y} 	& = -3\,dx \\
\int\frac{dy}{y}&= -3 \int \,dx \\
\ln y			&= -3 x + C \\
y				&= e^{-3x + C} \\
\end{align*}
$$\boxed{y = Ce^{-3x}}$$

\item

Solve the second order differential equation
$$ \frac{d^2 y}{dx^2} - A^2y = B $$
For the homogeneous equation, $\frac{d^2 y}{dx^2} - A^2y = 0$, try $y_c = X_1 e^{Ax} + X_2 e^{-Ax}$ as the complementary solution ($X_1$ and $X_2$ are constants). Then
$$ \frac{d^2 y}{dx^2} = X_1 A^2 e^{Ax} + X_2 A^2 e^{-Ax} $$
and
$$ (X_1 A^2 e^{Ax} + X_2 A^2 e^{-Ax}) - A^2(X_1 e^{Ax} + X_2 e^{-Ax}) = 0 $$
$$ A^2 (X_1 e^{Ax} + X_2 e^{-Ax}) - A^2 (X_1 e^{Ax} + X_2 e^{-Ax}) = 0 $$

This is \underline{true}.

For the inhomogeneous equation, $\frac{d^2 y}{dx^2} - A^2y = B$, $y_P = -B/A^2$ is the only particular solution satisfying the equation. The general solution is the sum of the complementary and particular solutions, $y = y_c + y_p$.
$$ y = X_1 e^{Ax} + X_2 e^{-Ax} - \frac{B}{A^2} $$
Now we can solve for $X_1$ and $X_2$.
$$ y(\frac{1}{A}) = X_1 e^{A(\frac{1}{A})} + X_2 e^{-A(\frac{1}{A})} - \frac{B}{A^2} = 0 $$
and
$$ y(-\frac{1}{A}) = X_1 e^{A(-\frac{1}{A})} + X_2 e^{A(\frac{1}{A})} - \frac{B}{A^2} = 0 $$
We can note that in these two equations, $X_1$ and $X_2$ can be interchanged freely, and so must be equal. We will say, $X_1 = X_2 = X$. Then, we have
$$ 0 = X e^{A(-\frac{1}{A})} + X e^{A(\frac{1}{A})} - \frac{B}{A^2} $$
$$ 0 = X (e^{-1} + e^{1}) - \frac{B}{A^2} $$
$$ X = \frac{B}{A^2(\frac{1}{e} + e)} $$
The we plug this $X$ into the final solution, which gives
$$ y = \frac{B}{A^2(\frac{1}{e} + e)} (e^{Ax} + e^{-Ax}) - \frac{B}{A^2} $$
\end{enumerate}
Note that $e^{Ax} + e^{-Ax}$ is similar in form to $\cosh(Ax)$ which we could have also used to solve this problem.


The grammar used in this section is the following:

\begin{align*}
    &S \to BA\\
    &A \to a\\
    &B \to BB|a|b
\end{align*}

\subsubsection{Number of iterations/recursive calls for each parser}

An interesting thing to notice is that with that grammar and a string of size $n$, no matter the used pattern, the amount of needed iterations or recursive calls will be the same for the naive, top-down and `boolean' bottom-up parsers.

The `string' bottom-up parser follows exactly the same behaviour as with the previous grammar.

By decomposing the behaviour of the naive parser one can find out that for any pattern it evolves following that sequence:
\begin{equation}\label{seq:naive}
n + (n - 1)^2
\end{equation}

It can be demonstrated like so:

\begin{itemize}
    \item[$-$] The initial call is $parse(S, 0, n)$
    \item[$-$] Then it calls $parse(B, 0, 1)$, which returns true no matter the character.
    \item[$-$] Then it calls $parse(A, 1, n)$, which returns false since A doesn't possess any \textit{non-terminals}.
    \item[$-$] Then it calls $parse(B, 0, 2)$, which returns true by doing 2 more recursive calls, $parse(B, 0, n)$ will always return true and do two more recursive calls each time it is called.
    \item[$-$] The process ends up when $parse(A, n - 1, n)$ is called, then it returns true or false depending on if the last character is an `a' or a `b'. No matter the result, and so the pattern, the algorithm will always make the same amount of recursive calls.
    \item[$-$] When it ends up the number of recursive calls is $1$ for the initial call, plus $n - 1$ for each time it has called parse(A, x, n), plus $(n - 1)^2$ for each time it has called parse(B, 0, x) and all the recursive calls it made. So the total number of recursive calls is $n + (n - 1)^2$ for the naive parser.
\end{itemize}

The top-down parser follows the exact same behaviour as the naive parser, to parse a string of size $n$ it will also need $n + (n - 1)^2$ recursive calls.

However some sub-problems will not need to be recomputed by the top-down parser, it is possible to show that, for a string of size $n$ and whatever pattern is used, it needs to solve that many sub-problems:
\begin{equation}\label{seq:top-down}
n + (n - 1)^2 - \dfrac{(n - 2) \cdot (n - 1)}{2}
\end{equation}

Here is the demonstration:

\begin{itemize}
    \item[$-$] The running process is exactly the same as for the naive one, the difference is that some results have already been computed and can be directly returned.
    \item[$-$] The amount of results already stored in the table and then asked again increases by one for each level of recursive depth when $parse(B, 0, x)$ is called.
    \item[$-$] This amount corresponds to a triangular number sequence: $\dfrac{n \cdot (n + 1)}{2}$. The formula needs to be twisted a little to match the string size $n$ and the number of asked already known results is $\dfrac{(n - 1) \cdot (n - 2)}{2}$.
    \item[$-$] Then the total amount of solved sub-problems is just the number of recursive calls minus the expression above: $n + (n - 1)^2 - \dfrac{(n - 2) \cdot (n - 1)}{2}$.
\end{itemize}

It is now possible to compare the anticipated behaviours of the naive, top-down and bottom-up parsers, using functions \ref{seq:naive} and \ref{eq:bottom-up_iterations}.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=2 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=`Boolean' bottom-up, xlabel=string size, ylabel=iterations]
    \addplot[red, domain=0:1000]{2 * x + ((3 / 6) * ((3 * (x^2) * (x + 1)) - (x * (x + 1) * ((2 * x) + 1))))};
    \nextgroupplot[title=Naive and top-down, xlabel=string size, ylabel=recursive calls]
    \addplot[blue, domain=0:50000]{x + (x - 1)^2};
    \end{groupplot}
\end{tikzpicture}
\caption{Anticipation of the behaviours of the parsers, grammar 3}
\end{figure}
\FloatBarrier

The `boolean' bottom-up parser follows a power function with an exponent of three as usual, but the top-down and naive parsers will follow a power function with an exponent of two with that grammar.

\subsubsection{Comparing the efficiency}

Except for the `string' bottom-up parser, with this grammar there is no particular cases, any pattern will be as long to parse as another by any of the three other parsers, the first used pattern is $a^n$.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=2 by 2},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Both bottom-up counters, ylabel=iterations, legend pos = north west]
    \addplot coordinates {
        (100, 1500150)
        (200, 12000300)
        (400, 96000600)
        (600, 324000900)
        (800, 768001200)
        (1000, 1500001500)};
    \addlegendentry{String}
    \addplot coordinates {
        (100, 333600)
        (200, 2667200)
        (400, 21334400)
        (600, 72001600)
        (800, 170668800)
        (1000, 333336000)};
    \addlegendentry{Boolean}
    \nextgroupplot[title=Both bottom-up running times, ylabel=seconds, legend pos = north west]
    \addplot coordinates {
        (100, 0.087481)
        (200, 0.652142)
        (400, 5.16778)
        (600, 16.9815)
        (800, 40.5165)
        (1000, 79.5284)};
    \addlegendentry{String}
    \addplot coordinates {
        (100, 0.012412)
        (200, 0.102093)
        (400, 0.82625)
        (600, 3.08987)
        (800, 8.56199)
        (1000, 18.7261)};
    \addlegendentry{Boolean}
    \nextgroupplot[title=Top-down/Naive counters, ylabel=recursive calls, xlabel=string size, legend pos = north west]
    \addplot coordinates {
        (100, 9901)
        (1000, 999001)
        (3000, 8997001)
        (5000, 24995001)
        (7000, 48993001)
        (9000, 80991001)};
    \nextgroupplot[title=Top-down/Naive running times, ylabel=seconds, legend pos=north west, xlabel=string size]
    \addplot coordinates {
        (100, 0.000158)
        (1000, 0.016837)
        (3000, 0.176553)
        (5000, 0.50593)
        (7000, 1.05917)
        (9000, 1.69652)};
    \addlegendentry{TD parsing}
    \addplot coordinates {
        (100, 0.00027)
        (1000, 0.020982)
        (3000, 0.207965)
        (5000, 0.595206)
        (7000, 1.23223)
        (9000, 1.99075)};
    \addlegendentry{TD total}
    \addplot coordinates {
        (100, 0.000091)
        (1000, 0.004109)
        (3000, 0.031383)
        (5000, 0.089245)
        (7000, 0.17303)
        (9000, 0.29419)};
    \addlegendentry{TD init}
    \addplot coordinates {
        (100, 0.000173)
        (1000, 0.01458)
        (3000, 0.134243)
        (5000, 0.374054)
        (7000, 0.776454)
        (9000, 1.17694)};
    \addlegendentry{Naive}
    \end{groupplot}
\end{tikzpicture}
\caption{The 4 parsers behaviours, grammar 3}
\end{figure}
\FloatBarrier

With strings of sizes greater than $9,000$ the top-down parser could not allocate the table so the used strings did not exceed that size.

The same string sizes are used for the top-down and naive parsers, it is possible to see on the graphs above that despite the fact that the naive parser and the top-down parsers need the same amount of recursive calls the naive parser has a lower running time.

The only parser that needs a various number of iterations is the `string' bottom-up parser, the pattern `$a\string^ n$' was as previously its worst case, its best case with that grammar is the pattern `$b\string^ n$' as previously.
It is not a suprise to find out that the `string' bottom-up parser is slower than the `boolean' version with that grammar, since it follows the same behaviour as before.

Next are the results obtained with the `string' bottom-up parser, using the last results for the `boolean' bottom-up parser as reference.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=2 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Both bottom-up counters, ylabel=iterations, xlabel=string size, legend pos = north west]
    \addplot coordinates {
        (100, 500250)
        (200, 4000500)
        (400, 32001000)
        (600, 108001500)
        (800, 256002000)
        (1000, 500002500)};
    \addlegendentry{String}
    \addplot coordinates {
        (100, 333600)
        (200, 2667200)
        (400, 21334400)
        (600, 72001600)
        (800, 170668800)
        (1000, 333336000)};
    \addlegendentry{Boolean}
    \nextgroupplot[title=Both bottom-up running times, legend pos = north west, xlabel=string size, ylabel=seconds]
    \addplot coordinates {
        (100, 0.042859)
        (200, 0.322342)
        (400, 2.60902)
        (600, 8.56177)
        (800, 20.5762)
        (1000, 40.6679)};
    \addlegendentry{String}
    \addplot coordinates {
        (100, 0.012412)
        (200, 0.102093)
        (400, 0.82625)
        (600, 3.08987)
        (800, 8.56199)
        (1000, 18.7261)};
    \addlegendentry{Boolean}
    \end{groupplot}
\end{tikzpicture}
\caption{The `string' bottom-up parser worst case, grammar 3}
\end{figure}
\FloatBarrier

As with the previous grammar the best case of the `string' bottom-up parser is once again slower than the `boolean' version.
The running time follows exactly the same curve as the number of iterations for the `string' bottom-up parser.

It would be possible to plot the area containing all the possible cases of the `string' bottom-up parser once again but it would be the exact same figure as before since the running time of the parser for each pattern remains the same with both grammars.


\section{Tensor Polynomial (First-Order)}
Associated external model: \texttt{tensor\_poly.py}

Tensor polynomial evaluation for quantity of interest $u$ with input parameters $Y=[y_1,\ldots,y_N]$:
\begin{equation}
u(Y) = \prod_{n=1}^N y_n+1.
\end{equation}
This test is specifically useful for its analytic statistical moments.  It is used as a benchmark in
\cite{ayreseaton2015}.

\subsection{Uniform, (-1,1)}
Let all $y_n$ be uniformly distributed between -1 and 1.  The first two statistical moments are:

\subsubsection{mean}
\begin{align}
\expv{u(Y)} &=\int_{-1}^1 dY \rho(Y)u(Y), \notag \\
  &=\int_{-1}^1 dy_1\cdots\int_{-1}^1 dy_N \prod_{n=1}^N \frac{y_n+1}{2},\notag \\
  &=\left[ \int_{-1}^1 dy \frac{y+1}{2}\right]^N,\\
  &=\left[\frac{1}{2}\left(\frac{y^2}{2}+y\right)\bigg|_{-1}^1\right]^N, \notag \\
  &=\left[\frac{2}{2}\right]^N, \notag \\
  &=1. \notag
\end{align}

\subsubsection{variance}
\begin{align}
\expv{u(Y)^2} &= \int_{-1}^1 dY \rho(Y)u(Y),\notag \\
  &=\int_{-1}^1 dy_1\cdots\int_{-1}^1 dy_N \frac{1}{2^N} \left(\prod_{n=1}^N y_n+1\right)^2,\notag \\
  &=\left[\frac{1}{2}\left(\int_{-1}^1 dy\ y^2+2y+1 \right)\right]^N,\\
  &=\left[\frac{1}{2}\left(\frac{y^3}{3}+y^2+y \right)\bigg|_{-1}^1 \right]^N, \notag \\
  &=\left[\frac{1}{3}+1\right]^N, \notag \\
  &=\left(\frac{4}{3}\right)^N.\notag \\
\text{var}[u(Y)] &= \expv{u(Y)^2}-\expv{u(Y)}^2,\notag \\
  &= \left(\frac{4}{3}\right)^N-1.
\end{align}

\subsubsection{numeric values}
Some numeric values for the mean and variance are listed below for several input cardinalities $N$.
\begin{table}[h!]
  \centering
  \begin{tabular}{c|c|c}
    $N$ & mean & variance \\ \hline
    2 & 1.0 & 0.77777777777 \\
    4 & 1.0 & 2.16049382716 \\
    6 & 1.0 & 4.61865569273
  \end{tabular}
\end{table}

\subsection{Uniform, (0,1)}
Let all $y_n$ be uniformly distributed between 0 and 1.  The first two statistical moments are:

\subsubsection{mean}
\begin{align}
\expv{u(Y)} &=\int_{0}^1 dY \rho(Y)u(Y),\\
  &=\int_{0}^1 dy_1\cdots\int_{0}^1 dy_N \prod_{n=1}^N y_n+1,\\
  &=\left[ \int_{0}^1 dy y+1\right]^N,\\
  &=\left[\left(\frac{y^2}{2}+y\right)\bigg|_{0}^1\right]^N,\\
  &=\left[\frac{3}{2}\right]^N.
\end{align}

\subsubsection{variance}
\begin{align}
\expv{u(Y)^2} &= \int_{0}^1 dY \rho(Y)u(Y),\\
  &=\int_{0}^1 dy_1\cdots\int_{0}^1 dy_N \left(\prod_{n=1}^N y_n+1\right)^2,\\
  &=\left[\left(\int_{0}^1 dy\ y^2+2y+1 \right)\right]^N,\\
  &=\left[\left(\frac{y^3}{3}+y^2+y \right)\bigg|_{0}^1 \right]^N,\\
  &=\left(\frac{7}{3}\right)^N.\\
\text{var}[u(Y)] &= \expv{u(Y)^2}-\expv{u(Y)}^2,\\
  &= \left(\frac{7}{3}\right)^N-\left(\frac{3}{2}\right)^{2N}.
\end{align}

\subsubsection{numeric values}
Some numeric values for the mean and variance are listed below for several input cardinalities $N$.
\begin{table}[h!]
  \centering
  \begin{tabular}{c|c|c}
    $N$ & mean & variance \\ \hline
    2 & 2.25      & 0.38194444444 \\
    4 & 5.0625    & 4.01306905864 \\
    6 & 11.390625 & 31.6377499009
  \end{tabular}
\end{table}

\subsection{Multivariate Normal}
Let $Y$ be $N$-dimensional, and have a multivariate normal distribution:
\begin{equation}
Y \thicksim N(\mu,\Sigma)
\end{equation}
with $N$-dimensional mean vector $\mu=[\mu_{y_1},\mu_{y_2},\ldots,\mu_{y_N}]$, and $N X N$ covariance matrix:
\begin{equation}
\Sigma = [Cov[y_i,y_j]],i = 1,2,\ldots,N; j = 1,2,\ldots,N
\end{equation}

To be simplicity, we assume there are no correlations between the input parameters. Then, the covariance matrix can be written
as:
\begin{equation}
\Sigma =
\begin{pmatrix}
\sigma_{y_1}^2 & 0 &\ldots & 0 \\
0 & \sigma_{y_2}^2 &\ldots & 0 \\
\vdots &\vdots &\ddots & \vdots \\
0 & 0 & \ldots & \sigma_{y_N}^2\\
\end{pmatrix}
\end{equation}
where $\sigma_{y_i}^2 = Cov[y_i,y_i]$, for $i = 1,2,\ldots,N$. Based on this assumption, the first two statistical moments are:

\subsubsection{mean}
\begin{align}
\expv{u(Y)} &=\int_{-\infty}^\infty dY \rho(Y)u(Y),\notag \\
  &=\int_{-\infty}^\infty dy_1 (1/\sqrt{2 \pi \sigma_{y_1}}e^{-\frac{(y_1-\mu_{y_1})^2}{2\sigma_{y_1}^2}})\cdots\int_{-\infty}^\infty dy_N (1/\sqrt{2 \pi \sigma_{y_N}}e^{-\frac{(y_N-\mu_{y_N})^2}{2\sigma_{y_N}^2}})\prod_{n=1}^N (y_n+1),\\
  &=\prod_{n=1}^5{(\mu_{y_n}+1)}.\notag
\end{align}

\subsubsection{variance}
\begin{align}
\text{var}[u(Y)]=\expv{(u(Y)-\expv{u(Y)})^2} &= \int_{-\infty}^\infty dY \rho(Y)(u(Y)-\expv{u(Y)})^2,\notag \\
  &=\int_{-\infty}^\infty dy_1 (1/\sqrt{2 \pi \sigma_{y_1}}e^{-\frac{(y_1-\mu_{y_1})^2}{2\sigma_{y_1}^2}}), \notag \\
  &\cdots\int_{-\infty}^\infty dy_N  (1/\sqrt{2 \pi \sigma_{y_N}}e^{-\frac{(y_N-\mu_{y_N})^2}{2\sigma_{y_N}^2}})(\prod_{n=1}^N (y_n+1-\expv{u(Y)})^2,\\
  &=\prod_{n=1}^5{[(1+\mu_{y_n})^2+\sigma_{y_n}^2]} - [\prod_{n=1}^5{(\mu_{y_n}+1)}]^2. \notag
\end{align}
\subsubsection{numeric values}
For example, for given mean $\mu = [0.5, -0.4, 0.3, -0.2, 0.1]$, and covariance
\begin{equation}
\Sigma =
\begin{pmatrix}
0.64 & 0 & 0 & 0 & 0 \\
0 & 0.49 & 0 & 0 & 0 \\
0 & 0 & 0.09 & 0 & 0 \\
0 & 0 & 0 & 0.16 & 0 \\
0 & 0 & 0 & 0 & 0.25 \\
\end{pmatrix}
\end{equation}
The mean and variance can computed using previous equation, and the results are:
\begin{equation}
\expv{u(Y)} = 1.0296
\end{equation}
\begin{equation}
\text{var}[u(Y)] = 4.047085600000002
\end{equation}


\subsection{Cardinality of the rationals}

\subsubsection{Cardinality of rational numbers}

We can see rational numbers as cartesian products of integers. That is:

\(\mathbb{Q}=Z.Z\)

We can order the rational numbers like so:

\(\{\dfrac{1}{1},\dfrac{2}{1},\dfrac{1}{2},\dfrac{1}{3},\dfrac{2}{2}\dfrac{3}{1}...\}\)

These can be mapped from natural numbers, so there is a bijunctive function.

So:

\(|\mathbb{Q} |=|\mathbb{Z}.\mathbb{Z} |=|\mathbb{N} |=\aleph_0\)

As: \(|\mathbb{Z}.\mathbb{Z} |=|\mathbb{Z}|^2\)

\(|\mathbb{N}|^n=\mathbb{N}\)


\section{The complex numbers}

\begin{outcome}
  \begin{enumerate}
  \item Add, subtract, multiply, and divide complex numbers.
  \item Find the conjugate and the magnitude of a complex number.
  \item Apply algebraic properties of the complex numbers to simplify
    equations.
  \end{enumerate}
\end{outcome}

\begin{definition}{The complex numbers}{complex-numbers}
  Let $i$ be an imaginary number such that $i\,^2=-1$. A \textbf{complex
    number}%
  \index{complex number} is a number of the form
  \begin{equation*}
    z = a + bi,
  \end{equation*}
  where $a$ and $b$ are real numbers. The set of all complex numbers
  is denoted $\C$.
\end{definition}

The form $z = a+bi$ is called the \textbf{standard form}%
\index{complex number!standard form}%
\index{standard form!of a complex number} or \textbf{Cartesian form}%
\index{complex number!Cartesian form}%
\index{Cartesian form!of a complex number} of the complex number $z$.
We refer to $a$ as the \textbf{real part}%
\index{complex number!real part}%
\index{real part!of a complex number} and to $b$ as the
\textbf{imaginary part}%
\index{complex number!imaginary part}%
\index{imaginary part!of a complex number} of $z$.

\textbf{Addition}%
\index{complex number!addition}%
\index{addition!of complex numbers}, \textbf{subtraction}%
\index{complex number!subtraction}%
\index{subtraction!of complex numbers}, and \textbf{multiplication}%
\index{complex number!multiplication}%
\index{multiplication!of complex numbers} of complex numbers are
defined in the obvious way, keeping in mind that $i\,^2=-1$. Namely, we
have
\begin{eqnarray*}
  (a+bi) + (c+di) &=& (a+c) + (b+d)i, \\
  (a+bi) - (c+di) &=& (a-c) + (b-d)i, \\
  (a+bi) (c+di)   &=& ac+adi+bci+bdi\,^2 ~=~ (ac-bd) + (ad + bc)i.
\end{eqnarray*}

\begin{example}{Addition, subtraction, and multiplication of complex numbers}{complex-add-subtract-multiply}
  \begin{itemize}
  \item $(3+5i) + (2-3i) = (3+2) + (5-3)i         = 5 + 2i$.
  \item $(3+5i) - (2-3i) = (3-2) + (5+3)i         = 1 + 8i$.
  \item $(3+5i) (2-3i)   = 6 - 9i + 10i - 15i\,^2 = 21 + i$.
  \end{itemize}
\end{example}

Division of complex numbers is more complicated. We first note that it
is easy to divide a complex number by a {\em real} number. Namely,
\begin{equation*}
  \frac{a+bi}{r} = \frac{a}{r} + \frac{b}{r}i.
\end{equation*}
But how can we divide by a complex number? We use the following
trick. Let $z=a+bi$ be a complex number, and consider the product
$(a+bi)(a-bi)$. It is equal to
\begin{equation*}
  (a+bi)(a-bi) = a^2 - b^2i\,^2 = a^2+b^2.
\end{equation*}
Therefore, $(a+bi)(a-bi)$ is always a {\em real} number, and therefore
easy to divide by. Therefore, we can compute the
\textbf{multiplicative inverse}%
\index{multiplicative inverse!of a complex number}%
\index{inverse!of a complex number}%
\index{complex number!inverse} of a complex number $z=a+bi$ as
follows:
\begin{equation*}
  z^{-1}
  = \frac{1}{z}
  = \frac{1}{a+bi}
  = \frac{1}{a+bi}\,\frac{a-bi}{a-bi}=\frac{a-bi}{a^2+b^2}.
\end{equation*}

\begin{example}{Inverse of a complex number}{complex-inverse}
  \begin{equation*}
    \frac{1}{2+5i}
    = \frac{2-5i}{2^2+5^2}
    = \frac{2}{29} - \frac{5}{29}i.
  \end{equation*}
\end{example}

You should verify that this is indeed the inverse, by multiplying
$2+5i$ by $\frac{2}{29} - \frac{5}{29}i$ and checking that the answer
is indeed $1$.

\textbf{Division}%
\index{complex number!division}%
\index{division!of complex numbers} of complex numbers can then be
defined in terms of the multiplicative inverse, i.e.,
$\frac{z}{w} = zw^{-1}$.

\begin{example}{Division of complex numbers}{complex-division}
  \begin{equation*}
    \frac{5+7i}{3-4i}
    = \frac{(5+7i)(3+4i)}{(3-4i)(3+4i)}
    = \frac{-13+41i}{3^2+4^2}
    = -\frac{13}{25} + \frac{41}{25}i.
  \end{equation*}
\end{example}

As a special case of division, note that
\begin{equation*}
  i\,^{-1} = \frac{1}{i} = \frac{1(-i)}{i(-i)} = -i.
\end{equation*}

The complex numbers form a {\em field}, i.e., they satisfy the nine
field axioms. See Section~\ref{sec:fields} for the definition of a
field.

\begin{proposition}{The complex numbers form a field}{complex-field}
  The complex numbers, with the operations of addition, subtraction,
  multiplication, and division, form a {\em field}%
  \index{complex number!field axioms}%
  \index{field!of complex numbers}%
  \index{properties of addition!complex numbers}%
  \index{properties of multiplication!complex numbers}. Specifically,
  this means that they satisfy the following properties:
  \begin{itemize}
  \item[(A1)] {Commutative law of addition:} $z+w=w+z$;
  \item[(A2)] {Associative law of addition:} $(z+w)+u = z+(w+u)$;
  \item[(A3)] {Unit law of addition:} $0+z = z$;
  \item[(A4)] {Additive inverse:} $z+(-z)=0$;
  \item[(M1)] {Commutative law of multiplication:} $zw=wz$;
  \item[(M2)] {Associative law of multiplication:} $(zw)u=z(wu)$;
  \item[(M3)] {Unit law of multiplication:} $1z=z$;
  \item[(M4)] {Multiplicative inverse:} when $z$ is non-zero: $zz^{-1}=1$;
  \item[(D)] {Distributive law:} $z(w+u)=zw+zu$.
  \end{itemize}
\end{proposition}

Another useful operation on complex numbers is the complex
conjugate. Let $z = a+bi$ be a complex number. Then the
\textbf{conjugate}%
\index{complex number!conjugate}%
\index{conjugate!of a complex number}%
\index{complex conjugate} of $z$, written $\conjugate{z}$, is given by
\begin{equation*}
  \conjugate{z} = a-bi.
\end{equation*}
Note that if $z=a+bi$ is a complex number, then
$z\conjugate{z} = (a+bi)(a-bi) = a^2+b^2$. Therefore, $z\conjugate{z}$
is always a real number and $z\conjugate{z}\geq 0$. We define the
\textbf{magnitude}%
\index{complex number!magnitude}%
\index{magnitude!of a complex number} of $z$ to be
\begin{equation*}
  \abs{z} = \sqrt{z\conjugate{z}} = \sqrt{a^2+b^2}.
\end{equation*}
The magnitude is also sometimes called the \textbf{absolute value}%
\index{complex number!absolute value|see{magnitude}}%
\index{absolute value!of a complex number|see{magnitude}} or the
\textbf{modulus}%
\index{complex number!modulus|see{magnitude}}%
\index{modulus!of a complex number|see{magnitude}} of the complex
number.

\begin{example}{Conjugate and magnitude of a complex number}{complex-conjugate}
  \begin{itemize}
  \item $\conjugate{3+5i} = 3-5i$.
  \item $\conjugate{i} = -i$.
  \item $\conjugate{7} = 7$.
  \item $\abs{3+5i} = \sqrt{3^2+5^2} = \sqrt{34}$.
  \item $\abs{i} = 1$.
  \item $\abs{-6} = 6$.
  \end{itemize}
\end{example}

Note that for a real number $a$, we have $\conjugate{a}=a$. Also, when
$a$ is real, the magnitude $|a| = \sqrt{a^2}$ is just the usual
absolute value of real numbers. The following two propositions
list some basic properties of the conjugate and of the magnitude.

\begin{proposition}{Properties of the conjugate}{properties-complex-conjugate}
  Let $z$ and $w$ be complex numbers. Then, the following properties
  of the conjugate hold.%
  \index{complex number!conjugate!properties}%
  \index{conjugate!of a complex number!properties}%
  \index{complex conjugate!properties}%
  \index{properties of complex conjugate}
  \begin{itemize}
  \item $\conjugate{z\pm w} = \conjugate{z} \pm \conjugate{w}$.
  \item $\conjugate{(zw)} = \conjugate{z}~ \conjugate{w}$.
  \item $\conjugate{z^{-1}} = \conjugate{z}^{-1}$.
  \item $\conjugate{z/w} = \conjugate{z} / \conjugate{w}$.
  \item $\conjugate{\conjugate{z}}=z$.
  \item $z$ is real if and only if $\conjugate{z}=z$.
  \end{itemize}
\end{proposition}

\begin{proposition}{Properties of the magnitude}{properties-complex-magnitude}
  Let $z,w$ be complex numbers.  The following properties hold.%
  \index{complex number!magnitude!properties}%
  \index{magnitude!of a complex number!properties}
  \begin{itemize}
  \item $\abs{z} \geq 0$, and $\abs{z}=0$ if and only if $z=0$.
  \item $\abs{zw} = \abs{z}\abs{w}$.
  \item $\abs{\conjugate{z}} = \abs{z}$.
  \item $\abs{z/w} = \abs{z}/\abs{w}$.
  \item \textbf{Triangle inequality}%
    \index{triangle inequality!in complex numbers}:
    $\abs{z+w} \leq \abs{z} + \abs{w}$.
  \end{itemize}
\end{proposition}


\paragraph{Numerical Experiment of Restriction}
We test to different restriction, on is 7-point (based on linear) and the other is 9 point (based on bilinear).

 The kernel of 7-point is
\begin{equation}
\begin{pmatrix}
0 &0.5 &0.5\\
0.5 &1 &0.5\\
0.5 &0.5 &0
\end{pmatrix}
\end{equation}
 The kernel of 9-point is
\begin{equation}
\begin{pmatrix}
0.25 &0.5 &0.25\\
0.5 &1 &0.5\\
0.25 &0.5 &0.25
\end{pmatrix}
\end{equation}

Suppose the prolongation is $P$,
we can write a step iteration of two layer multigrid method as:
\begin{equation}
x^{t+1}-x^*=[I-(D-U)^{-1}A][I-(D-L)^{-1}A][I-P(P^T A P)^{-1}P^T)A](x^t-x^*)
\end{equation}
write $I-BA=[I-(D-U)^{-1}A][I-(D-L)^{-1}A][I-P(P^T A P)^{-1}P^T)A]$

Some numerical experiment are briefly list below:

Our problem is a Possion equation on $n\times n$ conforming triangular grid. Boundary condition is Dirichlet or Neumann.

\begin{table}[!htp]
	\caption{Dirichlet Boundary}
	\begin{tabular}{l |c}
		n&  32\\ \hline
		7-point  & 0.2308	\\
		9-point  & 0.1984	\\
	\end{tabular}
\end{table}
\begin{table}[!htp]
	\caption{Neumann Boundary}
	\begin{tabular}{l |c}
		n&  32\\ \hline
		7-point  & 0.2325	\\
		9-point  & 0.2232	\\
	\end{tabular}
\end{table}

For multilayer, we choose $A_i = P_i^T A P_i$, then
\begin{equation}
x^{t+1}-x^*=[\Pi_{i=1}^{J}(I-T_i)](x^t-x^*)
\end{equation}
where $T_i=\Pi_iR_i\Pi_i^TA$, $R_i$ is the GS smoother of $A_i$.

\begin{table}[!htp]
	\caption{Dirichlet Boundary}
	\begin{tabular}{l |c c}
		Layer& 2&3 \\ \hline
		7-point  & 0.3234 &	0.7433\\
		9-point  & 0.2870 &	0.6256\\
	\end{tabular}
\end{table}

\begin{table}[!htp]
	\caption{Neumann Boundary}
	\begin{tabular}{l |c c}
		Layer& 2&3 \\ \hline
		7-point  & 0.3289 &	0.7167\\
		9-point  & 0.3155 &	0.6283\\
	\end{tabular}
\end{table}


\documentclass{article}
\usepackage{amsmath}
\begin{document}
\title{Advent of Code 2021: Day 7}
\author{Lyle Kopnicky}
\maketitle
\section{Introduction}
The crabs are located at positions $p_i$ for $0 \le i < n$.
The cost of fuel for a crab to move from position $x$ to position $y$ is
\begin{equation}\label{eq:fuel-cost}
  c(x, y) = \sum_{j=0}^{|x-y|} j.
\end{equation}
To minimize the amount of fuel expended by the crabs, we must choose a position $y$
such that the total cost for all crab movements,
\begin{equation}
  t(y) = \sum_{i=0}^{n-1} c(p_i, y),
\end{equation}
is minimized.

By finding the antidifference of \eqref{eq:fuel-cost}, we can see that the
fuel cost is very close to the square of the distance moved:
\begin{equation}
  c(x, y) = \frac{|x - y|(|x - y| + 1)}{2}
\end{equation}

This is very close to the square of the distance.
So, it seems possible that, since the arithmetic mean minimizes
the sum of squared errors, it might minimize the sum
of this fuel cost.

In fact, it does, and we will prove it. However, there's a hitch:
the mean may not be integral. In that case, it turns out that the
position of minimal cost will be one or both of the integers closest
to the mean, on either side.

Our proof consists of two steps:
\begin{enumerate}
  \item Prove that if a position is to the left of the mean, then
  moving further left will increase the total cost.
  \item Prove that if a position is to the right of the mean, then
  moving further right will increase the total cost.
\end{enumerate}
If we can prove these, then we can be sure that there are at most
two points with minimum cost, and if there are two, they are adjacent.
We can also calculate the mean and know that either:
\begin{enumerate}
  \item It falls on an integer: Then the mean is the unique position
        of minimum cost.
  \item It falls between integers: Then one or both of the positions
        to the left or right of the mean has the minimum cost.
\end{enumerate}
So, finding the position with minimum cost would involve calculating
the mean, then testing the adjacent points to see which has a lower
cost.

\section{Left of the mean}

Let's first show that for any position to the left of the mean,
moving further left increases the cost. First, recall the formula
for the mean:
%
\begin{equation}\label{eq:mean}
  m = \frac{1}{n} \sum_{i=0}^{n-1} p_i
\end{equation}

Now choose an arbitrary position $y$ less than or equal to the mean $m$.
We want to show that the total cost is higher at $y-1$ than at $y$.
To do this, we will show that $t(y-1)-t(y) > 0$.

\begin{equation*}
  \begin{split}
    t(y-1)-t(y) &= \sum_{i=0}^{n-1}
    \begin{cases}
      p_i - y, & \quad\text{if}\ p_i < y \\
      p_i - y + 1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
    &= -ny + \sum_{i=0}^{n-1}
    \begin{cases}
      p_i, & \quad\text{if}\ p_i < y \\
      p_i + 1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
    &= -n(y-m+m) + \sum_{i=0}^{n-1}
    \begin{cases}
      p_i, & \quad\text{if}\ p_i < y \\
      p_i + 1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
    &= -n(y-m) -nm + \sum_{i=0}^{n-1}
    \begin{cases}
      p_i, & \quad\text{if}\ p_i < y \\
      p_i + 1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
    &= n(m-y) -nm + \sum_{i=0}^{n-1}
    \begin{cases}
      p_i, & \quad\text{if}\ p_i < y \\
      p_i + 1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
    &= n(m-y) -\sum_{i=0}^{n-1} p_i + \sum_{i=0}^{n-1}
    \begin{cases}
      p_i, & \quad\text{if}\ p_i < y \\
      p_i + 1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
    &= n(m-y) + \sum_{i=0}^{n-1}
    \begin{cases}
      0, & \quad\text{if}\ p_i < y \\
      1, & \quad\text{if}\ p_i >= y \\
    \end{cases} \\
  \end{split}
\end{equation*}

Since $y<=m$, and $n$ must be positive, the term $n(m-y)$ must be
non-negative. The summation term must be positive as long as there
is at least one $i$ such that $p_i >= y$.

To show that there is at least one $p_i >= y$, recall that $y <= m$,
so we only need to show that there is at least one $p_i >= m$.
Then consider the definition of the mean \eqref{eq:mean}. If all
the $p_i$ were less than $m$, then $\sum_{i=0}^{n-1} p_i$ would be
less than $nm$, so $m$ would be less than $m$, a contradiction.
Thus there must be at least one $p_i$ greater than or equal to $m$,
and therefore greater than or equal to $y$.

Since we have at least one $p_i >= y$, the last summation term
in the calculation of $t(y-1) - t(y)$ must be positive, and since
the $n(m-y)$ term is non-negative, we have that $t(y-1) - t(y)$ is
positive. Therefore, the total fuel cost at $t(y-1)$ is greater than
the total fuel cost at $t(y)$, when $y <= m$.

\section{Right of the mean}

Secondly, we must prove that when $y >= m$, the total fuel cost
$t(y+1)$ is greater than the total fuel cost $t(y)$. The steps
are the same as the $y <= m$ case, but with some of the signs flipped:

\begin{equation*}
  \begin{split}
    t(y+1)-t(y) &= \sum_{i=0}^{n-1}
    \begin{cases}
      y - p_i, & \quad\text{if}\ p_i > y \\
      y - p_i + 1, & \quad\text{if}\ p_i <= y \\
    \end{cases} \\
    &= ny + \sum_{i=0}^{n-1}
    \begin{cases}
      -p_i, & \quad\text{if}\ p_i > y \\
      -p_i + 1, & \quad\text{if}\ p_i <= y \\
    \end{cases} \\
    &= n(y-m+m) + \sum_{i=0}^{n-1}
    \begin{cases}
      -p_i, & \quad\text{if}\ p_i > y \\
      -p_i + 1, & \quad\text{if}\ p_i <= y \\
    \end{cases} \\
    &= n(y-m) +nm + \sum_{i=0}^{n-1}
    \begin{cases}
      -p_i, & \quad\text{if}\ p_i > y \\
      -p_i + 1, & \quad\text{if}\ p_i <= y \\
    \end{cases} \\
    &= n(y-m) +\sum_{i=0}^{n-1} p_i + \sum_{i=0}^{n-1}
    \begin{cases}
      -p_i, & \quad\text{if}\ p_i > y \\
      -p_i + 1, & \quad\text{if}\ p_i <= y \\
    \end{cases} \\
    &= n(y-m) + \sum_{i=0}^{n-1}
    \begin{cases}
      0, & \quad\text{if}\ p_i > y \\
      1, & \quad\text{if}\ p_i <= y \\
    \end{cases} \\
  \end{split}
\end{equation*}

Since $y>=m$, and $n$ must be positive, the term $n(y-m)$ must be
non-negative. The summation term must be positive as long as there
is at least one $i$ such that $p_i <= y$.

To show that there is at least one $p_i <= y$, recall that $y >= m$,
so we only need to show that there is at least one $p_i <= m$.
Then consider the definition of the mean \eqref{eq:mean}. If all
the $p_i$ were greater than $m$, then $\sum_{i=0}^{n-1} p_i$ would be
greater than $nm$, so $m$ would be greater than $m$, a contradiction.
Thus there must be at least one $p_i$ less than or equal to $m$,
and therefore less than or equal to $y$.

Since we have at least one $p_i <= y$, the last summation term
in the calculation of $t(y+1) - t(y)$ must be positive, and since
the $n(y-m)$ term is non-negative, we have that $t(y+1) - t(y)$ is
positive. Therefore, the total fuel cost at $t(y+1)$ is greater than
the total fuel cost at $t(y)$, when $y >= m$.

\section{Conclusions}

These two proofs lead us to the following conclusions:
\begin{enumerate}
  \item The position $\lfloor m \rfloor$, the greatest integer less
        than or equal to the mean, must have the minimal cost of all
        positions less than or equal to the mean.
  \item The position $\lceil m \rceil$, the least integer greater than
        or equal to the mean, must have the minimal cost of all positions
        greater than or equal to the mean.
  \item Either position $\lfloor m \rfloor$, $\lceil m \rceil$, or
        both, must have minimal total fuel cost.
  \item If $m$ is integral, then that position has minimal total fuel
        cost.
\end{enumerate}

\end{document}


\section{Linear Support Vector Classifier}

Given $n$ training points, where each input $x_i$ has $m$ attributes, i.e., is of dimensionality $m$, and is in one of two classes $y_i=\pm1$, i.e., our training data is of the form:

\begin{equation}
	\{(x_i,y_i), x_i\in\Re^m, y_i=\pm1, i=1, \dots, n\} \label{eq:svc_data}
\end{equation}

For simplicity we first assume that data are (not fully) linearly separable in the input space $x$, meaning that we can draw a line separating the two classes when $m=2$, a plane for $m=3$ and, more in general, a hyperplane for an arbitrary $m$.

Support vectors are the examples closest to the separating hyperplane and the aim of support vector machines is to orientate this hyperplane in such a way as to be as far as possible from the closest members of both classes, i.e., we need to maximize this margin.

This hyperplane is represented by the equation $w^T x + b=0$. So, we need to find $w$ and $b$ so that our training data can be described by:

\begin{equation} \label{eq:svc_consts}
	\begin{aligned}
		& w^T x_i + b \geq +1 - \xi_i, \forall y_i=+1 \\
    	& w^T x_i + b \leq -1 + \xi_i, \forall y_i=-1 \\
    	& \xi_i \geq 0 \ \forall_i
	\end{aligned}
\end{equation}

where the positive slack variables $\xi_i$ are introduced to allow misclassified points. In this way data points on the incorrect side of the margin boundary will have a penalty that increases with the distance from it.

These two equations can be combined into:

\begin{equation} \label{eq:svc_const}
	\begin{aligned}
    	& y_i (w^T x_i + b) \geq 1 - \xi_i \ \forall_i \\
    	& \xi_i\geq 0 \ \forall_i
    \end{aligned}
\end{equation}

The margin is equal to $\displaystyle \frac{1}{\| w \|}$ and maximizing it subject to the constraint in~\eqref{eq:svc_const} while as we are trying to reduce the number of misclassifications is equivalent to finding:

\begin{equation} \label{eq:svc_obj}
    \begin{aligned}
        \min_{w,b,\xi} \quad & \| w \| + C \sum_{i=1}^n \xi_i \\
            \text{subject to} \quad & y_i (w^T x_i + b) \geq 1 - \xi_i \ \forall_i \\ & \xi_i \geq 0 \ \forall_i
    \end{aligned}
\end{equation}

Minimizing $\| w \|$ is equivalent to minimizing $\displaystyle \frac{1}{2} \| w \|^2$, so we need to find:

\begin{equation} \label{eq:quad_svc_obj}
    \begin{aligned}
        \min_{w,b,\xi} \quad & \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \xi_i \\
            \text{subject to} \quad & y_i (w^T x_i + b) \geq 1 - \xi_i \ \forall_i \\ & \xi_i \geq 0 \ \forall_i
    \end{aligned}
\end{equation}

where the parameter $C$ controls the trade-off between the slack variable penalty and the size of the margin.

\begin{figure}[h!]
	\centering
	\includegraphics[scale=0.6]{img/linear_dual_svc_hyperplane}
	\caption{Linear SVC hyperplane}
	\label{fig:linear_dual_svc_hyperplane}
\end{figure}

\subsection{Hinge loss}

The \emph{hinge} loss is defined as:

\begin{equation} \label{eq:hinge_loss1}
	\mathcal{L}_1 =
	\begin{cases}
		0 & \text{if} \ y (w^T x + b) \geq 1 \\
		1 - y (w^T x + b) & \text{otherwise} \\
	\end{cases}
\end{equation}

or, equivalently:

\begin{equation} \label{eq:hinge_loss2}
	\mathcal{L}_1 = \max(0, 1 - y (w^T x + b))
\end{equation}

and it is a nondifferentiable convex function due to its nonsmoothness in 1, but has a subgradient wrt $w$ that is given by:

\begin{equation} \label{eq:hinge_loss_der}
    \frac{\partial \mathcal{L}_1}{\partial w}=
        \begin{cases}
            -y x & \text{if} \ y (w^T x + b) < 1 \\
            0 & \text{otherwise} \\
        \end{cases}
\end{equation}

\subsubsection{Primal formulation}

The general primal unconstrained formulation takes the form:

\begin{equation} \label{eq:primal_svc}
    \min_{w,b} \mathcal{R}(w,b) + C \sum_{i=1}^n \mathcal{L}(w,b;x_i,y_i)
\end{equation}

where $\mathcal{R}(w, b)$ is the \emph{regularization term} and $\mathcal{L}(w,b;x_i,y_i)$ is the \emph{loss function} associated with the observation $(x_i,y_i)$~\cite{piccialli2018nonlinear}.

The quadratic optimization problem~\eqref{eq:quad_svc_obj} can be equivalently formulated as:

\begin{equation} \label{eq:svc_hinge}
    \min_{w,b} \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \max(0, 1 - y_i (w^T x_i + b))
\end{equation}

where we make use of the \emph{hinge} loss~\eqref{eq:hinge_loss1} or~\eqref{eq:hinge_loss2}.

The above formulation penalizes slacks $\xi$ linearly and is called $\mathcal{L}_1$-SVC.

\begin{figure}[h!]
	\centering
  	\includegraphics[scale=0.4]{img/svc_hinge_loss}
  	\caption{SVC Hinge loss with different optimization steps}
  	\label{fig:svc_hinge_loss}
\end{figure}

To simplify the notation and so also the design of the algorithms, the simplest approach to learn the bias term $b$ is that of including that into the \emph{regularization term}; so we can rewrite~\eqref{eq:svc_hinge} and~\eqref{eq:svc_squared_hinge} as follows:

\begin{equation} \label{eq:primal_svc_hinge1}
    \min_{w,b} \frac{1}{2} (\| w \|^2 + b^2) + C \sum_{i=1}^n \mathcal{L}(w;x_i,y_i)
\end{equation}

or, equivalently, by augmenting the weight vector $w$ with the bias term $b$ and each instance $x_i$ with an additional dimension, i.e., with constant value equal to 1:

\begin{equation} \label{eq:primal_svc_hinge2}
    \begin{aligned}
        \min_{w} \quad & \frac{1}{2} \| \bar{w} \|^2 + C \sum_{i=1}^n \mathcal{L}(w;\bar{x}_i,y_i) \\
            \text{where} \quad & \bar{w}^T = [w^T, b] \\ & \bar{x}_i^T = [x_i^T, 1]
    \end{aligned}
\end{equation}

with the advantages of having convex properties of the objective function useful for convergence analysis and the possibility to directly apply algorithms designed for models without the bias term.

Notice that in terms of numerical optimization the formulations~\eqref{eq:svc_hinge} and~\eqref{eq:svc_squared_hinge} are not equivalent to~\eqref{eq:primal_svc_hinge1} or~\eqref{eq:primal_svc_hinge2} since in the first one the bias term $b$ does not contribute to the \emph{regularization term}, so the SVM formulation is based on an unregularized bias term $b$, as highlighted by the \emph{statistical learning theory}. But, in machine learning sense, numerical experiments in~\cite{hsu2002simple} show that the accuracy does not vary much when the bias term $b$ is embedded into the weight vector $w$.

\subsubsection{Wolfe Dual formulation}

To reformulate the~\eqref{eq:quad_svc_obj} as a \emph{Wolfe dual}, we need to allocate the Lagrange multipliers $\alpha_i\geq 0, \mu_i \geq 0 \ \forall_i$:

\begin{equation} \label{eq:svc_wolfe_dual}
    \max_{\alpha,\mu} \min_{w,b,\xi} \mathcal{W}(w,b,\xi,\alpha,\mu) = \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \xi_i-\sum_{i=1}^n \alpha_i(y_i(w^T x_i + b)-1+\xi_i)-\sum_{i=1}^n\mu_i\xi_i
\end{equation}

We wish to find the $w$, $b$ and $\xi_i$ which minimizes, and the $\alpha$ and $\mu$ which maximizes $\mathcal{W}$, provided $\alpha_i\geq 0, \mu_i \geq 0 \ \forall_i$. We can do this by differentiating $\mathcal{W}$ wrt $w$ and $b$ and setting the derivatives to 0:

\begin{equation} \label{eq:svc_wolfe_der_w}
	\frac{\partial \mathcal{W}}{\partial w}=w-\sum_{i=1}^n \alpha_i y_i x_i \Rightarrow w=\sum_{i=1}^n \alpha_i y_i x_i
\end{equation}

\begin{equation} \label{eq:svc_wolfe_der_b}
	\frac{\partial \mathcal{W}}{\partial b}=-\sum_{i=1}^n \alpha_i y_i\Rightarrow\sum_{i=1}^n \alpha_i y_i=0
\end{equation}

\begin{equation} \label{eq:svc_wolfe_der_xi}
	\frac{\partial \mathcal{W}}{\partial\xi_i}=0\Rightarrow C=\alpha_i+\mu_i
\end{equation}

Substituting~\eqref{eq:svc_wolfe_der_w} and~\eqref{eq:svc_wolfe_der_b} into~\eqref{eq:svc_wolfe_dual} together with $\mu_i\geq 0 \ \forall_i$, which implies that $\alpha\leq C$, gives a new formulation being dependent on $\alpha$. We therefore need to find:

\begin{equation} \label{eq:svc_max_wolfe_dual}
	\begin{aligned}
    	\max_{\alpha} \mathcal{W}(\alpha) &= \sum_{i=1}^n \alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i\alpha_j y_i y_j \langle x_i, x_j \rangle \\
    	&= \sum_{i=1}^n \alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i Q_{ij}\alpha_j \ \text{where} \ Q_{ij} = y_i y_j \langle x_i, x_j \rangle \\
    	&= \sum_{i=1}^n \alpha_i - \frac{1}{2}\alpha^T Q\alpha \ \text{subject to} \ 0\leq\alpha_i\leq C \ \forall_i, \sum_{i=1}^n \alpha_i y_i=0
	\end{aligned}
\end{equation}

or, equivalently:

\begin{equation} \label{eq:svc_min_wolfe_dual}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2}\alpha^T Q\alpha+q^T\alpha \\
            \text{subject to} \quad & 0\leq\alpha_i\leq C \ \forall_i \\ & y^T\alpha=0
    \end{aligned}
\end{equation}

where $q^T = [1, \dots, 1]$.

By solving~\eqref{eq:svc_min_wolfe_dual} we will know $\alpha$ and, from~\eqref{eq:svc_wolfe_der_w}, we will get $w$, so we need to calculate $b.$

We know that any data point satisfying~\eqref{eq:svc_wolfe_der_b} which is a support vector $x_s$ will have the form:

\begin{equation} \label{eq:svc_sv_const1}
	y_s(w^T x_s + b)=1
\end{equation}

and, by substituting in~\eqref{eq:svc_wolfe_der_w}, we get:

\begin{equation} \label{eq:svc_sv_const2}
	y_s\big(\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle +b\big)=1
\end{equation}

where $s$ denotes the set of indices of the support vectors and is determined by finding the indices $i$ where $\alpha_i>0$, i.e., nonzero Lagrange multipliers.

Multiplying through by $y_s$ and then using $y_s^2=1$ from~\eqref{eq:svc_consts}:

\begin{equation} \label{eq:svc_sv_squared_const2}
	y_s^2\big(\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle +b\big)=y_s
\end{equation}

\begin{equation} \label{eq:svc_b}
	b=y_s-\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle
\end{equation}

Instead of using an arbitrary support vector $x_s$, it is better to take an average over all of the support vectors in $S$:

\begin{equation} \label{eq:svc_b_avg}
	b=\frac{1}{N_s}\sum_{s\in S} y_s-\sum_{m\in S}\alpha_m y_m \langle x_m, x_s \rangle
\end{equation}

We now have the variables $w$ and $b$ that define our separating hyperplane's optimal orientation and hence our support vector machine. Each new point $x'$ is classified by evaluating:

\begin{equation} \label{eq:svc_pred}
    y'=\operatorname{sgn}\big(\sum_{i=1}^n\alpha_i y_i\langle x_i, x' \rangle+b\big)
\end{equation}

From~\eqref{eq:svc_min_wolfe_dual} we can notice that the equality constraint $y^T \alpha = 0$ arises form the stationarity condition $\partial_{{b}} \mathcal{W}=0$. So, again, for simplicity, we can again consider the bias term $b$ embedded into the weight vector. We report below the box-constrained dual formulation~\cite{hsu2002simple} that arises from the primal~\eqref{eq:primal_svc_hinge1} or~\eqref{eq:primal_svc_hinge2} where the bias term $b$ is embedded into the weight vector $w$:

\begin{equation} \label{eq:svc_min_bcqp_wolf_dual}
    \begin{aligned}
        \min_{\alpha} \quad & \frac{1}{2} \alpha^T (Q + yy^T)\alpha+q^T\alpha \\
            \text{subject to} \quad & 0\leq\alpha_i\leq C \ \forall_i
    \end{aligned}
\end{equation}

\subsubsection{Lagrangian Dual formulation}

In order to relax the constraints in the \emph{Wolfe dual} formulation~\eqref{eq:svc_min_wolfe_dual} we define the problem as a \emph{Lagrangian dual} relaxation by embedding them into objective function, so we need to allocate the Lagrangian multipliers $\mu \geq 0, \lambda_+ \geq 0$, $\lambda_- \geq 0$:

\begin{equation} \label{eq:svc_lagrangian_dual}
	\begin{aligned}
		    \max_{\mu,\lambda_+,\lambda_-} \min_{\alpha} \mathcal{L}(\alpha,\mu,\lambda_+,\lambda_-) &= \frac{1}{2} \alpha^T Q\alpha+q^T\alpha - \mu^T (y^T \alpha) - \lambda_+^T (u - \alpha) - \lambda_-^T \alpha \\
    &= \frac{1}{2} \alpha^T Q\alpha + (q - \mu y + \lambda_+ - \lambda_-)^T \alpha - \lambda_+^T u
	\end{aligned}
\end{equation}

where the upper bound $u^T = [C, \dots, C]$.

Taking the derivative of the Lagrangian $\mathcal{L}$ wrt $\alpha$ and settings it to 0 gives:

\begin{equation} \label{eq:svc_lagrangian_der_a}
	\frac{\partial \mathcal{L}}{\partial \alpha}=0\Rightarrow Q \alpha + (q - \mu y + \lambda_+ - \lambda_-) = 0
\end{equation}

With $\alpha$ optimal solution of the linear system:

\begin{equation} \label{eq:svc_lagrangian_sol}
    Q \alpha = - (q - \mu y + \lambda_+ - \lambda_-)
\end{equation}

the gradient wrt $\mu$, $\lambda_+$ and $\lambda_-$ are:

\begin{equation} \label{eq:svc_lagrangian_der_mu}
	\frac{\partial \mathcal{L}}{\partial \mu}=-y \alpha
\end{equation}

\begin{equation} \label{eq:svc_lagrangian_der_lp}
	\frac{\partial \mathcal{L}}{\partial \lambda_+}=\alpha - u
\end{equation}

\begin{equation} \label{eq:svc_lagrangian_der_lm}
    \frac{\partial \mathcal{L}}{\partial \lambda_-}=-\alpha
\end{equation}

If the Hessian matrix Q is not positive definite, i.e., the Lagrangian function is not strictly convex since it will be linear along the eigenvectors correspondent to the null eigenvalues and so it will be unbounded below, the Lagrangian dual relaxation will be nondifferentiable, so it will have infinite solutions and for each of them it will have a different subgradient. In order to compute an approximation of the gradient, we will choose $\alpha$ in such a way as the one that minimizes the norm of the residual:

\begin{equation} \label{eq:svc_lagrangian_krylov_sol}
	\begin{aligned}
			\min_{\alpha_n \in K_n(Q, b)} \quad & \| Q \alpha_n - b \| \\
			\text{where} \quad & b = - (q - \mu y + \lambda_+ - \lambda_-)
	\end{aligned}
\end{equation}

Since we are dealing with a symmetric but indefinite linear system we will choose a well-known Krylov method that performs the Lanczos iterate, i.e., symmetric Arnoldi iterate, called \emph{minres}, i.e., symmetric \emph{gmres}, to compute the vector $\alpha_n$ that minimizes the norm of the residual $r_n = Q \alpha_n - b$ among all vectors in $K_n(Q, b) = span(b, Qb, Q^2b, \dots, Q^{n-1}b)$.

\bigskip

From~\eqref{eq:svc_min_wolfe_dual} we can notice that the equality constraint $y^T \alpha = 0$ arises form the stationarity condition $\partial_{{b}} \mathcal{W}=0$. So, again, for simplicity, we can again consider the bias term $b$ embedded into the weight vector. In this way the dimensionality of~\eqref{eq:svc_lagrangian_dual} is reduced of 1/3 by removing the multipliers $\mu$ which was allocated to control the equality constraint $y^T \alpha=0$, so we will end up solving exactly the problem~\eqref{eq:svc_min_bcqp_wolf_dual}.

\begin{equation} \label{eq:svc_bcqp_lagrangian_dual}
	\begin{aligned}
    	\max_{\lambda_+,\lambda_-} \min_{\alpha} \mathcal{L}(\alpha,\lambda_+,\lambda_-) &= \frac{1}{2} \alpha^T (Q + yy^T)\alpha+q^T\alpha - \lambda_+^T (u - \alpha) - \lambda_-^T \alpha \\
    &= \frac{1}{2} \alpha^T (Q + yy^T)\alpha + (q + \lambda_+ - \lambda_-)^T \alpha - \lambda_+^T u
	\end{aligned}
\end{equation}

where, again, the upper bound $u^T = [C, \dots, C]$.

Now, taking the derivative of the Lagrangian $\mathcal{L}$ wrt $\alpha$ and settings it to 0 gives:

\begin{equation} \label{eq:svc_bcqp_lagrangian_der_a}
	\frac{\partial \mathcal{L}}{\partial \alpha}=0\Rightarrow (Q + yy^T) \alpha + (q + \lambda_+ - \lambda_-) = 0
\end{equation}

With $\alpha$ optimal solution of the linear system:

\begin{equation} \label{eq:svc_bcqp_lagrangian_sol}
    (Q + yy^T) \alpha = - (q + \lambda_+ - \lambda_-)
\end{equation}

the gradient wrt $\lambda_+$ and $\lambda_-$ are:

\begin{equation} \label{eq:svc_bcqp_lagrangian_der_lp}
	\frac{\partial \mathcal{L}}{\partial \lambda_+}=\alpha - u
\end{equation}

\begin{equation} \label{eq:svc_bcqp_lagrangian_der_lm}
    \frac{\partial \mathcal{L}}{\partial \lambda_-}=-\alpha
\end{equation}

\subsection{Squared Hinge loss}

The \emph{squared hinge} loss is defined as:

\begin{equation} \label{eq:squared_hinge_loss1}
	\mathcal{L}_2 =
	\begin{cases}
		0 & \text{if} \ y (w^T x + b) \geq 1 \\
		(1 - y (w^T x + b))^2 & \text{otherwise} \\
	\end{cases}
\end{equation}

or, equivalently:

\begin{equation} \label{eq:squared_hinge_loss2}
	\mathcal{L}_2 = \max(0, 1 - y (w^T x + b))^2
\end{equation}

It is a strictly convex function and its gradient wrt $w$ is given by:

\begin{equation} \label{eq:squared_hinge_loss_der}
    \frac{\partial \mathcal{L}_2}{\partial w}=
        \begin{cases}
            - 2 y x & \text{if} \ y (w^T x + b) < 1 \\
            0 & \text{otherwise} \\
        \end{cases}
\end{equation}

\subsubsection{Primal formulation}

Since smoothed versions of objective functions may be preferred for optimization, we can reformulate~\eqref{eq:svc_hinge} as:

\begin{equation} \label{eq:svc_squared_hinge}
    \min_{w,b} \frac{1}{2} \| w \|^2 + C \sum_{i=1}^n \max(0, 1 - y_i (w^T x_i + b))^2
\end{equation}

where we make use of the \emph{squared hinge} loss that quadratically penalized slacks $\xi$ and is called $\mathcal{L}_2$-SVC.

\begin{figure}[h!]
	\centering
  	\includegraphics[scale=0.4]{img/svc_squared_hinge_loss}
  	\caption{SVC Squared Hinge loss with different optimization steps}
  	\label{fig:svc_squared_hinge_loss}
\end{figure}


\lab{Newton and Quasi-Newton Methods}{Newton and Quasi-Newton Methods}

\objective{Newton's method is the basis of several iterative methods for optimization.
Though it converges quickly, it is often very computationally expensive.
Variants on Newton's method, including BFGS, remedy the problem somewhat by numerically approximating Hessian matrices.
In this lab we implement Newton's method, BFGS, and the Gauss-Newton method for nonlinear least squares problems.}

% Newton's method is generally useful because of its fast convergence properties.
% However, Newton's method requires the explicit calculation of the second derivative (i.e., the Hessian matrix) at each step, which is computationally costly.
% Quasi-Newton methods modify Newton's method so that the Hessian does not have to be computed at each step, thus making computations faster.
% This generally comes at the cost of slower convergence speed, but the increased computation speed can make these methods more effective in many cases.

\section*{Newton's Method} % ==================================================

For $g:\mathbb{R}\rightarrow\mathbb{R}$, Newton's method finds a root $\bar{x}$ of the equation $g(x) = 0$ with the following rule.
\begin{equation}
x_{k+1} = x_{k} - \frac{g(x_k)}{g'(x_k)}
\label{eq:quasinewton-newton1d-root}
\end{equation}
Let $f:\mathbb{R}\rightarrow\mathbb{R}$.
Substituting $g = f'$ into \eqref{eq:quasinewton-newton1d-root} yields an iterative method for locating a critical point $x^*$ of $f$ satisfying $f'(x^*) = 0$.
\begin{equation}
x_{k+1} = x_{k} - \frac{f'(x_k)}{f''(x_k)}
\label{eq:quasinewton-newton1d}
\end{equation}

This technique generalizes to higher dimensions.
For $g:\mathbb{R}^n \rightarrow \mathbb{R}^n$, the following iterative technique finds $\bar{\x}$ such that $g(\bar{\x}) = \0$.
\begin{equation}
\x_{k+1} = \x_k - Dg(\x_k)^{-1}g(\x_k)
\label{eq:quasinewton-newtonroot-nd}
\end{equation}
Now let $f:\mathbb{R}^n\rightarrow\mathbb{R}$.
To calculate an optimal value $\x^*$ of $f$ satisfying $Df(\x^*) = \0$, plug $g = Df$ into \eqref{eq:quasinewton-newtonroot-nd} to get the following equation.
\begin{equation}
\x_{k+1} = \x_k - D^2f(\x_k)^{-1}Df(\x_k)\trp
\label{eq:quasinewton-newton-nd}
\end{equation}
Here the first derivative $Df:\mathbb{R}^n\rightarrow\mathbb{R}^n$ evaluates to the row vector $Df(\x) = [D_{1}f(\x)\ \ldots\ D_{n}f(\x)]$,
and the second derivative $D^2f:\mathbb{R}^n\rightarrow\mathbb{R}^{n\times n}$ evaluates to the $n\times n$ \emph{Hessian} matrix
\[
D^2f(\x) =
\left[\begin{array}{ccc}
D_1 D_1 f(\x) & \ldots & D_n D_1 f(\x) \\
D_1 D_2 f(\x) & \ldots & D_n D_2 f(\x) \\
\vdots        &        & \vdots        \\
D_1 D_n f(\x) & \ldots & D_n D_n f(\x)
\end{array}\right]
=
\left[\begin{array}{ccc}
\frac{\partial^2 f}{\partial x_1 \partial x_1} & \cdots & \frac{\partial^2 f}{\partial x_n \partial x_1} \\
\vdots & \ddots & \vdots \\
\frac{\partial^2 f}{\partial x_1 \partial x_n} & \cdots & \frac{\partial^2 f}{\partial x_n \partial x_n}
\end{array}\right].
\]

\begin{problem}
Write a function that accepts functions $Df:\mathbb{R}^n\rightarrow\mathbb{R}^n$ and $D^2f:\mathbb{R}^n\rightarrow\mathbb{R}^{n\times n}$, a starting point $\x_0 \in \mathbb{R}^n$,  a stopping tolerance \li{tol} defaulting to $1e^{-5}$, and an integer \li{maxiter} defaulting to $20$ .
Use Newton's method in \eqref{eq:quasinewton-newton-nd} to optimize $f$.
Return the final estimate $\x_k$, whether or not the method converged (\li{True} or \li{False}), and the number of iterations computed.

Your implementation should include the following items.
\begin{itemize}
    \item Iterate until either $\|Df(\x_{k})\|_\infty <$ \li{tol} or $k >$ \li{maxiter}.
    The criteria $\|\x_{k} - \x_{k-1}\| <$ \li{tol} is also common, but making sure $Df$ is near zero works better in many circumstances.
    \item Instead of inverting $D^2 f(\x_k)$ at each step, solve the equation $D^2 f(\x_k)\z_k = D f(\x_k)\trp$ and compute $\x_{k+1} = \x_k - \z_k$.
    In other words, use \li{la.solve()} instead of \li{la.inv()}.
    \item Avoid recomputing values by only computing $Df(\x_k)$ and $D^2f(\x_k)$ once for each $k$.
\end{itemize}
%
The \emph{Rosenbrock function} is a common test function for optimization methods.
\[
f(x,y) = 100(y - x^2)^2 + (1 - x)^2
\]
The minimizer is $\x^* = (1,1)$ with minimum value $f(1,1) = 0$.
Test your function by minimizing the Rosenbrock function using an initial guess $\x_0 = (-2,2)$.
The function and its derivatives are implemented as \li{rosen()}, \li{rosen_der()}, and \li{rosen_hess()} in \li{scipy.optimize}.
Compare your results to \li{scipy.optimize.fmin_bfgs()}.
\begin{lstlisting}
>>> from scipy import optimize as opt

>>> f = opt.rosen                   # The Rosenbrock function.
>>> df = opt.rosen_der              # The first derivative.
>>> d2f = opt.rosen_hess            # The second derivative (Hessian).
>>> opt.fmin_bfgs(f=f, x0=[-2,2], fprime=df, maxiter=50)
<<Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 35
         Function evaluations: 42
         Gradient evaluations: 42
array([ 1.00000021,  1.00000045])>>
\end{lstlisting}
\label{prob:quasinewton-newton}
\end{problem}

\begin{comment} % NONE OF THIS MAKES SENSE WITH THIS LAB.
    % We aren't talking about f:R->R and what is alpha?
\begin{info}
Remember, Newton's method has quadratic convergence if:
\begin{enumerate}
\item $f: \mathbb{R} \rightarrow \mathbb{R}$ is $C^2$ in an open neighborhood $ I = (x_* - r, x_* + r)$ with $r \geq |x_* - x_0|$
\item $Df(x_k) \neq 0$ for $x_k$ in the iteration
\item $x_0$ is sufficiently close to $\alpha$ that the Taylor series approximation is accurate enough to ignore higher order terms. Additional conditions of closeness can be found in Volume 2 text.
\end{enumerate}
\end{info}
\end{comment}

\section*{BFGS} % =============================================================

Newton's method enjoys quadratic convergence when the initial guess is good enough.
However, computing and inverting the Hessian matrix at each step of \eqref{eq:quasinewton-newton-nd} is often prohibitively expensive.
The idea behind \emph{quasi-Newton methods} is to numerically approximate the inverse of the Hessian at each step.
These methods sacrifice some convergence properties in exchange for becoming less computationally expensive.
They also make it possible to optimize functions where $D^2f$ is unknown.

\emph{Broyden's method} is a high-dimensional generalization of the secant method.
Just as the secant method approximates the second derivative of $f$ in \eqref{eq:quasinewton-newton1d} by using the first derivative at nearby points, Broyden's method uses the first derivative to update an approximated Hessian matrix.
\begin{equation}
\x_{k+1} = \x_k - A_{k}^{-1}Df(\x_k)\trp,
\qquad
A_{k+1} = A_k + \frac{\y_k-A_k\mathbf{s}_k}{\|\mathbf{s}_k\|^2}\mathbf{s}_k\trp,
\label{eq:quasinewton-broyden}
\end{equation}
where $\mathbf{s}_k = \x_{k+1} - \x_{k}$ and $\y_k = Df(\x_{k+1})\trp - Df(\x_{k})\trp$.

\begin{comment} % We don't need this much detail on Broyden.
To get an overview of how this method works, consider the quadratic Taylor series approximation used to construct Newton's method for a function:

\begin{equation}
g_k(\x) = f(\x_k) + Df(\x_k)(\x - \x_k) + \frac{1}{2}(\x - \x_k)\trp A_k(\x - \x_k)
\end{equation}

\noindent By assuming that the Jacobian is the same at $\x$ and $\x_k$, this equation can be rearranged to

\begin{equation} \label{Eq:BroydenRestraint}
Df(\x_{k+1})-Df(\x_k)=(\x_{k+1}-\x_k)\trp A_{k+1}.
\end{equation}

Though there are many possible matrices $A_{k+1}$ that can satisfy this equation, if an estimate of the Hessian $A_k$ at the point $\x_k$ is given, the best rank one approximation of $A_{k+1}$ (meaning that $A_{k+1} -  A_{k}$ is rank one) can be found to minimize $\|A_{k+1}-A_k\|$.
If $\y_k = Df(\x_{k+1})\trp -Df(\x_k)\trp $ and $\mathbf{s}_k=\x_{k+1}-\x_k$,  then $\y_k = A_{k+1}\trp \mathbf{s}_k$.
With this notation, $\|A_{k+1}-A_k\|$ can be shown to be uniquely minimized by the following:

\begin{equation} \label{Eq:BroydenHessian}
A_{k+1} = A_k + \frac{\y_k-A_k\mathbf{s}_k}{\|\mathbf{s}_k\|^2}\mathbf{s}_k\trp .
\end{equation}

This Hessian approximation can then be used in Newton's method as a replacement for the real Hessian as follows:
\begin{equation} \label{eq:quasinewton-broyden}
\x_{k+1} = \x_k - A_{k}^{-1}Df(\x_k)\trp
\end{equation}
\end{comment}

Though this method no longer explicitly calculates the Hessian $D^{2}f(\x_k)$, it still involves a matrix inversion.
The \emph{Sherman-Morrison-Woodbury} formula translates the update rule for $A_k$ in \eqref{eq:quasinewton-broyden} into the following update rule for $A_k^{-1}$.
\[
A_{k+1}^{-1} = A_{k}^{-1} + \frac{\mathbf{s}_k - A_{k}^{-1}\y_k}{\mathbf{s}_k\trp  A_{k}^{-1}\y_k} (\mathbf{s}_k\trp  A_{k}^{-1})
\]

Unfortunately, even if $D^2f(\x_k)$ is positive definite (which is desirable for minimization), the first-order approximation $A_k$ is not guaranteed to be positive definite, so Broyden's method is unreliable.
The \emph{Broyden-Fletcher-Goldfarb-Shanno} (BFGS) method remedies this problem by using the following positive definite second-order approximation for the Hessian.
\[
A_{k+1} = A_k + \frac{\y_k\y_k\trp }{\y_k\trp \mathbf{s}_k}-\frac{A_k\mathbf{s}_k\mathbf{s}_k\trp A_k}{\mathbf{s}_k\trp A_k\mathbf{s}_k}
\]
The Sherman-Morrison-Woodbury formula can also be applied in this situation to yield a computationally efficient form of BFGS.
\begin{align}
\x_{k+1} &= \x_k - A_{k}^{-1}Df(\x_k)\trp
\label{eq:quasinewton-bfgs-iteration}
\\
A_{k+1}^{-1} &= A_k^{-1} + \frac{(\mathbf{s}_k\trp \y_k + \y_k\trp A_k^{-1}\y_k)\mathbf{s}_k\mathbf{s}_k\trp }{(\mathbf{s}_k\trp \y_k)^2} - \frac{A_k^{-1}\y_k\mathbf{s}_k\trp  + \mathbf{s}_k\y_k\trp A_k^{-1}}{\mathbf{s}_k\trp \y_k}
\label{eq:quasinewton-bfgs-hessian}
\end{align}
Here $\mathbf{s}_k = \x_{k+1} - \x_{k}$ and $\y_k = Df(\x_{k+1})\trp - Df(\x_{k})\trp$ as before.

\begin{problem} % Implement BFGS.
Write a function that accepts a function $Df:\mathbb{R}^n\rightarrow\mathbb{R}^n$, a starting point $\x_0 \in \mathbb{R}^n$, a stopping tolerance \li{tol} defaulting to $1e^{-5}$, and an integer \li{maxiter} defaulting to $80$.
Use BFGS as given in \eqref{eq:quasinewton-bfgs-iteration} and \eqref{eq:quasinewton-bfgs-hessian} to optimize $f$, with $A_0^{-1} = I$ (the $n\times n$ identity matrix) as the initial approximation to the inverse of the Hessian.
Return the final estimate $\x_k$, whether or not the method converged, and the number of iterations computed.

This method is a little tricky and can have issues if $\x_0$ is chosen poorly.
Consider the following as you implement your function.
\begin{itemize}
    \item Use the same stopping criteria as in Problem \ref{prob:quasinewton-newton}, iterating until either $\|Df(\x_{k})\|_\infty <$ \li{tol} or $k >$ \li{maxiter}.
    The usual criteria $\|\x_{k} - \x_{k-1}\| <$ \li{tol} is \textbf{not} a good choice for BFGS.
    \item Avoid recomputing values by only calculating each $Df(\x_k)$, $\mathbf{s}_k$, $\y_k$, and $\mathbf{s}_k\trp\y_k$ once.
    \item Note that $\mathbf{s}_k\mathbf{s}_k\trp$, $\y_k\mathbf{s}_k\trp$, and $\mathbf{s}_k\y_k\trp$ are all \emph{outer products} that result in $n\times n$ matrices.
    Use \li{np.outer()} instead of \li{np.dot()} or the \li{@} operator for these computations.
    Carefully identify which parts of \eqref{eq:quasinewton-bfgs-hessian} are scalars and which parts are matrices.
    \item If $(\mathbf{s}_k\trp\y_k)^2 = 0$, terminate the iteration early to avoid dividing by zero.
\end{itemize}
Test your function on the Rosenbrock function as in Problem \ref{prob:quasinewton-newton}.

% % OLD TEST FUNCTIONS.
% Test this method on the following function:
% \[
% f(x,y) = e^{x-1}+e^{1-y}+(x-y)^2
% \]
% using starting point $(2,3)$, maximum iterations $1000$, and tolerance $10^{-2}$.
% Your results can be tested with the following:
% \begin{lstlisting}
% >>> import scipy.optimize as opt
% >>> f = lambda x : np.exp(x[0]-1) + np.exp(1 - x[1]) + (x[0] - x[1])**2
% >>> df = lambda x : np.array([np.exp(x[0]-1) + 2*(x[0]-x[1]), -1*np.exp(1-x[1]) - 2*(x[0]-x[1])])
% >>> minx = opt.fmin_bfgs(f=f,fprime=df,x0=[2,3],gtol=10**-2,maxiter=1000)
% \end{lstlisting}
\label{prob:quasinewton-bfgs}
\end{problem}

\begin{info}
The formula in \eqref{eq:quasinewton-bfgs-hessian} is not the only way to approximate the inverse Hessian.
For example, the \emph{Davidon-Fletcher-Powell} (DFP) method uses the following updating scheme.
\[
A_{k+1}^{-1} = A_k^{-1} +
\frac{\mathbf{s}_k\mathbf{s}_k\trp}{\mathbf{s}_k\trp \y_k}
- \frac{A_k^{-1}\y_k\y_k\trp A_k^{-1}}{\y_k\trp A_k^{-1}\y_k}
\]
This approximation works well for many problems, but BFGS is considered to be the superior method in general.
\end{info}

\begin{problem} % Compare Newton, BFGS, and SciPy.
Write a function that accepts an integer $N$ and performs the following $N$ times.
\begin{enumerate}
\item Sample a random initial guess $\x_0$ from the 2-D uniform distribution over $[-3,3]\times[-3,3]$.
\\(Hint: Use \li{np.random.uniform()} or \li{np.random.random()}.)
\item Time (separately) your implementation of Newton's method from Problem \ref{prob:quasinewton-newton}, your BFGS routine from Problem \ref{prob:quasinewton-bfgs}, and \li{scipy.optimize.bfgs_fmin()} for minimizing the Rosenbrock function with an initial guess of $\x_0$.
\item Record the number of iterations from each method.
For \li{scipy.optimize.fmin_bfgs()}, set \li{disp=False} to suppress printing the convergence message and \li{retall=True} to get the list of $\x_k$ at each iteration (to count the number of iterations).
\end{enumerate}

Plot the computation times versus the number of iterations with a log-log scale, using different colors for each method.
For $N = 100$, your plot should resemble the following figure.
Note that Newton's method consistently converges much faster than BFGS.
In addition, SciPy's BFGS algorithm will likely converge faster than your BFGS implementation because it employs a line search to choose an intelligent step size at each iteration.
% % OLD test functions.
% \begin{align*}
% f(x,y) &= 0.26(x^2+y^2) - 0.48xy\\
% g(x,y) &= \sin(x+y) + (x-y)^2 - 1.5x + 2.5y + 1
% \end{align*}
\begin{figure}[H]
    \includegraphics[width=.7\textwidth]{figures/newton_comparison.pdf}
\end{figure}
\end{problem}

\section*{The Gauss-Newton Method} % ==========================================

\subsection*{Non-linear Least Squares Problems}

Least Squares problems aim to fit a line (or model parameters) to a given set of data points.
These problems arise in many scientific fields, including economics, physics, and statistics and represent unconstrained optimization problems that minimize an objective function of the form
$$
f(\x) = \frac{1}{2}\displaystyle\sum_{j=1}^m r_j^2(\x),
$$
where each $r_i : \mathbb{R}^n \rightarrow \mathbb{R}$ is smooth and $m \geq n$.
This case of least squares problems can be solved with a Newton-like method.

Specifically, with data points $(t_1, y_1), (t_2, y_2), \ldots, (t_m, y_m)$, where $t_i,y_i \in \mathbb{R}$ for $i = 1, \ldots, m$.
Let $\phi(\x, \mathbf{t)}$ be a possible model for this data set, where $\x$ is a vector of parameters of the model, and $\mathbf{t} \in \mathbb{R}^n$.
The error at the $i$-th data point, called the \emph{residual}, is the value $$r_i(\x) := \phi(x_i, t_i) - y_i.$$
Summing the squares of these errors gives the following non-linear least squares objective function.
$$
f(\x) = \frac{1}{2} \displaystyle \sum_{j=1}^m  r_j^2(\x).
$$
The first and second derivatives of this function can then be expressed as
\begin{align*}
Df(\x) &= J(\x)\trp  r(\x), \\
D^2 f(\x) &= J(\x)\trp J(\x) + \displaystyle \sum_{j=1}^m r_j(\x) D^2r_j(\x).
\end{align*}
with $\mathbf{r}(\x) = [r_1(\x), r_2(\x), \ldots, r_m(\x)]\trp $ and
\[
J(\x) =
\left[\begin{array}{c}
\arrayrulecolor{lightgray}
D r_1(\x) \\\hline D r_2(\x) \\\hline \vdots \\\hline D r_m(\x)
\end{array}\right] \in \mathbb{R}^{m\times n}.
\]

The second term in the formula for $D^2 f$ involves second derivatives and can be problematic to compute.
In practice, the second term in the formula for $D^2 f$ is small, either because the residuals themselves are small, or because they are nearly affine in a neighborhood of the solution.
The simplest method for solving the nonlinear least squares problem, known as the \emph{Gauss-Newton Method}, exploits this observation, simply ignoring the second term and making the approximation
$$
D^2 f(\x) \approx J(\x)\trp J(\x).
$$
The method then proceeds in a manner similar to Newton's method.
Thus, at each iteration, we find $\x_{k+1}$ as follows:
\begin{equation} \label{Eq:GaussNewtonUpdate}
\x_{k+1} = \x_k-(J(\x_k)\trp J(\x_k))^{-1}J( \x_k)\trp  \mathbf{r}( \x_k).
\end{equation}

As an example, suppose we have data points generated from the function $y = 3\sin(x/2)$ and slightly perturbed by Gaussian noise.
To fit the data to a model $\phi(\x, t_i) = \phi(x_0, x_1, t_i) = x_0 \sin(x_1 t_i)$, we must select values for $\x = [x_0, x_1]\trp$ (since we know how the data was generated, we expect to find that $x_0 \approx 3$ and $x_1 \approx 1/2$).
Begin by writing functions for the proposed model, the residual vector, and the Jacobian of the residuals.

\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt

# Generate random data for t = 0, 1, ..., 10.
>>> T = np.arange(10)
>>> y = 3*np.sin(0.5*T)+ 0.5*np.random.randn(10)    # Perturbed data.

# Define the model function and the residual (based on the data).
>>> model = lambda x, t: x[0]*np.sin(x[1]*t)        # phi(x,t)
>>> residual = lambda x: model(x, T) - y            # r(x) = phi(x,t) - y

# Define the Jacobian of the residual function, computed by hand.
>>> jac = lambda x: np.column_stack((np.sin(x[1]*t), x[0]*t*np.cos(x[1]*t)))
\end{lstlisting}

By inspecting the data, an initial guess for the parameters could be $x_0 = (2.5, 0.6)$.
A function implementing Gauss Newton can then be used to find the least squares solution.

\begin{lstlisting}
>>> x0 = np.array([2.5,.6])
>>> x, conv, niters = gauss_newton(jac, residual, x0, tol=1e-3, maxiter=10)

# Plot the fitted model with the observed data and the data-generating curve.
>>> dom = np.linspace(0, 10, 200)
>>> plt.plot(T, y, '*')                         # Observed data.
>>> plt.plot(dom, 3*np.sin(.5*dom), '--')       # Data-generating curve.
>>> plt.plot(dom, model(x, dom))                # Fitted model.
>>> plt.show()
\end{lstlisting}

\begin{problem} % Gauss Newton for nonlinear least squares.
Write a function that accepts a function for the proposed model $\phi(\x)$, the model derivative $D\phi(\x)$, a function that returns the residual vector $r(\x)$, a callable function that returns the Jacobian of the residual $Dr(\x) = J(\x)$, a starting point $\x_0$, a stopping tolerance \li{tol} defaulting to $1e^{-5}$, and a max number of iterations \li{maxiter} defaulting to $10$.
This method should implement the Gauss-Newton Method and return a list containing: the minimizing x value, the number of iterations performed, and if the method converged as a boolean.

Test your function by using the Jacobian function, residual function, and starting point given in the example above.
Compare your results to \li{scipy.optimize.leastsq()}.
\begin{lstlisting}
>>> minx = opt.leastsq(func=residual, x0=np.array([2.5,.6]), Dfun=jac)
\end{lstlisting}
\end{problem}

% Non-linear least squares problems can be used to analyze trends in data or to predict future events and are ubiquitous in many academic fields as well as in industrial applications and machine learning.

\begin{problem}
The file \texttt{population.npy} contains census data from the United States every ten years since 1790 for 16 decades.
The first column ($\mathbf{t}$) gives the number of decades since 1790 in the decade ($0,1,\ldots$) and the second column ($\y$) gives the population count in millions of people.

By plotting the data, and with a little knowledge about population growth, it is reasonable to hypothesize an \emph{exponential model} for the population:
\[
\phi(x_1,x_2,x_3,t) = x_1\exp(x_2(t+x_3)).
\]
Use the initial guess $(1.5, .4, 2.5)$ for the parameters $(x_1, x_2, x_3)$ and your Gauss Newton function or \li{scipy.optimize.leastsq()} to fit this model.
Plot the resulting curve along with the actual data points.

Unfortunately, the exponential model isn't a very good fit for the data because the population grows exponentially for only the first 8 or so decades.\footnote{Fitting an exponential model to only the first 8 data points results in a good model for those points (but not for later data).}
Instead, consider the following \emph{logistic model}.
$$
\phi(x_1,x_2,x_3,t) = \frac{x_1}{1+\exp(-x_2(t+x_3))}.
$$
A reasonable initial guess for the parameters $(x_1, x_2, x_3)$ is $(150, .4, -15)$.
Write functions for the model and the corresponding residual vector, then fit the model.
Plot the data against the fitted curve (in the same plot as before).
It should be a much better fit than the exponential curve.
\end{problem}

\begin{comment}

%%% OLD GAUSS NEWTON
The individual functions $r_i$ that measure the error between the model and the data point are known as \emph{residuals},
and we can aggregate these functions into a \emph{residual vector}
$$
\mathbf{r}(\x) := (r_1(\x), r_2(\x), \ldots, r_m(\x))\trp .
$$
The Jacobian of $\mathbf{r}(\x)$ can be expressed in terms of the gradients of each $r_i$ as follows:
%% TODO: put this back in!
$$
J(\x) = \begin{bmatrix} D r_1(\x)\trp  \\ D r_2(\x)\trp  \\ \vdots \\ D r_m(\x)\trp  \end{bmatrix}
$$
You can further verify that
\begin{align*}
D f(\x) &= J(\x)\trp  r(\x), \\
D^2 f(\x) &= J(\x)\trp J(\x) + \displaystyle \sum_{j=1}^m r_j(\x) D^2r_j(\x).
\end{align*}

In particular, at the $k$-th iteration, we choose a search direction $p_k$ that solves the linear system
$$
J_k\trp J_kp_k = -J_k\trp r_k.
$$

For convenience, we summarize these steps in Algorithm \ref{alg:guassnewton}.

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Gauss-Newton}{}
    \State \textrm{Choose initial parameter vector } $x_0$
    \State $k \gets 0$
    \While{$J_k\trp r_k \neq 0$}
        \State \textrm{solve } $J_k\trp J_kp_k = -J_k\trp r_k$
        \State \textrm{choose step size } $\alpha_k$ \textrm{ satisfying Wolfe Conditions.}
        \State $x_{k+1} \gets x_k + \alpha_kp_k$
        \State $k \gets k+1$
    \EndWhile
\EndProcedure
\end{algorithmic}
\caption{Gauss-Newton Method}
\label{alg:guassnewton}
\end{algorithm}

Now we write Python functions for our model, the residual vector, the Jacobian, the objective function,
and the gradient. The calculations for all of these are straight forward.
\begin{lstlisting}
>>> def model(x, t):
>>>     return x[0]*np.sin(x[1]*t)
>>> def residual(x):
>>>     return model(x, t) - y
>>> def jac(x):
>>>     ans = np.empty((10,2))
>>>     ans[:,0] = np.sin(x[1]*t)
>>>     ans[:,1] = x[0]*t*np.cos(x[1]*t)
>>>     return ans
>>> def objective(x):
>>>     return .5*(residual(x)**2).sum()
>>> def grad(x):
>>>     return jac(x).T.dot(residual(x))
\end{lstlisting}
By inspecting our data, we might make an initial guess for the parameters $x_0 = (2.5, 0.6)$.
We are now ready to use our \li{gaussNewton} function to find the least squares solution.
\begin{lstlisting}
>>> x0 = np.array([2.5,.6])
>>> x = gaussNewton(jac, residual, x0, niter=10)
\end{lstlisting}
We can plot everything together to compare our fitted model with the data and the original sine
curve from which the data were generated.
\begin{lstlisting}
dom = np.linspace(0,10,100)
plt.plot(t, y, '*')
plt.plot(dom, 3*np.sin(.5*dom), '--')
plt.plot(dom, x[0]*np.sin(x[1]*dom))
plt.show()
\end{lstlisting}

The module \li{scipy.optimize} also has a method to solve non-linear least squares problem, and it is quite convenient.
The function is called \li{leastsq()}, and in its most basic use, you only need
to pass in the residual function and starting point as arguments.
In the example above, we simply need to execute the following code:
\begin{lstlisting}
>>> from scipy.optimize import leastsq
>>> x2 = leastsq(residual, x0)[0]
\end{lstlisting}
This should give us the same answer, but much faster.

%% TODO: put this back in!
Thus, your new model is
$$
\phi(x_1,x_2,x_3,t) = \frac{x_1}{1+\exp(-x_2(t+x_3))}.
$$
By inspection, find a reasonable
initial guess for the parameters $(x_1, x_2, x_3)$ (i.e. $(150, .4, -15)$).
Again, write Python functions for the model and the corresponding residual vector,
and fit the model. Plot the data against the fitted curve. It should be a good fit.
\end{comment}


%
% Chapter 5.4
%

\section*{5.4 Work}

The \textbf{force} \(F\) on an object (in the same direction) is given by Newton's Second Law of Motion as the product of its mass \(m\) and its acceleration \(a\):
\[ \text{force = mass} \times \text{acceleration} \quad \Leftrightarrow \quad F = ma = m \frac{d^2s}{dt^2} \]
For the case of constant acceleration, the force \(F\) is also constant and the work done is defined to be the product of the force \(F\) and the distance \(d\) that the object moves:
\[ \text{work = force} \times \text{distance} \quad \Leftrightarrow \quad W = Fd \]

\subsubsection*{Work Done Moving an Object from \(A\) to \(B\)}

\[ W = \lim_{n \to \infty} \sum_{i=1}^n f(x_1^*) \Delta x = \int_a^b f(x)dx \]

\subsubsection*{Hooke's Law}

The force required to maintain a spring stretched \(x\) units beyond its natural length is proportional to \(x\):
\[f(x)=kx\]
where \(k\) is a positive constant called the \textbf{spring constant}. Hooke's law holds as long as \(x\) is not too large.