%!TEX root = paper.tex
\subsection{Solitary wave on a composite beach} \label{sec:B_compositebeach}
This is an laboratory experiment conducted at the Coastal Engineering Laboratory of the U.S. Army Corps of Engineers and it is described in \url{http://nctr.pmel.noaa.gov/benchmark/Solitary_wave/}, \url{http://chl.erdc.usace.army.mil/chl.aspx?p=s&a=Projects;36}.
A linear solitary (better called single) wave is propagating over a stepwise increasing bathymetry and it is reflected at a vertical wall on the right boundary. Different wave gauges measure the surface elevation and the runup on the vertical wall.
The experimental data serve for validation of the models. Additionally to the experimental data, a linear analytic solution is provided, s.t. verification of our models is also tested.

Three different cases A, B, C with different target wave heights $a_t$, actual (measured) wave heights $a$ and distances $L$ of gauge $G4$ to the first step in the bathymetry at gauge $G5$. Table \ref{tab:compositebeach_cases} displays the three different cases and belonging data. We only consider case A at the moment. Measured runup data can be found in table \ref{tab:compositebeach_runup}, but are not compared yet to the simulations.

\begin{table}[htbp]
\begin{tabular}{lllll}
\textbf{Case} & \textbf{target $a_t / d$} & \textbf{actual $a / d$} & \textbf{dist. G4 to G5 [m]} & \textbf{dist. G4 to Wall [m]}  \\
\toprule
A       &     0.05   & 0.039    &   2.4   &  10.59     \\
B       &     0.30   & 0.264    &   0.89  &   9.17     \\
C       &     0.70   & 0.696    &    0.64  &   8.83    \\
\bottomrule
\end{tabular}
\caption{Data of three different cases}
\label{tab:compositebeach_cases}
\end{table}

\begin{table}[htbp]
\begin{tabular}{lll}
\textbf{Case} & \textbf{Runup R [cm]} & \textbf{R / d} \\
\toprule
A       &        2.74  &  0.13 \\
B       &       45.72  &  2.10 \\
C       &       27.43  &  1.26 \\
\bottomrule
\end{tabular}
\caption{Runup laboratory results of three different cases}
\label{tab:compositebeach_runup}
\end{table}

The initial condition is prescribed as
\begin{itemize}
 \item a linear analytic solitary wave solution
\begin{align}
\xi(\bx,t)&=a_t \ \text{cosh}^{-2}(K(x-ct-x_0)), \\
u(\bx,t)&=c\frac{\xi(\bx,t)}{d},
\end{align}
with the initial target amplitude $a_t$, propagation velocity $c=\csw$ on a stepwise reduced depth starting from $d=0.218 \, \text{m}$ with scale factor $K=\sqrt{\left(\frac{3a_t}{4d^3}\right)}$ and displacement $x_0=10.59$, s.t. the initial solitary wave has its maximum at gauge G4 while the entire domain length is $L=24 \, \text{m}$. The simulation time is $20$ seconds.
 \item the first option in \url{https://github.com/rjleveque/nthmp-benchmark-problems/blob/master/BP05-ElenaT-Solitary_wave_on_composite_beach_laboratory/BP5_description.pdf}. The velocities are set to zero and the doubled initial surface elevation $2\xi$ is prescribed at $x=0$. A doubled domain length of $L=48 \, \text{m}$ ensures that the waves reflected at the left boundary are not disturbing the solution. The simulation time is $40$ seconds.
\end{itemize}
We impose reflecting boundary conditions at the boundary in x-direction and periodic boundary conditions in y-direction. For the setup see figure \ref{fig:compositebeach_setup}.
Because the analytic solution belongs to the linear SWE, corresponding models are expected to represent very good coincidence with this analytic solution.

\begin{figure}[htbp]
\includegraphics[width=\textwidth]{compositebeach_setup}
\caption{Setup of the testcase solitary wave on a composite beach}
\label{fig:compositebeach_setup}
\end{figure}

\subsubsection{Results of \nh\ model}
The results of the first and the seconds option are shifted by 271.5s to match the initial wave at gauge G4. Both are also scaled with a factor of $\frac{0.037}{0.05}$. % because $0.037$m seems to be the actual amplitude instead of $0.039$.
No difference between both options is visible in the figures then. Hence, we only plot the results of the (cheaper) first option using the linear SWE model and also the following results obtained with the \nh\ model.

Figures \eqref{fig:nh_compositebeach_ana_nh_Lhy} and \eqref{fig:nh_compositebeach_lab_nh_Lhy} display the comparison of the linear SWE with the linear analytic solution and experimental data, respectively.
The models results compared to the analytic solution are very good, while at gauges G9 and G10 a mismatch is visible, because the nonlinear regime is entered here (d $\in [0.047,0.117]$m, $a\approx0.008$m), for which the linear model is not suited.
The comparison to the experimental data reveals less dispersion and reduced amplitudes, although the overall match is also satisfactory.

Figures \eqref{fig:nh_compositebeach_ana_nh_Nnh12} and \eqref{fig:nh_compositebeach_lab_nh_Nnh12} display the comparison of the \nh\ equations using both the linear and the quadratic pressure profile with the linear analytic solution and experimental data, respectively. The \nh\ wave profile represents well the amplitudes, the shape of the propagating wave, the propagation velocities and also the developed dispersive wave train after reflecting of the laboratory surface elevation. The agreement with the linear analytic solution is worse, but still okay.
When wave hits step, the numerical sea surface height is not as much increased as the amplitude of exp data.

Differences between both vertical pressure profiles are small, because dispersive effects become more significant after longer simulation time. However, the tendency of the linear profile to overestimate amplitudes, which was also visible in figure \ref{fig:nh_solitarywave}, is confirmed here, too. Additionally, we see that is also true not only for global maxima, but also for local maxima.

The analytic solution is a good test for the verification of the linear SWE model, whereas the laboratory data show its limits of applicability as well as that the dispersive model gives a more accurate physical representation of the experimental data.

Need to do: wq refl b.c. may be not good enough (not explicit treatment at the moment), edge integrals are missing.

\begin{figure}[htbp]
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G4_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G5_nh_Lhy}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G6_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G7_nh_Lhy}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G8_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G9_nh_Lhy}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G10_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_Wall_nh_Lhy}
\end{minipage}
\caption{Comparison of the analytical (black) sea surface height of the
solitary wave with the simulation results of the \nh\ model in its version of linear shallow water equations (blue)}
\label{fig:nh_compositebeach_ana_nh_Lhy}
\end{figure}

\begin{figure}[htbp]
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G4_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G5_nh_Lhy}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G6_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G7_nh_Lhy}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G8_nh_Lhy}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G9_nh_Lhy}
\end{minipage} \\
\begin{minipage}{0.48\textwidth}
\includegraphics[width=\textwidth]{compositebeach_lab_G10_nh_Lhy}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\begin{tabular}{lll}
\textbf{Data} & \textbf{Runup} & \textbf{R / d} \\
              & \textbf{R [cm]} &  \\
\toprule
Exp.  &  2.74   &  0.13 \\
Model hydr. &  2.13   &  0.0978 \\
\end{tabular}
\end{minipage}
\caption{Comparison of the experimental (black) sea surface height of the
solitary wave with the simulation results of the \nh\ model in its version of linear shallow water equations (blue). The model runup is also scaled with 0.75.}
\label{fig:nh_compositebeach_lab_nh_Lhy}
\end{figure}

\begin{figure}[htbp]
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G4_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G5_nh_Nnh12}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G6_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G7_nh_Nnh12}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G8_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G9_nh_Nnh12}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_G10_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_ana_Wall_nh_Nnh12}
\end{minipage}
\caption{Comparison of the analytical (black) sea surface height of the
solitary wave with the simulation results of the \nh\ model in its \nh\ version using the linear (yellow) and the quadratic vertical pressure profile (red)}
\label{fig:nh_compositebeach_ana_nh_Nnh12}
\end{figure}

\begin{figure}[htbp]
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G4_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G5_nh_Nnh12}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G6_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G7_nh_Nnh12}
\end{minipage} \\
\begin{minipage}{\textwidth}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G8_nh_Nnh12}
\includegraphics[width=0.48\textwidth]{compositebeach_lab_G9_nh_Nnh12}
\end{minipage} \\
\begin{minipage}{0.48\textwidth}
\includegraphics[width=\textwidth]{compositebeach_lab_G10_nh_Nnh12}
\end{minipage}
\begin{minipage}{0.45\textwidth}
\begin{tabular}{lll}
\textbf{Data} & \textbf{Runup} & \textbf{R / d} \\
              & \textbf{R [cm]} &  \\
\toprule
Exp.  &  2.74   &  0.13 \\
Model lin.&  2.18   &  0.10 \\
Model quadr.&  2.12   &  0.097 \\
\end{tabular}
\end{minipage}
\caption{Comparison of the experimental (black) sea surface height of the
solitary wave with the simulation results of the \nh\ model in its \nh\ version using the linear (yellow) and the quadratic vertical pressure profile (red). The model runup is also scaled with 0.75.}
\label{fig:nh_compositebeach_lab_nh_Nnh12}
\end{figure}


\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\begin{document}

\section*{Inline (within text) formulas}

The equation $x + x = 2x$ is inside a text, which uses the \TeX{} shorthand.
In addition, an equation can be written like \(x \cdot x = x^2\) as well, using the \LaTeX{} shorthand.
Finally, the environment \emph{math} can be used like this \begin{math}a^2 + b + c = 0\end{math}.

We can force symbols to be displayed like displayed formula. For example the formula $\sum_{k=0}^{10}k$ can be written $\displaystyle\sum_{k=0}^{10}k$ as well. The sum symbol is taller within a sentence using the command \emph{\textbackslash displaystyle}.

\section*{Displayed equations}

The recommended syntax to render a floating equation is to use the syntax below.

\[a \cdot x = ax \]

The use of the syntax \emph{\$\$$\cdots$\$\$} should be avoided, because, it will modify vertical spacing within equations, rendering them inconsistent.

Finally, the environment \emph{displaymath} produces the same effect.

\begin{displaymath}
	x^2 \cdot x^2 = x^4
\end{displaymath}

\section*{Equation numbering}

The environment \emph{equation} automatically numbers the equations.

\begin{equation}
	f(x)=(x+a)(x+b)
\end{equation}

\end{document}


In the appendix, we define a number of concepts and describe a number of
techniques that are utilized extensively throughout the thesis.

\section{Linear Algebra}
\inputsection{linear-algebra}

\section{Probability Theory}
\xlab{probability-theory}
\inputsection{probability-theory}

\section{Bayesian Statistics}
\xlab{bayesian-statistics}
\inputsection{bayesian-statistics}

\section{Probability Transformation}
\xlab{probability-transformation}
\inputsection{probability-transformation}

\section{Numerical Integration}
\xlab{numerical-integration}
\inputsection{numerical-integration}

\section{Hierarchical Interpolation}
\xlab{hierarchical-interpolation}
\inputsection{hierarchical-interpolation}

\section{Polynomial Chaos}
\xlab{polynomial-chaos}
\inputsection{polynomial-chaos}


\subsubsection{Hyperboloids of One Sheet}
\noindent
A hyperboloid of one sheet has 2 +'s and 1 - in its equation. It is one fully connected surface.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.33\textwidth]{./differentialMultivariableCalculus/one_sheet.png}
	\caption{A hyperboloid of one sheet}
\end{figure}


\chapter{Modal analysis}\label{chap:modalanalysis}
When external forces act on a multi degree-of-freedom system, the system
undergoes forced vibration.
For a system with three coordinates or degrees of freedom, the governing
equation of motion are a set of three coupled ordinary differential equations of
second second order.
The equations of motion of a multi degree-of-freedom system under external force
are given by:
\begin{equation}
  \label{eq:modal}
  [\mathbf{M}] \ddot{\vec{x}} + [\mathbf{K}] \vec{x} = \vec{\mathbf{F}}
\end{equation}
The equation \eqref{eq:modal} represents the system in the version without
damping, where \(\vec{\mathbf{F}}\) is the vector of the external force.
\section{Forced vibration of undamped system}\label{sec:eigen}
To solve the equation \eqref{eq:modal} by modal analysis, it is necessary first
to solve the eigenvalue problem and find the natural frequencies \(\omega_1,
\dots, \omega_n\) as reported below:
\(\omega_{1} = 8.27843\) [\si{\radian\per\second}],
\(\omega_{2} = 27.40935\) [\si{\radian\per\second}],
\(\omega_{3} = 41.87035\) [\si{\radian\per\second}];
and the corresponding normal modes:
\begin{equation}
	\label{eq:eiegnmodefree}
 	[\mathbf{U}] = \begin{bmatrix*}[r]
 		1.00000 & 1.00000 & 1.00000 \\
		0.86540 &-0.47555 &-2.44325 \\
		0.61945 &-1.28034 & 2.15533 \\
 	\end{bmatrix*}
\end{equation}
\section{Forced vibration of viscous damped system}\label{sec:viscoussystem}
Modal analysis, as presented in previous section \ref{sec:eigen}, applies only
to undamped systems.
In many cases, the influence of damping upon the response of a vibratory system
is minor and can be disregarded. However, it must be considered if the response
of the system is required for a relatively long period of time compared to the
natural periods of the system. Further, if the frequency of excitation is at or
near one of the natural frequencies of the system, damping is of primary
importance and must be taken into account. In general, since the effects are not
known in advance, damping must be considered in the vibration analysis of any
system, thus using the equation of motion:
\begin{equation}\label{eq:modaldamping}
  [\mathbf{M}] \ddot{\vec{x}} + [\mathbf{C}] \dot{\vec{x}} +[\mathbf{K}] \vec{x}
   = \vec{\mathbf{F}}
\end{equation}
%
Then we can consider for which the damping matrix can be expressed as a linear
combination of the mass and stiffness matrices:
\begin{equation}
\label{eq:moadlpropdamping}
	[C] = \alpha \cdot [\mathbf{M}] + \beta \cdot [\mathbf{K}]
\end{equation}
where \(\alpha\) and \(\beta\) are constant. By substituting in
\eqref{eq:modaldamping}, we obtain:
\begin{equation}\label{eq:modaldampingcomplete}
	[\mathbf{M}] \ddot{\vec{x}} +
	[\alpha [\mathbf{M}] + \beta[\mathbf{K}]] \dot{\vec{x}} +
	[\mathbf{K}] \vec{x}= \vec{\mathbf{F}}
\end{equation}
By expressing the solution vector \(\vec{x}\) as linear combination of the
natural mode of the undamped system. Substituting \(x(t) = [\mathbf{U}]q(t)\)
in the equation \eqref{eq:modaldampingcomplete} and pre-multiplying by
\([\mathbf{U}]^{\top}\) we obtain:
\begin{equation}\label{eq:modaldampingexpansed}
	[\mathbf{U}]^{\top}[\mathbf{M}][\mathbf{U}] \ddot{\vec{q}} +
	[\alpha[\mathbf{U}]^{\top}[\mathbf{M}][\mathbf{U}] +
   \beta[\mathbf{U}]^{\top}[\mathbf{K}]][\mathbf{U}] \dot{\vec{q}} +
	[\mathbf{U}]^{\top}[\mathbf{K}][\mathbf{U}]^{\top} \vec{q} =
  [\mathbf{U}]^{\top}\vec{\mathbf{F}}
\end{equation}
%
If the eigenvectors \(\vec{\mathbf{U}}^{(j)}\) are normalized we can rewrite in
decoupled form:
\begin{equation}\label{eq:decupled}
  \ddot{q}_{i}(t) + (\alpha + \omega_{i}^2\beta)\dot{q}_{i}(t) +
  \omega_{i}^{2}q(t) = Q_{i}(t)
\end{equation}
where: \(\omega_{1} = 8.27799\) [\si{\radian\per\second}],
\(\omega_{2} = 27.40863\) [\si{\radian\per\second}],
\(\omega_{3} = 41.91576\) [\si{\radian\per\second}] are
the natural frequencies of the undamped system.
The eigenvector are reported in \eqref{eq:eiegnmodeprop}.
\begin{equation}\label{eq:eiegnmodeprop}
	[\mathbf{U}] = \begin{bmatrix*}[r]
		1.00000 & 1.00000 & 1.00000 \\
		0.86499 &-0.48006 &-2.46147 \\
		0.61907 &-1.28528 & 2.16954 \\
 	\end{bmatrix*}
\end{equation}
In the equation \eqref{eq:decupled} substituting the relationship with
\(\alpha + \omega^{2}_{i}\beta = 2\,\zeta_{i}\,\omega_{i}\), where
\(\zeta_{i}\) is called \emph{modal damping ratio} for the \(i_\text{th}\) normal
mode, equation \eqref{eq:decupled} we obtain:
\begin{equation}\label{eq:decupled2}
  \ddot{q}_{i}(t) + 2\zeta_{i}\omega_{i}\dot{q}_{i}(t) +
  \omega_{i}^{2}q(t) = Q_{i}(t)
\end{equation}
It can be seen that each of then equations represented by this expression is
uncoupled from all of the others.
Hence we can find the response of the \(i_\text{th}\) mode in the same manner
as that of a viscously damped single-degree-of-freedom system.
%
The identification of the sources and magnitude of damping is difficult in most
practical problems.
The condition given by equation \eqref{eq:moadlpropdamping} is sufficient but not
necessary for the existence of normal modes in damped systems.
The necessary condition is that the transformation that diagonalizes the damping
matrix also uncouples the coupled equations of motion.
This condition is less restrictive than equation \eqref{eq:moadlpropdamping}
and covers more possibilities.
In the general case of damping, the damping matrix cannot be diagonalized
simultaneously with the mass and stiffness matrices.
A common procedure for finding the solution of the eigenvalue problem of a
damped system involves the transformation of the coupled second-order
equations of motion into \(2n\) uncoupled first-order equation.
%
\input{rayleigh}
\input{matrixiterationmethod}


\documentclass{article}

\usepackage{multicol}
\usepackage{caption}
\usepackage{amsmath}
\usepackage{tikz}
\usepackage{pgfplots}
\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
}

\usepackage{geometry}
\geometry{
	a3paper,
	noheadfoot=true,
	left=1.0in,
	right=1.0in,
	top=1.0in,
	bottom=1.0in
}
\usetikzlibrary{decorations.pathreplacing}
\usepgfplotslibrary{external}

\definecolor{myLightGray}{RGB}{191,191,191}
\definecolor{myGray}{RGB}{160,160,160}
\definecolor{myDarkGray}{RGB}{144,144,144}
\definecolor{myDarkRed}{RGB}{167,114,115}
\definecolor{myRed}{RGB}{255,58,70}
\definecolor{myGreen}{RGB}{0,255,71}

% Titling and Author
\title{Latex Tikz Examples, Linear Equation Plotting, Linear Spline}
\author{\href{https://fanwangecon.github.io/}{Fan Wang}\thanks{https://fanwangecon.github.io, repository: \href{https://fanwangecon.github.io/Tex4Econ/}{Tex4Econ}}}
\date{\today}

\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%
%%% Line Values
%%%%%%%%%%%%%%%%%%%%%
% A. slopes of lines a b c d
\def\ssa{0.8}
\def\ssb{1.0}
\def\ssc{2}
\def\ssd{0.6}
% B. y-intercepts of lines a b c d
\def\iia{+1.0}
\def\iib{+0.0}
\def\iic{-1.0}
\def\iid{-1.5}
% Other parameters
\def\rescale{4}

%%%%%%%%%%%%%%%%%%%%%
%%% Line Formatting
%%%%%%%%%%%%%%%%%%%%%
% A. color definitions
\def\cla{blue}
\def\clb{black}
\def\clc{red}
\def\cld{green}

%%%%%%%%%%%%%%%%%%%%%
%%% Draw Lines
%%%%%%%%%%%%%%%%%%%%%
\begin{multicols}{3}
[
\section{Tikz Four Lines}
Draw four lines, sum them up linear spline, then invert. A3 page has 29.7 cm width. Below, width is 7 cm for each plot, so 21 cm all together generally. Margins are 1 inch on each side, so about 5 cm together.
]

\subsection{Four Relative Allocation Lines}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N_m$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N_{m^{\prime}}$};
    \draw[line width=0.25mm,domain=-2:4,smooth,variable=\x, \cla] plot ({\x},{\iia+\x*\ssa});
    \draw[line width=0.25mm,domain=-2:4,smooth,variable=\x, \clb] plot ({\x},{\iib+\x*\ssb});
    \draw[line width=0.25mm,domain=-2:4,smooth,variable=\x, \clc] plot ({\x},{\iic+\x*\ssc});
    \draw[line width=0.25mm,domain=-2:4,smooth,variable=\x, \cld] plot ({\x},{\iid+\x*\ssd});
\end{tikzpicture}

\subsection{Inequality Constrained $Y > 0$}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N_m$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N_{m^{\prime}}$};
    \draw[line width=0.25mm,domain=(-\iia/\ssa):4,smooth,variable=\x, \cla] plot ({\x},{\iia+\x*\ssa});
    \draw[line width=0.25mm,domain=(-\iib/\ssb):4,smooth,variable=\x, \clb] plot ({\x},{\iib+\x*\ssb});
    \draw[line width=0.25mm,domain=(-\iic/\ssc):4,smooth,variable=\x, \clc] plot ({\x},{\iic+\x*\ssc});
    \draw[line width=0.25mm,domain=(-\iid/\ssd):4,smooth,variable=\x, \cld] plot ({\x},{\iid+\x*\ssd});
\end{tikzpicture}

\subsection{Sum $Y > 0$}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N_m$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N$};
    \draw[line width=0.25mm,domain=(-\iia/\ssa):4,smooth,variable=\x, \cla]
        plot ({\x},{(\iia+\x*\ssa)});
    \draw[line width=0.25mm,domain=(-\iib/\ssb):4,smooth,variable=\x, \clb]
        plot ({\x},{((\iia+\iib)+\x*(\ssa+\ssb))});
    \draw[line width=0.25mm,domain=(-\iic/\ssc):4,smooth,variable=\x, \clc]
        plot ({\x},{((\iia+\iib+\iic)+\x*(\ssa+\ssb+\ssc))});
    \draw[line width=0.25mm,domain=(-\iid/\ssd):4,smooth,variable=\x, \cld]
        plot ({\x},{((\iia+\iib+\iic+\iid)+\x*(\ssa+\ssb+\ssc+\ssd))});
\end{tikzpicture}

\subsection{Sum $Y > 0$ Rescale}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N_m$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N$};
    \draw[line width=0.25mm,domain=(-\iia/\ssa):4,smooth,variable=\x, \cla]
        plot ({\x},{(\iia+\x*\ssa)/\rescale});
    \draw[line width=0.25mm,domain=(-\iib/\ssb):4,smooth,variable=\x, \clb]
        plot ({\x},{((\iia+\iib)+\x*(\ssa+\ssb))/\rescale});
    \draw[line width=0.25mm,domain=(-\iic/\ssc):4,smooth,variable=\x, \clc]
        plot ({\x},{((\iia+\iib+\iic)+\x*(\ssa+\ssb+\ssc))/\rescale});
    \draw[line width=0.25mm,domain=(-\iid/\ssd):4,smooth,variable=\x, \cld]
        plot ({\x},{((\iia+\iib+\iic+\iid)+\x*(\ssa+\ssb+\ssc+\ssd))/\rescale});
\end{tikzpicture}

\subsection{$Y > 0$--Linear Spline}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N_m$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N$};
    \draw[line width=0.25mm,domain=(-\iia/\ssa):(-\iib/\ssb),smooth,variable=\x, \cla]
        plot ({\x},{(\iia+\x*\ssa)/\rescale});
    \draw[line width=0.25mm,domain=(-\iib/\ssb):(-\iic/\ssc),smooth,variable=\x, \clb]
        plot ({\x},{((\iia+\iib)+\x*(\ssa+\ssb))/\rescale});
    \draw[line width=0.25mm,domain=(-\iic/\ssc):(-\iid/\ssd),smooth,variable=\x, \clc]
        plot ({\x},{((\iia+\iib+\iic)+\x*(\ssa+\ssb+\ssc))/\rescale});
    \draw[line width=0.25mm,domain=(-\iid/\ssd):4,smooth,variable=\x, \cld]
        plot ({\x},{((\iia+\iib+\iic+\iid)+\x*(\ssa+\ssb+\ssc+\ssd))/\rescale});
\end{tikzpicture}

% (((\iia+\iib)+((-\iib/\ssb))*(\ssa+\ssb))/\rescale)
\subsection{Summed Lines Invert}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N_m$};
    \clip (-3,-3) rectangle (7,4);
    \draw[line width=0.25mm,
            domain=((\iia+(-\iia/\ssa)*\ssa)/\rescale):4,
            smooth,variable=\x, \cla]
        plot ({\x},{ ((\x*\rescale-\iia)/\ssa)});
    \draw[line width=0.25mm,
            domain=(((\iia+\iib)+(-\iib/\ssb)*(\ssa+\ssb))/\rescale):4,
            smooth,variable=\x, \clb]
        plot ({\x},{ ((\x*\rescale-(\iia+\iib))/(\ssa+\ssb)) });
    \draw[line width=0.25mm,
            domain=(((\iia+\iib+\iic)+(-\iic/\ssc)*(\ssa+\ssb+\ssc))/\rescale):4,
            smooth,variable=\x, \clc]
        plot ({\x},{ ((\x*\rescale-(\iia+\iib+\iic))/(\ssa+\ssb+\ssc)) });
    \draw[line width=0.25mm,
            domain=(((\iia+\iib+\iic+\iid)+(-\iid/\ssd)*(\ssa+\ssb+\ssc+\ssd))/\rescale):4,
            smooth,variable=\x, \cld]
        plot ({\x},{ ((\x*\rescale-(\iia+\iib+\iic+\iid))/(\ssa+\ssb+\ssc+\ssd)) });
\end{tikzpicture}

\subsection{Spline Invert}
\begin{tikzpicture}
    \draw[->] (-3,0) -- (4,0) node[right] {$N$};
    \draw[->] (0,-3) -- (0,4) node[above] {$N_m$};
    \clip (-3,-3) rectangle (7,4);
    \draw[line width=0.25mm,
            domain=((\iia+(-\iia/\ssa)*\ssa)/\rescale):(((\iia+\iib)+(-\iib/\ssb)*(\ssa+\ssb))/\rescale),
            smooth,variable=\x, \cla]
        plot ({\x},{ ((\x*\rescale-\iia)/\ssa)});
    \draw[line width=0.25mm,
            domain=(((\iia+\iib)+(-\iib/\ssb)*(\ssa+\ssb))/\rescale):(((\iia+\iib+\iic)+(-\iic/\ssc)*(\ssa+\ssb+\ssc))/\rescale),
            smooth,variable=\x, \clb]
        plot ({\x},{ ((\x*\rescale-(\iia+\iib))/(\ssa+\ssb)) });
    \draw[line width=0.25mm,
            domain=(((\iia+\iib+\iic)+(-\iic/\ssc)*(\ssa+\ssb+\ssc))/\rescale):(((\iia+\iib+\iic+\iid)+(-\iid/\ssd)*(\ssa+\ssb+\ssc+\ssd))/\rescale),
            smooth,variable=\x, \clc]
        plot ({\x},{ ((\x*\rescale-(\iia+\iib+\iic))/(\ssa+\ssb+\ssc)) });
    \draw[line width=0.25mm,
            domain=(((\iia+\iib+\iic+\iid)+(-\iid/\ssd)*(\ssa+\ssb+\ssc+\ssd))/\rescale):4,
            smooth,variable=\x, \cld]
        plot ({\x},{ ((\x*\rescale-(\iia+\iib+\iic+\iid))/(\ssa+\ssb+\ssc+\ssd)) });
\end{tikzpicture}

\end{multicols}

\end{document}


\documentclass[12pt, a4paper]{article}
\usepackage[margin=1.5in]{geometry}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{float}
\usepackage{listings}
\usepackage[shortlabels]{enumitem}

\setlength\parindent{0pt}
\newcommand{\code}{\lstinline[basicstyle=\small]}
\lstset{
    language=Python,
    basicstyle=\scriptsize
}

\title{EE2703: Applied Programming Lab \\ \Large Assignment 4: Fourier Approximations}
\author{Soham Roy \\ \normalsize EE20B130}
\date{\today}
\begin{document}

\maketitle % Insert the title, author and date

\section{Introduction}
Two functions, $exp(x)$ and $cos(cos(x))$ over the interval $[0,2\pi)$ will be modeled using
the fourier series:
\begin{equation*}
    a_0 + \sum_{n=1}^{\infty} \{a_n \cos(nx) + b_n \sin(nx)\}
\end{equation*}
Where,
\begin{align*}
    a_0 & = \frac{1}{2\pi} \int_{0}^{2\pi} f(x) dx       \\
    a_n & = \frac{1}{\pi} \int_{0}^{2\pi} f(x)cos(nx) dx \\
    a_n & = \frac{1}{\pi} \int_{0}^{2\pi} f(x)sin(nx) dx \\
\end{align*}
\pagebreak

\section{Subquestions}
\subsection{Define \& Plot Python Functions}
Two Python functions are defined:

% \par\rule{\linewidth}{0.4pt}
\begin{lstlisting}
    def exp(x):  # exponential function, supports value or vector
        return np.exp(x)

    def coscos(x):  # cos of cos function, supports value or vector
        return np.cos(np.cos(x))
\end{lstlisting}
% \par\rule[1.5ex]{\linewidth}{0.4pt}

The functions have been plotted with their periodic extensions ($2\pi$ period):
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{1a.png}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{1b.png}
\end{figure}
\pagebreak

\subsection{Evaluate Integrals}
The integrands have been defined as functions of \code{x}, \code{k}, and \code{f}, where
\code{f} is the Python function to be approximated, i.e. \code{exp} or \code{coscos}:
\begin{lstlisting}
    def u(x, k, f):  # f is either exp or coscos
        return f(x) * np.cos(k * x)

    def v(x, k, f):  # f is either exp or coscos
        return f(x) * np.sin(k * x)
\end{lstlisting}
The integrals to calculate the values of $a_n$ and $b_n$ have been evaluated using the following loop:
\begin{lstlisting}
    F = [exp, coscos]
    a_0 = np.zeros((2))
    a_n = np.zeros((2, 25))
    b_n = np.zeros((2, 25))

    for i in range(2):  # iterate over exp and coscos
        a_0[i] = quad(F[i], 0, 2 * np.pi)[0] / (2 * np.pi)

    for j in range(25):  # integration
        a_n[i, j] = quad(u, 0, 2 * np.pi, args=(j + 1, F[i]))[0] / np.pi
        b_n[i, j] = quad(v, 0, 2 * np.pi, args=(j + 1, F[i]))[0] / np.pi
\end{lstlisting}

\subsection{Plot the Fourier Coefficients}
The answer vector $c$ has been generated by:
\begin{lstlisting}
    c_n = np.zeros((2, 51))  # c_n[0] and c_n[1] are the answer vectors
    c_n[:, 0] = a_0
    c_n[:, 1::2] = a_n  # alternate elements starting at 1
    c_n[:, 2::2] = b_n  # alternate elements starting at 2
\end{lstlisting}

To plot in semilog scale, for example, we use:
\begin{lstlisting}
    plt.semilogy(0, np.abs(a_0[f_i]), 'ro', label="by integration")
    plt.semilogy(range(1, 26), np.abs(a_n[f_i]), 'ro')
    plt.semilogy(range(1, 26), np.abs(b_n[f_i]), 'ro')
\end{lstlisting}

The coefficients $|a_n|$ and $|b_n|$ have been plotted:
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.39]{3a.png}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.39]{3b.png}
\end{figure}

\begin{enumerate}[(a)]
    \item The $b_n$ coefficients for $cos(cos(x))$ are of the order of magnitude $-16$, i.e. nearly zero. \\
          This is because $cos(cos(x))$ is an even function, and thus does not have any odd component.
    \item $cos(cos(x))$ is a periodic function, comprising not many frequencies. \\
          $e^x$, on the other hand, is a non-periodic function, and thus its periodic extension has multiple
          discontinuities. Hence, high frequency components are required to represent this function as a sum
          of trigonometric functions.
    \item The $loglog$ plot is linear for $e^x$ because the $log$ of its coefficients vary linearly with
          $log(n)$, i.e. the coefficients depend on $\frac{1}{n^a}$ for some $a$. \\
          On the other hand, the $semilog$ plot is linear for $cos(cos(x))$ because the $log$ of its
          coefficients vary linearly with $n$, i.e. the coefficients decay exponentially.
\end{enumerate}

\subsection{Least Squares}
The equation to be solved by \code{scipy.linalg.lstsq} is:
\begin{equation*}
    Ac = B
\end{equation*}

The matrix $A$ has been generated by:
\begin{lstlisting}
    A = np.ones((400, 51))  # first column should be ones
    for k in range(1, 26):
        A[:, 2 * k - 1] = np.cos(k * x)
        A[:, 2 * k] = np.sin(k * x)
\end{lstlisting}

The solution matrices are \code{c[0]} for $e^x$ and
\code{c[1]} for $coscos(x)$:
\begin{lstlisting}
    for i in range(2):
        b = F[i](x)
        c[i] = lstsq(A, b)[0]
\end{lstlisting}

\subsection{Plot the Best Fit Coefficients}
To plot in semilog scale, for example, we use:
\begin{lstlisting}
    plt.semilogy(0, np.abs(c[f_i][0]), 'go', label="by least squares")
    plt.semilogy(range(1, 26), np.abs(c[f_i][1::2]), 'go')
    plt.semilogy(range(1, 26), np.abs(c[f_i][2::2]), 'go')
\end{lstlisting}

The coefficients $|a_n|$ and $|b_n|$ by least squares method have been contrasted with
those by integration:
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.39]{5a.png}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.39]{5b.png}
\end{figure}

\subsection{Compare Least Squares and Direct Integration}
The maximum errors between the least squares and direct integration methods has been calculated:
\begin{lstlisting}
    np.max(np.abs(c[0] - c_n[0]))  # for e^x
    np.max(np.abs(c[1] - c_n[1]))  # for coscos(x)
\end{lstlisting}

\begin{tabular}{lcl}
    The largest deviation of coefficients for & $e^x$         & is : 1.3327     \\
    The largest deviation of coefficients for & $cos(cos(x))$ & is : 2.6684e-15
\end{tabular}
\medskip

The error is significant for $e^x$ but negligible for $cos(cos(x))$. \\
This is because there are multiple discontinuities in the periodic extension of $e^x$, and thus
would require a much higher number of coefficients to be somewhat accurately represented. \\
This lack in accuracy is more apparent close to those discontinuities.

\subsection{Plot the Fourier Approximations}
The functions as represented by the Fourier coefficients calculated through the least squares method
have been plotted:
\begin{lstlisting}
    plt.plot(x[200:600], np.dot(A, c[1]), 'go', label="by least squares")
\end{lstlisting}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{7a.png}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[scale=0.6]{7b.png}
\end{figure}

The $cos(cos(x))$ plot agrees nearly perfectly, but the $e^x$ plot has a large deviation.
This is because unlike $cos(cos(x))$, which has periodic components, the $e^x$ function
is not inherently comprised of periodic trigonometric functions. The periodic extension of $e^x$
has multiple discontinuities, which causes the Fourier representation to overshoot around them
due to the Gibbs phenomenon.

\end{document}


\input{coredef.tex}

%opening
\def\ntitle{Principles of Quantum Mechanics}
\def\nauthor{mostanes}
\def\npart{II}
\def\nterm{Michelmas}
\def\nyear{2018}
\def\nlecturer{Skinner}

\renewcommand*{\H}{\mathcal{H}}

\begin{document}

\mktitlepage

\newpage

\setcounter{section}{-1}

\section{Preface}
This course draws heavily from material in IB, with references to Linear Algebra, Analysis II, Methods; as well as some references to IB Quantum Mechanics. Note however that this course is likely to prove a lot of the intuitive notions from IB QM, so it should be accessible even to those who were left baffled by IB QM. The author of these notes will make generous references to IB material without restating it, unlike the course lecturer. References to II material will also be made where appropriate.

\newpage

\section{Introduction}
\subsection{Comparison of Classical and Quantum Mechanics}
\subsubsection{Classical Mechanics}
Classical mechanics are governed by Newton's laws, which are 2nd order differential equations in the variables $\vec{x}$, $\vec{p}$. By combining the 2 variables as in classical dynamics, we obtain the \textbf{phase space}, which in Newtonian dynamics is $\R^{2n}$, with the particular case $n=3$ our universe.\\
In classical mechanics, the observables are simple quantities, represented by functions $\R^{2n} \rightarrow \R$.

\subsubsection{Quantum Mechanics}
Particles are instead represented by points in a Hilbert space (equivalent of the phase space).\\
Observables are represented by linear operators $\H \rightarrow \H$.

\subsection{Hilbert spaces}
\begin{remark}
	There is an entire chapter of linear analysis dedicated to Hilbert spaces.
\end{remark}

\begin{definition}
	A Hilbert space $\H$ is a vector space (over $\C$) with a complete inner product $(\cdot,\cdot) : \H \times \H \rightarrow \C$.
\end{definition}
Therefore Hilbert spaces satisfy the usual vector space and complex inner product properties and any Cauchy sequence converges to a vector within the space under the norm induced by the inner product.
\subsubsection{Examples}
Every finite dimensional Hilbert space (of dimension $n$) is isomorphic to $\C^n$ (see Linear Analysis).\\
A simple $\infty$-dimensional space is $l^2$, the space of infinite sequences (converging under the 2-norm; see Analysis II).\\
Another example is $L^2$, the function space of Lebesgue integrable functions (integrals converging under the 2-norm; see Probability and Measure). Their inner product is defined analogously to the inner product of $l^2$ and the norm of $L^2$.
\subsection{Dual spaces}
The dual $\H^\star$ of a Hilbert space $\H$ is space of linear operators $\H \rightarrow \C$. One way of obtaining such operators is by considering the inner product $(\chi,\cdot) : \H \rightarrow \C$.\\
By the Riesz representation theorem (see Linear Analysis), all elements in $\H^\star$ can be written in inner product form.
\subsection{Dirac notation}
Dirac invented the following notation which is standard today in QM:\\
\begin{tabular}{ccc}
	Elements of $\H$ & written $\ket{\cdot}$ & called 'ket'\\
	Elements of $\H^\star$ & written $\bra{\cdot}$ & called 'bra'\\
	Inner product & written $\bra{\cdot}\ket{\cdot}$ & called 'braket'\\
\end{tabular}

\end{document}


%!TEX root =  ../main.tex

\subsection{Anti-Derivative}

\objective{Distinguish and find anti-derivatives and integrals of functions}

Suppose we are given a formula and are told it is the derivative of what we want.
This isn't as abstract as it sounds: velocity is the derivative of position, and (at least in
many cars) it is easier to record velocity than it is position.  If the velocity
function is an algebraic equation, we simply need to apply the Power Rule
in reverse and we will have the anti-derivative of the equation.

The Power Rule states that the derivative of $x^n$ is $n\cdot x^{n-1}$.  In other words,
``take the exponent out front, and lower the exponent by one''.  If we wanted to
turn this backwards, in order to arrive at an exponent of $x^n$, we must have
begun at $x^{n+1}$.  However, if we take the derivative of $x^{n+1}$, we must
multiply by $n+1$.  To cancel that, we should multiply by $\frac{1}{n+1}$.

\begin{derivation}{Backwards Power Rule}\index{Power Rule!Backwards}
The anti-derivative of $x^n$ is $\frac{1}{n+1} x^{n+1} + C$, where $C$ is an unknown
constant.
\end{derivation}

What is C?  Consider whether of not $x^2 +1$ is the anti-derivative of $\frac{1}{2}x$.
Is $x^2-1$?  Is $x^2+\pi$?  Because constants differentiate to 0, a constant could be
part of our anti-derivative equation and we cannot know what it is, without more information.
What information?  Well, if we know the initial conditions (when $x=0$) then we can
solve for $C$ and know precisely which anti-derivative equation we want.

\subsection{Integral}\index{integral!definite}
The preceding definition of anti-derivatives is very helpful algebraically, but what about
graphically?  If a given graph is the derivative of what we seek, can we interpret the
graph to give us numerical information?

Consider the graph of a car with constant velocity:

\begin{figure}[h]
\begin{centering}
\begin{tikzpicture}[scale=0.6]
	\draw[help lines] (0,0) grid (8,6);
	\draw (-0.5,0) -- (8.5,0);
	\draw (0,-0.5) -- (0,6.5);
	\draw[thick,->] (0,4) node[anchor=east] {40 mph}-- (8.5,4);
	\draw (8,0) node[anchor=north] {0.8 hr};
	\draw [fill=gray!80,opacity =0.4] (0,0) rectangle (8,4);
\end{tikzpicture}
\caption{A car's velocity in 10's of mph, over tenths of an hour}
\end{centering}
\end{figure}

If we want position or distance, we have known a formula for a long time:
distance = rate $\cdot$ time.  The $y$-value is the rate.  The $x$-value is the
time.  As hard as it may be to conceive of, distance is the \emph{area} under the
graph.  In this case, 40 mph times 0.8 hours is 32 miles.   Notice that this number does
not depend upon the initial position: the car has travelled 32 positive miles, regardless
of where it began.

\begin{equation}
\int_0^{0.8}  40dx = \left.40x \right|_0^{0.8} = 40(.8) - 40(0) = 32
\end{equation}

What about more complicated velocity?  Let us begin with constant acceleration:

\begin{figure}[h]
\begin{centering}
\begin{tikzpicture}[scale=0.7]
	\draw[help lines] (0,-2) grid (6,6);
	\draw (-0.5,0) -- (6.5,0);
	\draw (0,-2.5) -- (0,6.5);
	\draw [thick,->] (0,-2) -- (6.5,4.5);
	\draw [fill=red!80,opacity=0.4] (0,-2) -- (2,0) -- (0,0);
	\draw [fill=gray!80,opacity=0.4] (2,0) -- (6,0) -- (6,4);
\end{tikzpicture}
\caption{A car beginning at -20 mph but steadily accelerating to 40 mph by 0.6 hours later}
\end{centering}
\end{figure}

The object begins with a negative velocity, so we must count that distance as negative.
-2 + 8 = 6, so the object has gained that many units in positive displacement from 0 to 6.

In some problems, we can simply count the squares below the graph and find the
definite integral.  In most cases, the graph will be curved and we will need to find an
anti-derivative equation and subtract the evaluation at the left from that of the right.

Finally, notice that we can integrate functions we cannot differentiate, at times.  It would
be impossible for a physical object to have a velocity graph like the \texttt{int()} function,
but it can still be meaningful to find the area under the graph.

\begin{example}{Bandwidth Rates}
\exProblem
Suppose an wifi hotspot charges start at 2.50 per hour when you begin, and the rate goes
up .50 every 20 minutes after that.  The rate is not incremented
continuously, but jumps every 1/5 hour.  Illustrate the cost of using the service for 0.9 hours
as a definite integral.

\exSolution
\begin{tikzpicture}[xscale=4,yscale=1]
	\draw (-0.05,0) -- (1,0) ;
	\draw (0,-.2) -- (0,5.0);
	\draw[thick] (0,2.50) -- (.2,2.5) -- (.2,3) -- (.4,3) -- (.4,3.5) -- (.6,3.5) -- (.6,4) -- (.8,4) -- (.8,4.5) -- (1,4.5);
	\draw [fill=gray!80,opacity=0.4] (0,0) rectangle (.2,2.5);
	\draw [fill=gray!80,opacity=0.4] (.2,0) rectangle (.4,3);
	\draw [fill=gray!80,opacity=0.4] (.4,0) rectangle (.6,3.5);
	\draw [fill=gray!80,opacity=0.4] (.6,0) rectangle (.8,4);
	\draw [fill=gray!80,opacity=0.4] (.8,0) rectangle (.9,4.5);
\end{tikzpicture}
As the rectangle illustrate, $(0.2)(2.5) + (0.2)(3.0) + (0.2)(3.5) + (0.2)(4.0) + (0.1)(4.5) = 3.05$.
\end{example}


\subsubsection{Complex Eigenvalues}
\begin{theorem}
	If an $n \times n$ matrix $A$ is not defective, then for each pair of complex eigenvalues $\alpha \pm \beta i$ with corresponding eigenvectors $\vec{a} \pm i\vec{b}$, the corresponding fundamental solutions are $e^{\alpha t}\left(\cos{(\beta t)}\vec{a}-\sin{(\beta t)\vec{b}}\right)$ and $e^{\alpha t}\left(\sin{(\beta t)}\vec{a}+\cos{(\beta t)\vec{b}}\right)$.
\end{theorem}

\ifodd\includeLinearSystemsExamples\input{./linearSystems/homogeneousSystems/complexEigenvalues_example.tex}\fi


\documentclass[../../main.tex]{subfiles}
\begin{document}
\subsubsection{At Key}
The operation $atKey$ will return the Value $v$ at some specified Key $k$.
\begin{schema}{AtKey[KV, K]}
  m? : KV \\
  v! : V \\
  k? : K \\
  atKey~\_ : KV \cross K \surj V
  \where
  v! = atKey(m?, k?) @ \\
  \t2 let ~~ coll == ((\seq m?) \filter (k?, m?_{k?})) \implies \langle (k?, m?_{k?}) \rangle \iff k? \in \dom m? \\
  \ \ \ = (second(head(coll)) \iff k? \mapsto m?_{k?} \in coll) ~\lor \\
  \t1 (\emptyset \iff k? \not \in \dom m?)
\end{schema}
In the schema above, $coll$ is the result of filtering for $(k?, m?_{k?})$ within $\seq m?$.
If the mapping was in the original $m?$, it will also be in the sequence of mappings. This means
we can filter over the sequence to look for the mapping and if found, it is returned as $\langle (k?, m?_{k?}) \rangle$.
To return the mapping itself, $head(coll)$ is used to extract the mapping such that the value mapped to $k?$ can be returned.
\begin{zed}
  v! = atKey(m?, k?) = second(head(coll)) = m?_{k?} @ m?_{k?} : V \iff k? \in \dom m?
\end{zed}
The following examples demonstrate the properties of $atKey$
\begin{argue}
  M = \ldata k_{0}v_{k_{0}}, k_{1}v_{k_{1}} \rdata \\
  \t1 k_{0} = abc \ \land  v_{k_{0}} = 123 & $k_{0}v_{k_{0}} = abc \mapsto 123$ \\
  \t1 k_{1} = def \ \land v_{k_{1}} = xyz \mapsto 456 & $k_{1}v_{k_{1}} = def \mapsto xyz \mapsto 456$ \\
  atKey(M, abc) = 123 \\
  atKey(M, def) = xyz \mapsto 456 \\
  atKey(M, foo) = \emptyset
\end{argue}
\end{document}


\chapter{The local field potential}
\label{ch:LFP}

This thesis discusses software for processing intracranial voltage recordings. This chapter describes the nature of such voltage signals.

At every location in the brain, the electric field vector $\E$ (in units of N/C or V/m) points in the direction that a positive charge would be pushed towards, if it was placed at that location. The electric potential $\phi$ is defined\footnotemark{} such that:
%
\begin{equation}
\label{eq:potential}
\E = -\grad{\phi},
\end{equation}
%
i.e. such that the electric field points from locations of high potential to locations of lower potential. The electric field potential $\phi$ is a scalar field, in units of V. When it is measured extracellularly, neuroscientists refer to it as the \emph{local field potential} (LFP) -- especially when only frequencies below about 500 Hz are considered.

\footnotetext{The electric field potential $\phi$ is only defined when the electric and magnetic fields do not vary too quickly over time, i.e. when $\pdv*{\E}{t} \approx 0$ and $\pdv*{\B}{t} \approx 0$. It then directly follows from Maxwell's equations that both fields become decoupled, and that the electric field becomes irrotational ($\curl{\E} = 0$), so that an electric potential may be defined as in \cref{eq:potential}. This so called \emph{quasi-static} assumption is met in electrophysiological conditions \cite{Nunez2006,Plonsey2007}.}

We now discuss two models used to estimate the LFP $\phi$. Both models calculate $\phi$ based on the mobile charge density $\rho$ at each location in the brain and throughout time. ($\rho$ is the net positive surplus of mobile charges, in units of C/\si{\metre^3}. Biological tissue contains a high number of such mobile charges. These are mostly ions, with \ce{Na+}, \ce{Cl-}, and \ce{K+} the most abundant ones \cite{Martinsen2015a}).

Both models consist of a single divergence equation (namely \cref{eq:div-charge,eq:div-current}), which, together with \cref{eq:potential} and an assumption of uniform and isotropic brain tissue, leads in both cases to a simple closed form equation to calculate the LFP $\phi$ (namely \cref{eq:phi-charge,eq:phi-current}).

\section{Model 1 (Gauss's law)}

The first model is a direct translation of the first of Maxwell's equations (Gauss's law), at macroscopic scale:\footnotemark{}
%
\begin{equation}
\label{eq:div-charge}
\div{\eps \E} = \rho,
\end{equation}
%
where $\eps$ is the local tissue permittivity, in units of F/m and generally a 3 $\cross$ 3 matrix.

\footnotetext{Gauss's law in its pure form ($\div{\E} = \rho_\total / \eps_0$) considers both mobile and ``bound'' charges: $\rho_\total = \rho_\free + \rho_\bound$. In \cref{eq:div-charge}, we only explicitly consider mobile charges $\rho = \rho_\free$, while bound charges are subsumed in the tissue permittivity $\eps$. This simplification is allowed if we assume that ``tissue polarisation is proportional to electric field strength'' \cite{Feynman2013}. This is a common assumption in physics, and is largely valid for brain tissue in normal conditions \cite{Nunez2006}.}

When we assume the permittivity $\eps$ to be uniform and isotropic throughout the tissue (i.e. $\eps$ is a constant scalar)\footnotemark{}, \cref{eq:div-charge} has the following solution for the LFP $\phi$ \cite{Feynman2013}:
%
\begin{equation}
\label{eq:phi-charge}
\phi = \frac{1}{4 \pi \eps} \int_V \frac{\rho \dd{V}}{r},
\end{equation}
%
where we summate over the entire brain volume $V$, with $\rho \dd{V}$ the surplus mobile charge in a small volume $\dd{V}$ of tissue (in Coulomb), and $r$ the distance of this small volume to the point where $\phi$ is calculated.

\footnotetext{The uniform and isotropic permitivitty assumption is questionable. Roughly, brain tissue consists of a dense alternation of two types of tissue (see \cref{fig:neuropil}): the seawater-like fluid inside and in between cells, and the fatty membranes around cells, organelles, and vesicles. The former has a permittivity $\eps$ of about 15 times larger than the latter \cite{Marszalek1991,Weaver2003,Martinsen2015}. Additionally, the strongly non-random organization of some brain regions may challenge the isotropy assumption. Nevertheless, this uniform and isotropic permittivity assumption is often made \cite{Nunez2006,Plonsey2007}.}

In other words, positive charge surplusses increase the nearby LFP, negative charge surplusses decrease it, and their effects summate linearly, weighted inverse-proportionally by distance. This model explains for example the resting ``membrane potential'' of neurons, where $\phi$ is lower inside the cell than outside: neurons at rest contain an excess of negative charges \cite{Dayan2001b}.

Although this model is arguably the most physically correct, it is often difficult to apply in practice: charge distributions $\rho$ are already quite complex in even the simplest of electronic circuits \cite[chapter 8]{Chabay2015}. For the highly entangled topologies of brain tissue (see \cref{fig:neuropil}), estimating charge distributions is intractable.

The next model is more useful in practice, as it calculates the LFP based not on charges, but on currents. Currents (and specifically transmembrane currents) are the bread and butter of electrophysiology \cite{Kandel2013a}.

\section{Model 2 (Current source density analysis)}
\label{sec:CSD}

This is the ``standard model of electric potentials in biological tissue'' \cite{Bedard2011}. Unlike the previous model, it is mostly empirical \cite{Bedard2011}. In fact, the assumptions on which it rests are strictly incompatible with \cref{eq:div-charge,eq:phi-charge}, and lead to a paradox when considered jointly. \Citeauthor{Bedard2011} explore this paradox \cite{Bedard2011}, and propose a more general formulation of CSD analysis that `solves' the paradox. In the remainder of this chapter however, we will present the `classic' CSD model, as it is presented in most electrophysiology texts \cite{Plonsey2007,Nunez2006,Mitzdorf1985,Nicholson1975,Linden2014,Buzsaki2012a}.

We first define the so called \emph{current source density}, $I$, as the rate at which the net charge surplus inside a small volume decreases (units A/\si{m^3}):
%
\begin{equation}
\label{eq:CSD}
I = - \pdv{\rho}{t}
\end{equation}
%
Charge conservation (which is a direct consequence of Maxwell's equations) dictates that
%
\begin{equation}
\label{eq:charge-conservation}
\div{\J} = I,
\end{equation}
%
where $\J$ is the current density (units of A/\si{\metre^2}, and $\J = \rho \vb{v}$ with $\vb{v}$ the local velocity of mobile charges). In other words, the decrease in net positive charges in a small volume ($I$) equals the net rate at which positive charges flow out of this volume ($\div{\J}$).

Next (and this is the assumption challenged in \cite{Bedard2011}), we assume that the extracellular medium is \emph{resistive} or \emph{ohmic}:
%
\begin{equation}
\label{eq:ohmic-medium}
\J = \sigma \E,
\end{equation}
%
where $\sigma$ is the local tissue conductivity, in units of S/m and generally a 3 $\cross$ 3 matrix. In other words, current is assumed to be linearly related to the electric field. (This needn't be the case in reality: the electric field dictates the acceleration of mobile charges, while the current density describes their velocity. These are instantaneously indepedent.)

Combining charge conservation and the ohmic assumption leads to the governing equation for CSD analysis:
%
\begin{equation}
\label{eq:div-current}
\div{\sigma \E} = I.
\end{equation}
%
Note the similarity with \cref{eq:div-charge}. And analogously as in \cref{eq:div-charge,eq:phi-charge}, when we assume the conductivity $\sigma$ to be uniform and isotropic throughout the tissue (i.e. $\sigma$ is a constant scalar)\footnotemark{}, \cref{eq:div-current} has the following solution for the LFP $\phi$ \cite{Plonsey2007}:
%
\begin{equation}
\label{eq:phi-current}
\phi = \frac{1}{4 \pi \sigma} \int_V \frac{I \dd{V}}{r}.
\end{equation}
%
In other words, so called ``current sources'' ($I > 0$) increase the nearby LFP, current sinks ($I < 0$) decrease it, and their effects again summate linearly, weighted inverse-proportionally by distance.

\footnotetext{Again, a questionable assumption: (extra)cellular plasma has a conductivity $\sigma$ of about $10^6$ times higher than the lipid bilayer membranes in between \cite{Michel2017,Martinsen2015,Marszalek1991,Nunez2006,Weaver2003}.}

\section{Transmembrane currents}
\label{sec:transmembrane-currents}

Active transmembrane currents are the main physical mechanism of signal propagation in the nervous system.\footnotemark{} At synapses, excitatory neurons cause a brief inward current in their target neuron, while inhibitory neurons cause a brief outward current. Along axons, action potentials (``spikes'') propagate by a cascade of inward currents.

\footnotetext{With 'active', we mean here that the current is caused by the opening of ion channels that are located at the same position in the membrane as the current.}

Each of these active currents leads to corresponding passive currents in other parts of the membrane, forming closed current loops. An excitatory, inward current at a synapse corresponds to outward currents at distal parts of the neuron (most notably at the cell body). The inward, active currents of a spike are surrounded by outward, passive currents.

\section{Forward CSD model of the LFP}
\label{sec:forward-CSD}

In CSD analysis, both active and passive transmembrane currents are used to estimate the LFP, by making use of the following `trick': the existence of the inside of the neuron is ignored. An inward current for example is then modelled as a current sink $I_m < 0$ in the extracellular space, i.e. an accumulation of mobile charges. In reality, charges do not quite accumulate, but rather dissipate in the neuron. Or, equivalently, a proportional current source $I'_m = -I_m$ exists inside the neuron, at virtually the same location as $I_m$, cancelling it out. In CSD analysis however, only the current sink $I_m$ (and not the current source $I'_m$) would be used for calculating the LFP $\phi$.

The justification for ignoring the inside of the neuron is often as follows (paraphrasing from \cite{Mitzdorf1985}, a foundational CSD theory paper): ``The extracellular space is independent of the intracellular space, because its boundaries (the cell membranes) have a very high resistance compared to the extracellular space.'' However, electric insulators (high resistance areas) do not shield the electric field, nor the electric field potential $\phi$.
%
\footnote{Another reason the inside of neurons is ignored might possibly be notation: at synapses, there are real currents, often denoted by $I$; but there are no real current sources or sinks, also often denoted by $I$. This may cause confusion.}
\footnote{A final reason to not only consider the extracellular space, is that there is not that much of it: only between 10\% and 20\% of hippocampus volume is extracellular space \cite{Sykova1997a}. See also \cref{fig:neuropil}.}

Nonetheless, in the so called `forward' CSD model (estimating the LFP from transmembrane currents), \cref{eq:phi-current} is used to calculate the LFP based on `unpaired' transmembrane current source densities $I_m$. From \cref{eq:phi-current}, an inward current then corresponds to a drop in the nearby LFP, and an outward current to an increase. This technique is applied in \cref{sec:physiology} to explain the physical basis of the sharp wave-ripple motif. Another application of this technique is e.g. \texttt{LFPy}, a software package used to numerically simulate the LFP \cite{Linden2014}.

Notwithstanding the theoretical problems outlined here and in \cite{Bedard2011}, CSD analysis seems to provide relatively accurate results in practice \cite{Buzsaki2012a,Bedard2011}. As mentioned earlier, \citeauthor{Bedard2011} \cite{Bedard2011} give a derivation of a CSD-like model that does not rest upon the `classic' derivation as presented here, instead concluding: ``[..] the results obtained with the classic CSD analysis are perfectly consistent with \emph{ionic diffusion} because diffusion gives a source term which is very close to the phenomenological model of current source density introduced by Pitts and Mitzdorf, but in a manner consistent with Maxwell-Gauss law.'' (Pitts \cite{Pitts1952} and Mitzdorf \cite{Mitzdorf1985} refers to the classical CSD model).


\subsection{Course Structure}

\begin{table}[h]
% \caption{Tested autoencoder structures for a 3D case}

\begin{tabular}{p{25mm}|p{100mm}|p{30mm}}
  \hline
Lecture 1-2	&
Introduction. \newline
Linear Algebra Four Fundamental Subspaces
(Null space, Row space, Projectors, Column and Left null spaces). \newline
Use cases for fundamental subspaces in Control. &
Individual \newline assignment
\\   \hline
%
Lecture 3-5 &
Least Squares and Quadratic Programming. \newline
Domains. Convex Domains. \newline
Linear inequality representation of convex domains.	&
Individual \newline assignment
\\   \hline
%
Lecture 6-9	&
Second order cone programming (SOCP). \newline
Semidefinite programming (SDP). \newline
Linear matrix inequalities, example in Finite-time stability, example in computational geometry. \newline
Mixed-integer convex programming. \newline Relaxation. Big M method. Example in path planning.	&
Individual \newline assignment
\\   \hline
%
Lecture 10-11	&
Barrier functions, center of constraints. \newline
Minimax.	&
Miniproject
\\  \hline
\end{tabular}
\end{table}


\chapter{From MDPs to Reinforcement Learning}
\label{chapter3}

\vspace{0.5cm}

\noindent As said before, when we have both the reward function $\mathcal{R}$ and the transition model $\mathcal{P}$ we can solve the MDP by means of dynamic programming.

Truth to be told, for most cases we cannot precisely predict neither $\mathcal{R}$ nor $\mathcal{P}$ and we have to approximate, explore and estimate functions and states in order to converge to the optimal solution (the optimal policy).

Reinforcement Learning in fact, is a technique to handle Markov Decision Processes in an environment we don't fully know.

\section{Value Function and the Bellman equation}
We spoke before about \textit{optimal policy} but we never defined it. The optimal policy, is the one that maximizes our long term expected reward, that is:
\begin{equation}
    \pi^\star = arg\,max_\pi \ \mathbb{E} \Big[ \sum_t \gamma^t \mathcal{R}(s_t) \ \vert \ \pi \Big]
\end{equation}
\noindent
We can see in this function, that the initial state is not defined (it's random), and that the optimal policy must be optimal starting from any state.

We can actually define also the \textbf{value function} that gives the expected long term reward following a policy starting from a state $s$:
\begin{equation}
    \mathbb{V}^\pi(s) = \mathbb{E} \Big[ \sum_t \gamma^t \mathcal{R}(s_t) \ \vert \ \pi, \ s_0 = s \Big]
\end{equation}
This function helps us to define in a better way the optimal policy:
\begin{equation} \label{eq:bellman-value}
    \pi^\star = arg\,max_a \ \sum_{s'} \mathbb{P}(s_{t+1} = \mathnormal{s'} \vert s_t = \mathnormal{s}, a_t = \mathnormal{a}) \ \mathbb{V}(s')
\end{equation}
This equation says that the optimal policy is the one that, for every state, returns the action that maximizes my expected value function and this allows us to write the value function in a much more famous form known as \textbf{Bellman Equation}:
\begin{equation}
    \mathbb{V}^\pi(s) = \mathcal{R}(s) +  \gamma\sum_{s'} \mathbb{P}(s_{t+1} = \mathnormal{s'} \vert s_t = \mathnormal{s}, a_t = \mathnormal{a}) \ \mathbb{V}(s')
\end{equation}
This equation has two terms and says that the expected sum of discounted return is equal to the sum of two terms: the initial, or immediate, reward received immediately simply for starting in state $s$, and a discounted sum of the future expected rewards (rewards after the first step) and can be rewritten as \begin{equation}
    \mathbb{V}^{\pi^\star}(s) = \mathcal{R}(s) +  \gamma\,max_a\sum_{s'} \mathbb{P}(s_{t+1} = \mathnormal{s'} \vert s_t = \mathnormal{s}, a_t = \mathnormal{a}) \ \mathbb{V}(s')
\end{equation}
to describe the optimal value function $\mathbb{V}^\star$.

\section{Action-value function}
\begin{comment}
\begin{figure}[ht]
    \centering
    \includegraphics[width=1\textwidth]{./pictures/bellman.eps}
    \caption{A representation of the Bellman equation.}
    \label{fig:bellman}
\end{figure}
\end{comment}
\begin{figure} [ht]
    \tikzstyle{squarenode} = [circle, draw, minimum size=5mm]

    \centering
    \begin{tikzpicture}[align = center, node distance = 9em, auto, thick]
        \node [squarenode] (s1) {$s_1$};
        \node [squarenode, right of=s1] (s2) {$s_2$};
        \node [squarenode, right of=s2] (s3) {$s_3$};
        \node [squarenode, right of=s3] (s4) {$s_4$};
        \node[squarenode,draw=none, right of=s4, xshift=-5em, yshift=1.6em] (s5) {};

        \draw [->] (s1) edge [out=90 in=90] node[right, above] {$\mathcal{R}(s_1, a_1)$} node[very near start] {$a_1$} (s2);
        \draw [->] (s2) edge [out=90 in=90] node[right, above] {$\mathcal{R}(s_2, a_2)$} node[very near start] {$a_2$}  (s3);
        \draw [->] (s3) edge [out=90 in=90] node[right, above] {$\mathcal{R}(s_3, a_3)$} node[very near start] {$a_3$}  (s4);
        \draw (s4) edge [dotted] node {}  (s5);

    \end{tikzpicture}
    \caption{A representation of Bellman equation.}
    \label{fig:bellman}
\end{figure}
The Bellman equation is fundamental and is a necessary condition for optimality. Visually, our system goes like the one depicted in figure \ref{fig:bellman} and the Bellman Equation~(\ref{eq:bellman-value}) describes this system immediately after state $s_1$, before taking action $a_1$, so when we talk about \textit{value} we talk about a state and eventually we will get into another state that can also be represented with another value function. This is why the Bellman equation can be depicted in a recursive way.

The purpose of explaining the Bellman equation graphically is to show that, as the value function shows the infinite recursion of the system starting from state $s$, we can actually represent this recursion, starting immediately after the action is taken.

This equation is called \textbf{action-value function} and is defined in two ways. The first one in the Bellman equation form:
\begin{equation}
    \mathbb{Q}^\pi(s, a) = \mathcal{R}(s, a) +  \gamma\sum_{s'} \mathbb{P}(s_{t+1} = \mathnormal{s'} \vert s_t = \mathnormal{s}, a_t = \mathnormal{a}) \ \,max_a\mathbb{Q}(s', a')
\end{equation}
and the second one in the expectation form:
\begin{equation}
    \mathbb{Q}^\pi(s, a) = \mathbb{E} \Big[ \sum_t \gamma^t \mathcal{R}(s_t) \ \vert \ \pi, \ s_0 = s, \ a_0 = a \Big]
\end{equation}
This equations can be explained as we are in state $s$, we take action $a$, we get the first reward that is the first term, and then we go on with the summation.

The optimal value and action value functions are related in the following way.
\begin{align*}
    \mathbb{V}^\star(s) &= \,max_a\mathbb{Q}^\star(s, a) \\
    \mathbb{Q}^\star(s, a) &= \mathcal{R}(s, a) +  \gamma\sum_{s'} \mathbb{P}(s_{t+1} = \mathnormal{s'} \vert s_t = \mathnormal{s}, a_t = \mathnormal{a}) \ \mathbb{V}^\star(s')
\end{align*}
One way of thinking about this is: we always go further in the cycle. For the value function, we need the best possible action and the state-action function has already taken an action, so I just get the max state-action function. Same way for the second function, state-action function starts after the action is taken, so I sum the reward and I continue at the landing state so with the value function.

Someone may ask themselves \textit{why are we bothering of creating this action-value function when we already have the value function?} \\
Well, the \textit{action-value function} is key in reinforcement learning as it allows to take expectations of $\mathbb{Q}^\pi(s,a)$ without knowing the transition function or the reward function.

\section{Algorithms}
Now that we have defined two important functions, we can describe a reinforcement learning algorithm.

\subsection{Introduction}
\begin{figure} [ht]
    \tikzstyle{block} = [rectangle, draw,
        text width=8em, text centered, rounded corners, minimum height=4em]

    \tikzstyle{line} = [draw, -latex]
    \centering
    \begin{tikzpicture}[align = center, node distance = 15em, auto, thick]
        \node [block] (Agent) {Agent ($\pi$)};
        \node [block, right of=Agent] (Environment) {Environment};
        \draw [->] (Agent) edge [bend left]  node {$a$} (Environment);
        \draw [->] (Environment) edge [bend left] node {$s$}  (Agent);
        \draw [->] (Environment) edge [dotted] node [above] {$\mathcal{R}(s,a)$} (Agent);
    \end{tikzpicture}
    \caption{The concept of reinforcement learning.}
    \label{fig:rldiagram}
\end{figure}
\noindent
To understand better reinforcement learning, it's better to see Figure~\ref{fig:rldiagram}. The environment reveals to the agent in forms of states and the agent replies with actions and the environment sends back a reward. All the computation or, in a more reinforcement learning way, the \textit{thinking}, is done in the agent \textit{mind} through the policy. \\
An important concept is that the information regarding the environment is only available through this interaction, meaning that the environment is not available in the agent's brain (the policy), but the agent is \textit{experiencing} the environment only by interacting with it.

\subsection{Different families of algorithms}
We can think of a RL algorithm as a method that, given a sequence of action-state-reward triplets, is able to find an optimal policy $\pi$ as depicted in Figure~\ref{fig:rlalg-general}.
\begin{figure} [ht]
    \tikzstyle{block} = [rectangle, draw,
        text width=8em, text centered, rounded corners, minimum height=4em]
    \tikzstyle{txtblock} = [rectangle,
        text width=8em, text centered, rounded corners, minimum height=4em]

    \tikzstyle{line} = [draw, -latex]
    \centering
    \begin{tikzpicture}[align = center, node distance = 15em, auto, thick]
        \node [txtblock] (sequence) {$\langle s, a, r \rangle^+$};
        \node [block, right of=sequence, xshift=-3em] (algorithm) {RL algorithm};
        \node [txtblock, right of=algorithm, xshift=-3em] (policy) {$\pi$};
        \draw [->] (sequence) edge  node {} (algorithm);
        \draw [->] (algorithm) edge node {}  (policy);
    \end{tikzpicture}
    \caption{Reinforcement learning algorithm}
    \label{fig:rlalg-general}
\end{figure}
The RL algorithm block can be split in different ways according to the way the policy is learned. We can divide algorithms in three main families.

\subsubsection{Model-based algorithms}
\begin{figure} [ht]
    \tikzstyle{block} = [rectangle, draw,
        text width=4em, text centered, rounded corners, minimum height=4em]
    \tikzstyle{txtblock} = [rectangle,
        text width=4em, text centered, rounded corners, minimum height=4em]

    \tikzstyle{line} = [draw, -latex]
    \centering
    \begin{tikzpicture}[align = center, node distance = 15em, auto, thick]
        \node [txtblock] (sequence) {$\langle s, a, r \rangle^+$};
        \node [block, right of=sequence, xshift=-9em] (model-learner) {model learner};
        \node [txtblock, text width=2em, right of=model-learner, xshift=-10em] (p-r) {$\mathcal{P}, \mathcal{R}$};
        \node [block, right of=p-r, xshift=-10em] (solver) {solver};
        \node [txtblock, text width=1em, right of=solver, xshift=-10em] (action-value) {$\mathbb{Q}^\star$};
        \node [block, right of=action-value, xshift=-10em] (argmax) {argmax};
        \node [txtblock, text width=2em, right of=argmax, xshift=-10em] (policy) {$\pi$};
        \draw [->] (sequence) edge  node {} (model-learner);
        \draw [->] (model-learner) edge node {}  (p-r);
        \draw [->] (p-r) edge node {}  (solver);
        \draw [->] (solver) edge node {}  (action-value);
        \draw [->] (action-value) edge node {}  (argmax);
        \draw [->] (argmax) edge node {}  (policy);
        \path [line] (p-r) |-++ (0cm, 1.2cm) -| (model-learner);

    \end{tikzpicture}
    \caption{A model-based algorithm}
    \label{fig:model-based}
\end{figure}
\noindent
A \textbf{model-based} algorithm takes the action-state-reward triples and sends them to a \textit{model learner} which learns the transition probability function $\mathcal{P}$ and the reward function $\mathcal{R}$. Once learned, a solver can compute an optimal action-value function. Once you have an optimal action-value function, for each state you just get the argmax to get the best action, and that's the policy.
The reason we have an arrow going back in the model learner is that the transition probability and the reward functions are initially estimated and are continuously updated with previous values of the functions.

\subsubsection{Value-function based algorithms}
\begin{figure} [ht]
    \tikzstyle{block} = [rectangle, draw,
        text width=4em, text centered, rounded corners, minimum height=4em]
    \tikzstyle{txtblock} = [rectangle,
        text width=4em, text centered, rounded corners, minimum height=4em]

    \tikzstyle{line} = [draw, -latex]
    \centering
    \begin{tikzpicture}[align = center, node distance = 15em, auto, thick]
        \node [txtblock] (sequence) {$\langle s, a, r \rangle^+$};
        \node [block, right of=sequence, xshift=-9em] (value-update) {value update};
        \node [txtblock, text width=1em, right of=value-update, xshift=-10em] (action-value) {$\mathbb{Q}$};
        \node [block, right of=action-value, xshift=-10em] (argmax) {argmax};
        \node [txtblock, text width=2em, right of=argmax, xshift=-10em] (policy) {$\pi$};
        \draw [->] (sequence) edge  node {} (value-update);
        \draw [->] (value-update) edge node {}  (action-value);
        \draw [->] (action-value) edge node {}  (argmax);
        \draw [->] (argmax) edge node {}  (policy);
        \path [line] (action-value) |-++ (0cm, 1.2cm) -| (value-update);

    \end{tikzpicture}
    \caption{A value-function based algorithm}
    \label{fig:value-function-based}
\end{figure}
\noindent
The \textbf{value-function} based algorithms are almost the same as the previous family of algorithms except that, as we can see from figure~\ref{fig:value-function-based}, one block is missing. The main difference is that this family of algorithms tries to learn a policy without explicitly learning the model or better, the policy is learned without the assumption of knowing the model, whereas in the first family, to learn the policy, the model \textit{must} be learned (the two functions that define a model are the transition probability function and the reward function).
This family of algorithm is in fact also called \textbf{model-free algorithms}.

In this family, the state-action function is directly estimated from the sequence of tuples from the environment and previous values of $\mathcal{Q}$. The $\mathcal{Q}$ doesn't have a $\star$ since it's just an estimation and it's not an optimal value but, with continuous iterations, it is proven that it will reach an optimal value.

\subsubsection{Policy search algorithms}
\begin{figure} [ht]
    \tikzstyle{block} = [rectangle, draw,
        text width=4em, text centered, rounded corners, minimum height=4em]
    \tikzstyle{txtblock} = [rectangle,
        text width=4em, text centered, rounded corners, minimum height=4em]

    \tikzstyle{line} = [draw, -latex]
    \centering
    \begin{tikzpicture}[align = center, node distance = 15em, auto, thick]
        \node [txtblock] (sequence) {$\langle s, a, r \rangle^+$};
        \node [block, right of=sequence, xshift=-9em] (policy-update) {policy update};
        \node [txtblock, text width=2em, right of=policy-update, xshift=-10em] (policy) {$\pi$};
        \draw [->] (sequence) edge  node {} (policy-update);
        \draw [->] (policy-update) edge node {}  (policy);
        \path [line] (policy) |-++ (0cm, 1.2cm) -| (policy-update);
    \end{tikzpicture}
    \caption{A policy search algorithm}
    \label{fig:policy-search}
\end{figure}
\noindent
The third family of algorithms updates directly the policy by feeding it back to the policy update that modifies it based on the sequence that it receives. This algorithm is much more direct, but the learning problem is much more difficult because the feedback you get from the policy usually is not that useful to update it. For example, in a game, knowing that an action moves you forward isn't sufficient to know when to use it.
\newline
\newline
As depicted in figures~\ref{fig:model-based}, \ref{fig:value-function-based} and \ref{fig:policy-search}, we can see that the three algorithms are getting simpler as going down, meaning that the policy learning is more direct but in the opposite way, going up, the algorithms become more supervised. Most of the time of this semester project has been spent in model-free value-based algorithms that provide a good balance between learning techniques and learning difficulties.

\subsection{Off-policy and on-policy algorithms}
A further distinction between algorithms can be made through another aspect, if they are \textit{off-policy} or \textit{on-policy}.

\begin{itemize}
    \item \textbf{off-policy} algorithms are those algorithms that learn about the greedy strategy (policy), while following a different policy that allows a better exploration of the state space;
    \item \textbf{on-policy} algorithms are those algorithms that estimate functions according to the current policy.
\end{itemize}

An on-policy algorithm is influenced by the exploration policy and can get stuck in a local maximum whereas an off-policy algorithm is independent from the exploration policy and reaches the global optimum. \\
Algorithmically, an on-policy algorithm updates its $\mathcal{Q}$-values using the $\mathcal{Q}$-value of the next state $s'$ and the current policy's action $a'$ so it estimates the return for state-action pairs assuming the \textit{current policy} continues to be followed. An off-policy algorithm updates its $\mathcal{Q}$-values using the $\mathcal{Q}$-value of the next state $s'$ and the greedy action $a'$ so it estimates the return for state-action pairs assuming a \textit{greedy policy} was followed despite the fact that it is not following a greedy policy. \\
This distinction disappears when the followed policy is a greedy policy but an agent that works like this wouldn't be as good since it would never explore the space of solutions.

%policy based
%value based
%model based
%model free

%off-on policy
%actor algorithm
%critic algorithms
%actor critic algorithms

\subsection{Deep Q-Network}

\textbf{Deep Q-Network}, or \textbf{DQN}, is the first popular reinforcement learning algorithm, proposed by DeepMind in 2013~\cite{pongpixels}.

It's an updated version of the Q-learning algorithm first proposed by Watkins in 1992~\cite{Watkins1992} and uses deep learning to estimate the action-value function. The Q-network is the neural network function approximator with parameters $\theta$.

Since it approximates and updates the action-value function, DQN is a \textit{model-free}, \textit{value-function based} algorithm: it solves the reinforcement learning task directly using samples from the emulator, without explicitly constructing an estimate of the model.

The two main characteristics regarding this algorithm are how it approximates the action-value function and the \textbf{experience replay}'s concept.
\subsubsection{Deep Reinforcement Learning}
Deep reinforcement learning can be defined as the connection between reinforcement Learning and neural networks, started in the late 90s~\cite{Tesauro1994} but with DQN it started to get exciting achievements and visibility. The deep reinforcement learning part is in the approximation of the action-value function with a convolutional neural network with 3 hidden layers. The input is an $84 \times 84 \times 4$ image, the first hidden layer convolves 16 $8 \times 8$ filters with stride 4, the second hidden layers convolves 32 $4 \times 4$ filters with stride 2, the final hidden layer is a fully-connected layer that consists of 256 rectifier units and the output layer is a fully-connected linear layer with a single output for each valid action.

The input image represents the state and the output actions are the actions that the agent can perform. It is interesting to see that this same network has been applied to seven Atari games and no preprocessing has been applied to the images (the algorithm worked directly with raw images from the games).

\subsubsection{Experience Replay}
The concept of experience replay was first introduced in 1992~\cite{Lin1992} as a way to reuse past experiences and replay them to the network in order to refresh them to it. The \textit{re-learning problem} says that  if an input pattern has not been presented for quite a while, the
network typically will forget what it has learned for that pattern and thus need to re-learn it when that pattern is seen again later. In DQN, a \textit{replay memory} is used to store past experiences and show them again in a random way.
\begin{spacing}{1.15}
\begin{algorithm}[ht]
\begin{algorithmic}
\State Initialize replay memory $\mathcal{D}$ to capacity $N$
\State Initialize action-value function $Q$ with random weights
%\State Require preprocessor $h(s)$ that maps histories to fixed-length representations.
\For{episode $=1,M$}
\State Initialise sequence $s_1 = \{x_1\}$ and preprocessed sequenced $\phi_1 = \phi(s_1)$
\For {$t=1,T$}
	\State With probability $\epsilon$ select a random action $a_t$
	\State otherwise select $a_t = \max_{a} Q^*(\phi(s_t), a; \theta)$
	\State Execute action $a_t$ in emulator and observe reward $r_t$ and image $x_{t+1}$
	\State Set $s_{t+1} = s_t,a_t,x_{t+1}$ and preprocess $\phi_{t+1} = \phi(s_{t+1})$
	\State Store transition $\left(\phi_t,a_t,r_t,\phi_{t+1}\right)$ in $\mathcal{D}$
	%\For {$k=1$ to $K$}
	\State Sample random minibatch of transitions $\left(\phi_j,a_j,r_j,\phi_{j+1}\right)$ from $\mathcal{D}$
	\State Set
	$y_j =
    \left\{
    \begin{array}{l l}
      r_j  \quad & \text{for terminal } \phi_{j+1}\\
      r_j + \gamma \max_{a'} Q(\phi_{j+1}, a'; \theta) \quad & \text{for non-terminal } \phi_{j+1}
    \end{array} \right.$
	\State Perform a gradient descent step on $\left(y_j - Q(\phi_j, a_j; \theta) \right)^2$
	%\EndFor
\EndFor
\EndFor
\end{algorithmic}
\caption{Deep Q-learning with Experience Replay}
\label{alg}
\end{algorithm}
\end{spacing}

\noindent
Algorithm \ref{alg} shows the pseudo code of DQN. We can see that after a random initialization of the convnet and of the replay memory, for each episode and for each time step (image in the episode), to the corresponding state $s_t$ an $\epsilon$-greedy policy is applied (off-policy learning). \\
Optimization is then performed using stochastic gradient descent based on the $\mathcal{Q}$-learning target and the $\mathcal{Q}$-network.

Note that the preprocessing on the sequence is just a rescaling of the image, for lower memory utilization, no other preprocessing has been performed.

This implementation has been tested with 7 Atari games. It showed incredible results, obtaining for every game higher results than previous algorithms and sometimes even outperforming humans.

\subsection{Asynchronous Advantage Actor Critic}
\textbf{A3C}~\cite{a3c} is a framework developed by the same team as before, DeepMind, that outperforms their first model, DQN, with the only use of CPU (DQN instead relies heavily on GPU). \\
The authors present four famous reinforcement learning algorithms in an asynchronous way ($\mathcal{Q}$-learning, SARSA, $n$-step $\mathcal{Q}$-learning and A3C). What does asynchronous mean? \\
It means that parallel agents are executed in parallel, on multiple instances of the environment. This parallelism allows a wider view since at any given time-step the parallel agents will be experiencing a variety of different states and after a certain period policies are combined. It has been shown that experience replay is no longer needed and that the same results can be reached by increasing the number of instances running in parallel.\\
The asynchronous architecture is achieved using multiple CPU threads on a single machine, allowing every consumer computer to use this implementation.

The best performing algorithm of the four presented is the \textbf{Asynchronous advantage actor-critic (A3C)}. \\
First of all, it's called actor-critic since actor and critic are synonyms of policy-based and value-based respectively~\cite{konda2003}. This means that this algorithm aims to combine the strengths of both families of algorithms: from the policy-based family of algorithms it gets the simplicity of the problem, optimizing the policy directly with no need to get the action-value function, whereas from the value-based algorithms it gets the ability to \textit{learn}, since, when working with the policy, when it gets updated, the gradients are computed independently from the previous estimates so there is no \textit{learning} in the sense of an accumulation and consolidation of informations. Actor-critic methods also have good convergence properties in contrast to critic-only methods.
\begin{spacing}{1.15}
\begin{algorithm}[ht]
\caption{Asynchronous advantage actor-critic - pseudo code for each actor-learner thread.}
\begin{algorithmic}
\small
\State \emph{// Assume global shared parameter vectors $\theta$ and $\theta_v$ and global shared counter $T=0$}
\State \emph{// Assume thread-specific parameter vectors $\theta'$ and $\theta'_v$}
\State Initialize thread step counter $t\gets 1$
\Repeat
\State Reset gradients: $d\theta \gets 0$ and $d\theta_v \gets 0$.
\State Synchronize thread-specific parameters  $\theta'=\theta$ and $\theta'_v=\theta_v$ %
\State $t_{start} = t$
\State Get state $s_t$
\Repeat
\State Perform $a_t$ according to policy $\pi (a_t|s_t;\theta')$
\State Receive reward $r_t$ and new state $s_{t+1}$
%
\State $t \gets t + 1$
\State $T \gets T + 1$
\Until terminal $s_t$ \textbf{or} $t-t_{start}==t_{max}$
\State $R =
    \left\{
    \begin{array}{l l}
      0  \quad & \text{for terminal } s_t\\
        V(s_t,\theta'_v) \quad & \text{for non-terminal } s_t \text{// Bootstrap from last state}
    \end{array}\right.$
\For {$i \in \{t-1,\ldots,t_{start} \}$}
\State $R \gets r_i + \gamma R$
\State Accumulate gradients wrt $\theta'$: $d\theta \gets d\theta + \nabla_{\theta'} \log\pi(a_i|s_i;\theta') (R - V(s_i;\theta'_v))$
\State Accumulate gradients wrt $\theta'_v$: $d\theta_v \gets d\theta_v + {\partial\left(R - V(s_i;\theta'_v)\right)^2}/{\partial \theta'_v}$
\EndFor
%
\State Perform asynchronous update of $\theta$ using $d\theta$ and of $\theta_v$ using $d\theta_v$.
%
\Until $T > T_{max}$
\end{algorithmic}
\label{a3c-alg}
\end{algorithm}
\end{spacing}

This algorithm uses a so called \textit{forward view} meaning that the algorithm selects actions using its exploration policy for up to a certain number of steps in the future and the agent will then receive up to the same number of rewards from the environment. Forward view in a sentence explains how far ahead you need to look to figure out the value of a state. \\
Deep learning techniques are used to estimate both the policy and the value function where for the policy there is a softmax output instead for the value function there is one linear output with all other layers of a convolutional neural network shared. \\
The updates on the policy and on the value function are then performed according to the gradient of an estimate of the advantage function that takes into account both the value function parameters and the policy parameters.

In algorithm~\ref{a3c-alg} the pseudo code for the A3C algorithm.


\documentclass[11pt]{article}
\usepackage{pgfplots}
\usepackage{amssymb}
\usepackage{url}
\pgfplotsset{compat=newest}
\setlength\topmargin{-0.6cm}
\setlength\textheight{23.4cm}
\setlength\textwidth{17.0cm}
\setlength\oddsidemargin{0cm}
\begin{document}
\title{Ling 572 HW9}
\author{Daniel Campos  \tt {dacampos@uw.edu}}
\date{03/13/2019}
\maketitle
\section{Q1}
\subsection{What does f'(x) intend to measure?}
The derrivative of a function measures the rate of change of a function rleative to the change in the agrument(in this case X).
\subsection{Let $h(x)=f(g(x))$ }
$h'(x)$ = $f'(g(x)) \cdot g'(x)$
\subsection{Let $h(x)=f(x)g(x)$ }
$h'(x) = f'(x)g(x) + f(x)g'(x)$
\subsection{Let $f(x)=a^x$ where $a>0$}
$ f'(x)$ =$ a^x log(a)$
\subsection{Let $f(x)= x^{10}-2x^8  \frac{4}{x^2} + 10$}
$f'(x) =10x^{9}-16x^7 + \frac{8}{x^3}$
\section{Q2}
The logistic function is $f(x)=\frac{1}{1+e^{-x}}$. The tanh function is $g(x)=\frac{e^x - e^{-x}}{e^x +e^{-x}}$.
\subsection{Prove that $f'(x)= f(x)(1-f(x))$}
1. $f'(x) = \frac{e^{-x}}{(1+e^{-x})^2}$ \\
2. $f'(x) =  \frac{1}{1+e^{-x}}-\frac{1}{(1+e^{-x})^2}$ \\
3. $f'(x) = \frac{1}{1+e^{-x}} \cdot (1-\frac{1}{1+e^{-x}})$  which means  $f'(x)= f(x)(1-f(x))$\\
\subsection{Prove that $g'(x)=1 - g^2(x)$}
1. $tanh(x) = \frac {sinh(x))}{cosh(x)}$ \\
2. $g'(x) = \frac{df}{dx} \frac{sinh(x)}{cosh(x)}$\\
3. $g'(x) = \frac{cosh^2(x) - sinh^2(x)}{cosh^2(x)}$\\
4. $g'(x) = \frac{cosh^2(x)}{cosh^2(x)} - {sinh^2(x)}{cosh^2(x)}$\\
5. $g'(x) = 1- {sinh^2(x)}{cosh^2(x)} $\\
6. $g'(x) = 1-tanh^2(x)$\\
\subsection{Prove that $g(x) = 2f(2x)-1$}
1. $f(2x) = \frac{1}{1+e^{-2x}}$ \\
2. $2f(2x) =\frac{2}{1+e^{-2x}}$ \\
3. $2f(2x) - 1 = \frac{2}{1+e^{-2x}}  -1$ \\
3. $2f(2x) - 1 = \frac{2}{1+e^{-2x}}  - \frac{1+e^{-2x}}{1+e^{-2x}}$ \\
4. $2f(2x) - 1 = \frac{2- e^{-2x}}{1+e^{-2x}}$ \\
5. $2f(2x) - 1 = \frac{(e^{x}-1)(e^{x}+1)}{1+e^{2x}}$ \\
6. $2f(2x) - 1 = \frac{e^{2x}-1)}{1+e^{2x}}$ \\
7.  $2f(2x) - 1  = \frac{e^{2x}-1}{e^{2x}+1} = tanh(x) = g(x)$  \\
\section{Q3}
\subsection{What is $f'_x$ trying to measure?}
A partial derrivative is trying to measure the change in a fucntion based on a variable assuming all other varriables in the function remain constant. In other words, our derrivative is representing the effect of a variable on the equation when no other variables are effecting the equation.
\subsection{$f(x,y)=x^3 + 3x^2y+y^3 + 2x$.}
$f'_x = 3x^2 + 6xy+2 $  \\ $f'_y = 3(x^2+y^2)$
\subsection{ $z = \sum_{i=1}^n w_i x_i$.}
$\frac{dz}{dw_i}= \sum{i=1}^n x_i $
\subsection{$f(z)=\frac{1}{1+e^{-z}}$ and $z = \sum_{i=1}^n w_i x_i$.}
$\frac{df}{dz}= f(z)* \frac{e^{-z}}{1 + e^{-z}} $ \\
$\frac{df}{dw_i}=f(z)*\frac{e^{- \sum{i=1}^n x_i }}{1+e^{- \sum{i=1}^n x_i }}$ \\
\subsection{$E(z)=\frac{1}{2}(t - f(z))^2$, $f(z)=\frac{1}{1+e^{-z}}$ and $z = \sum_{i=1}^n w_i x_i$.}
$\frac{dE}{dw_i} = -(t-f(z))*f(z)*\frac{e^{- \sum{i=1}^n x_i }}{1+e^{- \sum{i=1}^n x_i }}$

\section{Q4 Softmax Funciton}
\subsection{Where in NNs is the softmax function used and why?}
Softmax is used to normalize the outputs of NN to interval (0,1) and to make all components to add up to 1 so that they can be interpreted as regular probabilities. This tends to be implemented as the final step of a NN to get a probability distribution over all possible classes/predictions.  \\
\subsection{x is [1, 2, 3, -1, -4, 0], what is the value of softmax(x)}
[0.08607859048507978, 0.23398586833496002, 0.6360395340111326,\\ 0.011649470423906664, 0.0005799929804444501, 0.03166654376447658]
\section{Q5:  FNN}
\subsection{How many connections (i.e., weights) are there in this network?}
$ connections = \sum_{i=1}^{m-1} n_i*n_{i+1}$ since there are m-1 layers of connections.
\subsection{Given the input $x$, what is the formula for calculating the output of the first hidden layer?}
$Y_h = g(M_k*X) $
\subsection{Given the input $x$, what is the formula for calculating the output of the output layer?}
$Y_o = g(M_mg(M_m-1*...*g(M_k*X))) $  where m is total depth of network
\section{Q6 MNIST NNs}
\subsection{What's the loss function used in the digit recognition task?}
The loss function is Mean squared error(MSE) or $ \frac{1}{2n} \sum{x} ||y(x)-a||^2$.
\subsection{Why do they choose to minimize this function instead of maximizing classification accuracy?}
They choose do minimize this since the classification accuracy is not a smooth function that can be connected back to the weights and biases in the network. It is difficult to optimize small changes in weights and biases since they are unlikley to affect changes in training accuracy but will likley affect the MSE.
\subsection{In gradient descent, what's the formula for updating the weight matrix (or vector)? Why is that a good formula?}
$v \rightarrow v' = v- n*\triangledown C$ its a good formula because it is easy to calculate and the gradient vector, $\triangledown C$ and it gives us a way of repeatedly changing the position of v in order to find a minimum of the function C.
\subsection{What are the main ideas and benefits of stochastic gradient descent?}
The main idea with stochastic gradient descent(SGD) is to estimate the gradient $\triangledown C$ on a small sample of randomly chosen tarining inputs. By doing this average over this sample we can get a good estimate of the true gradient and speed up gradient descent. SGD is useful since it allows an efficient and accurate way of directing a NN to a global minimum is a measurable, stepable function without computing the gradient of the entire dataset.
\subsection{What is a training epoch?}
A training epoch represents all the mini-batches that needed to be run until the network has exhausted all training input. Once this happenes people usually report a current loss function and accuracy and start another epoch.
\subsection{Let $T$ be the size of the training data, $m$ be the size of mini-batch, and your training process contains $E$ training epoches. How many times is each weight in the NN updated? }
$updates = E*\frac{T}{m}$
\subsection{How can one choose the learning rate?}
The learning rate needs to be explored during hyperparameter tuning. There are various optimzation to choose learning rate such as ADAM but in general many selections of learning rates come from the type of task being modeled and tinkering.
\subsection{What's the risk if the rate is too big?}
If the learning rate is too big then the network may never learn the function properly since a high learning rate can effectively skip over maximums
\subsection{What's the risk if the rate is too small?}
If the learning rate is too small then the network will train extremely slowly and it may never leave a local minum.
\section{Q7 MNSIT NN in practice}
\begin{table}[h]
\centering
\caption{Results on digit recognition}
\label{table1}
\begin{tabular}{|c|r|l|l|l|r|} \hline
  Expt id & \# of hidden neurons & epoch \# & mini batch size & learning rate & accuracy \\ \hline
  1  & 30   & 30 & 10 & 3.0 & 0.9461 \\ \hline
  2  & 10   & 30 & 10 & 3.0 & 0.9172 \\ \hline
  3  & 30   & 30 & 10 & 0.5 & 0.9403 \\ \hline
  4  & 30   & 30 & 10 & 10  & 0.9457 \\ \hline
  5  & 30   & 30 & 100 & 3.0 & 0.9302 \\ \hline
\end{tabular}
\end{table}
 \end{document}


\subsection{Weak- and wide-sense stationarity}

Unconditional probabilities don't change over time.

So GDP would not be stationary, but random noise would. A random walk is not stationary, because the variance increases over time.

\subsection{Weak-sense stationary}

Mean and autocovarinance don't change over time.

\subsection{Wide-sense stationary}

All moments are the same.