\documentclass[main.tex]{subfiles}
\begin{document}

\section{The equation of state and degenerate gasses}

\marginpar{Tuesday\\ 2020-10-20, \\ compiled \\ \today}

% Still no answer about the quadrupole stuff

% 151879

We move away from the relativistic realm, and treat the more classical Equation of State (EoS).
In general \(P\) could be a function of \(\rho \), \(\mu \), \(T\) and other variables.
An often-used one is \(P = P (\rho )\); also sometimes we use \(u = u(\rho )\), where \(u\) is the internal energy density.

We will treat the equation of state of a completely degenerate gas.

Let us start for a very simple system: a \textbf{hydrogen plasma}.
It is a collection of \(e^{-}\) and protons \(p\).

We have complete collisional ionization for \(T \gtrsim \SI{e5}{K}\).
Under which conditions is this plasma relativistic or nonrelativistic? This is shown as the blue area in figure \ref{fig:relativisticity_degeneracy}.

The electrons are surely relativistic if \(k_B T \gtrsim m_e c^2\), which corresponds to \(T \gtrsim \SI{6e9}{K}\).
For the protons, an analogous equation yields \(T \gtrsim \SI{e13}{K}\): these temperatures are basically never reached for realistic astrophysical scenarios.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/relativisticity_degeneracy.pdf}
\caption{}
\label{fig:relativisticity_degeneracy}
\end{figure}

In the region of \(\SI{e5}{K} \lesssim T \lesssim \SI{e9}{K}\) (for low densities), the plasma will be ionized but not relativistic.

The ideal gas law for the electrons reads \(P_e = n_e k_B T\), for the ions \(P_i = n_i k_B T\).
However, electrons are much lighter than protons, so for the same forces they will accelerate more, so they will radiate away more energy.

Another complication is the following: consider a collection of \(N\) ionic species, then for each of these (labelled by \(k\)) we will have \(P_{i, k}= n_{i, k} k_B T\).
The total pressure can be calculated by summing over all of these, plus the electrons:
%
\begin{align}
P &= n_e k_B T + \sum _{k} n_{i, k} k_B T = k_B T \qty(n_e + \sum_k n_{i, k})  \\
&= \frac{nk_B T}{\mu }
\,,
\end{align}
%
where \(\mu \) is the \emph{mean molecular weight}, calculated through the total baryonic number density \(n\):
%
\begin{align}
\mu = \qty[\frac{n_e}{n} + \frac{\sum _{k} n_{i, k}}{n} ]^{-1}
\,.
\end{align}

For a pure hydrogen plasma, we have one electron per proton, so the baryonic density is equal to the hydrogen atom density and we find
%
\begin{align}
\mu_{\ce{H}} = \qty[1 + 1]^{-1} = \frac{1}{2}
\,.
\end{align}

For a pure helium plasma we have two electrons per Helium nucleus, which contains 4 baryons: so,
%
\begin{align}
\mu_{\ce{^{4}He}} = \qty[ \frac{2}{4} + \frac{4}{4}]^{-1} = \frac{4}{3}
\,.
\end{align}

This holds under the assumption that the plasma behaves like an ideal gas, and that the electrons and protons are nonrelativistic.
There is a crucial reason why the first assumption fails: \textbf{degeneracy}.

Electrons obey Dirac statistics, and if the density is high enough they can behave very similarly to the \(T = 0\) limit even for high temperatures. The region in which they behave like a fully degenerate gas is shown in pink in figure \ref{fig:relativisticity_degeneracy}.

The distribution function for a system of fermions reads
%
\begin{align}
\frac{ \dd{N}}{ \dd[3]{x} \dd[3]{p}} = \frac{2}{h^3} \underbrace{\qty[\exp(\frac{E }{k_B T} - \alpha ) + 1]^{-1}}_{f}
\,.
\end{align}

Here, the \emph{degneracy parameter} is \(\alpha = \mu / k_B T\), where \(\mu  \) is the chemical potential.
The energy is expressed as \(E = \sqrt{m^2 c^{4} + p^2 c^2}\).

If we want to compute \(\alpha \), we can just integrate:
%
\begin{align}
n &= \int_{\mathbb{R}^3} \frac{ \dd{N}}{ \dd[3]{x} \dd[3]{p}} \dd[3]{p}  \\
&= \frac{2}{h^3} \int_0^{\infty }
\qty[\exp(\frac{E }{k_B T} - \alpha ) + 1]^{-1} 4 \pi p^2 \dd{p}
\,.
\end{align}

This integral can be computed for any value of \(\alpha \) and \(T\): we then find \(n = n(\alpha , T)\). Inverting this relation, we find \(\alpha = \alpha (n, T)\).

The logarithm of the absolute value of \(\alpha k_B T = \mu \) can be plotted against \(\log n\) for different values of the temperature \(T\). For each \(T\), there is a cusp: \(\alpha \) changes sign. It can become very big and positive or very big and negative.

% \todo[inline]{Do plot!}

Let us start with the case \(\abs{\alpha } \gg 1 \) and \(\alpha < 0\): this corresponds to low \(n\), high \(T\).
Then, the \(- \alpha \) appearing in the distribution is large and positive: then, the distribution looks like \(f \sim \exp(- \frac{E}{k_B T})\), the Maxwell-Boltzmann distribution.
The gas is behaving like an ideal gas.

Another option is \(\abs{\alpha } \gg 1\), \(\alpha > 0\). This is the case in which we have low \(T\), high \(n\). Even in the \(T \to 0\) limit, the product \(\alpha k_B T \) stays finite: this is a function of \(n\), and is called \(E _F\).

The distribution then looks like \(f \sim \qty[\exp( (E - E_F) / k_B T) + 1]^{-1} \to [E \leq E_F] = [p \leq p_F]\) in the limit \(T \to 0\).
(I use the Iverson bracket: \([\text{proposition}]\) is 1 if the proposition is true, 0 if it is false).

The higher \(n\) is, the higher the \(T\) for which the behavior is close to the \(T \to 0\) limit.

In the \(T \to 0\) limit, we can do the integration analytically: this yields an explicit expression for \(n\) in terms of the Fermi momentum \(p_F\) corresponding to the Fermi energy \(E_F\):
%
\begin{align}
n &= \frac{2}{h^3} \int [p \leq p_F] 4 \pi p^2 \dd{p} = \frac{8 \pi p_F^3}{3 h^3} \\
p_F &= \sqrt[3]{\frac{3n }{8 \pi }} h
\,.
\end{align}

This momentum is a characteristic of \(n\) independently of the temperature: for \(T > 0\) it will not be a hard limit anymore, but it is still a good descriptor of the Fermi gas.

We can write
%
\begin{align}
E_F = \sqrt{m^2 c^{4} + p_F^2 c^2} = \sqrt{1 + x_F^2} m c^2
\,,
\end{align}
%
where \(x_F = p_F / mc^2\).
For a nonrelativistic particle distribution \(x_F \ll 1\), so \(E_F \approx mc^2 + x_F^2 mc^2/2\).
The dependence on the number density of the kinetic part is \(\sim x_F^{2} \sim n^{2/3}\).

On the other hand, in the ultrarelativistic limit \(E_F \approx x_F mc^2 \sim x_F \sim n^{1/3}\).

This is the reason why in figure \ref{fig:relativisticity_degeneracy} the pink boundary curves down in the blue (relativistic region).

It is useful to define this quantity in terms of proper density, in \SI{}{g/cm^3}, instead of number density.

We have
%
\begin{align}
x_F = \frac{p_F}{mc} = \qty( \frac{3h^3}{8 \pi })^{1/3} \frac{1}{mc} n^{1/3}
\,;
\end{align}
%
the only degenerate species we will consider will be electrons, so the \(n\) in this equation should be substituted for \(n_e\), which we can express
by multiplying above and below by the mean baryon mass \(m_b\) multiplied by the baryon density \(n\), whose product is the baryonic mass density \(\rho \):
%
\begin{align}
n_e = \frac{n_e m_b n}{n m_b}
= \rho \frac{n_e}{n} \frac{1}{m_b}
= \frac{\rho}{\mu _e m_b}
\,,
\end{align}
%
which gives us:
%
\begin{align}
x_F &\sim \qty( \frac{n_e}{\SI{6e35}{m^{-3}}})^{1/3} \\
x_F &\sim \qty( \frac{\rho / \mu _e}{\SI{e9}{kg / m^3}})^{1/3}
% x_F &=\sim \num{e-2} \qty(\frac{\rho }{m_e})^{1/3}
\,.
\end{align}

% \todo[inline]{Are we sure about this? I think I missed something.}
This corresponds to the vertical delimiter in figure \ref{fig:relativisticity_degeneracy}: even at cold \(T\) for high enough densities the gas becomes relativistic.

The internal energy density \(u\) (not denoted as \(\rho \) to avoid confusion between it and the mass density) is computed in general as
%
\begin{align}
u &= \frac{2}{h^3}\int E(p) \frac{ \dd{N}}{ \dd[3]{x} \dd[3]{p}} \dd[3]{p}
\,,
\end{align}
%
which in our case (complete degeneracy) is
%
\begin{align}
u &= \frac{2}{h^3} \int_{0}^{p_F} \sqrt{m^2c^{4} + p^2c^2} 4 \pi p^2 \dd{p}  \\
&= \frac{8}{h^3} \pi  mc^2 (mc)^3 \int_0^{p_F} \sqrt{1 + x^2} x^2 \dd{x}  \\
&= \frac{8 \pi m^4 c^{6}}{h^3} \frac{x_F^{4}}{4} I(x_F)
\,,
\end{align}
%
where \(I(x_F)\) can be computed analytically, but it is of order 1 as can be seen by the asymptotics of the integral.

The integral reads
%
\begin{align}
\int_0^{p_F} \sqrt{1 + x^2} x^2 \dd{x} &= \frac{1}{8} \qty[x_F (1 + 2 x_F)^2 \sqrt{1 + x_F^2} - \log(x_F + \sqrt{1 + x_F^2})]  \\
&\overset{\text{def}}{=} \frac{x_F^{4}}{4} I(x_F)
\,.
\end{align}

For the pressure we have a similar integral, which however is more complicated from the conceptual point of view.

The first law of thermodynamics states that
%
\begin{align}
\dd{U} + p \dd{V} = 0
\,,
\end{align}
%
as long as the transformation does not exchange heat with its surroundings.
Note that \(\dd{u} = \dd{(U/ V)} = V^{-1} \dd{U} - U V^{-2} \dd{V}\), which means that
%
\begin{align}
\frac{ \dd{U}}{V} = \dd{u} + \frac{u}{V} \dd{V}
\,.
\end{align}

Substituting into the first law of thermodynamics,
%
\begin{align}
\dd{u} + \frac{U}{V} \dd{V} + P \frac{\dd{V}}{V} &= 0  \\
\dd{u} + \qty( \frac{U}{V} + P) \frac{ \dd{V}}{V} &= 0
\,,
\end{align}
%
but since \(V \propto 1/n\) we have
%
\begin{align}
\frac{ \dd{V}}{V} = - \frac{ \dd{n}}{n}
\,.
\end{align}

Substituting this in, we get
%
\begin{align}
\dd{u} - (P+u) \frac{ \dd{n}}{n} = 0
\,.
\end{align}

This means that
%
\begin{align}
n \frac{\dd{u}}{ \dd{n}} = P + u
\,,
\end{align}
%
which allows us to compute the pressure!
The only step remaining is to replace \(\dd{n} / n\) with an expression in terms of \(x_F\), which is
%
\begin{align}
\frac{ \dd{x_F}}{x_F} = 3 \frac{ \dd{n}}{n}
\,.
\end{align}

This finally yields
%
\begin{align}
P = \frac{x_F}{3} \dv{u}{x_F} - u
\,.
\end{align}

Then we are almost done: we can compute the pressure with \(u\), for which we have an analytic expression, and \(\dv*{u}{x_F}\), which we can easily find since the original expression for \(u\) was an integral in \(\dd{x_F}\), from which we can read off the integrand.

This yields
%
\begin{align}
P &= \frac{8 \pi m^4 c^{5}}{h^3} \qty[
    \frac{1}{3} x_F^3 \sqrt{1 + x_F^2 }
    - \frac{1}{8} \qty(x_F (1 + 2 x_F)^2 \sqrt{1 + x_F^2}
    - \log(x_F + \sqrt{1 + x_F^2}))
]  \\
&= \frac{m^4 c^{5} \pi }{h^3}
\qty[
    x_F \sqrt{1 + x_F^2}
    \qty( \frac{8}{3} x_F^2 - \qty(1 + 2 x_F^2))
    + \log(x_F + \sqrt{1 + x_F^2})
]  \\
&= \frac{m^4 c^{5} \pi }{h^3} \qty[
    x_F \sqrt{1+ x_F^2}
    \qty( \frac{2}{3} x_F^2 - 1)
    +
    \log(x_F + \sqrt{1 + x_F^2})
]
\,.
\end{align}

\end{document}


\chapter{XOR, XNOR gates}
%\ref{sec:background}.

\section{Aim}
%\label{sec:objectives}
	To verify and interpret the logic and truth table for XOR, XNOR gates using Resistor Transistor Logic (RTL)

\section{Apparatus}
%\label{sec:objectives}
	\begin{itemize}
		\tightlist
		\item Kit for realization of gates
		\item Connecting Leads
	\end{itemize}

\section{Circuits}

\section{Theory}
	Logic gates are the basic building blocks of any digital system. Logic gates are electronic circuits having one or more than one input and only one output. The relationship between the input and the output is based on a certain logic. Based on this, logic gates are named as:
	\begin{enumerate}
		\tightlist
		\item AND gate
		\item OR gate
		\item NOT gate
		\item NAND gate
		\item NOT gate
		\item Ex-OR gate
		\item Ex-NOR gate
	\end{enumerate}

	\subsection{XOR gate}
	The 'Exclusive-OR' gate is a circuit which will give a high output if either, but not both of its two inputs are high. An encircled plus sign ($\oplus$) is used to show the Ex-OR operation.
	\begin{align*}
		Y &= A \oplus B
	\end{align*}
	Ex-OR gate is created from AND, NAND and OR gates.The output is high only when both the inputs are different.
	\begin{figure}[ht]
		\centering
		\subfloat[Symbol]
		{
			\begin{circuitikz} \draw
				(0,0) node[xor port] (myand1) {}
				(myand1.in 1) node[anchor=east] {A}
				(myand1.in 2) node[anchor=east] {B}
				(myand1.out) node[anchor=west] {Y}
				;
			\end{circuitikz}
			\label{fig:xor_symbol}
		}
		\hfill
		\subfloat[Truth Table]
		{
			\begin{tabular}{|c|c|c|}
				\hline
				\multicolumn{2}{|c|}{Input} & Output \\
				\hline
				$A$ & $B$ & $Y=A\oplus B$ \\
				\hline
				0 & 0 & 0 \\
				\hline
				0 & 1 & 1 \\
				\hline
				1 & 0 & 1 \\
				\hline
				1 & 1 & 0 \\
				\hline
			\end{tabular}
			\label{fig:xor_table}
		}
		\hfill
		\subfloat[RTL Design]{
			\includegraphics[width=0.25\textwidth,valign=c]{img/exp3/fig1}
			\label{fig:xor_circuit}
		}
		\caption{\textit{XOR gate}}
	\end{figure}

	\subsection{XNOR gate}
		The 'Exclusive-NOR' gate circuit does the opposite to the EX-OR gate. It will give a low output if either, but not both of its two inputs are high. The symbol is an EX-OR gate with a small circle on the output. The small circle represents inversion.
		\begin{align*}
			Y &= \overline{A \oplus B}
		\end{align*}
		Ex-NOR gate is created from AND, NOT and OR gates.The output is high only when both the inputs are same.
		\begin{figure}[ht]
			\centering
			\subfloat[Symbol]
			{
				\begin{circuitikz} \draw
					(0,0) node[xnor port] (myand1) {}
					(myand1.in 1) node[anchor=east] {A}
					(myand1.in 2) node[anchor=east] {B}
					(myand1.out) node[anchor=west] {Y}
					;
				\end{circuitikz}
				\label{fig:xnor_symbol}
			}
			\hfill
			\subfloat[Truth Table]
			{
				\begin{tabular}{|c|c|c|}
					\hline
					\multicolumn{2}{|c|}{Input} & Output \\
					\hline
					$A$ & $B$ & $Y=\overline{A\oplus B}$ \\
					\hline
					0 & 0 & 1 \\
					\hline
					0 & 1 & 0 \\
					\hline
					1 & 0 & 0 \\
					\hline
					1 & 1 & 1 \\
					\hline
				\end{tabular}
				\label{fig:xnor_table}
			}
			\hfill
			\subfloat[RTL Design]{
				\includegraphics[width=0.25\textwidth,valign=c]{img/exp3/fig2}
				\label{fig:xnor_circuit}
			}
			\caption{\textit{XNOR gate}}
		\end{figure}

\section{Procedure}
	\subsection{XOR gate}
		\begin{figure}[ht]
			\centering
			\subfloat[Simulator 1]{\includegraphics[width=0.45\textwidth,valign=c]{img/exp3/fig3}
				\label{fig:xor_sim:1}}
			\hfill
			\subfloat[Simulator 2]{\includegraphics[width=0.4\textwidth,valign=c]{img/exp3/fig4}
				\label{fig:xor_sim:2}}
			\caption{\textit{Simulator for realizing circuit for XOR gate}}
		\end{figure}
		\subsubsection{Simulator 1}
			\begin{enumerate}
				\tightlist
				\item Connect the supply(+5V) to the circuit.
				\item Press the switches for inputs "A" and "B".
				\item The bulb glows if one of the switches is ON and one of the switches is OFF else it won't glow.
				\item Repeat step-2 and step-3 for all state of inputs.
			\end{enumerate}
		\subsubsection{Simulator 2}
			\begin{enumerate}
				\tightlist
				\item Enter the Boolean input "A" and "B".
				\item Enter the Boolean output for your corresponding inputs.
				\item Click on "Check" Button to verify your output.
			\end{enumerate}

	\subsection{XNOR gate}
		\begin{figure}[ht]
			\centering
			\subfloat[Simulator 1]{\includegraphics[width=0.45\textwidth,valign=c]{img/exp3/fig5}
				\label{fig:xnor_sim:1}}
			\hfill
			\subfloat[Simulator 2]{\includegraphics[width=0.4\textwidth,valign=c]{img/exp3/fig6}
				\label{fig:xnor_sim:2}}
			\caption{\textit{Simulator for realizing circuit for XNOR gate}}
		\end{figure}
		\subsubsection{Simulator 1}
			\begin{enumerate}
				\tightlist
				\item Connect the supply(+5V) to the circuit.
				\item Press the switches for inputs "A" and "B".
				\item The bulb glows if both the switches are ON or if both the switches are OFF else it won't glow.
				\item Repeat step-2 and step-3 for all state of inputs.
			\end{enumerate}
		\subsubsection{Simulator 2}
			\begin{enumerate}
				\tightlist
				\item Enter the Boolean input "A" and "B".
				\item Enter the Boolean output for your corresponding inputs.
				\item Click on "Check" Button to verify your output.
			\end{enumerate}

\section{Observations}
	\subsection{XOR gate}
			\begin{figure}[ht]
				\centering
				\subfloat[If the Inputs are alternative, LED is ON]{\includegraphics[width=0.45\textwidth,valign=c]{img/exp3/fig7}
					\label{fig:xor_obs:1}}
				\hfill
				\subfloat[If both Inputs are same, LED is OFF]{\includegraphics[width=0.45\textwidth,valign=c]{img/exp3/fig8}
					\label{fig:xor_obs:2}}
				\caption{\textit{Observations for different Input Values}}
			\end{figure}
			\begin{figure}[h]
				\centering
				\includegraphics[width=0.85\linewidth]{img/exp3/fig9}
				\caption{\textit{Observations for verification of Truth Table of the XOR gate}}
				\label{fig:xor_obs_2}
			\end{figure}

\pagebreak
	\subsection{XNOR gate}
			\begin{figure}[ht]
				\centering
				\subfloat[If the Inputs are alternative, LED is OFF]{\includegraphics[width=0.45\textwidth,valign=c]{img/exp3/fig10}
					\label{fig:xnor_obs:1}}
				\hfill
				\subfloat[If both Inputs are same, LED is ON]{\includegraphics[width=0.45\textwidth,valign=c]{img/exp3/fig11}
					\label{fig:xnor_obs:2}}
				\caption{\textit{Observations for different Input Values}}
			\end{figure}
			\begin{figure}[h]
				\centering
				\includegraphics[width=0.85\linewidth]{img/exp3/fig12}
				\caption{\textit{Observations for verification of Truth Table of the XNOR gate}}
				\label{fig:xnor_obs_2}
			\end{figure}

\section{Precautions}
	\begin{enumerate}
		\tightlist
		\item Make the connections when power supply is OFF.
		\item Ensure that the connections are tight.
		\item Change the status of inputs only when power supply is OFF.
	\end{enumerate}


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{float}
\usepackage{subfigure}
\usepackage{subfloat}

\title{CS294 Deep RL Assignment 2: Policy Gradients}

\author{Mohamed Khodeir}

\date{\today}

\begin{document}
\maketitle

\section*{Problem 1. State Dependent Baselines}
\subsection*{(a)}
% I will use the notation $\tau_{t+1}^{(s_t,a_t)}$ to refer to the remainder of a trajectory that passes through $s_t, a_t$
% $$ \Sigma_{t=1}^{T} E_{\tau \sim P_{\theta}(\tau)}[ \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t)] $$

As given in the question, we can use the chain rule to deconstuct $P_\theta(\tau)$ as:
$$P_\theta(\tau) = P_{\theta}(s_t, a_t) P_\theta(\tau/s_t,a_t | s_t, a_t)$$

We can then use the law of iterated expectations to express the expectation over $\tau$ as:

$$  E_{\tau/s_t,a_t \sim P_\theta(\tau/s_t,a_t | s_t, a_t)}[ E_{(s_t, a_t) \sim P_{\theta}(s_t, a_t)}[\nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t)]] $$

% The expression inside the inner expectation is constant with respect to $\tau_{t+1}^{(s_t,a_t)}$ so the whole expression just reduces to:

Looking just at the inner expectation, we see:

$$  E_{(s_t, a_t) \sim P_{\theta}(s_t, a_t)}[ \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t)] $$

%  as a sum:

% $$  \int_{(s_t, a_t)}  P_{\theta}(s_t, a_t)( \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t)) $$

Expanding the expectation, we can rewrite that as a nested integral, the first over $s_t$ and the second over $a_t$. We can also substitute the full form of $P_{\theta}(s_t, a_t)$  as a product of the policy and the state marginal.

$$  \int_{s_t} \int_{a_t}  P_{\theta}(s_t)\pi_\theta(a_t | s_t) \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) $$

Taking terms in common
$$  \int_{s_t}  P_{\theta}(s_t) b(s_t) \int_{a_t} \pi_\theta(a_t | s_t) \nabla_\theta \log \pi_\theta (a_t|s_t) $$

Using the identity  $\pi_\theta(a_t | s_t) \nabla_\theta \log \pi_\theta (a_t|s_t) = \nabla_\theta
\pi_\theta (a_t|s_t)$, we get:

$$  \int_{s_t}  P_{\theta}(s_t) b(s_t) \int_{a_t} \nabla_\theta
\pi_\theta (a_t|s_t) $$

becomes by linearity of differentiation:

$$  \int_{s_t}  P_{\theta}(s_t) b(s_t) \nabla_\theta \int_{a_t}
\pi_\theta (a_t|s_t) $$

The inner integral, being an integral over a propper probility distribution just sums to 1.

$$  \int_{s_t}  P_{\theta}(s_t) b(s_t) \nabla_\theta (1)$$

becomes

$$  \int_{s_t}  P_{\theta}(s_t) b(s_t) (0) = 0$$
.
So going back to the full equation in (12):

$$\sum_{t=1}^T E_{\tau \sim P_\theta}[\nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t)] =$$
$$\sum_{t=1}^T E_{\tau/s_t,a_t \sim P_\theta(\tau/s_t,a_t | s_t, a_t)}[ E_{(s_t, a_t) \sim P_{\theta}(s_t, a_t)}[\nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t)]] = $$
$$\sum_{t=1}^T E_{\tau/s_t,a_t \sim P_\theta(\tau/s_t,a_t | s_t, a_t)}[0] = $$
$$ 0 $$

\subsection*{(b)}

\subsubsection*{(a)}

Let's consider $P_\theta(s_{t+1:T}, a_{t:T} | s_{1:t}, a_{1:t-1})$, the probability of the "rest" of the trajerctory after ($s_1, a_1, s_2, a_2, ... a_{t-1}, s_t$).

Because an MDP satisfies the Markov property, we know that given $s_t$ and $a_t$, the probability of $s_{t+1}$ is independent of previous states and actions.

Therefore $P_\theta(s_{t+1:T}, a_{t:T} | s_{1:t}, a_{1:t-1})$ should exactly equal $P_\theta(s_{t+1:T}, a_{t:T} | s_t)$

We can show this using Bayes rule and by substituting the full form of $P_\theta(\tau)$, where $\tau$ is the whole trajectory ($s_1, a_1, s_2, a_2, ... a_{T-1}, s_T$). See Appendix.

% $$\frac{P(s_1)\prod_{i = 2}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}{P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}$$
\subsubsection*{(b)}
I will start by rewriting the expression for the probability of the "rest" of the trajectory using bayes rule:
$$P_\theta(s_{t+1:T}, a_{t:T}|s_{1:t}, a_{1:t-1}) = P_\theta(s_{t+1:T}, a_{t+1:T}|s_{1:t}, a_{1:t})P_\theta(a_t | s_{1:t}, a_{1:t})$$

This allows us to write:
$$P_\theta(\tau) = P_\theta(s_{1:t}, a_{1:t-1}) P_\theta(a_t | s_{1:t}, a_{1:t}) P_\theta(s_{t+1:T}, a_{t+1:T}|s_{1:t}, a_{1:t})$$

Note that, in our case $P_\theta(a_t | s_{1:t}, a_{1:t}) = \pi_\theta(a_t|s_t)$.

% $$E_{\tau \sim P_\theta(\tau)} \bigg[ f \bigg] = E_{(s_{1:t}, a_{1:t-1}) \sim P_\theta(s_{1:t}, a_{1:t-1})} \bigg[ E_{a_t \sim P_\theta(a_t | s_{1:t}, a_{1:t})} \big[E_{s_{t+1:T}, a_{t+1:T} \sim P_\theta(s_{t+1:T}, a_{t+1:T}|s_{1:t}, a_{1:t})} \big[f\big]\big] \bigg]$$
$$E_{\tau \sim P_\theta} \bigg[  \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \bigg] =$$
$$E_{(s_{1:t}, a_{1:t-1}) \sim P_\theta} \bigg[ E_{a_t \sim P_\theta} \big[E_{s_{t+1:T}, a_{t+1:T} \sim P_\theta} \big[ \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \big]\big] \bigg]$$

$$\int_{(s_{1:t}, a_{1:t-1})}P_\theta(s_{1:t}, a_{1:t-1}) \bigg[ \int_{a_t} \pi_\theta(a_t|s_t) \big[\int_{s_{t+1:T}, a_{t+1:T}} P_\theta(s_{t+1:T}, a_{t+1:T}|s_{1:t}, a_{1:t})  \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \big] \bigg]$$

$$\int_{(s_{1:t}, a_{1:t-1})}P_\theta(s_{1:t}, a_{1:t-1}) \bigg[ \int_{a_t} \pi_\theta(a_t|s_t) \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \big[\int_{s_{t+1:T}, a_{t+1:T}} P_\theta(s_{t+1:T}, a_{t+1:T}|s_{1:t}, a_{1:t}) \big] \bigg]$$

$$\int_{(s_{1:t}, a_{1:t-1})}P_\theta(s_{1:t}, a_{1:t-1}) \bigg[ \int_{a_t} \pi_\theta(a_t|s_t) \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \big[Const\big] \bigg]$$

Making use of that useful identity again, and moving constants out of the inner integral:

$$\int_{(s_{1:t}, a_{1:t-1})}P_\theta(s_{1:t}, a_{1:t-1}) b(s_t) \big[Const\big] \bigg[ \nabla_\theta \int_{a_t} \pi_\theta (a_t|s_t) \bigg]$$

$$\int_{(s_{1:t}, a_{1:t-1})}P_\theta(s_{1:t}, a_{1:t-1}) b(s_t) \big[Const\big] \bigg[ \nabla_\theta Const \bigg]$$
$$\int_{(s_{1:t}, a_{1:t-1})}P_\theta(s_{1:t}, a_{1:t-1}) b(s_t) \big[Const\big] \bigg[0 \bigg] = 0$$

As we've shown that $$E_{\tau \sim P_\theta} \bigg[  \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \bigg] = 0$$ it follows that $$\sum_{t = 1}^T E_{\tau \sim P_\theta} \bigg[  \nabla_\theta \log \pi_\theta (a_t|s_t) b(s_t) \bigg] =0 $$

% Rewriting the numerator:
% $$\frac{P(s_1)(\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})(\prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}
% {P(s_1)\prod_{i = 2}^{i = t)}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}$$

% $$\prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) = P_\theta(s_{t+1}|a_{t}, s_{t})P_\theta(a_{t}|s_{t})\prod_{i = t+2}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})$$

\section*{Problem 4. CartPole}
\subsection*{Learning Curves for Small/Large Batch Sizes}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{sb_cartpole.png}
\caption{Small Batch}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{lb_cartpole.png}
\caption{Large Batch}
\end{figure}

\subsection*{Analysis questions}
\subsubsection*{Trajectory-Centric vs Reward-To-Go w/out Advantage Centering}
We can see that the reward to go estimator displays higher performance, though the effect seems to be significantly less pronounced with larger batch sizes.
\subsubsection*{Advantage Centering}
Advantage centering certainly seems to have helped by reducing the variance of the estimator, which we can see in the more stable learning curve for both small and larger batch sizes.
\subsubsection*{Batch Size}
The batch size also seems to be very effective at reducing the variance of the gradient estimators both in the reward-to-go estimator as well as the trajectory-centric one. It also converges more quickly in the number of iterations in all cases.

\section*{Problem 5. Inverted Pendulum}
\subsection*{Learning Curve for Smallest Batch Size and Largest Learning Rate}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{problem-5.png}
\caption{The learning rate used is 0.01, and the batch size is 1000.}
\end{figure}

\section*{Problem 7. Lunar Landing}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{lunar_landing.png}
\caption{S}
\end{figure}

\section*{Problem 8. HalfCheetah}
\subsection*{batch size and learning rate}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{half_cheetah_bsize_lr.png}
\caption{S}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{halfcheetah_rtg_baseline.png}
\caption{S}
\end{figure}
% \begin{table}[H]
% \centering
% \begin{tabular}{l|l|l|r}
% & Game & Mean & Std \\\hline
% Expert & HalfCheetah & 2374.79 & 772.96\\
% BC Agent & HalfCheetah & 2110.61 & 947.18\\
% Expert & Humanoid & 2908.21 & 929.65\\
% BC Agent & Humanoid & 45.01 & 13.77
% \end{tabular}
% \caption{Stats reported over 100 rollouts. The HalfCheetah BC agent used a network of 128 and 64 units respectively followed by an output layer over 6 dimensions of the action space. The Humanoid agent 256 and 128 units respectively followed by an output layer over 17 dimensions of the action space.}
% \end{table}

% \subsection*{2.3 Experimentation}

% \begin{figure}[H]
% \centering
% \includegraphics[width=1\textwidth]{{../2.3-propdata-halfcheetah}.png}
% \caption{\label{fig:propdata}I chose to look at the effect of more training data on the BC Agent's performance. I trained the HalfCheetah agent using subsets of the data between 10\% and 100\%.}
% \end{figure}

% \subsection*{3.2 DAgger}
% \begin{figure}[H]
% \centering
% \includegraphics[width=1\textwidth]{{../3.2 - Dagger Comparison}.png}
% \caption{\label{fig:dagger}I chose the task on which the behavior cloning agent performed poorest relative to the expert policy - Humanoid. I used the same architecture as in the behavior cloning experiments (i.e. 256 relu > 128 relu -> 17 lin). I ran 20 episodes of the algorithm, training the policy for 80000 steps from scratch each episode, and generating 10 rollouts worth of samples for the next iteration. }
% \end{figure}
\section*{Appendix}
\subsection*{Problem 1 (b)}

$$P_\theta(s_{t+1:T}, a_{t:T} | s_{1:t}, a_{1:t-1}) = \frac{P_\theta(\tau)}{P(s_{1:t}, a_{1:t-1})}$$

Recall that the numerator, $P_\theta(\tau)$ is:

$$P_\theta(\tau) = P(s_1)\prod_{i = 2}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})$$

We can equivalently represent that as:

$$P_\theta(\tau) = \Bigg( P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \Bigg) \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})$$

The denomenator $P(s_{1:t}, a_{1:t-1})$ simply marginalizes $P_\theta$ over all possible assignments of the remaining states and actions. i.e.

$$P(s_{1:t}, a_{1:t-1}) = \sum_{a_{t:T}}\sum_{s_{t+1:T}} P_\theta(\tau) = \sum_{a_{t:T}}\sum_{s_{t+1:T}} P(s_1)\prod_{i = 2}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) = $$

Factoring out the terms that dont depend on the summation domains:

$$ \Bigg(P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \Bigg) \sum_{a_{t:T}}\sum_{s_{t+1:T}} \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})$$

Now, substituting this for our numerator and denomenator we get:

$$P_\theta(s_{t+1:T}, a_{t:T} | s_{1:t}, a_{1:t-1}) = \frac{ \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}{ \sum_{a_t}^T\sum_{s_{t+1:T}} \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}$$

We can follow a similar procedure starting from $P_\theta(s_{t+1:T}, a_{t:T} | s_t)$ to show that they reduce to the same expression.

 $$P_\theta(s_{t+1:T}, a_{t:T} | s_t) = \frac{\sum_{a_1:t-1}\sum_{s_1:t-1} P_\theta(\tau)}{\sum_{a_1:t-1}\sum_{s_1:t} \sum_{a_t:T-1}\sum_{s_t+1:T} P_\theta(\tau)}$$

 Substituting our factored form for $P_\theta$, looking  only at numerator:

  $$\sum_{a_1:t-1}\sum_{s_1:t-1} \Bigg( P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \Bigg) \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) = $$

  $$\prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \sum_{a_1:t-1}\sum_{s_1:t-1} \Bigg( P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \Bigg)$$
 Now denomenator:

 $$\sum_{a_1:t-1}\sum_{s_1:t-1} \sum_{a_t:T-1}\sum_{s_t+1:T} \Bigg( P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \Bigg) \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) = $$

  $$\sum_{a_{1:t-1}}\sum_{s_{1:t-1}} \Bigg( P(s_1)\prod_{i = 2}^{i = t}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) \Bigg) \sum_{a_t:T-1}\sum_{s_t+1:T}  \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1}) = $$

  Putting it all together:
  $$P_\theta(s_{t+1:T}, a_{t:T} | s_t) = \frac{ \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}{ \sum_{a_t}^T\sum_{s_{t+1:T}} \prod_{i = t+1}^{i = T}P_\theta(s_i|a_{i-1}, s_{i-1})P_\theta(a_{i-1}|s_{i-1})}$$

\end{document}


\documentclass[%
 reprint,
%superscriptaddress,
%groupedaddress,
%unsortedaddress,
%runinaddress,
%frontmatterverbose,
%preprint,
%showpacs,preprintnumbers,
%nofootinbib,
%nobibnotes,
%bibnotes,
 amsmath,amssymb,
 aps,
%pra,
%prb,
%rmp,
%prstab,
%prstper,
%floatfix,
]{revtex4-1}

\usepackage{graphicx}% Include figure files
\usepackage{dcolumn}% Align table columns on decimal point
\usepackage{bm}% bold math
\usepackage{listings}

%\usepackage{hyperref}% add hypertext capabilities
%\usepackage[mathlines]{lineno}% Enable numbering of text and display math
%\linenumbers\relax % Commence numbering lines

\begin{document}

\title{List Comprehensions in Java}% Force line breaks with \\

\author{Franco Arolfo}

\date{\today}% It is always \today, today,
             %  but any date may be explicitly specified

\begin{abstract}
In many programming languages such as Haskell or python is popular to write list comprehensions, like we write in algebra $\{x \mid x \in \mathbb{R} \wedge x \equiv 0 \mod 2 \}$ as the list of all even numbers in $\mathbb{R}$. This article proposes an implementation of list comprehensions in Java, and then provides the implementation of the functions \emph{map} and \emph{filter} using the proposed code.
\end{abstract}

\pacs{Valid PACS appear here}% PACS, the Physics and Astronomy
                             % Classification Scheme.
%\keywords{Suggested keywords}%Use showkeys class option if keyword
                              %display desired
\maketitle

%\tableofcontents
\section{INTRODUCTION}

We may remember from algebra that we can define lists as lists comprehensions, which use a special notation called set-builder notation. For example,

	\[
		\{(x,y) \mid x \in \mathbb{R} \wedge y \in \mathbb{R} \wedge x * y = x + y\}
	\]

	denotes the set of all pairs \((x,y)\) such that $x$ and $y$ are real numbers and the product of both numbers equals the sum.\\

	Today, this syntactic construct is integrated in many programming languages, such as Haskell

	\begin{lstlisting}[frame=none]
[x * 2 | x <- [0 .. 99], x * x > 3]
	\end{lstlisting}

	python

	\begin{lstlisting}[frame=none]
S = [2*x for x in range(100) if x**2 > 3]
	\end{lstlisting}

	C\#

	\begin{lstlisting}[frame=none]
var ns = Enumerable.Range(0, 100)
        .Where(x => x*x > 3)
        .Select(x => x*2);
	\end{lstlisting}

	Scala

	\begin{lstlisting}[frame=none]
val s = for (x <- Stream.from(0);
        if x*x > 3) yield 2*x
	\end{lstlisting}

	Ruby

	\begin{lstlisting}[frame=none]
(0..100).select { |x| x**2 > 3 }
        .map { |x| 2*x }
	\end{lstlisting}

	and more. But the Java language does not provide a syntactic construct for this concept.\\

	The usage of list comprehensions in Haskell motivated this article. Bringing the set-builder notation to Java, this \emph{way of thinking problems}. Instead of using \emph{for} or \emph{while} statements in your programs, you may work with \emph{map} and \emph{filter} functions, but take into account that thinking problems as list comprehensions is also another good way to practice programming.\\

	The structure of this article follows:\\
		1. Set-builder notation and list comprehensions definitions as we saw in algebra.\\
		2. Some words about the implementation of Haskell's list comprehensions. The implementation of \emph{map} and \emph{filter} with list comprehensions.\\
		3. The proposed implementation of list comprehensions in Java. The implementation of \emph{map} and \emph{filter} with our new list comprehensions.\\
		4. Discussion. Is this useful at industrial level? Possible enhancements and future work.\\

\section{SET BUILDER NOTATION}

	In set theory there is a popular way of defining sets: \emph{set-builder notation}. Any list defined in this way is called a list comprehension.
    For example, let's define the set of all integers that are even and are bigger or equals than 5

    \[
    \{ x \mid x \in \mathbb{Z} \wedge x  \textrm{ is even} \wedge x \geq 5 \}
    \]

    Which is read as \emph{give me the set of all $x$ such that $x$ belongs to the set of the integer numbers, $x$ is even and $x$ is greater than 5}.

    Going formal, a set is composed by three sections: a variable, a colon or vertical bar separator, and a logical predicate, which are contained in curly brackets.\cite{wikipedia}.

    We may also quantify variables, either by using the existential quantifier,  like this definition of the set of all natural even numbers

    \[
    \{ k \mid k \in \mathbb{N} \wedge \exists n \in \mathbb{N}, k = 2n\}
    \]

    or the universal quantifier, negations, etc.

    Sometimes, we have to decide if we want our set-builder notation to be able to quantify not only under elements that belong to a set, but also under sets. If we do this, we may encounter some situations like the one described as the Russell's Paradox

    \[
    \{ S \mid S \notin S \}
    \]

    which is read as \emph{the set of all sets S such that S does not contain themselves}.

    So let's write the functions \emph{map} and \emph{filter} with the set-builder notation we learned. For the first one, we want the set of all the elements that are in a set S but with some transformation applied, let's say the function $f : S \longrightarrow U$. We could write our version of \emph{map} like this(with some syntax sugar on the variable section)

    \[
    \{ f x \mid x \in S\}
    \]

    And now we can do the same for the \emph{filter} function

    \[\{ x \mid x \in S \wedge p x \}\]

    where $x$ must belong to the set S and hold the predicate $p:S \longrightarrow bool$.

\section{LIST COMPREHENSIONS IN HASKELL}

	Haskell is one of the programming languages that supports writing list comprehensions. Let's consider the example

    \begin{lstlisting}[frame=none]
    [toUpper c | c <- s]
    \end{lstlisting}

	where s :: String is a string such as "Hello". Strings in Haskell are lists of characters; the generator $c <- s$ feeds each character of s in turn to the left-hand expression toUpper c, building a new list. The result of this list comprehension is "HELLO".\cite{haskellwiki}

    And with multiple generators, we have

    \begin{lstlisting}[frame=none]
    [(i,j) | i <- [1,2], j <- [1..4]]
    \end{lstlisting}

	yielding the result

    \begin{lstlisting}[frame=none]
    [(1,1),(1,2),(1,3),(1,4),(2,1),(2,2),
   (2,3),(2,4)]
    \end{lstlisting}

    In Haskell, list comprehensions are translated into equivalent definitions in terms of \emph{map}, and \emph{concat}\cite{wadler}. The translation rules are

    \begin{lstlisting}[frame=none]
[e |True] = [e]
[e | q] = [e | q, True]
[e | b, Q] = if b then [e | Q] else []
[e | p <- xs, Q] = let ok p = [e | Q]
                       ok _ = []
                   in concat (map ok xs)
    \end{lstlisting}

    Now, we can write the functions \emph{map} and \emph{filter} as a list comprehensions as we did on the previous section

    \begin{lstlisting}[frame=none]
       map f xs = [f x | x <- xs]
    filter p xs = [x | x <- xs, p x]
    \end{lstlisting}

\section{LIST COMPREHENSIONS IN JAVA}

	Here is the proposed solution for list comprehensions in Java, where a new class ListComprehension has been added.

    \[
    \{ x \mid x \in \{1,2,3,4\} \wedge x \mbox{ is even} \}
    \]

    would be now expressed in Java as

    \begin{lstlisting}[frame=none]
Predicate<Integer> even = x -> x % 2 == 0;

List<Integer> evens =
          new ListComprehension<Integer>()
  .suchThat(x -> {
     x.belongsTo(Arrays.asList(1, 2, 3, 4));
     x.is(even);
});
    \end{lstlisting}

    Note that we are using lambdas here, so Java 8 is required.
    One of the main goals when defining this Java API is that we want the user to read this code and read the same concepts as in the set-builder algebraic notation: \emph{give me the set (or list comprehension) of all x such that x belongs to the list of [1,2,3,4] and x is even}.

    And if you want to modify the output of the expression with some function, for example

    \[
    \{ x * 2 \mid x \in \{1,2,3,4\} \wedge x \mbox{ is even} \}
    \]

    we can write it in Java as

    \begin{lstlisting}[frame=none]
List<Integer> duplicated =
          new ListComprehension<Integer>()
  .giveMeAll((Integer x) -> x * 2)
  .suchThat(x -> {
     x.belongsTo(Arrays.asList(1, 2, 3, 4));
     x.is(even);
});
    \end{lstlisting}

    Now we can introduce our implementation of the functions \emph{map} and \emph{filter} with our new library

    \begin{lstlisting}[frame=none]
public List<T> map(List<T> list,
                   Function<T,T> f) {
        return new ListComprehension<T>()
          .giveMeAll((T t) -> f.apply(t))
          .suchThat(s -> {
             s.belongsTo(list);
        });
}

public List<T> filter(List<T> list,
                      Predicate<T> p) {
        return new ListComprehension<T>()
          .suchThat(s -> {
             s.belongsTo(list);
             s.holds(p);
        });
}
    \end{lstlisting}

	We have also the method \emph{is} as an alias of \emph{holds}.

    The full code is in the Apendix A and hosted at https://github.com/farolfo/list-comprehensions-in-java. Note how the code does not contain any \emph{for}, \emph{while} or \emph{if}, only \emph{maps} and \emph{filters}.

\section{DISCUSSION}

	We got able to implement the functions \emph{map} and \emph{filter} using only list comprehensions written in set-builder notation, Haskell and Java code, the latest one with an implementation of this article.

	Our new implementation lacks of some functionality though, such as quantifiers \emph{exists} and \emph{for-all} under elements, quantifiers under sets (which we have in set-builder notation), among others. Could it be possible to express quantification under sets in the Java implementation? How the Russells's Paradox case would behave?

    Another thing to take into consideration is the usage of list comprehensions in production code. We have seen how Java adopted some functional programming fundamentals in its newer releases (Java 8), such as lambdas, \emph{map}, \emph{filter}, etc; may be is not too crazy to think that Java will adopt list comprehensions at some point, as Scala, Clojure and other programming languages have done.

\begin{thebibliography}{9}
\bibitem{wikipedia}
Wikipedia, \emph{Set-builder notation} \url{https://en.wikipedia.org/wiki/Set-builder_notation}

\bibitem{haskellwiki}
Haskell Wiki, \emph{List Comprehension} \url{https://wiki.haskell.org/List_comprehension}

\bibitem{wadler}
Philip Wadler, \emph{Comprehending Monads} University of Glasgow

\end{thebibliography}

\end{document}


\section{Comparing Storkey and Hebbian learning}

When training a Hopfield network, we have employed two types of learning: the
first one is the Hebbian learning, while the second one is Storkey learning.
The advantage of the latter is that it increases the capacity of the network
and the basin of attraction of the clusters. We will now explain some
experiments we did with both types of learning in order to see how the
learning type affects the basin of attraction for clusters.

We will now show how the learning determines the basin of attraction of a
Gaussian distributed cluster. The generation was done using the T2 method
described in section ~\ref{sec:fexp}.

\begin{table}[h]
\centering
  \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    \tmtextbf{Learning} & N & Cluster size & $\mu$ & $\sigma$ & Average size
    of basin of attraction\\
    \hline
    Hebbian & 50 & 2 & 25 & 5 & 10.0\\
    \hline
    Storkey & 50 & 2 & 25 & 5 & 18.0\\
    \hline
    Hebbian & 50 & 4 & 25 & 5 & 1.5\\
    \hline
    Storkey & 50 & 4 & 25 & 5 & 4\\
    \hline
    Hebbian & 50 & 4 & 25 & 10 & 5.75\\
    \hline
    Storkey & 50 & 4 & 25 & 10 & 8.0\\
    \hline
    Hebbian & 50 & 5 & 25 & 5 & 4.4\\
    \hline
    Storkey & 50 & 5 & 25 & 5 & 4.8\\
    \hline
    Hebbian & 50 & 6 & 25 & 5 & 4.2\\
    \hline
    Storkey & 50 & 6 & 25 & 5 & 4.4\\
    \hline
    Hebbian & 50 & 6 & 25 & 10 & 4.45\\
    \hline
    Storkey & 50 & 6 & 25 & 10 & 5.61\\
    \hline
  \end{tabular}
  \caption{Results comparing Storkey and Hebbian learning for various parameters}
\end{table}

The results confirm what the mathematical theory showed us: Storkey learning
increases the average basin of attraction. We must note that this does not
come without a price: training the network using Storkey learning slowed down
our experiments, as it is more expensive. Depending to the application, this
is an acceptable trade off. All our functions and experiments can be performed
with both types of learning, just by changing a parameter (the learning type),
which enables any user of our libraries to make the choice depending on the
use case.

The following results show the difference between Storkey and Hebbian learning in the experiments we talked about before.
The aim of repeating the experiments with Storkey learning was to see what kind of impact the learning type has in respect to the average basin size.
Both curves (given by Hebbian and Storkey learning) follow the same shape, which also gives confidence in the initial results obtained with Hebbian learning: an initial fast drop in
average basin size, with a steadily increase afterwards.
\\*
The best way to express out results is by showing the two different trends, obtained with the 2 methods we have employed: T1 and T2.

\subsection{T1 Method}

For the T1 method, one can easily notice that Storkey learning gives bigger basin sizes. While this was expected when experimenting with one cluster
 (\ref{fig:plot-storkey-T1-onecluster}), the surprise comes from the second chart (\ref{fig:plot-storkey-T1-twoclusters}): the  increased basin size in the second cluster. Even though the network is trained with 2 clusters
and they affect each others basin sizes, both sizes increase with Storkey learning, without it giving a different ration between the average.

%T1

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T1-onecluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to a cluster generated using T1.}
\label{fig:plot-storkey-T1-onecluster}
\end{figure}

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T1-twocluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to two clusters generated using T1. The first cluster has fixed $p = 0.2$}
\label{fig:plot-storkey-T1-twoclusters}
\end{figure}

%T2
\clearpage
\subsection{T2 Method}

The results are different when looking at T2 type experiments:
the difference between the two learning methods is less straight forward.
When training a Hopfield network with one cluster (\ref{fig:plot-storkey-T2-onecluster})  there is a critical point from which the Storkey learning clearly gives a bigger basin size
(at a standard deviation approximately 5.0), but before that it seems like the two methods give very similar results.
\\*
Even more interesting results come from training the network with two clusters (as seen in \ref{fig:plot-storkey-T2-twoclusters}): surprisingly, when the standard deviation of the second cluster
is less than 9, the Hebbian learning provides a bigger basin size. After that point, Storkey learning acts as expected, by giving a
bigger basin size when compared to Hebbian. Note that the threshold point of this crossover is very close to the standard deviation of the first
cluster, which was fixed to 10.
\\* These results enables us to speculate that Storkey learning is more sensitive to non independent patterns. Both the results from T1 experiments with
2 clusters and the T2 experiments show this. However, we believe that more investigation needs to be done in order to have confidence in these results.

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T2-onecluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to a cluster generated using T2.}
\label{fig:plot-storkey-T2-onecluster}
\end{figure}

\begin{figure}[!ht]
  \centering
  \input{plot-storkey-T2-twocluster-100}
\caption{Comparing the impact of different learning methods on the average basin size of patterns belonging to two clusters generated using T2. The first cluster has fixed $\sigma = 10$}
\label{fig:plot-storkey-T2-twoclusters}
\end{figure}