% Alek Westover writeup draft

\documentclass[twocolumn, twoside, 12pt]{article}
\author{\vspace{-2ex}Alek Westover\vspace{-2ex}}
\title{\vspace{-8ex}Cache Efficient Parallel Partition\vspace{-2ex}}
\date{\vspace{-2ex}June 26, 2019\vspace{-2ex}}

\renewcommand{\thesubsection}{\thesection.\alph{subsection}}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{amsfonts}

\usepackage{amsmath}
% defines a new function
\def\polylog{\operatorname{polylog}}
\def\pred{\operatorname{pred}}
\def\E{\operatorname{\mathbb{E}}}

\usepackage{mathtools}
\DeclarePairedDelimiter{\floor}{\lfloor}{\rfloor}
% \floor*{\frac{x}{y}} actually does a nice looking floor!
\DeclarePairedDelimiter{\paren}{(}{)}
% paren*{ } gives the correct size
\newcommand{\defn}[1]       {{\textit{\textbf{\boldmath #1}}}}

\renewcommand{\paragraph}[1]{\vspace{0.09in}\noindent{\bf \boldmath #1.}}

\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{algorithm, algpseudocode}

%%%% Theorem stuff
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}

\begin{document}
\maketitle
\begin{abstract}
	This is a modified version of the "Algorithm Overview" paragraph in the "Analysis of Grouped Partition" section of the other writeup that I am doing.
\end{abstract}

\paragraph{Algorithm Overview}
We now describe the \defn{Grouped Partition} algorithm.
We logically divide the array $A = A[0],A[1],\ldots, A[n-1]$ into blocks $P_j$ each of $b$ adjacent elements, where $b$ is the \defn{block size}.
Note that the $P_j$s are defined in a different way then in the Strided algorithm when $P_j$ contained elements spaced throughout the array.
This is equivalent to setting $|P_j| = 1$ for the $P_j$s in the Strided Algorithm and then making $P_j$ a collection of cache blocks.
Define \defn{$\pred(j)$} to be the number of predecessors in $P_j$.
There are $n/b$ of these blocks $P_j$.

The algorithm generates a random array $X=X[0],X[1],\ldots,X[s-1]$ where each element in $X$ contains an integer chosen randomly at uniform from $[0,g-1]$, where $g$ is the number of groups.
The values in $X$ determine $g$ groups $G_0, G_1, \ldots, G_{g-1}$ of $P_j$s.
The first group is
\begin{multline*}
	G_0 = \{X[0], X[1]+g, X[2]+2\cdot g,\\
	\ldots,X[s-1]+ (s-1)\cdot g\}.
\end{multline*}
This means that group $G_0$ is a collection of indices for specific parts $P_j$, indicating that these parts belong to group $G_0$. Similarly, group $G_y$ is defined as
\begin{multline*}
	G_y = \{(X[0]+i) \bmod g, (X[1]+i) \bmod g + g, \\
	\ldots, (X[s-1]+i) \bmod g + (s-1)\cdot g \}.
\end{multline*}
Intuitively this means that, for group $G_y$, on each chunk of size $g$ of the array, take $X[j]$ and add the group's index $i$ (wrapping around if there is overflow beyond the number of groups) to get to the index of the $P_j$ that belongs to group $G_y$ from this chunk of the array.
Note that we do not need to store the indices of each $P_j$ that belongs to each group because between $X$ and the group index the $P_j$s that belong to the group $G_y$ are uniquely determined.
We call our algorithm in-place because the size $s$ of X is made $\Theta(\frac{\log n}{\delta^2})$, for $\delta \in (0, 1)$ of our choice, which is minuscule compared to the array of size $n$ that is an input to the partition problem.

Define $U_y$ to be the union of all parts that belong to group $G_y$, that is
$$U_y = \bigcup_{j\in G_y} P_j.$$
Define $\mu_y$ to be the number of predecessors in $U_y$ divided by the number of elements in $U_y$. This is analogous to the definition of $\mu$: the number of predecessors in $A$ divided by $n$.

Once the algorithm has generated $X$, it performs a serial partition on each collection $U_y$ in parallel.
After partitioning the collection $U_y$ the Grouped Partition algorithm stores the index $v_y$, which is the index in $A$ of the first successor in $G_y$, in array of size $s$.
Recall that $s$ is small, and that the algorithm already created an array $X$ of size $s$, so this does not change the asymptotics for memory consumption of the algorithm.
After the Grouped Partition algorithm finishes performing in parallel a serial partition of each group, the algorithm computes in serial $v_{min} =\min v_y$, and $v_{max} = \max{v_y}$ from the stored values $v_y$ generated by each serial partition.
Note that we cannot just have a variable that all threads can write to that stores the current lowest $v_y$ and current highest $v_y$ and then overwrite this in each thread when it finishes its group as we would try in a serial algorithm because this would cause data races in the parallel algorithm.
Thus, because for all $y$
$$v_{min}\leq v_{y} \leq v_{max},$$
we can determine that all elements of $A$ with index less than $v_{min}$ are predecessors and elements of the array with index greater or equal to than $v_{max}$ are successors.
Thus, by recursing on the subarray $A[v_{min}],\ldots,A[v_{max}-1]$, we complete the partitioning of the array.
We recursively apply the Grouped Partition algorithm to the subproblem.
The base case for the recursion is that when the algorithm can no longer make a substantial number of groups, in which case it partitions its input array in serial.

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{natbib}

\title{Weekly Report}
\author{Junior Team }
\date{July 2020}

\begin{document}

\maketitle

\section*{Introduction}

\section{Scaling Cost}
One idea that was proposed was to scale the differences of angular velocities ($\dot{\theta}_{observed}$ and $\dot{\theta}_{predicted}$) in the cost function used in \textit{fmincon}. The previous method did not account for units, and since the angular velocity changes were numerically much larger, they could dominate the optimization. In order to add a scale to our cost function, Dr. Posa suggested we use this formula:
\begin{gather}
cost = \frac{|| D *(v_{observed} - v_{predicted})||^2}{||D * v_{observed}||^2} \\[2ex]
D =
    \left [
        \begin{array}{ccc}
             1 & 0 & 0\\
             0 & 1 & 0\\
             0 & 0 & scale\\
        \end{array}
    \right ]
\end{gather}
\begin{figure}[h!]
\centering
\includegraphics[scale=0.15]{scaled.jpg}
\caption{Scaling $\dot{\theta}$ Differences for IRB}
\label{fig:scaled}
\end{figure}

\noindent As the figure above shows (Fig. \ref{fig:scaled}), scaling theta dot in the updates cost function (Eq. 1) does not lower the error given by the original cost function. When Dr. Fazeli and his coauthors chose to base their error comparison on the normalized $l_2$ norm of velocity, they made an implicit choice on their definition of accuracy. Since we have been using Dr. Fazeli's data as well as referring mostly to his papers, we have also stuck to their cost function. Depending on the application of interest however, it may make sense to update the cost function to prioritize certain needs. Perhaps also, there is a better metric for judging the effectiveness of impact models.

\section{Nonzero Error for IRB with Width }
For quite some time, Dr. Posa has been asking us why some of our IRB plots with width still have significant error, Since we have three equations and three variables we are optimizing over, we should be able to get nearly 0 cost, as long as we aren't running into any major issues with the energy constraints. It turns out some of us had different settings in \textit{fmincon} but once we all switched our step tolerance to $e^{-10}$ we were getting errors basically equal to zero (ignoring numerical and a few outliers). \\

\noindent When comparing our errors (Fig. \ref{fig:wIRB}) with the plot from \citet{nimaPaper} (Fig. \ref{fig:cIRB}), the IRB with width's error is effectively zero compared to the classical IRB which has an expected error of 0.6.

\begin{figure}[ht]
    \caption{Comparison of Errors for IRB With and Without Width}
    \centering
    \begin{subfigure}[b]{0.45\linewidth}
        \includegraphics[scale=0.45]{nima1.jpg}
        \caption{Classic IRB\cite{nimaPaper}}
        \label{fig:cIRB}
    \end{subfigure}
    \quad
    \begin{subfigure}[b]{0.45\linewidth}
       \includegraphics[scale=0.12]{errorWidth.jpg}
        \caption{IRB with Width}
        \label{fig:wIRB}
    \end{subfigure}
\end{figure}

\bibliographystyle{plainnat}
\bibliography{references}

\end{document}

Weekly Reports/Weekly Report 705/main.tex


\documentclass[12]{scrartcl}
\usepackage{amssymb,amsmath,gensymb,dsfont,calc,multicol,fullpage}
\makeatletter
\newcommand\Aboxed[1]{
   \@Aboxed#1\ENDDNE}
\def\@Aboxed#1&#2\ENDDNE{%
   &
   \settowidth\@tempdima{$\displaystyle#1{}$}
   \setlength\@tempdima{\@tempdima+\fboxsep+\fboxrule}
   \kern-\@tempdima
   \boxed{#1#2}
}
\makeatother

\begin{document}

\title{Homework 3, Section 1.3: 1-11 odd, 17-25 odd}
\author{Alex Gordon}
\date{\today}
\maketitle
\section*{Homework}
\subsection*{1.}
$ u =  \begin{bmatrix} -1&2 \end{bmatrix} $
$ v =  \begin{bmatrix} -3&-1 \end{bmatrix} $
\subsection*{3.}
$ u =  \begin{bmatrix} -1&2 \end{bmatrix} $
$ v =  \begin{bmatrix} -3&-1 \end{bmatrix} $
\subsection*{5.}
$3x_1 + 5x_2 = 2$
$-2x_1 + 0x_2 = -3 $
$8x_1 + -9x_2 = 8 $
\subsection*{7.}
$a = u - 2v$
$b = 2u - 2v $
$c = 2u - 2.5v $
$d = 3u - 4v$
\subsection*{9.}
$ x_1 \begin{bmatrix} 0&4&-1 \end{bmatrix} + x_2 \begin{bmatrix} 1&6&3 \end{bmatrix} + x_3 \begin{bmatrix} 5&-1&-8 \end{bmatrix} = \begin{bmatrix} 0&0&0 \end{bmatrix}  $

\subsection*{11.}
The linear system corresponding to M has a solution so 1 has a solution and b is therefore a linear combination of $a_1, a_2$ and $a_3$.
\subsection*{17.}
The weights are 1 and -1
\subsection*{19.}
If $v_1$ and $v_2$ are nonzero vectors then $Span \{v_1, v_2\}$ for the given vectors is the set of points on a line through $v_1$ and $0$.
\subsection*{21.}
The matrix corresponds to a consistent system for all $h$ and $k$, so $y$ is in Span of $\{u,v\}$
\subsection*{23.}
Span $\{u,v\}$ = span $\{u\}$
\subsection*{25. A)}
There are only three vectors in the set of columns $\{a_1, a_2, a_3\}$ in $A$ and $b$ is not one of them.
\subsection*{25. B)}
There are infinitely many vectors in $W = Span \{a_1, a_2, a_3\}$
\subsection*{25. C)}
$a_1$ is in $W$.

\end{document}


\documentclass[a4paper]{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{textcomp}
\usepackage[english]{babel}
\usepackage{amsmath, amssymb}
\usepackage{physics}
\usepackage{isomath}
\usepackage{listings}

% figure support
\usepackage{import}
\usepackage{xifthen}
\pdfminorversion=7
\usepackage{pdfpages}
\usepackage{transparent}
\newcommand{\incfig}[1]{%
	\def\svgwidth{\columnwidth}
	\import{./figures/}{#1.pdf_tex}
}

\DeclareMathSymbol{\C}{\mathalpha}{AMSb}{"43}
\DeclareMathSymbol{\R}{\mathalpha}{AMSb}{"52}

\pdfsuppresswarningpagegroup=1
\title{EE 636 - Matrix Computations}
\begin{document}
\section{Introduction}
The following are course notes made during the live lectures of
EE 636, taken by Prof. Debasattam Pal (Spring 2021)
\textbf{Logistics}
\begin{itemize}
	\item Google Classroom - join codes shared on Moodle
	\item login to classroom regularly to check for assignments, announcements
	\item live lectures
	\item do not rely on the lectures being recorded
\end{itemize}
\textbf{Reference books}
\begin{itemize}
	\item David Watkins - Matrix Computations (Main)
	\item Golub - Matrix Compuations
\end{itemize}
\textbf{Grading Policy}
\begin{itemize}
	\item TA Proctored quizes - 20\%
	\item Midsem - 20\%
	\item Endsem - 40\%
	\item Assignments - 10\%
	\item TakeHomeExam/viva/project/coding - 10\%
\end{itemize}

Take home exams will be significantly harder than assignments, which
will be standard problems. Deadlines will be stricter for take-home-exams.

Learning more about HPC in specific will require you to look into
topics that are beyond the scope of the course, but are in the
reference book.

As such the main aim of this course is to learn what happens behind
the scenes when we call library functions and learn how to write
better code. \textbf{It is important that we understand when the result
of a computation can be trusted or not.} We will see when a computer
is prone to make errors. It has something to do with the condition
number of the matrix.

Condition number is defined as
\[
	K_2(A) = {\norm{A}_2}{\norm{A^{-1}}_2}
.\]

We will spend a lot of time seeing how to solve $Ax = b$. We might
not look at specific examples, but the content covered will apply
more or less directly in some use cases e.g. image processing.

Starting off with a small simple assignment, to be submitted before
the next lecture.
\textbf{Assignment}
\begin{itemize}
	\item Go through the syllabus
	\item Write briefly about a problem (from your area of expertise) that needs knowledge from any of the syllabus topics
	\item Feel free to come up with multiple examples. The more examples $\implies$ more credits.
	\item Submission due by next Monday (11th Jan)
	\item To be submitted on the classroom
	\item submission format \texttt{.pdf}
\end{itemize}

Professor asked us here what exactly are we looking for in this course.
Some answers were given by students. The main purpose of this course
is to understand what happens behind the scenes when we call linalg
library functions, and consequently understand whether a particular
computation is trustworthy or not.

\subsection{Types of Problems in Matrix Computations}
%%%%% TODO: reformat, resolve TODOs.
\subsubsection{$\mathrm{A}\vec{x} = \vec{b}$}
where we solve for $x$, where $A \in \mathbb{R}^{m\times n}
		,b \in  \mathbb{R}^{m}
		,x \in \mathbb{R} ^{n}$. Also, note that $\R^{n} = \R^{n\times 1}$.

\subsubsection{$\text{argmin}_{\vec{x}}\norm{\mathrm A \vec{x} - \vec{b}}$}
	Sometimes we cannot find solve for $\vec{x}$ exactly, in which
	case we would like to minimize some norm of the above kind.
\subsubsection{$\mathrm a \vec{x}= \lambda \vec{x}$}
Here we need to solve for both $\vec{x}$ and $\lambda$. This is a
very important class of problems and also not straightforward. The
characteristic equation to be solved is
\begin{equation}\label{chareq}
	\left| \mathrm A - \lambda \mathbb{1} \right|  = 0
\end{equation}
And then we have the older problem of finding $\vec{x}$, the eigenvectors.
The equation \ref{chareq} is basically finding the roots of a polynomial,
which is very nontrivial ($\because$ in general there is no closed
form for degrees $\ge 5$, as shown by Henry Abel.)

Something called the QR algorithm can find both eigvals and eigvecs
in one shot.

\subsubsection{$\mathrm A \vec{u} = \mathrm \sigma \vec{v}$}
$\mathrm A$ is known, the rest are unknown. And
\begin{align}
	&\vec{u_1}\quad \vec{u_2}\ldots\vec{u_n} \\
	&\vec{v_1}\quad \vec{v_2} \ldots \vec{v_m} \\
	&\vec{u_i}
\end{align}
\begin{itemize}
	\item Singular value decomposition
	\item
\end{itemize}

Questions that we are concerned with, given some problems
\begin{itemize}
	\item How does a computer solve them?
	\item How trustworthy are the solutions?
	\item How efficient are the algorithms?
\end{itemize}

Something was said about the etymology of the word ``algorithm". The
prof then talks about the Caesar cypher.

The letter ``E" is used most often in English. If you have a very
large encrypted text, you could find the Caesar cypher key by
finding the most frequent letter in the encrypted text. Pretty neat
huh.

The enigma machine would randomly choose a key every day and the
decoder machine could automatically figure out the key from the
encrypted text.

Let's get down to computations and all. What does a computer do when
you tell it to $\mathrm{A}\vec{x} = \vec{b}$.
 \begin{equation}
	 \begin{bmatrix} a_{11} & a_{12} & a_{13} \ldots &a_{1n} \\
	 a_{21} & a_{22} & a_{23} \ldots &a_{2n}\\
	 \hdotsfor{4} \\
	 a_{m1} & a_{m2} & a_{m 3} \ldots &a_{mn}
 \end{bmatrix}
 \begin{bmatrix} x_1\\ \vdots\\ x_n \end{bmatrix}
\end{equation}

Internally, it computes something like
\begin{equation}
	\begin{pmatrix} \sum_{i=1}^{n} a_{1i}x_i\\ \vdots\\ \sum_{i=1}^{n} a_{mi}x_i \end{pmatrix}
\end{equation}
%%%% TODO: spent too much time figuring out the matrix snippets. Complete this part

\textbf{Task:} Compute the number of tasks required in matrix vector
multiplication. Here, task means FLOP. So, find the number of FLOP.
Prove that rowwise FLOP $= 2mn$ and columnwise FLOP $= 2mn$ and
$\mathrm{A}\vdot B$ FLOP $= 2mnp$

\section*{Summary}
The purpose of this course is to
\begin{itemize}
	\item learn how a computer can solve
the four types of problems mentioned a
	\item Effect of round-off errors
	\item When can a solution be trusted
	\item FLOPS to measure computation time
	\item Counted the FLOPS for basic matrix multiplication: $2mn$ for a matrix of size $m\times n$ and a vector of dim $n$.
\end{itemize}

\section*{Solvinig a linear system of linear equations}
\subsection*{Triangular Matrix}
\begin{equation}
	\mathrm{A}\vec{x} = \vec{b}
\end{equation}
where $\mathrm A \in \R^{m \times  n}$, $b \in  \R^{n}$. The first
step to learning how to solve such problems is learning about
triangular matrix. Consider the following system with a lower
triangular matrix
\begin{equation}
	%%%%% TODO: Complete
	\begin{bmatrix} l_{11} & 0 & 0 & \ldots & 0\\
		l_{21} & l_{22} & 0 & \ldots & 0 \\
		l_{31} & l_{32} & l_{33} & \ldots & 0\\
	\hdotsfor{5} \\
	l_{n 1} & l_{n 2} & l_{n 3} &\ldots & l_{nm} \end{bmatrix}
%	\begin{bmatrix}  \end{bmatrix} %%%% TODO
\end{equation}

\lstinputlisting{listings/ltr.c}

For the above algorithm (\emph{forward substitution}), FLOPS $= n(n-1) + n = n^2$.
Innermost loop has $2(k-1)$ FLOPS. The loop is called $n$ times for
$k = 1,2,\ldots n$. There is another division operation after a whole
pass over the the inner loop. So, we have $n + \sum_{j=1}^{n} a_n z^n$ TODO. %%% TODO

In general, we can have a column-oriented algorithm for a row-oriented
operation. The corresponding algo is called \emph{backward substitution}.
FLOPs will be the same as the row-oriented business.

\subsection*{The question of solvability}
In real life we do not really get such nice looking lower-triangular
or upper triangular matrices. Consider the following theorem

\textbf{Theorem:} Let $\mathrm A \in \R^{n\times n}$ and $b \in  \R^{n}$ be given. Then the following are equivalent
\begin{enumerate}
	\item $\mathrm A \vec{x}= \vec{b}$ has a unique solution
	\item $\mathrm A^{-1}$ exists
	\item $\text{det} A \neq 0$
	\item The rows of $\mathrm A$ are linearly independent
	\item The columns are linearly independent
	\item $\mathrm A\vec{y} = 0 \iff y = 0$
\end{enumerate}
How do we prove $1 \implies 2$? (Notation: $\vec{e_i}$ are the unit vectors).
$\vec{b} = \sum_{j=1}^{n} a_j\vec{e_j}$ And $\vec{x} = \sum_{j=1}^{n} x_i \vec{e_i}$

\[
	\mathrm A \vec{x} = \sum_{i=1}^{n} x_i \mathrm{A}\vec{e_i}
.\]
LOL IDK what I am doing. %%%% TODO: write full proof

\subsection*{Gaussian Elimination}
You know what Guassian elimination is from MA 214. Summarizing
na\"ive Gaussian elimination here. We use elementary transformation
$\mathrm{E}$ to reduce $\mathrm{A}$ to an upper triangular form
and solve then use backward substitution.

\begin{equation}
	\label{step_1}
	\begin{split}
	\begin{bmatrix}
	1 & 0 & 0 & \ldots & 0\\
	-m_{21} & 1 & 0 & \ldots & 0\\
	-m_{31} & 0 & 1 & \ldots & 0 \\
	\hdotsfor{5}\\
	-m_{n 1} & 0 & 0 & \ldots & 1\end{bmatrix}
	\begin{bmatrix}
	a_{11} & a_{12} & a_{13} & \ldots & a_{1n}\\
	a_{21} & a_{22} & a_{23} & \ldots & a_{2n}\\
	a_{31} & a_{32} & a_{33} & \ldots & a_{3 n}\\
	\hdotsfor{5}\\
	a_{n_1} & a_{n 2} & a_{n 3} & \ldots & a_{n n}\end{bmatrix} \\
	= \begin{bmatrix}
	a_{11} & a_{12} & a_{13} & \ldots & a_{1 n}\\
	0 & a_{22}^{(1)} & a_{23}^{(1)} & \ldots & a_{2 n}^{(1)}\\
	0 & a_{32}^{(1)} & a_{33}^{(1)} & \ldots a_{3 n}^{(1)}\\
	\hdotsfor{5}\\
	0 & a_{n2}^{(1)} & a_{n 3}^{(1)} &\ldots & a_{n n}^{(1)}
\end{bmatrix}
	\end{split}
\end{equation}
where
\begin{equation}
	m_{j 1} = \frac{a_{j1}}{a_{1 1}}, \quad a_{j 1} \neq  0
\end{equation}

This completes one step of procuring an upper triangular matrix. The
next step proceeds by pre-multiplying a similar elementary transformation
matrix to equation (\ref{step_1})
\begin{equation}
	\label{step_2}
	\begin{split}
		\begin{bmatrix}
	1 & 0 & 0 & \ldots & 0\\
0 & 1 & 0 & \ldots & 0\\
0 & -m_{32} & 1 & \ldots & 0\\
\hdotsfor{5}\\
0 & -m_{n 2} & 0 & \ldots & 1
\end{bmatrix}
\begin{bmatrix}
	a_{11} & a_{12} & a_{13} & \ldots & a_{1 n} \\
	0 & a_{22}^{(1)} & a_{23}^{(1)} & \ldots & a_{2 n}^{(1)}\\
	0 & a_{32}^{(1)} & a_{33}^{(1)} & \ldots & a_{3 n}^{(1)}\\
	\hdotsfor{5}\\
	0 & a_{n 2}^{(1)} & a_{n 3}^{(1)} & \ldots & a_{n n}^{(1)}
\end{bmatrix} 	\\
= \begin{bmatrix}
	a_{11} & a_{21} & a_{13} & \ldots & a_{1 n} \\
0 & a_{22}^{(1)} & a_{23}^{(1)} & \ldots & a_{2n}^{(1)}\\
0 & 0 & a_{33}^{(2)} & \ldots & a_{3 n}^{(2)}\\
\hdotsfor{5}\\
0 & 0 & a_{n 3}^{(2)} & \ldots & a_{n n}^{(2)}\end{bmatrix} ,
	\end{split}
\end{equation}
where
\begin{equation}
	m_{j 2} = \frac{a_{j2}^{(1)}}{a_{2 2}^{(1)}}.
\end{equation}

You can see the upper triangular matrix forming. In general, we
have
\begin{equation}
	\label{mij}
	m_{ij} = \frac{a_{ij}^{(j-1)}}{a_{jj}^{(j-1)}}.
\end{equation}

So, Gaussian elimination is basically the process
\begin{equation}
	E_{n-1}E_{n-2}\ldots E_2E_1A = \overline{A}
\end{equation}
where $\overline{A}$ is an upper-triangular matrix. The product
$E_{n -1}E_{n-2}\ldots E_2E_1$ ends up being an upper triangular matrix.

\subsubsection*{When can we proceed with Gaussian elimination?}
Of course, we would like to have $a_{jj}^{(j-1)}$ is non-zero.
Else there will be singularieites as seen in (\ref{mij}).

Of course, checking those values after each step sounds
very inefficient. An equivalent condition is - given $A$, we check
if all of $A_1, A_2, \ldots, A_{n}$ are all invertible, where
\[
	A_k = A(1:k, 1:k)
\]
is the $k\times k$ upper left submatrix of $A$. Formal proof ditch,
we could look at some informal arguments. Consider  $ k = 1$.
$A_1$ is invertible $\implies a_{11} \neq  0$. This could be our base
case.

Now, the most important point here is that $\text{det}A_k \forall k$
is invariant under elementary transformations.
At step $j$, we already have the upper left $k-1 \times  k-1$ matrix
is upper triangular. It can be easily shown that the next elementary
operation would create a $j \times  j$ upper triangular matrix.

Also, the

\input{lec09.tex}
\end{document}


\chapter{Theoretical Framework}
\section{Sequential Decision Making}
In robotics, sequential decision-making problems refer to tasks in which a robot must achieve a goal by interacting with the world as a consequence of decisions that are made sequentially, for example, navigating from one point to another, grasping an object, balancing, etc. It can be described as a step-by-step decision process, where depending on the information that is gathered by its sensors, the robot has to make decisions consecutively.

\subsection[Markov Decision Process]{Markov Decision Process (MDP)}
A Markov Decision Process (MDP) is meant to be a straightforward framing of sequential decision-making problems \cite{puterman2014markov, sutton2018reinforcement}. In an MDP we have an \emph{agent} that is continually interacting with an \emph{environment}. The agent is the learner and the decision maker. The environment is everything that the agent can interact with. In robotics, the agent is the robot and the environment is the real world setting in which the robot exists.

To keep things as simple as possible, MDPs are commonly modeled as discrete time interaction of the agent with the environment \cite{sutton2018reinforcement}. Thus, the time is divided into a sequence of \emph{time steps}, $t=0,1,2,3,...$ . At each time step the agent perceives an \emph{observation} $o_{t}$ of the environment, which is a representation of the environment's \emph{state} $s_{t}$. Depending on the state, the agent selects an \emph{action} $a_{t}$, which will ideally lead to a specific task being solved.

The state describes the current situation of the environment; the action is an input to the environment that the agent can select and that affects the environment's evolution. In robotics, this is information gathered with sensors such as RGB cameras, encoders, inertial measurement units, etc. If the observation contains sufficient information such that the agent can see the entire state of the environment, then the problem is said to be \emph{fully observed} (or fully observable); otherwise, it is said to be \emph{partially observed} (or partially observable). If a partially observed problem is modeled as a sequence of interactions between the agent and the environment, then is called a partially observable MDP (POMDP) \cite{kober2013reinforcement}.

The environment's evolution is the sequence of states visited by the agent. Transitioning from one state to another is a stochastic process defined by the nature of the environment and the actions taken, at each time step, by the agent. Additionally, an MDP is a process that satisfies the Markov property \cite{sutton2018reinforcement}. Consequently, the conditional probability distribution of future states depends only upon the present state, such that:

\begin{equation}
    p(s_{t}|s_{t-1}, a_{t-1}) = p(s_{t}|s_{t-1},...,s_{0},a_{t-1},...,a_{0})  \hspace{0.5cm} \forall t
\end{equation}

Finally, every time the agent executes an action, a numerical \emph{reward} is received by the agent. Better decisions will produce higher cumulative rewards.

In a nutshell, an MDP can be defined by the tuple $MDP=(\mathcal{S},\matchcal{A},\mathcal{R},\mathcal{P})$, where $\mathcal{S}$ represents the state space and $\mathcal{A}$ the action space. $\mathcal{R}$ represents the \emph{reward function}, which generates a scalar reward $r$ every time step. This reward may depend on $s_{t-1}$; $s_{t-1},a_{t-1}$ or $s_{t-1},a_{t-1},s_{t}$, so $\mathcal{R}$ can be defined with $\mathcal{R}: \mathcal{S} \to \R$, $\mathcal{R}: \mathcal{S} \times \mathcal{A} \to \R$ or $\mathcal{R}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \R$, respectively. Finally, $\mathcal{P}$ corresponds to the transition function, which is the probability $p(s_{t+1}|s_{t}, a_{t})$, where $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0, 1]$. Graphically, an MDP can be summarized in the following diagram:

\begin{figure}[H]
    \centering
    \includegraphics[width=0.6\linewidth]{imagenes/cap1/mdp.pdf}
    \caption{Finite part of a Markov Decision Process.}
    \label{fig:msim}
\end{figure}

\section[Reinforcement Learning]{Reinforcement Learning}
Reinforcement Learning aims to use the MDP framing of sequential decision-making problems so that agents learn to solve tasks from \emph{experience} \cite{sutton2018reinforcement}. In this case, experience is understood as the agent-environment interaction (see Figure \ref{fig:agent-environment}), which results in the generation of reward values that indicate the quality of the decisions made by the agent.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\linewidth]{imagenes/cap1/agent_environment.pdf}
    \caption{The agent-environment interaction.}
    \label{fig:agent-environment}
\end{figure}

The idea is to use this experience to learn a \emph{policy} that maximizes the total amount of reward the agent receives \cite{sutton2018reinforcement}. A policy $\pi$ is a function that maps states into actions. This mapping can be either deterministic $\pi: \mathcal{S} \to \mathcal{A}$ or stochastic $\pi: \mathcal{S} \times \mathcal{A} \to [0,1]$. To obtain a good performing policy, it is necessary to adequately combine \emph{exploration} with \emph{exploitation}. When exploring, the agent finds more information about the environment; when exploiting, it exploits known information to maximize the reward.

Commonly, in complex problems, $\pi$ is a parameterized model; then, the goal of reinforcement learning is to find the set of parameters $\theta$ that maximizes the sum of rewards, or \emph{return} $G$, that the agent will obtain when solving a task. As a consequence of the parameterized policy $\pi_{\theta}$, the agent will follow a trajectory $\tau = \{s_{0}, a_{0},...,s_{T}, a_{T}\}$, where $T$ is the number of time steps. When a problem has well-defined initial and final conditions it has a \emph{finite horizon} (the trajectory generated between the initial and final conditions is called \emph{episode}), otherwise, it has an \emph{infinite horizon} \cite{sutton2018reinforcement, kober2013reinforcement}. In the former, $T \in \N$; in the later, $T \to \infty$. Finally, $\tau$ follows the probability distribution $p_{\theta}(\tau)$ that depends on $p(s_{t+1}|s_{t}, a_{t})$, $\pi_{\theta}$ and $p_{0}=p(s_{0})$, which corresponds to the initial distribution of states.

With the above formulation, the goal of reinforcement learning is to \textbf{maximize the expected return} of a given task such that:

\begin{equation}
    J = E_{\tau \sim p_{\theta}(\tau)} \left[\sum_{t=0}^{T}r(s_{t}, a_{t})\right]
\end{equation}

\begin{equation}
    \theta^{*} = \argmax_{\theta}J(\theta)
\label{eq:objective}
\end{equation}

It can be observed that in the infinite horizon case, this definition can be numerically unstable and have divergence problems. To address this, the term $\gamma \in [0, 1)$, known as discount rate, can be introduced to construct a formulation of the goal with \emph{discounted rewards} \cite{sutton2018reinforcement, kober2013reinforcement}:

\begin{equation}
    J = E_{\tau \sim p_{\theta}(\tau)} \left[\sum_{t=0}^{T}\gamma^{t}r(s_{t}, a_{t})\right]
\end{equation}

Where $\gamma$ allows to control the horizon in which the agent maximizes the reward. This rate determines the present value of a reward received in the future, where immediate rewards are worth more than those received later.

\subsection{Learning Policies}

Under this framework, there are different approaches for learning policies \cite{sutton2018reinforcement, kober2013reinforcement}:

\begin{itemize}
    \item Value-Based
    \item Policy Search
    \item Actor-Critic
\end{itemize}

In \textbf{Value-Based} approaches, the policy is learned implicitly by learning a \emph{value function} parameterized by the weights vector $\theta$. This function estimates the expected return of the agent for every state, or in other words, how good is to be in a given state. The value for a given state will depend on the the agent's policy, thus, the value function can be written as:

\begin{equation}
    V^{\pi}(s_{t'})  = E_{\tau' \sim p_{\theta}(\tau')}\left[\sum_{t=t'}^{T}\gamma^{t}r_{t}(s_{t}, a_{t})|s_{t'}\right]
\end{equation}

Where $\tau'$ is the trajectory followed by the agent starting from time step $t'$.

Similarly, the value of taking the action \emph{a} in the state \emph{s} can be estimated by a function, which is called the \emph{action-value function}:

\begin{equation}
    Q^{\pi}(s_{t'}, a_{t'})  = E_{\tau' \sim p_{\theta}(\tau')}\left[\sum_{t=t'}^{T}\gamma^{t}r_{t}(s_{t}, a_{t})|s_{t'},a_{t'}\right]
\end{equation}

The main idea is to find a policy $\pi^{*}$, the \emph{optimal policy}, such that:

\begin{equation}
    V^{\pi^{*}}(s) \geq V^{\pi}(s) \hspace{0.5cm} \forall s, \pi
\end{equation}

\begin{equation}
    Q^{\pi^{*}}(s,a) \geq Q^{\pi}(s,a) \hspace{0.5cm} \forall s,a,\pi
\end{equation}

If these conditions are met, then, the optimal vector of weights $\theta^{*}$ is obtained. On the other hand, \textbf{Policy Search} approaches update $\theta$ by direct policy differentiation. In this case, the challenge is to estimate the gradient of $J(\theta)$ to update $\theta$ iteratively, using gradient ascent:

\begin{equation}
    \Delta \theta = \alpha \nabla_{\theta}J(\theta)
\end{equation}

\begin{equation}
    \theta \leftarrow \theta + \Delta \theta
\end{equation}

Where $\alpha$ is the learning rate.

Finally, \textbf{Actor-Critic} approaches combine Value Based and Policy Search methods. A policy (actor) and a value function (critic) are explicitly learned. In this case, the actor decides the action that the agent should take while the critic observes its performance and decides when the policy needs to be updated and how. Figure \ref{fig:rl_summ} summarizes these RL approaches:

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\linewidth]{imagenes/cap1/rl_summary.pdf}
    \caption{Reinforcement Learning methods.}
    \label{fig:rl_summ}
\end{figure}

Independently of the technique that is used to solve a problem with Reinforcement Learning, the anatomy of these algorithms is always the same: obtain experience (generate samples), fit a model (or estimate a return), improve the policy and repeat (see Figure \ref{fig:RL_anatomy}).

\begin{itemize}
    \item \textbf{Generate samples:} obtain $n$ transitions. A transition is the tuple $(s_{t}, a_{t}, r_{t+1}, s_{t+1})$, which can also be written as $(s, a, r, s')$.
    \item \textbf{Fit a model:} use the gathered samples to improve the estimation of a value function or to estimate $\nabla_{\theta} J(\theta)$.
    \item \textbf{Improve the policy:} use the estimated value function to obtain a policy or $\nabla_{\theta} J(\theta)$ to update $\theta$.
\end{itemize}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.45\linewidth]{imagenes/cap1/RL_anatomy.pdf}
    \caption{Anatomy of Reinforcement Learning.}
    \label{fig:RL_anatomy}
    \vspace{-0.3cm}
\end{figure}

\subsubsection{Model-Free vs Model-Based}
\vspace{-0.3cm}
Reinforcement Learning approaches may be either Model-Free or Model-Based \cite{sutton2018reinforcement, kober2013reinforcement}.

\textbf{Model-Free} algorithms learn a policy directly from experience. The dynamics of the environment are not directly learned.

\textbf{Model-Based} algorithms learn a \emph{model} of the dynamics of the environment from experience. Then, planning or optimal control techniques are applied to develop a policy using the model.

A model of the environment is a function $M: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ that predicts the next state as a function of the current state and action to take:
\vspace{-0.02cm}
\begin{equation} \label{eq:model}
    M(s_{t},a_{t}) = s_{t+1}
\end{equation}
\vspace{-0.02cm}
or, in the stochastic case, $M: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0, 1]$:
\vspace{-0.02cm}
\begin{equation}
    M(s_{t},a_{t}) = p(s_{t+1}|s_{t},a_{t})
\end{equation}

Figure \ref{fig:free_based_model} summarizes the difference between both approaches:

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\linewidth]{imagenes/cap1/free_based_model.pdf}
    \caption{Model-Free vs Model-Based.}
    \label{fig:free_based_model}
\end{figure}

\subsubsection[Experience Replay]{Experience Replay}
\label{sss:ER}

The anatomy of Reinforcement Learning (Figure \ref{fig:RL_anatomy}) suggests that every time the policy is improved, obtaining new samples is necessary to improve it again. This is only true in $\text{\emph{on-policy}}$ algorithms \cite{sutton2018reinforcement}. On-policy algorithms are those in which a policy can only be updated with data gathered by itself. Data gathered before the last update corresponds to data gathered by a past version of the current policy; thus, gathered by a different policy. So if the policy is changed, even a little bit, generating new samples becomes necessary.

In contrast, \emph{off-policy} algorithms are able to improve a policy without the aforementioned restriction \cite{sutton2018reinforcement}; thus, using samples collected by other policies or by past versions of itself. Under this context is where is useful to use an \emph{experience replay} \cite{lin1993reinforcement,atari}.

The key idea of experience replay is to store old transitions $(s, a, r, s')$ in a buffer (or memory buffer) and replay them constantly (randomly sampling a mini-batch) when updating a policy. Using experience replay has two advantages: (1) the data/samples are uncorrelated and (2) data efficiency is significantly improved \cite{zhang2017deeper}:

\begin{itemize}
    \item \textbf{Uncorrelated samples:} every sequential decision-making problem has the issue that the data generated when exploring the environment is correlated. If the data is correlated, then, every time the policy is updated, this will be with respect to a sequence, or various sequences. In some cases this is problematic because models may locally overfit to this data, constantly forgetting older experiences. This issue can be tackled by randomly sampling from a memory buffer.

    \item\textbf{Data efficiency:} depending on the the parameterized model of $\pi_{\theta}$, it may not be efficient to update its parameters with a set of samples just once. To successfully extract all the information contained in the samples, it would be necessary to update the model several times from them. This is achieved when using experience replay.
\end{itemize}

\section[Function Approximation]{Function Approximation}
Although $\pi$ can be represented by a look-up table, where every state has a corresponding action stored in a table, this representation is not scalable to complex problems with large state/action spaces. This is mainly because the states and/or actions are too many to be stored in memory and/or it would take too long to adequately explore the states. Thus, in robotics contexts, where state/action spaces are most likely continuous, the standard approach is to use function approximation \cite{kober2013reinforcement}. As mentioned before, this means that the policy is a parameterized model $\pi_{\theta}$.

There is a large list of different function approximators \cite{sutton2018reinforcement, kober2013reinforcement}, but in this work two of them are the most relevant: \textbf{linear models of basis functions (LCBFs)} \cite{busoniu2010reinforcement} and \textbf{artificial neural networks (ANNs)} \cite{goodfellow2016deep, nielsen2015neural}. Although the policy may not be directly represented by these models, as in the case of value function approximation, for simplicity, from now on we are going to assume that it always does. Also, we are only going to work with deterministic policies. So, if the function approximator is $\Psi$, then $\pi_{\theta} = \Psi(s;\theta)$ such that $\Psi: \mathcal{S} \to \mathcal{A}$.

\subsection[Linear Model of Basis Functions]{Linear Model of Basis Functions}
A well-known case of function approximation in RL is that in which $\pi_{\theta}$ is a linear function of the weight vector $\theta$. Specifically, we are going to work with linear combinations of gaussian radial basis functions (RBFs) \cite{busoniu2010reinforcement}. If $\phi(s)$ corresponds to a vector of basis functions which is weighted by $\theta$, then $\pi_{\theta}$ can be represented as:

\begin{equation}
    \pi_{\theta}(s) = \phi(s) \cdot \theta
    \label{eq:lcbf}
\end{equation}

In this case, $\phi(s)$ corresponds to normalized gaussians:

\begin{equation}
    \phi_{i}(s) = \frac{\phi'_{i}(s)}{\sum_{i'=1}^{N}{\phi'_{i'}(s)}}
    \hspace{0.5cm};
    \hspace{0.5cm}
    \phi'_{i}(s) = \exp\left( -\frac{1}{2}[s - c_{i}]B^{-1}_{i}[s-c_{i}]\right)
\end{equation}

\noindent Where $N$ is the number of basis functions and the sub index \emph{i} refers to the \emph{i-th} basis function.

\subsection[Artificial Neural Networks]{Artificial Neural Networks}

ANNs are a widely used model for non-linear function approximation. The idea behind these models is to build a network of interconnected units that have some properties inspired on neurons. There is a large variety of models \cite{goodfellow2016deep, nielsen2015neural}, the ones relevant to this work are the following:

\begin{enumerate}
    \item Feedforward Fully-Connected
    \item Convolutional
    \item Autoencoder
    \item Recurrent
\end{enumerate}

\subsubsection{1. Feedforward Fully-Connected}

Feedforward Fully-Connected or Feedforward neural networks (FNNs) are the quintessential ANN model \cite{goodfellow2016deep}. These models are called to be feedforward because the information flows in one sense from the input, trough intermediate representations, to the output. In the mathematical formalization, a network is understood as the model generated when many functions are composed. For instance, you may have three functions ($f^{(1)}$, $f^{(2)}$, $f^{(3)}$) connected in a chain such that:

\begin{equation}
    \Psi(s) = f^{(3)}(f^{(2)}(f^{(1)}(s)))
\end{equation}

Each of these functions represents a \emph{layer}, the overall number of layers is known as the \emph{depth} of the neural network. Commonly, these layers have $n\mathrm{-dimensional}$ outputs, and each dimension of the outputs corresponds to the output of a single \emph{neuron}. A neuron is a function $h$ composed by weights $w$, a bias $b$ and a nonlinear function $\sigma$, known as the activation function:

\begin{equation}
    h(x) = \sigma(w \cdot x + b)
    \label{eq:h}
\end{equation}

There are different types of activation functions, such as ReLU, sigmoid or hyperbolic tangent. Graphically, this function is described in Figure \ref{fig:neuron}.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.6\linewidth]{imagenes/cap1/neuron.pdf}
    \caption[Neuron.]{Neuron ($b$ considered as an extra dimension of $w$).}
    \label{fig:neuron}
\end{figure}

As mentioned before, a layer may have many neurons. The number of neurons in a layer is known as the \emph{width} of the layer. The inputs $x$ of a neuron are the outputs of every neuron of the past layer.  There are three types of layers: \textbf{input}, \textbf{hidden} and \textbf{output}. The input layer is the input of $\Psi$, which is $s$ in this context (which is not actually a layer). The output layer is the final function in the chain of functions that compose $\Psi$, which outputs the output of $\Psi$. Finally, the hidden layers are all the functions in between the input and output layers. Graphically, a FNN with one hidden layer is presented in Figure \ref{fig:FNN}.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.4\linewidth]{imagenes/cap1/nn.pdf}
    \caption{Feedforward Neural Network model with one hidden layer.}
    \label{fig:FNN}
\end{figure}

It can be observed that the nonlinearity that characterizes neural networks exists because of the activation functions, which are nonlinear. This is an essential condition for neural networks to be \emph{universal approximators} \cite{cybenko1989approximation, hornik1991approximation}. What this states is that a neural network with at least one hidden layer containing a large enough finite number of neurons can approximate any continuous function on a compact region of the network's input space to any degree of accuracy. This is a powerful property; nevertheless, finding the set of weights that parametrize the network in order to approximate an arbitrary function is another story, which will be discussed in Section \ref{sec:learning}.

Finally, going back to the concept of a parameterized policy $\pi_{\theta}$, in this case $\theta$ would correspond to the concatenation of all the weights (including biases) of the layers of a network. \newline

\subsubsection{2. Convolutional}

Convolutional Neural Networks (CNNs) are a specialized type of neural network. The main feature of these networks is that they are designed such that convolutions are employed in at least one layer \cite{goodfellow2016deep, nielsen2015neural}. In the area of signal processing convolutions are a well-known tool for applying filters to signals, and at the same time, filtering signals is a well-known way of extracting features out of them. The approach proposed by CNNs is to learn the weights of these filters (also known as kernels) as a part of the network's learning process, and that by construction, these filters are applied to the inputs of the convolutional layers.

The way to achieve this is to replace the dot product in Equation \ref{eq:h} by a convolution operation. Usually, in these problems we work with discrete data, so the discrete version of the convolution is used:

\begin{equation}
    s(x)_{i} = (x * w)_{i} = \sum_{l=0}^{L}w_{l} \cdot x_{i+l}
\end{equation}

The sub-index \emph{i} corresponds to the \emph{i-eth} term of the input $x$, and $w$ corresponds to the kernel, which has dimension $L$. It can be observed that the kernel is applied over a region of each index \emph{i}, which means that this operation cares about the order of the data, and as a consequence, these networks are used with inputs that have spatial or time correlations, such as images or time series. CNNs are widely use for image processing problems \cite{liu2017survey}, so it is common to use convolutions over more than one axis at a time. If we have a two-dimensional image as an input it seems natural to use a two-dimensional kernel $K$ as well (to exploit spatial correlations):

\begin{equation}
    s(x)_{i,j} = (x * K)_{i,j} = \sum_{l=0}^{L}\sum_{m=0}^{M}K_{l, m} \cdot x_{i+l, j+m}
\end{equation}

In this case K has dimension $L \times M$. Figure \ref{fig:conv} shows an example of a kernel operation over a pixel.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.57\linewidth]{imagenes/cap1/convolution.pdf}
    \caption{Convolution operation in an image.}
    \label{fig:conv}
\end{figure}

So, in CNNs, Equation \ref{eq:h} has the following shape:

\begin{equation}
    h(x)_{i} = \sigma(s(x)_{i} + b)
    \label{eq:h2}
\end{equation}

Each neuron will correspond to the output of a kernel applied over a region of the input; thus, the output of the full convolution operation will correspond to a set of neurons. As a consequence, the weights, which in this case conform the kernel, will be shared between neurons, which does not happens in the FNN case. It is important to note that it is also common to apply the convolution operation over images with arbitrary depth. For instance, RGB images have depth=3 (red, green and blue channels), so in these cases filters with arbitrary depth may be used as well. Also, considering the assumption that each filter extracts one kind of feature out of a signal, usually, several filters are applied over an input, where every filter generates a different image (set of neurons) as an output. These means that the depth of the output of a convolutional layer will be equal to the number of filters applied.

To modify the output of a convolutional layer further, it is common to use a \textbf{pooling function} \cite{goodfellow2016deep, nielsen2015neural}. This function summarizes the output of the convolution operations replacing every value of the outputs with a statistic of the nearby outputs. One example of a pooling function is the \emph{max pooling}, which takes the maximum value over a rectangular neighborhood, as it can be seen in Figure \ref{fig:maxpool}.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.45\linewidth]{imagenes/cap1/max_pool.pdf}
    \caption{Max pooling operation.}
    \label{fig:maxpool}
\end{figure}

Pooling helps the model to be approximately invariant to translations in the input space. An alternative to pooling, is to use a convolution with \textbf{stride} greater than one. The stride is the distance skipped between values of the input to where the kernel is applied, which in the default case is one (no skipping). It has been observed that replacing pooling by stride$>1$ has no loss in accuracy for several image recognition benchmarks \cite{springenberg2014striving}. Figure \ref{fig:cnn} shows a CNN used for classification.

\begin{figure}[H]
\centering
\includegraphics[width=\linewidth]{imagenes/cap1/cnn.pdf}
\caption{Convolutional Neural Network.}
\label{fig:cnn}
\end{figure}

\subsubsection{3. Autoencoder}

An autoencoder is a special type of ANN that aims to copy the input of the network to its output i.e. aims to be the identity function \cite{goodfellow2016deep, nielsen2015neural}. Although solving this problem may seem to be useless, it has some interesting properties. In this work, we are interested in using autoencoders for dimensionality reduction and feature learning.

This neural network is divided in two parts, the \textbf{encoder $e(x)$} and the \textbf{decoder $d(\mathcal{L})$}. When used for dimensionality reduction and feature learning, the encoder tries to encode the input into a space of lower dimension (a bottleneck), called the \textbf{latent space $\mathcal{L}$}, which represents a compressed representation of the input $x$, $\mathcal{L}=e(x)$. The decoder takes the latent space and tries to reconstruct the input $\widetilde x = d(\mathcal{L})$, where $\widetilde x$ is the reconstruction of $x$, see Figure \ref{fig:ae}. The objective of using autoencoders in this context is to obtain a latent space that captures the most salient features of the input.

\begin{figure}[H]
    \centering
    \includegraphics[width=\linewidth]{imagenes/cap1/ae.pdf}
    \caption{Autoencoder.}
    \label{fig:ae}
\end{figure}

The goal of the autoencoder is to minimize the loss function:

\begin{equation}
\label{eq:ae}
    L(x, \widetilde x = d(e(x)))
\end{equation}

Where $L$ is a dissimilarity function such as the mean squared error. This model may have convolutional, fully-connected, both or other types of layers.

\subsubsection{4. Recurrent}

Recurrent Neural Networks (RNNs), unlike FNNs or CNNs, have memory. They are designed to work with sequential data \cite{goodfellow2016deep, nielsen2015neural}. What this means, that past evaluations of the network affect the ones in the future. RNNs have an internal state, or hidden state $h_{state}^{t}$ ($t$ represents the time step), that is modified every time an input is fed to the network such that:

\begin{equation}
    h_{state}^{t} = f(h_{state}^{t-1}, x^{t}; \theta)
\end{equation}

In the \textbf{vanilla} RNN, one of the simplest models of RNNs, a neuron of a recurrent layer is understood as the $i\mathrm{-eth}$ term of the hidden state, $h_{state\mathrm{-}i}^{t}$. If compared with Equation \ref{eq:h}, an RNN neuron has the following structure:

\begin{equation}
    h_{state\mathrm{-}i}^{t}(x) = \sigma(w_{hh} \cdot h_{state}^{t-1} + w_{xh} \cdot x^{t} + b)
\end{equation}

Where the main difference is that the term $w_{hh} \cdot h_{state\mathrm{-}i}^{t-1}$ is incorporated, and the concatenation of the weights $w_{hh}$ and $w_{xh}$ corresponds to $w$. Figure \ref{fig:rnn_unfolded} represents a vanilla RNN in its \emph{unfolded} representation.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.4\linewidth]{imagenes/cap1/unfolded_rnn.pdf}
    \caption{Unfolded vanilla RNN.}
    \label{fig:rnn_unfolded}
\end{figure}

In RNN contexts, commonly, the hidden state is represented as a whole vector directly, and not by the equations of every neuron. A weight matrix $W_{h}$ is used, and corresponds to the concatenation of the weights of every neuron, including the bias, such that:

\begin{equation}
    h_{state}^{t}(x) = \sigma(W_{hh} h_{state}^{t-1} + W_{xh} x^{t}) = \sigma \left(W_{h} \begin{pmatrix} h^{t-1} \\ x^{t} \end{pmatrix} \right)
\end{equation}

In a vanilla RNN with one hidden state, $\sigma$ is the hyperbolic tangent function $tanh$ and one more operation is added to obtain an output $y$:

\begin{equation}
    y^{t}(x) = W_{hy}h_{state}^{t}(x)
\end{equation}

Nevertheless, vanilla RNNs have some structural problems related to the computation of their gradients (exploding and vanishing gradients) \cite{bengio1994learning, pascanu2013difficulty}. So in practice, more complex RNN architectures are used, such as \textbf{Long Short-Term Memory (LSTM)} \cite{hochreiter1997long} or \textbf{Gated Recurrent Unit (GRU)} \cite{cho2014learning} networks. The architecture relevant to this work is LSTM, and one of its layers can be described with the following equations:

\begin{equation}
    \begin{pmatrix} i \\ f \\ o \\ g \end{pmatrix} = \begin{pmatrix} \sigma_{f} \\ \sigma_{f} \\ \sigma_{f} \\ tanh \end{pmatrix} W \begin{pmatrix} h^{t-1} \\ x^{t} \end{pmatrix}
\end{equation}

\begin{equation}
    c^{t} = f \odot c^{t-1} + i \odot g
\end{equation}

\begin{equation}
    h^{t} = o \odot tanh(c^{t})
\end{equation}

Where $i$, $f$, $o$, $g$ are called \textbf{gates} and each one of them is a vector used for internal computations of the recurrent unit. $\sigma_{f}$ corresponds to the $sigmoid$ activation function, $\odot$ is the element-wise product (Hadamard product) operator and $W$ is a weight matrix containing the submatrices $W_{i}$, $W_{f}$, $W_{o}$ and $W_{g}$. A LSTM layer or unit has two internal states $c^{t}$ and $h^{t}$. $h^{t}$ is still known as the hidden state and $c^{t}$ is known as the cell state.

\subsection{Parameter Tuning}
\label{sec:learning}

Until now, we have discussed that policies can by represented as parameterized models, called function approximators, and that the two most relevant to this work are LCBFs and ANNs. Nevertheless, we have not discussed on how to tune the parameters of these models in order for them to approximate a specific function.

First, it is necessary to define a loss function, in this work we use the \textbf{Mean Squared Error (MSE)}, which gives us a measure of how close are the vector outputs generated by our model $\Psi(s;\theta)=\hat{y}$ with respect to observed target values $y$. Then, the objective is to minimize this function in the parameter space with some optimization method, in this case we use \textbf{Stochastic Gradient Descent (SGD)} \cite{robbins1951stochastic, saad1998online}. Given that this method needs to compute the gradient of the loss function $J(\theta)=MSE(\Psi(\theta))$, it is necessary to compute the gradient of $\Psi(\theta)$.

When using LCBFs, this computation is trivial, such that the gradient of \ref{eq:lcbf} would be:

\begin{equation}
    \nabla_{\theta} \Psi(\theta)_{LCBF} = \phi(s)
\end{equation}

But, in the case of ANNs computing the gradient is not direct. Given that we have a function composed by layers, it is necessary to propagate the error through all of these layers in order to update all of the weights of the network. \textbf{Backpropagation} \cite{rumelhart1988learning} is the technique used to achieve this and, in principle, it uses the chain rule to quantify how much changes in past layers affect later ones. Nowadays, there are several computer tools that have this algorithm incorporated and that automatically compute the gradients of ANNs using symbolic math and differentiable programming, such as Tensorflow \cite{tensorflow2015-whitepaper} or PyTorch \cite{paszke2017automatic}.

\subsection{Deep RL Era}

Reinforcement Learning can be divided in the \emph{pre Deep RL era} and the \emph{Deep RL era}, which is the current one. In the year 2013, the first work known for successfully solving sequential decision-making problems with high-dimensional state spaces using CNNs was presented \cite{atari}. This was a consequence of the new advances that where being made in the area of ANNs or \textbf{Deep Learning (DL)}. When ANNs have several layers, which was the case for the mentioned work, they are called \textbf{Deep Neural Networks (DNNs)}. The area that combines DL with RL is called \textbf{Deep RL}. Deep RL gives agents the possibility to build rich state representations of the environment without feature engineering on the side of the designer, which was always necessary in classical RL. This drew a lot of attention and made Deep RL increasingly popular over the past few years \cite{franccois2018introduction}.

Before the Deep RL era, ANNs where not a predominant function approximator for solving sequential decision-making problems. In the pre Deep RL era, one of the most widely used function approximators were LCBFs, as stated in a survey of the year 2013 \cite{kober2013reinforcement}.

\section[Interactive Machine Learning]{Interactive Machine Learning}

In the context of this work, Interactive Machine Learning (IML) is understood as a research area that gathers all the techniques in which humans/users take part in the agent's learning process for solving sequential decision-making problems \cite{Argall2009, amershi2014power,billing2010formalism, Chernova2014, cuayahuitl2013machine}. Other works refer to IML as an area that focuses on learning classifiers with the help of users \cite{fails2003interactive, ware2001interactive, amershi2012regroup, ngo2014efficient}.

As RL, IML can also be studied under the framework of MDPs \cite{Celemin2018AnInteractive, Knox:2009:ISA:1597735.1597738, Argall2009}. The main difference with RL is that in IML having an explicitly defined reward function is not a requirement. Some approaches may combine the information gathered from the human with the one given by a reward function, while others, may not need a reward function at all.

In this work, we divide IML into two areas: \textbf{Learning from Demonstration (LfD)} \cite{Argall2009, billing2010formalism} and \textbf{Learning from Feedback (LfF)} \cite{Celemin2018AnInteractive, argall2009learning, Knox:2009:ISA:1597735.1597738}.

\subsection{Learning from Demonstration}

LfD is a learning paradigm in which a demonstrator provides labelled data of a desired policy \cite{Argall2009, billing2010formalism}. Then, the agent tries to copy the demonstrator's behavior by shaping its policy in a supervised learning manner, using as database the data provided by the demonstrator (see Figure \ref{fig:b_cloning}). This is the most basic LfD approach and it takes the name of \textbf{Behavior Cloning}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\linewidth]{imagenes/cap1/b_cloning.pdf}
    \caption{Behavior cloning.}
    \label{fig:b_cloning}
\end{figure}

\subsubsection{Imitation Learning}

When the platform of the agent is different to the one in which the demonstrations are made, we talk about \textbf{Imitation Learning}\cite{Argall2009, billing2010formalism}. In this case, the learning agent must identify a mapping between the demonstrator and itself. This is a setting that arises naturally in real-world problems. For instance, let us suppose that we want to teach a robot how to walk. Using a camera, we could record a human making demonstrations (just by walking), but then, the robot should figure out how to map that observations into labeled data over its own state and action spaces. The problem of identifying this mapping is one of the main difficulties of LfD approaches and is known as \textbf{The Correspondence Problem} \cite{nehaniv2002correspondence}.

\subsubsection{State Distribution Mismatch}

The \textbf{state distribution mismatch} (also known as distributional drift) is an issue that every LfD approach could have if is not taken into account. This problem arises because the probability distribution of the trajectory followed by the policy $p_{\pi_{\theta}}$ \emph{drifts} from the one of the demonstrator $p_{data}$ \cite{ross2011reduction}. This happens because even though the agent imitates the demonstrator, it is very difficult for the agent to make a perfect imitation. Thus, small errors over several states accumulate into a larger error that eventually causes the agent to fail. This phenomenon can be observed in Figure \ref{fig:drift}, where the drift is shown over one dimension of the state.

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\linewidth]{imagenes/cap1/drift.pdf}
    \caption{Distributional Drift.}
    \label{fig:drift}
\end{figure}

A well-known algorithm that tackles this problem is called \textbf{DAgger} \cite{ross2011reduction}. This approach matches $p_{\pi_{\theta}}$ with $p_{data}$, such that $p_{\pi_{\theta}} = p_{data}$. The main idea is that instead of collecting samples from the demonstrator, collect them by running $\pi_{\theta}$ and then label them with the demonstrator. Samples are collected, labeled, added to the database $\mathcal{D}$ and used for training iteratively, in order to approach asymptotically to $p_{data}$. The basic structure of DAgger is the following:

\begin{enumerate}
    \item Train $\pi_{\theta}(a_{t}|o_{t})$ from human data $\mathcal{D}=\{o_{1},a_{1},...,o_{T},a_{T}\}$
    \item Run $\pi_{\theta}(a_{t}|o_{t})$ to get dataset $\mathcal{D}_{\pi_{\theta}}=\{o_{1},...,o_{M}\}$
    \item Ask human to label $\mathcal{D}_{\pi_{\theta}}$ with actions $a_{t}$
    \item Aggregate: $\mathcal{D} \leftarrow \mathcal{D} \cup \mathcal{D_{\pi_{\theta}}}$
\end{enumerate}

One of the problems that DAgger presents in real-world applications is that the process of labeling the data can be challenging or unfeasible. For instance, robots commonly execute continuous actions, but for a human is not intuitive to assign a continuous valued label to an observation.

\newpage

\subsection{Learning from Feedback}

We define LfF as the paradigms in where policies are shaped by occasional human feedback. In this case there are no demonstrations; instead, \textbf{corrections} or \textbf{evaluations} are given by a teacher over the decisions made by the learning agent \cite{Celemin2018AnInteractive, Knox:2009:ISA:1597735.1597738, Argall2009}. Figure \ref{fig:LfF} summarizes the LfF approach.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.47\linewidth]{imagenes/cap1/LfF.pdf}
    \caption{Learning from Feedback.}
    \label{fig:LfF}
\end{figure}

\subsubsection{Evaluative Feedback}

Evaluative feedback has been used similarly to RL in methods wherein a human teacher communicates the desirability of the executed action or policy. There are different evaluative feedback approaches, some of them are:

\begin{itemize}
    \item \textbf{Interactive Reinforcement Learning:} RL-based methodologies in which the reward signals are given by a human \cite{thomaz2006reinforcement}.
    \item \textbf{Human Reinforcement Modeling:} A reward function is modeled from the feedback given by a human. Then, this model is used as the reward function in a RL-based approach \cite{Christiano2017, akrour2014programming}.
    \item \textbf{The \emph{shaping} approach:} This is a LfF approach known as TAMER \cite{Knox:2009:ISA:1597735.1597738}. The policy is interactively shaped with evaluations that the teacher provides occasionally.
\end{itemize}

Evaluative feedback approaches have been validated in problems of state spaces of low dimensionality \cite{Knox:2009:ISA:1597735.1597738,akrour2011preference,macglashan2017interactive} and high dimensionality \cite{Christiano2017,Warnell2017}.

\subsubsection{Corrective Feedback}

In corrective feedback approaches, the teacher advices corrections directly in the action's domain. This information indicates how to modify an action (increase or decrease it value), instead of evaluating it. Thus, this kind of feedback is designed to work in continuous-action domains.

If the idea behind corrective feedback is applied in discrete-action domains (that are not discretized continuous spaces), such as choosing between the actions $\{left, up, right\}$, the feedback signals  are corrections and demonstrations at the same time \cite{chernova2009interactive, mericcli2010complementary}. This kind of feedback is known as \textbf{corrective demonstrations} and it also belongs to the area of LfD.

Given that in this work we work with continuous-action domains, we are going to used the term corrective feedback under this context.

Methodologies such as \textbf{Advice Operators} \cite{Argall2009, Argall2008} have been proposed to solve problems using corrective feedback. This work studies another approach known as the \textbf{COrrective Advice Communicated by Humans (COACH)} \cite{Celemin2018AnInteractive} framework, which is described below.

To the best of our knowledge, corrective feedback has been only validated in problems with state spaces of low dimensionality, such as the ones covered in the aforementioned approaches.

\subsubsection{COrrective Advice Communicated by Humans}
\label{sss:COACH}
In COACH, if the agent executes an action $a$ that the human considers to be erroneous, then s/he would indicate the direction in which the action should be corrected (thus, COACH was proposed for problems with continuous actions). Each dimension of the action would have a corresponding correction signal $h$ with values $0$, $-1$ or $1$ which produces an error signal with arbitrary magnitude $e$ that is used to shape directly the policy. Thus, the error would be:

\begin{equation}\label{eq:error}
    error=h \cdot e
\end{equation}

$h=0$ indicates that no correction has been advised. $h=\pm 1$ indicates the direction of the advised correction.

In this framework no value function is modeled, since no reward/cost is used in the learning process. A parameterized policy is directly learned in the parameter space, as in Policy Gradients RL.
The classic COACH algorithm shapes two functions parameterized as a linear model of basis functions. The objective of the first function is to learn the policy of the agent $\pi_{\theta}(s)=\phi(s) \cdot \theta$; the objective of the second function, $H_{\psi}(s)=\phi(s) \cdot \psi$, is to learn a prediction of the human feedback. The parameter vectors $\theta$ and $\psi$ are updated to shape the models. As it can be seen, $\phi(s)$ is the same vector for both the \textbf{Policy Model} $\pi(s)$ and the \textbf{Human Feedback Model}  $H(s)$. The Human Feedback Model is used to adapt the size of the error signal that is then used to update the weights of $\pi(s)$. Both functions are updated using stochastic gradient descent each time feedback is received. The pseudocode of COACH is shown in Algorithm \ref{algorithm:COACH}.

\begin{algorithm}[H]
\caption{Basic Structure of COACH}\label{algorithm:COACH}
\begin{algorithmic}[1]
\State \textbf{Require:} error magnitude $e$, human model learning rate $\beta$, time steps $N$
\For{t = 1,2,...,N}{}
\State \textbf{observe} state $s_{t}$
\State \textbf{execute} action $a_{t}=\pi(s_{t})$
\State \textbf{feedback} human corrective advice $h_{t}$
\If{$h_{t}$ is not 0}
\State \textbf{update} $H(s_{t})$ with $\Delta \psi = \beta\cdot (h_{t}-H(s_{t}))\cdot \phi(s_{t})$
\State $\alpha_{t} = |H(s_{t})|$
\State $error_{t} = h_{t}\cdot e$
\State \textbf{update} $\pi(s_{t})$ with $\Delta \theta = \alpha_{t} \cdot error_{t} \cdot \phi(s_{t})$
\EndIf
\EndFor
\end{algorithmic}
\end{algorithm}


\section{Representations}
We make use of layers of representation between the surface text of a word problem and the learned model.
The first of these is a domain-independent semantic representation derived directly from the text using pre-existing language processing tools.
This semantic representation is then transformed into a domain-specific intermediate representation of sets similar to the {\it Qset} representation used by \cite{koncel2015parsing}.
Finally, a pair of intermediate representations are used to produce a vector for training.
Each step of this process is described below.

\subsection{Semantic Representation}
\label{semantics}
The text of each sentence of a word problem is represented as a collection of predicates as provided by the Answer Constraint Engine parser \cite{ace}.
ACE parses text using HPSG grammars, and we use the English Resource Grammar \cite{flickinger2000building,flickinger2011accuracy}, a high precision, hand-built grammar for the English language.
The predications instantiate a collection of entity and event variables and express the different interactions of these variables according to the compositional meaning of the text.
We track all predications in which a given entity variable appears as an argument.
We also track relations between entities appearing as arguments of the same predication.

Relevant to the task of solving math word problems are those entities that are related to numbers.
These entities often appear as the ARG1 of a CARD\_REL whose CARG (and text span) is a number.
If a CARD\_REL has a non-entity ARG1, we associate it's CARG to an instantiated entity which shares the same label.
These distinguished entities form our domain-specific intermediate representation, which for the sake of simplicity we will call {\it Number Entities}.

\subsection{Vector Representation}
\label{vectors}
As mentioned above, our system learns from triples consisting of two text quantities and an operator which appears in an equation leading to the correct solution.
We represent a triple as a vector by considering the Number Entities corresponding to the text quantities in the triple.
We evaluate several methods of vectorizing these semantic representations:
\paragraph{All Predicates} For each Number Entity we add a 196 dimension vector, the number of predicates associated with any Number Entity in the training data. The vector contains a 1 if the Number Entity appears as an ARG$N$ for $N>0$ of the corresponding predicate.
\paragraph{Abstract Predicates}
 Due to the nature of predicate naming, specific content noun and verb lexemes are encoded in the predicates, and these can be used to determine the correct operation between Number Entities.
For example, problems about {\it seashells} (and thus ``\_seashell\_n\_1\_rel'') are generally addition or subtraction problems due to the nature of the data sources.
Such lexical overlap was shown to be a significant factor for the success of template-based methods; however, it is theoretically unsatisfying as we would like a method which does not ``memorize'' spurious features of the data but rather learns something more fundamental about the nature of the natural language representation of mathematical operations.
To combat this, we consider abstract predicates to be those MRS predicates whose names begin with an underscore (rather than an open-quotation).
45 predicates meet this definition.
\paragraph{Abstract Predicates + Word2Vec} Previous work \cite{koncel2015parsing} has shown that verb lexemes, along with distances between the two Number Entity's nouns and verbs respectively improve classification performance.
We use Word2Vec \cite{mikolov2013efficient} to obtain 200-dimensional representations of verbs and nouns associated with each Number Entity.
We incorporate the verb vectors of both Number Entities directly into our feature vector, as well as the cosine distances of their nouns and verbs.