\typeout{}\typeout{If latex fails to find aiaa-tc, read the README file!}
%

\documentclass[]{aiaa-tc}% insert '[draft]' option to show overfull boxes

\usepackage{amsmath}          % for formula writing (i.e. 'split', etc)
\usepackage{rotate}           %rotate/mirror images
\usepackage{cancel}           %draw lines through math to show "goes to zero"
\usepackage{xfrac}            %allows slated and side fractions
\usepackage{subcaption}       %allows captioning individual subfigures
\usepackage{multicol}         %enable environment with multiple columns
\usepackage[mode=buildnew]{standalone}% requires -shell-escape
  % compile with `pdflatex -shell-escape main` or `xelatex  -shell-escape main`

\usepackage{tikz}             %for creating vector graphics diagrams
\usetikzlibrary{backgrounds}  %put backgrounds behind tikz figures
\usetikzlibrary{calc}         %perform calculations within $$
\usetikzlibrary{positioning}  %position tikz elements using "right of, etc"
\usetikzlibrary{angles}       %label angles between lines with arcs
\usetikzlibrary{quotes}       %Put angle label in quotes
\usetikzlibrary{patterns}     %Patterns to fill shapes with

  \title{MAE 298 Aeroacoustics -- Homework \#3 \\ Generalized Differentiation and Farassat's Formulation}

\author{
  Logan D. Halstrom \\
  {\normalsize\itshape Graduate Student} \\
  {\normalsize\itshape Department of Mechanical and Aerospace Engineering} \\
  {\normalsize\itshape University of California, Davis, CA 95616}
       }

 % Define commands to assure consistent treatment throughout document
 \newcommand{\eqnref}[1]{(\ref{#1})}
 \newcommand{\class}[1]{\texttt{#1}}
 \newcommand{\package}[1]{\texttt{#1}}
 \newcommand{\file}[1]{\texttt{#1}}
 \newcommand{\BibTeX}{\textsc{Bib}\TeX}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle

% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \begin{abstract}

% Abstract about homework

% \end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Nomenclature}

\begin{multicols}{2}

\begin{tabbing}
  XXX \= \kill% this line sets tab stop
  $0$                 \> Subscript for undisturbed, quiescent parameters \\
  $e$                 \> Subscript for emission parameters \\
  $L$                 \> Subscript for loading parameters \\
  $t$                 \> Time \\
  $\tau$              \> Retarded time \\
  $ret$               \> Evaluated at retarded time \\
  $\hat{n}$           \> Unit surface normal vector \\
  $\vec{r}$           \> Radial direction vector \\
  $\hat{r}$           \> Radial unit vector \\
  $r$                 \> Magnitude of radial vector $|\vec{r}|$ \\
  $\theta$            \> Angle between $\hat{n}$ and $\hat{r}$ \\
  $f$                 \> Function of surfaces within a fluid space \\
  $\vec{V}$           \> General velocity vector \\
  $V_r$               \> Velocity component in radial direction \\
  $V_n$               \> Velocity component in surface normal direction \\
  $M$                 \> Mach number \\
  $c$                 \> Speed of Sound\\
  $\overline{\rho}$   \> Mean density \\
  $p$                 \> Pressure  \\
  $\widetilde{p}$     \> Pressure (Discontinuous across data surface)  \\
  $\overline{p}$      \> Mean pressure \\
  $p'$                \> Perturbation pressure \\
  $\Delta P$          \> Pressure difference from CFD solution \\
  $L$                 \> Pressure loading \\
  $\overline{\partial}$ \> Generalized derivative \\
  $\delta$            \> Dirac delta function \\
  \scriptsize{FW-H}   \> Ffowcs Williams-Hawkings\\

\end{tabbing}

\end{multicols}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Overview} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this assignment, we will first derive the wave equation for sound generated by a moving body using generalized differentiation.  This process is similar to that used by Ffowcs Williams and Hawkings in the derivation of their equation describing aeroacoustic noise.  Equations like these are useful for computing aeroacoustic effects once source terms such as body loading are calculated.

Secondly, we will derive Farassat's Formulation 1A of the Ffowcs Williams-Hawkings (FW-H) equation for pressure due to loading noise beginning with Formulation 1.  This result will demonstrate the process for solving the FW-H equation and may be applied to the other terms to obtain Farassat's complete Formulation 1A of the FW-H equation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 1 -- Generalized Differentiation of Wave Equation} \label{SecGenDiff}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In this section, we will derive the wave equation for sound generated by a moving body, which is known as the Kirchhoff formula for moving surfaces.  The general acoustic wave equation with no source term (homogeneous) is expressed as follows:

\begin{equation} \label{AcousticWaveHomo}
\boxed{\dfrac{1}{c^2}\dfrac{\partial^2p}{\partial t^2} - \nabla^2p = 0}
\end{equation}

We will used generalized differentiation to show that the wave equation whose sound is generated by an arbitrary moving body $f=0$ can be expressed as follows:

\begin{equation} \label{AcousticWaveBody}
\boxed{\dfrac{1}{c^2}\dfrac{ \overline{\partial}^2\widetilde{p}}{\partial t^2}
    - \overline{\nabla}^2 \widetilde{p}
= -\left[ \dfrac{M_n}{c} \dfrac{\partial p}{\partial t} + p_n \right] \delta(f)
    -\dfrac{1}{c} \dfrac{\partial}{\partial t} \left[ M_n p \delta(f) \right]
    -\nabla \cdot \left[ p \hat{n} \delta(f) \right]}
\end{equation}

\noindent where $\hat{n}$ is the unit normal vector on the surface and $p_n = \nabla p \cdot \hat{n}$.

We will use Green’s function of the wave equation in the unbounded space to find the unknown function $p(\vec{x},t)$, which exists everywhere in space. This result is called the Kirchhoff formula for moving surfaces.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Data Surface Definition}

The moving surface generating acoustic effects will be defined as $f(\vec{x},t) = 0$ and will be referred to as a ``data surface''.  The definition of $f$ allows the fluid to be defined at all points in space, which can be divided into three regions:

\begin{center}
\begin{tabular}{| r  l |}
  \hline
  Data Surface:    & $f=0$ \\
  Exterior Region: & $f>0$ \\
  Interior Region: & $f>0$ \\
  \hline
\end{tabular}
\end{center}

\noindent We will also assume that the gradient of $f$ is the surface outward unit normal vector $\hat{n}$:

\begin{equation} \label{Gradfnormal}
\nabla f = \hat{n}
\end{equation}

Under this definition of $f$, the fluid will exist in all regions of unbounded space and thus enable the usage of Green's function to solve for the sound field.

We will solve for the sound field in the surface exterior region, but we will assume that the fluid that extends into the interior region has the same conditions as an undisturbed, quiescent medium.  We can define a pressure variable with these properties as $\widetilde{p}$ using the embedding technique as follows:

\begin{equation} \label{Ptilde}
\begin{split}
\boxed{\widetilde{p} =
    \left\{ \begin{array}{lll}
        p, & f > 0 \\
        0, & f < 0
    \end{array} \right.}
\end{split}
\end{equation}

\noindent Thus, pressure will vary outside of the data surface and will be undisturbed inside. Though not continuous, this function of pressure will be defined at all points in an unbounded space and thus applicable to Green's function.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Generalized Differentiation}

Due to the discontinuity in $\widetilde{p}$ at the data surface $f=0$, we will not be able to use standard differentiation to solve for the sound field and must instead use a technique called generalized differentiation, which will be denoted with a bar over the differential operator (e.g. $\overline{d}, \overline{\partial}, \overline{\nabla}$, etc).

To illustrate the principles of generalized differentiation, we will use the piecewise-continuous pressure $\widetilde{p}$.  The following rules will also apply to other piecewise-continuous fluid properties such as density $\widetilde{\rho}$. Without a formal proof, we can state the rules of generalized differentiation as:

\begin{equation} \label{GenDiff}
\dfrac{\overline{d}\widetilde{p}}{dx}
    = \dfrac{d\widetilde{p}}{dx} + \Delta\widetilde{p} \delta(x - c)
\end{equation}

\noindent where $x$ is an arbitrary variable of differentiation, $\delta$ is the Dirac delta function, $x=c$ is the location of the discontinuity in $\widetilde{p}$, and $\Delta\widetilde{p}=\widetilde{p}(c^+)-\widetilde{p}(c^-)=p - p_0$ is the discontinuous difference in $\widetilde{p}$ at the location of the discontinuity, or the difference in the parameter between the exterior and interior regions of the defined space.

For the case of the arbitrary moving body $f=0$, where $x=f$ and $c=0$, the Dirac delta function $\delta(x-c)$ becomes $\delta(f)$.  Applying Eqn~\ref{GenDiff} to the gradient operator $\nabla$ and using the chain rule, we obtain:

\begin{align*}
\overline{\nabla} \widetilde{p}
&= \dfrac{\overline{\partial}\widetilde{p}}{\partial x_i}
= \dfrac{\overline{\partial}\widetilde{p}}{\partial f}
    \dfrac{\partial f}{\partial x_i} \\
&= \left( \dfrac{\partial\widetilde{p}}{\partial f}
    + \Delta \widetilde{p} \delta(f) \right)
    \dfrac{\partial f}{\partial x_i} \\
&= \dfrac{\partial\widetilde{p}}{\partial f} \dfrac{\partial f}{\partial x_i}
    + \Delta \widetilde{p} \dfrac{\partial f}{\partial x_i} \delta(f) \\
&= \dfrac{\partial\widetilde{p}}{\partial x_i}
    + \dfrac{\partial f}{\partial x_i} \Delta \widetilde{p} \delta(f)
\end{align*}

\noindent Substituting the gradient operator $\nabla = \dfrac{\partial}{\partial x_i}$:

\begin{equation} \label{GenGrad}
\overline{\nabla} \widetilde{p}
= \nabla\widetilde{p} + \nabla f \Delta\widetilde{p} \delta(f)
\end{equation}

Recall the assumption that gradient of $f$ is the outward normal vector, as summarized in Eqn~\ref{Gradfnormal}.  Additionally, we can apply the definition of $\widetilde{p}$ to the pressure difference at the discontinuity $\Delta\widetilde{p} = p - p_0 = p - 0 = p$.  Making these substitutions, we can rewrite Eqn~\ref{GenGrad}:

\begin{equation} \label{GenGradAssume}
\overline{\nabla} \widetilde{p}
= \nabla\widetilde{p} + \hat{n} p \delta(f)
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Kirchhoff Formula Derivation}

We will now apply the principles of generalized differentiation to Eqn~\ref{AcousticWaveHomo} to derive the general equation for the sound field generated by a moving surface.  Beginning with the first term on the LHS of Eqn~\ref{AcousticWaveHomo}, we will first take a single partial derivative WRT time and then perform another derivative on the resulting term:

\begin{align*}
\dfrac{1}{c}\dfrac{\overline{\partial}\widetilde{p}}{\partial t}
&= \dfrac{1}{c}\dfrac{\overline{\partial}\widetilde{p}}{\partial f}
    \dfrac{\partial f}{\partial t} \\
&= \dfrac{1}{c}\left( \dfrac{\partial\widetilde{p}}{\partial f}
    + \Delta \widetilde{p} \delta(f) \right)
    \dfrac{\partial f}{\partial t} \\
&= \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial f} \dfrac{\partial f}{\partial t}
    + p \dfrac{1}{c}\dfrac{\partial f}{\partial t} \delta(f) \\
&= \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t}
    + p \dfrac{\partial f / \partial t}{c} \delta(f)
\end{align*}

Recalling the total time derivative of $f$, which is equal to zero by definition, we can write the time derivative of $f$ in terms of normal velocity $V_n$:

\begin{gather*}
\dfrac{df}{dt}
= \dfrac{\partial f}{\partial t} \cancelto{1}{\dfrac{dt}{dt}}
  + \dfrac{\partial f}{\partial x_i} \dfrac{\partial x_i}{\partial t} = 0 \\
\dfrac{\partial f}{\partial t} + \hat{n} \cdot \overline{V} = 0 \\
\boxed{\dfrac{\partial f}{\partial t} = -V_n}
\end{gather*}

\noindent where $V_n$ is the local normal velocity component along the data surface $f=0$. Now, recalling that Mach number $M = \dfrac{V}{c}$, we can rewrite the previous expression of the first derivative WRT time in terms of Mach number:

\begin{align*}
\dfrac{1}{c}\dfrac{\overline{\partial}\widetilde{p}}{\partial t}
= \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t}
    + p \dfrac{-V_n}{c} \delta(f)
\end{align*}

\begin{equation} \label{GenTimeP}
\dfrac{1}{c}\dfrac{\overline{\partial}\widetilde{p}}{\partial t}
= \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t}
    - M_n p\delta(f)
\end{equation}

Applying the second partial derivative WRT time to Eqn~\ref{GenTimeP}, we can derive the first term of Eqn~\ref{AcousticWaveHomo}:

\begin{align*}
\dfrac{1}{c^2}\dfrac{\overline{\partial}^2\widetilde{p}}{\partial t^2}
    &= \dfrac{1}{c}\dfrac{\overline{\partial}}{\partial t} \left(
    \dfrac{1}{c}\dfrac{\overline{\partial}\widetilde{p}}{\partial t} \right)
= \dfrac{1}{c}\dfrac{\overline{\partial}}{\partial t} \left(
\dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t}
    - M_n p\delta(f) \right)
\end{align*}

\noindent Distributing the differential to the first term and applying the same assumptions and substitutions that were used to derive Eqn~\ref{GenGradAssume}:

\begin{align*}
\dfrac{1}{c}\dfrac{\overline{\partial}}{\partial t}
    \left( \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t} \right)
    &= \dfrac{1}{c}\dfrac{\overline{\partial}}{\partial f}
    \left( \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t} \right)
    \dfrac{\partial f}{\partial t} \\
&= \dfrac{1}{c} \left[
    \dfrac{1}{c} \dfrac{\partial}{\partial f} \left(
    \dfrac{\partial\widetilde{p}}{\partial t} \right)
    + \dfrac{1}{c} \dfrac{\partial\Delta\widetilde{p}}{\partial t} \delta(f)
    \right] \dfrac{\partial f}{\partial t} \\
&=  \dfrac{1}{c^2} \dfrac{\partial}{\partial t} \left(
    \dfrac{\partial\widetilde{p}}{\partial t} \right)
    + \dfrac{\partial p}{\partial t} \dfrac{1}{c}\dfrac{-V_n}{c} \delta(f)
\end{align*}

\noindent which results in the final expression for the first term of the second general derivative of pressure WRT time:

\begin{equation} \label{2ndDerTerm1}
\dfrac{1}{c}\dfrac{\overline{\partial}}{\partial t}
    \left( \dfrac{1}{c}\dfrac{\partial\widetilde{p}}{\partial t} \right)
=  \dfrac{1}{c^2} \dfrac{\partial^2\widetilde{p}}{\partial t^2}
    - \dfrac{M_n}{c} \dfrac{\partial p}{\partial t}  \delta(f)
\end{equation}

Now, distributing the differential to the second term, we notice that all three parameters are continuous at the surface $f=0$, thus making the generalized differentiation identical to standard differentiation $\left(\dfrac{\overline{\partial}}{\partial t} = \dfrac{\partial}{\partial t} \right)$

\begin{equation} \label{2ndDerTerm2}
\dfrac{1}{c}\dfrac{\overline{\partial}}{\partial t}\left[-M_n p\delta(f)\right]
    = -\dfrac{1}{c}\dfrac{\partial}{\partial t} \left[ M_n p\delta(f) \right]
\end{equation}

Finally, applying the results for the first (Eqn~\ref{2ndDerTerm1}) and second (Eqn~\ref{2ndDerTerm2}) terms of the second derivative, we derive the final expression:

\begin{equation} \label{2ndDer}
\dfrac{1}{c^2}\dfrac{\overline{\partial}^2\widetilde{p}}{\partial t^2}
= \dfrac{1}{c^2} \dfrac{\partial^2\widetilde{p}}{\partial t^2}
    - \dfrac{M_n}{c} \dfrac{\partial p}{\partial t}  \delta(f)
- \dfrac{1}{c}\dfrac{\partial}{\partial t} \left[ M_n p\delta(f) \right]
\end{equation}

Next, we will solve for the second term on the LHS of Eqn~\ref{AcousticWaveHomo}; the generalized Laplacian of $\widetilde{p}$.  Beginning with the results for a single gradient (Eqn~\ref{GenGradAssume}) and applying the same assumptions and substitutions as used in that derivation:

\begin{align*}
\overline{\nabla}^2 \widetilde{p}
    &= \overline{\nabla} \widetilde{p}
        \left( \overline{\nabla} \widetilde{p} \right)
    = \overline{\nabla}
        \left( \nabla\widetilde{p} + \hat{n} p \delta(f) \right) \\
&= \nabla^2\widetilde{p} + \hat{n}\cdot\nabla p \delta(f)
    + \nabla\cdot \left[ \hat{n} p \delta(f) \right]
\end{align*}

\noindent Letting $p_n = \hat{n}\cdot\nabla p$ be the spatial partial derivative of surface pressure in the outward normal direction of $f=0$, we can write the final form of the generalized Laplacian of pressure:

\begin{equation} \label{DoubleGrad}
\overline{\nabla}^2 \widetilde{p}
    = \nabla^2\widetilde{p} + p_n \delta(f)
    + \nabla\cdot \left[ p\hat{n} \delta(f) \right] \\
\end{equation}

Finally, we will apply generalized differentiation to Eqn~\ref{AcousticWaveHomo}, the homogeneous acoustic wave equation, and then substitute the results for Eqns~\ref{2ndDer} and~\ref{DoubleGrad} to derive the final form of Eqn~\ref{AcousticWaveBody}:

\begin{align*}
\dfrac{1}{c^2}\dfrac{\overline{\partial}^2 \widetilde{p}}{\partial t^2}
    - \overline{\nabla}^2 \widetilde{p}
&= \dfrac{1}{c^2} \dfrac{\partial^2\widetilde{p}}{\partial t^2}
    - \dfrac{M_n}{c} \dfrac{\partial p}{\partial t}  \delta(f)
    - \dfrac{1}{c}\dfrac{\partial}{\partial t} \left[ M_n p\delta(f) \right]
    -\nabla^2\widetilde{p} - p_n \delta(f)
    - \nabla\cdot \left[ p\hat{n} \delta(f) \right] \\
&= \underbrace{\dfrac{1}{c^2} \dfrac{\partial^2\widetilde{p}}{\partial t^2}
    -\nabla^2\widetilde{p}}_{=0}
    - \dfrac{M_n}{c} \dfrac{\partial p}{\partial t}  \delta(f)
    - p_n \delta(f)
    - \dfrac{1}{c}\dfrac{\partial}{\partial t} \left[ M_n p\delta(f) \right]
    - \nabla\cdot \left[ p\hat{n} \delta(f) \right]
\end{align*}

The terms grouped in the bracket are exactly equal to the Right Hand Side (RHS) of Eqn~\ref{AcousticWaveHomo} and are thus exactly equal to zero since this equation is homogeneous.  Grouping like terms in the resulting equation, we derive the final expression for the sound field generated by a moving surface $f=0$, which is equivalent to Kirchoff's formula for moving surfaces:

\begin{equation}
\boxed{\dfrac{1}{c^2}\dfrac{\overline{\partial}^2 \widetilde{p}}{\partial t^2}
    - \overline{\nabla}^2 \widetilde{p}
= -\left[\dfrac{M_n}{c} \dfrac{\partial p}{\partial t} + p_n  \right] \delta(f)
    - \dfrac{1}{c}\dfrac{\partial}{\partial t} \left[ M_n p\delta(f) \right]
    - \nabla\cdot \left[ p\hat{n} \delta(f) \right]}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Problem 2 -- Farassat Formulation 1A for Loading Noise} \label{SecFarassat}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

In Section~\ref{SecGenDiff}, we derived Eqn~\ref{AcousticWaveBody} for the sound field produced by an arbitrary moving body.  To solve equations of this form, we must apply Green's function.  This was achieved for the FW-H equation by Farassat, first in his ``Formulation 1'', which transformed the spatial derivatives into terms of observer time $t$ and then into ``Formulation 1A'', which further transformed $t$ into terms of retarded time $\tau$.

Separating the FW-H equation into separate source terms of thickness, loading, and quadrupole volume, we can look specifically at Farassat’s Formulation 1 for the loading noise, which is given as:

\begin{equation} \label{FarassatForm1Loading}
\boxed{4\pi p_{L}' = \dfrac{1}{c} \dfrac{\partial}{\partial t}
      \int_{f=0} \left[ \dfrac{L_r}{r   (1 - M_r)} \right]_{ret} ds
    + \int_{f=0} \left[ \dfrac{L_r}{r^2 (1 - M_r)} \right]_{ret} ds}
\end{equation}

\noindent where $L_r = \Delta P \hat{n} \cdot \hat{r} = \Delta P \cos\theta$ is the loading in the radial direction and $ret$ denotes that the expression is evaluated at the retarded time $\tau=\tau_e$.

In practice, Formulation 1 is difficult to compute since the observer time differentiation is outside the integrals. Farassat's Formulation 1A presents a much more efficient and practical approach, which can be derived by carrying the observer time derivate inside the integrals. In the following section, we will show that Formulation 1A for the loading noise becomes:

\begin{equation} \label{FarassatForm1ALoadingEx}
\begin{split}
4\pi p_{L}' = \dfrac{1}{c}
      \int_{f=0} \left[ \dfrac{\dot{L}_r}{r (1 - M_r)^2} \right]_{ret} ds
    + \int_{f=0} \left[ \dfrac{L_r - L_M}{r^2 (1 - M_r)} \right]_{ret} ds \\
+ \dfrac{1}{c} \int_{f=0} \left[
    \dfrac{L_r[r \dot{M}_r + c(M_r - M^2)] }{r^2 (1 - M_r)^3} \right]_{ret} ds
\end{split}
\end{equation}

\noindent where $L_M = \vec{L} \cdot \vec{M} $.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Formulation 1A Transformation}

To obtain Farassat's formulations, it is required to transform the time derivative into a derivative WRT to $\tau$.  This transformation will be performed using Eqn~\ref{t2tau} below:

\begin{equation} \label{t2tau}
\left. \dfrac{\partial}{\partial t} \right|_x
= \left[ \dfrac{1}{1 - M_r} \dfrac{\partial}{\partial \tau} \right]_{ret}
\end{equation}

\noindent where $\dfrac{\partial}{\partial t}$ is evaluated along some surface $x$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Loading Noise Formulation 1A}

Now that we have defined the transformation between Formulation 1 and Formulation 1A, we can apply Eqn~\ref{t2tau} to Eqn~\ref{FarassatForm1Loading} to derive Formulation 1A of the loading noise.  Specifically, we will perform the transformation on the first term of the RHS of Eqn~\ref{FarassatForm1Loading}, which contains a time derivative:

\begin{align} \label{DerInt}
\dfrac{\partial}{\partial t}
      \int_{f=0} \left[ \dfrac{L_r}{r   (1 - M_r)} \right]_{ret} ds
&= \int_{f=0} \left[
    \dfrac{1}{1 - M_r} \dfrac{\partial}{\partial \tau}
    \left( \dfrac{L_r}{r   (1 - M_r)}  \right)
    \right]_{ret} ds
\end{align}

Focusing on the term contained in the $\tau$ differential, we see that both the numerator and denominator contain functions of $\tau$, so the product rule of differentiation will be required to complete this derivation.  Before that, however, let us compute the derivatives of the individual components to simplify the final derivation.

The time derivative of the numerator is straightforward, as shown below:

\begin{equation} \label{DerL}
\dfrac{\partial}{\partial\tau} \left( L_r \right)
    = \dfrac{\partial L_r}{\partial\tau}
    = \dot{L}_r
\end{equation}

\noindent where the dot over $L_r$ denotes the time derivative, or the rate of change of the pressure loading with respect time time.

Next, we compute the time derivative of $\dfrac{1}{r}$ using the chain rule:

\begin{equation} \label{DerR}
\dfrac{\partial}{\partial\tau} \left( \dfrac{1}{r} \right)
= \dfrac{\partial}{\partial r} \left( \dfrac{1}{r} \right)
    \dfrac{\partial r}{\partial\tau}
= -\dfrac{1}{r^2} (-V_r)
= \dfrac{V_r}{r^2}
\end{equation}

\noindent where $\dfrac{\partial r}{\partial\tau}$ is:

\begin{equation} \label{drdtau}
\dfrac{\partial r}{\partial\tau}
= \dfrac{\partial r}{\partial y_i}\dfrac{\partial y_i}{\partial\tau}
= (-\hat{r})(\vec{V})
= -V_r
\end{equation}

\noindent where $V_r$ is the velocity component in the radial direction:

We will next compute the time derivative of $\dfrac{1}{1 - M_r}$, but will first calculate two derivatives on which it depends.  First, we calculate the derivative of the radial unit vector $\hat{r}$, substituting the results and assumptions of Eqn~\ref{DerR}:

\begin{align*}
\dfrac{\partial}{\partial\tau} \left( \hat{r} \right)
&= \dfrac{\partial}{\partial\tau} \left( \dfrac{\vec{r}}{r} \right)
    = \dfrac{1}{r} \dfrac{\partial \vec{r}}{\partial\tau}
    + \vec{r}\dfrac{\partial\vec{r}}{\partial\tau} \left(\dfrac{1}{r} \right)\\
&= \dfrac{-\vec{V}}{r} + \vec{r} \cdot \dfrac{V_r}{r^2}
    = \dfrac{-\vec{V}}{r} + \dfrac{1}{r} \dfrac{\vec{r}}{r} \cdot V_r \\
&= -\dfrac{\vec{V}}{r} + \dfrac{1}{r} \hat{r} \cdot V_r
\end{align*}

\noindent where $r = |\vec{r}|$ is the magnitude of the radial vector.  Finally, multiplying the equation by $c/c$ allows expression in terms of Mach number:

\begin{equation} \label{DerRhat}
\dfrac{\partial}{\partial\tau} \left( \hat{r} \right)
= -\dfrac{c}{r} \left( \vec{M} - \vec{r} \cdot M_r \right)
\end{equation}

Secondly, the derivative of Mach number in the radial direction can be expressed as:

\begin{align*}
\dfrac{\partial}{\partial\tau} \left( M_r \right)
&= \dfrac{\partial}{\partial\tau} \left( \vec{M} \cdot \hat{r} \right)
= \dfrac{\partial\vec{M}}{\partial\tau}
    \cdot\hat{r} + \vec{M} \dfrac{\partial\hat{r}}{\partial\tau} \\
&= \dot{\vec{M}}\cdot\hat{r} + \vec{M} \dfrac{c}{r}
    \left( -\vec{M} + \vec{r} \cdot M_r \right) \\
&= \dot{\vec{M}}\cdot\hat{r} + \dfrac{c}{r}
    \left( -\vec{M}\cdot\vec{M} + \vec{r} \cdot\vec{M}\cdot M_r \right)
\end{align*}

\noindent where Eqn~\ref{DerRhat} was substituted in for the time derivative of $\hat{r}$.  Letting $\dot{\vec{M}} = \dfrac{\partial \vec{M}}{\partial \tau}$ denote the time derivative of Mach number or the acceleration in terms of Mach, we obtain the final expression for the derivative of radial Mach number:

\begin{equation} \label{DerMr}
\dfrac{\partial}{\partial\tau} \left( M_r \right)
= \dot{\vec{M}}_r + \dfrac{c}{r}
    \left( -|\vec{M}|^2 + M_r^2 \right)
\end{equation}

Now, we now have all of the derivative components required to calculate the time derivative of $\dfrac{1}{1 - M_r}$:

\begin{align*}
\dfrac{\partial}{\partial\tau} \left( \dfrac{1}{1 - M_r} \right)
&= -\dfrac{1}{(1 - M_r)^2} \dfrac{\partial}{\partial\tau}(-M_r) \\
&= -\dfrac{1}{(1 - M_r)^2}(-1) \left[
    \dot{\vec{M}}_r +\dfrac{c}{r}\left( -|\vec{M}|^2 + M_r^2 \right) \right] \\
&= \dfrac{ \dot{\vec{M}}_r + \dfrac{c}{r} \left( -|\vec{M}|^2 + M_r^2 \right) }
    {(1 - M_r)^2}
\end{align*}

\noindent Multiplying by $r/r$:

\begin{equation} \label{DerMrDenom}
\dfrac{\partial}{\partial\tau} \left( \dfrac{1}{1 - M_r} \right)
= \dfrac{ r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) }
    {r (1 - M_r)^2}
\end{equation}

At long last, we have finally computed all of the necessary terms to complete the transformation of the first term in Eqn~\ref{FarassatForm1Loading}.  The time derivative must be performed using the product rule, as follows:

\begin{align*}
\dfrac{\partial}{\partial \tau} \left( \dfrac{L_r}{r (1 - M_r)}  \right)
& = \dfrac{\partial}{\partial \tau} \left\{ L_r \dfrac{1}{r (1 - M_r)}  \right\} \\
& = \dfrac{\partial L_r}{\partial \tau} \dfrac{1}{r (1 - M_r)}
    + L_r\dfrac{\partial}{\partial \tau} \left[ \dfrac{1}{r (1 - M_r)} \right] \\
& = \dfrac{\dot{L}_r}{r (1 - M_r)}
    + L_r \left[
    \dfrac{\partial}{\partial\tau} \left(\dfrac{1}{r}\right) \dfrac{1}{1 - M_r}
    + \dfrac{1}{r} \dfrac{\partial}{\partial\tau} \left(\dfrac{1}{1-M_r}\right)
    \right] \\
& = \dfrac{\dot{L}_r}{r (1 - M_r)}
    + L_r \left[
    \dfrac{1}{1 - M_r} \dfrac{-V_r}{r^2}
    + \dfrac{1}{r}
    \dfrac{ r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) }
    {r (1 - M_r)^2}
    \right] \\
& = \dfrac{\dot{L}_r}{r (1 - M_r)}
    + L_r \left[
    \dfrac{-V_r}{r^2 (1 - M_r)}
    + \dfrac{ r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) }
        {r^2 (1 - M_r)^2}
    \right] \\
&= \dfrac{\dot{L}_r}{r (1 - M_r)}
    + \dfrac{-L_r V_r}{r^2 (1 - M_r)}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^2} \\
&= \dfrac{\dot{L}_r}{r (1 - M_r)}
    + \dfrac{-(\vec{L}\cdot\hat{r}) (\vec{V}\cdot\hat{r})}{r^2 (1 - M_r)}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^2} \\
&= \dfrac{\dot{L}_r}{r (1 - M_r)}
    + \dfrac{-(\vec{L}\cdot\vec{V}) (\hat{r}\cdot\hat{r})}{r^2 (1 - M_r)}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^2}
\end{align*}

\noindent Recalling that any unit vector $\hat{r}$ that is multiplied by itself must equal unity due to the property:

\begin{equation} \label{UnitSquared}
\hat{r}\cdot\hat{r}
= \dfrac{\vec{r}}{|\vec{r}|}\cdot\dfrac{\vec{r}}{|\vec{r}|}
= \dfrac{\vec{r}\cdot\vec{r}}{|\vec{r}|\cdot|\vec{r}|}
= \dfrac{|\vec{r}|^2}{|\vec{r}|^2} = 1
\end{equation}

\noindent we can write our derivative term as such:

\begin{equation} \label{TauDer}
\dfrac{\partial}{\partial \tau} \left( \dfrac{L_r}{r (1 - M_r)}  \right)
= \dfrac{\dot{L}_r}{r (1 - M_r)}
    + \dfrac{-\vec{L}\cdot\vec{V}}{r^2 (1 - M_r)}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^2}
\end{equation}

Now that we have computed the derivative WRT $\tau$, we can substitute Eqn~\ref{TauDer} into Eqn~\ref{DerInt}:

\begin{align*}
\dfrac{\partial}{\partial t}
      \int_{f=0} \left[ \dfrac{L_r}{r   (1 - M_r)} \right]_{ret} ds
&= \int_{f=0} \left\{
    \dfrac{1}{1 - M_r} \left[
    \dfrac{\dot{L}_r}{r (1 - M_r)}
    + \dfrac{-\vec{L}\cdot\vec{V}}{r^2 (1 - M_r)}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^2} \right]
    \right\}_{ret} ds \\
&= \int_{f=0} \left[
    \dfrac{\dot{L}_r}{r (1 - M_r)^2}
    + \dfrac{-\vec{L}\cdot\vec{V}}{r^2 (1 - M_r)^2}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3}
    \right]_{ret} ds
\end{align*}

\noindent resulting in the Formulation 1A version of the derivative integral term:

\begin{equation} \label{Form1aInt}
\dfrac{\partial}{\partial t}
      \int_{f=0} \left[ \dfrac{L_r}{r   (1 - M_r)} \right]_{ret} ds
= \int_{f=0} \left[
    \dfrac{\dot{L}_r}{r (1 - M_r)^2}
    + \dfrac{-\vec{L}\cdot\vec{V}}{r^2 (1 - M_r)^2}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3}
    \right]_{ret} ds
\end{equation}

The final step in deriving Formulation 1A is to substitute Eqn~\ref{Form1aInt} into Eqn~\ref{FarassatForm1Loading} (Loading Noise Formulation 1):

\begin{align*}
4\pi p_{L}' &= \dfrac{1}{c} \dfrac{\partial}{\partial t}
      \int_{f=0} \left[ \dfrac{L_r}{r   (1 - M_r)} \right]_{ret} ds
    + \int_{f=0} \left[ \dfrac{L_r}{r^2 (1 - M_r)} \right]_{ret} ds \\
&= \dfrac{1}{c} \int_{f=0} \left[
    \dfrac{\dot{L}_r}{r (1 - M_r)^2}
    + \dfrac{-\vec{L}\cdot\vec{V}}{r^2 (1 - M_r)^2}
    + \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3}
    \right]_{ret} ds
    + \int_{f=0} \left[ \dfrac{L_r}{r^2 (1 - M_r)} \right]_{ret} ds
\end{align*}

\noindent Distributing the integral across summed terms:

\begin{align*}
% 4\pi p_{L}'
= \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{\dot{L}_r}{r (1 - M_r)^2} \right]_{ret} ds
    &+ \int_{f=0} \left[ \dfrac{L_r}{r^2 (1 - M_r)} \right]_{ret} ds
    + \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{-\vec{L}\cdot\vec{V}}{r^2 (1 - M_r)^2} \right]_{ret} ds
    \\ &+ \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3} \right]_{ret} ds
\end{align*}

\noindent Distributing the speed of sound term and combining like integrals:

\begin{align*}
% 4\pi p_{L}'
&= \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{\dot{L}_r}{r (1 - M_r)^2} \right]_{ret} ds
    + \int_{f=0} \left[ \dfrac{L_r}{r^2 (1 - M_r)}
        - \dfrac{\frac{1}{c} (\vec{L}\cdot\vec{V})}{r^2 (1 - M_r)^2}
        \right]_{ret} ds
    + \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3} \right]_{ret} ds \\
&= \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{\dot{L}_r}{r (1 - M_r)^2} \right]_{ret} ds
    + \int_{f=0} \left[
        + \dfrac{L_r - \vec{L}\cdot\vec{M}}{r^2 (1 - M_r)^2}
        \right]_{ret} ds
    + \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3} \right]_{ret} ds
\end{align*}

\noindent Substituting the definition $L_M = \vec{L}\cdot\vec{M}$, we derive the final expression for Formulation 1A of Loading Noise:

\begin{equation}
\begin{split}
4\pi p_{L}'
= \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{\dot{L}_r}{r (1 - M_r)^2} \right]_{ret} ds
    &+ \int_{f=0} \left[
         \dfrac{L_r - L_M}{r^2 (1 - M_r)^2}
        \right]_{ret} ds \\
    &+ \dfrac{1}{c} \int_{f=0} \left[
        \dfrac{ L_r \left[
        r\dot{\vec{M}}_r + c \left( M_r^2 - |\vec{M}|^2 \right) \right] }
        {r^2 (1 - M_r)^3} \right]_{ret} ds
\end{split}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Conclusion}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

We have now completed the derivation of an aeroacoustic equation describing the pressure fluctuations around a moving body and then demonstrated a method of solving equations such as these.  In a real life application, the equation being solved would be more complicated and include more source terms, like the complete FW-H equation does.  This equation can be solved by applying Farassat's Formulation 1A and implementing the solution numerically.  Source inputs for loading noise ($\Delta P$) would be obtained from an unsteady CFD simulation and used to inform the aeroacoustic solution.

\end{document}


    \filetitle{demean}{Remove constant and the effect of exogenous inputs from VAR object}{VAR/demean}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
V = demean(V)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{V} {[} VAR {]} - VAR object.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{V} {[} VAR {]} - VAR object with the constant vector,
  \texttt{K}, and the asymptotic assumptions for exogenous inputs,
  \texttt{X0}, reset to zero.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\documentclass[12pt]{article}
\usepackage{graphicx,bm,amsfonts,amsmath,setspace}
%\usepackage[nolists]{endfloat}
\usepackage{color,hyperref,verbatim,rotating,subfigure}
\usepackage{apacite}
%\usepackage{listings}
\usepackage{listings}
\usepackage{color}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  %language=Java,
  language=Matlab,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true,
  tabsize=3
}

%\renewcommand{\baselinestretch}{1.5}
\newcommand{\ve}[1]{\mbox{\boldmath $#1$}}
\newcommand{\diag}{\mathop{\rm diag}}
\def\argmax{\mathop{\rm arg\,max}}%
\setlength{\textheight}{9.8in}
\setlength{\textwidth}{6.53in}
\oddsidemargin=0mm
\evensidemargin=0mm
\topmargin=0mm
\hoffset=0mm
\voffset=-0.5in
\textheight=9.4in
%\setcounter{secnumdepth}{0}
% THEOREM Environments ---------------------------------------------------
 \newtheorem{thm}{{\bf Theorem}}
% \newtheorem{cor}[thm]{Corollary}
% \newtheorem{lem}[thm]{Lemma}
% \newtheorem{prop}[thm]{Proposition}
% \theoremstyle{definition}
% \newtheorem{defn}[thm]{Definition}
% \theoremstyle{remark}
% \newtheorem{rem}[thm]{Remark}
% \numberwithin{equation}{subsection}
%%% ----------------------------------------------------------------------
\bibliographystyle{apacite}
%\bibliographystyle{asa}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\title{MATLAB code for CLAD with MIP model}
\author{K. Florios\\\textit{Department of International and European Economic Studies,}\\\textit{Athens University of Economics and Business}\\\textit{cflorios@aueb.gr}}
\date{\today}\maketitle

{\vspace*{1cm}}

%========================================================================================

\newpage

\textbf{CladCompute.m}

\begin{lstlisting}

function [value,estimates,time,quality]=CladCompute()

%Main Program: Computes Max score by defining a MILP and calling milp.m
tic
[X,y,w]=readXyw();
[X,mu,sigma]=standardizeX(X,y);
[c,A,b]=definecAb(X,y,w);
[lb,ub, Aeq, beq, n, p, best]=definelbub(X,y);
%[x,cost,feasible]=milp(c,A,b,Aeq,beq,lb,ub,n, best);
 [x,cost,feasible, time]=milp_cplex(c,A,b,Aeq,beq,lb,ub,n, p, best);
estimatesNorm=x
value=cost
status=feasible
runtime=time

estimatesRaw=denormalizeEstimates(estimatesNorm,mu,sigma)
estimates=estimatesRaw

%estimatesRaw=denormalizeEstimates(estimatesNorm,mu,sigma)

estimates=estimatesRaw
value=cost
time=toc
quality=feasible;

end

\end{lstlisting}

\textbf{definecAb.m}

\begin{lstlisting}

function [c,A,b]=definecAb(X,y,w)

%Defines c,A,b for milp.m

n=size(X,1);
p=size(X,2);

c3=repmat(0,1,n);   %gammas integer
c1=repmat(0,1,p);    %betas
c2=repmat(0,1,n);     %phis
c4=repmat(1,1,n);      %sm's
c5=repmat(1,1,n);       %sp's

c=[c3 c1 c2 c4 c5];

d=20;
%d=5;
for i=1:1:n
    M(i)=abs(X(i,1))*d+abs(X(i,2:end))*repmat(d,1,p-1)';
end

for i=1:n         %constraint 3c in pdf ssrn
   A1(i,1:n)=0;  %gammas
   for j=1:p
   A1(i,(n+j))=X(i,j);  %betas
   end
   A1(i,(n+p+1):(2*n+p))=-1.*(1:n==i);   %phis
   A1(i,(2*n+p+1):(3*n+p))=0;             %sm's
   A1(i,(3*n+p+1):(4*n+p))=0;              %sp's
end

for i=1:n
   b1(i)=0;
end

for i=1:n       %constraint 3e in pdf ssrn
   A2(i,1:n)=M(i).*(1:n==i);   %gammas
   for j=1:p
   A2(i,(n+j))=-X(i,j);  %betas
   end
   A2(i,(n+p+1):(2*n+p))=1.*(1:n==i);        %phis
   A2(i,(2*n+p+1):(3*n+p))=0;       %sm's
   A2(i,(3*n+p+1):(4*n+p))=0;        %sp's
end

for i=1:n
   b2(i)=M(i);
end

for i=1:n       %constraint 3f in pdf ssrn
  A3(i,1:n)=-M(i).*(1:n==i);   %gammas
  A3(i,(n+1):(n+p))=0; %betas
  A3(i,(n+p+1):(2*n+p))=1.*(1:n==i);  %phis
  A3(i,(2*n+p+1):(3*n+p))=0;  %sm's
  A3(i,(3*n+p+1):(4*n+p))=0; %sp's
end

for i=1:n
   b3(i)=0;  % for left censoring at 0
end

A= [A1 ; A2 ; A3];
b= [b1  b2  b3]';

end

\end{lstlisting}

\textbf{definelbub.m}

\begin{lstlisting}

function [lb,ub, Aeq, beq, n, p, best]=definelbub(X,y)

%Defines lb,ub for milp.m
 d=20;
%d=10;
%d=5;
n=size(X,1);
p=size(X,2);

lb1=repmat(0,1,n);     %gammas
lb2=repmat(-d,1,p);     %betas
%lb3=repmat(-Inf,1,n);    %phis
lb3=repmat(0,1,n);          %phis, left censoring at zero (0)
lb4=repmat(0,1,n);          %sm's
lb5=repmat(0,1,n);           %sp's

lb=[lb1 lb2 lb3 lb4 lb5];

BIGNUM=+Inf;

ub1=repmat(1,1,n);     %gammas
ub2=repmat(d,1,p);      %betas
ub3=repmat(+BIGNUM,1,n);    %phis
ub4=repmat(+BIGNUM,1,n);     %sm's
ub5=repmat(+BIGNUM,1,n);      %sp's

ub=[ub1 ub2 ub3 ub4 ub5];

for i=1:n
   Aeq(i,1:n)=0; %gammas
   Aeq(i,(n+1):(n+p))=0; %betas
   Aeq(i,(n+p+1):(2*n+p))=-1.*(1:n==i);  %phis
   Aeq(i,(2*n+p+1):(3*n+p))=1.*(1:n==i);  %sm's
   Aeq(i,(3*n+p+1):(4*n+p))= -1.*(1:n==i);  %sp's
end

for i=1:n
   beq(i)= -y(i);
end

best=+Inf;

end

\end{lstlisting}

\textbf{milp.m}

\begin{lstlisting}

function [x,cost,feasible]=milp(c,A,b,Aeq,beq,lb,ub,n, best);
% Solves a integer lp using branch and bound proceeding in a
% depth-first manner (recursive)
% c: is cost        A: is constraint matrix   b: is constraint vector
% lb: lower bound     ub: upper bound   n: number of integer variables
% best: is best solution so far
% Note this traverses along the right most branch
% unlike the usual way of checking both branches before continuing. Also,
% it assumes first n variables must be integer.
% Written by Kevin Tomsovic
% Note: Last accessed, Autumn 2007, from K. Tomsovic's homepage.
% Note: by K.Florios, Date: 4.4.2016

% Initialization
feasible=0;         % 1 - Feasible, 0 - Infeasible
tol=1e-3;           % Tolerance on integers
%%%tol=1e-6;           % Tolerance on integers
cost=inf;

% Solve current linprog, suppress error messages
options=optimset('display','off');
[x,cost,how]=linprog(c,A,b,Aeq,beq,lb,ub,[],options);

% If infeasible or cost is greater than bound (best solution found)
%   then stop and return with infinite cost
if how<=0 | cost>best, cost=inf; return; end;   %KJF comments out 1.4.2016

% Take first non integer u and branch
branch=0;
for i=1:n   % Split LP on first non-integer u
   if abs(x(i)-round(x(i)))>tol, branch=i; break; end;
end;

% If integer solution then return optimal solution
if branch==0, feasible=1; % If valid integer solution (branch==0), return
   return;
else  % Solve subproblems by putting new upper and lower bounds on branch variable
      % Proceeding depth-first
   lba=lb; uba=ub;
   uba(branch)=floor(x(branch)); lba(branch)=ceil(x(branch));
   % Solve left branch (x<=floor(x))
   [x,cost,feasible]=milp(c,A,b,Aeq,beq,lb,uba,n,best);

   % Solve right branch (x>=ceil(x))with new best
   best=min(cost,best);
   [xb,cost,how]=milp(c,A,b,Aeq,beq,lba,ub,n,best);
   if cost<best, x=xb; % Better solution found
   else cost=best;
   end;
   feasible=feasible | how;    % Feasible solution found?
end;
return;

\end{lstlisting}

\textbf{milp\_cplex.m}

\begin{lstlisting}

function [x,deviation,feasible, time]=milp_cplex(c,A,b,Aeq,beq,lb,ub,n, p, best);
% Solves a mixed integer lp using gurobi 5.0.1
% c: is objective function coefficients A: is constraint matrix   b: is constraint vector
% lb: lower bound  ub: upper bound  n: number of 0-1 variables
% best: is best solution so far
% Note this uses the MATLAB/Gurobi Interface documented at
% http://www.gurobi.com/documentation/5.0/reference-manual/node650
% Also, it assumes first n variables must be integer.
% The MIP equations of the maximum score estimator are available at
% Florios. K, Skouras, S. (2008) Exact computation of maximum weighted
% score estimators, Journal of Econometrics 146, 86-91.
% Written by Kostas Florios, July 20, 2012
%
% gurobi 5.0.1
% cd c:/Users/jones/gurobi500/win64/matlab
% gurobi_setup
%
% cd C:\gurobi501\win32\matlab
% gurobi_setup
%
% cplex 12.6
% addpath('C:\Program Files\IBM\ILOG\CPLEX_Studio_Preview126\cplex\matlab\x86_win32')
% Greg corei3 2016
% addpath('D:\Program Files\IBM\ILOG\CPLEX_Studio_Preview126\cplex\matlab\x86_win32')

model.Aineq = sparse(A) ;
model.f = c ;
model.bineq = b ;
model.Aeq = sparse(Aeq) ;
model.beq = beq' ;
model.lb = lb ;
model.ub = ub ;
%model.ctype = [repmat('B',size(b,2),1) ; repmat('C',size(c,2)-size(b,2),1)]' ;
model.ctype = [repmat('B',n,1) ; repmat('C',p,1) ; repmat('C',n,1) ; repmat('C',n,1) ; repmat('C',n,1)]' ;

opt = cplexoptimset('cplex') ;
opt.mip.display = 4 ;
opt.mip.interval = 1000 ;

opt.timelimit= 1800 ;
opt.mip.tolerances.mipgap = 0.00 ;
opt.parallel = 1 ;
opt.threads = 1 ;
opt.mip.strategy.file = 3 ;
opt.workmem = 1024 ;
opt.emphasis.mip = 3 ;

opt.exportmodel = 'G:\KOSTAS\Core-i5-Laptop\matlab2011\milp-cplex-2016-for-CLAD\myModel.lp' ;

model.options = opt;

[x,fval,exitflag,output] = cplexmilp(model) ;

  fprintf('Optimization returned status: %s\n', output.message);
  fprintf('Objective Value: %e\n', fval);
  fprintf('(Wall clock) Time elapsed (s): %e\n', output.time);
  fprintf('Decision variables: show only the  betas\n');
  %disp(x)

x=x((n+1):(n+p))
deviation=fval;
feasible=output.message;
time=output.time;
return;

\end{lstlisting}

\textbf{readXyw.m}

\begin{lstlisting}

function [X,y,w]=readXyw()

%Reads X,y,w of given max score problem

X=load('X_200_4.txt');
%X=[X(:,2) X(:,3) X(:,4) X(:,5)];
X=X(:,2:end);
y=load('ys_200_4.txt');
y=[y(:,2)];
%w=load('w_numeric5.txt');
%w=[w(:,2)];
w=repmat(1,size(X,1),1);

end

\end{lstlisting}

\textbf{denormalizeEstimates.m}

\begin{lstlisting}

function [estimatesRaw]=denormalizeEstimates(estimatesNorm,mu,sigma);

%denormalized estimatesNorm obtained by Gurobi MIP to estimatesRaw, which
%are meaningful to the user

%quick and dirty implementation, based on GAMS and Fortran Analogues
p=size(estimatesNorm,1);
betaNorm=estimatesNorm;

for j=1:1:p
    betaRaw(j)=0;
    betaHelp(j)=0;
end

for j=1:1:p
    if (sigma(j) ~=0)
        betaHelp(j)= betaNorm(j)./sigma(j);
    end
    if (sigma(j) ==0)
        for jj=1:1:p
            if (sigma(jj) ~=0)
                betaHelp(j)=betaHelp(j)-betaNorm(jj).*mu(jj)./sigma(jj) ;
            else
                jj0=jj;
            end
        end
        betaHelp(j)=betaHelp(j)+betaNorm(jj0);
    end
end

for j=1:1:p
   %%betaRaw(j)=betaHelp(j)./betaHelp(1);
   betaRaw(j)=betaHelp(j);
end

estimatesRaw=betaRaw;

end

\end{lstlisting}

\textbf{standardizeX.m}

\begin{lstlisting}

function [X,mu,sigma]=standardizeX(X,y)

%Standardizes X
mu=mean(X);
sigma=std(X);
testX=(X-repmat(mu,size(X,1),1)) ./ repmat(sigma,size(X,1),1);
p=size(X,2);
for j=1:1:p
if isnan(testX(:,j))
   X(:,j) = X(:,j)
else
    X(:,j) = testX(:,j)
end

end

\end{lstlisting}

\section{Contained functions} \label{Sec:Contents}

\begin{enumerate}
\item \textbf{CladCompute.m}  $\rightarrow$  main
\item \textbf{definecAb.m}       $\rightarrow$  defines matrices c,A,b
\item \textbf{definelbub.m}      $\rightarrow$ defines bounds lb,ub and matrices Aeq,beq
\item \textbf{denormalizeEstimates.m}  $\rightarrow$ post-processing
\item \textbf{milp.m}  $\rightarrow$ calls linprog
\item \textbf{milp\_cplex.m} $\rightarrow$ calls cplexmilp commercial function
\item \textbf{readXyw.m}  $\rightarrow$ input from files X.txt and ys.txt
\item \textbf{standardizeX.m} $\rightarrow$ pre-processing
\end{enumerate}

\section{Guidelines for usage} \label{Sec:Guidelines}

We present the source code in Matlab in order to compute the CLAD estimator exactly (when technically possible) using MIP following \cite{Bilias:13}.
There are several options on which MIP solver to use in MATLAB.
Here are two options - for MATLAB $\textregistered$ versions before R2014a:

\begin{itemize}
\item
the milp.m solver by Prof. Kevin Tomsovic, for small scale examples (N $\leq$ 30),
which uses the linprog() command for the LP subproblems solution

\item
the cplexmilp.m solver by IBM ILOG CPLEX OPTIMIZATION STUDIO 12.6 $\textregistered$  which is formatted in the milp\_cplex() function we wrote,
and is the state-of-the-art solver for N $\approx$ 500
\end{itemize}

For users of versions after R2014a (including R2014a):

\begin{itemize}
\item
the intlinprog() solver by MATHWORKS $\textregistered$, can be used for medium scale problems (N=200-400 observations)
\end{itemize}

Typical values for p are p $\in$ \{2,3,4,5,6\}. Often CPLEX can tackle problems up to N  $\approx$ 500 and p $\leq$ 5.
\footnote{It is possible to choose the DGPs in such a way that problems with N $\approx$ 800 and p = 5 can be solved}.

\bibliography{bibfile1}

\end{document}


\part{Lecture 04: Monte Carlo Methods}
\title[RL Lecture 04]{Lecture 04: Monte Carlo Methods}
\date{}
\frame{\titlepage}
\frame{\frametitle{Table of Contents}\tableofcontents}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{General Idea and Differences to Dynamic Programming}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Monte Carlo Methods vs. Dynamic Programming %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Monte Carlo Methods vs. Dynamic Programming}
Dynamic Programming:
\begin{itemize}
	\item \hl{Model-based} prediction and control
	\item Planning inside \hl{known MDPs}
\end{itemize}
\vspace{1cm}
\pause
Monte Carlo methods:
\begin{itemize}
	\item \hl{Model-free} prediction and control
	\item Estimating value functions and optimize policies in \hl{unknown MDPs}
	\item But: still assuming finite MDP problems (or problems close to that)
	\item In general: broad class of computational algorithms relying on \hl{repeated random sampling} to obtain numerical results
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% General Monte Carlo Methods' Characteristics %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{General Monte Carlo (MC) Methods' Characteristics}
\begin{itemize}
	\onslide<1->\item \hl{Learning  from experience}, i.e., sequences of samples $\left\langle x_k, u_k, r_{k+1}\right\rangle$
	\onslide<2->\item Main concept: Estimation by \hl{averaging sample returns}
	\onslide<3->\item To guarantee well-defined returns: \hl{limited to episodic tasks}
	\onslide<4->\item Consequence: Estimation and policy updates only possible in an episode-by-episode way compared to step-by-step (online)
\end{itemize}
\onslide<1->\begin{figure}
\includegraphics[width=8cm]{fig/lec04/Monte_Carlo_City.jpg}
\caption{Monte Carlo port \\(source: \href{https://www.flickr.com/photos/flynn_nrg/40890124713/}{www.flickr.com}, by \href{https://www.flickr.com/photos/flynn_nrg/}{Miguel Mendez} \href{https://creativecommons.org/licenses/by/2.0/}{CC BY 2.0})}
\label{fig:MMonte_Carlo_City}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Basic Monte Carlo Prediction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Task Description and Basic Solution %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Task Description and Basic Solution}
\begin{block}{MC prediction problem statement}
\begin{itemize}
	\item Estimate state value $v_{\pi}(x)$ for a given policy $\pi$.
	\item Available are samples $\left\langle x_{k,j}, u_{k,j}, r_{k+1,j}\right\rangle$ for episodes  $j=1,\ldots,J$.
\end{itemize}
\end{block}
\pause
\vspace{0.25cm}
MC solution approach:
\begin{itemize}
	\item Average returns after visiting state $x_k$ over episodes $j=1,\ldots$
	\begin{equation}
		v_{\pi}(x_k) \approx \hat{v}_{\pi}(x_k)=\frac{1}{J}\sum_{j=1}^J g_{k,j}=\frac{1}{J}\sum_{j=1}^J\sum_{i=0}^{T_j} \gamma^i r_{k+i+1,j}\, .
		\label{eq:MC_average_v_basic}
	\end{equation}
	\item Above, $T_j$ denotes the \hl{terminating time step} of each episode $j$.\pause
	\item \hl{First-visit MC}: Apply \eqref{eq:MC_average_v_basic} only to the first state visit per episode.\pause
	\item \hl{Every-visit MC}: Apply \eqref{eq:MC_average_v_basic} each time visiting a certain state per episode (if a state is visited more than one time per episode).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic Implementation:MC-Based Prediction %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: MC-Based Prediction}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a policy $\pi$ to be evaluated}
\Output{estimate of $\bm{v}_{\mathcal{X}}^{\pi}$ (i.e., value estimate for all states				 $x\in\mathcal{X}$)}
\Init{$\hat{v}(x)\, \forall \, x\in\mathcal{X}$ arbitrary except $v_0(x)=0$ if $x$ is terminal\newline
$l(x)\leftarrow$ an empty list for every $x\in\mathcal{X}$}
 \For{$j=1,\ldots,J$ episodes}{
		Generate an episode following $\pi$: $x_{0}, u_{0}, r_{1},\ldots,x_{T_j}, u_{T_j}, r_{T_{j}+1}$ \;
		Set $g \leftarrow 0$\;
		\For{$k=T_j-1, T_j-2, T_j-3,\ldots, 0$ time steps}{
			$g \leftarrow \gamma g + r_{k+1}$\;
			\If{$x_{k}\notin \left\langle x_{0}, x_{1}, \ldots, x_{k-1}\right\rangle$}{
					Append $g$ to list $l(x_k)$\;
					$\hat{v}(x_k)\leftarrow \mbox{average}(\,l(x_k)\,)$\;}
		}
	}
\caption{First-visit MC state-value prediction}
\label{algo:MC_state_prediction_first_visit}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Incremental Implementation%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Incremental Implementation}
\begin{itemize}
	\item \algoref{algo:MC_state_prediction_first_visit} is inefficient due to large memory demand.
	\item Better: use \hl{incremental / recursive implementation}.\pause
	\item The sample mean $\mu_1,\mu_2,\ldots$ of an arbitrary sequence $g_1, g_2, \ldots$ is:
\end{itemize}
\begin{equation}
	\begin{split}
		\mu_j &= \frac{1}{j}\sum_{i=1}^j g_i,\\
					&= \frac{1}{j}\left[g_j + \sum_{i=1}^{j-1} g_i\right],\\
					&= \frac{1}{j}\left[g_j + (j-1) \mu_{j-1}\right],\\
					&=\mu_{j-1} + \frac{1}{j}\left[g_j-\mu_{j-1}\right].
	\end{split}
	\label{eq:inc_impl_MC_pred}
\end{equation} \pause
\begin{itemize}
	\item \hl{If a given decision problem is non-stationary}, using a forgetting factor $\alpha\in\left\{\mathbb{R}|0<\alpha<1\right\}$ allows for dynamic adaption:
\end{itemize}
\begin{equation}
	\mu_j = \mu_{j-1} + \alpha\left[g_j-\mu_{j-1}\right] .
	\label{eq:inc_impl_MC_pred_non_stat}
\end{equation}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Statistical Properties of MC-Based Prediction (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Statistical Properties of MC-Based Prediction (1)}
\onslide<1->{First-time visit MC:
\begin{itemize}
	\item Each return sample $g_j$ is independent from the others since they were drawn from separate episodes.
	\item One receives \hl{i.i.d. data} to estimate $\E{\hat{v}_{\pi}}$ and consequently this \hl{is bias-free}.
	\item The estimate's variance $\Var{\hat{v}_{\pi}}$ drops with $1/n$ ($n$: available samples).
\end{itemize}}
\onslide<2->{Every-time visit MC:
\begin{itemize}
	\item Each return sample $g_j$ is not independent from the others since they might be obtained from same episodes.
	\item One receives \hl{non-i.i.d.} data to estimate $\E{\hat{v}_{\pi}}$ and consequently this \hl{is biased} for any $n<\infty$.
	\item Only in the limit $n\rightarrow \infty$ one receives $\left(v_{\pi}(x)-\E{\hat{v}_{\pi}(x)}\right) \rightarrow 0$.
\end{itemize}}
\footnotesize
\vspace{0.5cm}
\onslide<1->More information: S. Singh and  R. Sutton, "Reinforcement Learning with Replacing Eligibility Traces", Machine Learning, Vol. 22, pp. 123-158, 1996
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Statistical Properties of MC-Based Prediction (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Statistical Properties of MC-Based Prediction (2)}

\begin{itemize}
	\item State-value estimates for each state are independent.
	\item One estimate does not rely on the estimate of other states \newline(\hl{no bootstrapping} compared to DP).
	\item Makes MC particularly attractive when one requires state-value knowledge of only one or few states.
	\begin{itemize}
		\item Hence, generating episodes starting from the state of interest.
	\end{itemize}
\end{itemize}
\begin{figure}
	\subfloat{
		\includegraphics[height=2.5cm]{fig/lec04/Back_Up_DP.pdf}
	}
	\hspace{1cm}
	\subfloat{
		\includegraphics[height=2.5cm]{fig/lec04/Back_Up_MC.pdf}
	}
\caption{Back-up diagrams for DP (left) and MC (right) prediction: shallow one-step back-ups with bootstrapping vs. deep back-ups over full epsiodes}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Predction Example: Forest Tree MDP%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Prediction Example: Forest Tree MDP (1)}
Let's reuse the forest tree MDP example with \textit{fifty-fifty policy} and discount factor $\gamma=0.8$
plus disaster probability $\alpha=0.2$:
\vspace{0.5cm}
\begin{figure}
	%\hspace*{-1.5cm}
	\includegraphics[width=9cm]{fig/lec04/Forest_Markov_Decision_Process_State_Value.pdf}
	\caption{Forest MDP with fifty-fifty-policy including state values}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Predction Example: Forest Tree MDP%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Prediction Example: Forest Tree MDP (2)}

\begin{figure}
	%\hspace*{-1.5cm}
	\includegraphics{fig/lec04/Forest_Tree_MC_Value_Prediction.pdf}
	\caption{State-value estimate of forest tree MDP initial state using MC-based prediction over the number of episodes being evaluated (mean and standard deviation are calculated based on 2000 independent runs)}
	\label{fig:Forest_Tree_MC_Value_Prediction}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Predction Example: Blackjack (1) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Prediction Example: Blackjack (1)}
\begin{itemize}
	\item \hl{States} (200 in total)
	\begin{itemize}
		\item Player's sum: 12-21 (automatically twist if player's sum $<$ 12)
		\item Usable ace (counts 1/11): yes/no
		\item Dealer's initial card: ace-10
	\end{itemize}\pause
	\item Action \hl{stick}: stop receiving cards (terminate episode)\pause
	\item Reward for \hl{stick}:
	\begin{itemize}
		\item +1 if player's sum $>$ dealer's sum
		\item 0 if player's sum $=$ dealer's sum
		\item -1 if player's sum $<$ dealer's sum
	\end{itemize}\pause
	\item Action \hl{twist/hit}: take another card (assuming no replacements)\pause
	\item Reward for \hl{twist/hit}:
	\begin{itemize}
		\item -1 if player's sum $>$  21 ('bust', terminates episode)
		\item 0 otherwise
	\end{itemize}\pause
	\item Dealer always sticks on any sum $\geq$ 17.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Predction Example: Blackjack (2) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Prediction Example: Blackjack (2)}
\begin{figure}
		\includegraphics[width=9cm]{fig/lec04/MC_Black_Jack.pdf}
		\caption{Approximate state-value functions by first-visit MC prediction for a \hl{player's policy that sticks only on 20 or 21} (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
		\label{fig:Blackjack_MC}
\end{figure}
}
%Remarks:
%Blackjack could be modeled as a MDP since one has full knowledge of the system 'dynamics'
%Problem: Analytical/exact calculation of all transition probabilities not easy / a lot of work
%MC offers an alternative: set up a simulation environment with less effort in order bypass the need of calculating the transition probabilities

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC Estimation of Action Values %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC Estimation of Action Values}
Is a \hl{model available} (i.e., tuple $\left\langle\mathcal{X}, \mathcal{U}, \bm{\mathcal{P}}, \mathcal{R}, \gamma \right\rangle$)?
\begin{itemize}
	\item \hl{Yes}: state values plus one-step prediction deliver optimal policy.
	\item \hl{No}: action values are very useful to directly obtain optimal choices.
	\item Recap policy improvement from last lecture.
\end{itemize}\pause
\hl{Estimating $q{_\pi}(x, u)$} using MC approach:
\begin{itemize}
	\item Analog to state values summarized in \algoref{algo:MC_state_prediction_first_visit}.
	\item Only small extension: a visit refers to a state-action pair $(x, u)$.
	\item First-visit and every-visit variants exist.
\end{itemize}\pause
Possible problem when following a deterministic policy $\pi$:
\begin{itemize}
	\item Certain state-action pairs $(x, u)$ are never visited.
	\item Missing degree of exploration.\pause
	\item Workaround: \hl{exploring starts}, i.e., starting episodes in random state-action pairs $(x, u)$ and thereafter following  $\pi$.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Basic Monte Carlo Control}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Applying Generalized Policy Iteration (GPI) to MC Control %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Applying Generalized Policy Iteration (GPI) to MC Control}
\onslide<1->{GPI concept is directly applied to MC framework using action values:
\begin{equation}
\label{eq:GPI_MC}
	\pi_0 \rightarrow \hat{q}_{\pi_0} \rightarrow \pi_1 \rightarrow \hat{q}_{\pi_1} \rightarrow \cdots \pi^* \rightarrow \hat{q}_{\pi^*} \, .
\end{equation}}
\begin{itemize}
	\onslide<2->{\item Degree of freedom: Choose number of episodes to approximate $\hat{q}_{\pi_i}$.}
	\onslide<3->{\item Policy improvement is done by greedy choices:
	\begin{equation}
		\pi(x)= \argmax_{u} q(x, u) \quad \forall x\in\mathcal{X}\, .
	\end{equation}}
\end{itemize}
\onslide<1->{\begin{figure}
\begin{minipage}[c]{0.35\textwidth}
    \includegraphics[width=4cm]{fig/lec04/MC_GPI.pdf}
  \end{minipage}\hfill
  \begin{minipage}[c]{0.65\textwidth}
    \caption{Transferring GPI to MC-based control (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
		\label{fig:GPI_MC}
  \end{minipage}
\end{figure}}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Policy Improvement Theoreom %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Policy Improvement Theorem}
Assuming that one is operating in an \hl{unknown MDP}, the policy improvement theorem \theoref{theo:Policy_improvement} is still valid for MC-based control:
\begin{block}{Policy improvement for MC-based control}
\begin{equation}
	\begin{split}
		q_{\pi_i}(x,\pi_{i+1}(x)) &= q_{\pi_i}(x,\argmax_{u} q_{\pi_i}(x,u)),\\
											&= \max_{u} q_{\pi_i}(x,u),\\
											&\geq q_{\pi_i}(x,\pi_{i}(x)),\\
											&\geq v_{\pi_i}(x).
	\end{split}
\end{equation}
\end{block}
\begin{itemize}
	\item Each $\pi_{i+1}$ is uniformly better or just as good (if optimal) as $\pi_{i}$.
	\item Assumption: All state-action pairs are evaluated due to sufficient exploration.
	\begin{itemize}
		\item For example using exploring starts.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic Implementation: MC-Based Control Using Exploring Start %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation: MC-Based Control}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Output{Optimal deterministic policy $\pi^*$}
\Init{$\pi_{i=0}(x)\in\mathcal{U}$ arbitrarily $\forall x\in\mathcal{X}$\newline
$\hat{q}(x,u)$ arbitrarily $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$\newline
$n(x,u)\leftarrow$ an empty list for state-action visits $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$}
\Repeat{$\pi_{i+1} = \pi_{i}$}{
	$i\leftarrow i+1$ \;
	Choose $\left\{x_{0}, u_{0}\right\}$ randomly such that all pairs have probability $>$ 0 \;
	Generate an episode from $\left\{x_{0}, u_{0}\right\}$ following $\pi_{i}$ until termination step $T_i$\;
	Set $g \leftarrow 0$\;
	\For{$k=T_i-1, T_i-2, T_i-3,\ldots, 0$ time steps}{
				$g \leftarrow \gamma g + r_{k+1}$\;
				\If{$\left\{x_{k}, u_{k}\right\}\notin \left\langle \left\{x_{0}, u_{0}\right\}, \ldots, \left\{x_{k-1}, u_{k-1}\right\}\right\rangle$}{
						$n(x_k,u_k)\leftarrow n(x_k,u_k)+1$\;
						$\hat{q}(x_k,u_k)\leftarrow \hat{q}(x_k,u_k) + 1/n(x_k,u_k)\cdot(g-\hat{q}(x_k,u_k))$\;
						$\pi_{i}(x_k)\leftarrow\argmax_{u}\,\, \hat{q}(x_k,u)$\;
				}
	}
}
\caption{MC-based control using exploring starts (first-visit)}
\label{algo:MC_ES}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Control Example: Blackjack %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Control Example: Blackjack}
\begin{figure}
		\includegraphics[width=9cm]{fig/lec04/MC_ES_Black_Jack.pdf}
		\caption{The optimal policy and state-value function for blackjack found by MC-based control using exploring starts. (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
		\label{fig:MC_ES_Black_Jack}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Extensions to Monte Carlo On-Policy Control}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Off- and On-Policy Learning %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Off- and On-Policy Learning}
\begin{itemize}
	\item \hl{On-policy learning}
	\begin{itemize}
		\item Evaluate or improve the policy used to make decisions.
		\item Agent picks own actions.
		\item Exploring starts (ES) is an on-policy method example.
		\item However: ES is a restrictive assumption and not always applicable \newline (in some cases the starting state-action pair cannot be choosen freely).
	\end{itemize}\pause
	\vspace{1cm}
	\item \hl{Off-policy learning}
		\begin{itemize}
		\item Evaluate or improve a policy different from that used to generate data.
		\item Agent cannot apply own actions.
		\item Will be focused in the next sections.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Epsilon-Greedy as an On-Policy Alternative %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{$\varepsilon$-Greedy as an On-Policy Alternative}
\begin{itemize}
	\item Exploration requirement:
	\begin{itemize}
		\item Visit all state-action pairs with probability:
		\begin{equation}
			 \pi(u|x) > 0 \quad \forall\, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}\, .
		\end{equation}
		\item Policies with this characteristic are called: \hl{soft}.
		\item Level of exploration can be tuned during the learning process.
	\end{itemize}\pause
	\vspace{0.5cm}
	\item \hl{$\varepsilon$-greedy on-policy learning}
		\begin{itemize}
		\item With probability $\varepsilon$ the agent's choice (i.e., the policy output) is overwritten with a random action.
		\item Probability of all non-greedy actions: \begin{equation}\varepsilon/|\mathcal{U}|\,.\end{equation}
		\item Probability of the greedy action: \begin{equation}1-\varepsilon+\varepsilon/|\mathcal{U}|\,.\end{equation}
		\item Above, $|\mathcal{U}|$ is the cardinality of the action space.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic implementation $\varepsilon$-Greedy MC-Control %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Algorithmic Implementation $\varepsilon$-Greedy MC-Control}
\vspace{-0.2cm}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Output{Optimal $\varepsilon$-greedy policy $\pi^*(u|x)$}
\Param{$\varepsilon\in\left\{\mathbb{R}|0<\varepsilon<<1\right\}$}
\Init{$\pi_{i=0}(u|x)$ arbitrarily soft $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$\newline
$\hat{q}(x,u)$ arbitrarily $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$\newline
$n(x,u)\leftarrow$ an empty list counting state-action visits $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$}
\Repeat{$\pi_{i+1} = \pi_{i}$}{
	Generate an episode following $\pi_i$: $x_{0}, u_{0}, r_{1},\ldots,x_{T_j}, u_{T_j}, r_{T_{j+1}}$ \;
	$i\leftarrow i+1$ \;
	Set $g \leftarrow 0$\;
	\For{$k=T_i-1, T_i-2, T_i-3,\ldots, 0$ time steps}{
				$g \leftarrow \gamma g + r_{k+1}$\;
				\If{$\left\{x_{k}, u_{k}\right\}\notin \left\langle \left\{x_{0}, u_{0}\right\}, \ldots, \left\{x_{k-1}, u_{k-1}\right\}\right\rangle$}{
						$n(x_k,u_k)\leftarrow n(x_k,u_k)+1$\;
						$\hat{q}(x_k,u_k)\leftarrow \hat{q}(x_k,u_k) + 1/n(x_k,u_k)\cdot(g-\hat{q}(x_k,u_k))$\;
						$\tilde{u}\leftarrow\argmax_{u}\,\, \hat{q}(x_k,u)$\;
						$\pi_i(u|x_k)\leftarrow\begin{cases}1-\varepsilon+\varepsilon/|\mathcal{U}|, \quad u=\tilde{u}\\ \varepsilon/|\mathcal{U}|, \quad u\neq\tilde{u}\end{cases}$\;
				}
	}
}
\caption{MC-based control using $\varepsilon$-greedy approach}
\label{algo:MC_eps_greedy}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% $\varepsilon$-Greedy Policy Improvement (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{$\varepsilon$-Greedy Policy Improvement (1)}
\begin{theo}{Policy improvement for $\varepsilon$-greedy policy}{policy_improv_eps_greedy}
Given an MDP, for any $\varepsilon$-greedy policy $\pi$ the $\varepsilon$-greedy policy $\pi'$ with respect to $q_\pi$ is an improvement, i.e., $v_{\pi'} > v_{\pi} \quad \forall x\in\mathcal{X}$.
\end{theo}\pause
Small proof:
\small
\begin{equation}
	\begin{split}
		q_{\pi}(x,\pi'(x)) &= \sum_{u}\pi'(u|x)q_{\pi}(x,u),\\
											&= \frac{\varepsilon}{|\mathcal{U}|}\sum_{u}q_{\pi}(x,u) + (1-\varepsilon)\max_{u}q_{\pi}(x,u),\\
											&\geq \frac{\varepsilon}{|\mathcal{U}|}\sum_{u}q_{\pi}(x,u) + (1-\varepsilon)\sum_{u}\frac{\pi(u|x)-\frac{\varepsilon}{|\mathcal{U}|}}{1-\varepsilon}q_{\pi}(x,u).
\end{split}
\end{equation}
\normalsize
In the inequality line, the second term is the weighted sum over action values given an $\varepsilon$-greedy policy. This weighted sum will be always smaller or equal than  $\max_{u}q_{\pi}(x,u)$.
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% $\varepsilon$-Greedy Policy Improvement (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{$\varepsilon$-Greedy Policy Improvement (2)}
Continuation:
\small
\begin{equation}
	\begin{split}
		q_{\pi}(x,\pi'(x)) &\geq \frac{\varepsilon}{|\mathcal{U}|}\sum_{u}q_{\pi}(x,u) + (1-\varepsilon)\sum_{u}\frac{\pi(u|x)-\frac{\varepsilon}{|\mathcal{U}|}}{1-\varepsilon}q_{\pi}(x,u),\\
											&= \frac{\varepsilon}{|\mathcal{U}|}\sum_{u}\left(q_{\pi}(x,u) - q_{\pi}(x,u)\right) + \sum_{u}\pi(u|x)q_{\pi}(x,u),\\
											&=\sum_{u}\pi(u|x)q_{\pi}(x,u),\\
											&=v_{\pi}(x).
	\end{split}
\end{equation}
\normalsize\pause
\begin{itemize}
	\item Policy improvement theorem is still valid when comparing $\varepsilon$-greedy policies against each other.
	\item But: There might be a non-$\varepsilon$-greedy policy which is better.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Control Example: Forest Tree MDP (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Control Example: Forest Tree MDP (1)}
\vspace{-0.5cm}
\begin{figure}
	\includegraphics{fig/lec04/Forest_Tree_MC_Control_eps_02.pdf}
	\caption{Different estimates of forest tree MDP ($\alpha=0.2, \gamma=0.8$) using MC control with \hl{$\varepsilon=0.2$} over the number of episodes. Mean is red and standard deviation is light blue, both calculated based on 2000 independent uns.}
	\label{fig:Forest_Tree_MC_Control_eps_02}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Control Example: Forest Tree MDP (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Control Example: Forest Tree MDP (2)}
\vspace{-0.5cm}
\begin{figure}
	\includegraphics{fig/lec04/Forest_Tree_MC_Control_eps_005.pdf}
	\caption{Different estimates of forest tree MDP ($\alpha=0.2, \gamma=0.8$) using MC control with \hl{$\varepsilon=0.05$} over the number of episodes. Mean is red and standard deviation is light blue, both calculated based on 2000 independent runs.}
	\label{fig:Forest_Tree_MC_Control_eps_005}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Control Example: Forest Tree MDP (3)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Control Example: Forest Tree MDP (3)}
Observations on forest tree MDP with $\varepsilon$-greedy MC-based control:
\begin{itemize}
	\item Rather slow convergence rate: quite a number of episodes is required. \pause
	\item Significant uncertainty present in a single sequence. \pause
	\item Later states are less often visited and, therefore, more uncertain. \pause
	\item Exploration is controlled by $\varepsilon$: in a totally greedy policy the state $x=3$ is not visited at all (cf. \figref{fig:Forest_Markov_Decision_Process_Optimal_Action_Value}). With $\varepsilon$-greedy this state is visited occasionally. \pause
	\item Nevertheless, the above figures highlight that MC-based control for the forest tree MDP tend towards the optimal policies discovered by dynamic programming (cf. \tabref{tab:Forest_tree_value_iteration}).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Greedy in the Limit with Infinite Exploration (GLIE) %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Greedy in the Limit with Infinite Exploration (GLIE)}
\begin{defi}{Greedy in the limit with infinite exploration (GLIE)}{GLIE}
A learning policy $\pi$ is called GLIE if it satisfies the following two properties:
\begin{itemize}
	\item If a state is visited infinitely often, then each action is chosen infinitely often:
	\begin{equation}
		\lim_{i\rightarrow\infty} \pi_i(u|x)=1 \quad \forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}\, .
	\end{equation}
	\item In the limit ($i\rightarrow \infty$) the learning policy is greedy with respect to the learned action value:
	\begin{equation}
		\lim_{i\rightarrow\infty} \pi_i(u|x)=\pi(x)= \argmax_{u} q(x, u) \quad \forall x\in\mathcal{X} \, .
	\end{equation}
\end{itemize}
\end{defi}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% GlIE Monte-Carlo Control %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{GLIE Monte Carlo Control}
\begin{theo}{Optimal decision using MC-control with $\varepsilon$-greedy}{GLIE_MC}
MC-based control using $\varepsilon$-greedy exploration is GLIE, if $\varepsilon$ is decreased at rate
\begin{equation}
	\varepsilon_i = \frac{1}{i}
\end{equation}
with $i$ being the increasing episode index. In this case,
\begin{equation}
	\hat{q}(x, u) = q^*(x, u)
\end{equation}
follows.
\end{theo}
\pause
Remarks:
\begin{itemize}
	\item Limited feasibility: infinite number of episodes required.
	\item $\varepsilon$-greedy is an undirected exploration strategy. Can that be the most efficient way of learning?
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Monte Carlo Off-Policy Prediction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Off-Policy Learning Background %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Off-Policy Learning Background}
Drawback of on-policy learning:
\begin{itemize}
	\item Only a compromise: comes with inherent exploration but at the cost of learning action values for a \hl{near-optimal policy}.
\end{itemize}
\vspace{0.75cm}\pause
Idea off-policy learning:
\begin{itemize}
	\item Use two separated policies:
	\begin{itemize}
		\item \hl{Behavior policy} $b(u|x)$: Explores in order to generate experience.
		\item \hl{Target policy} $\pi(u|x)$: Learns from that experience to become the optimal policy.
	\end{itemize}\pause
	\item Use cases:
	\begin{itemize}
		\item Learn from observing humans or other agents/controllers.
		\item Re-use experience generated from old policies ($\pi_0,\pi_1,\ldots$).
		\item Learn about multiple policies while following one policy.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Off-Policy Prediction Problem Statement %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Off-Policy Prediction Problem Statement}
\begin{block}{MC off-policy prediction problem statement}
\begin{itemize}
	\item Estimate $v_\pi$ and/or $q_\pi$ while following $b(u|x)$.
	\item Both policies are considered fixed (prediction assumption).
\end{itemize}
\end{block}
\vspace{0.75cm}\pause
Requirement:
\begin{itemize}
	\item \hl{Coverage}: Every action taken under $\pi$ must be (at least occasionally) taken under $b$, too. Hence, it follows:
	\begin{equation}
		\pi(u|x) > 0 \Rightarrow b(u|x) > 0 \quad \forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\} .
	\end{equation}\pause
	\item Consequences from that:
	\begin{itemize}
		\item In any state $b$ is not identical to $\pi$, $b$ must be stochastic.
		\item Nevertheless: $\pi$ might be deterministic (e.g., control applications) or stochastic.
	\end{itemize}
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Importance Sampling %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Importance Sampling}
What is the probability of observing a certain trajectory on random variables $U_k, X_{k+1}, U_{k+1}, \ldots, X_{T}$ starting in $X_{k}$ while following $\pi$?
\begin{equation}
\begin{split}
	&\Pb{U_k, X_{k+1}, U_{k+1}, \ldots , X_{T}| X_{k}, \pi},\\
	=&\pi(U_k|X_{k})p(X_{k+1}|X_{k}, U_{k})\pi(U_{k+1}|X_{k+1})\cdots\,,\\
	=&\prod_k^{T-1} \pi(U_k|X_{k})p(X_{k+1}|X_{k}, U_{k}).
	\end{split}
\end{equation}
Above $p$ is the state-transition probability (cf. \defref{defi:Markov_decision_process}).\pause
\begin{defi}{Importance sampling ratio}{import_sampl_ratio}
The relative probability of a trajectory under the target and behavior policy, the importance sampling ratio, from sample step $k$ to $T$ is:
\begin{equation}
	\rho_{k:T}=\frac{\prod_k^{T-1} \pi(U_k|X_{k})p(X_{k+1}|X_{k}, U_{k})}{\prod_k^{T-1} b(U_k|X_{k})p(X_{k+1}|X_{k}, U_{k})}=\frac{\prod_k^{T-1} \pi(U_k|X_{k})}{\prod_k^{T-1} b(U_k|X_{k})} .
	\label{eq:import_sampl_ratio}
\end{equation}
\end{defi}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Importance Sampling for Monte Carlo Prediction (1)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Importance Sampling for Monte Carlo Prediction (1)}
\begin{defi}{State-value estimation via Monte Carlo importance sampling}{MC_ord_import_sampl}
Estimating the state value $v_\pi$ following a behavior policy $b$ using (ordinary) importance sampling (OIS) results in scaling and averaging the sampled returns by the importance sampling ratio per episode:
\begin{equation}
	\hat{v}_\pi(x_k)=\frac{\sum_{k\in\mathcal{T}(x_k)}\rho_{k:T(k)}g_k}{|\mathcal{T}(x_k)|}.
	\label{eq:OIS}
\end{equation}
\end{defi}
Notation remark:
\begin{itemize}
	\item $\mathcal{T}(x_k)$: set of all time steps in which the state $x_k$ is visited the first or each time per episode.
	\item $T(k)$: Termination of a specific episode starting from $k$.
\end{itemize}\pause
General remark:
\begin{itemize}
	\item From \eqref{eq:import_sampl_ratio} it can be seen that $\hat{v}$ is bias-free (first-visit assumption).
	\item However, if $\rho$ is large (distinctly different policies) the estimate's variance is large (i.e., uncertain for small numbers of samples).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Importance Sampling for Monte Carlo Prediction (2)%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Importance Sampling for Monte Carlo Prediction (2)}
\begin{defi}{Weighted importance sampling for MC-based prediction}{MC_weighted_import_sampl}
Estimating the state value $v_\pi$ following a behavior policy $b$ using weighted importance sampling (WIS) results in a scaling and a weighted averaging of the sampled returns by the importance sampling ratio per episode:
\begin{equation}
\begin{split}
	\hat{v}_\pi(x_k)= \begin{cases}\frac{\sum_{k\in\mathcal{T}(x_k)}\rho_{k:T(k)}g_k}{\sum_{k\in\mathcal{T}(x_k)}\rho_{k:T(k)}}, \quad &\mbox{if}\,\,\sum_{k\in\mathcal{T}(x_k)}\rho_{k:T(k)}\neq0 \\ 0, &\mbox{if}\,\,\sum_{k\in\mathcal{T}(x_k)}\rho_{k:T(k)} = 0\quad \end{cases}.
\end{split}
\label{eq:WIS_MC}
\end{equation}
\end{defi}\pause
Comparison to OIS:
\begin{itemize}
	\item Weighting introduces bias (which reduces to zero in the limit).\pause
	\item But: weighting limits variance while the OIS's variance is unbounded.
	\begin{itemize}
		\item Largest weight per return sample for WIS is one, while OIS's weights are unbounded.
	\end{itemize}\pause
	\item Conclusion: WIS delivers 'better' estimates for few samples and, therefore, is often preferred in practice.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Importance Sampling Example: Blackjack%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Importance Sampling Example: Blackjack}
\begin{itemize}
	\item Behavior policy: stick or hit at equal probability.
	\item Target policy: stick if player has sum of 20 or 21 (cf. \figref{fig:Blackjack_MC}).
	\item True state value of target policy is $v=-0.2773$.\pause
\end{itemize}
\begin{figure}
	\includegraphics[width=8cm]{fig/lec04/Importance_Sampling_Blackjack.pdf}
	\caption{Off-policy state-value estimation of single state $\{$dealer's sum = 2, player's sum = 13 and player has usable ace$\}$ (source: R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018, \href{https://creativecommons.org/licenses/by-nc-nd/2.0/}{CC BY-NC-ND 2.0})}
	\label{fig:Importance_Sampling_Blackjack}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Weighted Importance Sampling: Incremental Implementation %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{WIS: Incremental Implementation}
\begin{itemize}
	\item For OIS, \eqref{eq:OIS} can be directly transformed into an incremental implementation following \eqref{eq:inc_impl_MC_pred}.\pause
	\item For WIS an incremental implementation is also highly desirable to save memory and computational resources.\pause
	\item Rewrite WIS \eqref{eq:WIS_MC} for a sequence of returns $g_1, g_2,\ldots$:
	\begin{equation}
		\hat{v}_i(x) = \frac{\sum_{k=1}^{i-1} w_k g_k}{\sum_{k=1}^{i-1} w_k} \quad \mbox{with} \quad w_k=\rho_{k:T(k)} .
	\end{equation}\pause
	\item The WIS recursive/incremental update rule is then:
	\begin{align}
	\hat{v}_{i+1}(x) &= \hat{v}_{i}(x) + \frac{w_i}{c_i}\left(g_i - \hat{v}_{i}(x)\right),\quad i \geq 1 \\
	c_{i+1} &= c_i + w_{i+1} \quad \mbox{with} \quad c_0=0.
	\end{align}
	\item Above, $c_i$ is the cumulative sum of weights over all considered state visits.
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic Implementation: MC-Based Off-Policy Prediction Using WIS %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Off-Policy Prediction Using WIS}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Input{a target policy $\pi$ to be evaluated}
\Init{$\hat{q}(x,u)$ arbitrarily $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$\newline
$c(x,u)\leftarrow 0$ cumulative sum of WIS weights $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$}
 \For{$j=1,\ldots,J$ episodes}{
	Choose an arbitrary behavior policy $b$ with coverage of $\pi$\;
	Generate an episode following $b$: $x_{0}, u_{0}, r_{1},\ldots,x_{T_j}, u_{T_j}, r_{T_{j+1}}$ \;
	Set $g \leftarrow 0$\;
	Set $w \leftarrow 1$\;
	\For{$k=T_i-1, T_i-2, T_i-3,\ldots, 0$ time steps while $w \neq 0$}{
				$g \leftarrow \gamma g + r_{k+1}$\;
				$c(x_k,u_k) \leftarrow c(x_k,u_k) + w$\;
				$\hat{q}(x_k,u_k) \leftarrow \hat{q}(x_k,u_k)+\frac{w}{c(x_k,u_k)}\left(g - \hat{q}(x_k,u_k)\right)$\;
				$w \leftarrow w\frac{\pi(u_k|x_{k})}{b(u_k|x_{k})}$\;
	}
}
\caption{MC-based off-policy prediction using WIS}
\label{algo:MC_off_policy_pred_WIS}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Off-Policy Prediction Using WIS: Forest Tree MDP %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Off-Policy Prediction: Forest Tree MDP}
\begin{itemize}
	\item Behavior policy $b$: '\textit{fifty-fifty}'
	\item Target policy $\pi$: optimal policy from \figref{fig:Forest_Markov_Decision_Process_Optimal_Action_Value}
\end{itemize}
\vspace{-0.5cm}
\begin{figure}
	\hspace*{-1.0cm}
	\includegraphics{fig/lec04/Forest_Tree_MC_Off_Policy_Prediction_WIS.pdf}
	\caption{Action-value estimates of forest tree MDP ($\alpha=0.2, \gamma=0.8$) using MC-based off-policy prediction with WIS over the number of episodes. Mean is red and standard deviation is light blue, both calculated based on 2000 independent runs.}
	\label{fig:Forest_Tree_MC_Off_Policy_Prediction_WIS}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Monte Carlo Off-Policy Control}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Table of Contents}
\tableofcontents[currentsection]
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Off-Policy Monte Carlo Control: Introduction %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Off-Policy Monte Carlo Control: Introduction}
Just put everything together:
\begin{itemize}
	\item MC-based control utilizing GPI (cf. \figref{fig:GPI_MC}),
	\item Off-policy learning based on \algoref{algo:MC_off_policy_pred_WIS}.
\end{itemize}
\vspace{1cm}
Requirement for off-policy MC-based control:
\begin{itemize}
	\item \hl{Coverage}: behavior policy $b$  has nonzero probability of selecting actions that might be taken by the target policy $\pi$.
	\item Consequence: behavior policy $b$ is \hl{soft} (e.g., $\varepsilon$-soft).
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Algorithmic Implementation: MC-Based Off-Policy Control Using WIS %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Off-Policy Control Using WIS}
\vspace{-0.175cm}
\setlength{\algomargin}{0.5em}
\begin{algorithm}[H]
\footnotesize
\SetKwInput{Input}{input}
\SetKwInput{Output}{output}
\SetKwInput{Init}{init}
\SetKwInput{Param}{parameter}
\Init{$\hat{q}(x,u)$ arbitrarily $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$\newline
$c(x,u)\leftarrow 0$ cumulative sum of WIS weights $\forall \, \left\{x\in\mathcal{X}, u\in\mathcal{U}\right\}$\newline
$\pi(x) \leftarrow \argmax_{u} \hat{q}(x,u)$ (with ties broken consistently)}
 \For{$j=1,\ldots,J$ episodes}{
	Choose an arbitrary soft policy $b$\;
	Generate an episode following $b$: $x_{0}, u_{0}, r_{1},\ldots,x_{T_j}, u_{T_j}, r_{T_{j+1}}$ \;
	Set $g \leftarrow 0$\;
	Set $w \leftarrow 1$\;
	\For{$k=T_i-1, T_i-2, T_i-3,\ldots, 0$ time steps}{
				$g \leftarrow \gamma g + r_{k+1}$\;
				$c(x_k,u_k) \leftarrow c(x_k,u_k) + w$\;
				$\hat{q}(x_k,u_k) \leftarrow \hat{q}(x_k,u_k)+\frac{w}{c(x_k,u_k)}\left(g - \hat{q}(x_k,u_k)\right)$\;
				$\pi(x_k) \leftarrow \argmax_{u_k} \hat{q}(x_k,u_k)$ (with ties broken consistently)\;
				\If{$u_k\neq\pi(x_k)$}{
						break/exit inner loop\;
				}
				$w \leftarrow w\frac{1}{b(u_k|x_{k})}$\;
	}
}
\caption{MC-based off-policy control using WIS}
\label{algo:MC_off_policy_control_WIS}
\end{algorithm}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% MC-Based Off-Policy Control Using WIS: Forest Tree MDP %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{MC-Based Off-Policy Control: Forest Tree MDP}
\begin{itemize}
	\item Behavior policy $b$: '\textit{fifty-fifty}'
	\item Initial target policy $\pi$: '\textit{fifty-fifty}'
\end{itemize}
\vspace{-0.5cm}
\begin{figure}
	\hspace*{-1.0cm}
	\includegraphics{fig/lec04/Forest_Tree_MC_Off_Policy_Control_WIS.pdf}
	\caption{Action-value estimates and policy trends of forest tree MDP ($\alpha=0.2, \gamma=0.8$) using MC-based off-policy control with WIS over the number of episodes. Mean is red and standard deviation is light blue, both calculated based on 2000 independent runs.}
	\label{fig:Forest_Tree_MC_Off_Policy_Control_WIS}
\end{figure}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Remark on MC-Based Off-Policy Control with WIS %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{Remark on MC-Based Off-Policy Control with WIS}
Potential problems with MC-based off-policy control:
\begin{itemize}
	\item Learns only 'from tails' of episodes if the remaining actions generated by $b$ are greedy with respect to $\pi$.
	\begin{itemize}
		\item See last and inner-most \textit{if} query in \algoref{algo:MC_off_policy_control_WIS}.
		\item Hence, a lot of samples generated by $b$ remain unused for training $\pi$.
	\end{itemize}\pause
	\item Slows down training.
	\item Adds uncertainty to the training process.
\end{itemize}
\vspace{0.5cm}\pause
Possible improvements/extensions:
\begin{enumerate}
	\item \hl{Per-decision importance sampling}
	\item \hl{Discounting-aware importance sampling} (if discount factor $\gamma<1$)
\end{enumerate}
\begin{itemize}
	\item Introduce modified value estimators for importance sampling.
	\item Main goal: decrease variance of the learning process.
	\item For details see chapter 5.8 \& 5.9 of R. Sutton and G. Barto, Reinforcement learning: an introduction, 2018 .
\end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Summary %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
\frametitle{Summary: What You've Learned Today}
\begin{itemize}
	\item MC methods allow model-free learning of value functions and optimal policies from experience in the form of sampled episodes.
	\item Using deep back-ups over full episodes, MC is largely based on averaging returns. \pause
	\item MC-based control reuses generalized policy iteration (GPI), i.e., mixing policy evaluation and improvement. \pause
	\item Maintaining sufficient exploration is important:
	\begin{itemize}
		\item Exploring starts: not feasible in all applications but simple.
		\item On-policy $\varepsilon$-greedy learning: trade-off between optimality and exploration cannot be resolved easily.
		\item Off-policy learning: agent learns about a (possibly deterministic) target policy from an exploratory, soft behavior policy.
	\end{itemize}\pause
	\item Importance sampling transforms expectations from the behavior to the target policy.
	\begin{itemize}
		\item This estimation task comes with a bias-variance-dilemma.
		\item Slow learning can result from ineffective experience usage in MC methods.
	\end{itemize}
\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Final Slide %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\frame{\frametitle{The End for Today}
\begin{figure}
\includegraphics[width=10cm]{fig/lec04/Dilbert.png}
\end{figure}
\vspace{1cm}
\centering
Thanks for your attention and have a nice week!
}


\documentclass[sigplan]{acmart}
\usepackage{coqtheorem}

\theoremstyle{coqtheorem}
\newcoqtheorem{theorem}{Theorem}
\newcoqtheorem{lemma}{Lemma}
\newcoqtheorem{fact}{Fact}

% Note that the double braces are mandatory
\setBaseUrl{{http://www.ps.uni-saarland.de/courses/cl-ss16/LectureNotes/html/}}
\setCoqFilename{LectureNotes.Base}

\begin{document}

\section{Bla}

% Leave the name field empty if you do not want a name
\begin{theorem}[][DM_or]
  DM for or holds.
\end{theorem}

\begin{lemma}[Name]
  Some lemma that is not in Coq.
\end{lemma}

\setCoqFilename{LectureNotes.Chapter1}
\begin{fact}[Boolean and is commutative][andb_com]
  Commutativity for and holds.
\end{fact}

\begin{fact}~
  \begin{enumerate}
  \coqitem[plus_O] $x + 0 = x$
  \coqitem[plus_S] $x + S y = S (x + y)$
  \item Thus $x + y = y + x$ (\coqlink[plus_com]{Link})
\end{enumerate}
\end{fact}

In Theorem \ref{coq:DM_or} we say something about logical or, in
Fact~\ref{coq:andb_com} something about boolean and.

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End: