    \filetitle{xsf}{Power spectrum and spectral density of model variables}{model/xsf}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
[S,D,List] = xsf(M,Freq,...)
[S,D,List,Freq] = xsf(M,NFreq,...)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{M} {[} model {]} - Model object.
\item
  \texttt{Freq} {[} numeric {]} - Vector of frequencies at which the
  XSFs will be evaluated.
\item
  \texttt{NFreq} {[} numeric {]} - Total number of requested
  frequencies; the frequencies will be evenly spread between 0 and
  \texttt{pi}.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\item
  \texttt{S} {[} namedmat \textbar{} numeric {]} - Power spectrum
  matrices.
\item
  \texttt{D} {[} namedmat \textbar{} numeric {]} - Spectral density
  matrices.
\item
  \texttt{List} {[} cellstr {]} - List of variable in order of
  appearance in rows and columns of \texttt{S} and \texttt{D}.
\item
  \texttt{Freq} {[} numeric {]} - Vector of frequencies at which the
  XSFs has been evaluated.
\end{itemize}

\paragraph{Options}\label{options}

\begin{itemize}
\item
  \texttt{'applyTo='} {[} cellstr \textbar{} char \textbar{}
  \emph{\texttt{Inf}} {]} - List of variables to which the option
  \texttt{'filter='} will be applied; \texttt{Inf} means all variables.
\item
  \texttt{'filter='} {[} char \textbar{} \emph{empty} {]} - Linear
  filter that is applied to variables specified by `applyto'.
\item
  \texttt{'nFreq='} {[} numeric \textbar{} \emph{\texttt{256}} {]} -
  Number of equally spaced frequencies over which the `filter' is
  numerically integrated.
\item
  \texttt{'matrixFmt='} {[} \emph{\texttt{'namedmat'}} \textbar{}
  \texttt{'plain'} {]} - Return matrices \texttt{S} and \texttt{D} as
  either \href{namedmat/Contents}{\texttt{namedmat}} objects
  (i.e.~matrices with named rows and columns) or plain numeric arrays.
\item
  \texttt{'progress='} {[} \texttt{true} \textbar{}
  \emph{\texttt{false}} {]} - Display progress bar on in the command
  window.
\item
  \texttt{'select='} {[} \emph{\texttt{@all}} \textbar{} char \textbar{}
  cellstr {]} - Return XSF for selected variables only; \texttt{@all}
  means all variables.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\subsection{Implementation of Gatys style transfer}
As a part of our project we have implemented a video version of the traditional style transfer proposed in Gatys\cite{Gatys:1}, where each frame is individually generated in the way described here and put together as a new video.\newline\newline
This was the first style transfer method we implemented, as it is the most known well method and also one of the simplest. Gatys style transfer is also the basis for many other research papers and is often referenced and used to compare results with. This includes the research papers we have discussed in the Related Works section. Our implementation also serves as the basis for our later implementations of Ruder \cite{Ruder:1} and Huang \cite{Huang:1} style transfer.  \newline\newline
The main method in which we generate our images is that we have two loss functions with separate weights. One loss function is for style loss and the other one is for content loss, where we use Gram Matrices to calculate the distances for the loss of style. We also implemented total variation loss to minimize the high frequency artifacts as presented by \cite{Zhaoyou:1}. Then we train our model to minimize the loss. All this is explained more in depth in Section \ref{sec:gatys}.\newline\newline
As described earlier, for Gatys implementation we run the style-image, content image and our generated image through the pre-trained VGG-19 network and extract pre-selected layers for content and style. In our implementation we use \textit{conv5.2} for content. For style we use 5 layers in the network: \textit{conv1.1, conv2.1, conv3.1, conv4.1 and conv5.1}\newline\newline
The original implementation of Gatys style transfer is made primarily to be used for style transfer on single images. But in our implementation we are making video style transfer. To do this we created a method to split a video into a list of all the single images in a video, with a frame rate chosen by us. Then we can use the Gatys style transfer on the images on by one, and at the end put them back together to a video. For the style transfer we use tensorflow, and to extract frames from videos and generating new videos with our generated frames, we use the library cv2.\newline\newline
As Gatys style transfer is primarily made for image style transfer, there arises some problems when we try to implement it in video form. When we train each frame in the video separate from each other, the randomness in the training will cause there to a different styling for each image. That means two frames which are almost identical will end up being quite different when styled. This causes the video to have a lot of noise and movement. In some cases this produces a very nice effect and can actually look better than other more advanced implementations. But in other cases it would be preferable to have less noise. \newline\newline

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.15]{report/Method/images/gatys_adjacent.png}
\caption{Two consecutive frames produced with ruder style transfer with the La Muse (picasso) as style reference}
\label{fig:Adjacent frames with ruder}
\end{center}
\end{figure}\newline
In the example above, we have two adjacent frames which show the randomness that appears. On the wall to the right there is some styling effect on the first frame, which does not appear in the next frame.

This leads us to our next implementation, which is based on the paper of Ruder \textit{et al.} \cite{Ruder:1}. In this implementation we try to improve the video style transfer we implemented from Gatys by introducing a temporal component to the loss function.


\hypertarget{structIir_1_1Custom_1_1SOSCascade}{}\doxysubsection{Iir\+::Custom\+::S\+O\+S\+Cascade$<$ N\+S\+OS, State\+Type $>$ Struct Template Reference}
\label{structIir_1_1Custom_1_1SOSCascade}\index{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}}

{\ttfamily \#include $<$Custom.\+h$>$}

Inheritance diagram for Iir\+::Custom\+::S\+O\+S\+Cascade$<$ N\+S\+OS, State\+Type $>$\+:\begin{figure}[H]
\begin{center}
\leavevmode
\includegraphics[height=2.000000cm]{structIir_1_1Custom_1_1SOSCascade}
\end{center}
\end{figure}
\doxysubsubsection*{Public Member Functions}
\begin{DoxyCompactItemize}
\item
\mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade_a0d7f370062345ba03190d8484977d018}{S\+O\+S\+Cascade}} ()
\item
\mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade_a60654fe4bd861799dff82a6dce9dfd42}{S\+O\+S\+Cascade}} (const double(\&sos\+Coefficients)\mbox{[}N\+S\+OS\mbox{]}\mbox{[}6\mbox{]})
\item
void \mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade_aae603eb1bd4330411a6003e09ba36efc}{setup}} (const double(\&sos\+Coefficients)\mbox{[}N\+S\+OS\mbox{]}\mbox{[}6\mbox{]})
\end{DoxyCompactItemize}

\doxysubsubsection{Detailed Description}
\subsubsection*{template$<$int N\+S\+OS, class State\+Type = Direct\+Form\+II$>$\newline
struct Iir\+::\+Custom\+::\+S\+O\+S\+Cascade$<$ N\+S\+O\+S, State\+Type $>$}

A custom cascade of 2nd order (S\+OS / biquads) filters.
\begin{DoxyParams}{Parameters}
{\em N\+S\+OS} & The number of 2nd order filters / biquads. \\
\hline
{\em State\+Type} & The filter topology\+: \mbox{\hyperlink{classIir_1_1DirectFormI}{Direct\+FormI}}, \mbox{\hyperlink{classIir_1_1DirectFormII}{Direct\+Form\+II}}, ... \\
\hline
\end{DoxyParams}

\doxysubsubsection{Constructor \& Destructor Documentation}
\mbox{\Hypertarget{structIir_1_1Custom_1_1SOSCascade_a0d7f370062345ba03190d8484977d018}\label{structIir_1_1Custom_1_1SOSCascade_a0d7f370062345ba03190d8484977d018}}
\index{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}!SOSCascade@{SOSCascade}}
\index{SOSCascade@{SOSCascade}!Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}}
\doxyparagraph{\texorpdfstring{SOSCascade()}{SOSCascade()}\hspace{0.1cm}{\footnotesize\ttfamily [1/2]}}
{\footnotesize\ttfamily template$<$int N\+S\+OS, class State\+Type  = Direct\+Form\+II$>$ \\
\mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade}{Iir\+::\+Custom\+::\+S\+O\+S\+Cascade}}$<$ N\+S\+OS, State\+Type $>$\+::\mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade}{S\+O\+S\+Cascade}} (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})}

Default constructor which creates a unity gain filter of N\+S\+OS biquads. Set the filter coefficients later with the \mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade_aae603eb1bd4330411a6003e09ba36efc}{setup()}} method. \mbox{\Hypertarget{structIir_1_1Custom_1_1SOSCascade_a60654fe4bd861799dff82a6dce9dfd42}\label{structIir_1_1Custom_1_1SOSCascade_a60654fe4bd861799dff82a6dce9dfd42}}
\index{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}!SOSCascade@{SOSCascade}}
\index{SOSCascade@{SOSCascade}!Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}}
\doxyparagraph{\texorpdfstring{SOSCascade()}{SOSCascade()}\hspace{0.1cm}{\footnotesize\ttfamily [2/2]}}
{\footnotesize\ttfamily template$<$int N\+S\+OS, class State\+Type  = Direct\+Form\+II$>$ \\
\mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade}{Iir\+::\+Custom\+::\+S\+O\+S\+Cascade}}$<$ N\+S\+OS, State\+Type $>$\+::\mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade}{S\+O\+S\+Cascade}} (\begin{DoxyParamCaption}\item[{const double(\&)}]{sos\+Coefficients\mbox{[}\+N\+S\+O\+S\mbox{]}\mbox{[}6\mbox{]} }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}}

Python scipy.\+signal-\/friendly setting of coefficients. Initialises the coefficients of the whole chain of biquads / S\+OS. The argument is a 2D array where the 1st dimension holds an array of 2nd order biquad / S\+OS coefficients. The six S\+OS coefficients are ordered \char`\"{}\+Python\char`\"{} style with first the F\+IR coefficients (B) and then the I\+IR coefficients (A). The 2D const double array needs to have exactly the size \mbox{[}N\+S\+OS\mbox{]}\mbox{[}6\mbox{]}.
\begin{DoxyParams}{Parameters}
{\em sos\+Coefficients} & 2D array Python style sos\mbox{[}N\+S\+OS\mbox{]}\mbox{[}6\mbox{]}. Indexing\+: 0-\/2\+: F\+I\+R-\/, 3-\/5\+: I\+I\+R-\/coefficients. \\
\hline
\end{DoxyParams}

\doxysubsubsection{Member Function Documentation}
\mbox{\Hypertarget{structIir_1_1Custom_1_1SOSCascade_aae603eb1bd4330411a6003e09ba36efc}\label{structIir_1_1Custom_1_1SOSCascade_aae603eb1bd4330411a6003e09ba36efc}}
\index{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}!setup@{setup}}
\index{setup@{setup}!Iir::Custom::SOSCascade$<$ NSOS, StateType $>$@{Iir::Custom::SOSCascade$<$ NSOS, StateType $>$}}
\doxyparagraph{\texorpdfstring{setup()}{setup()}}
{\footnotesize\ttfamily template$<$int N\+S\+OS, class State\+Type  = Direct\+Form\+II$>$ \\
void \mbox{\hyperlink{structIir_1_1Custom_1_1SOSCascade}{Iir\+::\+Custom\+::\+S\+O\+S\+Cascade}}$<$ N\+S\+OS, State\+Type $>$\+::setup (\begin{DoxyParamCaption}\item[{const double(\&)}]{sos\+Coefficients\mbox{[}\+N\+S\+O\+S\mbox{]}\mbox{[}6\mbox{]} }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}}

Python scipy.\+signal-\/friendly setting of coefficients. Sets the coefficients of the whole chain of biquads / S\+OS. The argument is a 2D array where the 1st dimension holds an array of 2nd order biquad / S\+OS coefficients. The six S\+OS coefficients are ordered \char`\"{}\+Python\char`\"{} style with first the F\+IR coefficients (B) and then the I\+IR coefficients (A). The 2D const double array needs to have exactly the size \mbox{[}N\+S\+OS\mbox{]}\mbox{[}6\mbox{]}.
\begin{DoxyParams}{Parameters}
{\em sos\+Coefficients} & 2D array Python style sos\mbox{[}N\+S\+OS\mbox{]}\mbox{[}6\mbox{]}. Indexing\+: 0-\/2\+: F\+I\+R-\/, 3-\/5\+: I\+I\+R-\/coefficients. \\
\hline
\end{DoxyParams}

The documentation for this struct was generated from the following file\+:\begin{DoxyCompactItemize}
\item
iir/Custom.\+h\end{DoxyCompactItemize}


\documentclass{article}
\usepackage{graphicx}
\usepackage[utf8]{inputenc}
\usepackage{amsmath, amssymb, latexsym}
\usepackage{neuralnetwork}
\usepackage{multicol}
\usepackage{expl3}

\usepackage{pgfplots}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{tikz}
\usepackage{nicefrac}
\pgfplotsset{every axis legend/.append style={
at={(0,0)},
anchor=north east}}
\usetikzlibrary{shapes,positioning,intersections,quotes}
\usetikzlibrary{arrows.meta,
                bending,
                intersections,
                quotes,
                shapes.geometric}

\definecolor{darkgreen}{rgb}{0.0, 0.6, 0.0}
\definecolor{darkred}{rgb}{0.7, 0.0, 0.0}
\makeatletter
\def\BState{\State\hskip-\ALG@thistlm}
\makeatother
\title{Week 13}
\begin{document}
\pagenumbering{gobble}
\maketitle
\newpage
\pagenumbering{arabic}

\section*{Unsupervised learning}

\begin{itemize}
  \item Try to figure out the structure of the data.
  \item The clustering algorithm organizes data based on data characteristics.
  \item Market segmentation is categorizing clients into different market categories.
  \item Social network analysis.
  \item Computer clusters and data centers are organized for network structure and location.
  \item Understanding galaxy creation through astronomical data analysis.
\end{itemize}

\section*{K-means algorithm}

\begin{itemize}
  \item Would you like an algorithm to automatically arrange data into coherent clusters?
  \item By far the most used clustering algorithm is K-means.
\end{itemize}

~\\
Algorithm overview:

\begin{enumerate}
    \item Assign k locations at random as cluster centroids.
    \item Go through each example and assign each point to one of the k clusters based on which center it is closest to.

\includegraphics[width=\textwidth]{resources/kclusters_1}

    \item Move to the average of the similarly allocated data-points for each centroid.

\includegraphics[width=\textwidth]{resources/kclusters_2}

    \item Repeat 2) and 3) until convergence.
\end{enumerate}

\section*{K-means for non-separated clusters}

\begin{itemize}
  \item So far, we've looked at K-means, which has well-defined clusters.
  \item However, K-means is frequently used on datasets with poorly defined clusters.
  \item As an example, consider t-shirt sizes. How large do you make them if you want three sizes (S,M,L)?
  \item As a result, three clusters are formed, even if they are not actually there.
  \item This is an example of market segmentation; create items that are tailored to the demands of your subpopulations.
\end{itemize}

\includegraphics[width=0.5\textwidth]{resources/t_shirt}

\section*{K means optimization objective}

\begin{itemize}
  \item K-means, like the supervised learning functions we've examined, has an optimization goal.
  \item While K-means is running, we keep track of two sets of variables.
  \item $c^i$ is the index of clusters ${1,2, ..., K}$ to which $x^i$ is currently assigned.
  \item $\mu_k$, is the cluster associated with centroid $k$.
  \item  $\mu_c^i$, is the cluster centroid of the cluster to which example $x^i$ has been assigned to.
  \item We may write the optimization objective using this notation:
\end{itemize}

$$J(c^{(1)}, ..., c^{(m)}, \mu_1, ...,\mu_K)=\frac{1}{m}\sum_{i=1}^{m}||x^{(i)}-\mu_{c^{(i)}}||^2$$

~\\
i.e. squared distances between training example $x^i$ and the cluster centroid to which $x^i$ has been assigned to.

\includegraphics[width=0.5\textwidth]{resources/cost_cluster}

~\\
When we look at the k-means method:

\begin{itemize}
  \item The cluster assigned step is minimizing $J(...)$ with respect to $c_1, c_2 ... c_i$ i.e. find the centroid closest to each example. Doesn't change the centroids themselves.
  \item The move centroid step. We can show this step is choosing the values of $\mu$ which minimizes $J(...)$ with respect to $\mu$.
  \item So, we're partitioning the algorithm into two parts: First part minimizes the $c$ variables. Second part minimizes the $J$ variables.
\end{itemize}

\section*{Random initialization}
Depending on the starting setting, K means might converge to different solutions.

\includegraphics[width=\textwidth]{resources/optimum_cluster}

\begin{itemize}
  \item Randomly initialize K-means.
  \item For each n (e.g. 100) random initialization run K-means.
  \item Then compute the distortion on the set of cluster assignments and centroids at convergent.
  \item End with n ways of cluster the data.
  \item Pick the clustering which gave the lowest distortion.
\end{itemize}

\section*{Elbow method}

\begin{itemize}
  \item How do we choose the number of clusters K?
  \item Vary K and compute cost function at a range of K values.
  \item $J(...)$'s minimum value should decrease as K rises (i.e. you decrease the granularity so centroids can better optimize).
  \item Look for the "elbow" on the graph ($K$ vs $J()$).
\end{itemize}

\includegraphics[width=0.47\textwidth]{resources/elbow}

\end{document}


% arara: makechapters: {items: [syllabus, dtt, pi, inductive, identity, equivalences, contractible, fundamental, hierarchy, funext, pullback, univalence]}

\documentclass[11pt]{memoir} %[ebook,10pt,oneside]

\usepackage{hott}

\title{Introduction to homotopy type theory}
\author{Egbert Rijke}
\date{Carnegie Mellon University\\Pittsburgh PA\\Summer 2019}%\\Version: \today}
%\address{Carnegie Mellon University}
%\email{erijke@andrew.cmu.edu}

\pretitle{\begin{center}\textsc\bgroup\LARGE}
\posttitle{\egroup\end{center}\vspace{2cm}}
\preauthor{\begin{center}\textsc\bgroup\Large}\postauthor{\egroup\end{center}\vfill}
\predate{\begin{center}\textsc\bgroup}{\postdate{\egroup\end{center}}

  % The following is to avoid overfull hboxes in the table of contents.
% https://tex.stackexchange.com/questions/49887/overfull-hbox-warning-for-toc-entries-when-using-memoir-documentclass
%\renewcommand*{\cftdotsep}{1}
\setpnumwidth{2em}
\setrmarg{3em}
\setlength{\cftchapternumwidth}{2em}
\setlength{\cftsectionindent}{2em}
\setlength{\cftsectionnumwidth}{2em}
\setlength{\cftsubsectionindent}{4em}
\setlength{\cftsubsectionnumwidth}{3em}

% We number sections independently of chapters, and subsections will be
% numbered too.

\counterwithout{section}{chapter}
\settocdepth{subsection}

% We set up the exercise environment, which produces list environment in a new
% unnumbered subsection that also gets mentioned in the table of contents.

\newlist{exenum}{enumerate}{1}
\setlist[exenum]{noitemsep,label=\thesection.\arabic*}
  %,ref=\thechapter.\arabic*}

\crefname{exenumi}{Exercise}{Exercises}

\newlist{subexenum}{enumerate}{1}
\setlist[subexenum]{noitemsep,label=(\alph*),ref=\theexenumi.\alph*}
\crefname{subexenumi}{Exercise}{Exercises}

\newenvironment{exercises}
{%
\subsection*{Exercises}%
\addcontentsline{toc}{subsection}{Exercises}%
\sectionmark{Exercises}%
\begin{exenum}}
{%
\end{exenum}}

\addbibresource{bibliography.bib}

\makeindex

\begin{document}

\begin{titlingpage}
  \maketitle
\end{titlingpage}

\mbox{}
\vfill
\doclicenseThis\thispagestyle{empty}

\cleardoublepage

\frontmatter

\tableofcontents

%\include{intro}

\chapter{Introduction}

To include introduction:
\begin{enumerate}
\item What are types in mathematics. Dependent types and dependent functions are everywhere in mathematics.
\item Why univalent foundations. Why should homotopy be in the foundation of mathematics
\item Constructive nature of homotopy type theory. Discuss differences with set theory.
\item What this course is about
\item How to use this book
\item Formal type theory versus informal type theory
\item Mention the formalization
\end{enumerate}

\begin{rmk}
  One difference between set theory and type theory is that every well-formed term is specified along with its type and with its context. One way of looking at this is that there are three sorts in type theory: contexts, types, and terms. On the other hand, there is only one sort in set theory: sets. Sets are governed by the elementhood relation: the formula $x\in y$ is a well-formed formula of set theory for any two sets $x$ and $y$. In particular, for a given set $x$, the formula $x\in y$ can be true for many sets $y$, which is very different to the situation in type theory, where every term is assigned a unique type.

  Another important difference between set theory and type theory is that set theory is formulated in the language of first order logic, whereas type theory is its own deductive system, not making use of any ambient logic. We will see in the present chapter and in the next few chapters what this deductive system looks like.
\end{rmk}

\mainmatter

\renewcommand{\thechapter}{\Roman{chapter}}
\setsecnumdepth{subsection}

\chapter{Martin-L\"of's dependent type theory}
\input{dtt}
\input{pi}
\input{nat}
\input{inductive}
\input{identity}
\input{universes-relations}
\input{number-theory}
\input{finite-types}

\chapter{Basic concepts of type theory}
\input{equivalences}
\input{contractible}
\input{fundamental}
\input{hierarchy}

\chapter{Univalent mathematics}
\input{funext}
\input{univalence}
\input{circle}

\chapter{Homotopy pullbacks and pushouts}
\input{pullback}
\input{pushout}
\input{cubical}
\input{descent}
\input{id-pushout}

\chapter{The homotopy image of a map}
\input{sequences}
\input{image}
\input{replacement}
\input{sets}
\section{The classifying type of a group}
\input{truncation}
\section{Truncations}

\chapter{Synthetic homotopy theory}
\input{homotopy_groups}
\input{les}
\input{connected}
\input{smash}
\input{blakers-massey}

%\include{open_problems}

\backmatter

\printbibliography

\printindex

\end{document}


\chapter{R Code for Data Analysis}
\label{apdx:appendixa}
\lstinputlisting[language=R]{data_analysis.R}


\subsection{Depth-limited search}

Depth-first search with a limit. This is useful if we know the solution is shallower than limit \(l\).

Informed: No

Time:

Space:

Complete:

Optimal:


    \filetitle{VAR}{Population VAR for selected model variables}{model/VAR}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
V = VAR(M,List,Range,...)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\item
  \texttt{M} {[} model {]} - Solved model object.
\item
  \texttt{List} {[} cellstr \textbar{} char {]} - List of variables
  selected for the VAR.
\item
  \texttt{Range} {[} numeric {]} - Hypothetical range, including
  pre-sample initial condition, on which the VAR would be estimated.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{V} {[} VAR {]} - Asymptotic reduced-form VAR for selected
  model variables.
\end{itemize}

\paragraph{Options}\label{options}

\begin{itemize}
\item
  \texttt{'order='} {[} numeric \textbar{} \emph{1} {]} - Order of the
  VAR.
\item
  \texttt{'constant='} {[} \emph{\texttt{true}} \textbar{}
  \texttt{false} {]} - Include in the VAR a constant vector derived from
  the steady state of the selected variables.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\subsubsection{Total number of messages sent}\label{subsubsec:ld2krmessages}

This is an indication of the energy efficiency of the entire network.

The maximum number of copies is the dominant factor (\(85.18\%\)), as in the
high density case. In the previous case the size of the hear window was the
second most important factor. Here, instead, we have the broadcast radius
(\(6.57\%\)) and its combination with the maximum number of copies. The
unexplained variation is extremely low \(0.31\%\).

From \figref{fig:ldperfmessagesR} we notice that increasing the broadcast radius
is useful to reduce the total number of messages sent when the maximum number of
copies is low. This can be explained by the fact that with an higher value for
\(m\) means that nearly all the users in the network will relay the message
after they have successfully heard it. So, if \(m\) is high, there is not much
to do to reduce the total number of messages sent.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.6\textwidth]{img/ld/messages-R-perfplot}
	\caption{Performance plot for the total number of messages sent. The
	broadcast radius can be used to reduce the total number of messages sent
	when the maximum number of copies is not
	high}\label{fig:ldperfmessagesR}
\end{figure}


\documentclass{dw2014}

\usepackage{mathptmx}
\usepackage{helvet}
\usepackage{courier}
\usepackage{graphicx}

\title{Identification of human control during perturbed walking}

\author{Jason K. Moore, Sandra K. Hnat, Antonie J. van den Bogert\\
  Human Motion and Control Laboratory, Cleveland State University, Cleveland, Ohio, USA\\
  {\it j.k.moore19@csuohio.edu, s.hnat@vikes.csuohio.edu, a.vandenbogert@csuohio.edu}}

\date{}

\begin{document}

\maketitle

\section*{Introduction}
%
Recent research and commercial activity have shown that gait-related powered
prosthetics will play an important role in assisting humans with disabilities.
These devices include a variety of sensors and actuators than can be coupled to
a control system to provide acutated gait assistance. However, the available
lightweight, lower extremity exoskeletons lack gait that resembles an
able-bodied human. To improve the gait of powered prosthetics, our intent is to
identify a simple, linear controller from a large set of data collected from
able-bodied subjects being perturbed by random longitudinal forces.
%
\section*{Methods}
The example data shown here was collected from a single subject (age: 29, mass:
63 kg, height: 172 cm) walking on an instrumented treadmill (V-Gait, Motek
Medical). The subject was longitudinally perturbed using random white noise
with 10\% std around a nominal 1.2 m/s belt speed. Data was recorded for 8
minutes at 100 Hz, which included approximately 500 steps of walking. Ankle
plantarflexion, knee flexion, and hip flexion angles, rates, and moments were
computed using 2D inverse dynamics.
%
\begin{figure}[hbt]
  \begin{center}
    \includegraphics[width=\columnwidth]{fig/gains.pdf}
    \caption{Gait phase percent scheduled gains for right (blue) and left (red) legs.}
    \label{fig:gains}
  \end{center}
\end{figure}

Joint angle, rate, and torque time series were sectioned into steps based on
the right foot's heel strike. Subsequently, 20 evenly spaced data points from
each series were interpolated along the gait cycle. We assume a simple
scheduled proportional derivative controller that generates the joint torques
given the joint angles and rates that fits the following form.
%
\begin{equation}
  \mathbf{m}(t) = \mathbf{m}^*(\varphi(t)) -
  \mathbf{K}(\varphi(t))\mathbf{s}(t)
\end{equation}
%
where $t$ is an instance of time, $\varphi(t)$ is the phase in the right leg
gait cycle, $\mathbf{m}(t)$ is a vector of joint torques,
$\mathbf{m}^*(\varphi(t))$ is a vector of the nominal joint torques,
$\mathbf{K}(\varphi(t))$ is a gain matrix scheduled with respect to gait
phase which multiplies the vector of joint angles and rates, $\mathbf{s}(t)$.
This equation is linear in the gains and the reference torques. Given
sufficient joint angle, rate, and torque measurements, the reference torques
and the gains can be solved for using linear least squares.
%
\section*{Results}
%
Here we present an example result from a controller structure which is limited
to joint torque generation only from error in the sensors from the same joint.
Figure \ref{fig:gains} shows the estimates of the scheduled gains with respect
to the percent gait cycle in each leg. Figure \ref{fig:fit} demonstrates an
example prediction of the measured ankle plantarflexion torque in the right leg
by the identified control model.
%
\begin{figure}[b]
  \begin{center}
    \includegraphics[width=\columnwidth]{fig/fit.pdf}
    \caption{Predicted torque compared to independent validation data.}
    \label{fig:fit}
  \end{center}
\end{figure}
%
\section*{Discussion}
%
We are able to identify a simple linear controller that exhibits larger gains
in the stance phase than in the swing phase. Additionally, similar gain
patterns in the right and left legs are observed that use both positive and
negative feedback. The controller is capable of predicting the measured joint
torques with greater than 65\% VAF in all joints. Results and conclusions from
a larger sample of subjects and conditions will be presented at the conference.

%\section*{Acknowledgments}
%
%This research was funded by the Ohio's Wright Center for Sensor Systems
%Engineering and the Parker Hannifin Corporation.
\end{document}


%!TEX root = ../thesis.tex
%*******************************************************************************
%****************************** Second Chapter *********************************
%*******************************************************************************

\chapter{The Rotating Sun}

\section{Differential Rotation}
The sun is known to rotate differentially, unlike the earth. This means that angular rate of rotation of a point in the sun about its spin axis is depends on depth and latitude, $\Omega = \Omega(r,\theta)$. Splitting of p-mode\footnote{pressure waves are also called p-modes} frequencies due to differential rotation is well understood and has a long history of inversion analysis \cite{ritzwoller,lavely92,schou98}.
This $\Omega$ is generally taken to be symmetric about the equitorial plane \cite{ritzwoller}.

As a result of Alfven's freezing theorem\footnote{Alfven's freezing theorem states that in a perfectly conducting plasma, magnetic flux through \textit{every} surface is conserved as it gets dragged along with the ambient flow. See \cite{goedbloed2004} for proof. This theorem implies that magnetic field lines are $frozen$ in the plasma as it flows.}, differential rotation is responsible for winding the solar magnetic field around its spin axis in an axisymmetric fashion. Thus we'll be investigating the effects of an axis symmetric magnetic field on the spectrum in this work, although the method we develop can very well accommodate non-axisymmetric fields too.
\section{Detection from frequency spectrum}
Because of axis symmetry of differential rotation, the flow profile is given as,
\begin{equation}
\vrot = \sum_{s = 1,3,5,...}^{\infty} -w_s^0(r) \partial_{\theta} Y_s^0 \ev{\phi}
\end{equation}

Note that $w_1^0$ is responsible for shell like (pure) rotation as it couples with $\partial_{\theta} Y_1^0 \sim \sin\theta$. Hence $w_3^0$ onwards components of flow are responsible for the differential part of the rotation. The fact that the azimuthal index for the $\vrot$ profile is $0$ means that it has no azimuthal dependence. Finding frequency splittings due to differential rotation is a problem in DPT or QDPT depending on whether we're using the isolated multiplet approximation or not. The isolated multiplet approximation is the assumption that there exists negligible cross coupling between different $\mode{n}{l}$ modes.
Below we outline the QDPT approach to the problem because when applied to a single multiplet $_n S_l$ it reduces to the DPT approach. It has been argued via analysis differential rotation coupling of \mode{n}{1} and \mode{n}{3} multiplets that eigenfrequency correction to DPT via QDPT from cross coupling between these two is negligible ($\sim 1 \mu Hz$) \cite{lavely92}. We'll show in Chapter 5 (figure \ref{fig:split_dr}) that for $l\sim 100$ modes, a frequency corrections of upto $600 nHz$ frequencies are obtained when QDPT is used.

%which is close to the correction obtained due presence of realistically strong magnetic fields too, and hence cannot be ignored in an analysis which accounts for both differential rotation and magnetic fields.

\subsection{QDPT Analysis}
The pertubation operator \dLd for a differential rotaional flow is given by
\begin{equation}
\dLd = -2i \omega \rho \vrot \cdot \grad
\end{equation}
where $\omega$ is the reference frequency in the problem, and $\rho$ is the static background density profile \cite{ritzwoller}.
The supermatrix element $Z_{k' k}$ for a pertubation \dLd is given by
\begin{equation}
Z_{k' k} = \Lamdr_{k'k} - \delta_{k'k} (\omref^2 - \omega_k^2)
\end{equation}
where $\Lamdr$ is the coupling matrix element $\Lamdr = \inner{\xiv_{k'}}{\dLd \xiv_{k}}$ .
Coupling matrix element is given by
\begin{equation}
\Lamdr_{k'k} = 8 \pi \omref \gam{l'}\gam{l} \oddsum{s} \gam{s} \wigred{-m}{0}{m} \ints dr
r^2 w_s^0(r) T_s(r)
\end{equation}
where the sensitivity kernel $T_s$ is given by
\begin{dmath}
T_s(r) = (1-(-1)^{s+l+l'}) \om{l'}{0} \om{l}{0} \wigred{-1}{0}{1} r^{-1}
 \enc{U'V+V'U-U'U-\frac{1}{2}\enc{l'(l'+1) + l(l+1) - s(s+1) V'V}}
\end{dmath}
where the rounded brackets represent Wigner 3j symbols \cite{ritzwoller}, and $\gam{l}$ and $\Omega_{l}^{N}$ are constants defined in Appendix \ref{app_conventions}.
\subsection{Selecition rules in mode coupling}\label{sec:selec_rules}
This matrix element enforces the following selection rules for inter-mode interaction which derive from the properties of Wigner 3j symbols \cite{lavely92}.
\begin{enumerate}
\item $m'=m$
\item $l'+l+s = \text{odd}$
\item $|l'-l| \leq s \leq l'+l$
\end{enumerate}
It should be noted here that even though only sum over odd s is considered, the expression for $T_s$ is general and holds for all $s$. This has been verified independently using the Mathematica packaged developed for the sake of this work \cite{GSH_repo}. Hence, as far as self coupling is concerned ($l'=l$), $T_s$ vanishes for even $s$. This means that even if $\Omega(\theta,\phi)$ had a component which is antisymmetric about the equatorial plane, DPT analysis would not reveal any signature of that in the frequency splittings.

\subsubsection{Form of supermatrix $Z_{k'k}$}

It is clear from selection rule (1) in \ref{sec:selec_rules} that $Z_{k'k}$ is going to be a sparse matrix consisting of a diagonal and a number of sub-diagonals (proportional to number of multiplets being considered). Also, as $s$ is always odd, $l-l'$ has to be even for non-zero coupling as consequence of selection rule (2). This influences our choice of modes whose inter-coupling will be studied hereafter. Figure (\ref{fig:coup_mat}) is a visual representation of a typical supermatrix consisting of three multiplets. The logarithm scale demonstrates the order of magnitude difference between self-coupling (main diagonal elements), and cross-coupling (subdiagonal elements). Also notice that largest elements (yellow) are in the first and third section in the main diagonal. These are large because these frequencies are placed away from $\omref$ which is taken to be mean of the three mode frequencies. The relative weakness of the sub-diagonal terms compared to the main-diagonals is because $l-l'\geq 2$ forces $s\geq 2$ which makes the largest component of differential rotation, i.e. $w_1^0$, unable to couple these modes; $w_3^0$ and $w_5^0$ are one and two orders of magnitude smaller than $w_1^0$ respectively.

\begin{figure}[h]
\includegraphics[scale=0.6,center]{Chapter2/figs/coup_mat}
\caption{Visualisation of $\log_{10}|Z_{k'k}|$ in $\mu Hz^2$ for the multiplets $\mode{0}{11}$, $\mode{0}{13}$, and $\mode{0}{15}$ with frequencies $603.69 \mu Hz$, $641.84\mu Hz$, and $677.55 \mu Hz$ respectively. Numbers on the X and Y axes represent cumulative $m$ of all three multiplets. White spaces correspond to $0$ value.}
\label{fig:coup_mat}
\end{figure}


\documentclass[a4paper,12pt]{article}

% Font
\usepackage[T1]{fontenc}
\usepackage{gentium}

% Math packages
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{bm}

% Define symbol shortcuts
\newcommand{\cc}{\mathcal{C}}
\newcommand{\dd}{\mathcal{D}}
\newcommand{\hh}{\mathcal{H}}
\newcommand{\xx}{{\bm x}}
\newcommand{\yy}{{\bm y}}

% Math environment
\newtheorem*{thm}{Theorem}

% Better list management:
% - vertical spacing in lists
% - items in lists start with dash not bullet point.
\usepackage{enumitem}
\setlist{label=\textemdash,
  itemsep=0pt, topsep=3pt, partopsep=0pt}

% Include graphics
\usepackage{graphicx}
\usepackage{subcaption}

% Page format
\usepackage[top=2cm,left=2cm,right=2cm,bottom=2cm]{geometry}
\usepackage{float}

\begin{document}
%%% HEADER
\raisebox{0.6in}[0in]{\makebox[\textwidth][r]{\it Unproofed version }}
\vspace{-0.7in}

\begin{center}
\bf\large MA2823 : Foundations of Machine Learning \\
Chapter 5 : Linear Regression
\end{center}

\noindent
Lecturer : Chlo\'e-Agathe Azencott
\hfill
Scribes : Emilie Leblanc, Antoine Salon, Hippolyte Jacomet

\noindent
\rule{\textwidth}{1pt}

\medskip

%%% NOTES START HERE
In this chapter we will see how to :
\begin{itemize}
\item define parametric methods ;
\item define the maximum likelihood estimator and compute it
for Bernoulli, multinomial and Gaussian densities ;
\item define the Bayes estimator and compute it for normal
priors ;
\item compute the maximum likelihood estimator / least-square
fit solution for linear regression ;
\item compute the maximum likelihood estimator for logistic
regression.
\end{itemize}

\section{Parametric methods}

In this chapter we would like to define different ways of building estimators that are part of a statistical model, and how to assess their performances.

\paragraph{Definitions and notations} Let us consider a training set of $n$ $p$-dimensional vectors denoted by $\mathcal{X}=\{x^i\}_{i=1,\dots,n}$, and let's assume that each of these observation follows a law of probability given a parameter $\theta$.

\[
\forall i \in [1,\dots,n],\ \xx^{i}\sim{}p(\xx|\theta)
\]\\
The {\em parametric estimation} consists in {\em assuming} a form for $p(\xx|\theta)$ and then {\em estimating} the parameter $\theta$ using our training set $\mathcal{X}$. \\
For example, if we assume that the observations of our training set follow a Gaussian distribution $p(x_{j}|\theta_j)\sim{}\mathcal{N}(\mu_j,{\sigma_j}^2)$ our goal would be to estimate the parameter $\theta=\{\mu_1,\sigma_1,\dots,\mu_p,\sigma_p\}$. \\
\\
To achieve that goal, the observations of the training set $\xx^i$ are usually assumed to be {\em independent and identically distributed} (i.i.d.). \\

In the following sections we sill study two different methods to estimate the parameters : the {\em maximum likelihood estimation} and the {\em Bayes estimation}.

\subsection{Maximum likelihood estimation}

This first method relies on the idea that $\theta$ should maximize the likelihood for $\mathcal{X}$ to be drawn from the law of probability we chose in our model.

\paragraph{Likelihood and log-likelihood} We call {\em likelihood of} $\theta$ given $\mathcal{X}$ the function : $\ell(\cdot{}|\mathcal{X}) : \theta \mapsto p(\mathcal{X}|\theta).$\\
In the case of an i.i.d. sample $\mathcal{X}$, this becomes :

\[\ell(\theta|\mathcal{X})=p(\mathcal{X}|\theta)=p(\xx^1|\theta)p(\xx^2|\theta)\dots{}p(\xx^p|\theta)\] \\
It is often useful to also consider the {\em log-likelihood}, simply defined by $\mathcal{L} (\cdot{}|\mathcal{X}) : \theta \mapsto \log \ell(\theta|\mathcal{X})$, which in our previously mentioned case becomes :
\[\mathcal{L}(\theta|\mathcal{X})=\log{p(\xx^1|\theta)}+\dots{}+\log{p(\xx^n|\theta)}\]
\\
Given these two functions, we will then try to find the estimator that maximizes them, called the {\em maximum likelihood estimator} or MLE, and defined by :

\[\hat{\theta}=\arg\underset{\theta}\max \  \ell(\theta|\mathcal{X})=\arg\underset{\theta}\max \  \mathcal{L}(\theta|\mathcal{X})\]\\

Let us now look at these MLE for Bernoulli, Multinomial and Normal laws.

\subsubsection{Bernoulli density}

Such a law corresponds to observations that reflect two states of either failure or success (think of a coin toss for example), and thus let us consider a data set $\mathcal{X}=\{x^i\}_{i=1,\dots,n}$ with  $x^i\in\{0,1\}$ $\forall i\in[1,\dots,n]$.\\
\\
The Bernoulli density is given by $P(X=x|p_0)={p_0}^{x}(1-p_0)^{(1-x)}$, let us try to find the MLE $\ \hat{p_0}$ of the parameter $p_0$.\\
\\
We first compute the log-likelihood :

\[\mathcal{L}(p_0|\mathcal{X})=\log{P(\mathcal{X}|p_0)}=\sum_{i=1}^n\left(x^i\log p_0 + (1-x^i)\log{(1-p_0)}\right)\]
This is a concave function which we can easily maximize by setting its gradient to 0 :

\[\frac{\sum_{i=1}^nx^i}{\hat{p_0}}-\frac{n}{(1-\hat{p_0})}+\frac{\sum_{i=1}^nx^i}{(1-\hat{p_0})}= 0
\]
Which yields the MLE of $p_0$ :

\[\hat{p_0}=\frac{1}{n}\sum_{i=1}^nx^i\]

\subsubsection{Multinomial density}

Let us consider $K$ mutually exclusive and exhaustive classes. The probability for each class to occur is $p_k$, with $\sum_{k=1}^Kp_k=1$. We represent these classes with $K$ indicator variables $x_1, x_2, \dots, x_K$ knowing
\[
    x_k=
\begin{cases}
    1& \text{if the outcome is class } k\\
    0              & \text{otherwise}
\end{cases}
\]
The joint probability distribution is given by :
\[P(x_1,x_2,\dots,x_K)=\displaystyle\prod_{k=1}^{K}p_k^{x_k}\]\\
\\
Let us compute the MLE $\bm{\hat{p}}=(\hat{p_1},\dots,\hat{p_K})$. The observations are i.i.d. thus the log-likelihood looks dramatically like :
\\
\[\mathcal{L}(\bm p|\mathcal{X})=\sum_{i=1}^n\sum_{k=1}^K x_k^i\log p_k
\]\\
And we now have to maximize it under the constraint $\displaystyle\sum_{k=1}^Kp_k=1$.\\
We must call on to a Lagrange multiplier $\lambda$ and define the Lagrangian function
\[\mathcal{F} : (\bm p, \lambda) \mapsto \mathcal{L}(\bm p|\mathcal{X})-\lambda \left(\sum_{k=1}^Kp_k - 1\right)\]
Setting its gradient to zero gently yields a system of equations from which we are happy to learn that $\lambda=n$ and consequently that

\[\hat{p}_k=\dfrac{1}{n}\displaystyle\sum_{i=1}^nx_k^i\ \ \ \ \ \ \ \ \forall k \in [1,\dots,K]\]

\subsubsection{Gaussian distribution}

Let us now assume that our observations $x^i$ are i.i.d. following a Gaussian distribution, i.e. :

$\forall i \in [1,\dots,n]$,
\[x^i\sim\mathcal{N}(\mu,\sigma^2)\]

\[p(x^i|\mu,\theta)=\dfrac{1}{\sqrt[]{2\pi}\sigma}\exp\left[-\frac{(x^i-\mu)^2}{2\sigma^2}\right]\]
\\

Setting the gradient of the log-likelihood to zero yields the estimators :

\[ \hat{\mu} = \frac{1}{n} \sum_{i=1}^nx^i\]
\[ \hat{\sigma}^2 = \frac{1}{n} \sum_{i=1}^n (x^i-\hat{\mu})^2\]

\subsection{Bias-variance trade-off}

We can assess the performance of the estimator (be it obtain through MLE or another method) by computing the {\em mean squared error} or MLE, defined by :

\[\mbox{MSE}(\hat{\theta}) = \mathbb{E}[(\hat{\theta}-\theta_0)^2]\]

\paragraph{Bias-variance trade-off} Let us try to link this MSE to two previously known notions that help characterize our estimator, the variance and the bias. As a reminder, the {\em bias} of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated : $\mbox{Bias}(\hat{\theta})=\mathbb{E}[\hat{\theta}]-\theta_0$.

\begin{align*}
\mbox{MSE}(\hat{\theta}) &= \mathbb{E}[(\hat{\theta}-\theta_0)^2]\\
&= \mathbb{E}\left[(\hat{\theta}-\mathbb{E}[\hat{\theta}])^2+2\hat{\theta}\mathbb{E}[\hat{\theta}]-\mathbb{E}[\hat{\theta}]^2-2\hat{\theta}\theta_0+\theta_0^2\right]\\
&=\mbox{Var}(\hat{\theta})+\mathbb{E}[\hat{\theta}]^2-2\mathbb{E}[\hat{\theta}]\theta_0+\theta_0^2\\
&=\mbox{Var}(\hat{\theta})+\mbox{Bias}^2(\hat{\theta})
\end{align*}\\
We see that the MSE is a balance between these two positive expressions, hence the idea of a {\em bias-variance trade-off}. We understand that a biased estimator may achieve better MSE than an
unbiased one, as illustrated below :

\begin{center}
\includegraphics[width=100mm]{biasvariance.png}
\end{center}

\subsection{Bayes estimator}
From now on let us study another way of building an estimator, using Bayes rule which in its general form is :
\[P(C|\xx)  = \dfrac{P(C)p(\xx|C)}{p(\xx)}\]\\
Here we choose to treat $\theta$ as a random variable with a given prior distribution $p(\theta)$. Considering an observation-set $\mathcal{X}$, the Bayes rule yields :

\[p(\theta|\mathcal{X})=\frac{p(x|\theta)p(\theta)}{p(\mathcal{X})}\]\\
\\
We define the {\em Bayes estimate} as the conditional expected value of $\theta$ given our data set $\mathcal{X}$ :
\[\hat{\theta}_{\mbox{Bayes}}=\mathbb{E}[\theta|\mathcal{X}]=\int{\!\theta p(\theta|\mathcal{X})d\theta}\]\\
\\
Here's a reminder of the estimates we have previously seen :\\

{\em Maximum a posteriori} estimate :
\[\hat{\theta}_{\mbox{MAP}}=\arg\underset{\theta}\max \  p(\theta|\mathcal{X})\]

{\em Maximum likelihood} estimate :
\[\hat{\theta}_{\mbox{MLE}}=\arg\underset{\theta}\max \  p(\mathcal{X}|\theta)\]\\

\subsubsection{Bayes estimator for normal priors}

Let us consider $n$ i.i.d. data points $x^i$ following a normal law, $x^i\sim\mathcal{N}(\theta,\sigma^2)$ and let's assume that $\theta$ is of Gaussian prior distribution : $\theta\sim\mathcal{N}(\mu,\sigma_0^2)$.\\
\\
We know that the MLE of $\theta$ is its sample mean : $\hat{\theta}_{\mbox{MLE}} = \frac{1}{n} \sum_{i=1}^nx^i$ and we would like to compare it to its Bayes estimator.\\
\\
After a long and troublesome computation, a quick look at $p(\theta|\mathcal{X})$ shows us that it follows a normal distribution with mean $m$ and variance $s^2$ :

\[m = \frac{n\hat{\theta}_{\mbox{MLE}}\sigma^2+\mu\sigma_0^2}{n\sigma^2+\sigma_0^2}
\]

\[s^2 = \frac{\sigma^2\sigma_0^2}{n\sigma^2+\sigma_0^2}
\]\\
\\
Knowing that :
\[\mathbb{E}[\theta|\mathcal{X}]=m\]
We can conclude with :
\[\hat{\theta}_{\mbox{Bayes}}=\frac{\frac{n}{\sigma_0^2}}{\frac{n}{\sigma_0^2}+\frac{1}{\sigma^2}}\hat{\theta}_{\mbox{MLE}}+\frac{\frac{1}{\sigma^2}}{\frac{n}{\sigma_0^2}+\frac{1}{\sigma^2}}\mu
\]\\
\\
It is interesting to notice that when n increases, $\hat{\theta}_{\mbox{Bayes}}$ gets closer to the sample average (uses
information from the sample) and When $\sigma$ is small, $\hat{\theta}_{\mbox{Bayes}}$ gets closer to $\mu$ (little uncertainty
about the prior).

\section{Linear regression}

The goal of a {\em linear regression} is to approximate the law followed by our data \(\textit{y}\) by a linear combination of observed variables \(\xx\) such as :
\[ f(\xx|\beta)= \displaystyle\sum_{j=1}^{p} \beta_j*x_j + \beta_0\]
with \(\xx^i \in \mathbb{R}^p\)  and \(\textit{y}^i \in \mathbb{R}\) and noting \(\dd = {\{\xx^i,\textit{y}^i\}}_{\textit{i}=1,..,\textit{n}}\).
\begin{center}
\includegraphics[width=100mm]{linear_regression_example.png}
\end{center}

We assume that the error \(\epsilon\) we have while doing a regression is \textbf{Gaussian distributed}. Therefore, with \(y = g(\xx) + \epsilon \),  \(f(\xx|\beta)\) being \(g\)'s estimator, we have :
\begin{center}
\(\epsilon = y - g(\xx)\) and \(\epsilon\sim{}\mathcal{N}(0,{\sigma}^2)\)
\end{center}

We therefore have our \(\dd = {\{\xx^i,\textit{y}^i\}}_{\textit{i}=1,..,\textit{n}}\) observations (the blue dots), the linear regression obtained \(\mathbb{E}(y|x) = \beta*x + \beta_0\) and, for each new \(x^*\) point, we can compute the hypothetical \(y^* = \mathbb{E}(y|x^*)\) value and the corresponding error \(\epsilon^* \) and confidence interval given by \(p(y|x^*)\) since :
\[p(y|\xx)\sim{}\mathcal{N}(f(\xx|\beta),{\sigma}^2)\]
\begin{center}
\includegraphics[width=100mm]{LR_2.png}
\end{center}

\subsection{Maximum likelihood estimation under Gaussian noise}

We are now going to use the \textit{maximum likelihood estimation} in order to find the best linear regression possible to fit the data and predict new points. \\
\\
Under Gaussian noise, considering i.i.d. observations and with \(p(y|\xx)\sim{}\mathcal{N}(f(\xx|\beta),{\sigma}^2)\), the log-likelihood defined earlier becomes :
\begin{align*}
\mathcal{L}(\beta|\dd) &= \log\displaystyle\prod_{i=1}^{n} p(y^i|\xx^i) + \log\displaystyle\prod_{i=1}^{n} p(\xx^i) \\  &= \log \left(\displaystyle\prod_{i=1}^{n} \frac{1}{\sqrt{2\pi}\sigma}*exp\left[ - \frac{(y^i - f(\xx^i|\beta))^2}{2\sigma^2} \right] \right ) + Cte  \\
& = Cte - \frac{1}{2\sigma^2} \displaystyle\sum_{i=1}^{n} (y^i - f(\xx^i|\beta))^2
\end{align*}
Since \(\log\displaystyle\prod_{i=1}^{n} p(\xx^i)\) is independent of \(\beta\), its value is added to a term \(Cte\) along with others. \\
Assuming Gaussian error, maximizing the likelihood \(\mathcal{L}(\beta|\dd)\) is therefore equivalent to minimizing the sum of squared residuals \(\displaystyle\sum_{i=1}^{n} (y^i - f(\xx^i|\beta))^2\).
This expression of the residual sum of squares can also be written in matrix form :
\begin{align*}
RSS(\beta) &= \displaystyle\sum_{i=1}^{n} (y^i - f(\xx^i))^2 \\  &= \displaystyle\sum_{i=1}^{n} \left( y^i - \beta_0 - \displaystyle\sum_{j=1}^{p} x_j^i \beta_j \right)^2 \\
& = (y - \textbf{\textit{X}} \beta)^T * (y - \textbf{\textit{X}} \beta)
\end{align*}
with \(\textbf{\textit{X}} = \left( \begin{array}{ccccc}
1 & x_1^1 & x_2^1 & ... & x_p^1 \\
1 & x_1^2 & x_2^2 & ... & x_p^2 \\
. & . & . & ... & .\\
. & . & . & ... & .\\
. & . & . & ... & .\\
1 & x_1^n & x_2^n & ... & x_p^n \end{array} \right) \).
Here, we added a vector \(\left( \begin{array}{c}
1 \\
1 \\
.\\
. \\
. \\
1 \end{array} \right)\) to \({\{\xx^i\}}_{\textit{i}=1,..,\textit{n}}\) in  \(\textbf{\textit{X}}\)'s expression in order to compute the scalar \(\beta_0\). \\
\\
Historically, the use of the residual sum of squares minimization can be attributed to Carl Friedrich Gauss (to predict the location of Ceres) and Adrien Marie Legendre.
\\
\\
\textbf{In this method, under which conditions is \(\beta\)'s estimator unique ? }
\\

\begin{itemize}
\item
In order to minimize \(RSS(\beta)\), \(\textbf{\textit{X}}\) must have a full column rank, hence \(\textbf{\textit{X}}^T\textbf{\textit{X}}\) \textit{invertible}. In this case, the \(\beta\) minimizing \(RSS(\beta)\) is :
\[\hat{\beta} = (\textbf{\textit{X}}^T\textbf{\textit{X}})^{-1} \textbf{\textit{X}}^T y\]
\item
If \(\textbf{\textit{X}}^T\textbf{\textit{X}}\) is not invertible (or \textit{rank-deficient}), we can use its pseudo-inverse. However, when doing so, we must keep in mind that the solution is \textit{not} unique. \\ A pseudo-inverse of\textbf{ A} is a matrix \textbf{G} such as \textbf{AGA = A}.
\end{itemize}

\subsection{Gauss-Markov Theorem}

\textbf{Gauss-Markov Theorem :} Under the assumption that \(\epsilon\sim{}\mathcal{N}(0,{\sigma}^2)\), the least-squares estimator of \(\beta\) is its best linear unbiased estimator. This Best Linear Unbiased Estimator is unique if \(\textbf{\textit{X}}^T\textbf{\textit{X}}\) is invertible. We call it the "\textit{BLUE}".
\\
\\
Indeed, with \(\hat{\beta} = (\textbf{\textit{X}}^T\textbf{\textit{X}})^{-1} \textbf{\textit{X}}^T y\), we can show that (demonstration of the Gauss-Markov Theorem) :
\begin{center}
\(\forall \beta^*\) unbiased estimator of \(\beta\), \(\mathbb{V}ar(\beta^*) \geq \mathbb{V}ar(\hat{\beta})\) and when \(\mathbb{V}ar(\beta^*) = \mathbb{V}ar(\hat{\beta}) => \beta^* = \hat{\beta} \)
\end{center}
\vspace{1cm}

\textbf{Proof of the Gauss-Markov theorem :}
\\ We have \(\hat{\beta} = (\textbf{\textit{X}}^T\textbf{\textit{X}})^{-1} \textbf{\textit{X}}^T y\) \textit{BLUE} estimator and let \(\beta^* = A y\) be another linear estimator of \(\beta\). As we're restricting to unbiased estimators, minimum mean squared error implies minimum variance. The goal is therefore to show that such an estimator has a variance no smaller than that of \(\hat{\beta}\).
We have \(A = (\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D\) where \(D\) is a non-zero matrix.
\\ In order to have \(\beta^*\) unbiased, we compute:
\begin{align*}
\mathbb{E}[\beta^*] &= \mathbb{E}[Ay] \\  &= \mathbb{E}[((\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D)(\textit{\textbf{X}}\beta + \epsilon)] \\
& = ((\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D)\textit{\textbf{X}}\beta + ((\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D)\mathbb{E}[\epsilon] \\ &= ((\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D)\textit{\textbf{X}}\beta \\ &=  (\textit{\textbf{I}} + D\textit{\textbf{X}})\beta
\end{align*}Since \(\mathbb{E}(\epsilon) = 0\). Therefore, in order to have an unbiased  \(\beta^*\) estimator, we must have \(DX = 0\). \\
Now, we calculate and compare the variances :
\begin{align*}
\mathbb{V}ar(\hat{\beta}) &= \mathbb{E}[((\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T \epsilon \epsilon^T \textit{\textbf{X}} (\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1}] \\  &= (\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T * \sigma \textit{\textbf{I}} * \textit{\textbf{X}} (\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \\
& = \sigma^2 (\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1}
\end{align*}
\begin{align*}
\mathbb{V}ar(\beta^*) &= \mathbb{V}ar(Ay)  \\  &= A\mathbb{V}ar(y)A^T \\
& = \sigma^2*AA^T \\ &= \sigma^2*((\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D)(\textit{\textbf{X}}^T \textit{\textbf{X}})^{-1} \textit{\textbf{X}}^T + D)^T \\ &= \sigma^2[(\textit{\textbf{X}}^T\textit{\textbf{X}})^{-1} + (\textit{\textbf{X}}^T\textit{\textbf{X}})^{-1}(D\textit{\textbf{X}})^T + D(\textit{\textbf{X}}(\textit{\textbf{X}}^T\textit{\textbf{X}})^{-1} + DD^T] \\ &= \sigma^2\textit{\textbf{X}}^T\textit{\textbf{X}})^{-1} + \sigma^2 DD^T \\ &= \mathbb{V}ar(\hat{\beta}) + \sigma^2 DD^T
\end{align*}since \(DX = 0\). \\
Therefore \(\mathbb{V}ar(\beta^*) = \sigma^2 D D^T +  \mathbb{V}ar(\hat{\beta})\) and since \(\sigma^2 D D^T \) is positive semi-defined and minimal for \(D = 0\),  \(\mathbb{V}ar(\beta^*)\) exceeds \(\mathbb{V}ar(\hat{\beta})\).

\subsection{Interpretation with correlated or uncorrelated variables}

The interpretation of the coefficients of a linear regression can be tricky. Indeed, once we have found \(\beta_0,...,\beta_p\) such as :
\[f(\textbf{\textit{X}}) = \beta_0 + \beta_1 x_1 + ... + \beta_p x_p\]
we first have to study the correlation of the variables.
\begin{itemize}
\item If the variables are \textit{decorrelated}, each coefficient can be estimated separately and the interpretation is thus easy : "A change in 1 in \(x_j\) is associated with a change of \(\beta_j\) in Y, while everything else stays the same."
\end{itemize}
\begin{itemize}
\item This interpretation is no longer true if the variables are \textit{correlated}. The correlations between variables cause problems : the variance of all coefficients tend to increase and the interpretation is much harder (when \(x_j\) changes, so does everything else).
\end{itemize}

\section{Logistic regression}

\paragraph{What about classification ?}
There are different conceptual issues with linear regression that make it unsuitable for classification. The main one is that we can't model the probability \(P(y=1|\xx)\) as a linear function of $\xx$ because it must be between 0 and 1. Moreover, this cannot model "diminishing returns" where the impact of a change in $\xx$ is not constant along the probability range. Indeed, if \(P(y=1|\xx)\) is close to 0 or +1, $\xx$ must change a lot for y to change and this is not the case when \(P(y=1|\xx)\) is close to 0.5. \\
Hence we use a \textbf{logit transformation} through a \textit{logistic regression}.\\

\begin{figure}[H]
\centering
\begin{minipage}[b!]{0.4\textwidth}
\includegraphics[width=70mm]{graphe1.png}
\end{minipage}
\hfill
\begin{minipage}[b!]{0.4\textwidth}
\includegraphics[width=70mm]{graphe2.png}
\end{minipage}
\end{figure}
\begin{center}
\(\log \dfrac{P(y=1|\xx)}{1 - P(y=1|\xx)} = \beta^T\xx + \beta_0\) with \(\beta = (\beta_0, ..., \beta_p)\)\\
\end{center}

\paragraph{Maximum likelihood estimation of logistic regression coefficients}

As usual, let's compute the log likelihood for this logistic regression, knowing our data \(\dd = {\{\xx^i,\textit{y}^i\}}_{\textit{i}=1,..,\textit{n}}\)  :\\
\\
\[\mathcal{L}(\beta|\dd) = \displaystyle\sum_{i=1}^{n} log P(y^i|\xx^i) + Cte = \displaystyle\sum_{i=1}^{n} (y^i \log g^i + (1-y^i) \log(1-g^i))\]
\begin{center}
with \(g = P(y=1|\xx) = \frac{1}{1+e^{-\beta^T\xx}}\)
\end{center}
We find the maximum of the log-likelihood by calculating its gradient \(\nabla_{\beta}\mathcal{L}\):

\[\nabla_{\beta}g^i = \xx^i g^i (1-g^i)\]

\[\nabla_{\beta}\mathcal{L} = \displaystyle\sum_{i=1}^{n} (y^i-g^i)\xx^i\]\\
\\
And solving for zero in the gradient formula :

\[\displaystyle\sum_{i=1}^{n} \left( y^i - \frac{1}{1 + e^{-\beta^T\xx^i}} \right) = 0\]\\
\\
However, this expression cannot be solved analytically. \\ Since $\mathcal{L}$ is concave, hence there is no local minima, we can use the gradient ascent method.
\\
\paragraph{Gradient ascent method} For a function J concave in $\beta$ :

\begin{itemize}
\item Update rule : \(\beta^{(t+1)} \leftarrow \beta^{(t)} + \eta \nabla_\beta J(\beta^{(t)})\)
\item Iterate until change is inferior to a chosen margin $\epsilon$
\item $\eta$ is the learning rate
\end{itemize}

\begin{center}
\includegraphics[width=100mm]{gradient.png}
\end{center}
Other methods remain possible such as the Newton method, conjugate gradient ascent or IRLS.

\section{Summary}

The main points to keep in mind from this lesson are the following :
\begin{itemize}
\item MAP estimate : \(\hat{\theta}_{\mbox{MAP}}=\arg\underset{\theta}\max \  p(\theta|\mathcal{X})\)
\item MLE : \(\hat{\theta}_{\mbox{MLE}}=\arg\underset{\theta}\max \  p(\mathcal{X}|\theta)\)
\item Bayes estimate : \(\hat{\theta}_{\mbox{Bayes}}=\mathbb{E}[\theta|\mathcal{X}]=\int{\!\theta p(\theta|\mathcal{X})d\theta}\)
\item Assuming a Gaussian error, maximizing the likelihood (MLE) is equivalent to minimizing the RSS (residual sum of squares).
\item Linear regression MLE : \(\hat{\beta} = (\textbf{\textit{X}}^T\textbf{\textit{X}})^{-1} \textbf{\textit{X}}^T y\)
\item Logistic regression : to solve with gradient ascent.
 \end{itemize}
\end{document}


\documentclass[runningheads]{llncs}
\usepackage[hyphens]{url}
\usepackage{graphicx}
\usepackage{amsmath}

\begin{document}

\title{Genetic Model Design for 2021 Huawei Delivery Optimization Competition}

\author{Zhenshuo Chen\inst{1}\orcidID{0000-0003-2091-4160} \and
    Guowen Liu\inst{2}\orcidID{0000-0002-8375-5729}}

\institute{Dublin City University, Dublin, Ireland \and
    Trinity College Dublin, Dublin, Ireland}

\maketitle

\begin{abstract}
We design a genetic model for 2021 Huawei Delivery Optimization Competition,
which aims to minimize multi-vehicle transportation costs within the constraints of vehicle specifications.
We use the one-dimensional array as genotype to represent the transportation routes of multiple vehicles.
This representation allows us to directly use existing genetic algorithms for multi-vehicle route optimization.

\keywords{Genetic Algorithm \and Multi-vehicle Route Optimization.}
\end{abstract}

\section{INTRODUCTION}
The competition task is to takes care of deliveries of consumables from a depot to customers, as shown in Fig.~\ref{fig:overview}.

\begin{figure}[htbp]
    \centerline{\includegraphics[width=0.8\linewidth]{figures/overview.png}}
    \caption{The overview}
    \label{fig:overview}
\end{figure}

\begin{itemize}
    \item
    The depot has a given set of home vehicles, each with a certain capacity for carrying consumables.

    \item
    Customers are represented as nodes in a graph, in which edges have an associated transportation cost equal to the distance between the nodes.

    \item
    Each customer node has an associated demand for a consumable.
\end{itemize}

The target is to minimize the total transportation cost by determining a set of routes that meet the following requirements:

\begin{itemize}
    \item Each vehicle has one route that starts and finishes at the depot.

    \item Each customer is visited exactly once.

    \item The total demand along each route is less than the vehicle capacity.
\end{itemize}

In a route, the depot is represented by node $1$ and customer numbers start from node $2$.
For example, the route $[1, 2, 3, 4, 1]$ means a vehicle starts from the depot ($1$), visits customer nodes $2$, $3$, $4$ and returns to the depot ($1$).

\section{MODEL DESIGN}
We used the genetic algorithm as our main process.
It simulates the process of natural selection which means those species who can adapt to changes in their environment are able to survive, reproduce and go to next generation as Fig.~\ref{fig:genetic}.

\begin{figure}[htbp]
    \centerline{\includegraphics[width=\linewidth]{figures/genetic.jpg}}
    \caption{A genetic model}
    \label{fig:genetic}
\end{figure}

\subsection{Individual}
For a task containing $N$ customers and $M$ vehicles, we used a permutation of $N + M - 1$ consecutive numbers (0-based) to represent a transportation plan, consisting of both customers and depots.
$M - 1$ depots can split the customers into $M$ segments. Each segment represents a route.
For instance, if a task contains eight customers and three vehicles, a transportation plan can be generated as Fig.~\ref{fig:individual}.
In the example individual, $0, \ldots, 7$ are customers and $8, 9$ are the depot.

\begin{figure}[htbp]
    \centerline{\includegraphics[width=\linewidth]{figures/individual.png}}
    \caption{Using an individual to represent multi-vehicle routes}
    \label{fig:individual}
\end{figure}

This representation can produce empty routes such as the individual $[3,0,1,2]$ for a task containing three customers and two vehicles.
The depot $3$ is in the first position, so the routes are $[1, 1]$ and $[1, 2, 3, 4, 1]$, transformed from an empty segment and $[0,1,2]$.

\subsection{Mutation}
In a mutation operation, we randomly swapped the order of two nodes as Fig.~\ref{fig:mutation}.

\begin{figure}[htbp]
    \centerline{\includegraphics[width=0.8\linewidth]{figures/mutation.png}}
    \caption{Mutation}
    \label{fig:mutation}
\end{figure}

\subsection{Crossover}
In a crossover operation, we executed ordered crossover on two parents to produce children, as shown in Fig.~\ref{fig:crossover}.

\begin{figure}[htbp]
    \centerline{\includegraphics[width=0.8\linewidth]{figures/crossover.png}}
    \caption{Ordered crossover}
    \label{fig:crossover}
\end{figure}

\subsection{Selection}
We use tournament selection, keeping transportation plans with the highest fitness from randomly chosen candidates.

The fitness is the cost of a transportation plan.
For overloaded plans, a large value is added to fitness as a penalty. This value increases as overload weight increases, see Equation.~\ref{eq:penalty}.

\begin{equation}
    Penalty = \sum_{i}\sum_{j}c_{i, j} + 2 \cdot W
    \label{eq:penalty}
\end{equation}

$c_{i, j}$ is the cost between the node $i$ and $j$, and $W$ is overloaded weight of the transportation plan.

\section{IMPLEMENTATION}
Our implementation is based on \verb|DEAP| framework \cite{deap}.
The source code is available on the GitHub \footnote{\url{https://github.com/Zhuagenborn/Huawei-Delivery-Optimization}}.

\bibliographystyle{splncs04}
\bibliography{huawei-delivery-optimization}

\end{document}


\subsection{Multi-Agent Proximal Policy Optimization}
\label{subsec:41}

Since the naive application of standard reinforcement learning algorithms performs poorly in multi-agent settings, our goal is to derive an algorithm that can operate under the following constraints:
\begin{itemize}
\item{The learned policies can only use local information (i.e. own observations) at execution time.}
\item{There is no particular communication method between the agents, since it can restrict the scalability of the solution.}
\item{We do not assume a differential model of the environment dynamics.}
\end{itemize}
Fulfilling the above desiderata would provide a general-purpose multi-agent learning algorithm that could be applied not just to cooperative games with explicit communication channels, but also to competitive games and games involving only physical interactions between agents.

We accomplish our goal by adopting the framework of centralized training with decentralized execution in which agent policies are composed of two separate networks with different parameters: a policy network (actor) that generates an action distribution and a critic network (critic) that predicts discounted future returns. The critic uses extra information to ease training process whilst actors take actions based solely on their own local observations. This is still a valid approach as long as the information used during the training of the agents are not used at test/execution time.

% For figures use
%
\begin{figure}[h!]
%\sidecaption
% Use the relevant command for your figure-insertion program
% to insert the figure file.
% For example, with the graphicx style use
\includegraphics[scale=.65]{images/MAPPO}
%
% If no graphics program available, insert a blank space i.e. use
%\picplace{5cm}{2cm} % Give the correct figure height and width in cm
%
\caption{Overview of our multi-agent decentralized actor, centralized critic approach. The centralized action-value function (critic) for each agent takes as input the actions and observations of all agents and predicts the discounted future returns (Q-values) for the agent while the actor produces an action distribution based solely on the agent's observations and actions.}
\label{fig:MAPPO}       % Give a unique label
\end{figure}

As illustrated in Fig. \ref{fig:MAPPO}, our centralized action-value function (critic) for each agent takes as input the actions and observations of all agents and predicts the discounted future returns for the agent. Additional information can also be included in the inputs. Since each action-value function is learned separately, each agent can have arbitrary rewards including conflicting rewards in competitive scenarios.

Furthermore, we extend the idea of \cite{schulman2017proximal} by adopting the proximal policy optimization algorithm (PPO) as the main policy optimization algorithm instead of deep deterministic policy gradient (DDPG) as presented in the paper. PPO is a family of policy optimization methods that use multiple epochs of stochastic gradient ascent to perform each policy update. It performs comparably to or better than DDPG while being much simpler to implement and tune.

\subsection{Cooperation and Competition}
\label{subsec:42}
As we seek the emergence of coordination strategies between the components of the micro-grid, the agents controlling the micro-grid components are trained using \textit{self-play} in an \textit{autocurricula} configuration. This acts as a natural curriculum as these agents are always in a mixed cooperative and competitive setting. For example, the energy loads (i.e. production machines) of the micro-grid are always in competitive settings. If there is not enough electrical energy available in the grids, then some production machines have to turn themselves off so that the production can be carried on by the other machines. In the same direction, cooperation can take place if, for example, the energy storage components need to be switched to the discharging state in order to increase the electrical energy available on the energy bus and thus enable the production machines to perform additional production tasks.

Adopting such a configuration also offers many advantages. If a new successful strategy is adopted by an agent, it implicitly changes the task distribution for the competing agents which are then forced to develop a better strategy. These evolutionary arms races create an implicit autocurriculum in which competing agents constantly create new tasks for each other and thus discover new powerful and robust strategies and counter-strategies. In addition, the multi-agent autocurriculum is open-ended. That means that the training will only stabilizes once equilibrium between the different agents is founded which can only be reached if a suitable coordination strategy between the agents has emerged. Due to the exploration factors, many coordination strategies can also emerge at different stages of the training.

In order to create a competitive setting, competing agents are rewarded with competitive rewards. If an agent wins the game, it receives a positive reward and all other competing agents are penalized by an opposite negative reward. If nobody wins the game, then all competing agent are penalized. In a cooperative setting, agents are given a team based reward; meaning that all members of the team receive a positive reward, if a team member wins the game. To confine agent behavior to a reasonable space, we also introduced an environmental-based penalization.

The success of agents in the autocurricula setting requires the agents to occasionally solve the task (win the game) by random actions. The probability of this happening in most games is minuscule as they require as a prerequisite some fundamental skills like the ability to properly control the operational state machine of the grid component. For example the agent must first quickly learn how to bring a production machine in the $execute$-state in order to start a production task or how to turn on/off the storage component without suspending/halting the production. To overcome this problem, we use simple dense rewards at each step so that the agents can first learn basic motor skills that increase the likelihood that random actions of the agent will result in a relatively small positive reward. We refer to this reward as the \textit{exploration reward}. The exploration reward is then gradually annealed to zero, in favor of the \textit{competition/cooperation reward}, to allow the agents to learn coordination strategies for the remaining training time. This is achieved using a linear annealing factor $\alpha$. We also introduce an \textit{environmental reward} which aims to penalize action sequences leading to dangerous outcomes (for example system configurations violating the constraints formulated in Eq. \ref{eq:constraint_energy_balance} and Eq. \ref{eq:constraint_load_profile}) and a \textit{productivity reward} which encourage production machines to execute production tasks more frequently.

So, at time-step $t$, if the exploration reward is $r_{explo}$, the competition reward is $r_{comp}$, the environmental reward $r_{env}$, the the productivity reward $r_{prod}$ and T is the termination time-step, then the total agent reward is:
\begin{equation}
\label{eq:total_reward}
	r_t = \alpha \cdot r_{explo} + (1 - \alpha) \cdot (r_{comp} + r_{env} + r_{prod})
\end{equation}

With this formulation, the agents are trained initially on the dense reward for only about 10-20\% of the training epochs in order to gain some basic skills first.


\section{Results}
\label{results}

Our Spark program is written in Scala using Hadoop distribution $2.7.2$, and run on a Intel (R) Xeon machine with two physical CPUs. Each CPU has $16$ E$5-2650$ @ $2.00$GHz cores, giving us a total $32$ cores. The machine also has $32$GB of RAM, and runs Ubuntu $16.04$.

\subsection{Input description}

As discussed earlier, our input graph $G$ is an undirected, unweighted graph with $679965$ edges and $899162$ vertices. Each vertex and edge is represented using 8 bytes and so the total amount of memory required by our graph is about $20$MB.

Our input graph also consists of $199004$ connected components, with the maximum component size at $145$ and the minimum at $2$. The distribution of component sizes is shown in Fig. \ref{fig:vertices-components-all}, where the numbers of components with sizes above, say $40$, were dwarfed by the components with smaller sizes. Fig. \ref{fig:vertices-components-40} (resp. Fig. \ref{fig:vertices-components-20}) is an alternative figure that reveals the distribution of larger (resp. smaller) components.

To investigate the scalability of our code for increasing input, we replicate the graph by doubling it (labeled graph $G_2$) then quadrupling it (labeled graph $G_4$), whilst preserving the same structure as far as connected components are concerned.

\subsection{Failures and connectivity loss analysis}

For each of the four scenarios described in Sec. \ref{methods}, we vary the number of threads and partitions, compute the percentage of loss, and plot this value against the number of nodes removed from the graph. In all the tables and figures below, $R$ corresponds to random failures, $D$ to degree based failures, $BC$ to betweenness centrality (load) failures, and $C$ to cascading failures. Fig. \ref{fig:loss-100} and \ref{fig:loss-1000} present a refined view demonstrating the progression of loss for the first 100 and 1000 failures on transmission and distribution nodes. Fig. \ref{fig:loss-all} shows the loss for the entire failures. The connectivity loss is faint and proportional to the number of failures in the random case. It is worse for the cascading scenario, whereas the connectivity losses associated with the load-based versus degree based scenarios are more or less comparable. These findings are confirmed in Table~\ref{resilience}, which presents, for each scenario, the percentage of failed nodes effecting in $60\%$ and $80\%$ (nearly total) connectivity loss. Some of these figures, particularly, the relatively high percentages required in each of the $BC$, $D$, and $R$ scenarios that precede total failure, can be interpreted to say that the Lebanese grid is highly redundant, and thus somehow resilient, thanks to its decomposition into numerous components and its reliance on local diesel generators that alleviate the effects of blackouts and failures.

\begin{table}[t]
\centering
\caption{Percentage of vertices to be removed to reach $60\%$ and $80\%$ losses}
\label{tab:threshold-percentage}
{\small
\begin{tabular}{||c||c|c|c|c||}
\hline
\textbf{Loss}	&\cellcolor{black!10}BC&	\cellcolor{black!10}C & \cellcolor{black!10} R & \cellcolor{black!10} D \\ \hline \hline
60\% &		11.6\%&	4.3\%	&59.8\%&	7.7\%	 \\ \hline
80\%	&		27.8\%&	14.6\%	&79.7\%&	29.5\%	 \\ \hline
\end{tabular}
}
\label{resilience}
\end{table}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/loss-100-crop.pdf}
\caption{Loss percentage: first 100 attacks}
\label{fig:loss-100}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/loss-1000-crop.pdf}
\caption{Loss percentage: first 1000 attacks}
\label{fig:loss-1000}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/loss-all-crop.pdf}
\caption{Loss percentage: overall attacks}
\label{fig:loss-all}
\end{figure}

\subsection{Run-time and parallel efficiency analysis}
\secondedited{
The run-time for each pair of ({\it thread},{\it partition}) values using our input graphs are shown in the tables~\ref{tab:graph1}, \ref{tab:graph2}, and \ref{tab:graph4}, for $G$, $G_2$, and $G_4$ respectively. Moreover, the corresponding performance efficiency plots are shown in Fig. \ref{fig:effbc1}, \ref{fig:effbc2}, \ref{fig:effbc4}, \ref{fig:effc1}, \ref{fig:effc2}, \ref{fig:effc4}, \ref{fig:effd1}, \ref{fig:effd2}, \ref{fig:effd4}, \ref{fig:effr1}, \ref{fig:effr2}, and \ref{fig:effr4}.
}
%
\edited{
Here, parallel efficiency is defined as $\frac{T_s}{p\cdot T_p}$, where $T_s$ denotes the serial run-time and $T_p$ denotes the parallel run-time given $p$ parallel processes.}
%
\secondedited{
The timings shown in the tables correspond to the parallel phase of the algorithm. The sequential run-time was omitted because it was extremely negligible compared to the parallel part that is of higher order (in contrast to linear running time in steps 7 and 8).
%
By default, Spark sets the partition size at 64 MB. This is too large for our given graph. As a result, we choose to override the default value by specifying the number of partitions at compile time. When this is done, Spark re-adjusts the size of each partition based on their total number as well as on the size of the input graph. The several connected components in each graph get mapped onto all partitions such that the number of vertices in each partition is balanced across all partitions. We experiment with a number of partitions ranging from $4$, $8$, $16$, and $32$ in order to explore the effect that the number of partitions have on run-time and parallel efficiency. From Fig. \ref{fig:vertices-components-all} and \ref{fig:vertices-components-40}, we gather that there would be enough connected components in each partition to engage all threads assigned to the partition. Also, the fact that many connected components have sizes greater than, say, 40, ensures that the distributed work is more or less balanced.}

As expected, for all three graphs, the fastest and yet the worst parallel efficiency correspond to the R (random failures) scenario that does not rely on any centrality measure computation. In contrast, the highest run-times and hence best efficiency are for the C (cascading scenario) that updates the betweenness centrality of all the nodes following each node removal. It is clear that the more work entailed by a certain scenario, the better it will be for the parallel program to compensate for the overheads associated with the partitioning and scheduling operations. Reading across the three tables for the same partition size and same scenario, our tool shows improved scalability as the input size grows.

For each given graph and for each scenario employed, we notice that the actual run-time has opposing trends that depend on the number of partitions. For all three tables, there is a cut-off value for the number of partitions, before which run-time continues to improve, and after which it starts to deteriorate. For graph $G$, the cut-off number is at $8$, for graph $G_2$ it is somewhere between $8$ and $16$, and for $G_4$, it is at $16$. We justify the improvement in run-time as we move closer to the cut-off threshold as follows. With a higher number of partitions, one would expect that the multiple threads will be spread about, sharing the work but on data that is more split into distinct regions of main memory. As such, we have reduced contention over the shared address space, as well as scheduling costs associated with managing threads on one single partition. We now argue that creating more partitions beyond the cut-off number drastically affects the performance and introduces a huge overhead -- particularly when the number of threads is low. For instance, in Table II, if we consider only one thread of the BC scenario, it takes 161 (resp. 423) seconds in case of 4 (resp. 32) partitions. We attribute this overhead to the shuffling and repartition operations taking place at the end of each stage that assigns one or more threads from one partition to another. In that phase, some transformations (e.g., \texttt{groupByKey}, \texttt{reduceByKey}, \texttt{join} operations) require shuffling and repartitioning of data, for instance, to group all the items with the same keys in the same partition.

With that said, we observe that efficiency actually improves for larger partitions. This isn't to be construed, however, to mean that the parallelisation has improved in any way, but rather that the rate of deterioration in the performance for a smaller number of threads (particularly, the case of one thread) is higher when the number of partitions exceeds the cut-off number, resulting in a higher-speedup as the number of threads grows larger.

%Our own Spark implementation of contingency analysis executes in real-time for the entire power grid, achieving a speed-up of ? on ? processors. We conclude with a spatial understanding of the hotspots on Lebanese soil where energy centers can be exposed and are at risk, using a spatial correlation supported by a binary search tree. The amenability of our work to big data processing makes it extendable to larger networks of networks, towards a fuller understanding of resilence at many vital levels beyond the power grid. Examples are as communications network, Internet networks, transportation networks, hospitals and medical centers networks, to name a few.

\secondedited{
Moreover, we notice that the parallel efficiency in some cases is above one (e.g., in case of betweenness centrality 32 partitions and 2 threads). This may due to the effect of the garbage collectors. In case of one thread and several partitions, the threshold of the garbage collector would be reached ahead of time. Whereas in case of more threads/processes, less data needs to be freed.
%
Additionally, another cause behind super-linear speedup can be attributed to the ``caching effect'', which results from the varying speeds in accessing different levels of the memory hierarchy on which the input graph and intermediate data are stored. As the number of threads increases, the data assigned to each thread decreases, rendering it into the smaller cache levels which are faster to access. As a result, the reduction in run-time is not solely explained by the increase in the number of working threads but also in the reduction of the time spent on I/O, which causes the theoretical estimate for parallel speedup to go beyond $p$ (given $p$ threads), or equivalently, for parallel efficiency to go beyond 1.}

%\begin{figure}
%\centering
%\includegraphics[scale=0.35]{bench/generated/frequencyall-crop.pdf}
%\caption{Components size -- overall distribution}
%\label{fig:cs-overall}
%\end{figure}

%\begin{figure}
%\centering
%\includegraphics[scale=0.35]{bench/generated/frequency-selected-crop.pdf}
%\caption{Components size -- cropped distribution}
%\label{fig:cs-selected}
%\end{figure}

\begin{table*}[t]
\begin{minipage}[b]{\textwidth}
\caption{Run-time \edited{(in seconds)} for graph $G$}
\label{tab:graph1}
{\small
\centering
\begin{tabular}{||c||c|c|c|c||c|c|c|c||c|c|c|c||c|c|c|c|}
\hline
\textbf{Threads}	&\cellcolor{black!10}BC-4&	\cellcolor{black!10}BC-8	&\cellcolor{black!10}BC-16	&\cellcolor{black!10}BC-32&	\cellcolor{black!10}C-4	&\cellcolor{black!10}C-8&	\cellcolor{black!10}C-16&	\cellcolor{black!10}C-32&	\cellcolor{black!10}D-4&	\cellcolor{black!10}D-8	&\cellcolor{black!10}D-16&\cellcolor{black!10}	D-32&	\cellcolor{black!10}R-4	&\cellcolor{black!10}R-8&	\cellcolor{black!10}R-16&	\cellcolor{black!10}R-32 \\ \hline \hline
64&		27	&17	&22	&41	&53	&34	&32	&52	&26	&15	&20	&41	&23	&17	&20	&39			\\ \hline
32&		24	&17	&19	&37	&52	&33	&31	&49	&20	&16	&18	&36	&20	&19	&19	&36	\\ \hline
16&		24	&18	&22	&39	&53	&35	&37	&54	&21	&16	&19	&37	&19	&16	&20&37	\\ \hline
8	&	25	&25	&30	&54	&55	&50	&56	&80	&21	&21	&26	&51	&20	&20	&26	&50	\\ \hline
4	&	41	&41	&53	&97	&87	&86	&98	&138	&33	&33	&45	&89	&30	&31	&44	&90	\\ \hline
2&		82	&84	&108	&206	&165	&165	&189	&287	&65	&66	&92	&188	&55	&62	&87	&186	\\ \hline
1&		161	&167	&219	&423	&312	&320	&369	&543	&124	&127	&187	&384	&104	&114	&174	&344	\\ \hline
\end{tabular}
}
\end{minipage}
\end{table*}

\begin{table*}[t]
\begin{minipage}[b]{\textwidth}
\caption{Run-time \edited{(in seconds)} for graph $G_2$}
\label{tab:graph2}
{\small
\begin{tabular}{||c||c|c|c|c||c|c|c|c||c|c|c|c||c|c|c|c|}
\hline
\textbf{Threads}	&\cellcolor{black!10}BC-4&	\cellcolor{black!10}BC-8	&\cellcolor{black!10}BC-16	&\cellcolor{black!10}BC-32&	\cellcolor{black!10}C-4	&\cellcolor{black!10}C-8&	\cellcolor{black!10}C-16&	\cellcolor{black!10}C-32&	\cellcolor{black!10}D-4&	\cellcolor{black!10}D-8	&\cellcolor{black!10}D-16&\cellcolor{black!10}	D-32&	\cellcolor{black!10}R-4	&\cellcolor{black!10}R-8&	\cellcolor{black!10}R-16&	\cellcolor{black!10}R-32 \\ \hline \hline
64		&127	&71	&48	&55	&119	&64	&60	&76	&108	&58	&45	&53	&45	&32	&31	&50		 \\ \hline
32		&120	&59	&45	&49	&116	&68	&56	&71	&108	&57	&42	&47	&40	&29	&29	&45		 \\ \hline
16		&128	&66	&39	&52	&115	&66	&65	&79	&112	&57	&36	&50	&41	&33	&30	&46 \\ \hline
8		&123	&66	&56	&77	&120	&97	&98	&121	&106	&54	&46	&68	&40	&34	&39	&63	 \\ \hline
4		&164	&104	&101	&142	&210	&191	&185	&223	&134	&83	&80	&119	&60	&57	&69	&113	 \\ \hline
2		&263	&203	&207	&292	&416	&353	&366	&462	&209	&159	&153	&239	&111	&110	&131	&224	 \\ \hline
1		&452&357	&371	&525	&761	&804	&793	&969	&355	&281	&286	&434	&196	&201	&230	&402 \\ \hline
\end{tabular}
}
\end{minipage}
\end{table*}

\begin{table*}[t]
\begin{minipage}[b]{\textwidth}
\caption{Run-time \edited{(in seconds)} for graph $G_4$}
\label{tab:graph4}
{\small
\begin{tabular}{||c||c|c|c|c||c|c|c|c||c|c|c|c||c|c|c|c|}
\hline
\textbf{Threads}	&\cellcolor{black!10}BC-4&	\cellcolor{black!10}BC-8	&\cellcolor{black!10}BC-16	&\cellcolor{black!10}BC-32&	\cellcolor{black!10}C-4	&\cellcolor{black!10}C-8&	\cellcolor{black!10}C-16&	\cellcolor{black!10}C-32&	\cellcolor{black!10}D-4&	\cellcolor{black!10}D-8	&\cellcolor{black!10}D-16&\cellcolor{black!10}	D-32&	\cellcolor{black!10}R-4	&\cellcolor{black!10}R-8&	\cellcolor{black!10}R-16&	\cellcolor{black!10}R-32 \\ \hline \hline
64 &		525&	317	&171&	117	&411	&179&	127	&123	&430	&220	&134	&104&	133&	79&	70	&87	 \\ \hline
32	&	491&	274	&153	&99	&387	&202	&123	&121 &449	&243&	149&	100	&122	&101&	64&	75\\ \hline
16	&	473	&311	&133	&96	&401	&184	&127	&137 &457	&236	&113	&77	&130	&86	&62	&78 \\ \hline
8	&	557&	250	&132	&126	&387	&234	&204	&212 &452	&222	&113	&105	&119&	93	&75	&93 \\ \hline
4		&750&	333	&221	&234	&612	&431	&374	&398 &601	&274 &168	&190	&151	&126&	120	&163 \\ \hline
2		&1041 &526 &430 &467	&1042	&791	&736	&793 &888	&436	&344	&395	&273	&235	&237	&337 \\ \hline
1		&1626&902&879 &1021 &2156 &1784 &1577 &1754 &1584 &905 &730 &783& 533 &458 &456 &647 \\ \hline
\end{tabular}
}
\end{minipage}
\end{table*}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-bc-1-crop.pdf}
\caption{Efficiency for the load based (BC) scenario and graph $G$}
\label{fig:effbc1}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-c-1-crop.pdf}
\caption{Efficiency for the cascading (C) scenario and graph $G$}
\label{fig:effc1}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-d-1-crop.pdf}
\caption{Efficiency for the degree-based (D) scenario and graph $G$}
\label{fig:effd1}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-r-1-crop.pdf}
\caption{Efficiency for the random (R) scenario and graph $G$}
\label{fig:effr1}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-bc-2-crop.pdf}
\caption{Efficiency for the load based (BC) scenario and graph $G_2$}
\label{fig:effbc2}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-c-2-crop.pdf}
\caption{Efficiency for the cascading (C) scenario and graph $G_2$}
\label{fig:effc2}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-d-2-crop.pdf}
\caption{Efficiency for the degree-based (D) scenario and graph $G_2$}
\label{fig:effd2}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-r-2-crop.pdf}
\caption{Efficiency for the random (R) scenario and graph $G_2$}
\label{fig:effr2}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-bc-4-crop.pdf}
\caption{Efficiency for the load based (BC) scenario and graph $G_4$}
\label{fig:effbc4}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-c-4-crop.pdf}
\caption{Efficiency for the cascading (C) scenario and graph $G_4$}
\label{fig:effc4}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-d-4-crop.pdf}
\caption{Efficiency for the degree-based (D) scenario and graph $G_4$}
\label{fig:effd4}
\end{figure}

\begin{figure}
\centering
\includegraphics[scale=0.35]{bench/generated/efficiency-r-4-crop.pdf}
\caption{Efficiency for the random (R) scenario and graph $G_4$}
\label{fig:effr4}
\end{figure}

\subsection{Spatial analysis}
The spatial correlation analysis shown in Fig. \ref{fig:correlation} reveals a long range correlated case with correlation decaying slowly with distance, particularly, in the linear regime where $C(r) \propto r^{-\gamma} $, $\gamma = 1.13$, which is in agreement with the literature results in \cite{DaqingAl14}.  Overload failures usually propagate through collective interactions among system components. Our results reveal that high failures in critical nodes has impact that propagates across long path lengths on Lebanese soil.

\begin{figure}
\centering
\includegraphics[scale=0.65]{bench/fixed/correlation-eps-converted-to.pdf}
\caption{Spatial correlation between nodes in the cascading scenario on a logarithmic scale.}
\label{fig:correlation}
\end{figure}