% --------------------------------------------------------------
% This is all preamble stuff that you don't have to worry about.
% Head down to where it says "Start here"
% --------------------------------------------------------------

\documentclass[12pt]{article}

\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm,amssymb}
\usepackage{color}
\usepackage{fancyhdr}
\usepackage{lastpage}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{url}
\usepackage{pdfpages}

\pagestyle{fancy}
\fancyhf{}
\fancyhead[L]{ID: 6110}
%\fancyhead[C]{\thepage}
%\fancyhead[R]{Module 5 Homework}
\fancyhead[C]{Project 4}
\fancyhead[R]{Prof. Hicken, MANE 4280}

\rfoot{Page \thepage \hspace{1pt} of \pageref{LastPage}}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}

\newenvironment{theorem}[2][Theorem]{\begin{trivlist}
		\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{lemma}[2][Lemma]{\begin{trivlist}
		\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{exercise}[2][Exercise]{\begin{trivlist}
		\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{problem}[2][Problem]{\begin{trivlist}
		\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{question}[2][Question]{\begin{trivlist}
		\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{corollary}[2][Corollary]{\begin{trivlist}
		\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}

\newenvironment{solution}{\begin{proof}[Solution]}{\end{proof}}
\usepackage{multicol}
\newcommand{\mysize}{0.5}
\usepackage{subcaption}
\usepackage{float}
\usepackage{listings}
\usepackage{color} %red, green, blue, yellow, cyan, magenta, black, white
\definecolor{mygreen}{RGB}{28,172,0} % color values Red, Green, Blue
\definecolor{mylilas}{RGB}{170,55,241}

\begin{document}

	% --------------------------------------------------------------
	%                         Start here
	% --------------------------------------------------------------

	\title{Project 4: Uncertain Loading of a UAV Spar}
	\author{ID: 6110\\ %replace with your name
		 MANE 4280, Design Optimization}

	\maketitle
	\tableofcontents
	\thispagestyle{empty}
	\newpage
	\setcounter{page}{1}
	\section{Executive Summary}

	A spar is a type of support found in the wing of an aeroplane. Used as early as the Wright brother's days, spars are used in plane wings today to provide structural support, and to transport the forces acting on the wings to the rest of the plane.

	\begin{figure}[h!]
		\centering
		\includegraphics[width=0.5\linewidth]{sparFlyer}
		\caption{Spars and ribs are visible in the aerofoils of the Wright brother's Flyer\cite{original1905wrightflyeriii}.}
		\label{fig:sparflyer}
	\end{figure}

	A spar similar to the type explored in this project is seen in figure \ref{fig:spardiagram} %\ref{fig:telescopicwing}\cite{afa}

	\begin{figure}[h!]
	\centering
	\includegraphics[width=0.7\linewidth]{sparDiagram}
	\caption{Diagram of spar with axis \cite{Hicken18}.}
	\label{fig:spardiagram}
\end{figure}

	The goal of this project is to develop an optimal design for a spar that will minimize the mass under uncertain loading subject to a given set of constraints. This report will explain the analysis methods and their limitations, discuss how the geometry was parameterized, investigate the optimization methods and their results.

	\section{Analysis Methods}
	Before the analysis begins the details of the spar should be listed. It has a length of 7.5 meters. It's density is 1,600 $\frac{kg}{m^3}$\cite{Hicken18Proj}. A diagram of the spar is seen in figure \ref{fig:spardiagram}. The minimum radius of the spar is one centimeter and the maximum radius is five centimeters. The thickness must be greater than 2.5 mm. The bending of the spar is modeled by Euler-Bernoulli Beam Theory using the following equations\cite{Hicken18}.

	\begin{equation}
	\frac{d^2}{dx^2}\left(EI_{yy}\frac{d^2 w}{dx^2}\right)=q,  \forall x\in[0,L]
	\end{equation}

	Where
	\begin{align}
	w& \text{ is the vertical displacement in the z direction (m)}\nonumber\\
	q(x)& \text{ is the applied load (N/m)}\nonumber\\
	E& \text{ is the elastic, or Young’s, modulus (Pa)}\nonumber\\
	I_{yy}& \text{ is the second-moment of area with respect to the y axis }\text{(m)}^2\nonumber
	\end{align}
	\begin{equation}
	I_{yy}=\int \int z^2 dz dy
	\end{equation}
	The spar is treated like a cantilever beam with the following boundary conditions:
	\begin{align}
	w(x=0)=0,&\text{   } \frac{d^2w}{dx^2}(x=L)=0\\
	\frac{dw}{dx}(x=0)=0,&\text{   }\frac{d^3w}{dx^3}(x=L)=0
	\end{align}
	This is to say that there is no vertical or angular displacement at the root and no stress at the tip. Once the vertical displacement is known, the stress may be solved by
	\begin{equation}
\sigma_{xx} (x)=-z_{max}E\frac{d^2w}{dx^2}
	\end{equation}

	The beam is discretized by the finite-element method. The finite-element equations come from the minimization of the potential energy. The solution is represented using Hermite-cubic shape functions. It is assumed that the spar does not experience any thermal effects due to the warping or the atmosphere's temperature.

	The load on the spar is described by

	\begin{equation}
	f ( x , \xi ) = f _ { \mathrm { nom } } ( x ) + \delta _ { f } ( x , \xi )
	\end{equation}
	Where
	\begin{equation}
	f _ { \mathrm { nom } } ( x ) = \frac { 2.5 W } { L } \left( 1 - \frac { x } { L } \right)
	\end{equation}
	And the probabilistic perturbation has the form

	\begin{equation}
	\delta _ { f } ( x , \xi ) = \sum _ { n = 1 } ^ { 4 } \xi _ { n } \cos \left( \frac { ( 2 n - 1 ) \pi x } { 2 L } \right)
	\end{equation}
	with
	\begin{equation}
	\xi _ { n } \sim \mathcal { N } \left( 0 , \frac { f _ { \mathrm { nom } } ( 0 ) } { 10 n } \right)
	\end{equation}

	\subsection{Limitations}
	This model is concerned only with the stress on a spar from a turn and the spar's mass. It does not consider any other aerodynamic problems the spar may face; such as flutter for instance. The model assumes the majority of the force will be applied at the root of the spar. This would be a poor assumption if a payload is mounted along the wing. Many UAVs do have wing mounted payloads, such as missiles.  Additionally the model does not take into account the manufacturability of the spar, some designs may provide lots of strength with little weight, but could never be manufactured in the real world.

	\section{Geometry parameterization}

	To simplify the analysis the spar's geometry was divided into small sections. As the sections get smaller and smaller they better approximate a spar with a shape determined by some sort of polynomial.

	The design variable is a vector X, with n nodes. The vector must represent the inner and outer radii of the spar at each node. The way this was implemented was as follows:
	\begin{align}
	\underbrace{X_{(2\times n,1)}}_\text{design variable}=[r_1,r_2,\dots,r_n,t_1,t_2,\dots,t_n]^T
	\end{align}
	Where r is the inner radius of the spar at a specific node and t is the thickness at a specific node.

	\section{Optimization}
	The objective function of this problem is to minimize the mass of the spar while ensuring that the mean plus six standard deviations of the stress remains below the ultimate strength of the carbon fiber. The dependent variable is an 'X' vector. This X vector, holds the inner radius and corresponding thickness of the cylinder at each node. \par
\begin{equation*}
\begin{aligned}
& \underset{X}{\text{minimize}}
& & \text{mass}(X) \\
& \text{subject to}
& & X_j \geq 0.01 \text{ meter},                    &j=1,2,\dots,n \\
&&& X_{n+j} \geq 0.0025 \text{ meter},            &j=1,2,\dots,n\\
&&& X_j + x_{n+j} \leq 0.05 \text {meter},       &j=1,2,\dots,n \\
&&& \mu(\text{Stress})+ 6 \sigma \leq \text{Stress}_\text{yeild},
&j=1,2,\dots,n
\end{aligned}
\label{eq:optim}
\end{equation*}

%	\begin{align}
%		\begin{array}
%		{ c l } { \underset { x } { \operatorname { minimize } } } & { m ( \boldsymbol { x } ) } %&{} \\
%		{ \text { subject to } } & { x _ { j } \geq 0.01 } & { j = 1,2 , \ldots , N _ { x } }\\
%		{ } & { x _ { N _ { z } + j } \geq 0.0025 } & { j = 1,2 , \ldots , N _ { x } }\\
%		{ } & { p _ { j } + p _ { N _ { x } + j } \leq 0.05 }
%		\end{array}
%	\end{align}

%	\begin{align}
%	\begin{array} { l l } { \underset { \boldsymbol { p } } { \operatorname { minimize } } } & { m ( \boldsymbol { p } ) } \\ { \text { subject to } } & { p _ { j } \geq 0.01 , } \\ { p _ { N _ { x } + j } \geq 0.0025 , } & { j = 1,2 , \ldots , N _ { x } } \end{array}
%	\end{align}

%	\begin{array} { l } { p _ { j } \geq 0.01 } \\ { p _ { N _ { x } + j } \geq 0.0025 } \\ { p _ { j } + p _ { N _ { x } + j } \leq 0.05 } \end{array}

The minimization of the mass of the spar will be accomplished via the fmincon function in MATLAB and the complex step method. As the minimum mass of the spar is sought, no manipulation of the output is need (if the maximum mass was sought, the MATLAB implementation would have to search for $1/\text{mass}$, as these methods traditionally seek a variable to minimize a functions output).

	\subsection{Optimization Methods}
	The optimization method used in this project was the complex step method. The complex step method is useful because it's step value is not limited by machine epsilon. As such it can take far smaller steps than Sequential Quadratic Programming (SQP)\cite{hoddinottP1}.\par

	The complex step method was used to provide gradients for both the objective function (the minimization of mass) and the nonlinear constraint (the stress). As previous reports have throughly explained the complex step method, it is not described any further in this report. A more detailed description may be found in \cite{hoddinottP2}.

	The uncertanty is implemented via stochastic collocation. Stochastic collocation operates as follows
	\begin{enumerate}
		\item Compute quadrature points, denoted $\xi^{(k)}$, that are specifically tailored to the given probability density function and the corresponding
		quadrature weights $w^{(k)}$.
		\item Adjust weights and quadrature points via equation \ref{eqn:adj} \cite{Hicken18}

		\begin{equation}
		\begin{array} { l } { w ^ { ( k ) } = \frac { w ^ { ( k ) } } { \sqrt { \pi } } } \\ { \xi ^ { ( k ) } =\sqrt { 2 } \sigma \xi ^ { ( k ) } + \mu } \end{array}
		\label{eqn:adj}
		\end{equation}
		\item Sample function f at quadrature points.
		\item Compute mean as the weighted sum using equation \ref{eqn:compMean} \cite{Hicken18}
		\begin{equation}
		\mu _ { f } \approx \sum _ { k = 1 } ^ { m } w ^ { ( k ) } f \left( \xi ^ { ( k ) } \right)
		\label{eqn:compMean}
		\end{equation}
	\end{enumerate}

	Table \ref{tab:exactGH} gives the first Hermite-Gauss quadrature points and weights. In the MATLAB code the Gauss Hermine quadrature locations and weights were computed using the GaussHermiteFunc code, which was developed from Geert Van Damme's Gauss-Hermite Quadrature function \cite{ghMATLABcode}.
	\begin{table}[H]\fontsize{10}{14}\selectfont
		\caption{Exact Hermite-Gauss quadrature points and weights \cite{mathTabBK}}
		\label{tab:exactGH}
		\begin{tabular}{|l|l|l|l|l|l|l|l|l|}
			\hline
			$n$ & 2 & 3 &  & 4 &  & 5 &  &  \\ \hline
			$x_i$ & $\pm0.707107$ & 0 & $\pm1.22474$ & $\pm0.524648$ & $\pm1.65068$ & 0 & $\pm0.958572$ & $\pm2.02018$ \\ \hline
			$w_i$ & 0.886227 & 1.18164 & 0.295409 & 0.804914 & 0.0813128 & 0.945309 & 0.393619 & 0.0199532 \\ \hline
		\end{tabular}
	\centering
	\end{table}\fontsize{12}{12}\selectfont

%	\begin{table}[H]
%	\caption{Exact GH values}
%	\label{tab:compGH}
%	\begin{tabular}{|l|l|l|l|l|l|l|l|l|}
%		\hline
%		$n$ & 2 & 3 & a & 4 & a & 5 & a & a \\ \hline
%		$x_i$ & a & 0 & a & a & a & 0 & a & a \\ \hline
%		$w_i$ & 0.886227 & 1.18164 & 0.295409 & 0.804914 & 0.0813128 & 0.945309 & 0.393619 & 0.0199532 \\ \hline
%	\end{tabular}
%\end{table}

	\subsection{Constraints}

	The inner radius of the spar can be no smaller than 1 cm, and the outer radius of the spar can be no lager than 5 cm. The distance between the inner and outer radii must be lager than 2.5 mm.

	%The spar must be able to handle the stress of 2.5 g maneuver (a maneuver where the force on the spar is 2.5 times the weight of the aircraft), where the force is linearly distributed starting at the root and ending at the tip plus six standard deviations of uncertanty. \par

	To accomplish these constraints the MATLAB fmincon function used it's lower bound, upper bound, linear inequality, and non linear constraints. For more details of keeping the the spar within it's geometrical bounds please see \cite{hoddinottP2}. What will be detailed here is the stress constraint under uncertain loading.

	The most important constraint is that the maximum stress must be six standard deviations below the spar's ultimate tensile/compressive strength, or:

	\begin{equation}
\mu(\text{Stress})+ 6 \sigma \leq \text{Stress}_\text{yeild}
	\end{equation}

	For a single variable Gauss-Hermine, with m = 3, the mean or expected stress is given by equation \ref{eqn:sing} \cite{Hicken18}.
	\begin{equation}
	\mu _ { f } ( x ) \approx \frac { 1 } { \sqrt { \pi } } \sum _ { k = 1 } ^ { 3 } w ^ { ( k ) } f \left( \sqrt { 2 } \sigma \xi ^ { ( k ) } + x \right)
	\label{eqn:sing}
	\end{equation}

	For p uncorrelated Gaussian random variables with $\xi _ { i } \sim \mathcal { N } \left( x _ { i } , \sigma _ { i } \right) _ { 1 } i = 1,2 , \ldots , p _ { i}$, the mean or expected stress is given by equation \ref{eqn:mult} \cite{Hicken18}.
	\begin{equation}
	\mu _ { f } ( x ) = \left( \prod _ { i = 1 } ^ { p } \frac { 1 } { \sqrt { 2 \pi \sigma _ { i } } } \right) \int _ { \mathbb { R } ^ { p } } f ( \xi ) \exp \left( - \sum _ { i = 1 } ^ { p } \frac { \left( \xi _ { i } - x _ { i } \right) ^ { 2 } } { 2 \sigma ^ { 2 } } \right) d \xi
	\label{eqn:mult}
	\end{equation}

	This may be approximated via equation \ref{eqn:approx} \cite{Hicken18}
	\begin{align}
	\mu _ { f } ( x ) = \left( \prod _ { i = 1 } ^ { p } \frac { 1 } { \sqrt { \pi } } \right) \sum _ { k _ { 1 } = 1 } ^ { m } \sum _ { k _ { 2 } = 1 } ^ { m } \ldots \sum _ { k _ { p } = 1 } ^ { m } \left( \prod _ { i = 1 } ^ { p } w ^ { \left( k _ { i } \right) } \right) 	\times\label{eqn:approx} \\\nonumber
 f \left( \sqrt { 2 } \sigma _ { 1 } \xi ^ { \left( k _ { 1 } \right) } + x _ { 1 } , \sqrt { 2 } \sigma _ { 2 } \xi ^ { \left( k _ { 2 } \right) } + x _ { 2 } , \ldots , \sqrt { 2 } \sigma _ { p } \xi ^ { \left( k _ { p } \right) } + x _ { p } \right)
	\end{align}
	The variance is computed from the mean by equation \ref{eqn:VarFmean}.
	\begin{equation}
	Var(x)=\mu(x^2)-\mu(x)^2
	\label{eqn:VarFmean}
	\end{equation}
	And the standard deviation is then computed from the variance via equation \ref{eqn:stdFVar}.
	\begin{equation}
	\sigma=\sqrt{\mu(x^2)-\mu(x)^2}=\sqrt{Var(x)}
	\label{eqn:stdFVar}
	\end{equation}
	In MATLAB this nonlinear inequality constraint is found in the stressConst function, where the following inequality constraint is passed to fmincon:

	%	From here
	%However this formula  requires the weights and quadrature locations to be computed every time the function is evaluated. This would increase the run time significiatly. So instead the weights and quadrature locations are precomputed, and then the expected value is found via
	%The

	%The most important constraint is that of the maximum stress must not exceed the spar's ultimate tensile/compressive strength. This is done by the nonlcon function within fmincon. By calculating the displacement from CalcBeamDisplacement.m then getting the stress from CalcBeamStress.m the function stressCon.m passed the following  inequality constraint that checks the stress is allowable:
	\begin{align}
	\sigma_\text{current stress}=\mu(Stress)+6\times \sigma\\\nonumber
	c=\frac{\sigma_\text{current stress}}{\sigma_\text{yeild stress}}-1\leq 0
	\end{align}

	 The code that generates these matrices and all of the other codes may be seen in the appendix.

	\section{Results}

	The nominal mass of the spar was given to be 29.32 kg, based on an 15 element analysis with a inner radius of 4.15 cm and a outer radius of 5 cm. The resulting stress from the nominal configuration may be seen in in figure \ref{fig:nomstress}. The objective was to obtain an optimal mass that was 70\% lower than the nominal mass.

	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{nomStress}
		\caption{Stress Distribution from Nominal Configuration}
		\label{fig:nomstress}
	\end{figure}
\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{codeNomMass}
	\caption{Stress Distribution from Nominal Configuration From Code}
	\label{fig:codenommass}
\end{figure}

	The MATAB code outputs the stress associated with the nominal configuration seen in figure \ref{fig:codenommass}. These plots are nearly identical, which means the code is working as we expect it to.

	Running the MATLAB code at 15 elements and 3 collection points gave a result of 8.56387 kg, a little more than 70\% less than nominal. The output from MATLAB's fmincon may be seen in figure \ref{fig:fminconoutput} and the resulting stress and geometry may be seen in figure \ref{fig:coderesultsall}.

	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{FminConOutPut}
		\caption{Output from fmincon}
		\label{fig:fminconoutput}
	\end{figure}

	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{codeResultsAll}
		\caption{Spar Stress and Geometry  at 15 elements, and 3 collection points. }
		\label{fig:coderesultsall}
	\end{figure}

	Comparing this with the results from project 2 (optimal mass of 4.343), seen in figure \ref{fig:geovstress} shows the spars have a similar shape. Both spars start at a maximum $I_{yy}$ at the root to handle the stress (as the highest force is applied at the root). From there it decreases the mass by first shrinking the thickness, until the minimum thickness is hit. Then it decreases the mass by shrinking the inner radius while keeping the spar able to handle the stress. Eventually it hits a point where the radius and the thickness cannot be shrunk, and this the spar stays at this minimum cross section. However the spar from project 2 does not have to deal with six standard deviations of uncertainty, and can minimize it's mass far more.

	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{geoVstress}
		\caption{Geometry of spar and stress of spar from project 2.}
		\label{fig:geovstress}
	\end{figure}

	What will happen if the code is run for more that three collection points? Running the code at six collection points provided a spar with a mass of 8.56387 kg and a geometry and stress seen in figure \ref{fig:fivecolcresults}.

	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{fiveColcResults}
		\caption{Spar Stress and Geometry  at 15 elements, and 5 collection points.}
		\label{fig:fivecolcresults}
	\end{figure}

	This the results at five collection points are no different than the results at three collection points. In fact if the code is run for different numbers of elements and collection points(as seen in figure \ref{fig:massallplt}) the mass never changes due to the number of collection points, it only changes due to the number of elements.

	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{massAllPlt}
		\caption{Mass vs number of collection points and number of elements}
		\label{fig:massallplt}
	\end{figure}

	As a final check on convergence, we plot the displacement of the tip of the spar vs the number of nodes. This relation, seen in figure \ref{fig:tipdisplacment} shows that the displacement of the tip of the spar changes rapidly with the number of elements at first, but soon settles to a value of 0.681 meters. It stays there even though the number of elements increases.

	\begin{figure}
		\centering
		\includegraphics[width=0.7\linewidth]{tipDisplacment}
		\caption{Spar Tip displacement}
		\label{fig:tipdisplacment}
	\end{figure}

	\section{Conclusions}
	%It is a common falicy that alimuinum is used in aeroplanes because it is lighter than steel.

	The MATLAB results are what would be expected. When dealing with a large uncertainty, the spar will be heavier than the spar that was developed in uncertainty free project 2. The spar has a similar geometry to project 2's spar, just scaled to handle the uncertainty. The $I_{yy}$ must be at it's maximum near the root, then as it moves down the wing it can shrink to decrease mass. It first shrinks the thickness, and when it hits the minimum thickness it shrinks the radius. However at a certain point the radius can shrink no more and it stays at the minimum radius and thickness for the rest of the spar length. Like project 2, the stress is kept at the appropriate factor of safety for as long as possible.\par

	What was unexpected was that the number of collection points had no difference in the resulting mass. The only thing that changed the mass was the number of elements the spar was made up of.

	Something that engineers must take into account when designing models is the run time cost. The time to run each calculation at a different number of nodes and collection points is seen in figure \ref{fig:comptime}.\par
	%\newpage
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.7\linewidth]{runTimePlt}
		\caption{Time to run optimization vs the number of collection points and nodes}
		\label{fig:comptime}
	\end{figure}

	 As the number of nodes and collection points increases, so too does the time to run the model increase. This is important as some models may take to long to run for their results to be useful. Additionally it can be seen that once the number of nodes reaches a certain number, the mass changes very little with subsequent increases. In figure \ref{fig:massallplt}, this point is around 40 nodes. Thus the return of accuracy over run time is so small that it is not worth running the code past this point.\par

	\section{References}
	\label{sec:References}
		\bibliographystyle{ieeetran}
	\bibliography{ref}%.bib}
	\noindent This report was written by ID: 6110 in Latex. If desired the source code may be provided.\par
	%\bibliographystyle{plain}
	%\newpage
	\section{Appendix: MATLAB Code}
	\lstset{language=Matlab,%
		%basicstyle=\color{red},
		breaklines=true,%
		morekeywords={matlab2tikz},
		keywordstyle=\color{blue},%
		morekeywords=[2]{1}, keywordstyle=[2]{\color{black}},
		identifierstyle=\color{black},%
		stringstyle=\color{mylilas},
		commentstyle=\color{mygreen},%
		showstringspaces=false,%without this there will be a symbol in the places where there is a space
		numbers=left,%
		numberstyle={\tiny \color{black}},% size of the numbers
		numbersep=9pt, % this defines how far the numbers are from the text
		emph=[1]{for,end,break},emphstyle=[1]\color{red}, %some words to emphasise
		%emph=[2]{word1,word2}, emphstyle=[2]{style},
	}
	%\subsection{Master Script}
	\includepdf[pages=-]{C:/Users/Philip/Documents/GitHub/DesignOpt/Project4/html/Project_4_master.pdf}

	%\newpage
	%\subsection{Spar Class}
	\includepdf[pages=-]{C:/Users/Philip/Documents/GitHub/DesignOpt/Project4/html/Spar.pdf}
	%\lstinputlisting{project_2_master.m}
	%\newpage

	%\subsection{fmincon\_handle.m}
	\includepdf[pages=-]{C:/Users/Philip/Documents/GitHub/DesignOpt/Project4/html/fmincon_handle.pdf}
	%\lstinputlisting{project_2_master.m}
	%\newpage

	%\subsection{Make Plots}
	\includepdf[pages=-]{C:/Users/Philip/Documents/MATLAB/Fall2018/DesignOpt/Project_4/Final_Version/html/makePlots.pdf}

	\iffalse
	\lstinputlisting{project_2_master.m}

	\subsection{runFmincon.m}
	\lstinputlisting{runFmincon.m}
	%\newpage
	\subsection{stressCon.m}
	\lstinputlisting{stressCon.m}
	\newpage
	\subsection{cStep.m}
	\lstinputlisting{cStep.m}

	\subsection{cStep\_nonlinCon.m}
	\lstinputlisting{cStep_nonlinCon.m}

	\subsection{Iyy\_zmax\_from\_rLU.m}
	\lstinputlisting{Iyy_zmax_from_rLU.m}

	\subsection{create\_x\_from\_nodes\_Best.m}
	\lstinputlisting{create_x_from_nodes_Best.m}
	\subsection{create\_x\_from\_nodes.m}
	\lstinputlisting{create_x_from_nodes.m}
	\subsection{create\_r\_from\_x.m}
	\lstinputlisting{create_r_from_x.m}
	\subsection{create\_constraints.m}
	\lstinputlisting{create_constraints.m}
	\subsection{makePlots.m}
	\lstinputlisting{makePlots.m}
	\fi

%    \bibliography{ref}%.bib}
%\bibliographystyle{plain}

	% --------------------------------------------------------------
	%     You don't have to mess with anything below this line.
	% --------------------------------------------------------------

\end{document}


\documentclass[12pt]{article}
\title{Assignment 1: Sequential Program Optimization}
\author{Louis Thomas}
\date{30 January 2018}
\usepackage[letterpaper,margin=1in]{geometry}
\usepackage{setspace}
\doublespacing
\usepackage{amsmath}
\begin{document}
	\maketitle
	\section{Problem Description}
	The goal of this assignment is to take some intentionally slow code and optimize it.
	The given code (written by Ross Walker) was purposefully written to let students take the code
	and, between using compiler optimizations and hand-written ones,
	make the resulting executable process the same data at a much faster pace
	while producing the same or "same-enough" result.

	The computations performed by the computer were modeled after a fictional function:
	\[ \displaystyle\sum_{j<i}\left(
			\frac{e^{(r_{i,j}*q_i)}*e^{(r_{i,j}*q_j)}}{r_{i,j}}
			-\frac{1}{a}
			\right) \text{ if } r_{i,j} \leq cut\]
	where $i$ and $j$ are some particle,
	$r_{i,j}$ is the distance from particles $i$ and $j$,
	$cut$ is the cut-off distance between two particles,
	$q_{i}$ is the energy scalar of particle $i$, and
	$a$ is a constant.

	The specific portion of the program that can benefit from optimization is a doubly-nested for-loop. Notable functions in the loop include $exp()$ and $sqrt()$.

	\section{Approach}

	The first optimization was to reduce a number of subtractions done within array indices on each loop iteration.
	The original loop counted from one to the number of atoms,
	however, $1$ was subtracted from $i$ and $j$ each time they were used inside the loop.
	The change included starting $i$ and $j$ from zero and changing the $\leq$ to a $<$, and removing all the $-1$'s from the array indices.

	Secondly, an if statement including a check for $i<j$ was removed and instead incorporated into the inner loop.
	This cuts about half of the iterations from only checking that $(j<i)$ is false and continuing.

	Third, a call to the square root function was moved until later.
	The original loop calculated $r_{i,j}$ in its entirety each iteration.
	This meant that, even if the value isn't included in the final sum, it was still calling the square root on it.
	Instead of comparing $r_{i,j}$ to the cutoff, $r_{i,j}^2$ is compared to $cut^2$.
	This required the small cost of calculating $cut^2$,
	but it also meant that $sqrt(r_{i,j}^2)$ was pushed back until it was necessary to be calcualted.

	Fourth, many dereferences were moved out of the inner-most loop.
	This causes some dereferences to only happen $1$ or $n$ times, as opposed to up to $\frac{1}{2}n^2$ times.
	The arrays that were acted on included the $coords$ and $q$ arrays.

	Fifth, the multiplying two exponents of the same base can be simplified to one exponentiation.
	From the original formula: $e^{r_{i,j}*q_i}*e^{r_{i,j}*q_j}$ can be simplified to $e^{r_{i,j}*q_i+r_{i,j}*q_j}$, and then $e^{r_{i,j}*(q_i+q_j)}$.
	An optimization in the code looks much like the previous simplification,
	thus not only reducing the number of function calls but also cutting the number of exponentiations by half.

	A simple optimization follows from a mathematical characteristic in sums.
	Specifically, $\displaystyle\sum_{i=1}^n{i} = n*i$.
	Thus, we can avoid subtracting $^1/_a$ each iteration, and instead subtract $x*{^1/_a}$ from the sum,
	where $x$ is the number of results within the cutoff.

	Finally, a optimization involved taking the two-dimensional \texttt{coords} array and converting it into a one-dimensional array of structs.
	The hope here was to imporve cache hits and reduce misses.

	These tests were run on node granger3.

	%\section{Specifications}

	%The computer used is a Dell Optiplex 960 SFF running with 4x2 GiB of 800MHz DDR2 RAM and an Intel Core 2 Quad Q9650 (4x3.00 GHz).
	%The operating system is Xubuntu 17.10 (GNU/Linux 4.13.0-32-generic).
	%The compiler is gcc 7.2.0-8ubuntu3.
	%The hard drive is a Western Digital 250 GB SSD.

	%The program was tested on 10,000 data points generated from generate\_input.c on seed 10.

	\section{Observations}

	Some optimizations had nearly no effect on processing time, while others substantially contributed to the program's efficiency.

	\singlespacing
	\begin{center}
	\begin{tabular}{| p{3cm} | p{4cm} | p{1.5cm} | p{1.5cm} | l |}
	\hline
	Compiler flags & Optimization & Time to calculate $E$ (s) & Total Time (s) & Result \\ \hline
	None & & 2.3691 & 2.3775 & 58261662.7646147534 \\ \hline
	-O1 & & 1.0806 & 1.0893 & 58261662.7646147534 \\ \hline
	-O2 & & 0.9236 & 0.9318 & 58261662.7646147534 \\ \hline
	-O3 & & 0.9155 & 0.9232 & 58261662.7646147534 \\ \hline
	-Ofast & & 0.7045 & 0.7130 & 58261662.7646147460 \\ \hline
	-O3 & For-loops start from 0 & 0.9620 & 0.9695 & 58261662.7646147534 \\ \hline
	-O3 & Re-aligned for-loops & 0.8882 & 0.8961 & 58261662.7646147534 \\ \hline
	-O3 & Bring out dereferences & 0.8993 & 0.9082 & 58261662.7646147534 \\ \hline
	-O3 & Written-out squaring & 0.9291 & 0.9380 & 58261662.7646147534 \\ \hline
	-O3 & Combining exponents & 0.6981 & 0.7071 & 58261662.7646147460 \\ \hline
	-O3 & Comparing $r_{i,j}^2$ to $cut^2$ & 0.8461 & 0.8547 & 58261654.5481551588 \\ \hline
	-O3 & Subtract $x*{^1/_a}$ later & 1.1425 & 1.1512 & 58261662.7646149173 \\ \hline
	-O3 & Convert 2D array to 1D struct array & 0.8772 & 0.8853 & 58261662.7646147534 \\ \hline
	-O3 & All above & 0.5425 & 0.5521 & 58261654.5481553227 \\ \hline
	-O3 -ffast-math & All above & 0.5132 & 0.5223 & 58261654.5481553227 \\ \hline
	-Ofast & All above & 0.5104 & 0.5188 & 58261654.5481553227 \\ \hline
	-Ofast -fno-stack-protector & All above & 0.4971 & 0.5047 & 58261654.5481553227 \\ \hline
	\end{tabular}
	\end{center}
	\doublespacing

	\section{Analysis}

	The first notable observation is the magnitude of time difference just the first -O1 flag provides.
	Also notable is the insignificant difference between -O3 and -O2 compilation flags on the original program.
	-O3 has at best no effect, likely a slight adverse effect, compared to -O2.

	Next, -Ofast on the original program produces not only a faster program, but also a slightly different result.
	The documentation indicates that -Ofast produces code that acts in a different manner from the IEEE specification, and that fact is apparent here.

	Some optimizations resulted in a increase of time taken.
	This may be because those optimizations work better with other optimizations,
	such as in writing out the squaring as opposed to using the \texttt{pow()} method.
	There is also a difference in the produced result on the change to checking $r_{i,j}^2$ against $cut^2$.
	The best explanation is the gained resolution from not using the \texttt{sqrt()} function.
	Since \texttt{sqrt()} is normally implemented through an approximation requiring many iterations of floating-point operations,
	resolution is likely lost through the \texttt{sqrt()}.

	Removing one of the buffer overflow defenses, checking for stack smashing, seems to produce a small time advantage.
	In a practical implementation, this may or may not be a preferrable optimization
	since it cuts out two-thirds of the protections against buffer overflow strikes, but it is interesting to see that it does provide a small time reduction.
	This optimization may prove to be more effective if there were more user-written functions called to during the execution of the program.

	\section{Conclusions}

	-O1 provides the majority of the compiler-level optimizations in this situation, and marginal improvements to execution time are added by -O2 or -O3.
	It is possible that some of the adverse affects of -O3 versus -O2 are due to general optimizations that would, on average, be effective optimizations.
	-Ofast enables -ffast-math and, thus, many unsafe but faster optimizations to the program.
	In addition, combining -O2 and -ffast-math is not faster than -Ofast, despite -Ofast is defined as -O3 -ffast-math when -O2 is when than -O3.
	This is perhaps because -O3 has optimizations that work better for -ffast-math than -O2.

	In general, making the program do less computations results in less execution time, but can result in varrying mathematical results.
	Varrying optimizations can have varrying magnitudes of change in the calculated result.

	Effective optimizations the programmer can take include reducing calls to heavy math functions,
		reducing dereferences to arrays,
		and putting off calculations until later when it is mathematically sound to do so and results in less computations.

	\section{Pledge}
	On my honor, I pledge that I did not give nor receive help on this assignment.

	\vspace{1cm}

	Louis Thomas

\end{document}


\chapter{Reinforcement Learning}
\label{chp:reinforcement-learning}

% %
% HEADER
% %
The nature of learning is strictly related with the process of learning by interacting with the environment.
Learning from interactions is a foundational idea underlying nearly all theories of learning and intelligence.
In fact, experimenting the consequences of actions allows to acquire knowledge about the environment and ourselves in order to achieve goals.

In this Chapter we introduce reinforcement learning, that is a computational approach to learning from goal-driven interactions with a dynamic environment.
%
In particular, we focus on $\mathcal{Q}$-learning, that is the most representative reinforcement learning technique.
%
For a more complete and general coverage of reinforcement learning, we refer the reader to \cite{sutton1998reinforcement}, \cite{szepesvari2010algorithms} and \cite{kaelbling1996reinforcement}.

% %
% MACHINE LEARNING BACKGROND
% %
\section{Machine Learning Background}
\label{sec:reinforcement-learning-machine-learning-background}

Machine learning is the field of artificial intelligence that gives computers the ability to learn without being explicitly programmed and to automatically improve learning and behavioral performance with experience.
%
To be more precise, as stated in \cite{mitchell2006discipline}, we say that a machine $M$ learns with respect to a particular task $T$, performance metric $P$, and type of experience $E$, if $M$ reliably improves its performance $P$ at task $T$, following experience $E$.

We do not want to provide an exhaustive coverage of the machine learning field, but giving the necessary background to better contextualize reinforcement learning as an attractive machine learning paradigm.
%
For a complete overview of machine learning, we refer the reader to \cite{samuel2000some}.

All machine learning tasks can be reduced to an archetypal machine learning task consisting of a learning agent (agent, for short) with the goal of inferring some functional relationship $F$ between inputs $i\in I$ and outputs $o\in O$.
%
We say that an input is \textit{labeled} if it is presented to the agent as a pair $(i,o).F(i)=o$, otherwise it is \textit{unlabeled}.
Furthermore, we say that an input is \textit{statically labeled} if $(i,o_{1}) \wedge (i,o_{2}) \Rightarrow o_{1}=o_{2}$, otherwise it is \textit{dynamically labeled}.

Machine learning tasks are traditionally classified into four broad categories: \textit{supervised learning}, \textit{unsupervised learning}, \textit{active learning} and \textit{reinforcement learning}.
%
Notice that in literature, both active and reinforcement learning are sometimes classified as special form of semi-supervised learning.
%
\textit{Supervised learning} consists of a learning agent that is presented with statically labeled inputs. The goal is to generalize the functional relationship so as to act correctly in situations not presented in the training set.
%
\textit{Unsupervised learning} consists of a learning agent that is presented with unlabeled inputs. The goal is to find patterns hidden in collections of unlabeled data.
%
\textit{Semi-supervised learning} consists of a learning agent that is presented with a minority of statically labeled inputs and a majority of unlabeled inputs.
%
\textit{Active learning} consists of a learning agent that queries the environment to obtain statically labeled inputs.
%
\textit{Reinforcement learning} consists of a learning agent that modifies the environment to obtain dynamically labeled inputs as a result of trial-and-error interaction with the environment.

In this work we focus on reinforcement learning because
(i) it learns from the environment by modifying it and
(ii) it can benefit from prior knowledge about the environment, without assuming it.
%
All these aspects make reinforcement learning suitable for the realization of smart elasticity, that is elastic resource management leveraging machine learning.
%
Furthermore, other aspects that make reinforcement learning really exciting are that
(i) it fruitfully interacts with other scientific disciplines, such as statistics, optimization, psychology and neuroscience,
(ii) that among all the forms of machine learning, it is the most closely inspired by biological learning systems, e.g. humans and animals, and
(iii) it leads the trend of artificial intelligence focused on learning driven by simple general principles

% %
% Reinforcement Learning Problem
% %
\section{Reinforcement Learning Problem}
\label{sec:reinforcement-learning-reinforcement-learning-problem}
Reinforcement learning is a machine learning paradigm focused on learning the best decision-making strategy to control a system from goal-driven interactions with an environment.
%
The reinforcement learning problem consists of an \textit{agent} determining the \textit{policy} to control a \textit{system} so as to maximize a the \textit{long-run value} of its behavior, through \textit{trial-and-error interactions} with the \textit{environment}.
%
In the following, we assume that the system to control and the environment to sense are the same, i.e. the agent learns how to control the system interacting with it.

The notions of \textit{trial-and-error interaction} and \textit{long-run value} are really important in reinforcement learning.
%
The concept of \textit{trial-and-error interaction} consists in the fact that each action taken by the agent on the system are immediately judged to be good or bad, accordingly to some rewarding mechanism.
%
The concept of \textit{long-run value} consists in the fact that the agent learns from delayed rewards, that is it may take a long sequence of actions, receiving insignificant reward, then finally arrive at a state with high reward.
%
Hence the agent must be able to learn which actions are desirable based on reward that can take place arbitrarily far in the future.

The standard reinforcement learning model is shown in Figure \ref{fig:reinforcement-learning-model}.
%
The environment communicates its own state $x$, from which the agent calculates
(i) the perceived state $\tilde{x} = S(x)$, that is the portion of the state that the agent is interested in, and
(ii) the reward $r=R(x)$, that measures how good has been the last state transition, hence how good has been the last action taken.
%
The agent calculates the action $a=\pi(x)$ to be executed on the environment; the latter transits to a new state, in response to the taken action; and the cycle is repeated.
%
The agent's objective is to determine the policy that maximizes the \textit{total reward} achieved in the long-run.
%
In the following we assume that $S$ is the identity function, because the agent is interested in the environment state as a whole; hence we omit the $\tilde{x}$ notation and we will denote with $x$ the state perceived by the agent.
%The learning problems differ in the details of how the data is collected and how performance is measured.

\begin{figure}
	\label{fig:reinforcement-learning-model}
	\centering
	\includegraphics[width=.7\columnwidth]{reinforcement-learning-model}
	\caption{The standard reinforcement learning model.}
\end{figure}

The most important distinguishing characteristics of reinforcement learning are that
(i) it is a closed-loop problem\footnote{a \textit{closed-loop problem} is a problem that is solved iteratively and each iteration affects the input on the next one.},
(ii) it does not have wired instructions to decide a priori what actions to take in response to state transitions,
(iii) it does not rely on supervision nor on a complete model of the environment and
(iv) consequences of actions are perceived as to play out over extended time periods.

Reinforcement learning is of great interest because of the large number of practical applications that it can be used to address, ranging from problems in artificial intelligence to operations research or control engineering. In Section \ref{sec:reinforcement-learning-applications} we review a few of the most representative applications.

A reinforcement learning model is made of an \textit{environment}, an \textit{agent}, a \textit{policy}, a \textit{reward signal}, a \textit{value function} and an optional \textit{model} of the environment.

The \textit{environment (or controlled system)} is a dynamic, non-deterministic and stationary finite-state automaton.
It is a finite state automaton in the sense that at any given time it is fully characterized by its \textit{state}.
It is dynamic in the sense that it changes its own state in response to external stimulus.
It is non-deterministic in the sense that the same action taken on the same state in different instants may not produce the same state transition.
It is stationary in the sense that the probabilities of making state transitions do not change over time.

The \textit{agent} senses the state of the environment and takes \textit{actions} on the environment. The agent must be able to
(i) sense the environment state,
(ii) take actions that affect it,
(iii) chose actions driven by goals related to the environment state, and
(iv) use its experience to improve its performance over time.
%
At the same time, the agent cannot predict the effect of its actions on the environment; thus it has to monitor the environment frequently.

The \textit{reward signal (or reinforcement signal)} defines the short run convenience of taking a specific action in a given state.
Every time the agent takes an action on the environment, the latter changes its state and sends to the agent a reward representing the convenience of the action.
The reward signal thus defines what are the good and bad events for the agent.
Typically, it is realized by a stochastic function of the environment state and the actions taken.
If an action selected by the policy is followed by low reward, then the policy may be changed to select some other action in that situation in the future.
The agent objective is to maximize the total reward it receives over the long run.
The agent cannot alter directly the process that generates the reward signal, but can do it indirectly by changing the state of the environment through actions.

The \textit{value function} defines the long run convenience of a state, that is the total amount of reward an agent can expect to accumulate over the future, starting from that state.
To better understand the difference between rewards and values, notice that a state might always yield a low reward but still have a high value because it is regularly followed by other states that yield high rewards; and vice versa.
The goal of the agent is to determine a policy that maximize values, hence action choices rely on judgments about values rather than rewards.
However, there could be no values without rewards.
Determining values are much harder than rewards because the former must be continuously re-estimated from observations over time, whereas the latter are given given directly and immediately by the environment.
The value function is arguably the most important component in a reinforcement learning algorithm because an effective value function can make search in the space of policies more efficient.
Furthermore, the use of value function to discriminate policies distinguishes reinforcement learning from evolutionary methods that search directly in policy space driven by scalar evaluation of entire policies.

The \textit{policy} defines how the agent behaves in response to state changes of the environment.
Basically, it is a mapping from perceived states of the environment to actions to be taken when in those states.
It is typically realized by a simple stochastic function, whereas in some case it may involve extensive computation, such as a search process.
An \textit{optimal policy} is the one that maximizes the long-run reward.

The \textit{model of the environment} defines the behavior of the environment, allowing to infer, given a state and an action, the resultant next state and next reward.
Models are used to consider possible future situations before they are actually experienced. For example, a model can be used by an agent to exclude a priori some actions with respect to others in a given state.
The main advantage of using models is the reduction of the exploration cost.
Reinforcement learning algorithms that leverage models are called \textit{model-based methods}, as opposed to simpler \textit{model-free methods} that are pure trial-and-error learners.

The comprehension of the difference between rewards and values is really important.
In literature, it is sometimes explained with analogies from the human domain, where the reward signal is somewhat like pleasure (if high) and pain (if low), whereas values correspond to a more refined and farsighted judgment of how pleased or displeased we are that our environment is in a particular state.

% %
% REINFORCEMENT LEARNING ALGORITHM
% %
\section{Reinforcement Learning Algorithm}
\label{sec:reinforcement-learning-algorith}
There are plenty of exact and heuristic techniques to solve the reinforcement learning problem.
Although they vary a lot, they share a common algorithmic structure, shown as pseudo-code in Algorithm \ref{alg:reinforcement-learning-general}.

\begin{algorithm}[t]
	\label{alg:reinforcement-learning-general}
	\SetKwInOut{Input}{input}
	\SetKwInOut{Input}{input}
	\SetKwInOut{Output}{output}

	\Input{state space $\mathcal{X}$, action space $\mathcal{A}$}

	\Output{none}

	initialize $V[X,A]$ \\
	observe current state $x$ \\
	\While{stop criterion $=$ False}{
		$a \leftarrow argmax_{a\in\mathcal{A}} V[x,a]$ \\
		carry out action $a$ \\
		observe new state $x'$ \\
		observe reward $r$ \\
		update $V[x,a]$ using $x'$ and $r$ \\
		$x \leftarrow x'$ \\
	}

	\caption{Pseudocode of the general reinforcement learning algorithm.}
\end{algorithm}

%Notice that, the set of admissible actions may differ from state to state. Given a state $x\in\mathcal{X}$, we denote with $\mathcal{A}_{x}\subseteq\mathcal{A}$ the subset of admissible actions for the state $x$.

\section{Experience and history}
\label{sec:reinforcement-learning-experience-history}
An \textit{experience} $e_{n}$ of an agent at step $n$ is a tuple

$$
e_{n}:=(x_{n},a_{n},r_{n},y_{n})
$$

which means that at $n$-th step the environment was in state $x_{n}$, then the agent took the action $a_{n}$ which receives a reward $r_{n}$ and makes a state transition to state $y_{n}$, that is equal to the start state $x_{n+1}$ in the next experience $e_{n+1}$.

The \textit{history} of an agent is a time-ordered sequence of experiences $h=(e_{n})_{}n\geq0$, with an obvious meaning.
%
In literature, a history is often denoted as

$$
h:=(x_{0},a_{0},r_{0},x_{1},a_{1},r_{1},x_{2},a_{2},r_{2},x_{3}, ...)
$$

where the item $y_{n}\in e_{n}$ has been elided for sake of brevity.

% %
% EXPLORATION AND EXPLOITATION
% %
\section{Exploration and Exploitation}
\label{sec:reinforcement-learning-exploration-exploitation}
As an agent must try a variety of actions and progressively favor those that appear to be best, one of the most important challenges that arise is the \textit{trade-off between exploration and exploitation}.
%
This trade-off is one of the most distinguishing challenges in reinforcement learning.

The \textit{exploration} is the act of experimenting actions with unknown reward; whereas the \textit{exploitation} is the act of applying the only actions with a reward that is known to be high.
Clearly, an agent prefers actions known to be effective in producing reward (exploitation) so as to maximize the long-run reward.
However, in order to discover such actions, it has to try not yet selected ones (exploration).
Furthermore, from a stochastic point of view, the agent must try actions many times to reliably estimate its expected reward.
If an agent performs too much exploration, it may not spend enough time in exploiting to advantage what is has learned.
Conversely, if an agent does too little exploration, it may miss discovering some alternative behaviors that would bring much higher returns, and it may spend all its time exploiting an initial mediocre strategy.
A good balance between exploration and exploitation provides the agent with a gradual improvement of its skills.
As both exploration and exploitation may induce task failures, this trade-off should be addressed taking into account the cost of mistakes during the learning process.

The well-known \textit{Multi-Armed Bandit Problem} in Section \ref{sec:reinforcement-learning-examples}, is often used in literature to better explain the exploration-exploitation trade-off.

% %
% EXAMPLES
% %
\section{Examples}
\label{sec:reinforcement-learning-examples}
Reinforcement learning is better understood when considering some examples and applications the lead its development.
In the following we present the most notable examples by which reinforcement learning has been presented in literature.

The \textit{Multi-Armed Bandit Problem}, discussed in \cite{Auer2002}, consists in an agent that is in a room with a collection of $k$ gambling machines\footnote{gambling machines are called \textit{"armed bandit"} in colloquial English, hence the name of the problem.}. The agent is permitted a fixed number of pulls, $h$. Any machine may be pulled on each turn. The machines do no require a deposit to play; the only cost is in wasting a pull playing with a suboptimal machine. When arm $i$ is pulled, machine $i$ pays off $1$ or $0$, according to some underlying probability $p$. Payoffs are independent and the probabilities $p$ are unknown.
%
The agent might believe that a particular arm has a fairly high payoff probability. Should it choose that arm al the time, or should it choose another one with uncertain payoff? Answers to this question depends on how long the agent is expected to play the game. The longer the game lasts, the worse the consequences of prematurely converging on a suboptimal arm, and the more the agent should explore.
%
There is a wide variety of solutions to this problem. We refer the reader to \cite{Auer2002} for a deeper discussion and theoretical results.

The \textit{Pole-Balancing Problem}, discussed in \cite{barto1983neuronlike}, consists in an agent ...

%The \textit{Tic-Tac-Toe Problem}, discussed in \cite{sutton1998reinforcement}, consists in an agent ...

% %
% THEORETICAL FOUNDATIONS
% %
\section{Theoretical Foundations}
\label{sec:reinforcement-learning-theoretical foundations}
Reinforcement learning uses a formal framework defining the interaction between and agent and its environment in terms of states, actions and rewards.
The reinforcement learning problem is best described in the framework of \textit{Markovian Decision Processes (MDP)}, introduced in \cite{bellman1957markovian}.
In fact, it can be fully specified as an optimal control problem over a MDP.
The standard approach to solve a MDP is to use \textit{Dynamic Programming (DP)}, introduced in \cite{bertsekas1995dynamic}, which transforms the problem of determining a good policy finding a good value function.
However, since DP is infeasible to solve MDP with a lot of states and actions and without prior knowledge about the environment, reinforcement learning algorithms extends it so to be applied in large-scale problems that do not rely on a prior model of the environment.
In the following, we restrict our attention to countable MDPs, that are MDP with finite state-set and action-set.
However, under some technical conditions, the results extend to continuous MDPs.

\subsection{Markovian Decision Processes}
\label{Reinforcement-learning-markovian-decision-processes}
Insert here MDP description.
UTILE DA INSERIRE PER COMPRENDERE IL BACKGROUND TEORICO, MA SUPERFLUO PER COMPRENDERE IL FUNZIONAMENTO DEL Q-LEARNING

\subsection{Dynamic Programming}
\label{Reinforcement-learning-dynamic-programming}
Insert here DP description.
UTILE DA INSERIRE PER COMPRENDERE IL BACKGROUND TEORICO, MA SUPERFLUO PER COMPRENDERE IL FUNZIONAMENTO DEL Q-LEARNING

% %
% METRICS
% %
%\section{Metrics}
%\label{sec:reinforcement-learning-metrics}
%CHIEDERE A PROF. R. BASILI PER LISTA COMPLETA DI METRICHE
%Given a fully characterized reinforcement learning system, the learning performance of the agent must be measured.
%The most widely adopted metrics are \textit{eventual convergence}, \textit{speed of convergence} and \textit{regret}.

%\textit{Eventual convergence} measures ...

%\textit{Speed of convergence} measures ...

%\textit{Regret} measures ...

% %
% Q-LEARNING
% %
\section{$\mathcal{Q}$-learning}
\label{sec:reinforcement-learning-q-learning}

$\mathcal{Q}$-learning is a form of model-free reinforcement learning, defined in \cite{watkins1989learning}.
%
%It can also be viewed as a method of asynchronous and incremental Dynamic Programming because of the step-by-step manner in which it determines the optimal policy.
% $\mathcal{Q}$-learning is a method to solve discounted infinite-horizon countable MDP.
%
\cite{watkins1989learning} and \cite{watkins1992q} should be consulted for a more extensive discussion of $\mathcal{Q}$-Learning, its relationship with Dynamic Programming and Temporal-Differences and convergence results.

In $\mathcal{Q}$-Learning, the agent's experience consists of a sequence of distinct stages or \textit{episodes}.
%
It works by successively improving its evaluations of the quality of particular actions at particular states.
%
In the $n$-th episode, the agent
\begin{enumerate}
	\item observes its current state $x_{n}\in\mathcal{X}$
	\item selects and performs an action $a_{n}\in\mathcal{A}$,
	\item observes the subsequent state $y_{n}\in\mathcal{X}$,
	\item receives an immediate reward $r_{n}$ and
	\item updates $\mathcal{Q}_{n}(x,a)$ taking into account $\mathcal{Q}_{n-1}$ values, the learning factor $\alpha_{n}$ and the discount factor $\gamma$, as states in the following Equation \ref{eqn:reinforcement-learning-q-learning-update}.
\end{enumerate}

\begin{equation}
\label{eqn:reinforcement-learning-q-learning-update}
\mathcal{Q}_{n}(x,a) =
\begin{cases}
(1-\alpha_{n})\mathcal{Q}_{n-1}(x,a) + \alpha_{n}(r_{n} + \gamma\max_{b} \mathcal{Q}_{n-1}(x_{n+1},b)) & \text{if } x=x_{n},a=a_{n} \\
\mathcal{Q}_{n-1}(x,a)       & \text{otherwise}           \\
\end{cases}
\end{equation}

where
(i) the learning rate $\alpha_{n}$ trades off the importance of sooner versus later $\mathcal{Q}$ values,
(ii) the discount factor $\gamma$ trades off the importance of sooner versus later rewards and
(iii) the term $\max_{b} \mathcal{Q}_{n-1}(x_{n+1},b)$ is the highest value of the state $x_{n+1}$ perceived by the agent before transit to it, accordingly to the current policy. This term is discounted by $\gamma$ because rewards received $s$ steps hence are worth less than rewards received immediately, by a factor $\gamma^{s}$.

Notice that Equation \ref{eqn:reinforcement-learning-q-learning-update} assumes
(i) a lookup table\footnote{\cite{watkins1989learning} shows that $\mathcal{Q}$-Learning may not converge correctly for different representations.} for the representation of $\mathcal{Q}_{n}(x,a)$ and that
(ii) $\mathcal{Q}_{0}(x,a)$ is given by some initialization step, for all states and actions.

% %
% MEANING OF THE UPDATE
% %
\subsection{Meaning of the update}
\label{sec:reinforcement-learning-meaning-of-the-update}
In this Section we consider how the Equation \ref{eqn:reinforcement-learning-q-learning-update} has been constructed to better understand its meaning.

Let us consider a standard reinforcement learning problem.
%
In the $n$-th episode, the agent receives the state $x_{n}\in\mathcal{X}$ from the environment and chooses to take the action $a_{n}\in\mathcal{A}$, accordingly to the policy $\pi$, that is $a_{n}=\pi(x_{n})$.
%
As the environment is modeled as a non-deterministic stationary dynamic finite-state automaton, the action $a_{n}$ in state $x_{n}$ produces
(i) a transition to state $y_{n}\in\mathcal{X}$ with probability $P_{x_{n},y_{n}}(a_{n}):=P(y_{n}=y|x_{n},a_{n})$
(ii) a probabilistic reward $r_{n}$, whose mean value $R_{x_{n}}(a_{n})$ depends only on the state and action.

Under a policy $\pi$, the value of state $x$ is defined as

\begin{equation}
\label{eqn:reinforcement-learning-value-state}
V^{\pi}(x) := R_{x}(\pi(x)) + \gamma \sum_{y} P_{x,y}(\pi(x))V^{\pi}(y)
\end{equation}

Equation \ref{eqn:reinforcement-learning-value-state} means that the value of a state $x$ is made of
(i) the \textit{immediate reward} $R_{x}(\pi(x))$ for performing the action indicate by the policy,
(ii) the overall \textit{future reward} of moving with probability $P_{x,y}(\pi(x))$ to a state $y$ whose value is $V^{\pi}(y)$, and
(iii) the discount factor $\gamma$ that discounts future reward with respect to immediate reward.

The agent's objective is to determine the \textit{optimal policy} $\pi^{*}$ as a solution to the following optimization problem

\begin{equation}
\label{eqn:reinforcement-learning-optimization-problem}
\pi^{*} = argmax_{\pi} V^{\pi}(x)\qquad\forall x\in\mathcal{X}
\end{equation}

As stated in \cite{bellman2015applied,ross2014introduction}, Dynamic Programming guarantees that there exists at least one optimal solution $\pi^{*}$ to the problem in Equation \ref{eqn:reinforcement-learning-optimization-problem}, such that

$$
V^{*}(x) \equiv V^{\pi^{*}}(x) = \max_{a\in \mathcal{A}} (R_{x}(a) + \gamma \sum_{y} P_{x,y}(a)V^{\pi^{*}}(y))
$$

The problem is that Dynamic Programming assumes that $R_{x}(a)$ and $P_{x,y}(a)$ are known.
%
Unfortunately, this assumption cannot hold in a reinforcement learning problem as it is equivalent to assume a complete knowledge of the environment.

Here is where $\mathcal{Q}$-learning comes at hand, as it determines $\pi^{*}$ without knowing neither $R_{x}(a)$ nor $P_{x,y}(a)$.
Let us see how to do it.
%
For a policy $\pi$, it defines the so called $\mathcal{Q}$-values (also named action-vales)

\begin{equation}
\label{eqn:reinforcement-learning-q-value}
\mathcal{Q}^{\pi}(x,a) = R_{x}(a) + \gamma \sum_{y} P_{x,y}(\pi(x))V^{\pi}(y)
\end{equation}

The $\mathcal{Q}$-value is the expected discounted reward for executing action $a$ at state $x$ and following the policy $\pi$ thereafter.
The goal of $\mathcal{Q}$-learning is to estimate $\mathcal{Q}$-values for optimal policy $\pi^{*}$, that is:

\begin{equation}
\mathcal{Q}^{*}(x,a) \equiv \mathcal{Q}^{\pi^{*}}(x,a) \qquad \forall x\in\mathcal{X},a\in\mathcal{A}
\end{equation}

As a consequence we have that:

\begin{equation}
V^{*}(x) = \max_{a\in\mathcal{A}} \mathcal{Q}^{*}(x,a) \qquad \forall x\in\mathcal{X},a\in\mathcal{A}
\end{equation}

Finally:

\begin{equation}
\pi^{*}(x) = a^{*} = argmax_{a\in\mathcal{A}} \mathcal{Q}^{*}(x,a) \qquad \forall x\in\mathcal{X}
\end{equation}

Herein lies the utility of $\mathcal{Q}$-values: If an agent can learn them, it can easily decide what it is the optimal action to execute in any given state.
Notice that, although there could be more than one optimal policy, $\mathcal{Q}^{*}$ values are unique.

When the $\mathcal{Q}$-values are nearly converged to their optimal values, it is appropriate for the agent to act greedily, taking in each episode the action with the maximum $\mathcal{Q}$-value.
%
During learning, the exploration-exploitation trade-off must be addressed, thus considering to explore unknown actions.
%
It is important to notice that there is not an exploration strategy that is valid for all situations: ad-hoc heuristics must be taken into account.
%
However, $\mathcal{Q}$-learning has been shown to be \textit{exploration insensitive} in \cite{watkins1992q}. This means that it converges to optimality regardless of the balancing between exploration and exploitation. For this reason $\mathcal{Q}$-learning rapidly became the most popular algorithm for model-free reinforcement learning.

% %
% THE Q-LEARNING ALGORITHM
% %
\subsection{The $\mathcal{Q}$-learning Algorithm}
\label{sec:reinforcement-learning-q-learning-algorithm}
Algorithm\ref{alg:reinforcement-learning-q-learning} show the pseudo-code of the $\mathcal{Q}$-Learning algorithm.

\begin{algorithm}[t]
	\label{alg:reinforcement-learning-q-learning}
	\SetKwInOut{Input}{input}
	\SetKwInOut{Output}{output}

	\Input{state space $\mathcal{X}$,action space $\mathcal{A}$, learning rate $\alpha$, discount factor $\gamma$}

	\Output{none}

	initialize $\mathcal{Q}[X,A]$ arbitrarily \\
	observe current state $s$ \\
	\While{stop criterion $=$ False}{
		$a \leftarrow argmax_{a\in\mathcal{A}} \mathcal{Q}[x,a]$ \\
		carry out action $a$ \\
		observe new state $x'$ \\
		observe reward $r$ \\
		$\mathcal{Q}[x,a] \leftarrow (1-\alpha) \mathcal{Q}[x,a] + \alpha(r + \gamma\max_{b} \mathcal{Q}[x',b])$ \\
		$x \leftarrow x'$ \\
	}

	\caption{Pseudocode of the $\mathcal{Q}$-Learning algorithm.}
\end{algorithm}

% %
% CONVERGENCE
% %
\subsection{Convergence}
\label{sec:reinforcement-learning-convergence}
It has been proven in \cite{watkins1992q} that, regardless of the way $\mathcal{Q}$ is initialized and the policy being followed, the learned $\mathcal{Q}$ function converges with probability 1 to optimality $\mathcal{Q}*$, under the condition that every state-action pair continues to be sampled.
%
For reader's sake, we report here the convergence theorem, referring the reader to \cite{watkins1992q} for the demonstration.

\begin{theorem}[Convergence of $Q$-Learning]
\label{thm:reinforcement-learning-q-learning-convergence}
	Given bounded reward $|r_{n}|\leq \mathcal{R}$, learning rate $\alpha_{n}\in[0,1)$ and

	$$
	\sum_{i=1}^{\infty} \alpha_{n^{i}(x,a)}=\infty,\sum_{i=1}^{\infty}(\alpha_{n^{i}(x,a)})^{2} < \infty \forall x\in\mathcal{X},a\in\mathcal{A}
	$$

	then $\lim_{n\rightarrow\infty} Q_{n}(x,a)=Q_{*}(x,a)\forall x\in\mathcal{X},a\in\mathcal{A}$ with probability 1.

	\begin{proof}
		Refer to \cite{watkins1992q}.
	\end{proof}

\end{theorem}

% %
% OTHER TECHNIQUES
% %
%\section{Other Techniques}
%\label{reinforcement-learning-other-techniques}

%In order to integrate the partial knowledge of the system into a learning algorithm, we rely on the post-decision state (PDS) concept, exploiting the generalized definition given in [10]. A PDS (also known as afterstate) describes the state of the system after the known dynamics take place, but before the unknown dynamics take place. where s  ̃ i fully reflects the consequences of the action a i , and the next state s i+1 incorporates the unknown system dynamics (i.e., the input rate variation).
%We exploit the PDS concept to design a learning algorithm that aims at finding an optimal policy in less time than Q-Learning. To this end, we adapt the algorithm proposed in [10] to our problem. We integrate that solution into the generic Algorithm 1 by extending the update phase. In particular, the Q function has only to deal with the known system dynamics, since the unknown parts are hidden by the PDS, for which we introduce a PDS value function V that is updated along with Q.

%It is worth noting that, since the unknown system dynamics do not depend on the selected action, randomized exploration is not required any more, and a greedy policy can be followed during the learning phase.

% %
% HISTORY
% %
\section{History}
\label{sec:reinforcement-learning-history}
Since the late 1960's, many researchers presumed that intelligence is due to possession of special purpose heuristics and cannot rely on general principles. Methods based on general principles were characterized as \textit{weak methods}, whereas those based on specific knowledge were called \textit{strong methods}. Thanks to reinforcement learning, this trend is less dominant.

The history of reinforcement learning has three main threads: \textit{trial-and-error learning} in the psychology of animal learning, \textit{optimal control} and \textit{temporal difference}.
%One thread concerns learning by trial and error that started in the psychology of animal learning.
%This thread runs through some of the earliest work in artificial intelligence and led to the revival of reinforcement learning in the early 1980s.
%The other thread concerns the problem of optimal control and its solution using value functions and dynamic programming.
%For the most part, this thread did not involve learning.
All three threads came together in the late 1980s to produce the modern field of reinforcement learning.

The first to express the essence of trial-and-error learning as a
principle of learning was Edward Thorndike in \cite{thorndike1911animal}.
%
The individual most responsible for the application of the trial-and-error learning within artificial intelligence was Harry Klopf, who recognized in \cite{klopf1972brain} that essential aspects of adaptive behavior were being lost as learning researchers came to focus almost exclusively on supervised learning.
%
In the 1960's the term \textit{reinforcement learning} was used in literature for the first time to describe engineering uses of trial-and-error learning \cite{waltz1965heuristic}.
%
Reinforcement learning has been neglected in artificial intelligence since C. Watkins published his PhD thesis (\cite{watkins1989learning}), where he defined the $\mathcal{Q}$-learning technique.
%
Finally, the success of Gerry Tesauro's backgammon playing program leveraging reinforcement learning, described in \cite{tesauro1995temporal}, brought great attention to the field in 1992.

% %
% SOFTWARE
% %
\section{Software}
\label{sec:reinforcement-learning-software}
The most notable software resources for the development and testing of reinforcement learning algorithms are \textit{Tensorflow}, \textit{Theano}, and \textit{OpenAI Gym}.

\textit{Tensorflow} is an open source Python library for machine learning, developed by Google both for research and production.
%
It allows to define numerical computation using data flow graphs, where nodes represent mathematical operations, while edges represent the multidimensional data arrays (named tensors) communicated between them.
%
It can be run both on CPU and GPU.
%
It was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well.
%
It is used by a lot of leading companies, such as Intel, Nvidia, AirBnb, Uber and SAP to power their machine learning tasks.
%
It exposes API for other mainstream languages such as Java, C and Go.

\textit{Theano} is an open source Python library for numerical computation, developed by the Montreal Institute for Learning Algorithms (MILA) group at University of Montreal.
%
It can be run both on CPU and GPU.
%
It has been originally developed for deep learning.

\textit{OpenAI Gym}, presented in \cite{openaigym}, is a Python toolkit for developing and comparing reinforcement learning algorithms, developed by OpenAI.
%
It consists of two parts:
(i) the gym open-source library, that is a collection of test problems that can be used to work out reinforcement learning algorithms without making any assumption on agent's structure and the numerical computation library adopted, and
(ii) the OpenAI Gym service, that is a web service allowing people to meaningfully compare performance of their trained agents.

% %
% APPLICATIONS
% %
\section{Applications}
\label{sec:reinforcement-learning-applications}
There are a lot of successful applications of reinforcement learning.
The most notable are focused on \textit{robotics}, \textit{games}, \textit{operation research} and \textit{economics}.

In the field of \textit{robotics} we have: controlling quadrupedales (\cite{kohl2004policy}), humanoid robots (\cite{peters2003reinforcement}), or helicopters (\cite{abbeel2007application}).
%
In the field of \textit{games} we have: Backgammon (\cite{tesauro1995temporal}) and Go (\cite{gelly2007combininge}).
%
In the field of \textit{operation research} we have: vehicle routing (\cite{proper2006scaling}), targeted marketing (\cite{abe2004cross}), inventory control
(\cite{chang2007recursive})
%
In the field of \textit{economics} we have: option pricing (\cite{tsitsiklis2001regression}).

For a more complete and general coverage of reinforcement learning applications, we refer the reader to \cite{rlApplicationsUalberta}, \cite{rlApplicationsPbworks} and \cite{rlApplicationsAikorea}.


\appendix
\chapter{Generic Programming}
\section{Instances Functor for Pattern Functors}
\label{app-inst-functor-patfun}
\begin{minted}{haskell}
instance Functor I where
  fmap f (I r) = I (f r)

instance Functor (K a) where
  fmap _ (K a) = K a

instance Functor U where
  fmap _ U = U

instance (Functor f, Functor g) => Functor (f :+: g) where
  fmap f (L x) = L (fmap f x)
  fmap f (R y) = R (fmap f y)

instance (Functor f, Functor g) => Functor (f :*: g) where
  fmap f (x :*: y) = fmap f x :*: fmap f y
\end{minted}


\hypertarget{collective-communication-in-distributed-memory-systems}{%
\section{Collective Communication in Distributed-Memory
Systems}\label{collective-communication-in-distributed-memory-systems}}

\begin{itemize}
\tightlist
\item
  designing parallel algorithms on a distributed-memory system requires
  data exchange between processes (nodes)
\item
  this exchange can significantly impact the efficiency because of
  interaction delays
\end{itemize}

\hypertarget{communication-costs}{%
\subsection{Communication Costs}\label{communication-costs}}

\begin{itemize}
\tightlist
\item
  Overhead in parallel programs

  \begin{itemize}
  \tightlist
  \item
    idling
  \item
    contention
  \item
    communication
  \end{itemize}
\item
  Communication costs depend on

  \begin{itemize}
  \tightlist
  \item
    communication model
  \item
    the network topology
  \item
    data handling and routing (e.g.~packet routing, cut-through routing)
  \item
    associated software protocols
  \end{itemize}
\end{itemize}

The message passing costs is the total time to transfer a message over a
network and is comprised of the following:

\begin{itemize}
\tightlist
\item
  Startup time ($t_s$)

  \begin{itemize}
  \tightlist
  \item
    executing the routing algorithm, programming routers, etc.
  \end{itemize}
\item
  Per-hop time ($t_h$)
\item
  Per-word transfer time ($t_w$)
\item
  $t_h$ is typically smaller than $t_s$ and $t_w$ , so the second term does
  not show, when $m$ is large
\item
  furthermore, it is often not possible to control routing and placement
  of tasks
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/communicationCosts.png}
\caption{Communication costs}
\end{figure}

\begin{tcolorbox}[colback=red!5!white,colframe=red!75!black]
The simplified cost model: $t_{comm} = t_s + m * t_w$
\end{tcolorbox}

\hypertarget{introduction-to-collective-communication}{%
\subsection{Introduction to Collective
Communication}\label{introduction-to-collective-communication}}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/globalCommunication.png}
\caption{Overview Global Communication}
\end{figure}

\hypertarget{one-to-all-broadcast-and-all-to-one-reduction}{%
\subsection{One-to-All Broadcast and All-to-One
Reduction}\label{one-to-all-broadcast-and-all-to-one-reduction}}

\begin{itemize}
\tightlist
\item
  One-to-All Broadcast

  \begin{itemize}
  \tightlist
  \item
    one processor has a message M (of size m) it needs to send to
    everyone
  \end{itemize}
\item
  All-to-One Reduction

  \begin{itemize}
  \tightlist
  \item
    the dual of one-to-all broadcast
  \item
    each processor has m units of data
  \item
    these data items must be combined piece-wise (using some associative
    operator, such as addition or min for example)
  \item
    the result made available at a target node
  \end{itemize}
\end{itemize}

\hypertarget{one-to-all-broadcast-on-a-ring}{%
\subsubsection{One-to-All Broadcast on a
Ring}\label{one-to-all-broadcast-on-a-ring}}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/one-to-all-broadcast.png}
\caption{One-to-All Broadcast}
\end{figure}

\begin{itemize}
\tightlist
\item
  node 0 is the source of the broadcast
\item
  each message transfer step is shown by an arrow from the source of the
  message to its destination
\item
  the number on an arrow indicates the time step during which the
  message is transferred
\end{itemize}

\hypertarget{all-to-one-reduction-on-a-ring}{%
\subsubsection{All-to-One Reduction on a
Ring}\label{all-to-one-reduction-on-a-ring}}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/all-to-one-reduction-ring.png}
\caption{All-to-One Reduction}
\end{figure}

\hypertarget{broadcast-and-reduction-algorithms}{%
\subsubsection{Broadcast and Reduction
Algorithms}\label{broadcast-and-reduction-algorithms}}

\begin{itemize}
\tightlist
\item
  These algorithms can also be used in a mash of p nodes

  \begin{itemize}
  \tightlist
  \item
    the first step does the operation along a row
  \item
    the second step along each column concurrently
  \end{itemize}
\item
  this process generalizes to higher dimensions as well

  \begin{itemize}
  \tightlist
  \item
    e.g.~a hypercube with $2^d$ nodes can be regarded as a d-D mesh with
    two nodes in each dimension
  \end{itemize}
\end{itemize}

\hypertarget{all-to-all-broadcast-and-reduction}{%
\subsection{All-to-All Broadcast and
Reduction}\label{all-to-all-broadcast-and-reduction}}

\begin{itemize}
\tightlist
\item
  Broadcast

  \begin{itemize}
  \tightlist
  \item
    Generalization of broadcast in which each node is the source as well
    as the destination
  \item
    A node sends the same m-word message to every other node, but
    different nodes may broadcast different messages
  \end{itemize}
\item
  Reduction

  \begin{itemize}
  \tightlist
  \item
    every node is the destination of an all-to-one reduction
  \item
    each node starts with p messages
  \end{itemize}
\end{itemize}

\hypertarget{all-to-all-broadcast-in-a-ring}{%
\subsubsection{All-to-All Broadcast in a
Ring}\label{all-to-all-broadcast-in-a-ring}}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/all-to-all-broadcast.png}
\caption{All-to-All Broadcast Ring}
\end{figure}

\begin{itemize}
\tightlist
\item
  each node first sends to one of its neighbors the data it needs to
  broadcast
\item
  in subsequent steps, it forwards the data received from one of its
  neighbors to its other neighbor
\item
  the algorithm terminates in p-1 steps
\item
  \textbf{Reduction} can be performed in an identical fashion by
  inverting the process
\end{itemize}

\hypertarget{all-to-all-broadcast-on-a-mesh}{%
\subsubsection{All-to-All Broadcast on a
Mesh}\label{all-to-all-broadcast-on-a-mesh}}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/all-to-all-broadcast-mesh.png}
\caption{All-to-All Broadcast Mesh}
\end{figure}

\begin{itemize}
\tightlist
\item
  each row of the mesh performs an all-to-all broadcast using the
  procedure for the linear array
\item
  in this phase, all nodes collect sqr(p) messages corresponding to the
  sqr(p) nodes of their respective rows
\item
  each node consolidates this information into a single message of size
  m * sqr(p)
\end{itemize}

\clearpage
\hypertarget{all-reduce}{%
\subsection{All-Reduce}\label{all-reduce}}

Each node starts with a buffer of size m and the final results of the
operation are identical buffers of size m on each node that are formed
by combining the original p buffers using an associative operator.

\begin{itemize}
\tightlist
\item
  inefficient approach

  \begin{itemize}
  \tightlist
  \item
    identical to all-to-one reduction followed by a one-to-all broadcast
  \end{itemize}
\item
  better approach

  \begin{itemize}
  \tightlist
  \item
    uses the pattern of all-to-all broadcast, but without increasing
    message size
  \item
    hypercube: $T = (t_s + mt_w) * log(p)$
  \end{itemize}
\end{itemize}

\hypertarget{scatter-and-gather}{%
\subsection{Scatter and Gather}\label{scatter-and-gather}}

\begin{itemize}
\tightlist
\item
  Scatter

  \begin{itemize}
  \tightlist
  \item
    a single node sends a unique message of size m to every other node
    (also called a one-to-all personalized communication)
  \item
    is fundamentally different from broadcast
  \item
    but the algorithmic structure is similar
  \end{itemize}
\item
  Gather

  \begin{itemize}
  \tightlist
  \item
    a single node collects a unique message from each node
  \item
    the gather operation is exactly the inverse of the scatter operation
    and can be frexecuted as such
  \end{itemize}
\end{itemize}

\hypertarget{all-to-all-personalized-communication-total-exchange}{%
\subsubsection{All-to-All Personalized Communication (Total
Exchange)}\label{all-to-all-personalized-communication-total-exchange}}

\begin{itemize}
\tightlist
\item
  each node has a distinct message of size m for every other node
\item
  this is unlike all-to-all broadcast, in which each node sends the same
  message to all other nodes
\end{itemize}

\clearpage
\hypertarget{all-to-all-personalize-communication-on-a-ring}{%
\subsubsection{All-to-All Personalize Communication on a
Ring}\label{all-to-all-personalize-communication-on-a-ring}}

\begin{itemize}
\tightlist
\item
  each node sends all pieces of data as one consolidated message of size
  m(p -- 1) to one of its neighbors
\item
  each node extracts the information meant for it from the data
  received, and forwards the remaining pieces of size m each to the next
  node
\item
  the size of the consolidated message reduces by m at each step
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/all-to-all-ring.png}
\caption{All-to-All Ring}
\end{figure}

\hypertarget{summary-of-communication-times}{%
\subsection{Summary of communication
times}\label{summary-of-communication-times}}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/summary-CommunicationTimes.png}
\caption{Summary of all communication times}
\end{figure}

\clearpage


% $Header: /cvsroot/latex-beamer/latex-beamer/solutions/conference-talks/conference-ornate-20min.en.tex,v 1.6 2004/10/07 20:53:08 tantau Exp $

\documentclass{beamer}

\mode<presentation>
{
  \usetheme{Hawke}
  % or ...

  \setbeamercovered{transparent}
  % or whatever (possibly just delete it)
}

\usepackage[english]{babel}
% or whatever

\usepackage[latin1]{inputenc}
% or whatever

\usepackage{times}
\usepackage[T1]{fontenc}

\usepackage{multimedia}

%%%%%%
% My Commands
%%%%%%

\newcommand{\bb}{{\boldsymbol{b}}}
\newcommand{\bx}{{\boldsymbol{x}}}
\newcommand{\by}{{\boldsymbol{y}}}
\newcommand{\bfm}[1]{{\boldsymbol{#1}}}

%%%%

\title[Lecture 16] % (optional, use only with long paper titles)
{Lecture 16 - Initial Value Problems}

\author[I. Hawke] % (optional, use only with lots of authors)
{I.~Hawke}

\institute[University of Southampton] % (optional, but mostly needed)
{
%  \inst{1}%
  School of Mathematics, \\
  University of Southampton, UK
}

\date[Semester 1] % (optional, should be abbreviation of conference name)
{MATH3018/6141, Semester 1}

\subject{Numerical methods}
% This is only inserted into the PDF information catalog. Can be left
% out.

\pgfdeclareimage[height=0.5cm]{university-logo}{mathematics_7469}
\logo{\pgfuseimage{university-logo}}

\AtBeginSection[]
{
  \begin{frame}<beamer>
    \frametitle{Outline}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Predictor-Corrector methods}

\subsection{Predictor-Corrector methods}

\begin{frame}
  \frametitle{Geometrical interpretation of Euler's method}

  Considering IVPs in the form
  \begin{equation*}
    \by'(x) = \bfm{f}(x, \by(x)).
  \end{equation*}

  The simple (first order accurate, explicit) Euler method is
  \begin{equation*}
    \by_{n+1} = \by_n + h \bfm{f}(x_n, \by_n).
  \end{equation*} \pause

  Euler's method is not sufficiently accurate for practical use. The
  geometrical interpretation is that the slope ($\by'$) over the
  interval $[x_n, x_{n+1}]$ is approximated by the slope at the
  \emph{beginning} of the interval,
  \begin{equation*}
    \by' \simeq \by'(x_n).
  \end{equation*} \pause

  Try a better approximation to the slope for a more accurate result.

\end{frame}

\begin{frame}
  \frametitle{Predicting and correcting}

  \begin{overlayarea}{\textwidth}{0.8\textheight}
    \only<1-3|handout:1>
    {
      Still using forward differencing for $\by'$, an ``average''
      slope is
      \begin{equation*}
        \frac{\by_{n+1} - \by_n}{h} = \frac{\bfm{f}(x_n, \by_n) +
          \bfm{f}(x_{n+1}, \by_{n+1})}{2}.
      \end{equation*}
    }
    \only<2-3|handout:1>
    {
      Rearrange: gives the unknown $\by_{n+1}$ as
      \begin{equation*}
        \by_{n+1} = \by_n + \frac{h}{2} \left( \bfm{f}(x_n, \by_n) +
          \bfm{f}(x_{n+1}, \by_{n+1}) \right).
      \end{equation*}
    }
    \only<3|handout:1>
    {
      But we need $\by_{n+1}$ to compute the right-hand-side.
    }
    \only<4-|handout:2>
    {
      \begin{equation*}
        \by_{n+1} = \by_n + \frac{h}{2} \left( \bfm{f}(x_n, \by_n) +
          \bfm{f}(x_{n+1}, \by_{n+1}) \right).
      \end{equation*}

      \vspace{1ex}

      Instead compute a \emph{guess} for $\by_{n+1}$, and use this
      guess when required. This is the \emph{predictor} step:
      \begin{equation*}
        \by_{n+1}^{(p)} = \by_n + h \bfm{f}(x_n, \by_n)
      \end{equation*}
      if we use Euler's method for the prediction.
    }
    \only<5-|handout:2>
    {

      \vspace{1ex}
      Using this, apply the \emph{corrector} step
      \begin{equation*}
        \by_{n+1} = \by_n + \frac{h}{2} \left( \bfm{f}(x_n, \by_n) +
          \bfm{f}(x_{n+1}, \by_{n+1}^{(p)}) \right),
      \end{equation*}
      which describes the (Euler) \emph{predictor-corrector} method.
    }
    \only<6|handout:2>
    {

      \vspace{1ex}
      The \emph{local} error is $\propto {\cal O}(h^3)$.
    }
  \end{overlayarea}

\end{frame}

\begin{frame}
  \frametitle{Example}

  Apply the Euler predictor-corrector method to
  \begin{equation*}
    y'(x) = - \sin(x), \quad y(0) = 1
  \end{equation*}
  to integrate up to $x = 0.5$. With $h = 0.1$:
  \begin{center}
    \begin{tabular}{c|c c c c c c}
      $n$ & $x_n$ & $y_n$ & $f(x_n, y_n)$ & $y_{n+1}^{(p)}$ & $f(x_{n+1},
      y_{n+1}^{(p)})$ & $\cos(x_n)$ \\
      \hline
      0 & 0.0 & 1.000 &  0.000 & 1.000 & -0.100 & 1.000 \\
      1 & 0.1 & 0.995 & -0.100 & 0.985 & -0.199 & 0.995 \\
      2 & 0.2 & 0.980 & -0.199 & 0.960 & -0.296 & 0.980 \\
      3 & 0.3 & 0.955 & -0.296 & 0.926 & -0.389 & 0.955 \\
      4 & 0.4 & 0.921 & -0.389 & 0.882 & -0.479 & 0.921 \\
      5 & 0.5 & 0.878 &        &       &        & 0.878
    \end{tabular}
  \end{center} \pause
  The error is $0.1\%$, not visible at this precision. With $h = 0.01$
  then the error is $0.001\%$, showing second order convergence.

\end{frame}

\begin{frame}
  \frametitle{Example: 2}

  Consider the system
  \begin{equation*}
    \left\{
      \begin{aligned}
        \dot{x} & = -y \\ \dot{y} & = x
      \end{aligned} \right., \quad x(0) = 1, \, \, y(0) = 0.
  \end{equation*}
  In polar coordinates this is $\dot{r} = 0$, $\dot{\phi} = 1$.
  \begin{columns}
    \begin{column}{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{0.4\textheight}
        \only<2-3|handout:1>
        {
          Use the predictor-corrector method with $h=0.1$. At $t=500$
          the result matches the correct answer to the eye.
        }
        \only<3|handout:1>
        {

          \vspace{1ex}
          The growth of the radius makes the errors visible.
        }
        \only<4-5|handout:2>
        {
          Use the predictor-corrector method with $h=0.01$. At $t=500$
          the result matches the correct answer to the eye.
        }
        \only<5|handout:2>
        {

          \vspace{1ex}
          Looking at the growth of the radius makes the errors
          visible even now.
        }
      \end{overlayarea}
    \end{column}
    \begin{column}{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{0.6\textheight}
        \only<2|handout:0>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/EulerPC1}
          \end{center}
        }
        \only<3|handout:1>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/EulerPC_rad1}
          \end{center}
        }
        \only<4|handout:0>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/EulerPC2}
          \end{center}
        }
        \only<5|handout:2>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/EulerPC_rad2}
          \end{center}
        }
      \end{overlayarea}
    \end{column}
  \end{columns}

\end{frame}

\section{Errors}

\subsection{Errors}

\begin{frame}
  \frametitle{Errors}

  \begin{columns}
    \begin{column}{0.5\textwidth}
      With quadrature using e.g.\ Simpson's rule the \emph{local}
      error on any subinterval gave
      \begin{equation*}
        |{\cal E}_{\text{global}}| \le \sum |{\cal E}_{\text{local}}|.
      \end{equation*} \pause

      For an IVP we still have the \emph{local} and \emph{global}
      errors.  But steps are not independent so the error bound is
      different.
    \end{column}
    \begin{column}{0.5\textwidth}
      \begin{center}
        \includegraphics<1|handout:0>[width=\textwidth]{figures/simpson2}
        \includegraphics<2>[width=\textwidth]{figures/Errors1}
      \end{center}
    \end{column}
  \end{columns}

\end{frame}

\begin{frame}
  \frametitle{Truncation vs.\ roundoff}

  All error (local and global) has two pieces: \emph{truncation} and
  \emph{roundoff} error.
  \begin{enumerate}
  \item Roundoff: inherent in the floating point
    representation. Usually extremely small, it cannot be controlled.
    \pause
  \item Truncation: the result of replacing the continuous
    differential equation with a discrete algorithm. Usually
    dominant. Typically depends on ``step size'' $h$ which can be
    controlled.
  \end{enumerate} \pause

  \vspace{1ex}

  The \emph{total error} (to minimize) is the sum of the global
  truncation and global roundoff errors.

  \vspace{1ex}

  We bound the global truncation error only as we expect roundoff
  error to be negligible.

\end{frame}

\begin{frame}
  \frametitle{Error accumulation}

  \begin{equation*}
    |{\cal E}_{\text{global, truncation}}| \ne \sum |{\cal
      E}_{\text{local, truncation}}|.
  \end{equation*}
  Each step depends on the previous answer: \emph{total} previous
  error changes the initial data for each step. \pause So first bound
  the effect of this uncertainty in the initial data:

  \vspace{1ex}

  {\bf Theorem}: For the IVP
  \begin{equation*}
    y' = f(x, y), \quad y(0) = s, \quad 0 \le x \le X > 0,
  \end{equation*}
  the variation of the solution dependent on the initial data is
  bounded by
  \begin{equation*}
    \left| \frac{\partial y}{\partial s} \right| \leq e^{\lambda x}
  \end{equation*}
  where $\partial_y f \leq \lambda$ in $0 \le x \le X$.

\end{frame}

\begin{frame}
  \frametitle{Error accumulation: 2}

  If the IVP
  \begin{equation*}
    y' = f(x, y), \quad 0 \le x \le X > 0,
  \end{equation*}
  is solved with \emph{different} initial data $y(0) = s$, $y(0) = s +
  \delta$, then above implies that two initially ``close'' solutions
  will behave as
  \begin{equation*}
    | y(x, s) - y(x, s + \delta)| \leq |\delta| e^{\lambda x}.
  \end{equation*}
  \pause

  \vspace{1ex}

  At best the error growth is proportional to the exponential of the
  step length \emph{and} proportional to the initial error. \pause

  \vspace{1ex}

  Put this all together by summing over all steps.

\end{frame}

\begin{frame}
  \frametitle{Error accumulation: 3}

  {\bf Global Error Theorem}: If $|{\cal E}_{\text{local,
      truncation}}| \le \delta \propto h^{p+1}$, the global truncation
  error after $n$ steps is bounded by
  \begin{equation*}
    \delta \frac{1 - e^{n \lambda h}}{1 - e^{\lambda h}}.
  \end{equation*} \pause

  \vspace{1ex}

  By induction compute the ``initial'' error for each step; the
  previous theorem bounds $\delta_n$ in terms of $\lambda$ and
  $h$. Result follows from a geometric progression.  \pause

  \vspace{1ex}

  To use this theorem \emph{assume} that $\lambda h \ll 1$. We have $n
  h = X = (b - a)$. Taylor expanding the error bound we have
  \begin{equation*}
    |{\cal E}_{\text{Global}}|  \leq \left| \delta \frac{(1 -
      e^{\lambda(b-a)})}{-\lambda h - {\cal O}(h^2)} \right|
    \leq {\cal O}(h^p) \left| 1 - e^{\lambda(b-a)} \right|.
  \end{equation*}

\end{frame}

\section{Summary}

\subsection{Summary}

\begin{frame}
  \frametitle{Summary}

  \begin{itemize}
  \item The errors at each step in an IVP solver are not independent.
    Hence the total error is not bounded by the sum of the local
    errors.
  \item If the local error is ${\cal O}(h^{n+1})$ then the global
    error is ${\cal O}(h^{n})$. Such a method is said to be of
    \emph{order $n$}.
  \item Euler's method has local error $\propto h^2$, hence global
    error $\propto h$.
  \item The Euler predictor-corrector method has local error $\propto
    h^3$, hence global error $\propto h^2$.
  \end{itemize}

\end{frame}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\documentclass[jou]{apa6}

\usepackage[american]{babel}

\usepackage{csquotes}
\usepackage[style=apa,sortcites=true,sorting=nyt,backend=biber]{biblatex}
\DeclareLanguageMapping{american}{american-apa}
\addbibresource{bibliography.bib}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Discrete Structures
%% The start of RBS stuff
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Working internal and external links in PDF
\usepackage{hyperref}
% Extra math symbols in LaTeX
\usepackage{amsmath}
\usepackage{gensymb}
\usepackage{amssymb}
% Enumerations with (a), (b), etc.
\usepackage{enumerate}

\let\OLDitemize\itemize
\renewcommand\itemize{\OLDitemize\addtolength{\itemsep}{-6pt}}

\usepackage{etoolbox}
\makeatletter
\preto{\@verbatim}{\topsep=3pt \partopsep=3pt }
\makeatother

% These sizes redefine APA for A4 paper size
\oddsidemargin 0.0in
\evensidemargin 0.0in
\textwidth 6.27in
\headheight 1.0in
\topmargin -24pt
\headheight 12pt
\headsep 12pt
\textheight 9.19in

\setlength\parindent{0pt}

\title{Sample Quiz 4}
\author{Discrete Structures, Spring 2020}
\affiliation{RBS}

\leftheader{Discrete Quiz 6}

\abstract{%
}

%\keywords{}

\begin{document}

%\thispagestyle{empty}

\twocolumn
\section{Quiz 6: Recurrent Sequences}

\vspace{10pt}
{\bf Question 1: Modifying Sudoku Rules.}\\
We have a $9 \times 9$ table; each cell contains one number from
$A_9 = \{ 1, 2,\ldots, 9 \}$.
Number $a_{i,j}$ describes the number written on Line $i$ and
Column $j$ in the table (and $i,j \in A_9$).

Identify the formula describing that the sum of all numbers in one
diagonal of the table $9 \times 9$ is equal to the sum of all numbers
on the other diagonal of the table.

{\bf (A)} ${\displaystyle \sum\limits_{r=0}^{8} a_{r+1,r+1} = \sum\limits_{r=0}^{8} a_{9-r,r+1}}$\\
{\bf (B)} ${\displaystyle \sum\limits_{r=0}^{8} a_{r,r} = \sum\limits_{r=0}^{8} a_{9-r,r}}$\\
{\bf (C)} ${\displaystyle \sum\limits_{r=0}^{8} a_{r+1,r+1} = \sum\limits_{r=0}^{8} a_{10-r,r+1}}$\\
{\bf (D)} ${\displaystyle \sum\limits_{r=0}^{8} a_{r,r} = \sum\limits_{r=0}^{8} a_{10-r,r}}$

\vspace{6pt}
{\bf Question 2: Computing sums.}\\
Find the values of these sum in a Sudoku table (it is known that each number from $A_9 = \{ 1, 2,\ldots, 9 \}$
is written exactly once in all rows, in all columns and also in all $3 \times 3$ blocks).

$${\displaystyle \sum\limits_{r=1}^{9} \sum\limits_{s=1}^{9} a_{r,s}}$$

Write in the number as your answer.

\vspace{6pt}
{\bf Question 3: Long set operations.} Denote $A_1 = \{ 1 \}$, $A_2 = \{ 1,2 \}$, etc.
In general, $A_k = \{ 1,2,\ldots,k\}$.\\
By $A \oplus B = (A - B) \cup (B-A)$
we denote the symmetric difference: All elements that belong to just one of the
sets $A,B$ (but not the other one).
Consider this set:
$$S = \bigoplus\limits_{j=1}^{100} A_{2j-1}.$$
Write a comma-separated list of the $10$ smallest elements of $S$ in
increasing order.

\vspace{6pt}
{\bf Question 4: Using recurrent formula.}
Find the first $6$ members of this infinite sequence ($C_0,C_1,C_2,\ldots$):
$$\left\{ \begin{array}{l}
C_0 = 1,\\
C_{n+1} = \sum\limits_{i=0}^n (C_i \cdot C_{n-i}).
\end{array} \right.$$

In your answer write comma-separated values:\\
$C_0,C_1,C_2,C_3,C_4,C_5$.

\vspace{6pt}
{\bf Question 5: Reccurent sequence.} A sequence of real numbers
$f\,:\,\mathbb{N} \rightarrow \mathbb{R}$ satisfies
the following properties:\\
{\bf (A)} $f(k+2) = 2f(k+1) + f(k)$ for all integers $k \geq 2$.\\
{\bf (B)} $f(n)$ is a geometric progression.

Find some quotient of this geometric progression (if there are several possibilities, pick any of them).
Round it to the nearest thousandth, i.e. specify the first three digits after the decimal point).

\vspace{6pt}
{\bf Question 6: Finding a limit.} Define the following sequence:
$$\left\{ \begin{array}{l}
x_0 = 1,\\
x_{n+1} = \frac{1}{3} \left( 2x_n + \frac{7}{x_n^2}\right),\;\text{if}\;n\geq 0\\
\end{array} \right.$$
Assume that there exists limit $L = \lim_{n \rightarrow \infty} x_n$.\\
Find that limit $L$ and round it to
the nearest thousandth.

\vspace{6pt}
{\bf Question 7: Taylor series} There is a formula known from calculus (practically
used to compute $y = \sin x$) for each $x \in \mathbb{R}$.
$$\sin x = \sum\limits_{n=0}^{\infty} \frac{(-1)^n \cdot x^{2n+1}}{(2n+1)!} = x - \frac{x^3}{3!}
+ \frac{x^5}{5!} - \ldots$$

Use Python or Scala to add the first $20$ terms of this infinite sum
to compute $\sin 1080^{\circ}$. Round the answer
to $15$ digits (this usually happens by itself: double precision numbers are output in this manner).

\newpage

\subsection{Answers}

\vspace{6pt}
{\bf Question 1.} Answer: {\tt A}\\
All other variants refer to $a_{ij}$, where $i$ or $j$ are outside the interval $[1;9]$,
so they are not defined. Answer (A), on the other hand, refers to the elements that
lay on both diagonals.

\vspace{6pt}
{\bf Question 2.} Answer: {\tt 405}\\
This summation is the total of all elements in the Sudoku table.
Each row contains all numbers from $1$ to $9$. So they add up to
$45 = \frac{9 \cdot (9+1)}{2}$. This has to be multiplied by $9$, since
there are $9$ rows.

\vspace{6pt}
{\bf Question 3.} Answer:\\ {\tt 2,3,6,7,10,11,14,15,18,19}\\
The smallest number $1 \not\in S$, because it belongs to all $100$ sets (so it cancels out, when
we compute the long symmetric sum).\\
Numbers $2$ and $3$ belong to exactly $99$ sets, therefore they are included.\\
Numbers $4$ and $5$ belong to exactly $98$ sets, so they are not included, and so on.

\vspace{6pt}
{\bf Question 4.} Answer: {\tt 1,1,2,5,14,42}\\
We can simply plug into the formulas:
$$\left\{ \begin{array}{l}
C_1 = C_0 \cdot C_0 = 1 \cdot 1 = 1\\
C_2 = C_0 \cdot C_1 + C_1 \cdot C_0 = 1 \cdot 1 + 1 \cdot 1 = 2\\
C_3 = C_0 \cdot C_2 + C_1 \cdot C_1 + C_2 \cdot C_0 = 5\\
\ldots
\end{array} \right.$$

These are also known as {\em Catalan numbers}.
For example $C_3$ shows, in how many ``essentially different'' ways you can put parentheses
in an expression with four letters and three binary operators:
$$(((a \circ b) \circ c) \circ d);\; ((a \circ (b \circ c)) \circ d);$$
$$((a \circ b) \circ (c \circ d));\; (a \circ ((b \circ c) \circ d));$$
$$(a \circ (b \circ (c \circ d)))$$

Here we assume that the $\circ$ operation is neither associative nor commutative.
Certainly, we could insert even more parentheses; but these $5$ ways differ
by the order of execution of these three operations.

\vspace{6pt}
{\bf Question 5.} Answer: {\tt 2.414} {\bf or} {\tt -0.414}\\
If we substitute the geometric progression $f(n) = f(0) \cdot q^n$
into the equation $f(k+2) = 2f(k+1) + f(k)$:
$$f(0) \cdot q^{k+2} = 2f(0) \cdot q^{k+1} + f(0) \cdot q^{k}.$$

Now there are three possibilities.\\
{\bf Case 1.} If $f(0) = 0$, then the sequence
$0,0,\ldots$ is a (degenerated) version of a geometric series.
Any number can be its quotient; so it is not interesting to solve this.\\
{\bf Case 2.} If $f(0) \neq 0$, we could still take $q = 0$, but this
is also a degenerated geometric series of all zeroes (except the first member $f(0)$).
This is not interesting either.\\
{\bf Case 3.} If $f(0)$ and $q$ are both nonzero, we can cancel them in the
above equation and get this: $q^2 = 2q + 1$.
The two roots are
$$q_{1,2} = \frac{2 \pm \sqrt{4 + 4}}{2} = 1 \pm \sqrt{2}.$$
Either answer $1 - \sqrt{2}$ or $1 + \sqrt{2}$ is valid.

\vspace{6pt}
{\bf Question 6.} If the limit $L$ exists, then both $x_n$ and $x_{n+1}$ both
go to that limit, and we must have the identity:
$$L = \frac{1}{3}\left( 2L + \frac{7}{L^2} \right).$$
When we express $L$ from that equation, we get $L = \sqrt[3]{7}$.

In fact, the sequence does converge to the cubic root:
\begin{verbatim}
x(0) = 1.000000000000000
x(1) = 3.000000000000000
x(2) = 2.259259259259259
x(3) = 1.963308018221572
x(4) = 1.914212754165601
x(5) = 1.912932040596942
x(6) = 1.912931182772774
x(7) = 1.912931182772389
x(8) = 1.912931182772389
\end{verbatim}

\vspace{6pt}
{\bf Question 7.} Answer:\\ {\tt -482.8813747415584} (or similar)

In theory $\sin 1080^{\circ} = 0$ (and Taylor series should converge to $0$),
but this nonsense result happens because of huge rounding errors.

{\footnotesize
\begin{verbatim}
object TaylorSeries {

  def factorial(n:Int):Double = {
    n match {
      case 0 => 1.0
      case n => n*factorial(n-1)
    }
  }

  def sin(x:Double,nTerms:Int):Double = {
    val terms =
      for (k <- List.range(0,nTerms))
        yield { math.pow(-1,k)*
        math.pow(x,2*k+1)/factorial(2*k+1) }
    terms.foldLeft(0.0)((a,b) => a+b)
  }

  def main(args:Array[String]): Unit = {
    val nTerms = 20
    val x = Math.PI*6 // (1080/180)*PI = 6PI
    println(sin(x,nTerms))
  }
}
\end{verbatim}
}

\end{document}