\appendix
\section{Appendix: Example Primer}
\label{sec:examplePrimer}
In this Appendix, a set of examples are reported. In order to be as general as possible, the \textit{Model} type ``ExternalModel'' has been used.
%%%% EXAMPLE 1
\subsection{Example 1.}
\label{subsec:ex1}
This simple example is about the construction of a ``Lorentz attractor'', sampling the relative input space. The parameters that are sampled represent the initial coordinate (x0,y0,z0) of the attractor origin.

\begin{lstlisting}[style=XML,morekeywords={debug,re,seeding,class,subType,limit}]
<?xml version="1.0" encoding="UTF-8"?>
<Simulation verbosity="debug">
<!-- RUNINFO -->
<RunInfo>
    <WorkingDir>externalModel</WorkingDir>
    <Sequence>FirstMRun</Sequence>
    <batchSize>3</batchSize>
</RunInfo>
<!-- Files -->
<Files>
    <Input name='lorentzAttractor.py' type=''>lorentzAttractor</Input>
</Files>
<!-- STEPS -->
<Steps>
    <MultiRun name='FirstMRun'  re-seeding='25061978'>
        <Input   class='Files'     type=''               >lorentzAttractor.py</Input>
        <Model   class='Models'    type='ExternalModel'  >PythonModule</Model>
        <Sampler class='Samplers'  type='MonteCarlo'     >MC_external</Sampler>
        <Output  class='DataObjects'     type='HistorySet'      >testPrintHistorySet</Output>
        <Output  class='Databases' type='HDF5'           >test_external_db</Output>
        <Output  class='OutStreams' type='Print'   >testPrintHistorySet_dump</Output>
    </MultiRun >
</Steps>
<!-- MODELS -->
<Models>
    <ExternalModel name='PythonModule' subType='' ModuleToLoad='externalModel/lorentzAttractor'>
       <variables>sigma,rho,beta,x,y,z,time,x0,y0,z0</variables>
    </ExternalModel>
</Models>
<!-- DISTRIBUTIONS -->
<Distributions>
    <Normal name='x0_distrib'>
        <mean>4</mean>
        <sigma>1</sigma>
    </Normal>
    <Normal name='y0_distrib'>
        <mean>4</mean>
        <sigma>1</sigma>
    </Normal>
    <Normal name='z0_distrib'>
        <mean>4</mean>
        <sigma>1</sigma>
    </Normal>
</Distributions>
<!-- SAMPLERS -->
<Samplers>
    <MonteCarlo name='MC_external'>
      <samplerInit>
        <limit>3</limit>
      </samplerInit>
      <variable name='x0' >
        <distribution  >x0_distrib</distribution>
      </variable>
      <variable name='y0' >
        <distribution  >y0_distrib</distribution>
      </variable>
      <variable name='z0' >
        <distribution  >z0_distrib</distribution>
      </variable>
    </MonteCarlo>
</Samplers>
<!-- DATABASES -->
<Databases>
  <HDF5 name="test_external_db"/>
</Databases>
<!-- OUTSTREAMS -->
<OutStreams>
  <Print name='testPrintHistorySet_dump'>
    <type>csv</type>
    <source>testPrintHistorySet</source>
  </Print>
</OutStreams>
<!-- DATA OBJECTS -->
<DataObjects>
    <HistorySet name='testPrintHistorySet'>
        <Input>x0,y0,z0</Input>
        <Output>time,x,y,z</Output>
   </HistorySet>
</DataObjects>
</Simulation>
\end{lstlisting}
The Python \textit{ExternalModel} is reported below:
\begin{lstlisting}[language=python]
import numpy as np

def run(self,Input):
  max_time = 0.03
  t_step = 0.01

  numberTimeSteps = int(max_time/t_step)

  self.x = np.zeros(numberTimeSteps)
  self.y = np.zeros(numberTimeSteps)
  self.z = np.zeros(numberTimeSteps)
  self.time = np.zeros(numberTimeSteps)

  self.x0 = Input['x0']
  self.y0 = Input['y0']
  self.z0 = Input['z0']

  self.x[0] = Input['x0']
  self.y[0] = Input['y0']
  self.z[0] = Input['z0']
  self.time[0]= 0

  for t in range (numberTimeSteps-1):
    self.time[t+1] = self.time[t] + t_step
    self.x[t+1]    = self.x[t] +  self.sigma*
                      (self.y[t]-self.x[t]) * t_step
    self.y[t+1]    = self.y[t] + (self.x[t]*
                      (self.rho-self.z[t])-self.y[t]) * t_step
    self.z[t+1]    = self.z[t] + (self.x[t]*
                          self.y[t]-self.beta*self.z[t]) * t_step
\end{lstlisting}
%%%% EXAMPLE 2
\subsection{Example 2.}
\label{subsec:ex1}
This example shows a slightly more complicated example, that employs the usage of:
\begin{itemize}
    \item \textit{Samplers:} Grid and Adaptive;
    \item \textit{Models:} External, Reduce Order Models and Post-Processors;
    \item \textit{OutStreams:} Prints and Plots;
    \item \textit{Data Objects:} PointSets;
    \item \textit{Functions:} ExternalFunctions.
\end{itemize}
The goal of this input is to compute the ``SafestPoint''.
It provides the coordinates of the farthest
point from the limit surface that is given as an input.
%
The safest point coordinates are expected values of the coordinates of the
farthest points from the limit surface in the space of the ``controllable''
variables based on the probability distributions of the ``non-controllable''
variables.

The term ``controllable'' identifies those variables that are under control
during the system operation, while the ``non-controllable'' variables are
stochastic parameters affecting the system behaviour randomly.

The ``SafestPoint'' post-processor requires the set of points belonging to the
limit surface, which must be given as an input.

\begin{lstlisting}[style=XML,morekeywords={debug,re,seeding,class,subType,limit}]
<Simulation verbosity='debug'>

<!-- RUNINFO -->
<RunInfo>
  <WorkingDir>SafestPointPP</WorkingDir>
  <Sequence>pth1,pth2,pth3,pth4</Sequence>
  <batchSize>50</batchSize>
</RunInfo>

<!-- STEPS -->
<Steps>
  <MultiRun name = 'pth1' pauseAtEnd = 'False'>
    <Sampler  class = 'Samplers'  type = 'Grid'           >grd_vl_ql_smp_dpt</Sampler>
    <Input    class = 'DataObjects'     type = 'PointSet'   >grd_vl_ql_smp_dpt_dt</Input>
    <Model    class = 'Models'    type = 'ExternalModel'  >xtr_mdl</Model>
    <Output   class = 'DataObjects'     type = 'PointSet'   >nt_phy_dpt_dt</Output>
  </MultiRun >

  <MultiRun name = 'pth2' pauseAtEnd = 'True'>
    <Sampler          class = 'Samplers'  type = 'Adaptive'      >dpt_smp</Sampler>
    <Input            class = 'DataObjects'     type = 'PointSet'  >bln_smp_dt</Input>
    <Model            class = 'Models'    type = 'ExternalModel' >xtr_mdl</Model>
    <Output           class = 'DataObjects'     type = 'PointSet'  >nt_phy_dpt_dt</Output>
    <SolutionExport   class = 'DataObjects'     type = 'PointSet'  >lmt_srf_dt</SolutionExport>
  </MultiRun>

  <PostProcess name='pth3' pauseAtEnd = 'False'>
    <Input    class = 'DataObjects'          type = 'PointSet'       >lmt_srf_dt</Input>
    <Model    class = 'Models'         type = 'PostProcessor'  >SP</Model>
    <Output   class = 'DataObjects'          type = 'PointSet'     >sfs_pnt_dt</Output>
  </PostProcess>

  <OutStreamStep name = 'pth4' pauseAtEnd = 'True'>
  	<Input  class = 'DataObjects'            type = 'PointSet'  >lmt_srf_dt</Input>
  	<Output class = 'OutStreams' type = 'Print'         >lmt_srf_dmp</Output>
    <Input  class = 'DataObjects'            type = 'PointSet'  >sfs_pnt_dt</Input>
  	<Output class = 'OutStreams' type = 'Print'         >sfs_pnt_dmp</Output>
  </OutStreamStep>
</Steps>

<!-- DATA OBJECTS -->
<DataObjects>
  <PointSet name = 'grd_vl_ql_smp_dpt_dt'>
    <Input>x1,x2,gammay</Input>
    <Output>OutputPlaceHolder</Output>
  </PointSet>

  <PointSet name = 'nt_phy_dpt_dt'>
    <Input>x1,x2,gammay</Input>
    <Output>g</Output>
  </PointSet>

  <PointSet name = 'bln_smp_dt'>
    <Input>x1,x2,gammay</Input>
    <Output>OutputPlaceHolder</Output>
  </PointSet>

  <PointSet name = 'lmt_srf_dt'>
    <Input>x1,x2,gammay</Input>
    <Output>g_zr</Output>
  </PointSet>

  <PointSet name = 'sfs_pnt_dt'>
    <Input>x1,x2,gammay</Input>
    <Output>p</Output>
  </PointSet>
</DataObjects>

<!-- DISTRIBUTIONS -->
<Distributions>
  <Normal name = 'x1_dst'>
    <upperBound>10</upperBound>
    <lowerBound>-10</lowerBound>
  	<mean>0.5</mean>
    <sigma>0.1</sigma>
  </Normal>

  <Normal name = 'x2_dst'>
    <upperBound>10</upperBound>
    <lowerBound>-10</lowerBound>
    <mean>-0.15</mean>
    <sigma>0.05</sigma>
  </Normal>

  <Normal name = 'gammay_dst'>
    <upperBound>20</upperBound>
    <lowerBound>-20</lowerBound>
    <mean>0</mean>
    <sigma>15</sigma>
  </Normal>
</Distributions>

<!-- SAMPLERS -->
<Samplers>
  <Grid name = 'grd_vl_ql_smp_dpt'>
    <variable name = 'x1' >
      <distribution>x1_dst</distribution>
      <grid type = 'value' construction = 'equal' steps = '10' upperBound = '10'>2</grid>
    </variable>
    <variable name='x2' >
      <distribution>x2_dst</distribution>
      <grid type = 'value' construction = 'equal' steps = '10' upperBound = '10'>2</grid>
    </variable>
    <variable name='gammay' >
      <distribution>gammay_dst</distribution>
      <grid type = 'value' construction = 'equal' steps = '10' lowerBound = '-20'>4</grid>
    </variable>
  </Grid>

  <Adaptive name = 'dpt_smp' verbosity='debug'>
    <ROM              class = 'Models'    type = 'ROM'           >accelerated_ROM</ROM>
    <Function         class = 'Functions' type = 'External'      >g_zr</Function>
    <TargetEvaluation class = 'DataObjects'     type = 'PointSet'  >nt_phy_dpt_dt</TargetEvaluation>
    <Convergence limit = '3000' forceIteration = 'False' weight = 'none' persistence = '5'>1e-2</Convergence>
      <variable name = 'x1'>
        <distribution>x1_dst</distribution>
      </variable>
      <variable name = 'x2'>
        <distribution>x2_dst</distribution>
      </variable>
      <variable name = 'gammay'>
        <distribution>gammay_dst</distribution>
      </variable>
  </Adaptive>
</Samplers>

<!-- MODELS -->
<Models>
  <ExternalModel name = 'xtr_mdl' subType = '' ModuleToLoad = 'SafestPointPP/safest_point_test_xtr_mdl'>
    <variables>x1,x2,gammay,g</variables>
  </ExternalModel>

  <ROM name = 'accelerated_ROM' subType = 'SciKitLearn'>
    <Features>x1,x2,gammay</Features>
    <Target>g_zr</Target>
    <SKLtype>svm|SVC</SKLtype>
    <kernel>rbf</kernel>
    <gamma>10</gamma>
    <tol>1e-5</tol>
    <C>50</C>
  </ROM>

  <PostProcessor name='SP' subType='SafestPoint'>
    <!-- List of Objects (external with respect to this PP) needed by this post-processor -->
    <Distribution     class = 'Distributions'  type = 'Normal'>x1_dst</Distribution>
    <Distribution     class = 'Distributions'  type = 'Normal'>x2_dst</Distribution>
    <Distribution     class = 'Distributions'  type = 'Normal'>gammay_dst</Distribution>
    <!- end of the list -->
    <controllable>
    	<variable name = 'x1'>
    		<distribution>x1_dst</distribution>
    		<grid type = 'value' steps = '20'>1</grid>
    	</variable>
    	<variable name = 'x2'>
    		<distribution>x2_dst</distribution>
    		<grid type = 'value' steps = '20'>1</grid>
    	</variable>
    </controllable>
    <non-controllable>
    	<variable name = 'gammay'>
    		<distribution>gammay_dst</distribution>
    		<grid type = 'value' steps = '20'>2</grid>
    	</variable>
    </non-controllable>
  </PostProcessor>
</Models>

<!-- FUNCTIONS -->
<Functions>
  <External name='g_zr' file='SafestPointPP/safest_point_test_g_zr.py'>
    <variable>g</variable>
  </External>
</Functions>

<!-- OUT-STREAMS -->
<OutStreams>
  <Print name = 'lmt_srf_dmp'>
  	<type>csv</type>
  	<source>lmt_srf_dt</source>
  </Print>

  <Print name = 'sfs_pnt_dmp'>
  	<type>csv</type>
  	<source>sfs_pnt_dt</source>
  </Print>
</OutStreams>

</Simulation>
\end{lstlisting}
The Python \textit{ExternalModel} is reported below:
\begin{lstlisting}[language=python]
def run(self,Input):
  self.g = self.x1+4*self.x2-self.gammay
\end{lstlisting}
The ``Goal Function'',the function that defines the transitions with respect the input space coordinates, is as follows:
\begin{lstlisting}[language=python]
def __residuumSign(self):
  if self.g<0 : return  1
  else        : return -1
\end{lstlisting}

%%%%% EXAMPLE 3
%\subsection{Example3}
%\label{subsec:ex1}
%example 3


\section{Loss function and Gradient Descent} \label{sec:loss_&_gradient_descent}
So far only the concept of what a neural network is was explained, but not how it can learn from data, to understand this it is worth to have first a brief summary of what consists a neural network.

A collection of units having nonlinear activation functions form a layer; layers are stacked from the input to the output with some optional hidden layers in-between; the weighted input of a unit is calculated by multiplying its inputs with some weight parameters and adding a bias term; the unit activation is the output of its activation function applied to the weighted input; and by varying the values of the weights and biases it is possible to reach different model behaviours \textbf{--} this section will only consider the case when the inputs of a unit come from the immediate previous layer and the output is calculated by passing the input values from layer to layer until the last layer (this is called a \textit{feedforward} neural network), other configurations like \acp{ResNet} \cite{resnet2015} and \acp{RNN} also exist, but it is possible to extend the arguments present here to also explain learning in these contexts.

For all neural networks, learning is the process of changing the weights and biases (and possibly other values) in order to produce the desired result. The set of values that are learned during training are called the \textit{parameters} of the network.

The number of parameters gives the degree of freedom that the network has, in a larger parameter space the network has more possible choices and can construct more complex relations. On the other hand, fewer parameters may make it difficult or impossible to represent the full complexity of the data. However, besides requiring more computational power, very complex models break the idea of sparsity and can more easily fall into the problem of overfitting (see \autoref{sub:overfitting}).

The number of parameters in the network is given by its architecture, how many layers it has, how many units in each layer, how the connections are made, and so on. This is a choice made when building the network and is not something that is learned by the algorithm. The set of properties that influence the network but that are not learned are usually called the \textit{hyperparameters}.

Another example of hyperparameter is the activation functions used in the units, they are chosen when constructing the neural network and can't be learned \textbf{--} however some activations like PReLU have internal learnable parameters \cite{prelu2015}.

Learning is then the process of finding optimal parameters for a network given some hyperparameters. But how can a machine automatically learn the parameters? To do this the network must have some way to quantify the results that it produces, a way to distinguish between good and bad outputs, this is what is called the \textit{loss function}.

Also called the cost or objective, the loss function $J(\bm{x}, \bm{\theta})$ is a function that returns how good the output of the network is, given an input $\bm{x}$ and a set of network parameters $\bm{\theta}$ (represents all trainable parameters: weights, biases, and any others). Learning can then be described as the process of adapting the parameters $\bm{\theta}$ in order to minimize or maximize the objective function (usually minimize, depends on the function used). The choice of function will depend on the type of problem, but two of the most common methods are \gls{MSE} and Categorical Cross Entropy.

\subsection{Mean Squared Error} \label{sub:mse}
This function is the standard squared euclidean distance between two vectors. For a given input $\bm{x}$, the network output $\hat{\bm{y}}$ for this input, and the true desired output $\bm{y}$ (e.g. the corresponding label), the squared distance is calculated as shown in \autoref{eq:squared_distance} \textbf{--} note that the subscript $2$ indicates the euclidean distance (the $\ell_2$ norm).
\begin{equation} \label{eq:squared_distance}
    \| \bm{y} - \hat{\bm{y}} \|_{2}^{2} = \sum_{i}{(y_j - \hat{y_i})^2}
\end{equation}

This distance can also be generalize to matrices or any $n$ dimensional values of $\bm{y}$ by just calculating the element-wise squared differences between $\bm{y}$ and $\hat{\bm{y}}$.

However, simply calculating the distance for one possible input is not good for evaluating how well the network is doing in general. To better evaluate the network's performance it is necessary to see how well it is doing in the entire dataset or some subset of it. For a set $\bm{X}$ of $n$ inputs, the \gls{MSE} is the mean of the squared distances as shown in \autoref{eq:mse}.
\begin{equation} \label{eq:mse}
    J(\bm{X}, \bm{\theta}) = \frac{1}{n}\sum^{n}||\bm{y} - \hat{\bm{y}}||_{2}^{2} = \frac{1}{n}\sum^{n}\sum_{i}{(y_i - \hat{y_i})^2}
\end{equation}

This loss function is usually used for regression problems, where the output can have a range of values and is not simply defined as a $0$ or a $1$. It is also useful to calculate the differences between images in pixel space, like used for autoencoders \cite{autoencoder1991}.

\subsection{Categorical and Binary Cross Entropy} \label{sub:cross_entropy}
Like \gls{MSE} this loss function is calculated by averaging the error over many different inputs, but in this case the error calculated is the cross entropy. Given two discrete probability distributions $\bm{y}$ and $\hat{\bm{y}}$, the cross entropy is calculated as shown in \autoref{eq:cross_entropy}.
\begin{equation}  \label{eq:cross_entropy}
    H(\bm{y}, \hat{\bm{y}}) = -\sum_{i}{y_i\log(\hat{y}_i + \epsilon)}
\end{equation}

The \gls{epsilon} value in this equation is not a part of the cross entropy definition, but when using computers to calculate the logarithm, numbers very close or equal to zero can introduce numerical instabilities. For this reason, in almost all practical implementations it is always added a small constant $\epsilon$ when calculating the cross entropy and other functions that can have similar issues \textbf{--} in Tensorflow \cite{tensorflow2015} the default value for $\epsilon$ is equal to $10^{-7}$.

The cross entropy applied over a set of inputs gives the categorical cross entropy loss as shown in \autoref{eq:categorical_cross_entropy}.
\begin{equation} \label{eq:categorical_cross_entropy}
    J(\bm{X}, \bm{\theta}) = \frac{1}{n}\sum^{n}{ H(\bm{y}, \hat{\bm{y}}) }
\end{equation}

This kind of loss function is useful when dealing with probability distributions, since the concept of entropy is deeply related with information and probability. This makes it a better alternative for classification problems when compared with \gls{MSE}, since the desired output is the class that the input belongs to (one hot encoded) and the network output is a probability distribution over all possible classes.

But for cases where the output can only assume two values (0 or 1), the categorical cross entropy is reduced to a binary cross entropy. For \acp{GAN}, this is the more useful loss function and it is calculated as shown in \autoref{eq:binary_cross_entropy}.
\begin{equation} \label{eq:binary_cross_entropy}
    J(\bm{X}, \bm{\theta}) = -\frac{1}{n}\sum^{n}\sum_{i}{
        \left( y_i\log(\hat{y_i} + \epsilon) + (1 - y_i) \log(1 - \hat{y_i} + \epsilon) \right)
    }
\end{equation}

\subsection{Minimizing the loss} \label{sub:minimizing_loss}
Having an understanding of what is the loss function, the remaining question is: how can the network use the loss in order to update its parameters? Note that since this function effectively measures how well the network is doing, it can directly be used to guide the parameter changes, the goal is to change the parameters in order to minimize the loss (for now on the goal will only be described as minimization, since it is the most common approach and any maximization problem is equivalent to minimizing the negative of what is being maximized).

Note that the inputs $\bm{x}$ are fixed, given that the dataset is also fixed, and that the goal of using neural networks is that they should learn to model the data and not just choose whatever inputs reduce their loss. So in the eyes of the network $J(\bm{x}, \bm{\theta})$ becomes only $J(\bm{\theta})$. This may seem obvious, but this shows an important intuition that the cost function is a high dimensional surface in $N$ dimensional space, $N$ being the number of parameters of the network. And this demonstrates the importance of having both the network and loss functions be continuous and differentiable functions, since this produces a continuous and differentiable surface that allows for updating the parameters in order to reach a minimum region.

To see how the minimum is reached, it is better to start in a very simplified case where the network has only one parameter $\theta$. Consider a case where the 1-dimensional loss surface assumes the form shown in \autoref{fig:1d_loss}.
\begin{figure}
    \centering
    \caption{Example of a one dimensional loss surface}
    \includegraphics[width=0.5\textwidth]{chapters/NeuralNets/figures/1D-Loss.pdf}
    \fonte{From the author (2021)}
    \label{fig:1d_loss}
\end{figure}

Consider that the network initially starts with the parameter $\theta_0$ and loss $J_0$ at the dot shown in \autoref{fig:1d_loss}. To reduce the loss is to update the parameter $\theta$ in the opposite direction of the surface derivative. This can be shown by the following derivation.
\begin{align}
    J_0 + \Delta J & < J_0 \nonumber \\
    J_0 + \Delta\theta \frac{dJ}{d\theta} & < J_0 \nonumber \\
    \Delta\theta \frac{dJ}{d\theta} & < 0 \nonumber \\
    \sign(\Delta\theta) & = -\sign\left( \frac{dJ}{d\theta} \right) \label{eq:delta_theta_sign}
\end{align}

This derivation above only holds true for small values of $\Delta\theta$, since the derivative is calculated on infinitesimal small values, bigger changes run the risk of overshooting the region where the derivative approximation is reasonable.
\autoref{eq:delta_theta_sign} shows that the parameter change must be in the opposite direction of the change in the loss, the parameter updates can then be written as shown in \autoref{eq:1d_update}.
\begin{equation} \label{eq:1d_update}
    \theta_{i+1} = \theta_{i} + \Delta\theta = \theta_{i} - \eta \frac{dJ}{d\theta}
\end{equation}

\glsunset{learning_rate}
The value \gls{learning_rate} is a positive real number that determines how big are the steps taken when updating the parameters. This value is also a hyperparameter and is called the \textit{learning rate} of the network. A small learning rate is more precise but can make the training very slow, while bigger values are faster, but run the risk of overshooting and zigzaging around the target, or even worse, diverging and never reaching the result. \autoref{fig:1d_weight_update} shows how an initial parameter value is updated when using different values for $\eta$, it can be seen that there is a critical value $\eta_{crit}$ that is able to minimize the loss in a single step, while smaller and bigger values will converge to the minimum in multiple steps, and much bigger values will diverge.
\begin{figure}
    \centering
    \caption{Parameter updates for 1D loss surface in function of learning rate}
    \includegraphics[width=0.5\textwidth]{chapters/NeuralNets/figures/1D-Weight-Update.pdf}
    \fonte{From the author (2021)}
    \label{fig:1d_weight_update}
\end{figure}

For one dimension it is relatively simple to find the critical learning rate, but for very high dimensional problems this is unfeasible or even impossible since a critical value for one parameter is not necessarily the same for all other parameters.

In practice, it is usually necessary to experiment with some values in order to find what best suits the loss surface of the problem, it is also common to employ some sort of dynamic update that slowly reduces the learning rate, allowing for big steps in the beginning while being more precise towards the end.

\subsection{Gradient Descent} \label{sub:gradient_descent}
The same derivation used to obtain \autoref{eq:delta_theta_sign} can be applied to generalize the minimization procedure to higher dimensional parameter spaces, the only difference is to consider the partial derivatives for each parameter. Doing this would reveal that, like in the 1-dimensional case, the change in a parameter $\theta^{(j)}$ must be in the opposite direction of the partial derivative of the loss with respect to this parameter. The parameter update would then be given by \autoref{eq:parameter_update}.
\begin{equation} \label{eq:parameter_update}
    \theta_{i+1}^{(j)} = \theta_{i}^{(j)} - \eta \frac{\partial J}{\partial \theta^{(j)}}
\end{equation}

There is however another way to look at these updates. Recall that the gradient of any scalar function is a vector field composed of all its partial derivatives, so the gradient of the loss function is the vector field given by \autoref{eq:gradient}.
\begin{equation} \label{eq:gradient}
    \renewcommand{\arraystretch}{1.2}
    \nabla_{\bm{\theta}} J = \begin{bmatrix}
        \frac{\partial J}{\partial\theta^{(1)}} \\
        \frac{\partial J}{\partial\theta^{(2)}} \\
        \vdots \\
        \frac{\partial J}{\partial\theta^{(n)}}
    \end{bmatrix}
\end{equation}

Also recall that for any function $f$, its gradient at any given point is a vector that points towards the direction of steepest increase to the function at that point and the negative of this vector points to the steepest decrease \cite[Chapter~4.6]{calculusIII2016}. So the negative of the gradient $\nabla_{\bm{\theta}} J$ gives the direction to update all parameters in order to most reduce the loss function, this is what was being shown element-wise in \autoref{eq:parameter_update}. By grouping the derivatives it is possible to write the parameter updates in a much cleaner way.
\begin{equation} \label{eq:gradient_descent}
    \bm{\theta}_{i+1} = \bm{\theta}_{i} - \eta \nabla_{\bm{\theta}} J(\bm{\theta}_i)
\end{equation}

By repeating the process of calculating the gradient of the loss function and then updating all the network's parameters (given a sufficiently small learning rate), then eventually all parameters will converge to values that minimize the loss function. This is what is referred to as \textit{learning} in a neural network and this iterative process is called \textit{Gradient Descent}, since the steps are taken using the gradient in order to decrease the loss function \textbf{--} for maximization, the only difference is to update in the positive direction of the gradient and this is called \textit{Gradient Ascent}.

One important thing that was left unmentioned is the fact that this algorithm will almost surely converge to a local minimum in the loss surface instead of the global minimum. In the example shown for the 1-dimensional case the surface was very simple, with a single minimum value to converge, but even in these very simple cases it is possible to find surfaces much more complex (e.g. the curve $y = x + 2\sin x$ has infinitely many local minima but no global minimum). For the very high dimensional spaces and very complex geometries of the loss surfaces in neural networks, there will be many valleys where gradient descent will converge, but rarely those will be the optimal solution. The local minimum found will depend on the starting point in parameter space, and there is usually no better to way than just randomly selecting a point\footnote{
    This does not mean that the starting values are picked without any rhyme or reason, the starting point can have a huge impact on the network and it is very important to choose good values. However, initialization techniques are concerned with finding a random distribution with the right values of mean and variance to sample from. The point here is explicitly about the random nature of the starting point and not that this nature has no thought behind it.
}.

So if gradient descent will almost never reach the optimal solution, why would anyone use it? How can someone make sure that the local minimum found is good enough for the problem? Or if someone is trying to build a neural network and the results are not being good, how can this person know if this is caused by a legitimate problem on the network or the dataset, instead of simply being the fact that the training process was unlucky and found a bad local minimum? And perhaps most important, how is it possible that neural networks are achieving so much ground-breaking results while relying on gradient descent to learn?

This is indeed a worry that many researchers in the field have, but there are many options to combat this problem, it can be cited regularization techniques (e.g. $\ell_1$ and $\ell_2$ norms \textbf{--} see \autoref{sub:regularization}) and optimizers (e.g. \glsunset{SGD}\gls{SGD}, momentum, RMSProp, \glsunset{Adam}\gls{Adam} \textbf{--} see \autoref{sub:optimizers}) as examples.

Besides that, \textcite{lossSurfaces2014} argue that global minimum are not necessarily the best option, since they have a high chance of overfitting to the data (see \autoref{sub:overfitting}), they also empirically verify that for big enough neural networks most local minima are very similar to one another and have high quality. Their conclusion is that in practice it is not worth to strive for the global minimum, given that for large networks local minima have high quality and may even generalize better to unseen data.


\documentclass{article}
\author{QianLiu}
\title{Insert Equation}
\date{\today}

\usepackage{amsmath}

\begin{document}
    \maketitle
    \tableofcontents

    \section{inline Equation}
    \subsection{dollar character}
    $f(x)=x^2+3x$

    \subsection{parenthesis}
    \(f(x)=x^2+3x\)

    \subsection{mathematical enviroment}
    \begin{math}
        f(x)=x^2+3x
    \end{math}

    \section{super/sub script}
    \subsection{superscript}
    $f(x)=x^2+3y^{20}$

    \subsection{subscript}
    \(f(x)=x^2+3x_{20}\)

    \section{greek alphabet}
    \subsection{lowercase}
    \(\alpha\)
    \(\beta\)
    \(\omega_1\)
    \(\gamma\)
    \(\pi\)

    \subsection{uppercase}
    \(\Gamma\)
    \(\Delta\)
    \(\Pi^5\)
    \(\Omega\)
    \ \(\alpha^2+\beta^3=\omega\)

    \section{mathematical functions}
    \(\log_{10}^2 x\)
    \ \(\sin^2 x\)
    \ \(\cos^3 y\)
    \ \(\arcsin x\)
    \ \(\arccos z\)
    \ \(\ln x\)
    \ \(\sqrt{3x^2+2y^4_2}\)
    \ \(\sqrt[5]{3x}\)

    \section{algebraic fraction}
    \(3/4\)
    \ \(\frac{x}{y}\)
    \ \(\frac{\sqrt{2y_2^5}}{\sqrt[6]{\sin^2 x+\arcsin z_4^2}}\)

    \section{equation between lines}

    use \$\$...\$\$ insert equation
    $$f(x)=3x+y^3$$

    \subsection{square brackets}
    use \textbackslash [equation...\textbackslash ] insert equation
    \[f(x)=3x+y^3\] % used commonly to insert equation between lines

    \subsection{environment of displaymath}
    \begin{displaymath}
        f(x)=3x+y^3
    \end{displaymath}

    \subsection{environment where equation enumerated automatically}
    Refer the equation \ref{equation_1}
    \begin{equation}
        f(x)=3x+z^3
        \label{equation_1}
    \end{equation}

    \subsection{environment where equation* are not enumerated }
    Refer the equation \ref{equation_2}
    \begin{equation*} %equation without numbers
        f(x)=3x+z^6
        \label{equation_2}
    \end{equation*}

    \subsection{Equation in Multiple lines}
    \begin{gather} % Need to use package <amsmath>
        f(x)=3x+z^3  \\
        \notag \\
        \frac{\sqrt{2y_2^5}}{\sqrt[6]{\sin^2 x+\arcsin z_4^2}} \notag \\
        f(z)=\frac{x_2^5}{y_3^7}
    \end{gather}

    \subsection{Split Equation}
    \begin{equation} % Single Equation but can split in multiple lines
        \begin{split} % Use \begin{split}...\end{split} to realize it
            f(x)=3x+4x^2 \\
            f(x)=\sin^2 x \\
            f(x)=\cos x^3
        \end{split}
    \end{equation}

    \subsection{Piecewise funtion}
    \begin{equation}
        D(x)=\begin{cases}
            3x & \text{if }  x \in R \\
            x^2 & \text{if } x \in Z \setminus Q
        \end{cases}
    \end{equation}

\end{document}


\section{PAC Learning}

% ===
\begin{highlightbox}
    \textbf{Want:} Distribution indep. error guarantees!
    % What we want: Distribution independent error guarantees!
    % However, without prior knowledge, those bounds are quite loose.
\end{highlightbox}

Expec./\textbf{Gener. error:}\enskip
$\mathcal R(\hat c_n) = \bm P_{X,Y}(\hat c_n(x) \neq c(x))$

\textbf{Empirical error:}\enskip
$\hat{\mathcal R}_n(\hat c_n) = \frac1n \sum_{i=1}^n \bm 1_{\brace{\hat c_n(x_i) \;\neq\; y_i}}$

\emph{PAC learnable:} $\mathcal A$ can learn a concept class $\mathcal C$ from $\mathcal H$ if, given a \textcolor{Green}{suff. large sample}, it outputs a hypothesis that \textcolor{VioletRed}{generalizes well} \textcolor{OrangeRed}{w/ high prob}.

\begin{highlightbox}
\vspace{-\fboxrule}\vspace{-\fboxsep}
    \begin{highlightbox}[white]
        (1) $\color{VioletRed} 0 < \epsilon < \sfrac12$, $\color{OrangeRed} 0 < \delta < \sfrac12$,
        (2) $\bm P_{X,Y}$ on $\mathcal X \times \brace{0,1}$:

        If\enskip $\color{Green} n \geq \mathit{poly} (\textcolor{VioletRed}{\sfrac1\epsilon},\; \textcolor{OrangeRed}{\sfrac1\delta},\; \mathit{dim}(\mathcal X))$,

        Then\enskip $\color{OrangeRed} \bm P_{X,Y}
        \paren*{ \color{VioletRed}
            \mathcal R(\hat c_n) - \inf\limits_{c\in\mathcal C} \mathcal R(c) \leq \epsilon
        } \geq 1 - \delta$.
    \end{highlightbox}%
\vspace{-\fboxrule}\vspace{-\fboxsep}
\end{highlightbox}

% A concept class $\mathcal C$ is \emph{PAC learnable} from a hypothesis class $\mathcal H$ if \ldots\ldots\ldots
If $\mathcal A$ runs in time polynomial in $\color{VioletRed} \sfrac1\epsilon$ and $\color{OrangeRed} \sfrac1\delta$, we say that $\mathcal C$ is \emph{efficiently PAC learnable}.

% ===
\subsection{VC Inequality
\hfill $\bm P(\cdots \geq \begingroup\color{VioletRed}\epsilon\endgroup) \leq \ldots \leq \color{OrangeRed}\delta$}

Select ERM:\enskip
$\hat c_n^\ast = \arg\min_{c\in\mathcal C} \hat{\mathcal R}_n (c)$

Under uniform convergence: %, the suboptimality of $\hat c_n^\ast$ can be bounded:
\scalebox{.9}{${
    \bm P \paren*{
        \mathcal R(\hat c_n^\ast) - \inf\limits_{c\in\mathcal C} \mathcal R(c) > \epsilon
    } \leq \bm P \paren*{
        \sup\limits_{c\in\mathcal C} \abs*{\hat{\mathcal R}_n(c) - \mathcal R(c)} > \dfrac\epsilon2
    }
}$}

\begin{tabular}{@{\textbullet\:} r @{$\:\leq\:$} l @{}}
    \emph{$\abs{\mathcal C}$ Finite:}\enskip
    $\bm P \paren*{ \sup \abs*{\cdots} > \epsilon }$
    & $2 \abs{\mathcal C} \exp(-2n\epsilon^2)$
    \\
    \emph{$\abs{\mathcal C}$ Unbounded:}\hfill
    $\bm P \paren*{ \cdots }$
    & $9 n^{\mathrm{VC}_{\mathcal C}} \exp(-\frac{n \epsilon^2}{32})$
\end{tabular}

% ===
\subsection{Rectangle Learning}

$\bm P(\mathcal(\hat c_n^\ast > \epsilon)
\leq \abs{\mathcal C} \cdot (1-\epsilon)^n
\leq \abs{\mathcal C} \cdot \exp(-n\epsilon)
\color{gray} < \delta$

\textbf{Union bound:}\enskip
$\bm P(\bigcup_i T_i) \leq \sum_i \bm P(T_i)$

% ===


\section{Algebraic Moment Closures}
\label{sec:algebraicClosure}

The two-moment model given by Eq.~\eqref{eq:angularMoments} is not closed because of the appearance of the second moments $\vect{\cK}$ (the normalized pressure tensor).
Algebraic moment closures for the two-moment model are computationally efficient as they provide the Eddington factor in Eq.~\eqref{eq:eddingtonTensor} in closed form as a function of the density $\cJ$ and the flux factor $h=|\vect{\cH}|/\cJ$.
For this reason they are used in applications where transport plays an important role, but where limited computational resources preclude the use of higher fidelity models.
Examples include simulation of neutrino transport in core-collapse supernovae \cite{roberts_etal_2016} and compact binary mergers \cite{foucart_etal_2015}.
Algebraic moment closures in the context of these aforementioned applications have also been discussed elsewhere (e.g., \cite{janka_etal_1992,pons_etal_2000,smit_etal_2000,just_etal_2015,murchikova_etal_2017}).
Here we focus on properties of the algebraic closures that are critical to the development of numerical methods for the two-moment model of fermion transport.
For the algebraic closures we consider, the Eddington factor in Eq.~\eqref{eq:eddingtonTensor} can be written in the following form \cite{cernohorskyBludman_1994}
\begin{equation}
  \chi(\cJ,h)=\f{1}{3}+\f{2\,(1-\cJ)\,(1-2\cJ)}{3}\,\Theta\Big(\f{h}{1-\cJ}\Big),
  \label{eq:eddingtonFactor}
\end{equation}
where the \emph{closure function} $\Theta(x)$ depends on the specifics of the closure procedure.
We will consider two basic closure procedures in more detail below: the maximum entropy (ME) closure and the Kershaw (K) closure.

In the low occupancy limit ($\cJ\ll1$), the Eddington factor in Eq.~\eqref{eq:eddingtonFactor} depends solely on $h$; i.e.,
\begin{equation}
  \chi(\cJ,h)\to\chi_{0}(h)=\f{1}{3}+\f{2}{3}\,\Theta\big(h\big).
  \label{eq:eddingtonFactorLow}
\end{equation}
This for of $\chi$ yields a moment closure that is suitable for particle systems obeying Maxwell-Boltzmann statistics.

\subsection{Maximum Entropy (ME) Closure}

The ME closure constructs an approximation of the angular distribution as a function of $\cJ$ and $\vect{\cH}$ \cite{cernohorskyBludman_1994,lareckiBanach_2011}.
The ME distribution $f_{\mbox{\tiny ME}}$ is found by maximizing the entropy functional, which for particles obeying Fermi-Dirac statistics is given by
\begin{equation}
  S[f_{\mbox{\tiny ME}}]
  = \int_{\bbS^{2}}\big[\,(1-f_{\mbox{\tiny ME}})\log(1-f_{\mbox{\tiny ME}}) + f_{\mbox{\tiny ME}}\log f_{\mbox{\tiny ME}}\,]\,d\omega,
  \label{eq:entropyFunctional}
\end{equation}
subject to the constraints
\begin{equation}
  \f{1}{4\pi}\int_{\bbS^{2}}f_{\mbox{\tiny ME}}(\omega)\,d\omega=\cJ
  \quad\text{and}\quad
  \f{1}{4\pi}\int_{\bbS^{2}}f_{\mbox{\tiny ME}}(\omega)\,\vect{\ell}(\omega)\,d\omega=\vect{\cH}.
  \label{eq:closureConstraints}
\end{equation}
The solution that maximizes Eq.~\eqref{eq:entropyFunctional} takes the general form \cite{cernohorskyBludman_1994}
\begin{equation}
  f_{\mbox{\tiny ME}}(\omega;a,\vect{b})=\f{1}{e^{a + \vect{b}\cdot\vect{\ell}(\omega)}+1},
  \label{eq:fME}
\end{equation}
where the Lagrange multipliers $a$ and $\vect{b}$ are implicit functions of $\cJ$ and $\vect{\cH}$.
The ME distribution function satisfies $0 < f_{\mbox{\tiny ME}} < 1$, but $a$ and $\vect{b}$ are unconstrained.
Specification of $a$ and $\vect{b}$ from $\vect{\cM}=(\cJ,\vect{\cH})^{T}$ gives $f_{\mbox{\tiny ME}}$, and any number of moments can in principle be computed.
Importantly, for the maximum entropy problem to be solvable, we must have $\vect{\cM}\in\cR$ \cite{lareckiBanach_2011}.

To arrive at an algebraic form of the ME closure, Cernohorsky \& Bludman \cite{cernohorskyBludman_1994} postulate (but see \cite{lareckiBanach_2011}) that, as a function of the flux saturation
\begin{equation}
  x := h/(1-\cJ),
  \label{eq:fluxSaturation}
\end{equation}
the closure function $\Theta$ is independent of $\cJ$ and can be written explicitly in terms of the inverse Langevin function.
To avoid inverting the Langevin function for $\Theta$, they provide a polynomial fit (accurate to $2\%$) given by
\begin{equation}
  \Theta_{\mbox{\tiny ME}}^{\mbox{\tiny CB}}(x)
  =\f{1}{5}\,\big(\,3-x+3\,x^{2}\,\big)\,x^{2}.
  \label{eq:closureMECB}
\end{equation}
More recently, Larecki \& Banach \cite{lareckiBanach_2011} have shown that the explicit expression given in \cite{cernohorskyBludman_1994} is not exact and provide another approximate expression
\begin{equation}
  \Theta_{\mbox{\tiny ME}}^{\mbox{\tiny BL}}(x)
  =\f{1}{8}\,\big(\,9\,x^{2}-5+\sqrt{33\,x^{4}-42\,x^{2}+25}\,\big),
  \label{eq:closureMEBL}
\end{equation}
which is accurate to within $0.35\%$.
On the interval $x\in[0,1]$, the curves given by Eqs.~\eqref{eq:closureMECB} and \eqref{eq:closureMEBL} lie practically on top of each other.
The closure functions given by Eqs.~\eqref{eq:closureMECB} and \eqref{eq:closureMEBL}, together with the Eddington factor in Eq.~\eqref{eq:eddingtonFactor} and the pressure tensor in Eq~\eqref{eq:eddingtonTensor}, constitute the algebraic maximum entropy closures for fermionic particle systems considered in this paper.
We will refer to the ME closures with $\Theta_{\mbox{\tiny ME}}^{\mbox{\tiny CB}}$ and $\Theta_{\mbox{\tiny ME}}^{\mbox{\tiny BL}}$ as the CB (Cernohorsky \& Bludman) and BL (Banach \& Larecki) closures, respectively.

We also note that using the closure function given by Eq.~\eqref{eq:closureMECB} with the low occupancy Eddington factor in Eq~\eqref{eq:eddingtonFactorLow} results in the algebraic maximum entropy closure attributed to Minerbo \cite{minerbo_1978}, which is currently in use in simulation of neutrino (fermion) transport in the aforementioned nuclear astrophysics applications.
In a recent comparison of algebraic (or analytic) closures for the two-moment model applied to neutrino transport around proto-neutron stars, Murchikova et al. \cite{murchikova_etal_2017} obtained nearly identical results when using the closures of CB and Minerbo.
For these reasons, we include the Minerbo closure in the subsequent discussion and in the numerical tests in Section~\ref{sec:numerical}.

\subsection{Kershaw (K) Closure}

Another algebraic closure we consider is a Kershaw-type closure \cite{kershaw_1976}, developed for fermion particle systems in \cite{banachLarecki_2017a}.
The basic principle of the Kershaw closure for the two-moment model is derived from the fact that the realizable set generated by the triplet of scalar moments
\begin{equation}
  \{\cJ,\cH,\cK\}=\f{1}{2}\int_{-1}^{1}f(\mu)\,\mu^{\{0,1,2\}}\,d\mu,
  \label{eq:scalarMoments}
\end{equation}
is convex.
For the moments in Eq.~\eqref{eq:scalarMoments}, the realizable set is the set of moments obtained from distribution functions satisfying $0<f(\mu)<1,\,\forall\mu\in[-1,1]$.
(The moments in Eq.~\eqref{eq:scalarMoments} are the unique moments obtained from the moments in Eq.~\eqref{eq:angularMoments} under the assumption that the distribution function is isotropic about a preferred direction, and $\mu$ is the cosine of the angle between this preferred direction and the particle propagation direction given by $\vect{\ell}$.)

For a bounded distribution $0<f<1$, it is possible to show (e.g., \cite{banachLarecki_2013}) that the second moment satisfies
\begin{equation}
  \cK_{\mbox{\tiny L}}(\cJ,h) < \cK < \cK_{\mbox{\tiny U}}(\cJ,h),
\end{equation}
where $\cK_{\mbox{\tiny L}}=\cJ\,\big(\,\f{1}{3}\,\cJ^{2}+h^{2}\,\big)$, $\cK_{\mbox{\tiny U}}=\cK_{\mbox{\tiny L}} + \cJ\,(1-\cJ)\,(1-x^{2})$, and $x$ is the flux saturation defined in Eq.~\eqref{eq:fluxSaturation}.
By convexity of the realizable set generated by the moments in Eq.~\eqref{eq:scalarMoments}, the convex combination
\begin{equation}
  \cK(\beta,\cJ,h)=\beta\,\cK_{\mbox{\tiny L}}(\cJ,h)+(1-\beta)\,\cK_{\mbox{\tiny U}}(\cJ,h),
  \label{eq:kershawAnsatz}
\end{equation}
with $\beta\in[0,1]$, is realizable whenever $(\cJ,\cH)^{T}\in\cR$.
The Kershaw closure for the two-moment model is then obtained from Eq.~\eqref{eq:kershawAnsatz} with the additional requirement that it be correct in the limit of isotropic distribution functions; i.e., $\cK(\beta,\cJ,0)=\cJ/3$.
One choice for $\beta$, which leads to a strictly hyperbolic and causal two-moment model (and a particularly simple closure function) \cite{banachLarecki_2017a}, is $\beta=(2-\cJ)/3$, so that $\cK_{\mbox{\tiny K}}(\cJ,h)=\chi_{\mbox{\tiny K}}(\cJ,h)\,\cJ$, where
\begin{equation}
  \chi_{\mbox{\tiny K}}(\cJ,h)=\f{1}{3}+\f{2\,(1-\cJ)\,(1-2\cJ)}{3}\,\Theta_{\mbox{\tiny K}}\Big(\f{h}{1-\cJ}\Big),
  \label{eq:eddingtonFactorKershaw}
\end{equation}
and the Kershaw closure function is given by
\begin{equation}
  \Theta_{\mbox{\tiny K}}(x)=x^{2}.
\end{equation}
\modified{For multidimensional problems, the Kershaw closure is obtained by inserting the Eddington factor $\chi_{\mbox{\tiny K}}$ in Eq.~\eqref{eq:eddingtonFactorKershaw} into the Eddington tensor in Eq.~\eqref{eq:eddingtonTensor}.}
Finally, we point out that for the two-moment Kershaw closure (see \cite{banachLarecki_2017a} for details), a distribution function $f_{\mbox{\tiny K}}(\omega,\cJ,\vect{\cH})$, satisfying $0 < f_{\mbox{\tiny K}} < 1$, and reproducing the moments $\cJ$, $\vect{\cH}$, and $\vect{\cK}$, can be written explicitly in terms of Heaviside functions.

\subsection{Realizability of Algebraic Moment Closures}

It is not immediately obvious that all the algebraic moment closures discussed above are suitable for designing realizability-preserving methods for the two-moment model of fermion transport.
In particular, the realizability-preserving scheme developed in this paper is based on the result in Lemma~\ref{lem:explicitStep}, which must hold for the adapted closure.
The Kershaw closure is consistent with a bounded distribution, $f_{_{\mbox{\tiny K}}}\in(0,1)$, and should be well suited, but the algebraic ME closures are based on approximations to the closure function, and we need to consider if these approximate closures remain consistent with the assumed bounds on the underlying distribution function.
To this end, we rely on results in \cite{levermore_1984,lareckiBanach_2011} (see also \cite{kershaw_1976,shohatTamarkin_1943}), which state that realizability of the moment triplet $\{\cJ,\vect{\cH},\vect{\cK}\}$ (with $\vect{\cK}$ given by Eq.~\eqref{eq:eddingtonTensor}), is equivalent to the following requirement for the Eddington factor
\begin{equation}
  \chi_{\mbox{\tiny min}}
  =\max\big(1-\f{2}{3\cJ},h^{2}\big)
  <\chi<\min\big(1,\f{1}{3\cJ}-\f{\cJ}{1-\cJ}h^{2}\big)=\chi_{\mbox{\tiny max}}.
  \label{eq:eddingtonFactorBounds}
\end{equation}
Fortunately, these bounds are satisfied by the algebraic closures based on Fermi-Dirac statistics.
(Note that for $\cJ\ll1$ the bounds in Eq.~\eqref{eq:eddingtonFactorBounds} limit to the bounds for positive distributions given by Levermore \cite{levermore_1984}; i.e., $h^{2}<\chi<1$.)

In Figure~\ref{fig:EddingtonFactorsWithDifferentClosure}, we plot the Eddington factor $\chi$ versus the flux factor $h$ for the various algebraic closures discussed above and for different values of $\cJ\in(0,1)$: $0.01$ (upper left panel), $0.4$ (upper right panel), $0.6$ (lower left panel), and $0.99$ (lower right panel).
The lower and upper bounds on the Eddington factor for realizable closures ($\chi_{\mbox{\tiny min}}$ and $\chi_{\mbox{\tiny max}}$, respectively) are also plotted.
We note that for all the closures, the Eddington factor $\chi\to1/3$ as $h\to0^{+}$.

When $\cJ=0.01$, the maximum entropy closures (CB, BL, and Minerbo) are practically indistinguishable, while the Eddington factor of the Kershaw closure is larger than that of the other closures over most of the domain.
When $\cJ=0.4$, the Eddington factor for the closures based on Fermi-Dirac statistics (CB, BL, and Kershaw) remain close together, while the Eddington factor for the Minerbo closure is larger than the other closures for $h\gtrsim0.2$.
The Eddington factor for all closures remain between $\chi_{\mbox{\tiny min}}$ and $\chi_{\mbox{\tiny max}}$ when $\cJ=0.01$ and $\cJ=0.4$.

When $\cJ=0.6$, the Eddington factor for the closures based on Fermi-Dirac statistics remain close together and within the bounds in Eq.~\eqref{eq:eddingtonFactorBounds}.
The dependence of the Eddington factor on $h$ for the Minerbo closure differs from the other closures (i.e., increases vs decreases with increasing $h$), and exceeds $\chi_{\mbox{\tiny max}}$ for $h\gtrsim0.34$.
When $\cJ=0.99$, the Eddington factor of the CB and BL closures (indistinguishable) and the Kershaw closure remain within the bounds given in Eq.~\eqref{eq:eddingtonFactorBounds}.
The Eddington factor of the Minerbo closure is nearly flat, and exceeds $\chi_{\mbox{\tiny max}}$ for $h\gtrsim0.006$.

\begin{figure}[H]
  \centering
  \begin{tabular}{cc}
    \includegraphics[width=0.5\textwidth]{figures/Closures0_01}
    \includegraphics[width=0.5\textwidth]{figures/Closures0_40} \\
    \includegraphics[width=0.5\textwidth]{figures/Closures0_60}
    \includegraphics[width=0.5\textwidth]{figures/Closures0_99}
  \end{tabular}
   \caption{Plot of Eddington factors $\chi$ versus flux factor $h$ for different values of $\cJ$ for various algebraic closures: $\cJ=0.01$ (upper left panel), $\cJ=0.4$ (upper right panel), $\cJ=0.6$ (lower left panel), and $\cJ=0.99$ (lower right panel).  In each panel we plot the Eddington factors of Kershaw (solid blue lines), Cernohorsky \& Bludman (CB, solid red lines), Banach \& Larecki (BL, dashed orange lines), and Minerbo (dash-dot purple lines).  We also plot $\chi_{\mbox{\tiny min}}$ and $\chi_{\mbox{\tiny max}}$ defined in Eq.~\eqref{eq:eddingtonFactorBounds} (lower and upper solid black lines, respectively).}
  \label{fig:EddingtonFactorsWithDifferentClosure}
\end{figure}
We have also checked numerically that for all the algebraic closures based on Fermi-Dirac statistics (CB, BL, and Kershaw), the bounds on the Eddington factor in Eq.~\eqref{eq:eddingtonFactorBounds} holds for all $\vect{\cM}\in\cR$.
Thus, we conclude that these closures are suited for development of realizability-preserving numerical methods for the two-moment model of fermion transport.

In Figure~\ref{fig:MabWithDifferentClosure}, we further illustrate properties of the algebraic closures by plotting $\vect{\cM}_{ab}$ as defined in Lemma~\ref{lem:explicitStep} for the maximum entropy closures of CB and Minerbo.
In both panels, we plot $\vect{\cM}_{ab}$ constructed from randomly selected pairs $\vect{\cM}_{a},\vect{\cM}_{b}\in\cR$ (each blue dot represents one realization of $\vect{\cM}_{ab}$).
Results for the maximum entropy closure of CB are plotted in the left panel, while results for the Minerbo closure are plotted in the right panel.
As expected for the closure consistent with moments of Fermi-Dirac distributions (CB), we find $\vect{\cM}_{ab}\in\cR$.
For the Minerbo closure, which is consistent with positive distributions, $\vect{\cM}_{ab}$ is not confined to $\cR$.
\begin{figure}[H]
  \centering
  \begin{tabular}{cc}
%    \includegraphics[width=0.5\textwidth]{figures/MabWithBLME}
    \includegraphics[width=0.5\textwidth]{figures/MabWithCBME}
%    \includegraphics[width=0.5\textwidth]{figures/MabWithBLKS}
    \includegraphics[width=0.5\textwidth]{figures/MabWithMI}
  \end{tabular}
   \caption{Illustration of $\vect{\cM}_{ab}$, as defined in Lemma~\ref{lem:explicitStep}, computed with algebraic maximum entropy closures of Cernohorsky \& Bludman (left) and Minerbo (right).  In each panel, $\vect{\cM}_{ab}$ was computed using the respective closure, using $10^{6}$ random pairs ($\vect{\cM}_{a},\vect{\cM}_{b}\in\cR$), and plotted as a light-blue point.  The solid black lines mark the boundary of $\cR$: $\gamma(\vect{\cM}) = 0$.}
  \label{fig:MabWithDifferentClosure}
\end{figure}


\section{The MST Problem}	\label{section:mst-problem}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Minimum Spanning Tree}
  $G = (V, E)$: connected, undirected, weighted graph ($w(e)$)

  \importikznocaption{0.40\textwidth}{0.80\textwidth}{tikz-in-beamer/spanning-tree-example-overlay.tex}

  \pause
  \vspace{0.30cm}

  Spanning tree $T = (V, E' \subseteq E)$: connected, acyclic \pause ($\Rightarrow n-1$ edges)

  \vspace{0.20cm}
  \uncover<6->{
	\[
	  w(T) = \sum\limits_{e \in E'} w(e)
	\]
  }
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Minimum Spanning Tree}
  % \begin{center}
  %   MST: Mimimize $w(T)$ over all possible STs
  % \end{center}

  \[
	\text{MST:} \quad \mathop{\mathrm{arg\,min}}_{T} \; w(T)
  \]

  \importikznocaption{0.60\textwidth}{0.80\textwidth}{tikz-in-beamer/mst-example-overlay.tex}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{A Simple Property}
  \fignocaption{width = 0.45\textwidth}{figs/mst-simple-property.pdf}

  \begin{center}
	\textcolor{red}{\bf Cut:} $V = (S, V \setminus S)$
  \end{center}

  \pause

  \begin{enumerate}
	\item MST in each connected component
	\item $ce$: \textcolor{brown}{a} \textcolor{red}{\bf lightest} edge across cut
  \end{enumerate}

  \pause

  \begin{center}
	\textcolor{brown}{\bf Copy\&Paste Argument; Exchange Argument}
  \end{center}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{A Wrong Divide \& Conquer Algorithm}
  \begin{description}
    \item[Input:] $G = (V, E, w)$
    \item[Divide:] $V = (S, V \setminus S)$; $\large| |S| - |V \setminus S| \large| \le 1$
    \item<2->[Conquer:] $T_1$: an MST of $S$; $T_2$: an MST of $V \setminus S$
    \item<3->[Combine:] $T_{1} + T_{2} + \set{e}$: $e$ is a lightest edge across $(S, V \setminus S)$
  \end{description}

  \pause
  \vspace{0.50cm}

  \fignocaption{width = 0.30\textwidth}{figs/divide-conquer-mst-counterexample.pdf}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{A Wrong Algorithm}
  \begin{alertblock}{What is wrong?}
	\fignocaption{width = 0.30\textwidth}{figs/divide-conquer-mst-counterexample.pdf}

	\centerline{\textcolor{brown}{The edges \textcolor{blue}{$bc$ and $ad$} do \textcolor{red}{\bf not} belong to any MST.}}
  \end{alertblock}

  \pause
  \vspace{0.50cm}

  \begin{alertblock}{What if:}
	\centerline{\textcolor{red}{Invariant:} Manages a set of edges $X$ which is a subset of \textcolor{red}{\bf some} MST.}
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\chapter{The algorithm}
\label{chap:algorithm}

\textbf{by Bastian Boll} \\

The training dataset consists of approximately 560h of subtitled audio. From this, we extract MFCC features and approximate timestamps for each single word. For each word $w$ we aim to define a $\mathcal{C}^1$ function \(D_w\colon \R \to \lbrack 0, \infty)\) such that $D_w(t)$ is large iff the audio around time $t$ sounds like the word $w$. For a given transcript $\mathcal{T}$ and audio $A$, we can find the function $f$ introduced in chapter \ref{chap:intro} as the solution to the minimization problem
\begin{align*}
	&\min_x -\sum_{j=1}^l D_{w_j}(x_j)\\
	&\hspace{0.5em}\text{s.t. }x_1\leq x_2\leq \dotsc\leq x_l
\end{align*}

\section{Defining the distance metric}

\textbf{by Bastian Boll} \\

The functions \(D_w\colon \R \to \lbrack 0, \infty)\) can be seen as equivalent to a metric for the distance between the sound of the word $w$ and the audio $A$ at $t$. They need to be constructed in a way such that
\begin{enumerate}
	\item $D_w$ is guaranteed to be continuously differentiable w.r.t. $t$ for every word $w$.
	\item $D_w$ corresponds to the probability, that the word $w$ is said as time $t$ in $A$. More specifically, the function mapping said probability $p$ to $D_w(t)$ should be monotonically increasing.
\end{enumerate}

One important motivating factor behind the construction of this optimization objective is, that the dataset does not contain exact time points for each word. Instead, for any given word $w$, we assume that the difference between the time point in the dataset $t_\text{data}$ and the actual time point in the audio at which $w$ is being said $t_\text{true}$ follows a normal distribution with mean $0$ and constant standard deviation $\sigma$.

Divide the audio into intervals of equal size. Label each interval $I_i$ with the probability of it containing $w$. This can be done by a variety of possible machine learning methods, our specific approach is discussed in section \ref{interval_word_prob}. These probabilities $p_i$ can be used to infer an observed probability distribution of where in the audio the machine learning model expects $w$ to be located. More precisely, because the same word might appear multiple times, we assume the $p_i$ to be generated by the sum of multiple normal distributions with different means (time points) and constant standard deviation $\sigma$. We also assume, that the specific instance $w$ of the word in question can be attributed to exactly one of these distributions with mean $t$.\\
In order to find the optimal time point $t_\text{true}$ we can use the observation, that two normal distributions with the same standard deviation have the same mean $t=t_\text{true}$ exactly if it holds for the probability density functions $\rho_{t,\sigma}$ and $\rho_{t_\text{true},\sigma}$ that
\[t = \arg\max \int_\R \rho_{t_\text{true},\sigma}(x)\rho_{t,\sigma}(x)\text{d}x\]
Hence, we can use the maximum value on the right hand side as a score for how similar the audio around time $t$ the model considers the word $w$ to be. This can be transformed into an objective in terms of the probabilities $p_i$ computed by the machine learning model
\begin{align*}
	\int_\R \rho_{t_\text{true},\sigma}(x)\rho_{t,\sigma}(x)\text{d}x &= \sum_{i} \int_{I_i} \rho_{t_\text{true},\sigma}(x)\rho_{t,\sigma}(x)\text{d}x\\
			&= \sum_i \rho_{t,\sigma}(\xi_i) \underbrace{\int_{I_i} \rho_{t_\text{true},\sigma}(x)\text{d}x}_{\approx p_i}
\end{align*}
by the mean value theorem for a sufficient $\xi_i\in I_i$ (because $\rho_{t_\text{true},\sigma}(x) > 0$ for all $x$). Equality still approximately holds if we set $\xi_i$ to be the center point of $I_i$ for sufficiently small intervals.
Hence, we can define $D_w$ through
\[
	D_w(t) = \sum_i p_i \rho_{t,\sigma}(t_i)
\]
for each center point $t_i$ of $I_i$. This construction satisfies the smoothness required for optimization. It also considers the inherent imprecision of word time points in the dataset and provides a means of leveraging information computed by the machine learning model without requiring perfect prediction accuracy.\\
Additionally, by interpreting the time alignment as a global optimization problem under word order constraints, we can make effective use of the known word order and reduce the impact of single (local) bad predictions by the model.

\section{Computing probabilities for each interval}
\label{interval_word_prob}

\textbf{by Paul Warkentin} \\

Given an audio file, we divide its MFCC features into $n$ intervals $I_i$ of equal lengths $l$. If the rest part of the data $I_{n+1}$ is shorter than the length $l$, it is discarded. The result of this section will be a matrix $p$ of shape $\lbrack l, c \rbrack$, where $c$ is the fixed number of words we selected for evaluation.

Since we interpolated the timing of each word in chapter \ref{data_preparation}, we can iterate for all talks through each word $w_j$ with its estimated timing $t_j$. Calculate the cumulative distribution function $\Phi_{t_j, 0.8}(x)$ with a mean value of $t_j$ and a standard deviation of 0.8. \\
The current word $w_j$ lies in the interval with index $k = \lceil \tfrac{t_j}{l} \rceil$. For the interval $I_k$ and its neighbors $\{I_{k-2}, I_{k-1}, I_{k+1}, I_{k+2}\}$, do the cumulative mapping
\[p_{j, s} \mapsto p_{j, s} + \Phi_{t_j, 0.8}((s - 1) l) - \Phi_{t_j, 0.8}(s l),\]
for $s = k-2, k-1, \dots, k+2$. \\

Hence, we got the features and its labels for the supervised learning model described in the section below. \\
The MFCC features for an audio file are reshaped into a matrix of shape $\lbrack l, m \rbrack$, where $m$ is the number of MFCC samples for an interval of length $l$.

\section{Defining the models}

\textbf{by Paul Warkentin} \\

% source for deep convnet: https://yerevann.github.io/2016/06/26/combining-cnn-and-rnn-for-spoken-language-identification/

For the main speech recognition part, we trained several supervised models. The following three delivered the best results.

\subsection*{Simple convolutional neural network}
This model is derived from  TensorFlows' convolutional neural network for speech recognition in the example \textit{speech\_commands}\footnote{\url{https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech\_commands/models.py}}.
\begin{figure}[h]
	\centering
	\makebox[\textwidth][c]{
		\begin{tabular}{clccc||clccc}
			\hline \textbf{Nr} & \textbf{Type} & \textbf{\#Ch.} & \textbf{Wd.} & \textbf{Hgt.} & \textbf{Nr} & \textbf{Type} & \textbf{\#Ch.} & \textbf{Wd.} & \textbf{Hgt.} \\ \hline
			0 & Input & 1 & 80 & 13 & 3 & Conv2D & 64 & 4 & 10 \\
			1 & Conv2D & 64 & 8 & 20 & & Bias & 64 & 4 & 10 \\
			& Bias & 64 & 8 & 20 & & ReLU & 64 & 4 & 10 \\
			& ReLU & 64 & 8 & 20 & & Dropout & 64 & 4 & 10 \\
			& Dropout & 64 & 8 & 20 & 4 & MatMul & 1500 & & \\
			& MaxPool & 1 & 2 & 1 & & Bias & 1500 & & \\
			2 & Conv2D & 64 & 4 & 10 & & & & & \\
			& Bias & 64 & 4 & 10 & & & & & \\
			& ReLU & 64 & 4 & 10 & & & & & \\
			& Dropout & 64 & 4 & 10 & & & & & \\
			& MaxPool & 1 & 2 & 1 & & & & & \\ \hline
		\end{tabular}
	}
	\caption[Simple convolutional structure]{Graph structure of simple convolutional neural network.}
\end{figure}
\FloatBarrier

\subsection*{Deep convolutional neural network}
This model is a combination of six consecutive convolutional neural networks, each followed by activation. The output is also here flattened and will then be encoded by two dense layers into the label space. We got the idea by the blog post\footnote{\url{https://yerevann.github.io/2016/06/26/combining-cnn-and-rnn-for-spoken-language-identification/}} by Harutyunyan and Khachatrian as they did reach an accuracy of 99.24\%.
\begin{figure}[!htbp]
	\centering
	\makebox[\textwidth][c]{
		\begin{tabular}{clccc||clccc}
			\hline \textbf{Nr} & \textbf{Type} & \textbf{\#Ch.} & \textbf{Wd.} & \textbf{Hgt.} & \textbf{Nr} & \textbf{Type} & \textbf{\#Ch.} & \textbf{Wd.} & \textbf{Hgt.} \\ \hline
			0 & Input & 1 & 80 & 13 & 4 & Conv2D & 32 & 3 & 3 \\
			1 & Conv2D & 16 & 26 & 10 & & Bias & 32 & 3 & 3 \\
			& Bias & 16 & 26 & 10 & & ReLU & 32 & 3 & 3 \\
			& ReLU & 16 & 26 & 10 & 5 & Conv2D & 32 & 3 & 3 \\
			& MaxPool & 1 & 2 & 1 & & Bias & 32 & 3 & 3 \\
			2 & Conv2D & 16 & 13 & 5 & & ReLU & 32 & 3 & 3 \\
			& Bias & 16 & 13 & 5 & 6 & Conv2D & 32 & 3 & 3 \\
			& ReLU & 16 & 13 & 5 & & Bias & 32 & 3 & 3 \\
			& Dropout & 16 & & & & ReLU & 32 & 3 & 3 \\
			& MaxPool & 1 & 2 & 1 & 7 & Dense & 1000 & & \\
			3 & Conv2D & 32 & 5 & 5 & & Sigmoid & 1000 & & \\
			& Bias & 32 & 5 & 5 & 8 & Dense & 1500 & & \\
			& ReLU & 32 & 5 & 5 & & Sigmoid & 1500 & & \\ \hline
		\end{tabular}
	}
	\caption[Deep convolutional structure]{Graph structure of deep convolutional neural network.}
\end{figure}
\FloatBarrier

\subsection*{Convolutional LSTM neural network}
This model is a combination of a convolutional neural network and a LSTM network. The input vector will be put into a dynamic RNN cell of type \newline\textit{tf.contrib.rnn.Conv2DLSTMCell} in TensorFlow. Its output is then flattened and will be encoded by two dense layers into the label space.
\begin{figure}[h]
	\centering
	\begin{tabular}{clccc}
		\hline \textbf{Nr} & \textbf{Type} & \textbf{\#Channels} & \textbf{Width} & \textbf{Height} \\ \hline
		0 & Input & 1 & 80 & 13 \\
		1 & Conv2DLSTM & 1 & 80 & 13 \\
		2 & Dense & 1000 & & \\
		& Sigmoid & 1000 & & \\
		3 & Dense & 1500 & & \\
		& Sigmoid & 1500 & & \\ \hline
	\end{tabular}
	\caption[Convolutional LSTM structure]{Graph structure of convolutional LSTM neural network.}
\end{figure}
\FloatBarrier


\section{(Sub-)Groups with prime power order}
\subsection{Elementary Properties}
\begin{definition}
    Let $G$ be a group.
    We say $G$ is a $p$-group for a prime $p$ if $|G|=p^n$ for some $n\in\mathbb N$.
\end{definition}
\begin{theorem}
    Let $G$ be a $p$-group, then $Z(G)\neq 1$.
\end{theorem}
\begin{proof}
    Consider the partition of $G$ by conjugacy classes, which are either $1$ or divisible by $p$.
    Note that $Z(G)$ is the union of all $1$-classes.
    But if $|Z(G)|=1$, then
    $$0\equiv p^n=|G|=1+\sum_{\exists g\notin Z(G),C=\operatorname{ccl}_G(g)}|C|\equiv 1\pmod{p}$$
    which is a contradiction.
\end{proof}
In particular $|Z(G)|\equiv 0\pmod{p}$.
\begin{corollary}[Classification of Simple $p$-groups]
    If $G$ is a simple $p$-group, then $G\cong C_p$.
\end{corollary}
\begin{proof}
    $1\neq Z(G)\unlhd G$.
    Since $G$ is simple we must have $Z(G)=G$, hence $G$ is abelian, but then necessarily $G\cong C_p$ since $G$ cannot have any proper subgroup to be simple.
\end{proof}
\begin{corollary}
    Let $G$ be a $p$-group of order $p^n$, then $G$ contains an element of order $p^r$ for any $0\le r\le n$.
\end{corollary}
\begin{proof}
    Consider the composition series of $G$
    $$1=G_0\lhd G_1\lhd\cdots\lhd G_m=G$$
    Where $G_{i+1}/G_i$ is simple.
    But it is an $p$-group, so we must have $G_{i+1}/G_i\cong C_p$ for any $i$.
    The claim follows.
\end{proof}
\begin{lemma}
    For $G$ a group.
    If $G/Z(G)$ is cyclic, then $G$ is abelian.
\end{lemma}
\begin{proof}
    Let $gZ(G)$ be a generator of $G/Z(G)$.
    Then every element of $G$ is of the form $p^iz$ where $z\in Z$.
    But for $z,z'\in Z(G)$, $g^izg^jz'=g^{i+j}zz'=g^jz'g^iz$, so $G$ is abelian.
\end{proof}
\begin{corollary}
    If $G$ has order $p^2$ for a prime $p$, then $G$ is abelian.
\end{corollary}
\begin{proof}
    We already know that $Z(G)\neq 1$.
    For $|Z(G)|=p$ then we have $G/Z(G)\cong C_p$, so $G$ is abelian by the preceding lemma, contradiction.
    For $|Z(G)|=p^2$ we have $Z(G)=G$, which means that $G$ is abelian.
    There are no other possibilities, so the proof is done.
\end{proof}
Sadly (or not) there are nonabelian groups of order $p^3$.
\subsection{Sylow's Theorems}
\begin{theorem}[Sylow's Theorems]
    Let $G$ be a finite group with order $p^am$ where $p$ is a prime and $p\nmid m$.
    Then\\
    1. The set $\operatorname{Syl}_p(G)=\{P\le G:|P|=p^a\}$ is nonempty.\\
    2. All elements of $\operatorname{Syl}_p(G)$ are conjugate.\\
    3. The number $n_p=|\operatorname{Syl}_p(G)|$ satisfies $n_p\equiv 1\pmod{p}$ and $n_p|m$.
\end{theorem}
\begin{corollary}
    If $n_p=1$, then there is a normal Sylow $p$-subgroup.
\end{corollary}
\begin{proof}
    Let $g\in G$ and $P$ be a Sylow $p$-subgroup of $G$.
    But then $gPg^{-1}\in\operatorname{Syl}_p(G)=\{P\}$, hence $gPg^{-1}=P\implies P\unlhd G$.
\end{proof}
\begin{example}
    There is no simple group of order $1000$.
    Suppose $G$ is a group of order $1000=2^35^3$, then $n_5\equiv 1\pmod{5}$ and $n_5|8$, but then we must have $n_5=1$.
    So there is a normal Sylow $5$-subgroup of $G$ which is obviously not the identity or $G$, hence $G$ is not normal.
\end{example}
\begin{proof}
    1. Consider the action of the group $G$ on the set $\Omega$ of all subsets of $G$ of size $p^a$ by $g\star X=\{gx:x\in X\}$, so
    $$|\Omega|=\binom{p^am}{p^a}\not\equiv 0\pmod{p}$$
    Hence this action has an orbit, say the orbit of $X\in\Omega$, which order is not a multiple of $p$.
    Then $|G_X||\operatorname{orb}_G(X)|=|G|=p^am$, so $p^a||G_X|$.
    On the other hand, $\bigcup_{g\in G}g\star X=G$, so $|G|\le |\operatorname{orb}_G(X)||X|$, so $|G_X|=|G|/|\operatorname{orb}_G(X)|\le |X|=p^a$, but $|G_X|\ge p^a$, therefore $|G_X|=p^a$.\\
    2. We shall prove a stronger statement: suppose $P\in\operatorname{Syl}_p(G)$ and $Q\le G$ a $p$-subgroup, then $Q\le gPg^{-1}$ for some $g\in G$.
    Consider the action of $Q$ on the set of left cosets $G/P$ by left multiplication.
    By orbit-stabiliser, any orbit divides $|Q|$, so its size must be either $1$ or a multiple of $p$.
    But $|G/P|=m$ which is coprime to $p$, hence there is at least one orbit $\operatorname{orb}_Q(gP)$ of size $1$, so for any $q\in Q$, $g^{-1}qg\in P\implies Q\le gPg^{-1}$.\\
    3. Let $G$ act transitively (by 2) on $\operatorname{Syl}_p(G)$ by conjugation.
    So by orbit-stabiliser, $n_p||G|$, so it suffices to show $n_p\equiv 1\pmod{p}$.
    Now let $P\in\operatorname{Syl}_p(G)$ and consider its action on $\operatorname{Syl}_p(G)$ by conjugation.
    Now the orbits divides $|P|=p^a$, so is either $1$ or divisible by $p$.
    We shall show that there is exactly one orbit of size $1$, which will establish the theorem.
    There is at least one orbit, namely $\operatorname{orb}_P(P)$.
    If $\operatorname{orb}_P(Q)$ is also an orbit of size $1$, then $P\le N_G(Q)$.
    Now $P,Q$ are Sylow $p$-subgroups of $N_G(Q)$, so they are conjugate by 2, therefore there is some $g\in N_G(Q)$ with $Q=gQg^{-1}=P$.
    The theorem is hence proved.
\end{proof}
\begin{example}
    Suppose $G$ is a simple group, then $|G|\neq 132$.
    Assume there is a simple group of order $132=2^2\cdot 3\cdot 11$, then by Sylow's Third Theorem, $n_3=1,4,22$ and $n_{11}=1,12$, but by simplicity neither of them is $1$, so $n_{11}=12$
    If $n_3=4$, then letting $G$ act on $\operatorname{Syl}_3(G)$ by conjugation gives a group homomorphism $G\to S_4$, but its kernel is not all $G$, so $G$ is isomorphic to a subgroup of $S_4$, but $|S_4|=24<132$, contradiction.\\
    So $n_3=22$.
    Now the Sylow $3$-subgroups are all of order $3$, this means that there are $22\times (3-1)=44$ elements of order $3$.
    Similarly $n_{11}=12$ gives $(11-1)\times 12=120$ elements of order $11$, but then $132=|G|\ge 120+44+1=165$, contradiction.
\end{example}
For sake of problem sheets, we mention the following definition.
\begin{definition}
    An automorphism $\operatorname{Aut}(G)\le\operatorname{Sym}G$ of a group $G$ is the group of all isomorphisms from $G$ to itself.
\end{definition}


\subsection{Convolutional Neural Network}
\label{subsec:cnn}
% \todo[inline]{Citations}

Convolutional neural networks are a type of deep learning architecture that is primarily used to classify images.
They are based on artificial neural networks and the mathematical convolution operation.
The efficiency in image classification of CNNs is largely responsible for the reputation of deep learning \cite{cnn_pathmind}.

The main problems when working with color images is that they are high-dimensional and therefore require a lot of processing power.
For this reason, convolutional neural networks use special layers to detect features and reduce the amount of data.
The three different layers used are convolution layers, pooling layers and fully-connected layers.
% An example of a CNN sequence to classify handwritten digits is shown in figure \ref{fig:cnn}.
Figure \ref{fig:cnn} shows an example of a CNN sequence to classify handwritten digits \cite{cnn_pathmind}.

\paragraph{Convolution Layer}
The convolution layer performs the convolution operations using so-called kernels or filters.
The goal of the convolution operation is to extract certain features --- such as edges --- from the image.
Different kernels are used to detect different features.
The first convolution layers are responsible for capturing low-level features, while following layers are responsible for capturing high-level features \cite{cnn_tds}.
\clearpage

\paragraph{Pooling Layer}
Pooling layers reduce the spatial size of the convolved features.
Pooling is done by only looking at the portion of the image masked by the kernel.
Maximum pooling yields the maximum value and average pooling yields the average value of the masked portion.
By doing so, the required computational power is significantly decreases and the dominant features are extracted \cite{cnn_tds}.
\\

\paragraph{Fully-Connected Layer}
The fully-connected layers are responsible for learning the non-linear combinations of the high-level convolved features.
They flatten the result of the convolution process into a vector of values.
Each value corresponds to the probability that a certain feature belongs to a label \cite{cnn_tds}.
% Each value corresponds to a probability that a certain feature belongs to a label \cite{cnn_tds}.
% Each value corresponds to a probability that a certain feature belongs to a certain label \cite{cnn_tds}.

\begin{figure}[t]
  \centering
  \includegraphics[width=\textwidth]{cnn}
  \caption{Example of a convolutional neural network \cite{cnn_tds}}
  \label{fig:cnn}
\end{figure}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Shell Method}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises for using the shell method.
\end{abstract}
\maketitle

\begin{exercise}
The region defined by the inequalities $\sqrt{1-x^2} \leq y \leq 1$ for $0 \leq y \leq 1$ is revolved around the $y$-axis. Compute the volume of the resulting solid using the shell method.
\begin{center}
\begin{image}
\includegraphics{shell/shell03.png}
\end{image}
\end{center}
\begin{itemize}
\item When the slicing variable is $x$, the radius of a shell is the \wordChoice{\choice[correct]{horizontal}\choice{vertical}} distance from an $x$-slice to the axis of rotation. Thus
\[ r(x) = \answer{x} - \answer{0}. \]
\item The height of an $x$-slice is equal to
\[ h(x) = \answer{1 - \sqrt{1 - x^2}}. \]
\item The volume is equal to the integral of $2 \pi r h$, so
\[ V = \int_{\answer{0}}^{\answer{1}} \answer{2 \pi x ( 1 - \sqrt{1-x^2}) } dx = \answer{\frac{\pi}{3}}. \]
(Note: to compute the integral, split it into two parts and make the substitution $u = 1-x^2$ for one of them.)
\end{itemize}
\end{exercise}

\begin{exercise}
The region in the plane bounded above by the graph $y = \sqrt{1+x^2}$, below by $y = -1 + x + \sqrt{1+x^2}$, and on the left by $x = 0$ is revolved around the axis $x = 1$. Compute the volume of the resulting solid using the shell method.
\begin{center}
\begin{image}
\includegraphics{shell/shell04.png}
\end{image}
\end{center}
\begin{itemize}
\item When the slicing variable is $x$, the radius of a shell is the \wordChoice{\choice[correct]{horizontal}\choice{vertical}} distance from an $x$-slice to the axis $x = 0$. Thus
\[ r(x) = \answer{1} - \answer{x}. \]
\item The height of an $x$-slice is equal to
\[ h(x) = \answer{-1+x}. \]
\item The volume is equal to the integral of $2 \pi r h$, so
\[ V = \int_{\answer{-1}}^{\answer{1}} \answer{2 \pi (1-x)^2 } dx = \answer{\frac{16\pi}{3}}. \]
\end{itemize}
\end{exercise}

\begin{exercise}
The region in the plane $y = \sqrt{x}$, $y=0$, and $x = 1$ is revolved around the $y$-axis. Use the shell method to compute the volume.
\[ V = \answer{\frac{4 \pi}{5}}. \]
\end{exercise}

\begin{exercise}
The same region as above (bounded by $y = \sqrt{x}$, $y=0$, and $x = 1$) is revolved around the axis $x=1$. Use the shell method to compute the volume.
\[ V = \answer{\frac{8 \pi}{15}}. \]
\end{exercise}

\begin{exercise}
The same region as above  (bounded by $y = \sqrt{x}$, $y=0$, and $x = 1$) is revolved around the $x$-axis. Use the shell method to compute the volume.
\begin{hint}
The ``height'' of a shell is $1-y^2$ in this case.
\end{hint}
\[ V = \answer{\frac{\pi}{2}}. \]
\end{exercise}

\begin{exercise}
For the same region as above (bounded by $y = \sqrt{x}$, $y=0$, and $x = 1$), use the shell method to compute the volume when revolved around the axis $y = 1$.
\[ V = \answer{\frac{5 \pi}{6}}. \]
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[ShellQuad001]

The region in the plane bounded below by the curve \(y=-x^2\), above by the curve \(y=x^2+2x+2\), on the right by the line  \(x = 0\), and on the left by the line \(x = -2\) is revolved around the axis \(x = -2\). Compute the volume of the resulting solid.
(Hints won't reveal until after you choose a response.)
\begin{multiplechoice}
\choice{\(4\pi\)}
\choice{\(6\pi\)}
\choice[correct]{\(8\pi\)}
\choice{\(10\pi\)}
\choice{\(12\pi\)}
\choice{\(14\pi\)}
\end{multiplechoice}
\begin{feedback}
The axis \(x = -2\) is parallel to the direction of slices using the integration variable \(x\), which indicates the shell method.
 The region lies to the right of the axis, which must be the case because the interval \(-2 \leq x \leq 0\) lies to the right of the axis \(x = -2\).
 \begin{hint}
The integral to compute equals \[ \begin{aligned} V &= \int_{-2}^{0}2 \pi (x-(-2))((x^2+2x+2)-(-x^2))~ dx\\
& = \pi \int_{-2}^{0} (4x^3+12x^2+12x+8)~ dx\\
& = \pi \left. \left(x^4+4x^3+6x^2+8x\right) \right|_{-2}^{0} = 8\pi. \end{aligned}\]
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[ShellQuad019]

The region in the plane bounded on the left by the curve \(x=-y^2+4y+1\), on the right by the curve \(x=y^2+2y+1\), and below by the line \(y = -1\) is revolved around the axis \(y = -1\). Compute the volume of the resulting solid.
(Hints won't reveal until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(\pi\)}
\choice{\(5\pi\)}
\choice{\(9\pi\)}
\choice{\(13\pi\)}
\choice{\(17\pi\)}
\choice{\(21\pi\)}
\end{multiplechoice}
\begin{feedback}
The axis \(y = -1\) is parallel to the direction of slices using the integration variable \(y\), which indicates the shell method. The lower endpoint of integration will be \(y = -1\); the upper endpoint can be determined by setting \(-y^2+4y+1 = y^2+2y+1\) and choosing the solution which is greater than \(-1\). This gives the range \(-1 \leq y \leq 0\).
 The region lies above the axis, which must be the case because the interval \(-1 \leq y \leq 0\) lies above the axis \(y = -1\).
 \begin{hint}
The integral to compute equals \[ \begin{aligned} V &= \int_{-1}^{0}2 \pi (y-(-1))((y^2+2y+1)-(-y^2+4y+1))~ dy\\
& = \pi \int_{-1}^{0} (4y^3-4y)~ dy\\
& = \pi \left. \left(y^4-2y^2\right) \right|_{-1}^{0} = 1\pi. \end{aligned}\]
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[shellalgsub2001]

The region in the plane between the  \(x\)-axis and the graph
\[ y = \frac{1}{2 \sqrt{\frac{x^{2}}{3} + 1}} \]
 in the range \(0 \leq x \leq 3\) is revolved around the axis \(x = 0\). Compute the volume of the resulting solid.
 (Hints won't reveal until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle \frac{3}{2} \pi\)}
\choice{\(\displaystyle \frac{8}{5} \pi\)}
\choice{\(\displaystyle \frac{5}{3} \pi\)}
\choice{\(\displaystyle 2 \pi\)}
\choice[correct]{\(\displaystyle 3 \pi\)}
\choice{\(\displaystyle 5 \pi\)}
\end{multiplechoice}
\begin{feedback}
If the variable \(x\) is used for slicing, then slices are parallel to the axis of rotation, which indicates the shell method should be used.
The radius of a shell is \(x\). The height of a shell is exactly \(\frac{1}{2 \sqrt{\frac{x^{2}}{3} + 1}}\).
\begin{hint}
The volume of the region is therefore given by
\[ \int_{0}^{3} \frac{\sqrt{3} \pi x}{\sqrt{x^{2} + 3}}\, dx. \]
\begin{hint}
 To compute the integral we can use the substitution \(u = x^{2} + 3\) which implies the equality \(du = \left(2 x\right)dx\) for the differentials. This gives the equality
\[ \begin{aligned} \int \frac{\sqrt{3} \pi x}{\sqrt{x^{2} + 3}}\, dx & = \int \frac{\sqrt{3} \pi}{2 \sqrt{u}}\, du \\
 & = \sqrt{3} \pi \sqrt{u}. \end{aligned} \]
Reversing the substitution gives
\[ \begin{aligned} \int_{0}^{3} \frac{\sqrt{3} \pi x}{\sqrt{x^{2} + 3}}\, dx & = \left. \left[\sqrt{3} \pi \sqrt{x^{2} + 3} \right] \right|_{0}^{3}\\ & = \left(6 \pi \right) - \left(3 \pi \right) = 3 \pi. \end{aligned} \]
\end{hint}
\end{hint}
\end{feedback}

\end{question}

\end{document}


\hypertarget{matrix}{%
\section{Matrix}\label{matrix}}

The Matrix class provides support for matrices. A matrix can be
initialized with a given size,

\begin{lstlisting}
var a = Matrix(nrows,ncols)
\end{lstlisting}

where all elements are initially set to zero. Alternatively, a matrix
can be created from an array,

\begin{lstlisting}
var a = Matrix([[1,2], [3,4]])
\end{lstlisting}

You can create a column vector like this,

\begin{lstlisting}
var v = Matrix([1,2])
\end{lstlisting}

Once a matrix is created, you can use all the regular arithmetic
operators with matrix operands, e.g.

\begin{lstlisting}
a+b
a*b
\end{lstlisting}

The division operator is used to solve a linear system, e.g.

\begin{lstlisting}
var a = Matrix([[1,2],[3,4]])
var b = Matrix([1,2])

print b/a
\end{lstlisting}

yields the solution to the system a*x = b.


\gotosection{1}{4}
\subsection{The geometry of $\mathbb{R}^n$}

\begin{exercise}{3}
  \begin{enumerate}
    \item Let $\Vect{x} = \begin{bmatrix}0\\1\\4\end{bmatrix}$,
          $\frac{\Vect{x}}{|\Vect{x}|}=
           \frac{\Vect{x}}{\sqrt{17}}=
           \begin{bmatrix}0\\\frac{1}{\sqrt{17}}\\\frac{4}{\sqrt{17}}\end{bmatrix}$

    \item Let $\Vect{x} = \begin{bmatrix}-3\\7\end{bmatrix}$,
          $\frac{\Vect{x}}{|\Vect{x}|}=
           \frac{\Vect{x}}{\sqrt{58}}=
           \begin{bmatrix}-\frac{3}{\sqrt{58}}\\\frac{7}{\sqrt{58}}\end{bmatrix}$

    \item Let $\Vect{x} = \begin{bmatrix}\sqrt{2}\\-2\\-5\end{bmatrix}$,
          $\frac{\Vect{x}}{|\Vect{x}|}=
           \frac{\Vect{x}}{\sqrt{31}}=
           \begin{bmatrix}
             \sqrt{\frac{2}{31}} \\
             -\frac{2}{\sqrt{31}} \\
             -\frac{5}{\sqrt{31}}
           \end{bmatrix}$
  \end{enumerate}
\end{exercise}

\begin{exercise}{5}
  \begin{enumerate}
    \item $cos(\theta) = \frac{1}{\sqrt{3}}$, $\theta = \arccos{1}{\sqrt{3}}$.
    \item $cos(\theta) = 0$, $\theta = \frac{\pi}{2}$.
  \end{enumerate}
\end{exercise}

\begin{exercise}{7}
  \begin{enumerate}
    \item $\det(M) = 1$, $M^{-1} = \begin{xmatrix}0&1\\-1&2\end{xmatrix}$.
    \item $\det(M) = 0$, $M^{-1}$ doesn't exist.
    \item $\det(M) = ad$, if $a, d \neq 0$, $M^{-1} = \frac{1}{ad}\begin{xmatrix}d&-b\\0&a\end{xmatrix}$.
    \item $\det(M) = 0$, $M^{-1}$ doesn't exist.
  \end{enumerate}
\end{exercise}

\begin{exercise}{8}
  \begin{enumerate}
    \item -4
    \item $adf$
    \item $adg - cbg$
  \end{enumerate}
\end{exercise}

\begin{exercise}{10}
  \begin{enumerate}
    \item This can be proven inductively: $k = 1$, $|A^1| = |A| = |A|^1$. $k = 2$, $|A^2| = |A \cdot A| \leq |A||A| = |A|^2$, from Cauchy-Schwarz's inequality. $k > 2$, $|A^k| = |A^{k-1} \cdot A| \leq |A^{k-1}||A| \leq |A|^k$. \rQED
          $$|A^3| = |\begin{xmatrix}7&10\\5&7\end{xmatrix}| = \sqrt{223}$$
          $$|A|^3 = (\sqrt{7})^3 = 7\sqrt{7}$$

    \item $|\Vect{u} \cdot \Vect{v}| = |\Vect{u}||\Vect{v}|$, because they are in opposite direction, namely $\Vect{u}$ is on the line spanned from $\Vect{v}$.

          $|\Vect{u} \cdot \Vect{w}| < |\Vect{u}||\Vect{w}|$, because the two vectors are not in the same or opposite directions, namely one is not on the line spanned from the other.

          $|\Vect{u} \cdot \Vect{v}| = 2 + 8 + 18 = 28 = \sqrt{14} \cdot \sqrt{56} = |\Vect{u}||\Vect{v}|$.

          $|\Vect{u} \cdot \Vect{w}| = 2 + 18 = 20$, $|\Vect{u}||\Vect{w}| = \sqrt{14} \cdot \sqrt{40} = 4\sqrt{35} > 20 = |\Vect{u} \cdot \Vect{w}|$.

    \item Let $\Vect{u} = \begin{bmatrix}-v_2\\v_1\end{bmatrix}$, namely rotating $\Vect{v}$ by $\frac{\pi}{2}$ counterclockwise. $\Vect{u} \cdot \Vect{w} = -v_2w_1 + v_1w_2 < 0$. Therefore, the angle between $\Vect{u}$ and $\Vect{w}$ is larger than $\frac{\pi}{2}$, which means $\Vect{w}$ lies clockwise from $\Vect{v}$.

    \item Let $\theta$ be the angle between $\Vect{v}$ and $\Vect{w}$. $\cos(\theta) = \frac{\Vect{v} \cdot \Vect{w}}{|\Vect{v}||\Vect{w}|}$, $\cos(\theta)|\Vect{v}||\Vect{w}| = 42$.

          If $\cos(\theta) = 1$, $|\Vect{v}||\Vect{w}| = 42$, or $|\Vect{w}| = 3\sqrt{14}$. This is the shortest $\Vect{w}$. As $\cos(\theta)$ approaches $0$, $|\Vect{w}|$ will approach $\infty$, so there is no longest $\Vect{w}$.
  \end{enumerate}
\end{exercise}

\begin{exercise}{13}
  First, we can prove that for any
    $\Vect{a} = \begin{xmatrix}a_1\\a_2\\a_3\end{xmatrix},
     \Vect{b} = \begin{xmatrix}b_1\\b_2\\b_3\end{xmatrix}$ and $x, y \in \mathbb{R}$, $x\Vect{a} \times y\Vect{b} = xy(\Vect{a} \times \Vect{b})$:

  $$x\Vect{a} \times y\Vect{b} =
    \begin{xmatrix}
       xa_2yb_3 - xa_3yb_2 \\
      -xa_1yb_3 + xa_3yb_1 \\
       xa_1yb_2 - xa_2yb_1
    \end{xmatrix} = xy
    \begin{xmatrix}
       a_2b_3 - a_3b_2 \\
      -a_1b_3 + a_3b_1 \\
       a_1b_2 - a_2b_1
    \end{xmatrix} = xy(\Vect{a} \times \Vect{b})$$

  For two vectors $\Vect{v}_1$, $\Vect{v}_2$ pointing in the same direction, they can be written as $n_1\Vect{u}$ and $n_2\Vect{u}$, respectively. Let $u_1, u_2, u_3$ be the three components of $\Vect{u}$, we have:

  $$\Vect{v}_1 \times \Vect{v}_2 = (n_1n_2) \Vect{u} \times \Vect{u} = (n_1n_2)
    \begin{xmatrix}
       u_2u_3 - u_3u_2 \\
      -u_1u_3 + u_3u_1 \\
       u_1u_2 - u_2u_1
    \end{xmatrix} = (n_1n_2)\Vect{0} = \Vect{0}$$

  \rQED
\end{exercise}

\begin{exercise}{16}
  \begin{enumerate}
    \item $|Area| = |\det(\begin{xmatrix}1&5\\2&1\end{xmatrix})| = |1 - 10| = 9$
    \item $|Area| = |\det(\begin{xmatrix}1&5\\2&-1\end{xmatrix})| = |-1 - 10| = 11$
  \end{enumerate}
\end{exercise}

\begin{exercise}{19}
  \begin{enumerate}
    \item $|\Vect{v}_n| = \sqrt{1 + \hdots + 1} = \sqrt{n}$
    \item $\cos{a_n} = \frac{\Vect{v}_n \cdot \Vect{e}_1}{|\Vect{v}_n||\Vect{e}_1|} = \frac{1}{\sqrt{n}}$, so $a_n = \arccos{\frac{1}{\sqrt{n}}}$. $\lim_{n \to \infty} a_n = \arccos{\frac{1}{\infty}} = \frac{\pi}{2}$.
  \end{enumerate}
\end{exercise}

\begin{exercise}{24}
  \def \Set {\Vect{v}^{\bot}}
  \begin{enumerate}
    \item Since $\Vect{0} \cdot \Vect{v} = 0$, $\Vect{0} \in \Set$.

          For any $\Vect{x}, \Vect{y} \in \Set$, $(\Vect{x} + \Vect{y}) \cdot \Vect{v} = \Vect{x} \cdot \Vect{v} + \Vect{y} \cdot \Vect{v} = 0 + 0 = 0$. Therefore, $(\Vect{x} + \Vect{y}) \in \Set$.

          For $\Vect{x} \in \Set$ and $c \in \mathbb{R}$, $(c\Vect{x}) \cdot \Vect{v} = c(\Vect{x} \cdot \Vect{v}) = c0 = 0$. \rQED

    \item $(\Vect{a} - \frac{\Vect{a} \cdot \Vect{v}}{|\Vect{v}|^2}\Vect{v}) \cdot \Vect{v} =
            \Vect{a} \cdot \Vect{v} - \frac{\Vect{a}\cdot\Vect{v}}{|\Vect{v}|^2}\Vect{v} \cdot \Vect{v} =
            \Vect{a} \cdot \Vect{v}(1 - \frac{\Vect{v}^2}{|\Vect{v}|^2}) = 0$. \rQED

    \item (b) suggests the existence of $t(\Vect{a})$. Suppose that there exists a distinct $t'(\Vect{a})$ such that $(\Vect{a} + t'(\Vect{a})\Vect{v}) \in \Set$. Then we have:

         $$\left\{
         \begin{aligned}
            (\Vect{a} + t(\Vect{a})\Vect{v}) \cdot \Vect{v} = 0 \\
            (\Vect{a} + t'(\Vect{a})\Vect{v}) \cdot \Vect{v} = 0
         \end{aligned} \quad \Rightarrow \quad
         \begin{aligned}
            \Vect{a} \cdot \Vect{v} + t(\Vect{a})\Vect{v}^2 = 0 \\
            \Vect{a} \cdot \Vect{v} + t'(\Vect{a})\Vect{v}^2 = 0
         \end{aligned} \quad \Rightarrow \quad
         \begin{aligned}
            t(\Vect{a}) = -\frac{\Vect{a} \cdot \Vect{v}}{\Vect{v}^2} \\
            t'(\Vect{a}) = -\frac{\Vect{a} \cdot \Vect{v}}{\Vect{v}^2}
         \end{aligned}
         \right.$$

         Since there could be only one possible value for $-\frac{\Vect{a} \cdot \Vect{v}}{\Vect{v}^2}$, $t(\Vect{a}) = t'(\Vect{a})$, namely $t(\Vect{a})$ is unique.

        $$\Vect{a} + t(\Vect{a})\Vect{v} = \Vect{a} -\frac{\Vect{a} \cdot \Vect{v}}{\Vect{v}^2}\Vect{v} = \Vect{a} - \frac{\Vect{a} \cdot \Vect{v}}{|\Vect{v}|^2}\Vect{v} = P_{\Set}(\Vect{a})$$

        \rQED
  \end{enumerate}
\end{exercise}

\begin{exercise}{26}
  \begin{enumerate}
    \item $$A\begin{xmatrix}x\\y\end{xmatrix} = \begin{xmatrix}x-2y\\3x+4y\end{xmatrix}$$

          $$a\begin{pmatrix}x\\y\end{pmatrix} = \arccos{\frac{x^2+xy+4y^2} {\sqrt{10}\sqrt{(x^2+y^2) (x^2+2xy+2y^2)}}}$$

    \item Suppose there existing a nonzero vector such that it is rotated by $\frac{\pi}{2}$:

          $$\frac{x^2+xy+4y^2} {\sqrt{10}\sqrt{(x^2+y^2) (x^2+2xy+2y^2)}} = \cos(\frac{\pi}{2})$$
          $$x^2+xy+4y^2=0$$
          $$(\frac{1}{2}x+2y)^2=-\frac{3}{4}x^2$$

          Because $(\frac{1}{2}x+2y)^2 \geq 0$ and $-\frac{3}{4}x^2 \leq 0$, both sides must equal $0$. Therefore, $x = 0, y = 0$, which contradicts the premise of nonzero vector. As a result, such vector doesn't exist.
  \end{enumerate}
\end{exercise}


\pagebreak
\section{Experiment-2}
\textbf{Q.1 What would you expect the steady-state velocity of the system to be if a step input of 2000
N were applied at t = 0?}
\\
Ans. \\
From newtons second law we have, dv/dt = (F-bv)/M.
\\ M is the mass, v is velocity and F is force, b is damping cofficient.
\\
At steady state, the change in velocity will be zero. i.e. dv/dt = 0.
\\
Therefore, F = bv
\\F = 2000N
\\b = 40 N*sec/m
\\2000 = 40*v
\\v= 50 m/sec.

\textbf{Q.2 Why do this step input and the ramp input with saturation we simulated have the same steady state velocity?}
\\Ans.The step input is similar to saturated ramp input, because when steady state is reached, both step input and saturated ramp input
have fixed value(input value). Therefore, if we apply same force to same system (under idential condition),
then system will attain the same velocity. That is why we get same steady state velocity.

\textbf{Q.3 Using which input, the step or the ramp with saturation, would you expect the system to reach its steady-state velocity more quickly?
}
\\Ans. With step input the system will reach it's steady state velocity more quickly. Because,
saturated ramp input will take more time to reach it's saturation value whereas step input
reaches to it's fixed value quickly.


\newpage
\subsection{A Full Example}
\label{subsec:compss_example}
This section intends to give the reader a more or less extensive insight on what writing a COMPSs application is. We think that this section may help to \textit{materialize} concepts and will avoid to give this document an excessively abstract tone.\\
\\
Lets suppose that we want to approximate the value of $\pi$. For this purpose we have thought on a simple, randomized algorithm:
\begin{enumerate}
\item Generate $N$ random 2D points with coordinates between $-1$ and $1$
\item Consider the set of points $S$ within distance $1$ or less to the origin
\item Assume that $\frac{|S|}{N} = \frac{\pi}{4}$
\end{enumerate}

\begin{figure}[ht!]
\centering
\includegraphics[scale=0.5]{figures/circle_square.png}
\caption{A graphical representation of the random experiment. The square has side length $2$, so the circle has radius $1$, and therefore area $\pi$. In ratio terms, $\frac{\pi}{4}$ of the points belong to the circle}
\label{fig:circle_square}
\end{figure}

A more graphical explanation on why this works can be found in figure \ref{fig:circle_square}.\\
\\
We know a little bit of Python, so we have decided to implement this program in it. Basically, our small application will consist of a \verb|test_random_point| function that generates a random point and return $1$ if this point lies inside our circle, and $0$ otherwise. We will call this function $N$ times, and consider the proportion $\frac{|S|}{N}$ to be equal to $\frac{\pi}{4}$.
\inputminted{python}{applications/PI_SQUARE/sequential.py}
This code can be straightforward \textit{optimized} by transforming the \verb|test_random_point| function into a COMPSs task and syncing the results in the main procedure.
\inputminted{python}{applications/PI_SQUARE/pycompss_naive.py}
Although this may be a good approach to \textit{parallelize} this application we must note that we want to make it run in a distributed environment. The main difference we can appreciate is that a COMPSs task may run in a different machine than the master code, so some coordination between two processes in different machines and the transfer of potentially big amounts of data are necessary. In other words, the tradeoff between task granularity and performance is much more punishing in distributed computing than in single-machine parallel cases.\\
\\
Another important thing to note is that a distributed application can still exploit lower level parallelism in each of its tasks. In our case, we can transform our \verb|test_random_point| function into a \verb|test_random_points| procedure that generates and tests various random points at the same time.

\inputminted{python}{applications/PI_SQUARE/pycompss_vectorized.py}

This last approach is what we consider a well \textit{COMPSsfied} application: it has a reasonable task count and granularity, and it exploits various levels of parallelism at the same time. This application also delegates most of the work to \verb|numpy| procedures, which are mainly written in C++ and OpenMP. This aspect is also important in PyCOMPSs, as Python is, by nature, a very slow programming language and it should be only used as an orchestrator.

COMPSs is mainly designed to run in HPC environments. Most HPC machines integrate some sort of queue system to manage its resources among all the demanding users. Our previous example can be run as a job in a queue system with the following command:

\inputminted{bash}{applications/PI_SQUARE/run_mn4.sh}

The \verb|enqueue_compss| command refers to a generic queueing script (see section \ref{subsec:hpc_queues}) which translates our request to enqueue this COMPSs job to a specific queue system. Some of the most common parameters of a COMPSs job can be found in table \ref{table:compss_queue_param}.\\
\\
\begin{table}[ht!]
\centering
\begin{tabular}{|l|l|}
\hline
Argument name   & Description                                                                           \\ \hline
\verb|exec_time|      & Job time limit                                                                        \\ \hline
\verb|num_nodes|      & \begin{tabular}[c]{@{}l@{}}Number of computing\\ nodes\end{tabular}                   \\ \hline
\verb|cpus_per_node| & \begin{tabular}[c]{@{}l@{}}Number of cores per\\ computing node\end{tabular}          \\ \hline
\verb|constraints|     & \begin{tabular}[c]{@{}l@{}}Additional constraints\\ (e.g: highmem nodes)\end{tabular} \\ \hline
\end{tabular}
\caption{Some example configuration parameters of the queue system. These parameters are usually passed as flags to the enqueue\_compss script.}
\label{table:compss_queue_param}
\end{table}


\section{Background and Notation} \label{background_and_notation}

This section gives more background on the sliding window technique \cite{keogh2004sliding}, DTW distance measure \cite{keogh2002exact}, and time series normalization \cite{das1998rule}, which are fundamental building blocks of our conducted online gesture recognition study \cite{lesti2017filter}. Table \ref{tab:notation} introduces the notation that we use for formal problem description.

\begin{table}
    \begin{center}
        \begin{tabularx}{\textwidth}{c X}
            \hline
            \textbf{Symbol} \qquad & \textbf{Description}\\
            \hline
            $Q$ & a time series of size $n$ with $Q = (q_1, q_2, \dots, q_i, \dots, q_n)$\\
            $Q[i,j]$ & a subsequence time series of $Q$ with $Q[i,j] = (q_i, q_{i+1}, \dots, q_{j})$\\
            $t$ & the current time\\
            $\mu$ & the mean of a time series $Q$\\
            $\sigma$ & the standard deviation of a time series $Q$\\
            $\eta$, $z$  & two different time series normalizations\\
            \hline
        \end{tabularx}
    \end{center}
    \caption{Notation used for formal problem description.}
	\label{tab:notation}
\end{table}

\input{background_and_notation/sliding_window_technique.tex}
\input{background_and_notation/dynamic_time_warping.tex}