\chapter{Parameter Estimation}\label{ch:parameterEstimation}
In this chapter, we discuss our approach toward parameter estimation.
Specifically, we aim to address the problem of maximum likelihood.

\section{Intractable Likelihood Functions}\label{sec:intractableLikelihoodFunctions}
\begin{figure}[t]
    \centering{\input{include/floats/likelihood-general.tex}}
    \caption{General figure depicting a smooth likelihood distribution (blue).
    The maximum likelihood point of this distribution is the intersection of the black dashed line and the blue line.
    }\label{fig:likelihoodGeneral}
\end{figure}

We denote $\theta$ to be a set of parameters values, $\Theta$ to be the space all of our parameters values reside in,
$\mathcal{D}$ to be a collection of observed data, and $\mathcal{D}'$ to
be a collection of generated data.
The \emph{likelihood} of some parameter values $\theta \in \Theta$ is the joint probability that $\theta$ produces
some observed data $\mathcal{D}$.
Let $\mathcal{M},\mathcal{V} : \theta \mapsto \mathcal{D}'$ define a function that maps parameter values $\theta$ and random variables
$\mathcal{V}$ to generated data $\mathcal{D}'$.
Given that $\mathcal{V}$ is random, different $\mathcal{D}'$ may be returned for the same parameter values
$\theta$.
%In other words, $\mathcal{M}$ represents some stochastic simulator model which produces some random data.
Given \emph{discrete and countable} observations $\mathcal{D}$, the likelihood of parameter values $\theta \in \Theta$
is the probability that our generated data equals our observed data~\cite{lintusaariFundamentalsRecentDevelopments2017}.
\begin{equation}\label{eq:likelihood1}
    \mathcal{L}(\theta) = \Pr(\mathcal{D} = \mathcal{M}(\theta))
\end{equation}

One technique to calculate a maximum likelihood for our parameter values involves finding the critical points of
$\mathcal{L}$ (where $\frac{d\mathcal{L}}{d\theta} = 0$) and choosing the point $\hat{\theta}$ with the largest value
of $\mathcal{L}$.
In~\autoref{fig:likelihoodGeneral}, the maximum of such a likelihood is displayed at the intersection of the blue and
black dashed lines.
There are however, several problems with this approach:
\begin{enumerate}
    \item A large range $\mathcal{D}'$ for simulator $\mathcal{M}$ indicates that the frequency of exact observed --
        generated matches will be too low to make inferences.
    \item This assumes that we can explicitly express $\mathcal{L}$.
        For simulator based $\mathcal{M}$ like ours, this is not trivial to do.
\end{enumerate}
Consequently, we must look into other approaches to \textit{infer} $\mathcal{L}$.
The two problems this chapter aims to address are (a) how to efficiently compute $\mathcal{L}$ for a single $\theta$
and (b) how to infer $\mathcal{L}$ for all $\theta$.

\section{Approximate Bayesian Computation}\label{sec:approximateBayesianComputation}
In this section, we discuss an approximate method to compute the likelihood $\mathcal{L}$ for some parameter values
$\theta$.

As previously mentioned, to find $\mathcal{L}$ is to find the joint probability that our model and parameters produce
the observed data.
We expand~\autoref{eq:likelihood1} to make this explicit for $r$ sets of observed data
in~\autoref{eq:jointProbabilityLikelihood}.
$\theta$ remains the same, however the randomness of $\mathcal{V}$ may result in different probabilities.
For brevity, $\mathcal{V}$ is implicit in $\mathcal{M}$ from this point forward.
\begin{equation}\label{eq:jointProbabilityLikelihood}
    \mathcal{L}(\theta) = \prod_{i=1}^{r} \Pr\left(\mathcal{M}(\theta) = \mathcal{D}_i \right)
\end{equation}

Let $\mathcal{D}$ represent the set of observed samples from the Columbian populace.
How do we determine the probability of our model generating some $\mathcal{D}_i \in \mathcal{D}$ to
determine $\mathcal{L}(\theta)$?
Here, we take a frequentist approach and perform the following steps:
\begin{enumerate}
    \item Run our simulator once to get $\mathcal{D}'_1$.
    \item Check if $\mathcal{D}'_1$ matches $\mathcal{D}_1$.
    \item Repeat steps 1 and 2 some number of times $T_1$, for different simulated samples but the same
        observed sample.
        We define $\Pr\left( \mathcal{M}(\theta) = \mathcal{D}_1 \right)$ as the frequency of exact matches.
    \item Repeat step 3 for all observed samples to get $\Pr\left( \mathcal{M}(\theta) = \mathcal{D}_2 \right)$,
        \ldots, $\Pr\left( \mathcal{M}(\theta) = \mathcal{D}_r \right)$.
    \item To find the $\mathcal{L}$ is to multiply all $r$ probabilities together.
\end{enumerate}

This approach seems simple enough, but there exists one caveat: the frequency of exact matches is too low to interpret
anything meaningful.
The microsatellite repeat length set $\mathbb{M}$ consists of roughly 30 elements, meaning that roughly 30 dimensions
must exactly match.
In addition to this, each frequency resides in some large set whose elements are between $[0, 1]$.
The solution we propose here is a technique known as \emph{Approximate Bayesian Computation}, or ABC for short.
ABC has two parts: the use of \emph{approximate} matches and the use of summary
statistics~\cite{lintusaariFundamentalsRecentDevelopments2017}.
In the following sections, I discuss how we define approximate, what summary statistics are, and why we are not using
them for this problem.

\begin{figure}[t]
    \centering{\input{include/floats/approximate-likelihood.tex}}
    \caption{General figure depicting a true likelihood surface (blue) and an approximate surface (red).
    The true likelihood may not be found in a reasonable amount time, whereas the approximate likelihood is wider but
    taller (making it more tractable, but less defined).
    }\label{fig:approximateLikelihood}
\end{figure}

\subsection{Approximate Matches: $\epsilon$}\label{subsec:approximateMatches}
Let us dissect $\mathcal{D}$ and $\mathcal{D}'$ further.
A given $\mathcal{D}_i \in \mathcal{D}$ and $\mathcal{D}'_i \in \mathcal{D}'$ define $| \mathbb{M} |$-sized tuples,
whose values exist in $[0, 1]$.
To compare a given $\mathcal{D}_i$ and $\mathcal{D}'_i$ is to iterate through each element (repeat unit) in both
tuples and measure their proximity to each other.
Let $\delta : \mathcal{D}_i,\mathcal{D}'_i \mapsto [0, 1]$ represent some function that accepts the observed and
generated sample, and outputs some distance between 0 and 1.
The comparison between our observed and generated samples is given by $\delta(\mathcal{D}_i, \mathcal{D}'_i)$.

We were only able to explore one $\delta$ function: the angular (or Cosine) distance.
The angular distance treats $\mathcal{D}_i$ and $\mathcal{D}'_i$ as $| \mathbb{M} |$-dimensional vectors
and aims to quantify some difference between the two.
An output of 0 indicates that both samples are completely similar, while an output of 1 indicates that two vectors
are maximally dissimilar (orthogonal).
The angular distance $\delta_A$ is defined below~\cite{chaComprehensiveSurveyDistance2007a}:
\begin{equation}
    \delta_A(\mathcal{D}_i, \mathcal{D}'_i) = \frac{2}{\pi} \arccos \left(
    \frac{\sum_{\ell=\kappa}^{\Omega} \mathcal{D}_i[\ell] \cdot \mathcal{D}'_i[\ell]}{
        \sqrt{\sum_{\ell=\kappa}^{\Omega} \left(\mathcal{D}_i[\ell]\right)^2} \cdot
        \sqrt{\sum_{\ell=\kappa}^{\Omega} \left(\mathcal{D}'_i[\ell]\right)^2}
    } \right)
\end{equation}
where $\mathcal{D}_i[\ell]$ and $\mathcal{D}'_i[\ell]$ represent the frequency of repeat length $\ell$ for the observed
and generated samples respectively.

With some distance is quantified, the next step is defining what ``approximate'' means.
According to ABC, two samples $\mathcal{D}_i$ and $\mathcal{D}'_i$ are approximate matches if their distance falls below
some threshold $\epsilon \in [0, 1]$~\cite{marjoramMarkovChainMonte2003}:
\begin{equation}
    \left(\delta(\mathcal{D}_i, \mathcal{D}'_i) < \epsilon \right) \Leftrightarrow
    \left(\mathcal{D}_i \text { and } \mathcal{D}'_i
    \text{ are approximate matches} \right)
\end{equation}
By increasing $\epsilon$, the frequency of exact observed -- generated matches increases as well.
%By making this problem more tractable though, we increase the noise associated with drawing from $\mathcal{L}$.
This results in a flatter curve, as seen in~\autoref{fig:approximateLikelihood}.

The next question that follows is ``How do we know which $\epsilon$ to use?''
If $\epsilon$ is too small, the problem becomes intractable.
If $\epsilon$ is too large, we draw values that are not representative of the original distribution.
There is no clear answer to this question, and Lintusaari et.\ al.\ states this choice is typically made by
experimenting with different $(\mathcal{D}, \theta)$ pairs~\cite{lintusaariFundamentalsRecentDevelopments2017}.
We define our threshold $\epsilon$ as a \emph{hyperparameter}, a parameter we must specify and often experiment with)
to find the parameters of interest $\theta$.

\subsection{Dimension Reduction: Summary Statistics}\label{subsec:dimensionReductionSummaryStatistics}
An alternative to using a distance function $\delta$ that deals with $| \mathbb{M} |$-dimensional tuples is to use a
function that reduces, or \emph{summarizes} the data into two points $\bar{\mathcal{D}}, \bar{\mathcal{D}'}$ of lower
dimensionality and finds a distance between both $\bar{\mathcal{D}}, \bar{\mathcal{D}'}$.
We specify a distance function $\delta_S$ that performs this transformation using functions
$h : \mathcal{D}_i \mapsto \bar{\mathcal{D}_i}$ and $h' : \mathcal{D}'_i \mapsto \bar{\mathcal{D}'_i}$ as such:
\begin{equation}
    \bar{\delta_S}(\mathcal{D}_i, \mathcal{D}'_i)  = \bar{\delta}(h(\mathcal{D}_i), h'(\mathcal{D}'_i))
\end{equation}
A common choice for $h, h'$ is the mean or median.
For us, it may make sense to use the focal unit computation $\hat{\ell}$ as our summary statistic.

Using summary statistics avoids the curse of dimensionality (see~\cite{bellmanDynamicProgramming2013}) for high
dimension distance functions, but this adds yet another item we must specify: ``Which summary statistic is the best?''.
If we summarize our data wrong, we again run into the problem of drawing values that do not represent our original
distribution.
We ran several trials without reducing our dimensionality and have not run into any problems thus far.
To reduce noise, we are only using the angular distance $\delta_A$ for $| \mathbb{M} |$-dimensional vectors.

\section{Markov Chain Monte Carlo}\label{sec:markovChainMonteCarlo}
In this section, we discuss the Markov Chain Monte Carlo (MCMC) approach to approximating a likelihood function.

\begin{figure}[t]
    \centering{\input{include/floats/likelihood-mcmc.tex}}
    \caption{General figure depicting the random walk of Metropolis sampler around some surface poportional to our
    likelihood (the posterior).
    We start at initial state $\theta^{(1)}$.
    We accept $\theta^{(2)}$ and $\theta^{(3)}$ which leads to regions of higher likelihood but later accept
    $\theta^{(4)}$, a less likely point due to the randomness of $\mathcal{V}$.
    }\label{fig:metropolisAlgorithm}
\end{figure}

\subsection{Monte Carlo}\label{subsec:monteCarlo}
In~\autoref{sec:approximateBayesianComputation}, we explored how to determine the likelihood of a single point $\theta$.
We are now interested in the most likely $\theta$ out of all possible parameter values $\Theta$.
We considered a naive approach to determining the most likely point $\hat{\theta}$
in~\autoref{sec:intractableLikelihoodFunctions}, which involved determining the derivative of some function we can
explicitly express.
Given that we cannot express our function as such, this is not an option.
The solution Monte Carlo algorithms propose is choosing $\theta$ randomly and select the most likely $\theta$ out of
all runs.

Let $p$ define a probability distribution that determines how we draw $\theta$.
$p$ is more commonly known as a \emph{prior distribution}, and allows us to insert any prior beliefs we have about our
likelihood before finding $\mathcal{L}(\theta)$ itself.
In Bayesian inference, the characterization of the uncertainty of some $\theta$ given observations $\mathcal{D}$ is
given by another distribution known as the
\emph{posterior distribution}~\cite{lintusaariFundamentalsRecentDevelopments2017}:
\begin{equation}
    \Pr(\theta \mid \mathcal{D}) \propto \mathcal{L}(\theta) p(\theta)
\end{equation}
According to Baye's law, the posterior is proportional to likelihood function we are trying to find.
If we can explicitly express our posterior, then the $\hat{\theta}$ that maximizes $\Pr(\theta \mid \mathcal{D})$ also
maximizes $\mathcal{L}(\theta)$.

\subsection{Markov Chains}\label{subsec:markovChainMonteCarlo}
With the Monte Carlo strategy, our general procedure now becomes:
\begin{enumerate}
    \item Draw $\theta_i$ from our prior $p(\theta)$.
    \item Determine the likelihood of this point $\mathcal{L}(\theta_i)$.
    \item Repeat until we have a representative set of samples.
    \item Fit our samples to a curve, and return the $\hat{\theta}$ that maximizes $\mathcal{L}(\theta)$.
\end{enumerate}
The main problem with this approach is our reliance on the prior.
A misinformed prior will produce samples that are not representative of the posterior.
As an example if $E(p(\theta)) = 500$ but our posterior is centered around $\theta = 1$, we will end up with a small or
nonexistent posterior.
\emph{Markov Chain Monte Carlo} methods solve this by sampling more often from regions of higher likelihood.

Markov Chain Monte Carlo methods work by constructing a Markov chain such that the posterior distribution is its
equilibrium distribution.
We start by defining a $\Theta$-sized vector $Y$, indexed by all distinct $\theta \in \Theta$ and whose values represent
probabilities associated with each $\theta$.
$Y$ is said to be a distribution here.
Next we define a matrix $G$ of size $\Theta \times \Theta$, whose entries describe the probability of transitioning
from one $\theta$ (row) to another $\theta$ (column).
$G$ is known as a \emph{transition matrix}.
We are able to move from distribution $Y^{(i)}$ to $Y^{(i + 1)}$ using $G$:
\begin{equation}
    Y^{(i + 1)} = Y^{(i)} G
\end{equation}
$Y$ is said to be at \emph{equilibrium} if the following holds true for some transition matrix $G$:
\begin{equation}
    Y^{(i)} = Y^{(i)} G
\end{equation}
The goal of MCMC is to draw samples $X$ from $Y$ such that $Y$ represents our posterior $\Pr (\theta \mid \mathcal{D})$.
Let $X$ represent this \emph{chain} of states, or parameter values $\theta$, that satisfy the
conditions below to draw from such a distribution~\cite{hanadaMarkovChainMonte2018}:
\begin{enumerate}
    \item \emph{$X$ is a Markov chain}.
        The probability of obtaining $X^{(i)} \in X$ from $X^{(i - 1)} \in X$ does not depend on any other configuration
        $X^{(i - 2)}, X^{(i - 3)}, \ldots, X^{(1)}$ other than $X^{(i - 1)}$ itself.
    \item \emph{$X$ is irreducible}.
        This states that we are able to travel to all of $\Theta$ from any given $\theta$ in a finite number of
        transitions.
    \item \emph{$X$ is aperiodic for all configurations}.
        A state $\theta$ is aperiodic if there exists a $i$ such that for all $j \geq i$:
        \begin{equation}
            \Pr \left( X^{(j)} = \theta \mid X^{(1)} = \theta \right) > 0
        \end{equation}
    \item \emph{$X$ is positive recurrent}.
        The states that the expected number of transitions to move to back to the same state is finite.
\end{enumerate}
MCMC comprises a class of algorithms that are able to produce $X$ that satisfy these conditions.

\subsection{Metropolis Algorithm}\label{subsec:metropolisAlgorithm}
As per our last section, we want to generate some chain of states $X$ such that the given conditions are satisfied.
In this section, we describe the Metropolis algorithm-- a procedure that is able to generate a chain of states such that
these conditions above are met.
There are three main steps to the Metropolis algorithm:
\begin{enumerate}
    \item \emph{Proposal}: We define some function $g : \Theta,\mathbb{V} \rightarrow \Theta$ which generates a
        new and random $\theta_i$ given an old $\theta'$.
        This relates to the transition matrix $G$ from~\autoref{subsec:markovChainMonteCarlo}, and constructs different
        $G$ for different values of $\theta_i$.
        The Metropolis algorithm is a special case of the \emph{Metropolis-Hastings algorithm} in which this proposal is
        \emph{symmetric}.
        A symmetric proposal indicates that the probability of drawing our new value given our current value is equal to
        the probability of drawing our current value given our new value.
    \item \emph{Calculate}: We determine the acceptance ratio $\alpha$, which is a ratio of the proposed $\theta^{(i)}$
        to the old $\theta'$.
    \begin{equation}
        \alpha = \frac{\mathcal{L}(\theta_i)}{\mathcal{L}(\theta)}
    \end{equation}
    \item \emph{Accept}: We save $\theta_i$ and $\mathcal{L}(\theta_i)$ to our collection of states if and only if
        $\theta_i$ is more likely than the old $\theta'$ \emph{or} the ratio of proposed to old likelihoods is greater
        than some uniform random variable $U(0, 1)$.
        If this is not true, then we go back to step (1) until we have run $T_2$ iterations.
\end{enumerate}

\begin{algorithm}[t]
    \SetAlgoLined
    \DontPrintSemicolon
    \Fn{MetropolisSampler \ {$(T_2, \mathcal{L}, \theta^{(1)}, g)$}} {
        \KwIn{number of algorithm iterations $T_2$, likelihood function $\mathcal{L}$, initial state $\theta_1$,
        proposal function $g$}
        \KwOut{samples from our posterior distribution $\Pr(\theta \mid \mathcal{D})$}
        $X \gets \emptyset$, $\theta' \gets \theta^{(1)}$ \;
        \For{$i \gets 1$ \KwTo $T_2$}{
            $\theta^{(i)} \gets g(\theta')$ \;

            \If{$\mathcal{L}(\theta^{(i)}) \cdot \mathcal{L}(\theta')^{-1} \leq \ \sim U(0, 1)$}{
                $\theta' \gets \theta_i$, $X \gets X \cup \{ \theta^{(i)} \}$ \;
            }
        }

        \Return $X$ \;
    }
%    \textbf{end} \;
    \caption{The Metropolis algorithm, used to produce samples from a posterior distribution proportional to our
    likelihood.}
    \label{alg:metropolis}
\end{algorithm}

Note that we have introduced three new hyperparameters here: our initial Markov chain position $\theta^{(1)}$,
our proposal function $g$, and the number of Metropolis sampler iterations $T_2$.
If $\theta^{(1)}$ is nowhere near the regions of high likelihood, if $g$ is too tightly or loosely distributed, or if we
do not run our sampler for enough iterations, we say that our Markov chain has not \emph{converged}.
Nonconvergence means that we cannot interpret anything meaningful from $X$, making the selection of these
hyperparameters \emph{and} some convergence verification process critical.

\section{Maximum Likelihood Estimation}\label{sec:maximumLikelihoodEstimation}
In this section, we explore a high level view of our approach to maximum likelihood estimation.
Using the Metropolis sampler, we are able to get a collection of states $X$ from a distribution proportional to our
likelihood.
We now want to tie this back into our original question: ``Which parameter values $\theta \in \Theta$ are the most
likely to produce our observations $\mathcal{D}$?''.
Given states $X$ from our posterior, we determine this most likely point $\theta$ by (a) constructing histograms, (b)
fitting the histograms to the equation of some distribution, and (c) determining the mean of this distribution.

We start by constructing our histogram.
For this project, each $\theta$ represents a 2-tuple of $c$ and $d$.
We construct sets $C, D$ which consist of each $c$ and $d$ element for all $\theta \in X$:
\begin{align}
    C &= \{c \mid c \in \theta \land \theta \in X \} \\
    D &= \{d \mid d \in \theta \land \theta \in X \}
\end{align}
We now want to partition each set $C$ and $D$ sets into consecutive, non-overlapping intervals or \emph{bins}
$C^\star$ and $D^\star$ respectively.
The partitioning itself is governed by the boundaries of our results $[\min(C), \max(C)]$,
$[\min(D),\max(D)]$, and the \emph{bin widths} $b_c$ and $b_d$ for the $c$ and $d$ parameters respectively.
This gives us the number of bins:
\begin{equation}
    \begin{aligned}
        | C^\star | &= \left\lceil \frac{\max(C) - \min(C)}{b_c} \right\rceil \\
        | D^\star | &= \left\lceil \frac{\max(D) - \min(D)}{b_d} \right\rceil
    \end{aligned}
\end{equation}
We build our histogram for $c$ by constructing the function $w_c : [0, |C^\star |] \rightarrow [0, 1]$ that accepts
an integer that enumerates our bins and outputs the frequency of associated with that bin.
The end result must follow the property in~\autoref{eq:densityFunction} to ensure our histogram
represents some probability density function.
\begin{equation} \label{eq:densityFunction}
    \int_{i=C^\star} w(i) di = 1
\end{equation}

The next step is to fit our histogram $w_c$ to some known distribution.
We explored two distributions here: the normal distribution and the gamma distribution.
If we assume that our posterior for some $C$ is normally distributed, we can use the function below:
\begin{equation}\label{eq:normalDistribution}
    w_{cN}(c) = \frac{\exp(\frac{\left( \sfrac{(c - \mathit{loc})}{\mathit{scale}} \right)^2}{2} )}
    {\mathit{scale} \cdot \sqrt{2\pi}}
\end{equation}
To get the point of maximum likelihood point $\hat{c}$ using $w_{N}(c)$ is to obtain its mean.
For the distribution in~\autoref{eq:normalDistribution}, this is the $\mathit{loc}$ parameter with variance
$\mathit{scale}$.
If we instead assume our posterior for some $C$ is gamma distributed, we can use the function below:
\begin{equation}
    w_{c\Gamma}(c) = \frac{(x-\mathit{loc})^{a-1}\exp(-\frac{c-\mathit{loc}}{\mathit{scale}})}{\mathit{scale}^a
    \cdot \Gamma(a)}
\end{equation}
where $\mathit{loc}$ represents a horizontal shifting parameter, $\mathit{scale}$ represents the scaling parameter of
our distribution, and $a$ represents the gamma distribution \emph{shape} or skew parameter.
Again, the maximum likelihood $\hat{c}$ using this $w_\Gamma(c)$ is the mean.
For the distribution given above, this is equal to $a \cdot \mathit{scale} + \mathit{loc}$ with variance
$a \cdot \mathit{scale}^2$.
For both cases, these steps are then repeated for $D$ to obtain $\hat{d}$.


%\section{Physical Human Factors Report: Tristan Griffith}
\rhead{\today}
\begin{center}
{\large  Tristan Griffith}\\
\vspace{2mm}
{\large Dr. James Hubbard Jr.}
\noindent\rule{\textwidth}{2pt}
\end{center}
\setcounter{section}{1}

%\begin{wrapfigure}{r}{0.45\textwidth}
%\centering
%\includegraphics[scale=.3]{../../../figures/demo_map.png}
%\caption{Modal Heatmap for Subject 32}
%\end{wrapfigure}
\subsection{Introduction}
We inquire concerning the nature of traveling waves in the brain. Under this head there are three points of inquiry:
\begin{enumerate}
\item Whether the brain has traveling waves?
\item Whether fMRI waves are connected to EEG waves?
\item If traveling waves exist, are they relevant to clinical outcomes?
\end{enumerate}
\subsection{Whether the brain has traveling waves?}
\textbf{Keyword differences:} First, note that the notion of a traveling wave is different for neuroscience and engineering. Neuroscientists would denote any brain wave that is observable across the sensor locations as a traveling wave. Engineers reserve traveling wave for behavior which is not in phase across the spatial sensors. In phase behavior is called a standing wave, while out of phase is a traveling wave. Neuroscientists would call both traveling waves. Going forward, I will use traveling wave in the engineering sense, denoting standing waves where needed. \\
\textbf{Objection 1:} Prior work has shown repeated decoherence among spatially distributed recordings of brain activity. Theta oscillations in humans are postulated to have only local mechanisms \cite{doi:10.1152/jn.00409.2005}.
\begin{displayquote}
 We found that, whereas nearby gated sites $(<20 \textit{mm})$ were often but not always coherent, distant gated sites were almost never coherent. Our results imply that there are local mechanisms for the generation of cortical theta.
\end{displayquote}
\textbf{Objection 2:} While inter-electrode spatial correlations in the gamma band have been discovered in animals, the same was not found in humans \cite{MENON199689}.
\begin{displayquote}
The findings suggest that the surface diameters of domains of spatially correlated activity underlying perceptual categorization in human gamma band ECoG are limited to less than 2 cm and that the intermittent synchronization observed across separations of 1 cm and 1.4 cm is not solely due to volume conduction. Thus, if such gamma band spatial patterns exist in the human brain, no existing technology would be capable of measuring them at the scalp, and subdural electrode arrays for cortical surface recording would have to have spacings under 5 mm.
\end{displayquote}
\textbf{Objection 3:} Using statistical methods, EEG coherence was found to decline substantially in space across all frequency bands \cite{BULLOCK1995161}.
\begin{displayquote}
In both the subdural surface samples and those from temporal lobe depth electrode arrays coherent declines with distance between electrodes of the pair, on the average quite severely in millimeters. This is nearly the same for all frequency bands.
\end{displayquote}
\textbf{On the contrary,} it has been found most recently that when \textbf{phase} is considered as part of the spatial coherence, traveling and standing waves are observed across a broad spectrum for most subjects \cite{ZHANG20181269}.\\

I argue that, in each of the objections above, the possibility of complex traveling waves was ignored. In each of these studies, it was assumed that the spatial activity of the brain's electrical activity would be described by perfect sine and cosine waves in two dimensions. This excludes the possibility of damping in the oscillatory behavior of the electrical activity. Many mechanical systems have damping, so why should we exclude this possibility in the analysis of the brain? To this end, our output only modal analysis has shown significant spatial dependence when both proportional and non-proportional damping is considered. Unlike the objections above, the naiveté of our engineering first approach is a strength, because it does not exclude the possibility of complex eigenmodes. Further, studies \cite{doi:10.1152/jn.00409.2005} and \cite{BULLOCK1995161} argue that low correlation of the band power is sufficient evidence to exclude the possibility of spatial dependence in the activity of the brain. I argue that a single, simple statistical tool is not sufficient proof for such a broad claim. There is no consideration of phase or damping.  Ultimately, we discovered that 20\% of the eigenmodes are damped on average, with 50\% showing some out of phase behavior. At present this issue is \textbf{closed.}
\subsection{Whether fMRI waves are connected to EEG waves?}
\textbf{Objection 1:} Using band power and statistical tests, EEG and fMRI were found to be inversely correlated on average, but not significantly enough to report \cite{LAUFS20031463}:
\begin{displayquote}
A more general problem is that our current understanding of EEG and fMRI (or PET) signals does not allow us to extrapolate whether and, if so, how the oscillatory synchronization of electrical synaptic activity in the alpha band translates into hemodynamic signals. The mere fact that synaptic activity takes on the temporal structure of an oscillation does not mean that net synaptic activity changes over time, and the oscillation frequencies of physiological rhythms themselves are beyond the temporal resolution of hemodynamic signals.
\end{displayquote}
\textbf{Objection 2:} In a comprehensive survey, \cite{SALEKHADDADI2003110} reports that despite improvements in sensing technology, disparate studies considering many different hypotheses have been unable to realize a unified connection between fMRI and EEG. Questions even arise concerning the nature of spontaneous EEG (e.g. is the rest state actually a rest state?). Because it is difficult to define consistent operating conditions from subject to subject a connection between fMRI and EEG is non-trivia:
\begin{displayquote}
We have reviewed the existing literature on this topic and have explored both the theoretical and practical limitations of EEG/fMRI with specific emphasis on studying the generators of IEDs. We envisage that further characterization of the BOLD correlates of spontaneous EEG activity and the development of ever more sophisticated fMRI models will constitute a significant effort over the next few years.
\end{displayquote}
\textbf{Further,} A despite a great amount of effort, experimental and theoretical work has been unable to explain the link between EEG and BOLD signals, as discussed in \cite{10.3389/fneur.2013.00001}:
\begin{displayquote}
The most consensual evidence comes from the recording of electrical activity using micro-electrodes implanted in the cortex of (non-human) animals, simultaneously with fMRI, indicating that the BOLD signal reflects mostly slow, post-synaptic input activity measured by local field potentials (LFPs), rather than fast, spiking output activity measured by single/multi-unit activity (S/MUA; Logothetis et al., 2001). In humans, a growing number of simultaneous EEG-fMRI studies on healthy subjects as well as epilepsy patients have now been reported (Goldman et al., 2002; Laufs et al., 2003, 2006; Moosmann et al., 2003; de Munck et al., 2009), and biophysical models of the neurovascular coupling have been proposed (Riera et al., 2006, 2007). Overall, reports in the literature do not provide a clear picture of the link between EEG and BOLD signals. In particular, contradictory results have been presented regarding the dependency of BOLD changes on the EEG power and spectral profiles. These include, for example, positive and negative BOLD correlations with specific frequency band power changes in the human EEG (de Munck et al., 2009), BOLD decoupling from LFP power in mice (Ekstrom, 2010), and negative BOLD associated with large increases in LFP and MUA during seizures also in mice (Schridde et al., 2008). Rosa et al. (2010b) addressed this topic by comparing different models of the transfer function between EEG and BOLD signals, in the prediction of fMRI data, in a visual stimulation experiment with human healthy subjects. The models explored included the EEG total power (TP; Wan et al., 2006); linear combinations of the power from different frequency bands (Goense and Logothetis, 2008); and several variations of a heuristic model proposed by Kilner et al. (2005) in which BOLD changes are assumed to be proportional to the root mean square frequency (RMSF) of the EEG spectrum. The results obtained showed a clear superiority of the RMSF metrics in predicting the BOLD signal, when compared to power-weighted metrics.
\end{displayquote}
\textbf{At present,} I argue that our modal decomposition method offers a compelling spatial representation of the data which may be more easily compared with a BOLD time series. \textcolor{red}{I estimate it would take approximately 3 weeks to process a data set with EEG and fMRI signals in order to compare the modes.} We have seen some empirical evidence that the shapes are similar, but more work is needed to verify frequencies and mode shapes are significant. At present this issue is \textbf{\textcolor{red}{open}} in both our work and the broader literature.

\subsection{If traveling waves exist, are they relevant to clinical outcomes?}
\textbf{Objection 1:} \cite{Massimini6862} found that while a standing wave was present during sleep, there was not enough evidence to say if it was a function of the cognitive state or the natural resting state of the brain.
\begin{displayquote}
The pattern of origin and propagation of sleep slow oscillations is reproducible across nights and subjects and provides a blueprint of cortical excitability and connectivity. The orderly propagation of correlated activity along connected pathways may play a role in spike timing-dependent synaptic plasticity during sleep.
\end{displayquote}
\textbf{Objection 2:} \cite{Takahashi0} found that as in animal brains, there are standing waves in human brains that propagate 180\degree to each other during motor tasks. However, the same wave was observed in the ``rest" state.
\begin{displayquote}
This study shows that the two properties of propagating beta waves are present in MI of a tetraplegic human patient while he was instructed to perform an instruction delay center-out task using a cursor controlled by the chin. Moreover, we show that beta waves are sustained and have similar properties whether the subject was engaged in the task or at rest.
\end{displayquote}
\textbf{Objection 3:} While local bursts of gamma activity were found to generate standing alpha waves, they were not connected to a task as subjects were in the ``rest" state\cite{Bahramisharif18849}:
\begin{displayquote}
In short, we have demonstrated here that bursts of gamma activity propagate over neocortex and the propagating gamma bursts are locked to the phase of traveling alpha waves. Our findings suggest that not only do alpha oscillations serve to coordinate the gamma activity in time, but also in space. In future work to uncover the functional role of this phenomenon would be of great interest.
\end{displayquote}
\textbf{On the contrary,} it was found that standing waves are generated reliably during a memory task, and accordingly return to a baseline level a few hundred milliseconds after the stimulus has passed \cite{doi:10.1152/jn.00409.2005}:
\begin{displayquote}
We confirmed that these patterns were reliable by measuring the time course of mean traveling wave DC at the group level. Following cue onset, traveling waves in the temporal and frontal lobes showed increases in DC above baseline levels (Figure 4E). Inversely, traveling waves from occipitoparietal clusters showed decreased DC during this same period, which was significantly different from the DC increase in the frontal and temporal lobes. Because frontal and temporal regions specifically show increased DC following cue onset, it indicates that traveling waves in these areas move more consistently during memory retrieval.
\end{displayquote}
I argue that simply observing a consistent pattern of traveling and standing waves is insufficient to draw conclusions from. Using our modal decomposition technique, we should seek to find mathematical connections between certain modes and certain tasks. As with the fMRI comparison, this is something we can look at in the coming months that has not been resolved in the larger neuroscience community. At present this issue is \textbf{\textcolor{red}{open}} in both our work and the broader literature.

%\begin{displayquote}
%The class of monotone DNF expressions is learnable via an algorithm $B$ that uses $L=L(h,d)$ calls of examples and $dt$ calls of oracle, where $d$ is the degree of the DNF expression $f$ to be learnt and $t$ the number of variables.
%\end{displayquote}

%\begin{wrapfigure}{r}{0.55\textwidth}
%\centering
%\includegraphics[scale=.5]{../figures/complexity.png}
%\caption{The Error-Complexity Trade Off}
%\end{wrapfigure}


\chapter{parreset}

\section{Introduction}

An LPE may have parameters of which the values are not used anymore after a particular state.
If such parameters can have different values after that state, they continue to add states to the state space for each of those values -- \emph{without} adding any new behavior!

The \texttt{parreset} command tries to determine whether the value of a parameter is used by an LPE after a particular summand.
If this is the case, the parameter is said to be \emph{relevant} for that summand; otherwise, the summand can `reset' the parameter, meaning that it can be assigned a default value.

To determine the relevance of parameters, \texttt{parreset} analyzes the reachability of summands in a generalized, symbolic manner.
The \texttt{datareset} command, serving a similar purpose (see \ref{datareset}), does a control-flow analysis instead.

\section{Formal background}

\subsection{Possible successors} \label{possiblesuccessors}

Consider summands $s_\alpha$ and $s_\beta$, referencing their elements conform \ref{summandelements}.
Summand $s_\beta$ is said to be a \emph{possible successor} of $s_\alpha$ if the following expression \emph{could be} satisfiable:
\begin{align*}
g_\alpha \land {g_\beta}[v \rightarrow q(v) \;|\; v \in \varsof{g_\beta} \setminus P][p \rightarrow v_\alpha(p) \;|\; p \in P]
\end{align*}

where $q(v)$ is a bijective function that relates variable $v$ to a fresh variable.

\section{Algorithm}

The algorithm is a generalization of an existing algorithm \cite{van2009state}.
It consists of two phases.

During the first phase (the preparation phase), we determine all successors of each summand of the LPE using the equation from the previous section.

The second phase (the iteration phase) follows these steps:

\begin{enumerate}

\item For each summand $s_\alpha$ of the LPE, create a set $R_\alpha$ that contains all parameters of the LPE.
This means that, initially, we assume that all parameters of the LPE are used by one or more of the successors of $s_\alpha$; that is, \emph{relevant} to $s_\alpha$.

\item For each summand $s_\alpha$ of the LPE, set the value of $R_\alpha$ to $\bigcup\limits_{s_\beta \in S_\alpha}^{} r(s_\beta)$ where $S_\alpha$ is the set of all successors of $s_\alpha$ (as determined during the preparation phase) and where $r$ is the function
\begin{align*}
r(s_\beta) = \left( \text{vars}(g_\beta) \cup \bigcup\limits_{x \in R_\beta}^{} \text{vars}(v_\beta(x)) \right) \setminus C_\beta
\end{align*}

\item Repeat the previous step until the fixpoint of $R_\alpha$ is reached for each summand $s_\alpha$ of the LPE.

\item For each summand $s_\alpha$ and for all $p \in P \setminus R_\alpha$, change the expression $v_\alpha(p)$ (which defines the value of LPE parameter $p$ after the application of $s$) to $v_I(p)$, the value of $p$ in this initial state of the LPE.

\end{enumerate}

\section{Example}

Consider the following LPE:

\begin{lstlisting}
//Process definition:
PROCDEF example[A :: Int, B](x, y :: Int)
  = A ? i [[x==0]] >-> example[A, B](1, i)
  + A ? i [[x==1 && i==y]] >-> example[A, B](2, y)
  + B [[x==2]] >-> example[A, B](3, y)
  + B [[x==3]] >-> example[A, B](0, y)
  ;

//Initialization:
example[A, B](0, 0);
\end{lstlisting}

Let the summands be represented by $s_1$ from $s_4$ (from top to bottom).

Finding the successors of each summand is easy: each summand has exactly one successor, namely the next one, except in case of $s_4$, where the $s_1$ is the successor.

It is also obvious that $x$ will always be in $R_\alpha$ for each summand $s_\alpha$, because each summand $s_\alpha$ uses $x$ in its guard.

Process parameter $y$ will always be in $R_1$ because $y$ is used in the guard of $s_1$'s successor, $s_2$.
After a few iterations, however, $y$ is removed from $R_2$, $R_3$, and $R_4$.
This means that $y$ is assigned a default value in $s_2$, $s_3$, and $s_4$.
Choosing the initial value of $y$ as its default value gives

\begin{lstlisting}
//Process definition:
PROCDEF example[A :: Int, B](x, y :: Int)
  = A ? i [[x==0]] >-> example[A, B](1, i)
  + A ? i [[x==1 && i==y]] >-> example[A, B](2, 0)
  + B [[x==2]] >-> example[A, B](3, 0)
  + B [[x==3]] >-> example[A, B](0, 0)
  ;

//Initialization:
example[A, B](0, 0);
\end{lstlisting}

\section{Benchmark results}

The following durations were measured with a benchmark for several models:
\begin{itemize}
\item The average duration of \txs{} to make 500 steps in a model after it has been converted to LPE form;
\item The average duration of \txs{} to make 500 steps in a model after it has been converted to LPE form and after the \texttt{parreset} operation has been applied.
\end{itemize}

When plotting the second series of measurements against the first (see Figure~\ref{parreset-vs-lpe-only:fig}), it is easy to see that the impact is insignificant in most cases.

\begin{figure}[!ht]
\begin{center}
\includegraphics[width=0.7\linewidth]{charts/parreset-vs-lpe-only}
\caption{Benchmark results: parreset vs LPE transformation}
\label{parreset-vs-lpe-only:fig}
\end{center}
\end{figure}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{caption}
\usepackage{subcaption}

\title{Weekly Report}
\author{Junior Team }
\date{June 2020}

\usepackage{natbib}
\usepackage{graphicx}

\begin{document}

\maketitle

\section*{IRB Data Analysis}
Once we had the IRB model set up and could optimize over $P_t$, $P_n$, and the maximum width, our goal was to begin to identify trends in the data.

\section{Normal and Tangential Impulses}
The first interesting correlation we found was between the tangential impulse ($P_t$) and the optimal patch width, which can also be interpreted as the moment applied due to the patch width.

\begin{figure}[h!]
\centering
\caption{Tangential Impulse vs Optimal Width}
\includegraphics[scale=0.2]{tanImpulse}
\label{fig:tanImpulse}
\end{figure}

\noindent In the figure above, the constraint $\mbox{width} \leq 1 \mbox{cm}$ was used for 1000 different impacts. The correlation coefficient between $P_t$ and width was $ -0.9283$. \\

\noindent  When examining normal impulse ($P_n$) we did not find a clear linear trend, however there seemed to be an impulse threshold below which the optimal width was only zero. From this, we hypothesized that perhaps, those cases to the left of the threshold were below a certain energy threshold, or were lower energy impacts. This would lead to a smaller increase in angular velocity after impact, and less energy in the restitution phase. As a consequence, a patch width which creates a moment and greatly increases the angular velocity ($\dot\theta$) would not be optimal.\\

\noindent To test this hypothesis, we plotted pre- and post-impact kinetic energy versus the optimal patch width. The plots seem to disprove the idea that there is any relationship between zero patch width being optimal and a lower energy collision.

\begin{figure}[ht]
    \caption{Kinetic Energy Analysis}
    \centering
    \begin{subfigure}[b]{0.45\linewidth}
        \includegraphics[scale=0.12]{preImpactEnergy.jpg}
        \caption{Pre Impact KE}
        \label{fig:preKE}
    \end{subfigure}
    \quad
    \begin{subfigure}[b]{0.45\linewidth}
       \includegraphics[scale=0.12]{postImpactEnergy.jpg}
        \caption{Post Impact KE}
        \label{fig:postKE}
    \end{subfigure}
\end{figure}

\noindent Additionally, we took a look at the correlation between the net moment and impulses generated from the impact. When inspecting the normal impulse, we found no correlation between the normal impulse and the net moment generated by the ellipse. This is expected, since the configuration of the ellipse as it contacts the surface can be varied greatly, so there should be no definite correlation if the trajectory of the ellipse is not meant to be uniform. \\

\noindent On the other hand, when inspecting the tangential impulse of the ellipse, we find an interesting scatter plot that illustrates a negative correlation between net moment and tangential impulse. Since we assume that the tangential impulse mostly consists of frictional forces and we know that friction tends to oppose motion, we would expect the tangential impulse to be negative when the ellipse has a positive rotation and vice versa. Hence the negative correlation which can be depicted in Fig \ref{fig:tanImpulse2}.

\begin{figure}[ht]
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[scale=0.35]{netMomentvsNormImpulse.jpg}
    \caption{Normal Impulse}
    \label{fig:normImpulse}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[scale=0.35]{netMomentvsTanImpulse.jpg}
    \caption{Tangential Impulse}
    \label{fig:tanImpulse2}
  \end{subfigure}
  \caption{Impulse vs Net Moment}
\end{figure}

\newpage
\section{Pre-Impact Angle}
The next thing we chose to explore was the relationship between the pre-impact angle ($\theta^+$) and the optimal width. After a plot with a clear trend whose shape resembled an X and Dr. Posa's recommendation, we adjusted the angle transformation to get a cleaner range of angles (wrapped from [$-\pi, \pi$]). We then found another interesting correlation, when the pre-impact angle was plotted with the optimal width, the curve resembled a sin curve.

\begin{figure}[h!]
\centering
\caption{Pre-Impact Angle vs. Optimal Width}
\includegraphics[scale=0.2]{preImpactAngle}
\label{fig:preAngle}
\end{figure}

\noindent The red curve Width $= \frac{1}{100}(sin(2\theta)+0.3)$ clearly correlates with the data. The next thing we checked was whether there was a similar relationship with angular velocity (either $\dot\theta^+$ or $\dot\theta^-$), but there appeared to be nothing. \\

\noindent Interestingly, when we plotted error vs. pre-impact angle for the AP Poisson model and the change in rotational velocity vs. pre-impact angle for the IRB model, the plots also followed a $sin(2\theta)$ trend. \\

\noindent We wanted to explore whether the width depended on the moment arms (Tangential moment arm: vertical distance from the contact point to the COM, normal moment arm: horizontal distance from the contact point to the COM). \\

\noindent The following figures were made without any sort of optimization or considering any impulses. These are just geometric relationships that occur because of the elliptical shape of the object. Whenever the optimal width graphs are mentioned, Figure \ref{fig:preAngle} of Optimal width vs Angle is referenced. \\

\noindent Figure \ref{fig:TanMomentArmVsAngle} shows the relation between the orientation of the ellipse and the tangential moment arm. It shows how that the optimal width is at its max when the tangential moment arm is at its minimum and vice versa, since the optimal width takes the form of a sine wave and the tangential moment arm takes the form of a flipped cosine wave. They both have the same frequency, y-offset, and scaling. The only difference is the amplitude.

\noindent Figure \ref{fig:NormMomentArmVsAngle} shows the relation between the orientation of the ellipse and the normal moment arm. It shows that the optimal width and the normal moment arm are \textbf{\textit{identical}} in every way, except for the y offset and the reflection of the sine graph of the normal moment arm on the x-axis. This means that there is a direct correlation between the horizontal distance between the contact point and the COM, and the optimal width to account for the rotational impulse.

\begin{figure}[ht]
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[scale=0.14]{TanMomentArmVsAngle.jpg}
    \caption{ Tangential Moment Arm}
    \label{fig:TanMomentArmVsAngle}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[scale=0.14]{NormMomentArmVsAngle.jpg}
    \caption{Normal Moment Arm}
    \label{fig:NormMomentArmVsAngle}
  \end{subfigure}
  \caption{Moment Arms vs Angle}
\end{figure}

\noindent These relations agree with our predictions and reason.

\section{Moment and Patch Size}
\noindent Initially, we investigated patch size by taking the absolute value of moment. With this, we would see a plot fairly symmetric about $x = 0$ which makes sense since extending the patch from either direction of the center of the contact point should theoretically yield moments equal in magnitude but opposite in sign. However, when we remove this restriction, we can see a linear relationship between net moment and width (Fig \ref{fig:momentPSize}). Dr.\ Posa believed that all of this was pointing towards the idea that error and net moment are correlated, and predicted moment is essentially off by a linear factor:
\begin{align}
    M_{actual} = k * M_{predicted}
\end{align}

\begin{figure}[h!]
\centering
\caption{Net Moment vs. Optimal Patch Size}
\includegraphics[scale=0.55]{momentPSize}
\label{fig:momentPSize}
\end{figure}

\begin{align}
    \mbox{Best Fit Line: } width = 10.06 * M_{net} - 0.0345
\end{align}
\begin{center}
    Where width and the net moment match the units from the axes in Fig \ref{fig:momentPSize}.
\end{center}

\newpage

\section{Clumping and Convexity}
When we were investigating different potential correlations we began to notice clumping around a patch width of 0 (Fig \ref{fig:tanImpulse}, Fig \ref{fig:preKE}, Fig \ref{fig:postKE}) when we capped the maximum width. For the optimal patch width to be around zero for a given trial when the width is capped below the optimal width, but then have a nonzero optimal width when the range is increased means our solutions were non-convex. This goes against expectations for this sort of problem.To look more closely into why the clumping was occurring, we made fixed width vs error plots for different trials. The shapes of the graphs however, did not give us much insight on why we found clumping, since they seemed to point towards a convex solution.

\begin{figure}[ht]
  \centering
  \caption{Set Width vs Error Plots for Individual Trials}
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[scale=0.4]{Nat1.jpg}
    \caption{Trial 8}
    \label{fig:cov1}
  \end{subfigure}
  \begin{subfigure}[b]{0.4\linewidth}
    \includegraphics[scale=0.4]{Nat2.jpg}
    \caption{Trial 31}
    \label{fig:cov2}
  \end{subfigure}
\end{figure}

\noindent This is something we will continue to investigate, but for now we will stick to not capping the width at the previously used $1cm$ since we know nearly all of the clumping can be avoided if the cap is raised just half a centimeter to $width_{max} = 1.5cm$.

\newpage

\section{AP Poisson and Wang Mason}
Another thing we did this week was see if we could replicate some of our findings from the IRB model in the two models we had previously worked on and whether we would see similar trends. For the Wang Mason model, we checked for a $x_1$ position we could apply the impulsive forces to minimize error that was within 30\% of it's original value. The plots involving tangential or normal impulse vs. moment matched nicely with those from the IRB model (Fig \ref{fig:normImpulse}, Fig \ref{fig:tanImpulse2}). \\

\noindent A similar approach was taking with the AP Poisson model, where the $x_1$ position was shifted to add an additional moment, also within 30\% of the original value. The tangential and normal impulse plots resembled both Wang and IRB but were not quite as similar. We did observe the same trend between error and pre-impact angle:

\begin{figure}[h!]
\centering
\caption{Error vs Pre-Impact Angle (AP Poisson)}
\includegraphics[scale=0.25]{andy1}
\label{fig:andy1}
\end{figure}

\noindent The figure shows how the normalized velocity error follows a similar $sin(2\theta)$ trend to what was observed in the IRB model.

\end{document}

Weekly Reports/Weekly Report 621/main.tex


\documentclass{article}

\title{\sc\LARGE CSCA67 Tutorial, Week 2\\
{\Large Sept. 21st-25th, 2015}}
\date{}
\author{\sc Compiled by {\em G. Singh Cadieux}\\[1ex]
\sc Adapted from\\
A. Bretscher, \href{http://www.utsc.utoronto.ca/~bretscher/a67/lectures/w2.pdf}{\em CSCA67 Week 2 Lecture Notes},\\
\href{http://www.intmath.com/counting-probability/3-permutations.php}{\em Interactive Mathematics: Permutations (Ordered Arrangements)},\\
\href{http://www.intmath.com/counting-probability/4-combinations.php}{\em Interactive Mathematics: Combinations (Unordered Selections)},\\
Hunter, David J. \textit{Essentials of Discrete Mathematics.} Jones \& Bartlett, 2012. \&\\
Lov\'{a}sz, et al. \textit{Discrete Mathematics: Elementary and Beyond.} Springer, 2003.}

\usepackage{fullpage}
\usepackage{amsmath,amssymb}
%\usepackage{color}
%\usepackage{multicol}
\usepackage{tikz}
\usepackage{hyperref}

\setlength{\parindent}{0pt}

\begin{document}
\maketitle

\section{\sc Review of week 2's lecture}
\subsection{\em Sum rule vs. product rule}
\begin{tabular}{p{0.5\textwidth}|p{0.5\textwidth}}
{\bf Sum Rule}&{\bf Product Rule}\\[0.5ex]\hline\hline\noalign{\smallskip}
Used for counting the \textbf{total \# of possible outcomes} of an operation which can be performed $n$ ways, each with some number of outcomes.&
Used for counting the \textbf{total \# of possible ways} an operation can be performed when the operation takes $k$ steps, each of which can be performed in some number of ways.\\\hline
\begin{equation*}
\sum\limits_{i=1}^n x_i
\end{equation*}
where way $i$ has $x_i$ possible outcomes.&
\begin{equation*}
\prod\limits_{i=1}^k x_i
\end{equation*}
where step $k$ can be performed in $x_i$ ways.\\\hline\noalign{\smallskip}
\multicolumn{2}{c}{\em From Week 1's pizza example:}\\\hline\noalign{\smallskip}
Used for counting the number of combinations of up to 5 pizza toppings.\newline
Recall:\newline
\# of combinations of up to 5 toppings =
\begin{flushright}
\# of combinations of no toppings +\\
\# of combinations of 1 topping +\\
\ldots +\\
\# of combinations of 5 toppings
\end{flushright}
&
Used for counting the number of combinations of 5 pizza toppings (and 4, 3, etc.).\newline
Recall:\newline
\# of (non-unique) combinations of 5 toppings =
\begin{flushright}
\# of ways to choose first topping $\times$\\
\# of ways to choose second topping $\times$\\
\ldots $\times$\\
\# of ways to choose fifth topping
\end{flushright}
\end{tabular}

\subsection{\em Permutation vs. combination}
\begin{tabular}{c|c}
{\bf Permutation}&{\bf Combination}\\[0.5ex]\hline\hline\noalign{\smallskip}
\textit{Ordered} arrangement of a group of objects.& \textit{Unordered} arrangement of a group of objects.\\[0.5ex]\hline\noalign{\smallskip}
Total \# of permutations of $n$ objects: $n!$&Total \# of combinations of $n$ objects: 1\\[0.5ex]\hline\noalign{\smallskip}
{\bf $r$-permutation:}&{\bf $r$-combination:}\\
permutation of $r$ objects from a group of $n$ objects&combination of $r$ objects from a group of $n$ objects\\[1ex]
Total \# of $r$-permutations: $\dfrac{n!}{(n-r)!}$&Total \# of $r$-combinations: $\dfrac{n!}{(n-r)!\,r!}$\\[0.5ex]\noalign{\smallskip}\hline\noalign{\smallskip}
Represented $P(n,r)$&Represented $C(n,r)$ or $\binom{n}{r}$
\end{tabular}\\[2ex]
\textsc{Consider first} the number of permutations of $n$ objects.\\[1ex]
This is equivalent to considering the number of ways we can arrange $n$ objects. We can see that this will require $n$ steps: the first step is selecting the first object in the arrangement; the second step is selecting the second object, and so on.\\[1ex]
Using the product rule, we determine that
\begin{multline*}
\text{\bf\# of ways of permuting }n\text{ \bf objects}=\text{\# of ways of selecting the first object }\times\\
\text{\# of ways of selecting the second object }\times\ldots\times\text{\# of ways of selecting the }n\text{th object}
\end{multline*}
When we select the first object, there are $n$ objects from which to choose, meaning that there are $n$ ways to select an object. When we select the second object, we select from the reduced group of $(n-1)$ objects, since we have already selected one; this means that there are $(n-1)$ ways to choose. Eventually, when we select the final, $n$th object, we select from a group of only 1 object, which can only be done in 1 way.\\[1ex]
Overall, we can see that
\begin{equation*}
\text{\# of ways of permuting }n\text{ objects}=n\times(n-1)\times\ldots\times 1=n!
\end{equation*}
\textsc{Using this result,} we can determine the number of $r$-permutations of $n$ objects.\\[1ex]
If we select a subset of $r$ objects from an overall group of $n$ objects, we ignore the remaining $(n-r)$ objects.\\[1em]
Consider the 2-permutations of \{A, B, C, D, E\}.\\[1ex]
If we select \{A, B\}, we have 2 possible 2-permutations: AB and BA. We can permute all 5 objects to obtain (among others) ABCDE, ABCED, and ABDEC. However, because we have selected only \{A, B\} to permute, and ABCDE, ABCED, and ABDEC contain the same permutation AB of \{A, B\}, these 3 are equivalent here.\\[1ex]
To determine the number of 2-permutations, we can diminish the number of 5-permutations (permutations of all 5 objects) by the number of 5-permutations we consider to be equivalent.\\[1ex]
Having selected 2 ordered objects, the remaining 3 objects can be permuted $3!$ times: by the same reasoning as above, we can select the third object from 3 in 3 ways; then we can select the fourth object from 2 in 2 ways; and finally, we can select the fifth object from the remaining 1 object in only 1 way.\\
Since we are interested only in the order of the first 2 objects we selected, there are $3!$ ways that the remaining 3 objects can be selected to produce permutations we consider equivalent. So of the $5!$ total 5-permutations, we consider only 1 out of $3!$ to be a unique 2-permutation.\\
And thus, there are $\dfrac{5!}{3!}=20$ 2-permutations of \{A, B, C, D, E\}.\\[1em]
We can generalize this to say that
\begin{equation*}
\text{total \# of }r\text{-perms.}=\dfrac{\text{total \# of perms. of }n\text{ objects}}{\text{total \# of perms. of the remaining }(n-r)\text{ objects}}=\dfrac{n!}{(n-r)!}
\end{equation*}\\[1em]
\textsc{We can use this result} to determine the number of $r$-combinations of $n$ objects.\\[1ex]
Unlike an $r$-permutation, an $r$-combination is unordered. From all possible $r$-permutations of $n$ objects, we have multiple equivalent $r$-combinations.\\[1em]
Consider again the 2-permutations of \{A, B, C, D, E\}.\\[1ex]
If we select \{A, B\}, we have 2 possible 2-permutations: AB and BA. However, these are equivalent $r$-combinations, since both are composed of the same objects. For every possible 2-permutation \{object1, object2\}, there is another 2-permutation \{object2, object1\} that contains the same objects in the reverse order.\\[1ex]
To determine the number of 2-combinations, we can diminish the total number of 2-permutations by the number of 2-permutations which contain the same group of 2 objects.\\
Here, there are $\dfrac{5!}{3!}\dfrac{1}{2}=\dfrac{20}{2}=10$ 2-combinations of \{A, B, C, D, E\}.\\[1em]
We can generalize this to say that, in a group of $n$ objects, the number of $r$-permutations containing the same group of $r$ objects is $r!$, the number of ways that $r$ objects can be ordered. So of the $\dfrac{n!}{(n-r)!}$ total $r$-permutations, only 1 out of $r!$ is a unique $r$-combination.
\begin{equation*}
\text{total \# of }r\text{-combs.}=\dfrac{\text{total \# of }r\text{-perms. of }n\text{ objects}}{\text{total \# of perms. of the selected }r\text{ objects}}=\dfrac{n!}{(n-r)!\,r!}
\end{equation*}

\subsection{\em Permutation with repetition}
Consider the group of letters that form the word ``banana": \{b, a, n, a, n, a\}.\\[1ex]
\textsc{Q: How many ways can we ``arrange" (here, permute) these letters?}\\[1em]
If all 6 letters were distinct, we know that there would be $6!$ arrangements. We come to this total by multiplying together the 6 ways to choose the first letter from 6 possibilities, and the 5 ways to choose the second letter from 5 possibilities, and so on.\\[1ex]
But because the letters here are not unique, there are fewer than 6 different ways to choose the first letter, and fewer than 5 different ways to choose the second, and so on.\\[1em]
\textsc{Suppose} that we convert this to a problem in which all the letters are distinct: \{b, a$_1$, n$_1$, a$_2$, n$_2$, a$_3$\}.\\[1ex]
Then the arrangements ``ba$_1$a$_2$a$_3$n$_1$n$_2$", ``ba$_1$a$_2$a$_3$n$_2$n$_1$", and ``ba$_1$a$_3$a$_2$n$_1$n$_2$" are distinct. But without the distinguishing subscripts - that is, where all `a's are considered equivalent and all `n's are considered equivalent - these arrangements are not distinct.\\[1ex]
We can re-order equivalent letters to form equivalent arrangements.\\
For example, we can form the same arrangement ``baanna" by ordering the `a's and `n's as ``ba$_1$a$_2$n$_1$n$_2$a$_3$", ``ba$_1$a$_2$n$_2$n$_1$a$_3$", ``ba$_1$a$_3$n$_1$n$_2$a$_2$", and ``ba$_2$a$_3$n$_1$n$_2$a$_1$", among other ways. What differs between these is the order of the equivalent letters, but not their overall position in the arrangement.\\[1ex]
Thus, the total number of equivalent arrangements is the total number of ways we can order the equivalent letters: we can order 3 `a's in $3!$ ways, and 2 `n's in $2!$ ways. Then, by the product rule, we can order both `a's and `n's in $3!\times 2!$ ways.
\begin{equation*}
\text{total \# of distinct arrangements}=\dfrac{\text{total \# of arrangements}}{\text{\# of equivalent arrangements}}=\dfrac{6!}{3!\,2!}=60
\end{equation*}

\textsc{Alternatively,} rather than constructing an arrangement by selecting the first letter of the arrangement, followed by the second letter, and so on, let us select one letter out of the group and assign it a place within the arrangement. Then, we select another letter from the group and assign it one of the remaining places in the arrangement. We continue this until all the letters from the group have been placed into the arrangement. For example,
\begin{center}
\begin{tikzpicture}
\draw (0,0) -- (0.5,0) node[below,align=center] {letter$_1$} -- (1,0);
\draw (1.25,0) -- (1.75,0) node[below,align=center] {letter$_2$} -- (2.25,0);
\draw (2.5,0) -- (3,0) node[below,align=center] {letter$_3$} -- (3.5,0);
\draw (3.75,0) -- (4.25,0) node[below,align=center] {letter$_4$} -- (4.75,0);
\draw (5,0) -- (5.5,0) node[below,align=center] {letter$_5$} -- (6,0);
\draw (6.25,0) -- (6.75,0) node[below,align=center] {letter$_6$} -- (7.25,0);
\node[above,align=center] at (1.75,0) {a};
\node[above,align=center] at (3,0) {b};
\node[above,align=center] at (5.5,0) {a};
\node[font=\large] at (8.5,0.5) {a};
\node[font=\large] at (9.5,0.4) {n};
\node[font=\large] at (8.75,0) {n};
\draw (8.25,-0.25) rectangle (9.75,0.75);
\end{tikzpicture}
\end{center}
Then, the total number of possible arrangements is the product of the number of places possible for the first letter selected, and the number of places possible for the second letter selected, and so on.\\[1ex]
If all 6 letters are distinct, then this gives us 6 possible places for the first letter, 5 possible places for the second, and so on, which is $6!$ in total, as expected.\\[1ex]
However, because the letters here are not all distinct, if we assign one letter to letter$_1$ and another instance of that letter to letter$_3$, this is equivalent to having first assigned that letter to letter$_3$ and then, another instance of it to letter$_1$. Yet we counted fewer possible places for the first assignment than the second assignment.\\[1ex]
To deal with these equivalences, we assign places to all identical letters simultaneously; that is, we assign places to all 3 `a's simultaneously, and to both `n's simultaneously.\\[1ex]
For example, this means that the number of possible places for `n's is $\dbinom{\text{\# of unassigned places}}{2}$.\\
Note that this is no different if the letters are all distinct: $\binom{6}{1}\binom{5}{1}\binom{4}{1}\binom{3}{1}\binom{2}{1}\binom{1}{1}=6\times 5\times 4\times 3\times 2\times 1=6!$.\\[1ex]
Thus, by this reasoning,
\begin{align*}
\text{total \# of distinct arrangements}& =\dbinom{\text{\# of unass. places}}{\text{\# of `a's}}\dbinom{\text{\# of unass. places}}{{\text{\# of `n's}}}\dbinom{\text{\# of unass. places}}{{\text{\# of `b's}}}\\
& =\dbinom{6}{3}\dbinom{6-3}{2}\dbinom{6-3-2}{1}\\
& =\dfrac{6!}{(6-3)!\,3!}\dfrac{3!}{(3-2)!\,2!}\dfrac{1!}{(1-1)!\,1!}\\
& =\dfrac{6!}{3!\,2!\,1!}=60
\end{align*}

\textsc{In general,} there are
\begin{equation*}
\dbinom{n}{r_1}\dbinom{n-r_1}{r_2}\ldots\dbinom{n-r_1-r_2-\ldots-r_{m-1}}{r_m}=\dfrac{n!}{r_1!\,r_2!\,\ldots\,r_m!}
\end{equation*}
distinct arrangements/permutations of $n$ objects divided into $m$ types, where $r_i$ is the number of objects of type $i=1,2,\ldots,m$.\\[1ex]
Note that the $n$ objects are \textit{partitioned} into $m$ types; that is, $r_1+r_2+\ldots+r_m=n$.

\subsection{\em Combination/selection with repetition}

Consider a situation in which a supermarket has 3 kinds of bagels: poppy seed, sesame seed, and plain.\\[1ex]
\textsc{Q: If you want to buy 6 bagels, how many combinations of different kinds can you get?}\\[1ex]
If we were selecting 6 bagels from a group of $n$ unique bagels, we know that there would be $\dbinom{n}{6}=\dfrac{n!}{(n-6)!\,6!}$ possible combinations.\\[1ex]
But because we are selecting 6 bagels \textit{with repetition} (we have only 3 types from which to choose, so we must have more than 1 of at least 1 type of bagel), some of these $\binom{n}{6}$ combinations are equivalent.\\[1ex]
For example, suppose that we convert this to a problem in which all the bagels are unique. Then the combinations
\begin{center}
\begin{tikzpicture}
\draw (0,0) rectangle (1,1);
\draw (1.25,0) rectangle (2.25,1);
\draw (2.5,0) rectangle (3.5,1);
\draw (3.75,0) rectangle (4.75,1);
\draw (5,0) rectangle (6,1);
\draw (6.25,0) rectangle (7.25,1);

\filldraw[red] (0.5,0.5) circle (0.35);
\filldraw[white] (0.5,0.5) circle (0.15);

\filldraw[red] (1.75,0.5) circle (0.35);
\filldraw[white] (1.75,0.5) circle (0.15);

\filldraw[red] (3,0.5) circle (0.35);
\filldraw[white] (3,0.5) circle (0.15);

\filldraw[cyan] (4.25,0.5) circle (0.35);
\filldraw[white] (4.25,0.5) circle (0.15);

\filldraw[green] (5.5,0.5) circle (0.35);
\filldraw[white] (5.5,0.5) circle (0.15);

\filldraw[green] (6.75,0.5) circle (0.35);
\filldraw[white] (6.75,0.5) circle (0.15);

\node [font=\small] at (0.5,0.5) {1};
\node [font=\small] at (1.75,0.5) {2};
\node [font=\small] at (3,0.5) {3};

\node [font=\small] at (5.5,0.5) {1};
\node [font=\small] at (6.75,0.5) {2};
\end{tikzpicture}\\
and\\[1ex]
\begin{tikzpicture}
\draw (0,0) rectangle (1,1);
\draw (1.25,0) rectangle (2.25,1);
\draw (2.5,0) rectangle (3.5,1);
\draw (3.75,0) rectangle (4.75,1);
\draw (5,0) rectangle (6,1);
\draw (6.25,0) rectangle (7.25,1);

\filldraw[red] (0.5,0.5) circle (0.35);
\filldraw[white] (0.5,0.5) circle (0.15);

\filldraw[red] (1.75,0.5) circle (0.35);
\filldraw[white] (1.75,0.5) circle (0.15);

\filldraw[red] (3,0.5) circle (0.35);
\filldraw[white] (3,0.5) circle (0.15);

\filldraw[cyan] (4.25,0.5) circle (0.35);
\filldraw[white] (4.25,0.5) circle (0.15);

\filldraw[green] (5.5,0.5) circle (0.35);
\filldraw[white] (5.5,0.5) circle (0.15);

\filldraw[green] (6.75,0.5) circle (0.35);
\filldraw[white] (6.75,0.5) circle (0.15);

\node [font=\small] at (0.5,0.5) {2};
\node [font=\small] at (1.75,0.5) {3};
\node [font=\small] at (3,0.5) {5};

\node [font=\small] at (5.5,0.5) {3};
\node [font=\small] at (6.75,0.5) {4};
\end{tikzpicture}
\end{center}
are distinct. But without the distinguishing identifiers - that is, where all bagels of the same type are considered equivalent - these combinations are not distinct.\\[1ex]
\textsc{Since all} bagels of the same type are equivalent, rather than considering how to select 6 bagels one by one for the arrangement, let us consider how to divide our arrangement into 3 groups of types of bagels.\\
For example, we can divide our 6 possibilities into a group of 1 poppy seed bagel, a group of 4 sesame seed bagels, and a group of 1 plain bagel:
\begin{center}
\begin{tikzpicture}
\draw (0,0) rectangle (1,1);
\draw (1.25,0) rectangle (2.25,1);
\draw (2.5,0) rectangle (3.5,1);
\draw (3.75,0) rectangle (4.75,1);
\draw (5,0) rectangle (6,1);
\draw (6.25,0) rectangle (7.25,1);
\draw (7.5,0) rectangle (8.5,1);
\draw (8.75,0) rectangle (9.75,1);

\filldraw[red] (0.5,0.5) circle (0.35);
\filldraw[white] (0.5,0.5) circle (0.15);

\filldraw[cyan] (3,0.5) circle (0.35);
\filldraw[white] (3,0.5) circle (0.15);

\filldraw[cyan] (4.25,0.5) circle (0.35);
\filldraw[white] (4.25,0.5) circle (0.15);

\filldraw[cyan] (5.5,0.5) circle (0.35);
\filldraw[white] (5.5,0.5) circle (0.15);

\filldraw[cyan] (6.75,0.5) circle (0.35);
\filldraw[white] (6.75,0.5) circle (0.15);

\filldraw[green] (9.25,0.5) circle (0.35);
\filldraw[white] (9.25,0.5) circle (0.15);

\draw (1.25,0) -- (2.25,1);
\draw (7.5,0) -- (8.5,1);
\end{tikzpicture}
\end{center}
To mark the separation between groups, we use an empty pigeonhole. To divide our bagels into 3 groups, we require 2 pigeonholes to act as separators, and 6 filled pigeonholes + 2 empty pigeonholes = 8 piegeonholes in total.\\[1ex]
\textsc{In general,} to partition any number of objects into $n$ groups, we require $n-1$ empty pigeonholes.\\[1ex]
We can divide our 6 possibilities into 3 groups by selecting which pigeonholes will be filled and which will be empty. For example, if we select all but these pigeonholes to be filled
\begin{center}
\begin{tikzpicture}
\draw (0,0) rectangle (1,1);
\draw (1.25,0) rectangle (2.25,1);
\draw (2.5,0) rectangle (3.5,1);
\draw (3.75,0) rectangle (4.75,1);
\draw (5,0) rectangle (6,1);
\draw (6.25,0) rectangle (7.25,1);
\draw (7.5,0) rectangle (8.5,1);
\draw (8.75,0) rectangle (9.75,1);

\draw (2.5,0) -- (3.5,1);
\draw (3.75,0) -- (4.75,1);
\end{tikzpicture}
\end{center}
then our groups are 2 poppy seed bagels, 0 sesame seed bagels, and 4 plain bagels:
\begin{center}
\begin{tikzpicture}
\draw (0,0) rectangle (1,1);
\draw (1.25,0) rectangle (2.25,1);
\draw (2.5,0) rectangle (3.5,1);
\draw (3.75,0) rectangle (4.75,1);
\draw (5,0) rectangle (6,1);
\draw (6.25,0) rectangle (7.25,1);
\draw (7.5,0) rectangle (8.5,1);
\draw (8.75,0) rectangle (9.75,1);

\filldraw[red] (0.5,0.5) circle (0.35);
\filldraw[white] (0.5,0.5) circle (0.15);

\filldraw[red] (1.75,0.5) circle (0.35);
\filldraw[white] (1.75,0.5) circle (0.15);

\filldraw[green] (5.5,0.5) circle (0.35);
\filldraw[white] (5.5,0.5) circle (0.15);

\filldraw[green] (6.75,0.5) circle (0.35);
\filldraw[white] (6.75,0.5) circle (0.15);

\filldraw[green] (8,0.5) circle (0.35);
\filldraw[white] (8,0.5) circle (0.15);

\filldraw[green] (9.25,0.5) circle (0.35);
\filldraw[white] (9.25,0.5) circle (0.15);

\draw (2.5,0) -- (3.5,1);
\draw (3.75,0) -- (4.75,1);
\end{tikzpicture}
\end{center}
\textsc{The question} then becomes: in how many ways can we choose our 2 empty pigeonholes, or our 6 filled pigeonholes?\\[1ex]
Since these pigeonholes are unordered, the answer is simply $\dbinom{8}{6}=28$.\\[1ex]
\textsc{In general,} there are
\begin{equation*}
\dbinom{\text{total \# of pigeonholes}}{\text{\# of objects being selected}}=\dbinom{r+(n-1)}{r}
\end{equation*}
distinct combinations of $r$ objects from a group of $n$ types of objects.

\section{\sc Counting problems: Permutations and combinations}

\subsection*{Q: {\em How many ways can the letters A, B, and C be arranged?}}
We are being asked how many different ways 3 distinct objects, the letters A, B, and C, can be permuted.\\[1ex]
There are $n!\Rightarrow 3!=6$ permutations of 3 objects.

\subsection*{Q: {\em How many ways can 4 different resistors be arranged in series?}}
Again, we are being asked how many different ways 4 distinct objects - 4 resistors - can be permuted.\\[1ex]
There are $n!\Rightarrow 4!=24$ permutations of 4 objects.

\subsection*{Q: {\em How many ways can a supermarket manager display 5 brands of cereal in 3 spaces on a shelf?}}
We are again being asked about an arrangement, or permutation, of multiple objects. However, here we must determine the number of $r$-permutations: permutations of 3 brands out of the 5 possible brands.\\[1ex]
There are $\dfrac{n!}{(n-r)!}\Rightarrow\dfrac{5!}{(5-3)!}=60$ permutations of 3 objects from 5.

\subsection*{Q: {\em How many different license plates for cars can be made if each contains 4 of the digits 0-9, followed by a letter A-Z, assuming}}
\subsubsection*{a) {\em no repetition of digits?}}
\begin{tikzpicture}
\draw[thick,rounded corners](0,0)rectangle(5.25,1.45);
\draw[thick,rounded corners](-.1,-.1)rectangle(5.35,1.55);
\node[color=blue] at (2.62,1.15) {\sc Ontario};
\draw(0.25,0.2) -- (0.625,0.2) node [align=center,above,font={\Large\bf},color=blue] {D$_1$} -- (1,0.2);
\draw(1.25,0.2) -- (1.625,0.2) node [align=center,above,font={\Large\bf},color=blue] {D$_2$} --(2,0.2);
\draw(2.25,0.2) -- (2.625,0.2) node [align=center,above,font={\Large\bf},color=blue] {D$_3$} --(3,0.2);
\draw(3.25,0.2) -- (3.625,0.2) node [align=center,above,font={\Large\bf},color=blue] {D$_4$} --(4,0.2);
\draw(4.25,0.2) -- (4.625,0.2) node [align=center,above,font={\Large\bf},color=blue] {L} --(5,0.2);
\end{tikzpicture}\\[1ex]
Let us consider all 4 digits as a single entity \textbf{D}. Then, using the multiplication principle,
\begin{align*}
\text{\# of combinations of \textbf{D} and \textbf{L}}& =\text{\# of ways of choosing \textbf{D}}\times\text{\# of ways of choosing \textbf{L}}\\
& =\text{\# of perms. of 4 digits}\times\text{\# of letters}
\end{align*}
The number of permutations of 4 digits, with no repetition of digits, is the number of 4-permutations of 10 objects, since there are 10 digits from which we select. So
\begin{align*}
\text{\# of combinations of \textbf{D} and \textbf{L}}& =\left(\dfrac{10!}{(10-4)!}\right)\times 26\\
& =5040\times 26\\
& =131\,040
\end{align*}

\subsubsection*{b) {\em possible repetition of digits?}}
If repetition of digits (using the same digit more than once) is possible, then we are no longer dealing with permutations. Instead, using the product rule, we determine that the number of ways of choosing \textbf{D} is the product of the number of ways of choosing the first digit D$_1$, and then the second digit D$_2$, and so on.\\[1ex]
Since repetition is possible, at each step, we are choosing from the same set of 10 digits, meaning that there are 10 ways to choose each digit.
\begin{align*}
\text{\# of combinations of \textbf{D} and \textbf{L}}& =\text{\# of ways of choosing \textbf{D}}\times\text{\# of ways of choosing \textbf{L}}\\
& =\left(\text{\# of ways of choosing D}_1\times\ldots\times\text{\# of ways of choosing D}_4\right)\times\text{\# of letters}\\
& =\left(10\times 10\times 10\times 10\right)\times 26\\
& =260\,000
\end{align*}

%\subsection*{Q: {\em How many different sets of 4 letters can be selected from the alphabet?}\\
%{\normalsize Eg., \{A, E, R, T\} and \{E, A, T, R\} are the same set, but are different from \{Q, D, P, T\}.}}
%We are being asked how many different ways 4 letters from the alphabet can be combined. We know that these are combinations rather than permutations because sets are, by definition, unordered.\\[1ex]
%There are $\dfrac{n!}{(n-r)!\,r!}\Rightarrow \dfrac{26!}{(26-4)!\,4!}=14\,950$ combinations of 4 objects from 26.
%
%\subsection*{Q: {\em In how many ways can 3 components be selected from a batch of 20 different components?}}
%Again, we are being asked how many different ways 3 objects from a group of 20 objects can be combined.\\[1ex]
%There are $\dfrac{n!}{(n-r)!\,r!}\Rightarrow \dfrac{20!}{(20-3)!\,3!}=1140$ combinations of 3 objects from 20.

\subsection*{Q: {\em In how many ways can a group of 4 boys be selected from 10 if}}
\subsubsection*{a) {\em the eldest boy is included in each group?}}
If the eldest boy is included in each group, we select only the remaining 3 boys in the group; and we select these other 3 group members from 9 rather than from 10, since we have already selected 1 boy.\\[1ex]
So we are asked how many different ways 3 objects can be combined from a group of 9 objects.\\[1ex]
There are $\dfrac{n!}{(n-r)!\,r!}\Rightarrow \dfrac{9!}{(9-3)!\,3!}=84$ possible groups that include the eldest boy.

\subsubsection*{b) {\em the eldest boy is excluded from all groups?}}
If the eldest boy is excluded from all groups, we select 4 boys to form a group, but we select from 9 rather than from 10, since we have eliminated 1 possible boy.\\[1ex]
There are $\dfrac{n!}{(n-r)!\,r!}\Rightarrow \dfrac{9!}{(9-4)!\,4!}=126$ possible groups that exclude the eldest boy.

\subsubsection*{c) {\em What proportion of all possible groups contain the eldest boy?}}
Since there are only two possibilities - either a group includes the eldest boy or it does not - and both possibilities cannot be true simultaneously - a group cannot simultaneously include \textit{and} exclude the eldest boy - the total number of possible groups must be the sum of the number of groups that include the eldest boy and the number of groups that do not.
\begin{align*}
\text{total \# of possible groups}& =\text{\# of groups incl. the eldest boy}+\text{\# of groups excl. the eldest boy}\\
& =84+126\\
& =210
\end{align*}
Alternatively, we can calculate the total number of possible groups as the total number of combinations of 4 boys from 10, with no restrictions:
\begin{equation*}
\text{total \# of possible groups}=\dfrac{10!}{(10-4)!\,4!}=210
\end{equation*}
The proportion of groups containing the eldest boy is then
\begin{equation*}
\dfrac{\text{\# of groups incl. the eldest boy}}{\text{total \# of possible groups}}=\dfrac{84}{210}=40\%
\end{equation*}

\section{\sc Counting problems with repetition}

\subsection*{Q: {\em How many different ways can the letters of the word ``mammal" be arranged?}}
This question is directly analogous to the ``banana" example above: we want to determine the number of arrangements of a group of objects, where some objects are of the same type.\\[1ex]
Specifically, we have $n=6$ letters, $r_1=3$ of which are `m's, $r_2=2$ of which are `a's, and $r_3=1$ of which are `l's.
\begin{equation*}
\dfrac{n!}{r_1!\,r_2!\,r_3!}\Rightarrow\dfrac{6!}{3!\,2!\,1!}=60\text{ ways to arrange ``mammal"}
\end{equation*}

\subsection*{Q: {\em How many different ways can 3 red, 4 yellow, and 2 blue bulbs be arranged in a string of Christmas lights with 9 sockets?}}
Again, we are asked to determine the number of arrangements of a group of objects with repetition.\\[1ex]
We have $n=9$ bulbs, $r_1=3$ of which are red, $r_2=4$ of which are yellow, and $r_3=2$ of which are blue.
\begin{equation*}
\dfrac{n!}{r_1!\,r_2!\,r_3!}\Rightarrow\dfrac{9!}{3!\,4!\,2!}=1260\text{ ways to arrange bulbs in 9 sockets}
\end{equation*}
\textsc{Note} that we are able to use the formula for calculating the number of arrangements with repetition, derived above, because we are asked to arrange 9 objects (in 9 sockets) and have $3+4+2=9$ objects in total.\\
That is,
\begin{equation*}
r_1+r_2+\ldots+r_m=3+4+2=9=n
\end{equation*}

%\subsection*{{\normalsize You want to send postcards to 12 friends. In the shop, there are only 3 kinds of postcards.}\\
%Q: {\em How many ways can you send the postcards if}}
%\subsubsection*{a) {\em there is a large number of each kind, and you want to send 1 card to each friend?}}
%If we are sending 1 postcard to 12 friends, we must select 12 postcards in total. However, because there are only $n=3$ kinds of postcards, some friends will receive the same kind of card.\\
%So we are being asked how many ways we can select $r=12$ postcards, with repetition. We can determine this using the formula for selection with repetition, derived above:
%\begin{equation*}
%\binom{r+(n-1)}{r}\Rightarrow\binom{12+(3-1)}{12}=\dfrac{14!}{12!\,(14-12)!}=91\text{ ways to select 12 postcards}
%\end{equation*}
%
%\subsubsection*{b) {\em there is a large number of each kind, and you are willing to send 1 or more cards to each friend, but no one should get more than 1 of the same kind of card?}}
%We are again being asked how many ways we can select \textit{at least} 12 postcards, with repetition. We cannot send more than 3 to any friend because there are only 3 types, and so, we would be sending at least more than 1 of the same type.\\[1ex]
%The number of cards we want to select can be as few as 12 (if we only send 1 to each friend) or as many as 36 (if we send 3 to each friend). This makes it difficult to use the same method as in \textbf{(a)}. Instead, we can use the product rule:
%\begin{equation*}
%\text{\# of ways to send 1-3 cards 12 times}=\underbrace{\text{\# of ways to send 1-3 cards}}_\text{friend1}\times\ldots\times\underbrace{\text{\# of ways to send 1-3 cards}}_\text{friend12}
%\end{equation*}
%Then, using the sum rule, we determine that, for each friend,
%\begin{multline*}
%\text{\# of ways to send 1-3 cards}=\\
%\text{\# of ways to send 1 card }+\text{ \# of ways to send 2 cards }+\text{ \# of ways to send 3 cards}
%\end{multline*}
%Since we cannot have repetition when selecting 2 or 3 postcards, and there are only 3 unique postcards from which to choose, we find the number of ways to choose $r=1,2,3$ postcards from $n=3$:
%\begin{align*}
%\text{\# of ways to send 1-3 postcards}& =\binom{3}{1}+\binom{3}{2}+\binom{3}{3}\\
%& =3+3+1\\
%& =7
%\end{align*}
%And finally, we determine that
%\begin{equation*}
%\text{\bf\# of ways to send 1-3 postcards 12 times}=7^{12}=13\,841\,287\,201
%\end{equation*}
%
%\subsubsection*{c) {\em there are only 4 of each kind, and you want to send 1 card to each friend?}}
%There are 4 of 3 kinds of postcards, meaning that there are $n=12$ cards in total from which to choose.\\[1ex]
%If we suppose the order in which we select the postcards to be meaningful - that is, we send the first card we select to friend1, the second to friend2, etc. - then we are being asked to determine the number of ways in which we can select 12 cards, where some cards are of the same type.
%\begin{equation*}
%\dfrac{n!}{r_1!\,r_2!\,r_3!}\Rightarrow\dfrac{12!}{4!\,4!\,4!}=34\,650\text{ ways to send 12 postcards}
%\end{equation*}

\subsection*{Q: {\em How many solutions are there to the equation $x_1+x_2+x_3+x_4+x_5=13$ if $x_1,\ldots,x_5$ must be non-negative integers?}}
We can consider this question to be asking how many ways we can distribute 13 units among the five variables $x_1,\ldots,x_5$. (We can make the question more intuitive by imagining that, as in the example above, we have five types of bagels and want to select 13 bagels in total.)\\[1ex]
For example, the solution $x_1=4,\,x_2=0,\,x_3=5,\,x_4=1,\,x_5=3$ distributes 13 1's into the five groups:
\begin{equation*}
\underbrace{\;1\;1\;1\;1\;}_{x_1}|\underbrace{\;}_{x_2}|\underbrace{\;1\;1\;1\;1\;1\;}_{x_3}|\underbrace{\;1\;}_{x_4}|\underbrace{\;1\;1\;1\;}_{x_5}
\end{equation*}
Note that a unique solution is a set of assignments to $x_1,\ldots,x_5$: thus, $x_1=4,\,x_2=0,\,x_3=5,\,x_4=1,\,x_5=3$ is a different solution than $x_1=5,\,x_2=1,\,x_3=3,\,x_4=4,\,x_5=0$.
\begin{equation*}
\binom{r+(n-1)}{r}\Rightarrow\binom{13+(5-1)}{13}=\dfrac{17!}{13!\,(17-13)!}=2380\text{ unique solutions}
\end{equation*}

\section{\sc Additional practice problems}

At a competition of 100 athletes, only the order of the first 10 is recorded.\\
{\bf Q: How many different outcomes does the competition have?}\\[1ex]
Suppose we record the order of all 100 athletes.\\
{\bf Q: How many different outcomes can we have then?}\hfill(1)\\[1ex]
{\bf Q: How many of the outcomes in {\normalfont(1)} give the same result for the first 10 places?}\hfill(2)\\[1ex]
{\bf Q: Show that the number of outcomes for the first 10 places can also be obtained using {\normalfont(1)} and {\normalfont(2)}.}\\[1em]
Possible grades for a class are A, B, C, D, and F. (No +/-'s).\\
{\bf Q: How many ways are there to assign grades to a class of seven students?}\\[1ex]
{\bf Q: How many ways are there to assign grades to a class of seven students, if nobody receives an F and exactly one person receives an A?}\\[1em]
{\bf Q: How many ways are there to arrange the letters in the word ``inaneness"?}\\[1em]
Two teams, A and B, play a best-of-seven match. The match ends when one team wins four games.\\
{\bf Q: How many different win or loss scenarios are possible?}

\end{document}


\chapter{Basic Statistics in Spreadsheets}

When you completed the problems in the last section, you probably noticed
how long it took to compute statistics like the mean, the median,
and variance by hand. Luckliy, computers were designed to free us from these
sorts of tedious tasks. The most basic tool for automating
calculations is the spreadsheet program.\index{spreadsheet}

There are lots of spreadsheet programs including Microsoft's Excel and
Apple's Numbers. Any spreadsheet program will work; they are all very
similar. The instructions and screenshots here will be from Google
Sheets -- a free spreadsheet program you use through your web browser.

\section{Your First Spreadsheet}

In whatever spreadsheet program you are using, create a new spreadsheet document.

A spreadsheet is essentially a grid of cells. In each cell you can put data (like numbers or text) and a formulas.

\includegraphics[width=0.6\textwidth]{BlankSheet.png}

Let's put some labels in the column:
\begin{itemize}
\item Select the first cell (A1) and type ``A number''.
\item Select the cell below it (A2) and type ``Another number''.
\item Select the cell below that one (A3) and type ``Their product''.
\item In the next column, type the number 5 in B1 and 7 in B2.
\end{itemize}

It should look like this:

\includegraphics[width=0.5\textwidth]{NoFormulas.png}

Now put a formula in cell B3. Select B3, and type ``= B1 + B2''. The spreadsheet knows this is a formula because it starts with `=`. It will look like this as you type:\index{Spreadsheet!Entering formula}

\includegraphics[width=0.5\textwidth]{TypingFirstFormula.png}

When you press Return or Tab, the spreadsheet will remember the formula, but display its value:

\includegraphics[width=0.5\textwidth]{FirstCalc.png}

If you change the values of cell B1 or B2, the cell B3 will automatically be recalculated. Try it.

\section{Formatting}

Every spreadsheet lets you change the formatting of your columns and cells. They are all a little different, so play with your spreadsheet a little now. Try to do the following:
\begin{itemize}
\item Set the background of the first column to light gray.
\item Right-justify the text in the first column.
\item Make the text in the first column bold.
\item Make the numbers in the second column have one digit after the decimal point.
\end{itemize}

It should look something like this:

\includegraphics[width=0.6\textwidth]{FirstFormatting.png}

That's a spreadsheet. You have grid of cells. Each cell can hold a
value or a formula that uses values from other cells. The cells with
formulas automatically update as you edit the values in the other
cells.

\section{Comma-Separated Values}

A lot of data is exchanged in a file format called
\textit{Comma-Separated Values} or just CSV. Each CSV file holds one
table of data. It is a text file, and each line of text corresponds to
one row of data in the table. The data in each column is separated by
a comma. The first line of a CSV is usually the names of the
columns. A CSV might look like this:

\begin{Verbatim}
studentID,firstName,lastName,height,weight
1,Marvin,Sumner,260,45.3
2,Lucy,Harris,242,42.2
3,James,Boyd,261,44.2
\end{Verbatim}

In your digital resources for this module, you should have a file
called \path{1000cars.csv}. It is a CSV with only one column called
``speed''. The first few lines look like this:

\begin{Verbatim}
speed
33.8000
29.9920
34.8699
27.9936
\end{Verbatim}

There is a title line and 1000 data lines.

Import this CSV into your spreadsheet program. In Google Sheets, it looks like this:

\includegraphics[width=0.5\textwidth]{ImportingCSV.png}

You should see a long, long column of data appear. (Mine goes from cell A2 through A1001.)

\includegraphics[width=0.5\textwidth]{ImportedCSV.png}

\section{Statistics in Spreadsheets}

Let's take the mean all 1000 numbers.  In cell B2, type in a label:
``Mean''. (Feel free to format your labels as you wish. Bolding is recommended.)

In cell C2, enter the formula ``=AVERAGE(A2:A1001)''. When
you press return, the cell will show the mean: 31.70441, if done correctly .

\includegraphics[width=0.4\textwidth]{Spread_mean.png}

Notice that by specifing that the function \pyfunction{AVERAGE} was to
be performed on a range of cells: cells A2 through A1001.

Do the calculations for variance, standard deviation, and median.

\begin{itemize}
\item The function for variance is \pyfunction{VAR}.
\item The function for standard deviation is \pyfunction{STDEV}.
\item The function for median is \pyfunction{MEDIAN}.
\end{itemize}

\includegraphics[width=0.4\textwidth]{var_stdev_median.png}

\section{Histogram}

Most spreadsheets have the ability to create a histogram. In Google
Sheets, you select the entire range A2:A1001 by selecting the first
cell and then shift-clicking the last. Then you choose
Insert$\rightarrow$Chart. In the inspector, change the type of the
chart to histogram. This will get you a basic histogram.
% Add: Define histogram or give example, defined in basic statistics, must come previous

\includegraphics[width=0.7\textwidth]{default_histogram.png}

Play with the formatting to see how unquie you can make data. Here is an example:

\includegraphics[width=0.8\textwidth]{final_histogram.png}

\begin{Exercise}[title={RMS}, label=rms_spreadsheet]

  In your spreadsheet, calculate the quadratic mean (the root-mean-squared) of the speeds.

  You will need the following three functions:
  \begin{itemize}
  \item \pyfunction{SUMSQ} returns the sum of the squares of a range of cells.
  \item \pyfunction{COUNT} returns the number of cells in a range that contain numbers.
  \item \pyfunction{SQRT} returns the square root of a number.
  \end{itemize}

\end{Exercise}
\begin{Answer}[ref=rms_spreadsheet]

The formula for the RMS is ``=SQRT(SUMSQ(A2:A1001)/COUNT(A2:A1001))''.
% KA: https://www.khanacademy.org/computing/ap-computer-science-principles/data-analysis-101/data-tools/a/learning-from-data-sets

\end{Answer}


\section{Proof of Theorem~\ref{th:main}}\label{se:proof}
\begin{proof}
To simplify our proof we introduce the notion of \emph{canonical scenario} for the the ledger $\ledger_2$. In a canonical scenario the ledger
$\ledger_2$ is executed in the standard way. More precisely,
we assume the existence of a genesis block and that $\asmp_2[\tau]$=1 for all $\tau \geq 0$. Let $\parties$ be the set of parties that is running $\ledger_2$. Also,
let $t_j$ be the smallest time slot in which $B_j$ appears in  $\state_2^{P_i}[t_j]$ for each $P_i\in\parties$ and let $t_{j+k}$ be smallest time slot in which $B_{j+k}$ appears in %\nnote{$\state_2^{P_i}[t_{j+k}]$}
$\state_2^{P_i}[t_{j+k}]$ for each $P_i\in\parties$.
We are now ready to prove the security of $\Pi$.

In the protocol $\Pi$, by assumption, we have that $\asmp_2[\tau]=1$ for all $\tau \geq t$.
%From the description of $\Pi$ we can claim that $t \leq t_{P_1}+k\cg^{-1}=t_{P_1}+\Delta_1$.
%That is, when an honest party is activated she waits to be sure that also the other honest parties are activated. %Hence, from the chain-growth and the common-prefix parameters
%each party needs to wait at most $\Delta_1=k\cg^{-1}$ time slots.
From the moment when $\asmp_2$ becomes true the activation process takes $\Delta\leq k \cg^{-1} + k \cg^{-1}$  time slots to be completed.
This is because the parties need to wait that the $(j+k)$-th block of $\state_1$ is part of $\check \state_1^{P_i}[t]$  for all $P_i$ and that $k$ blocks are generated in $\state_2$. Note that when $k$ blocks are generated in $\state_2$ at least
$k$ blocks are generated in $\state_1$ since $\ledger_2$ and $\ledger_1$ have the same parameters and that the honest parties that maintain $\ledger_1$ are greater or equal than the parties that maintain $\ledger_2$. Therefore, the parties need to wait the \emph{special genesis block} of $\ledger_2$ to appear
in $\state_2^P$ for each honest $P\in\activep$. Given that a block in $\ledger_1$ ($\ledger_2$) takes at most $\cg^{-1}$ time slots then we have that $\Delta \leq k \cg^{-1} + k \cg^{-1}$.
In the moment that a \emph{candidate} block $B^i_{j+k}$ becomes available to an honest party $P_i\in\activep$ (i.e., $B^i_{j+k}$ is part of $\check \state_1^{P_i}$) then she starts running $\ledger_2$
using ${B^i}'$ which is computed from $B^i_{j+k}$ as described earlier (we recall that at this time slot the assumption $\asmp_2$ holds).
Let $t'$ be the smallest time slot in which $B_{j+k}$ appears in $ \state_2^{P_i}[t']$
%\nnote{$\state_2^{P_i}[t']$}
 for each $P_i\in\parties$.
If we take the execution of the protocol from time $t$ and $t'$ this can be seen as a canonical execution of $\ledger_2$ given
that the parameters of $\ledger_1$ and $\ledger_2$ are the same. The only difference between this and the canonical scenario is
that the blocks $B_{j}, B_{j+1},\dots, B_{j+k}$  are generated using $\ledger_1$, but this does not represent an issue since we are assuming that
any block of $\state_1$ can be turned into a block of $\state_2$. \end{proof}