\documentclass{article}
\usepackage{cite}
\usepackage[utf8]{inputenc}
\newcommand\independent{\protect\mathpalette{\protect\independenT}{\perp}}
\def\independenT#1#2{\mathrel{\rlap{$#1#2$}\mkern2mu{#1#2}}}
\title{Report for Comprehensive Exam}
\author{Siyuan Zhao}
\date{April 2017}

\usepackage[linesnumbered,ruled]{algorithm2e}
\usepackage{amsmath, mathtools}
\usepackage{amsfonts}
\begin{document}

\maketitle

\tableofcontents

\newpage

\section{Topic 1: Ways to Personalized}
\subsection{Q1: Estimating Treatment Effect}
\paragraph{Question:} You have explored at least one way to personalize.  That is to try to learn in an RCT which conditions that should be given to which types of students.   Review the methods you are aware (1) Susan Athey’s Random Causal Forest and 2) Counterfactual inference for causal inference/individual treatment effect, , 3) individual treatment rules from the medical literature and 4) Susan Murph's methods) of and how are they different and similar.
\\ [0.1in]

\paragraph{Strong Ignorability}
The treatment assignment is defined to be strongly ignorable
\cite{rosenbaum1983central} if the following two conditions hold: 1).
$(Y(1), Y(0)) \independent t\mid X$ and 2). $0 < \Pr (t=1 \mid X) <
1$. The first condition says that treatment assignment is independent
of the potential outcomes conditional on the observed features of subjects. The second condition says that every subject has a nonzero
probability to receive either treatment. The aforementioned first
condition is also referred to as the "no-hidden confounding variables"
assumption that all factors determining the outcome of each condition
are observed.

\subsubsection{Random Causal Forests}~\label{sect:rcf}
\cite{wager2015estimation} proposed random causal forest (RCF) to
infer treatment effect. From the conceptual point of view, causal trees can be viewed as
nearest neighbor methods with an adaptive neighborhood metric. Assume
that there are $n$ observed independent samples $(X_i, Y_i,
W_i)_{i=1}^n$. The workflow of the RCF is that a causal tree is first
built by recursively splitting the feature space until all samples are
partitioned into a set of leaves $L$, each of which contains a few
training samples. Then, for a data point $x$, the predicted outcome $\hat{\mu}(x)$
is evaluated by identifying the leaf $L(x)$ containing $x$ and
calculating

$$\hat{\mu} = \frac{1}{\left | \left \{ i: X_i \in L(x) \right \}
  \right |} \sum_{\left \{ i: X_i \in L(x) \right \}} Y_i$$

Given a test point $x$, the closest points to $x$ are those fall in the same
leaf as it. The authors believe that the leaf is small enough that the
responses $Y_i$ are roughly identically distributed. Then the
treatment effect $\hat{\tau}$ for any $x \in L(x)$ is estimated as
following:

\begin{align}
\hat{\tau}(x) & = \frac{1}{\left | \left \{ i: W_i=1, X_i \in L(x) \right \}
                \right |} \sum_{\left \{ i: W_i = 1, X_i \in L(x)
                \right \}} Y_i \nonumber \\
              & - \frac{1}{\left | \left \{ i: W_i=0, X_i \in L(x) \right \}
  \right |} \sum_{\left \{ i: W_i = 0, X_i \in L(x) \right \}} Y_i \label{eq:tree-treatment-effect}
\end{align}

RCF assumes that
there is overlapping in the data, i.e., for some $\epsilon > 0$ and
all $x \in \left [ 0, 1\right ]^d$,

$$\epsilon < \mathbb{P}\left [ W=1 \mid X=x \right ] < 1-\epsilon$$

This condition effectively guarantees that, for large enough $n$,
there will be enough treatment and control units near any test point
$x$ for local methods to work.

\paragraph{Honest Trees and Forests}
The authors also define the Honest Trees and Forests in \cite{wager2015estimation}.
A tree is honest if, for each training example $i$, it only uses the
response $Y_i$ to estimate the within-leaf treatment effect $\tau$
using Equation~\ref{eq:tree-treatment-effect} or
to decide where to place the splits, but not
both. There are two causal forest algorithms proposed from
\cite{wager2015estimation} that satisfy this condition.

The first algorithm, which is called double-sample trees, achieves
honesty by dividing its training data into two halves
$\mathcal{I}$ and $\mathcal{J}$. Then, it uses the
$\mathcal{J}$-sample to place the splits, while holding out the
$\mathcal{I}$-sample to do within-leaf estimation. The details of the
algorithm is shown in Algorithm~\ref{algo:double-sample-trees}.

\LinesNumberedHidden
\begin{algorithm}[h]
  \SetKwInOut{Input}{Input}
  \Input{$n$ training examples of $(X_i, Y_i, W_i)$, where $X_i$ are
    features, $Y_i$ is the response, and $W_i$ is the treatment
    assignment. \\
    A minimum leaf size $k$.
  }

  \caption{Double-sample Causal Trees}~\label{algo:double-sample-trees}
  \ShowLn Draw a random subsample of size $s$ from $\{1, \ldots, n\}$ without
  replacement, and then divide it into two disjoint sets of size
  $|\mathcal{I}|=\left \lfloor s/2 \right \rfloor$ and
  $|\mathcal{J}|=\left \lceil s/2 \right \rceil$ \\
  \ShowLn Grow a tree via recursive partitioning. The splits are chosen using
  any data from the $\mathcal{J}$ sample and $X$- or $W$-observations
  from the $\mathcal{I}$ sample, but without using $Y$-observations
  from $\mathcal{I}$-sample. \\
  \ShowLn Estimate leaf-wise response using only the $\mathcal{I}$-sample
  observations. \\ [1\baselineskip]

The algorithm estimates $\hat{\tau}(x)$ using
Equation~\ref{eq:tree-treatment-effect} on the $\mathcal{I}$
sample. The splitting criteria is to maximize the variance of
$\hat{\tau}(X_i)$ for $i \in \mathcal{J}$. Each leaf of the tree must
contain $k$ or more $\mathcal{I}$-sample observations of each
treatment class.
\end{algorithm}
Another approach to build honest trees is inspired by the idea of
propensity score matching, which is called propensity trees. The idea
behind this approach is to train a classification tree to predict the
propensity score, which is the treatment assignments $W_i$, and ignore the outcome $Y_i$ when
placing splits. This method is useful in observational studies
since propensity score is aimed to reduce the selection bias by
equating groups based on these features. The details of the
algorithm is shown in Algorithm~\ref{algo:propensity-trees}.

\LinesNumberedHidden
\begin{algorithm}[h]
  \SetKwInOut{Input}{Input}
  \Input{$n$ training examples of $(X_i, Y_i, W_i)$, where $X_i$ are
    features, $Y_i$ is the response, and $W_i$ is the treatment
    assignment. \\
    A minimum leaf size $k$.
  }
  \ShowLn Draw a random subsample $\mathcal{I} \in {1, \ldots, n}$ of
  size $|\mathcal{I}| = s$ (no replacement). \\
  \ShowLn Train a classification tree using sample $\mathcal{I}$ where
  the outcome is the treatment assignment, i.e., on the $(X_i, W_i)$
  pairs with $i \in \mathcal{I}$. Each leaf of the tree must have $k$
  or more observations of each treatment class. \\
  \ShowLn Estimate $\tau(x)$ using
  Equation~\ref{eq:tree-treatment-effect} on the leaf containing $x$.
  \caption{Propensity Trees}~\label{algo:propensity-trees}
\end{algorithm}

\subsubsection{Counterfactual Inference}~\label{sect:bnn}
The problem of causal inference is often framed in terms of
counterfactual problems such as "Would this particular student benefit
more from the video hint or the text hint when the student cannot
solve a problem?" In the binary intervention set, there are two possible interventions
$\mathcal{T} = \left \{  0, 1 \right \}$, where intervention 1 is
often referred as the "treated" and intervention 0 is the "control."
Given a sample of subjects and a treatment, each subject has a pair of
potential outcomes: $Y_0$ and $Y_1$, the outcomes under the
control and the treatment, respectively. Let $t$ be an indicator
variable denoting the treatment received ($t = 0$ for the control and
$t=1$ for the treatment). Only one outcome, which is called factual
outcome, $y_F(\mathbf{x}) =t\cdot Y_1(\mathbf{x}) +
(1-t) \cdot Y_0(\mathbf{x})$, is observed for the
subject $\mathbf{x}$, where
$\mathbf{x} \in \mathcal{X}$ are the
observed features for the subject. The unobserved outcomes are referred to as the
counterfactual outcome, denoted as $y_{CF}(\mathbf{x}) = (1-t)\cdot Y_1(\mathbf{x}) +
t \cdot Y_0(\mathbf{x})$. In other words, when a subject $\mathbf{x}$ is assigned to the
"control" ($t = 0$), $y_F(\mathbf{x})$ is equal to $Y_1(\mathbf{x})$, and $y_{CF}(\mathbf{x})$ is
equal to $Y_0(\mathbf{x})$. The other way around, $y_F(\mathbf{x})$ is equal to
$Y_0(\mathbf{x})$, and $y_{CF}(\mathbf{x})$ is equal to $Y_1(\mathbf{x})$. The estimated
individual treatment effect (ITE) is then calculated by
$\mathrm{ITE}(\mathbf{x})=Y_1(\mathbf{x}) - Y_0(\mathbf{x})$. The goal
of counterfactual inference is to predict the counterfactual outcome
given the observed data $D_n=\{(\mathbf{x}_i,t_i,y_F^i)_{i=1}^n\}$ from either RCTs or observational studies in
order to estimate the ITE.

We assume $n$ samples $\left \{ (x_i, t_i, y_F^i) \right \}_{i=1}^n$
form an empirical distribution $\hat{p}^F  = \left \{ (x_i, t_i)
\right \}_{i=1}^n$. We call this empirical distribution $\hat{p}^F
\sim p^F$ the empirical factual distribution. In order to calculate
ITE, we need to infer the counterfactual outcome which is dependent on
the empirical distribution $\hat{p}^{CF}  = \left \{ (x_i, 1-t_i)
\right \}_{i=1}^n$, which is called the empirical counterfactual distribution $\hat{p}^{CF}
\sim p^{CF}$. The $p^{F}$ and $p^{CF}$ may not be equal because the
distributions of the control and the treated populations may be
different. The inequality of two distributions may cause the
counterfactual inference over a different distribution than the one
observed from the experiment. In machine learning terms, this scenario
is usually referred to as domain adaptation, where the distribution of
features in test data are different than the distribution of features
in training data.

\cite{Johansson2016-dh} proposed Balancing Neural Networks (BNN) which
can be applied to solve the counterfactual inference problem. They
used a form of regularizer to enforce the similarity between the
distributions of representations learned for populations with
different interventions, for example, the representations for students
who received text hints versus those who received video hints.This
reduces the variance from fitting a model on one distribution and
applying it to another. Because of random assignment to the
interventions in RCTs, the distributions of the populations within
different interventions are highly likely to be identical. However, in
the observational study, we may end up with the situation where only
male students receive video hints and female students receive text
hints. Without enforcing the similarity between the distributions of
representations for male and female students, it is not safe to make a
prediction of the outcome if male students receive text hints.

The Counterfactual Regression (CFR) \cite{Shalit2016-qk} is built on the BNN. The important difference between these two models is that the CFR uses a more powerful distribution metric in the form of IPMs to learn a balancing representation.

\begin{figure}[h]
  \centering
  \includegraphics[width=0.95\columnwidth]{cfr.png}
  \caption{CFR for ITE estimation. $L$ is a loss function, IPM is an integral probability metric}.
  ~\label{fg:cfr-model}
\end{figure}

The structure of CFR is illustrated in Figure~\ref{fg:cfr-model}. To learn a representation of deep features $\Phi$, the CFR uses fully connected layers with ReLu activation function, where $Relu(z) = max(0, z)$. We need to generalize from factual distribution to counterfactual distribution in the feature representation $\Phi$ to obtain accurate estimation of counterfactual outcome. The common successful approaches for domain adaptation encourage similarity between the latent feature representations w.r.t the different distributions. This similarity is often enforced by minimizing a certain distance between the domain-specific hidden features. The distance between two distributions is usually referred to as the discrepancy distance, introduced by \cite{Mansour2009-fh}, which is a hypothesis class dependent distance measure tailored for domain adaptation.

Integral Probability Metric (IPM) are used to measure the distance between two distributions $p_0 = p(x|t = 0)$, and $p_1 = p(x|t = 1)$, also known as the control and treated distributions. The IPM for $p_0$ and $p_1$ is defined as
$$\mathrm{IPM}_{\mathcal{F}}(p_0, p_1) := \sup_{f\in \mathcal{F}} \left |\int_S f dp_0 -\int_S f d p_1 \right |,$$

where $\mathcal{F}$ is a class of real-valued bounded measurable functions on $S$.

The choice of functions is the crucial distinction between IPMs \cite{Sriperumbudur2009-pf}. Two specific IPMs are used in our experiments: the Maximum Mean Discrepancy (MMD), and the Wasserstein distance. $\mathrm{IPM}_{\mathcal{F}}$ is called MMD, when $\mathcal{F} = \left \{ f : \left \| f \right \| _\mathcal{H}\leq 1\right \}$, where $\mathcal{H}$ represents a reproducing kernel Hilbert space (RKHS) with $k$ as its reproducing kernel. In other words, the family of norm-1 reproducing kernel Hilbert space (RKHS) functions lead to the MMD. The family of 1-Lipschitz functions $\mathcal{F} = \left \{ f:\left \| f\right \|_L \leq 1 \right \}$, where $\left \| f\right \|_L$ is the Lipschitz semi-norm of a bounded continuous real-valued function $f$, make IPM the Wasserstein distance. Both the Wasserstein and MMD metrics have consistent estimators which can be efficiently computed in the finite sample case \cite{Sriperumbudur2012-sz}. The important property of IPM is that $$p_0 = p_1~\mathrm{iff}~ \mathrm{IPM}_{\mathcal{F}}(p_0, p_1) = 0.$$

The representation with reduction of the discrepancy between the control and the treated populations helps the model to focus on balancing features across two populations when inferring the counterfactual outcomes. For instance, if in an experiment, almost no male student ever received intervention A, inferring how male students would react to intervention A is highly prone to error and a more conservative use of the gender feature might be warranted.

After obtaining the representation for subject $\mathbf{x_i}$, CFR
concatenates the treatment assignment $t_i$ to the representation $\Phi(\mathbf{x}_i)$ and feeds $[\Phi(\mathbf{x}_i), t_i]$ to another two fully connected layers to generate the predicted outcome.
\subsubsection{Individualized Treatment Rules}
An individual treatment rule (ITR) $d: \mathcal{X} \rightarrow \mathcal{T}$ is a deterministic decision
rule from subject $\mathbf{x} \in \mathcal{X}$ into the intervention space $\mathcal{T}$. In experiments, we observe a triplet $(\mathbf{x}, t, y)$ from each
subject, where $\mathbf{x}=(x_1, x_2, \ldots, x_n)^T \in
\mathcal{X}$ denotes the participant's observed features, $t \in \mathcal{T}
= {-1,1}$ denotes the treatment assignment, and $y \in Y$ is the
observed outcome, also called the "reward" in the literature on
reinforcement learning. Note that $t=-1$ means that the subject is
assigned to the control and $t=1$ mean that the subject is assigned to
the treatment in the
context of ITR. Let $p(t|\mathbf{x})$ be the probability of assigning the subject
with features $\mathbf{x}$ to the intervention $t$. \cite{Qian2011-vz}
showed that the
value of $d$ satisfies
$$V(d) = E \left [ \frac{y}{p(t|x)}\mathbb{I}_{t=d(\mathbf{x})}\right
],$$
where $\mathbb{I}(\cdot)$ is an indicator function. The goal of the
ITR is to find an optimal ITR, $d^{*}$, which is a rule that has the maximal value such
that,

$$d^{*} \in \operatorname*{arg\,max}_d V(d).$$
\cite{Qian2011-vz} also showed that finding $d^{*}$ is equivalent to minimizing the following equation:
\begin{equation} \label{eq:d_min}
d^{*} \in \operatorname*{arg\,min}_d E \left [
  \frac{y}{p(t|x)}\mathbb{I}_{t \neq d(\mathbf{x})}\right
]
\end{equation}

Assume that the observed data $\left \{
  (\mathbf{x}_i,t_i,y_i),~i=1,\ldots , n \right \}$ are collected
independently. For any decision function $f(\mathbf{x})$, let
$d_f(\mathbf{x}) = sign(f(\mathbf{x}))$ be the associated rule, where
$sign(u) = 1$ for $u > 0$ and $-1$ otherwise. The particular choice of
the value of $sign(0)$ is not important. With the observed data, the
weighted classification error in Equation \ref{eq:d_min} can be approximated by
the empirical risk

\begin{equation} \label{eq:d_empirical_min}
  \frac{1}{n}\sum_{i=1}^{n}\frac{y_i}{p(t_i|x_i)}\mathbb{I}_{t_i \neq
    d_f(\mathbf{x_i})} .
\end{equation}

\cite{Qian2011-vz} proposed a two-step procedure that first train a
model using the observed data to estimate
a conditional mean for the outcome $E(y|\mathbf{x},t)$, and then determines the treatment
rule by comparing the predicted value $E(y|\mathbf{x},t=1)$ between $E(y|\mathbf{x},t=-1)$.

In contrast, \cite{zhao2012estimating} proposed one-step procedure by
showing that find an optimal ITR is equivalent to a classification task
where we want to classify $t$ with $\mathbf{x}$ as the input. Outcome weighted learning (OWL) proposed by
\cite{zhao2012estimating} use the hinge loss function and the
regularization technique aiming to minimize

\begin{equation}
  \frac{1}{n}\sum_{i=1}^{n}\frac{y_i}{p(t_i|x_i)}(1-t_if(\mathbf{x}_i))_{+}+\lambda\left
    \| f \right \|^2 ~\label{eq:d_owl_empirical_min},
\end{equation}
where $(u)_+=\max(u,0)$ is the positive part of $u$, $\left \| f
\right \|$ is some norm for $f$, and $\lambda$ is a tuning parameter
controlling the trade-off between empirical risk and complexity of the
decision function $f$.

\cite{Zhou_undated-ps} showed that decision rule in
Equation~\ref{eq:d_owl_empirical_min} is
affected by a simple shift of the outcome $y$ and proposed Residual
weighted learning (RWL) to relieve this issue. The idea behind RWL is
to introduce a function $g$ to reduce the variance of
$\frac{y-g(\mathbf{x})}{p(t|\mathbf{x})}\mathbb{I}_{t \neq
  d(\mathbf{x})}$ and a reasonable candidate of $g$ is
\begin{equation}
  g^{*}(\mathbf{x}) = \frac{\mathbb{E}(y|\mathbf{x},
    t=1)+\mathbb{E}(y|\mathbf{x}, t=-1)}{2} = \mathbb{E} \left (
    \frac{y}{2p(t, \mathbf{x})} | \mathbf{x} \right ).~\label{eq:g-choice}
\end{equation}

RWL thus is to minimize the following empirical risk:
\begin{equation}
   \frac{1}{n}\sum_{i=1}^{n}\frac{y_i-\hat{g}^{*}(\mathbf{x}_i)}{p(t_i|\mathbf{x_i})}\mathbb{I}_{t_i \neq
    d_f(\mathbf{x_i})}
  ~\label{eq:rwl-risk}
\end{equation}
where $\hat{g}^{*}$ is an estimate of $g^{*}$. For simplicity, let
$\hat{r}_i = y_i - \hat{g}^{*}(\mathbf{x}_i)$. As in OWL,
\cite{Zhou_undated-ps} consider a surrogate loss function $T$ to replace
the 0-1 loss in Equation~\ref{eq:rwl-risk}. The non-convex loss $T$
has the following form:
\begin{equation}
T(u) =  \left\{\begin{array}{ll}
 0 & \mathrm{if} \: u \geq 1, \\
 (1-u)^2 & \mathrm{if} \: 0 \leq u < 1, \\
 2-(1+u)^2 & \mathrm{if} \: -1 \leq u < 0, \\
 2 & \mathrm{if} \: u < -1
\end{array}\right.
\end{equation}
It is called the smoothed ramp loss in \cite{Zhou_undated-ps}. By
incorporating the regularization, RWL is eventually aimed to minimize
the following empirical risk:
\begin{equation}
    \frac{1}{n}\sum_{i=1}^{n}\frac{\hat{r}_i}{p(t_i|x_i)}T(t_if(\mathbf{x}_i))_{+}+\lambda\left
    \| f \right \|^2 ~\label{eq:d_owl_empirical_min},
\end{equation}
where $\left \| f \right \|$ is the norm for $f$, and $\lambda$ is a
tuning parameter.
\subsubsection{Comparison}
All three aforementioned methods (RCF,
Counterfactual Inference, ITR) can be trained on the data from either RCTs or
observational studies and can be used to assign a subject to a
predicted optimal condition. Propensity trees from RCF \cite{wager2015estimation} and BNN from
counterfactual inference \cite{Johansson2016-dh} are specifically
optimized for observational studies. Propensity tress uses the idea
from propensity score matching to reduce possible variance in features
of subjects between two conditions, while BNN uses deep learning
approaches for domain adaptation to enforce the similarity between the
distributions of representations of subjects from two condition.

Since ITR is a deterministic rule aimed to assign a subject to optimal
condition based on the observed features, ITR does not focus on
estimating the treatment effect. BNN predicts two potential outcomes
(the treatment outcome and the control outcome)
for each subject, so BNN is designed to estimate the individual
treatment effect. RCF assumes that subjects on the same leaf have the
same potential outcomes, so RCF usually estimates the treatment effect
on a small subset of populations from two conditions.

\subsection{Q2: Bayesian Optimization}
\paragraph{Question:} You have been doing stuff with Bandits. When might
be learning a function to predict goodness of hints across students be
a good idea?  Look into Bayesian Optimization.  How is that different
than what you are planning on doing.  How might you use such methods?
\\ [0.1in]

Bayesian optimization is a sequential model-based approach for global
optimization of an unknown objective function $f$. The problem can be
mathematically expressed as:

\begin{equation}
  \mathbf{x}^* = \underset{\mathbf{x} \in \mathcal{X}}{\mathrm{arg \,
      max} \, f(\mathbf{x})}
\end{equation}
where $\mathcal{X}$ is the design space of interest. Bayesian optimization is the combination of two main
components: a surrogate probabilistic model which captures our beliefs
about the behavior of the unknown objective function $f$ and is updated
once new observation is gathered, and an acquisition function $\alpha: \mathcal{X} \rightarrow \mathbb{R}$ which performs the selection
of the optimal sequence of queries based on the previous model. Since
the objective function $f$ is unknown, the Bayesian
strategy is to place a prior over it which captures our beliefs about the behavior of the
function. After gathering the function evaluations, which are treated
as data, the prior is updated to form the posterior distribution
over the objective function. Equipped with the posterior distribution,
an acquisition function $\alpha$
is induced to determines what the
next query point should be. The trade-off between the exploration and
the exploitation is determined by the acquisition function leveraging the
uncertainty in the posterior. Examples of acquisition functions includes
probability of improvement, expected improvement, Bayesian expected
losses, upper confidence bounds (UCB), and mixtures
of these. The purpose of the trade-off is to
minimize the number of function evaluations. As such, Bayesian
optimization is well suited for functions that are very expensive to
evaluate. The details of Bayesian
Optimization is shown in Algorithm~\ref{algo:bo}.

The Gaussian process (GP) is the most popular surrogate model. The
objective function is modeled as a sample from the GP distribution. Attributes of the
GP such as mean and variance are used by the acquisition function to
decide the successive query points. More information about GP in the
bandit setting can be found in Section~\ref{sect:gp}.

\begin{algorithm}[h]
  \SetKwInOut{Input}{Input}
  \Input{$n$ observations $\mathcal{D}=\{ (\mathbf{x}_i,
    y_i)\}_{i=1}^n$ from objective function $f$}

 \caption{Bayesian optimization}~\label{algo:bo}
 \For{$t = n+1,n+2,\ldots$} {
   select new $\mathbf{x}_{t}$ by maximizing acquisition function
   $\alpha$
   $$
     \mathbf{x}_{t} = \underset{\mathbf{x}}{\mathrm{arg \, max}} \, \alpha(\mathbf{x};\mathcal{D})
     $$\\
     query objective function to observe $y_{t} =
     f(\mathbf{x}_{t})$ \\
     augment data $\mathcal{D}=\{ \mathcal{D},
     (\mathbf{x}_{t}, y_{t}) \}$\\
     update statistical model
 }
\end{algorithm}

\subsubsection{Thompson Sampling}
Thompson sampling for Beta-Bernoulli bandit perhaps is the simplest non-trivial multi-armed
bandit strategy in Bayesian optimization \cite{Shahriari2016-ho}. Bernoulli bandit problem is
the bandit problem when the rewards are either 0 or 1, and for arm $i$
the probability of success (reward=1) is $\mu_i$. The Thompson
sampling maintains Bayesian priors on the Bernoulli means
$\mu_i$'s. Beta distribution turns out to be a very convenient choice
of priors for Bernoulli rewards. The pdf of $\mathrm{Beta}(\alpha,
\beta)$ with parameters $\alpha > 0, \beta > 0$ is given by
\begin{equation}
  f(x;\alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}.
\end{equation}

The mean of $\mathrm{Beta}(\alpha,\beta)$ is $\alpha / (\alpha +
\beta)$; and higher the $\alpha, \beta$, tighter is the concentration
of $\mathrm{Beta}(\alpha,\beta)$ around the mean. If the prior
for each arm is a $\mathrm{Beta}(\alpha,\beta)$ distribution, then
after observing a Bernoulli trial, the posterior distribution is
simply $\mathrm{Beta}(\alpha + 1,\beta)$ or
$\mathrm{Beta}(\alpha,\beta +1)$, depending on whether the trial
resulted in a success or failure, respectively. Thompson sampling then
samples from these posterior distributions across all arms and chooses
the arm with largest sample value. This procedure is summarized in
Algorithm~\ref{algo:thompson-sampling}.

\begin{algorithm}[h] \label{algo:thompson-sampling}
  \SetKwInOut{Input}{Input}
  \Input{$\alpha, \beta$: hyperparameters of the beta prior}

  \caption{Thompson sampling for Beta-Bernoulli bandit}
  Initialize $n_{a,0}=n_{a,1}=i=0$ for all $K$ arms $a$ \\
  \Repeat{stopping criterion reached} {
    \For{$a = 1,\ldots, K$}{
      $w_a \sim \mathrm{Beta}(\alpha + n_{a,1}, \beta + n_{a,0})$
    }
    $a_i = \underset{a}{\mathrm{arg\, max}}\, w_a$ \\
    Observe $y_i$ by pulling arm $a_i$ \\
    \eIf{$y_i = 0$}{
      $n_{a_i,0} = n_{a_i,0} + 1$

    }{
      $n_{a_i,1} = n_{a_i,1} + 1$
    }
    $i = i + 1$ \\
 }
\end{algorithm}

In summary, Thompson sampling is the simplest non-trivial multi-armed
bandit strategy in Bayesian Optimization. Since the Thompson sampling
models the arms as independent, Thompson sampling must try every arm
at least once. It will be an issue if the number of arms becomes
large. In this case, Bayesian optimization with GP is an alternative
to the Thompson sampling and alleviate this issue.

\subsubsection{Choice between function predictor and multi-armed bandit algorithm}
In many experiments, the designs space available to the designers have
components that can be varied independently. For example, in designing
an advertisement, one has choices such arkwork, font style, and
size. If there are five choices for each, the total number of possible
configurations is 125.

In general, this number grows combinatorially in the number of
components. This presents challenges for approaches such as the Thompson sampling, since the Thompson sampling models the arms as independent, which will lead
to strategies that must try every arm at least once. This rapidly
becomes infeasible in the large design spaces.

Even if the total number of possible design configuration is
relatively small, it is still challenging for multi-armed bandit
algorithms, such as Thompson sampling, when the number of available subjects is limited for these
algorithms to figure out the optimal arm.

To alleviate this issue, a commonly used approach is to learn a
function, whose input is a feature vector of the arm and output is
associated reward of the arm, capturing dependence between the arms. Assume that each possible arm $a$ has an
associated feature vector $\mathbf{x}_a\in \mathbb{R}^d$. The expected
reward of each arm can be expressed as a function of this feature
vector, such as $f(a) = f(\mathbf{x}_a)$. The goal is to learn this
function $f:\mathbb{R}^d \rightarrow \mathbb{R}$ from the experiment
data in order to choose the arm with the highest reward among all
possible arms.

\subsection{Q3: Contextual Bandit vs. Causal Inference}
\paragraph{Question:} In what conditions to use different ways to
personalize: Bandits (contextual) or Causal inference (Including Susan
Athey's Random Causal Forest and things like counterfactual inference
for personalized learning) \\ [0.1 in]

The ultimate goal of contextual bandit algorithms and causal inference
is the same: find the good intervention for each subject based on the
observed features. The contextual bandit algorithms are widely used in
the sequential experiments. In this setting, single interventions from
a pre-defined set are repeatedly performed to evaluate their goodness
via observed feedback. But explicit experiments may be difficult to be
conducted in some research areas. Thus, contextual bandit algorithms
cannot be directly applied and causal inference models are commonly
used here. The idea behind causal inference models is to predict the
outcome of an intervention from the observed data without explicitly
performing it.

\subsection{Q4: Online Learning vs. Batch Learning}
\paragraph{Question:} Why can BKT not update itself in real-time while
Thompson sampling can update itself online? \\ [0.1in]

There are four parameters in BKT that are learned from enough training
data. Before deploying BKT online, we need to collect reliable
training data and learn these four parameters offline. On the
contrary, Thompson sampling is non-parametric model, so collecting
training data is not necessary for it. Thompson sampling is a sequential
method, which means it makes use of observations one at a time, or in
small batches. Thus, it can be used in real-time sequential experiments.

Four parameters in BKT are usually learned via
Expectation-Maximization (EM) algorithm. EM algorithm is an iterative
method to find the optimal values for parameters. An iterative method
is a sequence of improving approximate solutions and requires
computational time and resources. It will take some amount of time to
update BKT parameters in real-time. On the contrary, Thompson sampling
has a simple close form to update its posterior distribution after
receiving new observations.

\subsection{Q5: Semi-Supervised Learning}
\paragraph{Question:} What is Semi-supervised learning?  How is that
related to Mitchel's CoLearning? Is the same idea under a different
name?  Can it be used to help solve your problems related to
PeerASSIST?  \\ [0.1 in]

In the setting of semi-supervised learning, the training data consists
of $l$ labeled instances $\left \{ (\mathbf{x}_i, y_i)
\right \}_{i=1}^{l}$ and $u$ unlabeled instances $\left \{ (\mathbf{x}_j)
\right \}_{j=l+1}^{l+u}$, often with $l \ll u$. The goal of
semi-supervised learning is to learn a model with better performance
from the combination of labeled data and unlabeled data than from
labeled data alone. The benefit of semi-supervised learning is that
the labeled data can be hard and expensive to obtain since labels may
require human experts, and the unlabeled data is often cheap in large
quantity. Figure~\ref{fg:ssl_model} illustrates how unlabeled data
help models achieve a better performance.

\begin{figure}[h]
  \centering
  \includegraphics[width=0.95\columnwidth]{ssl.png}
  \caption{An illustrative example on how semi-supervised learning helps
    the model have a better performance.}~\label{fg:ssl_model}
\end{figure}

\subsubsection{Co-training}
Co-training \cite{blum1998combining} is a technique in semi-supervised learning that requires two
views of the data. It assumes that each example is described using two
different feature sets that provide different, complementary
information about the example. Co-training first learns a separate
classifier for each view using any labeled examples. The most
confident predictions of each classifier on the unlabeled data are
then used to iteratively construct additional labeled training
data. The details of co-training algorithm is described in Algorithm~\ref{algo:co-training}.
\begin{algorithm}[h] \label{algo:co-training}
  \SetKwInOut{Input}{Input}
  \Input{labeled data $\left \{ (\mathbf{x}_i, y_i)
\right \}_{i=1}^{l}$, unlabeled data $\left \{ \mathbf{x}_j
\right \}_{j=l+1}^{l+u}$ \\
each instance has two views $\mathbf{x}_i=\left [ \mathbf{x}_i^{(1)},
  \mathbf{x}_i^{(2)} \right ]$ \\
and a learning speed $k$
}
Let $L_1 = L_2 = \left \{ (\mathbf{x}_1, y_1), \ldots, (\mathbf{x}_l, y_l)
\right \}.$ \\
\Repeat {unlabeled data is used up}{
  Train view-1 $f^{(1)}$ from $L_1$, view-2 $f^{(2)}$ from $L_2$ \\
  Classify unlabeled data with $f^{(1)}$ and $f^{(2)}$ separately. \\
  Add $f^{(1)}$'s top $k$ most-confident predictions $(\mathbf{x},
  f^{(1)}(\mathbf{x}))$ to $L_2$ \\
  Add $f^{(2)}$'s top $k$ most-confident predictions $(\mathbf{x},
  f^{(2)}(\mathbf{x}))$ to $L_1$ \\
  Remove these from the unlabeled data.
}

\caption{Co-training Algorithm for Semi-Supervised Learning}
\end{algorithm}

There are three important assumptions which guarantee the performance of
co-training algorithm: 1). feature split $x=[x^{(1)};x^{(2)}]$
exists; 2). $x^{(1)}$ or $x^{(2)}$ alone is sufficient to train a good
classifier; 3). $x^{(1)}$ and $x^{(2)}$ are conditionally independent
given the class.

\subsubsection{Application in PeerASSIST}
Semi-supervised learning approaches come in handy in PeerASSIST when
we try to learn a function to predict the effectiveness of peer
explanations. In the setting of PeerASSIST, not every peer explanation will be received by students,
and thus, some of the explanations have no observed data on their
effectiveness (e.g., the popularity, next problem correctness). When
learning the function, these peer explanations without observed data
are discarded, which is a waste of the data. Moreover, if there are not enough
observed data, it becomes difficult to accurately learn the
function. Semi-supervised learning approaches can be applied in this
scenario to learn a better function.

\section{Topic 2: Application of Deep Learning to EDM and related issues}
\subsection{Q1: Natural Language Processing for Enhancing Learning}
\paragraph{Question:} You have done some work in using memory network to
predict NLP classifications in a Kaggle Data set.   What are some
applications you can put those skills to that will drive new designs
that could improve student learning? I think I have heard to throw out
a few ideas (1) the evaluation of the quality of open response from
students, or 2) Which kids are improving or slacking? Or 3) Sentiment
analysis of posts or comments from students.  Is there a reasonable
way to think about how to use NLP in your PeerASSIST feature? Pitch a
complete idea of something you think practical  (you don't have to
want to do it, but if you did that would be nice). \\ [0.1 in]

\subsubsection{Confusion Detector}
Given the enormous student/instructor ratio in a MOOC's discussion
forums, it is difficulty for an instructor to read all posts in a
MOOC's discussion forums. To address this issue, \cite{Agrawal2015-hp}
collected and created the Stanford MOOCPosts dataset: a corpus
composed of 29,604 anonymized learner forum posts from eleven Stanford
University public online classes. Each post in the MOOCPosts dataset
was scored across six dimensions -- confusion, sentiment, urgency,
question, answer, and opinion -- and subsequently augmented with
additional metadata. Then the authors built a confusion classifier from
the Stanford MOOCPost dataset to automatically detect confusion from
students posts and proposed a recommendation algorithm, which takes the
student confusion as input, to automatically recommended a
video lecture from a collection of several video lectures related to
the course.
\subsubsection{Automated Essay Scoring}
Automated grading is a critical part of Massive Open Online Courses (MOOCs) system and any intelligent tutoring systems (ITS) at scale. Essay writing is usually a common student assessment process in schools and universities. In this task, students are required to write essays of various length, given a prompt or essay topic. Some standard tests, such as Test of English as a Foreign Language (TOEFL) and Graduate Record Examination (GRE), assess student writing skills. Manually grading these essay will be time-consuming. Thus automated essay scoring (AES) systems has been used in these tests to reduce the time and cost of grading essays. Moreover, as massive open online courses (MOOCs) become widespread and the number of students enrolled in one course increases, the need for grading and providing feedback on written assignments are ever critical.

AES has employed numerous efforts to improving its performance. AES uses statistical and Natural Language Processing (NLP) techniques to automatically predict a score for an essay based on the essay prompt and rubric. Most existing AES systems are built on the basis of predefined features, e.g. number of words, average word length, and number of spelling errors, and a machine learning algorithm \cite{Chen2013-zw}. It is normally a heavy burden to find out effective features for AES. Moreover, the performance of the AES systems is constrained by the effectiveness of the predefined features. Recently another kind of approach has emerged, employing neural network models to learn the features automatically in an end-to-end manner \cite{Taghipour2016-ns}. By this means, a direct prediction of essay scores can be achieved without performing any feature extraction. The model based on long short-term memory (LSTM) networks in \cite{Taghipour2016-ns} has demonstrated promise in accomplishing multiple types of automated grading tasks.

\subsubsection{NLP in PeerASSIST}
The goal of PeerASSIST is to crowdsource the explanation (in the
format of texts) of how to solve a
given problem from students and apply multi-armed bandit algorithm to
efficiently select the most helpful and useful peer explanation from all collected
peer explanations for a given problem. The effectiveness of each peer
explanations is measured in three dimensions: teacher's rating (thumb
up or thumb down), student's rating (thumb up or thumb down), and the
student next problem correctness after receiving the peer explanation.

A potential application of NLP techniques to PeerASSIST is to build a
classifier to predict the probability of a peer explanation being an
effective one based on the content of the explanation. As mentioned
above, NLP techniques, especially deep learning approaches for NLP,
have achieved promising results on text classification tasks. The
predicted probabilities can potentially used to select the optimal
explanation when one is needed to be delivered to the students. In
this scenario, we can deliver the explanation with the highest
probability to students. Another possible use of the predicted
probabilities is to treat these values as a part of the contextual
information for the explanations and incorporate this contextual
information into contextual multi-armed bandit algorithms.

The explanations collected from students are not always of good
quality. Since the classifier can automatically detect explanations
with bad quality (i.e., the explanations with low probability values),
one can quickly filter out these potential bad explanations, which
saves multi-armed bandit algorithm from wasting attempts trying these
explanations. When the explanations
that students type in are detected as bad ones by the classifier, we
can also build an intervention by
telling students that the algorithm thinks that their explanations
need some improvement. Then the students have a second chance to
submitting new explanations. We hope that there will be observable
improvement on the quality of peer explanations for these students.

\subsection{Q2a: Reliable Crowdsourcing}
\paragraph{Question:} Besides assessing the learning gains associated
with each learning artifacts, it can also be useful to characterize
them for certain features. Suppose you ask many students to rate a
bunch of learning artifacts along different dimensions, e.g., media
format, pedagogical approach, difficulty to understand, etc. How do
you know which learners to trust, and how do you combine their
opinions into aggregate labels? What kinds of algorithms exist for
this purpose and how do they work? \\ [0.1 in]

During this process of asking
students to rate the learning artifacts, we might
gather noisy rating from students due to the fact that students may
have a wide ranging level of rating expertise which are unknown, and
in some cases may be adversarial. To learn the ground truth of these
learning artifacts, we need to aggregate their opinions to recover the
true, unknown label of each learning artifacts.

\subsubsection{Majority Voting}
Given each item is labeled by different workers, it is a
straightforward approach to take the majority label as the true
label. From reported experimental results on real crowdcourcing data
\cite{Snow2008-rm}, majority voting performs significantly better on
average than individual workers. However, majority voting considers
each item independently and gives the same weight across all workers
who label the item when aggregating true label.

\subsubsection{Dawid-Skene Model}
\cite{dawid1979maximum} were among the first to consider such a
problem setup. They assume each workers are conditionally independent
given the true labels and each worker is associated with a
probabilistic confusion matrix that generates her labels. Each entry
of the matrix indicates the probability that items in one class are
labeled as another. Given the
observed responses, the true labels for each items and the confusion
matrices for each worker can be jointly estimated by a maximum
likelihood method. The optimization can be implemented by the
expectation-maximization (EM) algorithm.

\subsubsection{GLAD}
\cite{NIPS2009_3644} proposed a richer graphic model which includes
item difficulty and the expertise of the worker. The difficulty of
item is modeled by the parameter $1/\beta_j \in [0,\infty)$. Here $1/\beta_j = \infty$
means the image is very ambiguous and hence the most proficient worker
has a random chance of labeling it correctly. $1/\beta_j = 0$ means the item is
so easy that even the most obtuse worker will always label it correctly.

The expertise of each worker $i$ is modeled using the parameter
$\alpha_i \in (-\infty, +\infty)$. Here $\alpha = +\infty$ means the
worker always labels items correctly; $-\infty$ means the worker
always labels items incorrectly.

The labels given by worker $i$ to item $j$ are denoted as $L_{ij}$
and are generated as follows:

\begin{equation} \label{eq:glad_label}
  p(L_{ij}=Z_j|\alpha_i , \beta_j) = \frac{1}{1+e^{-\alpha_i \beta_j}}
\end{equation}

As the difficulty $1/\beta_j$ of an item increases, the probability of
the label being correct moves toward 0.5. Similarly, as the worker's
expertise decreases (lower $\alpha_i$), the chance of correctly
labeling drops to 0.5. Figure~\ref{fg:glad_model} shows the structure
of the graphical model.

\begin{figure}[h]
  \centering
  \includegraphics[width=0.95\columnwidth]{glad.png}
  \caption{An illustrative example on how semi-supervised learning helps
    the model have a better performance.}~\label{fg:glad_model}
\end{figure}

\subsubsection{Deep Learning Approach}
\cite{Shaham2016-nh} showed that the Dawid and Skene model is
equivalent to a Restricted Boltzmann Machine (RBM) with a single
hidden node under the assumption that all workers are conditionally
independent. Thus the posterior probabilities of the true labels can
be estimated via a trained RBM.

A RBM is an undirected bipartite graphical model, consisting of a
visible layer $X$ and a hidden layer $H$. The visible layer consists
of $d$ binary random variables and the hidden layer $m$ binary random
variables. These two layers are fully connected to each other. A RBM
is parametrized by $\lambda = (W, a, b)$, where $W$ is the weight
matrix of the connections between the visible and hidden units, and
$a,b$ are the bias vectors of the visible and hidden layers,
respectively. An illustration of a RBM is depicted in Figure~\ref{fg:rbm_model}.

A RBM implies the conditional probabilities

\begin{align*}
p_{\lambda}(X_i=1|H) &= \sigma (a_i+W_{i\cdot}H) \\
p_{\lambda}(H_j=1|X) &= \sigma (b_j+X^{T}W_{\cdot j}),
\end{align*}
where $\sigma (z)$ is the sigmoid function, $W_{i\cdot}$ is the $i$-th
row of $W$ and $W_{\cdot j}$ is its $j$-th column.
\begin{figure}[h]
  \centering
  \includegraphics[width=0.95\columnwidth]{rbm.png}
  \caption{Diagram of a restricted Boltzmann machine with four visible
  units and three hidden units (no bias units)} ~\label{fg:rbm_model}
\end{figure}

To relax the assumption on the conditional independence of the
variables $X_1,\ldots,X_d$, a RBM-based Deep Neural Net (DNN) is
proposed to estimate the posterior probabilities
$p_{\theta}(Y|X)$. The mechanism to stack multiple RBMs is that the
hidden layer of each RBM is the input for the successive RBM. The RBMs
are trained one at a time from bottom to top. Specifically, given
training data $x^{(1)}, \ldots, x^{(n)} \in \{ 0,1 \}^d$, the bottom
RBM is trained first, and then obtain the hidden representation of the
first layer by sampling $h^{(i)}$ from the conditional RBM
distribution $p_{\lambda}(H|X=x^{(i)})$. The vector $h^{(1)}, \ldots,
h^{(n)}$ are then used as a training set for the second RBM and so
on. An illustration of a RBM is depicted in Figure~\ref{fg:rbm-dnn-model}.

\begin{figure}[h]
  \centering
  \includegraphics[width=0.75\columnwidth]{rbm-dnn.png}
  \caption{Diagram of RBM-based DNN with two hidden layers.}
  ~\label{fg:rbm-dnn-model}
\end{figure}

\subsubsection{Minimax Entropy}
\cite{Zhou2012-ry} proposed a minimax entropy principle to estimate
the ground truth given the observed labels by workers. The model is
illustrated in Figure~\ref{fg:minimax_entropy_model}. Each row
corresponds to a worker indexed by $i$ (from 1 to $m$). Each column
corresponds to an item to be labeled, indexed by $j$ (from 1 to
$n$). Each item has an unobserved label represented as a vector
$y_{jl}$, which is 1 when item $j$ is in class $l$ (from 1 to $c$),
and 0 otherwise. Observed data is a tensor of labels $z_{ijk}$, which
is 1 when the item $i$ is labeled as class $k$ by the worker $i$, and
0 otherwise. We assume that $z_{ij}$ are drawn from
$\pi_{ij}$. $\pi_{ij}$ can be represented as a tensor $\pi_{ijk}$,
which is the probability that worker $i$ labels item $j$ as class
$k$. The proposed model will estimate $y_{jl}$ from the observed
$z_{ij}$.
\begin{figure}[h]
  \centering
  \includegraphics[width=0.95\columnwidth]{minimax_entropy.png}
  \caption{Left: observed labels. Right: underlying
    distributions. Highlights on both tables indicate that rows and
    columns of the distributions are constrained by sums over observations.}
  ~\label{fg:minimax_entropy_model}
\end{figure}

$\pi_{ij}$ is modeled through the principle of maximum entropy with the
constraints from the ideas of majority voting and Dawid and Skene's
model. Using the principle of maximum entropy to estimate the
probability distribution provides the largest remaining uncertainty
(i.e., the maximum entropy) consistent with the constraints. Majority voting indicates that the number of observed votes per
class per item  $\sum_{i}z_{ijk}$ should match
$\sum_{i}\pi_{ijk}$. Dawid and Skene's model indicates that the
observed confusion matrix per worker $\sum_{j}y_{jl}z_{ijk}$ should match $\sum_{j}y_{jl}\pi_{ijk}$. Thus, the maximum entropy model for $\pi_{ij}$ given $y_{jl}$ is as
follows:

\begin{align}
\max_{\pi} & -\sum_{i=1}^{m}\sum_{j=1}^{n}\sum_{k=1}^{c} \pi_{ijk}
\ln \pi_{ijk} \nonumber \\
  \mathrm{s.t.} & \sum_{i=1}^{m}\pi_{ijk} = \sum_{i=1}^{m}z_{ijk},
                  \forall j,k, ~ \sum_{j=1}^{n}y_{jl}\pi_{ijk} =
                  \sum_{j=1}^{n}y_{jl}z_{ijk} \forall i,k,l, \nonumber
  \\
  & \sum_{k=1}^{c}\pi_{ijk}=1, \forall i,j, \pi_{ijk} \geq 0, \forall
    i,j,k. \label{eq:max_entropy}
\end{align}

To infer $y_{jl}$, the authors proposed to choose $y_{jl}$ to minimize the
entropy in Equation~\ref{eq:max_entropy}, which leaves $z_{ij}$ the
least random given $y_{jl}$.

\begin{align}
\min_{y}\max_{\pi} & -\sum_{i=1}^{m}\sum_{j=1}^{n}\sum_{k=1}^{c} \pi_{ijk}
\ln \pi_{ijk} \nonumber \\
  \mathrm{s.t.} & \sum_{i=1}^{m}\pi_{ijk} = \sum_{i=1}^{m}z_{ijk},
                  \forall j,k, ~ \sum_{j=1}^{n}y_{jl}\pi_{ijk} =
                  \sum_{j=1}^{n}y_{jl}z_{ijk} \forall i,k,l, \nonumber
  \\
  & \sum_{k=1}^{c}\pi_{ijk}=1, \forall i,j, \pi_{ijk} \geq 0, \forall
    i,j,k, \sum_{l=1}^{c}y_{jl}=1, \forall j, y_{jl} \geq 0, \forall
    j, l. \label{eq:minimax_entropy}
\end{align}

\subsubsection{Control Items}
Control items with known answers can be used to evaluation workers'
reliability and bias, and hence weight their answers accordingly on target
items with unknown answers. There is a trade-off in this
scenario: we can better estimate workers' reliability and bias by
letting them answer more control items, but there are fewer
opportunities left for the
target items. On the other hand, using fewer control items produces poor
estimates of worker's reliability and bias, which leads to a bad
result when aggregating workers' answers. \cite{Liu2013-ck} examined
the effectiveness of control items and provided insights on how many control items are enough under
different scenarios, which can help crowdsourcing practitioners make
their own decisions. Aforementioned methods
(except for majority voting) evaluate workers' reliability by
comparing their agreement with other workers and control items can be
incorporated into these methods.

\subsection{Q2b: Gaussian Processes in Contextual
  Bandit}\label{sect:gp}
\paragraph{Question:} For the purposes of identifying the most effective
learning artifact, how do contextual bandits relate to Gaussian
processes? How are they different/similar? \\ [0.1 in]

The reward function in contextual bandit setting is usually an unknown
function and expensive to evaluate (performing an arm on a
subject). In this case, Gaussian process (GP) is an ideal method to model
our belief on the behavior of the reward function. The reward function
is modeled as a sample from GP.

The Gaussian process $\mathrm{GP}(\mu_0,k)$ is a non-parametric model
that is fully characterized by its prior mean function $\mu_0 :
\mathcal{X}\rightarrow \mathbb{R}$ and its positive-definite kernel,
or covariance function, $k: \mathcal{X} \times \mathcal{X} \rightarrow
\mathbb{R}$.

Let $\mathcal{D}_n = \left \{( \mathbf{x}_i, y_i \right )\}$ denote a
set of $n$ observations and $\mathbf{x}$ denote the arbitrary test
point. The random variable $f(\mathbf{x})$ is also a GP distribution
conditioned on observations $\mathcal{D}_n$ with following mean
$\mu_n(\mathbf{x})$ and variance $\sigma_n^2(\mathbf{x})$:
\begin{align}
 \mu_n(\mathbf{x}) &= \mu_0(\mathbf{x}) +
                     \mathbf{k}(\mathbf{x})^T(\mathbf{K}+\sigma^2\mathbf{I})^{-1}(\mathbf{y}-\mathbf{m})
  \\
  \sigma_n^2(\mathbf{x}) &= k(\mathbf{x}, \mathbf{x}) -
                           \mathbf{k}(\mathbf{x})^T(\mathbf{K}+\sigma^2\mathbf{I})^{-1}\mathbf{k}(\mathbf{x})
\end{align}
where $m_i := \mu_0(\mathbf{x}_i)$, $K_{i,j} := k(\mathbf{x_i, x_j})$,
and $\mathbf{k}(\mathbf{x})$ is a vector of covariance terms between
$\mathbf{x}$ and $\mathbf{x}_{1:n}$.

The posterior mean and variance evaluated at any point $\mathbf{x}$
represent the model's prediction and uncertainty in the objective
function at the point $\mathbf{x}$. In order to apply GP under the
setting of multi-armed bandit, we need a acquisition function to select
the next query point given the posterior model. The acquisition
function should be carefully design to trade off exploration of the
search area and exploitation of current promising areas.

\subsubsection{GP-UCB}
To decrease uncertainty globally, one strategy could be to pick the
point which maximums the variance $\mathbf{x}_{n+1}=
\underset{\mathbf{x} \in
  D}{\mathrm{arg\,max}}~\sigma_n(\mathbf{x})$. However, this strategy
is not well suited for multi-armed bandit problem since it can be
wasteful. Another idea is to
select the point which maximizes the expected reward given the
posterior model $\mathbf{x}_{n+1}=
\underset{\mathbf{x} \in
  D}{\mathrm{arg\,max}}~\mu_n(\mathbf{x})$. However, this idea is too
greedy and tends to end up with a local optima. To balance exploration
and exploitation, a combined strategy is to choose
\begin{equation}
  \mathbf{x}_{n+1} = \underset{\mathbf{x} \in D}{\mathrm{arg \,
      max}}\,\mu_n(\mathbf{x})+\beta_n \sigma_n(\mathbf{x}),
\end{equation} \label{eq:gp-ucb-query}
where $\beta_n$ are constants. There are theoretically motivated
guidelines for setting and scheduling the hyperparameter $\beta_n$ to
achieve optimal regret.

Since Equation~\ref{eq:gp-ucb-query} is an upper confidence bound of
the marginal posterior $P(f(\mathbf{x})|\mathbf{y}_n)$, a natural
interpretation of this strategy is that it selects the point
$\mathbf{x}$ such that $f(\mathbf{x})$ is a reasonable upper bound on
$f(\mathbf{x})$. This algorithm is called Gaussian process upper
confidence bound (GP-UCB), introduced by \cite{Srinivas2010-hi}. The
GP-UCB selection rule is motivated by the UCB algorithm for the
classical multi-armed bandit problem.

\subsubsection{Contextual GP-UCB}
Motivated by GP-UCB algorithm mentioned above, \cite{Krause2011-sb}
extended the generalization of this algorithm by incorporating
contextual information
\begin{equation}
  \mathbf{x}_{n+1} = \underset{\mathbf{x} \in D}{\mathrm{arg \,
      max}}\,\mu_n(\mathbf{x}, \mathbf{z}_n)+\beta_n
  \sigma_n(\mathbf{x}, \mathbf{z}_n),
\end{equation} \label{eq:cgp-ucb-query}
where $z_n \in Z$ is the contextual information from a set $Z$ of
contexts, $\mu_n(\cdot)$ and $\sigma_n^2(\cdot)$ are the posterior mean
and variance of the GP conditioned on observations $\mathcal{D}_n =
\left \{( \mathbf{x}_i, \mathbf{z}_i, y_i \right )\}.$ The authors
called the selection rule the contextual Gaussian process UCB
algorithm (GCP-UCB).

To derive the kernel $k$ on the product space $Z\times X$ of contexts and
actions, a natural approach to start with kernel functions $k_Z:
Z\times Z \rightarrow \mathbb{R}$ and $k_X:X \times X \rightarrow
\mathbb{R}$ on the space of contexts and actions. \cite{Krause2011-sb}
proposed two possibilities of constructing composite kernel $k$ from
context kernel $k_Z$ and action kernel $k_X$. One is to calculate a
product kernel $k=k_Z\otimes k_X$, by setting
\begin{equation}
  (k_Z\otimes k_X)((\mathbf{z}, \mathbf{x}),(\mathbf{z}^{\prime},
  \mathbf{x}^{\prime})) = k_Z(\mathbf{z}, \mathbf{z}^{\prime})k_X(\mathbf{x}, \mathbf{x}^{\prime}).
\end{equation}

An alternative is to calculate the additive kernel $k=k_Z\oplus k_X$,
by setting

\begin{equation}
  (k_Z\oplus k_X)((\mathbf{z}, \mathbf{x}),(\mathbf{z}^{\prime},
  \mathbf{x}^{\prime})) = k_Z(\mathbf{z}, \mathbf{z}^{\prime})+k_X(\mathbf{x}, \mathbf{x}^{\prime}).
\end{equation}

\subsection{Q2c: Selection Bias \& Propensity Score Matching}

\paragraph{Question:} how precisely are estimates of the effectiveness of
a learning artifact biased when the student herself can choose which
artifact she receives (selection bias)? How can methods such as
propensity score matching partially reduce this bias? \\ [0.1 in]

In RCTs, students are randomly assigned to conditions so treatment is assigned by randomization. As a consequence of
randomization, an unbiased estimate of the ATE can be directly
computed from the data. An unbiased estimate of the ATE is
$E[Y_i(1)-Y_i(0)]=E[Y(1)] - E[Y(0)]$. When the student can choose
which artifact she receives, randomization of treatment assignment
does not exist. As a consequence, the treated students often differ
systematically from untreated students. In general, $E[Y(1)\mid t=1]
\neq E[Y(1)]$ holds. Thus, an unbiased estimate of the ATE cannot be
obtained by directly comparing outcomes between the the treated and the control groups.

A propensity score \cite{rosenbaum1983central} is the probability of a unit (e.g., student,
classroom, school) being assigned to a particular treatment given a
set of observed features. Propensity scores are used to reduce
selection bias by equating groups based on these features.

Suppose that we have a binary treatment $T=\{0,1\}$, an outcome $Y$, and
observed features $X$. The propensity score is defined as the
conditional probability of treatment given background variables:
$$p(x):= \Pr(T=1 \mid X=x)$$

Due to random treatment assignment in RCT, the propensity score is
known. However, the actual propensity score in observational experiments is
unknown and it can be estimated using statistical or machine
learning models from the experiment data. The most commonly used model
is a logistic regression model, in which treatment assignment $T$ is
the dependent variable and features $X$ are independent variables. Beyond logistic regression, the
use of bagging or boosting \cite{Lee2010-zr}, tree-based model
(Propensity trees) and causal random forests
\cite{wager2015estimation}, and neural networks
\cite{setoguchi2008evaluating} have been proposed to estimate the
propensity score.

The purpose of Propensity score matching (PSM) \cite{austin2011introduction} is to form matched sets
of the treated and
the control subjects who share a similar value of the predicted propensity
score. The most common
implementation of PSM is one-to-one or pair matching, in which pairs
of the control and the treated subjects are formed, such that matched
subjects have similar values of the propensity score. Once a matched
sample has been formed, the treatment effect can be estimated by
directly comparing outcomes between treated and untreated subjects in
the matched sample. PSM attempts to mimic randomization by creating a
sample of units that received the treatment that is comparable on all
observed covariates to a sample of units that did not receive the
treatment. In summary, the analysis of a propensity score matched sample can mimic that of an
RCT: one can directly compare outcomes between the treated and the control
subjects within the propensity score matched sample.

The aforementioned methods, propensity trees
\cite{wager2015estimation} in section~\ref{sect:rcf} and balancing
neural networks (BNN) \cite{Johansson2016-dh} in
section~\ref{sect:bnn}, can also be used to reduce the selection
bias. The propensity trees is inspired by the idea of propensity score
matching. The goal of propensity trees is to train a classification
tree to predict the treatment assignments. The subjects on the same
leaf are considered as matched subjects. Due to the selection bias,
two distributions of observed covariates for the control and the
treated groups may be different. Thus, the problem of counterfactual
inference may require inference over a different distribution than the
one from which samples are given. In other words, the features
distribution of the test data differs from that of the training
data. This is a special case of domain adaptation. BNN uses deep learning
approaches for domain adaptation to learn a balanced representation
between the control and the treatment groups to reduce the selection
bias.

\section{Topic 3: Crowdsourcing for Education.  More on the HCI side and HCOMP ideas}
\subsection{Q1: Peer Assessment}
\paragraph{Question:} Chris Schunn works on peer grading.  What
literature in that is relevant to your dissertation?  Do people make
good reviewers? Do they learn from reviewing?  Do authors benefit from
getting reviews?  (Do they essay they right get better if they get a
review?) \\ [0.1 in]

After going through Chris Schunn's publications, I find these papers
\cite{patchan2016understanding,patchan2015understanding,
schunn2016reliability, cho2006validity}
related to my dissertation. The summary of research findings from
these paper is listed below.

In \cite{cho2006validity, schunn2016reliability}, the authors run a
set of studies to investigate the reliability and validity of peer
assessment of writing on college level and high school level,
respectively. The authors use term reliability to refer to the extent
to which students' assessment of writing correlate with each other and
use validity to describe the extent to which students can accurately
judge the quality of the writing. These two papers reach the
conclusion that the assessments from students at college level and
high school level have strong reliability and validity. In other
words, peers in general are capable of providing valid ratings for
others' writing, and their feedback is usually thought to be helpful
by other students. Students improved their writing ability from peer
assessment, and specifically from providing feedback to peers.

\cite{patchan2016understanding} investigated the relationship between
writer ability and reviewer ability and found several interesting interactions
between these two factors. Often lower-ability writers were more
willing to improve their writing using feedback from other low-ability
reviewers than from high-ability reviewers, while higher-ability
writers benefited equally from receiving feedback from lower-ability
and higher-ability reviewers, which is inconsistent with student
beliefs that only feedback from high-ability peers is worthwhile.

In terms of the style of the feedback, \cite{patchan2015understanding} found that low reviewer provided more praise than high reviewers
whereas high reviewers provided more criticism than low
reviewers. This criticism described more problems and offered more
solutions. There was one interesting interaction between reviewer
ability and text quality -- high reviewers described more problems in
the low-quality texts than in the high-quality texts, whereas low
reviewers did not make this distinction.

\subsection{Q2: DALITE}
\paragraph{Question:} How would that be adapted to deal with DALITE
objects? \\ [0.1 in]

Distributed Active Learning Technology Integrated Environment
(DALITE) \cite{bhatnagar2015analysis} is a web-based application for an asynchronous peer
instruction. In DALITE, students are asked to submit their
explanations for their choice after answering a multiple-choice
question. Then they are prompted with several explanations from other
students. Half of these peer explanations are for the choice that
students select, and the other half of the peer explanations are for a
different choice. Students have a chance to reflect their thinking by
comparing their explanations with prompted explanations and make a
second choice. Students can also select the best explanation among
those displayed.

Similarly to \cite{cho2006validity, schunn2016reliability}, we should
first investigate in DALITE whether students can learn from providing their own explanations and from reflecting their thinking by
comparing their explanations with other students'
explanations. Another investigation should be focused on the quality
of peer explanations and students' perceptions about the asynchronous
peer instruction workflow.

\cite{patchan2016understanding} found that lower-ability writers were more
willing to improve their writing using feedback from other low-ability
reviewers than from high-ability reviewers, while higher-ability
writers benefited equally from receiving feedback from lower-ability
and higher-ability reviewers. This inspires us to investigate whether
this conclusion holds in the context of DALITE. We usually believe
that high-ability students produce high-quality explanations and both
high-ability and low-ability students benefit from these
explanations. If the aforementioned conclusion holds in DALITE, an
adaptive strategy can be applied to select the prompted explanations
for students based on their ability. In other words, for low-ability students,
explanations from other low-ability students should be selected.

Another interesting topic inspired by \cite{patchan2015understanding}
is the style of peer explanations. There are two factors of the
students which play an important role in this
setting: the ability of solving a question and the ability of providing
a high-quality explanations. We can look at the interactions between
the style and these two factors.

\subsection{Q3: Learnersourcing Subgoal Labels}
\paragraph{Question:} How is the work by Rob Miller and Juho about
learning subgoal related to PeerASSIST? \\ [0.1 in]

\cite{Weir2015-hg} presented a three-stage workflow for learners to generate
subgoal labels for how-to video while they are learning from the
videos. The goal of the first stage (Generation) is to let learners submit the
subgoal labels after a certain interval during watching the
videos. The goal of the second stage (Evaluation) is to let learners select the
most relevant answer for the video section, as well as discard
potential spam answers. The goal of the third stage (Proofreading) is
to let learners evaluate the most popular subgoal labels from stage 2
for quality and eventually agree on a final label for that
subgoal. During stage 2 and stage 3, learners always have a option to
refine the label.

In PeerASSIST, only students who receive 0 on a problem are
prompted with a peer explanation from one of students who answer the same
problem correctly. Then these students have a option to give a rating
(thumb up or thumb down) to the peer explanation that they
receive. The students, who answer the problem correctly in the first
attempt, do not have a chance seeing and rating the peer
explanations. If
a certain problem is relatively easy, there will be few students who
receive 0 and it is difficult to find the optimal peer explanations
based on students' ratings. Even if the problem is relatively
difficult, we miss the contribution from the students with correct
answer in the first attempt when deciding the optimal peer
explanations. Motivated by \cite{Weir2015-hg}, we can apply two-stage
workflow to collect ratings from students in terms of the quality of
the peer explanations after they submit an answer. Assume that there
are several peer explanations available for a given problem and the
ranking of these peer explanations are already decided by some
algorithm (e.g., multi-armed bandit algorithm) based on the history data.

On stage 1, students are prompted with several (i.e., 3) peer
explanations and have a option to rate these explanations after they
submit an answer. The goal of this stage is to collect students'
opinions on these explanations and use these data as one of the
indicators to determine the most effective explanation and discard
spam explanations. In terms of
which explanations should be prompted to students on this stage,
several simple approaches can be applied: random selection from all
available explanations or newly added explanations. Exploration strategy can be applied in this scenario. For the purpose of
exploration, explanations with high uncertainty are prompted with
students. This strategy can potentially reduce the uncertainty of
these explanations and help multi-armed bandit algorithm efficiently
find the optimal explanations.

On stage 2, student are prompted with the currently optimal peer explanation
from the ranking algorithm and asked to evaluate whether this peer
explanation is highly effective. The goal of this stage is to let
students to evaluate the optimal explanation for quality and analog to
the goal of exploration strategy in multi-armed bandit algorithm.

\subsection{Q4: HCOMP \& CSCW}
\paragraph{Question:} Look at the two most recent  HCOMP conference and
the CSCL conferences.   What do you see in those conferences that
relate to what you want to propose for your dissertation?   I assume
some will be methodological, while others will be on the design side.
For the purpose of this question let's look at the design side. \\
[0.1 in]

\begin{figure}[h]
  \centering
  \includegraphics[width=0.75\columnwidth]{learnersourcing.png}
  \caption{Reprinted from \cite{Glassman2016-yy}. In the
    self-reflection workflow, students generate hints by reflecting on
  an obstacle they themselves have recently overcome. In the
  comparison workflow, students compare their own solutions to those
  of other students, generating a hint as a byproduct of explaining
  how one might get from one solution to the other.}
  ~\label{fg:learnersourcing-model}
\end{figure}

The aforementioned paper, Learnersoucring Subgoal Labels for How-to Videos
\cite{Weir2015-hg}, is published in CSCW 2015. In this paper, the
authors showed that students were qualified to generate subgoal labels
for how-to videos based on their new expertise. Motivated by this prior
work, another paper, Learnersourcing Personalized Hints
\cite{Glassman2016-yy}, is published in CSCW 2016. In
\cite{Glassman2016-yy}, the authors designed two workflows aimed to
engage students in creating personalized hints. Results showed that students can create helpful hints for their
peers that augment or even replace teachers’ personalized assistance,
when that assistance is not available.

In the self-reflection workflow, students iteratively work on their
solutions to pass as many teacher-created tests as possible. There are
hints available for any verification. Students can ask for these hints
to help them pass the test case. After fixing a bug in their own
solution, students can generate a new hint for others struggling with
the same error.

In the comparison workflow, students compare their solution to
alternative solution submitted by other students or teachers. If their
solution performs better than a solution $W$, they are asked to submit
an optimization hint for students who have solutions like $W$. If
their solution performs worse than a solution $B$, they are prompted
to generate an optimization hint for students who have solutions like
their own. Figure~\ref{fg:learnersourcing-model} illustrates both workflows.

\cite{Yoon2016-nk} developed a new multi-modal annotation (e.g.,
gesture and waveform) tool, called
$\mathrm{RichReview}^{++}$. The aim of this tool is to texts, audios
and gestures to mimic face-to-face communication experience for
collaborators reviewing and providing feedback on documents. It is
also extended to support peer discussion.
\bibliographystyle{apalike}
\bibliography{references}
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\section{Lexer and Parser}
The parser is implemented as a simple $LL(\infty)$ parser using parser combinators so it may be quite slow on large inputs, but it's sufficient
for our purpose.

Our AST for terms is defined as follows, for brevity, AST for other objects are not described here.

\begin{center}
\begin{minted}{haskell}
type Name = String
type Index = Int

data Term =
    TmRel
      Name       -- name of the variable, used for pretty printing
      Index      -- 0 based DeBruijn index
  | TmVar
      Name       -- name of the variable
  | TmAppl
      [Term]     -- the first is the abstraction and the rest are the arguments
  | TmProd
      Name       -- name of the abstracted variable, used for pretty printing
      Term       -- type of the abstracted variable
      Term       -- body of the abstraction
  | TmLambda
      Name       -- name of the abstracted variable, used for pretty printing
      Term       -- type of the abstracted variable
      Term       -- body of the abstraction
  | TmFix
      Int        -- the index of the decreasing variable
      Term       -- body of the fix definition
  | TmLetIn
      Name       -- name of the local binding variable
      Term       -- type of the local binding
      Term       -- body of the local binding
      Term       -- let body, the binding will be added here
  | TmIndType
      Name       -- name of the inductive type constructor
      [Term]     -- argument list
  | TmConstr
      Name       -- name of the term constructor
      [Term]     -- argument list
  | TmType
  | TmTypeHigher
  | TmMatch
      Int        -- how many parameters does the inductive type need
      Term       -- the term pattern matching on
      Name       -- the name of the local binding of the term in the return type
      [Name]     -- the matching name list for the return type
      Term       -- return type
      [Equation] -- equations, described below
  deriving (Eq, Show)

data Equation =
    Equation
      [Name]     -- matching name list for the term constructor
      Term       -- body of the equation
    deriving (Eq, Show)
\end{minted}
\end{center}


\documentclass[10pt]{article}

\usepackage[mode=buildnew,subpreambles=true]{standalone}
\input{preamble}

\title{\bf Short summary}
\author{Yann-Edwin Keta}
\date{November 12th, 2019}

\begin{document}

\maketitle

\section{Model}

We consider an ensemble of $N$ spherical ABPs $i$, with positions $\underline{r}_i$ and orientations $\theta_i$, which are self-propelled along the direction $\underline{u}_i \equiv (\cos(\theta_i), \sin(\theta_i))$. We have the following dimensionless equations of motion \cite{nemoto_optimizing_2019},
\begin{equation}
\begin{aligned}
\dot{\underline{r}}_i(t) &= \frac{1}{3} \frac{\sigma}{l_p} \tilde{\underline{F}}_{i, ex}(t) + \underline{u}_i(t) + \sqrt{\frac{2}{3}\frac{\sigma}{l_p}} \underline{\eta}_i(t),\\
\dot{\theta_i}(t) &= \sqrt{2\frac{\sigma}{l_p}} \xi_i(t),
\end{aligned}
\label{EOM}
\end{equation}
where $\tilde{\underline{F}}_{i, ex}$ is the total force applied on particle $i$ deriving from a WCA potential, $l_p$ is the persistence length, $\sigma$ is the particle diameter, and $\underline{\eta}_i \equiv (\eta_{x, i}, \eta_{y, i})$ and $\xi_i$ are independent Gaussian white noises of unit variance and zero mean.\\

We define the normalised rate of active work,
\begin{equation}
w(t_0; \tau) = \frac{1}{N \tau} \, (S)\int_{t_0}^{t_0 + \tau} \sum_{i=1}^{N} \underline{u}_i(t) \cdot \text{d}\underline{r}_i(t) = \frac{1}{2 N \tau \, \Delta t} \sum_{t=0}^{\tau - 1} \sum_{i=1}^n \left(\underline{u}(\theta_{i, t_0 + t + 1}) + \underline{u}(\theta_{i, t_0 + t})\right) \cdot \left(\underline{r}_{i, t_0 + t + 1} - \underline{r}_{i, t_0 + t}\right),
\end{equation}
which we can write as a sum of three terms,
\begin{equation}
\begin{aligned}
w_f(t_0; \tau) &= \frac{1}{2 N \tau} \sum_{t=0}^{\tau - 1} \sum_{i=1}^n \left(\underline{u}(\theta_{i, t_0 + t + 1}) + \underline{u}(\theta_{i, t_0 + t})\right) \cdot \frac{1}{3} \frac{\sigma}{l_p} \tilde{\underline{F}}_{i, ex, t_0 + t},\\
w_{\theta}(t_0; \tau) &= \frac{1}{2}\left(1 + \frac{1}{N\tau} \sum_{t=0}^{\tau - 1} \sum_{i=1}^N \cos(\theta_{i,t_0 + t  + 1} - \theta_{i,t_0 + t})\right),\\
w_{\eta}(t_0; \tau) &= \frac{1}{2N \tau} \sum_{t=0}^{\tau - 1} \sum_{i=1}^N \left(\underline{u}(\theta_{i,t_0 + t  + 1}) + \underline{u}(\theta_{i, t_0 + t})\right) \cdot \sqrt{\frac{2}{3} \frac{\sigma}{l_p} \frac{1}{\Delta t}} \, \underline{\eta}_{i, t_0 + t + 1},
\end{aligned}
\end{equation}
respectively the \textit{force}, \textit{orientation}, and \textit{noise} part of the active work.\\

We also define an order parameter,
\begin{equation}
\underline{\nu}(t) = \frac{1}{N} \sum_{i=1}^N \underline{u}(\theta_i(t)),
\end{equation}
with mean and correlations \footnote{Refer to appendix E of \cite{nemoto_optimizing_2019} for the derivation of the order parameter norm dynamics which we use to infer the corresponding results of equation \ref{meanVarOrder}.}
\begin{equation}
\begin{aligned}
\left<\underline{\nu}(t_0)\right> &= 0,\\
\left<|\underline{\nu}(t_0)|\right> &= \frac{1}{\sqrt{2N}},\\
\left<\delta \underline{\nu}(t_0 + \tau) \cdot \delta\underline{\nu}(t_0)\right> &= \frac{1}{N} \exp\left(-\frac{\sigma}{l_p}\tau\right),\\
\left<\delta |\underline{\nu}|(t_0 + \tau) \, \delta |\underline{\nu}|(t_0)\right> &= \frac{1}{4N} \exp\left(- 2\frac{\sigma}{l_p} \tau\right),
\end{aligned}
\label{meanVarOrder}
\end{equation}
in the absence of symmetry breaking.

\section{One-particle quantities}

We check our algorithm and set benchmarks by considering the \textit{free} or \textit{single particle} case, $\tilde{\underline{F}}_{i, ex} = 0$.\\

We can analytically derive the mean squared displacement,
\begin{equation}
\left<(\underline{r}(t_0 + \Delta t) - \underline{r}(t_0))^2\right> = \frac{l_p}{\sigma}\left(\Delta t + \frac{l_p}{\sigma}\left(\exp\left(-\frac{\sigma}{l_p} \Delta t\right) - 1\right)\right) + \frac{4}{3} \frac{\sigma}{l_p} \Delta t.
\end{equation}
and the active work mean and correlations
\begin{equation}
\begin{aligned}
\left<w(t_0; \tau)\right> &= 1,\\
\forall \tau \geq \tau_0,~ \left<\delta w(t_0; \tau_0) \delta w(t_0; \tau)\right> &= \frac{2}{3} \frac{\sigma}{l_p} \frac{1}{\tau},\\
\forall \tau \geq \tau_0,~ \left<\delta w(t_0; \tau_0) \delta w(t_0 + \tau; \tau_0)\right> &= 0.
\end{aligned}
\end{equation}
These theoretical predictions match the numerical results.

\section{Active work covariances}

We argue that
\begin{equation}
\left<\delta w(t_0; \tau_0) \delta w(t_0 + \tau; \tau_0)\right> \approx \left<\delta w_f(t_0; \tau_0) \delta w_f(t_0 + \tau; \tau_0)\right> + \left<\delta w_{\eta}(t_0; \tau_0) \delta w_f(t_0 + \tau; \tau_0)\right>,
\end{equation}
since that the fluctuations of the orientation part of the active work are negligible for low integration time steps, and that the orientational and translational noises are independent.\\

We plot the covariances between the different part of the active work with a delay $\tau$ in figure \ref{crossCor}.

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{crossCor_Nm2000_Dk6500_En1000.eps}
\caption{Covariances of active work parts with a delay $\tau$, for $l_p/\sigma = 20$ (dashed lines) and $l_p/\sigma = 100$ (dash-dotted lines). All available intervals but those overlapping with $t_0 \leq 3 l_p/\sigma$ were used to compute the means. These curves are qualitatively independent of the choice of $\tau_0$, this particular choice is a balance between noise reduction and low time resolution. Purple curves correpond to the total active work covariance.}
\label{crossCor}
\end{figure}

We observe that
\begin{itemize}
  \item the covariance of the force part of the active work and itself with a delay $\tau$ is a positive, monotonically decreasing function of $\tau$, with a length scale which increases with persistence length,
  \item the covariance of the noise part of the active work and the force part with a delay $\tau$ is a negative, monotonically increasing function of $\tau$, with a length scale which is qualitatively independent of the persistence length,
  \item the covariance of the active work and itself with a delay $\tau$ is a non-monotonic function of $\tau$, with a peak at $\tau \approx 10^{-1}$.
\end{itemize}

\subsection{Covariance of noise and force part}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{crossCorNoiseForce_No1000_Dk6500_Em2000.eps}
\caption{Covariance of the noise part of the active workand the force part with a delay $\tau$.}
\label{crossCorNoiseForce}
\end{figure}

We have that for the few values of persistence length tested, $2 \leq l_p/\sigma \leq 20$, the time scale of decrease of the absolute value of $\left<\delta w_{\eta}(t_0; \tau_0) \delta w_f(t_0 + \tau; \tau_0)\right>$ is qualitatively independent of the persistence length (figure \ref{crossCorNoiseForce}).\\

We argue that this time scale is set by the interaction potential. Consider a single particle in a harmonic potential
\begin{align*}
V(\underline{r}(t)) = \frac{1}{2} k \underline{r}(t)^2.
\end{align*}
Motivated by figures \ref{crossCor} and \ref{crossCorNoiseForce}, we assume that $l_p/\sigma \gg \tau_d$, where $\tau_d$ is the relevant time scale of decrease, \ie we can consider that the orientation of the particle is constant. We then have the following equation of motion
\begin{equation}
\dot{\underline{r}}(t) = - \frac{1}{3} \frac{\sigma}{l_p} \nabla V(\underline{r}(t)) + \underline{u}_0 + \sqrt{\frac{2}{3} \frac{\sigma}{l_p}} \underline{\eta}(t).
\end{equation}
We can write
\begin{align*}
\text{d}\left<\left(\underline{\eta}_i(t_0) \cdot \underline{u}_0\right) \, \left(\nabla V(\underline{r}(t_0 + t)) \cdot \underline{u}_0\right)\right> &= \left<\left(\underline{\eta}_i(t_0) \cdot \underline{u}_0\right) \, \left(\underbrace{\nabla^2 V(\underline{r}(t_0 + t))}_{k} \,\text{d}\underline{r}(t_0 + t) \cdot \underline{u}_0\right)\right>\\
&= - k \frac{1}{3} \frac{\sigma}{l_p} \left<\left(\underline{\eta}_i(t_0) \cdot \underline{u}_0\right) \, \left(\nabla V(\underline{r}(t_0 + t)) \cdot \underline{u}_0\right)\right> \, \text{d}t,
\end{align*}
such that
\begin{equation}
\left<\left(\underline{\eta}_i(t_0) \cdot \underline{u}_0\right) \, \left(\nabla V(\underline{r}(t_0 + \tau)) \cdot \underline{u}_0\right)\right> \propto \exp\left(- k \frac{1}{3} \frac{\sigma}{l_p} \tau\right).
\end{equation}

\subsection{Covariance of force part and itself}

According to figure \ref{crossCorForceLP}, we have that $\left<\delta w_f(t_0; \tau_0) \delta w_f(t_0 + \tau; \tau_0)\right>$
\begin{itemize}
  \item decays algebraically at low persistence length $l_p/\sigma = 2$,
  \item decays logarithmically at high persistence length $l_p/\sigma = 20$.
\end{itemize}
We show the buildup of this covariance with increasing persistence length in figure \ref{crossCorForceN}.

\begin{figure}[H]
\centering
\includegraphics[width=0.49\textwidth]{crossCorForceForce_Dk6500_Ll2000_Em2000.eps}
\hfill
\includegraphics[width=0.49\textwidth]{crossCorForceForce_Dk6500_Lm2000_Em2000.eps}
\caption{Covariance of the force part of the active work and itself with a delay $\tau$ for $l_p/\sigma = 2$ (left) and $l_p/\sigma = 20$ (right).}
\label{crossCorForceLP}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{crossCorForceForce_No1000_Dk6500_Em2000.eps}
\caption{Covariance of the force part of the active work and itself with a delay $\tau$ for $2 \leq l_p/\sigma \leq 20$ at $N=10^3$ particles.}
\label{crossCorForceN}
\end{figure}

According to \cite{tociu_how_2018}, the mean force part of the normalised rate of active work in steady state is linked to the structure of the liquid,
\begin{equation}
\left<w_f\right> = \phi \int g(\underline{r}) \left\{[\nabla V(\underline{r})]^2 - \frac{2}{3}\frac{\sigma}{l_p} \nabla^2 V(\underline{r})\right\} \, \text{d}\underline{r} + \phi^2 \iint g_3(\underline{r}, \underline{r}^{\prime}) \nabla V(\underline{r}) \cdot \nabla V(\underline{r}^{\prime}) \, \text{d}\underline{r} \, \text{d}\underline{r}^{\prime},
\end{equation}
where $g$ and $g_3$ are the two- and three-body density correlations among particles.\\

Maybe then is the decay of $\left<\delta w_f(t_0; \tau_0) \delta w_f(t_0 + \tau; \tau_0)\right>$ linked to the relaxation of the structure. We have that at $l_p/\sigma = 2$ the system resembles an homogeneous liquid, while at $l_p/\sigma = 20$ denser regions -- clusters -- appear (figure \ref{screenshots}). Moreover, clusters are expected to get denser as the persistence length increases, and particles move more slowly in denser regions \cite{cates_motility-induced_2015}. We can thus imagine that the buildup of the covariance of the force part of the active work and itself with a delay $\tau$ is directly linked to this changing dynamical picture with increasing persistence length. Such a link remains to be demonstrated.

\begin{figure}[H]
\centering
\includegraphics[width=0.49\textwidth]{o_Nm2000_Dk6500_Ll2000.eps}
\hfill
\includegraphics[width=0.49\textwidth]{o_Nm2000_Dk6500_Lm2000.eps}
\caption{Screenshots of the system for $l_p/\sigma = 2$ (left) and $l_p/\sigma = 20$ (right). Colors refer to the orientation of the particles.}
\label{screenshots}
\end{figure}

\section{Order and active work}

We compute the correlation function between the flucutations of the order parameter norm at time $t_0$ and the flucutations of the normalised rate of active work on the interval $[t_0; t_0 + \tau]$ (figure \ref{Cawo}),
\begin{equation}
C^{(a)}_{wo}(\tau) = \frac{\left<\delta w(t_0; \tau) \, \delta |\underline{\nu}(t_0)|\right>}{\sqrt{\left<\delta w(t_0; \tau)^2\right> \, \left<\delta |\underline{\nu}(t_0)|^2\right>}}.
\label{eqCawo}
\end{equation}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{corWorkOrderAve_Nm2000_Dk6500_Em1000.eps}
\caption{Correlation function between the flucutations of the order parameter norm at time $t_0$ and the flucutations of the normalised rate of active work on the interval $[t_0; t_0 + \tau]$.}
\label{Cawo}
\end{figure}

First of all, we note that this correlation function is non-monontic. This can rationalised by the fact that the variance of the active work,
\begin{align*}
\tau \mapsto \left<\delta w(t_0; \tau)^2\right>,
\end{align*}
is a monotonically decreasing function of $\tau$.\\

We have that the fluctuations of the normalised rate of active work $w$ over an interval $[t_0; t_0 + \tau]$ are positively correlated with fluctuations of the order parameter $|\underline{\nu}|$ at the beginning of this interval. This may indicate that positive fluctuations of the order parameter norm at a given time leads to an increased active work at following times. This can be explained by the fact that nematically ordered particles -- \ie high $|\underline{\nu}|$ -- have fewer collisions, so that active forces translates more efficiently into particle motion \cite{nemoto_optimizing_2019}.\\

We also note that these correlations decay on a length scale larger than the rotational diffusion time -- the dashed black line is here for illustration purposes and does not reflect any theoretical predicition. Furthermore, we have that the correlations between the averaged active work and the order parameter norm are higher for $l_p/\sigma=100$ than for $l_p/\sigma = 20$ -- this may however be due, in part or completely, to the fact that the variance of the active work is higher in the latter.\\

We also compute the covariance of the order parameter norm and the normalised rate of active work with a delay $\tau$ (figure \ref{covWorkOrder}),
\begin{equation}
\tau \mapsto \left<\delta |\underline{\nu}(t_0)| \delta w(t_0 + \tau; \tau_0)\right>,
\end{equation}
which is linked to the numerator in equation \ref{eqCawo} by summation over the interval $[t_0; t_0 + \tau]$.

\begin{figure}[H]
\centering
\includegraphics[width=0.49\textwidth]{corWorkOrderIns_Nm2000_Dk6500_Lm2000_Em1000_Ip5000.eps}
\hfill
\includegraphics[width=0.49\textwidth]{corWorkOrderIns_Nm2000_Dk6500_Ln1000_Em1000_Ip5000.eps}
\caption{Covariance of the order parameter norm and the normalised rate of active work with a delay $\tau$ for $l_p/\sigma = 20$ (left) and $l_p/\sigma = 100$ (right).}
\label{covWorkOrder}
\end{figure}

We have a monotonic decrease of this covariance with a length scale $\sim \frac{1}{2} \frac{l_p}{\sigma}$ for both tested values of the persistence length.

%%%%%%%%%%%%%%
% REFERENCES %
%%%%%%%%%%%%%%

\bibliographystyle{unsrtnat}
{\renewcommand{\bibname}{References}\bibliography{ref}}

\end{document}


\chapter{Conclusion}\label{chap:Conclusion}
In this thesis, we investigated the Asymptotic Safety scenario for quantum gravity and studied its phase diagram within the Einstein-Hilbert truncation in a background field approximation. We used non-perturbative Functional Renormalization Group techniques to compute the running of Newton's constant $g_k$ and of the cosmological constant $\lambda_k$.  \\

In chapters 1 to 3 we introduced the background knowledge, needed for a general understanding of the conducted calculations. After briefly discussing the general idea of the Asymptotic Safety conjecture, in chapter \ref{chap:EHT} we solved the flow equation for a pure-gravity system in a transverse-traceless spin-two graviton approximation to get a first insight into the underlying structures of the theory setting. The mathematical tools we used to solve the flow equation, such as the York decomposition of the fluctuation field and the heat-kernel techniques to compute the functional traces, were introduced very detailed. We computed the $\beta$-functions for $g_k$ and $\lambda_k$ and determined the fixed points of the flow. Besides the non-interacting Gaussian fixed point at $(g_k, \lambda_k)=(0, 0)$ we found an UV-attractive fixed point at $(g_k, \lambda_k)=(0.86, 0.18)$, providing further evidence for the Asymptotic Safety scenario as a promising candidate for a non-perturbative renormalizable quantum field theory of gravity. In chapter \ref{chap:Matter} we extended our truncation: First of all, the previously neglected trace mode and the Faddeev-Popov ghosts associated with the graviton sector were included to complete the calculation from chapter \ref{chap:EHT}. Then we investigated the impact of minimally coupled scalars, fermions and gauge fields on the Non-Gaussian fixed point. We explained in full detail how to solve the functional traces for all three matter types separately and presented the most important and insightful steps. After finishing the computation of all contributions, we were able to determine the $\beta$-functions for $g_k$ and $\lambda_k$ as a function of the number of matter fields. Neglecting all the contributions from the different anomalous dimensions and expanding the $\beta$-functions in some neighborhood of the Gaussian fixed point up to second order in the couplings, we qualitatively analyzed the behavior of the values for both couplings. We found out, that for an increasing amount of scalar fields, the values of the couplings tend to increase drastically. The fermionic fields and especially the gauge fields seem to have a stabilizing effect on the system. These results are almost in agreement with the results from earlier investigations, where similar conventions have been chosen, see e.\,g \cite{DonaEichhornPercacci2013}.
At the end, in chapter \ref{chap:BGindependence}, we highlighted some problems associated with the background field approximation, e.\,g. the loss of background independence as a consequence of violating the Nielsen identities. \\

This work may not provide fundamentally new results, but it still demonstrates some of the central concepts and calculations related to the subject. Due to the fact, that we worked in a rather simple truncation and chose a Litim-type cutoff, we we able to solve all the problems in this thesis analytically. Interesting modifications of our setup could include the employment of more sophisticated regulators or the inclusion of higher-order curvature terms. Compared to more recent results, the outcomes of our calculations should be treated with care. Nevertheless, this thesis provides a suitable framework for further investigations of asymptotically safe quantum gravity. Based on our discussion in chapter \ref{chap:BGindependence}, one may consider abandoning the background field approximation in future projects.


\documentclass{article}
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{amssymb}

\setlength{\bigskipamount}{8em}
\setlength{\parindent}{3em}
\setlength{\parskip}{1em}

\newtheorem{theorem}{Theorem}
\newtheorem{question}{Q.}

\begin{document}
\author{Dave Neary}
\title{Fundamental Theorem of Algebra}

\maketitle

\section{Introduction}

We will talk about some of the number systems you will come across, and work
up to the Fundamental Theorem of Arithmetic: that positive whole numbers have
a unique representation as the product of prime numbers.

\section{Number systems}

There are several number systems that we will come across in number theory.
You might say that mathematics is a journey, inventing new number systems
and abstractions to allow us to solve new types of problems we could not
with the tools available before.

We start with the natural numbers $\mathbb{N} = \{1,2,3,\cdots\}$, or the
natural numbers including 0 $\mathbb{N}_0 = \{0,1,2,\cdots\}$ - this
is the foundation of mathematics, counting objects. The natural numbers are
closed under the operations addition and multiplication, but not under
subtraction. In general, we cannot find $a-b \in \mathbb{N}$ for $a,b \in
\mathbb{N}$.

To enable closure under subtraction, we can extend the natural numbers to
include negative numbers to produce the integers
\[\mathbb{Z} = \{\cdots,-3,-2,-1,0,1,2,3,\cdots\}\] This number system is
closed under addition, multiplication, \textbf{and} subtraction, but still
has a limitation under division. When we try to divide a whole number into
parts, we cannot do this in the integers.

To enable closure under division, we can extend the integers to include the
rational numbers - fractions: \[\mathbb{Q} = \{\frac{a}{b} | a,b\in \mathbb{Z},
b \neq 0 \}\]

Finally, for now, we can extend the rational numbers to include numbers which
cannot be represented by a fraction, but which can be measured on the number
line, to the real numbers $\mathbb{R}$, which are essentially the rational
numbers with the gaps filled in. We call a number which is in the real numbers,
but is not a rational number, an irrational number.

The irrational numbers can also be subdivided into two groups. The algebraic
numbers are a superset of the rational numbers which includes all
exact solutions of polynomial equations with rational coefficients. This group
includes all the square roots, cube roots, and in general the $n$th roots,
and any combination of them (for example, $\sqrt[3]{7+\sqrt{13}}$ is an
algebraic number).

Any number which is not the solution to any such polynomial is called a
transcendental number. Well known numbers like $\pi$ and $e$ are transcendental
numbers.

For now, we will focus mostly on the characteristics of the natural numbers and
the integers. There will be plento of opportunity in the future to get into the
other number systems - and to discover others!

\section{The Fundamental Theorem of Arithmetic}

Natural numbers can be grouped into different subsets. For example, we learn
early in school that the numbers 2,4,6,8,... are special - they are all whole
number multiples  of 2, and we call them the even numbers. If a number is not even,
it is odd.

There is nothing special about 2 though - we can find the set of nultiples of 3, and
can even define it in general as $S = \{3,6,9,\cdots\} = \{ 3n | n \in \mathbb{N}\}$
- that is, the set contains all of the numbers which can be expressed as $3k$ for some
$k\in \mathbb{N}$. We will often use this type of set definition short-hand.

Every natural number has natural number divisors. Some numbers can only be divided in
natural numbers by themselves and 1. We call these numbers \textbf{prime numbers}. The
first few prime numbers are 2, 3, 5, 7, 11, 13, 17. A natural number which is not a
prime number is called a composite number, which means that we can write the number
$n = a \cdot b$ for $a,b > 1 \in \mathbb{N}$. 4, 6, 8, 9 are the first composite numbers.

The number 1 is special - it is neither prime nor composite, by convention (which
means, it's just useful to think of it as being neither - people have fought about
that stuff in the past). Next, we will bring these threads together to state the
Fundamental Theorem of Algebra. It is very powerful, and almost completely obvious.

\begin{theorem}[The Fundamental Theorem of Arithmetic]

Every number $n \in \mathbb{N}$ can be expressed as a \textbf{unique}
product of primes. We can write $n = p_1^{\alpha_1}p_2^{\alpha_2}\cdots p_k^{\alpha_k}$
for some set of primes $\{p_i\}$ and exponents $\{\alpha_i\} \in \mathbb{N}$.
\end{theorem}

Starting from just this theorem, we can already prove some nice results.

\subsection{Problems}

\begin{question}Prove that $\sqrt{5}$ is irrational.\end{question}

\emph{Proof:} We will use proof by contradiction. Assume that $\sqrt{5}$ is rational,
that is, that there exist $a,b\in \mathbb{N}$ such that $\gcd(a,b)=1$ and
$\sqrt{5}=\frac{a}{b}$.

Then since:
\[ 5 = \frac{a^2}{b^2} \implies a^2=5b^2 \]
We know that $5 | a^2$ and therefore $5|a$ (since if there is a 5 in the prime
decomposition of $a^2$ there must also be a 5 in the prime decomposition of $a$).

So we can write $a=5m$ and $25m^2 = 5b^2 \implies 5m^2 = b^2$ - and by repeating the
same logic, we can show that $5 | b$. But if 5 divides both $a$ and $b$, then
$\gcd(a,b) = 5k$ for some $k \in \mathbb{N}$, which is a contradiction with our
initial assumption that $\gcd(a,b)=1$ - so our original assumption, that $\sqrt{5}$
is rational, must be false.
$\square$

\begin{question}How many positive integer divisors does 36 have?\end{question}

$36 = 2^2 \cdot 3^2$, so every factor of 36 will have the form
$2^{\alpha}\cdot 3^{\beta}$ with $\alpha \in \{0,1,2\}$ and $\beta \in \{0,1,2\}$.

We can find all of the factors of 36 by combining the possible values of $\alpha$
and $\beta$ in all possible combinations, so the total number of factors of 36 is
$3 \times 3 = 9$, since we have 3 choices for $\alpha$ and 3 choices for $\beta$.

\begin{question} $n! = n\cdot (n-1)\cdot (n-2) \cdots 2 \cdot 1$ (called $n$
factorial). How many trailing zeros are there in $25!$?\end{question}
\vspace*{\bigskipamount}

\begin{question}How many positive integer factors does 720 have?\end{question}
\vspace*{\bigskipamount}

\begin{question}Write all the factors of $7^{21}$ and compute their sum.\end{question}
\vspace*{\bigskipamount}

\end{document}