\section{Tactic}
Although proofs can be automatically checked given the system above, it requires user to input the whole
term of the specific type, which is tedious, exhaustive, mostly anti-intuitive.\par
So in this part, we introduced some tactics into the system
to assist users in better writing the proof. With the help of
tactics, the proof becomes more human-readable and intuitive.\par

\subsection{\it apply}
This tactic applies to any goal. The purpose of this tactic is to extract premises of
current goal.

\subsection*{Usage}
\begin{itemize}
\item $\it apply\ \tt t.$ : $\tt t$ is a term of a type whose consequent is the goal.
\item $\it apply\ \tt t\ \it in\ \tt H.$ : $\tt t$ is a term of a type whose antecedent is $\tt H$.
\end{itemize}

\subsection*{Implementation}
The tactic {\it apply} tries to match the current goal against the
final consequent of the type of term.
If it succeeds, it returns as many subgoals as
the number of non-dependent premises of the type of term.\par
Here is an example:
\begin{center}
\begin{minted}{coq}
n : nat
m : nat
o : nat
e : eq nat n m -> eq nat m o -> eq nat n o
(* Current Goal *)
eq nat n o

(* $\Downarrow$ apply n $\Downarrow$ *)

n : nat                                         n : nat
m : nat                                         m : nat
o : nat                                         o : nat
e : eq nat n m -> eq nat m o -> eq nat n o      e : eq nat n m -> eq nat m o -> eq nat n o
(* SubGoal 1 *)                                 (* SubGoal 2 *)
eq nat m o                                      e : eq nat n m
\end{minted}
\end{center}
Because \mintinline{coq}|e| has a type of \mintinline{coq}|eq nat n m -> eq nat m o -> eq nat n o|
and the goal is \mintinline{coq}|eq nat n o|,
the goal is convertible to the \textbf{consequent} of the type of \mintinline{coq}|e|.
When applied with \mintinline{coq}|e|,
all of the non-dependent premises of the type of \mintinline{coq}|e| become subgoals.

The tactic {\it apply $\tt tm$ in $\tt H$} tries to match the hypothesis $\tt H$ against the
antecedent of the type of $\tt tm$.
If it succeeds, the hypothesis $\tt H$ will be replaced by the consequent of $\tt tm$.
\begin{center}
\begin{minted}{coq}
n : nat
m : nat
o : nat
H : eq nat n m
e : eq nat n m -> eq nat m o -> eq nat n o
(* Current Goal *)
eq nat m o -> eq nat n o

(* $\Downarrow$ apply e in H $\Downarrow$ *)

n : nat
m : nat
o : nat
H : eq nat m o -> eq nat n o
e : eq nat n m -> eq nat m o -> eq nat n o
(* SubGoal 1*)
eq nat m o -> eq nat n o
\end{minted}
\end{center}
Because \mintinline{coq}|e| has a type of
\mintinline{coq}|eq nat n m -> eq nat m o -> eq nat n o|
and $\tt H$ is \mintinline{coq}|eq nat n m|,
$\tt H$ is convertible to the \textbf{antecedent} of the type of the term.
When applied with \mintinline{coq}|e|,
$\tt H$ is replaced by the consequent of $\tt e$.

\subsection{\it destruct}

When {\it destruct} is applied on a term, this term must be inductive type. Then {\it destruct} will divide current goal
into several classified subgoals based on the constructors of the inductive type. During this process, no induction
hypothesis is generated by {\it destruct}; the {\it destruct} uses {\tt match} to handle classified discussion.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it destruct}\ \tt t.$ : $\tt t$ is a term of inductive type.
\end{itemize}

\subsubsection*{Implementation}

This tactic not only divides current goal based on the constructors of the inductive
type, but requires the system to handle several equivalent relations proposed by the constructors as well.\par
Here is an example:
\begin{center}
\begin{minipage}{0.7\textwidth}
\begin{minted}{coq}
n, m : nat
e : eq nat n m                             n : nat
f : eq nat m m      (* destruct e => *)    f : eq nat n n
(* Current Goal *)                         (* Current Goal *)
eq nat m n                                 eq nat n n
\end{minted}
\end{minipage}
\end{center}
From the constructor \mintinline{coq}|eq_refl : eq T x x| and the type of \mintinline{coq}|e : eq nat n m|,
we have \mintinline{coq}|m = n|, so any occurrence of \mintinline{coq}|m| should be replaced by \mintinline{coq}|n|.\par
If some term depends on the destructed term, we should rewrite it.
\begin{center}
\begin{minipage}{0.7\textwidth}
\begin{minted}{coq}
n, m : nat
e : eq nat n m
f : eq nat m m
(* Current Goal *)
eq nat m n

(* $\Downarrow$ destruct m $\Downarrow$ *)

n : nat                    n : nat
e0 : eq nat n O            e0 : eq nat n (S m0)
f0 : eq nat O O            f0 : eq nat (S m0) (S m0)
(* SubGoal 1 *)            (* SubGoal 2 *)
eq nat O n                 eq nat (S m0) n
\end{minted}
\end{minipage}
\end{center}
The two cases above are viewed from user's perspective.
On the other hand, in the system we also need to construct the term for back substitution.
\begin{center}
\begin{minipage}{\textwidth}
\begin{minted}{coq}
n, m : nat
e : eq nat (S n) (S m)
(* Current Goal*)
eq nat (S m) (S n)

(* $\Downarrow$ destruct (S n) $\Downarrow$ *)

(* The Proof Object *)
(fun (n:nat)(m:nat)(e:eq nat (S n) (S m)) =>
    match (S n) as n0 in nat return (eq nat n0 (S m) -> eq nat (S m) n0) with
    | O => fun e0 : eq nat O (S m) => (* SubGoal 1*)
    | S n0 => fun e0 : eq nat (S n0) (S m) => (* SubGoal 2*)
    end e)
\end{minted}
\end{minipage}
\end{center}
After building the {\it return type}, we can build the branches by applying {\it type of branch}.

\subsection{\it exact}

This tactic checks if the input term is actually of the goal's type. If so, the proof of the current goal is finished and the
constructed proof object is exactly the input term.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it exact}\ \tt t.$ : $\tt t$ is a term of the goal's type.
\end{itemize}

\subsubsection*{Implementation}
It's easy to implement. Here is an example:
\begin{center}
\begin{minipage}{0.8\textwidth}
\begin{minted}{coq}
m : nat
(* Current Goal *)  (* exact (eq_refl nat m) => *)  (* No More Goals *)
eq nat m m
\end{minted}
\end{minipage}
\end{center}

\subsection{\it exists}

This tactic handles the exists qualifier. In our system, $\tt\exists (x:T),P\ x$ is represented by $\tt ex\ T\ P$.

Here is the definition of $\tt ex$.
\begin{center}
\begin{minipage}{0.6\textwidth}
\begin{minted}{coq}
Inductive ex (A:Type) (P:A -> Type) : Type :=
| ex_intro : forall (x:A), P x -> ex A P.
\end{minted}
\end{minipage}
\end{center}

The proof of $\tt\exists (x:T), P\ x$ needs the user to provide a value $\tt x$
and prove that it satisfies the predicate $\tt P$.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it exists}\ \tt t.$ : If the goal is $\tt ex\ (x:T),P\ t$, and $\tt t$ is a term of type $\tt T$.
\end{itemize}

\subsubsection*{Implementation}
The tactic automatically generates the subgoal $\tt P\ x$.
\begin{center}
\begin{minipage}{0.7\textwidth}
\begin{minted}{coq}
(* Current Goal *)
ex nat (fun (n:nat) => eq nat (S (S O)) (plus n n))

(* $\Downarrow$ exists S O $\Downarrow$ *)

(* SubGoal *)
eq nat (S (S O)) (plus (S O) (S O))
\end{minted}
\end{minipage}
\end{center}

The proof object built is
\begin{center}
\begin{minipage}{0.8\textwidth}
\begin{minted}{coq}
ex_intro nat (fun (n:nat) =>
                eq nat (S (S O)) (plus n n)) (S O) (* SubGoal *)
\end{minted}
\end{minipage}
\end{center}

\subsection{\it induction}

This tactic applies to any goal.
The argument term must be of inductive type and the tactic {\it induction} generates subgoals,
one for each possible form of term, i.e., one for each constructor of the inductive type.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it induction}\ \tt t.$ : $\tt t$ is a term of some inductive type.
\end{itemize}

\subsubsection*{Implementation}

The tactic generates the predicate by the goal and automatically generates
the subgoals for every constructor of the inductive type.
The equivalence relations introduced by the constructors are also handled.

If the term is a hypothesis, {\it induction} will find the dependencies,
and try to move them and replace the subterms to solve the dependencies.
And then erase the original hypothesis.

\begin{center}
\begin{minipage}{0.7\textwidth}
\begin{minted}{coq}

n : nat
(* Current Goal *)
eq nat (plus n O) n

(* $\Downarrow$ induction n $\Downarrow$ *)

(* SubGoal 1 *)
eq nat (plus O O) O

n0 : nat
e : eq nat (plus n0 O) n0
(* SubGoal 2 *)
eq nat (plus (S n0) O) (S n0)
\end{minted}
\end{minipage}
\end{center}

After {\it induction}, all we need to prove is the property holds for \mintinline{coq}|O| and
for any \mintinline{coq}|n|, if \mintinline{coq}|P n| holds,
then the two subgoals are generated and \mintinline{coq}|n| is replaced by the constructors.

The case above is viewed from user's perspective.
On the other hand, in the system we also need to construct the term for back substitution.
\begin{center}
\begin{minipage}{\textwidth}
\begin{minted}{coq}
n : nat
(* Current Goal*)
eq nat (plus n O) n

(* $\Downarrow$ induction n $\Downarrow$ *)

(* The Proof Object *)
(fun (n:nat) =>
    nat_rect
    (fun (n0:nat) => eq nat (plus n0 O) n0)
    (* SubGoal 1 *)
    (fun (n0:nat) (e:eq nat (plus n0 O) n0) => (* SubGoal 2 *))
    n)
\end{minted}
\end{minipage}
\end{center}

\subsection{\it intro}

If the current goal is a dependent product $\tt \forall x:T,\ U$, \textit{intro} puts
$\tt x:T$ in the context, and the new subgoal is $\tt U$.\par
If the current goal is a non-dependent product $\tt \forall \_:T,\ U$, \textit{intro} renames
$\tt \_$ into $\tt s$ not in $\tt T,U$ and context. Then put $\tt s:T$ in the context,
and the new subgoal becomes $\tt U$.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it intro}\ .$ : introduce the outermost argument of current goal and rename them automatically.
\item ${\it intro}\ a_1\ a_2\ \cdots\ a_n.$ : introduce $n$ outermost arguments of current goal and
try to rename them into $a_1,a_2,\cdots,a_n$.
\end{itemize}

\subsubsection*{Implementation}
Since the context automatically preserves the index, we
can safely move the arguments out into the context without worrying about the index.

If the argument is dependent in either the consequent or some hypotheses of the goal,
the argument is replaced by the appropriate constructor form in each of the resulting subgoals
and induction hypotheses are added to the local context.

Here is an example:
\begin{center}
\begin{minipage}{\textwidth}
\begin{minted}{coq}
                                                        P : Type
(* Current Goal *)                (* intro P => *)      (* SubGoal *)
forall (x:Type) (f:False), x                            forall (f:False), P
\end{minted}
\end{minipage}
\end{center}

\subsection{\it intros}

If the current goal is a dependent (or non-dependent) product with $m$ arguments, namely,
$\tt \forall x_1:T_1,\ \forall x_2:T_2,\ \cdots\ \forall x_m:T_m,\ U$ where $\tt U$ is not a product type.
Then \textit{intros} will put these $m$ arguments into the context and rename automatically if necessary.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it intros}.$ : introduce all the argument of current goal and rename them automatically.
\end{itemize}

\subsubsection*{Implementation}
Keep calling {\it intro} until all the goal has no arguments.\par
Here is an example:
\begin{center}
\begin{minipage}{\textwidth}
\begin{minted}{coq}
                                                       x : Type
                                                       f : False
(* Current Goal *)                (* intros => *)      (* SubGoal *)
forall (x:Type) (f:False), x                           x
\end{minted}
\end{minipage}
\end{center}

\subsection{\it left right}
These tactics apply to a goal that is of the disjunction type. Then we can proof one of the two sides to solve the goal.

\subsubsection*{Usage}
\begin{itemize}
    \item ${\it left}.$
    \item ${\it right}.$
\end{itemize}

\subsubsection*{Implementation}
The disjunction type in our system is defined as follows:
\begin{center}
\begin{minipage}{0.6\textwidth}
\begin{minted}{coq}
Inductive or (A:Type) (B:Type) : Type :=
| or_introl : A -> or A B
| or_intror : B -> or A B.
\end{minted}
\end{minipage}
\end{center}

The tactic ${\it left}$ builds the proof object with the \mintinline{coq}|or_introl| constructor.
The tactic ${\it right}$ builds the proof object with the \mintinline{coq}|or_intror| constructor.
Here is an example:
\begin{center}
\begin{minipage}{\textwidth}
\begin{minted}{coq}
(* Current Goal *)
or (eq nat O O) (eq nat (S O) (S O))

(* $\Downarrow$ left $\Downarrow$ *)

(* The Proof Object *)
or_introl (eq nat O O) (eq nat (S O) (S O)) (* SubGoal 1 *)
\end{minted}
\end{minipage}
\end{center}

\subsection{\it split}
This tactic applies to a goal that is of the conjunction type. Then we need to proof the both sides to solve the goal.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it split}$.
\end{itemize}

\subsubsection*{Implementation}
The conjunction type in our system is defined as follows:
\begin{center}
\begin{minipage}{0.6\textwidth}
\begin{minted}{coq}
Inductive and (A:Type) (B:Type) : Type :=
| conj : A -> B -> and A B.
\end{minted}
\end{minipage}
\end{center}

The proof object is built with the \mintinline{coq}|conj| constructor.
Here is an example:
\begin{center}
\begin{minipage}{\textwidth}
\begin{minted}{coq}
(* Current Goal *)
and (eq nat O O) (eq nat (S O) (S O))

(* $\Downarrow$ split $\Downarrow$ *)

(* The Proof Object *)
conj (eq nat O O) (eq nat (S O) (S O)) (* SubGoal 1 *) (* SubGoal 2 *)
\end{minted}
\end{minipage}
\end{center}

\subsection{\it reflexivity}
This tactic applies to a goal that has the form $\tt eq\ T\ t\ u$, it check that $\tt t$ and $\tt u$
are convertible and then solves the goal.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it reflexivity}.$
\end{itemize}

\subsubsection*{Implementation}
This tactic simply checks if the terms are convertible and build the proof object with \mintinline{coq}|eq_refl| constructor.

\begin{center}
\begin{minipage}{0.7\textwidth}
\begin{minted}{coq}
n : nat
(* Current Goal *)
eq nat n (plus O n)

(* reflexivity *)

(* The Proof Object *)
fun (n:nat) => eq_refl nat n
\end{minted}
\end{minipage}
\end{center}

\subsection{\it rewrite}
This tactic applies to any goal.
The type of term must have the form $\tt eq\ T\ t\ u$.
Then the subterms $\tt t$ in the goal will be rewritten by $\tt u$.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it rewrite\ \tt H}.$ : $\tt H$ should have the form $\tt eq\ T\ t\ u$,
all the subterms $\tt t$ in the goal will be replaced by $\tt u$.
\item ${\it rewrite\ \texttt{<-}\ \tt H}.$ : $\tt H$ should have the form $\tt eq\ T\ t\ u$,
    all the subterms $\tt u$ in the goal will be replaced by $\tt t$.
\item ${\it rewrite\ [\texttt{<-}/\texttt{->}]\ \tt H\ \it in\ \tt H1}.$ :
    $\tt H$ should have the form $\tt eq\ T\ t\ u$, for the {\tt ->} case, all the
    subterms $\tt t$ in $\tt H1$ will be replaced by $\tt u$.
\end{itemize}

\subsubsection*{Implementation}
This tactic builds the proof object with \mintinline{coq}|eq_rect| or \mintinline{coq}|eq_rect_r|.

\begin{center}
\begin{minipage}{0.7\textwidth}
\begin{minted}{coq}
a : nat                                        a : nat
b : nat                                        b : nat
e : eq nat a b                                 e : eq nat a b
(* Current Goal *)     (* rewrite e => *)      (* SubGoal *)
eq nat b a                                     eq nat b b

(* Proof Object *)
fun (a:nat) (b:nat) (e:eq nat a b) =>
  (fun (e0:eq nat b b) =>
    eq_rect_r nat b (fun (n:nat) (e1:eq nat b n) =>
      eq nat b n) e0 a e) (* SubGoal *)
\end{minted}
\end{minipage}
\end{center}

\subsection{\it simpl}
This tactic applies to any goal, it tries to reduce a term to something still readable instead of fully normalizing it.
It performs a sort of strong normalization with a key difference:
\begin{center}
\begin{minipage}{0.7\textwidth}
it unfold a constant if and only if it leads to a $\iota$-reduction, i.e., reducing a {\tt match} or unfolding a {\tt Fix}.
\end{minipage}
\end{center}

\subsubsection*{Usage}
\begin{itemize}
\item ${\it simpl}.$
\item ${\it simpl\ in}\ \tt H.$ : $\tt H$ is a hypothesis.
\end{itemize}

\subsubsection*{Implementation}

The tactic first applies $\beta\iota$-reduction, then it expands constants and tries to reduce
further using $\beta\iota$-reduction, but when no $\iota$ rule is applied after unfolding,
$\iota$-reductions are not applied. For instance, trying to use $\it simpl$ on
\mintinline{coq}|eq nat (plus n O) n| changes nothing.
\begin{center}
\begin{minipage}{0.8\textwidth}
\begin{minted}{coq}
n : nat                                          e : nat
(* Current Goal *)           (* simpl => *)      (* Current Goal *)
eq nat (plus n O) (plus O n)                     eq nat (plus n O) n
\end{minted}
\end{minipage}
\end{center}

The tactic can also be applied to the hypotheses, and the same reduction rules applies.
\begin{center}
\begin{minted}{coq}
n : nat                                                    n : nat
e : eq nat (plus n O) (plus O n)                           e : eq nat (plus n O) n
(* Current Goal *)                 (* simpl in e => *)     (* Current Goal *)
eq nat (plus n O) n                                        eq nat (plus n O) n
\end{minted}
\end{center}

The tactic would never change the current built proof object and won't modify the proof objects built after it.
It is transparent in the proof tree.

\subsection{\it exact}

This tactic checks if the input term is actually of the goal's type. If so, the proof of the current goal is finished and the
constructed proof object is exactly the input term.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it exact}\ \tt t.$ : $\tt t$ is a term of the goal's type.
\end{itemize}

\subsubsection*{Implementation}
It's easy to implement. Here is an example:
\begin{center}
\begin{minipage}{0.8\textwidth}
\begin{minted}{coq}
m : nat
(* Current Goal *)  (* exact eq_refl nat m => *)  (* No more goals *)
eq nat m m
\end{minted}
\end{minipage}
\end{center}

\subsection{\it symmetry}
This tactic applies to a goal that has the form $\tt eq\ T\ t\ u$
and changes it into $\tt eq\ T\ u\ t$.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it symmetry}$.
\end{itemize}

\subsubsection*{Implementation}
This tactic builds the proof object use the pre-defined theorem \mintinline{coq}|eq_sym|, here is an example for the proof object building.
\begin{center}
\begin{minipage}{0.6\textwidth}
\begin{minted}{coq}
m : nat
n : nat
e : eq nat m n
(* Current Goal*)
eq nat n m

(* $\Downarrow$ symmetry $\Downarrow$ *)

(* The proof object *)
fun (m:nat) (n:nat) (e:eq nat m n) =>
    eq_sym nat m n (* SubGoal *)
\end{minted}
\end{minipage}
\end{center}

\subsection{\it unfold}
This tactic applies to any goal, the argument should denote a defined constant (i.e., not a hypothesis or a axiom).
The tactic applies the $\delta$ rule to each occurrence of the constant in the current goal and then replaces it with its $\beta\iota$-normal form.

\subsubsection*{Usage}
\begin{itemize}
\item ${\it unfold}\ \tt i.$ : $\tt i$ is an identifier.
\item ${\it unfold}\ \tt i\ \it in\ \tt H.$ : $\tt i$ is an identifier and $\tt H$ is a hypothesis.
\end{itemize}

\subsubsection*{Implementation}
It finds the occurrences of the constant and applies the $\delta$ rule, the nameless index is automatically preserved.

\begin{center}
\begin{minipage}{0.8\textwidth}
\begin{minted}{coq}
Definition addTwo (n : nat) : nat := S (S n).

n : nat
m : nat
e : eq nat m n
(* Current Goal *)
eq nat (addTwo n) (S (S m))

(* $\Downarrow$ unfold addTwo $\Downarrow$ *)

n : nat
m : nat
e : eq nat m n
(* SubGoal *)
eq nat (S (S n)) (S (S m))

(* $\Downarrow$ symmetry $\Downarrow$ *)

(* The proof object *)
(fun (n:nat) (m:nat) (e:eq nat m n) => (* SubGoal *))
\end{minted}
\end{minipage}
\end{center}


\documentclass{article}

\usepackage{enumitem}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{parskip}
\usepackage{graphicx}
\usepackage{mathtools}
\usepackage{mathrsfs}
\usepackage{amsmath,amsthm,hyperref}
\usepackage{amssymb}
\usepackage{amsmath}
\DeclareMathOperator{\Tr}{tr}
\usepackage{bbm}
\usepackage{dsfont}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\Tr}{Tr}

% Margins
\usepackage[top=2.5cm, left=3cm, right=3cm, bottom=4.0cm]{geometry}
% Colour table cells
\usepackage[table]{xcolor}

% Get larger line spacing in table
\newcommand{\tablespace}{\\[1.25mm]}
\newcommand\Tstrut{\rule{0pt}{2.6ex}}         % = `top' strut
\newcommand\tstrut{\rule{0pt}{2.0ex}}         % = `top' strut
\newcommand\Bstrut{\rule[-0.9ex]{0pt}{0pt}}   % = `bottom' /

% my new commands
\newcommand\partialkj{\frac{\partial^2}{\partial\theta_k\partial\theta_j}}
\makeatletter
\newcommand*\bigcdot{\mathpalette\bigcdot@{.5}}
\newcommand*\bigcdot@[2]{\mathbin{\vcenter{\hbox{\scalebox{#2}{$\m@th#1\bullet$}}}}}
\makeatother
\newcommand{\minus}{\scalebox{0.5}[1.0]{$-$}}
\newcommand{\zero}{\scalebox{0.6}[0.75]{$^{(0)}$}}
\newcommand{\supx}[1]{\scalebox{0.6}[0.75]{$^{(#1)}$}}
\newcommand{\supi}{\scalebox{0.6}[0.75]{$^{(i)}$}}
\newcommand{\sumim}{\begin{align*}
    \sum_{i=1}^m
\end{align*}
}
\newcommand{\sumzi}{\begin{align*}
    \sum_{z\supi}
\end{align*}
}

\newcommand{\bigDash}{\scalebox{3.0}[1.0]{$-$}}

%%%%%%%%%%%%%%%%%
%     Title     %
%%%%%%%%%%%%%%%%%
\title{Problem Set #4:  Unsupervised Learning and Reinforcement Learning}
\author{Eitan Joseph \and Caroline Wang}
\date{\today}

\begin{document}
\maketitle

%%%%%%%%%%%%%%%%%
%   Problem 1   %
%%%%%%%%%%%%%%%%%
\section*{Problem 1}
\textbf{EM for supervised learning}\\
In class we applied EM to the unsupervised learning setting. In particular, we represented $p(x)$ by marginalizing over a latent random variable \begin{align*}
    p(x) = \sum_{z} p(x,z) = \sum_{z} p(x|z)p(z)
\end{align*}
However, EM can also be applied to the supervised learning setting, and in this problem we discuss a “mixture of linear regressors” model; this is an instance of what is often call the Hierarchical Mixture of Experts model. We want to represent $p(y|x)$, $x\in \mathbb{R}^n$ and $y\in\mathbb{R}$, and we do so by again introducing a discrete latent random variable
\begin{align*}
    p(y|x) = \sum_z p(y,z|x) = \sum_{z} p(y|x,z) p(z|x)
\end{align*}
For simplicity we’ll assume that z is binary valued, that p(y|x, z) is a Gaussian density, and that $p(y|x)$ is given by a logistic regression model. More formally
\begin{align*}
    p(z|x;\phi ) =& g(\phi^Tx)^z(1-g(\phi^Tx))^{1-z}\\
    p(y|x,z = i; \theta_i) = &\frac{1}{\sqrt{2\pi}\sigma}\exp\left(\frac{-(y-\theta_i^Tx)^2}{2\sigma^2}\right)\quad i = 0,1
\end{align*}where $\sigma$ is a known parameter and $\phi, \theta_0, \theta_1 \in \mathbb{R}$ are parameters of the model (here we use the subscript on $\theta $ to denote two different parameter vectors, not to index a particular entry in these vectors).\\
Intuitively, the process behind model can be thought of as follows. Given a data point $x$, we first determine whether the data point belongs to one of two hidden classes $z=0$ or $z = 1$, using a logistic regression model. We then determine $y$ as a linear function of $x$ (different linear functions for different values of z) plus Gaussian noise, as in the standard
linear regression model. For example, the following data set could be well-represented by the model, but not by standard linear regression.
\begin{enumerate}[label=(\alph*)]
    \item Suppose $x$, $y$, and $z$ are all observed, so that we obtain a training set
${(x\supx{1}, y\supx{1}, z\supx{1}), . . . , (x\supx{m}, y\supx{m}, z\supx{m})}$. Write the log-likelihood of the parameters,
and derive the maximum likelihood estimates for $\phi$, $\theta_0$, and $\theta_1$. Note that because $p(z|x)$ is a logistic regression model, there will not exist a closed form estimate of $\phi$. In this case, derive the gradient and the Hessian of the likelihood with respect to $\phi$; in practice, these quantities can be used to numerically compute the ML estimate.\\\\
    \textit{answer: }\\\\
    The log-likelihood can be written as
    \begin{align*}
         \ell(\phi,\theta_0, \theta_1)  =& \sumim \log p(y\supi|x\supi;\phi,\theta_0, \theta_1)\\
         = & \sumim \log p(y\supi |x\supi, z\supi;\theta_0, \theta_1 )p(z\supi | x\supi;\phi)\\
         =& \sumim \mathds{1}\{z\supi=0\}\log \left((1-g(\phi^Tx\supi))\left(\frac{1}{\sqrt{2\pi}\sigma}\exp\left(\frac{-(y\supi-\theta_k^Tx\supi)^2}{2\sigma^2}\right)\right)\right)\\ &+ \sumim \mathds{1}\{z\supi=1\}\log \left((g(\phi^Tx\supi))\left(\frac{1}{\sqrt{2\pi}\sigma}\exp\left(\frac{-(y\supi-\theta_1^Tx\supi)^2}{2\sigma^2}\right)\right)\right)
    \end{align*}
    The maximum likelihood estimation for $\theta_k$ can be derived by taking the gradient with respect to $\theta_k$ and setting the result to zero.\begin{align*}
        \nabla_{\theta_0}\ell(\phi,\theta_0, \theta_1) \overset{\mathrm{set}}{=}0
    \end{align*}
    Then by extracting the non-relevant constants to $\theta_k$: \begin{align*}
        &{}\nabla_{\theta_k}\sumim-(y-\theta_k^Tx\supi)^2 = 0\\
        &{}\sumim-2(y\supi-\theta_k^Tx\supi)\cdot x\supi=0\\
        \implies&{}\sumim x\supi y\supi= \sumim \theta_k^Tx\supi^Tx\supi \\
        \implies&{}X^T\Vec{y} = \theta_k^T X^TX\\
        \implies&{}\theta_k = (X^TX)^{-1}X^T\Vec{y}
    \end{align*}Therefore the specific estimations for $\theta_0$ and $\theta_1$ are: \begin{align*}
        \theta_0 = (X_0^TX_0)^{-1}X_0^T\Vec{y_0}\\
        \theta_1 = (X_1^TX_1)^{-1}X_1^T\Vec{y_1}
    \end{align*}
    Where $X_0$, $y_0$, $X_1$, $y_1$ are the associated matrices and vectors derived in each of the two separate summations that combine to equal all of $z$.\\

    The next step is to find the gradient vector and the Hessian matrix respect with respect to $\phi$.\\
    We can once again remove the terms not relating to $\phi$:\begin{align*}
        \nabla_{\phi } \ell(\phi,\theta_0, \theta_1)=& \nabla_{\phi}\sumim \mathds{1}\{z\supi=0\}\log (1-g(\phi^Tx\supi))+ \sumim \mathds{1}\{z\supi=1\}\log((g(\phi^Tx\supi))\\
        =& \nabla_{\phi} \sumim (1-z\supi) \log (1-g(\phi^Tx\supi))+ z\supi \log((g(\phi^Tx\supi))
    \end{align*}
    From previous classes, the derivative of a sigmoid is known to be $\frac{\partial}{\partial z}g(z) = g(z)(1-g(z))$, therefore \begin{align*}
        \nabla_{\phi }\ell(\phi,\theta_0, \theta_1) = &\sumim - \frac{1-z\supi}{(1-g(\phi^Tx\supi))}\frac{\partial}{\partial \phi}g(\phi^Tx\supi)\cdot x\supi + \frac{z\supi}{g(\phi^Tx\supi)}\frac{\partial}{\partial \phi}g(\phi^Tx\supi)\cdot x\supi\\
        =& \sumim - \frac{1-z\supi}{(1-g(\phi^Tx\supi))}g(\phi^Tx\supi)(1-g(\phi^Tx\supi))\cdot x\supi + \frac{z\supi}{g(\phi^Tx\supi)}g(\phi^Tx\supi)(1-g(\phi^Tx\supi))\cdot x\supi\\
        =& \sumim -(1-z\supi)g(\phi^Tx\supi) \cdot x\supi + z\supi(1-g(\phi^Tx\supi))\cdot x\supi\\
        =& \sumim x\supi (z\supi -g(\phi^Tx\supi))
    \end{align*}
    Finally, this tells us that the gradient is\begin{align*}
        \nabla_{\phi }\ell(\phi,\theta_0, \theta_1) = X^T(\Vec{z}-\Vec{g})\quad \text{where} \quad \Vec{g}_i = g(\phi^Tx\supi)
    \end{align*}
    Based on the linear algebra review sheet, the Hessian Matrix can be derived by taking the derivative of the gradient, which is equivalent to looking at each $i$th entry of the gradient vector, taking the gradient of that entry, and setting that to be the $i$th column of the Hessian. The $i$th entry of the gradient vector is \begin{align*}
        x\supi^T \cdot (\Vec{z} -\Vec{g})
    \end{align*}and after taking the derivative with respect to that $i$th entry we get \begin{align*}
        \nabla_{\phi} x\supi^T (\Vec{z} -\Vec{g}) = x\supi^T  (\Vec{g}\cdot (1-\Vec{g}))\cdot x\supi
    \end{align*}Which gives the matrix \begin{align*}
        H = X^TDX\quad \text{where} \quad D_{ii} = g(\phi^Tx\supi)(1-g(\phi^Tx\supi))
    \end{align*}
    \item Now suppose $z$ is a latent (unobserved) random variable. Write the log-likelihood of the parameters, and derive an EM algorithm to maximize the log-likelihood. Clearly specify the E-step and M-step (again, the M-step will require a numerical solution, so find the appropriate gradients and Hessians). \\\\
     \textit{answer: }\\\\
     The log-likelihood can now be written as
     \begin{align*}
         \ell(\phi,\theta_0, \theta_1)
         &{}= \sumim\log p(y\supi|x\supi;\phi,\theta_0, \theta_1)\\
         &{}= \sumim\log p(y\supi|x\supi,z\supi;\theta_0, \theta_1) p(z\supi|x\supi;\phi)\\
         &{}= \sumim\log \left(\left(1-g(\phi^Tx\supi)\right)^{1-z\supi}\left(\frac{1}{\sqrt{2\pi}\sigma}\exp\left(\frac{-(y\supi-\theta_k^Tx\supi)^2}{2\sigma^2}\right)\right)\right) \\&+ \log \left(\left(g(\phi^Tx\supi)^{z\supi}\right)\left(\frac{1}{\sqrt{2\pi}\sigma}\exp\left(\frac{-(y\supi-\theta_1^Tx\supi)^2}{2\sigma^2}\right)\right)\right)
     \end{align*}
     Since each $z\supi$ is unobserved they cannot be separated explicitly into two cases - this is because we have no way of knowing how to separate the summation into two groups for each cluster of $z\supi$ values when their values are hidden.\\

     In order to derive the new EM algorithm, instead of explicitly maximizing $\ell$ we will use the same approach as explained in class which will be to repeatedly construct a lower-bound on $\ell$ (E-step), and then optimize that lower-bound (M-step). \\

     In order to accomplish this, we first define $Q_i$ to be the distribution over $z_i$ for all $i$ (implying $\sum_z Q_i(z) = 1$ and $Q_i(z) \geq 0$). Next we write the following:
     \begin{align}
         \sumim\log p(y\supi|x\supi;\phi,\theta_0, \theta_1)&{}=
         \sumim \log\sum_{z\supi} Q_i(z\supi)\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{Q_i(z\supi)}\\
         &{}\geq\sumim\sum_{z\supi} Q_i(z\supi)\log\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{Q_i(z\supi)}
     \end{align}
     Where step (2) utilizes Jensen's Inequality.\\\\
     We note here that the term
     \begin{align*}
         \sum_{z\supi} Q_i(z\supi)\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{Q_i(z\supi)}
     \end{align*}
     is just the expectation over the quantity
     \begin{align*}
        \frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{Q_i(z\supi)}
     \end{align*}
     according to the distribution $Q_i$. By Jensen's Inequality we know that
     \begin{align*}
         f\left(\mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{Q_i(z\supi)}\right]\right) \geq \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[f\left(\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{Q_i(z\supi)}\right)\right]
     \end{align*}
     Since (2) holds true for any set of distributions $Q_i$ we can choose a specific distribution for $Q_i$ which makes the inequality hold with equality at $\phi$, $\theta_0$, $\theta_1$. Since Jensen's Inequality holds with equality over a constant valued random variable it suffices to find $Q_i(z\supi) = c \times p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi) $ for some $c \in \mathbb{R}, c \neq 0$. We can solve for $Q_i(z\supi)$ using many iterations of Bayes' Theorem as follows:
     \begin{align*}
         Q_i(z\supi) &{}=
         \frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{\sum_{z\supi}p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}\\
         &{}=\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi|x\supi;\phi)}{p(y\supi|x\supi;\theta_0, \theta_1)}\\
         &{}=\frac{p(y\supi|x\supi, z\supi;\theta_0, \theta_1)p(z\supi, x\supi;\phi)}{p(y\supi|x\supi;\theta_0, \theta_1)p(x\supi;\phi)}\\
         &{}=\frac{p(y\supi, x\supi, z\supi;\phi, \theta_0, \theta_1)}{p(y\supi, x\supi;\phi, \theta_0, \theta_1)}\\
         &{}= p(z\supi|x\supi, y\supi;\phi, \theta_0, \theta_1)
     \end{align*}
     With this solved we can now write the following EM algorithm
     \begin{align*}
         \text{(E-step) For each i, j set}\\
         &{}w_j^{(i)} := p(z\supi = j | x\supi, y\supi;\phi,\theta_j)\\
         \text{(M-step) Update the parameters of}\\
         &{}\sumim\sum_j w_j^{(i)}\log\frac{p(y\supi|x\supi, z\supi;\theta_j)p(z\supi|x\supi;\phi)}{w_j^{(i)}}
     \end{align*}
     In order to maximize the equation in the M-step with respect to the parameters $\phi$, $\theta_0$, $\theta_1$ we need to take the gradient with respect to each parameter and set the resulting expression to zero. We can first generalize the $\theta$s as one gradient and solve
     \begin{align*}
         \nabla_{\theta_k}\sumim\sum_j w_j^{(i)}\log\frac{p(y\supi|x\supi, z\supi;\theta_j)p(z\supi|x\supi;\phi)}{w_j^{(i)}} \overset{set}= 0
     \end{align*}
     After dividing out all the non variable multipliers and taking the gradient we are left with the equation
     \begin{align*}
         &{}\sumim-2(y\supi-\theta_k^Tx\supi)x\supi^T w_k^{(i)} = 0\\
         \implies&{}\sumim x\supi^T w_k^{(i)}y\supi - x\supi^T w_k^{(i)}\theta_k^Tx\supi = 0\\
         \implies&{}\sumim x\supi^T w_k^{(i)}\theta_k^Tx\supi = \sumim x\supi^T w_k^{(i)}y\supi\\
         \implies&{}\sumim x\supi^T w_k^{(i)}\theta_k^Tx\supi = \sumim x\supi^T w_k^{(i)}y\supi\\
        \implies&{}\theta_k^T\sumim x\supi^T w_k^{(i)}x\supi = \sumim x\supi^T w_k^{(i)}y\supi\\
        \implies&{}\theta_k^T =  \sumim (x\supi^T w_k^{(i)}x\supi)^{-1}x\supi^T w_k^{(i)}y\supi\\
        \implies&{}\theta_k = (X^T W X)^{-1}X^TWy \quad \text{where} \quad W_{i,i} = w_k^{(i)}\\
    \end{align*}
     Finally we must derive the gradient vector and Hessian matrix \begin{align*}
         &\nabla_{\phi } \sumim\sum_j w_j^{(i)}\log\frac{p(y\supi|x\supi, z\supi;\theta_j)p(z\supi|x\supi;\phi)}{w_j^{(i)}}\\
         =&\nabla_{\phi }\sumim w_0^{(i)}\log (1-g(\phi^Tx\supi))+ w_1^{(i)} \log((g(\phi^Tx\supi))
     \end{align*}
     Since $w_0^{(i)}=1-w_1^{(i)}$ we have\begin{align*}
         &\nabla_{\phi }\sumim (1-w_1^{(i)})\log (1-g(\phi^Tx\supi))+ w_1^{(i)} \log((g(\phi^Tx\supi))\\
         =&\sum_ix\supi (w_1^{(i)} -g(\phi^Tx\supi))
     \end{align*}The matrix representation turns out to be \begin{align*}
         \nabla_{\phi } = X^T (\Vec{w}-\Vec{g})\quad \text{where} \quad\Vec{g}_i = g(\phi^Tx\supi)
     \end{align*}
    Finding the Hessian for $\phi$ in part (b), yields the same result as part (a) \begin{align*}
         H = X^TDX\quad \text{where}\quad D_{i,i} = g(\phi^Tx\supi)(1-g(\phi^Tx\supi))
     \end{align*}
\end{enumerate}
\section*{Problem 2}
\textbf{Factor Analysis and PCA}\\
In this problem we look at the relationship between two unsupervised learning algorithms we discussed in class: Factor Analysis and Principle Component Analysis.\\
Consider the following joint distribution over (x, z) where $z\in \mathbb{R}^k$ is a latent random variable\begin{align*}
    z\quad &\sim \quad \mathscr{N}(0,I)\\
    x|z\quad &\sim \quad \mathscr{N}(Uz,\sigma^2I)
\end{align*}
where $U\in \mathbb{R}^k$ is a model parameters and $\sigma^2$ is assumed to be a known constant. This model is often called Probabilistic PCA. Note that this is nearly identical to the factor analysis model except we assume that the variance of $x|z$ is a known scaled identity matrix rather than the diagonal parameter matrix $\Phi$, and we do not add an additional $\mu$ term to
the mean (though this last difference is just for simplicity of presentation). However, as we will see, it turns out that as $\sigma^2 \xrightarrow[]{}0$, this model is equivalent to PCA.\\
For simplicity, you can assume for the remainder of the problem that $k = 1$, i.e., that $U$ is a column vector in $\mathbb{R}^n$.
\begin{enumerate}[label=(\alph*)]
\item Use the rules for manipulating Gaussian distributions to determine the joint distribution over $(x, z)$ and the conditional distribution of $z|x$. [Hint: for later parts of
this problem, it will help significantly if you simplify your solution for the conditional distribution using the identity we first mentioned in problem set #1: $(\lamba I+BA)^{-1}B =
B(\lambada I + AB)^{-1}.]$\\\\
    \textit{answer:}\\\\
    According to the definition of $x$, the expectation of $x$ is \begin{align*}
        E[x] = E[Uz+\epsilon] = UE[z] + E[\epsilon] = 0
    \end{align*}
    where $\epsilon \sim \mathscr{N}(0,\sigma^2I)$ indicates the added covariance noise.
    We can now obtain the four different components of the matrix $\Sigma$ using the substitution $x=Uz+\epsilon$\begin{align*}
        \Sigma_{zz} =& E[zz^T] = I\\
        \Sigma_{xz} =& E[(Uz+\epsilon)z^T] = (UE[zz^T] + E[\epsilon z^T]) = U\\
        \Sigma_{zx} =& E[z(Uz+\epsilon)^T] = U^TE[zz^T] + E[z\epsilon^T] = U^T\\
        \Sigma_{xx} =& E[(Uz+\epsilon)(Uz+\epsilon)^T] = UU^TE[zz^T]+UE[z\epsilon^T] + U^TE[\epsilon z^T]+E[\epsilon\epsilon^T] = UU^T +\sigma^2I
    \end{align*}
    Therefore our join distribution is written
    \begin{align*}
        \begin{bmatrix}z \\ x\end{bmatrix}\quad \sim \quad  \mathscr{N}\left(\begin{bmatrix}0 \\ 0\end{bmatrix}, \begin{bmatrix}I & U^T \\ U & UU^T+\sigma^2I\end{bmatrix}\right)
    \end{align*} Then we can apply the formula to define the conditional distribution $z|x \sim \mathscr{N}(\mu_{z|x}, \Sigma_{z|x})$ such that \begin{align*}
        \mu_{z|x} =& \mu_z+\Sigma_{zx}\Sigma_{xx}^{-1}x = U^T(UU^T+\sigma^2I)^{-1}x = \frac{U^Tx}{UU^T+\sigma^2}\\
        \Sigma_{z|x} =& \Sigma_{zz}-\Sigma_{zx}\Sigma_{xx}^{-1}\Sigma_{xz} = I-U^T(UU^T+\sigma^2I)^{-1}U = 1-\frac{U^TU}{UU^T+\sigma^2}
    \end{align*}
    In conclusion, \begin{align*}
        z|x \quad \sim \quad \mathscr{N}\left(\frac{U^Tx}{UU^T+\sigma^2}, 1-\frac{U^TU}{UU^T+\sigma^2}\right)
    \end{align*}
\item Using these distributions, derive an EM algorithm for the model. Clearly state the E-step and the M-step of the algorithm.\\\\
    \textit{answer:}\\\\
    For the E-step, we need to compute $Q_i(z\supi)$ where
    \begin{align*}
        Q_i(z\supi) = p(z\supi |x\supi ;U)
    \end{align*}
   For the M-step we need to maximize the equation \begin{align*}
        &\sumim \int_{z\supi}Q_i(z\supi)\log\frac{p(x\supi,z\supi: U)}{Q_i(z\supi)}\\
        =&\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[\log p(x\supi |z\supi; U)+\log p(z\supi ) -\log(Q_i(z\supi))\right]
    \end{align*}Then we take the gradient with respect to $U$ and set it to $0$ in order to maximize the equation above.
    \begin{align*}
        &\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[\log p(x\supi |z\supi; U)]\\
        =&\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[\log\frac{1}{\sqrt{2\pi^n}|\sigma^2I|}\exp \left(-\frac{1}{2\sigma^2}(x\supi -Uz\supi)^T(x\supi-Uz\supi)\right)\right]\\
        =&\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[\frac{1}{2}\log|\sigma^2I|-\frac{n}{2}\log(2\pi)-\frac{1}{2\sigma^2}(x\supi -Uz\supi)^T(x\supi-Uz\supi)\right]
    \end{align*}We can drop the terms not dependent on $U$ before taking the gradient. \begin{align*}
        &\nabla_U\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[\frac{1}{2\sigma^2}(x\supi -Uz\supi)^T(x\supi-Uz\supi)\right]\\
        =&-\frac{1}{2\sigma^2}\sumim \nabla_U\mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[\Tr  z\supi^TU^TUz\supi-2\Tr z\supi^TUx\supi\right]\\
        =&-\frac{1}{2\sigma^2}\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[2z\supi z\supi^TU^T-2z\supi^Tx\supi\right]\\
        =&-\frac{1}{\sigma^2}\sumim \left[U\mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[z\supi z\supi^T]-x\supi\mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[z\supi^T]\right]\overset{\mathrm{set}}{=}0\\
        \implies &U = \left(\sumim x\supi \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[z\supi^T]\right)\left(\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[z\supi z\supi^T]\right)^{-1}
    \end{align*}From the definition of $Q_i$ being Gaussian we know that \begin{align*}
    \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[z\supi^T] =& \mu_{z\supi|x\supi}^T\\
    \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}[z\supi z\supi^T] =& \mu_{z\supi|x\supi}\mu_{z\supi|x\supi}^T
    \end{align*}Therefore, the final update of the M-step is \begin{align*}
        U=\left(\sumim \:x\supi\mu_{z\supi|x\supi}^T\right)\left(\sumim \mu_{z\supi|x\supi}\mu_{z\supi|x\supi}^T\right)^{-1}
    \end{align*}
\item As $\sigma^2 \xrightarrow[]{}0$, show that if the EM algorithm convergences to a parameter vector $U^*$ (and such convergence is guaranteed by the argument presented in class), then $U^*$ must be an eigenvector of the sample covariance matrix $\Sigma = \frac{1}{m}\sum x\supi x\supi^T$
— i.e., $U^*$ must satisfy
\begin{align*}
    \lambda U^* = \Sigma U^*
\end{align*}
    \textit{answer:}\\\\
    The first thing to notice is that as $\sigma^2 \xrightarrow[]{}0$ the E-step only needs to compute the means and not the variances, since $\Sigma_{z|x}\xrightarrow[]{}0$ as well. If we let $w \in \mathbb{R}^m$ be the vector that contains all the means such that $w_i = \mu_{z\supi|x\supi}$ then we can compute the new E-step to be
    \begin{align}
        w_i = \mu_{z\supi|x\supi} =&{} U^T(\sigma^2 I - UU^T)^{-1}(Uz\supi + \epsilon)\\
        =&{} U^T (UU^T)^{-1}x\supi\\
        =&{} \frac{U^Tx\supi}{UU^T}\\
        =&{} \frac{x\supi^TU}{U^TU}
    \end{align}
    Where step (6) uses the fact that step (5) yields a Real. When this is written in matrix form we see that
    \begin{align*}
        w = \frac{XU}{U^TU}
    \end{align*}
    Next we need to derive the new M-step. We can use the result from (b) in order to obtain the new update rule
    \begin{align}
        U &{}= \left(\sumim \:x\supi\mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[z\supi^T\right]\right)\left(\sumim \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[z\supi z\supi^T\right]\right)^{-1}\\
        &{}= \left(\sumim \:x\supi\mu_{z\supi|x\supi}^T\right)\left(\sumim \mu_{z\supi|x\supi}\mu_{z\supi|x\supi}^T\right)^{-1}\\
        &{}= \left(\sumim \:x\supi w_i\right)\left(\sumim w_i w_i\right)^{-1}
    \end{align}
    Where step (8) is derived from the definition of $Q_i$ being Guassian with mean $\mu_{z\supi|x\supi}$ and variance $\Sigma_{z\supi|x\supi}$ yielding
    \begin{align*}
        \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[z\supi^T\right] &{}= \mu_{z\supi|x\supi}^T\\
        \mathop{\mathbb{E}}_{z\supi \mathtt{\sim} Q_i}\left[z\supi z\supi^T\right] &{}= \mu_{z\supi|x\supi}\mu_{z\supi|x\supi}^T + \Sigma_{z\supi|x\supi}
    \end{align*}
    Now to write expression (9) in matrix form we see that
    \begin{align*}
        \left(\sumim w_i w_i\right)^{-1} &{}= \frac{1}{w^Tw}
    \end{align*}
    however to write the numerator in matrix form will be a little more difficult. We want to represent that each column of $X$ from $i\in{1,...,m}$ gets multiplied by its the $i$th entry of $w$ and then summed together. We cannot simply write this as $X^Tw$ so we will instead have to construct a new diagonal matrix $W$ where $W_{i,i} = w_i$. Then, by multiplying $XW$ we get
    \begin{align*}
        XW &{}=
        \begin{bmatrix}
        | & | & &|\\
        x\supx{1} & x\supx{2} & \dots & x\supx{m}\\
        | & | & &|
        \end{bmatrix}_{n\times m}
        \begin{bmatrix}
        w_1 & & &\\
        & w_2 & &\\
        & & \ddots &\\
        & & & w_m
        \end{bmatrix}_{m\times m}\\
        &{}= \begin{bmatrix}
        | & | & &|\\
        w_1x\supx{1} & w_2x\supx{2} & \dots & w_mx\supx{m}\\
        | & | & &|
        \end{bmatrix}_{n\times m}
    \end{align*}
    Next we just need to sum up each column of $XW$ which we can accomplish by constructing new vector $\Vec{1}\in \mathbb{R}^m$ such that $\Vec{1}_i = 1$ for all $i\in{1,...,m}$ and transforming it according to $XW$ as follows
    \begin{align*}
    XW\Vec{1} &{}= \begin{bmatrix}
        | & | & &|\\
        w_1x\supx{1} & w_2x\supx{2} & \dots & w_mx\supx{m}\\
        | & | & &|
        \end{bmatrix}_{n\times m}
    \begin{bmatrix}
        1\\
        \vdots\\
        1
    \end{bmatrix}_{m\times1}\\
    &{}= \left(\sumim \:x\supi w_i\right)
    \end{align*}
    Finally we can write the expression for U in matrix form
    \begin{align*}
        U = \frac{XW\Vec{1}}{w^T w}
    \end{align*}
    We know that $U$ has converged when its value has not changed after the E-step, so we can substitute our value for $w$ in the E-step and set it equal to our original U.
    \begin{align*}
        &{}U = \frac{XW\Vec{1}}{\frac{U^TX^T XU }{(U^TU)^2}}\\
        \implies&{}(U^TX^T XU)U = (U^TU)^2 XW\Vec{1}\\
        \implies&{}X^TXU = U^TU XW\Vec{1}\\
        \implies&{}X^TXU = UU^T XW\Vec{1} = U^T XW\Vec{1}U\\
        \implies&{}\Sigma U = \lambda U
    \end{align*}
    where we know that $\Sigma = X^TX$ and we take $\lambda = U^T XW\Vec{1} \in \mathbb{R}$ and therefore proving the desired result.
\end{enumerate}
\section*{Problem 4}
    \textbf{Convergence of Policy Iteration}\\
    In this problem we show that the Policy Iteration algorithm, described in the lecture notes, is guaranteed to find the optimal policy for an MDP. First, define $B^\pi$ to be the Bellman operator for policy $\pi$, defined as follows: if $V^\prime = B(V)$, then\begin{align*}
        V^\prime (s) = R(s)+\gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V(s^\prime)
    \end{align*}
\begin{enumerate}[label=(\alph*)]
\item Prove that if $V_1(s) \leq  V_2(s$) for all $s\in \mathcal{S}$, then $B(V_1)(s) \leq  B(V_2)(s)$ for all $s\in \mathcal{S}$.\\\\
\textit{answer:}\\\\
Given that $V_1(s) \leq  V_2(s$) for all $s\in \mathcal{S}$, we can also write the inequality
\begin{align*}
    &R(s) + V_1(s^\prime) \leq R(s) + V_2(s^\prime)\\
    \implies& R(s) + \gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V_1(s^\prime) \leq R(s) + \gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V_2(s^\prime)
\end{align*}
However, it turns out that
\begin{align*}
    &R(s) + \gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V_1(s^\prime) = B(V_1)(s)\\
    &R(s) + \gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V_2(s^\prime) = B(V_2)(s)
\end{align*}
So we have shown that $B(V_1)(s) \leq B(V_2)(s)$ for all $s \in \mathcal{S}$
\item Prove that for any $V$, \begin{align*}
    ||B^\pi(V)-V^\pi ||_\infty\leq \gamma ||V-V^\pi ||_\infty
\end{align*}where $||V||_\infty = \max_{s\in \mathcal{S}}|V(s)|$. Intuitively, this means that applying the Bellman operator $B^\pi $ to any value function $V$, brings that value function “closer” to the value function for $\pi, V^\pi $. This also means that applying $B^\pi $ repeatedly (an infinite number of times):\begin{align*}
    B^\pi(B^\pi (B^\pi\dots B^\pi(V)\dots )))
\end{align*}will result in the value function $V^\pi$.\\\\
\textit{answer:}\\\\
We can first observe that
\begin{align*}
    ||B^\pi(V)-V^\pi ||_\infty &= \max_{s\in \mathcal{S}}\big |R(s)+\gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V(s^\prime)-R(S)-\gamma \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )V^\pi(s^\prime)\big |\\
    &= \gamma \max_{s\in \mathcal{S}}\big |\sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )(V(s^\prime)-V^\pi (s^\prime))\big |
\end{align*}
Then since $P_{s,a}$ is a probability distribution it must be true that \begin{align}
    P_{s,\pi(s)}(s^\prime )\geq 0\\
    \sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime) = 1
\end{align}
we can apply the fact that for any $a, x\in \mathbb{R}^n$, if $\sum_i a_i = 1$, and $a_i\geq 0$, then $\sum_i a_ix_i\leq \max_i x_i$ to (10) and (11) to write
\begin{align*}
    \gamma \max_{s\in \mathcal{S}}\big |\sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime )(V(s^\prime)-V^\pi (s^\prime))\big |\leq \gamma\max_{s^\prime\in \mathcal{S}} |V(s^\prime)-V^\pi(s^\prime)|
\end{align*}
Which is, by definition, the same inequality as
\begin{align*}
     ||B^\pi(V)-V^\pi ||_\infty\leq \gamma ||V-V^\pi ||_\infty
\end{align*}
\item Now suppose that we have some policy $\pi$, and use Policy Iteration to choose a new policy $\pi\prime$ according to
\begin{align*}
    \pi^\prime(s) = \argmax_{a\in A}\sum_{s^\prime\in S}P_{s,a}(s^\prime)V^\pi (s^\prime)
\end{align*}
Show that this policy will never perform worse that the previous one — i.e., show that for all $s \in \mathcal{S}$, $V^\pi(s) \leq V^{\pi^\prime}(s)$.\\\\
\textit{answer:}\\\\
We know that
\begin{align*}
    V^\pi(s) &{}= R(s) + \gamma\sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime)V^\pi(s^\prime)\\
    &{}\leq\\
    B^{\pi^\prime}(V^\pi)(s) &{}= R(S)+ \gamma\sum_{s^\prime \in \mathcal{S}}P_{s,\pi^\prime(s)}(s^\prime)V^\pi(s^\prime)
\end{align*}
due to the fact that
\begin{align*}
    \gamma\sum_{s^\prime \in \mathcal{S}}P_{s,\pi(s)}(s^\prime)V^\pi(s^\prime) \leq \gamma\max_{a\in A}\sum_{s^\prime \in \mathcal{S}}P_{s,a}(s^\prime)V^\pi(s^\prime) = \gamma\sum_{s^\prime \in \mathcal{S}}P_{s,\pi^\prime(s)}(s^\prime)V^\pi(s^\prime)
\end{align*}
We can further expand this to show
\begin{align*}
    B^{\pi^\prime}(V^\pi)(s) &{}= R(S)+ \gamma\sum_{s^\prime \in \mathcal{S}}P_{s,\pi^\prime(s)}(s^\prime)V^\pi(s^\prime)\\
    &{}= R(S)+ \gamma\max_{a\in \mathcal{A}}\sum_{s^\prime \in \mathcal{S}}P_{s,a}(s^\prime)V^\pi(s^\prime)\\
    &{}= R(S)+ \gamma\max_{a\in \mathcal{A}}\sum_{s^\prime \in \mathcal{S}}P_{s,a}(s^\prime)(R(s^\prime) + \gamma\sum_{s^{\prime\prime} \in \mathcal{S}}P_{s^\prime,\pi(s^\prime)}(s^{\prime\prime})V^\pi(s^{\prime\prime}))\\
    &{}\leq R(S)+ \gamma\max_{a\in\mathcal{ A}}\sum_{s^\prime \in \mathcal{S}}P_{s,a}(s^\prime)(R(s^\prime) + \gamma\max_{a^\prime \in \mathcal{A}}\sum_{s^{\prime\prime} \in \mathcal{S}}P_{s^\prime,a^\prime}(s^{\prime\prime})V^\pi(s^{\prime\prime}))\\
    &{}= B^{\pi^\prime}(B^{\pi^\prime}(V^\pi))(s)
\end{align*}
Using part (b) we also know that\begin{align*}
    ||B^\pi(V)-V^\pi ||_\infty\leq \gamma ||V-V^\pi ||_\infty
\end{align*}
implies that applying $B^\pi$ repeatedly (an infinite number of times) results in the convergent identity \begin{align*}
    B^{\pi^\prime}(B^{\pi^\prime} (B^{\pi^\prime}\dots B^{\pi^\prime}(V)\dots ))) = V^{\pi^\prime}
\end{align*} which finally proves the desired result that $V^\pi(s) \leq V^{\pi^\prime}(s)$ for all $s \in \mathcal{S}$.
\item Use the proceeding exercises to show that policy iteration will eventually converge (i.e., produce a policy $\pi^\prime = \pi$). Furthermore, show that it must converge to the optimal policy $\pi^*$. For the later part, you may use the property that if some value function satisfies
\begin{align*}
    V(s) = R(s) + \gamma\max_{a\in A}\sum_{s^\prime \in \mathcal{S}}P_{s,a}(s^\prime)V(s^\prime)
\end{align*}
then $V = V^*$.\\\\
\textit{answer:}\\\\
Given that $|S|+|A| < \infty$ (i.e. that the number of states and the number of \\\m actions are both finite), we let $\pi_n$ be the resulting policy after the $n$th iteration of the Policy Iteration, and let
\begin{align*}
    \lim_{n\xrightarrow[]{}\infty}\pi_n = \pi_{\infty}
\end{align*}
We know from part (c) that $V^\pi(s) \leq V^{\pi^\prime}(s)$ for all s, where $\pi^\prime$ is the resulting policy after one iteration of Policy Iteration, therefore we have that
\begin{align*}
    V^{\pi_n}(s) \leq V^{\pi_\infty} \quad\forall s \in \mathcal{S},  \quad \forall n < \infty
\end{align*}
However, since there are only $k = |A|^{|S|}$ possibly policies it must be the case that
\begin{align*}
    \lim_{n\xrightarrow[]{}\infty}\pi_n = \pi_{k}
\end{align*}
Therefore Policy Iteration converges to $\pi_{k}$ and further,
\begin{align*}
    V^{\pi_{k}}(s) =&{} R(s) + \gamma\sum_{s^\prime \in \mathcal{S}}P_{s,\pi_{|A|^|S|}}(s^\prime)V^{\pi_{k}}\\
    =&{} R(s) + \gamma\max_{a\in \mathcal{A}}\sum_{s^\prime \in \mathcal{S}}P_{s,a}(s^\prime)V^{\pi_{k}}
\end{align*}
which proves that $V^{\pi_{k}} = V^*$
\end{enumerate}
\end{document}

Unsupervised Learning and Reinforcement Learning/Problem Set 4 - Unsupervised Learning and Reinforcement Learning.tex


\subsection{Exponents}

Eigenmath requires parentheses around negative exponents.
For example,

\begin{Verbatim}[formatcom=\color{blue}]
10^(-3)
\end{Verbatim}

\noindent
instead of

\begin{Verbatim}[formatcom=\color{blue}]
10^-3
\end{Verbatim}

\noindent
The reason for this is that the binding of the negative sign is not always
obvious.
For example, consider

\begin{Verbatim}[formatcom=\color{blue}]
x^-1/2
\end{Verbatim}

\noindent
It is not clear whether the exponent should be $-1$ or $-1/2$.
So Eigenmath requires

\begin{Verbatim}[formatcom=\color{blue}]
x^(-1/2)
\end{Verbatim}

\noindent
which is unambiguous.

\bigskip
\noindent
In general, parentheses are always required when the exponent
is an expression.
For example, \verb$x^1/2$ is evaluated as $(x^1)/2$ which
is probably not the desired result.

\begin{Verbatim}[formatcom=\color{blue}]
x^1/2
\end{Verbatim}

\noindent
$\displaystyle \tfrac{1}{2}x$

\bigskip
\noindent
Using \verb$x^(1/2)$ yields the desired result.

\begin{Verbatim}[formatcom=\color{blue}]
x^(1/2)
\end{Verbatim}

\noindent
$\displaystyle x^{1/2}$


\lab{Finite Volume Methods}{Finite Volume Methods}
\label{lab:finitevolume}

When solving a PDE numerically, how do we deal with discontinuous initial data?
The Finite Volume method has particular strength in this area.
It is commonly used for hyperbolic PDEs whose solutions can spontaneously develop discontinuities as they evolve in time.
These solutions are often called shock waves.

\section*{Conservation Laws}
Consider the conservation law
\begin{align}
\label{eqn:1D_continuous}
u_t + f(u)_x = 0,
\end{align}
where $u$ is a (spatially) one-dimensional conserved quantity, and $f(u)$ is the flux of $u$.
The continuous integral formulation of  \eqref{eqn:1D_continuous} states that
\begin{align*}
	\frac{d}{dt}\int_a^b u(x,t) dx + \int_a^b f(u)_x \,dx = 0.
\end{align*}
$\frac{d}{dt}\int_a^b u(x,t) dx$ may be thought of as the time evolution of the total `mass' of $u$ across the domain $[a,b]$, and is dependent only on the flux through the boundaries, since
\begin{align*}
\frac{d}{dt}\int_a^b u(x,t) dx = f(u(a))-f(u(b)).
\end{align*}
This fact is an important idea utilized by finite volume methods, which generally consider the evolution of $u$ not at a given point, but instead in volume-averaged regions.
For example, let $\{x_i\}$ be a grid of equally spaced points with spacing $\Delta x$, and let $C_i$ be the $i$-th `volume' (subinterval) defined by $(x_{i-1/2},x_{i+1/2})$.
We are interested in the evolution of the volume average of $u$ over this interval,
\begin{align*}
U_i^n  = \frac{1}{\Delta x}\int_{C_i} u(x,t^n)dx,
\end{align*}
where $\{t^n\}$ is the time discretization.

\begin{figure}
\centering
\includegraphics[trim= 20mm 195mm 0mm 30mm, clip]{flux_form.pdf}
\caption{A schematic of the fluxes for the finite volume method as indicated by \eqref{eqn:flux_form}.}
\end{figure}

The evolution of these volume-averaged quantities will depend only on the flux through the cell edges, so that
\begin{align}
\label{eqn:1D_semi_continuous}
\frac{d}{dt}\int_{C_i} u(x,t) dx = f(u(x_{i-1/2},t))-f(u(x_{i+1/2},t)).
\end{align}
We can then construct a time-stepping method where $\sum_i U_i^n\Delta x$ (the total `mass' of the system) is conserved from one time step $n$ to the next.

Let $F^n_{i-1/2} = \frac{1}{\triangle t}\int_{t^n}^{t^{n+1}} f(u(x_{i-1/2},t)) \, dt$.
Then
\begin{align*}
\int_{t^n}^{t^{n+1}} \left[ \frac{d}{dt}\int_{C_i} u(x,t)\, dx \right]
&{}= \int_{C_i} u(x,t^{n+1}) - u(x,t^{n})\, dx ,\\
&{ }= \triangle t \left(F^n_{i-1/2} - F^n_{i+1/2}\right).
\end{align*}

Thus, by integrating \eqref{eqn:1D_semi_continuous} in time, we may approximate the evolution of the cell (`volume') averages with the method
\begin{align}
\label{eqn:flux_form}
U_i^{n+1} = U_i^n - \frac{\Delta t}{\Delta x} \left(F^n_{i+1/2}-F^n_{i-1/2}\right).
\end{align}
where $U_i^n = \frac{1}{\triangle x} \int_{C_i} u(x,t^n) \, dt$.
% where
% \begin{gather}
% F_{i-1/2}^n = \frac{1}{\Delta t} \int_{t_n}^{t_{n+1}}f(u(x_{i-1/2},t))dt.
% \end{gather}
This formulation guarantees the conservation properties that are so desirable for conservation laws, if the time-averaged fluxes $F_{i-1/2}^n$ can be discretized in a natural way.

The key contribution of finite volume methods is the computation of $F_{i-1/2}^n$.
For a truly nonlinear $f(u)$ this can be rather complicated and messy, and typically will involve solving what is usually referred to as the Riemann problem for the conservation law.
The interested student can look at \cite{Le2002} for a very thorough introduction and discussion on the subject.
We will consider the linear problem in one dimension.
The analog to higher dimensions is obtained by considering the eigenvector decomposition of any linear system.
Nonlinear equations complicate things further.

\section*{The linear advection equation and upwinding}
The simplest conservation law describes the advection or transport of a quantity.
The PDE is given by
\begin{align}
u_t + au_x = 0, \label{eqn:fv_advection}
\end{align}
and describes the motion of a concentration of some constituent $u$ by a constant velocity one-dimensional `wind' $a>0$.
%(a slight generalization of this would be to consider $a=a(x)$, but the general concepts are exactly the same as those viewed here, just a more local consideration is necessary).
In higher dimensions this is an important problem in many fields, for example the transport of chemicals in the atmosphere and oceans, proper mixing of various properties in metallurgy, and the passing of information along a network.

Note that whenever $u(x,t)$ is a solution of the advection equation, then $u(x-at,t_0)$ (for any fixed $t_0$) is also a solution.
Thus, if $u(x,0) = u_0(x)$ then the solution for all time can be represented by $u(x,t) = u_0(x-at)$.
This is an important property of \eqref{eqn:fv_advection}, and gives a new meaning to the term advection: this equation merely takes the initial conditions and passively transports them with velocity $a$.

For this equation the computation of the flux appears straightforward: $F_{i-1/2}^n = a\overline{U}^n_{i-1/2}$ where the $\overline{U}^n_{i-1/2}$ refers to the time average of $U_{i-1/2}$ over the interval $t_n$ to $t_{n+1}$.
Let us determine how to approximate this time average.
% To understand how this might be done, note from Figure \ref{something} that for the advection equation with $a>0$ the flux that determines $U_i^{n+1}$ will be dependent on the value of $U_{i-1}^n$ so that it makes sense that we approximate the flux here as $F_{i-1/2}^n = aU_{i-1}^n$ (recall that the $U_i^n$ are cell averages of the continuous quantity $u(x)$).
Note from Figure \ref{fig:fv_LW_reconstruction} that when $a>0$ the flux that determines $U_i^{n+1}$ will be dependent on the value of $U_{i-1}^n$. Thus, one possibility is to approximate the flux by $F_{i-1/2}^n = aU_{i-1}^n$.
Using this approximation of the flux together with the flux differencing formula \eqref{eqn:flux_form} yields the first order upwind method, given by
\begin{align*}
U_i^{n+1} = U_i^n - \frac{a\Delta t}{\Delta x}\left(U_i^n-U_{i-1}^n\right).
\end{align*}
% (recall that the $U_i^n$ are cell averages of the continuous quantity $u(x)$).
% Similarly, if $a<0$ then $F_{i-1/2}^n = aU_{i}^n$.
% This is called the first order upwind method
% and gives a completely discretized version of the problem as \eqref{eq:flux_form} with
% \begin{align*}
% F_{i-1/2}^n = a^- U_i^n + a^+U_{i-1}^n,
% \end{align*}
% where $a^- = \min (a,0)$ and $a^+ = \max (a,0)$.
% Note that this formulation allows for an easy extension to the case where $a=a(x)$ is a more general function of $x$.
% From now on we will consider the simpler case where $a>0$ is constant.
% From now on we will consider $a>0$ only, the generalization to other cases is relatively straightforward.
% In either case, the flux differencing formula \eqref{eq:flux_form} yields
% \begin{align*}
% U_i^{n+1} = U_i^n - \frac{a\Delta t}{\Delta x}\left(U_i^n-U_{i-1}^n\right)
% \end{align*}
% for the upwind method.
Another way to derive the upwind method is to instead suppose that what we want to do is reconstruct $u(x)$ at each time step $n$ inside each cell $(x_{i-1/2},x_{i+1/2})$from the mean values in that cell and its surrounding neighbors. This reconstructed $\tilde{u}(x)^n$ is then defined piecewise for each cell $i$.
The solution at the next time step can be found as $\tilde{u}^{n+1}(x) = \tilde{u}^n(x-a\triangle t)$ which allows us to determine the fluxes $F^n_{i-1/2}$ once we have settled on a method for determining $\tilde{u}^n(x)$ in each cell.
The simplest approach is
\[\tilde{u}^n(x) = U_i^n \text{ for }x \in (x_{i-1/2},x_{i+1/2})\]
This leads to fluxes given by
\begin{align}
F_{i-1/2}^n &= \frac{a}{\triangle t}\int_{0}^{t_{n+1}-t_n} \tilde{u}^n(x_{i-1/2},t)\, dt , \\
&= \frac{a}{\triangle t}\int_{0}^{\triangle t} \tilde{u}^n(x_{i-1/2}-at)\, dt  ,\\
&= aU_{i-1}^n.\nonumber
\end{align}

\begin{figure}
\includegraphics[width=\textwidth, trim = 20mm 65mm 20mm 65mm, clip]
{LW_reconstruction.pdf}
\caption{The piecewise linear reconstruction for the upwind and Lax-Wendroff methods.
The solid lines represent the simplest reconstruction of the cell averages leading to the upwind method, and the dashed lines are those whose slope is obtained via the Lax-Wendroff method.
Note that the LW method introduces a spurious maximum at $i+3/2$ (the cell edge between $U_{i+1}$ and $U_{i+2}$) and the minimum at $i-3/2$ will be unphysical exaggerated.
The upwind method avoids this difficulties, but clearly loses a significant amount of the available information.
This provides the motivation for the slope limiters.}
\label{fig:fv_LW_reconstruction}
\end{figure}

The following code solves the problem
\begin{align}
\begin{split}
	u_t +a u_x &= 0, \quad 0 < x < 1, \\
	u(x,t) &= f(x),\\
	u(0,t) &= u(1,t),
\end{split}\label{eqn:fv_exercise}
\end{align}
where $f$ represents a signal with two parts: one is smooth and the other is discontinuous.
Notice that this PDE has periodic boundary conditions.
Essentially we are evolving the signal around the unit circle.
This allows us to evolve the signal much further to test our numerical methods, since we only have to discretize the interval $[0,1]$ instead of a much larger domain.
To see how to implement the boundary conditions, consider a grid $0 = x_0< x_1< \ldots< x_{N-1}< x_N = 1$ of evenly spaced points.
Since $x_N = x_0$, it is sufficient to track $x_0, \ldots , x_{N-1}$.

\begin{lstlisting}
import numpy as np
from matplotlib import pyplot as plt
from math import floor

def upwind(u0, a, xmin, xmax, t_final, nt):
    """ Solve the advection equation with periodic
    boundary conditions on the interval [xmin, xmax]
    using the upwind finite volume scheme.
    Use u0 as the initial conditions.
    a is the constant from the PDE.
    Use the size of u0 as the number of nodes in
    the spatial dimension.
    Let nt be the number of spaces in the time dimension
    (this is the same as the number of steps if you do
    not include the initial state).
    Plot and show the computed solution along
    with the exact solution. """
    dt = float(t_final) / nt
    # Since we are doing periodic boundary conditions,
    # we need to divide by u0.size instead of (u0.size - 1).
    dx = float(xmax - xmin) / u0.size
    lambda_ = a * dt / dx
    u = u0.copy()
    for j in xrange(nt):
        # The Upwind method. The np.roll function helps us
        # account for the periodic boundary conditions.
        u -= lambda_ * (u - np.roll(u, 1))
    # Get the x values for the plots.
    x = np.linspace(xmin, xmax, u0.size+1)[:-1]
    # Plot the computed solution.
    plt.plot(x, u, label='Upwind Method')
    # Find the exact solution and plot it.
    distance = a * t_final
    roll = int((distance - floor(distance)) * u0.size)
    plt.plot(x, np.roll(u0, roll), label='Exact solution')
    # Show the plot with the legend.
    plt.legend(loc='best')
    plt.show()

# Define the initial conditions.
# Leave off the last point since we're using periodic
# boundary conditions.
nx = 30
nt = nx * 3 // 2
x = np.linspace(0., 1., nx+1)[:-1]
u0 = np.exp(-(x - .3)**2 / .005)
arr = (.6 < x)  &  (x < .7 )
u0[arr] += 1.

# Run the simulation.
upwind(u0, 1.2, 0, 1, 1.2, nt)
\end{lstlisting}

Try running the previous code block with \li{nx} set to $30$, $60$, $120$, and $240$.
You will notice that the numerical solution diffuses with time.
It diffuses especially fast at the points of discontinuity.

\begin{comment}
Another way to derive the upwind method is to instead suppose that what we want to do is reconstruct $u(x)$ at each time step $n$ inside each cell $(x_{i-1/2},x_{i+1/2})$ from the mean values in that cell and its surrounding neighbors.
This reconstructed $\tilde{u}(x)^n$ is then defined piecewise for each cell $i$.
The solution at the next time step can be found as $\tilde{u}^{n+1}(x) = \tilde{u}^n(x-a\Delta t)$ which allows us to determine the fluxes $F_{i-1/2}^n$ once we have settled on a method for determining $\tilde{u}^n(x)$ in each cell.
The simplest approach is
\begin{gather*}
\tilde{u}^n(x) = U^n_i \mbox{ for } x \in (x_{i-1/2},x_{i+1/2}).
\end{gather*}
% This leads to fluxes given by
% \begin{gather}
% F_{i-1/2}^n = \frac{a}{\Delta t} \int_{t_n}^{t_{n+1}}\tilde{u}(x_{i-1/2}-a(t-t_n),t)dt = \frac{a}{\Delta t} \int_{t_n}^{t_{n+1}}U_i^n dt = aU_i^n.
% \end{gather}
In either case, we can see that using the flux differencing formula \eqref{eq:flux_form} that
\begin{gather}
U_i^{n+1} = U_i^n - \frac{a\Delta t}{\Delta x}\left(U_i^n-U_{i-1}^n\right)
\end{gather}
for the upwind method.
\end{comment}

\section*{Piecewise linear reconstruction and slope limiters}
The upwind method is formally only first order, and actually does relatively poorly in terms of actually transporting the initial data with velocity $a$.
You can notice from the example code that the upwind method has errors that are `diffusive' meaning that the initial data is diffused as time evolves, losing the peaks and fine details.
This is because the error for the upwind method is on the order of the second derivative of $u$ which is of a diffusive nature.
To get an improved method, consider a better reconstruction inside each cell, i.e.
\begin{gather}
\tilde{u}^n(x) = U_i^n + m_i^n(x-x_i) \mbox{ for } x \in (x_{i-1/2},x_{i+1/2})
\end{gather}
where the slope of this linear reconstruction $m_i^n$ is determined as a function of the neighboring cell averages at time $n$ and $U_i^n$ itself.
Then the flux is given by
\begin{align}
\begin{split}
F_{i-1/2}^n
&= \frac{a}{\triangle t}\int_{0}^{t_{n+1}-t_n} \tilde{u}^n(x_{i-1/2}-at)\, dt  ,\\
&= \frac{a}{\triangle t}\int_{0}^{\triangle t} U_{i-1}^n + m_i^n(x_{i-1/2} - at - x_i), \\
&= a \left( U_{i-1}^n + \frac{m_{i-1}^n}{2} (\triangle x - a \triangle t)\right).
\end{split}
\label{eqn:finite_volume_improved_flux}
\end{align}
One of the most natural approaches is to just estimate the slope depending on the cell $i$ and a neighboring cell $i+1$ or $i-1$.
This leads to two popular methods, the Lax-Wendroff method and the Beam-Warming method (that really is the name).
The Lax-Wendroff method has a slope chosen as
\begin{gather}
m_i^n = \frac{U^n_{i+1}-U_{i}^n}{\Delta x}.
\end{gather}
% This leads to
% \begin{gather}
% U_i^{n+1} = U_i^n - \frac{\Delta t}{2\Delta x} a \left(U_{i+1}^n-U_{i-1}^n\right) + \frac{1}{2}\left(\frac{\Delta t}{\Delta x}\right)^2 a^2 \left(U_{i-1}^n - 2U_i^n+U_{i+1}^n\right)
% \end{gather}
which it turns out is formally second-order accurate.
It turns out though that the errors for this method are dispersive, meaning that near very steep gradients, the method will generate very rapid oscillations (due to the third derivative of $u$ not being approximated accurately).
Another way to consider how these errors arise is to notice from Figure \ref{fig:fv_LW_reconstruction} that if the piecewise linear reconstruction is advocated by some positive wind $a$ then there will be places where the discontinuous nature of the reconstruction will introduce spurious maxima or minima into the solution.
These become the spurious waves seen in simulations using the Lax-Wendroff method.

A solution to this dilemma between balancing the diffusive and dispersive errors comes from constructing slopes $m_i^n$ that ensure no such non-monotonic transport takes place.
The basic idea is to constrain the slope so that the reconstructed piecewise linear function $\tilde{u}^n(x)$ will not generate unphysical extremal values when it is advocated by some finite wind $a$.
The Minmod limiter chooses the slope as
\begin{gather}
m_i^n = minmod\left(\frac{U_i^n-U_{i-1}^n}{\Delta x},\frac{U_{i+1}^n-U_{i}^n}{\Delta x}\right)
\end{gather}
where
\begin{gather}
minmod(a,b) = \left\{\begin{array}{ccc}a  \mbox{ if } |a| < |b|  \mbox{ and } ab>0\\
b  \mbox{ if } |b|<|a|  \mbox{ and } ab>0\\ 0  \mbox{ if } ab < 0 .\end{array}\right.
\end{gather}

% \begin{itemize}
% \item Minmod limiter
% \begin{gather}
% m_i^n = minmod\left(\frac{U_i^n-U_{i-1}^n}{\Delta x},\frac{U_{i+1}^n-U_{i^n}}{\Delta x}\right)
% \end{gather}
% where
% \begin{gather}
% minmod(a,b) = \left\{\begin{array}{ccc}a  \mbox{ if } |a| < |b|  \mbox{ and } ab>0\\
% b  \mbox{ if } |b|<|a|  \mbox{ and } ab>0\\ 0  \mbox{ if } ab < 0 .\end{array}\right.
% \end{gather}
% \item Superbee limiter
% \begin{gather}
% m_i^n = maxmod(m^{(1)}_i,m^{(2)}_i)
% \end{gather}
% where the maxmod function selects $m^{(k)}_i$ with the larger modulus, and
% \begin{gather}
% m^{(1)}_i = minmod \left(\frac{U_{i+1}^n-U_i^n}{\Delta x},2\frac{U_i^n-U_{i-1}^n}{\Delta x}\right)\\
% m^{(2)}_i = minmod \left( 2\frac{U_{i+1}^n-U_i^n}{\Delta x},\frac{U_i^n-U_{i-1}^n}{\Delta x}\right).
% \end{gather}
% \item Monotonized central-difference limiter
% \begin{gather}
% m_i^n = minmod\left(\frac{U_{i+1}^n-U_{i-1}^n}{2\Delta x},2\frac{U_i^n-U_{i-1}^n}{\Delta x},2\frac{U_{i+1}^n-U_i^n}{\Delta x}\right)
% \end{gather}
% \end{itemize}

% \begin{problem}
% Code up the Superbee and Minmod %MC
% limiters for an advection problem with constant wind.
% Use different initial data that is smooth or has steep gradients (such as the square wave in the example code) to see which of the three limiters may be more optimal.
% \end{problem}

\begin{problem}
Implement the Lax Wendroff method and use it to solve \eqref{eqn:fv_exercise}.
Compare it to the upwind method for $N = 30, 60, 120$, and 240.
You should be able to tell that the Lax Wendroff method approximates the smooth portion of the signal much better, as it does not struggle with diffusion.
Unfortunately, it has some difficulty with the discontinuous portion, where unphysical oscillations are seen.
Recall that we saw something similar in the waves lab when there were discontinuous initial conditions.

Hint: Use equations \ref{eqn:finite_volume_improved_flux} and \ref{eqn:flux_form}.
\end{problem}

\begin{problem}
Implement the Minmod method and use it to solve \eqref{eqn:fv_exercise}.
Compare it to the upwind and Lax Wendroff methods for $N = 30, 60, 120$, and 240.
Be sure to vectorize the minmod operation.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{FiniteVolume_Comparisons.pdf}
\caption{Solutions of \eqref{eqn:fv_exercise} at time $t = 1.2$ using various methods.
Here the advection coefficient is $a = 1.2$, and there are  $N = 100$ subintervals in space, 150 subintervals in time.}
\label{fig:fv_Comparisons}
\end{figure}

\section*{Beyond piecewise linear reconstructions}
As you can imagine, using a linear approximation is not the only option.
There are a host of high order finite volume methods that consider polynomial reconstructions of $\tilde{u}^n$ inside each cell.
The key is then to use some nonlinear limiting technique that will ensure that when $\tilde{u}^n(x)$ is advocated that no new extrema are introduced.
Choosing the correct limiter for the given application then becomes an art unto itself.


%!TEX root = ../main.tex

\section{Vectors problems}
\label{sec:vec_problems}

\vspace{-2mm}

You learned a bunch of vector formulas and you saw some vector diagrams,
but did you really learn how to solve problems with vectors?
There is only one way to find out: test yourself by solving problems.

I've said it before and I don't want to repeat myself too much,
but it's worth saying again: the more problems you solve, the better you'll understand the material.
It's now time for you to try the following vector problems to make sure you're on top of things.

\medskip

{ \small

\begin{problems}{ch3}

	\begin{problem}
		Given the vectors $\vec{u}=(1,1,1)$, $\vec{v} = (2,3,1)$, and $\vec{w}=(-1,-1,2)$,
		compute the following products:

		\threecol
			\textbf{a)}~$\vec{u} \cdot \vec{v}$

			\textbf{b)}~$\vec{u} \cdot \vec{w}$

			\textbf{c)}~$\vec{v} \cdot \vec{w}$
		\endthreecol

		\threecol
			\textbf{d)}~$\vec{u} \times \vec{v}$

			\textbf{e)}~$\vec{u} \times \vec{w}$

			\textbf{f)}~$\vec{v} \times \vec{w}$
		\endthreecol

		\begin{answer}\textbf{a)}~$6$.
					\textbf{b)}~$0$.
					\textbf{c)}~$-3$.
					\textbf{d)}~$(-2, 1, 1)$.
					\textbf{e)}~$(3, -3, 0)$.
					\textbf{f)}~$(7, -5, 1)$.\end{answer}
	\end{problem}

	\begin{problem}
		Given the vectors $\vec{p} =(1,1,0,3,3)$ and $\vec{q}=(1,2,3,4,5)$, calculate the following expressions:
		\threecol
			\textbf{a)}~$\vec{p}+\vec{q}$

			\textbf{b)}~$\vec{p}-\vec{q}$

			\textbf{c)}~$\vec{p} \cdot \vec{q}$
		\endthreecol

		\begin{answer}\textbf{a)}~$(2,3,3,7,8)$.
					\textbf{b)}~$(0,-1,-3,-1,-2)$.
					\textbf{c)}~$30$.\end{answer}
	\end{problem}

	\begin{problem}
		Find a unit vector that is perpendicular to both  $\vec{u}=(1, 0, 1)$ and $\vec{v} =(1, 2, 0)$.
		\begin{hint}
			Use the cross product.
		\end{hint}
		\begin{answer}$(-\frac{2}{3}, \frac{1}{3}, \frac{2}{3})$ or $(\frac{2}{3}, -\frac{1}{3}, -\frac{2}{3})$.\end{answer}
		\begin{solution}See \href{http://bit.ly/1cOa8yo}{\texttt{bit.ly/1cOa8yo}} for calculations.
		\end{solution}
	\end{problem}

	\begin{problem}
		Find a vector that is orthogonal to both $\vec{u}_1 = (1, 0, 1)$ and $\vec{u}_2 =(1, 3, 0)$,
		 and whose dot product with the vector  $\vec{v}= (1, 1, 0)$ is equal to $8$.
		%	\begin{hint}
		%	\end{hint}
		\begin{answer}$(12, -4, -12)$.\end{answer}
		\begin{solution}
			Any multiple of the vector $\vec{u}_1 \times \vec{u}_2 = (-3,1,3)$
			is perpendicular to both $\vec{u}_1$ and 	$\vec{u}_2$.
			We must find a multiplier $t \in \mathbb{R}$ such that
			$t(-3,1,3) \cdot (1, 1, 0) = 8$.
			Computing the dot product we find $-3t + t = 8$, so $t=-4$.
			The vector we're looking for is $(12, -4, -12)$.
			See \href{http://bit.ly/1nmYH8T}{\texttt{bit.ly/1nmYH8T}} for calculations.
		\end{solution}
	\end{problem}

\end{problems}

} % /small


In this section, we first define the problem we address, then the mathematical notation we use, and finally our  evaluation metrics.

%In this section, we will define our addressing problem with the notation. Also, the background about this paper will be provided.

\subsection{Problem definition}

The problem we address in this paper is related to the need of an AUI
to ease the task of monitoring alerts in
large-scale displayed networks. The new interface is expected
to filter irrelevant information to provide localized context for
each event or alert (defined loosely as a relevant related content localized in time, space, and/or keyword usage). Hence, we define the problem we are addressing in this
paper as a problem of filter optimization.
We assume that selection of information elements to display in visual interface is obtained via settings of three filters that can jointly express a global filter. These three sub-filters are the following:

%The problem we address in this paper is the proper display of portion in the network for user's easy to investigate large-scale network elements. The new system is expected to filter information to provide localized and relevant element in the interface. This problem is able to be defined as a filter search/recommendation problem, We expect to employ standard IR theory to solve this problem based on our argument about the similarity of AUI and IR system. We assume that display of the elements in the network is obtained via three filters that can jointly express a global filter. These three sub-filters are:
\begin{itemize}
\item Time: expresses the time window of a retrieved set.
\item Keyword: expresses a set of keyword to include or exclude from a retrieved set.
\item Location: expresses a bounding box of a retrieved set. The location can be expressed as longitude/latitude values, or pure coordinates obtained using a given display layout.
\end{itemize}

The problem we study is how to optimize one or multiple filter selection settings to maximize retrieval of relevant content.
%best set of elements that is assessed using one of the metrics described above, i.e., expected precision, expected recall, or expected F1-score

\subsection{Mathematical Notation}
Throughout this paper, we present all algorithms for Greedy and Optimal search using the following mathematical notation:

\begin{itemize}
\item An element $j$, for which three types of metadata are associated: (i) a textual content, which is composed of a set of terms of size $n$, (ii) a timestamp $t_e$, which may represent the creation date of $j$, and (iii) a position coordinates $(x_{e},y_{e})$.
\item Three variables $I(j)$, $B(j)$ and $S(j)$ are associated to each element $j$. $I(j)$ refers to whether element $j$ is in the retrieved set (E). $B(j)$ is the boolean variable about relevance of element $j$. $S(j)$ is a probabilistic score of  element $j$ being relevant. Note that $I(j)$ is correlated with the UI system. $B(j)$ and $S(j)$ are independent of the system. %$\emph{I(j), B(j)} \in \{0, 1\}$, $\emph{S(j)} \in \left[0, 1\right]$.
Also, $B(j)$ follows a \emph{Bernoulli} distribution with parameter $S(j)$, and hence, the expectation of $B(j)$ is $S(j)$, i.e.,
  $\mathbb{E_S}[B(j)] = S(i)$.

%which allows to derive the expectation of \emph{B(j)} as follows:
%\begin{equation}
 % \mathbb{E_S}[B(j)] = 0*(1 - S(i))+ 1*S(i) = S(i)
%\end{equation}

\item A global collection of elements $GC$ with size $m$. %Two subsets of $GC$ are particularly important in this research: retrieved set $E$ and relevant set $RS$.
\item A retrieved set $E$ with $E \subseteq  GC$, for which we use $E^*$ to refer to any subset, i.e., $E^*\subseteq E$. This variable depends on our user-interface filter system. Note that size of $E$ can be represented the sum of $I(j)$ among the global collection. Therefore, we have $|E| = \sum_{j=1}^m I(j)$.
\item A relevant set $RS$ of $|RS|$ elements, which contains all relevant elements. This is independent of the UI-filter system. Note that size of $RS$ is equal to the sum of $B(j)$ among the global collection. Therefore, we have $|RS| = \sum_{j=1}^m B(j)$. %However, $B(j)$ is not available for our estimation of $RS$ size in practice. We have to use expected $RS$ size $|RS|$, $\mathbb{E_S}|RS|$, to approximate $|RS|$.
%\begin{equation}
  %|RS| \approx \mathbb{E_S}|RS| = \sum_{j=1}^m \mathbb{E_S}[B(j)] = \sum_{j=1}^m S(j)
%\end{equation}

\item A keyword query $Q_k=\{\neg t_{1}^{*},\dots \neg t_{k}^{*}\}$, which expresses query term exclusion, i.e., find elements that don't contain the terms $t_{1}^{*},\dots t_{k}^{*}$.
\item  A time query $Q_t=[t_{start},t_{end}]$, which expresses the search for elements in the time window $[t_{start},t_{end}]$.
\item A position query $Q_p=[(x_{min},y_{min}),(x_{max},y_{max})]$, which expresses the search of elements falling in the bounding box represented by the  lower and upper bound coordinates -- respectively $(x_{min},y_{min})$ and $(x_{max},y_{max})$.
\item A query filter $Q$, which combines the three sub-filters $Q_k$, $Q_t$, and $Q_p$ in a conjunction, i.e., $Q=[Q_k\wedge Q_t\wedge Q_p]$.

\end{itemize}

%The goal is to explore an algorithm to obtain an optimal filter setting to retrieve a set of elements with maximum score of the specific metric.

%\subsection{Comparison to standard IR search}
%The comparison between information retrieval for web search and information retrieval for filtering in AUIs can be summarized in Table \ref{tbl:Comparaison2IR}. Obviously , there are important differences between web search and filtering for AUIs, especially in the indirect selection of results through filter settings. This unexplored field provides new possibilities and challenges of research in this novel information retrieval setting:
%\begin{itemize}
%\item Evaluation metrics and human factors: Are there new evaluation metrics specific to this AUI filtering setting? What evaluation metrics correlate with AUI user performance?
%\item Optimization and algorithms: How do we optimally select filter settings to maximize evaluation metrics in expectation? How can we interpret simple heuristics like average and cumulative relevance? (Answer: expected precision.) How do MILPs, relaxed LP approximations with guarantees, or greedy approaches compare in terms of time and metric quality? Are there properties of different filters (1D for time, 2D for bounding box, or discrete choices for property selection) that lend themselves to specialized greedy approaches?
%\item Robustness: As the signal-to-noise ratio varied in the quality of the relevance scoring, how do various algorithms perform?
%\item Explanation: Can we provide explanations for filter settings to allow users to understand the reasons for the suggestions?
%\item Personalization and learning: Can we learn from observations of manual adaption of the suggested filter settings to understand how to improve the third-party scoring systems?
%\item Collaborative filtering: Can we generalize learning across multiple
%users in a collaborative filtering approach?
%\item Learning from implicit feedback: How can we leverage implicit user
%feedback such as clicks and dwell time to indirectly measure the relevance
%of filtered content and improve system performance.
%\end{itemize}

%\subsection{Optimization technique definition}

%The \emph{greedy algorithm} is an algorithmic paradigm that aims to obtain a global optimum of a problem in terms of making the locally optimal decision at each step \cite{Black2005}. In a search problem, a greedy algorithm does not in general produce an optimal solution, but it still yields locally optimal solutions that approximate a global optimum in a reasonable time.

%The greedy algorithms described in this paper are coupled with a \emph{Top-down} search strategy, which basically begins with the whole search space, and then partition into several sub-spaces in a lower level for the local optimization search heuristic.

%An \emph{optimization-based} search is the problem of finding the best solution from all feasible solutions. Usually, the standard form of an optimization problem is defined as the minimization/maximization of a given objective function subject to a set of constraints.

%The search problem will be transformed into Mixed integer linear programming (MILP), which involves problems in which only some of the variables, $x_{i}$, are constrained to be integers, while other variables are allowed to be non-integers.

\subsection{Evaluation metrics}

We adopt the Boolean relevance framework of IR and thus assume that all information elements $j$ have a ground truth relevance assessment $B(j)$ available at evaluation time.
%%However, unlike previous document-based methods for estimating relevance at retrieval-time, we do not rely on text associated with information elements and instead rely on a third-party to supply an estimated probability (score) of relevance $S(j)$.
%Traditionally, Boolean labels indicate the relevance of documents is used to evaluate retrieval sets (e.g., precision).
%In practice, IR models are based on probabilistic scores to measure relevance of the elements, such as TF-IDF and cosine similarity.
%In practice, we are able to employ third-party technique to provide probabilistic scores to measure relevance of the document in IR system, such as TF-IDF and cosine similarity. Consequently,
Because filters are equivalent to Boolean retrieval (they either select or do not select content as relevant) and we have a probabilistic estimate of relevance $S(j)$, we propose to evaluate
expected variants of standard precision, recall, and F1-score.
%Boolean retrieval evaluation metrics based on the relaxation of Boolean labels to probabilistic scores resulting in expected metrics, i.e., \emph{expected precision} (EP), \emph{expected recall} (ER) and \emph{expected F1-Score} (EF1).
Specifically because the filter that selects all information elements would maximize expected recall, the filter that selects the highest probability singleton information element would maximize expected precision, we focus on expected F1-score as a trade-off to balance expected precision and recall.

Given a retrieved information element set $E$, a relevant element set $RS$ selected by a filter setting, the global information element collection $GC$ with size $m$, the precision of a retrieved set $P(E)$ is defined as follows:
\begin{equation}
   P(E) = \dfrac{\sum_{j \in RS} B(j)}{|E|} = \dfrac{\sum_{j=1}^m B(j)I(j)}{\sum_{j=1}^m I(j)}
\end{equation}
The relaxation of the Boolean label $B(j)$ to probabilistic score $S(j)$ gives the following definition of \emph{expected precision}:
\begin{equation}
EP(E)=\mathbb{E_{S}}[\dfrac{\sum_{j=1}^{m}B(j)I(j)}{\sum_{j=1}^{m}I(j)}]=\dfrac{\sum_{j=1}^{m}\mathbb{E_{S}}[B(j)]I(j)}{\sum_{j=1}^{m}I(j)}=\dfrac{\sum_{j=1}^{m}S(j)I(j)}{\sum_{j=1}^{m}I(j)}
\end{equation}
Similarly the recall of a retrieved set $R(E)$ is defined as:
\begin{equation}
   R(E) = \dfrac{\sum_{j \in RS} B(j)}{|RS|} = \dfrac{\sum_{j=1}^m B(j)I(j)}{|RS|}
\end{equation}
Taking a 1st order Taylor expansion, we have the following expectation approximation %$\mathbb{E}(X/Y)\approx\dfrac{\mathbb{E}(X)}{\mathbb{E}(Y)}$
$\mathbb{E}(X/Y)\approx \mathbb{E}(X)/ \mathbb{E}(Y)$ for two dependent random variables $X$ and $Y$~\cite{Kempen2000}. Hence,
we can now define an \emph{approximated expected recall}:
%Given a retrieved element set $E$  and a relevant element set $RS$ among a global element collection $GC$ with size $m$, we propose the following definition of \emph{expected precision} (EP), \emph{expected recall} (ER) and \emph{expected F1-Score} (EF1):
\begin{equation}
   	\emph{ER(E)}=\mathbb{E_{S}}[\dfrac{\sum_{j=1}^{m}B(j)I(j)}{|RS|}] \approx \dfrac{\sum_{j=1}^{m}\mathbb{E_{S}}[B(j)]I(j)}{\mathbb{E_{S}}[|RS|]} = \dfrac{\sum_{j=1}^m S(j)I(j)}{\sum_{j=1}^m S(j)}
\end{equation}
Finally, we define the \emph{approximated expected F1-Score} using the \emph{expected precision} and the \emph{approximated expected recall} as follows:
\begin{align}
    \emph{EF1(E)}  \approx \dfrac{2\times EP\times ER}{EP+ER} = \dfrac{2\times\sum_{j=1}^m S(j)I(j)}{\sum_{j=1}^m I(j) + \sum_{j=1}^m S(j)}
\end{align}

%Conventional metrics are based on Boolean relevance label $B(j)$ instead of probabilistic score $S(j)$. To link standard precision with our expected precision,  The definition of standard precision $P(RS)$ is as follows:

%Now, we  derive expectation of precision $P(RS)$ as follows:
%\begin{align}
%	\mathbb{E_S}[P(E)] &= \mathbb{E_S}[\dfrac{\sum_{j=1}^m B(j)I(j)}{\sum_{j=1}^m I(j)}] = \dfrac{\sum_{j=1}^m %\mathbb{E_S}[B(j)]I(j)}{\sum_{j=1}^m I(j)} \notag \\
%    &= \dfrac{\sum_{j=1}^m S(j)I(j)}{\sum_{j=1}^m I(j)} = EP(E)
   	%\mathbb{E_S}[R(RS)] &= \dfrac{\sum_{j=1}^m \mathbb{E_S}[B(j)] I(j)}{\sum_{j=1}^m \mathbb{E_S}[B(j)]} = \dfrac{\sum_{j=1}^m S(j)I(j)}{\sum_{j=1}^m S(j)} = ER(RS)  \\
    %\mathbb{E_S}[F1(RS)] &= \dfrac{2*\sum_{j=1}^m \mathbb{E_S}[B(j)]I(j)}{\sum_{j=1}^m I(j) + \sum_{j=1}^m \mathbb{E_S}[B(j)]} \notag \\
    %&= \dfrac{2*\sum_{j=1}^m S(j)I(j)}{\sum_{j=1}^m I(j) + \sum_{j=1}^m S(j)} = EF1(RS)
%\end{align}

%\begin{equation}
%\emph{\ensuremath{EP(RS)}}=\dfrac{{\displaystyle \sum_{j\in RS}S(j)}}{|RS|} =  \dfrac{\sum_{j=1}^m S(j)I(j)}{\sum_{j=1}^m I(j)}
%\end{equation}

%Analogously, we define the  \emph{expected recall} (ER) as follows:
%\begin{equation}
%\emph{\ensuremath{ER(RS)}}=\dfrac{{\displaystyle \sum_{j\in RS}S(j)}}{|RE|} = \dfrac{\sum_{i=1}^m S(i)I(i)}{\sum_{i=1}^m S(i)} = \dfrac{\sum_{i=1}^m S(i)I(i)}{C}
%\end{equation}

%\noindent where $|RE|$ is the size of the entire relevant element set. Finally, we define the  \emph{expected F1-Score} (EF1) as follows:

%\begin{equation}
%\emph{\ensuremath{EF1(RS)}}=\dfrac{2\times EP\times ER}{EP+ER} = \dfrac{2*\sum_{i=1}^m S(i)I(i)}{\sum_{i=1}^m I(i) + C}
%\end{equation}

In Figure \ref{fig:F1_vs_EF1}, we experimentally show that while EF1 is only an approximation of the true expectation, EF1 serves as an excellent surrogate for F1, which is our main concern when considering filter optimization.  That is, maximizing EF1 score with respect to a noisy classifier (noise decreases to 0 as $\lambda \to 1$) is strongly correlated with maximizing F1 score evaluated on the ground truth.   Specifically, while the EF1 and F1 scores are not perfectly calibrated along the diagonal, there is a linear correlation in that as the EF1 score increases for a scenario, the F1-score proportionally increases on average (as shown by the best fit linear regression in the plots).

\begin{figure}[H]
\begin{centering}
\par\end{centering}
\begin{centering}
\includegraphics[width=8.5cm]{imgs/Enron_results/scatter_plot_EF1\lyxdot vs\lyxdot F1}
\par\end{centering}
\caption{Scatter plot showing EF1-Score vs. F1-Score.}
\label{fig:F1_vs_EF1}
\end{figure}


\section{Homogeneous Systems}
\noindent
Homogeneous systems of linear equations have the form
\begin{equation*}
	\vec{x}' = A\vec{x}
\end{equation*}
We'll see how to find solutions to these types of systems using the eigenvalue method. Finding solutions to these systems will allow us to find homogeneous solutions to heterogeneous systems of equations.

% Eigenvalue Method
\input{./linearSystems/homogeneousSystems/eigenvalueMethod.tex}