%!TEX root = thesis-kdyoung.tex

% \begin{savequote}%[45mm]
% If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.
% \qauthor{John von Neumann}
% \end{savequote}
\begin{savequote}%[45mm]
---Everything should be made as simple as possible, but not simpler.
\qauthor{Albert Einstein}
\end{savequote}

\chapter{Introduction}
\label{chap:intro}
Assembly lines are flow-based production systems widely used by
the manufacturing industry.
Designing these systems to meet the requirements set
by the industry gives rise to the Assembly Line
Balancing Problem (\albp{}).
An assembly line problem consists of a sequence
of \emph{stations} and a set of \emph{tasks}.
Stations are fixed points along the assembly
line which are operated by either human personnel
or autonomous machinery.
A workpiece is launched down the line and
a subset of the tasks are performed on it at each station.
Once all required tasks are finished, the completed product
exits the assembly line.
% The construction of a single product is completed
% once all tasks have performed on it
% To completely construct a single product
% the all tasks are performed along the line.
The workload of tasks must be divided among the stations
to optimize some performance measure,
while also respecting physical restrictions.
A standard example of such a restriction would be a
precedence relation, which can require
one task to be completed before another can begin
processing on the product.
The line itself is a mechanism used to
transport the products between stations;
most commonly the line is a conveyor belt.

There are two common objectives when designing an assembly
line, which each lead a distinct problem.
The first is called the type-1 problem and is concerned with
the design of a new assembly line where the production rate
of the product is predefined and the aim is to construct the line
with as few stations as possible, in-order to meet this demand.
The second problem, called type-2, considers the redesign of an existing
assembly line where the number of stations is fixed and the rate
of production is optimized.
The second case can arise when a manufacturing company alters their
production process or changes the products they offer.
Our thesis focuses on solving the type-2 problem.

The \emph{cycle time} of an assembly line defines the time each station
is given to perform its assigned tasks.
As such, finding the minimal possible cycle time
is the primary aim when optimizing the production
rate of an assembly line.
Since all stations are connected by a single line,
the station with the largest workload will define the
overall cycle time of the line.
To achieve the minimal cycle time value,
we aim to balance the workload of the tasks
as evenly as possible among the stations.

The structure of the paper is as follows.
The remainder of Chapter \ref{chap:intro}
provides illustrative examples of the problem
and closes with some motivation.
In Chapter \ref{chap:theory} we detail a selection
of the relevant theory for our problem, spanning
the fields of Operations Research and Artificial Intelligence.
Chapter \ref{chap:mip} presents the mixed-integer
programs which were used to compare our
solution methodology against.
The construction of our logic-based Benders
decomposition is presented in Chapter \ref{chap:benders},
together with many possible modelling choices.
The results of our computational experiments are
given in Chapter \ref{chap:exp}.
We conclude in Chapter \ref{chap:conc}
with some remarks on the performance of our
method and finally note a range of possible directions
for future research on this topic.

\section{Preliminaries and Examples}
\label{sec:intro:prelim}
The fundamental problem in this area is called the
Simple \albp{}, or \sab{}; we denote
the type-2 version by \sab{2}.
We now detail some straightforward examples
to illustrate how the workload
of an assembly line can be balanced.
In Example \ref{ex:intro:simple} we provide an instance
of the \sab{2} with a possible feasible solution and the optimal solution given
in Figures \ref{fig:intro:exSchedFeas} and \ref{fig:intro:exSchedOpt}
respectively. Note, that $t_i$ denotes the processing time of task $i$ and
$c$ denotes the cycle time of the assembly line.

\begin{example}\label{ex:intro:simple}
	Consider an instance of the problem with four tasks $T_1,\ldots,T_4$,
	and three stations $S_1,\ldots,S_3$,
	along the assembly line.
	The aim is to find an assignment of the tasks to the stations
	which minimizes the cycle time.
	This assignment must respect the precedence relations listed in Figure \ref{fig:intro:exPrec},
	\eg task $T_1$ must be completed before task $T_2$.
	The processing time of each task is included in Figure \ref{fig:intro:exPrec}
	next to each task.

	A feasible solution to this problem is given in Figure \ref{fig:intro:exSchedFeas},
	where the stations are spaced along the horizontal axis.
	Each station's workload begins at the horizontal axis and proceeds
	upward through time until all assigned tasks are completed.
	In this solution, the workload of $S_3$ is the largest
	and as such defines the cycle time as $t_3+t_4=2+9=11$.

	This feasible solution can be improved by moving task $T_3$ from station
	$S_3$ to $S_2$.
	For this assignment to remain feasible, $T_3$ must be processed
	after $T_2$, due to the precedence relations.
	This new assignment leads to the optimal solution listed in
	Figure \ref{fig:intro:exSchedOpt} with a cycle time of nine.	\qed
\end{example}

\begin{figure}[tpb]
	\centering
	\caption{Precedence graph}
	\vspace{2mm}
	\includegraphics[width=0.45\textwidth]{images/precgraphSimple.eps}
	\label{fig:intro:exPrec}
\end{figure}

\begin{figure}[tpb]
	\centering
	\begin{minipage}{0.45\textwidth}
		\centering
		\caption{Feasible solution}
		\vspace{2mm}
		\includegraphics[width=\linewidth]{images/exSimpleFeas.eps}
		\label{fig:intro:exSchedFeas}
	\end{minipage}
	\hfill
	\begin{minipage}{0.45\textwidth}
		\centering
		\vfill
		\caption{Optimal solution}
		\vspace{2mm}
		\includegraphics[width=\linewidth]{images/exSimpleOpt.eps}
		\label{fig:intro:exSchedOpt}
	\end{minipage}
\end{figure}

The variant of the \albp{} which we are concerned with adds the consideration
of sequence-dependent setup times between consecutive tasks
within a station's workload.
Theoretical approaches to the \albp{} usually assume that the assembly
line workers can decide on an arbitrary precedence-feasible sequence
to execute their assigned tasks, which will not affect the station's total
processing time.
However, in practice there can be non-trivial setup costs, due
to walking times or tool changes, which can account for a considerable amount
of a station's processing time.
Adding this consideration to the problem
leads to a scheduling problem arising within each station.
This variant of the problem is called the SetUp Assembly Line Balancing and
Scheduling Problem (\sua{}).

To realistically model the setup costs that occur
in assembly lines, two types of setups are introduced;
\emph{forward} and \emph{backward} setups.
A forward setup time, denoted by $\phi$, occurs between two consecutive
tasks within a station's task sequence if both tasks
are performed on the same product.
Whereas a backward setup time, denoted by $\beta$, occurs between the last task
performed on a product and the first task performed on
the next product along the line.

\begin{figure}[tpb]
	\centering
	\caption{Cyclic task sequence of a station (adapted from \cite{Scholl2013})}
	\vspace{2mm}
	\includegraphics[width=0.8\textwidth]{images/IntroForwBackSetupEx.eps}
	\label{fig:intro:forwBackDifference}
\end{figure}

To see how these setups are differentiated,
Figure \ref{fig:intro:forwBackDifference} depicts the sequence of tasks
$T_i,\:T_j,\:T_k$ which are performed in a cyclic manner on consecutive
products.
The execution of tasks proceeds chronologically from right to left,
\ie $T_1$ is the first task performed in cycle $p$.
Forward setups arise between consecutive tasks operating
on the same product.
A backward setup cost is incurred between the last
task of cycle $p$ and the first task of the next cycle, $p+1$.
Each cycle begins at time zero with the execution of task $T_i$
taking $t_i$ time units to process.
Then after performing the forward setups $\phi_{ij}$ and $\phi_{jk}$
together with processing times $t_j$ and $t_k$,
the station operator must perform the necessary backward
setup operation before moving to the next
workpiece along the line.
Thus, when including this backward setup time $\beta_{ki}$,
the following condition must be satisfied to have a
feasible cycle time
$t_i+\phi_{ij}+t_j+\phi_{jk}+t_k+\beta_{ki} \leq c$.
In Figure \ref{fig:intro:forwBackDifference}, the gray arrow  from $T_k$ to $T_i$ implies
the cyclic nature of the sequence.

% Example \ref{ex:intro:simpleSetup} alters the previous
% example into an instance of the \sua{2}.

\begin{example}\label{ex:intro:simpleSetup}
	Again consider the instance from Example \ref{ex:intro:simple},
	but now with sequence-dependent setup times
	between tasks.
	The arrays of forward and backward setup costs
	can be found in Tables \ref{tab:intro:forwSetupTimes}
	and \ref{tab:intro:backSetupTimes} respectively.
	Note, that some of the entries in these arrays are
	omitted as the corresponding sequence of tasks is not
	possible due to the precedence relations or
	logical restrictions.

	The previous optimal solution is amended
	to include the required setup costs and is given in Figure
	\ref{fig:intro:exSchedSetupFeas}.
	Note that the setup cost of any task to itself is
	defined as zero.
	% This results in a large forward setup, $\phi_{23}$,
	% occurring on station $S_2$.
	To reach the optimal solution we must now consider
	the sequence of tasks within each station.

	By moving task $T_3$ to station $S_3$ and considering
	the sequencing of $T_3$ and $T_4$, we find the optimal
	solution to this problem, given in Figure \ref{fig:intro:exSchedSetupOpt}.
	The optimal cycle time is 13 in this case.	\qed
\end{example}

\begin{table}[tpb]
	\centering
	\begin{minipage}{0.45\textwidth}
		\def\arraystretch{1.1}
		\centering
		\caption{Forward setup times}
		\vspace{2mm}
		\begin{tabular}{lllll}
			\toprule
			$\phi$ & $T_1$ & $T_2$ & $T_3$ & $T_4$ \\\midrule\midrule
			$T_1$ & -- & 3 & 3 & 3 \\
			$T_2$ & -- & -- & 2 & 3 \\
			$T_3$ & -- & -- & -- & 1 \\
			$T_4$ & -- & -- & 2 & -- \\
			\bottomrule
		\end{tabular}
		\label{tab:intro:forwSetupTimes}
	\end{minipage}
	\hfill
	\begin{minipage}{0.45\textwidth}
		\def\arraystretch{1.1}
		\centering
		\caption{Backward setup times}
		\vspace{2mm}
		\begin{tabular}{lllll}
			\toprule
			$\beta$ & $T_1$ & $T_2$ & $T_3$ & $T_4$ \\\midrule\midrule
			$T_1$ & 0 & -- & -- & -- \\
			$T_2$ & 3 & 0 & -- & -- \\
			$T_3$ & 3 & 5 & 0 & 3 \\
			$T_4$ & 3 & 3 & 1 & 0 \\
			\bottomrule
		\end{tabular}
		\label{tab:intro:backSetupTimes}
	\end{minipage}
\end{table}

\begin{figure}[tpb]
	\centering
	\begin{minipage}{0.47\textwidth}
		\centering
		\caption{Feasible solution with setup}
		\vspace{2mm}
		\includegraphics[width=\linewidth]{images/exSimpleSetupFeas.eps}
		\label{fig:intro:exSchedSetupFeas}
	\end{minipage}
	\hfill
	\begin{minipage}{0.47\textwidth}
		\centering
		\caption{Optimal solution with setup}
		\vspace{2mm}
		\includegraphics[width=\linewidth]{images/exSimpleSetupOpt.eps}
		\label{fig:intro:exSchedSetupOpt}
	\end{minipage}
\end{figure}

\section{Motivation}
\label{sec:intro:motiv}
In this section we provide motivation for why the \albp{}
with setup costs
is an important problem to be solved and why the approach we
chose to utilize is well-suited to the problem.

\subsection{Why the SUALBSP?}
Assembly lines were originally designed for the production
of a single type of product in high volumes.
Although, assembly lines able to produce only a single
product are not suitable for consumer-centric markets,
where there is a need to tailor the products
more closely to the user's needs.
Sequence-dependent setup times are commonly considered
in job shop scheduling problems and a range of other
production problems (\cf \cite{Allahverdi2008a,Allahverdi2008b,Allahverdi2015}).
In those problems, the setup times arise because the products
are typically assumed to be highly diverse,
leading to additional configuration effort being required
between any two tasks.
In the context of operational planning of assembly lines,
sequence-dependent setup times have been sparsely
considered by the literature.
\authciteb{Scholl2013} list a number of possible
practical settings where setup times between tasks should not
be taken as negligible. Some include the following:
\begin{itemize}
	\item In automotive assembly lines where large workpieces (vehicles)
	need to be constructed, work can be performed at numerous
	mounting positions on the item.
	The size of the car body can result in walking distances
	between mounting positions being non-trivial.
	Thus, the preparation time (setup) can
	be a critical aspect when sequencing the required tasks.
	\item Fixed material containers can be placed along the moving
	conveyor system to allow workers to collect parts or tools needed.
	Further to walking times between these containers and the workpiece,
	the time taken to retrieve what is required from the container
	must also be accounted for.
	At a major German car manufacturer, these times contributed $10-15\%$
	of the total cycle time \cite{Scholl2013}.
	\item Specific tools are often required to perform a task.
	If consecutive tasks need different tools for their
	execution a tool-change between tasks will be necessary.
	Robotic assemblers used to perform the tasks of a station
	can be built to be highly flexible in the types of tasks
	they can perform.
	However, this flexibility can mean that numerous tool changes
	are required to switch from one task to another
	in a sequence.
	Thus, these sequence-dependent setup costs from tool-changes
	are typical in robotic assembly lines.
\end{itemize}
Consequently, accurately modelling the setup costs incurred
can be an important aspect when balancing an assembly line.

In practice, setup times are accounted for in simpler ways, such as
using an approximation or incorporating the setup time
into the task's processing time.
\citeauthor{Scholl2013} note that by using such approximations
``the planning team need to strike a fragile balance between an underestimation
of setups, which lead to infeasible line balances, and an overestimation
of setups, which leads to an allocation of excessive resources.''
The procedure used to estimate the cost of setups
can be time consuming and is prone to getting stuck in sub-optimal solutions.

For these reasons we feel that further research into the
effect of setup times on the \albp{} is of interest to the
academic community and the manufacturing industry.

\subsection{Why Benders Decomposition?}
Benders decomposition is a well-known approach
to optimization problems.
It is naturally suited
to problems where the decisions can be separated into
two distinct sets.
Advances in the past few decades to the theory of
this method has broadened its applicability to a
wider array of problems.
In our case, a solution to the \sua{2} can naturally
be divided into the assignment portion, which
assigns each task to a station, and the scheduling
portion, which decides on the time that each task
begins execution within a station.

We can view this separation of concerns in the following
way.
The manager of an assembly line needs a new item
to be put into production and so assigns the required
set of tasks to the stations along the line.
When calculating the line's cycle time the manager only estimates the setup times
within each station.
Each station operator must then decide if a possible sequencing
of their assigned tasks exists which can respect the manager's cycle time;
or if a precedence-feasible sequence exists at all.
% The worker at each station must then decide if there exists
% a possible sequencing of their assigned tasks which can
% respect the manager's cycle time; or if there a
% precedence-feasible sequence exists at all.
If at least one worker cannot find a feasible task-sequence
that respects the manager's cycle time
estimate, then a revision
may need to be made to the assignment of tasks.
The give-and-take between the assignment and scheduling portions
continues until a mutually satisfactory solution is found.
This informal process of communicating information between
the station workers and the assembly line manager could be time
consuming and not guarantee optimality.

The informal feedback loop between the two halves of the
decision problem suggests that Benders decomposition could be
employed to mimic this process.
The decision making process can naturally be divided
into a master problem, which assigns the tasks
to the stations along the line,
and several independent scheduling problems.
In total, there is a sub-problem
for each of the $m$ stations, which needs to
decide the exact execution time of all tasks
that station has been assigned.
Each sub-problem is similar to the asymmetric
Travelling Salesperson Problem (TSP)
with some forbidden paths due to the precedence
relations.
With this interpretation, the tasks are viewed as the
cities and distances between them are the setup times.
To complete the iterative loop, the sub-problems
relay their information back to the master as
Benders cuts and the assignment problem is re-optimized.
In the words of J. N. Hooker,
``the Benders cuts added to the master problem are
the mathematical equivalent of telephone calls''
from the station workers to the line manager \cite{Hooker2007}.

The reader may be familiar with the classical version
of the Benders decomposition method due to the
work of \authciteb{Benders1962} and \authciteb{Geoffrion1972},
however for the \sua{}, this approach is inappropriate.
Due to the sub-problems being highly-combinatorial discrete scheduling
problems, formulating them as a linear or non-linear program
will not be practical.
We instead explore how the more general method of \emph{logic-based}
Benders decomposition can be used to solve the \sua{2}.

By employing this approach we are able to exploit the comparative
advantages of multiple solving technologies to tackle each
portion of the problem.
Mixed-Integer Programs (MIPs) are well-suited to the assignment
problem of the master and Constraint Programming (CP), from
the Computer Science discipline, is an effective solving technology
for scheduling problems.

\section{Problem Definition}
\label{sec:intro:probDef}
Here we present the core notation that will be used for the
remainder of the thesis.
Along the conveyor belt of the assembly line
there are \emph{work stations} $K=\{1,2,\ldots,m\}$.
Workpieces (or products/items) move down the
conveyor belt from station to station.
A workpiece remains at each station for
one cycle, which has a duration given by the \emph{cycle time}.
The objective of the \albp{} is to optimally partition
the total work required among the stations with respect to
a performance measure.

The work required to complete a single piece is separated into
a set of \emph{non-preemptive tasks} $V=\{1,2,\ldots,n\}$,
each with a discrete processing time $t_i$.
Physical and technical conditions impose a set of
\emph{precedence relations} $E\subseteq V\times V$,
which prevent some non-allowed task orderings from occurring.
If we consider the graph $G=(V,E)$ with tasks as the vertex set
and directed edges defined by the precedence relations,
then $G$ is a directed acyclic graph (DAG).
Without loss of generality, we may assume that the vertices
are numbered topologically so that the following holds:
$(i,j) \notin E$ if $i>j$.

Between each pair of tasks, discrete
\emph{forward} and \emph{backward setup} times
are predefined.
We denote the forward setup time between $i$ and $j$
by $\phi_{ij}$ and the backward setup time
by $\beta_{ij}$.
When $i=j$, the forward setup time is undefined
as a task can never follow itself in a station's
forward work load.
The backward setup $\beta_{ii}$, \ie the setup cost
of a task to itself, is defined as zero.

To fully specify a feasible solution to the \sua{2},
one must give the following:
an assignment of tasks to the stations
and the execution time window of each task within
its assigned station.
From this specification, the cycle time is calculated by
the total processing time of the station with the
largest workload.


\section{SEDflow} \label{sec:sedflow}
In this section, we present \sedflow, which applies ANPE to galaxy SED modeling
for a scalable and accelerated approach.
For our SED model, we use the state-of-the-art PROVABGS model from
\cite{hahn2022}.
Although many SED models have been recently used in the
literature~(\emph{e.g.} {\sc Bagpipes}, \citealt{carnall2018};
{\sc Prospector}, \citealt{leja2017, johnson2021}), we choose PROVABGS because
it will be used to analyze >10 million galaxy spectrophotometry measured by the
DESI Bright Galaxy Survey~(Hahn~\etal~in prep.).
Below, we describe the PROVABGS model, the construction of the
\sedflow~training data using PROVABGS, and the training procedure for \sedflow.

% section explaining specific SED set up
\subsection{SED Modeling: PROVABGS} \label{sec:provabgs}
We use the state-of-the-art SPS model of the
PROVABGS~\citep{hahn2022}.
The SED of a galaxy is modeled as a composite of stellar populations defined by
stellar evolution theory (in the form of isochrones, stellar spectral
libraries, and an initial mass function) and its star
formation and chemical enrichment histories (SFH and ZH), attenuated by
dust~\citep[see][for a review]{walcher2011, conroy2013}.
The PROVABGS model, in particular, utilizes a non-parametric SFH with a
starburst, a non-parametric ZH that varies with time, and a flexible dust
attenuation prescription.

% highlight advantages of provabgs
The SFH has two components: one based on non-negative matrix factorization
(NMF) bases and the other, a starburst component.
The SFH contribution from the NMF component is a linear combination of four NMF
SFH basis functions, derived from performing NMF~\citep{lee1999, cichocki2009,
fevotte2011} on SFHs of galaxies in the Illustris cosmological hydrodynamical
simulation~\citep{vogelsberger2014, genel2014, nelson2015}.
The NMF SFH prescription provides a compact and flexible representation of the
SFH.
The second starburst component consists of a single stellar population (SSP)
and adds stochasticity to the SFH.

The ZH is similar defined using two NMF bases dervied from Illustris.
This ZH prescription enables us to flexibly model a wide range of ZHs and,
unlike most SED models, it does not assume constant metallicity over time,
which can significantly bias inferred galaxy properties~\citep{thorne2021}.
The stellar evolution theory is based on Flexible Stellar Population
Synthesis~\citep[FSPS;][]{conroy2009, conroy2010c} with the MIST
isochrones~\citep{paxton2011, paxton2013, paxton2015, choi2016, dotter2016},
the \cite{chabrier2003} initial mass function (IMF), and a combination of the
MILES~\citep{sanchez-blazquez2006} and BaSeL~\citep{lejeune1997, lejeune1998,
westera2002} libraries.
The SFH and ZH are binned into 43 logarithmically-space time and SSPs are
evalulated at each time bin using FSPS.
The SSPs are summed up to get the unattenuated rest-frame galaxy SED.

Lastly, PROVABGS attenuates the light from the composite stellar population
using the two component \cite{charlot2000} dust attenuation model with
diffuse-dust (ISM) and birth cloud (BC) components.
All SSPs are attenuated by the diffuse dust using the \cite{kriek2013}
attenuation curve.
Then, the BC component provides extra dust attenuation on SSPs younger than 100
Myr with young stars that are embedded in modecular clouds and HII regions.
In total the PROVABGS SED model has 12 parameters: stellar mass ($M_*$),
six SFH parameters ($\beta_1, \beta_2, \beta_3, \beta_4, t_{\rm burst}, f_{\rm
burst}$), two ZH parameters ($\gamma_1, \gamma2$), and three dust attenuation
parameters ($\tau_{\rm BC}, \tau_{\rm ISM}, n_{\rm dust}$).
Each PROVABGS model evaluation takes ${\sim}340$ ms.

\begin{figure}
\begin{center}
\includegraphics[width=0.85\textwidth]{figs/training.pdf}
    \caption{\label{fig:data}
    The distribution of SED model parameters, redshift, photometric
    magnitudes, and uncertainties of the data used to train \sedflow.
    We plot a subset of the parameters ($\log M_*$, $\beta_1$) and photometric
    bands for clarity.
    The training data was constructed by sampling SED model parameters from the
    prior and forward modeling optical photometry using the PROVABGS and noise
    models (Section~\ref{sec:sedflow}).
    For comparison, we present the distribution of redshift, magnitudes, and
    uncertainties for galaxies in the NSA catalog (blue).
    \emph{The training set encompasses the observations; thus, {\sc SEDflow}
    can be used to infer the posterior for the NSA galaxies}.
    }
\end{center}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% training data
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Training Data} \label{sec:training}
In this section, we describe how we construct the training data for
\sedflow~using the PROVABGS SED model.
First, we sample $N_{\rm train}$ model parameters from a prior: $\theta'\sim p(\theta)$.
We use the same priors as \cite{hahn2022}: uniform priors over $M_*,
t_{\rm burst}, f_{\rm burst}, \gamma_1, \gamma_2, \tau_{\rm BC}, \tau_{\rm ISM},
n_{\rm dust}$ with broad conservative ranges and Dirichlet prior over $\beta_1,
\beta_2, \beta_3, \beta_4$, chosen to normalize the NMF SFH.
For each $\theta'$, we also uniformly sample a redshift within the range of the
NSA: $z' \sim \mathcal{U}(0., 0.2)$.
Next, we forward model mock observables.
We calculate the rest-frame galaxy SED from PROVABGS and redshift it:
$F(\lambda;\theta', z)$.
Afterwards, we convolve $F$ with optical broadband filters, $R_X$, to generate
noiseless photometric fluxes:
\begin{equation}
    f_X(\theta', z') = \int F(\lambda;\theta', z') \, R_X(\lambda) \, {\rm d}\lambda
\end{equation}
The next step of the forward model is to apply noise.
We assign photometric uncertainties, $\sigma'_X$, by sampling an estimate of
the observed $p(\sigma_X | f_X)$ of NSA galaxies.
Then, we apply Gaussian noise
\begin{equation} \label{eq:noise}
    \hat{f}_X(\theta', z', \sigma'_x) = f_X(\theta', z') + n_X  \quad {\rm where}~n_X \sim \mathcal{N}(0, \sigma'_X)
\end{equation}
to derive the forward modeled photometric flux.

For our estimate of $p(\sigma_X | f_X)$, we use an empirical estimate based on
NSA photometry and measured uncertainties.
For each of the five optical bands, we separately estimate
\begin{equation}
    \hat{p}(\sigma_X | f_X) = \mathcal{N} \big( \mu_{\sigma_X}(f_X),
    \sigma_{\sigma_X}(f_X) \big)
\end{equation}
as a Gaussian in magnitude-space.
$\mu_{\sigma_X}$ and $\sigma_{\sigma_X}$ are the median and standard deviation
of $\sigma_X$ as a function of $f_X$ that we estimate by evaluating them in
$f_X$ bins and interpolating over the bins.
Any $\theta'$ that is assigned a negative $\sigma'_X$ is removed from our
training data.
We also remove any training data with $f_X(\theta')$ outside the range of NSA
photometry.

As SBI requires an accurate noise model, one might be concerned about the
simplicity of our model.
If the noise model is incorrect, estimates of the parameters $\theta$ would be
biased by an amount that is impossible to predict.
We therefore add the noise variances $\sigma_X$ to the conditioning variables
of the posterior model, \emph{i.e.} we train and evaluate
$p_\phi(\btheta\given\bfi{x},\{\sigma_X\})$.
This means we merely have to ensure that $\sigma'_X$ spans the observed
$\sigma_X$ values in order to have a posterior that is robust to our choice of
noise model.
We discuss this further in Section~\ref{sec:discuss}.

In total, we constuct $N_{\rm train} = 1,131,561$ sets of SED parameters,
redshift, photometric uncertainties, and mock NSA photometry.
In Figure~\ref{fig:data}, we present the distribution of the training data
$\{(\theta', z, \sigma'_X, \hat{f}_X) \}$ (black).
We include select SED model parameters ($\log M_*$, $\beta_1$), redshift,
photometry in the $g$ and $r$ bands, and photometric uncertainty in the $r$
band.
We also include the $(z, \sigma_X, f_X)$ distribution of NSA galaxies (blue),
for comparison.
The photometry and uncertainties are in magnitude-space.
The distribution of the training data spans the distribution of NSA galaxies.

\subsection{Training \sedflow} \label{sec:anpe_train}
For \sedflow, we use a MAF normalizing flow model (Section~\ref{sec:flow}) with
15 MADE blocks, each with 2 hidden layers and 500 hidden units.
In total, the model has 7,890,330 parameters, $\bphi$.
We determine this architecture through experimentation.
Our goal is to determine $\bphi$ of the MAF model,
$p_\phi(\btheta\given\bfi{x})$, so that it accurately estimates the
posterior probability distribution $p(\btheta\given\bfi{x})$.
$\btheta$ represent the SED parameters and $\bfi{x} = (f_X, \sigma_X, z)$.
We do this by minimizing the KL divergence between
$p_\phi(\btheta\given\bfi{x})$ and $p(\btheta\given\bfi{x})$:
$D_{\rm KL} (p\,||\,p_\phi)$.

In practice, we split the training data into a training and validation set with
a 90/10 split.
Afterwards, we maximize the total log likelihood
$\sum_i \log p_\phi(\btheta_i\given \bfi{x}_i)$ over training set, which is
equivalent to minimizing $D_{\rm KL} (p\,||\,p_\phi)$.
We use the {\sc Adam} optimizer~\citep{kingma2017} with a learning rate of $5\times10^{-4}$.
To prevent overfitting, we evaluate the total log likelihood on the validation
data at every training epoch and stop the training when the validation log
likelihood fails to increase after 20 epochs.
Training our model with a batch size of 50 takes roughly a day on a single 2.6
GHz Intel Skylake CPU.
Given our small batch size, we find similar training times when using CPUs or
GPUs.


% Filename: p2_set_and_functions@tikz_for_teachers.tex
% This code is part of LaTeX with Vim.
%
% Description: TikZ for teachers is free book about TikZ and Sage.
%
% Created: 30.03.12 08:25:52 PM
% Last Change: 30.03.12 08:26:12 PM
%
% Author: Raniere Gaia Costa da Silva, r.gaia.cs@gmail.com
% Organization:
%
% Copyright (c) 2010, 2011, 2012, Raniere Gaia Costa da Silva. All rights
% reserved.
%
% This file is license under the terms of a Creative Commons Attribution
% 3.0 Unported License, or (at your option) any later version. More details
% at <http://creativecommons.org/licenses/by/3.0/>.
\chapter{Conjuntos e Fun\c{c}\~{o}es}

\section{No\c{c}\~{o}es de L\'{o}gica}

\section{Conjuntos}

\example{codes/sets01@tikz_for_teachers}

\example{codes/sets02@tikz_for_teachers}

\example{codes/sets03@tikz_for_teachers}

\example{codes/sets04@tikz_for_teachers}

\section{Conjuntos Num\'{e}ricos}

\section{Rela\c{c}\~{o}es}

\section{Fun\c{c}\~{o}es}


\documentclass[bigger]{beamer}

\input{header-beam} % change to header-handout for handouts

% ====================
\title[Lecture 19]{Logic I F13 Lecture 19}
\date{November 21, 2013}
% ====================

\include{header}

\setlength{\fitchprfwidth}{14em}

\section{Numerical Quantification}

\subsec{Numerical Quantification}{

\bit
\item Cardinal numbers can be determiners:
\bit
\item \emph{Three cubes} are left of \emph{two dodecahedra}
\eit
\item Not always clear if ``three cubes'' means \emph{exactly} or \emph{at least} three
\item We'll assume the latter---the ``exactly'' is implicated, not implied
\bit
\item Do you have two dollars? Yes, I have two dollars. (Uncontroversially true even if you have more than 2\$)
\item How much money do you have? I have two dollars. (True but misleading if you have more.)
\eit
\item FOL can express all of:
\bit
\item \emph{At least $n$} cubes are \dots
\item \emph{Exactly $n$} cubes are \dots
\item \emph{At most $n$} cubes are \dots
\eit
\eit

}

\subsec{At least $n$}{

\bit
\item At least 1 cube is small:
\[
\exists x(Cube(x) \land Small(x))
\]
\item At least 2 cubes are small:
\[
\exists x\exists y(x \neq y \land Cube(x) \land Small(x) \land Cube(y) \land Small(y))
\]
\eit
}

\subsec{$\exists^{\ge n}x\, A(x)$}{

\bit
\item $\exists^{\ge n}x\, A(x)$:
\begin{align*}
\exists x_1\dots\exists x_n(x_1 \neq x_2 \land x_1 \neq x_3 \land \dots \land x_1 \neq x_n & {} \land {}\\
x_2 \neq x_3 \land \dots \land x_2 \neq x_n & {} \land {}\\
\ddots\qquad &\\
x_{n-1} \neq x_n & {} \land {}\\
 A(x_1) \land \dots \land A(x_n))
\end{align*}
\eit
}

\subsec{$\exists^{\ge n}x\, A(x)$}{

\bit
\item NB: must state that \emph{every pair} of variables is different, e.g.,
\[\sf\exists x_1\exists x_2\exists x_3(x_1 \neq x_2 \land x_2 \neq x_3 \land Cube(x_1) \land Cube(x_2) \land Cube(x_3))\]
only says ``There are at least two cubes''!
\item At least $n$ As are Bs:\[
\exists^{\ge n} x(A(x) \land B(x))
\]
\eit
}

\subsec{Exactly One}{

\bit
\item There is exactly one cube:
\[
\exists x(Cube(x) \land \lnot \exists y(Cube(y) \land x \neq y))
\]
\item This is equivalent to:
\[
\exists x(Cube(x) \land \forall y(Cube(y) \to x = y))
\]
\item In general: ``x has property A \emph{uniquely}'':
\begin{align*}
A(x) \land & \forall y(A(y) \to x = y)\\
\text{or just:\qquad} & \forall y(A(y) \liff x = y)
\end{align*}
\eit
}

\subsec{$\exists^{=n} x\, A(x)$}{

\bit
\item There are exactly $n$ things that satisfy A:
\begin{align*}
\exists x_1\dots\exists x_n(x_1 \neq x_2 \land x_1 \neq x_3 \land \dots \land x_1 \neq x_n & {} \land {}\\
x_2 \neq x_3 \land \dots \land x_2 \neq x_n & {} \land {}\\
\ddots\qquad &\\
x_{n-1} \neq x_n & {} \land {}\\
\only<1>{ A(x_1) \land \dots \land A(x_n) &{} \land {}}\\
\forall y(A(y) \only<1>{\to}\only<2>{\leftrightarrow} (y = x_1 \lor \dots \lor y=x_n)))
\end{align*}
\item Exactly $n$ As are Bs:
\[
\exists^{=n}x (A(x) \land B(x))
\]
\eit

}

\subsec{At most $n$}{

\bit
\item There are \emph{at most $n$} As $\IFF$ There aren't \emph{at least $n+1$} As
\[
\exists^{\emph{\le n}}x\, A(x) \IFF \emph{\lnot}\exists^{\emph{\ge(n+1)}}x\, A(x)
\]
\item $\lnot\exists^{\ge(n+1)}x\, A(x)$ is equivalent to:
\begin{align*}
\forall x_1\dots\forall x_{n+1}((A(x_1) \land \dots \land A(x_{n+1})) & {} \to {}\\
(x_1 = x_2 \lor x_1 = x_3 \lor \dots \lor x_1 = x_{n+1} & {} \lor {}\\
x_2 = x_3 \lor \dots \lor x_2 = x_{n+1} & {} \lor {}\\
\ddots\qquad &\\
x_n = x_{n+1} & ))
\end{align*}
\eit
}

\subsec{Proofs With Numerical Claims}{

\fitchprf{
\pline{\exists x\, P(x)}\\
\pline{\forall x\forall y ((P(x) \land P(y)) \to x = y)}}{
\pline{\exists x(P(x) \land \forall y(P(y) \to x = y))}
}
}

\section{``The'', ``Both'', ``Neither''}

\subsec{Definite Descriptions}{

\bit
\item Definite description: \emph{the so-and-so}
\item Russell's analysis of definite description: to say\\
\bit
\item ``The A is B''\\
\eit
is to say:
\bit
\item There is a unique A
\item It is B
\eit
\item In FOL:
\[
\exists x(A(x) \land \forall y(A(y) \to x = y) \land B(x))
\]
\item or more succinctly:
\[
\exists x(\forall y(A(y) \liff x = y) \land B(x))
\]
\eit

}

\subsec{The vs. Exactly One}{

\bit \item Compare:
\ben
\item The cube is small
\[\exists x(Cube(x) \land \forall y(Cube(y) \to x = y) \land Small(x))\]
\item There is exactly one small cube
\[\exists x(Cube(x) \land \forall y((Cube(y) \emph{{}\land Small(y)}) \to x = y) \land Small(x))\]
\een
\item (2) can be true without (1), but not vice versa
\item So (2) is a FO consequence of (1), but not vice versa
\eit
}

\subsec{Strawson's Analysis}{

\bit
\item According to Russell, ``The cube is small'' is false if there is no cube, or if there is more than one
\item Strawson disagrees
\item ``There is a unique cube'' is not part of what is \emph{said}, but is only \emph{presupposed}
\eit

}

\subsec{``Both'' and ``Neither''}{

\bit
\item ``Both cubes are small'': There are exactly 2 cubes, and each of them is small:
\begin{align*}
\exists x\exists y(x \neq y \land Cube(x) \land Cube(y) &{} \land {}\\
\forall z(Cube(z) \to (z = x \lor z = y)) & {}\land {}\\
\forall z(Cube(z) \to Small(z))&)
\end{align*}
\item NB: ``Both cubes are small'' implies ``There are exactly two small cubes'', but not vice versa!
\item ``Neither cube is small'': There are exactly 2 cubes, and none of them are small:
\begin{align*}
\exists x\exists y(x \neq y \land Cube(x) \land Cube(y) & {}\land {}\\
\forall z(Cube(z) \to (z = x \lor z = y)) &{} \land {}\\
\forall z(Cube(z) \to \lnot Small(z))&)
\end{align*}
\eit

}

\section{Truth-Functional Completeness}

\subsec{Truth Functions}{

\begin{block}{Definition}

An ($n$-place) \emph{truth function} $t$ is a mapping of $n$-tuples of \T{} and \F\ to either \T\ or \F.
\medskip

$n$-place truth functions correspond to truth tables of sentence with $n$ atomic sentences.
\medskip

A sentence $S$ containing the atomic sentences $A_1$, \dots, $A_n$
\emph{expresses} the truth function $t$ iff the truth value of $S$ on
the truth-value assignment which assigns $v_i$ to $A_i$ is $t(v_1,
\dots, v_n)$.
\medskip

An $n$-place truth function is \emph{expressible} if there is a
sentence containing atomic sentences $A_1$, \dots, $A_n$ that
expresses it.

\end{block}
}

\subsec{Examples}{

\begin{tabular}{cc@{\qquad}cc}
$\begin{array}{cc|c}
&& t_\land\\ \hline
\T & \T & \T \\
\T & \F & \F \\
\F & \T & \F \\
\F & \F & \F
\end{array}$
&
$A_1 \land A_2$ &
$\begin{array}{cc|c}
&& t_\lor \\ \hline
\T & \T & \T \\
\T & \F & \T \\
\F & \T & \T \\
\F & \F & \F
\end{array}$
& $A_1 \lor A_2$
\end{tabular}
}

\subsec{Examples}{

\begin{tabular}{cc@{\qquad}cc}
$\begin{array}{cc|c}
& & t_1\\ \hline
\T & \T & \T \\
\T & \F & \T \\
\F & \T & \F \\
\F & \F & \F
\end{array}$
& $A_1$ &
$\begin{array}{cc|c}
& & t_{XOR}\\ \hline
\T & \T & \F \\
\T & \F & \T \\
\F & \T & \T \\
\F & \F & \F
\end{array}$
&
$(A_1 \lor A_2) \land \lnot(A_1 \land A_2)$ \\ \ \\
$\begin{array}{cc|c}
& & t_2\\ \hline
\T & \T & \T \\
\T & \F & \F \\
\F & \T & \T \\
\F & \F & \F
\end{array}$
& $A_2$
\end{tabular}
}

\subsec{Truth-Functional Completeness}{

\begin{block}{Definition}

A set of connectives is \emph{truth-functionally complete} if every
truth function is expressible by a sentence containing only those
connectives.

\end{block}

}

\subsec{$\{\land, \lor\}$ Not Truth-Functionally Complete}{

\bit
\item $\{\land, \lor\}$ is not truth-functionally complete
\item Remember: To be truth functionally complete, \emph{every} truth
  function would have to be expressible using only $\land$ and $\lor$
\item Which 2-place truth-functions can be expressed using $\land$ and
  $\lor$?
\item Not this one:
\[
\begin{array}{cc|c}
\T & \T & \F \\
\T & \F & \T \\
\F & \T & \T \\
\F & \F & \F
\end{array}
\]
  \eit

}

\subsec{Course Evaluation}{

\bit
\item Please fill out TWO TA evaluations, one for Jared and one for Dan.
\item More TopHat instant feedback questions? Or is TopHat not worth the trouble?
\item Three tests good, or would you prefer a midterm
and a final after classes end?
\item Timing of homeworks---prefer Friday or Monday?
\item PASS sessions: did you go? Was it useful?
\eit
}
\end{document}

\subsec{$\{\land, \lor, \lnot\}$ is Truth-functionally Complete}{
\[
\begin{array}{lll|l@{\qquad}l}
A_1 & A_2 & A_3 & t_\mathrm{ODD} & S \\
\hline
\T & \T & \T & \T & \uncover<2->{(A_1 \land A_2 \land A_3)} \uncover<6->{{}\lor}\\
\T & \T & \F & \F \\
\T & \F & \T & \F \\
\T & \F & \F & \T & \uncover<3->{(A_1 \land \lnot A_2 \land \lnot A_3)} \uncover<6->{{}\lor}\\
\F & \T & \T & \F \\
\F & \T & \F & \T & \uncover<4->{(\lnot A_1 \land A_2 \land \lnot A_3)} \uncover<6->{{}\lor}\\
\F & \F & \T & \T & \uncover<5->{(\lnot A_1 \land \lnot A_2 \land A_3)} \uncover<6->{{}\lor}\\
\F & \F & \F & \F
\end{array}\]
}

\subsec{$\{\land, \lor, \lnot\}$ is Truth-Functionally Complete}{

\bit
\item $\lnot A_1 \land A_2 \land \lnot A_3$ is true in, and only in line \F\ \T\ \F
\item Combine using $\lor$: make $S$ true in all (and only) the lines where it is supposed to be true
\item NB: Procedure results in DNF!
\eit

}

\subsec{The ``neither\dots nor \dots'' connective: $\downarrow$}{

\[
\begin{array}{cc|c}
P & Q & (P \downarrow Q)\\
\hline
\T & \T & \F\\
\T & \F & \F\\
\F & \T & \F\\
\F & \F & \T
\end{array}
\]

}

\subsec{$\{\downarrow\}$ is Truth-Functionally Complete}{

\bit
\item Already know that $\{\lnot, \land, \lor\}$ is truth-functionally
  complete, i.e.,
\item Every truth-function can be expressed using only $\lor$, $\land$, $\lnot$
\item To show $\downarrow$ is truth-functionally complete, suffices to
  show that \emph{every sentence containing only $\lnot$, $\lor$, $\land$ is
  tautologically equivalent to one containing only $\downarrow$}
\item For that, it suffices to show that any negated sentence,
  conjunction, disjunction, can be expressed using only $\downarrow$
\eit

}

\subsec{Expressing $\lnot$ Using $\downarrow$}{

\begin{columns}
\begin{column}{3cm}
\[\begin{array}{cc|c}
P & Q & (P \downarrow Q)\\
\hline
\T & \T & \F\\
\T & \F & \F\\
\F & \T & \F\\
\F & \F & \T
\end{array}\]
\end{column}
\begin{column}{7cm}
\bits
\item Note how $P \downarrow Q$ is \F{} in the first line and \T{} in the last (when $P$ and $Q$ have same truth value)
\item So $P \downarrow P$ is \F{} if $P$ is \T, and \T{} if $P$ is \F, i.e., \[
\lnot P \Leftrightarrow (P \downarrow P)
\]
\eit
\end{column}
\end{columns}

}

\subsec{Expressing $\lor$ Using $\downarrow$}{

\begin{columns}
\begin{column}{3cm}
\[\begin{array}{cc|c}
P & Q & (P \downarrow Q)\\
\hline
\T & \T & \F\\
\T & \F & \F\\
\F & \T & \F\\
\F & \F & \T
\end{array}\]
\end{column}
\begin{column}{7cm}
\bits
\item $P \downarrow Q$ is the ``neither \dots nor'' connective, which can also
be expressed as $\lnot(P \lor Q)$, i.e.,
\[ \lnot(P \lor Q) \Leftrightarrow P \downarrow Q \]
\item Negate both sides:
\[
P \lor Q \Leftrightarrow \lnot(P \downarrow Q)
\]
\item Apply what we figured out in last slide:
\[
P \lor Q \Leftrightarrow (P \downarrow Q)\downarrow(P \downarrow Q)
\]
\eit
\end{column}
\end{columns}

}

\subsec{Expressing $\land$ Using $\downarrow$}{

\begin{columns}
\begin{column}{3cm}
\[\begin{array}{cc|c}
P & Q & (P \downarrow Q)\\
\hline
\T & \T & \F\\
\T & \F & \F\\
\F & \T & \F\\
\F & \F & \T
\end{array}\]
\end{column}
\begin{column}{7cm}
\bits
\item $P \downarrow Q$ is the ``neither \dots nor'' connective, which can also
be expressed as $\lnot P \land \lnot Q$, i.e.,
\[ (\lnot P \land \lnot Q) \Leftrightarrow P \downarrow Q \]
\item Equivalence holds for \emph{all sentences} $P$, $Q$, so also if we replace $P$ by $\lnot R$ and $Q$ by $\lnot S$:
\[
\lnot\lnot R \land \lnot\lnot S \Leftrightarrow (\lnot R \downarrow \lnot S)
\]
\item Delete $\lnot\lnot$'s, and express $\lnot$ using $\downarrow$:
\[
R \land S \Leftrightarrow (R \downarrow R)\downarrow(S \downarrow S)
\]
\eit
\end{column}
\end{columns}

}

\subsec{Truth-Functionally Complete Sets of Connectives}{

\bit
\item De Morgan's Law: $\land$ can be expressed by $\lor$ and $\lnot$
\item Similarly: $\lor$ can be expressed by $\land$, $\lnot$
\item So $\{\lor, \lnot\}$ and $\{\land, \lnot\}$ are
  truth-functionally complete
\item $\{\to, \bot\}$ is truth-functionally complete (HW)
\item $\{\to, \lnot\}$ is truth-functionally complete
\item No other sets of connectives that don't contain one of these sets are complete
\item ``Neither \dots nor'' is truth-functionally complete by itself
\item ``Not both'' connective is truth-functionally complete by itself (HW)
\item No other 2-place connectives are complete by themselves
\eit

}

\end{document}


\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{a4}
\usepackage[T1]{fontenc}
\usepackage[cyr]{aeguill}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{authblk}
\usepackage{listings}
\usepackage{subfigure}
\usepackage{float}
\usepackage{geometry}
\usepackage[a4paper]{geometry} % Marges plus larges
\geometry{hmargin=2.5cm,vmargin=1.65cm}
\usepackage[sectionbib]{chapterbib}

\title{TS114 Signal processing \\ MICA Project}

\author{ABIED Imad \\ Email: imad.abied@bordeaux-inp.fr

\and AHALLI Mohamed \\ Email: mohamed.ahalli@bordeaux-inp.fr

\and BIGI Mohamed \\ Email:  mohamed.bigi@bordeaux-inp.fr }

\date{11/05/2021}

\sloppy

\begin{document}
\maketitle
\begin{center}
    ENSEIRB-MATMECA
\end{center}

\tableofcontents

\newpage

\section{Introduction}

The goal of this project was to analyse signals obtained from measuring heart electrical activity. Such a signal is called electrocardiogram, ECG for short.
ECG respects the pattern described in figure 1. Detecting this pattern consists of identifying the four characteristic points P, Q, R,S and T in each R-R interval from a real ECG signal, as shown in figure 2. The algorithm for this task is presented in the technical part below.

\begin{figure}[htbp]

\centerline{\includegraphics[scale = 0.5]{figure_1.png}}
\caption{ECG pattern}

\centerline{\includegraphics[scale = 0.3]{figure_2.png}}
\caption{Real ECG signal}

\end{figure}

Furthermore, cardiac pathologies are defined from the PQRST mathematical properties. Therefore, the identification of cardiac pathologies can be automatic. Algorithms for automatic identification are discussed in the technical part of this report.

PQRST mathematical properties can change over time for different reasons. For example, the heart rate increases after doing a physical activity. This is why the spectrogram was used during the project. The spectrogram was employed to understand how ECG characteristics change in the course of time, enabling us to choose the right time interval to apply desired algorithms. For that reason the first section of the technical part is dedicated to spectrograms.

\section{Data visualization (spectrogram) }
Spectrograms are used to locally represent the spectrum of a signal. As the spectrum is a statistical measurement, it’s necessary to define the term “locally” by a window of a certain length noted N. The larger the window, the greater frequency precision is obtained at the price of losing time accuracy. Conversely, The shorter window, the greater time accuracy is obtained at the cost of losing frequency precision. This is because there is more data to process in order to establish the spectre, which is a statistical measure. Nevertheless, the spectrum computed by a large window is not “very local".

\begin{figure}[H]

\begin{subfigure}

\centerline{\includegraphics[scale = 0.6]{figure_3_large_window.png}}
\caption{Spectrogram with a large window.}
\end{subfigure}

\begin{subfigure}

\centerline{\includegraphics[scale = 0.2]{figure_4_short_window.png}}
\caption{Spectrogram with a short window.}
\end{subfigure}

\end{figure}

This duality between frequency precision and time accuracy can be shown by the example ./src/duality\_time\_frequence.m. In this script the spectrogram of the same signal is plotted using a large window in figure 3 and shorter one in figure 4.

In figure 3, the yellow line is thin compared to the one in figure 3, facilitating to us the reading of frequency value. On the other hand, it’s very difficult to read the transition at 0.25s in comparison with figure 4.

\section{QRS complex Detection}

The QRS complex is going to be detected using the Pan and Tompkins algorithm.The first step of this algorithm eliminates the interference of T and P waves with the QRS complex. A well designed filter is proposed by Pan and Tompkins for this task. Its transfer function is given by :

\begin{equation}
    H(z) = \frac{(1-z^{-6})^{2}}{(1-z^{-1})^{2}}\times\frac{(-1 + 32z^{-16} -32z^{-17} +z^{-32})}{(1-z^{-1})}
\end{equation}

Observing the fact that the R wave is very sharp,the derivative of the ECG signal is going to take big absolute values at the QRS complex.Pan and Tompkins suggests this time a filter with a transfer function equal to :

\begin{equation}
    H(z) = \frac{1}{8T_s}\times(-z^{-2} - 2z^{-1} +2z +z^{2})
\end{equation}

As a consequence, if a well chosen window is used for moving window integration,the result will be a signal which takes big values for every QRS complex.

In order to enhance the QRS complex domain, a thresholding operation is applied with a threshold equal to the mean of ECG signal after integration.

At this point, it’s certain that the computed maximum for every QRS domain corresponds to the unique pick called R.

Once the R pick is detected, Q and S are detected by searching consecutively the first minimum on the left and the first minimum on the right.

All those steps are applied to an ECG signal. The result at every step are plotted in figure 5.

\begin{figure}[htbp]

\centerline{\includegraphics[scale = 0.4]{figure_5.png}}
\caption{ECG signal in each step.}

\end{figure}

\begin{tabular}{ |p{5cm}|p{5cm}|p{5cm}|  }
\hline
\multicolumn{3}{|c|}{Used filters analyse} \\
\hline
band-pass filter &  high-pass filter & five-point differentiation filter\\
\hline
Nature : band-pass & Nature : high-pass & Nature : differentiation\\
Type : Infinite Impulse Response & Type : Infinite Impulse Response & Type : Finite Impulse Response \\
Causal : Yes & Causal : Yes   & Causal : No \\
Group delay : 5 samples &Group delay : 16.49 samples & Group delay : 0 \\
Linear phase : Yes & Linear phase : Yes & Linear phase : Yes\\
\hline
\end{tabular}

\newpage

\section{P and T wave detection}
\subsection{About P and T waves}

Generally, P and T waves in an  ECG (electrocardiogram) signal are lower in amplitude compared to a QRS complex, and, they are contaminated with noise from various sources. These factors make the detection of P and T waves within an ECG a challenging task. Unlike a P wave, T waves are slightly asymmetrical,  the peak of the wave is a little closer to its end than to its beginning.
\subsection{P and T waves detection method}
T waves are considered to be the highest peak between the first R peak and 0.7 times the R-R interval. While P waves have the highest peak in the remaining of the interval, as shown in the figures below.

\begin{figure}[htbp]

\centerline{\includegraphics[scale=0.5]{T_70.png}}
\caption{Highest peak on 70\% of R-R interval.}

\end{figure}

\begin{figure}[htbp]

\centerline{\includegraphics[scale=0.5]{P_30_30.png}}
\caption{Highest peak on the remaining 30\% of R-R interval.}

\end{figure}

In order to detect P and T waves, two filters are used:
 \begin{equation}
     G_1(z) = 1 -z^{-6}
 \end{equation}

 and

 \begin{equation}
     G_2(z) = \frac{1-z^{-8}}{1-z^{-1}}
 \end{equation}

 The first filter \textit{$G_1(z)$} is a differentiator, it allows the detection of maximum, minimum, as well as null values. This is achieved by determining where the signal, after applying the differentiator \textit{$G_1(z)$}, crosses the level 0.

 The Algorithm functions as follows: The R-R interval is divided into two parts, a part containing 70\% of the R-R interval, in which T waves are located; and another part containing the remaining 30\% of the R-R interval, in which P waves are located. In each part, the locations in which the signal (after crossing both filers (\textit{$G_1(z)$} and (\textit{$G_2(z)$}) crossed the level 0 were determined. As mentioned before, these locations correspond to either a maximum, minimum, or null values in the original ECG signal. Therefore, the locations that provide a maximum value, are where P and T waves are located.

 The figures below illustrate the locations of P and T waves in the original signal,as well as their location after each filter.

\begin{figure}[H]
\centerline{\includegraphics[scale=0.5]{ori_sig.png}}
\caption{Original ECG with locations of P and T waves}
\end{figure}

\begin{figure}[H]
\centerline{\includegraphics[scale=0.5]{sig_G1.png}}
\caption{ECG after undergoing first filter}
\end{figure}

\begin{figure}[H]
\centerline{\includegraphics[scale=0.5]{sig_G2.png}}
\caption{ECG after undergoing second filter}
\end{figure}

\section{Tachycardia/Bradycardia}

\subsection{Bradycardia}

Bradycardia is defined as a heart rate (HR) of less than 50 or 60 bpm (beats per minute), compared to a normal heart rate of 60 to 100 bpm. A slow heart rate is in general a sign of good health and fitness. However, a heart rate that is too slow (\textit{ie} Bradycardia ), is a sign of a problem with the heart's electrical system. It means that the heart's natural pacemaker isn't working right or that the electrical pathways of the heart are disrupted. The heart beat can be so slow that it can't pump enough blood for the body. Which could be life-threatening if left untreated.

\subsection{Tachycardia}

Tachycardia is defined as a heart rate (HR) of over 100 bpm (beats per minute). With atrial or supraventricular tachycardia, electrical signals in the heart’s upper chambers fire abnormally. This interferes with the heart’s natural pacemaker, and causes abnormal heart rates. This rapid heartbeat keeps the heart’s chambers from filling completely between contractions, which compromises blood flow to the rest of the body.

\subsection{Detecting cardiac rhythm anomalies: Tachycardia and Bradycardia}

An algorithm that calculates the heart rate of a patient, would make it possible to detect cardiac rhythm anomalies. The R waves mark the moment in which the heart beats. Therefore, the duration between consecutive R peaks would help determine the heart rate.
The value calculated is:

\begin{equation}
    \overline{\Delta} = \frac{1}{N}\times\sum_{n=0}^{N-1}\Delta_n
\end{equation}

$\overline{\Delta}$ is the mean of R-R intervals duration. It's value was converted to bpm (beats per minute) by multiplying with \textit{$F_s$} and by 60.

\section{Other pathologies}

\subsection{Ectopic beats}
An ectopic heartbeat is when the heart either skips a beat, or adds an extra beat. They are also called premature heartbeats. Ectopic heartbeats are usually not a cause for concern, and they may occur for no known reason. Despite the skipped or added beat, the heart  functions normally.

\subsection{Detecting ectopic beats}

Detecting premature heartbeats, comes down to detecting irregularities in R-R intervals, since, as mentioned earlier (\textit{section 5.3}), R peaks characterize the moment in which the heart beats.

The algorithm devised, calculated the length of different R-R intervals,and chose their maximum value, since an ectopic beat is placed either to close or too far from a regular beat, either a maximum or a minimum value would have been appropriate.
By calculating the same value for normal patients, a threshold $\epsilon$ was determined as $\epsilon = 1$.
This value (\textit{\textbf{max}}) was calculated for different patients, and compared with $\epsilon$.
The results were consistent, \textit{\textbf{max}} was Superior to $\epsilon$ in patients with ectopic beats, and inferior or equal on other cases.

\section{Fibrillation}

\subsection{Atrial fibrillation}

Atrial fibrillation is characterized by a strong heart rhythm irregularity until the point of considering it as white noise. The main property of a white noise is that its autocorrelation function is null except in zero. That is why autocorrelation function of the studied ECG is estimated with the formula :

\begin{equation}
    \hat{\gamma_k} = \frac{1}{N-k-1}\times\sum_{n=0}^{N-k-1} (\Delta_{n+k} - \overline{\Delta})(\Delta_{n} - \overline{\Delta})
\end{equation}

\begin{figure}
\centerline{\includegraphics[scale=0.4]{figure_6_edge_effect.png}}
\caption{Edge effect.}
\end{figure}

Figure 11 is an example of an estimated autocorrelation function. At the right of the figure, big random values are remarked because, in this area,the autocorrelation is estimated using a small number of samples so the mean is not representative enough. For that reason, the studied domain of the autocorrelation function is not going to include the values at the right edge.

The heart rhythm is going to be considered as a white noise if 40\% of its autocorrelation function at zero is still a maximum of the autocorrelation function.

\subsection{Ventricular fibrillation}

Ventricular fibrillation, or V-fib, is considered the most serious cardiac rhythm disturbance.
Disordered electrical activity causes the heart’s lower chambers (ventricles) to quiver, or fibrillate, instead of contracting (or beating) normally. This prohibits the heart from pumping blood. Thus leading to collapse and cardiac arrest

\subsection{Detecting ventricular fibrillation}

Two main properties are used to detect Ventricular fibrillation: The similarity between the ECG (electrocardiogram) of the patient and the pure sine; as well as a rapid heart rate between 240 and 600 bpm (beats per minute).
The similarity between Ventricular fibrillation ECG and the pure sine function was detected by resorting to the Fourier transform. Plotting the two Fourier transforms, illustrates the similarity between them, as shown below.
\textbf{N.B: FT = Fourier transform}

\newpage

\begin{figure}
\centerline{\includegraphics[scale=0.6]{Fourrier_VF.png}}
\caption{TF of ECG}
\end{figure}
\begin{figure}
\centerline{\includegraphics[scale=0.6]{sin_fou.png}}
\caption{TF of sine}
\end{figure}

\newpage

The FT are similar, with two important peaks at symmetric values of frequency. In order to detect the two peaks,  a threshold was applied to the FT,in order to calculate the number of values superior to $0.8\times\textit{n}$; where \textit{n} is the maximum value of the FT. An ECG with a ventricular fibrillation generally has a low number of values above  $0.8\times\textit{n}$, compared to other ECGs, combining this with the condition of a heart rate between 240 and 600 bpm, a case of ventricular fibrillation can be determined

\section{Conclusion}
This report has discussed the implementation of algorithms used to detect PQRST waves and automatic heart pathology identification. Besides that, it has introduced the utility of spectrograms as well as their limitations. The results obtained were satisfying. However, we really wished to develop a graphic user interface to make our work valuable but we could not because we did not have enough time.

\newpgae

\begin{thebibliography}{2}
   \bibitem[1]{cle} P and T waves https://pubmed.ncbi.nlm.nih.gov/29484531/
   \bibitem[2]{cle} Bradycardia   https://www.uofmhealth.org/health-library/aa107571
   \bibitem[3]{cle} tachy https://pubmed.ncbi.nlm.nih.gov/29484531/
   \bibitem[4]{cle} ectopic https://www.medicalnewstoday.com/articles/323202#what-is-an-ectopic-heartbeat
   \bibitem[5]{cle} Ventr fibr https://www.heart.org/en/health-topics/arrhythmia/about-arrhythmia/ventricular-fibrillation

\end{thebibliography}

\end{document}


% Template file for TeXShop by Michael Sharpe, LPPL
\documentclass[11pt]{article}
\usepackage[margin=0.9in]{geometry}
\usepackage[parfill]{parskip}% Begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amsmath,amsthm}
\usepackage{natbib}

\title{MIT 6.862 Applied Machine Learning \\ Project Progress Report}
\author{Matthew West, Fall 2019}
\begin{document}
\date{}
\maketitle

\section{Introduction}
This is the progress report for my MIT course 6.862 project, which aims to explore classification in the context of medical diagnosis, and specifically in datasets derived from images of breast cancer. The focus thus far has been on the establishment of a data pipeline and exploratory data analysis, before comparing multiple candidate machine learning models on the basis of classification accuracy.

This report focuses on the Wisconsin Breast Cancer dataset, \cite{Dua:2019} an analysis of which first appeared in Street et al., which used a decision tree-based algorithm to classify the data with approximately 97\% accuracy. \cite{street1993nuclear, mangasarian1993mathematical} As outlined in the proposal, it was decided that comparison between traditional machine learning on structured data derived from images and the raw images themselves was a desirable direction to take the project in. To this end, I have contacted an original author of this paper in an attempt to acquire the corresponding raw images, though this has not proved fruitful as of yet. There remains the option to extend the analysis to further datasets outlined in the proposal, though such a comparison still won't be straightforward and will introduce substantial roadblocks in the form of data extraction and image segmentation.

The report is structured as follows: Following this introduction, an account of the progress so far will be given regarding the data, methods, results, and discussion, followed by a brief outline of what will be done with the time remaining, and outlining a specific plot that will appear in the final report.

\section{Progress}
\subsection{Data}
The data used in the initial portion of the project thus far is the Wisconsin Breast Cancer dataset, collected by a team at the University of Wisconsin-Madison in the early 1990's. The dataset uses features derived from a set of fine-needle aspirate (FNA) images of breast tissue in patients with either malignant or benign tumours. It is a structured dataset with 569 instances and 30 features, an ID number, and a label, with 212 malignant examples and 357 benign examples. There are 10 base features (radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry, fractal dimension), and for each of these the mean, standard error, and largest/worst example of these features are included over each image, resulting in 30 features in total.

The dataset was checked for missing values, before dropping a non-explanatory ID column, an unexplained column of all NaN values, and separating the labels from the feature vectors.

\subsection{Method}

The machine learning problem at hand is one of classification. If we denote our hypothesis by $\text{h}(x; \lambda)$ where the feature vector for each tumour instance is denoted as $x^{(i)}$ and $\lambda$ denotes a specific vector of parameters that determine a hypothesis, our problem is to identify the optimal $\lambda$ subject to minimising

$$\frac{1}{n}\sum_{i=1}^{n} L(\text{h}(x; \lambda), y^{(i)}),$$

where $L(g, a)$ is the loss function chosen for a given algorithm, providing a measure of how far away the guess from the hypothesis $g$ is from the actual instance label, $a$. Specifically, this will be evaluated using 5-fold cross validation, as minimising training error will not be a relevant or interesting metric. Selected hypotheses from many such algorithms will be compared to see which of these minimises validation error and is therefore the preferred model in terms of classification accuracy. Multiple algorithms have been investigated so far, with promising results from xgboost, naive Bayes and logistic regression, as seen in table \ref{table:1}. These were mainly implemented using scikit-learn in Python 3 on my local machine.

A principle component decomposition was done on the dataset, allowing 2 or 3 principle components to represent a substantial portion of the variation across all 30 features. This allows for an intuitive understanding of how separable the dataset is, as two or three principle components can be plotted and understood visually, something that isn't readily doable for higher dimensional data. It is also possible to directly train and test models on the principle component representation of the data, though one loses the interpretability for which features are most important in predicting the outcome variable.

\subsection{Results}
The first result was, having used PCA to represent the data in both 2 and 3 dimensions, it was confirmed that the dataset would be somewhat separable, if not perfectly linearly separable. This was achieved by confirming that the PCA plots showed natural clusters of malignant and benign instances in the reduced latent space.

Furthermore, it was determined preliminarily that tree-based methods such as xgboost are performing better than other algorithms, with xgboost consisently obtaining 96\% classification accuracy without too much parameter tuning or feature selection. An additional bonus of using tree-based methods is that they allow some quantitative insight into which features are most important in classification, which appear to be those derived from texture and concave points.

\begin{table}[h!]
	\centering
	\begin{tabular}{ |c|c|c|c| }
		\hline
		Algorithm & 5-fold accuracy set & 5-fold accuracy ($\%$) \\
		\hline
		\textbf{xgboost}	& [0.930, 0.957, 0.992, 0.965, 0.973] & $96\pm2$ \\
		\textbf{Naive Bayes} & [0.922, 0.922, 0.956 0.947, 0.956] & $94\pm2$ \\
		\textbf{Logistic regression} & [0.930, 0.939, 0.973, 0.947, 0.965] & $95\pm2$ \\
		\hline
	\end{tabular}
\caption{Table of cross-validated results from selected classification algorithms}
\label{table:1}
\end{table}

\subsection{Discussion}
Getting rather good results using default hyperameters can be straightforward for some datasets. Marginal performance increases then come from the selection of algorithms together with the set of hyperparameters tuned for that algorithm. Without exploring this vast solution space comprehensively, it has still been possible to obtain classification accuracies close to that initial benchmark described in the orginal paper. One insight that has helped to overcome the `noise' associated with the accuracy metric across this solution space was the decision to use 5-fold cross validation when investigating accuracy.

A key lesson learnt thus far was how to think about exploratory data analysis in a systematic way, such as by using PCA to represent the data. This paralleled nicely with the content covered in class on autoencoders, but simplifying it to a linear combination of original features. This has allowed me to abstract conceptually from thinking about raw features to latent spaces, which is an invaluable tool in applied machine learning.

\section{Looking Ahead}
An outline for the next month leading up to the conclusion of the project is given in this section.
\begin{itemize}
	\item An investigation into precision and recall. In the context of medical diagnostics, the use of machine learning algorithms is controversial due to the direct impact of such diagnostics on human survival. Therefore, perhaps unlike many other domains in which machine learning is applied, special consideration must be given to precision and recall, as false positives are in general much less serious than false negatives. This presents a different set of potential metrics, and some of these will be investigated to see if a different algorithm is preferred when we take the domain-specific selection of metric into account. Deadline: 11/14
	\item It will hopefully be possible to acquire a set or perhaps representative subset of the original Wisconsin Breast Cancer images from the researchers that first collected this data. If this is the case, then the focus of this milestone will be to investigate the use of convolutional neural networks (CNN's) on these raw images, and compare performance on any relevant metrics. It is of particular interest to me to see if there are advantages to using CNN's in the small dataset regime, despite not expecting a substantial increase in resultant accuracy (e.g. interpretability, automated feature extraction). I identified this as ``time-allowing" in the project proposal because of the potential time commitment of pre-processing these larger datasets for use in CNN's and also deriving structured datasets from these similar in form to the Wisconsin dataset, for the purposes of making this comparison. Deadline: 12/01
\end{itemize}

\section{Projected Results Section}
The key result of this initial stage of the project will be a comparison of accuracies on the Wisconsin dataset using one model for each of the algorithms selected for comparison, by means of a violinplot or boxplot. The horizontal axis will be discrete in the different algorithms, and the vertical axis will be classification accuracy.

Each of these models will have been tuned to select the optimal set of any relevant hyperparameters, using 5 or 10-fold cross-validation. This plot allows the reader to assess visually which algorithm performs better on this dataset, as well as how much variance there is in each estimate of accuracy across the set of folds used for validation. I envision including a set of similar figures that concern performance not just in terms of accuracy, but also on other metrics that take into account sensitivity and specificity.
\nocite{*}
\bibliographystyle{plain}
\bibliography{MIT}

\end{document}