\chapter{Functional Programming and Closures \label{chapter:closures}}
We have already mentioned that \setlx\ is a full-fledged functional language:
Functions can be used both as arguments to other functions and as return values.  There is
no fundamental difference between the type of a function and, say, the type of a rational
number, as both can be assigned to variables, converted to strings, used as arguments of other
functions, or returned from a function.

Additionally, \setlx\ supports
\href{http://en.wikipedia.org/wiki/Closure_(computer_science)}{\emph{closures}}
in a way similar to languages like \textsl{Scheme} \cite{sussman:75}.
In order to introduce closures, we will first present some simple examples.  After that, we show
a more complex case study: We discuss a program that transforms a regular expression into a
non-deterministic finite state machine.

\section{Introductory Examples}
In this section we will first introduce the basic ideas of functional programming.
After that, we discuss the notion of a closure.

Using functions as arguments to other functions is something that is not often seen in
conventional programming languages.  Figure \ref{fig:reduce.stlx} on page \pageref{fig:reduce.stlx}
shows the implementation of the function \texttt{reduce}, that takes two arguments:
\begin{enumerate}
\item The first argument is a list $l$.
\item The second argument is a binary function $f$.  This function is expected to take two
      arguments.
\end{enumerate}

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    reduce := procedure(l, f) {
        match (l) {
            case []     : return;
            case [x]    : return x;
            case [x,y|r]: return reduce([f(x,y) | r], f);
        }
    };
    add      := [x, y] |-> x + y;
    multiply := [x, y] |-> x * y;

    l := [1 .. 36];
    x := reduce(l, add     );
    y := reduce(l, multiply);
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Implementing a second order function.}
\label{fig:reduce.stlx}
\end{figure}
The function \texttt{reduce} combines successive arguments of the list $l$ using the function $f$
and thereby reduces the list $l$ into a single value.  For example, if the function $f$ is the
function \texttt{add} that just adds its arguments, then $\texttt{reduce}(l,
\texttt{add})$ computes the sum of
the elements of $l$.  Therefore, in line 12 the invocation of
\texttt{reduce(l, add)} computes the sum of all elements of the list $l$.  Since this list
contains the natural numbers from 1 up to 36, \texttt{reduce} effectively computes the sum
\\[0.2cm]
\hspace*{1.3cm}
 $\sum\limits_{i=1}^{36} i$.
\\[0.2cm]
Note that the expression ``\texttt{+/ $l$}'' is another way to compute this sum.
If instead the second arguments of the function \texttt{reduce} is the function
\texttt{multiply} that returns the product of its arguments, then
$\texttt{reduce}(l, \texttt{multiply})$ computes the product of all elements of the list
$l$ and hence is equivalent to the expression ``\texttt{*/ $l$}''.
Finally, it should be noted that in the case that the list $l$ is empty,
\texttt{reduce} returns the value \texttt{om}.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    sort := procedure(l, cmp) {
        if (#l < 2) { return l; }
        m := #l \ 2;
        [l1, l2] := [l[.. m], l[m+1 ..]];
        [s1, s2] := [sort(l1, cmp), sort(l2, cmp)];
        return merge(s1, s2, cmp);
    };
    merge := procedure(l1, l2, cmp) {
        match ([l1, l2]) {
            case [[], _] : return l2;
            case [_, []] : return l1;
            case [[x1|r1], [x2|r2]] :
                 if (cmp(x1, x2)) {
                     return [x1 | merge(r1, l2, cmp)];
                 } else {
                     return [x2 | merge(l1, r2, cmp)];
                 }
        }
    };
    less    := procedure(x, y) { return x < y; };
    greater := procedure(x, y) { return y < x; };
    l  := [1,3,5,4,2];
    s1 := sort(l, less   );
    s2 := sort(l, greater);
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A generic sort function.}
\label{fig:merge-sort.stlx}
\end{figure}

Notice that the function \texttt{reduce} is a so called
\href{https://en.wikipedia.org/wiki/Higher-order_function}{higher order function} because
one of its arguments is itself a function.  While the example given above might seem
artificial, there are common applications of this scheme.  One example is sorting.
Figure \ref{fig:merge-sort.stlx} on page \pageref{fig:merge-sort.stlx} shows a generic
implementation of the algorithm \href{https://en.wikipedia.org/wiki/Merge_sort}{\emph{merge sort}}.
Here, the function \texttt{sort} takes two arguments:
\begin{enumerate}
\item The first argument $l$ is the list to be sorted.
\item The second argument \texttt{cmp} is a binary function that compares two  elements
      and returns either \texttt{true} or \texttt{false}.  If we call the function
      \texttt{sort} with second argument \texttt{less}, where the function \texttt{less} is
      defined in line 20, then the resulting list will be sorted ascendingly.
      If instead we use the function \texttt{greater} defined in line 21 as the second
      argument, then the list $l$ will be sorted descendingly.

      The second argument \texttt{cmp} of the procedure \texttt{sort} enables us to sort a list that
      contains elements that are not numbers.  In order to do so, we just have to implement an
      appropriate version of the function \texttt{cmp}.
\end{enumerate}

\subsection{Implementing Counters via Closures}
The concept of a closure is non-trivial, therefore we introduce the idea using some simple
examples.  Figure \ref{fig:counter-closure.stlx} shows the function
\texttt{createCounter}.  This function initializes the variable \texttt{count} to a given
value and then returns a procedure that increments this value.  Note that this procedure is not
defined via the keyword \texttt{procedure} but rather via the keyword \texttt{closure}.  The reason
is that this function needs access to the variable \texttt{count} that is defined outside of the
closure \texttt{counter}.  By defining the function \texttt{counter} as a \texttt{closure} rather
than as a \texttt{procedure},  the function \texttt{counter} has access to the variable \texttt{count}:
It can both read its value and even change the value.  When the function \texttt{counter} is
defined, the variable \texttt{count} is safely tucked away together with the function
\texttt{counter}.  Therefore, although the variable \texttt{count} goes out of scope once the
function \texttt{createCounter} terminates, the function \texttt{counter} still has access
to a
copy of this variable.  Hence, when we call the function \texttt{createCounter} in line 9,
the variable \texttt{ctr0} is assigned a version of the function \texttt{counter} where the variable
\texttt{count} initially has the value 0.  Later, when we call the function \texttt{ctr0}
in line 12, this counter is incremented 3 times.  Therefore, after line 12 is executed,
the variables \texttt{u}, \texttt{v}, and \texttt{w} have the values $1$, $2$, and $3$,
respectively.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    createCounter := procedure(i) {
        count   := i;
        counter := closure() {
                       count += 1;
                       return count;
                   };
        return counter;
    };
    ctr0 := createCounter(0);
    ctr9 := createCounter(9);

    u := ctr0(); v := ctr0(); w := ctr0();
    x := ctr9(); y := ctr9(); z := ctr9();
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Creating a counter as a closure.}
\label{fig:counter-closure.stlx}
\end{figure}

You should notice that the function \texttt{ctr9} created in line 10 gets its
\underline{own} copy of the variable \texttt{count}.  In the case of the function
\texttt{ctr9}, this copy of \texttt{count} is initialized with the value 9.  Therefore, after
we have called the function \texttt{ctr9} in line 13, the variables \texttt{x}, \texttt{y}, and
\texttt{z} will have the values $10$, $11$, and $12$.

\subsection{Lambda Definitions for Closures}
There is a short-hand notation for the definition of a closure that can be used if the closure only
contains a single \texttt{return} statement. This short-hand notation relates to closures as lambda definitions relate to
procedures.  This short-hand notation is called a \emph{lambda closure}.  Here is a simple example:
The following two lines define a function \texttt{f} that computes the cube of its argument.
\\[0.2cm]
\hspace*{1.3cm} \texttt{c := 3;}            \\
\hspace*{1.3cm} \texttt{f := x |=> x ** c;}
\\[0.2cm]
These two lines are equivalent to the following two lines:
\\[0.2cm]
\hspace*{1.3cm} \texttt{c := 3;}            \\
\hspace*{1.3cm} \texttt{f := closure(x) \{ return x ** c; \};}
\\[0.2cm]
In general, the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{x |=> \textsl{expr}}
\\[0.2cm]
is equivalent to the following expression:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{closure(x) \{ return \textsl{expr}; \}}
\\[0.2cm]
If there is more than one argument to a closure, the arguments have to be enclosed in square
brackets.  For example,
\\[0.2cm]
\hspace*{1.3cm}
\texttt{f := [x,y] |=> sqrt(x ** c + y ** c);}
\\[0.2cm]
is equivalent to
\\[0.2cm]
\hspace*{1.3cm}
\texttt{f := closure(x,y) \{ return sqrt(x ** c + y ** c); \};}

\subsection{Closures as Return Values}
So far, the functions we have discussed did not return functions.  The next example will change
that.  In mathematics, given a function
\\[0.2cm]
\hspace*{1.3cm}
$f: \mathbb{N} \rightarrow \mathbb{N}$
\\[0.2cm]
mapping natural numbers into natural numbers, the
\href{https://en.wikipedia.org/wiki/Finite_difference}{\emph{discrete derivative}} of $f$ is again a
function mapping natural numbers to natural numbers and is denoted
as $\Delta f$.  For $n \in \mathbb{N}$, the value $(\Delta f)(n)$ is defined as
\\[0.2cm]
\hspace*{1.3cm}
$(\Delta f)(n) := f(n+1) - f(n)$.
\\[0.2cm]
For example, given the function $f: \mathbb{N} \rightarrow \mathbb{N}$ defined as $f(n) := 2^n$,
we have
\\[0.2cm]
\hspace*{1.3cm}
$(\Delta f)(n) = 2^{n+1} - 2^n = 2^n = f(n)$.
\\[0.2cm]
To give another example, the discrete derivative of the identity function is the function that
maps every natural number to the number $1$, since $(n+1) - n = 1$.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    delta := procedure(f) {
        return n |=> f(n+1) - f(n);
    };
    g := n |-> n;
    h := n |-> 2 ** n;
    deltaG := delta(g);
    deltaH := delta(h);

    print([ deltaG(n) : n in [1 .. 10]]);
    print([ deltaH(n) : n in [1 .. 10]]);
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Computing the discrete derivative of a given function.}
\label{fig:finite-difference.stlx}
\end{figure}

\noindent
Figure \ref{fig:finite-difference.stlx} shows the implementation of the function \texttt{delta}
that takes a function \texttt{f} as input and then returns the discrete derivative of \texttt{f}.  In line 6
and 7, this function is applied to the identity function and to the function $n \mapsto 2^n$,
respectively.  The resulting functions \texttt{deltaG} and \texttt{deltaH} are then applied to
the sequence of natural numbers from 1 to 10.  In the first case, the result is a list
containing the number one ten times, while in the second case the the result is a list of the
first ten powers of two.

Note that we had to define the return value of the function \texttt{delta} as a closure.  The reason
is that otherwise the value \texttt{f} referring to the function would not be in the local scope of
the function that is returned by \texttt{delta} and hence \texttt{f} would be undefined.

\section[Closures in Action]{Closures in Action: Generating Finite State Machines from Regular Expressions}
In this section we demonstrate a non-trivial application of closures.  We present a program that
takes a regular expression $r$ as input and  converts this regular expression into a
\href{http://en.wikipedia.org/wiki/Nondeterministic_finite_automaton}{\emph{non-deterministic finite state machine}}
using the
\href{http://en.wikipedia.org/wiki/Thompson's_construction_algorithm}{\emph{Thompson construction}}
\cite{hopcroft:06}.  To this end, we first define a \emph{alphabet} as a finite set of characters.
Then, for the purpose of this section, \emph{regular expressions over an alphabet $\Sigma$} are
defined inductively as follows.
\begin{enumerate}
\item Every character $c\in \Sigma$ is a regular expression matching exactly this character and
      nothing else.
\item If $r_1$ and $r_2$ are regular expressions,  then $r_1 \cdot r_2$ is a regular
      expression.  If $r_1$ matches the string $s_1$ and $r_2$ matches the string $s_2$,
      then $r_1 \cdot r_2$ matches the string $s_1s_2$, where $s_1s_2$ is understood as
      the concatenation of $s_1$ and $s_2$.
\item If $r_1$ and $r_2$ are regular expressions,  then $r_1 \mid r_2$ is a regular
      expression.  The regular expression $r_1 \mid r_2$ matches any string that is matched
      by either $r_1$ or $r_2$.
\item If $r$ is a regular expression, then $r^*$ is a regular expression.
      This regular expression matches the empty string and, furthermore, matches any string
      $s$ that can be decomposed as
      \\[0.2cm]
      \hspace*{1.3cm}
      $s = t_1 t_2 \cdots t_n$
      \\[0.2cm]
      where each of the substrings $t_i$ is matched by the regular expression $r$.
\end{enumerate}
For the purpose of the program we are going to develop, composite regular expressions will be
represented by terms.  In detail, we have the following representation of regular
expressions.
\begin{enumerate}
\item A regular expression $r$ of the form $r = c$ where $c\in \Sigma$ is represented the character $c$.
\item A regular expression $r$ of the form $r = r_1 \cdot r_2$ is represented by the term
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{@Cat(r1, r2)}
      \\[0.2cm]
      where \texttt{r1} is the representation of $r_1$ and \texttt{r2} is the
      representation of $r_2$.  The name \texttt{Cat} reminds us of con\underline{cat}enation.
\item A regular expression $r$ of the form $r = r_1 \mid r_2$ is represented by the term
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{@Or(r1, r2)}
      \\[0.2cm]
      where \texttt{r1} is the representation of $r_1$ and \texttt{r2} is the
      representation of $r_2$.
\item A regular expression $r$ of the form $r = r_0^*$ is represented by the term
      \\[0.2cm]
      \hspace*{1.3cm}
      \texttt{@Star(r0)}
      \\[0.2cm]
      where \texttt{r0} is the representation of $r_0$.
\end{enumerate}
The \href{https://en.wikipedia.org/wiki/Thompson%27s_construction_algorithm}{\emph{Thompson construction}}
of a non-deterministic finite state machine from a given regular
expression $r$ works by induction on the structure of $r$.  Given a regular expression
$r$,  we define a non-deterministic finite state machine $A(r)$ by induction on $r$.
For the purpose of this section, a finite state machine $A$ is a 4-tuple
\\[0.2cm]
\hspace*{1.3cm}
$A = \langle Q, \delta, q_0, q_f \rangle$
\\[0.2cm]
where $Q$ is the set of states,
$\delta$ is the transition function and maps a pair $\langle q, c \rangle$ where $q$ is a
state and $c$ is a character to a set of new states.  If we denote the set of characters
as $\Sigma$, then the function $\delta$ has the signature
\\[0.2cm]
\hspace*{1.3cm}
$\delta: Q \times \Sigma \rightarrow 2^Q$.
\\[0.2cm]
The understanding  of $\delta(q,c)$ is that if the finite state machine $A$ is in the
state $q$ and reads the character $c$, then it can switch to any of the states in
$\delta(q,c)$.
Finally, $q_0$ is the start state and $q_f$ is the accepting state.  In the following, we
abbreviate the notion of a
\emph{finite state machine} as fsm.
\begin{enumerate}
\item For a letter $c$ the fsm $A(c)$ is defined as
      \\[0.2cm]
      \hspace*{1.3cm}
      $A(c) = \langle \{ q_0, q_1 \},
                      \{ \langle q_0, c \rangle \mapsto q_1\}, q_0, q_1  \rangle$.
      \\[0.2cm]
      The set of states is given as $\{ q_0, q_1 \}$, on reading the character $c$ the
      transition function maps the state $q_0$ to $q_1$, the start state is $q_0$ and the
      accepting state is $q_1$.  This fsm is shown in Figure
      \ref{fig:aChar.eps}.  The start state $q_0$ is marked by an incoming arrow, while the
      accepting state $q_1$ is marked by a doubled boundary.

      \begin{figure}[!ht]
        \centering
      \epsfig{file=aChar.eps, scale=0.5}
      \caption{The finite state machine $A(c)$.}
      \label{fig:aChar.eps}
      \end{figure}

      \begin{figure}[!ht]
      \centering
      \begin{Verbatim}[ frame         = lines,
                        framesep      = 0.3cm,
                        firstnumber   = 1,
                        labelposition = bottomline,
                        numbers       = left,
                        numbersep     = -0.2cm,
                        xleftmargin   = 0.8cm,
                        xrightmargin  = 0.8cm,
                      ]
    genCharNFA := procedure(c, rw ctr) {
        q0 := getNewState(ctr);
        q1 := getNewState(ctr);
        delta := closure(q, d) {
                     if (q == q0 && d == c) {
                         return { q1 };
                     } else {
                         return {};
                     }
                 };
        return [ {q0, q1}, delta, q0, q1 ];
    };
      \end{Verbatim}
      \vspace*{-0.3cm}
      \caption{Generating the fsm to recognize the character $c$.}
      \label{fig:genCharNFA.stlx}
      \end{figure}

      Figure \ref{fig:genCharNFA.stlx} shows the function \texttt{genCharNFA} that takes a
      character $c$ and generates the fsm $A(c)$.  The code corresponds to the
      diagram shown in Figure \ref{fig:aChar.eps}.
      \begin{enumerate}
      \item We generate two new states $q_0$ and $q_1$ using the function
            \texttt{getNewState}.  The implementation of this function is shown in Figure
            \ref{fig:getNewState.stlx} on page \pageref{fig:getNewState.stlx}.  Every time this
            function is called, it creates a new unique string that is interpreted as a state.
      \item The transition function checks whether the state $q$ given as input is equal
            to the start state $q_0$ and whether, furthermore, the character $d$ that is
            read is identical to the character $c$.  If this is the case, the fsm switches
            into the state $q_1$ and therefore the set of possible next states is the singleton
            set $\{ q_1 \}$.  Since this is the only transition of the fsm, in all other
            cases the set of next states is empty.
      \end{enumerate}

\item In order to compute the fsm $A(r_1 \cdot r_2)$
      we have to assume that the sets of states of the fsms
      $A(r_1)$ and $A(r_2)$ are disjoint.  This is true since the function \texttt{getNewState} does
      create a new state every time it is called.
      Let us assume that the fsms  $A(r_1)$ and $A(r_2)$ have the following form:
      \begin{enumerate}
      \item $A(r_1) = \langle Q_1, \delta_1, q_1, q_2 \rangle$,
      \item $A(r_2) = \langle Q_2, \delta_2, q_3, q_4 \rangle$, \quad where
      \item $Q_1 \cap Q_2 = \{\}$.
      \end{enumerate}
      Using $A(r_1)$ and $A(r_2)$ we construct the fsm $A(r_1 \cdot r_2)$ as
      \\[0.2cm]
      \hspace*{0.8cm}
       $\langle Q_1 \cup Q_2, \Sigma,
                \{ \pair(q_2,\varepsilon) \mapsto q_3 \}
                   \cup \delta_1 \cup \delta_2, q_0, q_4 \rangle$
      \\[0.2cm]
      The notation $\{ \pair(q_2,\varepsilon) \mapsto q_3 \} \cup \delta_1 \cup \delta_2$
      specifies that the transition function $\delta$ contains all the transitions
      from the transition functions  $\delta_1$ and $\delta_2$.  Additionally,
      there is an  $\varepsilon$-transition from $q_2$ to
      $q_3$.  Formally, the transition function could also be specified as follows:
      \\[0.2cm]
      \hspace*{1.3cm}
      $\delta(q,c) := \left\{
      \begin{array}{ll}
        \{ q_3 \}       & \mbox{if $q = q_2$ and $c = \varepsilon$}, \\[0.2cm]
        \delta_1(q, c)  & \mbox{if $q \in Q_1$ and $\pair(q,c) \not= \pair(q_2,\varepsilon)$}, \\[0.2cm]
        \delta_2(q, c)  & \mbox{if $q \in Q_2$.}
      \end{array}\right.
      $
      \\[0.2cm]
      Figure \ref{fig:aConcat.eps} depicts the fsm $A(r_1 \cdot r_2)$.

      \begin{figure}[!ht]
        \centering
      \epsfig{file=aConcat.eps, scale=0.8}
      \caption{The finite state machine $A(r_1 \cdot r_2)$.}
      \label{fig:aConcat.eps}
      \end{figure}

      Figure \ref{fig:catenate.stlx} on page \pageref{fig:catenate.stlx} shows the
      implementation of the function \texttt{catenate}.  This function takes two
      finite state machines \texttt{f1} and \texttt{f2} and concatenates them in the way
      depicted in Figure \ref{fig:aConcat.eps}.  This function can be used
      to compute $A(r_1 \cdot r_2)$, since we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $A(r_1 \cdot r_2) = \mathtt{catenate}\bigl(A(r_1), A(r_2)\bigr)$.

    \begin{figure}[!ht]
    \centering
    \begin{Verbatim}[ frame         = lines,
                      framesep      = 0.3cm,
                      firstnumber   = 1,
                      labelposition = bottomline,
                      numbers       = left,
                      numbersep     = -0.2cm,
                      xleftmargin   = 0.8cm,
                      xrightmargin  = 0.8cm,
                    ]
    catenate := procedure(f1, f2) {
        [m1, delta1, q1, q2] := f1;
        [m2, delta2, q3, q4] := f2;
        delta := closure(q, c) {
                     if (q == q2 && c == "") {
                         return { q3 };
                     } else if (q in m1) {
                         return delta1(q, c);
                     } else if (q in m2) {
                         return delta2(q, c);
                     } else {
                         return {};
                     }
                 };
        return [ m1 + m2, delta, q1, q4 ];
    };
    \end{Verbatim}
    \vspace*{-0.3cm}
    \caption{The function to compute $A(r_1 \cdot r_2)$}
    \label{fig:catenate.stlx}
    \end{figure}
\item In order to define the fsm  $A(r_1 + r_2)$ we assume that we have already computed
      the fsms $A(r_1)$ and $A(r_2)$ and that their sets of states are disjoint.
      If  $A(r_1)$ and $A(r_2)$ are given as
      \\[0.2cm]
      \hspace*{1.3cm}
      $A(r_1) = \langle Q_1, \delta_1, q_1, q_3 \rangle$ \quad and \quad
      $A(r_2) = \langle Q_2, \delta_2, q_2, q_4 \rangle$
      \\[0.2cm]
      then the fsm $A(r_1 + r_2)$ can be defined as
      \\[0.2cm]
      \hspace*{0.0cm}
       $\langle \{ q_0, q_5 \} \cup Q_1 \cup Q_2,
                \{ \pair(q_0,\varepsilon) \mapsto q_1, \pair(q_0,\varepsilon) \mapsto q_2,
                   \pair(q_3,\varepsilon) \mapsto q_5, \pair(q_4,\varepsilon) \mapsto q_5 \}
                   \cup \delta_1 \cup \delta_2, q_0, q_5 \rangle$.
      \\[0.2cm]
      This finite state machine is shown in Figure \ref{fig:aPlus.eps}.
      In addition to the states of $A(r_1)$ and $A(r_2)$ there are two additional
      states:
      \begin{enumerate}
      \item $q_0$ is the start state of the fsm $A(r_1 + r_2)$,
      \item $q_5$ is the accepting state of $A(r_1 + r_2)$.
      \end{enumerate}
      In addition to the transitions of the fsms  $A(r_1)$ and $A(r_2)$ we have four $\varepsilon$-transitions.
      \begin{enumerate}
      \item There are $\varepsilon$-transitions from $q_0$ to the states $q_1$ and $q_2$.
      \item There are $\varepsilon$-transitions from  $q_3$ and $q_4$ to $q_5$.
      \end{enumerate}

      \begin{figure}[!ht]
        \centering
      \epsfig{file=aPlus.eps, scale=0.5}
      \caption{The finite state machine $A(r_1 + r_2)$.}
      \label{fig:aPlus.eps}
      \end{figure}

    \begin{figure}[!ht]
    \centering
    \begin{Verbatim}[ frame         = lines,
                      framesep      = 0.3cm,
                      firstnumber   = 1,
                      labelposition = bottomline,
                      numbers       = left,
                      numbersep     = -0.2cm,
                      xleftmargin   = 0.8cm,
                      xrightmargin  = 0.8cm,
                    ]
    disjunction := procedure(f1, f2, rw ctr) {
        [m1, delta1, q1, q3] := f1;
        [m2, delta2, q2, q4] := f2;
        q0 := getNewState(ctr);
        q5 := getNewState(ctr);
        delta := closure(q, c) {
                     if (q == q0 && c == "") {
                         return { q1, q2 };
                     } else if (q in { q3, q4 } && c == "") {
                         return { q5 };
                     } else if (q in m1) {
                         return delta1(q, c);
                     } else if (q in m2) {
                         return delta2(q, c);
                     } else {
                         return {};
                     }
                 };
        return [ { q0, q5 } + m1 + m2, delta, q0, q5 ];
    };
    \end{Verbatim}
    \vspace*{-0.3cm}
    \caption{The function to compute $A(r_1 + r_2)$.}
    \label{fig:disjunction.stlx}
    \end{figure}

      Figure \ref{fig:disjunction.stlx} on page \pageref{fig:disjunction.stlx} shows the
      implementation of the function \texttt{disjunction}.  This function takes two
      finite state machines \texttt{f1} and \texttt{f2} and combines them in the way
      depicted in Figure \ref{fig:aPlus.eps}.  This function can be used
      to compute $A(r_1 + r_2)$, since we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $A(r_1 +r_2) = \mathtt{disjunction}\bigl(A(r_1), A(r_2)\bigr)$.
\item In order to define $A(r^*)$ we assume $A(r)$ is given as
      \\[0.2cm]
      \hspace*{1.3cm}
      $A(r) = \langle Q, \delta, q_1, q_2 \rangle$.
      \\[0.2cm]
      Then  $A(r^*)$ is defined as
      \\[0.2cm]
      \hspace*{0.8cm}
       $\langle \{ q_0, q_3 \} \cup Q,
                \{ \pair(q_0,\varepsilon) \mapsto q_1, \pair(q_2,\varepsilon) \mapsto q_1,
                   \pair(q_0,\varepsilon) \mapsto q_3, \pair(q_2,\varepsilon) \mapsto q_3 \}
                   \cup \delta, q_0, q_3  \rangle$.
      \\[0.2cm]

      \begin{figure}[!ht]
        \centering
      \epsfig{file=aStar.eps, scale=0.5}
      \caption{The finite state machine $A(r^*)$.}
      \label{fig:aStar.eps}
      \end{figure}

      Figure \ref{fig:aStar.eps} depicts the fsm $A(r^*)$.
      In addition to the states of the fsm  $A(r)$ there are two new states:
      \begin{enumerate}
      \item $q_0$ is the start state of $A(r^*)$,
      \item $q_3$ is the accepting state of $A(r^*)$.
      \end{enumerate}
      Furthermore, there are four additional  $\varepsilon$-transitions.
      \begin{enumerate}
      \item There are two $\varepsilon$-transitions form the start state $q_0$ to $q_1$ and $q_3$.
      \item From $q_2$ there is an  $\varepsilon$-transition back to $q_1$ and also an
            $\varepsilon$-transition to $q_3$.
      \end{enumerate}
      Figure \ref{fig:kleene.stlx} on page \pageref{fig:kleene.stlx} shows the
      implementation of the function \texttt{kleene}.  This function takes a
      finite state machine \texttt{f} and transforms it in the way
      depicted in Figure \ref{fig:aStar.eps}.  This function can be used
      to compute $A(r^*)$, since we have
      \\[0.2cm]
      \hspace*{1.3cm}
      $A(r^*) = \mathtt{kleene}\bigl(A(r)\bigr)$.

    \begin{figure}[!ht]
    \centering
    \begin{Verbatim}[ frame         = lines,
                      framesep      = 0.3cm,
                      firstnumber   = 1,
                      labelposition = bottomline,
                      numbers       = left,
                      numbersep     = -0.2cm,
                      xleftmargin   = 0.8cm,
                      xrightmargin  = 0.8cm,
                    ]
    kleene := procedure(f, rw ctr) {
        [m, delta0, q1, q2] := f;
        q0 := getNewState(ctr);
        q3 := getNewState(ctr);
        delta := closure(q, c) {
                     if (q == q0 && c == "") {
                         return { q1, q3 };
                     } else if (q == q2 && c == "") {
                         return { q1, q3 };
                     } else if (q in m) {
                         return delta0(q, c);
                     } else {
                         return {};
                     }
                 };
        return [ { q0, q3 } + m, delta, q0, q3 ];
    };
    \end{Verbatim}
    \vspace*{-0.3cm}
    \caption{The function to compute $A(r^*)$.}
    \label{fig:kleene.stlx}
    \end{figure}
\end{enumerate}

Figure \ref{fig:getNewState.stlx} shows the procedure \texttt{getNewState} that is used to create
new states.  Since the parameter \texttt{ctr} of this procedure is preceded by the keyword
\texttt{rw}, this parameter has a \emph{call-by-name} semantics and therefore the procedure
\texttt{getNewState} is able to increment this variable.  Hence, the states generated
by \texttt{getNewState} are unique.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    getNewState := procedure(rw ctr) {
        ctr += 1;
        return "q" + ctr;
    };
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A function to generate unique states.}
\label{fig:getNewState.stlx}
\end{figure}

Finally, Figure \ref{fig:regexp2NFA.stlx} shows how to generate a finite state machine
from a given regular expression $r$.  The function $\texttt{regexp2NFA}(r)$ is defined by
recursion on $r$:
\begin{enumerate}
\item If $r$ is a single character $c$, then the fsm $A(c)$ is computed using
      the function $\texttt{genCharNFA}(c)$.
\item If $r = r_1 \cdot r_2$, the function \texttt{regexp2NFA} recursively computes
      finite state machines $A(r_1)$ and $A(r_2)$.  These fsms are then combined
      using the function $\texttt{catenate}\bigr(A(r_1), A(r_2)\bigr)$.
\item If $r = r_1 + r_2$, the function \texttt{regexp2NFA} recursively computes
      finite state machines $A(r_1)$ and $A(r_2)$.  These fsms are then combined
      using the function $\texttt{disjunction}\bigr(A(r_1), A(r_2)\bigr)$.
\item If $r = r_0^*$, the function \texttt{regexp2NFA} recursively computes
      the finite state machines $A(r_0)$.  This fsm is then transformed
      using the function $\texttt{kleene}\bigr(A(r_0)\bigr)$.
\end{enumerate}
The reader should note that we have made heavy use of closures to implement the functions
\texttt{genCharNFA}, \texttt{catenate}, \texttt{disjunction}, and \texttt{kleene}.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.0cm,
                  xrightmargin  = 0.0cm,
                ]
    regexp2NFA := procedure(r, rw ctr) {
        match (r) {
            case c | isString(c):
                 return genCharNFA(c, ctr);
            case @Cat(r1, r2):
                 return catenate(regexp2NFA(r1, ctr), regexp2NFA(r2, ctr));
            case @Or(r1, r2):
                 return disjunction(regexp2NFA(r1, ctr), regexp2NFA(r2, ctr), ctr);
            case @Star(r0):
                 return kleene(regexp2NFA(r0, ctr), ctr);
        }
    };
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Generating a non-deterministic fsm from a regular expression.}
\label{fig:regexp2NFA.stlx}
\end{figure}
Let us take a step back and consider what we have achieved so far.  We have implemented a function
for converting regular expressions to non-deterministic finite state machines.
It is rather straightforward to implement a function that scans a given string using a non-deterministic
finite state machine. If we would implement this function, we would then have implemented our own
support for regular expressions.

\section{Decorators \label{sec:decorators}}
The programming language \href{https://www.python.org}{\textsl{Python}} has a nice feature called
\href{http://en.wikipedia.org/wiki/Python_syntax_and_semantics#Decorators}{\emph{decorators}} that
makes it easy to modify existing functions.  For example, a \emph{trace decorator} is a function that can
modify a given function $f$ so that every invocation of the function $f$ and every value returned by
the function $f$ are automatically traced.  \setlx\ supports similar techniques as \textsl{Python} and,
therefore, it is easy to implement decorators in \setlx, too.  In this section, we show how a simple
albeit powerful trace decorator can be implemented in just a few lines of \setlx\ code.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    gcd := procedure(n, m) {
        if (m == 0) {
            return n;
        }
        return gcd(m, n % m);
    };
    myTracer := tracer();
    gcd := myTracer.trace(gcd, "gcd");
    n   := gcd(36, 27);
    gcd := myTracer.untrace("gcd");
\end{Verbatim}
\vspace*{-0.3cm}
\caption{Using the trace decorator.}
\label{fig:trace-decorator-use.stlx}
\end{figure}

Figure \ref{fig:trace-decorator.stlx} on page \pageref{fig:trace-decorator.stlx} shows the class
\texttt{tracer}.  While classes will be discussed in full detail in a later chapter, it is convenient
to use them for this example.
This class \texttt{tracer} implements a trace decorator.  Before we discuss the details of its
implementation, lets us show how this decorator is used.  Figure \ref{fig:trace-decorator-use.stlx}
shows the implementation of the function \texttt{gcd} computing the greatest common divisor of two
natural numbers using the \href{http://en.wikipedia.org/wiki/Euclidean_algorithm}{Euclidean algorithm}.
Suppose we want to trace every invocation of the function \texttt{gcd}.  We do so by first creating
an object of class \texttt{tracer} in line 7 of Figure \ref{fig:trace-decorator-use.stlx}.  Then the line
\\[0.2cm]
\hspace*{1.3cm}
\texttt{gcd := myTracer.trace(gcd, "gcd");}
\\[0.2cm]
activates tracing for the function \texttt{gcd}.  Note that the first argument of the method
\texttt{trace} is the function we want to trace, while the second argument is the name of the function.
After executing this assignment statement, the function \texttt{gcd} will be traced.
For example, the call \texttt{gcd(36, 27)} produces the following output:
\begin{verbatim}
    calling gcd(36, 27)
    calling gcd(27, 9)
    calling gcd(9, 0)
    gcd(9, 0) = 9
    gcd(27, 9) = 9
    gcd(36, 27) = 9
    ~< Result: 9 >~
\end{verbatim}
This shows that every time the function \texttt{gcd} is called, a statement of the form
\\[0.2cm]
\hspace*{1.3cm}
\texttt{calling gcd($m$, $n$)}
\\[0.2cm]
is printed.  This is true even for the recursive invocations of \texttt{gcd}.  Furthermore,
every time the function \texttt{gcd} returns a result, this is also printed together with the
arguments used to compute this result.

To stop tracing the function \texttt{gcd}, we simply issue the statement:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{gcd := myTracer.untrace("gcd");}
\\[0.2cm]
After this assignment, the function \texttt{gcd} is restored to its original state and will not print
anything when called.

\begin{figure}[!ht]
\centering
\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    class tracer() {
        mStoredProcedures := {};

        trace := procedure(function, functionName) {
            mStoredProcedures[functionName] := function;
            tracedFunction := closure(*args) {
                argsString := join(args, ", ");
                print("calling $functionName$($argsString$)");
                result := function(*args);
                print("$functionName$($argsString$) = $result$");
                return result;
            };
            return tracedFunction;
        };
        untrace := procedure(functionName) {
            return mStoredProcedures[functionName];
        };
    }
\end{Verbatim}
\vspace*{-0.3cm}
\caption{A class providing a trace decorator.}
\label{fig:trace-decorator.stlx}
\end{figure}

Let us proceed to discuss how the class \texttt{tracer} is implemented in \setlx.  Figure
\ref{fig:trace-decorator.stlx} shows the implementation of this class.  We discuss this
implementation line by line.
\begin{enumerate}
\item Line 2 defines the member variable \texttt{mStoredProcedures}.  This variable will store a map
      that is represented as a binary relation.  The keys of this maps are procedure names, while the
      values entered into this map are the procedures themselves.  Initially, this map is empty.  Every time a
      procedure is traced, the old value of the procedure is stored in this map.  This is needed in
      order to implement the method \texttt{untrace} that restores a given procedure to its original
      state.
\item The workhorse of the class \texttt{tracer} is the method \texttt{trace} defined in line 4.
      This method takes two arguments:  The first argument is a procedure, while the second argument
      is the name of the procedure.  The function will return a new procedure that behaves like the
      given procedure but, furthermore, traces the invocation of the original procedure as discussed
      previously in the example.  The procedure \texttt{trace} works as follows:
      \begin{enumerate}
      \item In line 5, the old value of the procedure to be traced is saved in the dictionary
            \\[0.2cm]
            \hspace*{1.3cm}
            \texttt{mStoredProcedures}.
            \\[0.2cm]
            This value is needed to restore the procedure to its original value when we do not want
            to trace the procedure any more.
      \item Next, line 6 defines a closure called \texttt{tracedFunction}.  This is a function
            called with a variable number of arguments.  The reason is that we do
            not know how many
            arguments the procedure \texttt{function} that is given as argument to the method
            \texttt{trace} actually takes.  In the example discussed above, we had traced the
            function \texttt{gcd} that takes two arguments.  We could just as well have traced a
            procedure taking only one argument as we could have traced a procedure taking nine
            arguments.
      \item In line 7 and 8, the closure \texttt{tracedFunction} constructs a string containing the
            arguments and the name of the function and then prints this string.  This
            \texttt{tracedFunction} refers to \texttt{function} and \texttt{functionName}.  As these
            variables  are only defined in the method \texttt{trace}, the function \texttt{tracedFunction} needs to
            be defined as a \texttt{closure} so that it can access these variables.
      \item Line 9 is the most important line of the whole implementation:  It is here that the
            original procedure is called with the given arguments.  Note how prefixing the list of
            arguments \texttt{args} with the operator ``\texttt{*}'' makes it possible to call this
            function without knowing how many arguments the procedure actually takes.
      \item Line 10 prints the result that has been computed.
      \item This result is then returned in line 11.
      \item Finally, the method \texttt{trace} returns the newly created closure.  By assigning this
            closure to the name of the original procedure we can thus trace this procedure.
      \end{enumerate}
\item As a convenience, the class \texttt{tracer} also provides the method \texttt{untrace} in
      line 16.  This procedure looks up the original procedure stored under the given name in the
      dictionary \texttt{mStoredProcedures} and returns it.  Hence, assigning the value returned by
      this procedure to the name of a procedure stops tracing this procedure.
\end{enumerate}

%%% Local Variables:
%%% Mode: Latex
%%% TeX-master: "tutorial"
%%% End:


\documentclass[ 12pt, a4paper]{article}
% Use the option doublespacing or reviewcopy to obtain double line spacing
% \documentclass[doublespacing]{elsart}

\usepackage[utf8]{inputenc}
\usepackage[backend = biber, maxcitenames=2,uniquelist=minyear]{biblatex}

\AtEveryBibitem{\clearfield{number}}
\AtEveryBibitem{\clearfield{doi}}
\AtEveryBibitem{\clearfield{url}}
\AtEveryBibitem{\clearfield{issn}}
\AtEveryBibitem{\clearfield{isbn}}

\addbibresource{references.bib}

\usepackage{color,graphicx,tikz}
\usetikzlibrary{positioning,arrows}
% The amssymb package provides various useful mathematical symbols
\usepackage{mathtools,amssymb,amsmath,mathdots}
\usepackage[mathscr]{eucal} %just for the font \mathscr
\usepackage{setspace}
\usepackage{hyperref}

\usepackage{tikz}

\renewcommand{\vec}[1]{\boldsymbol{#1}}
\renewcommand{\thefootnote}{\fnsymbol{footnote}}

\input{macros}

\begin{document}

\title{Acoustic multiple scattering}
\author{
Artur L. Gower$^{a}$,\\
\footnotesize{$^{a}$ School of Mathematics, University of Manchester, Oxford Road, Manchester, M13 9PL,UK}
}
\date{\today}
\maketitle

\begin{abstract}
Here we show and deduce the T-matrix and multiple scattering for acoustics. Before reading this document, it may be helpful to read the general multiple scattering formulation shown in \href{multiplescattering.pdf}{multiplescattering.pdf}.
\end{abstract}

\noindent
{\textit{Keywords:} Multiple scattering, T-matrix, Scattering matrix}

\section{2D acoustics}
Much of the notation is define in \href{multiplescattering.pdf}{multiplescattering.pdf}. For the 2D acoustics some good references are \parencite{ganesh_far-field_2010,ganesh_algorithm_2017}.

For 2D acoustics we have that
\begin{align}
  & \mathrm u_{n}(k\rv) = J_{n}(k r) \ee^{\ii n \theta},
  \\
  & \mathrm v_{n}(k\rv)= H_{n}(k r) \ee^{\ii n \theta}.
\end{align}
When truncating up to some order $N$ we would sum over $n = -N, -N +1, \ldots, N-1, N$.

\subsection{Circular cylinder}
Let $\rho$ and $c$ be the background density and wavespeed, and let $\rho_j$, $c_j$ and radius $a_j$ be the mass density, wavespeed, and radius for a circular scatterer with density.

Let $u = \ui + \us$ be the total field outside the particle, and $\vi$ the total field inside the particle, then from the acoustic boundary conditions:
\[
u = \vi, \quad \frac{1}{\rho} \frac{\partial u}{\partial r} = \frac{1}{\rho_j} \frac{\partial \vi}{\partial r}, \quad \text{for} \;\; r = a_j,\
\]
we can deduce the T-matrix
\begin{equation}
  T_{nm} = - \delta_{nm} \frac{\gamma_j J_m' (k a_j) J_m (k_j a_j) - J_m (k a_j) J_m' (k_j a_j) }{\gamma_j H_m '(k a_j) J_m(k_j a_j) - H_m(k a_j) J_m '(k_j a_j)},
  \label{eqn:circular_t-matrix}
\end{equation}
where $\gamma_j = (\rho_j c_j)/(\rho c)$ and $k_j = \omega/c_j$.

We can also calculate the coefficients $b_n$ from
\begin{equation}
  b_n = \frac{f_n}{T_{nn} J_n(k_ja_j)}\left[ T_{nn} H_n(ka_j) + J_n(k a_j) \right]
\end{equation}

\subsection{Circular cylindrical capsule}

\begin{figure}[t]
\centering
  \begin{tikzpicture}[scale=0.6]
  % Outer box
  \draw (-5,-5) -- (5,-5) -- (5,5) -- (-5,5) -- (-5,-5);

  % Concentric circles
  \draw (0,0) circle (2);
  \draw (0,0) circle (3);

  % Line to show a_1
  \draw (-3.02, 0) -- (-0.02,0);
  \draw (-3.02, 0.3) -- (-3.02,-0.3);
  \draw (-0.02, 0.3) -- (-0.02,-0.3);
  \node at (-1.5,-0.3) { $a_1$ };

  % Line to show a_0
  \draw (2.02, 0) -- (0,0);
  \draw (2.02, 0.3) -- (2.02,-0.3);
  \draw (0.02, 0.3) -- (0.02,-0.3);
  \node at (1,-0.3) { $a_0$ };

  \node at (-4,4) { $c,\rho$ };
  \node at (0,1) { $c_0,\rho_0$ };
  \node at (0,2.5) { $c_1,\rho_1$ };

  \end{tikzpicture}

  \label{fig:capsule}
\end{figure}

\begin{align}
  & \psi^0 = \sum_{n=-\infty}^\infty g_n^0 J_{n}(k_0 r) \ee^{\ii n \theta},
  \\
  & \psi^1 = \sum_{n=-\infty}^\infty \left [ g_n^1 J_{n}(k_1 r) + f_n^1 H_{n}(k_1 r) \right ] \ee^{\ii n \theta}.
\end{align}
Applying the boundary conditions,
\begin{align}
  	& \psi^0 = \psi^1 \quad \text{and} \quad \frac{1}{\rho_0} \frac{\partial \psi^0}{\partial r} = \frac{1}{\rho_1} \frac{\partial \psi^1}{\partial r}, \quad \text{on} \;\; r = a_0,
    \label{eqn:inner_bc}
    \\
  	& \psi^1 = \psi^\scat + \psi^\inc \quad \text{and} \quad \frac{1}{\rho_1} \frac{\partial \psi^1}{\partial r} = \frac{1}{\rho} \frac{\partial (\psi^\scat + \psi^\inc)}{\partial r}, \quad \text{on} \;\; r = a_1.
    \label{eqn:outer_bc}
\end{align}
Solving these boundary conditions (see \href{capsule-boundary-conditions.nb}{capsule-boundary-conditions.nb}) leads to
\begin{multline}
  T_{nn} = - \frac{J_n(k a_1)}{H_n(k a_1)} - \frac{Y^n_{'}(k a_1, k a_1)}{H_n(ka_1)} \left[Y^n(k_1 a_1,k_1 a_0) J_n'(k_0 a_0) - q_0 J_n(k_0 a_0) Y^n_{'}(k_1 a_1,k_1 a_0) \right]
  \\ \times \big[
    J_n'(k_0 a_0)(q H_n(k a_1)Y^n_{'}(k_1 a_0,k_1 a_1) + H_n'(k a_1) Y^n(k_1 a_1,k_1 a_0))
    \\
    + q_0 J_n(k_0 a_0)(q H_n(k a_1)Y^n_{''}(k_1 a_1,k_1 a_0) - H_n'(k a_1) Y^n_{'}(k_1 a_1, k_1 a_0))
  \big]^{-1}.
\end{multline}
where $q = \rho c/(\rho_1 c_1)$, $q_0 = \rho_0 c_0/( \rho_1 c_1)$, and
\begin{align}
  & Y^n(x,y) = H_n(x) J_n(y) - H_n(y) J_n(x), \\
  & Y^n_{'}(x,y) = H_n(x) J_n'(y) - H_n'(y) J_n(x), \\
  & Y^n_{''}(x,y) = H_n'(x) J_n'(y) - H_n'(y) J_n'(x).
\end{align}

% Using~\eqref{eqn:inner_bc} we get
% \begin{align}
%   & g_n^0 J_{|n|}(k_0 a_0) = g_n^1 J_{|n|}(k_1 a_0) + a_n^0 H_{|n|}(k_1 a_0),
%   \\ \notag
%   & g_n^1 \left [ \frac{J_{|n|}'(k_0 a_1)}{\rho_0 c_0} \frac{J_{|n|}(k_1 a_0)}{J_{|n|}(k_0 a_0)} - \frac{J_{|n|}'(k_1 a_1)}{\rho_1 c_1} \right ] =
%   a_n^1 \left [ \frac{J_{|n|}'(k_0 a_1)}{\rho_0 c_0} \frac{H_{|n|}(k_1 a_0)}{J_{|n|}(k_0 a_0)} - \frac{H_{|n|}'(k_1 a_1)}{\rho_1 c_1} \right ].
% \end{align}
%
% \begin{gather}
%    g_n^1 J_{|n|}(k_1 a_1) + a_n^1 H_{|n|}(k_1 a_1) =  g_n J_{|n|}(k a_1) + a_n H_{|n|}(k a_1)
%    \\
%    g_n^1 \frac{1}{\rho_1 c_1} J_{|n|}'(k_1 a_1) + a_n^1 \frac{1}{\rho_1 c_1} H_{|n|}'(k_1 a_1)
%    =  g_n \frac{1}{\rho c} J_{|n|}'(k a_1) + a_n \frac{1}{\rho c} H_{|n|}'(k a_1)
% \end{gather}

\subsection{Multiple scattering in 2D}

Graf's addition theorem in two spatial dimensions:
\begin{align}
  & H_n(k R_\ell)\ee^{\ii n \Theta_\ell} =
  \sum_{m=-\infty}^\infty H_{n-m}(k R_{\ell j})\ee^{\ii(n-m)\Theta_{\ell j}} J_{m}(k R_j)\ee^{\ii m \Theta_j}, \;\;\text{for}\;\; R_j < R_{\ell j},
\label{eqn:Graf}
\end{align}
where $(R_{\ell j},\Theta_{\ell j})$ are the polar coordinates of $\vec r_j - \vec r_\ell$. The above is also valid if we swap $H_n$  for $J_n$, and swap $H_{n-m}$ for $J_{n-m}$.

Particle-$j$ scatters a field
\begin{equation}
  \label{eqn:outwaves}
  u_j = \sum_{n} f_n^j \mathrm u_{n}(k\rv - k \rv_j), \quad \text{for} \;\; |\rv - \rv_j| > a_j,
\end{equation}
% where $(R_j,\Theta_j)$ are the polar coordinates of $\vec r - \vec r_j$,
where $\vec r_j$ is the centre of particle $j$.

Let the incident wave, with coordinate system centred at $\vec r_j$, be
\begin{equation}
  \label{eqn:incident}
  \ui = \sum_{n} g_n^j \mathrm v_{n}(k\rv - k \rv_j),
\end{equation}
then the wave exciting particle-$j$ is
\begin{equation}
  \label{eqn:exciter}
  u_j^E = \sum_{n} F^n_j \mathrm v_{n}(k\rv - k \rv_j)
\end{equation}
where
\begin{equation}
  F_n^j = g_n^j + \sum_{\ell\not = j} \sum_{p=-\infty}^\infty f_p^\ell H_{p-m}(k R_{\ell j})\ee^{\ii(p-m)\Theta_{\ell j}}.
\end{equation}
Using the T-matrix of particle-$j$ we reach $f_n^j = \sum_m T_{nm}^j F_m^j$, which leads to
\begin{equation}
f_q^j  = \sum_{m} T_{qm}^j g_m^j + \sum_{\ell\not = j} \sum_{m,p=-\infty}^\infty f_p^\ell T_{qm}^j H_{p-m}(k R_{\ell j})\ee^{\ii(p-m)\Theta_{\ell j}}.
\label{eqn:As}
\end{equation}
The above simplifies if we substitute $f_q^j =  T_{qd}^j \alpha_d^j$, and then multiple across by $\{T_{qn}^j\}^{-1}$ and sum over $q$ to arrive at
% f_\ell^p = \alpha_\ell^d T^{dp}_\ell
% f_\ell^p = \alpha_\ell^d T^{dp}_\ell
\begin{equation}
\alpha_n^j = g_n^j + \sum_{\ell\not = j} \sum_{m,p=-\infty}^\infty  H_{p-n}(k R_{\ell j})\ee^{\ii(p - n)\Theta_{\ell j}} T_{pm}^\ell \alpha_m^\ell.
\label{eqn:As}
\end{equation}
As a check, if we use~\eqref{eqn:circular_t-matrix}, then we arrive at equation (2.11) in \cite{gower_reflection_2017}.

In the general formulation below we would have
\[
\mathcal U_{n'n}(k R_{\ell j}) = H_{n'-n}(k R_{\ell j})\ee^{\ii(n'-n)\Theta_{\ell j}}.
\]
Note that swapping $\ell$ for $j$ would result in $\Theta_{\ell j} = \Theta_{j \ell } + \pi$.
% \begin{equation}
% f_j^n  = -  f^n_j - \sum_{\ell\not = j} \sum_{p=-\infty}^\infty f_\ell^p Z^p_\ell H_{p-n}(k R_{\ell j})\ee^{\ii(p-n)\Theta_{\ell j}}.
% \end{equation}

\section{3D acoustics}

For all the details on acoustics in three spatial dimensions see~\cite{gower2020effective}. Here we all only provide:
\begin{equation}
\left\{\begin{aligned}
    & \mathrm u_{n}(k\rv) = {\mathrm{h}}_\ell^{(1)}(kr)\mathrm{Y}_{n}(\rvh),
    \label{eqn:outgoing_waves_and_regular_waves}
    \\
    & \mathrm v_{n}(k\rv) = \mathrm{j}_\ell(kr)\mathrm{Y}_{n}(\rvh),
 \end{aligned}\right.
\end{equation}
where $r=|\rv|$, $n=\{\ell,m\}$, with summation being over $\ell=0,1,2,3\ldots$ and $m=-\ell,-\ell+1,\ldots,-1,0,1,\ldots,\ell$, and the spherical Hankel and Bessel functions are denoted ${\mathrm{h}}_\ell^{(1)}(z)$ and $\mathrm{j}_\ell(z)$, respectively.

\section{A sphere}
Let $\rho$ and $c$ be the background density and wavespeed, then for a spherical particle with density $\rho_j$, soundspeed $c_j$ and radius $a_j$, we have that
\begin{equation}
  T_{nq} = - \delta_{nq} \frac{\gamma_j j_q' (k a_j) j_q (k_j a_j) - j_q (k a_j) j_q' (k_j a_j) }{\gamma_j h_q '(k a_j) j_q(k_j a_j) - h_q(k a_j) j_q'(k_j a_j)},
  \label{eqn:circular_t-matrix}
\end{equation}
where $\gamma_j = (\rho_j c_j)/(\rho c)$ and $k_j = \omega/c_j$.

\printbibliography

\end{document}


\chapter{Related Work}
\paragraph{Homotopy Type Theory} \cite{hottbook}
\paragraph{Cubical Type Theory} \cite{cohenCubicalTypeTheory2016}
\paragraph{Cubical Agda} \cite{vezzosiCubicalAgdaDependently2019}
\paragraph{Constructive Finiteness}
\begin{itemize}
  \item First paper on the topic, defines 4 notions of finiteness (split
    enumerability, there called enumerated, bounded, Noetherian, streamless):
    \cite{coquandConstructivelyFinite2010}
  \item More exploration of Noetherianness
    \cite{firsovVariationsNoetherianness2016}
  \item More exploration of streamless sets
    \cite{parmannInvestigatingStreamlessSets2015} (in particular closure under
    product).
  \item Paper exploring programming with finite sets for e.g. proof search
    \cite{firsovDependentlyTypedProgramming2015} (basically only enumerable sets
    though, only in MLTT)
  \item Finite sets in Homotopy Type Theory, especially Kuratowski
    \cite{fruminFiniteSetsHomotopy2018} (but no finite function arrows).
  \item Kuratowski's original paper on finiteness
    \cite{kuratowskiNotionEnsembleFini1920}.
  \item \cite{smolkaHereditarilyFiniteSets2016}.
\end{itemize}
\paragraph{Sets/Toposes}
\begin{itemize}
  \item Paper that sets in HoTT form a topos (under certain conditions
    etc) \cite{rijkeSetsHomotopyType2015}.
    This paper is adapted into a chapter in the HoTT book.
  \item Category theory in cubical Agda
    \cite{iversenUnivalentCategoriesFormalization2018}.
  \item Topos from cardinal finite \cite{henryToposesGeneratedCardinal2018}.
  \item Category of finite sets \cite{solovevCategoryFiniteSets1983}.
\end{itemize}
\paragraph{Species}
\begin{itemize}
  \item Brent Yorgey's thesis \cite{yorgeyCombinatorialSpeciesLabelled2014}.
  \item \cite{uszkaySpeciesMakingAnalytic2008}
\end{itemize}
\paragraph{Exhaustability}
\begin{itemize}
  \item Definition of limited principle of omniscience:
    \cite{bishopFoundationsConstructiveAnalysis1967}.
  \item \cite{escardoExhaustibleSetsHighertype2008}
  \item \cite{escardoInfiniteSetsThat2007}
  \item \cite{escardoInfiniteSetsThat2013}
\end{itemize}
\paragraph{Propositional Truncation algo}
\cite{krausGeneralUniversalProperty2015}
\paragraph{Countdown}
\begin{itemize}
  \item \cite{huttonCountdownProblem2002}
  \item \cite{birdCountdownCaseStudy2005}
  \item \cite{birdFunctionalPearlTrouble2003}
\end{itemize}
\paragraph{Generate and Test}
\begin{itemize}
  \item \cite{claessenQuickCheckLightweightTool2011}
  \item \cite{runcimanSmallCheckLazySmallCheck2008}
  \item \cite{oconnorApplicationsApplicativeProof2016}
  \item (for the generator syntax) \cite{allaisGenericLevelPolymorphic2019}.
\end{itemize}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../paper"
%%% End:


\section*{List of Symbols}
\begin{table}[H]
    %\caption{\label{tb:los} A list of coordinates, which was used to generate the different degrees of deformation.}
    \centering
\begin{tabular}{c | c }
    \toprule
    \textbf{symbol} & \textbf{meaning} \\ [0.5ex]
    \midrule
    h & height\\
    J & Jacobian matrix\\
    m & Weibull module\\
    N & interpolation matrix\\
    NoE & number of elements\\
    NoN & number of nodes\\
    n & shape function\\
    $\sigma$ & stress\\
    V & Volume\\
    x, y & global coordinates\\
    $\xi$, $\eta$ & natural/local coordinates\\[1ex]
    \bottomrule
\end{tabular}
\end{table}

\section*{List of Indices}
\begin{table}[H]
    \centering
\begin{tabular}{c | c}
    \toprule
\textbf{index} & \textbf{meaning} \\ [0.5ex]
\midrule
$eff$ & effective\\
$f$ & rapture or failure\\
i & element number\\
j & node index in element\\
k & index of principle stress\\
$max$ & maximum\\ [1ex]
\bottomrule
\end{tabular}
\end{table}


\pdfbookmark[0]{Abstract}{Abstract}
\section*{\Huge Abstract}
\label{sec:abstract}
With the increasing amount of high-throughput sequencing data becoming available, the proper integration of differently sized and heterogeneous molecular and clinical blocks of variables is becoming more and more critical in cancer survival models. Still, the Cox Proportional-Hazards model (Cox PH) using only clinical data remains one of the best performing methods due to the difficulty of building models that can successfully integrate omics data with clinical information \citep{herrmann2021large}. This finding motivates the need for new models which can successfully perform multi-omics integration in survival models. Furthermore, there is a need to make multi-omics models more interpretable to encourage their usage in clinical settings.

We developed several novel neural architectures based on supervised hierarchical autoencoders combined with \glsxtrfull{sgl} regularization. Our best architecture performed competitively with the best performing statistical models used for multi-omics integration as measured by Harrell's concordance. We also outperformed the Cox Proportional Hazards model using only clinical data across the $17$ \glsxtrfull{tcga} cancers considered within our study. We further studied the information captured by our best model by visualizing its latent space and showed that our model could learn relationships between clinical and, if necessary, molecular variable groups. We also proved that global surrogate modeling of multi-omics methods enables the usage of arbitrarily simple regression methods while maintaining high performance. In addition, global surrogate models provided high group-level and overall sparsity and thus an overall more interpretable model. Lastly, we investigated to what extent multi-omics integration could be reduced to a feature selection problem, that is, whether the performance differences between algorithms would vanish if all models were trained on a particular (small) subset of input variables.

All in all, our work proved that supervised autoencoders could perform competitively with the state-of-the-art statistical models in multi-omics survival analysis, even without transfer learning. In addition, we validated that these models learned good representations, although we caution against over-relying on this point. Also, we showed that global surrogate modeling was an effective technique for transferring multi-omics survival models to clinical settings.

Our full implementation is available at https://github.com/dnwissel/msc\_thesis. The preprocessed datasets are available at https://www.kaggle.com/davidwissel/shae-data.


\documentclass{article}

\usepackage[left=1in,right=1in,top=1in,bottom=1in]{geometry}
\usepackage{epsfig}
\usepackage{amsmath}
\usepackage{subfigure}
\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}
\newcommand{\Hrule}{\rule{\linewidth}{0.3mm}}

\makeatletter% since there's an at-sign (@) in the command name
\renewcommand{\@maketitle}{
  \parindent=0pt% don't indent paragraphs in the title block
  \begin{center}
    \MakeUppercase{\Large \bf \@title}
    \HRule%
  \end{center}%
  \textit{\@author \hfill \@date}
  \par
}
\makeatother% resets the meaning of the at-sign (@)

\title{System Identification for IFFL}
\author{\bf SUMIT MUKHERJEE}
\date{\bf  mukhes3@uw.edu}

\begin{document}

\maketitle% prints the title block
\vspace{5 mm}

\section{Introduction}

The aim of this work is to identify the model parameters for the different cell lines implementing the Incoherent Feed Forward Loop (IFFL). The parameters have then been used in gillespie simulations to generate stochastic trajectories.

\subsection{Identification Data Set}

Provided by Alex Rosenberg. Comprises of several cell lines :-
\begin{itemize}
\item Open Loop
\item Containing 4 sites complimentary to mir-124a (pv)
\item Containing 3 sites (d3)
\item Containing 2 sites (d23)
\item Containing 1 site (d123)
\end{itemize}

The data set has different time evolving values of mRNA, miRNA (both from FISH and qPCR) and protein (mCherry) levels for each of the cell lines.

\section{Model}

Initial assumed model :-
\begin{align*}
\frac{dm}{dt} &= \alpha_{m} - \beta_m m - \gamma_s m s \\
\frac{ds}{dt} &= \alpha_s - \beta_s s \\
\frac{dp}{dp} &= \alpha_p m - \beta_p p
\end{align*}

Where, $m$ is the mRNA concentration, $s$ is the miR-124a concentration, $p$ is the protein concentration, $\alpha_m$ is mRNA production rate, $\alpha_s$ is the miRNA production rate, $\alpha_p$ is the protein production rate (not constitutive), $\beta_m$ is the mRNA degradation rate, $\beta_s$ is the miRNA degradation rate, $\beta_p$ is the protein degradation rate and $\gamma_s$ is the miRNA degradation rate for mRNA.

\section{Parameter Identification}

\subsection{Cost Function}
We observe that the time evolution of the miRNA is independent of the mRNA and protein concentration, so we identify it separately and club the mRNA and protein into one identification routine.  The cost function used is :-
\begin{align*}
J = \sum\limits_{j = 1}^{j = l}  a_j \sum\limits_{i = 1}^{i = k} \frac{{(x_{j}(i) - x_{act,j}(i))}^2}{{x_{act,j}(k)}^2}
\end{align*}
Where, $x_j(i)$ represents the simulated concentration of $j$th specie at $i$th time point, $x_{act,j}(i)$ represents the actual concentration of $j$th specie at $i$th time point, $k$ is the number of time points, $l$ is the number of species and $a_j$ is the weight on the $j$th specie. The term in the denominator ensures that the species get normalized (as mRNA, protein and miRNA concentrations are often vastly different). The parameters being optimized here are $\alpha_m, \alpha_s, \alpha_p, \beta_m, \beta_s, \beta_p, \gamma_s$.

\subsection{Parameter Limits}

The decay rate limits are chosen based on practical values of half lifes and are chosen to be within one order of magnitude each way. The maximal limit on the production limit is chosen to be a realistic value based on the plots. The limits used for the various parameters are given in Table~\ref{MinMaxTable} :-

\begin{table}[h!]
\begin{center}
\begin{tabular}{|l|l|l|}
\hline
Parameter & Min & Max \\\hline
$\beta_m$ (1/h) &.01 &.5 \\\hline
$\beta_s$ (1/h) &.02 &.04 \\\hline
$\beta_p$ (1/h) &.001 &1 \\\hline
$\alpha_m$ (conc./h) &100 &3000 \\\hline
$\alpha_s$ (conc./h) &0 &500 \\\hline
$\alpha_p$ (1/h) &0 &5000 \\\hline
$\gamma_s$ (1/h conc.) &0 &.00001 \\\hline
\end{tabular}
\end{center}
\caption{Minimum and Maximum values of parameters}
\label{MinMaxTable}
\end{table}

\subsection{Identified Parameters}

The identified parameters are given in Table~\ref{paramTable}.

\begin{table}[h!]
\begin{center}
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Parameter & PV & d3 & d23 & d123 & in paper\\\hline
$\beta_m$ (1/h) &0.26645&0.13174&0.5&0.21695 &.1\\\hline
$\beta_s$ (1/h) &0.029091&0.028633&0.02363&0.04 & .03\\\hline
$\beta_p$ (1/h) &0.054131&0.03673&0.030614&0.018773 & .08\\\hline
$\alpha_m$ (conc./h) &524.9764&517.0953&208.5493&270.1188 &0-150\\\hline
$\alpha_s$ (conc./h) &50.0004&70.7653&22.036&49.0797 & 0-60\\\hline
$\alpha_p$ (1/h) &1.5638&1.1415&3.3885&1.6738 & 10\\\hline
$\gamma_s$ (1/h conc.) &0.001&0.00051711&0.00097197&0.00028637 & .00075\\\hline
\end{tabular}
\end{center}
\caption{Identified values for different parameters}
\label{paramTable}
\end{table}

\subsection{Plots}
The plots for each of the classes are shown below :-
\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv/model4mRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv/model4miRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv/model4pcont.jpg}}
\caption{Simulation vs Real plots for PV : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsPV}
\end{figure}

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d3/model4mRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d3/model4miRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d3/model4pcont.jpg}}
\caption{Simulation vs Real plots for d3 : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsd3}
\end{figure}

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d23/model4mRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d23/model4miRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d23/model4pcont.jpg}}
\caption{Simulation vs Real plots for d23 : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsd23}
\end{figure}

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d123/model4mRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d123/model4miRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_d123/model4pcont.jpg}}
\caption{Simulation vs Real plots for d123 : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsd123}
\end{figure}

\pagebreak

\subsection{Open Loop Plots and Parameters}

Table~\ref{paramTableOpenFish} shows the values of the parameters identified with the open loop data (the mRNA data is used from the FISH data for open loop). The plots of the open loop identification are shown below in Figure~\ref{resultsP}.

\begin{table}[h!]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Parameter & P \\\hline
$\beta_m$ (1/h) &0.3169\\\hline
$\beta_s$ (1/h) &0.0247\\\hline
$\beta_p$ (1/h) &0.0201\\\hline
$\alpha_m$ (conc./h) &140.7183\\\hline
$\alpha_s$ (conc./h) &51.0592\\\hline
$\alpha_p$ (1/h) &6.5936\\\hline
$\gamma_s$ (1/h conc.) &0\\\hline
\end{tabular}
\end{center}
\caption{Identified values for different parameters}
\label{paramTableOpenFish}
\end{table}

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_p_FISH/model4mRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_p_FISH/model4miRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_p_FISH/model4pcont.jpg}}
\caption{Simulation vs Real plots for P (open loop) : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsP}
\end{figure}

\subsection{Re-identification with FISH data}

We have mRNA data from FISH (Fluorescence in situ hybridization) for both P (open loop) and  PV cell lines. The re-identification of the PV cell line yields the parameter values as seen in table and the plots can be seen in figure

\begin{table}[h!]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Parameter & P \\\hline
$\beta_m$ (1/h) &0.2615\\\hline
$\beta_s$ (1/h) &0.0290\\\hline
$\beta_p$ (1/h) &0.0514\\\hline
$\alpha_m$ (conc./h) &194.9786\\\hline
$\alpha_s$ (conc./h) &49.967\\\hline
$\alpha_p$ (1/h) &3.9719\\\hline
$\gamma_s$ (1/h conc.) &0\\\hline
\end{tabular}
\end{center}
\caption{Identified values for different parameters}
\label{paramTableOpenFish}
\end{table}

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4mRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4miRNAcont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4pcont.jpg}}
\caption{Simulation vs Real plots for PV : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsPV_Fish}
\end{figure}

\subsection{Open loop plots with PV-FISH parameters}

Theoretically, the open loop plot can be obtained by setting $gs = 0$ and making the initial conditions to be the same as the initial conditions for the open loop case. This yields the plots seen in Figure~\ref{resultsPV_FishOpen}.

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4mRNAopencont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4miRNAopencont.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4popencont.jpg}}
\caption{Simulation vs Real plots for PV : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsPV_FishOpen}
\end{figure}

Clearly this does not lead to very well fitted plots. We assume that this could be a result of a translation inhibition. If only that was the case, the mRNA levels would roughly remain the same but the steady state value seems of the open loop case seems to be around half the steady state value of the PV cell line. Hence we fix all other parameters and just vary $\alpha_p$ and $\alpha_m$ to see if we can better fits. The plots obtained are shown in figure

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4mRNAopencontupdated.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4miRNAopencontupdated.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4popencontupdated.jpg}}
\caption{Simulation vs Real plots for PV : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsPV_FishUpdated}
\end{figure}

The identified values of parameters are shown in table

\begin{table}[h!]
\begin{center}
\begin{tabular}{|l|l|}
\hline
Parameter & P \\\hline
$\alpha_m$ (conc./h) &104.91\\\hline
$\alpha_p$ (1/h) &11.3891\\\hline
\end{tabular}
\end{center}
\caption{Identified values for different parameters}
\label{paramTableOpenFishUpdated}
\end{table}

This yields slightly better fits. However, we are interested to see whether this is just due to cell line variability or whether translation inhibition has a role to play here. So, now we just vary the $\alpha_m$ (change it to re-identified value) and keep everything else fixed (to the first case) and observe the results in Figure

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4mRNAopencontupdated2.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4miRNAopencontupdated2.jpg}}
\subfigure[]{\includegraphics[scale=.1]{IFFL_Alex_pv_FISH/model4popencontupdated2.jpg}}
\caption{Simulation vs Real plots for PV : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
\label{resultsPV_FishUpdated}
\end{figure}

So clearly, there is translation inhibition happening which causes the $\alpha_p$ value to be higher in the re-identified case.

%\section{Stochastic Simulations}
%
%Stochastic simulations were performed on the Control and Full Vamp3 strains using the Tau Leaping method to calculated the statistics. The obtained images are shown below :-
%\begin{figure}[h!]
%\centering
%\subfigure[]{\includegraphics[scale=.1]{TauL_IFFL_FISH_p/m}}
%\subfigure[]{\includegraphics[scale=.1]{TauL_IFFL_FISH_p/s}}
%\subfigure[]{\includegraphics[scale=.1]{TauL_IFFL_FISH_p/p}}
%\subfigure[]{\includegraphics[scale=.1]{TauL_IFFL_FISH_pv/m}}
%\subfigure[]{\includegraphics[scale=.1]{TauL_IFFL_FISH_pv/s}}
%\subfigure[]{\includegraphics[scale=.1]{TauL_IFFL_FISH_pv/p}}
%\caption{Tau Leaping plots for Control : a) mRNA b) miRNA (mir-124a) c) protein (mCherry), Tau Leaping plots for Full Vamp3 : a) mRNA b) miRNA (mir-124a) c) protein (mCherry)}
%\label{stochPV_FishUpdated}
%\end{figure}

\section{Identification at multiple Dox levels}

Protein
fluorescence data from at different Dox induction levels was available for various cell lines. Unfortunately the miRNA and mRNA time course data aren't available. Thus the problem reduces to finding
the initial conditions along with the parameters. However, it is expected that the other parameters remain
the same and only the $a_m$ and $b_m$ varies. However, their ratio is assumed to be constant, so we need to only identify $a_m$ for different Dox level. So, the optimization variables are m0, s0 and $a_m$. The plots for the
protein fits is seen in Figure~\ref{MultiDoxFits} for the Full Vamp3 cell line. The predicted plot for
fluorescence and mRNA levels at different
Dox induction levels is shown in the Figure~\ref{MultiDoxpredict}.

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox800pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox1000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox1500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox2000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox2500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox5000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox15000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_pv_mod/Dox1000000pCont}}
\caption{Protein conc. fits at different Dox induction level a)800 ng b)1$\mu g$ c)1.5$\mu g$ d)2$\mu g$ e) 2.5$\mu g$ f)5$\mu g$ g)15$\mu g$ h)1$mg$}
\label{MultiDoxFits}
\end{figure}
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.15]{MultiDox_pv_mod/multidoxplotmrna}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_pv_mod/multidoxplot}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_pv_mod/multidoxplotActualIncl}}
\caption{Predicted plots at different Dox induction levels for a) mRNA b) Protein c) Protein with observed data}
\label{MultiDoxpredict}
\end{figure}

The same is performed for the open loop data. The plots for the
protein fits is seen in Figure~\ref{MultiDoxFitsOpen} for the control cell line. The predicted plot for fluorescence and mRNA levels at different Dox induction levels is shown in the Figure~\ref{MultiDoxpredictOpen}.
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox800pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox1000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox1500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox2000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox2500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox5000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox15000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_p/Dox1000000pCont}}
\caption{Protein conc. fits at different Dox induction level a)800 ng b)1$\mu g$ c)1.5$\mu g$ d)2$\mu g$ e) 2.5$\mu g$ f)5$\mu g$ g)15$\mu g$ h)1$mg$}
\label{MultiDoxFitsOpen}
\end{figure}
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.15]{MultiDox_p/multidoxplotmrna}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_p/multidoxplot}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_p/multidoxplotActualIncl}}
\caption{Predicted plots at different Dox induction levels for a) mRNA b) Protein c) Protein with observed data}
\label{MultiDoxpredictOpen}
\end{figure}
\pagebreak

The same analysis is now performed on two other cell lines (d3 and d23) and can fe seen in Figures~\ref{MultiDoxFitsd3},~\ref{MultiDoxpredictd3},~\ref{MultiDoxFitsd23},~\ref{MultiDoxpredictd23}.
\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox800pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox1000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox1500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox2000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox2500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox5000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox15000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d3/Dox1000000pCont}}
\caption{Protein conc. fits at different Dox induction level a)800 ng b)1$\mu g$ c)1.5$\mu g$ d)2$\mu g$ e) 2.5$\mu g$ f)5$\mu g$ g)15$\mu g$ h)1$mg$}
\label{MultiDoxFitsd3}
\end{figure}
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d3/multidoxplotmrna}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d3/multidoxplot}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d3/multidoxplotActualIncl}}
\caption{Predicted plots at different Dox induction levels for a) mRNA b) Protein c) Protein with observed data}
\label{MultiDoxpredictd3}
\end{figure}
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox800pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox1000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox1500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox2000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox2500pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox5000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox15000pCont}}
\subfigure[]{\includegraphics[scale=.1]{MultiDox_d23/Dox1000000pCont}}
\caption{Protein conc. fits at different Dox induction level a)800 ng b)1$\mu g$ c)1.5$\mu g$ d)2$\mu g$ e) 2.5$\mu g$ f)5$\mu g$ g)15$\mu g$ h)1$mg$}
\label{MultiDoxFitsd23}
\end{figure}
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d23/multidoxplotmrna}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d23/multidoxplot}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d23/multidoxplotActualIncl}}
\caption{Predicted plots at different Dox induction levels for a) mRNA b) Protein c) Protein with observed data}
\label{MultiDoxpredictd23}
\end{figure}

The comparison of the the steady state protein level with varying levels of Dox induction for the four different cell lines are shown in Fig~\ref{DoxVsSteadyState} :-
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.15]{MultiDox_pv_mod/DoxVsProteinIFFL}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_p/DoxVsProteinIFFL}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d3/DoxVsProteinIFFL}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d23/DoxVsProteinIFFL}}
\caption{Dox Vs Steady State protein values for a) PV b) P c)d3 d)d23}
\label{DoxVsSteadyState}
\end{figure}

\subsection{Model for $\alpha_m$ variance with induction}
We then hypothesize that the $\alpha_m$ values will have a similar functional relationship with Dox induction levels. For the purpose of the study we assume two models i) $\alpha_m = \frac{a x}{b x + 1}$, ii) $\alpha_m = \frac{a x^n}{b x^n + 1}$. The matlab function 'fit' was used to fit these models. The plots are seen in Figure~\ref{DoxVsAmPred}.    The fits are seen in Figure~\ref{DoxVsAmPred}.
\pagebreak

\begin{figure}[h!]
\centering
\subfigure[]{\includegraphics[scale=.15]{MultiDox_pv_mod/AmVsDoxPred}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_p/AmVsDoxPred}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d3/AmVsDoxPred}}
\subfigure[]{\includegraphics[scale=.15]{MultiDox_d23/AmVsDoxPred}}
\caption{Dox Vs $\alpha_m$ for a) PV b) P c)d3 d)d23}
\label{DoxVsAmPred}
\end{figure}

\begin{table}[h!]
\begin{center}
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Cell Line & Model Type & a & b & n & SSE \\\hline
PV & Linear & 0.0812 & 5.82e-4 & 1 & 385.5\\\hline
PV & Nonlinear & 0.0053 & 4.04e-5 & 1.3825 & 186.6\\\hline
P & Linear & 0.0798 & 6.644e-4 & 1 & 471.7\\\hline
P & Nonlinear & 0.0241 & 2.061e-4 & 1.17 & 433.7\\\hline
d3 & Linear & 0.205 & 4.61e-4 & 1 & 9913.7\\\hline
d3 & Nonlinear & 0.0114 & 2.73e-5 & 1.405 & 8044.7\\\hline
d23 & Linear & 0.1362 & 5.89 e-4 & 1 & 1183.6 \\\hline
d23 & Nonlinear &0.0031 & 1.42e-5 & 1.53 & 183.3 \\\hline
\end{tabular}
\end{center}
\caption{Parameter Values for $\alpha_m$ Vs Dox plots}
\label{CFam}
\end{table}

\pagebreak

\section{Incorporating Translation Inhibition into Models}

It has been demonstrated in several papers that miRNA's not only bind to mRNA's causing them to degrade but also play a role in translational inhibition. This is demonstrated in Figure~\ref{miRNAtrans}.
\begin{figure}[h!]
\centering
\includegraphics[scale=.8]{DocPics/miRNAtrans}
\caption{Modes of translation inhibition in miRNAs}
\label{miRNAtrans}
\end{figure}
\subsection{Modeling}
Based on Figure~\ref{miRNAtrans}, miRNA not only causes enhanced mRNA degradation by binding to it but also competes for cap binding and causes ribosome dropoff once they are attached to a mRNA. Both these effects cause the translation rate to decrease for the fraction of mRNA's that are attached to the miRNA's.

\section{Future Direction}

\begin{itemize}
\item Identify the exact order and mechanism of miRNA based repression
\item Come up with a mathematical model to capture the translation inhibition
\item Demonstrate how this could work out to our benefit in synthetic control circuits
\end{itemize}

\end{document}


%\documentclass[handout]{beamer}
\documentclass{beamer}

\usepackage[utf8]{inputenc}
%%%%%%%%%%%%%%%% math packages %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{amssymb, amsmath}
\usepackage{amsthm}
\usepackage{bm}
\usepackage{comment}

%%%%%%%%%%%%%%%% to insert images %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{graphicx}
\graphicspath{{./images/}}

%%%%%%%%%%%%%%%% use costum theme %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usetheme{metropolis}
\setbeamercolor{block title}{bg=red!20}

%%%%%%%%%%%%%%%% to insert diagram %%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{smartdiagram}
%%%%%%%%%%%%%%%%%%%%%% declaration for argmax %%%%%%%%%%%%%%%%%
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\title{English Word Suggestion Based on Part of Speech Ngram}
\subtitle{Hamana Laboratory, Gunma University}
\author{Borann Chanrathnak}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Table of Contents}
\tableofcontents
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Motivation}
\begin{frame}{Motivation}
    \begin{itemize}
        \item Word suggestion for note-taking app on mobile devices
    \end{itemize}
    \centering
    \only<1>{\includegraphics[height=7cm, width=6cm]{one}}
    \only<2>{\includegraphics[height=7cm, width=6cm]{two}}
    \only<3>{\includegraphics[height=7cm, width=6cm]{three}}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{comment}
    \begin{frame}{How to \textcolor[rgb]{0.8,0.2,0}{Predict}?}
        \centering

        "I am japanese, so I speak" $\rightarrow$ \visible<2->{"japanese"}
    \end{frame}
\end{comment}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Part Of Speech Ngram (POS-Ngram)}
\subsection{Definitions}

\begin{frame}{What does POS-Ngram mean?}
    \visible<1->{
        \textbf{N-gram}
        is a contiguous sequence of n items from a given sample of text or speech\\
            \textbf{Ex: } I study english
            \begin{itemize}
                \item \textbf{unigram (1-gram)} : (I,), (study,), (english,)
                \item \textbf{bigram (2-gram)} : (I, study), (study, english)
                \item \textbf{trigram (3-gram)} : (I, study, english)
            \end{itemize}
        }

    \visible<2->{
        \textbf{N-gram model}
        is one of statistical langauge models for predicting the next item (word) based on Markov assumption, and usually abbreviated as N-gram.
    }

    \visible<3->{
        \textbf{POS-Ngram}
            is an improved model using part of speech as a class indicator.
        }
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{N-gram}

\begin{frame}{How To Compute Probability of a Sentence?}
    How can we compute the joint probability of words in a sentence?\\
    \textbf{Ex:} "an apple is on the table" \\
    $$P(an\ apple\ is\ on\ the\ table) = \text{?}$$

\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Notation}
    \begin{block}{\textcolor[rgb]{0.8,0.2,0}{Notation}}
        \visible<1->{\begin{itemize}
            \visible<2->{\item To represent the probability of a particular random variable $X_i$ taking on the value "the", or $P(X_i="the")$ we will use simplification $P(the)$.}
            \visible<3->{\item We represent a sequence of N words either as $w_1\cdots w_n$ or $w_1^n$}
            \visible<4->{\item \textbf{Ex:} "an apple is on the table" \\$\rightarrow$ $w_1="an"$, $w_2="apple"$, $w_3="is", \cdots$\\
                \visible<5->{$\rightarrow w_1^4="an\ apple\ is\ on"$}}
        \end{itemize}}
    \end{block}

\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{Chain Rule Of Probability}
    \visible<1->{\begin{block}{In General}
        $$P(X_1\cdots X_n) = P(X_1)P(X_2\mid X_1)P(X_3\mid X_1^2)\cdots P(X_n\mid X_1^{n-1})$$
    \end{block}}

    \visible<2->{\begin{block}{By applying chain rule to a sequence of words}
        $$P(w_1\cdots w_n) = P(w_1)P(w_2\mid w_1)P(w_3\mid w_1^2)\cdots P(w_n\mid w_1^{n-1})$$
    \end{block}}

    \visible<3->{\begin{block}{Examples}
        \begin{itemize}
            %\item $P(an\ apple) = P(an)\times P(apple\mid an)$
            \item $P(an\ apple\ is) = P(an)\times P(apple\mid an) \times P(is\mid an\ apple)$
        \end{itemize}
    \end{block}}

\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{Let's Apply Chain Rule}
    By applying chain rule to the phrase $"an\ apple\ is\ on\ the"$:
    \begin{align*}
        P("an\ apple\ is\ on\ the") &= P(an)\times P(apple\mid an) \times P(is\mid an\ apple)\\
                                  &\times P(on\mid an\ apple\ is) \times P(the\mid an\ apple\ is\ on)
    \end{align*}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{In practice chain rule does not help}
    \begin{itemize}[<+->]
        % \item Language is creative, and any particular context might have never occured before.
        \item We don't know the way to compute the exact probability of a word given a long sequence of preceding words, $P(w_n\mid w_1^{n-1})$.
    \end{itemize}

\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{The Presence of N-gram}
    The general equation for this n-gram approximation to the conditional probability of the next word in a sequence is
    $$P(w_n\mid w_1^{n-1}) \approx P(w_n\mid w_{n-N+1}^{n-1})$$

    \visible<2->{\begin{block}{In case of Bigram (2-gram)}
        When we use bigram model to predict the conditional probability of the next word, we can approximate by
        $$P(w_n\mid w_1^{n-1}) \approx P(w_n\mid w_{n-1})$$
    \end{block}}

    \visible<3->{\begin{block}{Example}
        $$P(the\mid \ an\ apple\ is\ on) \approx P(the\mid \ on)$$
    \end{block}}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{Bigram in Practice}
By supposing that C is the counts of a sequence of words from a corpus.
    $$P(w_n\mid w_{n-1}) = \frac{C(w_{n-1}w_n)}{\sum_{w\in dict}C(w_{n-1}w)}$$

    \visible<2->{\begin{block}{Example}
        $$P(the\mid \ on) = \frac{C(on\ the)}{\sum_{x}C(on\ x)}$$
    \end{block}}
    \visible<3->{(\textit{on x}) means (on the), (on board), (on time), (on that), $\cdots$}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{The probability of a complete word sequence}
    Given the \textbf{Bigram} assumption for the probability of an individual word, we can compute the probability of complete word sequence by:
    $$P(w_1^n) \approx \prod_{k=1}^nP(w_k\mid w_{k-1})$$

    % Compare Chain Rule with Bigram based Calculation
    \visible<2->{
        \textbf{Chain Rule}
            \begin{align*}
                P("an\ apple\ is\ on\ the") &= P(an)\times P(apple\mid an)
                                               \times P(is\mid an\ apple)\\
                                            &\times P(on\mid an\ apple\ is)
                                            \times P(the\mid an\ apple\ is\ on)
            \end{align*}
    }
    \visible<3->{\\
        \textbf{Bigram Assumption}
            \begin{align*}
            P("an\ apple\ is\ on\ the") &= P(apple\mid an) \times P(is\mid apple) \\
                                        &\times P(on\mid is) \times P(the\mid on)
            \end{align*}
    }

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{OOV \& Smoothing}
    \textbf{OOV} stands for \textbf{Out Of Vocabulary}\\
    \begin{itemize}
        \item Words appear only in a \textcolor{purple}{test set} but not in the \textcolor{purple}{training set}.
        \item OOV problem occurs even when we work on big data.
    \end{itemize}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{Types of smoothing}
    \begin{itemize}
        \item Laplace smoothing (Add-one smoothing)
        \item Add-k smoothing
        \item Back-off
        \item \only<1>{Linear interpolation}
              \onslide<2->{\textcolor{purple}{Linear interpolation}}
        \item $\cdots$
    \end{itemize}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{Linear Interpolation}

    \visible<1->{
    We mix the probability estimates from all the n-gram estimators, weighing and combining the trigram, bigram, and unigram counts.
    }

    \visible<2->{
    \begin{block}{Bigram + Unigram}
        $$\hat{P}(table\mid the) = \lambda_1P(table\mid the) + \lambda_2P(table)$$
    \end{block}
    }

    \visible<3->{\begin{block}{Trigram + Bigram + Unigram}
        \begin{align*}
            \hat{P}(table\mid on\ the) &= \lambda_1P(table\mid on\ the)\\
                                         &+ \lambda_2P(table\mid the)\\
                                         &+ \lambda_3P(table)
        \end{align*}
    \end{block}}

    \visible<4->{where we choose $\lambda_i$ such that $\sum_i\lambda_i = 1$}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{How are the $\lambda$ values set?}
    \begin{itemize}
        \item They can be learned from a \textcolor{purple}{held-out} corpus
        \item Can be found by \href{https://www.cs.cmu.edu/~roni/11761/Presentations/degenerateEM.pdf}{\textcolor{purple}{EM}} algorithm.
        \item For the purpose of this project, we assume without loss of generality that $\lambda_i > \lambda_j (\forall i < j)$
    \end{itemize}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{POS-Ngram}
\begin{frame}{POS-Ngram}

    \begin{block}{Formula}
        $$P(w_n\mid w_1^{n-1}) = \sum_{c_n}P(w_n\mid c_n)\times P(c_n\mid c_{n-N+1}^{n-1})$$
        where $c_n$ is a class of $w_n$\\
        (In this context, $c_n$ is considered a part of speech of the word $w_n$)
    \end{block}

    \visible<2->{\begin{block}{Example}
        \centering
        \begin{tabular}{|l|l|}
            \hline
            cat, dog, thought, $\cdots$ & $c_1$:Noun\\\hline
            go, speak, $\cdots$         & $c_2$:Verb \\\hline
            play, $\cdots$              & $c_1$:Noun, $c_2$:Verb\\\hline
        \end{tabular}\\
    \end{block}}

    \flushleft

    \visible<3->{It means that to predict the next word $w_n$
    \begin{enumerate}
        \item $P(w_n\mid c_n)=\frac{C(w_n,c_n)}{C(c_n)}$
        \item Compute $P(c_n\mid c_{n-N+1}^{n-1}) \onslide<4->{\sim P(w_n\mid w_{n-N+1}^{n-1})$}
        \item $w_n^\ast$ is determined by $\argmax_{w_n} P(w_n\mid w_1^{n-1})$
    \end{enumerate}}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{comment}
\begin{frame}{POS-Ngram}
    \begin{block}{Example}
        \begin{itemize}
            \item $"I\ study\ english"$
            \item $Pronoun\ Verb\ Noun"$
            \item $Pronoun\ Noun\ Adjective"$
            \item $\cdots$
        \end{itemize}
    \end{block}
\end{frame}
\end{comment}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Implementation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Tools}
\begin{frame}{Implementation}
    \begin{itemize}
        \item Programming Language : Python
        \item Ui : Tkinter
        \item NLTK (Natural Language ToolKit)
    \end{itemize}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Programming Structure}
\begin{frame}{Programming Structure}
    \smartdiagramset{}
    \begin{center}
        \smartdiagram[priority descriptive diagram]{PosNgram.py,Predict.py,Ui.py}
    \end{center}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}
    \centering
    \Huge DEMO
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}{References}
    \begin{thebibliography}{4}
        \bibitem{Stanford}
            Speech and Language Processing (Chapter 3) (Daniel Jurafsky - Stanford University, 1999)
            % \url{https://web.stanford.edu/~jurafsky/slp3/edbook_oct162019.pdf}
        \bibitem{Lecture Note}
            N-gram Language Modeling Tutorial (Lecture notes courtesy of Prof. Mari Ostendorf, 2007-06-21)\\
            % \url{http://ssli.ee.washington.edu/WS07/notes/ngrams.pdf}
        \bibitem{The University of Tokyo}
            Probabilistic Language Model (Chapter 3) (Kenji KITA, University of Tokyo Press, 1999)
    \end{thebibliography}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% lecture video
% https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=c1e90862-bf43-430f-a7cf-0410081ad95d
\end{document}