\lab{State Machines}{State Machines}
\objective{Explore deterministic and non deterministic finite automatons.}

A state machine is the simplest machine that can be considered a computer.
It is very similar to a Markov chain.
Both have a set of states and have a defined set of transitions between states.

\begin{definition}
A deterministic finite automaton (DFA) is described by a 5-tuple $(Q, \Sigma, \delta, q_0, A)$,
where
\begin{itemize}
\item $Q$ is a finite set of states
\item $\Sigma$ is a finite alphabet of recognized input symbols
\item $\delta$ is the machine's transition function
\item $q_0 \in Q$ is the machine's start state
\item $A \subseteq Q$ is the set of accept states
\end{itemize}
\end{definition}

A DFA describes a language or set of accepted inputs.
If we give a DFA a particular input, it will tell us if that input exists in its language or not.
Let's look at a simple example in Figure \ref{fig:simple_dfa}.
We assume a two symbol alphabet, \{0, 1\}.
All the other elements of the DFA are shown graphically ($q_1$ is an accept state, denoted by the double circle).
\begin{figure}
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial](q0) {$q_0$};
\node[state,accepting] (q1) [right=of q0]{$q_1$};

\path[->] (q0) edge node {0} (q1);
\end{tikzpicture}
\caption{A simple deterministic finite automaton.}
\label{fig:simple_dfa}
\end{figure}
This only input accepted by this DFA is the string \texttt{0}.
That means, only the string \texttt{0} will cause the machine to transition to an accepting state, namely $q_1$.
Once we have consumed all the input, we stop.
If the final state is an accepting state, then the string is a member of the language represented by this DFA.
You can think of every state having a transition to an implied failure state on unmatched input.
For example, if we gave this DFA the input \texttt{1} or \texttt{01} we would transition immediately to the failure state on the unmatched symbol.
A symbol is matched if there is a transition defined for it.
Otherwise, we automatically transition to a failure state.
One possible way to represent this DFA in Figure \ref{fig:simple_dfa} is:
\begin{lstlisting}
def consume(in_str):
    for char in in_str:
        yield char

def state0(scanner):
    try:
        if scanner.next() == '0':
            return state1(scanner)
    except:
        pass
    return False

def state1(scanner):
    try:
        c = scanner.next()
    except StopIteration:
        return True
    return False

def dfa(in_str):
    scanner = consume(in_str)
    return state0(scanner)
\end{lstlisting}

\begin{problem}
Write a Python function that will simulate any given DFA.
The above approach will not work (each state requires a new function).
Your function should accept as parameters an input string and a 4-tuple, \li{machine}, that describes the DFA as $(\Sigma, \delta, q_0, A)$ in the following format
\begin{itemize}
\item[$\Sigma$:] A set with all symbols of the language's alphabet represented as strings.
\item[$\delta$:] A dictionary that defines the transition function.
It should map tuples of the form \li{(m, c)} to values, $n$, to represent the transition $q_m \xrightarrow{c} q_n$ ($m, n \in \mathbb{Z}$ and $c$ is a string character).
\item[$q_0$:] An integer, $i$, representing $q_i$, the initial state.
\item[$A$:] A set of integers representing the accepting states.
\end{itemize}
You function should return true if the given input string is a member of the language described by the DFA and false otherwise.
Your function should test the entire input string.  Be careful not to return true prematurely.
\label{prob:dfa_sim}
\end{problem}

\begin{problem}
Define the following DFAs using the format in Problem \ref{prob:dfa_sim} for describing a DFA.

\begin{figure}[H]
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial,accepting] (q0) {$q_0$};
\node[state] (q1) [right=of q0] {$q_1$};

\path[->]
(q0) edge[loop above] node {a} (q0)
(q1) edge[loop above] node {a} (q1)
(q0) edge[bend left] node {b} (q1)
(q1) edge[bend left] node {b} (q0);
\end{tikzpicture}
\caption*{\ref{prob:reprDFA}(a)}
\label{prob:reprDFA_a}
\end{figure}

\begin{figure}[H]
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial] (q0) {$q_0$};
\node[state] (q1) [right=of q0] {$q_1$};
\node[state,accepting] (q2) [right=of q1] {$q_2$};
\node[state] (q3) [right=of q2] {$q_3$};

\path[->]
(q0) edge[loop above] node {1} (q0)
(q0) edge node {0} (q1)
(q1) edge[loop above] node {1} (q1)
(q1) edge node {0} (q2)
(q2) edge[loop above] node {1} (q2)
(q2) edge node {0} (q3)
(q3) edge[loop above] node {0,1} (q3);
\end{tikzpicture}
\caption*{\ref{prob:reprDFA}(b) The self-loop on $q_3$ transitions to itself on either a 0 or 1.}
\label{prob:reprDFA_b}
\end{figure}

\begin{figure}[H]
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial] (q0) {$q_0$};
\node[state,accepting] (q1) [right=of q0] {$q_1$};
\node[state,accepting] (q2) [above right=of q0] {$q_2$};
\node[state] (q3) [right=of q1] {$q_3$};
\node[state] (q4) [right=of q3] {$q_4$};

\path[->]
(q0) edge node {1} (q1)
(q0) edge node {0} (q2)
(q1) edge node {1} (q3)
(q1) edge node {0} (q2)
(q2) edge[loop above] node {0,1} (q2)
(q3) edge node {1} (q4)
(q3) edge node {0} (q2)
(q4) edge node {0,1} (q2);
\end{tikzpicture}
\caption*{\ref{prob:reprDFA}(c)}
\label{prob:reprDFA_c}
\end{figure}
\label{prob:reprDFA}
\end{problem}

\section*{Non-Deterministic Finite Automatons}
A non-deterministic finite automaton (NFA) are a generalization of deterministic state machines.
The difference between and NFA and a DFA is the transition function.
A transition for a DFA maps the current state and the input symbol to exactly one result state.
In contrast, an NFA can map the current state and the input symbol (or the empty string, $\epsilon$) to a set of resulting states.
If the set of resulting states has a cardinality greater than 1, the computation separates into independent branches.
For example, in Figure \ref{fig:simple_nfa}, given the input \texttt{0100}, we start at state $q_0$.  We match the first 0 and the active state is again $q_0$.  Next, we match a 1 and the resulting states are \{$q_0$, $q_1$, $q_2$\}.
At this point, our machine is in three states simultaneously (we transition to state $q_2$ on the $\epsilon$ transition).
This is the non-deterministic behavior of our state machine.
Our next input symbol is 0.
It is helpful to think of three separate and independent branches of computations.
\begin{enumerate}
\item State $q_0$ transitions back to $q_0$
\item State $q_1$ transitions to $q_2$
\item State $q_2$ transitions to $q_3$
\end{enumerate}
Our matching could terminate here, but since we have not completely read out input, we must still keep going.
The most we can say at this point is the string \texttt{010} is in the NFA's language.
Our current states are the set \{$q_0$, $q_2$, $q_3$\}.
Our last symbol is another 0.
\begin{enumerate}
\item State $q_0$ transitions back to itself
\item State $q_2$ transitions to $q_3$
\item State $q_3$ transitions to the implied fail state
\end{enumerate}
Now that we have read all of our input, and one of our final resulting states is an accepting state, we accept the entire string.
Does the string \texttt{01000} exist in the NFA's language?

You might think that this new power of NFAs will allow us to represent many more languages.
However, every language that can be represented as an NFA can also be represented by an equivalent DFA.
This is to say that an NFA is equivalent to a DFA in computing power.
In fact a method exists that will transform any NFA into an equivalent DFA.
This is, however, beyond the scope of this lab.
NFAs are quite useful though because they can often represent complex languages very simply and intuitively.
\begin{figure}
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial] (q0) {$q_0$};
\node[state] (q1) [right=of q0] {$q_1$};
\node[state] (q2) [right=of q1] {$q_2$};
\node[state,accepting] (q3) [right=of q2] {$q_3$};

\path[->]
(q0) edge node {1} (q1)
(q0) edge[loop above] node {0,1} (q0)
(q1) edge node {0,1,$\epsilon$} (q2)
(q2) edge node {0,1} (q3);
\end{tikzpicture}
\caption{A simple non-deterministic finite automaton.}
\label{fig:simple_nfa}
\end{figure}

\begin{problem}
Draw a non-deterministic finite automaton that can match the following language.

\emph{Let $w \in L$ where the number of a's is a multiple of 3, or contains least 4 b's, or where the number of c's is odd.}
\label{prob:drawnfa}
\end{problem}

\section*{Operations on Regular Languages}
We can build machines that represent more complex languages by concatenating several simpler DFAs or NFAs together.
The format expression for concatenating the languages $A$ and $B$ is $A \circ B$.
We can also union the machines and thus union the languages they each represent (expressed as $A \cup B$).
There are two more basic operations, \emph{star} and \emph{plus}, that are extremely useful.
Given a language, $L$, $L^*$ means that we can concatenate any number of strings in $L$ to get another string in $L^*$.  $L^+$ denotes one or more strings of $L$ have been attached to form a new string in $L^+$.
Note that $L^*$ and $L^+$ are different languages than $L$.

\section*{Introduction to Regular Expressions}
Regular expressions are not trivial or easy, but understanding them can be very straightforward.
The best way to think of regular expressions is the view them as operations on regular languages.
The entire regular expression represents a regular language.
In fact, the definition of being a regular language, $L$ means that there is a regular expression that represents $L$.
Since a every regular language can be represented by a DFA, we have an equivalence between regular expressions and DFAs.
And, because every NFA is equivalent to some DFA, we have a three-way equivalence (DFA=NFA=RE).
Regular epxressions are a \emph{very} compact way of representing complex state machines.
Sometimes it can be easier to use a DFA representation (much easier to write and define) or other times it can be easier to use a regular expression (more expressive, more compact).
As noted, regular expressions can only represent a language if it is regular.
There exist languages that are non-regular.
There are ways to rigorously prove that a language is non-regular, but that is beyond the scope of this lab.
A regular language is any language that can be represented as a regular expression, an NFA, or a DFA.

Regular expressions are especially useful and powerful for finding and matching patterns in text.
Many tools can take advantage of the powerful matching features of regular expressions.
You can think of regular expressions as operations on state machines.
For example, the regular expression $(a^*b)^* b^*a$ could be represented as the NFA in \ref{fig:nfa_concat}.
\begin{figure}
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial] (q0) {$q_0$};
\node[state] (q1) [right=of q0] {$q_1$};
\node[state] (q2) [right=of q3] {$q_2$};
\node[state] (q3) [right=of q1] {$q_3$};
\node[state,accepting] (q4) [right=of q2] {$q_4$};

\path[->]
(q0) edge node {$\epsilon$} (q1)
(q0) edge[bend right] node {$\epsilon$} (q2)

(q1) edge[loop above] node {a} (q1)
(q1) edge node {b} (q3)
(q3) edge node {$\epsilon$} (q2)
(q2) edge[loop below] node {b} (q2)
(q2) edge node {a} (q4);
\end{tikzpicture}
\caption{An NFA for the regular expression $(a^*b)^* b^*a$.}
\label{fig:nfa_concat}
\label{fig:simple_nfa}
\end{figure}
Which is simply a concatenation of the NFAs \ref{fig:nfa1} and \ref{fig:nfa2}.
\begin{figure}
\centering
\begin{subfigure}[b]{.5\textwidth}
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial] (q1) {$q_1$};
\node[state,accepting] (q3) [right=of q1] {$q_3$};

\path[->]
(q1) edge[loop above] node {a} (q1)
(q1) edge node {b} (q3);
\end{tikzpicture}
\caption{NFA for $a^*b$.}
\label{fig:nfa1}
\end{subfigure}

\begin{subfigure}[b]{.5\textwidth}
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,on grid,auto]
\node[state,initial] (q2) {$q_2$};
\node[state,accepting] (q4) [right=of q2] {$q_4$};

\path[->]
(q2) edge[loop above] node {b} (q2)
(q2) edge node {a} (q4);
\end{tikzpicture}
\caption{NFA for $b^*a$.}
\label{fig:nfa2}
\end{subfigure}
\end{figure}

\begin{problem}
For parts a and b in Problem \ref{prob:reprDFA}, write a  regular expression that is equivalent to state machine.
\label{prob:equiv_regex}
\end{problem}

Because of a regular expression's ability to compactly represent complex state machines, they are very useful in matching patterns in a string (both trivial and non-trivial).
Instead of matching exact strings (a standard string search find and replace), the regular expression matches the form or pattern of a string (like a formatted date or email address pattern).
For example, we can use a regular expression to match phone numbers, email addresses, HTTP addresses.

There are many libraries that implement regular expression matching.
Most programming languages also have their own regular expression library.
Unfortunately, the syntax for regular expressions may be slightly different in each implementation.
There is no universal standard for all regular expressions across all platforms.
However, a common syntax with a few variations is very widely used, so most regular expression patterns can be used with any library.

\begin{warn}
Theoretically, regular expressions have a linear runtime.
However, because of more advanced features that many regular expression libraries have, poorly written regular expressions can have exponential runtime!
Profiling regular expressions is tricky because of their syntax.
A single character or two  could mean the difference between an efficient regular expression and a regular expression that has practically infinite runtime for long inputs.
\end{warn}

\section*{Using Regular Expressions in Python}
The Python Standard Library includes a module that will handle regular expressions.
You can read the documentation for the module at \url{http://docs.python.org/2/library/re.html}.
We begin by importing the \li{re} module from the standard library.
\begin{lstlisting}
>>> import re
>>> m = re.match(r"(a*b)*b*a", "abba")
>>> type(m)
_sre.SRE_Match
\end{lstlisting}
We have just performed our first match!
As you can see, \li{re.match} returns an \li{SRE_Match} object (or MatchObject).
If there was not match to the regular expression, \li{None} is returned.
The MatchObject object has several methods and attributes that give us information about the match.
We mention a few here, but all of them are explained in the standard library documentation.
To view what a regular expression matched, we can use the \li{group()} method.
\begin{lstlisting}
>>> m = re.match(r"(.*) (.*) (.*)", "These are a lot of words!")
>>> m.group()
'These are a lot of words!'
>>> m.groups()
('These are a lot', 'of', 'words!')
>>> m.group(1)
'These are a lot'
>>> m.group(3)
'words!'
\end{lstlisting}
As you can see, we matched the entire string.
Our regular expression split the string into 3 groups.
We can access all the groups with \li{groups()} or individual subgroups with \li{group()}.
We can know the span of the group with the \li{span()} method.
\begin{lstlisting}
>>> m.span(1)
(0, 15)
>>> m.span(3)
(19, 25)
>>> m.string[19:25]
'words!'
>>> m.regs #gives the span of each group
((0, 25), (0, 15), (16, 18), (19, 25))
\end{lstlisting}

We have compiled a table of the important characters and their meanings in Python.
You can think of each of these characters representing simple state machines.
A worthwhile exercise would be to define the state machines that correspond to each row in Table \ref{table:regexchars}.
\begin{table}
\begin{tabular}{|l|l|}
\hline
Character & Meaning \\
\hline
\li{.} & Matches any character (except newlines) \\
\li{^} & Matches at the start of a line \\
\li{\$} & Matches at the end of a line \\
\li{*} & Matches 0 or more repetitions of a regular expression (greedy) \\
\li{+} & Matches 1 or more repetitions of a regular expression (greedy) \\
\li{?} & Matches 0 or 1 repetition of a regular expression \\
\li{(m)} & Matches exactly $m$ repetitions of a regular expression \\
\li{(m, n)} & Matches between $m$ and $n$ repetitions of a regular expression \\
\li{[]} & Matches a set or range of characters \\
\li{[^ ]} & Matches the complement of a range of characters \\
\li{A|B} & A, B are regular expression.  Match either A or B \\
\li{\\w} & Matches any alphanumeric character and underscore \\
\li{\\W} & Matches any non-alphanumeric character \\
\li{\\s} & Matches any whitespace character \\
\li{\\S} & Matches any non-whitespace character \\
\hline
\end{tabular}
\caption{Common character sequences in regular expressions.}
\label{table:regexchars}
\end{table}
Notice that some of the operator characters in Table \ref{table:regexchars} are \emph{greedy} operators.
This means that this part of the regular expression will try to consume as much of the input as possible needed to make a match.
There are cases where this is not desirable.
We can make these greedy operators non-greedy, or lazy, by suffixing them with a question mark.
The lazy star operator is then \texttt{*?}.
A lazy operator will consume as little of the input as possible needed to make a match.
\begin{lstlisting}
# greedy + operator
>>> re.match(r"<.+>", 'Superman').group()
Superman
# non-greedy/lazy + operator
>>> re.match(r"<.+?>", 'Superman').group()

\end{lstlisting}

\subsection*{Matching, Searching, Splitting, and Substituting}
Having now an understanding of the MatchObject, we can begin to talk about how to use the powerful features of Python's regular expression library.
Most of your regular expression use will revolve around \li{re.match()}, \li{re.search()}, \li{re.split()}, and \li{re.sub}.
\begin{description}
\item \emph{re.match(pattern, string, flags=0)}:  Apply a pattern beginning at start of \emph{string}.  Returns a MatchObject if successful, otherwise None.
\item \emph{re.search(pattern, string, flags=0)}:  Look in \emph{string} for any occurrence of \emph{pattern}.  Returns a MatchObject if the pattern is found.
\item \emph{re.split(pattern, string, maxsplit=0, flags=0)}: Split \emph{string} by occurrences of \emph{pattern}.  Returns a list of all matching substrings.
\item \emph{re.sub(pattern, repl, string, count=0, flags=0)}: Search \emph{string} for occurrences of \emph{pattern}, and replace each occurrence with the result of \emph{repl} (can be a string or function that returns a string).
\end{description}
Notice that all four of these commands operate from the same workflow.
Each accepts a pattern or regular expression and a string.
That pattern is run on the string and then results are returned.
In the case of splitting and substituting, we manipulate the string based off the matching parts of the pattern.
\begin{lstlisting}
# match one or more words (non-alphanumeric characters)
>>> re.split('\W+', 'Hi there! This is a splitted sentence.')
['Hi', 'there', 'This', 'is', 'a', 'splitted', 'sentence', '']
\end{lstlisting}

\begin{problem}
Write an expression that can be used to validate most email addresses of the form a@b.c where a is any character (including periods, dashes and underscores), b can be any domain (including periods, dashes, and underscores), and c is any three or four character alphanumeric sequence.

Your expression should split the email address into three groups: a, b, and c (using the notation above).
Substitute the letter x for group b.
For example, your substitution should transform abc123@email.com to abc123@xxxxx.com (the number of x's should be equal to the number of characters in group b).
\label{prob:emailvaldiator}
\end{problem}

\begin{problem}
Using Table \ref{table:regexchars}, describe the parts of the following regular expressions:
\begin{enumerate}
\item \li{^([a-z0-9]\{32\})\$}
\item \li{^(19|20)\\d\\d[- /.](0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])\$}
\item \li{^(([0-1]?[0-9])|([2][0-3])):([0-5]?[0-9])(:([0-5]?[0-9]))?\$}
\end{enumerate}
\label{prob:readingregex}
\end{problem}


\section{Gauss-Jordan elimination}

\begin{outcome}
  \begin{enumerate}
  \item Find the {\rref} of a matrix.
  \item Solve a system of linear equations using Gauss-Jordan elimination.
  \end{enumerate}
\end{outcome}

In the previous section, we saw how to solve a system of equations by
using Gaussian elimination and back substitution. The back
substitution step can be quite confusing and error prone, especially
when there are parameters. For example, in
Example~\ref{exa:two-parameter-set-of-solution}, we had to substitute $y=s$,
$z=2-t$, and $w=t$ into the equation $x=3-2y+2z-2w$, which required
another simplification step.

In this section, you will learn an alternative procedure called {\em
  Gauss-Jordan elimination} which eliminates the need for back
substitution, at the expense of doing a few additional row operations.
The key to this technique is a special kind of {\ef} called a
{\em {\rref}}.

\begin{definition}{\Rref}{rref}
  An augmented matrix is in \textbf{\rref}%
  \eindex{\rref}%
  \eindex{\ef!reduced} if

  \begin{enumerate}
  \item It is in {\ef}.

  \item Each leading entry is equal to $1$.

  \item All entries above a leading entry are zero.
  \end{enumerate}
\end{definition}

\begin{example}{{\Rref}}{rref-matrices}
  The following augmented matrices are in {\rref}. The leading entries
  have been circled for emphasis. Note how all of the leading entries
  are equal to $1$, and they have zeros above them.
  \begin{equation*}
    \begin{mymatrix}{rrrrr|r}
      \circled{1} & 2 & 0 & 5 & 0 & 3 \\
      0 & 0 & \circled{1} & 2 & 0 & 0 \\
      0 & 0 & 0 & 0 & \circled{1} & 1 \\
      0 & 0 & 0 & 0 & 0 & 0
    \end{mymatrix},\quad\begin{mymatrix}{rrr|r}
      \circled{1} & 0 & 0 & 0 \\
      0 & \circled{1} & 0 & 0 \\
      0 & 0 & \circled{1} & 0 \\
      0 & 0 & 0 & \circled{1} \\
      0 & 0 & 0 & 0
    \end{mymatrix}, \begin{mymatrix}{rrr|r}
      \circled{1} & 0 & 0 & 4 \\
      0 & \circled{1} & 0 & 3 \\
      0 & 0 & \circled{1} & 2
    \end{mymatrix}
  \end{equation*}
\end{example}

We can carry every augmented matrix to {\rref} by doing elementary row
operations.

\begin{algorithm}{Gauss-Jordan elimination}{gauss-jordan}
  This algorithm provides a method for using row operations to take a
  matrix to its {\rref}%
  \eindex{\rref!algorithm}%
  \index{Gauss-Jordan elimination}.
  \begin{enumerate}
  \item First, use Gaussian elimination
    (Algorithm~\ref{algo:gaussian-elimination}) to reduce the matrix to
    {\ef}.
  \item Moving from right to left, consider each pivot entry. Without
    changing the row containing the pivot entry, or any rows below it,
    use row operations to create zeros in the column above the pivot
    entry. Finally, divide the row by its pivot entry, to make the
    pivot entry equal to $1$.
  \end{enumerate}
\end{algorithm}

\begin{example}{Gauss-Jordan elimination}{gauss-jordan1}
  Solve the system of equations from
  Example~\ref{exa:system-with-one-solution} using Gauss-Jordan elimination.
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l}
      x+4y+3z &=& 11 \\
      2x+10y+7z &=& 27 \\
      x+y+2z &=& 5.
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  In Example~\ref{exa:system-with-one-solution}, we had already reduced the
  system to {\ef}:
  \begin{equation*}
    \begin{mymatrix}{rrr|r}
      \circled{1} & 4 & 3 & 11 \\
      0 & \circled{2} & 1 & 5 \\
      0 & 0 & \circled{1} & 3
    \end{mymatrix}.
  \end{equation*}
  We reduce it to {\rref} by performing the following row operations:
  \begin{equation*}
    \begin{mymatrix}{rrr|r}
      \circled{1} & 4 & 3 & 11 \\
      0 & \circled{2} & 1 & 5 \\
      0 & 0 & \circled{1} & 3
    \end{mymatrix}
    \!\!\!\stackrel{R_2\rowop R_2-R_3}{\stackrel{R_1\rowop R_1-3R_3}{\roweq}}\!\!
    \begin{mymatrix}{rrr|r}
      \circled{1} & 4 & 0 & 2 \\
      0 & \circled{2} & 0 & 2 \\
      0 & 0 & \circled{1} & 3
    \end{mymatrix}
    \!\!\stackrel{R_1\rowop R_1-2R_2}{\roweq}\!\!
    \begin{mymatrix}{rrr|r}
      \circled{1} & 0 & 0 & -2 \\
      0 & \circled{2} & 0 & 2 \\
      0 & 0 & \circled{1} & 3
    \end{mymatrix}
    \!\!\stackrel{R_2\rowop \frac{1}{2}R_2}{\roweq}\!\!
    \begin{mymatrix}{rrr|r}
      \circled{1} & 0 & 0 & -2 \\
      0 & \circled{1} & 0 & 1 \\
      0 & 0 & \circled{1} & 3
    \end{mymatrix}\!\!.
  \end{equation*}
  The resulting matrix is in {\rref}. Note that the final system of
  equations is especially easy to solve, because the three equations
  are $x=-2$, $y=1$, and $z=3$. No back substitution is needed.
\end{solution}

\begin{example}{Gauss-Jordan elimination}{gauss-jordan2}
  Solve the system of equations from
  Example~\ref{exa:two-parameter-set-of-solution} using Gauss-Jordan
  elimination.
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l}
      x + 2y - 2z + 2w &=& 3 \\
      x + 2y - z  + 3w &=& 5 \\
      x + 2y - 3z + 1w &=& 1.
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  In Example~\ref{exa:two-parameter-set-of-solution}, we had obtained the
  following {\ef}:
  \begin{equation*}
    \begin{mymatrix}{rrrr|r}
      \circled{1} & 2 & -2 & 2 & 3 \\
      0 & 0 & \circled{1} & 1 & 2 \\
      0 & 0 & 0 & 0 & 0
    \end{mymatrix}.
  \end{equation*}
  We reduce this to {\rref} by performing the following additional
  step:
  \begin{equation*}
    \begin{mymatrix}{rrrr|r}
      \circled{1} & 2 & -2 & 2 & 3 \\
      0 & 0 & \circled{1} & 1 & 2 \\
      0 & 0 & 0 & 0 & 0
    \end{mymatrix}
    \stackrel{R1\rowop R1+2R2}{\roweq}
    \begin{mymatrix}{rrrr|r}
      \circled{1} & 2 & 0 & 4 & 7 \\
      0 & 0 & \circled{1} & 1 & 2 \\
      0 & 0 & 0 & 0 & 0
    \end{mymatrix}.
  \end{equation*}
  The first equation states that $x=7-2y-4w$, and the second equation
  states that $z=2-w$. Using the free variables $y$ and $w$ as
  parameters, we obtain the following general solution:
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l}
      x &=& 7-2y-4w \\
      y &=& y \\
      z &=& 2-w \\
      w &=& w.
    \end{array}
  \end{equation*}
  Note that we did not really have to do back substitution; all we had
  to do is to shift parts of the equations to the right-hand side. If
  the solution looks strange, because it has equations like ``$y=y$''
  in it, keep in mind that this means that $y$ and $w$ are arbitrary
  numbers, i.e., parameters. We can replace $y$ and $w$ by parameters
  $s$ and $t$ on the right-hand side, as before:
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l}
      x &=& 7-2s-4t \\
      y &=& s \\
      z &=& 2-t \\
      w &=& t.
    \end{array}
  \end{equation*}
\end{solution}

\begin{discussion}{Which procedure is better?}{g-vs-gj}
  Which one is the better procedure to use, Gaussian elimination with
  back substitution, or Gauss-Jordan elimination? The answer is: it
  depends. In certain applications, it is not necessary to completely
  solve a system of equations. Sometimes it is sufficient just to figure
  out whether the system is consistent or inconsistent, or whether the
  solution is unique or not. In those situations, you already get the
  required information from the {\ef} and there is no need to do the
  additional steps to reduce the system to {\rref}. Also, in some
  situations, Gauss-Jordan elimination can introduce fractions into your
  augmented matrix, making the matrix more complicated to work with. In
  such cases, it may sometimes be easier to do back substitution. But in
  most cases, Gauss-Jordan elimination is simpler to do than back
  substitution, and therefore I recommend using the Gauss-Jordan method
  most of the time.
\end{discussion}

One situation where Gauss-Jordan elimination excels is when you have
to solve many systems of equations that all have the same coefficient
matrix.

\begin{example}{Multiple systems sharing the same left-hand side}{multiple-systems}
  Solve the following two systems of equations.
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l@{~}}
      x       +  z &=& 1 \\
      2x +  y + 3z &=& 2 \\
      3x + 2y + 5z &=& 4 \\
    \end{array}
    \quad\quad
    \begin{array}{r@{~}c@{~}l@{~}}
      x       +  z &=& 2 \\
      2x +  y + 3z &=& 5 \\
      3x + 2y + 5z &=& 8 \\
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  We could certainly solve each system of equations separately. But
  since the left-hand sides are the same, we will perform exactly the
  same row operations on both systems. We can save some work by
  solving both systems together. Instead of a usual augmented matrix
  with only one constant vector, we create an augmented matrix
  containing both constant vectors at the same time.
  \begin{equation*}
    \begin{mymatrix}{rrr|rr}
      1 & 0 & 1 & 1 & 2 \\
      2 & 1 & 3 & 2 & 5 \\
      3 & 2 & 5 & 4 & 8 \\
    \end{mymatrix}
  \end{equation*}
  Then we row-reduce the coefficient matrix to {\rref} as usual. (We
  do not need to bother reducing the right-hand size to {\rref}).
  \begin{equation*}
    \begin{mymatrix}{rrr|rr}
      1 & 0 & 1 & 1 & 2 \\
      2 & 1 & 3 & 2 & 5 \\
      3 & 2 & 5 & 4 & 8 \\
    \end{mymatrix}
    \stackrel{R_2\rowop R_2-2R_1}{
      \stackrel{R_3\rowop R_3-3R_1}{\roweq}}
    \begin{mymatrix}{rrr|rr}
      1 & 0 & 1 & 1 & 2 \\
      0 & 1 & 1 & 0 & 1 \\
      0 & 2 & 2 & 1 & 2 \\
    \end{mymatrix}
    \stackrel{R_3\rowop R_3-2R_2}{\roweq}
    \begin{mymatrix}{rrr|rr}
      1 & 0 & 1 & 1 & 2 \\
      0 & 1 & 1 & 0 & 1 \\
      0 & 0 & 0 & 1 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  We see that the first system is inconsistent, because it contains a
  row of the form $\mat{0~0~0~|~1}$. The second system is consistent,
  and we get the general solution $z=t$, $y=1-t$, $x=2-t$.
\end{solution}


\subsection{Primal and dual problems}

\subsubsection{The primal problem}

We already have this.

\subsubsection{The dual problem}

We can define the Lagrangian dual function:

\(g(\lambda, \nu ) = \inf_{x\in X} \mathcal{L}(x, \lambda ,\nu )\)

That is, we have a function which chooses the returns the value of the optimised Lagrangian, given the values of \(\lambda \) and \(\nu\).

This is an unconstrained function.

We can prove this function is concave (how?).

The infimum of a set of concave (and therefore also affine) functions is concave.

The supremum of a set of convex (and therefore also affine) functions is convex.

Given a function with inputs \(x\), what values of \(x\) maximise the function?

We explore constrained and unconstrained optimisation. The former is where restrictions are placed on vector \(x\), such as a budget constraint in economics.

\subsubsection{The dual problem is concave}

\subsubsection{The duality gap}

We refer to the optimal solution for the primary problem as \(p^*\), and the optimal solution for the dual problem as \(d^*\).

The duality gap is \(p^*-d^*\).


\subsection{Linear second order PDE}

\begin{Notation}{coordinates}
  Dimension of ``physical space'' will be denoted by $d$.  We denote
  coordinates in $\R^d$ as
  \begin{gather*}
    \vx = (x_1,\dots,x_d)^T  .
  \end{gather*}
  In the special cases $d=2,3$ we also write
  \begin{gather*}
  \vx =
  \begin{pmatrix}
    x\\y
  \end{pmatrix},
  \qquad\qquad
  \vx = \begin{pmatrix}
    x\\y\\z
  \end{pmatrix},
  \end{gather*}
  respectively.
  The Euclidean norm on $\R^d$ is denoted as
  \begin{gather*}
     \abs{\vx} = \sqrt{\sum_{i=1}^d x_i^2}.
  \end{gather*}
\end{Notation}

\begin{Notation}{partial-derivative}
  Partial derivatives of a function $u\in C^1(\R^d)$ are denoted by
  \begin{gather*}
    \frac{\d u(\vx)}{\d x_i} = \tfrac{\d}{\d x_i} u(\vx)
    = \d_{x_i} u(\vx) = \d_i u(\vx).
  \end{gather*}

  The \define{gradient} of $u \in C^1$ is the row vector
  \begin{gather*}
    \nabla u = (\d_1u,\dots,\d_du)
  \end{gather*}

  The \define{Laplacian} of a function $u\in C^2(\R^d)$ is
  \begin{gather*}
    \Delta u = \d_1^2 u + \dots + \d_d^2 u = \sum_{i=1}^d \d_i^2 u
  \end{gather*}
\end{Notation}

\begin{Notation}{elim-coord}
  When we write equations, we typically omit the independent variable
  $\vx$. Therefore,
  \begin{gather*}
    \Delta u \equiv \Delta u(\vx).
  \end{gather*}
\end{Notation}

\begin{Definition}{lin-pde-2order}
  A linear PDE of second order in divergence form for a function
  $u\in C^2(\R^d)$ is an equation of the form
  \begin{gather}
    -\sum_{i,j=1}^d \d_i \bigl(a_{ij}(\vx) \d_j u\bigr)
    + \sum_{i=1}^d \bigl(b_i(\vx) \d_i u\bigr) + c(\vx) u = f(x)
  \end{gather}
\end{Definition}

\begin{Definition}{poisson-eqn}
  An important model problem for the equations we are going to study
  is \define{Poisson's equation}
  \begin{gather}
    \label{eq:Poisson}
    -\Delta u = f.
  \end{gather}
\end{Definition}

\begin{intro}
  Already with ordinary differential equations we experience that we
  typically do not search for solutions of the equation itself, but
  that we ``anchor'' the solution by solving an initial value problem,
  fixing the solution at one point on the time axis.

  It does not make sense to speak about an initial point in
  $\R^d$. Instead, it turns out that it is appropriate to consider
  solutions on certain subsets of $\R^d$ and impose conditions at the
  boundary.
\end{intro}

\begin{Definition}{domain}
  A \define{domain} in $\R^d$ is a connected, open set of $\R^d$. We
  typically use the notation $\domain\subset\R^d$.

  The \define{boundary} of a domain $\domain$ is denoted by
  $\d\domain$. To any point $\vx\in\d\domain$, we associate the outer
  unit \define{normal vector} $\vn \equiv \vn(\vx)$.

  The symbol $\d_n u \equiv (\nabla u) \vn$ denotes the \define{normal
    derivative} of a function $u\in C^1(\overline{\domain})$ at a point
  $\vx\in\d\domain$.
\end{Definition}

\begin{Definition}{boundary-conditions}
  We distinguish three types of boundary conditions for Poisson's
  equation, namely for a point $\vx\in\d\domain$ with a given function $g$
  \begin{enumerate}
  \item Dirichlet:
    \begin{gather*}
      u(\vx) = g(\vx)
    \end{gather*}
  \item Neumann:
    \begin{gather*}
      \d_n u(\vx) = g(\vx)
    \end{gather*}
  \item Robin: for some positive function $\alpha$ on $d\domain$
    \begin{gather*}
      \d_n u(\vx) + \alpha(\vx) u(\vx) = g(\vx)
    \end{gather*}
  \end{enumerate}
  While only one of these boundary conditions can hold in a single
  point $\vx$, different boundary conditions can be active on
  different subsets of $\d\domain$. We denote such subsets as
  $\Gamma_D$, $\Gamma_N$, and $\Gamma_R$.
\end{Definition}

\begin{Definition}{dirichlet-problem-differential}
  The \define{Dirichlet problem} for \putindex{Poisson's equation} (in
  differential form) is: find
  $u\in C^2(\domain)\cap C(\overline{\domain})$, such that
  \begin{subequations}
    \begin{xalignat}2
      -\Delta u(\vx) &= f(\vx) & x&\in \domain, \\
      u(\vx) &= g(\vx) & x&\in \d\domain.
    \end{xalignat}
  \end{subequations}
  Here, the functions $f$ on $\domain$ and $g$ on $\d\domain$ are data
  of the problem.

  The Dirichlet problem is called \define{homogeneous}, if $g\equiv 0$.
\end{Definition}

\begin{Theorem*}{Dirichlet-principle}{Dirichlet principle}
  If a function $u\in C^2(\domain)\cap C(\overline{\domain})$ solves
  the \putindex{Dirichlet problem}, then it minimizes the
  \define{Dirichlet energy}
  \begin{gather}
    \label{eq:Dirichlet-energy}
    E(v) = \int_{\domain} \tfrac12 \abs{\nabla v}^2 \dvx - \int_{\domain} f v \dvx,
  \end{gather}
  among all functions $v$ from the set
  \begin{gather}
    V_g = \bigl\{ v\in C^2(\domain)\cap C(\overline{\domain})
    \big| v_{|\d\domain} = g \bigr\}.
  \end{gather}
  This minimizer is unique.
\end{Theorem*}

\begin{proof}
  Using variation of $E$, we will show that
  \begin{gather*}
    \frac{\diffd}{\diffd \varepsilon} E(u+\varepsilon v)
    \Big|_{\varepsilon=0} = 0
  \end{gather*}
  for all $v \in V_0$ since this implies $u + \varepsilon v = g$
  on $\d\domain$. By evaluating the square we have
  \begin{gather*}
    \frac{\diffd}{\diffd \varepsilon} E(u+\varepsilon v)
    = \int_\domain \nabla u \nabla v + \varepsilon \abs{\nabla v}^2 - fv \dx
  \end{gather*}
  Since we are intersted in $E(u)$, we now consider $\varepsilon=0$. We
  get that $u$ minimizes $E(u+\varepsilon v)$ at $\varepsilon = 0$ implies
  \begin{gather*}
    \int_\domain \nabla u \nabla v \dx = \int_\domain fv \dx,
    \qquad\forall v\in V_0.
  \end{gather*}
  By Green's formula
  \begin{gather*}
    \int_\domain \nabla u \nabla v \dx = \int_\domain - \Delta u v \dx
    + \int_{\d\domain} \d_n u v \ds
  \end{gather*}
  we obtain that if $u$ minimizes $E(\cdot)$, then
  \begin{gather*}
    \int_\domain \nabla u \nabla v \dx = \int_\domain fv \dx,
    \qquad\forall v\in V_0,
  \end{gather*}
  since $v \in V_0$ vanishes on $\d\domain$. In summary, we have
  proven so far that if $u$ solves Poisson's Equation, then it is a
  stationary point of $E(\cdot)$. It remains to show that
  $E(u) \le E(u+v)$ for any $v\in V_0$.  Using
  $\int_\domain fv \dx = \int_\domain \nabla u \nabla v$ yields
  \begin{multline*}
    E(u+v) - E(u) = \frac 12 \int_\domain |\nabla (u+v)|^2 - 2 \nabla (u+v) \nabla u
    + |\nabla u|^2 \dx
    = \frac 12 |\nabla v|^2 \dx \ge 0.
  \end{multline*}
  This also proves uniqueness.
\end{proof}

\begin{Lemma}{Dirichlet-Cauchy}
  A minimizing sequence for the Dirichlet energy exists and it is a
  Cauchy sequence.
\end{Lemma}

\begin{proof}
  The Dirichlet energy $E(\cdot)$ is bounded from
  below and hence an infinum exists. Thus, there also exists a series
  $\{u^{(n)}\}_{n \in \mathbb{N}}$ converging to this infinum, i.e.
  \begin{gather*}
    \lim_{n \to \infty} E(u^{(n)}) = \inf_{v \in V_0} E(v).
  \end{gather*}
  Second, we show that $\{u^{(n)}\}_n$ is a Cauchy sequence.

  For the first part we use Friedrich's inequality
  \begin{gather*}
    \norm{v}_{L^2(\domain)} \le \lambda(\Omega)
    \norm{\nabla v}_{L^2(\domain)} \qquad v \in V_0.
  \end{gather*}
  The proof of this result will be given later. Using Hölder's inequality
  we obtain
  \begin{gather*}
    E(v) = \frac 12 \norm{\nabla v}^2 _{L^2(\domain} - \int_\domain fv \dx
    \ge \frac 12 \norm{\nabla v}^2 _{L^2(\domain}
    - \norm{f}_{L^2(\domain} \norm{v}_{L^2(\domain}
  \end{gather*}
  Applying Friedrich's inquality yields that the above expression is
  greater or equal than
  \begin{gather*}
    \frac 12 \norm{\nabla v}^2 _{L^2(\domain}
    - \norm{\nabla v}_{L^2(\domain} \frac 1{\lambda (\domain)}
    \norm{f}_{L^2(\domain}.
  \end{gather*}
  Finally, we apply Young's inequality $ab \le \nicefrac 12 (a^2 + b^2)$ to obtain
  \begin{gather*}
    \frac 12 \norm{\nabla v}^2 _{L^2(\domain}
    - \norm{\nabla v}_{L^2(\domain}^2 - \frac 1{2\lambda (\domain)} \norm{f}_{L^2(\domain}^2
  \end{gather*}
  which yields $E(v) \ge - \frac 1{2\lambda (\domain)^2} \norm{f}_{L^2(\domain}^2$
  as a lower bound independent of $v$. To prove the second part,
  we use the parallelogram identity $\abs{v+w}^2 + \abs{v-w}^2 = 2\abs{v}^2 + 2 \abs{w}^2$.
  Let $m, n$ be natural numbers, then
  \begin{align*}
    \snorm{u^{(n)} - u^{(m)}}^2 _1 =& 2 \snorm{u^{(n)}}^2 _1
                                      + 2 \snorm{u^{(m)}}^2_1- 4 \snorm{\nicefrac 12 (u^{(n)} + u^{(m)}}^2_1 \\
    =& 4 E(u^{(n)}) + 4\int f u^{(n)} \dx + 4 E(u^{(m)}) + 4\int f u^{(m)} \dx \\
                                    &- 8 E(\nicefrac 12 (u^{(n)} + u^{(m)}) - 8 \int \nicefrac 12 f(u^{(n)} + u^{(m)}) \\
    =& 4 E(u^{(n)}) + 4 E(u^{(m)}) - 8 E(\nicefrac 12 f(u^{(n)} + u^{(m)}))
  \end{align*}
  Taking the limit $m,n\to \infty$ yields $4 E(u^{(n)}) + 4 E(u^{(m)})
  \to 8 \inf_{v \in V_0} E(v)$. Lastly, $-E(\nicefrac 12 f(u^{(n)} + u^{(m)}))$ can
  be bounded by $\inf_{v \in V_0} E(v)$. It follows that $\limsup_{m,n\to\infty}
  \snorm{u^{(n)}-u^{(m)}}^2_1 \le 0$ and consequently as desired
  \begin{gather*}
    \lim_{m,n\to\infty} \snorm{u^{(n)}-u^{(m)}}^2_1 = 0.
  \end{gather*}
\end{proof}

\begin{notes}{Dirichlet-proof}
  Dirichlet's principle proved essential for the development of a
  rigorous solution theory for Poisson's equation.  Its proof will be
  deferred to the next theorem.
\end{notes}

\subsection{Variational principle and weak formulation}
\begin{Theorem}{Dirichlet-variational-principle}
  A function $u\in V_g$ minimizes the Dirichlet energy, if and only if
  there holds
  \begin{gather}
    \int_{\domain} \nabla u\cdot\nabla v \dx
    = \int_{\domain} fv\dx, \qquad\forall v\in V_0.
  \end{gather}
  Moreover, any solution to the Dirichlet problem in
  \slideref{Definition}{dirichlet-problem-differential} solves this
  equation.
\end{Theorem}

\begin{Corollary}{Dirichlet-uniqueness}
  If a minimizer of the Dirichlet energy exists, it is necessarily unique.
\end{Corollary}

\begin{Lemma}{reduction-to-zero-bc}
  A function $u\in V_g$ minimizes the Dirichlet energy admits the
  representation $u = u_g + u_0$, where $u_g\in V_g$ is arbitrary and
  $u_0\in V_0$ solves
  \begin{gather}
    \int_{\domain} \nabla u\cdot\nabla v \dx
    = \int_{\domain} fv\dx
    - \int_{\domain} \nabla u_g\cdot\nabla v \dx,
    \qquad\forall v\in V_0.
  \end{gather}
  The function $u_0$ depends on the choice of $u_g$, but not the minimizer $u$.
\end{Lemma}

\begin{Notation}{l2}
  The inner product of $L^2(\domain)$ is denoted by
  \begin{gather*}
    \form(u,v) \equiv \form(u,v)_{\domain}
    \equiv \form(u,v)_{L^2(\domain)}
    = \int_{\domain} u v \dvx.
  \end{gather*}
  Its norm is
  \begin{gather*}
    \norm{u} \equiv \norm{u}_{\domain} \equiv \norm{u}_{L^2(\domain)}
    \equiv \norm{u}_{L^2} = \sqrt{\form(u,v)_{L^2(\domain)}}.
  \end{gather*}
\end{Notation}
\begin{Lemma*}{Friedrichs-continuous}{Friedrichs inequality}
  For any function in $v\in V_0$ there holds
  \begin{gather}
      \norm{v}_{\domain}
      \le \diam(\domain) \norm{\nabla v}_{\domain}.
  \end{gather}
\end{Lemma*}

\begin{Lemma}{h1-norm}
  The definitions
  \begin{gather}
    \begin{split}
      \abs{v}_1 &= \norm{\nabla v}_{L^2(\domain)},\\
      \norm{v}_1 &= \sqrt{\norm{v}^2_{L^2(\domain)}
        + \abs{v}^2_1},
    \end{split}
  \end{gather}
  both define a norm on $V_0$.
\end{Lemma}

\begin{Problem}{Friedrichs}
  Prove the Friedrichs inequality.
\end{Problem}

\begin{Lemma}{Dirichlet-energy-boundedness}
  The Dirichlet energy with homogeneous boundary conditions is bounded
  from below and thus has an infimum. In particular, there exists a
  \define{minimizing sequence} $\{u^n\}$ such that as $n\to\infty$,
  \begin{gather}
    E(u^n) \to \inf_{v\in V_0} E(v).
  \end{gather}
\end{Lemma}

\begin{Lemma}{minimizing-sequence}
  The minimizing sequence for the Dirichlet energy is a
  \putindex{Cauchy sequence}.
\end{Lemma}

\begin{Definition}{h10}
  The completion of $V_0$ under the norm $\norm{v}_1$ is the
  \define{Sobolev space} $H^1_0(\domain)$.
\end{Definition}

\begin{Lemma*}{Friedrichs-h1}{Friedrichs inequality}
  For any function in $v\in H^1_0$ there holds
  \begin{gather}
      \norm{v}_{\domain}
      \le \diam(\domain) \norm{\nabla v}_{\domain}.
  \end{gather}
\end{Lemma*}

\begin{proof}
  Let $v\in H^1_0(\domain)$.  We make use of the fact, that by
  definition of $H^1_0(\domain)$, there is a sequence $v_n \to v$ with
  $v_n \in V_0$. By~\slideref{Lemma}{Friedrichs-continuous},
  Friedrichs' inequality holds for $v_n$ uniformly in $n$. We conclude
  \begin{align*}
    \norm{v}_\domain &\le \norm{v - v_n}_\domain + \norm{v_n}_\domain \\
    & \le \norm{v - v_n}_\domain + \diam\domain \norm{\nabla v_n}_\domain \\
    & \le \norm{v - v_n}_\domain + \diam\domain
      \bigl(\norm{\nabla v_n - \nabla v}_\domain + \norm{\nabla v}_\domain\bigr)
  \end{align*}
  As $n\to\infty$, the norms of the differences converge to zero, such
  that the desired result holds in the limit.
\end{proof}

\begin{Definition}{weak-formulation}
  The \putindex{Dirichlet problem} for Poisson's equation in weak form
  reads: find $u\in H^1_g(\domain)$ such that
  \begin{gather}
    \int_{\domain} \nabla u\cdot\nabla v \dx
    = \int_{\domain} fv\dx, \qquad\forall v\in H^1_0(\domain).
  \end{gather}
\end{Definition}

\begin{Theorem}{weak-unique-solution-1}
  The weak formulation in \slideref{Definition}{weak-formulation} has
  a unique solution.
\end{Theorem}

\subsection{Boundary conditions in weak form}

\begin{Lemma}{neumann-weak}
  Let $u\in V=H^1(\domain)$ be a solution to the weak formulation
  \begin{gather}
    \int_{\domain} \nabla u\cdot\nabla v \dx
    = \int_{\domain} fv\dx, \qquad\forall v\in V(\domain).
  \end{gather}
  If $u\in C^2(\domain) \cap C^1(\overline{\domain})$ and $\domain$
  has $C^1$-boundary, then $u$ solves the boundary value problem
  \begin{gather}
    \begin{aligned}
      -\Delta u &= f &\qquad \text{in } &\domain\\
      \d_n u &= 0 &\text{on } &\d\domain.
    \end{aligned}
  \end{gather}
\end{Lemma}

\begin{Definition}{natural-bc}
  A boundary condition inherent in the weak formulation and not
  explicitly stated is called \define{natural boundary condition}. If
  boundary values are obtained by constraining the function space it
  is called \define{essential boundary condition}.

  We also call a boundary condition in strong form, if it is a
  constraint on the function space, and in weak form, if it is part of
  the weak formulation.
\end{Definition}

\begin{remark}
  Dirichlet and homogeneous Neumann boundary conditions are examples
  for essential and natural boundary conditions.
\end{remark}

\begin{Lemma}{mixed-bc-weak}
  The boundary value problem
  \begin{gather}
    \begin{aligned}
      -\Delta u &= f &\qquad \text{in } &\domain\\
      u &= 0 &\text{on } &\Gamma_D \subset \d\domain\\
      \d_n u + \alpha u &= g &\text{on } &\Gamma_R \subset \d\domain,
    \end{aligned}
  \end{gather}
  has the weak form: find $u\in V$ such that
  \begin{gather}
    \int_{\domain} \nabla u\cdot\nabla v \dx
    + \int_{\Gamma_R}\alpha u v \ds
    = \int_{\domain} f v\dx
    + \int_{\Gamma_R} g v \ds, \qquad\forall v\in V(\domain).
  \end{gather}
\end{Lemma}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{The Basics}

\subsection{Exercises}

\subsubsection{Exercise 2.1}
Let $a = nb + k$. If $n \geq 2$, then $\frac{a}{2} \geq b > a \mod b$ since $a \mod b \leq b-1$. For $n = 1$
we have
\begin{align*}
        k < b \implies \frac{k}{2} < \frac{b}{2} \implies k < \frac{b}{2} + \frac{k}{2} = \frac{a}{2}
\end{align*}
Since Euclid's algorithm ``alternates'' between performing divisions on $a$ and $b$ (since $a$ is replaced
by $b$ after a step), the number of divisions it performs is bounded by $2\log_2 a$.

\subsubsection{Exercise 2.2}
The maximum divisor of $a$ is bounded by $\sqrt{a}$. We can find all primes less than or equal to $\sqrt{a}$
in polynomial time by using a prime number sieve approach (e.g. Sieve of Eratosthenes). Once we have
the primes, the number of operations it takes to compute the prime factorization of $a$ is bounded by
$\sqrt{a} \log_2 a$, since the number of primes is bounded by $\sqrt{a}$ and  2 is the smallest prime.

\subsubsection{Exercise 2.3}
Changing logarithm base amounts to multiplying by a constant.

\subsubsection{Exercise 2.4}
We have the following
\begin{itemize}
        \item $4\log_2 n = \log_2 n^4 \implies n^4$
        \item $4\sqrt{n} = \sqrt{16n} \implies 16n$
        \item $4n \implies 4n$
        \item $4n^2 = (2n)^2 \implies 2n$
        \item $4 * 2^n = 2^{n+2} \implies n + 2$
        \item $4 * 4^n = 4^{n+1} \implies  n + 1$
\end{itemize}

\subsection{Problems}

\subsubsection{Problem 2.18}
An adjacency matrix requires a single bit for each vertex pair, so we need $n^2$ bits to specify it. A list
of edges consists of  $m$ tuples of the form $(a, b)$ for $1 \leq a, b \leq n$, so it requires $2m\log n$
bits. Thus, we are better off using the adjacency list format for sparse graphs and the adjacency matrix
format for dense graphs. If, however, we consider multigraphs, then the adjacency matrix requires $n^2 \log n$
bits (as we need to store the number of edges from $a$ to $b$). Thus, in the multigraph case, we can always
choose to use adjacency lists.

\subsubsection{Problem 2.19}
Suppose $f(n) = O(n^a)$ and $g(n) = O(n^b)$. Then we have that
\begin{align*}
        \exists c_1, c_2, N \: | \: f(n) &\leq c_1 n^a, \: g(n) \leq c_2 n^b \quad \forall n \geq N\\
        \implies g(f(n)) &\leq g(c_1 n^a) \leq c_2 (c_1 n^a)^b = O(n^{ab})
\end{align*}
so $\text{poly}(n)$ is closed under composition. Thus, any polynomial time algorithm that calls another
polynomial time algorithm remains polynomial.

\subsubsection{Problem 2.20}
Since $f(n) = 2^{\Theta(\log^k n)}$ and $g(n) = O(n^a)$, there exists $N$ such that for all $n \geq N$
we have $f(n) \geq 2^{c_1 \log^k n}$ and $g(n) \leq c_2 n^a$ for some constants $c_1, c_2$. Thus,
\begin{align*}
        \lim_{n \to \infty} \frac{f(n)}{g(n)} &\geq \lim_{n \to \infty} \frac{2^{c_1 \log^k n}}{c_2 n^a} \\
                                              &= \lim_{n \to  \infty} \frac{(n^{c_1})^{\log^{k - 1} n}}{c_2 n^a}
\end{align*}
Since $\lim_{n \to \infty}\log^{k - 1} n = \infty$, we have that $f(n) = \omega(g(n))$. To see that
$f(n) = o(h(n))$, we substitute $n = 2^x$ to get
\begin{align*}
        \lim_{x \to \infty} \frac{f(2^x)}{g(2^x)} &\leq \lim_{x \to \infty} \frac{2^{a_1 x^k}}{2^{a_2 2^{c x}}} \\
                                                  &= 0
\end{align*}
Where we used the fact that $\lim_{x \to \infty} \frac{a_1 x^k}{a_2 2^{c x}} = 0$. Finally, to see that
$\text{QuasiP}$ is closed under composition, we can check that
\begin{align*}
        2^{\log^{k_1} 2^{\log^{k_2} n}} = 2^{\log^{k_1 k_2} n}
\end{align*}


\chapter{Classical Computational Complexity Calculus }

\section{Introduction and Motivation}
This chapter will present the traditional methodology of calculus in the field of algorithm's computational complexity. The complexity will be expressed as a function $f:\mathbb{N}\longrightarrow\mathbb{R}$, where the function is characterized by the size of the input, while the evaluated value $f(n)$, for a given input size $n$, represents the amount of resources needed in order to compute the result. While the most often analyzed resource is time, expressed in primitive computational operations, the mathematical model is self-reliant for others natures of resources, including space reasoning or hybrid metrics.

All the results of this chapter are well known in the literature and they represent the reference standard in calculating algorithm complexity in the field \textit{Computer Science}.

Because of the difficulty of having a rigorous calculus of the complexity, an asymptotic computation approach for an algorithm's complexity offers a valuable insight about the real computational cost without requiring a precisely, flawless calculus.

\section{Family of Bachmann–Landau notations}
The following notations and names will be used for describing the asymptotic behavior of a algorithm's complexity characterized by a function, $f:\mathbb{N}\longrightarrow\mathbb{R}$. \\
We define the set of all complexity calculus $\mathcal{F}= \lbrace f:\mathbb{N}\longrightarrow\mathbb{R} \rbrace$
\\Assume that $n, n_{0}\in\mathbb{N}$. Also, we will consider an arbitrary complexity function $g \in \mathcal{F}$. \\

\begin{definition}
    Assume that two continuous and derivable functions $v,w:\mathbb{R}\longrightarrow\mathbb{R}$ are considered to be similar in an \textbf{asymptotic analysis} iff:
    \[\lim_{x\to\infty} \frac{v(x)}{w(x)} = c \in (-\infty, 0) \cup (0,\infty) \]
\end{definition}

\begin{lemma}
    Consider that if we analyses asymptotic behavior of functions defined over $\mathbb{N}$ the functions $v,w:\mathbb{N}\longrightarrow\mathbb{R}$ are similar iff there exists a function $r:\mathbb{N}\longrightarrow\mathbb{R}$, with $r(x) = \frac{v(x)}{w(x)}\ \forall\ x\in\mathbb{N}$, such that
    \[\lim_{x\to\infty} r(x) = \lim_{x\to\infty} \frac{v(x)}{w(x)} = C \in (-\infty, 0) \cup (0,\infty) \]
\end{lemma}

The following function classes can be therefore defined for any function $g:\mathbb{N}\longrightarrow\mathbb{R}$ that describes a convergent sequence or a divergent sequence with a limit that tends to infinity:
\begin{definition}
    \textbf{Big Theta}: This set defines the group of mathematical functions similar in magnitude with  $g(n)$ in the study of asymptotic behavior. A set-based description of this group can be expressed as:
    \[\Theta(g(n))= \lbrace f \in \mathcal{F}\ |\ \exists c_{1}, c_{2} \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\ \ c_{1} \cdot g(n) \leq f(n) \leq c_{2} \cdot g(n)\ ,\  \forall n \geq n_{0} \rbrace\]
\end{definition}

\begin{lemma}
    Another useful method of establishing a Big-Theta acceptance, if the composed sequence defined by the ratio of the two functions exists and also $ \lim_{x\to\infty} \frac{f(x)}{g(x)}$ exists, is by applying the sufficient condition:
    \[ f \in \Theta(g(n))\ \Leftrightarrow\ \lim_{x\to\infty} \frac{f(x)}{g(x)} = C \in (-\infty, 0) \cup (0,\infty) \]
\end{lemma}

Remark that both $f,g$ can define divergent sequence with limits that tend to infinity, while the sequence $r(x) = \frac{f(x)}{g(x)}\ \forall\ x\in\mathbb{N}$ can be convergent and can have a nonzero limit. \\
Iff $ r(x) = \frac{f(x)}{g(x)}\ $ exists, a sufficient condition can be defined using the formal limit definition for an arbitrary sequence if the following condition holds:
\[\exists \  C \in \mathbb{R}, \ \ C \neq 0 \ s.t. \ \forall \epsilon > 0,\exists n_{0} \in \mathbb{N}\ \ s.t.\ \forall n\geq n_{0}, \ \  |r(n) - C| < \epsilon  \]
Thus, if $ \exists \  C \in \mathbb{R}, C \neq 0 \  \Rightarrow\  f \in \Theta(g(n))\ $

\begin{definition}
    \textbf{Big O}: This set defines the group of mathematical functions that are known to have a similar or lower
    asymptotic performance in comparison with  $g(n)$. The set of such functions is defined as it follows:
    \[\mathcal{O}(g(n)) = \lbrace f \in \mathcal{F}\ |\ \exists c \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) \leq c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}
\begin{lemma}
    In order to establish a Big-O acceptance, the sufficient condition is that the $ \lim_{x\to\infty} \frac{f(x)}{g(x)}$ exists and it is finite.
    \[ f \in \mathcal{O}(g(n))\ \Leftrightarrow\ \lim_{x\to\infty} \frac{f(x)}{g(x)} = C \in (-\infty, \infty) \]
\end{lemma}

\begin{definition}
    \textbf{Big Omega}: This set defines the group of mathematical functions that are known to have a similar or higher asymptotic performance in comparison with  $g(n)$. The set of all function is defined as:
    \[\Omega(g(n)) = \lbrace f \in \mathcal{F}\ |\ \exists c \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) \geq c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}

\begin{lemma}
    In order to establish a Big-Omega acceptance, a sufficient condition is one of the following:
    \begin{enumerate}[label=(\roman*)]

        \item  \[  \lim_{x\to\infty} \frac{f(x)}{g(x)} = -\infty \Rightarrow f \in \Omega(g(n))\ \]
        \item  \[ \lim_{x\to\infty} \frac{f(x)}{g(x)} = \infty \Rightarrow f \in \Omega(g(n))\  \]
        \item  \[ \lim_{x\to\infty} \frac{f(x)}{g(x)} = C \in (-\infty, 0) \cup (0,\infty) \Rightarrow f \in \Omega(g(n))\ \]
    \end{enumerate}
\end{lemma}

\begin{definition}
    \textbf{Small O}:
    This set defines the group of mathematical functions that are known to have a humble
    asymptotic performance in comparison with  $g(n)$. The set of such functions is defined as it follows:
    \[o(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) < c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}
\begin{lemma}
    In order to establish a Small-O acceptance, the sufficient condition is that the $ \lim_{x\to\infty} \frac{f(x)}{g(x)}$ exists and it is finite.
    \[ f \in o(g(n))\ \Leftrightarrow\ \lim_{x\to\infty} \frac{f(x)}{g(x)} = 0 \]
\end{lemma}

\begin{definition}
    \textbf{Small Omega}:
    This set defines the group of mathematical functions that are known to have a commanding asymptotic performance in comparison with  $g(n)$.
    The set of such functions is defined as it follows:
    \[\omega(g(n)) = \lbrace f \in \mathcal{F}\ |\ \forall c \in \mathbb{R}^{*}_{+}, \exists n_{0} \in \mathbb{N}^{*}\ s.t.\  f(n) > c \cdot g(n),\  \forall n \geq n_{0} \rbrace\]
\end{definition}
\begin{lemma}
    In order to establish a Small-Omega acceptance, the sufficient condition is that the $ \lim_{x\to\infty} \frac{f(x)}{g(x)}$ exists and one of the following occurs:
    \begin{enumerate}[label=(\roman*)]
        \item \[  \lim_{x\to\infty} \frac{f(x)}{g(x)} = +\infty \Rightarrow f \in \omega(g(n))\ \]
        \item  \[  \lim_{x\to\infty} \frac{f(x)}{g(x)} = -\infty \Rightarrow f \in \omega(g(n))\ \]
    \end{enumerate}
\end{lemma}

\section{Common properties}
We will study few aspects~\cite{mogocs2015new};
with impact on defining equivalence relations over classes defined in \textit{Bachmann–Landau} notations.
\hfill\break
\begin{theorem}
    The reflexivity property holds for Big Theta, Big O and Big Omega notations.  \\  $ f \in \Theta(f(n)) $ \\$ f \in \mathcal{O}(f(n)) $ \\$ f \in \Omega(f(n)) $
\end{theorem}

\begin{remark}
    The reflexivity property does not hold for Small O and Small Omega notations \\
    $ f \notin \theta(f(n)) $ \\$ f \notin \omega(f(n)) $
\end{remark}

\begin{theorem}
    The transitivity property holds for \textit{Bachmann–Landau} notations.  \\  $ f \in \Theta(g(n)), g \in \Theta(h(n)) \Rightarrow  f \in \Theta(h(n))$ \\
    $ f \in \mathcal{O}(g(n)), g \in \mathcal{O}(h(n)) \Rightarrow  f \in \mathcal{O}(h(n))$ \\
    $ f \in \Omega(g(n)), g \in \Omega(h(n)) \Rightarrow  f \in \Omega(h(n))$ \\
    $ f \in o(g(n)), g \in o(h(n)) \  \Rightarrow  f \in o(h(n))$ \\
    $ f \in \omega(g(n)), g \in \omega(h(n)) \Rightarrow  f \in \omega(h(n))$
\end{theorem}

\begin{theorem}
    The symmetry property holds for Big Theta notation. \\  $ f \in \Theta(g(n)) \Rightarrow g \in \Theta(f(n)) $
\end{theorem}

\begin{theorem}
    The transpose symmetry property for \textit{Bachmann–Landau} notations.  \\  $ f \in \mathcal{O}(g(n)) \Leftrightarrow g \in \Omega(f(n)) $
    \\  $ f \in o(g(n)) \Leftrightarrow g \in \omega(f(n)) $
\end{theorem}

\begin{theorem}
    The projection property for Big Theta, Big O and Big Omega notations. \\  $ f \in \Theta(g(n)) \Leftrightarrow f \in \mathcal{O}(g(n)), f \in \Omega(g(n)) $
\end{theorem}

\section{Addition properties}
Calculus in $Bachmann-Landau$ notations (including Big-O arithmetic) is extremely powerful and when it comes to addition operations, it is straightforward. The following relations hold for any correctly defined functions $f, g, h:\mathbb{N}\longrightarrow\mathbb{R}$ :

\begin{lemma}
    Addition in \textbf{Big Theta}

    If  $ h(n) = f(n) + g(n)\  \forall n \in  \mathbb{N} $ and $\exists n_{0} \in \mathbb{N},\ s.t.\ \forall n \in \mathbb{N} \ \Rightarrow f(n) \leq g(n)$, then:
    \[  h \in \Theta(g)\]

\end{lemma}
\begin{remark}
    For each theorem, we will expose a corollary using a relax notation (consider that by any $Bachmann-Landau$ class notation, we denote an arbitrary function part of the class), some relations that can be settled.
\end{remark}
\begin{corollary}
    Addition in \textbf{Big Theta} using the relaxed notation:
    \[  \Theta(f) + \Theta(g) = \Theta(g)\]
\end{corollary}

\begin{lemma}
    Addition in \textbf{Big O}:

    If  $ h(n) = f(n) + g(n)\  \forall n \in  \mathbb{N} $ and $\exists n_{0} \in \mathbb{N},\ s.t.\ \forall n \in \mathbb{N} \ \Rightarrow f(n) \leq g(n)$, then:
    \[  h \in \mathcal{O}(g)\]

\end{lemma}
\begin{corollary}
    Addition in \textbf{Big O} using the relaxed notation:
    \[  \mathcal{O}(f) + \mathcal{O}(g) = \mathcal{O}(g)\]
\end{corollary}

\begin{lemma}
    Addition in \textbf{Big Omega}:

    If  $ h(n) = f(n) + g(n)\  \forall n \in  \mathbb{N} $ and $\exists n_{0} \in \mathbb{N},\ s.t.\ \forall n \in \mathbb{N} \ \Rightarrow f(n) \leq g(n)$, then:
    \[  h \in \Omega(g)\]

\end{lemma}
\begin{corollary}
    Addition in \textbf{Big Omega} using the relaxed notation:
    \[  \Omega(f) + \Omega(g) = \Omega(g)\]
\end{corollary}

\begin{lemma}
    Addition in \textbf{Small O}:

    If  $ h(n) = f(n) + g(n)\  \forall n \in  \mathbb{N} $ and $\exists n_{0} \in \mathbb{N},\ s.t.\ \forall n \in \mathbb{N} \ \Rightarrow f(n) \leq g(n)$, then:
    \[  h \in o(g)\]

\end{lemma}
\begin{corollary}
    Addition in \textbf{Small O} using the relaxed notation:
    \[  o(f) + o(g) = o(g)\]
\end{corollary}

\begin{lemma}
    Addition in \textbf{Small Omega}:

    If  $ h(n) = f(n) + g(n)\  \forall n \in  \mathbb{N} $ and $\exists n_{0} \in \mathbb{N},\ s.t.\ \forall n \in \mathbb{N} \ \Rightarrow f(n) \leq g(n)$, then:
    \[  h \in \omega(g)\]

\end{lemma}
\begin{corollary}
    Addition in \textbf{Small Omega} using the relaxed notation:
    \[  \omega(f) + \omega(g) = \omega(g)\]
\end{corollary}


\section{Designing a Greatest Common Divisor Circuit}
\subsection{Theory}
Greatest Common Divisor (GCD) of two non zero integers is the largest number that is a common divisor of both integers. In this implementation, we are more focused on circuit size and speed rather than optimizing the algorithm. The common way of computing the GCD can be accomplished by the following algorithm:

\begin{algorithm}
\caption{Computing GCD algorithm}\label{euclid gcd}
\begin{algorithmic}[1]
\Procedure{GCD(A, B)}{}
\BState \emph{begin}:
\If {(A==B)} \textbf{goto} \emph{end}
\EndIf
\If {(A>B)} $\textit A \gets A-B$
\EndIf
\If {(B<A)} $\textit B \gets B-A$
\EndIf
\State \textbf{goto} \emph{begin}
\BState \emph{end}:
\State $\textit return \ A$
\EndProcedure
\end{algorithmic}
\end{algorithm}

The algorithm subtracts the largest number between the two inputs, from the other one. It will then update the largest input by the result of subtraction. This procedure will be continued until the two numbers are equal. This algorithm can be recursively implemented in System-Verilog as follows:

\begin{minted}{verilog}
function int gcd_fcn(int a, int b);
    return b == 0 ? a : gcd_fcn(b, a % b);
endfunction
\end{minted}

\subsection{Implementation}
The behavioral model presented in previous section cannot be implemented in hardware. For that we need to extract the essential computational blocks from the algorithm. We are also going to need a state machine to keep track of states we are in. The state machine should also compute the next state. Adding state machine would increase our design size. This is due to the gate level implementation of registers that are required by the state machine to hold values between clock cycles. This means that one of the best ways of reducing the design size is to reduce the number of states that we need. In the first implementation of GCD, we used 3 states to compute the GCD. The following listing shows only the combinational logic part (the source code for this implementation can be found in \href{https://github.com/hossein1387/random_hw_experiments/blob/master/GCD/GCDopt.sv}{link to GCD repository}):

\begin{minted}{verilog}
    case (state)
      s0: begin
            a_next      = a_in;
            b_next      = b_in;
            gcd         = 32'd0;
            state_next  = s1;
            done        = 0;
        end
      s1: begin
        if(a!=b) begin
            a_next= (a >= b) ? a - b : a;
            b_next     = (b >  a) ? b - a : b;
            gcd        = 32'd0;
            state_next = s1;
        end else begin
            state_next = s2;
            end
        end
      s2: begin
            gcd = a;
            state_next = s0;
            done = 1;
        end
        default: begin
            state_next = s0;
        end
    endcase
\end{minted}

In the code above, \mintinline{latex}{s0} samples the input and pulls down the $done$ signal (that indicates the output is ready to be sampled). In \mintinline{latex}{s1}, we subtract the largest number from the other one and update the next state. Finally \mintinline{latex}{s2} changes the next state to \mintinline{latex}{s0} and more importantly, pulls up the ``done'' signal and outputs the result on the output line. Figure \ref{SCHEMATIC} shows the schematic view of the circuit:

\begin{figure}[H]
    \centering
    \includegraphics[width=1.0\textwidth]{Images/schematic.png}
    \caption{High level Schematic view of GCD implementation showing combinational and sequential logic }
    \label{SCHEMATIC}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=1.0\textwidth]{Images/GCDNormal.png}
    \caption{GCD optimized}
    \label{GCDNORMAL}
\end{figure}

Figure \ref{GCDNORMAL} shows the waveform generated by GTKWave. As it is illustrated, GCD 14 and 161 is computed after 12 clock cycles and after that the ``done'' signal stays high for one clock cycle. As described earlier, by reducing number of states in the state machine, we can reduce the circuit size.

To reduce the number of states, we can combine states 1 and 2 so that only 1 bit is required to hold the state and we will end up with less number of registers. The following listing shows how this can be achieved:
\begin{minted}{verilog}
    case (state)
      s0: begin
            a_next      = a_in;
            b_next      = b_in;
            gcd         = 32'd0;
            state_next  = s1;
            done        = 0;
        end
      s1: begin
        if(a!=b) begin
            a_next= (a >= b) ? a - b : a;
            b_next     = (b >  a) ? b - a : b;
            gcd        = 32'd0;
            state_next = s1;
        end else begin
            gcd = a;
            state_next = s0;
            done = 1;
            end
        end
        default: begin
            state_next = s0;
        end
\end{minted}

Figure \ref{GCDOPT} shows that the same result is achieved but with less number of registers. The synthesis of this circuit is done using Yosys 3.0 available at
\href{https://www.edaplayground.com/x/4reN}{Synthesis Result}.

\begin{figure}[H]
    \centering
    \includegraphics[width=1.0\textwidth]{Images/GCDOpt.png}
    \caption{GCD optimized}
    \label{GCDOPT}
\end{figure}

\subsection{Results and Source Code}

\markdownInput{GCD.md}