\documentclass[12pt]{article}
\usepackage{lipsum}
\usepackage{authblk}
\usepackage{fancyhdr}

\usepackage{amssymb}
\usepackage{amsmath}

\usepackage[english]{babel}
\usepackage{tikz}
\usetikzlibrary{positioning}
\usepackage[colorlinks=true,
            urlcolor=blue,
            linkcolor=black,
            citecolor=black]{hyperref}
\usepackage[doublespacing]{setspace}
\usepackage{geometry}
  \geometry{
    a4paper,
    right=25mm,
    left=25mm,
    bottom=25mm,
    top=25mm
  }

\usepackage{amsmath}
\usepackage{listings}

\pagestyle{fancy}
\setlength{\headheight}{15pt}

\usepackage{algpseudocode}
\usepackage{algorithm}

%\usepackage{cite}
\usepackage[nottoc,numbib]{tocbibind}
\usepackage{natbib}

\usepackage{../presentation/colordef}
\usepackage{../presentation/lvblisting}

\usepackage{wrapfig}
\usepackage{booktabs}
%\raggedbottom
\usepackage{tikz}
\usetikzlibrary{shapes.geometric, arrows}

\tikzstyle{io} = [trapezium, trapezium left angle=70,
                  trapezium right angle=110, minimum width=3cm,
                  minimum height=0.5cm, text centered,
                  draw=black, fill=blue!30]
\tikzstyle{process} = [rectangle, minimum width=3cm,
                       minimum height=0.5cm, text centered,
                       draw=black, fill=orange!30]
\tikzstyle{decision} = [rectangle, minimum width=3cm,
                        minimum height=0.5cm, text centered,
                        draw=black, fill=green!30]
\tikzstyle{algo} = [circle, minimum width=0.5cm, draw=black, fill=orange!30]
\tikzstyle{arrow} = [thick,->,>=stealth]

\renewenvironment{abstract}{%

\begin{center}
\begin{minipage}{0.9\textwidth}
\rule{\textwidth}{1pt}}
{\par\noindent\rule{\textwidth}{1pt}
\end{minipage}
\end{center}}

\begin{document}

\title{Numerical Methods for solving Eigenvalue-Problems}
\author{Thomas Siskos}
\date{}

\begin{titlepage}
  \begin{center}

  \includegraphics[scale=1.25]{../presentation/hulogo.pdf} \par
  {\scshape\LARGE Humboldt Universit{\"a}t zu Berlin \par}

  {\scshape\Large Seminar Paper\par}

  {\huge\bfseries Numerical Methods for solving Eigenvalue-Problems\par}

\vspace{1cm}

  {\Large\itshape Thomas Siskos (580726)\par}

  {\Large\scshape Numerical Introductory Course\par}

  \vfill
  Supervised by: \par
  {\Large Prof. Dr. Brenda L{\'o}pez Cabrera \par}
  \vfill
  {\large \today\par}
  \end{center}

\end{titlepage}

\tableofcontents
\newpage
\listofalgorithms
\listoffigures
\listoftables
\newpage

\section{Motivation}

\begin{singlespacing}
\begin{abstract}
\centerline{\textbf{Abstract}} \\
\small
Eigenvalues and eigenvectors are often the solution to multidimensional optimization problems, however computing them by hand for anything but trivial matrices is most of the time infeasible or inpractical. To this extend we would like to deploy an automated procedure which yields the correct eigenvectors and eigenvalues. We demonstrate the relevance of eigenvalues and eigenvectors by revising two applications from statistics, Principal Component Analysis and Fisher's Linear Discriminant Analysis, which we follow up by investigating four algorithms suited for eigenvalue problems. Finally we provide a compound solution that takes advantage of each algorithm's strengths.
\end{abstract}
\vspace{3mm}
\end{singlespacing}

For many statistical applications eigenvectors provide a formidable solution. Be it dimensionality reduction in terms of a Principal Component Analysis or classification by Fisher's Linear Discriminant Analysis, both come in the guise of optimization problems. But what are eigenvalues and eigenvectors?

If $A$ is an $n \times n$ matrix, $v$ is a non-zero vector and $\lambda$ is a scalar, such that

\begin{equation}
\label{eigenvalue-def}
Av = \lambda v
\end{equation}

then $v$ is called an \textit{eigenvector} and $\lambda$ is called an \textit{eigenvalue} of the matrix $A$.
An eigenvalue of A is a root of the characteristic equation,

\begin{equation}
\label{eigenvalue-solve}
det\left(A - \lambda I \right) = 0.
\end{equation}

Each \textit{eigenvalue} is associated with an \textit{eigenvector}. Geometrically speaking, we require a vector which, when multiplied by matrix $A$, will not get rotated but only elongated by the \textit{eigenvalue} $\lambda$ \citep[p. 13]{NME}.

When confronted with a high-dimensional data matrix $X \in \mathbb{R}^{n \times m}$ an analyst often wishes to find a lower-dimensional representation, while conserving as much of the data's structure as possible. One way of achieving this goal is to choose a standardized linear combination of features that aim to maximize the variance of the projection $\delta^{\prime} X$. We can formalize this as

\begin{equation}
	\label{pca_obj}
    	max\ \delta^{\prime} Var \left(X\right) \delta \;\; s.t. \;\; \sum \delta_i^2 = 1.
\end{equation}

The solution to this optimization problem is given by
\begin{equation}
\label{pca_sol}
	Y = \Gamma^{\prime} \left(X - \mu\right)
\end{equation}

where $Y \in \mathbb{R}^{n \times m}$ is the matrix of rotations,
	  $\Gamma \in \mathbb{R}^{m \times m}$ is the matrix of eigenvectors,
	  $\mu \in \mathbb{R}^m$ is the vector of sample means \citep[p. 234]{MVA}.

A second use for Eigenvalues lies in Fisher's Linear Discriminant Analysis for classification problems. In this setting we would like to find the rotation that is best suited to separate the classes. To this effect we try to maximize the between-class-variance while simoultaneously trying to minimize the within-class variance. This is equivalent to maximizing the so called Rayleigh-coefficient

\begin{equation}
\label{LDA}
max J = \frac{J_b(w)}{J_w(w)} = \frac{w^{\prime} S_{b} w}{w^{\prime} S_{w} w}.
\end{equation}

The between and within class variance matrices are defined as $S_b = \sum\limits_{c=1}^{C}(\mu_c - \mu)(\mu_c - \mu)^{\prime}$ and $S_w = \sum\limits_{c=1}^{C}\sum\limits_{i \in c}(x_i - \mu_c)(x_i - \mu_c)^{\prime}$. The solution to this maximization problem is given by
\begin{equation}
S_B^{\frac{1}{2}} S_W^{-1} S_B^{\frac{1}{2}} v = \lambda v,
\end{equation}

which again is an eigenvalue problem of a square, symmetric matrix and can be readily computed \citep[p. 219]{duda}.

In the following we will try to extract the eigenvalues of matrices by reformulating any complicated eigenvalue problem into a straightforward one by diagonalizing the matrix in question, without altering the eigenvalues we would like to compute. In section two we provide the mathematical tools, which allow us to do so. We follow these justifications by proposing two main algorithms for computing eigenvalues, first the Jacobi-Method for symmetric matrices, then the QR-Method for arbitrary square matrices in section 3. Additionally, for the QR-Method we define two extensions which try to increase the initial QR-algorithm's speed.
For all algorithms we provide implementations in the \texttt{Python}-programming-language, all plots have been produced with the \texttt{matplotlib}-module and the custom algorithms rely solely on modules from \texttt{Python}'s \texttt{scipy} environment for scientific computing \citep{python, matplotlib, scipy}. All scripts are available \href{https://github.com/thsis/NIS18}{online}. In section 4 we will analyse the implemented routines by critically reflecting upon the accuracy of the obtained results as well as their efficiency. In the final section we provide a final algorithm which combines the strengths of the defined procedures by chosing the algorithm that is most fit for the underlying problem.
% ==============================================================================
\section{Similarity Transformations}

In general we want to reformulate the eigenvalue problem of a complicated matrix into an eigenvalue problem of a simple matrix, which yields the same eigenvalues. Simple matrices in our case will be diagonal matrices, since with them it is possible to identify their eigenvalues as entries on the main diagonal. A transformation that conserves the eigenvalues of a matrix is called a \textit{similarity transformation}.

Two $n \times n$ matrices $A$ and $B$ are called \textit{similar} if there exists an invertible matrix $P$ such that

\begin{equation}
\label{similarity}
A = P^{-1} B P.
\end{equation}

It is obvious that the similarity relationship is commutative as well as transitive. If $A$ and $B$ are similar, it holds that

\begin{align*}
B - \lambda I &= P^{-1} B P - \lambda P^{-1} I P \notag \\
              &= A - \lambda I.
\end{align*}

 Hence $A$ and $B$ have the same eigenvalues. This fact also follows immediately from the transitivity of the similarity relationship and the fact that a matrix is similar to the diagonal matrix formed from its eigenvalues, as stated in the spectral-decomposition. Important types of similarity transformations are based around orthogonal matrices. If $Q$ is orthogonal and

$$ A = Q^{\prime} B Q, $$

$A$ and $B$ are called \textit{orthogonally similar} \citep{NLA}. We will use \textit{orthogonal similarity transformations} to diagonalize matrices we wish to know the eigenvalues of. For our custom implementations we will make use of Givens-Rotations and Householder Reflectors (for a detailed discussion consider the appendix).

% ==============================================================================
\section{Algorithms}
% ------------------------------------------------------------------------------
\subsection{Jacobi Method}

The \textit{Jacobi-Method} for computing the eigenvalues of a symmetric matrix $A \in \mathbb{R}^{n \times n}$ deploys a sequence of orthogonal similarity transformations that eventually results in
$$ A = P \Lambda P^{-1} \Leftrightarrow \Lambda = P^{-1} A P,$$

where $\Lambda$ is diagonal and $P$ consists of a sequence of matrix multiplications $P = \prod\limits_{k=1}^{K} V_{p_k, q_k}(\theta_k)$ and $V_{p_k, q_k}(\theta_k)$ is a matrix that performs a Givens-Rotation. Such a rotation is performed by an identity matrix with four of its elements replaced, such that
\small
\begin{equation}
\label{givens_rotator}
V_{pq}(\theta) = \begin{bmatrix}
                      1 \\
                        & \ddots \\
                        &        & \cos\theta    &        & \sin\theta  \\
                        &        &               & \ddots     \\
                        &        & -\sin\theta   &        & \cos\theta   \\
                        &        &               &        &            &  \ddots \\
                        &        &               &        &            &         & 1 \\
                 \end{bmatrix}
\end{equation}
\normalsize
\newpage
The \textit{Jacobi iteration} is defined as

\begin{equation}
A^{(k)} = V^{\prime}_{p_k, q_k}(\theta_k) A^{(k-1)} V_{p_k, q_k}(\theta_k),
\end{equation}

where $p_k, q_k$ and $\theta_k$ are chosen such that $A^{(k)}$ resembles more a diagonal matrix than $A^{(k-1)}$. Specifically they will be chosen as to reduce the sum of squares of the off-diagonal elements. Also, note that $A^{(k)}$ and $A^{(k-1)}$ are similar, since $V^{\prime}_{p_k, q_k}(\theta_k)$ is an orthogonal matrix for every step $k$. The benefit of matrices as in (\ref{givens_rotator}) is, that it is easy to choose an angle $\theta_k$ in order to introduce a zero in a single multiplication step. Here we use the rotations in the context of a similarity transformation, so it is a little more complicated.

We require that $a_{pq}^{(k)} = 0$, this implies
\begin{equation}
\label{theta-troubles}
a^{(k-1)}_{pq} (\cos^2\theta - \sin^2\theta) + \left( a^{(k-1)}_{pp} - a^{(k-1)}_{qq} \right) \cos\theta \sin\theta = 0.
\end{equation}
We can use the trigonometric identities
\begin{align*}
\cos(2\theta) &= \cos^2 \theta \sin^2 \theta \\
\sin(2\theta) &= 2 \cos\theta \sin\theta,
\end{align*}

in (\ref{theta-troubles}) we have
$$\tan(2\theta) = \frac{2a^{(k-1)}_{pq}}{a^{(k-1)}_{pp} - a^{(k-1)}_{qq}}.$$

From this we can retrieve the angle and obtain the rotation matrix in each iteration \citep[p. 128]{NLA}.

The algorithm converges if the off-diagonal elements are sufficiently small. The best index pair at a given iteration is the pair $(p, q)$ that satisfies

\[
|a^{(k-1)}_{pq}| = \mathop{\max_{i<j}} |a^{(k-1)}_{ij}|.
\]

If this choice is made, the Jacobi Method can be shown to converge \citep[p. 128]{NLA}.

\begin{figure}[H]
\begin{center}
\caption{\href {https://github.com/thsis/NIS18/tree/master/media/plots}{Progress Jacobi-Method}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
  \label{j-plot}
  \includegraphics[scale=0.6]{../media/plots/jacobi.png}
\end{center}
\end{figure}

Figure \ref{j-plot} visualizes the progress of the \textit{Jacobi}-method on a symmetric $5 \times 5$ matrix. As we can see, in the first iteration the element $a_{43}$ is eliminated. In the subsequent operations the \textit{Jacobi}-method continues to eliminate any non-zero entries on the off-diagonal until the algorithm convergences after 10 iterations.

\begin{algorithm}
\caption{\href {https://github.com/thsis/NIS18/blob/master/algorithms/eigen.py}{\texttt{jacobi}}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}

\label{j-algo}
\begin{algorithmic}[1]
  \Require symmetric matrix $A$
  \Ensure $0 < precision < 1$
  \Statex \textbf{initialize: } $L \gets A$; $U \gets I$; $L_{max} \gets 1$
  \While{$L_{max} > precision$}
    \State Find indices $i$, $j$ of largest value in lower triangle of $abs(L)$
        \State $L_{max} \gets L_{i,j}$
            \State $\theta \gets \frac{1}{2}\cdot \arctan(\frac{2A_{i, j}}{A_{i, i}-A_{j, j}})$
    \State $V \gets I$
    \State $V_{i, i}, V_{j, j} \gets \cos \theta$; $V_{i, j}, V_{j, i} \gets -\sin \theta,\; \sin \theta$
    \State $A \gets V^{\prime} A V$; $U \gets UV$

  \EndWhile\\
  \Return $diag(A),\; U$
\end{algorithmic}
\end{algorithm}

% ------------------------------------------------------------------------------
\subsection{QR-Method}

The most widely used algorithm to extract eigenvalues is the so called \textit{QR}-method. The most important advantage of the \textit{QR}-method over the \textit{Jacobi}-method is that it can be applied to non-symmetric matrices. Note however, that it is simpler for symmetric matrices, since the eigenvalues are real-valued.

The \textit{QR}-method to extract the eigenvalues of a square matrix $A \in \mathbb{R}^{n \times n}$ is performed by first computing the titular \textit{QR} decomposition of $A$.

\begin{equation}
\label{qr_a}
A = QR,
\end{equation}

where $Q$ is an orthogonal and $R$ is an upper triangular matrix. Then define the \textit{QR} iteration as

\begin{equation}
\label{qr-method}
  A^k = Q_{k-1}^{\prime} A_{k-1} Q_{k-1} = R_{k-1}Q_{k-1}
\end{equation}

Note hereby that all matrices in the sequence $\{A_k\}$ share the same eigenvalues, since this procedure is a similarity transformation due to $Q$'s orthogonality \citep[p. 121]{NME}. Additionally, for practical applications it is preferable to compute the \textit{QR}-iteration in the way shown at the rightmost part of equation (\ref{qr-method}). Although, mathematically, each statement is exactly identical there is a non-negligible difference due to computational imperfections and limited machine precision. The reason is that the computation of $Q_{k-1}^{\prime} A_{k-1} Q_{k-1}$, obviously, requires two matrix multiplications whereas the result of $R_{k-1}Q_{k-1}$ can be readily obtained by one. When combining multiple steps over a long sequence of \textit{QR} iterations the additional computations lead to additional rounding errors, which can have an influence on the accuracy of the obtained results. Besides, less computations lead of course to a faster procedure in general.

\begin{figure}
\begin{center}
\label{qrm1-plot}
\caption{\href {https://github.com/thsis/NIS18/tree/master/media/plots}{Progress basic QR-Method}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
  \includegraphics[scale=0.6]{../media/plots/qrm1.png}
\end{center}
\end{figure}

Figure \ref{qrm1-plot} visualizes the progress of the basic \textit{QR}-method on the same $5 \times 5$ matrix as in Figure \ref{j-plot}. Compared to the \textit{Jacobi}-method it does not explicitly pick a single element that will be eliminated per iteration. Instead, the \textit{QR}-method extracts the eigenvalues by a process that is called \textit{chasing}. By that we mean that alternating steps are being performed, which create non-zero eintries in positions $(i+2, i)$, $(i+3, i)$ and $(i+3, i+1)$ and restore them to zero, as the nonzero entries are moved farther down the matrix \citep[p. 130]{NLA}. We can also see that compared to the \textit{Jacobi}-Method, so far, the \textit{QR}-algorithm lacks in speed. Where the \textit{Jacobi}-method was almost done diagonalizing the matrix in iteration 10, the basic \textit{QR}-algorithm still had multiple non-zero entries left. Thus we would like to make minor improvements on the algorithm's efficiency.
\begin{algorithm}
\caption{\href {https://github.com/thsis/NIS18/blob/master/algorithms/eigen.py}{\texttt{QRM1}}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
\label{qr1-meth}
  \begin{algorithmic}[1]
    \Require square matrix $A$
    \Statex \textbf{initialize: } $conv \gets False$
    \While{not $conv$}
      \State $Q, R \gets$ QR-Factorization of $A$
      \State $A \gets RQ$
      \If{$A$ is diagonal}
        \State $conv \gets \texttt{True}$
      \EndIf
    \EndWhile\\
    \Return $diag\left(A\right),\; Q$
  \end{algorithmic}
\end{algorithm}

\subsubsection{Hessenberg Variant}

In order to speed up the \textit{QR}-method it is advisable to transform the matrix to its upper \textit{Hessenberg} form. A matrix $A$ is of upper \textit{Hessenberg} form if it is upper triangular except for the first subdiagonal, which may be non-zero. In particular $a_{ij} = 0\; \forall i > j + 1$:
\small
$$
\begin{bmatrix}
X & X & X  & \dots &  X & X\\
X & X & X &  \dots &  X & X\\
0 & X & X &  \dots &  X & X\\
0 & 0 & X &  \dots &  X & X \\
\vdots &  \vdots & & \ddots  & \vdots  & \vdots\\
0 & 0 & 0 &  \dots &  X & X \\
\end{bmatrix}$$
\normalsize
A matrix can be reduced to \textit{Hessenberg} form in a finite number of similarity transformations using Householder transformations or Givens rotations. For symmetric matrices the transformation into a \textit{Hessenberg}-form results in a tridiagonal matrix. But even for non-symmetric matrices, the \textit{Hessenberg}-form allows a large saving in subsequent computations. After the transformation we can deploy the previously defined \textit{QR}-method \citep[p. 130]{NLA}.

\begin{figure}
\centering
\caption{\href {https://github.com/thsis/NIS18/tree/master/media/plots}{Progress Hessenberg-QR-Method}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
  \label{qr2-plot}
  \includegraphics[scale=0.6]{../media/plots/qrm2.png}
\end{figure}

Figure \ref{qr2-plot} visualizes the progress of the \textit{Hessenberg} variant of the \textit{QR}-method. In order to make it comparable to the previous algorithms the same $5 \times 5$-dimensional matrix is being evaluated. We can readily see, that the transformation to \textit{Hessenberg}-form results in a tridiagonal matrix. This facilitates computations and explains the vastly improved resulting matrix after 10 iterations compared to the basic \textit{QR}-method. However, it still does not match the progress of the \textit{Jacobi}-method after the same number of iterations.

\begin{algorithm}
\caption{\href {https://github.com/thsis/NIS18/blob/master/algorithms/eigen.py}{\texttt{QRM2}}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
\label{qr2-meth}
\begin{algorithmic}[1]
  \Require square matrix $A$
  \State $A \gets \texttt{hessenberg(}A\texttt{)}$
  \State continue with: \Call {QRM1} A
\end{algorithmic}
\end{algorithm}

\subsubsection{Accelerated Variant}

We could already improve the \textit{QR}-method and cut down on computational cost. However, we still cannot match the results of the \textit{Jacobi} method. To this effect we present one additional adjustment on the \textit{QR}-method that aims to improve convergence speed. The general idea is, that we deliberately create an additional zero entry on the main diagonal by subtracting a scalar on each element, perform the \textit{QR}-iteration and finally undo the subtraction. In particular, we define

\begin{equation}
\label{qrm3-prop}
T^{m} = \begin{bmatrix}

\alpha^{m}_1 & \beta^{m}_1  & 0            & 0            & \dots            & 0                                  \\
\beta^{m}_1  & \alpha^{m}_2 & \beta^{m}_2                                                                         \\
0            & \beta^{m}_2  & \alpha^{m}_3 & \beta^{m}_3  &                  & \vdots                             \\
             &              & \ddots       & \ddots       & \ddots                                                \\
             &              &              &              & \beta^{m}_{n-2}  & \alpha^{m}_{n-1} & \beta^{m}_{n-1} \\
0            &              &              &              &                  & \beta^{m}_{n-1}  & \alpha^{m}_n    \\

\end{bmatrix}
\end{equation}

\begin{align*}
T^{m}       &= T - t _{n, n} I \\
T^{m}       &= QR \\
T^{m+1}     &= T^{m} + t _{n, n} I
\end{align*}

After this we can define the accelerated iteration step as
\begin{align}
\label{qrm3}
R_m &= Q^{\prime}_m\left(T^m - \alpha_n^{m}I\right) \\
T^{m+1} &= Q^{\prime}_m\left(T^m - \alpha_n^{m}I\right)Q_m + \alpha_n^{m}I \\
        &= Q^{\prime}_m T^m Q_m
\end{align}

Again $T^{m+1}$ is similar to $T^m$.

\begin{figure}[H]
\centering
\caption{\href {https://github.com/thsis/NIS18/tree/master/media/plots}{Progress Accelerated QR-Method}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
  \label{qr3-plot}
  \includegraphics[scale=0.6]{../media/plots/qrm3.png}
\end{figure}

Figure \ref{qr3-plot} visualizes the progress of the accelerated \textit{QR}-method. For comparability, the matrix used, is the same $5 \times 5$ matrix as before. Most notably is, that the accelerated method still performs worse than the \textit{Jacobi} method. So far results seem similar compared to the \textit{Hessenberg} variant of the \textit{QR}-method. The case can be made that the accelerated method performs considerably better than the basic \textit{QR}-method and slightly better than the \textit{Hessenberg} variant after 10 iterations. Howbeit, this claim warrants further analysis.

\begin{algorithm}
\begin{algorithmic}[1]
\caption{\href {https://github.com/thsis/NIS18/blob/master/algorithms/eigen.py}{\texttt{QRM3}}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}

\Require square matrix $A \in \mathbb{R}^{p \times p}$
\State $T \gets \texttt{hessenberg}(A),\ conv \gets False$
\While{not $conv$}
    \State $Q, R \gets$ QR-Factorization of $T - t_{p-1, p-1} I$
    \State $T \gets RQ + t_{p-1, p-1}I$
    \If{$T$ is diagonal}
        \State $conv \gets True$
    \EndIf
\EndWhile\\
\Return $diag\left(T\right),\; Q$
\end{algorithmic}
\end{algorithm}

% ==============================================================================
\section{Analysis}

In order to test the quality of our implemented algorithms we will follow a two pronged approach. First, we will scrutinize the quality of the obtained results. Once we can be sure that the implemented routines are reliable or, to be more precise, under which conditions our routines are reliable we can analyze their convergence behavior and compare the efficiency accross algorithms in terms of computational cost.
% ------------------------------------------------------------------------------
\subsection{Accuracy}

If we wish to investigate the accuracy of our algorithms, we first need to define what we mean by that. We set up an environment of unit tests that compare the absolute difference between the vector of true eigenvalues of a matrix and the vector of computed eigenvalues by our custom algorithms. If any element of this $L_1$-norm is larger than a threshold of $10^{-5}$ we consider the test as failed. In order to obtain matrices with known eigenvalues we will reverse the spectral decomposition, which states that every square, symmetric matrix $A \in \mathbb{R}^{m \times m}$ can be written as
\begin{equation}
\label{spectral-decomp}
A = \Gamma \Lambda \Gamma^{\prime} = \sum\limits_{j=1}^m \lambda_j \gamma_j \gamma^{\prime}_j,
\end{equation}
where $\Lambda = diag(\lambda_1, \dots, \lambda_m)$ is a diagonal matrix with the eigenvalues of $A$ on its main diagonal and where $\Gamma=(\gamma_1, \dots, \gamma_m)$ is the matrix containing the associated eigenvectors \citep{MVA}. This means we can draw a random vector of eigenvalues $\lambda_{true}$ and a random orthogonal matrix to construct a test matrix $A_{test}$ \citep{lapack}. Once the matrix $A_{test}$ is initialized, we can plug it into the eigenvalue-routine we wish to test.

\begin{table}[b]
\centering
\caption{\href {https://github.com/thsis/NIS18/tree/master/tests/tests_eigen.py}{Failed unit tests accross matrix-sizes}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
\label{acc-res}
\scriptsize
\begin{tabular}{ccrrrrr}
\toprule
Algorithm & Maximum number & \multicolumn{5}{c}{Dimension}\\
\cline{3-7}
          & of iterations &    3 &    4 &    5 &     6 &     7 \\
\toprule
\bottomrule
\textit{Jacobi} & - &    0 &    0 &    0 &     0 &     0 \\
\textit{QRM1}   & 10 &  785 &  952 &  994 &   999 &  1000 \\
                & 100 &  110 &  224 &  349 &   474 &   565 \\
                & 1000 &    7 &   16 &   33 &    38 &    45 \\
                & 10000 &    1 &    1 &    1 &     1 &     3 \\
                & 100000 &    0 &    0 &    0 &     0 &     0 \\
\textit{QRM2}   & 10 &  826 &  975 &  997 &  1000 &  1000 \\
                & 100 &  111 &  191 &  360 &   467 &   602 \\
                & 1000 &    9 &   16 &   22 &    35 &    58 \\
                & 10000 &    0 &    0 &    3 &     2 &     0 \\
                & 100000 &    0 &    0 &    0 &     0 &     0 \\
\textit{QRM3}   & 10 &  251 &  616 &  896 &   983 &   997 \\
                & 100 &   29 &   52 &  116 &   222 &   311 \\
                & 1000 &    6 &    2 &    8 &    19 &    21 \\
                & 10000 &    0 &    0 &    1 &     1 &     3 \\
                & 100000 &    0 &    0 &    0 &     0 &     0 \\
\bottomrule
\end{tabular}
\normalsize
\end{table}

The number of failures out of 1000 unit tests on matrices of different dimensions are catalogued in Table \ref{acc-res}. Most notably, we see that the \textit{Jacobi}-method outperforms any of the \textit{QR}-methods in terms of accuracy. This is mostly due to its convergence criterion being defined differently than the convergence criterion of the \textit{QR}-methods. Simply put, the \textit{Jacobi}-method will always converge in this setup. Also keep in mind, that since the \textit{Jacobi}-method is only allowed for symmetric matrices, it essentially eliminates two non-zero off-diagonal elements at once. Comparing only the \textit{QR}-methods amongst each other one can clearly see, that the quality of the obtained results increases, the more willing an analyst is to wait. Convergence is slower for the \textit{QR}-methods and a premature stop leads to poor eigenvalues. One the other hand, if the \textit{QR}-methods converge, they unanimously pass all tests. It would therefore be possible to always achieve an arbitrary grade of precision by allowing for a higher number of iterations. For the highest possible accuracy one can set the maximum number of iterations to infinity.

% ------------------------------------------------------------------------------
\subsection{Efficiency}

\begin{figure}[b]
\centering
\caption{\href {https://github.com/thsis/NIS18/blob/master/tests/tests_eigen.py}{Unit-tests: Iterations}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
  \label{algo_efficiency}
  \includegraphics[width=\textwidth, height=6.5cm]{../media/plots/iterations_boxplot.png}
\end{figure}

Now that we can be confident that each of the routines is able to offer accurate results we are interested in which method is most appropriate for which kind of matrix. Especially, we are interested in convergence speed. Figure \ref{algo_efficiency} shows the distribution of steps performed before the algorithms reached convergence or their maximum allowed number of iterations. Again, the \textit{Jacobi}-method outperforms every \textit{QR}-method. For matrices of small dimensions, i.e. eleven or less, the accelerated \textit{QR}-method outperforms the other \textit{QR}-methods. However, for larger matrices the \textit{Hessenberg} variant seems to be slightly more appropriate, since the accelerated method tends to have a wider range of needed steps. Additionally, the accelerated method's distribution has a wider tail whenever the dimension of the test matrices is larger than eleven. This means that high waiting times, operating on the same matrix, are more likely to occur using the \textit{accelerated} variant compared to the \textit{Hessenberg} variant or even the basic variant. This can only mean that the supplementary additions and subtractions of the accelerated method provide a questionable aid. In some cases they do seem to help for small matrices, however when confronted with larger matrices the cost of the additions and subtractions outweighs the benefit of the facilitated \textit{QR}-step. Surprisingly, there seems to be no striking difference between the distributions of the basic and the \textit{Hessenberg} variants. Looking closely, the only effect of performing a \textit{Hessenberg} transformation prior to running a \textit{QR}-algorithm on a test matrix is to mitigate outliers in terms of computation steps needed before convergence. And even that is true, only for large matrices. This can only mean that the cost of computing the \textit{Hessenberg}-form of the matrix is roughly as high as the reduction of steps needed before convergence is reached.

% ==============================================================================
\section{Conclusion}
\begin{figure}
\centering
\caption{\href {https://github.com/thsis/NIS18/tree/master/algorithms/eigen.py}{Decision process of the final eigenvalue routine}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
\begin{tikzpicture}
\node (in1) [io] {Input: matrix A};
\node (dec_sym) [decision, below of=in1, yshift=-0.25cm] {Is A symmetric?};
\node (algo1) [algo, below of=dec_sym, xshift=-2cm, yshift=-0.5cm] {\texttt{Jacobi}};
\node (dec_small) [decision, below of=dec_sym, xshift=2cm, yshift=-0.5cm] {rows of A $\leq$ 11?};
\node (algo2) [algo, below of=dec_small, xshift=-1cm, yshift=-0.5cm] {\texttt{QRM3}};
\node (algo3) [algo, below of=dec_small, xshift=1cm, yshift=-0.5cm] {\texttt{QRM2}};

\draw [arrow] (in1) -- (dec_sym);
\draw [arrow] (dec_sym) -- node[anchor=east] {yes} (algo1);
\draw [arrow] (dec_sym) -- node[anchor=west] {no} (dec_small);
\draw [arrow] (dec_small) -- node[anchor=east] {yes} (algo2);
\draw [arrow] (dec_small) -- node[anchor=west] {no} (algo3);
\end{tikzpicture}
\end{figure}

Finally, we would like to present a compound solution which combines the strengths of all presented algorithms and automatically picks the procedure that fits best to the problem. For obvious reasons, it is best to chose the \textit{Jacobi}-method, whenever applicable. That is, when the input-matrix is symmetric. Whenever the matrix is not symmetric, a second decision based on the matrix' size will determine which \textit{QR}-method will be used. If the matrix' dimension is smaller than eleven it will chose the accelerated \textit{QR} method. Otherwise, the \textit{Hessenberg} variant of the \textit{QR}-method will be deployed.

We have thus demonstrated the practical importance of eigenvalues in the context of Principal Component Analysis and Fisher's Linear Discriminant Analysis. For computing the eigenvalues we have shown various numerical procedures. We examinated the \textit{Jacobi}- and the \textit{QR}-method, as well as a selection of simple improvements towards the basic \textit{QR}-method. We have shown that the implemented routines can achieve an arbitrary grade of precision and provided a guided procedure for when to use which of the provided algorithms.

% ==============================================================================
\newpage
\section{Appendix}
\subsection{Householder-Reflections}

Our goal is to diagonalize a matrix in order to programmatically extract its eigenvalues. In section 2 we have seen that there exist such transformations that conserve the eigenvalues of a given matrix. However, we require transformations that, additionally, eliminate non-zero entries on the off-diagonal elements of a particular matrix. A greedy technique, that eliminates all but the first elements of a vector is proposed in the form of Householder-Reflections.

Let $u$ and $v$ be orthonormal vectors and let $x$ be a vector in the space spanned by $u$ and $v$, such that
$$x = c_1 u + c_2 + v$$
for some scalars $c_1$ and $c_2$. The vector
$$\tilde{x}=-c_1 u + c_2 v$$
is a \textit{reflection} of x through the line difined by the vector u. Now consider the matrix

\begin{equation}
P = I - 2 uu^{\prime}.
\end{equation}
Note that
\begin{align*}
Px &= c_1 u + c_2 v - 2c_1 uuu^{\prime} - 2 c_2 v uu^{\prime} \\
   &= c_1 u + c_2 v - 2c_1 u^{\prime}uu - 2 c_2 u^{\prime} v u \\
   &= -c_1 u + c_2 v\\
   &= \hat{x}.
\end{align*}

The matrix $P$ is called a reflector. The usefulness of Householder-Reflections stems from the fact that it is easy to transform a vector of the form

$$x = (x_1, x_2, \dots, x_n)$$

into a vector
$$\hat{x} = (\hat{x}_1, 0, \dots, 0).$$

If $Qx = \hat{x}$, then $||x||_2 = ||\hat{x}||_2$ and thus $\hat{x_1} = \pm ||x||_2$, since it is the only non-zero entry. To construct the reflector let
\begin{equation}
\label{house-con}
v = (x_1 + sign(x_1)||x||_2, x2, \dots, x_n)
\end{equation}
and $u = \frac{v}{||v||_2}$ \citep{NLA}. We use the $sign$-function, which simply returns the sign of its argument in order to avoid the numerical problem known as \textit{catastrophic cancellation}. It can occur when adding two very close, but different, floating point numbers of differing signs. In some unfortunate cases both of these numbers get represented by the same computer number and, because of their opposing signs cancel each other out. In our case this would mean, that we reflect the vector onto the origin. Fortunately, by making use of the \textit{sign} function we can make sure that both summands will share the same sign, thus mitigating any concerns about catastrophic cancellation.

We use reflectors to compute the so called $QR$ factorization of an aribitrary square matrix $A \in \mathbb{R}^{n \times n}$.

\begin{equation}
\label{QR-prop}
A = QR
\end{equation}

where $Q$ is orthogonal and $R$ is upper triangular. We use Householder transformations to reflect the $i^{th}$ column and produce zeros below the $(i, i)$ element. The QR-factorization of a matrix $A \in \mathbb{R}^5$ would therefore consist of five Householder-reflections with $Q=P_5 P_4 P_3 P_2 P_1$. The number of computations for the \textit{QR} factorization in this fashion is $2n^3 / 3$ multiplications and $2n^3 / 3$ additions \citep[p. 110]{NLA}.
\newpage
% ------------------------------------------------------------------------------
\subsection{Givens-Rotations}

\begin{wrapfigure}[7]{r}{0.35\textwidth}
\centering
\caption{\href {https://github.com/thsis/NIS18/blob/master/media/plots}{Rotation of \textit{x}}  \protect\includegraphics[scale=0.05]{qletlogo.pdf}}
\includegraphics[scale=0.5]{../media/plots/givens.png}
\end{wrapfigure}

Another way of forming the \textit{QR}-factorization is by using orthogonal transformations which rotate a vector in a way such that a specific element becomes 0 and only one other element in the vector being changed. These transformations are called \textit{Givens transformations, Givens Rotations} or \textit{Jacobi transformations}

Using orthogonal transformations we can also rotate a vector in such a way that a specified element becomes 0 and only one other element in the vector is changed. The basic idea can be seen in a two-dimensional space. We wish to rotate the vector $x = (x_1, x_2)$ to $\tilde{x} = (\tilde{x_1}, 0)$ as with a reflector.

It is easy to see that the orthogonal matrix
$$Q=\begin{bmatrix}
\cos\theta & \sin\theta \\
-\sin\theta & \cos\theta
\end{bmatrix}$$

performs the desired rotation, if $\cos\theta = \frac{x_1}{||x||_2}$ and $\sin\theta = \frac{x_2}{||x||_2}$

In general, we can construct an orthogonal $matrix V_{pq}$, that will transform the vector $$x = (x_1,\dots, x_p, \dots x_q, \dots, x_n)$$ to $$\tilde{x} = (x_1,\dots, \tilde{x}_p, \dots 0, \dots, x_n)$$. The matrix that does this is
\scriptsize
\begin{equation}
\label{givens-rotation}
V_{pq}(\theta) = \begin{bmatrix}
                      1 \\
                        & \ddots \\
                        &        & \cos\theta    &        & \sin\theta  \\
                        &        &               & \ddots     \\
                        &        & -\sin\theta   &        & \cos\theta   \\
                        &        &               &        &            &  \ddots \\
                        &        &               &        &            &         & 1 \\
                 \end{bmatrix}
\end{equation}
\normalsize
	where $\cos\theta = \frac{x_p}{||x||}$ and $\sin\theta = \frac{x_q}{||x||}.$

A rotation matrix is therefore the same as an identitiy matrix, in which we change four elements \citep[p. 105]{NME}. We will use Givens rotations primarily in the Jacobi-Method.
\newpage
\subsection{Code}
\subsubsection{Eigenvalue Routines}
  \lstinputlisting[language=Python]{../algorithms/helpers.py}
  \lstinputlisting[language=Python]{../algorithms/eigen.py}
  \newpage
\subsubsection{Analysis: Figures}
  \lstinputlisting[language=Python]{../analysis/analysis.py}
  \newpage
\subsubsection{Analysis: Unit tests}
  \lstinputlisting[language=Python]{../tests/tests_eigen.py}
  \newpage
% ==============================================================================
\bibliographystyle{apalike}
\bibliography{references}

\end{document}


% Note on compiling: requires external programs dot (from the graphvix package
% on Debian/Ubuntu) and dot2tex.
% You need to pass the --shell-escape option to pdflatex
% (or use the Makefile, which required latexmk).
%\documentclass[aspectratio=169,handout]{beamer}
%\documentclass[aspectratio=169,handout,hyphens]{beamer} % hyphens option for url package which beamer loads.
\documentclass[aspectratio=169,hyphens]{beamer} % hyphens option for url package which beamer loads.

\def\UrlFont{\small\tt}

\usepackage{minted}
\usepackage{color}
%\usepackage[pgf,dot]{dot2texi}
%\usepackage{tikz}
%\usetikzlibrary{shapes,arrows}

\usetheme{Pittsburgh}
\usecolortheme{beaver}
% \useoutertheme{infolines}

\title{Constraint Programming in Haskell}
\subtitle{Melbourne Haskell Users Group}
\author{David Overton}
\date{29 October 2015}

\AtBeginSection[]
{
	\begin{frame}
		\frametitle{Table of Contents}
		\tableofcontents[currentsection]
	\end{frame}
}

%\lstnewenvironment{code}{\lstset{language=Haskell,basicstyle=\small}}{}

\newminted[code]{haskell}{fontsize=\small}
\newmint[hask]{haskell}{fontsize=\small}
\newminted{prolog}{fontsize=\small}

\definecolor{mygreen}{rgb}{0,0.6,0}
\definecolor{mygrey}{rgb}{0.5,0.5,0.5}
\definecolor{mymauve}{rgb}{0.58,0,0.82}

\newtheorem{observation}[theorem]{Observation}

\newcommand\myheading[1]{%
  \par\bigskip
  {\large\color{blue}#1}\par\smallskip}
\begin{document}

\frame{\titlepage}

\section{Constraint programming}

\begin{frame}
    \frametitle{Constraint programming}
    Constraint programming is a declarative programming paradigm for solving constraint satisfaction problems.
    \pause
    \begin{itemize}
        \item A set of \emph{constraint variables} over a \emph{domain}, e.g. Booleans, integers, reals, finite domain.
    \pause
        \item A set of \emph{constraints} between those variables.
    \pause
        \item A \emph{solver} to find solutions to the constraints, i.e. assignments of variables to values in the domain such that all constraints are satisfied.
    \end{itemize}
    \pause
    Applications: planning, scheduling, resource allocation, computer graphics, digital circuit design, programming language analysis, \ldots
\end{frame}

\section{Constraint logic programming}

\begin{frame}
    \frametitle{Constraint logic programming}
    \begin{itemize}
        \item Constraint programming and logic programming work well together.
    \pause
        \item Many Prolog implementations have built in constraint solvers.
    \pause
        \item Basic idea:
            \begin{itemize}
                \item add constraints to the constraint store
    \pause
                \item constraint solver works behind the scenes to propagate constraints
    \pause
                \item use Prolog's backtracking search mechanism to generate solutions
    \pause
            \end{itemize}
        \item Advantages over pure logic programming:
            \begin{itemize}
                \item ``constrain-and-generate'' rather than ``generate-and-test''
    \pause
                \item constraint solver can greatly reduce the search space required compared to Prolog's built-in depth-first-search
    \pause
                \item much more powerful than relying on just unification and backtracking
            \end{itemize}
    \end{itemize}
\end{frame}

\section{Finite domain constraints}

\begin{frame}[fragile]
    \frametitle{Finite domain constraints}
\begin{itemize}
    \item One of the most widely used varieties of constraint solver.
        \pause
    \item Variables range over a finite domain of integers.
        \pause
    \item Simple equality and inequality constraints: $=$, $\neq$, $<$, $>$, $\leq$, $\geq$
        \pause
    \item Also simple arithmetic expressions: $+$, $-$, $\times$, \texttt{abs}
\end{itemize}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Arc consistency}
Solver uses an \emph{arc consistency algorithm}, e.g.\ AC-3
\begin{itemize}
    \item Constraint store holds the set of constraints to be checked.
        \pause
    \item For each constraint, the domains of the variables involved are checked to
        ensure they are consistent with the contraint.
        \pause
    \item Any values in the domains that break consistency are removed.
        \pause
    \item If the domain of a variable changes then all other constraints involving that variable are
        rechecked.
\end{itemize}
\pause
\begin{example}
\begin{displaymath}
    \begin{array}{c}
    x \in \{1, 2, 3\} ~~\land~~ y \in \{1, 2, 3\} \\
    \pause
    \textrm{add constraint~} x < y \\
    \pause
    \Rightarrow ~~~ x \in \{1, 2\} ~~\land~~ y \in \{2, 3\} \\
    \pause
    \textrm{add constraint~} y = 2 \\
    \pause
        \Rightarrow ~~~ x \in \{1\} ~~\land~~ y \in \{2\}
    \end{array}
\end{displaymath}
\end{example}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Example: $n$ queens in SWI-Prolog}

\begin{columns}[t]
\column[c]{0.35\paperwidth}
\begin{minted}[fontsize=\small]{prolog}
n_queens(N, Qs) :-
        length(Qs, N),
        Qs ins 1..N,
        safe_queens(Qs).

safe_queens([]).
safe_queens([Q|Qs]) :-
    safe_queen(Qs, Q, 1),
    safe_queens(Qs).

safe_queen([], _, _).
safe_queen([Q|Qs], Q0, D0) :-
        Q0 #\= Q,
        abs(Q0 - Q) #\= D0,
        D1 #= D0 + 1,
        safe_queen(Qs, Q0, D1).
\end{minted}
\column[c]{0.5\paperwidth}
\includegraphics[width=0.45\paperwidth]{8queens.png}
\end{columns}
\end{frame}

\section{Constraint programming in Haskell}

\subsection{Basic equality and inequality}

\begin{frame}[fragile]
    \frametitle{Constraint programming in Haskell}
How can we do something similar in Haskell?\pause{}
\textbf{Use a monad!}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Example: $n$ queens in SWI-Prolog and Haskell}

\begin{columns}[t]
\column[t]{0.33\paperwidth}
\begin{minted}[fontsize=\footnotesize]{prolog}
n_queens(N, Qs) :-
        length(Qs, N),
        Qs ins 1..N,
        safe_queens(Qs).

safe_queens([]).
safe_queens([Q|Qs]) :-
    safe_queen(Qs, Q, 1),
    safe_queens(Qs).

safe_queen([], _, _).
safe_queen([Q|Qs], Q0, D0) :-
        Q0 #\= Q,
        abs(Q0 - Q) #\= D0,
        D1 #= D0 + 1,
        safe_queen(Qs, Q0, D1).
\end{minted}
\pause
\column[t]{0.59\paperwidth}
\begin{minted}[fontsize=\footnotesize]{haskell}
nQueens :: Int -> FD [FDExpr]
nQueens n = do
    qs <- news n (1, n)
    safeQueens qs
    return qs

safeQueens :: [FDExpr] -> FDConstraint
safeQueens [] = return ()
safeQueens (q : qs) = do
    safeQueen qs q 1
    safeQueens qs

safeQueen :: [FDExpr] -> FDExpr -> FDExpr -> FDConstraint
safeQueen [] _ _ = return ()
safeQueen (q : qs) q0 d0 = do
   q0 #\= q
   abs (q0 - q) #\= d0
   safeQueen qs q0 (d0 + 1)
\end{minted}
\end{columns}
\end{frame}
\begin{frame}[fragile]
\begin{itemize}
    \item List monad provides backtracking / search / multiple solutions.
        \pause
    \item Wrap it in a state monad transformer to keep track of the constraint store.
        \pause
\end{itemize}
\begin{code}
type FD a = StateT FDState [] a
\end{code}
\pause
\begin{code}
type FDConstraint = FD ()
\end{code}
\pause
\begin{code}
-- Run the monad to obtain a list of solutions.
runFD :: FD a -> [a]
runFD fd = evalStateT fd initState
\end{code}
\end{frame}

\begin{frame}[fragile]
\begin{code}
newtype FDVar = FDVar { _unwrapFDVar :: Int } deriving (Ord, Eq)

type VarSupply = FDVar

data Domain
    = Set IntSet
    | Range Int Int

data VarInfo = VarInfo { _delayedConstraints :: !FDConstraint
                       , _domain :: !Domain }

type VarMap = Map FDVar VarInfo

data FDState = FDState { _varSupply :: !VarSupply, _varMap :: !VarMap }

initState :: FDState
initState = FDState { _varSupply = FDVar 0, _varMap = Map.empty }
\end{code}
\end{frame}

\begin{frame}[fragile]
\begin{code}
newVar :: ToDomain a => a -> FD FDVar
newVar d = do
    v <- use varSupply
    varSupply . unwrapFDVar += 1
    let vi = initVarInfo & domain .~ toDomain d
    varMap . at v ?= vi
    return v

newVars :: ToDomain a => Int -> a -> FD [FDVar]
newVars n d = replicateM n (newVar d)
\end{code}
\end{frame}

\begin{frame}[fragile]
\begin{code}
-- Look up the current domain of a variable.
lookup :: FDVar -> FD Domain
lookup x =
    use $ varMap . ix x . domain

-- Update the domain of a variable and fire all delayed constraints
-- associated with that variable.
update :: FDVar -> Domain -> FDConstraint
update x i = do
    vi <- use $ varMap . ix x
    varMap . ix x . domain .= i
    vi ^. delayedConstraints

-- Add a new constraint for a variable to the constraint store.
addConstraint :: FDVar -> FDConstraint -> FDConstraint
addConstraint x constraint =
    varMap . ix x . delayedConstraints %= (>> constraint)
\end{code}
\end{frame}

\begin{frame}[fragile]
\begin{code}
-- Useful helper function for adding binary constraints between FDVars.
type BinaryConstraint = FDVar -> FDVar -> FDConstraint
addBinaryConstraint :: BinaryConstraint -> BinaryConstraint
addBinaryConstraint f x y = do
    let constraint  = f x y
    constraint
    addConstraint x constraint
    addConstraint y constraint

-- Constrain two variables to have the same value.
same :: FDVar -> FDVar -> FDConstraint
same = addBinaryConstraint $ \x y -> do
    xv <- lookup x
    yv <- lookup y
    let i = xv `intersection` yv
    guard $ not $ Domain.null i
    when (i /= xv) $ update x i
    when (i /= yv) $ update y i
\end{code}
\end{frame}

\begin{frame}[fragile]
\begin{code}
-- Constrain two variables to have different values.
different :: FDVar -> FDVar -> FDConstraint
different = addBinaryConstraint $ \x y -> do
    xv <- lookup x
    yv <- lookup y
    guard $ not (isSingleton xv) || not (isSingleton yv) || xv /= yv
    when (isSingleton xv && xv `isSubsetOf` yv) $
        update y (yv `difference` xv)
    when (isSingleton yv && yv `isSubsetOf` xv) $
        update x (xv `difference` yv)

-- Constrain a list of variables to all have different values.
varsAllDifferent :: [FDVar] -> FDConstraint
varsAllDifferent (x:xs) = do
    mapM_ (different x) xs
    varsAllDifferent xs
varsAllDifferent _ = return ()
\end{code}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Labelling}
\emph{Labelling} is used to obtain valid solutions for a set of variables.
The embedded list monad allows us to search for and return all possible solutions.
\begin{code}
-- Label variables using a depth-first left-to-right search.
varsLabelling :: [FDVar] -> FD [Int]
varsLabelling = mapM label where
    label var = do
        vals <- lookup var
        val <- lift $ elems vals
        var `hasValue` val
        return val
\end{code}
\end{frame}

\begin{frame}[fragile]
We now have enough to solve Sudoku!
\begin{code}
sudoku :: [Int] -> [[Int]]
sudoku puzzle = runFD $ do
    vars <- newVars 81 (1, 9)
    zipWithM_ (\x n -> when (n > 0) (x `hasValue` n)) vars puzzle
    mapM_ varsAllDifferent (rows vars)
    mapM_ varsAllDifferent (columns vars)
    mapM_ varsAllDifferent (boxes vars)
    varsLabelling vars

rows, columns, boxes :: [a] -> [[a]]
rows = chunk 9
columns = transpose . rows
boxes = concat . map (map concat . transpose) . chunk 3 . chunk 3 . chunk 3

chunk :: Int -> [a] -> [[a]]
chunk _ [] = []
chunk n xs = ys : chunk n zs where
    (ys, zs) = splitAt n xs
\end{code}
\end{frame}

\subsection{Arithmetic expressions}

\begin{frame}[fragile]
    \frametitle{Arithemtic expressions}
    \begin{itemize}
        \item So far we have seen how to declare contraint variables and define simple equality constraints
            between them.
            \pause
        \item We also want to be able to write constraints involving simple arithmetic expressions.
    \end{itemize}
\end{frame}
\begin{frame}[fragile]
    \begin{code}
data FDExpr
    = Int !Int
    | Var !FDVar
    | Plus !FDExpr !FDExpr
    | Minus !FDExpr !FDExpr
    | Times !FDExpr !FDExpr
    | Negate !FDExpr
    | Abs !FDExpr
    | Signum !FDExpr
    \end{code}
\pause
    \begin{code}

-- Num instance allows us to use the usual arithmetic operators
-- and integer literals
instance Num FDExpr where
    (+) = Plus
    (-) = Minus
    (*) = Times
    negate = Negate
    abs = Abs
    signum = Signum
    fromInteger = Int . fromInteger
    \end{code}
\end{frame}

\begin{frame}[fragile]
    \begin{code}
-- Define new variables and return as expressions
new :: ToDomain a => a -> FD FDExpr
new d = newVar d <&> Var

news :: ToDomain a => Int -> a -> FD [FDExpr]
news n d = replicateM n $ new d
    \end{code}
    \pause
    \begin{code}

-- Interpret an FDExpr and return an FDVar representing it
interpret :: FDExpr -> FD FDVar
interpret (Var v) = return v
interpret (Int i) = newVar [i]
interpret (Plus e0 e1) = interpretBinary (+) e0 e1
interpret (Minus e0 e1) = interpretBinary (-) e0 e1
interpret (Times e0 e1) = interpretBinary (*) e0 e1
interpret (Negate e) = interpretUnary negate e
interpret (Abs e) = interpretUnary abs e
interpret (Signum e) = interpretUnary signum e
    \end{code}
\end{frame}

\begin{frame}[fragile]
    \begin{minted}[fontsize=\footnotesize]{haskell}
interpretBinary :: (Int -> Int -> Int) -> FDExpr -> FDExpr -> FD FDVar
interpretBinary op e0 e1 = do
    v0 <- interpret e0
    v1 <- interpret e1
    d0 <- lookup v0
    d1 <- lookup v1
    v <- newVar [n0 `op` n1 | n0 <- elems d0, n1 <- elems d1]
    let pc  = constrainBinary (\n n0 n1 -> n == n0 `op` n1) v v0 v1
        nc0 = constrainBinary (\n0 n n1 -> n == n0 `op` n1) v0 v v1
        nc1 = constrainBinary (\n1 n n0 -> n == n0 `op` n1) v1 v v0
    addConstraint v0 $ pc >> nc1
    addConstraint v1 $ pc >> nc0
    addConstraint v  $ nc0 >> nc1
    return v

constrainBinary :: (Int -> Int -> Int -> Bool) -> FDVar -> FDVar -> FDVar -> FDConstraint
constrainBinary pred v v0 v1 = do
    d <- lookup v
    d0 <- lookup v0
    d1 <- lookup v1
    let d' = toDomain [n | n <- elems d, n0 <- elems d0, n1 <- elems d1, pred n n0 n1]
    guard $ not $ Domain.null d'
    when (d' /= d) $ update v d'
    \end{minted}
\end{frame}

\begin{frame}[fragile]
    \begin{code}
infix 4 #\=
(#\=) :: FDExpr -> FDExpr -> FDConstraint
a #\= b = do
    v0 <- interpret a
    v1 <- interpret b
    v0 `different` v1

allDifferent :: [FDExpr] -> FDConstraint
allDifferent = varsAllDifferent <=< mapM interpret

labelling :: [FDExpr] -> FD [Int]
labelling = varsLabelling <=< mapM interpret
    \end{code}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Example: $n$ queens in SWI-Prolog and Haskell}

\begin{columns}[t]
\column[t]{0.33\paperwidth}
\begin{minted}[fontsize=\footnotesize]{prolog}
n_queens(N, Qs) :-
        length(Qs, N),
        Qs ins 1..N,
        safe_queens(Qs).

safe_queens([]).
safe_queens([Q|Qs]) :-
    safe_queen(Qs, Q, 1),
    safe_queens(Qs).

safe_queen([], _, _).
safe_queen([Q|Qs], Q0, D0) :-
        Q0 #\= Q,
        abs(Q0 - Q) #\= D0,
        D1 #= D0 + 1,
        safe_queen(Qs, Q0, D1).
\end{minted}
\pause
\column[t]{0.59\paperwidth}
\begin{minted}[fontsize=\footnotesize]{haskell}
nQueens :: Int -> FD [FDExpr]
nQueens n = do
    qs <- news n (1, n)
    safeQueens qs
    return qs

safeQueens :: [FDExpr] -> FDConstraint
safeQueens [] = return ()
safeQueens (q : qs) = do
    safeQueen qs q 1
    safeQueens qs

safeQueen :: [FDExpr] -> FDExpr -> FDExpr -> FDConstraint
safeQueen [] _ _ = return ()
safeQueen (q : qs) q0 d0 = do
   q0 #\= q
   abs (q0 - q) #\= d0
   safeQueen qs q0 (d0 + 1)
\end{minted}
\end{columns}
\end{frame}
\begin{frame}[fragile]
    \begin{verbatim}
        SEND
     +  MORE
     -------
       MONEY
    \end{verbatim}

    \begin{code}
sendMoreMoney = runFD $ do
    vars@[s, e, n, d, m, o, r, y] <- news 8 (0, 9)
    s #\= 0
    m #\= 0
    allDifferent vars

    1000 * s + 100 * e + 10 * n + d
       + 1000 * m + 100 * o + 10 * r + e
       #== 10000 * m + 1000 * o + 100 * n + 10 * e + y

    labelling vars
    \end{code}
\end{frame}

\section{Conclusion}
\begin{frame}
    \frametitle{Consclusion}
    \begin{itemize}
        \item Haskell can do constraint logic programming -- all you need is monads.
            \pause
        \item Advantages of Haskell
            \begin{itemize}
                    \pause
                \item Awesomeness of Haskell.
                    \pause
                \item Type safety.
                    \pause
                \item Leverage libraries, such as monad combinators, in a very natural way.
                    \pause
            \end{itemize}
        \item Disadvantages
            \begin{itemize}
                    \pause
                \item Not full Prolog, e.g. missing unification between terms, multi-moded predicates.
                    \pause
                \item Some Prolog implementations have very powerful and efficient built-in solvers, which Haskell can't use.
            \end{itemize}
    \end{itemize}
    \pause
    Github repository: \url{https://github.com/dmoverton/finite-domain}
\end{frame}
\end{document}


\documentclass[12pt]{article}
\input{physics1}
\begin{document}

\section*{NYU Physics I---Problem Set 9}

Due Thursday 2018 November 08 at the beginning of lecture.

\paragraph{\problemname~\theproblem}\refstepcounter{problem}%
What is the pressure gradient (units of $\Pa\,\m^{-1}$) in air at sea level at STP?
What is the pressure gradient in water?
What is the pressure gradient in liquid mercury?
What would be the pressure gradient in rock, if we treated the rock like a fluid?

Compute the ratio of the pressure gradient in mecury to the pressure gradient
in water. How does this ratio compare to the ratio of $33\,\ft$ to $760\,\mm$?
Why did I ask that?

\paragraph{\problemname~\theproblem}\refstepcounter{problem}%
In a bungy jump, the bungy cord has a rest (unstretched) length of
$\ell_0 = 5\,\m$, and has a spring constant $k$ such that it stretches
by 1~m for every 400~N of force.

\textsl{(a)} If an adult of mass $M=80\,\kg$ jumps off of a very tall
bridge at time $t=0$ with this bungy cord attached between her or
himself and also the bridge, to what maximum distance $h_\mathrm{max}$
below the bridge will he or she fall?  (You might use energy
conservation). Give your answer in terms of the symbols $M$, $\ell_0$,
$k$, and $g$ as well as numerically. That is, we want the numerical
and symbolic answers both.

\textsl{(b)} What is the maximum tension $T$ in the bungy? Again, give
both answers.

\textsl{(c)} If you stiffen the cord (increase $k$), but keep
the natural length and adult's mass fixed, do you increase or decrease the maximum
tension?

\textsl{(d)} What acceleration $a$ does the adult feel at the maximum
extension of the bungy (that is, at the bottom)? Again, give your
answer both symbolically (in terms of the same symbols) and numerically.

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
Here we consider the construction and tuning of a standard grand piano.
Each string of the piano has a mass $M$, a length $L$, and a tension $T$.

\textsl{(a)} Use dimensional analysis to estimate the natural angular
frequency $\omega$ of a piano string with these properties. That is,
what combination has units of frequency?

\textsl{(b)} Look up the natural frequency of a guitar or piano string
in its lowest harmonic. You might have to look up ``standing wave'' or
something like that, and you might also have to look up ``transverse
wave speed'' in a string. A string fixed at both ends (like a piano
string) is different from an open organ pipe! How does what you find
in your research compare to what you found in part \textsl{(a)}?

\textsl{(c)} Look inside a piano at or near middle C. Roughly what are
the diameters of the strings? And what are the lengths of the strings?
Use these quantities and the density of steel to estimate the masses
of the strings.

\textsl{(d)} Given what you know about the piano---the number of keys,
the number of strings per key (which isn't one for most keys), and the
range of frequencies and string lengths, estimate \emph{very roughly}
the total force compressing a piano frame, in Newtons. That is,
estimate the total of all the tension forces. Do you understand why
piano frames and harps are so heavy?

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Everything submerged in the Earth's atmosphere is subject to a buoyant
force from the air.  In the following, use a sensible (reasonably
accurate) measure of the density of air at STP.

\textsl{(a)} When you measure your weight on a standard bathroom
scale, you are measuring the \emph{normal force} between yourself and
the floor.  This normal force opposes the \emph{combination} of
gravity and buoyancy.  What is the correction to your weight coming
from buoyancy, roughly?  Express it as a \emph{fraction} of the
gravitational force.  Is this correction positive or negative---that
is, does it increase or decrease the weight measured by the scale?

\textsl{(b)} Look up the ``volume'' of the Goodyear blimp model GZ-22.
Imagine that it is floating in an air atmosphere at STP, and that the
gas inside the blimp is \emph{also} at STP.  What is the approximate
buoyant force on the blimp if it is filled with helium?  What about if
it were filled with hydrogen (molecular hydrogen)?  Compare these
numbers with the gross weight and capacity of the blimp.

\textsl{(c)} Will the buoyant force increase, decrease, or stay the
same as you decrease the temperature?
Assume that the blimp contents are always at the same pressure as the
exterior air (and therefore the volume of the blimp must change).

\paragraph{Extra Problem (will not be graded for credit):}%
A child of mass $m$ sits exactly on top of a hemispherical mound of
frictionless ice of radius $R$.  If the child is displaced a tiny (ie,
small relative to $R$) horizontal distance $x$ from the top of the
mound of ice, what is the $x$-component $F_x$ of the net force on the
child?  Write down the differential equation relating the $x(t)$ to
its second derivative (with respect to time).  Use the small-angle
approximation to get rid of trigonometric functions! What functions
$x(t)$ solve your equation?  Try to be as general as possible.
\emph{Hint: Try exponentials!}.

\paragraph{Extra Problem (will not be graded for credit):}%
Look up the Youngs modulus for steel, and estimate the spring
potential energy stored in the middle C string, using what you figured
out in the piano problem. Now estimate the total mechanical energy
stored in the tuned piano!

\end{document}