\documentclass[a4paper,12pt]{article}

 \addtolength{\oddsidemargin}{-.5in}
\addtolength{\evensidemargin}{-.5in}
\addtolength{\textwidth}{1.0in}
\addtolength{\topmargin}{-.5in}
\addtolength{\textheight}{1.0in}

\usepackage{cite}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{xcolor}
\usepackage{color}
\usepackage{bbding}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}

\begin{document}

\title{Group 4: The Kaczmarz Algorithm}
\author{Wei Deng, Nicole Eikmeier, Nate Veldt, and Xiaokai Yuan}
\maketitle{}

\section{Introduction}
Project Objective:

Given a large sparse linear system $Ax = f$, of order $n = 10^6$ that can be effectively reduced to a banded matrix using the reordering scheme ``reverse Cuthill-McKee", use the method of Row Projection, accelerated via the Conjugate Algorithm to yield a parallel solver. Compare the robustness and parallel scalability/speed with preconditioned Krylov subspace methods preconditioned via approximate LU-factorization.

\vspace{.2in}

The Kaczmarz Algorithm is a row projection method which is equivalent to solving $AA^Ty = f$ using the Gauss-Siedel iteration, with $x = A^Ty$. In Kaczmarz we consider each equation as a hyperplane:
$$S_i=\{x:A_ix-b_i=0\}$$ for $i=1,2,...$, where $A_i$ and $b_i$ are $i$th row of matrix $A$ and vector $b$. So the problem is transformed into finding the coordinates of the point of intersection of these hyperplanes.  See figure \ref{fig:ClassKacz} for a visualization of the problem.

\begin{figure}[htbp]
\begin{centering}
%\begin{minipage}[b]{0.8\linewidth}
\includegraphics[width=5in]{Images/Classical_Kaczmarz}
\caption{2-Partition Case of the Kaczmarz Algorithm}
\label{fig:ClassKacz}
%\end{minipage}
\end{centering}
\end{figure}

From this basic idea we use symmetrization and acceleration via the CG method to improve the procedure. The details will be given carefully in the following sections.

\section{Implementation}

%------------------------------------------------
%------------------------------------------------
% REVERSE CUTHILL-MCKEE
%------------------------------------------------
%------------------------------------------------

\subsection{Reverse Cuthill-McKee and the Woodbury Formula}
The first step in our implementation is to transform the sparse matrix $A$ to a banded form using Reverse Cuthill-McKee. We can perform a symmetric permutation with matrix $P$ so that $P^TAP$ is banded. In figure \ref{fig:v3}, we see the result from Matlab of Reverse Cuthill Mckee on the stomach data set. The band size is nice and small.

%\begin{figure}[ht]
%\centering
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/SparseA.jpg}
%\caption{$A$ sparse, non-symmetric}
%\end{minipage}
%\quad
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/Arcm.jpg}
%\caption{$P^TAP$ (banded)}
%\label{fig:banded}
%\end{minipage}
%\end{figure}

\begin{figure}[ht]
\centering
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/SparseA_v5.jpg}
\caption{$A$ sparse, non-symmetric}
\label{fig:v5}
\end{minipage}
\quad
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/Arcm_v3.jpg}
\caption{$P^TAP$ (banded)}
\label{fig:v3}
\end{minipage}
\end{figure}

For some problems the bandwidth could be too large after using Reverse Cuthill Mckee. See for example figure \ref{fig:approxbanded}, where we see the result of RCM on the lns$\_$131 data set, again in Matlab.  If this is the case, instead we could choose $P$ so that $P^TAP$ is narrow banded plus a low rank matrix. See figure \ref{fig:woodbury}.

\begin{figure}[ht]
\centering
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/SparseA_v4.jpg}
\caption{$A$ is sparse and non-symmetric}
\label{sparseA}
\end{minipage}
\quad
\begin{minipage}[b]{0.45\linewidth}
\includegraphics[width=2in]{Images/ApproxBanded_v2.jpg}
\caption{$P^TAP$ narrow band+low rank}
\label{fig:approxbanded}
\end{minipage}
\end{figure}

%\begin{figure}[ht]
%\centering
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/SparseA.jpg}
%\caption{$A$ is sparse and non-symmetric}
%\end{minipage}
%\quad
%\begin{minipage}[b]{0.45\linewidth}
%\includegraphics[width=2in]{Images/ApproxBanded.jpg}
%\caption{$P^TAP$ narrow band + low rank}
%\end{minipage}
%\label{fig:bandedplus}
%\end{figure}

\begin{figure}[htbp] %  figure placement: here, top, bottom, or page
   \centering
   \includegraphics[trim = 5mm 30mm 0mm 40mm, clip, width=4in]{Images/Slide1.jpg}
    \caption{Breaking up $A$ into a banded matrix plus a low rank matrix}
    \label{fig:woodbury}
\end{figure}

In this case we use the Woodbury Formula to solve $Ax = b$. Recall first that $ Ax = b$  implies $x = A^{-1} b$. Then,
%\begin{block}{Woodbury Formula}
\begin{align*}
A^{-1} &= (B - USV^T)^{-1}\\
	  &= B^{-1} - B^{-1}UTV^TB^{-1}
\end{align*}
where $T = (V^TB^{-1}U - S^{-1})^{-1}$. And
%\end{block}
\begin{align*}
x &= A^{-1}b\\
  &= {\bf B^{-1} b} - B^{-1}UTV^T {\bf B^{-1} b} \\
   &= {\bf a} - B^{-1}UT(V^T {\bf a}) \\
   &= {\bf a} - B^{-1}UT {\bf c}  \hspace{.5cm} \mbox{( solve ($V^TB^{-1}U-S^{-1}) \textbf{d} = \textbf{c}$)}\\
   &= {\bf a} - B^{-1}U\textbf{d}\\
   &= {\bf a} - B^{-1}\textbf{h}
\end{align*}

The Woodbury formula is useful because all systems involving $B$ are relatively easy to solve. We did not implement the Woodbury formula in our experiments, due to time contraints. This means that we transformed $A$ to a completely banded matrix, no matter how large the band size. We expect there would be improvements in our final results if the Woodbury formula was implemented.

In table \ref{tab:Bandsize}, the band size after permutation is shown. Matlab performed the best across the board, but all three permutations did well for our largest test case. You can see also that for the bayer01 matrix, our band size is very large, and so we did not get reasonable results in our parallel Kaczmarz implementation.

\begin{table}
\begin{center}
\begin{tabular}{| l | l | c  c  c |}
\hline

    matrix           & size         &    Matlab & PETSc     &  rcm.cpp        \\
 \hline
 lns                   & 131             &     32        &       \color{red} $\times$ 111         & \color{red} $\times$ 113  \\
 Jac2-db            & 21,982     &         545       &     \color{red} $\times$            &  \color{red} $\times$  \\
 bayer01           & 57,735      &     \color{red} $\times$ 18,322        &     \color{red} $\times$     &  \color{red} $\times$   \\
 venkat25         & 62,424      &     1,515        &    1,515    & 1,495                   \\
 stomach          & 213,360    &      1,133      & 2,216 & 2,239                    \\
 atmosmodd     & 1,270,432 &    7,772       & 7,772&  7,772                   \\
 \hline

\end{tabular}
\caption{Bandsize after RCM using various codes}
\label{tab:Bandsize}
\end{center}

\end{table}

%------------------------------------------------
%------------------------------------------------
%KACZMARZ
% NEEDS SOME RE-WRITING
%------------------------------------------------
%------------------------------------------------
%
\subsection{The Kaczmarz method}

Once we have a banded matrix, $A$, we move on to the Kaczmarz algorithm, which was introduced earlier. Suppose we partition $A$ into two pieces $$A = \begin{pmatrix} A_1^T \\ A_2^T \end{pmatrix}  $$ and we also partition the right hand size vector $f^T = (f_1^T, f_2^T)$.

In the classical Kaczmarz method at each iteration we compute:
$$x_k=x_k+\overrightarrow{n_k}=x_k+\frac{r_k^i}{||A_i||^2}A_i^T$$
where  $\overrightarrow{n_k}=\bigtriangleup{x_k}cos\theta=\frac{\langle\,A_i{,}\,\bigtriangleup{x_k}\rangle}{||A_i||^2}A_i^T=\frac{b_i-\langle\,A_i{,}\,x_k\rangle}{||A_i||^2}A_i^T=\frac{r_k^i}{||A_i||^2}A_i^T$.
$$
\left\{
\begin{array}{ll}
                  AA^Ty = f\\
                  x = A^Ty\\
 \end{array}
 \right.
 \implies AA^T = \begin{pmatrix} A_1^T \\ A_2^T \end{pmatrix} \begin{pmatrix} A_1, A_2 \end{pmatrix}
  $$

  From Gauss-Seidel, use $x^{k+1} = \begin{pmatrix} A_1, A_2 \end{pmatrix} \begin{pmatrix} y_1^{k+1} \\ y_2^{k+1} \end{pmatrix} $.

  $$ \begin{pmatrix} A_1^TA_1 & 0 \\ A_2^T A_1 & A_2^T A_2 \end{pmatrix} \begin{pmatrix} y_1^{k+1} \\ y_2^{k+1} \end{pmatrix} =  \begin{pmatrix} 0 & -A_1A_2 \\ 0 & 0\end{pmatrix} + \begin{pmatrix} y_1^{k+1} \\ y_2^{k+1} \end{pmatrix}\begin{pmatrix} f_1 \\ f_2 \end{pmatrix}$$

  Replacing some parameters with projection operator $P_i$, we can get:
  $$x^{k+1} = A_1y_1^{k+1} + A_2y_2^{k+1} = Qx_k +b$$ where $Q = (I-P_2)(I-P_1)$. Similarly, for an m-partition, $$x^{k+1} = Q_u x^k + f_u = (I-P_m)(I-P_{m-1}) \ldots (I-P_1)x^k + f_u.$$

  However, the spectral radius of $Q_u$ may interfere with the convergence speed, and the distribution of eigenvalues could also influence the performance. To handle this, we can symmetric $Q_u$ to get an accelerated iteration:
  $$ x^{k+1} = Q(\omega)x^k + T(\omega) f$$ where $Q(\omega) = (I-\omega P_1)(I-\omega P_2) \ldots (I-\omega P_m) \ldots (I-\omega P_2) (I-\omega P_1)$, and $T(\omega) = A^T(D+\omega L)^{-T}D(D+\omega L)^{-1},$ where $A^TA = L+ D+L^T$ is the splitting into block lower, block upper, and block diagonal pieces of $A^TA$. Since $(I-Q)$ is symmetric positive definite, the conjugate gradient method is suitable to accelerate the basic scheme.

 Once we have a banded matrix, we considered using various partitions of the rows. A very simple example is shown in figure \ref{fig:permutation}. A permutation like this gives several benefits. The first is that we can have an outer level of parallelism for each projection. Also, when we split the matrix this way we create several small independent least squares problems, which reduces the time.

 \begin{figure}[htbp]
\begin{minipage}[b]{1\linewidth}
\centering
\includegraphics[width=4in]{Images/permutation}
\end{minipage}
\caption{Re-ordering the banded matrix}
\label{fig:permutation}
\end{figure}

 In Figure \ref{quality} we consider different types of partitions, both in the inner and outer levels. We found that have more independent blocks resulted in faster implementation of Kaczmarz. We also found that more partitions, i.e. taking 4 partitions: $$A = \begin{pmatrix} A_1^T  \\A_2 ^T \\ A_3^T \\ A_4^T \end{pmatrix} $$ instead of 2 partitions is slower.

  Furthermore, it has been proven that the optimal value of $\omega$ is 1.0 if we use two partitions, and $Q(1)$ has the minimal spectral radius. For these reasons we decided to use 2 outer partitions for our implementation. In this case the problem can be simplified as follows: $$(I-Q)x = Tf$$

\begin{figure}[ht]
\begin{centering}
%\begin{minipage}[b]{1\linewidth}
\includegraphics[width=6in]{Images/quality.jpg}
%\end{minipage}
\end{centering}
\caption{Speed for different partition types.}
\label{quality}
\end{figure}

%\begin{figure}[ht]
%\centering
%\begin{minipage}[b]{1.0\linewidth}
%\includegraphics[width=4.8in]{Images/permute.jpg}
%\caption{$A$ is partitioned into 2 parts with several independent submatrices}
%\end{minipage}
%\end{figure}

%------------------------------------------------
%------------------------------------------------
% SYMMETRIZE KACZMARZ
% NEEDS SOME HELP
%------------------------------------------------
%------------------------------------------------
\subsection{Calculation of $c$}
To transform the system $Ax = f$ into solving the symmetric positive definite system $(I-Q)x = c$, we need to get the new right hand side vector $c$. Recall that we use $m=2$ partitions in our work. We let

 $$A = \begin{pmatrix} A_1^T \\ A_2^T \end{pmatrix}  $$ and we also partition the right hand size vector $$f^T = (f_1^T, f_2^T).$$

 For two partitions, if we expand $c = Tf$ to be:

 $$\textbf{Tf} = \begin{bmatrix} (I + (I-P_2)(I-P_1))(A_1^T)^+ && (I-P_1)(A_2^T)^+ \end{bmatrix} \begin{bmatrix} f_1  \\ f_2 \end{bmatrix}= c $$

 We see that in order to perform this we need to solve two pseudo inverse problems:

 $$\hat{f}_i = (A_i^T)^+f_i$$

 and whenever we see a product of the form $(I-P_i)u = v$ we are really performing a least squares operation involving the block $A_i$. We will discuss this least squares problem in more detail later.

Computing $c$ can be accomplished by solving a saddle point problem. In our implementation we spent some of our efforts trying to compute $c$, but due to time constraints and difficulties with other parts of our algorithm we were not able to ever correctly incorporate a proper computation of $c$ in our work. For this reason, for our code we generated a right hand side vector $c$ by setting $u$ to be the all ones vector, and then performing $c := (I-Q)\cdot u$ via a sequence of least squares operations. Then we were able to use our conjugate gradient framework to solve the system $(I-Q)x = c.$

In general for Kaczmarz we expect the time taken in running the CG scheme and calling the least squares function to be the bottleneck in our computations. So even though we were unable to properly generate the right hand side vector from a starting vector $f$, we chose to instead focus on the CG scheme and least squares framework in our implementation.

\subsection{Least Squares Computations}
Recall that the new system we are dealing with is $(I-Q)x = c$, where $c = Tf$, and $$Q=(I-P_{1})(I-P_{2})(I-P_{1})$$
 with $P_{i}=A_{i}(A_{i}^{T}A_{i})^{-1}A_{i}^{T}$.

It is not stable to form $P_{i}$ directly since it will square the condition number of $A_{i}$, and at the same time will cost time in solving a system with the matrix $(A^{T}_{i}A_{i})$. Instead, given a vector $u$ we note that $v=(I-P_{j})u \Leftrightarrow \min\limits_{v}||u-A_{j}w||_{2}$. We therefore solve $\min\limits_{v}||u-A_{j}w||_{2}$ to obtain the vector $v$.

%In PETSc, we use the CG method on the normal equations: $A_{j}^{T}A_{j}w=A_{j}^{T}u$. The parallel step is:
%        \begin{equation*}
%            \begin{split}
%                v&=\min\limits_{v}||u-A_{j}w||_{2}\Leftrightarrow \min\limits_{v_{i}}||u_{i}-A_{j,i}w_{i}||_{2},\\
%                v&^{T}=[v_{1}^{T},v_{2}^{T},...,v_{k}^{T}],\\
%                w&^{T}=[w_{1}^{T},w_{2}^{T},...,w_{k}^{T}].
%            \end{split}
%        \end{equation*}

Because of the way we have split our matrix, when we solve a least squares problem $(I-P_i)u = v$ we in theory can do so independently because $A_i$ is a matrix with many independent blocks. For proper parallel implementation, we would perform the following steps:

\begin{enumerate}
\item At the beginning of the algorithm, send different blocks of $A_i$ (for each sub matrix $i$) to different processors.
\item In each least squares computation, independently perform least squares operations on the blocks, and then gather the final result after the simultaneous, independent least squares operations.
\end{enumerate}

\subsubsection*{QR Factorizations}
One very effective way to compute the least squares problem in parallel is to use the QR factorization via Given's rotations on each independent block of each sub matrix $A_i$. The computation and storage of the QR factorization of the entire matrix $Q$ would be extremely expensive. However, because $A_i$ is made up of independent diagonal blocks that are all very small in size, we can perform QR factorizations of each of these blocks with great success.

There are two clear benefits of using the QR factorization on the independent sub blocks of $A_i$. First of all, we can find this factorization simultaneously for each block of $A_i$ at the very beginning of the algorithm, then in all future calls to our least squares function we can use the factorization without needing to recompute it. Secondly, the storage required for the factorization of the blocks is very small in comparison with the overall system.

\subsubsection*{Difficulties}

Unfortunately in our project we were not able to make full use of the parallelism that is theoretically possible by the Kaczmarz framework. Our difficulty was that splitting up a parallel matrix $A_i$ into independent pieces is extremely difficult in PETSc. This causes a number of issues in our final code, including the following:

\begin{itemize}
\item We cannot be sure that the blocks are perfectly split up among different processors, since PETSc assigns rows automatically by itself without our control.
\item Even though we have multiple processors at work on a least squares problem, the least squares computation is still performed globally on the entire sub matrix $A_i,$ rather than simultaneously on independent blocks of $A_i$.
\item We are unable to factorize pieces of $A_i$ ahead of time to save time in later calls to our sequential least squares algorithm
\end{itemize}

With a better knowledge of PETSc these problems could be remedied, but time constraints rendered it impossible to master the software enough so that we could reach the full potential of the Kaczmarz framework.

\subsubsection{Attempt to Improve Speed by LSQR}

In the end, in order to solve the least squares problem, we implemented LSQR. LSQR is an iterative method for solving $Ax = b$ where $A$ is large, sparse, and overdetermined. It is equivalent at each iteration to CG.

\begin{algorithm}
\caption{LSQR}
\begin{algorithmic}[1]
%\Require{$P_1$ an initiator matrix; $n$, where we want to use $P_n$}
\State{$\beta_{1}u_{1}=b, \alpha_{1}v_{1}=A^{T}u_{1}, w_{1}=v_{1}, x_{0}=0, \bar{\phi_{1}}=\beta_{1},\bar{\rho_{1}}=\alpha_{1}$}
\For{$i = 1,2,3,\cdots$}
\State{Bidiagonalization:}
\State{$\beta_{i+1}u_{i+1}=Av_{i}-\alpha_{i}u_{i}$}
\State{$\alpha_{i+1}v_{i+1}=A^{T}u_{i+1}-\beta_{i+1}v_{i}$}
\State{Orthogonal Transformation:}
\State{$\rho_{i}=\sqrt{(\bar{\rho_{i}^2}+\beta_{i+1}^2)}$}
\State{$c_{i}=\bar{\rho_{i}}/\rho_{i},$}
\State{$s_{i}=\beta_{i+1}/\rho_{i},$}
\State{$\theta_{i+1}=s_{i}\alpha_{i+1},$}
\State{ $\bar{\rho_{i+1}}=-c_{i}\alpha_{i+1},$}
\State{$\phi_{i}=c_{i}\bar{\phi_{i}},$}
\State{$\bar{\phi_{i+1}}=s_{i}\bar{\phi_{i}}.$}
\State{Update x:}
\State{$x_{i}=x_{i-1}+(\phi_{i}/\rho_{i})w_{i},$}
\State{$w_{i+1}=v_{i+1}-(\theta_{i+1}/\rho_{i})w_{i}.$}
\EndFor

\end{algorithmic}
\end{algorithm}

    \begin{figure}[htbp]
    \begin{center}
        %\begin{minipage}[b]{0.8\linewidth}
            \includegraphics[width=3in]{Images/LSQR}
        %\end{minipage}
        \caption{Convergence of LSQR}
        \label{fig:LSQR}
        \end{center}
    \end{figure}

We had hoped that using this method and truncating the number of iterations used in the least squares calculations would allow us to obtain good overall results for our Kaczmarz algorithm. Note the plot of the convergence of the LSQR method in Figure \ref{fig:LSQR}. We can see that the algorithm mostly converges in the first several iterations, so we hoped that by truncating the number of steps taken we would make the least squares computation faster without harming the accuracy too much. We know that Krylov subspace methods are robust against poor matrix-vector products, so the hope was that our outer CG scheme would still progress towards an accurate overall solution of $(I-Q)x = c.$ . However, our final results are still far from satisfactory in terms of accuracy if we truncate the computations too much.

One thing we tried was to increase the maximum number of inner LSQR iterations used as the number of outer CG scheme iterations increased. We did this to try to find a balance between speed and accuracy. However, to achieve any type reasonable accuracy for our larger test cases, we did eventually need to make the iterative least squares method run for many iterations (over 100 for each least squares computation). This leads to very slow runtimes. We provide our results in a later section.

 %------------------------------------------------
%------------------------------------------------
% ACCELERATION VIA CG
%MAYBE NEEDS SOME HELP
%------------------------------------------------
%------------------------------------------------

 \subsection{Acceleration via the CG Method}
As stated previously, since $(I-Q)$ is symmetric positive definite, we accelerate solving $(I-Q)x = Tf$  using the conjugate gradient method. This is the framework of our entire algorithm. The outline follows:

\vspace{.2in}
 \noindent \textbf{Step 1} : $x_0 = c$ \\
\hspace{.5in} Compute $r_0 = Tf - \boxed{(I-Q)c} = Qc$\\
\hspace{.5in} Set $p_0 = r_0, i = 0$ \\
\textbf{Step 2}:  Compute: \\
\hspace{.5in} $\alpha_i = (r_i, r_i)/(p_i,\boxed{(I-Q)p_i})$ \\
\hspace{.5in} $ x_{i+1} = x_i + \alpha_i p_i $ \\
\hspace{.5in} $\beta_i = (r_{i+1}, r_{i+1})/(r_i, r_i) $ \\
\hspace{.5in} $p_{i+1} = r_{i+1} + \beta_i p _i $ \\
\textbf{Step 3}: If convergence criterion is satisfied, terminate the iterations; else set $i = i+1$ and return to Step 2.

 We can take a convergence criterion as : $$ \frac{ ||r_i||}{||r_0||} \leq \epsilon$$

The boxed out portions in our framework are very important. Remember that whenever we see: $$(I - Q) u$$ for any vector $u$, we really mean to solve multiple least squares problems:
$$(I-Q) u = (I-P_1)(I-P_2)(I-P_1) u $$
$$(I-P_i) u \Rightarrow \text{min}_{v} ||u-A_i w|| $$

 %------------------------------------------------
%------------------------------------------------
% LAST SECTION: RESULTS
%------------------------------------------------
%------------------------------------------------

\section{Results and Conclusions}

\subsection{Results}

We will now discuss the results of our parallel solver. Because of our slow least squares computations, we did not reach a very low tolerance for any of our test cases. For this reason we chose to simply truncate our CG loop at 100 iterations and report the runtime and the tolerance attained by this point. The results are given in table $\ref{tab:badresults}$.

\begin{table}
\begin{center}
\begin{tabular}{| l | l | c  c | c  c |}
\hline
                         &                & Our Parallel & Solver & Krylov & ILU\\
                          \hline
    matrix           & size         &  tol achieved & time (s)     &  num-its & time (s)        \\
 \hline
 lns                   & 131             &     $7.5E{-03}$   &         5.6  & \color{red} $\times$  &  \color{red} $\times$\\
 Jac2-db            & 21,982     &   $2.0E{-04}$     &    283    &  \color{red} $\times$  & \color{red} $\times$ \\
 venkat25         & 62,424      & $2.5E{-03}$      &     874  & 374                   & 14.17 \\
 stomach          & 213,360    &   $2.1E{-06}$    & 194 & 16                   & 2.21 \\
 atmosmodd     & 1,270,432 &  $3.0E{-03}$     &7100 &  266                   & 130.72 \\
 \hline

\end{tabular}
\end{center}
\caption{Runtimes of our algorithm (100 iterations of CG) in comparison with GMRES}
\label{tab:badresults}
\end{table}

To find a benchmark of how our method performed, we compared with a Krylov Subspace method. The implementation of this consisted of first using the MC64 software to re-order the matrix. This software finds a permutation which brings non-zeros to the diagonal. Once this permutation is used, then we called a Krylov Subspace method with ILU pre-conditioner in PETSc. The Krylov subspace method used is GMRES.

Note that the ILU preconditioner fails for some of our data. The reason for this could be that the eigenvalues of these matrices are distributed too much across the positive and negative spectrum. The ILU preconditioner is not well equipped to handle these so called ``highly indefinite'' matrices. Table \ref{tab:MC64} shows which of our data failed with this method.
\begin{table}
\begin{center}
\begin{tabular}{| l | l | c |}
\hline
matrix             & size        & Converged? \\
\hline
lns                  & 131         &    \color{red} $\times$  \\
std1-Jac2-db  & 21,982    & \color{red}  $\times$  \\
venkat25        & 62,424     & \color{green} \Checkmark   \\
stomach         & 213,360   &  \color{green}  \Checkmark  \\
atmosmodd    & 1,270,432 &  \color{green} \Checkmark  \\
\hline
\end{tabular}
\end{center}
\caption{ILU preconditioner}
\label{tab:MC64}
\end{table}

We note that the one success of our algorithm was its ability to make progress in solving the first few test cases where we failed to converge for GMRES. We see that the framework of our algorithm is good for solving these difficult systems, though runtimes are terrible for reasons already addressed.

\subsection{Conclusions}
In conclusion, it was a mistake to use PETSc. PETSc is better for high level implementation, not for optimization and efficiency. PETSc was also not well suited specifically for grabbing the exact sub-matrices that we needed in order to call the parallel Least Squares.

In the end we tried to salvage our project by implementing LSQR, but even with this attempt we did not obtain the desired outcome. If we had left PETSc out of our project we know our results would have been even better.

\nocite{KamathSameh1988,GallopoulosPhilippeSameh2016,Paige,HSL}

\newpage

\bibliography{Ref}
\bibliographystyle{plain}

\end{document}

Parallel Solver for a Large Sparse Linear System/slides/Group4_Final_Report_Complete.tex


\subsection{Simulation}

To verify the validity of the \textbf{PD} controller for this task, simulations were run in MATLAB. The simulation set the motors to a common speed, so that the controller always makes the robot move forward when $\theta=0$ and lower the speed of a motor on one side to turn. This is the same way as the controller is implemented on the robot. Both motors speed can therefore be set into one variable that is defined as the difference between the motor speeds ($\Delta v = v_L-v_R$). \\
\indent The simulation used a sinusoid as input to the controller, simulating that the robot travelled along a line that consistently turn from left to right and back again. The results can be seen in Fig. \ref{fig:sim_t_error} and \ref{fig:sim_LR_motor}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{img/theta_error.eps}
    \caption{The error between the robots pose and the line.}
    \label{fig:sim_t_error}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{img/LR_motors.eps}
    \caption{The motor output for the results in Fig.\ref{fig:sim_t_error}.}
    \label{fig:sim_LR_motor}
\end{figure}

The results presented above is evidence that a \textbf{PD} controller can be used for a line follower.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{tcolorbox}
\usepackage{float}
\usepackage{amsfonts}
\usepackage{svg}
\date{}

\usepackage{qcircuit}

\title{\textbf{Quantum Computing: An Applied Approach}\\\vspace*{1cm}
Chapter 8 Problems: Building a Quantum Computer
}

\begin{document}

\maketitle

\section{}

The circuit model can be defined as the union of three distinct components: (1) a set of $n$ initialized input qubits, (2) a set of qubit lines consisting of one- or multi-qubit unitary gate operations, and (3) a set of up to $n$ observables to measure that project the qubit states onto the subspace spanned by $n$ classical bits.

The query model, on the other hand, describes some function that maps $n$ input qubits onto $n$ output qubits via the action of some oracle. In this sense, it consists of an input state $\{|x_i\rangle\}_{i=1}^n$ and a corresponding collection of classical bits $\{c_i\}_{i=1}^n$.

\section{}

This proof appears relatively elementary. The query complexity involves up to a single oracle call on each of the $n$ input qubits. The circuit, or gate, complexity involves an unbounded number of operations on each of $n$ qubit lines. The query model complexity then represents a lower bound on the circuit complexity, as the circuit complexity involves at least $O(1)$ unitary operations on each qubit.

\section{}

There is a single single-qubit gate per line in the quantum Fourier transform (QFT), so there are $n$ single-qubit gates. The second part of the question clearly means to ask how many double-qubit gates are needed; there are $i-1$ double-qubit gates on the $i^\text{th}$ line yielding $\sum_{i=1}^{n-1}i=\boxed{\frac{n^2-n}{2}}$ gates overall.

\section{}
$n=1$ qubits:
\newline
\Qcircuit @C=1em @R=.7em {
& \gate{H} & \qw
}
\newline\newline
$n=2$ qubits:
\newline
\Qcircuit @C=1em @R=.7em {
& \gate{H} & \gate{R_{\pi/2}} & \qw & \qw \\
& \qw & \ctrl{-1} & \gate{H} & \qw
}
\newline\newline
$n=3$ qubits:
\newline
\Qcircuit @C=1em @R=.7em {
& \gate{H} & \gate{R_{\pi/2}} & \gate{R_{\pi/4}} & \qw & \qw & \qw & \qw \\
& \qw & \ctrl{-1} & \qw & \gate{H} & \gate{R_{\pi/2}} & \qw & \qw \\
& \qw & \qw & \ctrl{-2} & \qw & \ctrl{-1} & \gate{H} & \qw
}

\section{}

The intuition behind this proof is that, when initialized in the zero-state $|0\rangle^{\otimes n}$, all of the controlled-rotation gates reduce to the identity. Consequently, the QFT reduces to a set of one Hadamard operation per line yielding:
$$
QFT_n|0\rangle^{\otimes n}=H^{\otimes n}|0\rangle ^{\otimes n}
$$
as described.

\section{}

\subsection{}

The approach of period finding proposed in this paper by Ekera and Hastad addresses the modular exponentation step and offers several different proposals for reducing the complexity of this step. Overall, especially for low-bit ($n$) inputs, their approach greatly reduces the number of Toffoli gates required and in doing so, lowers the overall number of qubits required to solve problems such as 2048-bit RSA encryption.

\subsection{}

In terms of the complexity of the modular exponentiation step, Shor's Algorithm involves a Toffoli gate count of $20n_en^2$, where $n_e$ is the number of modular multiplication operations to perform and $n$ is the number of input bits in the integer to be factored. This step dominates the complexity of Shor's Algorithm as a whole.

\subsection{}

The square-and-multiply approach begins with an initial register qubit $x$ in the state $|1\rangle$. Afterwards, a controlled modular multiplication is run for each exponent qubit $e_j$, with $j$ iterating from 0 to $n_e-1$. After this process, including multiple iterations of the modular multiplication subprocess, the initial qubit $x$ stores the exponentiated value $x=g^e$.

\subsection{}

Windowed arithmetic scales down the number of qubits needed for operations in intermediate steps, by replacing subprocesses such as modular multiplication with lookup functions that operate in clusters. In the example shown, a window size of 4 is used to lookup classically known values for the expression $g^{e[4n:4(n+1)]2^{4n}}$, reducing computation time by a factor of 4.

In this paper, this technique is used in both the modular addition and modular multiplication subroutines, modifying the Toffoli gate count from $4n^2n_e$ to $\frac{2n_en}{c_\text{mul}c_\text{exp}}(2n+2^{c_\text{mul}+c_\text{exp}})$. A tradeoff takes place, as the scalar decrease in the gate counts with the window sizes $c$ is balanced by the number of Toffoli gates involved in the lookup operation (factor of $2^{c_\text{mul}+c_\text{exp}}$).

\subsection{}

These implications are explored in Table 1. Although the complexity of the approach studied in this paper is asymptotically greater than the others, for small values of $n$ it reaches a practical baseline for gate complexity in solving 2048-bit RSA decryption around two orders of magnitude lower than the next best.

Specifically, the minimum number of abstract qubits necessary to factor a 2048-bit key has been reduced from 340 in Fowler et al. (2012) to 2.7.

\section{}

Under the approximate QFT, there are at most $m$ two-qubit gates per line, and there are $n$ lines in total. There is also one single-qubit gate per line, yielding a gate complexity of $O(nm+n)=O(nm)$.

In determining the total number of gates exactly, first notice that the number of single-qubit gates remains the same at $n$. The number of double-qubit gates continues to increase by 1 per line beginning at 0, but stops at $m$ since there are at most $m$ rotations $\theta_{jk}$ where $j-k>m$.

There are then $\frac{m(m-1)}{2}$ double-qubit gates in the first $m$ lines, and $(n-m)m$ in the remaining $n-m$. Overall, including the $n$ single-qubit gates this yields
$$
\frac{m^2-m}{2}+\frac{2m(n-m)}{2}=\boxed{\frac{m(2n-m+1)}{2}}
$$
gates.

Since $n>m$ is assumed, this expression has complexity $O(mn)$ as described. It disagrees with the expression listed in the problem set assignment, but matches my own calculations for various values of $m$ and $n$.

For example, setting $m=2$ and $n=5$ for the AQFT yields a quantum circuit with 5 single-qubit gates and 0+1+2+2+2=9 two-qubit gates. Assigning these values into the expression above, this yields $5+\frac{2(2\cdot 5-2+1)}{2}=5+9=14$ gates, as expected.

\end{document}


The \eslmod{scorematrix} module implements standard residue pairwise
alignment scoring systems (such as BLOSUM matrices), and their
probabilistic interpretation \citep{Altschul91,YuAltschul03}.

Table~\ref{tbl:scorematrix_api} lists the functions in the
\eslmod{scorematrix} API. The module implements one object,
\ccode{ESL\_SCOREMATRIX}, which holds a score matrix with integer
scores.

\begin{table}[hbp]
\begin{center}
{\small
\begin{tabular}{|ll|}\hline
\apisubhead{The \ccode{ESL\_SCOREMATRIX} object.}\\
\hyperlink{func:esl_scorematrix_Create()}{\ccode{esl\_scorematrix\_Create()}} & Create an \ccode{ESL\_SCOREMATRIX}.\\
\hyperlink{func:esl_scorematrix_SetWAG()}{\ccode{esl\_scorematrix\_SetWAG()}} & Parameterize matrix from the WAG evolutionary model.           \\
\hyperlink{func:esl_scorematrix_SetFromProbs()}{\ccode{esl\_scorematrix\_SetFromProbs()}} & Set matrix scores from target and background probabilities.\\
\hyperlink{func:esl_scorematrix_Compare()}{\ccode{esl\_scorematrix\_Compare()}} & Compare two matrices for equality.\\
\hyperlink{func:esl_scorematrix_Max()}{\ccode{esl\_scorematrix\_Max()}} & Returns maximum value in score matrix.\\
\hyperlink{func:esl_scorematrix_Min()}{\ccode{esl\_scorematrix\_Min()}} & Returns minimum value in score matrix.\\
%\hyperlink{func:esl_scorematrix_RelativeEntropy()}{\ccode{esl\_scorematrix\_RelativeEntropy()}} & Returns relative entropy of a matrix.\\
\hyperlink{func:esl_scorematrix_IsSymmetric()}{\ccode{esl\_scorematrix\_IsSymmetric()}} & Returns \ccode{TRUE} for symmetric matrix.\\
\hyperlink{func:esl_scorematrix_Destroy()}{\ccode{esl\_scorematrix\_Destroy()}} & Frees a matrix.\\
\apisubhead{Reading/writing score matrices.}\\
\hyperlink{func:esl_scorematrix_Read()}{\ccode{esl\_scorematrix\_Read()}} & Read a standard matrix input file.\\
\hyperlink{func:esl_scorematrix_Write()}{\ccode{esl\_scorematrix\_Write()}} & Write a BLAST-compatible score matrix file.\\
%\apisubhead{Interpreting matrices probabilistically.}\\
%\hyperlink{func:esl_scorematrix_ObtainPij()}{\ccode{esl\_scorematrix\_ObtainPij()}} & Obtain $P_{ij}$ from score matrix with known $\lambda$ and background $f$'s.\\
%\hyperlink{func:esl_scorematrix_SolveLambda()}{\ccode{esl\_scorematrix\_SolveLambda()}} & Find $\lambda$ for score matrix, given background.\\
%\hyperlink{func:esl_scorematrix_ReverseEngineer()}{\ccode{esl\_scorematrix\_ReverseEngineer()}} & Calculate the probabilistic basis of a score matrix.\\

\hline
\end{tabular}
}
\end{center}
\caption{The \eslmod{scorematrix} API.}
\label{tbl:scorematrix_api}
\end{table}

\subsection{An example of using the scorematrix API}

Figure~\ref{fig:scorematrix_example} shows an example of reading a
matrix file from disk, reverse engineering it \citep{YuAltschul03} to
obtain its target probabilities, background frequencies, and lambda,
then printing information about it:

\begin{figure}
\input{cexcerpts/scorematrix_example}
\caption{An example of using the \eslmod{scorematrix} module.}
\label{fig:scorematrix_example}
\end{figure}


\documentclass[a4paper,12pt]{report}

\usepackage{amsmath,amsfonts,mathtools}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{hyperref}

\begin{document}
\title{PHY294 Abridged}
\author{Aman Bhargava}
\date{Janaury 2020}
\maketitle

\tableofcontents

\chapter{Wave Applications}
\section{Polarization}

We learned last year that wave-particle duality leads to \textbf{predictable probabilities} and unpredictable individual behavior. The simplest example of this is polarization.
\begin{itemize}
\item Light oscillates perpendicular to its direction of propagation.
\item There are two possible polarization states: $\{\uparrow, \rightarrow\}$. A diagonal state is just a superposition of these two.
\item A \textbf{polarizer} allows only one polarization state through.
\end{itemize}
$$I = I_0 \cos^2(\theta)$$

\paragraph{What's with the $\cos^2$? } Remember how we learned that the energy of a wave is proportional to the amplitude$^2$? It's just like that here. There's this idea of a \textbf{probability amplitude}. If you square the probability amplitude, you get the actual \textbf{probability} (kind of like how squaring the amplitude gets you energy when considering oscillators).

\paragraph{What's the point of probability amplitudes? } The utility of using probability amplitudes is that, when you have two probabilities that affect eachother (e.g. in an additive way), you get the \textbf{correct answer} when you add the probability \textbf{amplitudes} and THEN square them to get the actual probabilities. This becomes particularly important in interference problems where probability waves can \textit{interfere} with eachother.

\paragraph{A note on unpolarized light: } All the light that we deal with in this course will be polarized. Unpolarized light doesn't have as nice of a mathematical characterization and is overall a pain to deal with.

\section{Interference}
\section{Double Slit}
Remember the double slit experiment? If you do, then you just need to know the following formula:

$$I(y) = 4I_0 \cos^2(\frac{\pi d y}{D \lambda})$$

\textit{Where $y$ is how far along the screen away from the centre you are, $d$ is the slit distance, $D$ is the distance to the screen}

The interference patterns occur due to a difference in path length. The probability waves of the photons interfere with eachother because they are not in phase, leading to the patterns. Just takes some trigonometry to derive! The $\cos^2$ is there, as again, due to the difference between probability amplitudes and probabilities.

\chapter{Schroedinger's Equation}
This chapter discusses the mathematical background required to understand, validate, and use Schroedinger's equation.
\section{Fourier Series}

We're going to re-do Fourier series with complex exponentials instead of the usual sin and cos functions because that's going to make actual analysis a lot easier.
\paragraph{Handy formulae: }
$$e^{i2\pi n x/L} = \cos(2\pi n x / L) + i\sin(2\pi n x/L)$$
$$cos(x) = \frac{1}{2}[e^{ix} + e^{-ix}]$$
$$\sin(x) = \frac{1}{2i}[e^{ix} - e^{-ix}]$$

\paragraph{Fourier Equations:}
$$f(x) = \sum_{n=-\infty}^{\infty} c_n e^{i2\pi n x / L} $$
$$c_0 = \frac{1}{L} \int_0^L f(x) dx$$
$$c_n = \frac{1}{L} \int_0^L f(x) e^{-i 2\pi n x /L}dx$$

Where $c_0 = A_0$, $c_n = \frac{a_n-ib_n}{2}$ for $n > 0$ and $c_n = \frac{a_n + ib_n}{2}$ for $n < 0$.

One of the main benefits of this complex exponential form is that it makes it easier for us to solve for when $L \to \infty$.

\subsection{Infinite Fourier Coefficients}

Since we have uncountably infinite Fourier bands when we expand the limits to $\pm \infty$, we can make a \textbf{function} that gives us the Fourier coefficients for any real number.

$$\phi(x) = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}} \tilde{\psi(k)} e^{ikx} dk $$
$$\tilde{\phi(k)} = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}} \phi(x) e^{-ikx} dx $$

\section{Matter Waves}

That Fourier series stuff becomes quite useful when we start thinking of \textbf{matter as waves}. The \textbf{de Broglie Hypothesis} states that all particles have wavelengths ($\lambda$).

$$p = \hslash k = \frac{h}{\lambda}; \,\,\, k = \frac{n\pi}{L}=\frac{\omega}{c};\,\,\,$$
$$\hslash = \frac{h}{2\pi}$$

Therefore, if we can represent a particle as a wave, we are good to go to analyze the components using fourier analysis! If the particle is described as a non-repeating pulse, then you need to use the $\lim_{L\to\infty}$ we talked about at the end of the Fourier series section. This allows us to convert from a plot that shows \textbf{location} and another that shows the \textbf{frequency (momentum)}.

\subsection{Uncertainty Principle}

Remember R.M.S. from ECE159? Me neither. Basically it's just a way of denoting how spread out a distribution of values on a graph is. For the example of a particle that is described by a local \textbf{pulse} of probability, we can find the RMS value for its position $\Delta x$.

We can also take the \textit{infinite bounds Fourier transform} and to get a distribution of the wave number intensities ($k$) and get the RMS value for that. But the wave number is just momenutum times a constant! So really, we know both the $\Delta x$ and the $\Delta k$ values!

Here's the really cool thing: We can derive a mathemtical relationship between the two! And that is: $$\Delta k \Delta x \geq \frac{1}{2}$$

Which makes a lot of sense! If you think about a function with an infinitely small RMS in the $k$-domain, you think of a function described with just one Fourier coefficient. That's just an infinite wave -- it has literally infinite RMS in the real domain. Same goes vice-versa. We did the actual math with a Gaussian function in class, but that's kind of extra for a review document like this. Just take my word for it and know it's pretty cool.

\subsection{Solving Problems with Uncertainty Principle}

Personally I didn't remember a lot of this stuff from PHY293, so here goes. In the real world, the uncertainty principle is: $\Delta x \Delta p \geq \frac{\hslash}{2}$. They key to solving is having an intuition for what happens to a randomly distributed variable when you square it.

Let's say that $x$ is normally distributed around $0$. The average value of $x$ ($<x>$) is obviously zero. But what about the average value of $x^2$? All those negative $x$ values turn positive when you square them, so $<x^2>$ must be positive. Here's the expression for a general variable:
$$<x^2> = <x>^2 + <\Delta x>^2$$

It makes a lot of sense for the square of the uncertainty to correlate with the mean squared value. The fact that $<x^2>$ is non-zero for anything where $\Delta x > 0$ is really important when you start thinking about \textbf{minimizing energy} in a system. If kinetic energy is $KE = \frac{1}{2m}p^2$, you're in a good position to solve some interesting problems if you know the bounds on $x$!

If you have an expression for energy that relies on momentum and/or position and you have some other boundary condition in place to fully constrain $p$ and $x$, you can generally use the uncertainty principle to deduce the \textbf{minimum possible allowable energy}.

\section{Schroedinger Equation}

\paragraph{GOAL: } Specify the \textbf{state} of the system and predict the \textbf{future state} of the system perfectly.

In classical terms, we would achieve this goal via the equations of motion ($\vec{F} = m\vec{a}$). For quantum, however, we have more of a focus on \textbf{probability amplitudes} rather than absolute values. Therefore, we have an equation for a wave function:

$$i\hslash \frac{\partial}{\partial t} \Psi(x, t)  =   \frac{-\hslash^2}{2m} \frac{\partial^2}{\partial x^2} \Psi(x, t) + V(x) \Psi(x, t)$$

$$E\phi(x) = \frac{-\hslash^2}{2m} \frac{d^2\phi(x)}{dx^2} + V(x)\phi(x)$$ % TODO: Populate this with the time-independent Schroedinger equation.

\begin{itemize}
\item Term 1: Representative of Total Energy.
\item Term 2: Representative of Kinetic Energy.
\item Term 3: Representative of Potential Energy.
\end{itemize}

Schroedinger arrived at this conclusion via educated trial and error. The solutions are of the form:

$$\Psi(x, t) = e^{i(kx-\omega t)}$$

This reduces to the following relationships:
$$\hslash \omega = \frac{( \hslash k )^2}{2m};\,\,\, \hslash \omega = E;\,\,\, (\hslash k)^2 = p^2$$

\paragraph{Separation of variables: } To find more solutions, we can try $\Psi(x, t) = \phi(x)f(t)$. We arrive at
$$\frac{i\hslash}{f(t)} \frac{df}{dt} = -\frac{\hslash^2}{2m}\frac{1}{\psi(x)}\frac{d^2\phi}{dx^2} + V(x) = C_{onst}$$

Since the left side is time dependent and the right side is space dependent, they must be equal to a constant. Bsed on this we have the following constraints:
\begin{enumerate}
\item $i \hslash \frac{df}{dt} = af(t)$
\item $\frac{-\hslash^2}{2m} \frac{d^2 \phi}{dx^2} + v(x)\phi(x) = a\phi(x)$
\item $a = \hslash \omega = E_{tot}$
\item Therefore, $$\Psi(x, t) = \phi_a(x) f(0)e^{-iat/\hslash}$$
\end{enumerate}

Know that $a$ can only come in \textbf{quantized} energy states. This is due to the fact that a non-quantized $a$ makes the equation very difficult to solve, and the experiments tend to agree that this is the case in quantum mechanics.

\subsection{Particle in a Box}
We now know the Schroedinger equation. When we try to `solve' it for a particle in an infinite square well, we must apply the following boundary conditions and assumptions:
\begin{enumerate}
\item For $0 < x < L$, $V = 0$. Elsewhere $V = \infty$
\item $\phi(x)$ is continuous.
\item $\phi(x) = 0$ for $x = L;\,\, x=0$.
\end{enumerate}

Handily, the equation simplifies to the following (we can use our amazing knowledge of wave mathematics to solve it):

$$\frac{-\hslash^2}{2m}\frac{d^2\phi(x)}{dx^2} = E\phi(x)$$

So clearly $\phi(x) = A e^{i\alpha x} + B e^{-i\alpha x}$ where $\alpha = \sqrt{\frac{2mE}{\hslash}}$. The solutions, including time-dependence, are as follows:

$$\Psi_n(x, t) = \sqrt{\frac{2}{L}} \sin(\frac{n\pi x}{L}) f_n(0) e^{-i E_n t/\hslash}$$

And the most general form is $\Psi = \sum_{n=1}^{\infty} \Psi_n(x, t)$. The coefficients that describe the varying intensity of each component wave is inside of $f_n(0)$. Also, make sure you know the \textbf{energy levels for a particle in a box}:

$$E_n = \frac{\hslash^2 n^2 \pi^2}{2mL^2},\,\,\,\, n \in N^*$$

\subsection{Time Dependence in Superposition}

When we just have one $n$ value (i.e. $\Psi$ is a perfect sinusoidal wave), it's not really time dependent. When we take the superposition of \textbf{more than one energy level}, we get time dependence in the form of what our professor calls `cross terms'.

$$P = \Psi^*(x, t) \Psi(x, t)$$

The rate at which time-dependent \textbf{sloshing} that occurs when you superimpose two energy levels is: $\omega = \frac{E_1-E_2}{\hslash}$.

\subsection{Determining the Likelihood of a Given Superposition}

When you make an observation of a particle whose state is described as superposition of many energy levels/states, it will collapse to just one of those states.

The probability of a given state being actualized can be deduced using the following steps:
\begin{enumerate}
\item We begin with a general wave function $\Psi(x)$ and a specific state $\psi_n(x)$. We wish to deduce the probability $P(\phi_n)$.
\item Remember that each $\phi_n$ comprises a basis set. $\Psi = \sum c_n \phi_n$. We can deduce $c_n$ by the inner product of $\Psi$, $\phi_n$ which is
$$c_n = \int_0^L \phi_n^*(x)\Psi(x)dx$$
\item $c_n$ gives the probability amplitude of $\phi_n$. We get $P(\phi_n)$ via $$P(\phi_n) = |c_n|^2$$.
\end{enumerate}

If we want to get the \textbf{expected value} of energy or wavelength, we just need to take the weighted average of that value with its associated probability.

\subsection{Quantum Harmonic Oscillator}

This classic example gives you a curve that describes the potential energy of a particle at a given distance from an atom. It's generally bowl shaped, but slightly irregular. Here are the steps taken to solve it:

\begin{itemize}
\item To simplify the potential energy curve, we Taylor expand at $r_0$. We use this as the $V(x)$ term in the Schroedinger equation such that $V(r) = V(r_0) + \frac{1}{2}\frac{d^V}{dr^2}$
\item Set $m$ to the \textbf{reduced mass} of the system ($\frac{1}{\frac{1}{m_1} + \frac{1}{m_2}}$)
\item Plug that value into the \textbf{time-independent} Schroedinger equation.
\item $\frac{-\hslash^2}{2m} \frac{d^2\phi}{dx^2} + \frac{1}{2}Kx^2 \phi(x) = E\phi(x)$ where $K$ is the spring constant.
\item Boundary conditions: $lim_{|x|\to\infty}\phi(x) = 0$
\item At large $|x|$, the first term might be there, the second term is definitely there due to the $x^2$, and the last term is definitely not there since it's finite.
\item We make a guess that, at large $x$, $\phi(x) \approx \exp(-\sqrt{\frac{mk}{\hslash^2}\frac{x^2}{2}})$
\item Since that function is $\approx 1$ at around $x = 0$, we can say that the final form is $\phi(x) = g(x) \exp(-\sqrt{\frac{mk}{\hslash^2}}\frac{x^2}{2})$. $g(x) = \beta_n$.
\end{itemize}

\paragraph{ENERGY OF A QHO: } $$E_n = (n+\frac{1}{2})\hslash \omega$$
\paragraph{Useful Facts: }
\begin{itemize}
\item $\omega = \frac{2\pi c}{\lambda}$
\item $\omega_0^2 = \frac{k}{m}$
\item Reduced mass: $$\frac{1}{ \frac{1}{m_1} + \frac{1}{m_2} }$$
\end{itemize}

\subsection{Probability Density and Current}

Do you remember anything whatsoever from the fluid dynamics portion of AER210? Me neither. Anyway, the idea here is that flowing incompressible fluids have similar behavior to probability moving in time and space.

$$\frac{\partial P}{\partial t} = -\frac{\partial J_x}{\partial x}$$

Where $P$ is the fluid density and $J_x$ is the current density. We can make the analogy that probability is equivalent to fluids. From Schroedinger's equation, we can simplify:

$$\frac{\partial P}{\partial t} = \frac{\hslash}{2mi} (\Psi\frac{\partial^2 \Psi^*}{\partial x^2} - \Psi^* \frac{\partial^2 \Psi}{\partial x^2})$$

From this we get that the \textbf{probability current} is $J_x = \frac{\hslash}{2mi} (\Psi^* \frac{\partial \Psi}{\partial x} - \Psi \frac{\partial \Psi^*}{\partial x})$

\subsection{Eherenfeist Theorem}

The macroscopic manifestations of quantum probabilities are usefully approximated as the \textbf{expected values} of the wavefunctions. For example:
$$\frac{d<x>}{dt} = \frac{<p>}{m};\,\,\, \frac{d<p>}{dt} = <\frac{-\partial V}{\partial x}>$$
In order to find the average value of a given variable in quantum, you must take the following integral:
$$<x> = \int \Psi^*\Psi x dx$$
So, for example: $<p> = \int P(x, t) [-i\hslash \frac{d}{dx}dx = \int \Psi^*(k, t) \Psi(k, t) \hslash k dk$

\chapter{Quantum Wave Transmission}
\section{Index of Refraction}
Generally speaking, the light phenomena taught in Grade 10 Ontario High School (e.g. index of refraction, optics, etc.) have solid analogues in quantum mechanics. That should make sense intuitively because light is best characterized by quantum mechanical models in many cases.

The refraction index for light is characterized by the varying \textbf{speeds of light} that exist in different media. That difference causes refraction, reflection, and a whole lot of other jazz.

\paragraph{Quantum version: } Instead of varying the speed of light in media, we consider a \textbf{potential energy step function}. The following attributes apply:
\begin{itemize}
\item Total energy remains the same from region I to II.
\item Potential energy goes from $V_{I} \to V_{II}$.
\item $P_I = \sqrt{2m(E-V_I)}$ and $P_{II} = \sqrt{2m(E-V_{II}}$.
\item $\lambda_I = h/P_I$, $\lambda_{II} = h/P_{II}$.
\end{itemize}

\paragraph{Wave function form: }
$$\phi_I(x) = Ae^{ik_1x} + Be^{-ik_1x}$$
$$\phi_{II}(x) = Ce^{ik_2x}$$
That second equation for after the potential energy step only has one term because only transmitted light gets through (by definition) and it's only characterized by one wavelength.

\subsection{Case I: $E > V_0$}

\paragraph{Boundary conditions: }
\begin{itemize}
\item $\phi(x)$ is continuous.
\item $\phi'(x)$ is continuous.
\end{itemize}

The continuity arises because a discontinuous function would lead to an infinite $\frac{d^2\phi}{dx^2}$ in the Schroedinger equation that doesn't make sense because all the other terms definitely go to zero when integrated over very short periods.

\paragraph{Conclusions on the wave functions: }
\begin{enumerate}
\item $A + B = C$
\item $(A-B) = \frac{k_2}{k_1}C$
\end{enumerate}

Classically, transmission coefficient is $T = \frac{J_{transmitted}}{J_{incident}}$ and the reflection coefficient is $R = \frac{J_{reflected}}{J_{incident}}$. When we apply \textbf{quantum equations...}

$$J_{trans} = |C|^2\frac{\hslash k_2}{m};\,\,\, J_{inc} = |A|^2\frac{\hslash k_1}{m};\,\,\, J_{refl} = |B|^2 \frac{-\hslash k_1}{m}$$
$$T = \frac{|C|^2}{|A|^2}(\frac{k_2}{k_1});\,\,\, R = \frac{|B|^2}{|A|^2}$$

\subsection{Case II: $E < V_0$}

$$A = \frac{1}{2}C(1 + \frac{i\beta}{k_1});\,\,\, B = \frac{1}{2}C(1-\frac{i\beta}{})$$
$$\frac{J_{refl}}{J_{inc}} = \frac{|B|^2}{|A|^2} = 1$$

Unexpectedly, the results are what one would intuitively reason given your experience with the material world. If the ball doesn't have enough energy to overcome the potential energy of a step, it's not getting up there. All particles in that situation are reflected back.

\paragraph{HOWEVER: } There's a weird pooling of probability right around the cusp of the step function. The wave coming to the step is a regular complex exponential (it's oscillating. The wave function right after the step is a \textbf{non-complex decaying exponential}! Because of course there has to be comething confusing about each topic in quantum :)

So yeah, $J_trans$, the probability current of the particle going above $x = 0$ is still 0, but the actual probability is greater than 1.

Low key this is actually seen in lens optics. When you have total internal reflection, there is this thing called the \textbf{evanescent wave} that is $\approx \lambda$ distance away from the surface of the medium that does have EM radiation.

\section{Quantum Tunneling}

\chapter{Quantum Measurements, Operators and the Like}

\section{Hamiltonians and Eigen Things}

If we manipulate the time-independent Schroedinger equation, we can get the following: $$[-\frac{\hslash^2}{2m} \nabla^2 + V(r)]\phi_n(x) = E_n\phi_n$$

If you look carefully, you can see the Eigen value form! Remember how the whole thing was that $\pmb{A}\vec{x} = \lambda \vec{X}$? Well here we have this big
\textbf{operator} in the square brackets times the \textbf{state} $\phi_n(x)$ being set equal to the scalar value for total energy $E$ times the same state $\phi_n(x)$

We call the stuff in the brackets the \textbf{Hamiltonian operator}. The state $\phi_n(x)$ is an \textbf{eigen state} or \textbf{eigen vector} of the Hamiltonian and the value of $E_n$ is an \textbf{eigen value} of the Hamiltonian.

\paragraph{Problem Solving Tips: }
\begin{itemize}
\item You can always re-derive the conventional Hamiltonian operator from the time-independent Schroedinger equation.
\item If you are asked to confirm that a state is an \textbf{eigen state}, just run it through the given operator. You should get a scalar multiple of the original state.
\item If you are asked to find the energy level of a given state, use the Hamiltonian operator (or whatever other operators you have access to, the rest will be discussed later)
\end{itemize}

\end{document}


In this chapter we describe how the solver for the MKC problem were
implemented. This implementation is a sample for a column generation scheme;
no cut generation is done. Since this problem is not so well known, first we
will describe the problem setting then the implementation details.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The MKC Problem}
\label{mkc:problem}

MKC stands for {\em Multiple Knapsack problem with Color constraints} as it is
derived by generalizing the multiple knapsack problem along two directions:
(i) adding assignment restrictions on items which can be assigned to a
knapsack, (ii) adding a new attribute (called ``color'') to the items and then
adding the associated ``color'' constraints which restrict the number of
distinct colors which can be assigned to a knapsack to two.

This problem is motivated by the surplus inventory matching problem in the
steel industry (\cite{KDTL}): before planning production, an attempt is made
to satisfy orders using leftover slabs from surplus inventory. The goal of
inventory matching is to maximize the total weight of the orders satisfied
from the leftover and to minimize the leftover weight of each slab used in the
matching. For each order we can identify a set of applicable slabs from the
surplus inventory.  These assignment restrictions are based on quality and
physical dimension considerations.  For any given order only slabs which are
of the same quality or better can be applied.  In addition, the thickness and
width requirements for each order need to be compatible with those of the slab
applicable.  These considerations restrict the number of applicable slabs for
each order. The color constraints place restrictions on the sets of orders
that can be matched to the same slab in the surplus inventory. Because of
processing considerations in the finishing line of a steel mill not all orders
assignable to a slab can be packed together on the slab. There is a route
associated with each order that specifies the set of process operations that
need to be applied in the finishing mill. Orders with different routes require
different process operations and are referred to as being of different types.
Slabs packed with different order types need to be cut before they are
processed in the finishing mill.  Since cutting slabs is expensive and often
the cutting machine is a bottleneck, strong constraints are posed in terms of
the number of allowed cuts per slab. The simplest and most commonly used
constraint used is to limit the number of required cuts to one; i.e., no more
that two order types are allowed on a slab.  In order to describe this
constraint formally we associate a unique {\em color} with each route code and
restrict the number of colors on a slab to be no more than two.  Notice that
this implies that we associate a color with each order based on its route
code.  This restricts the number of different order types on a slab to two and
the number of required cuts to be no more than one.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Natural formulation for MKC}

This formulation has three sets of variables and four sets of constraints
modeling the various restrictions.
\begin{eqnarray}[r@{\eqsep}c@{\eqsep}lqql]
\multicolumn{3}{c}{
\max \sum_{i = 1}^{N}\sum_{j \in N^i} w^i x^i_j -
     \sum_{i = 1}^{N} (W_j - \sum_{j \in N^i} w^i x^i_j) z_j
} & \nonumber \\
\sum_{i \in N_j} w^i x^i_j & \le & W_j z_j & 1 \le j \le M \label{con-ks} \\
\sum_{j \in N^i} x^i_j     & \le & 1       & 1 \le i \le N \label{con-order} \\
\sum_{c \in C_j} y^c_j     & \le & 2       & 1 \le j \le M \label{con-col1} \\
x^i_j & \le & y^{c^i}_{j}  & 1 \le i \le N, ~j \in N_{i} \label{con-col2} \\
x^i_j & \in & \{0,1\}      & 1 \le i \le N, ~j \in N_{i} \nonumber \\
y^c_j & \in & \{0,1\}      & \forall c \in C_j, ~1 \le j \le M \nonumber \\
z_j   & \in & \{0,1\}      & 1 \le j \le M \nonumber
\end{eqnarray}

\begin{table}[ht]
\caption{List of notations}
\begin{center}
\begin{tabular}{|l@{ : }l|}
\hline
$N$ & Total number of orders.\\
$M$ & Total number of slabs. \\
$N^i$ & Set of slabs incident to order $i$. \\
$N_j$ & Set of orders incident to slab $j$. \\
$w^i$ & Weight of order $i$. \\
$W_j$ & Weight of slab $j$. \\
$C_j$ & Set of colors incident on slab $j$. \\
$c^i$ & The color of order $i$. \\
$x^i_j$ & 1 if order $i$ is assigned to slab $j$; 0 otherwise. \\
$y^c_j$ & 1 if orders of color $c$ obtain material from slab $j$; 0
otherwise.\\
$z_j$ & 1 if any order is incident to slab $j$; 0 otherwise. \\
\hline
\end{tabular}
\end{center}
\end{table}

The total number of variables in this formulation is
$$
\sum_{i=1}^{N} |N^i| + \sum_{j=1}^{M} |C_j| + M \quad=\quad
\sum_{j=1}^{M} |N_j| + \sum_{j=1}^{M} |C_j| + M
$$
while the total number of constraints is $2\sum_{i=1}^{N} |N^i| + 2M + N$.

Constraints (\ref{con-ks}) specify that if a slab is used then the total
weight of the orders assigned to the slab cannot exceed the weight of the
slab; Constraint (\ref{con-order}) describes that each order will be made at
most once; while constraints (\ref{con-col1}) and (\ref{con-col2}) enforce the
coloring restriction.

Notice that the objective function is non-linear. However, since $z_j = 0$
forces $x^i_j$ to be zero for all $i \in N_j$ and $z_j = 1$ implies
$x^i_j z_j = x^i_j$, for all feasible solutions the objective function is
equivalent to
$$
\sum_{i = 1}^{N}\sum_{j \in N^i} w^i x^i_j -
    \sum_{i = 1}^{N} ( W_j z_j - \sum_{j \in N^i} w^i x^i_j ) =
\sum_{i = 1}^{N}\sum_{j \in N^i} 2w^i x^i_j - \sum_{i = 1}^{N} W_j z_j
$$.

The final observation is that the objective function just combines the two
stated goals (maximizing satisfied orders and minimizing wasted parts of
slabs) with equal weights. This may or may not be the best composite
objective, but this is how the creator of the application specified the
problem. Also, all that a different composite weight would change is the
multiplier $2$ for $w^i$ (the coefficient of $x^i_j$) and the multiplier $1$
for $W_j$ (the coefficient of $z_j$); nothing in the proposed algorithms would
need to be changed.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{A formulation suitable for column generation}

This new formulation has significantly more columns than the original
formulation, on the other hand it results in a well studied problem, the set
packing problem (\cite{NW}).

There are two types of constraints in this formulation. The first type
corresponds to the slabs in the problem, the second type to the orders. The
variables represent feasible production patterns, that is, variable $u$ has a
$1$ in the row corresponding to the slab the production pattern is to be made
of and $1$'s in the rows corresponding to the orders in the production
pattern. Each variable is a binary variable indicating whether that production
pattern is chosen in the solution or not. Let us introduce the following
notation:
\begin{itemize}
\item $P$ is the set of feasible production patterns;
\item $P_j$ is the set of set of feasible patterns manufacturable from slab
  $j$;
\item $P^i$ is the set of set of feasible patterns containing order $i$;
\item $R_k$ is the row (constraint) corresponding to the slab the production
  pattern corresponding to $u_k$ is made of; and
\item $R^k$ is the set of rows (constraints) corresponding to the orders in
  the production pattern corresponding to $u_k$.
\end{itemize}
Let the cost of variable $u_k$ be $\bar{c}_k = \sum_{i\in R^k} 2w^i - W_{R_k}$
and create the following set packing problem:
\begin{eqnarray}[rclqql]
\multicolumn{3}{l}{\max \sum_{k\in P} \bar{c}_k u_k} & \nonumber \\
\sum_{k\in P^i} u_k & \le & 1 & \forall 1 \le i \le N \\
\sum_{k\in P_j} u_k & \le & 1 & \forall 1 \le j \le M \\
\multicolumn{3}{l}{u_k\in\{0,1\}} & \forall k \in P \nonumber
\end{eqnarray}

It is very easy to see that there is a one to one correspondence between the
feasible solutions of this set packing problem and the feasible solutions of
the original formulation. Moreover, the construction of the $\bar c$ cost
vector ensures that the corresponding solutions have identical objective
values. Therefore optimizing this problem is the same as optimizing the
original formulation.

The obvious problem with this formulation is that the number of feasible
production patterns is enormous.

\subsection{Generating columns with positive reduced costs}
To improve the solution evenly, for each slab we generate a production pattern
whose corresponding column has the highest reduced cost, i.e., the most
positive if there is one with positive reduced cost. Finding these columns is
again a set of optimization problems, since for a dual vector $\pi$ the
reduced cost of variable $u_k$ whose production pattern is made of slab $j$ is
simply
\begin{equation}
\bar{c}_k - \pi_j - \sum_{i\in R^k} \pi^i =
\sum_{i\in R^k}2w^i - W_j - \pi_j - \sum_{i\in R^k} \pi^i =
- (W_j + \pi_j) + \sum_{i\in R^k} (2w^i - \pi^i)
\end{equation}
and we want to maximize this over the set of production patterns that can be
manufactured from slab $j$. For a fixed $j$ the first term is constant. The
feasible production patterns from slab $j$ are those that satisfy the capacity
and color constraints, thus this problem is equivalent to (using the notation
from the original formulation):
\begin{eqnarray}[rcl]
\multicolumn{3}{l}{\max\sum_j (2w^i - \pi_i) x^i_j} \\
\sum_i w^i x^i_j & \le & W_j \\
\sum_i y^{c(i)}_j & \le & 2 \\
\multicolumn{3}{l}{x^i_j \in \{0,1\}}
\end{eqnarray}
which is a knapsack problem with the side constraint that selected objects
must have no more than two different colors. Moreover, the constant term in
the reduced cost implies that we are only looking for production patterns
whose reduced cost exceeds $W_j + \pi_j$. Since solving the LP relaxation of
the knapsack problem (even with the side constraint) is rather simple, this
required lower bound on the reduced cost can be very helpful in quickly
concluding that there is no improving pattern for a particular slab.

\subsection{Upper bounding}

The previous subsection addresses the issue of how to solve the full LP
relaxation by iteratively solving smaller LP relaxations and generating
columns, but we need something more. We need to be able to derive an upper
bound on the optimal objective value of the full LP relaxation in every
iteration. There are two reasons for this. The first is that in a
Branch-and-Price algorithm we can fathom a search tree node if the upper bound
on the optimal objective value of the LP relaxation at the node is already
lower than the value of a currently known feasible solution. Since we may not
be able to solve the subproblems that generate the columns (after all, even
though the knapsack problem is considered relatively easy, it {\em is}
NP-complete) we still want to have an upper bound for fathoming purposes. The
second reason is that without an upper bound on the optimal value of the LP
relaxation of the full problem we couldn't tell how close we are to
optimality, we wouldn't have a proven gap.

Fortunately, upper bounding is very easy using Dantzig-Wolfe decomposition
\cite{dantzig-wolfe}. Since the sum of all variables in $P_i$ is not more than
$1$, the objective value of the LP relaxation cannot change more than the
highest reduced cost (or an upper bound on that value) as a result of changing
the values of the variables in $P_i$. To get an upper bound on the reduced
cost we can use the LP relaxation of the subproblem (the side constrained
knapsack problem) which is very easy to solve. Now adding all ``per slab''
upper bounds to the optimal objective value of the current LP relaxation
yields an upper bound on the optimum of the full LP relaxation.

\subsection{Finding integral feasible solutions}

Another advantage of the column generation based formulation is that it is
very easy to generate feasible solutions. In each iteration we considered the
fractional solution and started by including every variable above $0.5$ in the
solution. From the set of remaining fractional variables we exluded all that
intersected the already selected variables. Whatever remained afterwards was
always such a small set that we could solve the set packing problem on that
set by enumeration.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Implementation details}

\subsection{Cuts, variables and solutions}
First of all, we did not have to worry about anything cut generation related,
since we were not generating cuts. Since the number of constraints is not too
great (number of orders + number of slabs) we decided to treat all of them as
core constraints, thus completely eliminating the need to bother about cuts.

For the variables first we had to decide which ones are going to be core
variables and which ones will be extra variables. Since we had no
reason to believe that any one particular pattern was more likely to be in
an optimal solution than some other pattern we decided not to have core
variables at all (this also simplified coding somewhat). Since the variables
are the feasible production patterns, they do not lend themselves to any
enumeration scheme, so we decided not to have indexed variables either.
Therefore all our variables are algorithmic ones. Actually, we had two kind of
algorithmic variables, one for the production patterns and another one for
branching, but we will discuss that latter in Section \ref{mkc:branching}. Both
types of variables are derived from \code{BCP\_var\_algo} and are defined in
\code{MKC\_var.hpp}.

We have defined our solution class for two reasons. First, all our pattern
variables are binary variables so there is no reason to include the value of
the non-zero ones. Second, there might be branching variables (not pattern
variables) that are at nonzero level as we go down in the tree, and we didn't
want to include those in a feasible solution. Still, if we wanted to, we could
have used a generic solution type. We just thought that using our own solution
type makes the code clearer.

\subsection{Branching}
The problem with branching when generating columns is that we must be able to
generate columns after branching, too. In other words, every generated column
must conform to whatever branching decisions have been made to that point.
That means that branching on a regular variable is out of question. On one
side (when it is fixed to 1) we'd have great results, it would significantly
shrink the search space. However, on the other side (fixing the variable to 0)
the restriction is that we cannot regenerate that variable. But that variable
will almost always ``want to be regenerated'' (definitely immediately after
branching), since its reduced cost will make it attractive (after all, we have
forcibly moved it away from where it ended up in the LP-optimal solution). So
for our problem this would mean that after one branching we have to check the
optimal solution to the knapsack subproblem and if it is the forbidden
variable then we have to find the second best solution. After two branchings
we may have to find the third best solution, etc. This is impossible.

Instead, the following logic is introduced. A branching object will specify
whether a particular order $O$ is manufactured from slab $S$ or not.

\subsection{Packing and unpacking}

Packing and unpacking of user objects is really straightforward. For example,
look at the \code{MKC\_var\_(un)pack()} functions. The packing function packs
the
type of each variable and invokes the pack member of the variables while the
unpacking function unpacks the types and invokes the appropriate constructor.

In general, when an object is packed it is simply torn down to built-in types
and those are packed. On the other side the date is unpacked in the same order
and the appropriate objects are constructed. In the following subsection we
will not mention the (un)packing member methods.

\subsection{MKC\_init}

This is the implementation of the intializer class. The TM initializer reads
in the problem and the parameters. Unfortunately this piece is rather
complicated since the problem is specified as an MPS file and we have to
extract order and slab information from it. The problem is loaded into the
\code{kss} member of the \code{MKC\_tm} class. See the \code{MKC\_knapsack.hpp}
file for data structures. Once the problem is read in a pointer to the
\code{MKC\_tm} class is returned. The LP initializer just returns a pointer to
an empty \code{MKC\_lp} object.

\subsection{MKC\_tm}

There were only four methods (besides the (un)packing ones) we had to deal
with. Initializing the core consisted of simply specifying the core cuts as we
had no core variables (hence no core matrix). Since we had no core variables
we had to add some extra variable in creating the root. There are two options
for this, one is to add those variables that were read in from a file (maybe
as a result of a previous run), or we could generate columns for the all zero
dual solution (which is in some sense the optimal solution if we have no
variables...). Displaying the solution has the option to test the solution
that it really satisfies the original formulation (we have used this for
debugging purposes) and then the solution is printed in two different ways.
Finally, there will be only one phase and we will generate columns in it, so
we set this in the \code{init\_new\_phase()} method.

\subsection{MKC\_lp}


\chapter{Natural Gas Flow}
\label{chap:fund_NGF}

The steady-state Natural Gas Flow (NGF) problem for transmission networks is aimed at finding the value for a set of state-variables that satisfy the flow balance in all nodes. We show how the NGF can be derived in a similar way as the Power Flow (PF) problem is introduced for power systems. In  particular, a set of nonlinear equations must be solved where the definition of the state-variables depends on the selected models for all of the elements of the system. In this section, we derive the NGF problem and introduce the modeling for the main elements considered in \mpng{}: nodes, wells, pipelines, compressors, and storage units.

\section{Modeling}
\label{sec:gas_modeling}

An exact description of the natural gas flow in transmission networks requires applying the laws of fluid mechanics and thermodynamics \cite{Osiadacz2001}. Complex analyses provide an accurate description for variables such as temperature, pressure, flow, adiabatic head, among others, for all time instants. However, as the primary concern of \matpower{} (and of \mpng{}) is the system operation in steady-state, we define some models to describe the main elements of the default natural gas network, as explained below.

\subsection{Nodes}
\label{subsec:nodes}

By definition, a node is the location of a natural gas system where one or more elements are connected. Users are commonly associated with a node where a stratified demand is modeled as different market segments that get different priorities. Figure \ref{fig:node} shows the $i$-th node of a gas network with some traditional markets connected to form the nodal demand $f_{dem}=\sum_{j} f_{\text{dem}_j}$. The primary variable related to a node is pressure $p_i$, although, as explained later, the actual state variable used in \mpng{} is the quadratic pressure.

\begin{figure}[!ht]
	\centering
	\includegraphics[scale=1.2]{Figures/Node}
	\caption{A natural gas node and some traditional markets.}
	\label{fig:node}
\end{figure}

\subsection{Wells}
\label{subsec:wells}

Natural gas is extracted from deep underground and injected into the system from wells. Depending on the well capacity, the injection can be made either at constant pressure, where a control system regulates the amount of gas flow such that pressure behaves constant, or at constant flow, where pressure is adjusted such that injected flow remains constant. Figure \ref{fig:well} shows a well connected to the $i$-th node whose operation depends on two principal variables, the injected gas flow $f_{inj}^w$, and the nodal pressure $p_i$.

\begin{figure}[!ht]
	\centering
	\includegraphics[scale=0.9]{Figures/Well}
	\caption{A natural gas well.}
	\label{fig:well}
\end{figure}

\subsection{Pipelines}
\label{subsec:pipelines}

In general, the flow of gas through pipes is studied using the energy equation of fluid mechanics~\cite{Banda2006}. However, in practice, the relationship between the gas flow in the pipe and the upstream and downstream pressures can be described by various equations. The Weymouth's general flow equation is a frequent choice in gas industry applications to model the steady-state flow in pipes in gas transports networks~\cite{Woldeyohannes2011}. Figure \ref{fig:pipeline} shows a pipeline $o$ whose gas flow from node $i$ to node $j$ is represented by $f_{ij}^o$. The Weymouth equation states the relationship between $f_{ij}^o$, $p_i$, and $p_j$ in the following form:

\begin{equation}
	\label{eq:Weymouth_eq1}
	\text{sgn}(f_{ij}^o)(f_{ij}^o)^2 = K_{ij}(p_i^2-p_j^2).
	\vspace{0.3cm}
\end{equation}

\begin{figure}[!ht]
	\centering
	\includegraphics[scale=1]{Figures/Pipeline}
	\caption{A natural gas pipeline.}
	\label{fig:pipeline}
\end{figure}

In Equation \ref{eq:Kij}, $\text{sgn}(\cdot)$ represents the sign function, and $K_{ij}$ is the  Weymouth constant\footnote{Measured in Million Standard Cubic Feet per Day (MMSCFD) over psia. Different expressions for $K_{ij}$ can be derived depending on the parameters used in the flow equations. See \cite{Woldeyohannes2011} and reference therein for details.} of the pipeline defined in terms of the pipe length and diameter as below~\cite{Wolf2000}:

\begin{equation}
	\label{eq:Kij}
	K_{ij} = \sqrt{5.695756510\times 10^{-13}\:\frac{D^5}{\lambda Z T L\delta}}\quad \left[\frac{\text{MSCFD}}{\text{psia}}\right],
	\vspace{0.3cm}
\end{equation}

where:

\begin{equation}
	\label{eq:lambda_Kij}
	\frac{1}{\lambda} = \left[2\log \left(\frac{3.7D}{\varepsilon}\right)\right]^2,
	\vspace{0.3cm}
\end{equation}

with:

\begin{labeling}{alligator}
	\item [$\qquad \qquad  D$]  \hspace{0.8cm} Diameter [in].
	\item [$\qquad \qquad  L$]  \hspace{0.8cm} Length [km].
	\item [$\qquad \qquad  T$] \hspace{0.8cm} Gas temperature [K].
	\item [$\qquad \qquad  \varepsilon$] \hspace{0.8cm} Absolute rugosity [mm].
	\item [$\qquad \qquad  \delta$] \hspace{0.8cm} Gas density relative to air [-].
	\item [$\qquad \qquad  Z$] \hspace{0.8cm} Gas compressibility factor [-].
\end{labeling}

For mathematical convenience, we rewrite Equation \ref{eq:Weymouth_eq1} as follows:

\begin{equation}
	\label{eq:Wymouth_eq_2}
	f_{ij}^o = K_{ij} \;\text{sgn}(\pi_i - \pi_j)\sqrt{|\pi_i-\pi_j|},
	\vspace{0.3cm}
\end{equation}

\noindent where $\pi=p^2$ is defined as the quadratic pressure.

As seen, the gas flow through a pipeline is a nonlinear function of the quadratic pressures of the initial and final nodes, that is, $f_{ij}^o=g(\pi_i,\pi_j)$.

\subsection{Compressors}
\label{subsec:compressors}
As seen in Equation \ref{eq:Wymouth_eq_2}, there exists a downstream pressure drop when transporting large flows through pipes caused by energy losses. Analogous to the transformer in power systems, compressors are installed in the gas network to compensate pressure drops. Figure \ref{fig:compressor} shows a compressor $c$ that increases the discharge pressure $p_j$ with respect to the suction pressure $p_i$ by compressing gas in a way that a flow $f_{ij}^c$ passes through it. Assuming an adiabatic process, the power demanded by the compressor, $\psi_c$, states the relationship between the flow and the suction and discharge pressures in the following way~\cite{Shabanpour2016}:

\begin{equation}
	\label{eq:comp_flow}
	f_{ij}^c = \frac{\psi_c}{B_c \left[\left(\frac{\pi_j}{\pi_i}\right)^{\frac{Z_c}{2}}-1\right]},
	\vspace{0.3cm}
\end{equation}

where $B_c$ is the compressor constant that describes its construction features, $Z_c$ is the compressibility factor, and $\pi=p^2$ is again the quadratic pressure.

\begin{figure}[!ht]
	\centering
	\includegraphics[scale=0.9]{Figures/Compressor}
	\caption{A natural gas compressor.}
	\label{fig:compressor}
\end{figure}

Moreover, the compressor ratio, $\beta_c$, is defined as below:

\begin{equation}
	\label{eq:comp_ratio}
	\beta_c = \frac{\pi_j}{\pi_i}, \quad \beta_c\geq 1.
	\vspace{0.3cm}
\end{equation}

In general, there exist two types of compressors: the power-driven compressors, whose demanded energy is supplied from the power system, and the gas-driven compressors, that require additional gas to operate. In the latter, the additional gas demanded at the suction node, $\phi_c$, can be expressed as a quadratic function of the power as~\cite{Chen2017}:

\begin{equation}
	\label{eq:f_cons_gas_comp}
	\phi_c = x+y\psi_c+z\psi_c^2,
	\vspace{0.3cm}
\end{equation}

where $x,y,z \in \Real$.

Notice that the gas flow through a compressor (as well as the consumed flow for a gas-driven compressor) is a function of the consumed power and the quadratic suction and discharge pressures, that is, $f_{ij}^c=h(\pi_i,\pi_j,\psi_c)$.

\subsection{Storage Units}
\label{subsec:sto_units}

The possibility of storing natural gas provides flexibility with regards to production and transportation decisions~\cite{Midthun2007}. A storage unit is a reservoir that allows both storing from and injecting gas to the gas network. Figure \ref{fig:storage} shows a storage unit located at node $i$ with an associated gas flow $f_s$ that could be either an \textit{outflow} in the case of injection to the system or an \textit{inflow} in the case of a storing operation. Then, in a node with a specific demand and injection, the (known) value of $f_s$ could be added to the nodal demand when it is a storage inflow or could be summed to the injection flow of a constant-flow well when it is a storage outflow.

\begin{figure}[!ht]
	\centering
	\includegraphics[scale=1]{Figures/Storage_unit}
	\caption{A natural gas storage unit.}
	\label{fig:storage}
\end{figure}

\section{Deriving the Natural Gas Flow Problem}
\label{sec:NGF_problem}

Let us consider the transmission natural gas network shown in Figure \ref{fig:nodal_balance}. According to the principle of conservation of mass, the balance equation applied to the node $i$ states:

\begin{equation}
	\label{eq:balance_eq}
	f_{inj} - f_{dem} \pm f_s = \sum_{\substack{j=1 \\ j \neq i}}^{m} f_{ij},
\end{equation}

where $f_{dem}$ is the known demand flow, $f_s$ is either the  inflow (negative) or outflow (positive) of the storage unit, and $f_{inj}$ is the injected flow of the well.

\begin{figure}[!ht]
	\centering
	\includegraphics[scale=0.95]{Figures/Gas_flow_problem}
	\caption{Nodal balance in a transmission natural gas network.}
	\label{fig:nodal_balance}
\end{figure}

Notice in equation \ref{eq:balance_eq} that the left hand side is a known value if injection is produced by a constant-flow well. However, for a constant-pressure well, the value of $f_{iny}$ must be determined. In turn, the algebraic sum of the right hand side depends on the nature of the elements connected between nodes $i$ and $j$ such that $f_{ij}$ takes the form of $f_{ij}^o$ or $f_{ij}^c$, according to Equations \ref{eq:Wymouth_eq_2} and \ref{eq:comp_flow}, for a pipeline or a compressor, respectively. Moreover, for a gas-driven compressor, the additional gas consumption $\phi_c$ must be considered as an outflow in Equation \ref{eq:balance_eq} if node $i$ matches the suction node for compressor $c$.

%such that $f_{ij}$ is $g(\pi_i,\pi_j)$, $h(\pi_i,\pi_j,\psi_c)$, or $h(\pi_i,\pi_j,\psi_c)+\phi_c(\psi_c)$ for a pipeline, a power-driven compressor, and a gas-driven-compressor, respectively.

As a consequence, we can rewrite the balance equation applied at node $i$ in a functional form as follows:

\begin{equation}
	\label{eq:balance_eq_funtional}
	\mathbb{F}_i\left(\pi,\psi_c,f_{iny}^{w}\right) = 0.
	\vspace{0.3cm}
\end{equation}

In practice, for a given gas network with $n_n$ nodes, $n_c$ compressors, and $n_{w_p}$ constant-pressure wells, the application of the balance equation for all nodes will produce a set of $n_n$ nonlinear equations with a number of $(n_n-n_{w_p})+n_c+n_{w_p} = n_n+n_c$ unknown variables. To get a square system with the same number of equations and variables, the $n_c$ missing equations are obtained from the compressor ratios of all compressors. Then, the NGF problem can be formulated as follows:

\begin{equation}
	\label{eq:F_i=0}
	\mathbb{F}_i\left(\pi,\psi_c,f_{iny}^{w}\right) = 0, \quad \forall i\in\mathcal{N}, c\in\mathcal{C}; w\in\mathcal{W}_p,
\end{equation}

\begin{equation}
	\label{eq:R_c=0}
	\beta_c = \frac{\pi_j}{\pi_i}, \quad \forall c\in\mathcal{C}, \;  i,j\in\mathcal{N},
	\vspace{0.3cm}
\end{equation}

where

\begin{labeling}{alligator}
	\item [$\qquad \qquad  \mathcal{N}$]  \hspace{0.85cm} Set of gas nodes, $|\mathcal{N}|=n_n$.
	\item [$\qquad \qquad  \mathcal{C}$]  \hspace{1cm} Set of compressors, $|\mathcal{C}|=n_c$.
	\item [$\qquad \qquad  \mathcal{W}_p$] \hspace{0.65cm} Set of constant-pressure wells, $|\mathcal{W}_p|=n_{w_p}$.
\end{labeling}


\documentclass{subfile}

\begin{document}
	\section{AzNO}\label{sec:azno}

		\begin{problem}[$2020$ National Olympiad, problem $3$]
			$a,b,c$ are positive real numbers such that $a+b+c=3$. Prove that
				\begin{align*}
					\sum\dfrac{a^{2}+6}{2a^{2}+2b^{2}+2c^{2}+2a-1}
						& \leq3
				\end{align*}
		\end{problem}

		\begin{problem}[$2015$ National Olympiad, problem $1$]
			Let $a,b,c$ be positive real numbers such that $abc=\frac{1}{8}$. Prove that
				\begin{align*}
					a^{2}+b^{2}+c^{2}+a^{2}b^{2}+b^{2}c^{2}+c^{2}a^{2}
						& \geq\dfrac{15}{16}
				\end{align*}
		\end{problem}

		\begin{problem}[$2016$ Team Selection Test, problem $1$, day $3$]
			Let $a_{1},a_{2},\ldots$ be a sequence of positive real numbers such that
				\begin{align*}
					a_{k+1}
						& \geq\dfrac{ka_{k}}{a_{k}^{2}+k-1}
				\end{align*}
			for every positive integer $k$. Prove that
				\begin{align*}
					a_{1}+\ldots+a_{n}
						& \geq n
				\end{align*}
			for every positive integer $n$.
		\end{problem}

		\begin{problem}[$2016$ Balkan Mathematical Olympiad Team Selection Test $4$, problem $1$]
			Let $a,b,c$ be non-negative real numbers. Prove that
				\begin{align*}
					3(a^{2}+b^{2}+c^{2})
						& \geq (a+b+c)\left(\sqrt{ab}+\sqrt{bc}+\sqrt{ca}\right)+(a-b)^{2}+(b-c)^{2}+(c-a)^{2}
						  \geq (a+b+c)^{2}
				\end{align*}
		\end{problem}
\end{document}


\section{Conclusions}

For what about the SVM formulations, it is known, in general, that the \emph{primal formulation}, is suitable for large linear training since the complexity of the model grows with the number of features or, more in general, when the number of examples $n$ is much larger than the number of features $m$, i.e., $n \gg m$; meanwhile the \emph{dual formulation}, is more suitable in case the number of examples $n$ is less than the number of features m, i.e., $n < m$, since the complexity of the model is dominated by the number of examples, or more in general when the training data are not linearly separable in the input space.

\bigskip

From all these experiments we can see as all the \emph{custom} implementations underperforms all the others, i.e., both \emph{cvxopt}~\cite{vandenberghe2010cvxopt} and \emph{sklearn} implementations, i.e., \emph{liblinear}~\cite{fan2008liblinear} and \emph{libsvm}~\cite{chang2011libsvm} implementations, in terms of \emph{time} obviously due to the different core implementation languages, i.e., Python and C respectively.

In the \emph{primal} formulations the \emph{liblinear}~\cite{fan2008liblinear} implementation uses an optimization method called \emph{Coordinate Gradient Descent} which minimizes one coordinate at a time.

Meanwhile, for what about the \emph{Wolfe dual} formulations we can notice as \emph{cvxopt}~\cite{vandenberghe2010cvxopt} underperforms the \emph{sklearn} implementation, i.e., \emph{libsvm}~\cite{chang2011libsvm} implementation, in terms of \emph{time} since it is a general-purpose QP solver and it does not exploit the structure of the problem, as SMO does. An intresting consideration can be made about the number of \emph{iterations} of \emph{custom} SMO implementation wrt that in \emph{libsvm} which seems to be always lower thanks to the improvements described in~\cite{keerthi2001improvements, shevade1999improvements} for classification and regression respectively.

Finally, in the \emph{Lagrangian dual} formulations the goodness of the solution in terms of \emph{accuracy} or \emph{r2} values depends on the residue in the solution of the \emph{Lagrangian dual} at each step provided by \emph{minres} algorithm. Moreover, we can see as fitting the intercept in an explicit way, i.e., by adding Lagrange multipliers to control the equality constraint always get lower scores wrt the \emph{Lagrangian dual} of the same problem with the bias term embedded into the weight matrix.


\documentclass{article}

%----------------------------------------------------------------------------------------

\usepackage{listings} % Required for inserting code snippets
\usepackage{geometry}
\geometry{margin=0.7in}
\usepackage[usenames,dvipsnames]{color} % Required for specifying custom colors and referring to colors by name
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{tikz}
\usepackage{enumerate}
\usepackage[T1]{fontenc}

\delimitershortfall-1sp
\newcommand\abs[1]{\left|#1\right|}

\definecolor{DarkGreen}{rgb}{0.0,0.4,0.0} % Comment color
\definecolor{highlight}{RGB}{255,251,204} % Code highlight color

%----------------------------------------------------------------------------------------

\begin{document}

%----------------------------------------------------------------------------------------

\subsubsection*{Group Problem 5.5}
\begin{flushleft}
Recall that a function $f \in \mathbb{R}^2$ is called a polynomial if there is a non-negative integer $n$ (called "the degree" of $f$) and real numbers $c0, c1, ..., cn$, such that for every $x \in \mathbb{R}$.
\begin{center}
$f(x) = \sum_{i=0}^{n} c_ix^i$
\end{center}
Use induction and the theorem about combining the limits to prove that every polynomial is continuous on $\mathbb{R}$. (Note: the results about combining limits apply to at most two functions at a time. Also don't forget that you must check the case when $n = 0$, specifically, when $f$ is a constant function).
\end{flushleft}

\subsubsection*{Proof}
\begin{flushleft}
First, we'll prove the base case of $n=1$. When $n=1$,
\begin{center}
$\sum_{i=0}^{1} c_ix^i = c_0 + c_1x$.
\end{center}
Since the limit of the resulting function is constructed based on the two limit definitions of
\begin{center}
\begin{tabular}{c c}
$\lim_{x \to a} c = c$ & $\lim_{x \to a} x = a$,
\end{tabular}
\end{center}
we can combine them to show that $c_0 + c_1x$ is continuous if the limit exists for $\lim_{x \to a} c_0 + c_1x$. Thus, we know the polynomial of $n=1$ is also continuous.

\vspace{1.5cm}

Next, for the induction step, we assume a polynomial $P_n$ is continuous as defined $f(x) = c_0 + c_1x + ... + c_nx^n$. Let $g(x)=c_{n+1}x^{n+1}$. We know $c_1x$ and $c_nx^n$ are continuous from our initial assumption about $P_n$, so we can see that $c_{n+1}(x^n \cdot x)$ is also continuous for any $c \in \mathbb{R}$ as a result of

\begin{center}
\begin{tabular}{l}
$\lim_{x \to a} (f \cdot g)(x)$ \\
$\lim_{x \to a} (cf)(x)$, $\forall c \in \mathbb{R}$. \\
\end{tabular}
\end{center}

Since $g(x)$ is also continuous, we know by $\lim_{x \to a} (f + g)(x)$ that $P_{n+1}$ is also continuous. This completes the proof.
\end{flushleft}

%----------------------------------------------------------------------------------------

\end{document}


\subsection{Locally connected spaces}


\section{Equations}
	\label{sec:typesetting_equations}

	Typesetting equations is one of the things that LaTeX does best. It has packages for different fonts and symbols for many different mathematical notations. However, to person learning how to typeset in LaTeX for the first time it can be a daunting and unwieldy user experience. Almost all LaTeX packages have documentation available in pdf format online, and documentation for packages specifically relating to fonts and symbols usually have tables enumerating the names and codes for all of the fonts symbols, organized by intended usage.

	\subsection{Inline equations}

	Small equations like $x = 0$ can be written directly within the text by using LaTeX's maths mode shorthand controlled by dollar signs \lstinline|$ math mode $|. As long as it is not becoming cumbersome to the reader, equations such as $\mathbb{P}({A} \cap {B}) = \mathbb{P}({B} \cap {A})$ are quite neatly displayed in this fashion.

	\newpage

	\subsection{Block equations}

		For long equations it is best to provide a break in the main text of the document and format the equation using a \lstinline|\begin{equation}...\end{equation}| environment.

		\begin{equation} \label{eq:veclen}
			\left\lvert a \right\rvert = \left\lvert \left[\begin{array}{c} a_0\\ a_1\\ \vdots\\ a_n\end{array}\right] \right\rvert = \sqrt{a_0^2 + a_1^2 + \hdots + a_n^2}
		\end{equation}

		Equation \ref{eq:veclen} demonstrates formatting a larger equation and uses an \lstinline|\begin{array}...\end{array}| environment to structure a column vector of sub-equations. Block equations should be located at a relevant point directly as they are being referred to in the text. When referred to from other locations in the document you should use the \lstinline|\ref{key}| command to insert the correct equation number.

		\subsubsection{Aligning multi-line block equations}

			When equations become even larger they may need cross over multiple new lines. When this happens it is desirable to align relevant parts of the equation on each line to one another for aesthetic reasons and to help imply structure to the reader.

			\begin{equation} \label{eq:rendering_equation}
				\begin{split}
					\mathcal{L}_o\left(x, \omega_o, \lambda, t\right) &= \mathcal{L}_e\left(x, \omega_o, \lambda, t\right)\\
					&+ \int_\Omega f\left(x, \omega_i, \omega_o, \lambda, t\right) \mathcal{L}_i\left(x, \omega_i, \lambda, t\right) \left(\omega_i \bullet n\right) d\omega_i\\
					&\text{where} \quad \mathcal{L}_i\left(x, \omega_i, \lambda, t\right) = \mathcal{L}_o\left(x^\prime, -\omega_i, \lambda, t\right)\\
				\end{split}
			\end{equation}

			Equation \ref{eq:rendering_equation}, known as Kajiya's Rendering Equation \cite{kaj86} demonstrates the use of the \lstinline|\begin{split}...\end{split}| environment which uses a single un-escaped \& symbol placed on each line of the equations LaTeX code to indicate where each line should be co-aligned. In this example the \&'s were placed on the =, +, and w (in where) characters.

	\subsection{A masochistic approach to learning to typeset mathematics in LaTeX}

		\input{./graphics/texnique}

		TeXnique \cite{texnique} is web-browser based game for practising how to typeset equations in LaTeX. The game will present you with a rendered equation and your task is to type LaTeX code into the box below it such that your code produces the same (or closely matching / pixel equivalent) rendered equation. Figure \ref{fig:texnique} shows the game during play, the bottom rendered equation is bordered in green to indicate it is a valid match with the target.

		% An example of how to center a passage of text, control local fontsize,
		% and create a properly formatted and clickable URL.
		\begin{center}
		{\small \url{https://texnique.xyz}}
		\end{center}

		This is one of the more painful parts of typesetting a document, so it really takes a special kind of sadism to come up with such a game. Least to say, graduate students and researchers can be an odd bunch, and when we found this it was surprisingly addictive to compete over.