\chapter{Introduction}
In this short chapter, I would like to motivate why it is that you have to learn logic when you study computer
science.  After that, I will give a short overview of the lecture.

\section{Motivation}
Modern software systems are among the most complex systems developed by mankind.  You can get a
sense of the complexity of these systems if you look at the amount of work that is necessary to
build and maintain complex software systems.  For example, in the telecommunication industry it is
quite common that software projects require more than a thousand collaborating developers to
develop a new system.  Obviously, the failure of a project of this size is very costly.
The page
\\[0.2cm]
\hspace*{1.3cm}
\href{http://spectrum.ieee.org/static/the-staggering-impact-of-it-systems-gone-wrong}{Staggering Impact of IT Systems Gone Wrong}
\\[0.2cm]
presents a number of examples showing big software projects that have failed and have subsequently caused huge
financial losses.  These examples show that the development of complex software systems requires a high level
of precision and diligence.  Hence, the development of software needs a solid scientific
foundation.  Both \href{https://en.wikipedia.org/wiki/Mathematical_logic}{mathematical logic} and
\href{https://en.wikipedia.org/wiki/Set_theory}{set theory}
are important parts of this foundation.  Furthermore, both set theory and logic have immediate applications in
computer science.
\begin{enumerate}
\item Logic can be used to specify the \blue{interfaces} of complex systems.
\item The correctness of digital circuits can be verified using \blue{automatic theorem provers} that are based on
      propositional logic.
\item Set theory and the theory of relations is one of the foundations of \blue{relational databases}.
\end{enumerate}
It is easy to extend this enumeration.  However, besides their immediate applications,
there is another reason you have to study both logic and set theory: Without the proper use of
{\color{blue}abstractions}, complex software systems cannot be managed.  After all, nobody is able to keep
millions of lines of program code in her head.  The only way to construct and manage a software system of this
size is to introduce the right abstractions and to develop the system in layers.  Hence, the ability
to work with abstract concepts is one of the main virtues of a modern computer scientist.
Exposing students to logic and set theory trains their abilities to work with abstract concepts.

From my past teaching experience I know that many students think that a good programmer already is a
good computer scientist.  However, a good programmer need not be a scientist, while a
{\color{blue}computer \underline{scientist}}, by its very name, is a
{\color{blue}scientist}.  There is no denying that {\color{blue}mathematics} in general and
{\color{blue}logic} in particular is an important part of science, so you should master it.  Furthermore, this
part of your education is much more permanent than the knowledge of a particular programming
language.  Nobody knows which programming language will be \emph{en vogue} in 10 years from now.  In three
years, when you start your professional career, quite a lot of you will have to learn a new
programming language.  What will count then will be much more your ability to quickly grasp new
concepts rather than your skills in a particular programming language.

\section{Overview}
The first lecture in theoretical computer science creates the foundation that is needed for future lectures.
This lecture deals mostly with mathematical logic and is structured as follows.
\begin{enumerate}
\item We begin our lecture with a short introduction of set theory.  A basic understanding of set theory is
      necessary for us to formally define the semantics of both propositional logic and first order logic.
\item We proceed to introduce the programming language \textsl{Python}.

      As the concepts introduced in this lecture are quite abstract, it is beneficial to clarify the main
      ideas presented in this lectures via programs.  The programming language
      \href{http://python.org}{\textsl{Python}} supports both sets and their operations and is therefore
      suitable to implement most of the abstract ideas presented in this lecture.  According to the
      \href{http://ieee.org}{\textsc{Ieee}} (\underline{I}nstitute of \underline{E}lectrical and
      \underline{E}lectronics \underline{E}gineers),  \textsl{Python} is now the
      \href{https://spectrum.ieee.org/static/interactive-the-top-programming-languages-2018}{most popular programming language}.
      Furthermore, \textsl{Python}
      \href{https://cacm.acm.org/blogs/blog-cacm/176450-python-is-now-the-most-popular-introductory-teaching-language-at-top-u-s-universities/fulltext}{is
        now the most popular introductory teaching language at top U.S. universities}.   For these reasons I
      have decided to base these lectures on \textsl{Python}.

\item Next, we investigate the limits of computability.

      For certain problems there is no algorithm that can solve the problem algorithmically.
      For example, the question whether a given program will terminate for a given input is not
      \blue{decidable}.  This is known as the \href{https://en.wikipedia.org/wiki/Halting_problem}{halting problem}.
      We will prove the \blue{undecidability} of the halting problem in the third chapter.
\item The fourth chapter discusses \href{https://en.wikipedia.org/wiki/Propositional_calculus}{propositional logic}.

      In logic, we distinguish between  \blue{propositional logic},
      \blue{first order logic}, and \blue{higher order logic}.  Propositional logic is only
      concerned with the \blue{logical connectives}
      \\[0.2cm]
      \hspace*{1.3cm}
      ``$\neg$'', ``$\wedge$'', ``$\vee$'', ``$\rightarrow$'' und ``$\leftrightarrow$'',
      \\[0.2cm]
      while first-order logic also investigates the \blue{quantifiers}
      \\[0.2cm]
      \hspace*{1.3cm}
      ``$\forall$'' and ``$\exists$'',
      \\[0.2cm]
      where these quantifiers range over the objects of the \blue{domain of discourse}.
      Finally, in \blue{higher order logic} the quantifiers also range over functions and predicates.

      As propositional logic is easier to grasp than first-order logic, we start our investigation
      of logic with propositional logic.  Furthermore, propositional logic has the advantage of
      being \blue{decidable}:  We will present an algorithm that can check whether a propositional formula
      is universally valid.  In contrast to propositional logic, first-order logic is not decidable.

      Next, we discuss applications of propositional logic:  We will show how the \blue{8 queens problem}
      can be reduced to the question, whether a formula from propositional logic is satisfiable.  We present
      the algorithm of \blue{Davis and Putnam} that can decide the satisfiability of a propositional formula.
      This algorithm is therefore able to solve the 8 queens problem.
\item Finally, we discuss \href{https://en.wikipedia.org/wiki/First-order_logic}{first-order logic}.

      The most important concept of the last chapter will be the notion of a \blue{formal proof} in
      first order logic.  To this end, we introduce a \blue{formal proof system} that is
      \blue{complete} for first order logic.  \blue{Completeness} means that we will develop an
      algorithm that can \blue{prove} the correctness of every first-order formula that is
      universally valid.  This algorithm is the foundation of \blue{automated theorem proving}.

      As an application of theorem proving we discuss the systems \href{https://www.cs.unm.edu/~mccune/mace4/}{Prover9} and
      \href{https://www.cs.unm.edu/~mccune/mace4/}{Mace4}. \blue{Prover9} is an automated theorem prover, while
      \blue{Mace4} can be used to refute a mathematical conjecture.
\end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "logic"
%%% End:


\documentclass[main.tex]{subfiles}
\begin{document}

\subsection{Mass terms for neutrinos}

\marginpar{Monday\\ 2021-12-13}

We will discuss the difference between Weyl, Dirac and Majorana neutrinos for
the single-generation case.
Remember that under a Lorentz transformation \(\Lambda (\vec{\omega}, \vec{n}) \) they change as
%
\begin{align}
\phi_R \to \exp( i \vec{\omega} \cdot \frac{\vec{\sigma}}{2} + \vec{n} \cdot \frac{\vec{\sigma}}{2}) \phi _R \\
\phi_L \to \exp( i \vec{\omega} \cdot \frac{\vec{\sigma}}{2} - \vec{n} \cdot \frac{\vec{\sigma}}{2}) \phi _L
\,.
\end{align}

It can be shown that if \(\phi _R\) is right-handed, \(i \sigma_2 \phi_R^{*}\) is left-handed
(and similarly with \(L \leftrightarrow R\)).

We can therefore parametrize our full spinor in terms of two right-handed spinors, as
%
\begin{align}
\Psi = \left[\begin{array}{c}
\phi _R \\
\phi _L
\end{array}\right]
=
\left[\begin{array}{c}
u \\
i \sigma_2 v^{*}
\end{array}\right]
\,.
\end{align}

Recall also the projectors
%
\begin{align}
\psi_{L, R} = P_{L, R} \psi
\qquad \text{where} \qquad
P_{L, R} = \frac{1 \mp \gamma^{5}}{2}
\,.
\end{align}

We also have \(\overline{\psi} = \psi ^\dag \gamma^{0}\) and \(\psi^{c} = C(\psi ) = i \gamma^{2} \psi^{*}\).

The convention for the order of operations is as follows:
\begin{enumerate}
    \item projectors \(P_{L, R}\) act before
    \item charge conjugation \(C( \cdot)\) which acts before
    \item conjugation \(\overline{\cdot}\).
\end{enumerate}

In terms of our parametrization \(\psi (u, v)\) the charge conjugate swaps \(u\) and \(v\):
%
\begin{align}
\psi^{c} = \left[\begin{array}{c}
v \\
i \sigma_2 u^{*}
\end{array}\right]
\,.
\end{align}

The Dirac case is the one in which \(u \neq v\), the Majorana case is the one in which \(u \simeq v\).
The equality in the Majorana case can actually be relaxed including an arbitrary phase:
\(\psi = \psi^{c} e^{i \phi }\) for some fixed \(\phi \), called the \emph{Majorana phase}.

The Weyl case is the one in which \(m = 0\), meaning that \(\psi = \psi _R\) or \(\psi = \psi _L\).
The spinor just has two degrees of freedom.

The Weyl-Majorana case would then be \(\psi = \psi _R + \psi _R^{c} = \psi^{c}\).

The Dirac case is the general, massive, 4 degrees of freedom one.

A Dirac mass term looks like \(m \overline{\psi} \psi\),
a Majorana one would look like \((1/2) m \overline{\psi} \psi \).

In the Dirac case, this would read
%
\begin{align}
\overline{\psi} \psi = \overline{\psi}_L \psi_R + \overline{\psi}_R \psi _L
\,,
\end{align}
%
while in the Majorana, left-handed case we would have
%
\begin{align}
\overline{\psi} \psi = \overline{\psi}_L \psi _L^{c} + \overline{\psi}_L^{c} \psi _L
\,,
\end{align}
%
and the right-handed one is analogous.

The most general Lagrangian would then include a term
%
\begin{align}
\mathscr{L} &\ni m_D
\left(
    \overline{\psi}_L \psi _R + \overline{\psi}_R \psi _L
\right)
+
\frac{1}{2} m_L
\left(
    \overline{\psi}_L^{c} \psi _L + \overline{\psi}_L \psi _L^{c}
\right)
+
\frac{1}{2} m_R
\left(
    \overline{\psi}_R^{c} \psi _R + \overline{\psi}_R \psi _R^{c}
\right)  \\
&= \frac{1}{2} \left[\begin{array}{cc}
\overline{\psi}_L + \overline{\psi}_L^{c}, &
\overline{\psi}_R + \overline{\psi}_R^{c}
\end{array}\right]
\left[\begin{array}{cc}
m_L & m_D \\
m_D & m_R
\end{array}\right]
\left[\begin{array}{c}
\psi _L + \psi_L^{c} \\
\psi _R + \psi _R^{c}
\end{array}\right]
\,,
\end{align}
%
which we can diagonalize!
The eigenvectors will in general be linear combinations of Majorana fields.

What does the Standard Model tell us about these masses?
The \(m_D\) term is allowed, is comes from the SSB of the Yukawa terms as long as we have right-handed neutrinos.

Specifically, it will be \(m_D \sim y_\nu v / \sqrt{2}\) where \(v \approx \SI{200}{GeV}\).
In order for this to work we must have \(y_\nu \lesssim \num{e-11}\).

In the SM we cannot have Majorana mass terms \(m_L\), since they break the
electroweak symmetry.

\todo[inline]{Clarify this point}

The term \(m_R\), on the other hand (\(m_R \overline{\psi}_R \psi _R^{c} + \text{h. c.}\)), is allowed by the SM gauge group, but the \(m_R\) term is a new mass scale of the theory.

The mass matrix can be thought to look like
%
\begin{align}
\left[\begin{array}{cc}
m_L & m_D \\
m_D & m_R
\end{array}\right]
=
\left[\begin{array}{cc}
0 & \sim v \\
\sim v &  \sim \Lambda
\end{array}\right]
\,,
\end{align}
%
where \(\Lambda \) is a scale from Beyond-the-Standard-Model physics.

These ideas came out when people were considering very large gauge groups.

The diagonalization of that matrix yields a light neutrino \(\nu \sim \nu _L + \nu _L^{c}\) with mass of the order \(\order{v^2 / \Lambda } \ll v\), as well as a massive neutrino \(\nu \sim \nu _R + \nu _R^{c}\) with mass of the order \(\order{\Lambda }\).

This is the see-saw mechanism.

A technical detail: one mass comes out negative, but if \(\psi \) has a negative mass \(\gamma^{5} \psi \) has a positive mass.

What happens if we have more than one neutrino generation?

In the SM, we have determined that the flavor eigenstates are linear combinations of the mass eigenstates with a unitary mixing matrix \(U\).

We can also have a number \(N_s\) of sterile neutrino states which do not couple to the \(W^{\pm}\) and \(Z\) bosons.
The number \(N_s\) is not limited to the right-handed components of the three neutrinos, there could be more. The full mixing matrix will then be
%
\begin{align}
\left[\begin{array}{cc}
3 \times 3 & 3 \times N_s \\
N_s \times 3 & N_s \times N_s
\end{array}\right]
\sim
\left[\begin{array}{cc}
\sim U _{\text{PMNS}} & \sim \order{v / \Lambda } \\
\sim \order{v / \Lambda } & U_{N \times N}
\end{array}\right]
\,.
\end{align}

A phenomenologically motivated situation is already the \(N_s = 1\) one,
with a fourth sterile neutrino with mass \(\sim \SI{1}{eV}\).
At present, though, the evidence for a sterile neutrino is very controversial.

The process of leptogenesis, by some process which satisfies Sakharov's conditions, is relevant for this discussion.

We need both C and CP violation.
If we can find CP violation at low energy scales, it would be a good indication!

CP violation in the quark sector, \(V _{\text{CKM}} = V^{*} _{\text{CKM}}\), cannot explain matter-antimatter asymmetry.

It is possible that a CP violating mechanism in the weak sector may give an imprint in the primordial GW background.

If the CP violating decay of heavy neutrinos is to blame, there might be a connection between the low- and high-energy manifestations of CP violation.

\subsubsection{The effect of neutrino masses and mixing in propagation}

We have
%
\begin{align}
\left[\begin{array}{c}
\nu _e \\
\nu _\mu  \\
\nu _\tau
\end{array}\right]
=
U _{\text{PMNS}}
\left[\begin{array}{c}
\nu_1   \\
\nu_2  \\
\nu_3
\end{array}\right]
\,,
\end{align}
%
where \(U _{\text{PMNS}} U _{\text{PMNS}} ^\dag = 1\), while in general \(U \neq U ^\dag\).
We will assume \(E \gg m \).

The convention used for this matrix is as follows, since it can be shown that it contains three angles and a CP-violating phase:
%
\begin{align}
\left[\begin{array}{ccc}
U_{e1} & U_{e2} & U_{e3} \\
U_{\mu 1} & U_{\mu 2} & U_{\mu 3} \\
U_{\tau 1} & U_{\tau 2} & U_{\tau 3}
\end{array}\right]
=
\underbrace{\left[\begin{array}{ccc}
1 & 0 & 0 \\
0 & c_{23} & s_{23} \\
0 & -s_{23} & c_{23}
\end{array}\right]}_{R(\theta_{23})}
\underbrace{\left[\begin{array}{ccc}
c_{13}  & 0 & s_{13} e^{-i \delta } \\
0 & 1 & 0 \\
s_{13} e^{i \delta } & 0 & c_{13}
\end{array}\right]}_{R(\theta_{13})}
\underbrace{\left[\begin{array}{ccc}
c_{12}  & s_{12}  & 0 \\
-s_{12}  & c_{12}  & 0 \\
0 & 0 & 1
\end{array}\right]}_{R(\theta_{12})}
\underbrace{\left[\begin{array}{ccc}
1 & 0 & 0 \\
0 & e^{i \psi ' } & 0 \\
0 & 0 & e^{i \psi ''}
\end{array}\right]}_{\text{Majorana}}
\,.
\end{align}

If we have Majorana neutrinos the spinor is equal to its conjugate up to a phase, but only two of the phases are physically meaningful.
However, these play no role in oscillations.

We have measured these angles: for \(\theta_{23}\) there is (almost) maximal mixing, \(\theta_{23} \approx \pi /4\) or \(s^2_{23} \approx \num{.5}\).
This is called \emph{octant ambiguity}, since we do not know the precise value.

The value of \(s^2_{12}\) is approximately \(\num{.3}\).

The value of \(s^2_{13}\) is approximately \(\num{.02}\).

What about the mass spectrum?
Well, the energy is approximately
%
\begin{align}
E = \sqrt{p^2 + m^2}
\approx p + \frac{m^2}{2 p}
\approx p + \frac{m^2}{2E}
\,.
\end{align}

The masses are numbered \(1\), \(2\) and \(3\);
by convention we always take \(m^2_2 > m^2_1\).

We have \(\delta m^2 = m_2^2 - m_1^2 \approx \SI{7.5e-5}{eV^2}\), while \(\abs{\Delta m^2} = \abs{m_3^2 - m_1^2} \approx \SI{2.5e-3}{eV^2}\).
We do not know the sign of \(\Delta m^2\)!

As long as we deal with three neutrino masses and mixing, we have 5 known quantities and 5 unknowns: we know
\begin{enumerate}
    \item \(\delta m^2\);
    \item \(\abs{\Delta m^2}\);
    \item \(s^2_{23}\);
    \item \(s^2_{12}\);
    \item \(s^2_{13}\),
\end{enumerate}
%
while we do not know some parameters which are hard or impossible to measure with oscillations:
\begin{enumerate}
    \item the value of the CP-violating \(\delta \) --- but we do have some weak \(\sim 2 \sigma \) indications that it might be nonzero;
    \item the sign of \(\Delta m^2\);
    \item whether \(\theta_{13} \lessgtr \pi /4 \);
    \item the absolute mass scale (we only have some bounds);
    \item whether neutrinos are Dirac or Majorana.
\end{enumerate}

There is a search for right-handed neutrinos at many scales, even at LHC.

Tomorrow we will start studying neutrino oscillations, dealing with wavefunctions as opposed to spinors, approximating time with position \(t \sim x\); and using the fact that \(E \approx p + m^2 / 2E\).

The Schrödinger equation will read
%
\begin{align}
i \dv{}{t} \left[\begin{array}{c}
\nu_1  \\
\nu_2  \\
\nu_3
\end{array}\right]
= H
\left[\begin{array}{c}
\nu_1  \\
\nu_2  \\
\nu_3
\end{array}\right]
\,,
\end{align}
%
where, in a vacuum, \(H\) is diagonal, with the three energies of the neutrinos.
There will be corrections in matter.

We will have
%
\begin{align}
H = p + \frac{1}{2E} \underbrace{\left[\begin{array}{ccc}
m_1^2  & 0 & 0 \\
0 & m_2^2 & 0 \\
0 & 0 & m_3^2
\end{array}\right]}_{M^2}
\,.
\end{align}

\end{document}


\documentclass[onecolumn, conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}

\title{Tensor applications in neural networks\\
% {\footnotesize \textsuperscript{*}Note: Sub-titles are not captured in Xplore and
% should not be used}
% \thanks{Identify applicable funding agency here. If none, delete this.}
}

\author{
    \IEEEauthorblockN{Junjie He}
\IEEEauthorblockA{\textit{School of Information Science and Technology} \\
\textit{ShanghaiTech University}\\
Shanghai, China \\
hejj1@shanghaitech.edu.cn}
\and
\IEEEauthorblockN{Jiaqiong Zhang}
\IEEEauthorblockA{\textit{School of Information Science and Technology} \\
\textit{ShanghaiTech University}\\
Shanghai, China \\
zhangjq@shanghaitech.edu.cn}
% \IEEEauthorblockN{3\textsuperscript{rd} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{4\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{5\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{6\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
}

\maketitle

\begin{abstract}
Neural networks has obtain the state of the art performance in many problems, while it will spend much time to train and inference the whole neural network. Due to weights in neural networks can be seen as tensors, which implies that we can use tensor technique to accelerate neural networks.
Recently, tensor techniques also show its performance in Graph Neural Network (GNN), we also extend our experiments.
\end{abstract}

\begin{IEEEkeywords}
Tensor, Tensor-Train decomposition, neural network, Graph neural network
\end{IEEEkeywords}

\section{Introduction}
\par Main topic for our project are tensor methods and neural network. Deep neural networks currently demonstrate state-of-the-art performance in several domains. Such as computer vision, speech recognition, text processing, etc.
These advances have become possible because of algorithmic advances, large amounts of available data, and modern hardware. For example, convolutional neural networks (CNNs) [1][2]show by a large margin superior performance on the task of image classification.
These models have thousands of nodes and millions of learnable parameters and are trained using millions of images[3] on powerful Graphics Processing Units (GPUs).The necessity of expensive hardware and long processing time are the factors that complicate the application of such models on conventional desktops and portable devices.
\par In tradition neural network, this layer has a linear transformation of a high dimension input signal to a high dimension output signal. For example, the data set CIFAR10 widely used in deep learning course is a collection of pictures. When it used as input signal into neural network, the pictures in it will be divided 32$\times$32$\times$3 pixels. particularly, 32$\times$32 means a picture divided into 32$\times$32 pixel blocks, and 3 means three channels(RGB).Then, the input signal will be reshaped a 32$\times$32$\times$3 dimensional vector .Obviously, such an operation will largely increase the dimension of input signal, and leads to  further complexity of the calculations.
The data set CIFAR10[5] is already a very simple data set in the field of deep learning. However, in the convolutional neural network model used in practical application the dimensions of the input and output signals of the fully-connected layers are of the order of thousands, bringing the number of parameters of the fully-connected layers up to millions. This is undoubtedly a very demanding requirement for hardware facilities.
\par Consequently, a large number of works tried to reduce both hardware requirements (e. g. memory demands) and running times. To solve this problem, We consider the most frequently used layer of neural network: fully-connected layer. We use a compact tensor train data set to represent the matrix of the fully-connected layers using few parameters while keeping enough flexibility to perform signal transformations[6].And, the layer transformed should be compatible with the existing training algorithms for neural network, because all the derivatives required by the back propagation algorithm[4] can be computed using the properties of Tensor train set.
Tensors are natural multidimensional generalizations of matrices and have attracted tremendous interest in recent years. Multilinear algebra, tensor analysis, and the theory of tensor approximations play increasingly important roles in computational mathematics and numerical analysis[7][8][9][10]. An efficient representation of a tensor (by tensor we mean only an array with d indices) by a small number of parameters may give us an opportunity and ability to work with d-dimensional problems, with d being as high as 10, 100, 1000 or even one million. Problems of such sizes cannot be handled by standard numerical methods due to the curse of dimensionality, since everything (memory, amount of operations) grows exponentially in d. So, Tensor train decomposition will a effective way to solve this problem.
\par The tensor operation can be applied in addition to the traditional CNN, it can also applied in graphs neural networks(GNNs). Graphs are popular data structures used to effectively represent interactions and structural relationships between entities in structured data domains. Inspired by the success of deep neural networks for learning representations in the image and language domains, recently, application of neural networks for graph representation learning has attracted much interest. A number of graph neural network architectures have been explored in the contemporary literature for a variety of graph related tasks and applications. Tensor based approaches have been shown to perform well in many image and video processing applications. And, recently, a new tensor framework called the tensor M-product framework was proposed that extends matrix based theory to high-dimensional architectures.
We will apply our method to popular network architectures proposed for data set CIFAR10.
\par We will experimentally use the networks with tradition fully-connected layer and the tensor fully-connected layer to train a neural network model. Then, we will compare the performance of two models.

\section{Preliminary ideas}

\par

\par

\section{Overview of existing work}
% \section{Formulation}
\subsection{TT-format}

In various fields, low-rank approximation was applied to reduce the computation cost and memory usage. In \cite{b11}, they generalize the idea of low-rank. The authors do not find low-rank approximation of weight matrix in fully-connected layers, they treat the matrix as multidimensional tensor and employ Tensor Train decomposition \cite{b6} to accelerate the computation.
Usally, wider neural network can achieve better performance than narrow neural network. But wide neural networks imply large dense matrix, amount of computation resources are used in per step when training neural networks. By using Tensor Train decomposition for weight matrix, wide neural network can be developed for applications with moderate computation cost and memory usage.
\cite{gilboa2013scaling} shows that wide and shallow neural networks has competitive performance with the state-of-art deep neural networks by traing a shallow network oin the outputs of a trained deep neural network.
They report the improvement of performance with the increase of the layer size and used up to 30 000 hidden units while restricting the matrix rank of the weight matrix in order to be able to keep and to update it during the training. Restricting the TT-ranks of the weight matrix (in contrast to the matrix rank) allows to use much wider layers potentially leading to the greater expressive power of the model.

\par CP-decomposition algorithm was applied to compress convolution kernel in CNNs. And they also using properties of CP-decomposition to speed up the inference time.
tp speed up computation of matrix-by-vector, properties of the Kronecker product of matrices was exploited. These matrices have the same structure as TT-matrices with unit TT-ranks. We can generalize this idea to formulate the weight matrix with TT-matrices with unit TT-ranks.
The Tucker format and the canonical format will meet the curse of dimensionality, TT-format is immune to the cues of dimensionality and its algorithm are robust.
\par A $d$-dimensional array (tensor) $ \mathcal{A}$ is said to be TT-format if for each dimension $k=1,...,d$ and for each possible value of the $k$-th  dimension index $j_k=1,...,n_k$ there exists a matrix $\mathbf{G}_k[j_k]$ such that all elements of $\mathcal{A}$ can be computed as the following matrix product:
\begin{equation}
    \label{eq1}
    \mathcal{A}(j_1,...,j_d)=\mathbf{G}_1[j_1]\mathbf{G}_2[j_2]\cdots\mathbf{G}_d[j_d]
\end{equation}
All the matrices $\mathbf{G}_k[j_k]$ related to the same dimension $k$ are restricted to be of the same size $r_{k-1}\times r_k$.
The values $r_0$ and $r_d$ equal to 1 in order to keep the matrix product (\ref{eq1}) of size $1\times 1$.
In what follows we refer to the representation of a tensor in the TT-format as the TT-representation or d the TT-decomposition.
The sequence $\{r_k \}^d_{k=0}$ is referred to as the TT-ranks of the TT-representation of $\mathbf{A}$
(or the ranks for short), its maximum – as the maximal TT-rank of the TT-representation $n$ of $\mathcal{A}: r = \max_{k=0,...,d} r_k$ .
The collections of the matrices $(\mathbf{G}_k [j_k ])_{j_k}^{n_k} =1$ corresponding to the same dimension (technically, 3-dimensional arrays $\mathcal{G}_k$ ) are called the cores.

We use the symbols $\mathbf{G}_k[j_k](\alpha_{k-1},\alpha_k)$ to denote the element of the matrix $\mathbf{G}_k[j_k]$ in the position $(\alpha_{k-1},\alpha_k)$,

where $\alpha_{k-1}=1,...,r_{k-1},\alpha_k=1,...,r_k$. Equation (\ref{eq1}) can be equivalently rewritten as the sum of the products of the elements of the cores:
\begin{equation}
    \label{eq2}
    \mathcal{A}(j_1,...,j_d)=\sum_{\alpha_0,...,\alpha_d} \mathbf{G}_1[j_1](\alpha_{0},\alpha_1)\cdots \mathbf{G}_d[j_d](\alpha_{d-1},\alpha_d)
\end{equation}
The representation of a tensor $\mathcal{A}$ via the explicit enumeration of all its elements requires to store
$\prod_{k=1}^d n_k $ numbers compared with $\sum_{k=1}^d n_kr_{k-1}r_k$ numbers if the tensor is stored in the TT-format.
Thus, the TT-format is very efﬁcient in terms of memory if the ranks are small.

An attractive property of the TT-decomposition is the ability to efﬁciently perform several types of operations on tensors if they are in the TT-format: basic linear algebra operations, such as the addition of a constant and the multiplication by a constant, the summation and the entrywise product of tensors (the results of these operations are tensors in the TT-format generally with the increased ranks); computation of global characteristics of a tensor, such as the sum of all elements and the Frobenius norm.
See [17] for a detailed description of all the supported operations.

\subsection{Training with TT-formate}
Neural networks are usually trained with the stochastic gradient descent algorithm where the gradient is computed using the back-propagation procedure.
Back-propagation allows to compute the gradient of a loss-function $L$ with respect to all the parameters of the network.
The method starts with the computation of the gradient of $L$ w.r.t. the output of the last layer and proceeds sequentially through the layers in the reversed order while computing the gradient w.r.t. the parameters and the input of the layer making use of the gradients computed earlier.
Applied to tensorizing fully-connected layers  the back-propagation method computes the gradients w.r.t. the input $\bf{x}$ and the parameters $\bf{W}$ and $\bf{b}$ given the gradients $\frac{\partial L}{\partial {\bf y}}$ w.r.t to the output ${\bf y}$:
\begin{equation}
    \label{gradient}
    \frac{\partial L}{\partial {\bf x}}={\bf W}^T \frac{\partial L}{\partial {\bf y}},
    \frac{\partial L}{\partial {\bf W}}={\bf W}^T \frac{\partial L}{\partial {\bf y}}{\bf x}^T,
    \frac{\partial L}{\partial {\bf b}}={\bf W}^T \frac{\partial L}{\partial {\bf y}}
\end{equation}

In what follows we derive the gradients required to use the back-propagation algorithm with the tensorizing layers.
To compute the gradient of the loss function w.r.t. the bias vector ${\bf b}$ and w.r.t. the input vector ${\bf x}$ one can use equations (\ref{gradient}).
The latter can be applied using the matrix-by-vector product (where the matrix is in the TT-format) with the complexity of $\mathcal{O}(dr^2n\max\{m,n\}^d)=\mathcal{O}(dr^2n\max\{M,N\})$.
To perform a step of stochastic gradient descent, we can use traditional back-propagation in computational graph to compute gradient of loss function w.r.t the weight matrix $\mathbf{W}$, then we convert the gradient matrix into the TT-format using TT-SVD algorithm. Another way to learn the TensorNet parameters is to
compute gradient of loss function w.r.t the cores of the TT-representations of $\mathbf{W}$.

\subsection{TT-SVD for High-dimensional matrices}
For high-dimensional matrices, the TT-SVD algorithm will meet curse of dimensionality, i.e., computation cost will increase quickly such as exponentially. Then we have difficulty to employ TT-format in neural networks. To
A Randomized Tensor Train Singular Value
Decomposition
Each of the existing TT decomposition algorithms,
including the TT-SVD and randomized TT-SVD, is successful in the field, but
neither can both accurately and efficiently decompose large-scale sparse tensors. \cite{b13} proposes a new quasi-best fast TT
decomposition algorithm for large-scale sparse tensors with proven correctness
and the upper bound of its complexity is derived. In numerical experiments,
authors verify that the proposed algorithm can decompose sparse tensors faster than
the TT-SVD, and have more speed, precision and versatility than randomized
TT-SVD\cite{b14}, and it can be used to decomposes arbitrary high-dimensional tensor
without losing efficiency when the number of non-zero elements is limited.
Faster TT-SVD algorithm can be integrated into tensorizing neural networks, and it should be more efficiently to solve the problem in large scale.

\subsection{Dynamic graph embedding}
Our approach is inspired by the first order GCN by Kipf and Welling (2016) for static graphs, owed to its simplicity and effectiveness. For a graph with adjacency matrix ${\bf A}$ and feature matrix ${\bf X}$, a
GCN layer takes the form $Y = \sigma({\bf AXW})$, where $\tilde{\bf A} = \tilde{\bf D}^{-1/2}({\bf A} + {\bf I})\tilde{\bf D}^{-1/2}$, $\tilde{\bf D}$ is diagonal with $\tilde{\bf D}_{ii} = 1 + \sum_j {\bf A}_{ij}$ , ${\bf I}$ is the matrix identity, ${\bf W}$ is a matrix to be learned when
training the NN, and $\sigma$ is an activation function, e.g., ReLU.
Our approach translates this to a tensor model by utilizing the M-product framework. We first introduce a tensor activation function $\hat{\sigma}$ which operates in the transformed space.
Let $\mathcal{A} \in \mathbb{R}^{I\times J\times T}$ be a tensor and $\sigma$ an elementwise activation function. We define $$\hat{\sigma}(\mathcal{A}) = \sigma(\mathcal{A}\times {\bf M})\times {\bf M}^{-1}$$.

\section{Criticism of the existing work}

\section{New Contribution}

\section{Numerical results}
In all experiments we will use MATLAB extension\footnote{https://github.com/Bihaqo/TensorNet} of the MatConvNet framework\footnote{http://www.vlfeat.org/matconvnet}. For the operations related to the TT-format we use the TT-Toolbox\footnote{https://github.com/oseledets/TT-Toolbox} implemented in MATLAB as well.
To show the properties of the TT-layer and compare different strategies for setting its parameters: dimensions of the tensors representing the input/output of the layer and the TT-ranks of the compressed weight matrix.
We run the experiment on the MNIST dataset for the task of handwritten-digit recognition.
As a baseline we use a neural network with two fullyconnected layers (1024 hidden units) and rectiﬁed linear unit (ReLU) and compute error on the test set.
For more reshaping options we resize the original 28 × 28 images to 32 × 32.

Futhermore, we will train several networks differing in the parameters of the single TT-layer. The networks contain the following layers: the TT-layer with weight matrix of size 1024×1024, ReLU, the fully-connected layer with the weight matrix of size 1024 × 10.
We test different ways of reshaping the input/output tensors and try different ranks of the TT-layer.
As a simple compression baseline in the place of the TT-layer we use the fully-connected layer such that the rank of the weight matrix is bounded (implemented as follows: the two consecutive fully-connected layers with weight matrices of sizes 1024 × r and r ×1024, where r controls the matrix rank and the compression factor).

\subsection{CIFAR 10}
CIFAR-10 dataset [12] consists of 32 × 32 3-channel images assigned to 10 different classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The dataset contains 50000 train and
10000 test images. Following [10] we preprocess the images by subtracting the mean and performing global contrast normalization and ZCA whitening.
As a baseline we use the CIFAR-10 Quick [22] CNN, which consists of convolutional, pooling and
non-linearity layers followed by two fully-connected layers of sizes 1024 × 64 and 64 × 10. We fix
the convolutional part of the network and substitute the fully-connected part by a 1024×N TT-layer followed by ReLU and by a N × 10 fully-connected layer.
\subsection{Time-varing graph}
Here, we present results for edge classification on four datasets1: The Bitcoin Alpha and OTC transaction datasets (Kumar et al., 2016), the Reddit body hyperlink dataset (Kumar et al., 2018),
and a chess results dataset (Kunegis, 2013).
The bitcoin datasets consist of transaction histories for users on two different platforms. Each node is a user, and each directed edge indicates a transaction and is labeled with an integer between -10 and 10 which indicates the senders trust for the receiver.
We convert these labels to two classes: positive (trustworthy) and negative (untrustworthy). The Reddit dataset is build from hyperlinks from one subreddit to another.
Each node represents a subreddit, and each directed edge is an interaction which is labeled with -1 for a hostile interaction or +1 for a friendly interaction.
We only consider those subreddits which have a total of 20 interactions or more. In the chess dataset, each node is a player, and each directed edge represents a match with the source node being the white player and the target node being the black player.
Each edge is labeled -1 for a black victory, 0 for a draw, and +1 for a white victory.
\section{Conclusion}

% \section{Prepare Your Paper Before Styling}
% Before you begin to format your paper, first write and save the content as a
% separate text file. Complete all content and organizational editing before
% formatting. Please note sections \ref{AA}--\ref{SCM} below for more information on
% proofreading, spelling and grammar.

% Keep your text and graphic files separate until after the text has been
% formatted and styled. Do not number text heads---{\LaTeX} will do that
% for you.

% \subsection{Abbreviations and Acronyms}\label{AA}
% Define abbreviations and acronyms the first time they are used in the text,
% even after they have been defined in the abstract. Abbreviations such as
% IEEE, SI, MKS, CGS, ac, dc, and rms do not have to be defined. Do not use
% abbreviations in the title or heads unless they are unavoidable.

% \subsection{Units}
% \begin{itemize}
% \item Use either SI (MKS) or CGS as primary units. (SI units are encouraged.) English units may be used as secondary units (in parentheses). An exception would be the use of English units as identifiers in trade, such as ``3.5-inch disk drive''.
% \item Avoid combining SI and CGS units, such as current in amperes and magnetic field in oersteds. This often leads to confusion because equations do not balance dimensionally. If you must use mixed units, clearly state the units for each quantity that you use in an equation.
% \item Do not mix complete spellings and abbreviations of units: ``Wb/m\textsuperscript{2}'' or ``webers per square meter'', not ``webers/m\textsuperscript{2}''. Spell out units when they appear in text: ``. . . a few henries'', not ``. . . a few H''.
% \item Use a zero before decimal points: ``0.25'', not ``.25''. Use ``cm\textsuperscript{3}'', not ``cc''.)
% \end{itemize}

% \subsection{Equations}
% Number equations consecutively. To make your
% equations more compact, you may use the solidus (~/~), the exp function, or
% appropriate exponents. Italicize Roman symbols for quantities and variables,
% but not Greek symbols. Use a long dash rather than a hyphen for a minus
% sign. Punctuate equations with commas or periods when they are part of a
% sentence, as in:
% \begin{equation}
% a+b=\gamma\label{eq}
% \end{equation}

% Be sure that the
% symbols in your equation have been defined before or immediately following
% the equation. Use ``\eqref{eq}'', not ``Eq.~\eqref{eq}'' or ``equation \eqref{eq}'', except at
% the beginning of a sentence: ``Equation \eqref{eq} is . . .''

% \subsection{\LaTeX-Specific Advice}

% Please use ``soft'' (e.g., \verb|\eqref{Eq}|) cross references instead
% of ``hard'' references (e.g., \verb|(1)|). That will make it possible
% to combine sections, add equations, or change the order of figures or
% citations without having to go through the file line by line.

% Please don't use the \verb|{eqnarray}| equation environment. Use
% \verb|{align}| or \verb|{IEEEeqnarray}| instead. The \verb|{eqnarray}|
% environment leaves unsightly spaces around relation symbols.

% Please note that the \verb|{subequations}| environment in {\LaTeX}
% will increment the main equation counter even when there are no
% equation numbers displayed. If you forget that, you might write an
% article in which the equation numbers skip from (17) to (20), causing
% the copy editors to wonder if you've discovered a new method of
% counting.

% {\BibTeX} does not work by magic. It doesn't get the bibliographic
% data from thin air but from .bib files. If you use {\BibTeX} to produce a
% bibliography you must send the .bib files.

% {\LaTeX} can't read your mind. If you assign the same label to a
% subsubsection and a table, you might find that Table I has been cross
% referenced as Table IV-B3.

% {\LaTeX} does not have precognitive abilities. If you put a
% \verb|\label| command before the command that updates the counter it's
% supposed to be using, the label will pick up the last counter to be
% cross referenced instead. In particular, a \verb|\label| command
% should not go before the caption of a figure or a table.

% Do not use \verb|\nonumber| inside the \verb|{array}| environment. It
% will not stop equation numbers inside \verb|{array}| (there won't be
% any anyway) and it might stop a wanted equation number in the
% surrounding equation.

% \subsection{Some Common Mistakes}\label{SCM}
% \begin{itemize}
% \item The word ``data'' is plural, not singular.
% \item The subscript for the permeability of vacuum $\mu_{0}$, and other common scientific constants, is zero with subscript formatting, not a lowercase letter ``o''.
% \item In American English, commas, semicolons, periods, question and exclamation marks are located within quotation marks only when a complete thought or name is cited, such as a title or full quotation. When quotation marks are used, instead of a bold or italic typeface, to highlight a word or phrase, punctuation should appear outside of the quotation marks. A parenthetical phrase or statement at the end of a sentence is punctuated outside of the closing parenthesis (like this). (A parenthetical sentence is punctuated within the parentheses.)
% \item A graph within a graph is an ``inset'', not an ``insert''. The word alternatively is preferred to the word ``alternately'' (unless you really mean something that alternates).
% \item Do not use the word ``essentially'' to mean ``approximately'' or ``effectively''.
% \item In your paper title, if the words ``that uses'' can accurately replace the word ``using'', capitalize the ``u''; if not, keep using lower-cased.
% \item Be aware of the different meanings of the homophones ``affect'' and ``effect'', ``complement'' and ``compliment'', ``discreet'' and ``discrete'', ``principal'' and ``principle''.
% \item Do not confuse ``imply'' and ``infer''.
% \item The prefix ``non'' is not a word; it should be joined to the word it modifies, usually without a hyphen.
% \item There is no period after the ``et'' in the Latin abbreviation ``et al.''.
% \item The abbreviation ``i.e.'' means ``that is'', and the abbreviation ``e.g.'' means ``for example''.
% \end{itemize}
% An excellent style manual for science writers is \cite{b7}.

% \subsection{Authors and Affiliations}
% \textbf{The class file is designed for, but not limited to, six authors.} A
% minimum of one author is required for all conference articles. Author names
% should be listed starting from left to right and then moving down to the
% next line. This is the author sequence that will be used in future citations
% and by indexing services. Names should not be listed in columns nor group by
% affiliation. Please keep your affiliations as succinct as possible (for
% example, do not differentiate among departments of the same organization).

% \subsection{Identify the Headings}
% Headings, or heads, are organizational devices that guide the reader through
% your paper. There are two types: component heads and text heads.

% Component heads identify the different components of your paper and are not
% topically subordinate to each other. Examples include Acknowledgments and
% References and, for these, the correct style to use is ``Heading 5''. Use
% ``figure caption'' for your Figure captions, and ``table head'' for your
% table title. Run-in heads, such as ``Abstract'', will require you to apply a
% style (in this case, italic) in addition to the style provided by the drop
% down menu to differentiate the head from the text.

% Text heads organize the topics on a relational, hierarchical basis. For
% example, the paper title is the primary text head because all subsequent
% material relates and elaborates on this one topic. If there are two or more
% sub-topics, the next level head (uppercase Roman numerals) should be used
% and, conversely, if there are not at least two sub-topics, then no subheads
% should be introduced.

% \subsection{Figures and Tables}
% \paragraph{Positioning Figures and Tables} Place figures and tables at the top and
% bottom of columns. Avoid placing them in the middle of columns. Large
% figures and tables may span across both columns. Figure captions should be
% below the figures; table heads should appear above the tables. Insert
% figures and tables after they are cited in the text. Use the abbreviation
% ``Fig.~\ref{fig}'', even at the beginning of a sentence.

% \begin{table}[htbp]
% \caption{Table Type Styles}
% \begin{center}
% \begin{tabular}{|c|c|c|c|}
% \hline
% \textbf{Table}&\multicolumn{3}{|c|}{\textbf{Table Column Head}} \\
% \cline{2-4}
% \textbf{Head} & \textbf{\textit{Table column subhead}}& \textbf{\textit{Subhead}}& \textbf{\textit{Subhead}} \\
% \hline
% copy& More table copy$^{\mathrm{a}}$& &  \\
% \hline
% \multicolumn{4}{l}{$^{\mathrm{a}}$Sample of a Table footnote.}
% \end{tabular}
% \label{tab1}
% \end{center}
% \end{table}

% \begin{figure}[htbp]
% \centerline{\includegraphics{fig1.png}}
% \caption{Example of a figure caption.}
% \label{fig}
% \end{figure}

% Figure Labels: Use 8 point Times New Roman for Figure labels. Use words
% rather than symbols or abbreviations when writing Figure axis labels to
% avoid confusing the reader. As an example, write the quantity
% ``Magnetization'', or ``Magnetization, M'', not just ``M''. If including
% units in the label, present them within parentheses. Do not label axes only
% with units. In the example, write ``Magnetization (A/m)'' or ``Magnetization
% \{A[m(1)]\}'', not just ``A/m''. Do not label axes with a ratio of
% quantities and units. For example, write ``Temperature (K)'', not
% ``Temperature/K''.

% \section*{Acknowledgment}

% The preferred spelling of the word ``acknowledgment'' in America is without
% an ``e'' after the ``g''. Avoid the stilted expression ``one of us (R. B.
% G.) thanks $\ldots$''. Instead, try ``R. B. G. thanks$\ldots$''. Put sponsor
% acknowledgments in the unnumbered footnote on the first page.

% \section*{References}

% Please number citations consecutively within brackets \cite{b1}. The
% sentence punctuation follows the bracket \cite{b2}. Refer simply to the reference
% number, as in \cite{b3}---do not use ``Ref. \cite{b3}'' or ``reference \cite{b3}'' except at
% the beginning of a sentence: ``Reference \cite{b3} was the first $\ldots$''

% Number footnotes separately in superscripts. Place the actual footnote at
% the bottom of the column in which it was cited. Do not put footnotes in the
% abstract or reference list. Use letters for table footnotes.

% Unless there are six authors or more give all authors' names; do not use
% ``et al.''. Papers that have not been published, even if they have been
% submitted for publication, should be cited as ``unpublished'' \cite{b4}. Papers
% that have been accepted for publication should be cited as ``in press'' \cite{b5}.
% Capitalize only the first word in a paper title, except for proper nouns and
% element symbols.

% For papers published in translation journals, please give the English
% citation first, followed by the original foreign-language citation \cite{b6}.
\bibliography{references}
% \begin{thebibliography}{00}
%     \bibitem{b1} A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 (NIPS), 2012, pp. 1097–1105.
%  \bibitem{b2}K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”in International Conference on Learning Representations (ICLR), 2015.
%  \bibitem{b3}O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision (IJCV), 2015.
%  \bibitem{b4}D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,”Nature, vol. 323, no. 6088, pp. 533–536, 1986.
%  \bibitem{b5}A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, Computer Science Department, University of Toronto, 2009.
%  \bibitem{b6}I. V. Oseledets, “Tensor-Train decomposition,” SIAM J. Scientific Computing, vol. 33, no. 5, pp. 2295–2317, 2011.
%  \bibitem{b7}L. de Lathauwer, B. de Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1253–1278.
%  \bibitem{b8}L. de Lathauwer, B. de Moor, and J. Vandewalle, On best rank-1 and rank-(R1, R2,...,RN ) approximation of high-order tensors, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1324–1342.
%  \bibitem{b9}R. Bro, PARAFAC: Tutorial and applications, Chemometrics Intell. Lab. Syst., 38 (1997), pp. 149–171.
%  \bibitem{b10}L. Grasedyck, Existence and computation of low Kronecker-rank approximations for large systems in tensor product structure, Computing, 72 (2004), pp. 247–265.
%  \bibitem{b11}Oseledets, and V. I. . "Tensor-Train Decomposition." Siam Journal on entific Computing 33.5(2011):2295-2317.
% \end{thebibliography}
% \vspace{12pt}
% \color{red}
% IEEE conference templates contain guidance text for composing and formatting conference papers. Please ensure that all template text is removed from your conference paper prior to submission to the conference. Failure to remove the template text from your paper may result in your paper not being published.

\end{document}


\section{Experimental Results}
\label{sec:results}

In this section, we discuss the performance results of several experiments on
a multicore Intel Xeon workstation and a Blue Gene/P (both at Argonne). The
workstation has dual quad-core E5462 Xeon processors (8 cores total) running
at 2.8 GHz (1600 MHz FSB) with 32 KB L1 cache, 12 MB of L2 cache (6 MB shared
per core pair), and 2 GB of DDR2 FBDIMM RAM, running Linux kernel version
2.6.25 (x86-64). Each compute node of the Blue Gene/P is equipped with four
850 MHz IBM PowerPC 450 processors with a dual floating-point unit and 2 GB
total memory per node, private L1 (32 KB) and L2 (4 MB) caches, a shared L3
cache (8 MB), and running a proprietary lightweight operating system. We used
version 10.1 of the Intel and version 9.0 of the IBM XL C/C++ V9.0 compilers
on the Xeon and Blue Gene/P, respectively. Because of space considerations,
here we discuss a limited number of computational kernels; more performance
data for different computations is available at the Orio project
website~\cite{OrioURL}.

\subsection{Sequence of Linear Algebra Operations}
\label{sec:axpy4-results}

In this experiment, we tuned the performance of the AXPY-4 operation (see
Figure~\ref{fig:orio-example}) on a single node of the Blue Gene/P machine
using the IBM xlc compiler.  We measured the performance for two scenarios:
using a single core per node and using all four cores. The results are shown
in Figures~\ref{fig:axpy4-bgp-seq} and~\ref{fig:axpy4-bgp-par},
respectively. The single-core scenario was compiled with the following options:
\texttt{-O3 -qstrict -qarch=450d -qtune=450 -qhot -qsmp=noauto};  the
multicore scenario differs in the use of \texttt{-qsmp=auto} for the non-Orio
versions. The parallel Orio version contains OpenMP parallelization
directives in the generated code, thus necessitating the use of the
\texttt{-qsmp=omp:noauto} compiler option. Included are performance numbers for four
code variants: a simple loop implementation without any library calls
(labeled ``Compiler-optimized''), two BLAS-based implementations that use
Goto BLAS~\cite{Goto:2006fk} and the ESSL~\cite{ESSL} libraries, and the
Orio-tuned version.

\begin{figure}%[b]
\begin{tabular}{c}
\begin{minipage}[t]{.43\textwidth}
\centering
\vspace{0.2in}
~~~~~~~~\includegraphics[width=1.0\textwidth]{figures/spmv.eps}
\end{minipage}
\\
(a)
\\
\begin{minipage}[t]{.3\textwidth}
\scriptsize
\begin{verbatim}

for (i=0; i<num_rows; i++)
  for (j=Aptr[i]; j<Aptr[i+1]; j++)
    y[i] += Aval[j]*x[Aind[j]];

\end{verbatim}
\end{minipage}
\\
(b)
\\
\end{tabular}

\caption{(a) Compressed sparse row (CSR) format; (b) Basic implementation of CSR-based SpMV.}
\label{fig:spmv}
\end{figure}

\begin{figure*}%[t]
\begin{center}
  \subfigure[Sequential (single core)]{
  \includegraphics[width=.45\textwidth]{figures/axpy4_bgp/seq.eps}
  \label{fig:axpy4-bgp-seq}
  }
  \subfigure[Parallel (four cores)]{
  \includegraphics[width=.45\textwidth]{figures/axpy4_bgp/par.eps}
  \label{fig:axpy4-bgp-par}
  }
\end{center}
\caption{Performance of AXPY-4 operations on Blue Gene/P.}
\label{fig:axpy4-bgp-results}
\end{figure*}

The performance results shown in Figure~\ref{fig:axpy4-bgp-results} indicate
that the code tuned by Orio consistently outperforms the other three versions
for both the sequential and parallel cases. We observe that even for a simple
algebraic operation, such as the composed AXPY routines, the compiler alone
is unable to yield performance comparable to the empirically tuned
version. Moreover, implementations that rely on calls to multiple tuned
library routines (e.g., Goto BLAS and ESSL) suffer from loss of both spatial
and temporal localities, resulting in inferior memory performance.

%The sequential BLAS-based implementation perform better than the
%sequential single loop implementation. while for the parallel case, the
%simple loop implementation is more efficient than the BLAS codes.

\subsection{Sparse Matrix Computations}

In this section we examine the effectiveness of Orio in optimizing key
computations in PETSc~\cite{petsc-user-ref}, a toolkit for the parallel
numerical solution of partial differential equations, by empirically tuning
one of its heavily used kernels, sparse matrix-vector multiplication
(SpMV). SpMV dominates the performance of various scientific applications;
yet, traditional implementations of sparse kernels exhibit relatively poor
performance because of the use of indirect addressing for accessing the
matrix data elements.
%In
%order to attain higher performance, SpMV requires selecting a compact data
%structure and code transformations that best exploit properties of both the
%sparse matrix and the underlying architecture. Hence, the need for
%optimization and runtime tuning is a big distinction from the dense
%matrix-vector multiplication.

The SpMV operation computes $\forall{A_{i,j}} \neq 0:y_{i}
\leftarrow y_{i} + A_{i,j} \cdot x_{j}$, where $A$ is a sparse matrix,
and $x$, $y$ are dense vectors. Each element of $A$ is used only once,
and element reuse is possible only for $x$ and $y$. Thus, to optimize SpMV,
one should use compact data structures for $A$ and try to maximize temporal
reuse of $x$ and $y$. One of the most commonly used data structures for
storing a sparse matrix is compressed sparse row (CSR) storage
~\cite{vuduc-thesis},
illustrated in Figure~\ref{fig:spmv}(a). Elements in each row of $A$ are
packed together in a dense array, \texttt{Aval}, and a corresponding array of
integers \texttt{Aind} stores the column indices. The \texttt{Aptr} array
designates where each sparse row begins in \texttt{Aval} and
\texttt{Aind}. An implementation of SpMV using CSR storage is
shown in Figure~\ref{fig:spmv}(b).

PETSc's implementation of SpMV used in this experiment exploits matrix
structure by using \textit{inodes}, which represent rows with identical
nonzero structure.  PETSc detects inodes automatically in arbitrary sparse
matrices. For example, the sparse matrix shown in Figure~\ref{fig:spmv}(a)
contains three inodes. The first inode is of size two because it holds two
adjacent rows (i.e., the first and the second rows) with the same nonzero
structure. The second inode contains three consecutive rows with identical
nonzero structure, and the last inode has only a single row. Knowing the
properties of each inode at runtime enables maximum reuse of vector $x$ since
multiple elements of $x$ can be loaded and used exactly once for all rows in
the same inode structure. To accomplish this, one must use a
register-blocking transformation. To optimize the inode-based SpMV routine,
we incorporated a new transformation module inside Orio that implements
various optimization strategies including register blocking, SIMDization,
memory alignment optimization, loop-control optimization, accumulator
expansion, and thread-level parallelization (with OpenMP). We then used Orio
to automatically select the best version of the optimized inode SpMV. Only
the outer loop that iterates over inodes was parallelized by using OpenMP.

To evaluate the performance of the tuned SpMV routine, we conducted the
experiment using a 2-D driven cavity flow simulation
application~\cite{coff:kell:keye} (SNES ex27 in PETSc), on both the Xeon and
Blue Gene/P. The ``Compiler-optimized'' label is used as a base case that
represents the performance of the naive implementation of SpMV (shown in
Figure~\ref{fig:spmv}(b)) optimized by the native compiler. We also tested
the performance of the hand-tuned (by PETSc developers) SpMV code, which is
included in PETSc releases.

\begin{figure*}%[ht]
\begin{center}
  \subfigure[SMP: $p=1$, $t/p=8$]{
  \includegraphics[width=.315\textwidth]{figures/ex27_cookie/smp.eps}
  \label{fig:ex27-cookie-smp} } \subfigure[Dual: $p=4$, $t/p=2$]{
  \includegraphics[width=.315\textwidth]{figures/ex27_cookie/dual.eps}
  \label{fig:ex27-cookie-dual} } \subfigure[VN: $p=8$, $t/p=1$]{
  \includegraphics[width=.315\textwidth]{figures/ex27_cookie/vn.eps}
  \label{fig:ex27-cookie-vn} }
\end{center}
\caption{Performance of inode SpMV on eight-core Intel machine; $p$ is the number of processes, and $t/p$ is the threads per process.}
\label{fig:ex27-cookie-results}
\end{figure*}

\begin{figure*} %[htb]
\begin{center}
  \subfigure[1 node, SMP mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n1_smp.eps}
  \label{fig:ex27-bgp-smp-n1}
  }
  \subfigure[1 node, Dual mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n1_dual.eps}
  \label{fig:ex27-bgp-dual-n1}
  }
  \subfigure[1 node, VN mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n1_vn.eps}
  \label{fig:ex27-bgp-vn-n1}
  }
  \subfigure[8 nodes, SMP mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n8_smp.eps}
  \label{fig:ex27-bgp-smp-n8}
  }
  \subfigure[8 nodes, Dual mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n8_dual.eps}
  \label{fig:ex27-bgp-dual-n8}
  }
  \subfigure[8 nodes, VN mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n8_vn.eps}
  \label{fig:ex27-bgp-vn-n8}
  }
%  \subfigure[32 nodes, SMP mode]{
%  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n32_smp.eps}
%  \label{fig:ex27-bgp-smp-n32}
%  }
%  \subfigure[32 nodes, Dual mode]{
%  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n32_dual.eps}
%  \label{fig:ex27-bgp-dual-n32}
%  }
%  \subfigure[32 nodes, VN mode]{
%  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/n32_vn.eps}
%  \label{fig:ex27-bgp-vn-n32}
%  }
\end{center}
\caption{Performance of inode SpMV on Blue Gene/P for 1 and 8 nodes.}
\label{fig:ex27-bgp-results1}
\end{figure*}

\comment{
\begin{figure}
\begin{center}
  \subfigure[32x32, SMP mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s32_smp.eps}
  \label{fig:ex27-bgp-smp-32x32}
  }
  \subfigure[32x32, Dual mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s32_dual.eps}
  \label{fig:ex27-bgp-dual-32x32}
  }
  \subfigure[32x32, VN mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s32_vn.eps}
  \label{fig:ex27-bgp-vn-32x32}
  }
  \subfigure[64x64, SMP mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s64_smp.eps}
  \label{fig:ex27-bgp-smp-64x64}
  }
  \subfigure[64x64, Dual mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s64_dual.eps}
  \label{fig:ex27-bgp-dual-64x64}
  }
  \subfigure[64x64, VN mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s64_vn.eps}
  \label{fig:ex27-bgp-vn-64x64}
  }
  \subfigure[128x128, SMP mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s128_smp.eps}
  \label{fig:ex27-bgp-smp-128x128}
  }
  \subfigure[128x128, Dual mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s128_dual.eps}
  \label{fig:ex27-bgp-dual-128x128}
  }
  \subfigure[128x128, VN mode]{
  \includegraphics[width=.315\textwidth]{figures/ex27_bgp/s128_vn.eps}
  \label{fig:ex27-bgp-vn-128x128}
  }
\end{center}
\caption{Performance of inode SpMV on Blue Gene/P for problem sizes 32x32, 64x64, and 128x128.}
\label{fig:ex27-bgp-results2}
\end{figure}
}

Figure~\ref{fig:ex27-cookie-results} shows the performance results on the
Intel Xeon. The problem size is the number of grid points in the $x$ and $y$
directions, not the size of the sparse input matrix. The matrix dimension is
based on the grid size and the number of field components computed for each grid
point; for example, an $8 \times 8$ problem involves sparse matrices of
dimension 900 with 17,040 nonzero entries. In this experiment, we ran in
three multicore modes: SMP (one MPI process, eight
threads/process), dual (four MPI processes, two threads/process), and virtual
node or VN (eight MPI processes, one thread/process). The code tuned by Orio
consistently outperforms the others. Furthermore, the performance of
the hand-tuned code is almost equivalent to that of the icc-optimized simple
code. Performance differences between the Orio version and the hand-tuned
version become more significant as more threads per process are employed to
facilitate OpenMP parallelization. The SMP and VN results show that
optimizations using loop-level parallelism (through OpenMP) achieve much
better performance than using coarse-grained MPI parallelism.
%Moreover, for the multithread cases, tuning using Orio determined that
%the performance benefit of SIMDization is more significant than that of
%thread-level parallelization only when the problem sizes are small. The
%reason for this is due to the overhead introduced by OpenMP from coordinating
%its threads across processors.
%So, each optimized
%code for the many-thread cases comprises two distinct tuned codes to handle

The Blue Gene/P results in Figure~\ref{fig:ex27-bgp-results1} again
show that empirical optimization using Orio produces the best
performance for all cases. On this architecture, the performance gap
between the xlc-optimized and the hand-tuned codes is now larger than
in the Intel experiment. Similarly, by exploiting thread-level
parallelism, the Orio-optimized code performs better than the
hand-tuned version when the number of threads per process increases
and the number of nodes decreases. For the single-thread cases (VN
mode), thread-level parallelism is not exploited; nevertheless, the
Orio version still performs better than the other two implementations.

%This validates the
%effectiveness of Orio in empirically tuning sequential codes.

\subsection{Evaluation of the Pluto-Orio Integrated System}

This section discusses the performance evaluation of the integrated
Pluto-Orio system (discussed in Section~\ref{sec:integration}) on the
multicore Intel Xeon by using a number of application kernels that are
nontrivial to optimize and parallelize. We compare the performance of the
code tuned by Orio with the base code and the Pluto-generated codes. The
Pluto code was obtained by running the base code with
Pluto-0.3.0~\cite{pluto030} using \texttt{--tile} to employ loop tiling for
L1 cache, \texttt{--l2tile} to employ loop tiling for L2 cache, and
\texttt{--unroll} to employ loop unrolling, and for parallel code generation,
an additional \texttt{--parallel} option. We used Pluto's default tile
sizes (L1: 32x32 or 32x32x32; L2: 256x256 or 256x256x256) and default
unroll factors (64 for 1-D unrolling and 8x8 for 2-D
unroll/jamming). All codes were compiled with the Intel C compiler
using the \texttt{-O3} optimization flag to enable auto-vectorization
and other advanced optimizations, and
\texttt{-parallel} (or \texttt{-openmp} in the case of manual OpenMP parallelization)
to enable code parallelization.

In the following sections, we refer to the icc-optimized base code as
``ICC,'' the Pluto-generated code with L1 tiling and
unroll/jamming as ``Pluto (L1 tiling),'' and the Pluto-generated code with L1
and L2 tilings and unroll/jamming as ``Pluto (L1+L2 tiling).'' The code
tuned by Orio is referred to as ``Pluto+Orio''.

\begin{figure*}%[htb]
\begin{center}
  \subfigure[Sequential (T=500)]{
  \includegraphics[width=.45\textwidth]{figures/fdtd-2d-cookie/seq.eps}
  \label{fig:fdtd-2d-cookie-seq} } \subfigure[Parallel (T=500,
  N=2000)]{
  \includegraphics[width=.45\textwidth]{figures/fdtd-2d-cookie/par.eps}
  \label{fig:fdtd-2d-cookie-par} }
\end{center}
\caption{2-D FDTD performance on eight-core Intel Xeon.}
\label{fig:fdtd-2d-cookie-results}
\end{figure*}

\begin{figure*}%[htb]
\begin{center}
  \subfigure[Sequential (T=500)]{
  \includegraphics[width=.45\textwidth]{figures/seidel-cookie/seq.eps}
  \label{fig:seidel-cookie-seq} } \subfigure[Parallel (T=500,
  N=4000)]{
  \includegraphics[width=.45\textwidth]{figures/seidel-cookie/par.eps}
  \label{fig:seidel-cookie-par} }
\end{center}
\caption{3-D Gauss-Seidel performance on eight-core Intel Xeon.}
\label{fig:seidel-cookie-results}
\end{figure*}

\begin{figure*}%[htb]
\begin{center}
  \subfigure[Sequential]{
  \includegraphics[width=.45\textwidth]{figures/lu-cookie/seq.eps}
  \label{fig:lu-cookie-seq} } \subfigure[Parallel (N=4000)]{
  \includegraphics[width=.45\textwidth]{figures/lu-cookie/par.eps}
  \label{fig:lu-cookie-par} }
\end{center}
\caption{LU decomposition performance on eight-core Intel Xeon.}
\label{fig:lu-cookie-results}
\end{figure*}

\subsubsection{2-D Finite-Difference Time-Domain Method for Computational Electromagnetics}
We consider the two-dimensional finite difference time domain (FDTD)
algorithm, a popular method for solving the time-dependent Maxwell's
equations in the context of computational electrodynamic problems. As
shown in Figure~\ref{fig:fdtd-2d-code}, the 2-D FDTD method is
implemented as an outer iteration over time containing four
imperfectly nested loops.  The arrays $ex$ and $ey$ denote the
electric field components, and the array $hz$ denotes the magnetic
field.

\begin{figure}
\begin{center}
\begin{minipage}{3in}
\scriptsize
\begin{verbatim}
for(t=0; t<tmax; t++) {
  for (j=0; j<ny; j++) ey[0][j] = t;
  for (i=1; i<nx; i++)
    for (j=0; j<ny; j++)
      ey[i][j] -= 0.5*(hz[i][j] - hz[i-1][j]);
  for (i=0; i<nx; i++)
    for (j=1; j<ny; j++)
      ex[i][j] -= 0.5*(hz[i][j] - hz[i][j-1]);
  for (i=0; i<nx; i++)
    for (j=0; j<ny; j++)
      hz[i][j] -= 0.7*(ex[i][j+1] - ex[i][j]
                  + ey[i+1][j] - ey[i][j]);
}
\end{verbatim}
\end{minipage}
\end{center}
\caption{2-D FDTD code.}
\label{fig:fdtd-2d-code}
\end{figure}

\begin{figure}%[ht]
\begin{center}
\begin{minipage}{3in}
\scriptsize
\begin{verbatim}
for (t=0; t<=T-1; t++)
  for (i=1; i<=N-2; i++)
    for (j=1; j<=N-2; j++)
      A[i][j] = (A[i-1][j-1] + A[i-1][j]
          + A[i-1][j+1] + A[i][j-1] + A[i][j]
          + A[i][j+1] + A[i+1][j-1] + A[i+1][j]
          + A[i+1][j+1]) / 9.0;
\end{verbatim}
\end{minipage}
\end{center}
\caption{3-D Gauss-Seidel code}
\label{fig:seidel-code}
\end{figure}

\begin{figure}%[ht]
\begin{center}
\begin{minipage}{2.8in}
\scriptsize
\begin{verbatim}
for (k=0; k<=N-1; k++) {
  for (j=k+1; j<=N-1; j++)
    A[k][j] /= A[k][k];
  for(i=k+1; i<=N-1; i++)
    for (j=k+1; j<=N-1; j++)
      A[i][j] -= A[i][k]*A[k][j];
}
\end{verbatim}
\end{minipage}
\end{center}
\caption{LU decomposition code.}
\label{fig:lu-code}
\end{figure}

The performance of the sequential 2-D FDTD code for $tmax=500$ and $nx=ny$ is
shown in Figure~\ref{fig:fdtd-2d-cookie-seq}. The base code optimized by icc
alone performs better than Pluto for small problem sizes since all input
arrays fit in the L2 cache (insufficient computation to offset Pluto's tiling
overhead). As the array sizes increase, the lack of data reuse impairs the
base code's performance, whereas the Pluto performance remains about the
same.
%We used Orio to generate two code variants: one for small problem sizes
%and another for large problem sizes.
When the input arrays are small, Orio discovers that applying
Pluto's polyhedral transformations is not beneficial, and therefore
it employs only its syntactic transformations on the original FDTD
code. For large problem sizes, Orio exploits some of the Pluto's code
transformations and enhances these further with its syntactic
optimizations, resulting in performance consistently and significantly
higher than both the base and Pluto codes (up to 86\% over Pluto).

%By tuning the sequential code with Orio,
%the attainable performance enhancement over Pluto is up to 86\%.

Figure~\ref{fig:fdtd-2d-cookie-par} shows the multicore performance
obtained for $tmax=500$ and $nx=ny=2000$. The results indicate that
whereas icc is unable to auto-parallelize the code, Pluto detects the
existence of pipelined parallelism and then successfully parallelizes
the code. Locality-exploiting optimizations by Orio additionally
improve the Pluto performance by up to 78\%. Moreover, we observe
that because of memory contention between the two quad-core Intel
processors, the speedup of both the Pluto and Orio codes slightly
decreases when the number of cores used is greater than four.

%%BN: commented out since this needs more explanation (no room in this version)
%It is to be noted that collecting performance results of the Pluto
%code with L1 and L2 tilings was not possible because of the large code
%size of the generated loop nest, which led to an impractically huge
%compile time.

\subsubsection{3-D Gauss-Seidel Successive Overrelaxation Method}
Figure~\ref{fig:seidel-code} shows the 3-D Gauss-Seidel computation,
which is sometimes referred to as
\emph{successive displacement method}, indicating the dependence of the
iterations on the ordering. If the ordering is changed, the components
of the new iterations will change as well.

Figure~\ref{fig:seidel-cookie-seq} contains the sequential performance
results for $T=500$, which show that applying Pluto's polyhedral
tiling on the original code always delivers performance boosts, which
range from 126\% to 142\%. When using Pluto, performing two-level
tiling (for both L1 and L2 caches) yields 13\% lower performance than
performing only L1 tiling. This is also reflected in the best
sequential code found by Orio, where one-level tiling (for L1 cache
only) is always found to be better than two-level tiling. The
sequential speedup obtained from tuning the Pluto code with Orio is
significant, ranging from 51\% to 133\%.

The parallel performance results for $T=500$ and $N=4000$ are shown in
Figure~\ref{fig:seidel-cookie-par}. Again we observe that icc fails to
parallelize the code, whereas Pluto is able to parallelize it and Orio
improves the performance of the one-level tiled Pluto code further by up to
128\% (and up to 548\% for the two-level tiled Pluto code). We observe that
tiling over both L1 and L2 can result in worse performance than L1-only
tiling because the number of tiles can be less than the number of cores.
%Similar earlier trend also exists
%in these performance results: Pluto parallelizes the code through its
%polyhedral transformations (i.e., skewing and tiling), while the icc
%compiler alone does not parallelize the loop. Empirical tuning via
%Orio creates up to 122\% speedups over the one-level tiled Pluto code,
%and up to 548\% speedups over the two-level tiled Pluto code. Another
%finding is that tiling for both L1 and L2 caches contributes to a poor
%scalability because the number of tiles can be less than the number of
%processors, resulting in underutilization of the available resources.

\subsubsection{LU Factorization}

LU factorization or decomposition is a numerical method for the
solution of linear systems of equations; a simple implementation is
shown in Figure~\ref{fig:lu-code}.

The sequential performance results are shown in
Figure~\ref{fig:lu-cookie-seq}. Similarly to the 2-D FDTD results, the icc-optimized code is more
efficient than the Pluto-generated codes when the input arrays are small and
fit in the L2 cache. Because of better data locality, however, the
performance of the Pluto-tiled codes is better than that of the base code as
the input arrays get larger.
%Again, there are two different codes generated
%by Orio: one for small problem sizes and one for large problem sizes. And
Consequently, Orio employs Pluto's polyhedral transformations only for large
input sizes prior to applying its own syntactic transformations.
%Unlike 3-D
%Gauss-Seidel, this time Orio chooses to apply both L1 and L2 tilings when
%generating the best tuned code.
Compared to the two Pluto codes, the
Orio-tuned code yields performance improvements ranging between 26\% and 277\%.

The parallel performance results shown in Figure~\ref{fig:lu-cookie-par} were
obtained for $N=4000$. We observe that icc is not able to parallelize the
code, whereas Pluto achieves higher performance by exploiting multicore
parallelism. Furthermore, both Pluto-generated codes have comparable
performance, with slightly better scalability exhibited by the single-level tiled
code. Orio further improves the performance of the Pluto-generated versions
by a factor of 51\% to 120\%.


\newcommand{\defitem}[2]{\item[{\bf #1}]\label{itm:#2} }
\newcommand{\notitem}[1]{\item[]#1}
\section{Glossary}
\begin{compactenum}
\defitem{as complete}{defAsComplete} a phrase contrasting 2 trees.
    Tree $A$ is as complete as tree $B$ $\iff$
    leaf label set of $B$ is a subset of the leaf label set of $A$.\\
    In notation: $\leafLabels{B} \subseteq \leafLabels{A}$.
\defitem{cluster}{defCluster} the set of leaf nodes that are descendants
    of a node, $x$, is the cluster of $x$
\defitem{complete}{defComplete} as an adjective for a tree. Tree $A$ is complete $\iff$ the leaf label set
    of tree $A$ is identical to the the set of terminal taxa.
    In notation: $\leafLabels{A} = \leafLabels{\taxonomy}$
\defitem{conflicting}{defConflicting}. 2 \pss conflict with each other if there does not exist
    any tree that displays both of them.\\
    Since we are dealing with rooted statements that can partially overlap in term of their leaf sets,
    a procedure for checking for conflict: restrict both statements to just the overlapping labels
    (remove any labels that are not in the intersection of the leaf sets of the two statements), the
    \pss are in conflict if their include groups overlap but neither include group is a subset of the
    other.
\defitem{contested}{defContested} as an adjective for a non-terminal taxon. Taxon $A$ is contested $\iff$
    there is at least one tree in the set of input trees that has a node which is in conflict
    with the taxon.
    Note that if a tree has members of taxon as children of a polytomy that also contains other taxa, then
        the tree does not display the taxon, but it is also does not contest the taxon.
    Thus ``contests'' is not equivalent to ``does not display'' (though, if a tree displays a taxon, then
        it cannot contest that taxon).
\defitem{display}{defDisplay} Tree $A$ displays a node, $x$, from tree $B$ $\iff$
    that $A$ is as complete as tree $B$ and the induced tree of $A$ limited to the
    leaf label set of $B$ contains a cluster with labels identical to the label set of the cluster of $x$.\\
    An equivalent characterization in terms of the full leaf set of $A$ would be:
    tree $A$ is as complete as $B$ and there is a node in $A$ which has a leaf label set
    that is a superset of \leafLabels{x} and which does not contain any member of $\leafLabels{B} \setminus \leafLabels{x}$.\\
    We say that a tree $A$ displays a \ps $x$, if you were to translate
        $x$ tree with single internal node $y$, and tree $A$ displays $y$.
\defitem{internal node}{defInternalNode} a node that is not the root or a leaf.
\defitem{label}{defLabel} When speaking of the label of a node in this document, we are referring to unique taxonomic
    identifier for that node.
    This document does not discuss any issues associated with mapping strings to taxonomic names.
\defitem{leaf labels}{defLeafLabels} For our problems, the input trees have been aligned to a common
    taxonomy.
    So referring to the leaf label set of a tree can be interpreted as
    ``the set of taxonomic identifiers that are mapped to the leaves of the tree''.
    The leaf label set of a node is the set of labels associated with the cluster of the node.\\
    Note that a leaf label is not necessarily a terminal taxon.
    In notation, $\leafLabels{x}$ is the leaf label set of $x$.
\defitem{more complete}{defMoreComplete} tree $A$ is {\em more complete} than tree $B$ $\iff$ the leaf
    label set of $B$ is a proper subset of the leaf label set of $A$. \\
    In notation, $\leafLabels{B} \subset \leafLabels{A}$.
\defitem{phylogenetic statement}{defPS} (we are considering ``rooted split'' or ``rooted bipartition'' for this phrase)
    This is a pair of sets of taxonomic id sets: the include set and the exclude set (called ``ingroup'' and ``outgroup''
    on the google doc).\\
    The interpretation of a \ps is: if the statement is true, then
    any pair of elements in the include are more closely related to each
        other than they are to any element in the the exclude set.
    Thus if there are $N$ labels in the include set, and $M$ in the exclude set, the \ps implies
    ${N \choose 2}M$ distinct rooted triples.\\
    The include set and the exclude set must be disjunct. Their union is referred to as the leaf label set
        for the \ps.
    When we use the phrase ``\ps'' without qualifier, we mean a nontrivial statement.
    For a \ps to be nontrivial there must be at least 2 elements in the include group and at least
        1 in the exclude group.\\
    We use ``\vvps{\mbox{include}}{\mbox{exclude}}'' to denote a \ps.
    This is very similar to split syntax in unrooted phylogenetics, but with the arrow
        in place of $\mid$ to emphasize that the statement is oriented.\\
    Each internal node in a tree maps to a nontrivial \ps that can created by setting
        the include set equal to the leaf labels of node and setting the exclude set equal
        to the leaf label set of the tree minus the leaf labels of the node.
        In notation, node $x$ in tree $T$, maps to: $\lvps{x}{[\leafLabels{T} \setminus \leafLabels{x}]}$.\\
    A tree can be converted to a set of \pss: one for each internal node in the tree.
\defitem{terminal taxon}{defTerminalTaxon} a taxonomic identifier for at taxonomic node that
    has no children {\em in the taxonomic tree}.
\defitem{trivial}{defTrivial} a trivial \ps is one which must be found and any tree that has
    the leaf labels set of the \ps.
    This includes

\end{compactenum}
\section{Notation}
\begin{compactenum}
    \notitem{\leafLabels{x}} is the leaf label set of $x$.
    \notitem{\taxonomy} the taxonomic tree.
    %\notitem{\leafComp{x}{T}} is the exclude group for node $x$ on tree $T$. It is the
        %set complement of \leafLabels{x} with respect to the
    \notitem{\displaysPred{T}{x}} is an identity function that evaluates to 1 if tree $T$
        displays $x$, and to 0 otherwise.
\end{compactenum}