%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chapter{Using Trilinos}
\label{TRILINOS}
Trilinos has a number of packages and provides a large collection of both direct and indirect solvers.  We refer the reader to \cite{TrilinosWeb} for details.  Escript needs to be installed with Trilinos to be able to use the Trilinos solvers.  See the install guide for details.  We show a few examples for the Trilinos options with a simple example.

Consider Laplacian in domain ($\Omega\in \mathbb{R}^3$) with a simple right hand side,
\begin{align}
 -\nabla^t\; \nabla u &= 1,  &&\text{ in } \Omega, \label{CONST1a}\\
 u &= 0, &&\text{ on } \Gamma_D,\label{BC1}\\
 \mathbf{n}^t \; \nabla u &= 0, &&\text{ on }\Gamma_N,\,\label{BC2}
\end{align}
with $\mathbf{n}$ the outward normal, with $\Gamma_D$ the left boundary and $\Gamma=\partial\Omega\backslash\Gamma_D$.  Gravity forward weak form of PDE (\ref{CONST1a})-(\ref{BC2}), where $(~,~)$ is the standard $L^2$ inner product on $\Omega$, is
\begin{equation}\label{weak}
(\nabla u ~,~\nabla v ) = -(f ~,~v ),
\end{equation}
for all admissible potential functions $v$.

For this example, we just consider a simple, unstructured, 3D domain.  The mesh is created with GMSH using simplemesh.geo (file \ref{simplemesh}) see /escript/doc/examples/usersguide/simplemesh.geo.  To test AMG, a finer mesh is created using    /escript/doc/examples/usersguide/simplemeshfine.geo.

\begin{python}[caption=simplemesh.geo,label=simplemesh]
// dimensions and mesh size
xdim = 100.;
ydim = 200.;
zdim = 50.;
mtop = 2.;
mbase = 5.;

//Points
Point(1) = {0., 0., 0., mbase};
Point(2) = {xdim, 0., 0., mbase};
Point(3) = {0., ydim, 0., mbase};
Point(4) = {xdim, ydim, 0., mbase};
Point(5) = {0., 0., zdim, mtop};
Point(6) = {xdim, 0., zdim, mtop};
Point(7) = {0., ydim, zdim, mtop};
Point(8) = {xdim, ydim, zdim, mtop};

//Lines and surfaces
Line(1) = {1, 2};
Line(2) = {3, 4};
Line(3) = {1, 3};
Line(4) = {2, 4};
Line(5) = {5, 6};
Line(6) = {7, 8};
Line(7) = {5, 7};
Line(8) = {6, 8};
Line(9) = {1, 5};
Line(10) = {3, 7};
Line(11) = {2, 6};
Line(12) = {4, 8};
Line Loop(1) = {-1, 3, 2, -4};
Plane Surface(1) = {1};
Line Loop(2) = {5, 8, -6, -7};
Plane Surface(2) = {2};
Line Loop(3) = {1, 11, -5, -9};
Plane Surface(3) = {3};
Line Loop(4) = {-2, 10, 6, -12};
Plane Surface(4) = {4};
Line Loop(5) = {-3, 9, 7, -10};
Plane Surface(5) = {5};
Line Loop(6) = {4, 12, -8, -11};
Plane Surface(6) = {6};

// domain
Surface Loop(1) = {1:6};
Volume(1) = {1};
\end{python}

The most basic escript script to solve this pde using escript defaults is in program \ref{basic}.  The computed solution is saved in a silo file "asimple.silo" and can be visualized using \VisIt.
\begin{python}[caption=basic solve using defaults only, label=basic ]
from esys.escript import *
from esys.weipa import saveVTK, saveSilo
from esys.escript.linearPDEs import LinearSinglePDE
from esys.finley import ReadGmsh

domain=ReadGmsh("simplemesh.msh", 3,  optimize=True )

pde = LinearSinglePDE(domain, isComplex=False)
pde.setSymmetryOn()
x = domain.getX()
pde.setValue(A=kronecker(3), Y = 1., q = whereZero(x[0]-inf(x[0])))

u=pde.getSolution()
saveSilo("asimple", u=u)
\end{python}

This script uses default Trilinos solver PCG with Jacobi preconditioner. (It takes 271 iterations to reach the default tolerance of $10^{-8}$).  Tolerance and solver output can be controlled by adding to the basic listing.
 \begin{python}[caption=tolerance , label=tolerance ]
options = pde.getSolverOptions()
options.setTolerance(1e-8)
options.setVerbosityOn()
\end{python}

\section{Direct Solvers}
There are a number of options that can be set for solving the pde.  The simplest way to choose a direct solver is to use the default Trilinos direct solver KLU2.  Escript can only use this feature if it is available in Trilinos.

To choose the default Trilinos direct solver, set the tolerance for the PDE solver to $10^{-8}$,  use the Trilinos suite of solvers and output information about the solvers, we add code to listing \ref{basic}.
\begin{python}[caption=default Direct , label=defaultdirect ]
options = pde.getSolverOptions()
options.setSolverMethod(SolverOptions.DIRECT)
options.setPackage(SolverOptions.TRILINOS)
options.setVerbosityOn()
\end{python}
The default Trilinos Direct solver is the Amesos2 LU factorisation KLU. It is a serial, unsymmetric sparse, partial-pivoting, direct matrix solver.  The last line above ensures that the output includes details of the methods used.

To use SUPERLU the code that needs to be added to listing \ref{basic} is
\begin{python}[caption=SuperLU, label=superLU ]
options = pde.getSolverOptions()
options.setTolerance(1e-8)
options.setPackage(SolverOptions.TRILINOS)
options.setSolverMethod(SolverOptions.DIRECT_SUPERLU)
options.setVerbosityOn()
\end{python}
Trilinos is the default solver package so the setPackage line is not really necessary.

\section{Preconditioned Conjugate Gradient}
If we want to use preconditioned conjugate gradient, then we remover the DIRECT solver line and add the line
\begin{python}[caption=Preconditioned conjugate gradient defaults, label=PCG]
options.setSolverMethod(SolverOptions.PCG)
\end{python}
This takes 271 iterations to reach tolerance using a Jacobi preconditioner, BELOS Pseudo Block CG with Ifpack2.  To change the preconditioner to Gauss-Siedel, we add the line
\begin{python}
options.setPreconditioner(SolverOptions.GAUSS_SEIDEL)
\end{python}
This takes 120 iterations to reach tolerance.

\section{Multigrid}
Geometric multigrid methods were introduced for structured grids to maximise the advantages of iterative methods for solving matrix equations derived from discretised partial differential equations.  Iterative methods for these problems are effective in reducing oscillatory error but stall for smooth error and smooth error appears oscillatory when restricted to a coarser grid.  The idea is to have a succession of grids from fine to coarse and to remove error by iterating on each of the grids in turn reducing the oscillatory error on that grid.  The coarsest grid is chosen small enough so that it can be quickly solved directly or iteratively. If $n$ is the size of the problem then a multigrid algorithm is order $n$.

The matrix equation,  derived from the PDE, is
\begin{equation}\label{matrixEQ}
    \mathbf{A}\mathbf{u}=\mathbf{f},
\end{equation}
where $\mathbf{u}$ is the unknown $n\times 1$ vector, $\mathbf{A}$ is the $n\times n$ matrix and $\mathbf{f}$ is the nown right hand side $n\times 1$ vector.  The residual $\mathbf{r}$ is defined
\begin{equation}\label{res}
    \mathbf{r}=\mathbf{f}-\mathbf{A}\mathbf{u}%=\mathbf{A}\mathbf{u}^*-\mathbf{A}\mathbf{u}=\mathbf{A}\mathbf{u}^*-\mathbf{u}),
\end{equation}
and the residual equation is defined
\begin{equation}\label{resEQ}
    \mathbf{A}\mathbf{e}=\mathbf{r},
\end{equation}
where $\mathbf{e}=\mathbf{u}^*-\mathbf{u}$ is the error and $\mathbf{u}^*$ is the exact solution.  Relaxing on \ref{resEQ}) with the residual as the right hand side is the same as relaxing on (\ref{matrixEQ}) with the original right and side.  We use superscripts on these terms to indicate the discretization representing element length, $h$, for a fine grid and $H$ for one level coarser discretization, $h<H$.  For a structured mesh, $H=\frac{h}{2}$ and $\mathbf{A}^h$ is an $n\times n$ matrix and $\mathbf{A}^H$ is an $N\times N$ matrix obtained in the coarsening process with $N<n$.  Prolongation operator, $\mathbf{P}^h_H$, an $n\times N$ matrix, is used to interpolate the error from the coarse grid to the fine grid and restriction operator $\mathbf{P}_h^H$, an $N\times n$ matrix, is used to restrict the residual from the fine grid to the coarse grid.  It is not necessary for the restriction operator to be the transpose of the interpolation operator.  The coarse grid matrix operator $\mathbf{A}^H$ is computed by multiplying the fine grid matrix operator $\mathbf{A}^h$ on the left by the restriction operator and the right by the interpolation operator resulting in an $N\times N$ matrix.

The MG algorithm is best described using a recursion algorithm.  After iteration on a fine grid, the fine grid error is smooth and the residual can be restricted to a coarse grid. The error on the coarse grid appears more oscillatory and is (smoothed) reduced by iteration.  If this is the coarsest grid then the matrix equation is solved using a direct method or sufficient iterations of the solver to get within discretization error, otherwise the algorithm is called again with the coarse grid replacing the fine grid and the next coarser grid as the coarse grid.  The coarse grid error, is interpolated to the fine grid where it corrects the fine grid approximation and post-smoothing iteration smooths the error.  For structured grids, the choices for interpolating from a coarse grid to a fine grid or restricting from a fine grid to a coarse grid are reasonably obvious, see \cite{Briggs2000}.

\begin{table}\center
\begin{tabular}{|l|l|ll}
\hline
AMG($\mathbf{u}^h;\mathbf{A}^h,\mathbf{f}^h)$ &\\ \hline
    \quad $\mathbf{A}^h\mathbf{u}^h=\mathbf{f}^h$ & \textbf{p steps pre-smoothing iteration}\\
	\quad $\mathbf{r}^h=\mathbf{f}^h-\mathbf{A}^h \mathbf{u}^h$ & \textbf{fine grid residual}\\
	\quad $\mathbf{r}^H=\mathbf{R}_h^H \mathbf{r}^h$ & \textbf{restrict residual} \\
	\quad $\mathbf{A}^H=\mathbf{R}_h^{H} \mathbf{A}^h \mathbf{P}_H^h$& \textbf{restrict operator}\\
	\quad if not coarsest  &\\
	  \quad\quad AMG($\mathbf{e}^H;\mathbf{A}^H,\mathbf{r}^H$) &\textbf{recursion}\\
	  \quad else &\\
	  \quad\quad $\mathbf{A}^H\mathbf{e}^H=\mathbf{r}^H$ & \textbf{coarsest solve}\\
	\quad$\mathbf{e}^h=\mathbf{P}_H^h \mathbf{e}^H$ & \textbf{interpolate error} \\
	\quad$\mathbf{u}^h=\mathbf{u}^h+\mathbf{e}^h$ & \textbf{fine grid correction}\\
	\quad$\mathbf{A}^h\mathbf{u}^h=\mathbf{f}^h$ & \textbf{q steps post-smoothing iteration}\\
	\quad return $\mathbf{u}^h$ &\\
\hline
\end{tabular}\caption{An AMG V(p.q) cycle algorithm to solve $\mathbf{A}\mathbf{u}=\mathbf{f}$, where $h$ and $H$ represent grid sizes with $h<H$.}
\end{table}

Algebraic multigrid methods were developed for unstructured grids and do not reference the grid but instead use interpolation and restriction operators derived from the matrix (see \cite{Briggs2000, Stuben2001281, Vanek1996, Tuminaro2000}).  The terms "coarse grid" and "fine grid" are still used but do not refer to actual grids.  "Smooth error" is defined to be the error not reduced by iteration and "oscillatory error" is the error reduced by iteration.  Coarse levels are chosen from the relative sizes of the off diagonal terms in the fine matrix.
Once the coarse grid is chosen the restriction and interpolation operators are computed. Restriction operators need to be chosen so that "smooth error" on a fine grid will appear "oscillatory" on a coarse grid ensuring that it can be reduced by iterating on this grid. There are a number of algorithm options available in Trilinos to compute the coarse grid and the choice will depend on the original PDE and smoothing options. For any multigrid method, there are basic choices:
\begin{itemize}%[topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]
    \item pre-smoothing iterative solver and number of iterations
    \item post-smoothing iterative solver and number of iterations
    \item choosing coarse grids
    \item number of coarse grids or size of coarsest grid
    \item interpolation operator
    \item restriction operator
    \item coarsest grid solver
    \item cycle type
\end{itemize}
We use Trilinos solvers and it is possible to access Trilinos options either within the escript script or, if more complicated control is needed, in an XML file. There are many Trilinos packages that can be used by escript including
MueLu   - setup of AMG, Belos   - linear solvers - Pseudo Block CG, Ifpack2 - iterative solvers (Jacobi, Gauss-Siedel), Amesos2 - direct solvers for coarse level and Voltan or Voltan2 - repartitioning for caorse grids

\subsection{Default MueLu Trilinos options for algebraic multigrid preconditioned conjugate gradient (AMG-PCG)}
More detail on the various options can be found in the MUELU user guide and other Trilinos user guides \cite{TrilinosWeb}. MUELU uses other Trilinos packages and to access these parameters the XML file must be used.
Recall $\mathbf{R}_h^H$ is the restriction operator and $\mathbf{P}^h_H$ is the interpolation (prolongation) operator.  The default values used are shown in Table \ref{defaultAMG}.  To use AMG-PCG with default paramerters we use script \ref{basicAMG}.

\begin{table}\center
\begin{tabular}{|r|l|}
\hline
    number of equations & 1\\
    problem: symmetric & True, $\mathbf{R}_h^H = (\mathbf{P}^h_H)^t$ \\
    pre-smoothing iterative solver & Symmetric Gauss-Seidel\\
    post-smoothing iterative solver & Symmetric Gauss-Seidel\\
    pre-smoothing iterations & 1 \\
    post-smoothing iterations & 1 \\
    minimum aggregate size & 2 \\
    maximum aggregate size & unlimited \\
    aggregation & uncoupled \\
    maximum number of levels & 10\\
    maximum size of coarsest grid & 2000\\
    choosing coarse grids & classical smoothed aggregation\\
    coarsest grid solver &  SuperLU\\
    cycle type & V(1,1) \\
    \hline
\end{tabular}\caption{Default parameters for AMG-PCG}\label{defaultAMG}
\end{table}

\begin{python}[caption=basic Trilinos PCG-AMG defaults only script, label=basicAMG ]
from esys.escript import *
from esys.weipa import saveVTK, saveSilo
from esys.escript.linearPDEs import LinearSinglePDE
from esys.finley import ReadGmsh

domain=ReadGmsh("simplemesh.msh", 3,  optimize=True )

pde = LinearSinglePDE(domain, isComplex=False)
pde.setSymmetryOn()
x = domain.getX()
pde.setValue(A=kronecker(3), Y=1, q=whereZero(x[0]-inf(x[0])))

options = pde.getSolverOptions()
options.setPackage(SolverOptions.TRILINOS)
options.setSolverMethod(SolverOptions.PCG)
options.setPreconditioner(SolverOptions.AMG)

u=pde.getSolution()
saveSilo("asimple",u=u)
\end{python}

Only two grids were used for the simple mesh.  The first coarse grid, replaces, if possible, 27 fine grid nodes represented by  one coarse grid node.  In 2D this would be 9 fine grid nodes represented with one coarse grid node. The ratio of the fine to coarse grid in this example is 19.21.  For a finer mesh, with mtop = 2 and mbase=1 in the geo file, 3 levels of grids and the coarsest grid is solved with a direct method.  AMG-PCG took 11 iterations to reach tolerance.

\subsection{Altering MUELU parameters}
To access MUELU parameters in the python script we use the general form
\begin{python}
options.setTrilinosParameter( "A", "B")
\end{python}
where "A" is the Trilinos parameter and "B" is its string value. It is extremely important to have correct spaces in the strings.

\subsubsection{Debug output}
Options are "none", "low", "medium", "high" and "extreme".
\begin{python}
options.setTrilinosParameter("verbosity", "low")
\end{python}
Options are\\
\var{"low"} - setup time,\\
\var{"medium"} - basic AMG data, mesh sizes, smoothers + "low" \\
\var{"high"} - input data, relaxation solvers and data, aggregate data + "medium"\\
\var{"extreme"} - may include solver details that MueLu calls + "high "\\

\subsubsection{Problem type}
Changes default multigrid algorithm, block size and smoother.
Options are
\begin{itemize}
    \item \var{"unknown"}: default
    \item \var{"Poisson-2D" or "Poisson-3D"} : using smoothed aggregation, Chebyshev smoother and block size of 1,
    \item \var{"Elasticity-2D" and "Elasticity-3D"}: using smoothed aggregation, Chebyshev smoother and block size of 2 or 3 respectively,
    \item \var{"Poisson-2D-complex" and "Poisson-3D-complex"}: using smoothed aggregation, symmetric Gauss-Seidel and block size 1,
    \item \var{"Elasticity-2D-complex" and "Elasticity-3D-complex"}: using smoothed aggregation, symmetric Gauss-Seidel and block size 2 and 3 respectively,
    \item \var{"ConvectionDiffusion"}: using Petrov-Galerkin AMG, Gauss-Seidel and 1 block,
    \item \var{"MHD"}: using unsmoothed aggregation and Additive Schwarts method with one level of overlap and ILU(0) as a subdomain solver.
\end{itemize}
\begin{python}
options.setTrilinosParameter("problem:type", "Poisson-3D")
\end{python}

\subsubsection{number of equations}
Number of PDE equations at each grid node.
\begin{python}
options.setTrilinosParameter("number of equations", 1)
\end{python}

\subsubsection{AMG algorithm}
The multigrid algorithm for computing the coarse levels and interpolation and restriction operators is controlled with \var{"multigrid algorithm"}. The default value is smoothed aggregation and is selected with \var{"sa"} and a damping factor can be imposed.  The other options are \var{"unsmoothed"}, no Jacobi prolongation improvement step; \var{"pg"}, $\mathbf{A}$ prolongation smoothing and $\mathbf{A}^T$ restriction smoothing; \var{"emin"} basis functions for grid transfer using energy constrained minimisation; \var{"interp"}, piecewise constant ("interpolation order" set to 0) or linear interpolation (\var{"interpolation order"} set to 1) from coarse to fine and is only possible with structured aggregation; and \var{"semicoarsen"}, coarsen fully in z direction (will need to set rate in this direction).   It is also possible to use an implicit transpose for the restriction operator.
\begin{python}
# smoothed aggregation
options.setTrilinosParameter("multigrid algorithm", "sa")
options.setTrilinosParameter("sa: damping factor", 1.3)
options.setTrilinosParameter("sa: use filtered matrix", True)
options.setTrilinosParameter("filtered matrix: use lumping", True)
options.setTrilinosParameter("filtered matrix: reuse eigenvalue", True)
# unsmoothed
options.setTrilinosParamter("multigrid algorithm", "unsmoothed")
# pg
options.setTrilinosParameter("multigrid algorithm", "pg")
# interpolation
options.setTrilinosParameter("multigrid algorithm", "interp")
options.setTrilinosParameter("interp: interpolation order", 1)
                                                          # 0, 1
options.setTrilinosParameter("interp: build coarse coordinates", True)
# emin
options.setTrilinosParameter("multigrid algorithm", "emin")
options.setTrilinosParameter("emin: iterative method", "cg")
                                                     # "cg", "gmres", "sd"
options.setTrilinosParameter("emin: num iterations", 2)
options.setTrilinosParameter("emin: num reuse iterations", 1)
options.setTrilinosParameter("emin: pattern", "AkPtent")
options.setTrilinosParameter("emin: pattern order", 1)
# semicoarsen
options.setTrilinosParameter("multigrid algorithm", "semicoarsen")
options.setTrilinosParameter("semicoarsen: coarsen rate", 3)
#
options.setTrilinosParameter("transpose: use implicit", False)
\end{python}

\subsubsection{Maximum levels, coarse mesh, coarse solver}
It is possible to limit the size of the coarsest level as well as limit the number of levels.  The default for the size of the coarsest level is 2000.  So once the size of the coarse level is less than 2000 then no more coarse levels are created.  Additionally, it is possible to limit the number of levels by setting "max levels" in the hierarchy.  This includes the fine grid.  The default value is 10 but depending on fine grid size, changing this could improve performance of the algorithm.  The coarsest level can be solved using a direct solver.  Possibilities are KLU, KLU2, SuperLU, SuperLU\_dist, Umfpack and Mumps

\begin{python}
options.setTrilinosParameter("max levels", 10)
options.setTrilinosParameter("coarse: max size", 2000)
options.setTrilinosParameter("coarse: type", "SuperLU")
\end{python}

\subsubsection{Aggregation}
It is possible to influence aggregation options.  If the fine mesh is a structured grid then aggregates can be created in a  "structured" way and the aggregation attempts to form hexahedral coarse levels.  This uses a default coarsening rate of 3 in each direction. The option "hybrid" allows user determined "structured" or "unstructured" aggregation for each level, To get optimal size coarse mesh ($3^d$ in $d$ dimensions) "uncoupled" or "coupled" is used with "coupled" allowing aggregates to span processors.  It is suggested that "coupled" should be used with care.  "brick" attempts to make rectangular aggregates. Some of the options are below with more detail and more options in the MUELU user guide.
\begin{python}
options.setTrilinosParameter("aggregation: type", "structured")
options.setTrilinosParameter("aggregation: ordering", "natural")
                                                 # "natural", "graph", "random"
options.setTrilinosParameter("aggregation: drop scheme", "classical")
                                                 # "classical", "distance laplacian"
options.setTrilinosParameter("aggregation: drop tol", 0.0)
options.setTrilinosParameter("aggregation: min agg size", 2)
options.setTrilinosParameter("aggregation: max agg size", -1)
                                                # -1 means unlimited
options.setTrilinosParameter("aggregation: Dirichlet threshold", 1e-5)
\end{python}

\subsubsection{Relationship between $\mathbf{R}_h^H$  and $\mathbf{P}^h_H$}
For $\mathbf{R}_h^H=(\mathbf{P}^h_H)^t$
\begin{python}
options.setTrilinosParameter("problem: symmetric", True)
\end{python}
this is the default.

\subsubsection{Smoothers}
In the escript script it is possible to choose smoother type, "RELAXATION", "CHEBYSHEV" and "ILUT" or "RILUT" but for more specific control the XML file needs to be used.  It is possible to use different pre and post smoothers.  "RELAXATION" could use Jacobi, Gauss-Seidel, symmetric Gauss-Seidel, multithreaded Gauss-Seidel.  To specify which one the XML file must be used.  Some examples for this are
\begin{python}
options.setTrilinosParameter("smoother: pre or post", "both")
options.setTrilinosParameter("smoother: type", "RELAXATION")
options.setTrilinosParameter("smoother: pre type", "CHEBYSHEV")
options.setTrilinosParameter("smoother: post type", "RELAXATION")
\end{python}

\subsubsection{Cycle type}
Allowable cycle types are "V" and "W".  The default is a "V" cycle.
\begin{python}
options.setTrilinosParameter("cycle type", "V")
\end{python}

\subsubsection{reuse}
If multiple PDEs are being solved the reuse strategy can use elements of previous computations.  The level of reuse varies from none to full.  Options are \var{"none"}; \var{"S"}, symbolic coarse levels information; \var{"tP"}, reuse tentative prolongation operator; \var{"emin"}, reuse old prolongator for initial guess; \var{"RP"}, reuse smoothed restrictor and prolongator; \var{"RAP"}, compute only fine level smoothers and reuse all other operators, and \var{"full"}, reuse everything.
\begin{python}
options.setTrilinosParameter("reuse: type", "full")
\end{python}

\subsubsection{repartitioning}
If there are multiple processors it might be benificial to repartition as the mesh are coarsened including perhaps using only one processor for the coarsest grid.  This is to reduce communication costs for the caorser grids.
\begin{python}
options.setTrilinosParameter("repartition: enable", False)
options.setTrilinosParameter("repartition: start level", 2)
options.setTrilinosParameter("repartition: min rows per proc", 800)
options.setTrilinosParameter("repartition: max imbalance", 1.2)
options.setTrilinosParameter("repartition: remap parts", True)
options.setTrilinosParameter("repartition: rebalance P and R", False)
\end{python}

\subsection{commands in XML file}
All the previous commands can be placed into an XML file.  The XML file option allows the user to choose parameters for the programs that MUELU calls, so more control is possible on the iterative solvers.
\begin{python}
from esys.escript import *
from esys.weipa import saveVTK, saveSilo
from esys.escript.linearPDEs import LinearSinglePDE
from esys.finley import ReadGmsh

domain=ReadGmsh("simplemesh.msh", 3,  optimize=True )

pde = LinearSinglePDE(domain, isComplex=False)
pde.setSymmetryOn()
x = domain.getX()
pde.setValue(A=kronecker(3), Y=1, q=whereZero(x[0]-inf(x[0])))

options = pde.getSolverOptions()
options.setPackage(SolverOptions.TRILINOS)
options.setSolverMethod(SolverOptions.PCG)
options.setPreconditioner(SolverOptions.AMG)
options.setTrilinosParameter("xml parameter file", "simplebob.xml")

u=pde.getSolution()
saveSilo("asimple",u=u)
\end{python}

It is possible to specify how many sweeps of the iterative solvers for pre and post smoothing and we could choose cycles with different numbers of pre and post sweeps.  Amesos2 provides direct solvers including superLU and Mumps. MueLu passes parameters directly to solver library.  To specify CHEBYSHEV parameters, for example, an XML file must be used.

Ifpack2 or Ifpack provides iterative matrix solvers Jacobi, Gauss Seidel, polynomial, distribution relaxation, domain decomposition solvers and incomplete factorizations.

A very simple XML file is in file \ref{simplebob}
\begin{python}[caption=simplebob.xml,label=simplebob]
<ParameterList name="MueLu">
  <Parameter name="verbosity"               type="string"    value="high"/>
  <Parameter name="max levels"              type="int"       value="4"/>
  <Parameter name="coarse: max size"        type="int"       value="200"/>
  <Parameter name="multigrid algorithm"     type="string"    value="sa"/>
  <Parameter name="reuse: type"             type="string"    value="full"/>
  <Parameter name="transpose: use implicit" type="bool"      value="true"/>
  <Parameter name="sa: damping factor"      type="double"    value="0.1"/>
  <Parameter name="sa: use filtered matrix" type="bool"      value="true"/>
</ParameterList>
\end{python}

A more complicated example that controls the number of pre and post sweeps is in the listing \ref{complicatedbob}
\begin{python}[caption=complicatedbob.xml,label=complicatedbob]
<ParameterList name="MueLu">
  <!--    General    -->
  <Parameter name="verbosity"               type="string"    value="high"/>
  <Parameter name="max levels"              type="int"       value="4"/>
  <Parameter name="coarse: max size"        type="int"       value="200"/>
  <Parameter name="multigrid algorithm"     type="string"    value="sa"/>
  <Parameter name="reuse: type"             type="string"    value="full"/>
  <Parameter name="transpose: use implicit" type="bool"      value="true"/>
  <Parameter name="sa: damping factor"      type="double"    value="0.1"/>
  <Parameter name="sa: use filtered matrix" type="bool"      value="true"/>

  <!-- Smoothing -->
  <Parameter name="smoother: pre or post"        type="string"  value="both"/>

  <Parameter name="smoother: pre type"           type="string"  value="CHEBYSHEV"/>
  <ParameterList name="smoother: pre params">
    <Parameter name="relaxation: type"           type="string"  value="Symmetric Gauss-Seidel"/>
    <Parameter name="relaxation: sweeps"         type="int"     value="5"/>
    <Parameter name="relaxation: damping factor" type="double"  value="0.9"/>
  </ParameterList>

  <ParameterList name="smoother: params">
    <Parameter name="chebyshev: degree"           type="int"     value="3"/>
    <Parameter name="chebyshev: ratio eigenvalue" type="double"  value="15"/>
  </ParameterList>

  <Parameter name="smoother: post type"           type="string"  value="RELAXATION"/>
  <ParameterList name="smoother: post params">
    <Parameter name="relaxation: type"           type="string"  value="Symmetric Gauss-Seidel"/>
    <Parameter name="relaxation: sweeps"         type="int"     value="5"/>
    <Parameter name="relaxation: damping factor" type="double"  value="0.9"/>
  </ParameterList>

  <!-- Aggregation -->
  <Parameter name="aggregation: type"           type="string"  value="uncoupled"/>
  <Parameter name="aggregation: min agg size"   type="int"     value="3"/>
  <Parameter name="aggregation: max agg size"   type="int"     value="27"/>

  <!--  for different level parameter list -->
  <ParameterList name="level 2">
    <Parameter name="smoother: type" type="string" value="CHEBYSHEV"/>
  </ParameterList>

</ParameterList>
\end{python}


\chapter{Binary search algorithms}
\Label{cha:binary-search}

In this chapter, we consider the four
\emph{binary search} algorithms of the \cxx Standard Library \cite[\S
28.7.3]{cxx-17-draft}, namely

\begin{itemize}
\item \lowerbound in \S\ref{sec:lowerbound}

\item \upperbound in \S\ref{sec:upperbound}

\item two variants for the implementation of  \equalrange in \S\ref{sec:equalrange}

\item two variants for the formal specification of \binarysearch in \S\ref{sec:binarysearch}
\end{itemize}

As in the case of the of maximum/minimum algorithms from Chapter~\ref{cha:maxmin}
the binary search algorithms primarily use the less-than operator~\inl{<}
(and the derived operators \inl{<=}, \inl{>} and \inl{>=}) to determine whether a particular
value is contained in an increasing range.
Thus, different to the \find algorithm in \S\ref{sec:find},
the equality operator~\inl{==} will play only a supporting part
in the specification of binary search.

In order to make the specifications of the binary search algorithms
more compact and (arguably) more readable we re-use the predicates
\logicref{LowerBound}, \logicref{StrictLowerBound},
\logicref{UpperBound}, and \logicref{StrictUpperBound}.

All binary search algorithms require that their input array is arranged in
increasing order.
The following listing shows two versions of predicate \logicref{Increasing}.
The first one defines when a section of an array is in increasing order.
The second version uses the first one to express that the whole array is in increasing order.

\input{Listings/Increasing.acsl.tex}

%\clearpage

There is also the overloaded predicate \logicref{WeaklyIncreasing} that we will user for
the verification of other algorithms.

\input{Listings/WeaklyIncreasing.acsl.tex}

Users inexperienced in formal verification often have a blind spot at the
difference between \Increasing and \WeaklyIncreasing.
%
Both versions are logically equivalent,
and proving that \Increasing implies \WeaklyIncreasing is even trivial.
%
However, proving the converse direction is not, and requires an induction on the
array size~\inl{n}, employing the transitivity of \inl{<=} in the induction step.
%
Humans are trained to perform such inductions unnoticed,
but none of the automated provers supported by \framac is able to perform induction.
%
The following Listing contains several lemmas on the relationship of
\WeaklyIncreasing and \Increasing.

\input{Listings/IncreasingLemmas.acsl.tex}

\clearpage

We usually exploit the relationship of the predicates \Increasing and \WeaklyIncreasing
in the following way:

\begin{itemize}
\item We use the predicate \Increasing in the preconditions and postconditions of
      function contracts.
\item The \WeaklyIncreasing is employed for assertions and loop invariants
      whenever we have to verify that an algorithm (typically a sorting algorithm)
      produces an increasing array.
\item Finally, to conclude that a \emph{weakly increasing} array is in fact \emph{increasing}
      we rely on lemma\\
      \logicref{WeaklyIncreasingIncreasing} .
\end{itemize}

\clearpage

\input{binary-search/lower_bound}
\input{binary-search/upper_bound}
\input{binary-search/equal_range}
\input{binary-search/binary_search}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  CERNLIB manual in LaTeX form   	                          %
%                                                                 %
%  Michel Goossens (for translation into LaTeX)                   %
%  Version 2.00                                                   %
%  Last Mod.  6 Oct 1992  1030   MG                               %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentstyle[epsfig,11pt,fleqn,amssym,cernlib,cerndoc]{cernman}
\newcommand{\Title}{CERN Program Library}%       Title for document
\newmathalphabet*{\mathtt}{cmtt}{m}{n}
\newmathalphabet*{\mathbf}{cmr}{b}{n}
%\romanfont{times}
%\PScommands% Initialize PS boxes
%\makeindex
\begin{document}
%  ==================== Front material ============================
%\include{cernlibf}
%\cleardoublepage
%  ==================== Body of text ==============================
\pagenumbering{arabic}
\setcounter{page}{1}

%\chapter{General Information}

%%%%%%   Catalog of Program packages and entries%%%%

\def\Rtnr{Catalog}%Dummy routine name to appear at bottom of page
\include{crnlbcat}
\cleardoublepage

\let\LARGE\large
\let\Large\large

% Here come the different files to be included

%%     A part     %%

\Sectitle{Arithmetic Routines}

\include{a105}

%%     B part     %%

\Sectitle{Elementary Functions}

\include{b100}
\include{b101}
\include{b102}

%%     C part     %%

\Sectitle{Equations and Special Functions}

\include{c200}
\include{c201}
\include{c202}
\include{c205}
\include{c207}
\include{c208}
\include{c209}
\include{c210}
\include{c300}
\include{c301}
\include{c302}
\include{c303}
\include{c304}
\include{c306}
\include{c307}
\include{c308}
\include{c309}
\include{c310}
\include{c312}
\include{c313}
\include{c315}
\include{c316}
\include{c317}
\include{c318}
\include{c320}
\include{c321}
\include{c322}
\include{c323}
\include{c324}
\include{c327}
\include{c328}
\include{c330}
\include{c331}
%\include{c332}
\include{c333}
\include{c334}
\include{c335}
\include{c336}
\include{c337}
\include{c338}
\include{c339}
\include{c340}
%\include{c341}
\include{c342}
\include{c343}
\include{c344}
\include{c345}
\include{c346}
\include{c347}
\include{c348}
\include{c349}
%\include{c351}

%%     D part     %%

\Sectitle{Integration, Minimization, Non-linear Fitting}

\include{d101}
\include{d103}
\include{d104}
\include{d105}
\include{d107}
\include{d108}
\include{d110}
\include{d111}
\include{d113}
\include{d114}
\include{d115}
\include{d151}
\include{d201}
\include{d202}
\include{d203}
\include{d209}
\include{d300}
\include{d302}
\include{d401}
\include{d506}
\include{d507}
\include{d508}
\include{d509}
\include{d510}
\include{d601}
\include{d700}
\include{d701}
\include{d702}
\include{d703}
\include{d704}
%\include{d999}

%%     E part     %%

\Sectitle{Interpolation, Approximations, Linear Fitting}

\include{e100}
\include{e102}
\include{e103}
\include{e104}
\include{e105}
\include{e106}
\include{e207}
\include{e208}
\include{e211}
\include{e220}
\include{e221}
\include{e230}
\include{e250}
\include{e255}
\include{e401}
\include{e406}
\include{e407}
\include{e410}
%\include{e999}

%%     F part     %%

\Sectitle{Matrices, Vectors and Linear Equations}

\include{f001}
\include{f002}
\include{f003}
\include{f004}
\include{f010}
\include{f011}
\include{f012}
\include{f105}
\include{f106}
\include{f112}
\include{f116}
\include{f117}
\include{f118}
\include{f121}
\include{f122}
\include{f123}
\include{f150}
\include{f202}
\include{f220}
\include{f221}
\include{f222}
\include{f223}
\include{f224}
\include{f225}
\include{f230}
\include{f406}
\include{f500}
\include{f600}

%%     G part     %%

\Sectitle{Statistical Analysis and Probability}

\include{g100}
\include{g101}
\include{g102}
\include{g103}
\include{g104}
\include{g105}
\include{g106}
\include{g110}
\include{g111}
\include{g900}
\include{g901}
%\include{g999}

%%     H part     %%

\Sectitle{Operation Research Techniques and Management Science}

\include{h100}
\include{h300}

%%     I part     %%

\Sectitle{Input/Output}

\include{i101}
\include{i202}
\include{i302}
\include{i303}
%\include{i999}

%%     J part     %%

\Sectitle{Output and Graphical Data Presentation}

\include{j200}
\include{j401}
\include{j403}
\include{j509}
\include{j511}
\include{j530}
%\include{j551}
%\include{j999}

%%     K part     %%

%%     L part     %%

\Sectitle{Executive Routines}

\include{l210}
\include{l400}

%%     M part     %%

\Sectitle{Data Handling}

\include{m101}
\include{m103}
\include{m104}
\include{m107}
\include{m108}
\include{m109}
\include{m214}
\include{m215}
\include{m216}
\include{m218}
\include{m220}
\include{m224}
\include{m231}
\include{m232}
\include{m233}
\include{m250}
\include{m251}
\include{m400}
\include{m409}
\include{m410}
\include{m416}
\include{m421}
\include{m422}
\include{m423}
\include{m426}
\include{m427}
\include{m428}
\include{m429}
\include{m431}
\include{m432}
\include{m433}
\include{m434}
\include{m436}
\include{m437}
\include{m438}
\include{m439}
\include{m440}
\include{m441}
\include{m442}
\include{m501}
\include{m502}
\include{m503}
\include{m506}
\include{m507}
\include{m508}

%%     N part     %%

\Sectitle{Debugging, Error Handlng}

\include{n001}
\include{n002}
\include{n100}
\include{n103}
\include{n105}
\include{n203}

%%     Q part     %%

\Sectitle{Service or Housekeeping Programming Aids}

\include{q100}
\include{q120}
\include{q121}
\include{q122}
\include{q123}
\include{q124}
\include{q180}
\include{q210}
\include{q901}
\include{q902}
\include{q904}
%\include{q999}

%%     R part     %%

\Sectitle{Logical and Symbolic}

\include{r205}

%%     T part     %%

\Sectitle{Magnet and Beam Design, Electronics}

\include{t604}

%%     U part     %%

\Sectitle{Quantum Mechanics, Particle Physics}

\include{u100}
\include{u101}
\include{u102}
\include{u110}
\include{u501}
%\include{u999}

%%     V part     %%
\Sectitle{Random Numbers and General Purpose Utilities}

\include{v100}
\include{v101}
\include{v102}
\include{v103}
\include{v104}
\include{v105}
\include{v106}
\include{v107}
\include{v108}
\include{v109}
\include{v110}
\include{v111}
\include{v112}
\include{v113}
\include{v114}
\include{v130}
\include{v150}
\include{v151}
\include{v202}
\include{v300}
\include{v301}
\include{v302}
\include{v304}
\include{v306}
\include{v401}
\include{v700}
%\include{v999}

%%     W part     %%

\Sectitle{High Energy Physics Simulation, Kinematics, Phase Space}

\include{w150}
\include{w151}
\include{w505}
\include{w515}
%\include{w999}

%%     X part     %%

%\Sectitle{Particle Detection, Measurement, Reconstruction}

%\include{x999}

%%     Y part     %%

\Sectitle{Statistical Data Analysis and Presentation}

\include{y201}
\include{y250}
\include{y251}
%\include{y999}

%%     Z part     %%

\Sectitle{Miscellaneous System-Dependent Facilities}

\include{z001}
\include{z007}
\include{z008}
\include{z009}
\include{z020}
\include{z029}
\include{z034}
\include{z035}
\include{z036}
\include{z037}
\include{z041}
\include{z042}
\include{z044}
\include{z100}
\include{z203}
\include{z204}
\include{z262}
\include{z264}
\include{z265}
\include{z267}
\include{z300}
\include{z301}
\include{z303}
\include{z304}
\include{z305}
\include{z306}
\include{z307}
\include{z308}
\include{z309}
\include{z310}
\include{z311}
\include{z312}

%  ==================== Backmaterial ===========================
\bibliographystyle{myunsrt} % style for bibliography
\bibliography{/user/goossens/cnasall/cnasbibl}   % Master BibTeX file for CNAS docs

\input{\jobname.ind} % index

\end{document}


\documentclass[a4paper]{article}
\title{Implementation of Gouraud shading in GLSL}
\author{Hiroka IHARA (The Univ. of Tokyo)}
\date{2016/10/05}
\usepackage{bm}
\begin{document}
\maketitle
\section{Basics for per-vertex shading}
\[
I_{diffuse} = k_{diffuse} \cdot I_{light} \cdot max \left( \bm{N} \cdot \bm{L}, 0 \right)
\]
where $\bm{N}$ is the normalized vertex normal in viewspace, and $\bm{L}$ is the normalized viewspace vector pointing to the light.
\[
I_{ambient} = k_{ambient} \cdot I_{light}
\]
\[
I_{specular} = k_{specular} \cdot I_{light} \cdot max \left( \bm{R} \cdot \bm{V}, 0 \right)^{k_{reflectivity}}
\]
where $\bm{R}$ is calculated as $2\bm{N}\left(\bm{N}\cdot\bm{L}\right)-\bm{L}$ or simply $reflect\left(\bm{N},-\bm{L}\right)$,
and $\bm{V}$ is the normalized viewspace vector pointing to the view, which is equivalent to the negated projection vector.
\end{document}


\chapter{A LIBRARY FOR LOCAL DIFFERENTIAL PRIVACY}

\section{Introduction in Local DP}

As we mentioned in previous chapters, there are two major forms of Differential Privacy. Having analyzed and tested the first one, \emph{Global D.P.}, it is now time to examine \emph{Local D.P.}, by explaining some possible protocols, as well as building our own.

In Local D.P., there is a significant difference compared to Global DP: there is \emph{no trusted curator} between the data and the users, as they just want to send their data, while already being anonymized. Thus, an algorithm must perturb the data before sending it to the untrusted curator, who will then transmit it to the analysts.

In order to achieve that goal, the user must randomize the value before making it public (i.e. sending it to the untrusted curator). Then, the curator which collects the data (we will reference to him as aggregator moving forward), collects the data and tries to retrieve their original values, with a goal of producing the most accurate results possible.

Thus, each LDP algorithm has the following steps:

\begin{itemize}
    \item Each user encodes, and then perturbs the private value that he wants to make public
    \item Each user sends out the result of the perturbation process, with that being only the final value, as they keep the intermediate results for themselves
    \item The untrusted data curator collects each user's value, and implements some kind of aggregation in order to retrieve the stats that he wants from the data given to him.
\end{itemize}

In comparison with Global D.P., the Local model has advantages, as well as disadvantages.
Its main advantages are:
\begin{itemize}
    \item The user is not forced to trust the data curator, as only the perturbed value is reported
    \item Simpler implementation of the algorithms, due to the district steps taken by both sides.
\end{itemize}

while the main disadvantages are the following:

\begin{itemize}
    \item The noise added should be larger than the Global model, in order to satisfy the definition, thus the number of people in the dataset should be significant for accurate results to be produced.
    \item Because this is not always possible, many real-world applications use extremely high values of epsilon compared to what we got used to during our testing in the Global models.
\end{itemize}

During this Thesis, concern was raised for the main disadvantage of L.D.P., and thus\emph{ we will present a new protocol aiming to reduce the need for many users, while still covering the definition.} However, the definition for L.D.P. is quite different than the Global model one's.

\section{Definition of Local DP}

Having a general idea in how Local D.P. functions, it is now time to give a strict definition that we are going to depend our work on moving forward.

We can say that an algorithm $A$ satisfies ε-Local Differential Privacy, if and only if for any input $v_1$, $v_2$, we have

$$ \forall y \in Range(A):\ Pr[A(v_1) = y] \leq e^{\epsilon} * Pr[A(v_2) = y] $$

where $Range(A)$ denotes the set of all possible outputs of the algorithm $A$.

As mentioned in Chapter 2, this definition can have many interpretations by different algorithms or protocols, but each one must produce a probabilistic space whose elements must satisfy the above equation.

\section{Simple Application of LDP}

The most simple of L.D.P. protocols is already mentioned in this Thesis, and is no other than the \emph{Randomized Response} protocol. This algorithm implements the three steps mentioned in the introduction, as the user chooses a value (Yes or No), perturbs it (by the flipping of the coins), reports the perturbed value, with the sole job of the aggregator being to collect, normalize and report the values provided. It meets the definition of L.D.P., as the fraction of a pair of probabilities in the space of possible outputs (Yes, No) has always the ceiling of a real number.

Our goal is to now find this ceiling, and thus denote the level of privacy that randomized response offers. In order to do this, we are going to select the possibility of the user having chose the answer "Yes". A simple case analysis shows that $Pr[Yes | Truth] = \frac{3}{4}$, and of course $Pr[Yes | False] = \frac{1}{4}$. Thus, by the definition of L.D.P., we have

\begin{align*}
    \frac{Pr[Yes | Truth]}{Pr[Yes | False]} = \frac{\frac{3}{4}}{\frac{1}{4}} = 3 = e^\epsilon \Longleftrightarrow \epsilon = ln(3)
\end{align*}

Thus, R.R. offers  $ln(3)$-differential privacy to its users. This is quite a good setting, but the restriction is that the user can only report 2 values, something not suitable for modern problems and surveys.

In R.R., we care about the total true answers of the users, and not the individual responses. Thus the metric we are going to use is the \emph{absolute difference of the sum of the 2 vectors: the one with the truthful answers, and the one with the reported answers}. We are going to divide this result with the number of the users, in order to get the scale of the error depending on the size of the vector that was reported. The metric is expressed from the following function:

\begin{align*}
    \text{Error} = \frac{|\sum \text{true\_values} - \sum \text{reported\_values}| }{\text{number\ of \ users}}
\end{align*}

As always, during the creation of probabilistic distributions, one run is not enough, because of the extreme amount of noise that can occur. Thus, for each number of users we are going to run the R.R. protocol 100 times, and the final accuracy error will be produced by the mean value of those runs.

Having implemented R.R. in Python, we can now display the accuracy error of R.R. as the number of users rises. The results of the testings are shown bellow in \textbf{Figure 4.1}.

\begin{figure}[!htb]\centering
    \includegraphics[width=0.8\textwidth]{images/rr_results.png}
    \caption{Accuracy Error in R.R for increasing values of epsilon}
\end{figure}

We observe that the plot behaves as expected: the protocol produces a logarithmic curve for the accuracy error, while for a large number of users (over 3000), the error stabilizes bellow $0.1$.


\documentclass[12pt,a4paper]{article}

\usepackage{amsmath}
\usepackage[font=footnotesize]{caption}
\usepackage[section]{algorithm}
\captionsetup[algorithm]{font=footnotesize}
\usepackage[numbered]{algo}

% Following packages only required for this sample, not for using algo.sty in the first place!
\usepackage{color}
\usepackage{listings}
\lstset{language=TeX}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{dkgray}{rgb}{0.25,0.25,0.25}

\lstset{%
  	backgroundcolor=\color{white},
  	basicstyle=\footnotesize,
  	breakatwhitespace=false,
  	breaklines=true,
  	captionpos=b,
  	commentstyle=\color{dkgreen},
  	deletekeywords={...},
  	escapeinside={\%*}{*)},
  	extendedchars=true,
  	frame=single,
  	keepspaces=true,
  	keywordstyle=\color{blue},
  	language=Octave,
  	morekeywords={*,...},
  	numbers=left,
  	numbersep=5pt,
  	numberstyle=\tiny\color{dkgray},
  	rulecolor=\color{dkgray},
  	showspaces=false,
  	showstringspaces=false,
  	showtabs=false,
  	stepnumber=1,
  	tabsize=2,
  	title=\lstname
}

\begin{document}

	\section{Quick Shift}

	To use the \lstinline!alog.sty! package for a book, update the following lines of \lstinline!algo.sty!:

\begin{lstlisting}
% Set counter to include chapter:
% \renewcommand{\thealgorithm}{\thechapter .\arabic{algo}}
\renewcommand{\thealgorithm}{\thesection .\arabic{algo}}
\end{lstlisting}

	\begin{algorithm}[h]
		\begin{algo}{QS}{\label{algo:related-work-qs}\qinput{color image $I$}\qoutput{superpixel segmentation $S$}}
			\qfor $n = 1$ \qto $N$\\
				initialize $t(x_n) = \boldsymbol 0$\qrof\\
			\qfor $n = 1$ \qto $N$\\
				\qcom{$N_R(x_n)$ is the set of all pixels in the local neighborhood of size $R \times R$ around pixel $x_n$:}\\
				calculate $p(x_n) = \sum_{x_m \in N_R(x_n)} \exp\left(\frac{-d(x_n,x_m)^2}{(2/3) R}\right)$ \qrof\\
			\qfor $n = 1$ \qto $N$\\
				set $t(x_n) = \arg\max_{x_m \in N_R(x_n): p(x_m) > p(x_n)} \{p(x_m)\}$\qrof\\
				\qcom{$t$ maps each pixel to its neighbor $x_m$ with highest $p(x_m)$ if $p(x_m) > p(x_n)$;}\\
			\qcom{$t$ can be interpreted as forest, where all pixels $x_n$ with $t(x_n) = \boldsymbol 0$ are roots.}\\
			derive superpixel segmentation $S$ from $t$\\
			\qreturn $S$
		\end{algo}
		\caption{The superpixel algorithm \textbf{QS} proposed in \cite{QuickShift}.}
		\label{fig:related-work-qs-algorithm}
	\end{algorithm}

	\begin{thebibliography}{1}
		\bibitem{QuickShift}
		A. Vedaldi,
		S. Soatto,
		\emph{Quick shift and kernel methods for mode seeking},
		ECCV,
		2008.
	\end{thebibliography}

\end{document}


%%%
%%% Statistics
%%%
\part{Statistics}
\begin{frame}
\thispagestyle{empty}
\textbf{\huge{Statistics}}
\end{frame}

\begin{frame}{Statistics Contents}
 \tableofcontents
\end{frame}

\section{Statistic}
\begin{frame}[fragile]{Statistic I}
Arithmetic mean and some additional statistics \index{Mean!mean} \index{Describe!summarize} \index{Describe!detail} \index{Mean} \index{Statistic}
\begin{lstlisting}
  mean height
  summarize height
  summarize height, detail
\end{lstlisting}

  \begin{tikzpicture}[transform shape, rotate=10, overlay]
\node at (8,-1.5) [mybox] (box) {%
    \begin{minipage}[t!]{0.35\textwidth}
    \tiny\textcolor{black}{\texttt{mean returns the standard error: $s/\sqrt{n}$. summarize returns the standard deviation: $\sqrt{\sum(x-\bar{x})^2)/n}$.}}
    \end{minipage}
    };
\end{tikzpicture}
\end{frame}

\begin{frame}[fragile]{Statistic II}
Minimum, maximum, arithmetic mean, median, number of observations and quartils. \index{Minimum} \index{Minimum!min} \index{Maximum} \index{Maximum!max} \index{Mean!mean} \index{Mean} \index{Mean!arithmetic mean} \index{Mean!Median} \index{Quantile} \index{tabstat} \index{Range} \index{Range!range} \index{Quantile!q}
\begin{lstlisting}
  tabstat height, statistic(min max mean median p50)
  tabstat height, statistic(min max range mean count q) by(county)
\end{lstlisting}

  \begin{tikzpicture}[transform shape, rotate=10, overlay]
\node at (8,-1.5) [mybox] (box) {%
    \begin{minipage}[t!]{0.35\textwidth}
    \tiny\textcolor{black}{\texttt{range = max - min}}
    \end{minipage}
    };
\end{tikzpicture}
\end{frame}

\begin{frame}[fragile]{Statistics III}
Standard deviation, standard error, variance und interquartil range. \index{Standard deviation} \index{Standard error} \index{Variance} \index{Interquartil range} \index{tabstat} \index{Standard deviation!sd} \index{Variance!var} \index{Standard error!sem} \index{Interquartil range!iqr} \index{Quantile!q} \index{Skewness} \index{Kurtosis} \index{Skewness!skewness} \index{Kurtosis!kurtosis}
\begin{lstlisting}
 tabstat height, statistic(sd sem var q iqr)
\end{lstlisting}
Skewness and kurtosis
\begin{lstlisting}
  tabstat height, statistics(skewness kurtosis)
\end{lstlisting}

  \begin{tikzpicture}[transform shape, rotate=10, overlay]
\node at (8,-1.5) [mybox] (box) {%
    \begin{minipage}[t!]{0.35\textwidth}
    \tiny\textcolor{black}{\texttt{iqr = q3 - q1}}
    \end{minipage}
    };
\end{tikzpicture}
\end{frame}

\section{Tables}
\begin{frame}[fragile]{Tables} \index{Table!tab} \index{Table!tab, summarize()} \index{Table}
We already had \texttt{summarize} and \texttt{tab}, let us combine them
\begin{lstlisting}
  tab county, summarize(height)
\end{lstlisting}
\end{frame}

\subsection{Cross tabs}
\begin{frame}[fragile]{Cross tabs} \index{Table!tab} \index{Table!tab1} \index{Table!tab2} \index{Table!cross tab}
Create a cross tab with
\begin{lstlisting}
  tab school county
\end{lstlisting}
And some more tables
\begin{lstlisting}
  ** tabs of each variable
  tab1 sex county school
  ** cross tabs of ab bc ac
  tab2 sex county school
\end{lstlisting}
\end{frame}

\subsection{Chi, V and Phi}
\begin{frame}[fragile]{Cross tabs II} \index{Table!cross tab} \index{Table!$\chi^2$} \index{Table!chi-square} \index{Table!chi} \index{Table!tab} \index{Table!Cramers V} \index{Table!tab exp} \index{Table!tab col} \index{Table!tab row}
Let us recapitulate Statistics I, we will do some tests \footnote{or see e.g. \textcite{Krebs10} oder \textcite{Agresti09}.}
\begin{lstlisting}
** chi^2
tab sex county, chi
** cramers v
tab sex county, V

** manual
tab sex county, exp col row
** phi and for 2x2 tabs V
di (1013*1002 - 925*1131) / (2144*1927*1938*2133)^(1/2)
** chi^2
di 4071 *(-.00753735)^2
** usual way to compute V
di (.23128021 / 4071 * (2 - 1) )^(1/2) // V
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Numlabel}
You can add numeric values to your variable labels \index{Label!numlabel}
\begin{lstlisting}
  ** numeric labels on for all variables
  numlabel _all, add
  ** and off
  numlabel _all, remove
\end{lstlisting}
\end{frame}

\section{Inference}
\begin{frame}[fragile]{T-Test} \index{T-Test!ttest} \index{T-Test}
And finally let us have a look at some test-statistics
\begin{lstlisting}
  ** compare
  tab sex, sum(bmi)
  ttest bmi, by(sex)
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{And much more to come \dots} \index{Regression!regress} \index{Regression!logit} \index{Regression!mlog}
Regression
\begin{lstlisting}
  regress
  logit
  mlog
\end{lstlisting}
\end{frame}

\begin{frame}
\thispagestyle{empty}
 To be continued.
\end{frame}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath, amssymb, amsthm}
\usepackage{geometry}
\usepackage{graphicx}

\geometry{top=1in, bottom=1in, left=1in, right=1in}

\title{Analysis of Music in Machine Learning}
\author{Eben Kadile}
\date{July 2020}

\begin{document}

\maketitle

\section{Objective}

We want to better understand the temporal and statistical structure of music, and how machine learning systems detect this structure and exploit it for music prediction and synthesis

\section{Regression}

Because fitting a linear regression model is a convex optimization problem, we would like to start by looking at the result of fitting the next set of notes in piano roll music based on the last set of notes.

\begin{figure}
    \centering
    \includegraphics{figures/regression_weights.png}
    \caption{Here we plot the weights of the trained 1-step logistic regression model. Red indicates positivity and blue indicates negativity. As can be inferred, only notes 27 through 75 appear in the dataset. If a note is played in one time step, it is likely to be played in the next, and the probability of a note far away from the current note being played is low.}
\end{figure}

\begin{figure}
    \centering
    \includegraphics{figures/regression_bias.png}
    \caption{Here we plot the bias of the trained 1-step logistic regression model. The color scale is different from that of the weights.}
\end{figure}

Using the information encoded in the weights and bias, we should be able to recover the fact that all the songs were in C major, but I don't know how to do that yet.

\section{Latent LDS}

One can initialize a latent LDS so that it forgets everything at every time step and behaves like the regression model. Hopefully, this initialization is good for finding a superior solution to the problem using the LDS. We can also use the weights of the trained latent LDS to initialize a tanh RNN. First, we investigate the various ways the hidden weights of the latent LDS can be initialized.

\begin{figure}
    \centering
    \includegraphics{figures/jsb_lds_losses.png}
    \caption{Loss of different latent LDS models compared to regression.}
\end{figure}

\begin{figure}
    \centering
    \includegraphics{figures/jsb_lds_accs.png}
    \caption{Accuracy of different latent LDS models compared to regression.}
\end{figure}

As we can see from these comparisons, scaled down critical (power-law) initialization performs the best while overfitting the least (having weights high in magnitude seems to encourage overfitting, especially for tanh RNNs).

We will plot the weights and hidden eigenvalues of this model.

\begin{figure}
    \centering
    \includegraphics{figures/scrit_lds_in_weights.png}
    \caption{Input weights of scaled down critical initialization.}
\end{figure}

\begin{figure}
    \centering
    \includegraphics{figures/scrit_lds_hid_weights.png}
    \caption{Hidden weights of scaled down critical initialization.}
\end{figure}

\begin{figure}
    \centering
    \includegraphics{figures/scrit_lds_out_weights.png}
    \caption{Output weights of scaled down critical initialization.}
\end{figure}

\begin{figure}
    \centering
    \includegraphics{figures/scrit_lds_eigs.png}
    \caption{Eigenvalues of hidden weight matrix after training of scaled down power law LDS.}
\end{figure}

\section{RNNs}

A tanh RNN can be initialized using the weights of the trained LDS. This increases performance noticeably, but in the case of some initializations it drastically increases overfitting.

\end{document}