\lispy/ is statically typed. This means type safety (i.e.\ that there are no discrepancies between expected types of values and actual types of values) is verified at compile time by analysing the source code. The language has  a small set of types and a limited (but simple) type system. The language has type inference for everything except function parameters, meaning it can determine the type of a value at compile time without relying on being explicitly told the type by the programmer.

\subsection{int and float}
There are both integer and rational numbers. The former are \texttt{int}s and the latter are \texttt{float}s (a fixed-point representation). The syntax for them is a bit laxer than it is in LISP 1.5 (see~\nameref{sec:syntax}). However, octal number literals are unsupported. There are also \texttt{float} constants for \texttt{inf} (floating-point positive infinity) and \texttt{nan} (``not a number'').

The behaviour of precision and overflow is implementation-defined. For the reference implementation, these types are implemented using Python's \texttt{int} and \texttt{float} types, so they are subject to the same limitations in \lispy/ as they are in Python. That further depends on the Python interpreter used to run the compiled code.

\subsection{bool}
The \texttt{bool} type is represented by the literal atoms \texttt{true} and \texttt{false}. This is a replacement for the \texttt{T} and \texttt{NIL} used by predicates in LISP 1.5. Having a single Boolean type makes it simple from a type checking perspective to implement predicates in \lispy/.

% TODO: mention that recursion is possible.
\subsection{func} \label{subsec:func}
A \texttt{func} is a function as described in~\nameref{subsec:simpleforms}. The fundamental way to create a function is with a special form know as a \textit{lambda expression}. The resulting function is called a \textit{lambda function}. The distinction from just \textit{function} is that a lambda function is not bound to any name. Thus, to evaluate a lambda function, a composed form has to be used. In fact, the use of a lambda expression in place of a function name is what was being alluded to briefly in the description of~\nameref{subsec:composedforms}. As with function names, the lambda expression is evaluated first in the form, and then the arguments are evaluated.

\begin{figure}[htp]
    \centering
    \begin{cminted}[autogobble=true, escapeinside=??]{lisp}
        (?\tikzmark{exp_start}?(lambda ?\tikzmark{param_start}?((?\tikzmark{atom_start}?x?\tikzmark{atom_end}? int) (y ?\tikzmark{type_start}?int?\tikzmark{type_end}?))?\tikzmark{param_end}? ?\tikzmark{form_start}?(div x y)?\tikzmark{form_end}?)?\tikzmark{exp_end}? ?\tikzmark{arg_start}?1 2?\tikzmark{arg_end}?)
    \end{cminted}
    \begin{tikzpicture}[
        remember picture,
        overlay,
        thick,
        font=\scriptsize,
        every node/.style={pos=0.5, black}
    ]
        \draw[decorate, decoration={calligraphic brace, mirror, raise=0.25em, amplitude=1pt}]
            (pic cs:atom_start) -- (pic cs:atom_end)
            node[below=0.5em]{atomic literal};

        \draw[decorate, decoration={calligraphic brace, mirror, raise=0.25em, amplitude=3pt}]
            (pic cs:type_start) -- (pic cs:type_end)
            node[below=0.5em]{type};

        \draw[decorate, decoration={calligraphic brace, mirror, raise=1.75em, amplitude=3pt}]
            (pic cs:param_start) -- (pic cs:param_end)
            node[below=2em]{parameters};

        \draw[decorate, decoration={calligraphic brace, mirror, raise=0.25em, amplitude=3pt}]
            (pic cs:form_start) -- (pic cs:form_end)
            node[below=0.5em]{body};

        \draw[decorate, decoration={calligraphic brace, mirror, raise=0.25em, amplitude=3pt}]
            (pic cs:arg_start) -- (pic cs:arg_end)
            node[below=0.5em]{arguments};

        \draw[decorate, decoration={calligraphic brace, mirror, raise=3.25em, amplitude=5pt}]
            (pic cs:exp_start) -- (pic cs:exp_end)
            node[below=3.5em]{lambda expression};
    \end{tikzpicture}
    \captionsetup[figure]{font=small, skip=5em}
    \captionof{figure}{A lambda function being evaluated.}
\end{figure}

A function's evaluated value (the \textit{return value}) is the result of evaluating the function body with the function variables, which are bound to some values (the \textit{arguments}) prior to evaluation.

Function parameters are defined as pairs, of which there can be 0 or more. The first element of the pair is the name of the parameter. The second element is the type of that parameter. All parameters must be specified with their types; they are never type-inferred. Note that neither element of this pair is evaluated (this is possible because a lambda expression is a special form).

Unlike parameters, the return value and arguments \textit{will} be type inferred. In fact, there is no mechanism in the language for explicitly specifying the type of a return value or argument.

\subsubsection{First-class Functions}
Functions are \textit{first-class citizens} in the language. This means that, like other values, a function can be passed as an argument to another function, can be returned from other functions, and can be assigned to variables. As a consequence, \textit{higher-order functions} are supported, which are functions that takes a function as an argument and/or return a function.

\begin{figure}[htp]
    \centering
    \begin{cminted}[autogobble=true]{lisp}
        (lambda ((f (func (int int) int))
                 (i int))
          (f 3 i)
        )
    \end{cminted}
    \captionsetup[figure]{font=small}
    \captionof{figure}{A higher-order function that takes a function \texttt{f} as an argument. \texttt{f} has two \texttt{int} parameters and an \texttt{int} return value.}
\end{figure}

\paragraph{Special Forms}
Unlike regular functions, special forms are not first-class citizens. However, this can be worked around by defining a lambda function which ``wraps'' the special form.

\begin{figure}[htp]
    \centering
    \begin{cminted}[autogobble=true]{lisp}
        ((lambda ((f (func (int int) int))
                 (i int))
          (f 3 i))
        (lambda ((a int) (b int)) (prod a b)) 2)
    \end{cminted}
    \captionsetup[figure]{font=small}
    \captionof{figure}{A lambda function wrapping a special form and being passed to another function. This is effectively doing $3 \times 2 = 6$.}
\end{figure}

\subsubsection{Named Functions}
Lambda functions are not inherently bound to any variable. However, it can be quite useful to define a function so it can be re-used later. Since lambda functions are first-class citizens, it is valid to assign a lambda expression to a variable.

One way to do this has actually already been shown: rely on the binding mechanism of a lambda expression. A lambda can be bound to another lambda's parameters, and then the lambda will be accessible via a variable within the other lambda. However, this is tedious. A more convenient way to define functions with names is to use the \texttt{let} special form. This will be discussed in~\nameref{sec:bindingassign}.

\subsection{list}
The \texttt{list} type stores a homogenous (i.e.\ all of the same type) sequence of elements. Lists always have \texttt{nil} as the final element.

Lists can be created with the special form \texttt{cons}. For example, \texttt{(cons 1 (2 (3 nil)))} creates the list \texttt{(1 2 3)} (the presence of \texttt{nil} is implicit). Notice that the second argument must be a list which holds elements that have the same type as the first argument. Alternatively, the special form \texttt{list}, which takes an indefinite number of arguments, can create the same list: \texttt{(list 1 2 3)}.

The first element of a list can be retrieved using the special form \texttt{car}. For example, \texttt{(car (list 1 2 3))} returns the value \texttt{1}. Conversely, \texttt{cdr} can be used to retrieve the remaining list: \texttt{(cdr (list 1 2 3))} returns the value \texttt{(2 3)}. Both \texttt{car} and \texttt{cdr} only accept one argument, which must be a \texttt{list}.

\begin{figure}[htp]
    \centering
    \begin{cminted}[autogobble=true]{lisp}
        (lambda ((n int) (l (list int)))
          (prod n (car l)))
    \end{cminted}
    \captionsetup[figure]{font=small}
    \captionof{figure}{A lambda with a parameter that's a \texttt{list} of \texttt{int}s.}
\end{figure}

\subsubsection{nil}
The literal atom \texttt{nil} represents an empty list. Thus, it also has a type of \texttt{list}. In fact, an alternative way to represent \texttt{nil} is \texttt{()}. For \texttt{nil}, \texttt{car} and \texttt{cdr} are unsupported -- attempting to do so will result in a runtime error.

\texttt{nil} is a built-in ``global'' variable that's accessible everywhere. However, it's special as far as variables go because it cannot be assigned a new value.


\documentclass[letterpaper,final,12pt,reqno]{amsart}

\usepackage[total={6.3in,9.2in},top=1.1in,left=1.1in]{geometry}

\usepackage{times,bm,bbm,empheq,fancyvrb,graphicx,amsthm,amssymb}
\usepackage[dvipsnames]{xcolor}
\usepackage{longtable}
\usepackage{booktabs}

\usepackage{tikz}
\usetikzlibrary{decorations.pathreplacing}

\usepackage[kw]{pseudo}
\pseudoset{left-margin=15mm,topsep=5mm,idfont=\texttt}

\usepackage{float}

% hyperref should be the last package we load
\usepackage[pdftex,
colorlinks=true,
plainpages=false, % only if colorlinks=true
linkcolor=blue,   % ...
citecolor=Red,    % ...
urlcolor=black    % ...
]{hyperref}

\renewcommand{\baselinestretch}{1.05}

\allowdisplaybreaks[1]  % allow display breaks in align environments, if they avoid major underfulls

\newtheoremstyle{cstyle}% name
  {5pt}% space above
  {5pt}% space below
  {\itshape}% body font
  {}% indent amount
  {\itshape}% theorem head font
  {.}% punctuation after theorem head
  {.5em}% space after theorem head
  {\thmname{#1}\thmnumber{ #2}\thmnote{ (#3)}}% theorem head spec
\theoremstyle{cstyle}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{assumptions}[theorem]{Assumptions}

\newtheoremstyle{cstyle*}% name
  {5pt}% space above
  {5pt}% space below
  {\itshape}% body font
  {}% indent amount
  {\itshape}% theorem head font
  {.}% punctuation after theorem head
  {.5em}% space after theorem head
  {\thmname{#1}}% theorem head spec
\theoremstyle{cstyle*}
\newtheorem{assumptions*}{Assumptions}

\newtheoremstyle{dstyle}% name
  {5pt}% space above
  {5pt}% space below
  {}%{\itshape}% body font
  {}% indent amount
  {\itshape}% theorem head font
  {.}% punctuation after theorem head
  {.5em}% space after theorem head
  {\thmname{#1}\thmnumber{ #2}\thmnote{ (#3)}}% theorem head spec
\theoremstyle{dstyle}

\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}

% numbering
\numberwithin{equation}{section}
\numberwithin{figure}{section}
\numberwithin{table}{section}
\numberwithin{theorem}{section}

\newcommand{\eps}{\epsilon}
\newcommand{\RR}{\mathbb{R}}

\newcommand{\grad}{\nabla}
\newcommand{\Div}{\nabla\cdot}
\newcommand{\trace}{\operatorname{tr}}

\newcommand{\hbn}{\hat{\mathbf{n}}}

\newcommand{\bb}{\mathbf{b}}
\newcommand{\be}{\mathbf{e}}
\newcommand{\bbf}{\mathbf{f}}
\newcommand{\bg}{\mathbf{g}}
\newcommand{\bn}{\mathbf{n}}
\newcommand{\br}{\mathbf{r}}
\newcommand{\bu}{\mathbf{u}}
\newcommand{\bv}{\mathbf{v}}
\newcommand{\bw}{\mathbf{w}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\bF}{\mathbf{F}}
\newcommand{\bV}{\mathbf{V}}
\newcommand{\bX}{\mathbf{X}}
\newcommand{\bxi}{\bm{\xi}}
\newcommand{\bzero}{\bm{0}}

\newcommand{\cK}{\mathcal{K}}
\newcommand{\cV}{\mathcal{V}}

\newcommand{\rhoi}{\rho_{\text{i}}}

\newcommand{\ip}[2]{\left<#1,#2\right>}

\newcommand{\mR}{R^{\bm{\oplus}}}
\newcommand{\iR}{R^{\bullet}}

\newcommand{\nn}{{\text{n}}}
\newcommand{\pp}{{\text{p}}}
\newcommand{\qq}{{\text{q}}}
\newcommand{\rr}{{\text{r}}}

\newcommand{\supp}{\operatorname{supp}}
\newcommand{\Span}{\operatorname{span}}

\begin{document}
\title[On multilevel constraint decomposition methods]{On multilevel constraint decomposition methods \\ for nonlinear variational inequalities}

\author{Ed Bueler}

\date{\today}

\begin{abstract} FIXME
\end{abstract}

\maketitle

%\tableofcontents

\thispagestyle{empty}
%\bigskip

\newfloat{pseudofloat}{t}{xyz}[section]
\floatname{pseudofloat}{Algorithm}

\section{Introduction} \label{sec:intro}

The goal of this paper is to generalize the constraint decomposition (CD) method of X.-C.~Tai \cite{Tai2003} to nonlinear VI problems to which it has not been previously applied.  The convergence of this method, for several types of decompositions, was proven by Tai for coercive variational inequality (VI) problems which arise from minimization of a convex functional over a convex set.  In its multilevel form the method has been shown to have optimal complexity for elliptic, linear obstacle problems \cite[Subsection 5.4]{Tai2003}; see also \cite[Theorem 4.6 and Algorithm 4.7]{GraeserKornhuber2009}.  We construct a new proof of convergence which removes the assumption that the problem arises from (constrained) optimization.  Furthermore we address its finite element (FE) implementation in nonlinear cases, and we demonstrate the resulting performance.

The algorithms and theory presented in \cite{Tai2003} are extended in four particular directions:
\renewcommand{\labelenumi}{\emph{(\roman{enumi})}}
\begin{enumerate}
\item We do not assume that the continuum VI problem arises from optimization of a scalar objective.  Nonetheless we can prove convergence in $H^1$ norm at the same rate as the original method converges in energy (Section \ref{sec:convergence}); % HOPE
examples are explored in Sections \ref{sec:vi} and \ref{sec:results}.

\item We make the observation that, if implemented appropriately, multilevel ``up-smoothing'' is intrinsically more efficient than ``down-smoothing'' (Section \ref{sec:multilevel}).  This observation seems to be new; compare the comments on V(1,0) and V(1,1) cycles in \cite{GraeserKornhuber2009,Tai2003}.  A strong preference for up-smoothing is, apparently, special to multilevel CD methods and does not arise in corresponding unconstrained problems.

\item We make the multilevel CD algorithm more practical by addressing efficient storage of intermediate quantities.  In Section \ref{sec:results} we show results from a full approximation storage (FAS; see \cite{Brandt1977}) implementation for nonlinear operators, one which avoids global (Newton) linearization; compare \cite{GraeserKornhuber2009}.

\item For problems of porous-media type the nonlinear operator $f$ is not coercive (Appendix A), so the convergence theory in \emph{(i)} does not apply.  However, a full-cycle implementation of the multilevel CD method is demonstrated numerically.  By ``freezing' the solution-dependent coefficient, the operator is approximated by a coercive operator, nonlinear in general, for the duration of the V-cycle.  The resulting scheme is highly-effective for doubly-nonlinear diffusion operators (Section \ref{sec:results}).
\end{enumerate}

The iterates from the CD method are always admissible, and thus the operator need only be defined for admissible states; Section \ref{sec:results} includes a nontrivial example.  Admissible-iterate methods should permit direct solutions of certain VI problems, such as fluid-layer dynamics problems \cite{Bueler2021conservation,JouvetBueler2012}, for which non-admissible methods, such as semi-smooth methods \cite{BensonMunson2006}, require unnatural modifications of the operator formula.  The full-cycle scheme in \emph{(iv)} above, which applies to non-coercive VIs, is also designed for these problems, which are characterized by the solution of an auxiliary PDE on a domain determined inside the VI residual evaluation.

Especially for the classical obstacle problem with a Laplacian operator, certain multilevel techniques are known to improve performance relative to the multilevel CD method \cite{GraeserKornhuber2009}.  These improved methods either track the active set in the discretization or modify the nodal basis functions, and in this sense they are discrete algorithms.  By contrast the CD method applies at the level of the continuum problem (Sections \ref{sec:cd} and \ref{sec:convergence}), and thus it is a more fundamental method than the truncated monotone multigrid method \cite{Kornhuber1994}, for example.  In any case, acceleration of our nonlinear multilevel CD method via active-set and/or basis-level manipulations represents a potential extension of the method here, though as a topic for future research.

% A BRIDGE TOO FAR:  In one example at the end of this paper (Section \ref{sec:resultsnonlocal}) we consider a nonlocal residual functional, that is, one which is not a partial differential operator.  Each evaluation of this functional requires the solution of a Stokes problem for a layer of fluid \nocite{SayagWorster2013} on a substrate (which forms the obstacle), and thus the corresponding FE operator discretization is also not sparse.  In this case we cannot prove coercivity but we nonetheless succeed in demonstrating near-optimal complexity of the Section \ref{sec:multilevel} multilevel CD algorithm in practice.

\section{Coercive variational inequalities} \label{sec:vi}

Suppose $\cV$ is a real, reflexive Banach space with norm $\|\cdot\|$ and topological dual space $\cV'$.  Denote the dual pairing of $\phi \in \cV'$ and $v\in\cV$ by $\ip{\phi}{v} = \phi(v)$, and note that $\|\phi\|_{\cV'} = \sup_{\|v\|=1} |\ip{\phi}{v}|$ defines a (Banach space) norm on $\cV'$.

Let $\cK \subset \cV$ be a nonempty closed and convex subset, the \emph{constraint set}; elements of $\cK$ are said to be \emph{admissible}.  For a continuous, but generally nonlinear, operator $f:\cK \to \cV'$ and \emph{source} $g\in \cV'$ we consider the following \emph{variational inequality} (VI) for the (exact) solution $u^*\in \cK$, if it exists:
\begin{equation}
\ip{f(u^*)}{v-u^*} \ge \ip{g}{v-u^*} \qquad \text{for all } v\in \cK. \label{eq:vi}
\end{equation}
Because $f$ is a (generally) nonlinear map, $g$ is not strictly needed when posing this VI, but its presence is helpful to the algorithms of Section \ref{sec:multilevel}.  (Note that by redefining $f$ we may take $g=0$.)

VI \eqref{eq:vi} generalizes the nonlinear system of equations $f(u^*)=g$ from a vector space $\cV$ to the constrained case (over $\cK$).  Informally, if we conceptualize the dual pairing as an inner product then \eqref{eq:vi} says that the angle between $f(u^*)-g$ and any arbitrary vector $v-u$ pointing from $u$ into $\cK$ is at most $90^\circ$.  That is, \eqref{eq:vi} says that $f(u^*)-g$ may not be zero but it points directly into $\cK$.  In particular, if $u^* \in \cK^\circ$ (interior) then \eqref{eq:vi} implies $f(u^*)=g$.  The following definitions are also standard \cite{KinderlehrerStampacchia1980}.

\begin{definition} A map $f:\cK \to \cV'$ is \emph{monotone} if
\begin{equation}
\ip{f(u)-f(v)}{u-v} \ge 0 \qquad \text{for all } u,v \in \cK, \label{eq:monotone}
\end{equation}
\emph{strictly monotone} if equality in \eqref{eq:monotone} implies $u=v$, and \emph{coercive} if there exists $w \in \cK$ so that
\begin{equation}
\frac{\ip{f(u)-f(w)}{u-w}}{\|u-w\|} \to +\infty \qquad \text{as } \|u\|\to +\infty. \label{eq:coercive}
\end{equation}
We say VI \eqref{eq:vi} is \emph{monotone} if $f$ is monotone, and likewise for strictly monotone and coercive. \end{definition}

It is well-known that if $f:\cK \to \cV'$ is continuous, monotone, and coercive then VI \eqref{eq:vi} has a solution \cite[Corollary III.1.8]{KinderlehrerStampacchia1980}, and also that the solution $u^* \in \cK$ is unique when $f$ is strictly monotone.  As in the calculus of variations \cite{Evans2010}, coercivity permits a compactness argument for unbounded sets $\cK$; recall that the bounded, closed subsets of a reflexive Banach space are weakly compact.  The condition of continuity can be weakened to only apply on finite-dimensional subspaces \cite{KinderlehrerStampacchia1980}, but the stronger condition will apply in our examples.

The coercive VIs solved in this paper satisfy a stronger inequality than \eqref{eq:coercive}, and thus they are well-posed.

\begin{definition}  Let $p>1$.  The map $f:\cK \to \cV'$ is \emph{$p$-coercive} if there exists $\kappa>0$ such that
\begin{equation}
\ip{f(u)-f(v)}{u-v} \ge \kappa \|u-v\|^p \qquad \text{for all } u,v \in \cK. \label{eq:pcoercive}
\end{equation}
\end{definition}

Note \cite{Tai2003} uses ``coercive'' for $2$-coercive.  It is easy to see that if $f$ is $p$-coercive then it is monotone, strictly monotone, and coercive, and thus the following result holds.

\begin{theorem}  \label{thm:viwellposed}  If $f:\cK \to \cV'$ is continuous and $p$-coercive (for $p>1$) then there exists a unique $u\in \cK$ solving VI \eqref{eq:vi}.
\end{theorem}

When $f$ is monotone, VI \eqref{eq:vi} generalizes the problem of minimizing a convex function over $\cK$.  Suppose $F:\cK \to \RR$ is lower semi-continuous and (G\^ateau) differentiable with continuous derivative $F':\cK \to \cV'$.  Then $F$ is convex if and only if $F'$ is monotone \cite[Proposition I.5.5]{EkelandTemam1976}.  Furthermore, Proposition II.2.1 in \cite{EkelandTemam1976} shows that if $F$ is convex then \eqref{eq:vi} holds for $f=F'$ and $g=0$ if and only if
\begin{equation}
u^* = \operatorname{arg-min}_{v\in\cK} F(v). \label{eq:minimization}
\end{equation}
The CD methods of Tai \cite{Tai2003} address problem \eqref{eq:minimization} under the hypothesis that $F'$ is coercive.

From now on $\Omega \subset \RR^d$ denotes a bounded, open set with smooth or piecewise-smooth (e.g.~polygonal) boundary.  Also, Sobolev spaces \cite{Evans2010} are denoted by $W^{k,p}(\Omega)$, for integer $k$ and $1\le p \le \infty$, with $W^{k,2}$ denoted by $H^k$.

The following example includes the classical obstacle problem for the linear Laplacian \cite{GraeserKornhuber2009} and the obstacle problem for the $p$-Laplacian \cite{ChoeLewis1991}.

\begin{example}  \label{ex:plaplacian}  Suppose $a\in L^\infty(\Omega)$ such that $a(x)\ge a_0$ a.e.~for some constant $a_0>0$, and $p\ge 2$.  For $u,v \in \cV = W^{1,p}_0(\Omega)$ define $f:\cV \to \cV'$ by
\begin{equation}
\ip{f(u)}{v} = \int_\Omega a(x) |\grad u|^{p-2} \grad u \cdot \grad v\,dx. \label{eq:plaplacian}
\end{equation}
Now, if $x,y\in\RR^d$ then $(|x|^{p-2} x - |y|^{p-2} y)\cdot (x-y) \ge 2^{2-p} |x-y|^p$ \cite[see Appendix A and references therein]{Bueler2021conservation}.  Thus it follows from the Poincar\'e inequality that
    $$\ip{f(u) - f(v)}{u-v} \ge 2^{2-p} a_0 \|\grad u - \grad v\|_p^p \ge 2^{2-p} a_0 C \|u-v\|^p$$
for some $C>0$, and thus $f$ is $p$-coercive.  (The map in \eqref{eq:plaplacian} is coercive if $1<p<2$, but the proof is somewhat different \cite[Theorem 4.4]{Bueler2021conservation}.  In Section \ref{sec:results} we need only the $p\ge 2$ case.)  For $g\in\cV'$ define
    $$F(u) = \int_\Omega \frac{a(x)}{p} |\grad u|^p\,dx - g(u).$$
Then $F'(u) = f(u) - g$, $F$ is a convex functional (since $f$ is coercive), and, for any closed and convex $\cK\subset \cV$, VI problem \eqref{eq:vi} for is equivalent to optimization problem \eqref{eq:minimization}.
\end{example}

However, not all VI problems arise from optimization.  We give two such examples next, first a coercive and linear advection-diffusion problem, and then a nonlinear porous-medium-type problem.  Each of these examples is important in applications.  The first is preceded by a lemma.

\begin{lemma}  \label{lem:advectionskew}  \cite{Elmanetal2014}\,  Suppose $\bX :\Omega \to \RR^d$ is a smooth and bounded vector field on $\Omega$ with zero divergence ($\Div \bX=0$).  For $u,v \in H^1(\Omega)$ let $b(u,v) = \int_\Omega (\bX \cdot \grad u) v\,dx$.  Then $b(u,u) = \frac{1}{2} \int_{\partial \Omega} u^2 \bX\cdot \bn\,dx$ where $\bn$ is the outward normal on $\partial \Omega$.
\end{lemma}

\begin{proof}
Integration by parts gives $b(u,v) = - b(v,u) + \int_{\partial \Omega} uv \bX\cdot \bn\,dx$, so the result follows.
\end{proof}

\begin{example}  \label{ex:advectiondiffusion}  Suppose $\partial\Omega$ is partitioned into Dirichlet and Neumann portions, i.e.~$\partial\Omega = \partial_D\Omega \cup \partial_N\Omega$, with $\partial_D\Omega$ of positive measure.  Let $\cV = H_0^1(\Omega)$ be the space of functions with value zero on $\partial_D\Omega$.  (For simplicity we consider zero boundary data.)  Consider a smooth and bounded velocity field $\bX$ on $\Omega$ such that $\Div \bX=0$, but additionally assume that the flow is outward on the Neumann boundary, i.e.~$\bX \cdot \bn \ge 0$ at points of $\partial_N\Omega$.  For $u,v \in \cV = H_0^1(\Omega)$ and $\eps>0$ define
\begin{equation}
\ip{f(u)}{v} = \int_\Omega \eps \grad u \cdot \grad v - (\bX \cdot \grad u) v\,dx. \label{eq:advectiondiffusion}
\end{equation}
Consider VI \eqref{eq:vi} for any closed and convex $\cK \subset \cV$ and any $g\in\cV'$.  It is easy to see that $|\ip{f(u)}{v}| \le (\eps + \|\bX\|_\infty) \|u\| \|v\|$, thus that $f:\cK \to \cV'$ is continuous (and Lipschitz).  On the other hand, Lemma \ref{lem:advectionskew} says that $s(u,v)$ is skew-symmetric up to a nonnegative term.  However, by the outward flow assumption and the Poincar\'e inequality,
\begin{align*}
\ip{f(u)-f(v)}{u-v} &= \eps \int_\Omega |\grad u - \grad v|^2\,dx + b(u-v,u-v) \\
                    &= \eps \int_\Omega |\grad u - \grad v|^2\,dx + \frac{1}{2} \int_{\partial_N\Omega} (u-v)^2 \bX\cdot\bn \ge \eps C \|u-v\|^2.
\end{align*}
Thus $f$ is 2-coercive, and so VI problem \eqref{eq:vi} is well-posed for any closed and convex $\cK \subset \cV$ and $g\in \cV'$.
\end{example}

If $\bX \ne 0$ then VI \eqref{eq:vi} for $f$ in \eqref{eq:advectiondiffusion} is not a (constrained) minimization problem.  Indeed, the Jacobian of $f$, namely the bilinear form $J_f(u,v) = \int_\Omega \eps \grad u \cdot \grad v\,dx - b(u,v)$, is not symmetric in that case.\footnote{Assume $\bX \ne 0$ is continuous for simplicity.  For $u,v$ which are zero on $\partial \Omega$, $J_f(u,v) - J_f(v,u) = -2 b(u,v)$.  There exist $u,v$ so $b(u,v)$ is nonzero; construct $u,v$ locally near some point where $\bX$ is nonzero.}  Thus $f$ in \eqref{eq:advectiondiffusion} cannot be the Hessian of a scalar objective $F$.  References \cite{Bueler2021conservation,ChangNakshatrala2017} consider such advection-diffusion VI problems over $\cK = \{v\ge 0\}$.

\begin{example}  \label{ex:porous}  FIXME
\begin{equation}
\ip{f(u)}{v} = \int_\Omega \phi(u) \grad u \cdot \grad v\,dx \label{eq:porous}
\end{equation}
\end{example}

Numerical solver performance for Examples \ref{ex:plaplacian}, \ref{ex:advectiondiffusion}, and \ref{ex:porous} will be considered in Section \ref{sec:results}.  Further examples of nonlinear VI problems appear in ice sheet models \cite{Calvoetal2002,JouvetBueler2012} and other geophysical fluids \cite{Bueler2021conservation}.

\section{Constraint decomposition, the basic algorithm} \label{sec:cd}

Suppose there are $m<\infty$ closed subspaces $\cV_i \subset \cV$ so that the sum
\begin{equation}
\cV = \sum_{i=0}^{m-1} \cV_i \label{eq:subspacedecomp}
\end{equation}
holds in the sense that if $w \in \cV$ then there exist $w_i \in \cV_i$ so that $w = \sum_i w_i$; this is called a \emph{subspace decomposition} \cite{Xu1992}.  Suppose further that $\cK_i \subset \cV_i$ are nonempty, closed, and convex subsets such that
\begin{equation}
\cK = \sum_{i=0}^{m-1} \cK_i. \label{eq:constraintdecomp}
\end{equation}
The sum in \eqref{eq:constraintdecomp} is required to hold in two senses: \emph{(i)}~if $w \in \cK$ then there exist $w_i \in \cK_i$ so that $w = \sum_i w_i$, and \emph{(ii)}~if $z_i \in \cK_i$ for each $i$ then $\sum_i z_i \in \cK$.\footnote{Sense \emph{(ii)} for \eqref{eq:subspacedecomp} is automatic because the $\cV_i$ are subspaces.}

Note that neither decomposition \eqref{eq:subspacedecomp} or \eqref{eq:constraintdecomp} is required to be unique.  Also, $\cK_i \not\subset \cK$ in many applications; see the cartoon in Figure \ref{fig:cartoon}.

Finally, for each $\cK_i$ we assume that there are bounded, (generally) nonlinear restriction operators $R_i : \cK \to \cK_i$ such that if $v \in \cK$ then
\begin{equation}
v = \sum_{i=0}^{m-1} R_i v.  \label{eq:constraintrestrictionsum}
\end{equation}
A \emph{constraint decomposition} (CD) of $\cK$ is a choice of $\cV_i,\cK_i,R_i$ satisfying \eqref{eq:subspacedecomp}--\eqref{eq:constraintrestrictionsum} \cite{Tai2003}.

\begin{figure}[ht]
\includegraphics[width=0.55\textwidth]{genfigs/cartoon.pdf}
\caption{A constraint decomposition (CD) in a one-sided obstacle problem ($\mathcal{K}=\{v\ge \psi\}$) might look like this if $\mathcal{V}$ were 2-dimensional.}
\label{fig:cartoon}
\end{figure}

In Section \ref{sec:multilevel} we will introduce discretizations and describe practical algorithms, but the fundamental CD concept applies even at the level of the continuum problem.  The following two examples illustrate this for obstacle problems \cite{GraeserKornhuber2009}.  First we consider an overlapping domain decomposition.

\begin{example}  \label{ex:domaindecomposition}  Consider a bounded domain $\Omega \subset \RR^d$, let $\cV = W_0^{k,p}(\Omega)$ for $k\ge 0$ and $p\ge 1$, and suppose the obstacle $\psi \in W^{k,p}(\Omega)$ satisfies $\psi|_{\partial \Omega} \le 0$.  Let $\cK = \{v \ge \psi\} \subset \cV$.  Suppose further that $\{\phi_i\}_{i=0}^{m-1}$ is a smooth partition of unity on $\Omega$, satisfying $0 \le \phi_i\le 1$ and $\sum_i \phi_i = 1$, and let $\Omega_i$ be the support of $\phi_i$.  Let $\cV_i = \{w \in \cV:w|_{\Omega \setminus \Omega_i} =0 \}$, $\cK_i = \{v \in \cV_i: v \ge \phi_i \psi\}$, and $R_i(v) = \phi_i v$.  Then \eqref{eq:subspacedecomp}, \eqref{eq:constraintdecomp}, and \eqref{eq:constraintrestrictionsum} all hold.
\end{example}

Our second example is a disjoint frequency decomposition.  A multilevel FE CD, e.g.~the one proposed in Section \ref{sec:multilevel}, approximates such a frequency decomposition.

\begin{example}  \label{ex:frequencydecomposition}  For simplicity suppose $\Omega = (0,a)^d \subset \RR^d$ is a cube, and let $\cV = H_{\text{per}}^k(\Omega)$, $k\ge 0$, be the periodic functions.  Suppose $\psi \in H_{\text{per}}^k(\Omega)$ and let $\cK = \{v \ge \psi\} \subset \cV$.  Without using any detailed notation for Fourier representation, but noting that the frequencies are discrete, suppose $\{\cV_i\}$ are $m<\infty$ subspaces of $\cV$ defined by an (nonoverlapping) partition by frequency, thus satisfying \eqref{eq:subspacedecomp} as an orthogonal decomposition.  Suppose $P_i:\cV \to \cV_i$ are the corresponding orthogonal projections, satisfying $I = \sum_i P_i$.  Let $\cK_i = \{v \ge P_i \psi\} \subset \cV_i$ and $R_i = P_i$.  Then \eqref{eq:constraintdecomp} and \eqref{eq:constraintrestrictionsum} also hold.
\end{example}

Note that $\cK_i \not\subset \cK$ in most cases.  In Example \ref{ex:domaindecomposition}, if $\psi$ is positive over portions of $\Omega$ where the decomposition into overlapping subdomains $\Omega_i$ is nontrivial, then $\cK_i \not\subset \cK$, and similarly for Example \ref{ex:frequencydecomposition}.  Instead, the important inclusion is $\cK_i \subset \cV_i$.

Algorithm \ref{alg:basiccd} below states the basic CD algorithm as an iteration which solves smaller VI problems over each set $\cK_i$.  (By contrast, the algorithms in \cite{Tai2003} solve optimization problems over each $\cK_i$.)  The algorithm starts from a current iterate $u \in \cK$ and computes a new iterate $w\in\cK$, an improved approximation of the solution $u^* \in \cK$ of \eqref{eq:vi}, using a damping parameter $0<\alpha\le 1$.  There are parallel (additive) and successive (multiplicative) versions of the algorithm, generalizing the Jacobi and Gauss-Seidel iterations \cite{Greenbaum1997}, respectively.  For the parallel version the \textbf{for} loop can be computed in any order.

\begin{pseudofloat}[H]
\begin{pseudo*}
\pr{cditeration}(u\in\cK)\text{:} \\+
    if \pr{parallel}: \\+
        for $i \in \{0,\dots,m-1\}$: \\+
            $\hat w_i\in \cK_i$: \\+
                 $\boxed{\ip{f(u - R_i u + \hat w_i)}{v_i-\hat w_i} \ge g(v_i-\hat w_i)} \quad \forall v_i\in \cK_i$ \\---
    else: \\+
        for $i = 0,\dots,m-1$: \\+
            $\hat w_i\in \cK_i$: \\+
                $\displaystyle \boxed{\ip{f\Big(\sum_{j<i} w_j + \hat w_i + \sum_{j>i} R_j u\Big)}{v_i-\hat w_i} \ge g(v_i-\hat w_i)} \quad \forall v_i\in \cK_i$ \\-
            $w_i = (1-\alpha) R_i u + \alpha \hat w_i\in\cK_i$ \\--
    $\hat w = \sum_i \hat w_i\in\cK$ \\
    return $w=(1-\alpha) u + \alpha \hat w\in\cK$
\end{pseudo*}
\caption{The basic constraint decomposition (CD) algorithm for VI problem \eqref{eq:vi}.}
\label{alg:basiccd}
\end{pseudofloat}

The reader may confirm that, inside each boxed VI in Algorithm \ref{alg:basiccd}, the argument of $f$ is an element of $\cK$.  While the difference $v_i - \hat w_i \in \cV_i$ appears, note that by \eqref{eq:constraintdecomp} and \eqref{eq:constraintrestrictionsum} one may write it as a difference of admissible vectors (i.e.~from $\cK$), namely
\begin{align*}
[u - R_i u + v_i] - [u - R_i u + \hat w_i] &= v_i - \hat w_i, \label{eq:admissibledifference} \\
\left[\sum_{j<i} w_j + v_i + \sum_{j>i} R_j u\right] - \left[\sum_{j<i} w_j + \hat w_i + \sum_{j>i} R_j u\right] &= v_i - \hat w_i,  \notag
\end{align*}
for the two versions, respectively.

In other words, we solve VI \eqref{eq:vi} over perturbations from the sets $\cK_i$.  Define
\begin{equation}
e_i = \hat w_i - R_i u \in \cV_i \label{eq:ithupdate}
\end{equation}
as the $i$th update from a pass through either \textbf{for} loop in Algorithm \ref{alg:basiccd}.  Then $\hat w = u + \sum_{i} e_i$ and $w = u + \alpha \sum_i e_i$.  Easy identities $u - R_i u + \hat w_i = u + e_i$ and $\hat w = u^* + \sum_i \hat w_i - R_i u^*$ also hold.  We will use these identities in Section \ref{sec:convergence}.

The convergence results in Section \ref{sec:convergence} require substantial damping when applying the parallel algorithm, i.e.~$\alpha \le 1/m$, the same as in \cite{Tai2003}, but the successive version can be shown to converge without damping ($\alpha=1$).  In Section \ref{sec:results} we demonstrate practical convergence for a larger range of $\alpha$ than suggested by the theory.  % HOPE

In order for Algorithm \ref{alg:basiccd} to have a practical and efficient FE implementation (Section \ref{sec:multilevel}), we need additional notions when $f$ has any of the following properties: \emph{(i)} $f$ is defined only on $\cK$, or \emph{(ii)} $f$ is nonlinear, or \emph{(iii)} $f$ is non-local.  The issue of practical implementation for any such $f$ seems not to have been addressed in the CD literature, which is why we call Algorithm \ref{alg:basiccd} the ``basic'' algorithm.  In particular  \cite{GraeserKornhuber2009,Tai2003} only apply the CD method to the classical obstacle problem.  The following example explains the simplifications available in that easy case.

\begin{example}  \label{ex:fnice} Suppose $f:\cV \to \cV'$ is linear and defined on all of $\cV$.  Furthermore suppose $f$ is local in the sense that a basis $\{\phi_i\}$ of $\cV_i$ exists, with each support small in $\Omega$, such that $\ip{f(\phi_i)}{z}$ can be computed by an integral over the support of $\phi_i$.  (When $\cV$ is an FE space then this is the usual case for evaluating PDE weak forms over a basis of hat functions; see Section \ref{sec:multilevel}.  Constrast integral operators where $\ip{f(\phi_i)}{z}$ requires an integral over $\Omega$ even if the support of $\phi_i$ is small.)  Considering only the parallel version of Algorithm \ref{alg:basiccd} for simplicity, the boxed VI over $\cK_i$ can be written as
\begin{equation}
\ip{f(e_i)}{v_i-\hat w_i} \ge \tilde g(v_i-\hat w_i) \label{eq:linearlocalvi}
\end{equation}
where $\tilde g(z) = g(z) - \ip{f(u)}{z}$.  Noting $e_i = \hat w_i - R_i u \in \cV_i \notin \cK$, in general, VIs \eqref{eq:linearlocalvi} will only make sense because $f$ is defined over $\cV$.  Each problem \eqref{eq:linearlocalvi} can be solved using a stored residual $\ip{f(u)}{\cdot}$ (already included into the source term).  The VI \eqref{eq:vi} is then approximately solved in any incremental and efficient manner, by computations over the basis supports, during one application of \pr{cditeration}.
\end{example}

In other words, when $f$ has all the nice properties proposed in Example \ref{ex:fnice} then the decomposed VI problems can be computed from inexpensive data.  Note that the actual solution method for VI \eqref{eq:linearlocalvi} is not the concern here; we are observing that the data of problem \eqref{eq:linearlocalvi}, and the cost of residual evaluation, is small in practice.  A solver which would work for Example \ref{ex:fnice} can thus have implementation efficiencies which are unavailable in general.

Additional ideas, like those in Section \ref{sec:multilevel}, are needed for practical application of CD methods to more difficult problems.  In particular, we will extend the basic Algorithm \ref{alg:basiccd} to nonlinear $f$ by applying the full approximation storage (FAS) idea of Brandt \cite{Brandt1977}, in which case the decomposed problems only evaluate $f$ over (admissible) elements of $\mathcal{K}$, and we propose restrictions which allow inexpensive approximations of the decomposed problems.  However, extending the algorithm to non-local residual functionals is a topic for future research.

\section{Convergence of the basic algorithm} \label{sec:convergence}

We now seek a proof of the convergence of Algorithm \ref{alg:basiccd}.  This will be possible if we restrict to $2$-coercive operators and make certain assumptions as in \cite{Tai2003}.  First we define a new quantity related to VI problem \eqref{eq:vi}.

\begin{definition} Suppose $f:\cK \to \cV'$ and $g \in \cV'$.  For $u,v \in \cK$ let
\begin{equation}
  E(v,u) = \ip{f(v)}{v-u} - g(v-u).  \label{eq:normlikedefn}
\end{equation}
\end{definition}

If $u^*$ solves \eqref{eq:vi} then $E(v,u^*)$ is somewhat like an ``merit function,'' something used in the context of solving nonlinear equations \cite{NocedalWright2006} to replace an objective (scalar) functional.  (Note that $E(v,u^*)$ is \emph{not} a residual for \eqref{eq:vi}.)  The next lemma, which follows directly from $p$-coercivity, shows $E(v,u^*)$ is bounded below.

\begin{lemma} \label{lem:normlike}  Suppose $f:\mathcal{K} \to \mathcal{V}'$ is $p$-coercive and $u^* \in \mathcal{K}$ solves \eqref{eq:vi}.  For $v \in \mathcal{K}$,
\begin{equation}
  E(v,u^*) \ge \kappa \|v-u^*\|^p.  \label{eq:normlikebound}
\end{equation}
\end{lemma}

\begin{proof}
\begin{align*}
E(v,u^*) &= \ip{f(v)}{v-u^*} - \ip{f(u^*)}{v-u^*} + \ip{f(u^*)}{v-u^*} - g(v-u^*) \\
   &\ge \kappa \|v-u^*\|^p + 0.  \qedhere
\end{align*}
\end{proof}

Next we make two assumptions which are essentially the same as (7) and (8) in \cite{Tai2003}.

\begin{assumptions*}  There exists a constant $C_1>0$ so that
\begin{equation}
\left(\sum_{i=0}^{m-1} \|R_i u - R_i v\|^2\right)^{1/2} \le C_1 \|u-v\| \label{as:lipschitzrestrictions}
\end{equation}
for all $u,v\in\cK$.  Furthermore, there exists $C_2>0$ so that for all $v_i \in \cV_i$ and $y_j \in \cV_j$,
\begin{equation}
\sum_{i=0}^{m-1} \sum_{j=0}^{m-1} \left|\ip{f(w_{ij} + v_i) - f(w_{ij})}{y_j}\right| \le C_2 \left(\sum_{i=0}^{m-1} \|v_i\|^2\right)^{1/2} \left(\sum_{j=0}^{m-1} \|y_j\|^2\right)^{1/2} \label{as:lipschitzresidual}
\end{equation}
for all $w_{ij} \in \cK$ such that $w_{ij} + v_i \in \cK$ for all $i$ and $j$.
\end{assumptions*}

These inequalities might be described as ``totally Lipschitz'' requirements for the maps $R_i$ and $f$, respectively.  We make the following observations:
\begin{itemize}
\item Assumption \eqref{as:lipschitzrestrictions} addresses only the CD, and not the map $f$.
\item It is sometimes the case that \eqref{as:lipschitzresidual} addresses only a subspace decomposition property of $f$ (and not the CD itself).  For example, in the classical obstacle problem one may verify \eqref{as:lipschitzresidual} over all $w_{ij} \in \cV$.
\item The unconstrained versions of \eqref{as:lipschitzrestrictions} and \eqref{as:lipschitzresidual} are (13), (14) in \cite{TaiXu2002}, respectively.
\item If we assume $f$ itself is Lipschitz then the existence of $C_2$ is clear \cite{TaiXu2002}.  However, our bound for the convergence rate for Algorithm \ref{alg:basiccd} is improved when $C_1,C_2$ are made smaller.
\item Tai \cite{Tai2003} gives values of the constants $C_1$ in \eqref{as:lipschitzrestrictions} for obstacle problem constraint decompositions using $P_1$ FE spaces over shape-regular and quasi-uniform triangulations.  In particular, constants $C_1$ are known for overlapping domain decompositions (with or without an additional coarse mesh) and standard multilevel hierarchies.  Thus the cases we will demonstrate in Section \ref{sec:results} are already covered.
\item The constants $C_1,C_2$ generally depend on $m$.  However, regarding the convergence proof below, \cite{Tai2003} observes that if repeated application of Algorithm \ref{alg:basiccd} generates a bounded sequence of iterates in $\mathcal{K}$ then constants $C_1,C_2$ would be permitted to depend on $u,v,w_{ij},v_i,y_i$ as well.
\end{itemize}

From the above definitions and assumptions we have the following estimate for the parallel version of Algorithm \ref{alg:basiccd}.  Our proof follows \cite{Tai2003}.

\begin{lemma} \label{lem:core}  Suppose $f$ is $2$-coercive and $u^* \in \mathcal{K}$ solves \eqref{eq:vi}.  Recall $E(v,u^*)$ is defined in \eqref{eq:normlikedefn}.  Suppose $\mathcal{V}_i$, $\mathcal{K}_i$, $R_i$, and $f$ satisfy \eqref{as:lipschitzrestrictions} and \eqref{as:lipschitzresidual}.  Suppose $\hat w$ is computed from $u$ by the parallel version of Algorithm \ref{alg:basiccd}, and recall $e_i = \hat w_i - R_i u$.  Then
\begin{equation}
   E(\hat w,u^*) \le C_2 \sum_{i=0}^{m-1} \|e_i\|^2 + \kappa^{-1} C_1 C_2 \left(\sum_{i=0}^{m-1} \|e_i\|^2\right)^{1/2} E(u,u^*)^{1/2}. \label{eq:core}
\end{equation}
\end{lemma}

\begin{proof}  Recalling that $\hat w = u^* + \sum_i \hat w_i - R_i u^*$, by the (parallel version) boxed VI in Algorithm \ref{alg:basiccd} with $v_i = R_i u^*$ we get
\begin{align}
E(\hat w,u^*) &= \sum_{i=0}^{m-1} \ip{f(\hat w)}{\hat w_i - R_i u^*} - g(\hat w_i - R_i u^*) \label{eq:startcore} \\
    &\le \sum_{i=0}^{m-1} \ip{f(\hat w)}{\hat w_i - R_i u^*} + \ip{f(u + e_i)}{R_i u^* - \hat w_i} \notag \\
    &= \sum_{i=0}^{m-1} \ip{f(\hat w) - f(u + e_i)}{\hat w_i - R_i u^*}. \notag
\end{align}
For $i\in \{0,1,\dots,m-1\}$ use wrapped indices to define elements $\phi_{i,j}$:
\begin{align}
\phi_{i,0} &= u + e_i, \label{eq:gridcore} \\
\phi_{i,1} &= u + e_i + e_{i+1}, \notag \\
  &\vdots \notag \\
\phi_{i,m-1} &= u + e_i + e_{i+1} + \dots + e_{m-1} + e_0 + \dots + e_{i-1}, \notag
\end{align}
and $\phi_{i,j} \in \cK$ by \eqref{eq:constraintdecomp} and \eqref{eq:constraintrestrictionsum}.  Note that $\phi_{i,m-1} = u + \sum_i e_i = \hat w$.  From \eqref{eq:startcore}, apply a telescoping sum using the elements in \eqref{eq:gridcore}:
\begin{align}
E(\hat w,u^*) &\le \sum_{i=0}^{m-1} \sum_{j=1}^{m-1} \ip{f(\phi_{i,j}) - f(\phi_{i,j-1})}{\hat w_i - R_i u^*} \label{eq:nextcore} \\
  &\le \sum_{i=0}^{m-1} \sum_{j=0}^{m-1} \left|\ip{f(\phi_{i,j}) - f(\phi_{i,j-1})}{\hat w_i - R_i u^*}\right|. \notag
\end{align}
By an easy renumbering of the indices $i,j$ we may apply assumption \eqref{as:lipschitzresidual}, then the triangle inequality, and then assumption \eqref{as:lipschitzrestrictions}; we denote $\left(\sum_{i=0}^{m-1} \|e_i\|^2\right)^{1/2}$ by $Z$:
\begin{align}
E(\hat w,u^*) &\le C_2 Z \left(\sum_{i=0}^{m-1} \|\hat w_i - R_i u^*\|^2\right)^{1/2} = C_2 Z \left(\sum_{i=0}^{m-1} \|\hat w_i - R_i u + R_i u - R_i u^*\|^2\right)^{1/2} \label{eq:nextnextcore} \\
  &\le C_2 Z \left(Z + \left(\sum_{i=0}^{m-1} \|R_i u - R_i u^*\|^2\right)^{1/2}\right) \le C_2 Z \left(Z + C_1 \|u-u^*\|\right). \notag
\end{align}
Finally apply \eqref{eq:normlikebound} to give \eqref{eq:core}.
\end{proof}

FIXME NOW THE ENTIRE BATTLE IS TO GET A CONVEXITY RESULT $E(w,u) \ge C \sum_{j=0}^{m-1} \|e_j\|^2$ for some reasonable $C$

\section{Finite elements and multilevel constraint decomposition} \label{sec:multilevel}

In practice we will solve VI \eqref{eq:vi} over a finite-dimensional space $\cV$ based on a choice of a mesh over $\Omega$ and a finite element (FE) space.

FIXME triangulation

\begin{example}  FIXME if $\cV_i=\Span\{\phi_i\}$ for $\phi_i\in\cV$ are 1d spaces and if $\cK = \{v \ge \psi\} \subset \cV$ (obstacle problem), and if $R_i : \cK \to \cK_i$ for obstacle problem; note $\cK_i \not\subset \cK$ when $\psi>0$
\end{example}

FIXME state essentially Algorithm 4.7 \cite{GraeserKornhuber2009} but with $\text{V}(\nu_1,\nu_2)$ cycles which works for linear; observe that up-smoothing is more efficient; state the FAS version which has $O(m)$ residual evaluation complexity on each level

\section{Results for local variational inequalities} \label{sec:results}

FIXME

% A BRIDGE TOO FAR:  \section{Results for a nonlocal variational inequality} \label{sec:resultsnonlocal}

\small
\bibliography{mcd2}
\bibliographystyle{siam}

\normalsize
\appendix

\section{Porous-medium problems are never monotone}

Based on the examples in Section \ref{sec:vi}, the property of $p$-coercivity would seem to be a mild generalization of uniform ellipticity (including its quasi-linear meaning \cite[Section 8.3]{Evans2010}).  Indeed, weak-form operators with certain lower-order terms (Example \ref{ex:advectiondiffusion}) or degenerate quasi-linear type (Example \ref{ex:plaplacian}) can be monotone and coercive.  However, even the weaker property of monotonicity is ``fragile'' in a pointwise sense, as explained next, while ellipticity is more robust.  In particular, we show that no nontrivial porous-medium type functional is monotone, nor $p$-coercive.  We have not found any such explanation in the literature, but the observation of non-monotonicity in \cite{RocknerWang2008} is related.

Consider the expression $(Ax - By)\cdot (x-y)$ where $A,B \in \RR^{d\times d}$ are symmetric matrices and $x,y\in \RR^d$.  This expression is the quadratic form for a certain symmetric block matrix.  In fact, regarding $x,y$ as column vectors, we observe that
\begin{equation}
(Ax - By)\cdot (x-y) = \begin{bmatrix} x^\top & y^\top \end{bmatrix} \begin{bmatrix}
                        A & -\frac{1}{2}(A+B) \\
                        -\frac{1}{2}(A+B) & B \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}. \label{eq:blockmatrix}
\end{equation}
Let $M_{A,B}$ denote the symmetric $2d \times 2d$ matrix in \eqref{eq:blockmatrix}.  Clearly the expression is only bounded below if $M_{A,B}$ is nonnegative definite.  Stated as the following Lemma, the important point here is that exact equality $A=B$ is required for nonnegativity of the expression; knowing $A$ is close to $B$ is not helpful.

\begin{lemma}  Suppose $A,B \in \RR^{d\times d}$ are symmetric matrices which commute.  Then $(Ax - By)\cdot (x-y)\ge 0$ for all $x,y\in \RR^d$ if and only if $A=B$ is nonnegative definite. \end{lemma}

\newcommand{\sbvec}[2]{\left[\begin{smallmatrix} #1 \\ #2 \end{smallmatrix}\right]}
\newcommand{\sbmat}[4]{\left[\begin{smallmatrix} #1 & #2 \\ #3 & #4 \end{smallmatrix}\right]}

\begin{proof}
If $A=B$ is nonnegative definite then nonnegativity of the expression follows immediately.  Conversely, since $A$ is symmetric there exists an orthogonal basis $\{x_i\}$ of eigenvectors, with $Ax_i = \lambda_i x$.  By commutativity these are also eigenvectors of $B$, with $Bx_i = \mu_i x_i$.  Let $\big\{\sbvec{x_1}{x_1}$, $\sbvec{x_1}{-x_1}$, $\dots$, $\sbvec{x_d}{x_d}$, $\sbvec{x_d}{-x_d}\big\}$ be an ordered basis of $\RR^{2d}$.  It is easy to confirm that in this basis $M_{A,B}$ is block diagonal with $2\times 2$ diagonal blocks $\sbmat{0}{-\frac{1}{2}(\lambda_i+\mu_i)}{-\frac{1}{2}(\lambda_i+\mu_i)}{\lambda_i+\mu_i}$.  Thus the eigenvalues of $M_{A,B}$ come in pairs $\frac{1}{2}(\lambda_i + \mu_i) \pm \sqrt{\frac{1}{2}(\lambda_i^2 + \mu_i^2)}$ for $i=1,\dots,d$.  By the strict concavity of the square root, $\sqrt{\frac{1}{2}(\lambda_i^2 + \mu_i^2)} > \frac{1}{2}(|\lambda_i| + |\mu_i|)$ if $\lambda_i\ne \mu_i$.  Thus if any eigenvalues of $A$ differ from those of $B$, i.e.~$\lambda_i\ne \mu_i$ for any $i$, then there exists a negative eigenvalue of $M_{A,B}$.
\end{proof}

It is not clear if the hypotheses of symmetry and commutativity can be removed.  However, when the matrices are scalar multiples of the identity then the Lemma asserts that $(ax-by)\cdot(x-y) \ge 0$ for all $x,y\in\RR^d$ if and only if $a=b\ge 0$.  This scalar case suffices for the following result.

\begin{theorem}
Suppose $\phi:[0,\infty) \to (0,\infty)$ is continuous.  For $\Omega \subset \RR^d$ open and nonempty, $\cV = H_0^1(\Omega)$, and $g\in \cV'$, consider the following functional on $\cK = \{v\ge 0\} \subset \cV$:
\begin{equation}
\ip{f(u)}{v} = \int_\Omega \phi(u) \grad u\cdot \grad v\,dx - g(v).  \label{eq:porousagain}
\end{equation}
If $\phi$ is not constant then $f$ is not monotone.
\end{theorem}

\begin{proof}
Suppose $\phi(\alpha)\ne \phi(\beta)$ for $\alpha > 0$ and $\beta \ge 0$.  (That is, suppose $\phi$ is not constant.)  For $x\in\Omega$, construct $u\in \cK$ smooth so that $u(x)=\alpha$ and $\xi = \grad u(x) \ne 0$.  Using the Lemma, choose $\eta \in \RR^d$ so that $(\phi(\alpha) \xi - \phi(\beta) \eta) \cdot (\xi - \eta) < 0$.  FIXME NOT SURE IF THIS IS GOING TO WORK
\end{proof}

The main idea is that even the slightest variation in the coefficient $\phi(u)$ violates monotonicity.  A strong and tight ellipticity hypothesis, e.g.~the existence of $c_0>0$ and $\eps>0$ so that $c_0 \le \phi(u) \le (1+\eps)c_0$, will not imply monotonicity.  Contrast the nontrivial coefficient dependence on $\grad u$ in the $p$-coercive $p$-Laplacian Example \ref{ex:plaplacian} (with $p>2)$.

No nontrivial functional \eqref{eq:porousagain} is monotone, nor is it ever $p$-coercive for any $p>1$.
\end{document}


%
% File acl2017.tex
%
%% Based on the style files for ACL-2015, with some improvements
%%  taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2017}
\usepackage{times}
\usepackage{latexsym}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{amsmath}
\DeclareMathOperator*{\argmax}{argmax} % thin space, limits underneath in displays
\usepackage{url}
\usepackage[inline,shortlabels]{enumitem}
\usepackage{caption}% <-- added

\usepackage{tabularx,booktabs}
\newcolumntype{C}{>{\centering\arraybackslash\hsize=.5\hsize}X} % centered version of "X" type
\setlength{\extrarowheight}{1pt}

\usepackage{adjustbox}
\usepackage{float}

\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} %  Enter the acl Paper ID here

%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\newcommand\BibTeX{B{\sc ib}\TeX}

\title{CS395T: Neural Networks for Sentiment Analysis Project Report}

\author{Zeyuan Hu \\
  Computer Science Department \\
  University of Texas at Austin \\
  Austin, Texas \\
  {\tt iamzeyuanhu@utexas.edu} \\
}

\date{}

\begin{document}
\maketitle

\begin{abstract}
In this project, I implement a Feedforward Neural Network based on
averaged word vectors over the input sentence and a convolutional neural
network (CNN) to perform the sentiment analysis on the sentences from
Rotten Tomatoes introduced by Pang and Lee~\shortcite{Pang:2005}.
\end{abstract}

\section{Collaborators}
Zhan Shi, Danlu Wang

\section{Introduction}

Sentiment analysis is an umbrella term for several closely-related tasks. One of them is to
classifiy the sentences as either negative or positive. Traditionally, Naive Bayes or SVM
bag-of-words models are well-suited for those tasks. However, neural network approaches
including convolutional networks \cite{Kim14f} and feedforward networks \cite{Iyyer2015}
can substitute those traditional models to achieve similar performance. In this project,
I implement a feedford network and a convolutional neural networks for the sentiment analysis.
In addition, I explore various architectural and parameter options to
tune the systems to gain good classification performance.

\section{Implementation Details}

\subsection{Feedforward Neural Networks}

The implementation of the feedforword neural network (FFNN) follows closely to
the architecture of the deep averaging network (DAN)
described by Iyyer et al.~\shortcite{Iyyer2015}. In the paper, the authors
break down the architecture into three simple steps: 1. take the average of
word embeddings of words in a given sentence 2. pass the average into a
FFNN with one or more hidden layers 3. perform classification
on the final output. I follow those three steps closely and my network contains
one hidden layer. Essentially, I calculate $P(\boldsymbol{y}|\boldsymbol{x})$ in the network.
\verb|one_best| contains the classifiction result of the sentences in the computataion graph \verb|graph|.
I implement batching and during each epoch, I shuffle
the training examples at the very beginning. For the optimizer, I experiment with both the Adam
and the stochastic gradient descent (SGD). The parameter tuning and the impact of the optimizer
to the performance will be disccued in details in the experiment section.

\subsection{CNN}

The implementation of CNN follows Zhang and Wallace~\shortcite{Zhang15}'s empiricial study of
one-layer CNN hyperparameter setting on sentence classification. In my implementation, I use
one convolution layer with multiple filter widths and feature maps and then I use 1-max pooling
suggested in the paper. The code structure is divided into five steps:
\begin {enumerate*}[1) ]
\item construct embedding layer by looking up pre-trained word embeddings,
\item create convolution layer and implement 1-max pooling strategy,
\item combine all the pooled features,
\item add dropout, and
\item make final predictions.
\end {enumerate*}
The rest of work is to focus on the hyperparameter setting, which
includes filter region size(s) (i.e., \verb|filter_sizes|),
the number of filters to use with each size (i.e., \verb|num_filters|),
the activation function(s) (i.e., \verb|activation_func|), and so on.

\section{Experiments}

The experiment is carried out in a similar way as Zhang and Wallace~\shortcite{Zhang15}'s.
For each exploration task, I report the mean, minimum, and maximum average
accuracy values observed over three replications.

\subsection{Feedforward Neural Networks}

For the FFNN, I explore the impact of the architecture and
the choice of the optimizer to the system performance. Without special note, the experiments
is based on the hyperparameter setting in table 1.

\begin{table}
\captionsetup{size=footnotesize}
\caption{FFNN Configuration} \label{tab:freq}
%\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}cc}
\toprule
  Description  & Values  \\
\midrule
 batch size & 10      \\
 number of hidden layer & 1 \\
 number of hidden units & 150        \\
 number of epochs & 50        \\
 initial learning rate & 0.1        \\
 decay steps & 10 \\
 learning rate decay factor & 0.99 \\
 optimizer & SGD \\
\bottomrule
\end{tabular*}
\end{table}

\textbf{Optimizer:} I compare the performance between Adam and SGD, which is controlled by
the environment variable \verb|OPT| in the code. One surprising finding is that whether
supplying \verb|global_step| argument inside the function call \verb|apply_gradients|
can have impact on the performance of the system . I use tensorflow version 1.3.0 and the result
is shown in table 1.

\begin{table}[!htb]
\captionsetup{size=footnotesize}
\caption{Optimizer Performance with ``global\_step"} \label{tab:freq}
\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}rccr}
\toprule
  Optimizer  & Arg & Accuracy (training) & Accuracy (dev) \\
\midrule
 SGD & Yes   & 76.53 (76.44, 76.59)    & 74.83 (74.67, 75.05)      \\
 ADAM & Yes   & 77.58 (76.72, 78.23)    & \textbf{74.98 (74.48, 75.33)}     \\
 SGD & No  & \textbf{81.72 (81.15, 82.70)}     & 74.42 (74.20, 74.77)      \\
 ADAM & No   & 50.06 (50.06, 50.06)     & 49.06 (49.06, 49.06)      \\
\bottomrule
\end{tabular*}
\end{table}

As shown in the table, \verb|global_step| does not have impact for SGD
measured by the accuracy on the dev set. Adam has similar performance
as SGD with \verb|global_step|. However, without \verb|global_step|,
Adam performs poorly and SGD has significant increase in training set
accuracy compared with \verb|global_step| unset case.

\textbf{Architecture:} The architecture of FFNN is determined by the number of
hidden layers. In this experiment, I explore the FFNN with zero and one
hidden layers. The result is shown in table 3.

\begin{table}[!htb]
\captionsetup{size=footnotesize}
\caption{FFNN architecture} \label{tab:freq}
\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}rcr}
\toprule
  Hidden layers  &  Accuracy (training) & Accuracy (dev) \\
\midrule
 0   &  71.73 (66.66, 77.40)   &  69.48 (65.76, 73.92)     \\
 1    & \textbf{81.72 (81.15, 82.70)}    & \textbf{74.42 (74.20, 74.77)}     \\
 2   & 51.22 (49.94, 52.05)     & 51.88 (50.38, 53.00)     \\
\bottomrule
\end{tabular*}
\end{table}

As shown in the table, there is a significant performance gain
when we increase our hidden layer from zero to one. However, with
the given configuration, two layers architecture has notable performance
drop. This suggests that hyperparameter setting and architecture has
to be carefully picked to achieve good performance.

%
%\begin{table*}
%\caption{Impact of the parameters on the UAS score}
%\label{data-table}
%\begin{tabularx}{\textwidth}{@{}l*{8}{C}c@{}}
%\toprule
%Parser    & Optimizer & Learning Rate & Epoches & Sentences     & Batch Size     & UAS    \\
%\midrule
%Greedy    & SGD        & 0.2              & 10         & 5000          & 1         & 72.27  \\
%Greedy    & SGD        & 0.2            & 20      & 5000           & 1                & 74.67  \\
%Greedy    & SGD       & 0.2             & 30        & 5000          & 1               & 75.81  \\
%Greedy    & SGD       & 0.2              & 40         & 5000         & 1               & 75.74  \\
%\addlinespace
%Greedy    & SGD       & 0.2              & 30         & 6000          & 1              & 76.35  \\
%Greedy    & SGD       & 0.2              & 30      & 7000         & 1               & 76.43  \\
%Greedy    & SGD       & 0.2              & 30      & 39832          & 1                & \textbf{78.60}  \\
%Global    & Adagrad(0.0001, 5) & 5    & 10      & 39832          & 20               & \textbf{75.67}  \\
%\bottomrule
%\end{tabularx}
%\end{table*}

\subsection{CNN}

For the CNN, I explore the impact of the hyperparameter setting to the system performance.
Without special note, the experiments is based on the configuration in table 4. Figure 1
shows the performance of the CNN versus the number of epochs.

\begin{figure}[h]
\includegraphics[scale=0.5]{cnn-epoch.png}
\caption{Accuracy vs. number of epochs in dev set}
\end{figure}

\begin{table}
\captionsetup{size=footnotesize}
\caption{CNN Configuration}
%\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}cc}
\toprule
  Description  & Values  \\
\midrule
input word vectors & GloVe-300      \\
filter region size & (3,4,5) \\
feature maps &    128    \\
activation functions & ReLU        \\
pooling & 1-max pooling        \\
dropout rate & 0.5 \\
$l2$ norm constant & 0 \\
optimizer & Adam \\
number of epochs & 20 \\
batch size & 64 \\
\bottomrule
\end{tabular*}
\end{table}

In the following experiments, I train the CNN for $5$ epochs.
\textbf{Filter region size(s):}
In this experiment, I examine whether the filter region sizes will have
the impact on the CNN performance. The choice of filter region sizes are
picked based on Zhang and Wallace~\shortcite{Zhang15}'s empirical study.
The result is shown in table 5.

\begin{table}[!htb]
\captionsetup{size=footnotesize}
\caption{Filter region sizes and CNN performance} \label{tab:freq}
\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}cc}
\toprule
  Filter region sizes  &   Accuracy (dev) \\
\midrule
(3,4,5)    &  \textbf{77.52 (77.39, 77.77)}     \\
(7,7,7,7)  &  77.20 (76.92, 77.58)     \\
(7)        &  76.92 (76.08, 77.39)     \\
\bottomrule
\end{tabular*}
\end{table}

I try different region sizes and find out that the impact of the region sizes
to the performance is not significant. Even we have one region size $7$, there
is almost no performance drop.

\textbf{Feature maps:}
Zhang and Wallace~\shortcite{Zhang15} suggests that the number of feature maps
for each filter region size is ranged between $100$ and $600$. This is quite large
range. I experiment with three feature maps with one value between $100$ and $200$,
one value between $200$ and $400$, and one value between $400$ and $600$. The result is
shown in table 6.

\begin{table}[!htb]
\captionsetup{size=footnotesize}
\caption{Feature maps and CNN performance} \label{tab:freq}
\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}cc}
\toprule
  Feature maps  &   Accuracy (dev) \\
\midrule
128    &  77.52 (77.39, 77.77)     \\
300  &    \textbf{77.86 (77.67, 78.05)} \\
600    &  77.61 (76.83, 78.52)    \\
\bottomrule
\end{tabular*}
\end{table}

\textbf{Activation functions:}
In this experiment, I experiment with two activation functions: \verb|tanh| and \verb|relu|.
The result is summarized in the table 7.

\begin{table}[!htb]
\captionsetup{size=footnotesize}
\caption{Activation functions and CNN performance} \label{tab:freq}
\setlength\tabcolsep{0pt} % let LaTeX compute intercolumn whitespace
\footnotesize\centering
%This table provides the frequencies.

\smallskip
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}cc}
\toprule
  Activation function  &   Accuracy (dev) \\
\midrule
 \verb|tanh|   &   77.49 (77.30, 77.77)     \\
 \verb|relu|    &  \textbf{77.52 (77.39, 77.77)}     \\
\bottomrule
\end{tabular*}
\end{table}

As shown in the table 7, the choice of the activation function does not have impact on the
CNN performance.

\section{Conclusion and Future Work}

In this project, I implement a feedforward neural network and a CNN for sentiment analysis.
I explore the impact of architecture choice and hyperparameter setting on the performance of
the neural networks to the task. In the future, I can carry out similar empirical study towards
other NLP tasks with different type of neural networks (i.e., LSTM).

\bibliography{acl2017}
\bibliographystyle{acl_natbib}

\end{document}


\documentclass[a4paper,11pt,captions=tableheading,DIV=12]{scrartcl}
\pdfoutput=1

% ----------------------------------------------------------- Packages
\usepackage{amsmath,amssymb,url,cite,slashed,cancel,booktabs,graphicx,listings,subcaption}
\usepackage[colorlinks=true,urlcolor=blue,citecolor=magenta]{hyperref}
%%%UNUSED%%% \usepackage{feynmp,enumerate,multirow,wrapfig}
\renewcommand\citepunct{,\penalty1000\hskip.13emplus.1emminus.1em\relax} % no line-break in \cite

% MATH NOTATION
\newcommand\w[1]{_{\mathrm{#1}}}
\newcommand\vc[1]{{\boldsymbol{#1}}}
\newcommand\dd{\mathrm{d}}
\newcommand\ee{\mathrm{e}}
\newcommand\ii{\mathrm{i}}
\newcommand\pmat[1]{\begin{pmatrix}#1\end{pmatrix}}
\newcommand\Order{\mathop{\mathcal{O}}}
\newcommand\Br{\mathop{\mathrm{Br}}}
\newcommand{\dn}[3]{\frac{\dd^#1 #2}{\dd #3^#1}}
\newcommand{\pdn}[3]{\frac{\partial^#1 #2}{\partial #3^#1}}
\newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}}

\newcommand\unit[1]{\,\mathrm{#1}}
\newcommand\eV{\unit{eV}}
\newcommand\keV{\unit{keV}}
\newcommand\MeV{\unit{MeV}}
\newcommand\GeV{\unit{GeV}}
\newcommand\TeV{\unit{TeV}}
\newcommand\fb{\unit{fb}}
\newcommand\pb{\unit{pb}}
\newcommand\iab{\unit{ab^{-1}}}
\newcommand\ifb{\unit{fb^{-1}}}
\newcommand\ipb{\unit{pb^{-1}}}
\newcommand\fm{\unit{fm}}

\makeatletter
\def\EE{\@ifnextchar-{\@@EE}{\@EE}}
\def\@EE#1{\ifnum#1=1 \times\!10 \else \times\!10^{#1}\fi}
\def\@@EE#1#2{\times\!10^{-#2}}
\makeatother

\newcommand{\fracp}[3]{\ifnum1=#3\relax\biggl(\frac{#1}{#2}\biggr)\else\biggl(\frac{#1}{#2}\biggr)^{#3}\fi}
\newcommand{\neut}[1][]{{\tilde\chi^0_{#1}}}
\newcommand{\chrg}[1][]{{\tilde\chi^\pm_{#1}}}
\newcommand{\chrgP}[1][]{{\tilde\chi^+_{#1}}}
\newcommand{\chrgM}[1][]{{\tilde\chi^-_{#1}}}

\lstset{columns=[l]fullflexible,basicstyle=\small\ttfamily,xleftmargin=2em,frame=L,keepspaces=true}
\bibliographystyle{utphys27mod}

% ---------------------------------------------------- For Sho's Notes
\usepackage{scrlayer-scrpage,color,soul}
\usepackage[hhmmss]{datetime}
\newdateformat{mydate}{\THEDAY\;\shortmonthname.\;\THEYEAR}
\addtokomafont{pagehead}{\small\normalfont}
\rohead*{\texttt{[\jobname~@~\mydate\today~\currenttime]}}

\newcommand{\TODO}[1]{\begingroup\color{red}\textbf{$\clubsuit$#1$\clubsuit$}\endgroup}

\title{SLHA convention compared against References}
\author{Sho Iwamoto}
\date{}
\newcommand{\wL}{_{\mathrm L}}
\newcommand{\wR}{_{\mathrm R}}
\newcommand{\bU}{\bar U}
\newcommand{\bD}{\bar D}
\newcommand{\bE}{\bar E}
\newcommand{\tra}{^{\mathrm T}}
\begin{document}
\maketitle
%---------------------------------------------------------------------
\section{Comparison}
\subsection{Higgs potential}

\subsubsection{SLHA convention}
The SLHA~\cite{SLHA} is based on Gunion--Haber's notation~\cite{GH}:
\begin{align}
\begin{split}
  W &\supset-\epsilon_{ab}
 \mu H_1^aH_2^b
 \\&=-\mu(H_1^1H_2^2-H_1^2H_2^1)
 \\&\equiv \mu(-H_1^0H_2^0+H_1^-H_2^+),
\end{split}
 \tag{SLHA:3}\\
\begin{split}
  V_2 &\supset
  m_{H_{1}}^{2}|H_1|^2
 + m_{H_{2}}^{2}|H_2|^2
 - (m_3^2\epsilon_{ab}H_1^aH_2^b +\text{h.c.}),
\\
&=
  m_{H_{1}}^{2}|H_1|^2
 + m_{H_{2}}^{2}|H_2|^2
+ \left[m_3^2 (-H_1^0H_2^0+H_1^-H_2^+) +\text{h.c.}\right].
\end{split}
 \tag{SLHA:7}
\end{align}
where $\epsilon_{12}=\epsilon^{12}=+1$.
The parameter $m_A^2$, set by \texttt{EXTPAR 24}, is then defined as
\begin{equation}
 m_A^2 = \frac{2m_3^2}{\sin2\beta}.
\tag{SLHA:8}
\end{equation}
The Higgs vacuum expectation value is defined as
\begin{align}
 \langle{H_i^0}\rangle&=\frac{v_i}{\sqrt{2}};&
 m_Z^2 = \frac14(g'^2+g^2)(v_1^2+v_2^2)=\frac14(g'^2+g^2)v^2,
\end{align}
i.e., $v\simeq246\GeV$ is given in the \texttt{HMIX 3}.

\subsubsection{Comparison to GH convention}
In Gunion--Haber~\cite{GH}, the definitions are as follows, with their errata applied:
\begin{align}
 W &\supset -\epsilon_{ab}\mu H_1^aH_2^b,\tag{GH:3.3--3.4}
\\
V\w{soft}&\supset m_1^2|H_1|^2 + m_2^2 |H_2|^2 - (m_{12}^2\epsilon_{ab}H_1^a H_2^b+\text{h.c.}),
\tag{GH:3.9}
%
\end{align}
where $\bE\equiv\tilde R$ in their notation, with the same definition $\epsilon_{12}=1$ (found below Eq.~(3.2)).
So, with the identification $(m_1^2,m_2^2,m_{12}^2)\equiv(m_{H_1}^2,m_{H_2}^2,m_3^2)$, this is identical to SLHA convention.
Also we note the difference of the vacuum expectation values:
\begin{align}
 \langle H_1\rangle &= \pmat{\bar v_1\\0},&
 \langle H_2\rangle &= \pmat{0\\\bar v_2},
\tag{GH:3.7}
\end{align}
with $\bar v_1>0$, $\bar v_2>0$ (3.24), and $\tan\beta=\bar v_2/\bar v_1=v_2/v_1$ (2.8).
Here we use bars to denote $v$s under this definition.

We can then use their results, which say
\begin{align}
 &m_{H_1}^2 = -|\mu|^2 + 2\lambda_1 \bar v_2^2 - m_Z^2/2,
\tag{GH:3.21c}
\\
 &m_{H_2}^2 = -|\mu|^2 + 2\lambda_1 \bar v_1^2 - m_Z^2/2,
\tag{GH:3.21d}
\\
 &m_{H_1}^2 + m_{H_2}^2 + 2|\mu|^2
= m_{3}^2(\tan\beta+\cot\beta)
= \frac{2m_{3}^2}{\sin2\beta},
\tag{GH:3.22}
\end{align}
and, noting that
\begin{align}
 m_Z^2&=(g^2+g'^2)\bar v^2/2,&
 m_W^2&=g^2\bar v^2/2,
\end{align}
which are found below Eq.~(3.19),
\begin{equation}
 \lambda_1
= \frac{g^2+g'^2}{4} + \frac{m_3^2}{2\bar v_1\bar v_2}
= \frac{m_Z^2}{2(\bar v_1^2+\bar v_2^2)} + \frac{m_3^2}{2\bar v_1\bar v_2};
\end{equation}
combining them, we have
\begin{align}
 m_{H_1}^2&=-|\mu|^2-\frac{m_Z^2}{2}\cos2\beta + m_3^2\tan\beta,\\
 m_{H_2}^2&=-|\mu|^2+\frac{m_Z^2}{2}\cos2\beta + m_3^2\cot\beta.
\end{align}
Also the Higgs parameters are given by
\begin{align}
 m_{H^\pm}^2&=(4\lambda_1-g'^2)\bar v^2=m_W^2+m_A^2,\tag{GH:3.16}\\
 m_{H_3^0}^2&=m_{H^\pm}^2-m_W^2=m_A^2,\tag{GH:3.17}\\
 m_{H_1^0,H_2^0}^2 &=
 \frac12\left[m_{H_3^0}^2+m_Z^2\pm\sqrt{
 (m_{H_3^0}^2+m_Z^2)^2-4m_Z^2 m_{H_3^0}^2\cos^22\beta
}\right],\tag{GH:3.18}\\
\tan2\alpha&=\frac{m^2_{H_3^0}+m_Z^2}{m_{H_3^0}^2-m_Z^2}\tan2\beta\tag{GH:3.19}.
\end{align}

\subsubsection{Comparison to SUSY Primer convention}
We here compare the SLHA/GH notation with Martin's SUSY primer~\cite{Martin:1997ns}.
The potentials are given by
\begin{align}
 W\w{MSSM}
&\supset
 \mu H\w u H\w d
\equiv
\mu (H_2^+H_1^- - H_1^0H_2^0),\tag{SP:6.1.1--3}\\
%
\begin{split}
-\mathcal L\w{soft}&\supset
  m_{H\w d}^{2}|H\w d|^2
 + m_{H\w u}^{2}|H\w u|^2
 + (b\epsilon^{ab}H\w u^aH\w d^b +\text{h.c.}),
\end{split}\tag{SP:6.3.1}
\\
&\bar v_u = \langle{H\w u^0}\rangle,\quad \bar v_d = \langle{H\w d^0}\rangle,
\tag{SP:8.1.5}
%
% (|\mu|^2+{m^2_{H\w u}})|H_u|^2
%+(|\mu|^2+{m^2_{H\w d}})|H_d|^2
%+\left(b(H\w u^+H\w d^- - H\w u^0 H\w d^0)+\text{c.c.}\right),\tag{SP:8.1.1}
\end{align}
where $\epsilon^{12}=+1$ (2.13).
So the parameters are identified by the replacement
\begin{align}
\mu&=\mu,\\
m_3^2&=b,
\\
m_{H_1}^2 &= m_{H\w d}^2,
\\
m_{H_2}^2 &= m_{H\w u}^2,
\end{align}
where the LHS are the SLHA parameters and the RHS are those in SUSY primer.
With this identification, we can confirm that the above-shown formulae agree with SUSY Primer's equations (8.1.8)--(8.1.11) and (8.1.19)--(8.1.22).

\subsection{Interaction terms}
Here, to simplify the notation, we omit the SU(2) indices with assuming
\begin{equation*}
 AB=-BA\equiv A^1B^2-B^1A^2.
\end{equation*}
\subsubsection{SLHA convention}
The SLHA~\cite{SLHA} convention for the interection terms are
\begin{align}
 W &=
 - H_2 QY_U\bU
 + H_1 QY_D\bD
 + H_1 LY_E\bE,
\tag{SLHA:3}\\
 V_3 &=
 - H_2 \tilde Q T_U \tilde u^*
 + H_1 \tilde Q T_D \tilde d^*
 + H_1 \tilde L T_E \tilde e^*
\tag{SLHA:5}\\
 V_2 &=
  \tilde Q^*m_{Q}^{2} \tilde Q
 + \tilde L^*m_{L}^{2} \tilde L
 + \tilde u  m_{u}^{2} \tilde u^*
 + \tilde d  m_{d}^{2} \tilde d^*
 + \tilde e  m_{e}^{2} \tilde e^*
\tag{SLHA:7}\\
 \mathcal L_G &= \frac{M_1}{2}\tilde b\tilde b+\frac{M_2}{2}\tilde w\tilde w+\frac{M_3}{2}\tilde g\tilde g + \text{h.c.}
\tag{SLHA:9}
\end{align}
The neutralino and charginos are defined as
\begin{align}
 \tilde\psi^0&=(-\ii\tilde b,-\ii\tilde w^3,\tilde h_1,\tilde h_2)\tra,
&
\tilde\psi^+&=(-\ii\tilde w^+,\tilde h_2^+),
&
\tilde\psi^-&=(-\ii\tilde w^-,\tilde h_1^-).
\tag{SLHA:10+13}
\end{align}

\subsubsection{Comparison to SUSY Primer convention}
In SUSY primer~\cite{Martin:1997ns}, the interaction terms are defined by
\begin{align}
 W\w{MSSM}
&\supset \bar U y_u Q H\w u - \bar d y_d Q H\w d - \bar e y_e L H\w d
\tag{SP:6.1.1}\\
&=
 -H_2Q y_u\tra \bar U + H_1 Qy_d\tra\bar d  +H_1 Ly_e\tra\bar e
\notag\\
V_3 &=
 \tilde u^* a_u \tilde Q H\w u
-\tilde d^* a_d \tilde Q H\w d
-\tilde e^* a_e \tilde L H\w d
\notag\\
&=
-H_2 \tilde Q a_u\tra \tilde u^*
+H_1 \tilde Q a_d\tra \tilde d^*
+H_1 \tilde L a_e\tra \tilde e^*
\notag\\
V_2 &\supset
 \tilde Q^* m_Q^2 \tilde Q
+\tilde L^* m_L^2 \tilde L
+\tilde u^* m_u^2 \tilde u
+\tilde d^* m_d^2 \tilde d
+\tilde e^* m_e^2 \tilde e
\notag\\
&=
 \tilde Q^* m_Q^2 \tilde Q
+\tilde L^* m_L^2 \tilde L
+\tilde u (m_u^2)\tra \tilde u^*
+\tilde d (m_d^2)\tra \tilde d^*
+\tilde e (m_e^2)\tra \tilde e^*
\notag\\
\mathcal L&\supset
-\frac{M_1}{2}\tilde b\tilde b-\frac{M_2}{2}\tilde w\tilde w-\frac{M_3}{2}\tilde g\tilde g + \text{h.c.},\tag{SP:8.1.1}
\end{align}
and the neutralinos and charginos are
\begin{align}
 \tilde\psi^0&=(\tilde b,\tilde w^3,\tilde h_1,\tilde h_2)\tra,
&
\tilde\psi^+&=(\tilde w^+,\tilde h_2^+),
&
\tilde\psi^-&=(\tilde w^-,\tilde h_1^-).
\tag{SP:8.2.1+15}
\end{align}
So the notations can be matched with
\begin{align}
 Y_U, Y_D, Y_E &= y_u\tra, y_d\tra, y_e\tra\\
 T_U, T_D, T_E &= a_u\tra, a_d\tra, y_e\tra\\
 m_Q^2, m_L^2  &= m_Q^2, m_L^2\\
 m_u^2, m_d^2, m_e^2  &= (m_u^2)\tra,(m_d^2)\tra,(m_e^2)\tra,\\
 M_1, M_2, M_3 &= M_1, M_2, M_3,
\end{align}
where the LHS are the SLHA parameters and the RHS are those in SUSY primer.

\section{Reverting sfermion rotation}
In SLHA2 convention~\cite{SLHA2}, the sfermion mass basis $\tilde f_i$ and super-CKM/PMNS basis $\tilde{\vc f}$ are related as
\begin{equation}
 \tilde f_i = R_{iA}\tilde f_A,
\end{equation}
where $R$ is the rotation matrix (e.g.~\texttt{USQMIX}), capital (small) indices are used to denote super-CKM/PMNS (mass) eigenstates, and
the super-CKM/PMNS eigenstates are defined as, e.g.,
\begin{equation}
 \tilde {\vc u} = (
\tilde u\w L, \tilde c\w L, \tilde t\w L,
\tilde u\w R, \tilde c\w R, \tilde t\w R)\tra.
\end{equation}

If the model has no flavor (and CP) violation and the left--right mixings in the first and second generation are negligible, we can convert the SLHA2 output to SLHA1 output.
The SLHA1 mixing matrix $R'$ is obtained as
\begin{equation}
R' = \begin{pmatrix}
  R_{a3} & R_{a6}\\
  R_{b3} & R_{b6}
\end{pmatrix}
\end{equation}
with all elements are non-zero and $a<b$, and thus $\tilde f_a\equiv\tilde f^{\text{3rd}}_1$
and  $\tilde f_b\equiv\tilde f^{\text{3rd}}_2$.
The first and second generation fermions, $\tilde f_A$, are identified with $\tilde f_i$, where $|R_{iA}|>|R_{jA}|$ for all $j\neq i$.
\bibliography{bibliography}
\end{document}


\section{Other concepts} \label{sec:other_NN_concepts}
This section will briefly describe some other concepts that are relevant to this document but not warrant a thorough explanation.

\subsection{Overfitting} \label{sub:overfitting}
One fundamental principle, not only for neural networks but many other machine learning techniques, is the idea of generalization. An algorithm that learned by training on some given data should be able to produce good results not only on this seen data, but also on similar and never before seen data. In the case of supervising learning problems for example, all the data is already labeled, there is no need to create a complex algorithm that can find these labels since the answer is already know. When teaching machines to think, the goal is not to make them memorize all the inputs and outputs, it is instead to make them learn how to abstract the data, to capture the fundamental patterns, and to interpret the input in a high level (e.g. seeing shapes, and objects instead of just pixels).

Since in the majority of cases the data available for training is only a fraction of the possible values expected for the problem, a machine learning algorithm must be able to use this relatively small amount of data to project how all the input space is distributed. This is as much a task for the algorithm as is for the dataset itself, the training data should contain enough samples to offer an accurate representation of how the input space is truly distributed. Ideally this data should be diverse enough to represent well the possible inputs (e.g. when training to differentiate between cats and dogs, the data should contain photos of both cats and dogs of multiple colors, in various angles, and with different poses) and it also should contain as much samples as possible\footnote{
    Although having too much representation for one region of the input space in relation with the others can introduce bias in the model. For example, \cite{networkBias2018} found that some gender classification systems had a significant higher error rates for dark skinned women (as high as 30\% higher than for white males) and that some facial analysis datasets underrepresented minorities in the samples. A similar problem happened to amazon, where an AI recruiting system was discriminating women for jobs since most of the resumes that it was trained on came from men \cite{amazonBias2018}. One proposed approach to combat this problem is to train another network to learn how the data is distributed and sample more from underrepresented data \cite{debiasingVAE2019}.
}.

\begin{figure} [hbt]
    \centering
    \caption{Visual representation of a learning algorithm fit to noisy data}
    \includegraphics[width=0.8\textwidth]{chapters/NeuralNets/figures/fitting.pdf}
    \fonte{From the author (2021)}
    \label{fig:fitting}
\end{figure}
When the dataset is insufficient or the learning algorithm is not capable enough, the final model will only partially represent the data, but will not be able to capture all the details of the representation; this is called \textit{Underfitting}, the model was not able to fully represent the structure of the problem. In the other hand, the algorithm can also go to the opposite extreme and learn the dataset too well, this means that it will pick up particularities of the training data that don't generalize to the whole input distribution; this is called \textit{Overfitting} and it is facilitated by small datasets that don't represent much variety (noise in a sample has more influence in the signal to noise ratio of the whole dataset), or by networks that have too many degrees of freedom. \autoref{fig:fitting} visually represents the different fits for a 1-dimensional case.

Overfitting is however so common that it is almost natural to expect its appearance, it is one of the most prevalent problems in machine learning and there are many existing techniques to avoid it. The first step is to have a good quality dataset, data augmentation methods can be used to easily multiply the number of samples without much downsides. In the algorithmic side, next subsection will discuss regularization and how it can combat overfitting among other things.

\subsection{Regularization} \label{sub:regularization}
According to \cite[p. 117]{deepLearningBook2016}: ``\textit{Regularization is any modiﬁcation we make to a learning algorithm that is intended to reduce its generalization error but not its training error}''. This is a general definition that encompasses many different techniques for regularization, but for this document it will suffice to describe only dropout and the $\ell_1$ and $\ell_2$ norms.

\subsubsection{L1 and L2 norms}
Since one of the reasons overfitting happens is because the models are given a large number of free parameters to adjust to the dataset (worth mentioning again the 175 billion parameters present in GPT-3 \cite{gpt3_2020}), then a common way of reducing overfitting is to limit the choice of parameters by adding a penalization term $\Omega(\bm{\theta})$ to the loss function as shown in \autoref{eq:regularization_penalty} \cite[p. 226]{deepLearningBook2016}.
\begin{equation} \label{eq:regularization_penalty}
    \tilde{J}(\bm{X}, \bm{\theta}) = J(\bm{X}, \bm{\theta}) + \lambda\Omega(\bm{\theta})
\end{equation}

The \gls{lambda} term is a positive hyperparameter that determines how much the penalty is relevant to the overall loss. For $\ell_1$ and $\ell_2$ norms, $\Omega(\bm{\theta})$ will be the respective norm of the $\bm{\theta}$ parameters. The $\ell_2$ norm is the standard distance in euclidean space, so for this norm the penalty term is given by \autoref{eq:l2_norm}.
\begin{equation} \label{eq:l2_norm}
    \ell_2(\bm{\theta}) = \|\bm{\theta}\|_2 = \sqrt{\theta_0^2 + \theta_1^2 + \dots + \theta_n^2}
\end{equation}

For the $\ell_1$ norm, the value calculated is what is called the Manhattan distance or taxicab distance \cite[p. 102]{dataDrivenScience2019} and is calculated as shown is \autoref{eq:l1_norm}
\begin{equation} \label{eq:l1_norm}
    \ell_1(\bm{\theta}) = \|\bm{\theta}\|_1 =  |\theta_0| + |\theta_1| + \dots + |\theta_n|
\end{equation}

Both norms will penalize large parameter values and the modified loss function in \autoref{eq:regularization_penalty} will make gradient descent favor solutions that use smaller parameters. The \gls{lambda} hyperparameter then determines how much to favor these simpler solutions.

A simple explanation on why smaller values would reduce overfitting can be given by considering the case of fitting a polynomial to some data. Consider the overfitting case seen before in \autoref{fig:fitting}, fitting a polynomial to reduce the square error on that noisy data results in a reasonable good approximation around the data points, but the curve suddenly changes directions in the extremes because of the terms with the highest powers in the polynomial. To make the curve fit the noisy data, a least squares approach will usually produce very high coefficients to balance the higher powers, this reduces the error but will almost never produce the true behaviour behind the data. Penalizing high coefficients with regularization is a way to force the solution to assume simpler values.

This is in no way a rigorous argument for regularization, but it can be used to have an intuition. The goal of this section is just to introduce the concept, regularization is an active area of research and a lot of support for it is based on empirical evidence, there is no complete theory to explain how and why it works so well \cite[chap. 3]{NN&DL2015}.

One last thing to mention is the fact that $\ell_1$ norm promotes sparsity in the parameters, this means that this approach will usually find solutions where multiple parameters will equal zero, larger values of \gls{lambda} will promote more sparse models.

\subsubsection{Dropout}
Since the local minimum reached by a training process will depend on the starting values for the parameters, one common way to increase the accuracy of a model is create an ensemble of different networks trained in the same dataset but with different starting points and/or architectures \cite[Chapter 6]{NN&DL2015}. The model output can then be a combination of the outputs of the ensemble, the combination could be majority voting, an average between outputs, or any other sensible function. This gives better results because it is not expected that randomly initialized networks would make the same mistakes in all situations \cite[p. 253]{deepLearningBook2016}.

Dropout is a technique that can be interpreted as a way to approximate an exponentially large ensemble of networks while using a single network and without needing to train more models \cite{dropout2012}. The idea behind it is that during training, when feedforwarding the input through the network, some units should have a probability $p$ of being dropped from the calculation, that is, their activations are set to zero.

The idea of removing units during calculation may sound counter-intuitive, but like the $\ell$ norms it is a way of restricting the learning process so that the network can't build very complex connections. By removing random units in each iteration, the network must be able to learn to represent the data even if a great number of units are removed, it no longer can expect that a combination of units will be present when evaluating the result and so it cannot depend on complex inter-correlations between units \cite{dropout2012}.

By dropping different units, dropout effectively trains the ensemble of all sub-networks of the base network. Typical dropout rates are 20\% for the input layer and 50\% for hidden layers \cite[p. 255, 257]{deepLearningBook2016}.

\subsection{Optimizers} \label{sub:optimizers}
Gradient descent has the theoretical basis for convergence, but it has big problems for use in practice. The first problem is that the whole training set must be averaged when calculating the loss $J(\bm{\theta})$, even for small datasets this is quite restrictive, for larger datasets containing hundreds of thousands or millions of samples this can make each iteration take an immense amount of time. This type of approach is commonly called Batch Gradient Descent.

Another problem is that, even if calculating the gradient was very fast, the simple parameter update (recall \autoref{eq:gradient_descent}) can become very slow, specially later in training. The point in parameter space can reach a region in the loss surface where the curvature is minimal, making the gradients very small; the updates could also jump around a valley in the loss surface, never reaching the local minimum \cite{momentumWorks2017}.

For calculating the gradients faster there is the approach called \glsreset{SGD}\gls{SGD}. There are several different methods for better traversing the loss surface, these are called optimizers. For the purposes of this document only the Momentum, RMSProp, and \gls{Adam} optimizers will be briefly explained.

\subsubsection{Stochastic Gradient Descent}
Instead of calculating the gradient $\nabla_{\bm{\theta}} J$ for all inputs in the dataset, \gls{SGD} instead approximates this value by calculating it for a single sample on the input. This makes the updates much faster, but will cause the updates to fluctuate heavily because of the gross approximation; this however can be beneficial since it can make the updates jump to potentially better local minima and it has been shown that, for small learning rates, \gls{SGD} will also converge \cite{optimizers2016}.

Another way to approximate the gradient is by calculating it only for a mini-batch of the dataset, that is, a subset of $m$ samples of the data. This is called Mini-batch Gradient Descent but is also more commonly referred to as \gls{SGD} as well. Using mini-batches instead of a single sample gives more stable updates while still being very fast, the values for $m$ usually fall between $50$ and $256$ \cite{optimizers2016}, but lower values are also common.

Another advantage of the stochastic approach is that the noise in the gradient approximation has an added regularization effect \cite[p. 5]{practical_gradient_recomendations2012}.

\subsubsection{Momentum, RMSProp, and Adam}
\glsreset{Adam}
The simple parameter update (\autoref{eq:gradient_descent}) for Gradient Descent has some problems that make it difficult to converge in practice. Some of the problems mentioned by \textcite{optimizers2016} are: that it is hard to choose a good value for the learning rate; only using a single learning rate can be insufficient; the parameters can get stuck in difficult regions of the loss surface, especially saddle points.

Between the many different approaches to combat this, one of the most popular that is also recommended when training \acp{GAN} \cite[p. 20, 27]{nipsGAN2017} is the \gls{Adam} optimizer. This optimizer is essentially a combination of Momentum and RMSProp, two other very popular optimizers.

Momentum, as the name suggests, is a way to give some inertia to the gradient updates, accelerating in the consistent directions while dampening oscillations. This technique has a strong theoretical basis and can give a quadratic speedup on many functions \cite{momentumWorks2017}. Momentum changes the parameter updates by adding a velocity term controlled by a new hyperparameter $\beta$ as shown in the following equations.
\begin{gather} \label{eq:momentum}
    \bm{m}_{i} = \beta \bm{m}_{i-1} + \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1}) \\
    %
    \bm{\theta}_{i} = \bm{\theta}_{i-1} - \eta \bm{m}_{i}
\end{gather}

The RMSProp algorithm also tries to reinforce movement to the most relevant directions, it does this by keeping a exponentially moving average $v$ of the gradient accumulation \cite[303-304]{deepLearningBook2016}. It also introduces another $\beta$ hyperparameter and changes the updates as follows.
\begin{gather}
    \bm{v}_{i} = \beta \bm{v}_{i-1} + (1 - \beta)\nabla_{\bm{\theta}} J(\bm{\theta}_{i-1})^2 \\
    %
    \bm{\theta}_{i} = \bm{\theta}_{i-1} - \frac{\eta}{\sqrt{\bm{v}_{i} + \epsilon}} \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1})
\end{gather}

Recall that although these equations are represented in vector form, all operation are performed element-wise. Finally for the \gls{Adam} algorithm, it can be seen as a combination of the two previous algorithms. It introduces two new hyperparameters $\beta_1$ and $\beta_2$ and the parameter updates work as follows \cite{adam2017}.
\begin{gather}
    \bm{m}_i = \beta_1 \bm{m}_{i-1} + (1 - \beta_1) \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1}) \\[5pt]
    \bm{v}_i = \beta_2 \bm{v}_{i-1} + (1 - \beta_2) \nabla_{\bm{\theta}} J(\bm{\theta}_{i-1})^2 \\[5pt]
    \hat{\bm{m}_i} = \frac{\bm{m}_i}{1 - \beta_1^i} \\[5pt]
    \hat{\bm{v}_i} = \frac{\bm{v}_i}{1 - \beta_2^i} \\[5pt]
    \bm{\theta}_i = \bm{\theta}_{i-1} - \eta \frac{\hat{\bm{m}_i}}{\sqrt{\hat{\bm{v}_i} + \epsilon}}
\end{gather}

\textcite{adam2017} mentions that good default values for the hyperparameters are $\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$. The Tensorflow library \cite{tensorflow2015} follows this default, only changing $\epsilon$ to a order of magnitude higher, that is, $\epsilon = 10^{-7}$.

\subsection{Batch Normalization}
One important aspect in training neural networks is the distribution of the inputs for a layer. As mentioned in \autoref{subsub:sigmoid} about sigmoid, \textcite{efficientBackprop2012} showed that inputs that are not zero centered will have some bias effect on parameter updates, to avoid this the layer inputs should all be shifted to have a mean of zero; the authors also suggest to normalize all inputs in order to have the same covariance and, if possible, to have their values be uncorrelated. This should significantly speed up the learning process since the network will not have to adapt to a different distribution for every input.

Normalizing inputs is then an easy and effective way for better learning, therefore being a very commonly used technique that will also be used for the experiments proposed in this document.

However \textcite{batchnorm2015} note that for any layer in the network, the normalization idea for its inputs also applies; that is, any hidden layer feeding its output to the next layer can be considered as an input layer for a sub-network consisting of all the subsequent layers. Therefore, the same advantages for normalized inputs would be beneficial by normalizing hidden layer outputs.

Looking from the training perspective, each training iteration updates all parameters at the same time, but the gradient change in a layer gives the best change considering that all other layers remain constant \cite[p.313-314]{deepLearningBook2016}. In reality, changing the parameters of the previous layers will affect the distribution of inputs for the current layer, this constant change to the inputs of a layer resulting from changes to previous layers is what \textcite{batchnorm2015} call \textit{internal covariate shift}.

To address these issues \textcite{batchnorm2015} proposed a technique known as \gls{BN}, this method can be applied to any hidden layer and it consists of normalizing the layer inputs by the mean and variance of all the inputs in the current mini-batch. So, for a mini-batch $\mathcal{B}$ of size $m$ and inputs $(\bm{x}_1, \dots, \bm{x}_m)$, the normalized inputs $\hat{\bm{x}}_i$ are calculated using the mean $\bm{\mu}_{\mathcal{B}}$ and variance $\bm{\sigma}_{\mathcal{B}}$ of the batch. Equations \ref{eq:batch_mean}, \ref{eq:batch_variance}, and \ref{eq:batch_normalized_input} show how the mean, variance, and normalized inputs are calculated.
\begin{gather}
    \bm{\mu}_{\mathcal{B}} = \frac{1}{m}
    \sum_{i=1}^{m}{\bm{x}_i} \label{eq:batch_mean} \\[5pt]
    %
    \bm{\sigma_}{\mathcal{B}}^2 = \frac{1}{m}
    \sum_{i=1}^{m}{ \left( \bm{x}_i - \bm{\mu}_{\mathcal{B}} \right)^2 } \label{eq:batch_variance} \\[5pt]
    %
    \hat{\bm{x}}_i = \frac{ \bm{x}_i - \bm{\mu}_{\mathcal{B}} }
    { \sqrt{\bm{\sigma}_{\mathcal{B}}^2 + \epsilon} } \label{eq:batch_normalized_input}
\end{gather}

Note that these operations are applied element-wise for each component of the vectors, and again $\epsilon$ is used for numerical stability.

These operations result in the vectors $\hat{\bm{x}}_i$ having a mean of $0$ and variance of $1$ for each of their dimensions. However, by limiting the distribution to only these values also limits the representation power of the network \cite{batchnorm2015}, so the last step is to scale each one of the normalized activations by two new learnable parameters, $\gamma$ and $\beta$, unique for each activation, to obtain the transformed input $\bm{y}_i$ that can have any mean and variance. \autoref{eq:batch_transformed_input} shows how to scale the dimension $k$ of the normalized input $i$.
\begin{equation} \label{eq:batch_transformed_input}
    y_i^{(k)} = \gamma^{(k)} x_i^{(k)} + \beta^{(k)}
\end{equation}

This method has proven to be very powerful, with the original paper being able to reach the same accuracy as the, at the time state of the art image classification model, in 14 times less training iterations \cite{batchnorm2015}. The authors also cite a regularization effect of \gls{BN} that can reduce the need for other techniques like dropout.

When the authors originally proposed this method, they recommended applying the batch normalization operation directly to the weighted inputs of the network and only after apply the activation to the normalized values. However, since then \textcite{CaffeNetBench2017} empirically showed that applying \gls{BN} after the activation produces better results.

\subsection{Vanishing Gradients} \label{sub:vanishing_gradients}
Recall that for Equations \ref{eq:delta_last_layer} and \ref{eq:delta_hidden_layer} the $\delta$ term used to calculate the gradients, was directly dependent on the derivative of the activation function with relation to the weighted inputs $f'(\bm{z}^{(l)})$. Also note that the gradients are backpropagated through the network, which means that the $\delta$ values in a hidden layer are calculated from the $\delta$ in the next layer.

Consider now the case for a sigmoid or \gls{tanh} activation function which is applied to an weighted input far from zero, the activation in this case will be very close to the maximum or minimum value that the function can produce. When a unit outputs this kind of value it is said that it saturated, any slight deviation in the weighted input will barely have any difference in the activation value, in other words the derivative of the activation is close to zero.

Combining the points made in the last paragraphs it is possible to recognize a problem, when a unit saturates its gradient will be very small since the derivative of the activation is close to zero. Even worse than that is the fact that this small gradient will be backpropagated through the network, reducing the gradients of all previous layers. When many units saturate, this effect can compound, making the gradients for deeper layers very small and greatly reducing the training speed; this is the problem known as the \textit{Vanishing Gradients} problem.

One alternate case is when the gradients are high and compound to make the gradients in deeper layers become very high, making the steps too large and not converging to any local minima. This is called the \textit{Exploding Gradients} problem.

Both situations are undesirable and are specially worrying when trying to train very deep neural networks. Ideally the gradients should all be close to $1$ to make training consistent, but for saturating activation functions like sigmoid, there will usually be vanishing gradients \cite{NN&DL2015}. That is one of the main reasons to use non-saturating activation functions like \gls{ReLU} in the hidden layers of the network. Other common ways to combat this problem is to carefully initialize the network parameters and use small learning rates \cite{batchnorm2015}, these authors also suggest that \gls{BN} can be used on saturating units since it can reduce the chance that the input of the units will fall to the saturating regions.