\chapter{Mathematical Background}\label{math}
\section{Introduction}
Discontinuous Galerkin finite element method is an amalgamation of the element based Galerkin schemes and
the finite volume method. It is a compact, higher-order accurate scheme where the solution can be
discontinuous between two elements.
The DG scheme requires the physical domain ($\Omega$) discretized with $Ne$ finite elements (written as $\Omega^k$).
The finite element cells are assumed to be one of the following types.
\begin{enumerate}
	\item Hexahedral ($8$ edges, $8$ nodes, $6$ faces)
	\item Tetrahedral ($6$ edges, $4$ nodes, $4$ faces)
	\item Prism ($9$ edges, $6$ nodes, $5$ faces)
	\item Pyramid ($8$ edges, $5$ nodes, $5$ faces)
\end{enumerate}
The solution is assumed to be continuous (along with continuous derivatives depending on the governing equations) inside
the element. However, the solution can be discontinuous at the edges and faces. The value of the conserved variable (and
the numerical fluxes) at the faces and edges are found out by solving the Riemann problem which arises because of the
discontinuity.

The DG method essentially starts with the `variational' formulation of the problem. Consider a
differential equation given by,
\begin{equation}\label{diffeqn}
	\mathbf{L}(u) = f(u),  u\in{H^1(\Omega)}
\end{equation}
(and suitable boundary conditions, not mentioned here).
For example, $\mathbf{L} = \frac{d}{dx}$ and $f(u) = u$ results in,
\begin{equation}\label{diffeqn_ex1}
	\frac{du}{dx} = u
\end{equation}
We will be using the general notation given by equation \ref{diffeqn} henceforth.
Then, if $u^k$ is the approximate solution inside of the element, then instead of the solving equation \ref{diffeqn}
with $u^k$, we solve the following variational formulation of equation \ref{diffeqn}.
\begin{equation}\label{diffeqnvr}
	\int_{\Omega_k} \phi \mathbf{L}(u^k) d\Omega = 	\int_{\Omega_k} \phi f(u^k) d\Omega
\end{equation}
$\phi$ is the test function. The approximate solution $u^k$ is obtained by polynomial approximation of $u$ inside $\Omega^k$.
A modal interpolation (basis functions consisting of modes, i.e. frequencies) or a nodal interpolation (basis functions
consist of the cardinal polynomials such as the Lagrange polynomials) can be used for this.
In the DG methods, the basis functions serve as the test functions.
The integrations in equation \ref{diffeqn_ex1} are solved numerically using quadrature formulas.
The resulting algebraic system is solved iteratively (or ODE is solved numerically if time dependant).
This process is repeated till convergence (for steady state
problems) or till the final time is reached (for transient problems).
It is easier to solve the problem on a reference cell (canonical element) and then map the solution on to the physical cell.
This requires a mapping to be defined between the reference cell and the physical cell.
Based on this general process, the following mathematical concepts are revised in this chapter.
\begin{enumerate}
	\item Mapping functions (Jacobian ($\mathbf{J}$) of transformation, inverse Jacobian ($\mathbf{J}^{-1}$) and ($|\mathbf{J}|$))
	\item Modal and nodal basis functions
	\item Polynomial interpolation and differentiation
	\item Numerical quadrature (integration)
	\item Cell matrices (Vandermonde matrix, Mass matrix, differentiation matrix and flux matrix)
	\item Other mathematical functions
\end{enumerate}
The discussion is kept brief and concise and only those concepts concerned with MEAN-DG are discussed.
The user is encouraged to go through the references for in-depth theory of DG schemes.

\begin{note}
	We use the c++ style indexing. i.e. indices start at $0$ (not $1$).
\end{note}

\section{Mapping on the Reference Element}
Corresponding to each physical cell, there exists a canonical (reference) element.
The physical cell is expressed in terms of the physical coordinate system $(x,y,z)$. The reference coordinate system is
written in terms of  $(r,s,t)$.
Thus, each of the physical coordinates can be written in terms of the $(r,s,t)$ coordinate system. In the code, we
describe the physical coordinate system as $XYZ$, and the reference system as $RST$.
The mapping from the physical coordinate system to the reference coordinate system is characterized by the Jacobian of
transformation $\mathbf{J}$.
In this section, we describe the Jacobian of transformation for the four types of the cells.

\subsection{Hexahedral cells}
Hexahedral, or Hex in short, cells are mapped to a cube in 3D. The Hex cell has $8$ nodes given by the array,
$\left((x_0,y_0,z_0), (x_1, y_1, z_1), ... ,(x_7,y_7,z_7)\right)$.
We define the linear shape functions on the reference element as follows:
\begin{equation}\label{shape0}
	N_0 = \frac{1}{8} (1-r) (1-s) (1-t), \hspace{5mm} \{r,s,t \in V | \forall v\in V \subset  {\mathbb{R}} , ~ v \in [-1,1]\}
\end{equation}
\begin{equation}\label{shape1}
	N_1 = \frac{1}{8} (1+r) (1-s) (1-t)
\end{equation}
\begin{equation}\label{shape2}
	N_2 = \frac{1}{8} (1+r) (1+s) (1-t)
\end{equation}
\begin{equation}\label{shape3}
	N_3 = \frac{1}{8} (1-r) (1+s) (1-t)
\end{equation}
\begin{equation}\label{shape4}
	N_4 = \frac{1}{8} (1-r) (1-s) (1+t)
\end{equation}
\begin{equation}\label{shape5}
	N_5 = \frac{1}{8} (1+r) (1-s) (1+t)
\end{equation}
\begin{equation}\label{shape6}
	N_6 = \frac{1}{8} (1+r) (1+s) (1+t)
\end{equation}
\begin{equation}\label{shape7}
	N_7 = \frac{1}{8} (1-r) (1+s) (1+t)
\end{equation}

The mapping then is simply given as,

\begin{equation}\label{mapx}
	x(r,s,t) = \sum_{i=0}^7 N_i x_i
\end{equation}

\begin{equation}\label{mapy}
	y(r,s,t) = \sum_0^7 N_i y_i
\end{equation}

\begin{equation}\label{mapz}
	z(r,s,t) = \sum_0^7 N_i z_i
\end{equation}

\begin{definition}\label{Jacobian}
	The Jacobian of transformation ($\mathbf{J}$) is defined as :
	\begin{equation}\label{J}
		{\setstretch{2}
		\mathbf{J} = \left[\begin{array}{c c c}
			\frac{\partial x}{\partial r} & \frac{\partial x}{\partial s} &  \frac{\partial x}{\partial t} \\
			\frac{\partial y}{\partial r} & \frac{\partial y}{\partial s} &  \frac{\partial y}{\partial t} \\
			\frac{\partial z}{\partial r} & \frac{\partial z}{\partial s} &  \frac{\partial z}{\partial t} \\
												                   \end{array} \right]
			= \left[\begin{array}{c c c}
				J_{00} & J_{01}  & J_{02}   \\
				J_{10} & J_{11}  & J_{12}   \\
				J_{20} & J_{21}  & J_{22}   \\
												                   \end{array} \right]
	        }
	\end{equation}
\end{definition}
The partial derivatives in equation \ref{J} can be easily computed from equations \ref{shape0}-\ref{shape7} and \ref{mapx}, \ref{mapy}, \ref{mapz}.
In the code, the above implementation is found in Cell.cpp.
\begin{verbatim}
Cell::calculateJacobian3DTensor(double r,double s,double t)
\end{verbatim}
\begin{note}
	The first name indicates the name of the class (Cell), the second symbol (::) indicates the scope (in this case, indicating
	'belongs to Cell' the third name is the name of the class method (calculateJacobian3DTensor) and terms in the bracket being
	arguments passed to this function.
\end{note}
It is to be noted that the name `Tensor' in the above function indicates that the functional space for a Hex cell is
obtained by a tensor product (outer product) of 1D functional spaces.

The inverse of the Jacobian is found out by the standard process of inversion of a matrix.

\begin{equation}\label{JInv}
	{\setstretch{2}
	\mathbf{J^{-1}} = \left[\begin{array}{c c c}
		\frac{\partial r}{\partial x} & \frac{\partial r}{\partial y} &  \frac{\partial r}{\partial z} \\
		\frac{\partial s}{\partial x} & \frac{\partial s}{\partial y} &  \frac{\partial s}{\partial z} \\
		\frac{\partial t}{\partial x} & \frac{\partial t}{\partial y} &  \frac{\partial t}{\partial z} \\
											                   \end{array} \right]
        }
\end{equation}
These are given as:
\begin{equation}\label{metricTerms00}
	\frac{\partial r}{\partial x} = \frac{1}{|{\mathbf{J}}|} \left( J_{22}J_{11} - J_{21}J_{12}\right)
\end{equation}
\begin{equation}\label{metricTerms01}
	\frac{\partial r}{\partial y} = \frac{-1}{|{\mathbf{J}}|} \left( J_{22}J_{01} - J_{21}J_{02}\right)
\end{equation}
\begin{equation}\label{metricTerms02}
	\frac{\partial r}{\partial z} = \frac{1}{|{\mathbf{J}}|} \left( J_{12}J_{01} - J_{11}J_{02}\right)
\end{equation}
\begin{equation}\label{metricTerms10}
	\frac{\partial s}{\partial x} = \frac{-1}{|{\mathbf{J}}|} \left( J_{22}J_{10} - J_{20}J_{12}\right)
\end{equation}
\begin{equation}\label{metricTerms11}
	\frac{\partial s}{\partial y} = \frac{1}{|{\mathbf{J}}|} \left( J_{22}J_{00} - J_{20}J_{02}\right)
\end{equation}
\begin{equation}\label{metricTerms12}
	\frac{\partial s}{\partial z} = \frac{-1}{|{\mathbf{J}}|} \left( J_{12}J_{00} - J_{10}J_{02}\right)
\end{equation}
\begin{equation}\label{metricTerms20}
	\frac{\partial t}{\partial x} = \frac{1}{|{\mathbf{J}}|} \left( J_{21}J_{10} - J_{20}J_{11}\right)
\end{equation}
\begin{equation}\label{metricTerms21}
	\frac{\partial t}{\partial y} = \frac{-1}{|{\mathbf{J}}|} \left( J_{21}J_{00} - J_{20}J_{01}\right)
\end{equation}
\begin{equation}\label{metricTerms22}
	\frac{\partial t}{\partial z} = \frac{1}{|{\mathbf{J}}|} \left( J_{11}J_{00} - J_{10}J_{01}\right)
\end{equation}
where, the determinant of the Jacobian $|{\mathbf{J}}|$ is calculated as:
\begin{equation}\label{detJ}
	|{\mathbf{J}}| = J_{00} (J_{22}J_{11} - J_{21}J_{12}) - J_{01} (J_{22}J_{10} - J_{20}J_{12})  + J_{02} (J_{21}J_{10} - J_{20}J_{11})
\end{equation}
The inverse of Jacobian is implemented in
\begin{verbatim}
Cell::calculateInverseJacobianMatrix3DTensor(arguments)
\end{verbatim}

\begin{note}
	The inverse of Jacobian is called as {\bf dRST\_by\_dXYZ} in the code. Each cell stores these matrices for all the quadrature
	points within that cell. The metric terms given by \ref{metricTerms00}--\ref{metricTerms22}
	are used in computations involving derivatives as seen in the subsequent sections and chapters.
\end{note}
The inverse Jacobian is stored for all the quadrature points for all the cells. It is an attribute of the cell
(i.e. Cell::dRST\_by\_dXYZ) and has dimensions $nQuad \times 3 \times 3$.

{\bf Use of the Jacobian}
Consider the following integration:
\begin{equation}\label{J_example}
	I = \int_{\Omega^k} \phi(x,y,z) d\Omega^k
\end{equation}
The integration can be written as,
\begin{equation}
	I = \int_{\Omega^s} \phi(r,s,t) |\mathbf{J}| d\Omega^s
\end{equation}
where, $\Omega^s$ is a reference element and $\mathbf{J}$ is the Jacobian matrix.
The main benefit of performing the above conversion is that, the function $\phi(r,s,t)$ needs to be computed only once
for the standard element, each cell will have its own Jacobian matrix, so number of computations reduce drastically.

In the current code, the vertices and faces are numbered in a standardized fashion as shown in Fig.\ref{RC}.

\begin{figure}[h]
	\begin{minipage}[t]{0.5 \textwidth}
		\begin{center}{\includegraphics[width=0.9\textwidth]{RefCellFace.pdf}}
		\end{center}
	\end{minipage}%
	\begin{minipage}[t]{0.5 \textwidth}
		\begin{center}{\includegraphics[width=0.9\textwidth]{RefCellVertex.pdf}}
		\end{center}
	\end{minipage}%
	\caption{\label{RC}a. Reference cell and faces relative numbering \hspace{2mm} b. Reference cell and vertices numbering}
\end{figure}

The corresponding implementation can be found in:
\begin{verbatim}
Cell::orderDefiningPoints()
\end{verbatim}

\section{Mapping onto the Reference Face}
DG method involves evaluation of surface integrations in order to find the numerical flux at the cell-interface.
The surface integrations in turn require mapping of a given arbitrary surface on the reference surface.
The parameterization of a plane surface requires two parameters $r$ and $s$.

\subsection{Quadrilateral Faces}
For a quadrilateral faces, the parameters assume the following values: $(r,s) \in [-1,1]$.
However, the given surface can be arbitrarily oriented in the space, and thus requires $(x,y,z)$ position
to be completely defined.
We thus need to orient the surface such as one of the variables becomes redundant. This means rotating
the surface such as to make it parallel to $xy$ plane, for example.
Thus, each surface needs to be re-oriented in space such that the face normal $\hat{n}$ becomes
parallel to the $z$ axis. After the we can map the rotated face onto the standard element.
Let $\theta_z$ be the angle made by the face normal with the $z$ axis and $\theta_x$ be the angle made by
the face normal with the $x$ axis.
The rotation matrix rotates the face through $\theta_x$ first (counterclockwise) and the through $\theta_z$
counterclockwise. The resulting rotation matrix $R$ is given as:
\begin{equation}\label{rotMat1}
	R = <R_z, R_x>
\end{equation}
where,
\begin{equation}\label{rotMat}
	{\setstretch{1.5}
	R_x =
	 \left[\begin{array}{c c c}
				cos(\theta_x) & sin(\theta_x)  & 0  \\
			       -sin(\theta_x) & cos(\theta_x)  & 0  \\
				       0      &       0        & 1  \\
         \end{array} \right];
\hspace{5mm}
	R_z =
	 \left[\begin{array}{c c c}
				cos(\theta_z)   & 0 & -sin(\theta_z) \\
				       0        & 1 &       0       \\
			        sin(\theta_z)   & 0 & cos(\theta_z) \\
         \end{array} \right];
 }
 \end{equation}
 and $<a,b>$ indicates the dot product between matrices $a$ and $b$.
 Matrix $R^{-1}$ indicates the reverse transformation.
 The rotation matrix is calculated in,
 \begin{verbatim}
 Face::calculateRotationMatrixParallelToXY()
 \end{verbatim}
 and the inverse matrix is computed in,
 \begin{verbatim}
 Face::calculateInverseRotationMatrix()
 \end{verbatim}
 In order to describe the procedure to compute the Jacobian of transformation ($|{\mathbf{J}^f}|$) for face,
 we use the following nomencleture:
 \begin{itemize}
	 \item $(x,y,z)$: physical coordinate system (actual coordinates)
	 \item $(x', y', z')$: Rotated coordinates such that the face is $||$ to the $xy$ plane
	 \item $(r,s)$: Reference coordinates such that $r,s \in [-1,1]$
 \end{itemize}
 There are two ways the Jacobian can be computed.

 \subsubsection{Method 1}
 First, rotate each vertex of the face using $R$.
 i.e.
 \begin{equation}
	 {\setstretch{1.5}
	 \left[\begin{array}{c}
			  x'_i  \\
			  y'_i  \\
			  z'_i \\
	  \end{array} \right]
	  =
	  R\left(
	 \left[\begin{array}{c}
			  x_i  \\
			  y_i  \\
			  z_i \\
	  \end{array} \right]
	  \right), \hspace{5mm} i \in \{0,1,2,3\}
  }
 \end{equation}
 Each of the rotated points is then mapped to the standard element.
 After rotation of the face, $\frac{\partial z'}{\partial r} = 0, \frac{\partial z'}{\partial s}=0$ since
 $z'$ remains constant.
The mapping from the rotated face $f'$ to the standard face $f^s$ can be defined through the shape functions.
\begin{equation}\label{shape0f}
	N_0 = \frac{1}{4} (1-r) (1-s) , \hspace{5mm} \{r,s \in V | \forall v \in V \subset {\mathbb{R}}  , ~ v \in [-1,1]\}
\end{equation}
\begin{equation}\label{shape1f}
	N_1 = \frac{1}{4} (1+r) (1-s)
\end{equation}
\begin{equation}\label{shape2f}
	N_2 = \frac{1}{4} (1+r) (1+s)
\end{equation}
\begin{equation}\label{shape3f}
	N_3 = \frac{1}{4} (1-r) (1+s)
\end{equation}
Then,
\begin{equation}\label{faceMap}
	x' = \sum_{i=0}^3 N_i x'_i, \hspace{4mm}y' = \sum_{i=0}^3 N_i y'_i,
\end{equation}
Equation\ref{faceMap} can be used to find partial derivatives of $x'$ and $y'$ with respect to $r$ and $s$.
The required face Jacobian is:
 \begin{equation}\label{method1J}
	 {\setstretch{1.5}
	 \mathbf{J}^f =
	 \left[\begin{array}{c c}
			 \frac{\partial x}{\partial r}  & \frac{\partial x}{\partial s}  \\
			 \frac{\partial y}{\partial r}  & \frac{\partial y}{\partial s}  \\
			 \frac{\partial z}{\partial r}  & \frac{\partial z}{\partial s}  \\
         \end{array} \right]
 }
 \end{equation}
 which can be written as,
 \begin{equation}\label{method1J}
	 {\setstretch{1.5}
	 \mathbf{J}^f =
	 \left[\begin{array}{c}
			  x  \\
			  y  \\
			  z  \\
	  \end{array} \right].
	  \left[\begin{array}{c c}
		  \frac{\partial}{\partial r} &  \frac{\partial}{\partial s}\\
		  \end{array} \right]
		  = R^{-1}
	 \left[\begin{array}{c}
			  x'  \\
			  y'  \\
			  z'  \\
	  \end{array} \right].
	  \left[\begin{array}{c c}
		  \frac{\partial}{\partial r} &  \frac{\partial}{\partial s}\\
		  \end{array} \right]
 }
 \end{equation}
i.e.
\begin{equation}\label{method1J1}
	\mathbf{J}^f = R^{-1} \mathbf{J}^{f'}
\end{equation}
where,
\begin{equation}\label{method1J'}
        {\setstretch{1.5}
		\mathbf{J}^{f'} =
        \left[\begin{array}{c c}
       		 \frac{\partial x'}{\partial r}  & \frac{\partial x'}{\partial s}  \\
       		 \frac{\partial y'}{\partial r}  & \frac{\partial y'}{\partial s}  \\
				0		 &             0		   \\
        \end{array} \right]
}
\end{equation}
The procedure is summarized as,
\begin{enumerate}
        \item Rotate the face so as to make it parallel to the $xy$ plane.
	\item Find the Jacobian of transformation from the rotated face to the standard face ($\mathbf{J}^{f'}$).
	\item Find the Jacobian of the face as given by equation \ref{method1J1}.
	\item $|\mathbf{J}^{f}|$ is given as $\sqrt{|\left(\mathbf{J}^{f}\right)^{T}\left(\mathbf{J}^{f}\right)|}$
\end{enumerate}

\subsubsection{Method 2}
In this method, we parametarize the surface $x,y,z$ with parameters $r,s$.
Let $$X = x\hat{i} + y\hat{j} + z\hat{k}$$
We use the standard shape functions to map the physical element to parameters $r,s$.
i.e.
\begin{equation}\label{faceMap1}
	x = \sum_{i=0}^3 N_i x_i, \hspace{4mm}y = \sum_{i=0}^3 N_i y_i, \hspace{4mm}z = \sum_{i=0}^3 N_i z_i,
\end{equation}
where same shape functions are used as earlier.
\begin{note}
In order to find the vertex mapping of a randomly oriented face with the standard
element, it would be advisable to first rotate the face to make it parallel to the $xy$ plane and then
find the vertex mapping. Then find the shape function and proceed with equation \ref{faceMap1}.
However, we do not use the coordinates of the rotated face here.
\end{note}
After that, we find $\frac{\partial X}{\partial r}$ and $\frac{\partial X}{\partial s}$.
Then $|\mathbf{J}^{f}|$  is simply $\left|\left| \frac{\partial X}{\partial r} \times \frac{\partial X}{\partial s}\right|\right|$
Both of these methods are implemented in
\begin{verbatim}
	Face::calculateJacobianQuadFace(double r, double s)
\end{verbatim}
Any given quadrature point $(r,s)$ can be mapped into $(x,y,z)$ space using the two operations:
\begin{enumerate}
	\item First find mapping from $(r,s)$ to $(x',y',z')$ using equations \ref{shape0f}-\ref{shape3f} and
		\ref{faceMap}.
	\item Then map from $(x',y',z')$ to physical coordinates using $R^{-1}$.
\end{enumerate}
The global positioning of all the quadrature points on each face is
stored in array
\begin{verbatim}
	Face::quadPointsGlobalLocation
\end{verbatim}
Index $i$ runs over the surface quadrature points and index $j$ runs over the $x,y,z$ axes.
\begin{note}
	Some of the volume quadrature points coincide with the surface quadrature points. Function \begin{verbatim}DG::mapFaceQuadPointsToCellQuadPoints() \end{verbatim} performs the mapping by direct comparison of global positioning of surface and volume quadrature points.
\end{note}
\noindent
The mapping of surface to volume quadrature points is stored in arrays \newline Face::mapOwnerQuadPoints[~] and Face::mapNeighbourQuadPoints[~].

Thus, if we are finding a $2^{nd}$-order polynomial for a face, then it should have $9$ quadrature points (QP).
The corresponding cell will meanwhile have $27$  quadrature points, $9$ of which will coincide with the face QP.
Then if
\begin{verbatim}
	Face::mapOwnerQuadPoints = [3, 2, 21, 14, ... , 7]
\end{verbatim}
Then, it means the $0^{th}$ quad point on face is same as the $3^{rd}$ quad point of owner cell and so on.

\begin{note}
	In the improved version of the code, the face-vertex numbering is standardized, such that the vertex numbering follows the right-hand rule.
	i.e. if the curled fingers indicate the sequence of vertices, then the stretched thumb points in the direction of hte face-normal.
	In that case, the rotation matrices are not required. i.e. both method 1 and method 2 are obsolete.
	Refer:
	\begin{verbatim}
	Face::orderDefiningPoints()
	\end{verbatim}
\end{note}

\section{Modal and Nodal Basis Functions}
Basis functions are the independant dimensions on which any function can be projected. In this chapter, we present a few
modal and nodal basis functions.

\subsection{Modal Basis Functions}
These are the set of basis functions, where each dimension corresponds to a `mode' or frequency. A simple example is
a set of monomials defined on the interval $[a,b]$
\begin{note}
This is just an example, in reality $x$ could be valid over $(-\infty, \infty)$
\end{note}
\begin{equation}
	V_{mono}^K = \{1, x, x^2, x^3, ..., x^{K-1}\} ,\hspace{4mm} x\in C^0[a,b]
\end{equation}
A polynomial up to order $K-1$ can be represented exactly using $K$ modal functions.
Thus, any general polynomial can be written as
\begin{equation}
	P^K(x) = \sum_{i=0}^{i=K-1} \alpha_{i} x^{i}, ,\hspace{4mm} x\in C^0[a,b]
\end{equation}
The monomials is not a `good' choice of the basis functions as we will see later, thus we need a set of basis functions
where each of the functions is `orthogonal' (or better `orthonormal') to every other functions.
One such important set of orthogonal functions
is called as `Jacobi' polynomials. A special case of `Jacobi' polynomials is `Legendre' polynomials.
In our code, we have extensively used Legendre polynomials.
A general $i^{th}$  modal basis function is indicated by symbol $\psi_i$.
Other popular example of modal basis functions is a set of Chebyshev polynomials.
\begin{figure}[H]
	\begin{center}
		\includegraphics[width=0.8\textwidth]{Legendre.pdf}
		\caption{\label{Legendre} First $9$ Legendre polynomials}
	\end{center}
\end{figure}
Fig.\ref{Legendre} shows first $9$ Legendre polynomials. Observe that the `frequency' of the polynomials increases  (for increasing
count) indicating higher modes. Fig.\ref{Cheb} shows Chebyshev polynomials.
\begin{figure}[H]
	\begin{center}
		\includegraphics[width=0.8\textwidth]{Chebyshev.pdf}
		\caption{\label{Cheb} First $9$ Chebyshev polynomials}
	\end{center}
\end{figure}
\noindent The $1$D modal functions are implemented in:
\begin{verbatim}
FunctionalSpace::monomials1D(args),
FunctionalSpace::legendrePolynomial1D(args),
FunctionalSpace::chebyshevPolynomial1D(args) and
FunctionalSpace::jacobiPolynomial1D(args)
\end{verbatim}
It is indeed difficult to comprehend how the polynomials shown in Fig.\ref{Legendre} and Fig.\ref{Cheb} be `orthogonal'.
The main reason for this is that we try to stick to our understanding of the $3$D space and the notion of
orthogonality that comes with it, however, the polynomial space is essentially $\infty$ dimensional space. For
this space, the definition of orthogonality depends on the definition of the Inner-Product and `metric' of that space.
For $L^2$ space, the orthogonality is defined as,
\begin{equation}
	\int_{\Omega} \psi_i \psi_j d\Omega = \alpha \delta_{ij}, \hspace{4mm}\forall \psi_i, \psi_j \in L^2\{\Omega\} ~\& ~\alpha \in \mathbb{R}
\end{equation}
where, $\delta_{ij}$ is a `Kronecker Delta' which means $\delta_{ij} = 1 $ if $i=j$ and $0$ otherwise.
The functions $\psi_i$ and $\psi_j$ are orthonormal if $\alpha = 1$.

\subsection{Nodal Basis Functions}
The `nodal' basis functions are the set of functions which are `Cardinal' in nature. What that means is, at the
interpolation point, the basis function will have value $1$ and for all other collocation points, the value of the
basis function will be zero. Thus, the interpolated value of the function agrees perfectly at the collocation points
but may have some error at other places.
The general symbol used for nodal basis functions is $\phi$.
The interpolation itself is given as,
\begin{equation}
	f^N(x) = \sum_{i=0}^{N-1} f(x_i) \phi_i(x), \hspace{4mm} x \in C^0[a,b]
\end{equation}
and, $\phi_i(x_j) = \delta_{ij}$.
We use Lagrange polynomials as the nodal basis functions. The polynomials in $1$D are give as,
\begin{equation}
	\phi_i(x) = \prod_{j=0 , j \ne i}^{N-1} \frac{(x-x_j)}{(x_i - x_j)}
\end{equation}
it is clear that at $x=x_i$, $\phi_i(x_i) = 1$.
One crucial point that needs to be taken care of is selection of the collocation points $\left[x_0, x_1, ..., x_{N-1}\right]$.
It is seen that the equi-spaced arrangement of collocation points gives poor interpolation for very high orders of
accuracies. Therefore we take these points as roots of orthonormal modal polynomials.
Popular choices for collocation points are Gauss-Legendre (LG) collocation points, Legendre-Gauss-Lobatto (LGL) points
or Chebyshev points. An important distinction between LG and LGL points is that, LGL points include the `end-points'
whereas LG points include only the internal points. For DG methods, this doesn't have a significant difference since
solving a Riemann problem at the face remains unchanged.

\subsection{Multidimensional Interpolation}
For number of dimensions greater than $1$, the interpolating polynomials are obtained depending on the type of the
element we wish to interpolate on. For hex elements, these are obtained using tensor product of $1$D polynomials.
i.e.
\begin{equation}
	\phi(x,y,z) = \phi_{1D}(x)  \phi_{1D}(y)  \phi_{1D}(z)
\end{equation}
Whereas, for other types of elements, special multidimensional polynomials are used. However, the concept of
modal and nodal basis functions remains essentially the same irrespective of the number of dimensions.


\section{Results}

The hyperparameters of the \acrshort{ESN} used in Figure \ref{fig:ESN1} and
Figure \ref{fig:ESN2} were randomly assigned and therefore not optimized. In
spite of this, preliminary predictions track reasonably well with grid demand.
The current iteration shows a potentially
misleading relationship between accuracy and training length. By inspection,
Figure \ref{fig:ESN2} is more accurate but the comparison is unfair because
the two ESNs are predicting different time periods.
We also conducted a single grid search for the optimal combination of
spectral radius ($\rho$) and noise injection (for regularization of reservoir
neurons), shown in Figure \ref{fig:gridsearch}, following the recommendations from
\cite{lukosevicius_practical_2012}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\columnwidth]{scaled_esn_network2.png}
  \caption{A simple ESN with a prediction of 100 hours into the future after
  training on 1000 hours of historical data.}
  \label{fig:ESN1}
\end{figure}
\begin{figure}[h]
  \centering
  \includegraphics[width=\columnwidth]{scaled_esn_network.png}
  \caption{A simple ESN with a prediction of 100 hours into the future. After
  training on 3500 hours of historical data.}
  \label{fig:ESN2}
\end{figure}
\begin{figure}[h]
  \centering
  \includegraphics[width=0.8\columnwidth]{noise_spectral_radius.png}
  \caption{A grid search over a range of spectral radii and noise levels. The
  optimal set minimizes the mean squared error.}
  \label{fig:gridsearch}
\end{figure}


\section{Two-Moment Model}\label{se:Two-MomentModel}

\subsection{Transport Equations}
\modified{We consider neutrino transport through a static background in a non-relativistic framework.
For simplicity, we only include neutrino--matter interactions due to emission, absorption, and isoenergetic scattering.
It is possible to include other interactions, such as neutrino--electron scattering and pair processes, in the two-moment model.
However, these interactions result in dense coupling across neutrino energies and across neutrino species, and we have not yet considered the realizability-preserving property of our scheme in this case.
We leave this for future work.}

After scaling to dimensionless units, the Boltzmann equation can be written as
\begin{equation}
  \pd{f}{t}+\vect{\ell}\cdot\nabla f
  =\f{1}{\tau}\,\cC(f),
  \label{eq:boltzmann}
\end{equation}
where the distribution function $f = f(\omega,\varepsilon,\vect{x},t)$ gives the number of neutrinos propagating in the direction $\omega\in\bbS^{2}$, with energy $\varepsilon\in\bbR^{+}$, at position $\vect{x}\in\bbR^{3}$ and time $t\in\bbR^{+}$.
$\vect{\ell}(\omega)\in\bbR^{3}$ is the unit vector parallel to the neutrino three-momentum: $\vect{p}=\varepsilon\,\vect{\ell}$.
On the right-hand side, $\tau$ is a collision time scale.
In opaque regions, where neutrinos have frequent interactions with the background, $\tau\ll1$.
In transparent regions, where neutrinos rarely interact and stream freely, $\tau\gg1$.
The collision term, $\cC(f)$, which models emission, absorption, and isoenergetic scattering is given by
\begin{equation}
  \cC(f)=\xi\,\big(\,f_{0}-f\,\big)
  +(1-\xi)\,\big(\,\f{1}{4\pi}\int_{\bbS^{2}}f\,d\omega-f\,\big),
  \label{eq:collisionTerm}
\end{equation}
where $\xi = \sigma_{\Ab} / (\sigma_{\Ab}  + \sigma_{\Scatt} )$ is the ratio of the absorption opacity $\sigma_{\Ab}$ to the total opacity.
The scattering opacity is $\sigma_{\Scatt}$.
The limit $\xi = 1$, when $\sigma_{\Scatt} = 0$, corresponds to pure absorption, while $\xi = 0$, when $\sigma_{\Ab} = 0$, corresponds to pure scattering.
The equilibrium distribution function for neutrinos is given by the Fermi-Dirac distribution:
\begin{equation}
  f_{0}(\vect{z})=\f{1}{e^{(\varepsilon-\mu(\vect{x}))/T(\vect{x})}+1},
  \label{eq:fermiDirac}
\end{equation}
where $\vect{z}:=\{\varepsilon,\vect{x}\}$, $T$ is the material temperature in energy units and $\mu$ is the neutrino chemical potential.
Both $T$ and $\mu$ depend on the spatial position $\vect{x}$.

\subsection{Two-Moment Model}

Approximate solutions to the Boltzmann equation, Eq.~\eqref{eq:boltzmann}, can be found by solving the two-moment model.
To this end, define the angular moments of the distribution function as follows:
\begin{equation}
  \big\{\,\cJ,\vect{\cH},\vect{\cK}\,\big\}(\vect{z},t)
  =\f{1}{4\pi}\int_{\bbS^{2}}f(\omega,\vect{z},t)\,\{\,1,\vect{\ell},\vect{\ell}\otimes\vect{\ell}\,\}\,d\omega.
  \label{eq:angularMoments}
\end{equation}
The zeroth moment, $\cJ$, is referred to as the particle density.
The first moment, $\bcH$, is the particle flux, and the second moment, $\bcK$, is proportional to the stress tensor.
By integrating Eq.~\eqref{eq:boltzmann} over the momentum-space angular dimension we obtain equations for the zeroth and the first moments:
\begin{equation}
  \pd{\vect{\cM}}{t}+\nabla\cdot\vect{\cF}=\f{1}{\tau}\,\vect{\cC}(\vect{\cM}),
  \label{eq:momentEquations}
\end{equation}
with $\vect{\cM}=(\cJ,\vect{\cH})^{T}$, $\vect{\cF}=(\vect{\cH},\vect{\cK})^{T}$, and
\begin{equation}
  \vect{\cC}(\vect{\cM})=\vect{\eta}-\vect{\cD}\,\vect{\cM},
  \label{eq:collisionTermMoments}
\end{equation}
where $\vect{\eta}=(\xi\,f_{0},\vect{0})^{T}$, $\vect{\cD}=\mbox{diag}(\xi,\vect{I})$, and
$\vect{I}$ is the identity matrix.
Hence, the process of solving the Boltzmann equation, Eq.~\eqref{eq:boltzmann}, for the neutrino distribution function $f(\omega,\vect{z},t)$, is replaced by solving the two-moment equations for the neutrino number density, $\cJ(\vect{z},t)$, and flux, $\bcH(\vect{z},t)$.

\subsection{Algebraic Closures}

The moment equation for $\bcH$ involves the higher moment $\bcK$ and the two-moment model is open.
To close the two-moment model, we consider algebraic closures.
For the two-moment model, algebraic closures give an approximation to $\bcK$ using the lower moments:
\begin{equation}
  \bcK = \vect{k} \cJ,
\end{equation}
where $\vect{k}$ is the Eddington tensor.
By assuming that the distribution function is symmetric about a preferred direction $\widehat{\vect{h}}=\vect{\cH}/|\vect{\cH}|$, Levermore~\cite{levermore_1984} proposed a simple form for the Eddington tensor:
\begin{equation}
  \vect{k}=\f{1}{2}\big[\,\big(1-\chi\big)\,\vect{I}+\big(3\,\chi-1\big)\,\widehat{\vect{h}}\otimes\widehat{\vect{h}}\,\big],
  \label{eq:eddingtonTensor}
\end{equation}
where $\chi=\chi(\cJ,|\vect{\cH}|)$ is the Eddington factor.
Thus, the two-moment model is closed by specifying the scalar $\chi$ in terms of $\cJ$ and $|\vect{\cH}|$.

\subsection{Constraints on the Moments}

Neutrinos are fermions and obey the Pauli exclusion principle.
Because of this, the neutrino distribution function is bounded; i.e.~$f \in [0,1]$.
As a result, the angular moments $\cJ$ and $\vect{\cH}$ and the Eddington factor $\chi$ satisfy the following bounds~\cite{levermore_1984,lareckiBanach_2011,kershaw_1976,shohatTamarkin_1943}:
\begin{align}
\cJ \in[0,1], \quad &(1-\cJ)\cJ-|\vect{\cH}| \geq 0, \label{eq:MomentsBounds} \\
  \chi_{\mbox{\tiny min}}
  =\max\big(1-\f{2}{3\cJ},h^{2}\big)
  \leq & \chi \leq \min\big(1,\f{1}{3\cJ}-\f{\cJ}{1-\cJ}h^{2}\big)=\chi_{\mbox{\tiny max}},
  \label{eq:eddingtonFactorBounds}
\end{align}
where $h = |\bcH|/\cJ$ is the flux factor.
\modified{The inequalities in Eqs.~\eqref{eq:MomentsBounds} and \eqref{eq:eddingtonFactorBounds} hold for moments as defined in Eq.~\eqref{eq:angularMoments} with $f\in[0,1]$.
In a relativistic model (e.g., \cite{shibata_etal_2011,cardall_etal_2013}), the evolved quantities --- functions of the moments in Eq.~\eqref{eq:angularMoments} --- satisfy different bounds.
We are currently exploring extensions of the realizability-preserving scheme presented here to the relativistic case.}

The constraints in Eq.~\eqref{eq:MomentsBounds} define realizable moments $\bcM$.
For fermions, realizable moments can only be constructed from a distribution satisfying the bounds $f \in [0,1]$.
Moreover, the set of realizable moments is convex: let $\cR$ be the realizability set and $\bcM_{1}, \bcM_{2} \in \cR$, then $\lambda \bcM_{1} + (1-\lambda)\bcM_{2} \in \cR$ for any $\lambda \in [0,1]$~\cite{chu_etal_2018}.
As we will see later in Section~\ref{se:SpatialDiscretization}, this convexity makes it possible to design a realizability-preserving discretization for solving the two-moment model numerically.

The inequalities in Eq.~\eqref{eq:eddingtonFactorBounds} deserve further attention.
They are as important as those in Eq.~\eqref{eq:MomentsBounds}, in maintaining consistency of the two-moment model with respect to Fermi-Dirac statistics.
When designing a numerical scheme for the two-moment model that maintains realizable $\bcM$, it is also necessary for the Eddington factor to satisfy the bounds in Eq.~\eqref{eq:eddingtonFactorBounds}.

However, recently reported CCSN simulations using two-moment neutrino transport with algebraic closures have employed Eddington factors that can violate the bounds in Eq.~\eqref{eq:eddingtonFactorBounds}.
As examples, we consider the Eddington factors discussed in~\cite{murchikova_etal_2017}, where the suitability of several algebraic closures for two-moment neutrino transport was evaluated.
In Fig.~\ref{fig:EddingtonFactorsWithDifferentClosure}, we plot the Eddington factor versus the flux factor for two occupancies: $\cJ = 0.1$ (low occupancy) and $\cJ = 0.9$ (high occupancy).
Of the algebraic closures plotted, few satisfy the bounds on the Eddington factor in Eq.~\eqref{eq:eddingtonFactorBounds}.
Kershaw~\cite{kershaw_1976}, Wilson~\cite{wilson_1975,leblancWilson_1970}, Levermore~\cite{levermore_1984}, Minerbo~\cite{minerbo_1978}, and Janka~2~\cite{janka_1992} closures may work fine when the occupancy is low.
When the occupancy is high, the Eddington factor due to these closures exceeds the upper bound for Fermi-Dirac statistics.
The Eddington factor of Janka~1~\cite{janka_1991} may violate both the upper and lower bound on the Eddington factor.
Only the closure due to Cernohorsky~\&~Bludman~\cite{cernohorskyBludman_1994} satisfies both the upper and lower bounds.
This is not surprising as this is the only closure based on Fermi-Dirac statistics.
Although the Levermore and Minerbo closures do not satisfy the bounds in Eq.~\eqref{eq:eddingtonFactorBounds}, they are widely used in simulations of neutrino transport in CCSNe and compact binary mergers; e.g., O'Connor \& Couch~\cite{oConnorCouch_2018}, Pan et al.~\cite{pan_etal_2018}, Glas et al.~\cite{glas_etal_2018}, Just et al.~\cite{just_etal_2018}, and Foucart et al.~\cite{foucart_etal_2015} use the Minerbo closure, while Vartanyan et al.~\cite{vartanyan_etal_2018}, Cabezon et al.~\cite{cabezon_etal_2018}, Kuroda et al.~\cite{kuroda_etal_2016}, and Fujibayashi et al.~\cite{fujibayashi_etal_2017} use the Levermore closure.
When employing these closures in conditions of high occupancy, a numerical scheme may evolve the moments outside the realizable domain of Fermi-Dirac statistics given in Eq.~\eqref{eq:MomentsBounds}.
If this were to happen, the update step may give $\cJ>1$.
Considering the fact that the collision term contains blocking factors --- i.e.~$(1-\cJ)\times$ something positive --- $\cJ > 1$ would change the blocking factors' sign, and it would be difficult to predict the impact of the subsequent induced errors on the simulation outcome.
Besides, even with the simplified collision term Eq~\ref{eq:collisionTermMoments}, $\cJ>1$ could result in information loss (such as loss to the neutrino distribution for the groups that exceed the bounds), closure failure, and other unpredictable consequences.
If treatments are developed to map the unrealizable moments into the realizable domain, they should be developed to conserve lepton number, energy, and momentum.

\begin{figure}[h]
  \centering
  \begin{tabular}{cc}
    \includegraphics[width=0.5\textwidth]{figures/Closures0_10}
    \includegraphics[width=0.5\textwidth]{figures/Closures0_90}
  \end{tabular}
   \caption{Plot of Eddington factors $\chi$ versus flux factor $h$ for different values of $\cJ$ for various algebraic closures: $\cJ=0.1$ (left panel, low occupancy) and $\cJ=0.9$ (right panel, high occupancy).  In each panel we plot the Eddington factors of two-moment closures due to Kershaw (red), Wilson (yellow), Levermore (green), Minerbo (light blue), Cernohorsky \& Bludman (blue), Janka~1 (purple), and Janka~2 (pink) .  We also plot $\chi_{\mbox{\tiny min}}$ and $\chi_{\mbox{\tiny max}}$, defined in Eq.~\eqref{eq:eddingtonFactorBounds} (lower and upper dashed black lines, respectively).}
  \label{fig:EddingtonFactorsWithDifferentClosure}
\end{figure}


\documentclass{article}

\include{stddefs}
\include{imodefs}

\chapterno{4}

\begin{document}

\chapter{What is optimization?}\label{whatisopt}

In this chapter we will denote the set of column vectors with $d$ rows by $\RR^d$.
The arithmetic
of $d\times 1$ matrices apply i.e., we may add vectors in $\RR^d$ and
multiply them by a number in $\RR$.

In the next chapter we will introduce them as \emph{euclidean} vector spaces. The
term euclidean refers to a norm: a function measuring the size of a vector. In this
chapter we only need the structure as column vectors.

\section{What is an optimization problem?}

An optimization problem consists of maximizing or minimizing a function subject
to constraints.

Below are two classical examples related to minimizing (non-linear) functions subject to
(non-linear) constraints. These are actually examples of
convex optimization problems. More about that later.

\begin{example}

  A cylindrical can is supposed to have a volume of $V$ $\text{m}^3$. The material used
  for the top and bottom costs $T$ DKK per $\text{m}^2$ and the material used for
  the side costs $S$ DKK per $\text{m}^2$. Give the dimensions $r$ and $h$ of the
  can minimizing the price of the materials.

\includegraphics{cylinder.svg}

The cost of the top and bottom pieces are $2 \pi r^2 T$. The cost of the
side material is $2\pi r h S$. The constraint is that the volume must be
$V$. This is expressed in the equation $\pi r^2 h = V$. All in all
the optimization problem is
\begin{align*}
  &\text{Minimize} &2\pi r^2 T + 2\pi r h S&\\
  &\text{with constraints}\\
  &&\pi r^2 h &= V\\
  &&r &\geq 0\\
  &&h &\geq 0,
\end{align*}
where $V, T$ and $S$ are constants.

\beginshex
Can you see a way of solving this optimization problem by eliminating $h$ in
the constraint $\pi r^2 h = V$?

\begin{hideinbutton}{Hint}
$$
\pi r^2 h = V\iff h = \frac{V}{\pi r^2}
$$
and $h$ can be inserted in $2\pi r^2 T + 2 \pi r h S$. Why is this helpful?
\end{hideinbutton}
\endshex
\end{example}

\begin{example}
  A person is in distress $D$ meters from the beach. The life guard
  spots the situation, but is $d$ meters from where he would naturally jump
  in the water as indicated below. The life guard runs $8$ m/s on the
  beach and swims $1$ m/s in the water. How far ($x$) should he run along the
  beach before jumping into the water in order to minimize the time needed
  to reach the person in distress?

  \includegraphics{lifeguard.png}

  The time spent moving with a speed of $v$ over a distance of $s$ is
  $$
  t = \frac{s}{v}.
  $$
  If the life guard jumps in the water at the point $x$ he will have
  to swim a distance of
  $$
  \sqrt{D^2 + (d-x)^2}
  $$
  using the Pythagorean theorem. Therefore the optimization problem becomes
\begin{align*}
  &\text{Minimize} &\frac{x}{8} + \sqrt{D^2 + (d-x)^2}&\\
  &\text{with constraints}\\
  &&x\geq 0\\
\end{align*}
Strictly speaking we do not need the constraint $x\geq 0$, as the life guard is free
to run in the other direction. So the optimization problem is simply to minimize
$$
\frac{x}{8} + \sqrt{D^2 + (d-x)^2}
$$
with no strings attached i.e., $x\in \RR$ is just assumed to be any number.

\begin{sage}
D = 200
d = 100
plot(x/8 + sqrt(D**2 + (d-x)**2), (x, 0, d))
\end{sage}

\end{example}

\beginshex
  You need to build a rectangular fence in front of your house for a herb garden.
  Your house will make up one side of the rectangle, so you only need to build three
  sides. Suppose you have 10 m of wire. What is the maximum area of the herb garden you can
  wall in?
\endshex

\section{General definition}

An optimization problem consists of a subset $D\subseteq \RR^d$ and
a function $f:D\rightarrow \RR$. We will consider optimization problems in the context
of minimization. Optimize in this situation means minimize.

\begin{definition}[emph]\label{defopt}
  In our most general setting an optimization problem looks like
\begin{align*}
  &\text{Minimize} &f(x)&\\
  &\text{with constraint}\\
  &&x\in C,
\end{align*}
     where $C$ and $D$ are subsets of $\RR^d$ with $C\subseteq D$ and $f: D\rightarrow \RR$ is a function.
A solution to the optimization problem is a vector $x_0\in C$, such that
$$
f(x_0) \leq f(x)
$$
for every $x\in C$. Here $x_0$ is called an \emph{optimum} and $f(x_0)$ is called the \emph{optimal value}.

We will often write the optimization problem defined above in short form as
\begin{align*}
  &\text{min}\, f(x)\\
  &x\in C.
\end{align*}
\end{definition}

  The complexity of the problem depends very much on the nature of $C$ and $f$. Also,
  we cannot even be certain that an optimization problem has a solution. Consider
  the problem

  \begin{align*}
    &\text{min}\,x\\
    &x\leq 0
  \end{align*}

  Here $x$ can be made arbitrarily small subject to the constraint $x\leq 0$ and
  the problem has no solution.

  \begin{remark}[emph]
    We have deliberately not included maximization problems in Definition \ref{defopt}. This is
because a maximization problem, such as
\begin{align}\label{maxprob}
  &\text{Maximize} &f(x)&\\
  &\text{with constraint}\\
  &&x\in C
\end{align}

 can be formulated as the minimization problem
\begin{align}
  &\text{Minimize} &-f(x)&\\
  &\text{with constraint}\\
  &&x\in C.
\end{align}

Again, we will use the short notation
\begin{align*}
  &\text{max}\,f(x)\\
  &x\in C
\end{align*}
for the maximization problem in \eqref{maxprob}.
A solution to \eqref{maxprob} is a vector $x_0\in C$, such that
$$
f(x_0) \geq f(x)
$$
for every $x\in C$. Again, $x_0$ is called an optimum and $f(x_0)$ the
optimal value.

\end{remark}

\beginshex
Suppose that the maximization problem
\begin{align}\label{maxprob1}
  &\text{max}\, f(x)\\
  &x\in C
\end{align}
is formulated as
the minimization problem
\begin{align}\label{minmaxprob}
  &\text{min}\, -f(x)\\
  &x\in C.
\end{align}
Show that $-m$ is the optimal value
and $x_0$ the optimum for \eqref{maxprob1} if
$m$ is the optimal value
and $x_0$ the optimum for \eqref{minmaxprob}.
\endshex

    \beginshex\label{mothercopt}
    Suppose that $a > 0$. Solve the optimization problem
\begin{align*}
    &\text{Minimize} &a x^2 + b x + c\\
    &\text{with constraint}\\
    &&x\in \RR.
  \end{align*}

    \endshex

    \section{Convex optimization}

    Particularly well behaved optimization problems are the convex ones. These are optimization
    problems, where $C\subseteq \RR^d$ is a convex subset and $f:C\rightarrow \RR$ a
    convex function in Definition \ref{defopt}. To define these concepts we first introduce
    the notion of
    a line in $\RR^d$.

    \begin{definition}[emph]\label{defline}
      A line $L\subseteq \RR^d$ is a subset of the form
      $$
      L = \{u + t v \mid t\in \RR\},
      $$
      where $u, v\in \RR^d$ with $v\neq 0$.
    \end{definition}

\begin{example}
A line $L$ in the plane $\mathbb{R}^2$ is (usually) given by its equation
\begin{equation}\label{lineeq}
y = a x + b.
\end{equation}
This means that it consists of points $(x, y)\in \mathbb{R}^2$ satisfying $y = a x + b$.
Here $a$ can be interpreted as the slope of the line and $b$ the intersection with the
$y$-axis.

What about all the points $(x, y)$ with $x = 0$? Certainly they also deserve to be called
a line. However, they do not satisfy an equation like \eqref{lineeq}. Informally, this line
has infinite slope.

Therefore we introduce the parametric representation of a line: a line is the
set of points of the form
$$
\begin{pmatrix} x_0 \\ y_0 \end{pmatrix} +
t \begin{pmatrix} u_0 \\ v_0 \end{pmatrix},
$$
where $t\in \mathbb{R}$,
$$
\begin{pmatrix} x_0 \\ y_0\end{pmatrix}
$$
is any point on the line and
$$
\begin{pmatrix} u_0 \\ v_0\end{pmatrix}
$$
is a non-zero (directional) vector.

\begin{figure}
  \includegraphics{line.png}
  \begin{center}
    Example of a line in $\RR^2$ with (directional) vector $v\in \RR^2$ through the point $u\in \RR^2$.
  \end{center}
\end{figure}

\end{example}

\beginshex
Compute the parametric representation of the line through the points $(1, 1)$ and $(2, 3)$.
\endshex

\beginshex
What is the parametric representation of the line with the equation
$y = a x + b$? What is the parametric representation of the
line consisting of the points $(x, y)$ with $x=0$?
\endshex

    \beginshex
    Show in Definition \ref{defline} that if $L$ is given by $u$ and $v$, then
    you might as well replace $v$ by $s v$, where $s$ is a real number and $s\neq 0$.  It gives
    the same line.
    \endshex

\begin{exercise}[emph]
  Show that there is a unique line passing through two distinct points
    $x, y\in \RR^d$ and that it is given by $u = x$ and $v = y - x$ in Definition \ref{defline}.
\end{exercise}

    \beginshex
    Do the points
    $$
    \begin{pmatrix}1\\ 2\\ 3\end{pmatrix}, \quad \begin{pmatrix} 4\\ 5\\ 6\end{pmatrix}
    \quad\text{and}\quad \begin{pmatrix} 7\\ 8\\ 9\end{pmatrix}
    $$
    lie on the same line in $\RR^3$?
    \endshex

    \beginshex
    Show that the line through two distinct points $x, y\in \RR^d$ is equal to the subset
    $$
    \{ (1-t) x + t y \mid t\in \RR\} \subseteq \RR^d.
    $$
    \endshex

    \begin{definition}[emph]\label{Def:convexsubset}
    A convex subset $C\subseteq \RR^d$ is a subset that contains the
    line segment between any two of its points $x, y\in C$ i.e.,
    $$
    (1 - t) x + t y\in C
    $$
    for every number $t$ with $0\leq t \leq 1$.
    \end{definition}

\begin{figure}
  \includegraphics{nonconvex.png}
  \begin{center}
    Example of non-convex subset of $\RR^2$.
  \end{center}
\end{figure}

\begin{quizexercise}[showhide]
  \begin{quiz}
    \question
    Which of the subsets below are convex?
    \answer{T}
    $$C=\{1\}\subseteq \RR$$
    \answer{F}
    $$C=\{1, 2\}\subseteq \RR$$
    \answer{T}
    The points $C$ on a line $y = a x + b$ in $\RR^2$.
    \answer{F}
    $$
    C=\{(x, y)\in \RR^2 \mid x y \geq 0\}.
    $$
  \end{quiz}
\end{quizexercise}

    \beginshex
      A closed interval in $\RR$ is a subset of the form
      $$
      [a, b] = \{x \mid a \leq x \leq b\}
      $$
      for $a \leq b$. Prove that $[a, b]$ is a convex subset of $\RR$.

\begin{hideinbutton}{Hint}

Keep cool and just apply the definitions! First of all, $x\in [a, b]$ if and only if
\begin{equation}\label{intdef}
a\leq x \land x \leq b.
\end{equation}
Now pick any $t\in [0, 1]$. We must show that if $x\in [a, b]$ and $y\in [a, b]$, then
$$
(1 - t)x + t y \in [a, b].
$$
You may also write this out as
$$
a\leq x \land x \leq b\qquad\land\qquad a\leq y \land y \leq b
$$
implies that
$$
a\leq (1 - t)x + t y \qquad \land \qquad (1 - t)x + t y \leq b.
$$

\begin{hideinbutton}{Hint}
$$
a\leq x\implies (1-t) a \leq (1-t) x\qquad\land\qquad a\leq y \implies t a \leq t y
$$
implies that
$$
(1 - t) a + t a \leq (1-t)x + t y.
$$
What is $(1-t)a + a$?
\end{hideinbutton}

\end{hideinbutton}

\endshex

    \beginshex
    Let $A$ and $B$ be convex subsets of $\RR^d$. Prove that $A\cap B$ is a
    convex subset of $\RR^d$. Generalize this to show that if $A_1, \dots, A_n$
    are any number of convex subsets of $\RR^d$, then their intersection
    $$
    A_1 \cap \cdots \cap A_n
    $$
    is a convex subset of $\RR^d$. Is the union of two convex subsets necessarily convex?
    \endshex

    \begin{definition}[emph]\label{Def:convexfunction}
    A convex function is a function $f: C\rightarrow \RR$
    defined on a convex subset $C\subseteq \RR^d$, such that
    $$
    f((1 - t) x + t y) \leq (1-t) f(x) + t f(y)
    $$
    for every number $t$ with $0\leq t \leq 1$.
    \end{definition}

\begin{figure}
  \includegraphics{convexfct.png}
  \begin{center}
    Graph of convex function. The line segment between $(x, f(x))$ and $(y, f(y))$
    lies above the graph.
  \end{center}
\end{figure}

    \beginshex\label{exconvfct}
    Let the function $f: \RR \rightarrow \RR$ be given by
    $f(x) = a x + b$, where $a, b\in \RR$. Show that $f$ is a convex
    function.

    \begin{hideinbutton}{Hint}
      Try the case $a = 0$ first.
    \end{hideinbutton}

    Can you at this point prove that $f(x) = x^2$ is a convex function?

    \begin{hideinbutton}{Hint}
      Simplify
      $$
      (1 - t) x^2 + t y^2 - ((1-t) x + t y)^2
      $$
      to an expression that has to be non-negative.

      \begin{hideinbutton}{Hint}
        \begin{sage}
from sympy.abc import x, y, t
f = (1-t)*x^2 + t*y^2 - ((1-t)*x + t*y)^2
factor(f)
        \end{sage}
    \end{hideinbutton}
  \end{hideinbutton}

  Using that $f(x) = x^2$ is a a convex function, prove that $g(x) = x^4$ is a convex function and in general, that $h(x) = x^m$ is a convex function, where
  $m = 2n$ is an even natural number.

\begin{hideinbutton}{Hint}
      $$g(x) = f(x)^2$$
    \end{hideinbutton}

    It is a fact that $f(x) = x^3$ is not a convex function, but can
    you explain this using the definition of a convex function?

    \begin{hideinbutton}{Hint}
      Try $x = -1, y = 0$ and $t =\frac{1}{2}$.
    \end{hideinbutton}
    \endshex

    \begin{lemma}[emph]
    Let $f: \RR^d\rightarrow \RR$ be a convex function. Then the subset
    $$
    C = \{x\in \RR^d \mid f(x) \leq a\}
    $$
    is a convex subset of $\RR^d$, where $a\in \RR$.
    \end{lemma}
    \begin{proof}
    Suppose that $u, v\in C$ and $t\in [0, 1]$. Looking at Definition \ref{Def:convexsubset}
we must prove that
$$
(1-t) u + t v \in C.
$$
By the definition of $f$ being convex (Definition \ref{Def:convexfunction}), it follows that
$$
f((1-t) u + t v) \leq (1- t) f(u) + t f(v).
$$
But, since $f(u)\leq a$ and $f(v)\leq a$ we have
\begin{align*}
(1- t) f(u) &\leq (1-t) a\\
t f(v) &\leq t a
\end{align*}
and therefore
$$
(1- t) f(u) + t f(v) \leq (1-t) a + t a = a.
$$
Therefore,
$$
f((1-t) u + t v)\leq a
$$
and $(1-t) u + t v\in C$.
\end{proof}

    We do not have the tools yet to prove the crucial
    result about convex optimization problems, but at least we can state it.

    \begin{remark}[emph]
    In hunting for optimal solutions to an optimization problem one is often stuck with a
    point $x_0\in \RR^d$, which is optimal locally. This means that $f(x_0)\leq f(x)$ for
    every $x$ that is sufficiently close to $x_0$ (we will explain what this means in the next
    chapter). The remarkable thing that happens in a convex optimization problem is
    that if $x_0$ is optimal locally, then it is a global optimum! It satisfies
    $f(x_0)\leq f(x)$ not only for $x$ close to $x_0$, but for every $x\in C$.
\end{remark}

    The optimization problem in Exercise \ref{mothercopt} is a very typical convex optimization
    problem.

    Below you see a plot of the function (press Compute)
    $$
    f(x) = x^3 + 2 x^2 + x + 1
    $$
    restricted
    to the interval $[-1.5, 0]$. You can see that it has a local minimum around $-0.3$ and
    also that this minimum is not a global minimum (certainly $f(-1.4)$ is smaller). So $f(x)$
    is not a convex function on this interval (but if you look at it more locally on the
    interval $[-0.6, 0]$ it is a convex function.

    \begin{sage}
      plot(x**3+2*x**2+x+1, (x, -1.5, 0))
    \end{sage}

    \beginshex
    Solve the optimization problem
    \begin{align*}
    &\text{Minimize} &x^3 + 2 x^2 + x + 1\\
    &\text{with constraint}\\
      &&x\in C
    \end{align*}
    for $C = [-0.6, 0]$ and $C = [-2, 0]$.
    \endshex

  \section{Linear optimization}

  The simplest convex optimization problems are the linear ones. Recall that a
  linear function $f: \RR^d\rightarrow \RR$ has the form
  $$
  f\begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix} =
  c_1 x_1 + \cdots + c_n x_n
  $$
  for $c_1, \dots, c_n\in \RR$. Usually we write this with matrix notation as
  $$
  f(x) = c^T x,
  $$
  where
  $$
  c =
  \begin{pmatrix} c_1 \\ \vdots \\ c_n \end{pmatrix}\qquad \text{and} \qquad
  x = \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix}.
  $$

  \beginshex
  Show that a linear function is convex.
  \endshex

  A linear optimization problem is not about minimizing a linear function over
  an arbitrary convex subset. We choose the convex subset as an intersection of
  subsets of the form
  $$
  \{x\in \RR^d\mid a^T x \leq b\},
  $$
  where $a\in \RR^d$ is a non-zero vector and $b\in \RR$ a number i.e.,
  a linear optimization problem has the form
  \begin{align*}
    &\text{min}\,c^T x\\
    &x\in C,
  \end{align*}
  where
  \begin{align}\label{linsubs}
    C &= \{x\in \RR^d \mid a_1^T x \leq b_1, \dots, a_m^T x\leq b_m\}\\
    &= \{x\in \RR^d \mid a_1^T x \leq b_1\} \cap \cdots \cap \{x\in \RR^d \mid a_m^T x \leq b_m\}
  \end{align}
  and $c, a_1, \dots, a_m\in \RR^d$ and $b_1, \dots, b_m\in \RR$.

  \beginshex
  Use a selection of previous exercises to show that
  the subset $C$ defined in \eqref{linsubs} is a convex subset of $\RR^d$.
  \endshex

  Using matrix notation we write $C$ as
  $$
  C = \{x\in \RR^d \mid A x \leq b\},
  $$
  where $A$ is the $m\times d$ matrix with row vectors $a_1^T, \dots, a_m^T$ and
  $$
  b =
  \begin{pmatrix}
    b_1 \\ \vdots \\ b_m
  \end{pmatrix}.
  $$

  \begin{example}\label{exlinopt}

     Here is a concrete example for $d = 2$. The optimization problem
  \begin{align*}
    &\text{Maximize} &x + y&\\
    &\text{with constraints}\\
    &&2 x + y &\leq 1\\
    &&x + 2 y &\leq 1\\
    &&x &\geq 0\\
    &&y &\geq 0
  \end{align*}
  translates into matrix notation with the matrices
  \newcommand{\mi}{\hphantom{-}}
  $$
  c = \begin{pmatrix} 1 \\ 1 \end{pmatrix}, \qquad
  A = \begin{pmatrix} \mi 2 & \mi 1 \\ \mi 1 & \mi 2 \\ -1 & \mi 0 \\ \mi 0 & - 1\end{pmatrix}\qquad \text{and}\qquad
  b = \begin{pmatrix} 1 \\ 1 \\ 0 \\ 0 \end{pmatrix}.
  $$

  In this case it is helpful to draw the optimization problem in the plane $\RR^2$. This is done below.

  \includegraphics{LP.png}
  \begin{center}
    Constraints pictured as shaded area above. Optimum occurs in a vertex (corner).
    \end{center}

  \end{example}

  We will give a general (but rather slow) algorithm below for solving
  linear optimization problems. In fact it all boils down to solving
  systems of linear inequalities. Sometimes linear optimization is
  referred to as \url{linear
    programming}{https://en.wikipedia.org/wiki/Linear_programming}. The
  basic theory of linear programming was pioneered, among others, by
  one of the inventors of the modern computer, John von Neumann.

\includegraphics{vonNeumann.gif}
\begin{center}
  \url{John von Neumann}{https://en.wikipedia.org/wiki/John_von_Neumann} (1903-1957). Picture from \url{LANL}{https://en.wikipedia.org/wiki/Los_Alamos_National_Laboratory}.
\end{center}

  Sage has much more advanced algorithms built in for solving (integer) linear optimization
  problems. I have translated the linear optimization problem in Example \ref{exlinopt} into
  Sage below.

\begin{example}\label{sageLP}
\begin{sage}
LP = MixedIntegerLinearProgram(solver = "GLPK")

# generate variables (do not require integer solution)
x = LP.new_variable(integer=False)

# Linear function to be maximized (notice maximized!)

LP.set_objective(x[1] + x[2])

# Add constraints
LP.add_constraint( 2*x[1] + x[2] <= 1 );
LP.add_constraint( x[1] + 2*x[2] <= 1 );
LP.add_constraint( x[1] >= 0 );
LP.add_constraint( x[2] >= 0 );

LP.show()

oval = LP.solve()
xopt = LP.get_values(x);

print('Optimal Value:', oval)
print('occurs in')
for i, v in xopt.items():
    print('x_%s = %f' % (i, v))
\end{sage}
\end{example}

\section{Fourier-Motzkin elimination}

Fourier-Motzkin elimination is a classical method (dating back to 1826) for solving linear inequalities.
It is also a key ingredient in an algorithm for solving linear optimization problems.

I am convinced that
the best way to explain this method is by way of an extended example. For more formalities you may
consult \url{Chapter 1}{https://www.worldscientific.com/doi/suppl/10.1142/8527/suppl_file/8527_chap01.pdf} of my book \url{Undergraduate Convexity}{https://www.worldscientific.com/worldscibooks/10.1142/8527}.

Consider the linear optimization problem

\begin{equation}\label{optfmex}
  \begin{array}{llrrrr}
    \text{Maximize} &x + y\\
    \text{with constraints}\\
    &2 x &+ &y &\leq &6\\
    &x &+ &2 y &\leq &6\\
    &x &+ &2 y &\geq &2\\
    &x && &\geq &0\\
    &&&y &\geq &0.
  \end{array}
\end{equation}

We might as well write this as

$$
  \begin{array}{llrrrl}
    \text{Maximize} &z\\
    \text{with constraints}\\
    &&& z &= &x + y\\
    &2 x &+ &y &\leq &6\\
    &x &+ &2 y &\leq &6\\
    &x &+ &2 y &\geq &2\\
    &x &&&\geq &0\\
    &&&y &\geq &0
  \end{array}
$$

  by adding the extra variable $z$. This enables us to reformulate the problem as follows: Find
  the maximal value of $z$, such that there exists $(x, y)\in \RR^2$ with
  $$
  (x, y, z)\in P,
  $$
  where $P\subseteq \RR^3$ is the set of solutions to the system

\begin{equation}\label{Pset}

  \begin{array}{llrrrl}
    &&& z &= &x + y\\
    &2 x &+ &y &\leq &6\\
    &x &+ &2 y &\leq &6\\
    &x &+ &2 y &\geq &2\\
    &x &&&\geq &0\\
    &&&y &\geq &0
  \end{array}

\end{equation}

  of \footnote{inequalities}{An equality $a = b$ is logically equivalent to the two
    inequalities $a\leq b$ and $a \geq b$ in the sense that $(a\leq b) \wedge (a\geq b) \iff a = b$.}.

  We have the Gauss elimination method for solving systems of linear equations. How do we now solve
  \eqref{Pset}, where we also have inequalities?

  Well, at first we can actually do a Gauss elimination step by eliminating $x$ in the equation
  $z = x + y$ i.e., by putting $x = z - y$. This is then inserted into the inequalities in
  \eqref{Pset}:

$$
\begin{array}{llrrrl}
    &2 (z - y) &+ &y &\leq &6\\
    &(z - y) &+ &2 y &\leq &6\\
    &(z - y) &+ &2 y &\geq &2\\
    &(z - y) &  &    &\geq &0\\
    &          &  &  y &\geq &0
\end{array}
$$

and we get the system

$$
  \begin{array}{llrrrl}
    &2 z &- &y&\leq &6\\
    &z &+ &y &\leq &6\\
    &z &+ & y &\geq &2\\
    &z &- &y &\geq &0\\
    &  &  &y   &\geq &0
  \end{array}
$$
  of inequalities
  in the variables $z$ and $y$. Now we only have inequalities left and we have to invent a trick for
  eliminating $y$. Let us isolate $y$ on one side of the inequality signs $\leq$ and $\geq$:

  $$
    \begin{array}{llrrrl}
    &2 z &- &6 &\leq &y\\
    &6 &- &z &\geq &y\\
    &2 &- &z &\leq &y\\
    & & &z &\geq &y\\
    &&&0 &\leq &y
  \end{array}
$$

  Written a little differently this is the same as

\begin{equation}\label{fmex}
  \begin{array}{llllllllll}
    &2 z &- &6&\leq &{\color{red} y} \\
    &&&&&{\color{red} y} &\leq &6 &- &z\\
    &2&- &z&\leq &{\color{red} y} \\
    &&&&&{\color{red} y} &\leq &z\\
    &&&0&\leq &{\color{red} y}
  \end{array}
\end{equation}

Now the scene is set for elimination of $y$. Listen carefully. First the inequalities in
\eqref{fmex} can be boiled down to the following two inequalities

\begin{equation}\label{fmex1}
  \begin{array}{lllll}
\max(2 z - 6, 2 - z, 0) &\leq &{\color{red} y}\\
&& {\color{red} y} & \leq &\min(6 - z, z)
  \end{array}
\end{equation}
by using (repeatedly) that  $\max(a, b) \leq c \iff a\leq c \wedge b\leq c$ and
  $c\leq \min(a, b) \iff c\leq a \wedge c\leq b$ for three
numbers $a, b, c\in \RR$.

Then, finally comes the (Fourier-Motzkin) elimination step: The existence of a solution to \eqref{fmex1} is
equivalent to the single inequality

\begin{equation}\label{singleineq}
\max(2 z - 6, 2 - z, 0) \leq \min(6 - z, z).
\end{equation}

This single inequality can be exploded or expanded (see Exercise \ref{explex}) into the following $6 = 3\cdot 2$ inequalities

\begin{align*}
2 z - 6 &\leq 6 - z\\
2 z - 6 &\leq z\\
2 - z &\leq 6 - z\\
2 - z &\leq z\\
0 &\leq 6 - z\\
0 &\leq z.
\end{align*}

Similarly to \eqref{fmex} we now isolate $z$ from the above inequalities:

\begin{equation*}
  \begin{array}{lllll}
&&{\color{red} z} &\leq &4\\
&&{\color{red} z} &\leq &6\\
1&\leq &{\color{red} z}\\
&&{\color{red} z} &\leq &6\\
0&\leq &{\color{red} z}
\end{array}
\end{equation*}

and find that

\begin{equation*}
  \begin{array}{lllll}
\max(1, 0) = 1 &\leq &{\color{red} z}\\
&& {\color{red} z} & \leq &\min(4, 6) = 4.
  \end{array}
\end{equation*}

Therefore the maximum in the optimization problem \eqref{optfmex} is $z = x + y = 4$.
How do we now find numbers $x, y\in \RR$ satisfying the constraints
in the optimization problem \eqref{optfmex} with $z = x + y = 4$?

This is simply done inserting first $z = 4$ in \eqref{fmex1}. Here you get the two inequalities
$2 \leq y$ and $y\leq 2$. Therefore $y = 2$. Since we had $x = z - y$ from the very beginning
we therefore get $x = 2$ and we have the unique solution to the optimization problem.

\beginshex
What is the solution if we replace Maximize with Minimize in the optimization problem \eqref{optfmex}?
\endshex

\beginshex\label{explex}

Prove the following:

    Let $x_1, \dots, x_m, y_1, \dots, y_n\in \RR$ be $m + n$ numbers. Then
    $$
    \max(x_1, \dots, x_m) \leq \min(y_1, \dots, y_n)
    $$
    if and only if the $m n$ inequalities
    $$
    \begin{matrix}
      &x_1 \leq y_1 &x_1 \leq y_2 &\dots &x_1\leq y_n\\
      \\
      &\vdots & \vdots &\ddots &\vdots\\
      \\
      &x_m \leq y_1 &x_m \leq y_2 &\dots &x_m\leq y_n
    \end{matrix}
    $$
    are satisfied.
\endshex

\beginshex

The following is Exercise 1.8 from my book \url{Undergraduate Convexity}{https://www.worldscientific.com/worldscibooks/10.1142/8527}.

A vitamin pill $P$ is produced using two ingredients $M_1$
  and $M_2$. The pill needs to satisfy four constraints for the vital
  vitamins $V_1$ and $V_2$. It must contain at least $6$ milligrams and
  at most $15$ milligrams of $V_1$ and at least $5$ milligrams and at
  most $12$ milligrams of $V_2$. The ingredient $M_1$ contains $3$
  milligrams of $V_1$ and $2$ milligrams of $V_2$ per gram.  The
  ingredient $M_2$ contains $2$ milligrams of $V_1$ and $3$ milligrams
  of $V_2$ per gram:

$$
\def\arraystretch{1.5}
\begin{array}{c|cc}
& V_1 & V_2\\ \hline
M_1 & 3 & 2\\
M_2 & 2 & 3
\end{array}
$$

Let $x$ denote the amount of $M_1$ and $y$ the amount of $M_2$
  (measured in grams) in the production of a vitamin pill. Write down
  a system of linear inequalities in $x$ and $y$ describing the
  constraints above.

  We want a vitamin pill of minimal weight satisfying the
  constraints. How many grams of $M_1$ and $M_2$ should we mix?

Use Fourier-Motzkin elimination to solve this
  problem.

Check your solution by modifying the input to the Sage code in Example \ref{sageLP}.

\endshex

\section{Application in machine learning and data science}\label{sec:mlds}

To start with, consider a toy example of a machine learning problem: we wish
to tell the gender of a person based on a data point consisting of
the height and weight of the person.

To do this we train our model by measuring the height and weight of a
lot of people. Each of these measured data points are labeled
female or male according to the gender of the person.

\includegraphics{sephp.png}

Given a new data point, we wish to tell if the person is female or male. Here we
consider a very simple model for doing this. First we need to introduce some
new mathematical terms. We will introduce the terms generally for
data points in $\RR^d$ and not just in $\RR^2$ as above.

A hyperplane in $\RR^d$ is a generalization of a line $y = a x + b$ in the plane. In general
a hyperplane is defined as the set of points $x\in \RR^d$ satisfying
$$
a^T x = b,
$$
where $a\in \RR^d$ is a non-zero vector and $b$ is a number. A hyperplane divides $\RR^d$
into two subsets: the points above or on the hyperplane satisfying $a^T x \geq b$ and the ones
below the hyperplane satisfying $a^T x < b$.

Suppose we are given a data set as a finite set of points in $\RR^d$ and that each of
these points are labeled with either a blue or a red color. We wish to
find a hyperplane, such that the blue points are above and the red points
are below the hyperplane.

\includegraphics{sephpWithline.png}

We may then use the hyperplane to predict the label of a point. This could be
gender, if you win or lose money buying a stock, anything with
a binary classifier.

  \subsection{Formulation as a linear optimization problem}

  Suppose that the points labeled blue are $x_1, \dots, x_m\in \RR^d$ and
  the points labeled red are $y_1, \dots, y_n\in \RR^d$. Then we wish to
  find $a\in \RR^d$ and $b\in \RR$, such that
  $$
  a^T x_i > b
  $$
  for $i = 1, \dots, m$ and
  $$
  a^T y_j < b
  $$
  for $j = 1, \dots, n$. One can show that these strict inequalities may be solved
  for $a$ and $b$ if and only if the inequalities
  \begin{align*}
  a^T x_i &\geq b + 1\\
  a^T y_j &\leq b - 1
  \end{align*}
  are solvable for $a$ and $b$, where $i = 1, \dots, m$ and $j = 1, \dots, n$.

  It is, however, not realistic to expect data to behave this nicely. Instead one invents
  the rather ingenious linear optimization problem

  \begin{align}\label{linoptnice}
    &\text{Minimize} &\frac{1}{m}(u_1 + \cdots + u_m) + \frac{1}{n}(v_1 + \cdots + v_n)\\
    &\text{with constraints}\\
    &&a^T x_i + u_i&\geq b + 1\\
    &&a^T y_j - v_j &\leq b - 1\\
    &&u_i&\geq 0\\
    &&v_j&\geq 0
  \end{align}

  for $i = 1, \dots, m$ and $j = 1, \dots, n$. This linear optimization problem has optimal value zero
  if and only if data can be separated strictly. Otherwise, it finds a hyperplane minimizing the mean errors
  for the points involved.

  The linear optimization problem \eqref{linoptnice} may look untied to the real world, but it has
  been used very successfully in the diagnosis and prognosis of breast
  cancer. See \url{Mangasarian et al.}{https://www.jstor.org/stable/171686}

  In the sage window below we have implemented the solution of the
  linear optimization problem \eqref{linoptnice}, where the output is
  a graphical illustration of the optimal line, that separates the
  points in \texttt{xpts} and \texttt{ypts} with the smallest mean
  error as defined in the function to be minimized in \eqref{linoptnice}.

\begin{sage}
# Enter blue data points below
xpts = [(3, 1), (4, 2), (4, 3), (3, 5)]
# Enter red data points below
ypts = [(2, 2), (2.5, 4.5), (3, 2), (4, 1)]

LP = MixedIntegerLinearProgram(solver = "GLPK", maximization=False)

# generate variables (do not require integer solution)
a = LP.new_variable(integer=False)
b = LP.new_variable(integer=False)
u = LP.new_variable(integer=False, nonnegative=True)
v = LP.new_variable(integer=False, nonnegative=True)

# Linear function to be minimized

LP.set_objective(float(1)/len(xpts)*sum(u[i] for i in range(1, len(xpts)+1)) + float(1)/len(ypts)*sum(v[i] for i in range(1, len(ypts)+1)))

# Add constraints

for i in range(1, len(xpts)+1):
  LP.add_constraint(xpts[i-1][0]*a[1] + xpts[i-1][1]*a[2] + u[i] >= b[1] + 1)

for j in range(1, len(ypts)+1):
  LP.add_constraint(ypts[j-1][0]*a[1] + ypts[j-1][1]*a[2] - v[j] <= b[1] - 1)

# LP.show()

oval = LP.solve()
aopt = LP.get_values(a);
bopt = LP.get_values(b)

# Compute frame for showing separating line

pts = xpts + ypts
xvals = list(map(lambda x: x[0], pts))
yvals = list(map(lambda y: y[1], pts))

xmin = min(xvals)
xmax = max(xvals)
ymin = min(yvals)
ymax = max(yvals)

# Intersection of separating line with frame (cumbersome!):

if (aopt[2] == 0):
  x =  bopt[1]/aopt[1]
  p1 = (x, ymin)
  p2 = (x, ymax)
else:
  a = -aopt[1]/aopt[2]
  b = bopt[1]/aopt[2]
  y = a*xmin + b
  if (y >= ymin and y <= ymax):
    p1 = (xmin, y)
    x = (ymax - b)/a
    if (x > xmin and x <= xmax):
      p2 = (x, ymax)
    else:
      x = (ymin - b)/a
      if (x >= xmin and x <= xmax):
        p2 = (x, ymin)
      else:
        p2 = (xmax, a*xmax+b)
  else:
    x = (ymax - b)/a
    if (x >= xmin and x <= xmax):
      p1 = (x, ymax)
      y = a*xmax + b
      if (y >= ymin and y < ymax):
        p2 = (xmax, y)
      else:
        x = (ymin -b)/a
        p2 = (x, ymin)
    else:
      p1 = (xmax, a*xmax + b)
      x = (ymin - b)/a
      p2 = (x, ymin)

show(points(xpts, rgbcolor=(0, 0, 1), pointsize=30) + points(ypts, rgbcolor=(1,0,0), pointsize=30) + line([p1, p2], rgbcolor=(100, 100, 100), linestyle='dashed'))
\end{sage}

\end{document}


\subsection{Minimax}
\subsubsection{Games}

Games are different to search. In a search algorithm we are looking for a sequence of actions. In a game we are looking for a reaction function. Unlike in seach, there are other players.

We can use iterative deepening search.

\subsubsection{Heuristics}

The search space can be too big to look through all the nodes.

Rather than look for win states, we evaluate a non-terminal state using heuristics.

\subsubsection{Stochastic games}

Can use expectminimax

For max node, return highest expectminimax of children

For min node ---

For chance node, average of children weightted

Minimax: two players, max min

Max goes first, maximises results. min minimises results

A node's minimax value is the best outcome against best player.

To find optimal strategy, depth first search of game tree.

Propagate minimax values up tree as terminal nodes are discovered

If a state is terminal, its value is utility of state

If a state is max node, highest value of children

If a state is min node, lowest value of children

Minimax is optimal, complete (for finite trees)


\textit{Combinatorics. Logic. Programming.} This dissertation is an attempt to
explore how each entity relates to the others. Seen as a set of rules that
characterize our methodology, we apply and use them to study the topic of
\textit{infinite~sequences}.  Our approach is two-fold, it allows us, first, to
prove theoretical results about them; second, to built an orbit around them
composed of side-track techniques that helps our main goal, such as programming
techniques and logical reasoning.

In particular, the research field of interest concerns (i)~the manipulation of
a set of matrices that can be algebraically defined, which deserve interest
both as standalone objects and as tools to study other combinatorial
structures; (ii)~the \textit{practice of programming} that exposes our way of
thinking to its paradigms, functional and relational in particular, and
(iii)~the rigor and power of \textit{mechanized logic}.

In normal conditions, it is hard to tackle a problem in our context with both
entities present at the same time; however, we sacrifice a direct approach to
solve the given questions to get the most out of the process that uses the
three tenets together. In this philosophy the constant delay, needed to sharpen
our knowledge in each individual field, is balanced by the discovery of
relations among apparently unrelated subjects that when mixed together yield
nicer, more elegant and possibly unexpected solutions.

For this reason we spread our focus over many topics of mathematics and
computer science instead of composing a mono-theme discussion; pairwise,
we deepen into
\begin{description}

    \item[Combinatorics and Programming] the implementation of enumeration
    techniques for classes of combinatorial objects from both the algebraic
    and applicative points of view;

    \item[Programming and Logic] the study of a family of languages designed
    for relational programming, using a general purpose inference engine to
    perform deductions over domain specific objects;

    \item[Logic and Combinatorics] the formalization of proofs to which
    corresponds certified enumerations of classes of objects, using an extended
    theorem prover based on Higher Order Logic.

\end{description}

Even though abstract reasoning took place most of the time, we make
some room for practical stuff; in particular, we would like to provide a suite
of tools that helps interacting with the Online Encyclopedia of Integer
Sequences, to automate fetching, printing and graphing the networks they
compose. Moreover, some practice with bitmasking and backtracking techniques is
done to show some interesting tiling and enumeration problems.

Another force that drove our work is a pedagogical approach to problem solving,
hence we preferred to refine a first, naive solution over and over to cut away
unnecessary details and complexities; moreover, we don't seek for efficiency in
all cases in favor of simple and beauty definitions. This methodology allows us
to use many programming languages and environments to support theoretical
derivations and this dissertation collects this heterogeneous pool of techniques.

\section*{Structure}

This dissertation has 5 main chapters that rely on the first one, which
introduces basic definitions needed by the following ones; so, except for the
first, the others can be read in any order,
\begin{itemize}
\item Chapter \ref{ch:backgrounds} quickly recalls the theory characterizing
the Riordan group and provides a set of constructors to define those matrices
programmatically, after the introduction of the symbolic module Sympy
implemented on top of the Python programming language. Moreover, it shows our
programming style which is based on consecutive manipulation of symbolic
equations, as mush as possible close to paper and pencil derivations.
\item Chapter \ref{ch:Riordan-matrices-function} presents a theoretical and
practical framework that lifts a scalar function to a matrix function, toward
application to Riordan matrices. To the best of our knowledge, this is the
first implementation that allows the user to perform its computations fully
symbolically, postponing to the end the substitution of ground values to fill
matrices with numbers. Moreover, the Jordan Canonical Form of Riordan matrices
is also studied.
\item Chapter \ref{ch:algebraic-gfs-languages-avoiding-Riordan-patterns}
studies the enumeration of languages of binary words avoiding a given pattern,
provided that some contraints over the structure of each word are taken into
account.  In particular, when the pattern to avoid is a Riordan pattern then
the problem can be solved using Riordan arrays, deriving new series
developments about enumerations with respect to the number of $1$-bits and to
the length. Finally, some combinatorial interpretations are shown, at least for
simpler families of languages.
\item Chapter \ref{ch:OEIS:tools} implements a suite of tools that interact with
the Online Encyclopedia of Integer Sequences; in particular, a (i)~crawler fetches
sequences and their cross references recursively, using asynchronous primitive
to optimize network delays, a (ii)~(pretty) printer for the fetched sequences that
allows the user to filter the sections to be rendered both in the terminal and in
web interfaces and, finally, a (iii)~grapher that draw graphs where vertices
are sequences and edges are references among them.
\item Chapter \ref{ch:queens-tilings-polyominoes} is an exercise in
backtracking techniques to solve tiling and placement problems; for the sake of
efficiency, bit masking manipulation and encodings are used. Moreover, it
provides an implementation of a classic enumeration methodology that allows us
to clearly show generations of combinatorial objects starting from concise and
recursive symbolic definitions; finally, counting all of them is a check of
the correctness of our implementation.
\item Chapter \ref{ch:scilp} proposes an extension to the tactic mechanism
actually present in the HOL Light theorem prover inspired by the relational
paradigm. This prototype makes explicit use of meta-variables to support
substitutions and allows backtracking facilities during the interactive proof
process; nonetheless, the proposed generalization offers compositionality just
as the current system does. So this extension is stressed against an evaluator
for a subset of the Lisp language to find a \textit{quine} program.
\end{itemize}

Moreover, a final paragraph concludes with a quick summary of our main results
and the bibliography ends this dissertation.

\section*{Typographical and typesetting conventions}

This dissertation was typeset using the \LaTeX\,'s style \verb|tufte-book|,
which splits each page into two columns: the left-most is greater in width and
holds the main body, while the right-most is lesser and holds captions,
contextual data and code comments; in particular, code chunks are printed in
\verb|verbatim| and highlighted wherever possible, while outputs that denote
mathematical objects are printed in math style, as usual -- when their size
overflows over the right column we prefer to allow it to take the entire
page width for the sake of clarity.