\section{Interpolation Operator: Discrete Fourier Expansion}

    The continuous Fourier series method requires the evaluation of the coefficients
    \begin{align}
        \hat{u}_n = \displaystyle \frac{1}{2\pi} \int_{0}^{2\pi} u(x) e^{-inx} dx.
    \end{align}
    In general, these integrals cannot be computed analytically, and one resorts to the approximation of the Fourier integrals by using quadrature formulas. This procedure defines a discrete transform between the set of values of $u$ at the quadrature points and the set of approximate, or discrete, coefficients. The finite series defined by the discrete transform is actually the interpolate of $u$ at the quadrature nodes. If the properties of accuracy (in particular the spectral accuracy) are retained by replacing the finite transform with the discrete transform, then the interpolant series can be used instead of the truncated series to approximate functions. Also, quadrature formulas differ based on the exact position of the grid points, and the choice of an even or odd number of grid points results in slightly different schemes.

    \subsection{The Even Expansion}

    Define an equidistant grid, consisting of an even number $N$ of gridpoints $x_j \in [0, 2\pi)$, defined by
	\begin{align*}
        x_j = \frac{2 \pi j}{N} , \hspace{0.5cm} j\in [0, \cdots , N -1].
    \end{align*}
    The trapezoidal rule yields the discrete Fourier coefficients $\widetilde{u}_n$, which approximate the continuous Fourier coefficients $\hat{u}_n$ given as follows
    \begin{align}
        	\widetilde{u}_n = \frac{1}{N}  \displaystyle \sum_{j = 0}^{N - 1} u(x_j) e^{-in x_j}.
    \end{align}
	The difference between the continuous and the discrete approximation is very clear since here we only need precision in the points $ x_j $. This may somehow be an advantage in the numerical calculation because in some cases it is possible to obtain the same order of precision, as shown in the following theorem when trigonometric polynomials are involved, the trapezoidal quadrature rule is a very natural approximation. \\

	\begin{teor}
	\label{Exactness_Even}
	For the points $x_j$ defined as above, the quadrature formula
    \begin{align*}
         \frac{1}{2\pi}\displaystyle \int^{2\pi}_{0} f(x) dx = \frac{1}{N} \displaystyle \sum^{N-1}_{j=0} f(x_j),
    \end{align*}
    is exact for any trigonometric polynomial $f(x) = e^{inx}$ , $|n| < N$.
    \end{teor}
	\begin{proof}
	Given a function $f(x) = e^{in x}$, It is easy to observe that

    \begin{align*}
        \frac{1}{2\pi}\displaystyle \int^{2\pi}_{0} f(x) dx =  \left \lbrace \begin{array}{ll}
    	1 \hspace{3mm} & \text{if } n = 0, \\
    	0 \hspace{3mm} & \text{otherwise.}
    	\end{array}  \right .
    \end{align*}

	\noindent On the other hand,
    \begin{align*}
        \frac{1}{N}  \displaystyle \sum_{j = 0}^{N - 1} f(x_j) &= \frac{1}{N}  \displaystyle \sum_{j = 0}^{N - 1} e^{in (\frac{2\pi j}{N})} \\
        &= \frac{1}{N}  \displaystyle \sum_{j = 0}^{N - 1} q^j
    \end{align*}
    where $q = e^{i \frac{2\pi n}{N}}$. If $n$ is an integer multiple of $N$ , i.e., $n = m N$, then, we have to
    \begin{align*}
    	\displaystyle \frac{1}{N} \sum^{N-1}_{j=0}  e^{i Nm (\frac{2\pi j}{N})} = \frac{1}{N} \sum^{N-1}_{j=0}  e^{i(2\pi j m)} = 1
    \end{align*}
	Otherwise,
	\begin{align*}
		\displaystyle \frac{1}{N} \sum^{N-1}_{j=0} q^j = \frac{q^{N} - 1}{q - 1} = 0
	\end{align*}
	Thus, the quadrature formula is exact for any function of the form $f(x) = e^{inx}$, $|n| < N$.
	\end{proof}

	Moreover, we can see that the quadrature formula is exact for $f(x) \in \hat{B}_{2N-2}$ where $\hat{B}_N$ is defined as before. Then using the trapezoid rule, the discrete Fourier coefficients become
	\begin{align}
	\label{coeficients_IN}
	    \widetilde{u}_n = \frac{1}{N \widetilde{c}_n}  \displaystyle \sum_{j = 0}^{N - 1} u(x_j) e^{-in x_j},
	\end{align}
	where we introduce the coefficients
	\begin{align}
	\label{constants_IN}
	    \widetilde{c}_n = \left \lbrace \begin{array}{ll}
	    2  \hspace{0.25cm}\text{if} & |n| =  N/2, \\
	    \\
	    1  \hspace{0.25cm} \text{if} & |n| < N/2.
	\end{array}  \right .
	\end{align}

	\noindent These relations define a new projection of $u$
	\begin{equation}
	\label{collocation_operator_even}
		\mathcal{I}_N u(x) =  \displaystyle \sum_{ |n| \leq \frac {N}{2}} \widetilde{u}_n e^{inx}
	\end{equation}
	This is the complex discrete Fourier transform, based on an even number of quadrature points. From the above, we can see that
	\begin{align*}
	    \widetilde{u}_{-N/2} = \widetilde{u}_{N/2},
	\end{align*}
	so we have exactly $N$ independent Fourier coefficients, corresponding to the $N$ quadrature points. As a consequence, $\mathcal{I}_N \sin( \frac{N}{2} x) = 0$, so that the function $\sin( \frac{N}{2} x)$ is not represented in the above expansion. Therefore, the space $\hat{B}_N$ does not include $\sin( \frac{N}{2} x)$, and the correct space must be as follows
	\begin{align*}
	    \widetilde{B}_N = span\left\{\left(\cos(nx), \hspace{0.2cm} 0 \leq n \leq \frac{N}{2} \right)\cup  \left(\sin(nx), \hspace{0.2cm} 1 \leq n \leq \frac{N}{2} - 1 \right)\right\},
	\end{align*}
	which has dimension dim$(\widetilde{B}_N) = N$.\\

	\noindent In the same way, as in the previous subsection using the discrete expansion for Examples \ref{Example1} and \ref{Example2}, we can observe the same behavior as with continuous expansion, but now we have that the error at each point $x_j$ of the grid is zero.

	\begin{example}
	    Consider the $C^{\infty}_p [0, 2 \pi]$ function
    	\begin{align}
    		\label{Example4}
    	    u(x) = \frac{1}{5 - 4 \cos(x)}.
    	\end{align}
    	Its expansion coefficients are
    	\begin{align*}
    	     \hat{u}_{n} = \frac{2^{-|n|}}{3}.
    	\end{align*}
    	\begin{figure}[H]
        \includegraphics[width=\textwidth]{preliminaries/figures/example23.png}
        \caption{(a) Discrete Fourier series approximation of the equation (\ref{Example4}). (b) Pointwise error of approximation for increasing resolution.}
        \label{fig3}
        \end{figure}
	\end{example}

	\begin{example}
	    The expansion coefficients of the function
    	\begin{align}
    		\label{Example5}
    	    u(x) = \frac{\pi}{2} \sin(\frac{x}{2}),
    	\end{align}
   		are given by
    	\begin{align*}
    	     \hat{u}_{n} = \frac{1}{(1 - 4n^2)}.
    	\end{align*}
    	\begin{figure}[H]
        \includegraphics[width=\textwidth]{preliminaries/figures/example24.png}
        \caption{(a) Discrete Fourier series approximation of the equation (\ref{Example5}). (b) Pointwise error of approximation for increasing resolution.}
        \label{fig4}
        \end{figure}
	\end{example}

	Therefore, we can see that the discrete expansion is, in fact, an interpolation operator as mentioned. This can be shown in the following theorem.\\

	\begin{teor}
	Let the discrete Fourier transform be defined by Equations (\ref{coeficients_IN})-(\ref{collocation_operator_even}). For any periodic function, $C^{0}_p [0, 2\pi]$, we have
	\begin{align*}
		\mathcal{I}_N u(x_j) = u(x_j), \hspace{0.3cm} \forall x_j = \frac{2 \pi j}{N} , \hspace{0.3cm} j = 0, \dots, N - 1.
	\end{align*}
	\end{teor}

	\begin{proof}
	Substituting Equation (\ref{coeficients_IN}) into Equation (\ref{collocation_operator_even}) we obtain

	\begin{equation*}
    	\mathcal{I}_N u(x) =  \displaystyle \sum_{ |n| \leq \frac {N}{2}} \left(\frac{1}{N \widetilde{c}_n}  \displaystyle \sum_{j = 0}^{N - 1} u(x_j) e^{-in x_j}\right) e^{inx}.
	\end{equation*}
	Exchanging the order of the sum gives
	\begin{align}
	    \mathcal{I}_N u(x) = \displaystyle \sum_{j=0}^{N-1} u(x_j) g_j (x),
	\end{align}
	where
	\begin{align*}
	    g_j (x) &= \displaystyle \sum_{ |n| \leq \frac {N}{2}} \frac{1}{N \widetilde{c}_n} e^{in(x -x_j)}\\
	    &= \frac{1}{N} \sin\left[N \frac{x - x_j}{2} \right] \cot\left[\frac{x - x_j}{2} \right]
	\end{align*}
	by summing as a geometric series. It is easily verified that $g_j (x_i) = \delta_{ij}$\\
	\\
	We still need to show that $g_j (x) \in \widetilde{B}_N$. Clearly, $g_j (x) \in \hat{B}_N$ as $g_j (x)$ is a polynomial of degree $\leq N/2$. However, since

	\begin{align*}
	    \frac{1}{2} e^{-i \frac{N}{2} x_j} = \frac{1}{2} e^{i \frac{N}{2} x_j} = \frac{(-1)^j}{2},
	\end{align*}

	and, by convention $\widetilde{u}_{-N/2} = \widetilde{u}_{N/2}$, we do not get any contribution from the term $\sin(\frac{N}{2} x)$, hence $g_j (x) \in \widetilde{B}_N$.
	\end{proof}

	\subsection{The Odd Expansion}

	Similarly, we define a grid with an odd number of grid points as follows
	\begin{align*}
        x_j = \frac{2 \pi}{N + 1} j , \hspace{0.5cm} j\in [0, \dots , N],
    \end{align*}
    and using the trapezoidal rule we get
	\begin{align}
	\label{coefficients_JN}
        \widetilde{u}_n = \frac{1}{N + 1}  \displaystyle \sum_{j = 0}^{N} u(x_j) e^{-in x_j},
    \end{align}
    to obtain the interpolation operator
    \begin{equation}
    \label{Interpolation_operator_odd}
    	\mathcal{J}_N u(x) =  \displaystyle \sum_{ |n| \leq \frac {N}{2}} \widetilde{u}_n e^{inx}.
	\end{equation}

    \noindent Again as before, the quadrature formula is highly accurate. \\
    \begin{teor}
    \label{Exactness_Odd}
    For the points $x_j$ defined as above, the quadrature formula
    \begin{align*}
         \frac{1}{2\pi}\displaystyle \int^{2\pi}_{0} f(x) dx = \frac{1}{N+1} \displaystyle \sum^{N}_{j=0} f(x_j),
    \end{align*}
    is exact for any $f(x) = e^{inx}$ , $|n| < N$, i.e., for all $f(x) \in \widetilde{B}_{2N}$.
    \begin{proof}
    	Given a function $f(x) = e^{in x}$, It is easy to observe that

    	\begin{align*}
    		\frac{1}{2\pi}\displaystyle \int^{2\pi}_{0} f(x) dx =  \left \lbrace \begin{array}{ll}
    			1 \hspace{3mm} & \text{if } n = 0, \\
    			0 \hspace{3mm} & \text{otherwise.}
    		\end{array}  \right .
    	\end{align*}
    	On the other hand,
    	\begin{align*}
    		\frac{1}{N+1}  \displaystyle \sum_{j = 0}^{N} f(x_j) &= \frac{1}{N+1}  \displaystyle \sum_{j = 0}^{N} e^{in (\frac{2\pi j}{N+1})} \\
    		&= \frac{1}{N+1}  \displaystyle \sum_{j = 0}^{N} q^j
    	\end{align*}
    	where $q = e^{i \frac{2\pi n}{N+1}}$. If $n$ is an integer multiple of $N+1$ , i.e., $n = (N+1)m$, then we have to
    	\begin{align*}
    		\displaystyle \frac{1}{N+1} \sum^{N}_{j=0}  e^{i(N+1)m (\frac{2\pi j}{N+1})} = \frac{1}{N+1} \sum^{N}_{j=0}  e^{i (2\pi jm)} = 1
    	\end{align*}
    	 Otherwise,
    	\begin{align*}
    		\displaystyle \frac{1}{N+1} \sum^{N}_{j=0} q^j = \frac{q^{N+1} - 1}{q - 1} = 0
    	\end{align*}
    	Thus, the quadrature formula is exact for any function of the form $f(x) = e^{inx}$, $|n| < N$.
    \end{proof}

    \end{teor}
    The scheme may also be expressed through the use of a Lagrange interpolation polynomial,
    \begin{align}
    \label{Lagrange_Odd}
    	\mathcal{J}_N u(x) =  \displaystyle \sum_{j=0}^{N} u(x_j) h_j (x)
	\end{align}
	where
	\begin{align}
	    h_j (x) = \displaystyle \frac{1}{N+1} \sum_{|k| \leq \frac{N}{2}} e^{ik (x - x_j)} = \frac{1}{N + 1} \frac{\sin(\frac{N+1}{2}(x - x_j))}{\sin(\frac{x - x_j}{2})}
	\end{align}
    One easily shows that $h_j (x_l) = \delta_{jl}$ and that $h_j (x) \in \hat{B}_N $.

	\newpage
    \subsection{Differentiation of the discrete expansions}

    Similarly, as in continuous expansion, we require compute derivatives of the discrete approximation. In the following subsections, we assume that our function $u$ and all its derivatives are continuous and periodic on $[0, 2\pi]$.\\
    \\
    We consider the case of an even number of grid points. Using expansion coefficients given the values of the function $u(x)$ at the points $x_j$ , differentiating the basis functions in the interpolant yields
    \begin{align}
        \frac{d}{dx} \mathcal{I}_N u(x) = \displaystyle \sum_{|n| \leq N/2} in \widetilde{u}_n e^{inx}, \hspace{2mm} \widetilde{u}_n = \displaystyle \frac{1}{N \widetilde{c}_n} \sum_{j=0}^{N-1} u(x_j) e^{-in x_j},
    \end{align}
    where $\widetilde{c}_n$ is given by (\ref{constants_IN}). Higher order derivatives can be obtained simply by further differentiating the basis functions.\\

    \noindent Similarly, for the case of an odd number of grid points
    \begin{align}
    	\frac{d}{dx} \mathcal{J}_N u(x) = \displaystyle \sum_{|n| \leq N/2} in \widetilde{u}_n e^{inx}, \hspace{2mm} \widetilde{u}_n = \displaystyle \frac{1}{N + 1} \sum_{j=0}^{N} u(x_j) e^{-in x_j},
    \end{align}
    The procedure for differentiating using expansion coefficients can be described as follows: first, we transform the point values $u(x_j)$ in physical space into the coefficients $\widetilde{u}_n$ in mode space. We then differentiate in mode space by multiplying  $\widetilde{u}_n$ by $in$, and return to physical space.\\

    There are other ways to obtain these derivatives, which may have greater advantage and be more efficient to calculate. In the literature, it can commonly find the use of differentiation matrices, for which there is a great variety. We will present some matrices that have been studied in \cite{gottlieb2007}, \cite{Canuto2012}, and we will observe the difference between the cases of an even and odd number of grid points. \\

    \paragraph{Differentiation Matrix.} Recall that to the case of an even number of grid points, the interpolation operator can be written as
    \begin{align*}
        \mathcal{I}_N u(x) = \displaystyle \sum^{N-1}_{j=0} u(x_j) g_j (x),
    \end{align*}
    where $g_j$ are the Lagrange interpolation polynomials given by
    \begin{align*}
        g_j (x) = \frac{1}{N} \sin\left[N \frac{x - x_j}{2} \right] \cot\left[\frac{x - x_j}{2} \right].
    \end{align*}
    Then, by differentiating the interpolation directly, it can get an approximation to the derivative of $u(x)$ at the points $x_j$ as follows
    \begin{align*}
        \displaystyle \frac{d}{dx} \mathcal{I}_N (x) \Big|_{x_l} = \sum^{N-1}_{j=0} u(x_j) \frac{d}{dx} g_j (x) \Big|_{x_l} = \sum^{N-1}_{j=0} D_{lj} u(x_j),
    \end{align*}
    where $D_{lj}$ are the differentiation matrix entries given by
    \begin{align}
    \label{matrix_DN_even}
        D_{ij} = \frac{d}{dx} g_j (x) \Big|_{x_i} = \begin{cases} \frac{(-1)^{i+j}}{2} \cot \left[ \frac{x_i - x_j}{2}\right] &   i \neq j, \\ \hspace{1mm} 0 &  i=j, \end{cases}
    \end{align}
    it is also well known that $D$ is circulant and skew-symmetric matrix. In the same way, the entries of the second order differentiation matrix $D^{(2)}$ gives us
    \begin{align}
    \label{matrix_D2N_even}
        D_{ij}^{(2)} = \frac{d^2}{dx^2} g_j (x) \Big|_{x_i} = \begin{cases} -\frac{(-1)^{i+j}}{2} \left[\sin \left[ \frac{x_i - x_j}{2}\right]\right]^{-1} &   i \neq j, \\ -\frac{N^2 + 2}{12} &  i=j. \end{cases}
    \end{align}

    The approximation of higher derivatives follows exactly the same route, and similarly to obtain the entries of the differentiation matrix $\widetilde{D}$ for the interpolation based on an odd number of points given by
    \begin{align}
    \label{matrix_DN_odd}
        \widetilde{D}_{ij} = \begin{cases} -\frac{(-1)^{i+j}}{2} \left[\sin \left[ \frac{x_i - x_j}{2}\right]\right]^{-2} &   i \neq j, \\ \hspace{1mm} 0 &  i=j. \end{cases}
    \end{align}

    It is also known that $\widetilde{D}$ is a circulant, skew-symmetric matrix. The advantage of this method is that the differentiation matrix takes us from physical space to physical space, and the act of differentiation is hidden in the matrix itself. \\

    It is interesting to observe that the differentiation operator for the interpolation based on an odd number of grid points, takes elements of $\widetilde{B}_N$ out of $\widetilde{B}_N$ and then
    \begin{align*}
    	\mathcal{I}_N \frac{d^2}{dx^2} \mathcal{I}_N \neq \left( \mathcal{I}_N \frac{d}{dx} \right)^2 \mathcal{I}_N.
    \end{align*}

	But for the interpolation based on an odd number of grid points the differentiation operator remain in $\hat{B}_N$ when takes elements of $\hat{B}_N$, and thus,
	\begin{align*}
		\mathcal{J}_N \frac{d^2}{dx^2} \mathcal{J}_N = \left( \mathcal{J}_N \frac{d}{dx} \right)^2 \mathcal{J}_N
	\end{align*}
    Moreover, for all values of $q$ we have
    \begin{align*}
        \widetilde{D}^{(q)} = \mathcal{J}_N \frac{d^q}{dx^q} \mathcal{J}_N = \widetilde{D}^q
    \end{align*}
	allowing us to calculate approximate high derivatives by just multiplying the $D$ matrix as many times as necessary.

    For the above, and for some interesting properties that we will see later about interpolation operator based on an odd number of grid points, it has been decided to use it for the study of this work.

    \subsection{Approximation theory for Discrete Expansion.}

    Based on the theory developed in \cite{gottlieb2007} with respect to the interpolation operator analysis for the case of an even number of grid points, we will adapt the results for the case of an odd number of grid points in the most detailed way possible. \\

    \noindent First of all, we can define a discrete version of the inner product $L^2$ as follows
    \begin{align*}
    	\langle f_N, g_N \rangle_N = \displaystyle \frac{1}{N + 1} \sum_{j = 0}^{N} f_N (x_j) \bar{g}_N (x_j),
    \end{align*}
	and the associated norm
	\begin{align*}
		\| f_N \|_N^2 = \langle f_N, f_N \rangle_N
	\end{align*}
	where $f_N$ , $g_N \in \hat{B}_N$ and there are an odd number of grid points $x_j$ , $j = 0, \dots , N$. Note also that the interpolant $\mathcal{J}_N u$ of a continuous function $u$ and for all $v \in \hat{B}_N$, satisfies trivially the identity
	\begin{align*}
		\langle \mathcal{J}_N u, v \rangle_N = \langle u, v \rangle_N.
	\end{align*}

	Moreover, as a consequence of the exactness of the quadrature rule for trigonometric functions, as have seen in Theorem \ref{Exactness_Even}, we have
	\begin{align}
	\label{Coincidence_Inner}
		\langle f_N, g_N \rangle_N = \displaystyle \frac{1}{2 \pi} \int_{0}^{2 \pi} f_N \bar{g}_N dx, \hspace{3mm} \|f_N\|_{L^2 [0, 2 \pi]} = \|f_N\|_N
	\end{align}

	Hence, in $\hat{B}_N$ , the continuous and discrete inner product are the same.\\

	The situation is different when we discuss an even number of grid points. If $f_N , g_N \in \tilde{B}_N$ and we have an even number of grid points $x_j$ , the discrete inner product
	\begin{align*}
		\langle f_N, g_N \rangle_N = \displaystyle \frac{1}{N} \sum_{j = 0}^{N - 1} f_N (x_j) \bar{g}_N (x_j), \hspace{3mm} \|f_N\|_N^2 = \langle f_N, f_N \rangle_N
	\end{align*}
	is not equal to the continuous inner product. However, using the fact that $f_N \in L^2 [0, 2 \pi]$ it can be shown that there exists a $K > 0$ such that
	\begin{align}
	\label{equivalent_discrete_continous}
		K^{-1} \|f_N\|^2_{L^2 [0, 2 \pi]} \leq \|f_N\|^2_N \leq K \|f_N\|^2_{L^2 [0, 2 \pi]}.
	\end{align}

    Something very interesting and useful in the use of discrete expansion to approximate functions and their derivatives is that the behavior is very similar to that shown in the previous subsection for continuous expansion. We will see that the approximation theory for the discrete expansion yields essentially the same results as for the continuous expansion. The proofs are based on the fact that the Fourier coefficients of the discrete approximation are sufficiently close to those of the continuous approximation. \\

    Recall that the interpolation operator associated with an odd number of grid points is given by
    \begin{align*}
    	\mathcal{J}_{2N} u = \displaystyle \sum_{|n|\leq N} \widetilde{u}_n e^{inx},
    \end{align*}
    with expansion coefficients
    \begin{align*}
    	\widetilde{u}_n = \displaystyle \frac{1}{2N + 1} \sum^{2N}_{j=0} u(x_j)e^{-in x_j}, \hspace{2mm} x_j = \frac{2 \pi j}{2N+1}.
    \end{align*}

    First we observe the following, the interpolation operator associated with an odd number of grid points are based on the points $x_j$, for which the $(n + Mm)$th mode, where $M = 2N + 1$, is indistinguishable from  the $n$th mode, i.e.,
    \begin{align*}
    	e^{i(n + Mm)x_j} = e^{in x_j} e^{i 2 \pi m j} = e^{in x_j}
    \end{align*}
	This phenomenon is known as aliasing.\\

	Moreover, due to the orthogonality relation as before seen we have to
    \begin{align*}
    	\frac{1}{M} \displaystyle \sum^{M-1}_{j=0} e^{-inx_j} =  \left \lbrace \begin{array}{ll}
    	1 \hspace{4mm} &\text{if} \hspace{2mm} n=M m, \hspace{2mm} m = 0, \pm 1, \pm 2, \dots, \\
    	0 &\text{otherwise.}
    	\end{array}  \right .
    \end{align*}

    The relationship between the discrete expansion coefficients $\widetilde{u}_n$ , and the continuous expansion coefficients $\hat{u}_n$, is given in the following lemma.

    \begin{lemma}
    \label{lemma_2.1}
    Consider $u(x) \in W^q_p [0, 2\pi]$, where $q > 1/2$. For $|n| \leq N$ we have
	    \begin{align}
	    \label{alias}
	        \widetilde{c}_n \widetilde{u}_n = \displaystyle \hat{u}_n + \sum_{\substack{|m|\leq \infty \\ m \neq 0}} \hat{u}_{n + Mm}
	    \end{align}
	\end{lemma}
	\begin{proof}
    Substituting the continuous Fourier expansion into the discrete expansion yields
    \begin{align*}
        \widetilde{c}_n \widetilde{u}_n = \frac{1}{M} \displaystyle \sum_{j=0}^{M - 1} \sum_{|l| \leq \infty} \hat{u}_l e^{i(l -n)x_j}
    \end{align*}
    To interchange the two summations we must ensure uniform convergence, i.e., $\sum_{|l| \leq \infty} |\hat{u}_l| < \infty$. This is satisfied, since if $q> 1/2$ then, as before there is $m \in \mathbb{N}$ such that for $l \geq m$ we have to
    \begin{align*}
    	(1 + |l|)^{2q} \leq 2q(1 + l^{2q})
    \end{align*}
	taking $m$ as follows
	\begin{align*}
		\frac{1}{m} \leq (2q)^{\frac{1}{2q}} - 1
	\end{align*}

	\noindent Therefore
    \begin{align*}
        \displaystyle \sum_{|l| \leq \infty} |\hat{u}_l| &= \sum_{|l| \leq \infty} (1 + |l|)^q \frac{|\hat{u}_l|}{(1 + |l|)^q}  \\
        &\leq \left( 2q \sum_{|l| \leq \infty}  (1 + l^{2q}) |\hat{u}_l|^2 \right)^{1/2} \left(\sum_{|l| \leq \infty} (1 + |l|)^{-2q}  \right)^{1/2},
    \end{align*}
    where the last expression follows from the Cauchy-Schwarz inequality. As $u(x) \in W^q_p [0, 2\pi]$ the first part is clearly bounded. Furthermore, the second term is a $p$-series and then converges provided $q > 1/2$, ensuring boundedness. \\

    Interchanging the order of summation and using orthogonality of the exponential function at the grid yields the desired result
    \begin{align*}
    	\widetilde{c}_n \widetilde{u}_n &= \frac{1}{M} \displaystyle \sum_{j=0}^{M - 1} \sum_{|l| \leq \infty} \hat{u}_l e^{i(l -n)x_j} =  \sum_{|l| \leq \infty} \frac{1}{M} \sum_{j=0}^{M - 1} \hat{u}_l e^{i(l -n)x_j} \\
    	&= \sum_{|m| \leq \infty}  \frac{1}{M} \sum_{j=0}^{M - 1} \hat{u}_{n + Mm} e^{i(n + Mm)x_j} \\
    	&= \frac{1}{M} \sum_{j=0}^{M - 1} \hat{u}_n e^{inx_j}
    	+ \sum_{\substack{|m|\leq \infty \\ m \neq 0}} \frac{1}{M} \sum_{j=0}^{M - 1} \hat{u}_{n + Mm} e^{i(n + Mm)x_j} \\
    	&= \hat{u}_n + \sum_{\substack{|m|\leq \infty \\ m \neq 0}} \hat{u}_{n + Mm}
    \end{align*}
	\end{proof}

	The conclusions of the previous discussion are equally valid in the number of odd or even points. An equivalent formulation of (\ref{alias}) is
	\begin{align*}
		\mathcal{J}_N u = \mathcal{P}_N u + \mathcal{A}_N u
	\end{align*}
	It is orthogonal to the truncation error, $u - \mathcal{P}_N u$, so that
	\begin{align*}
		\| u - \mathcal{J}_N u \|^2 = \|u - P_N u \|^2 + \| \mathcal{A}_N u \|^2
	\end{align*}
	Hence, the error due to the interpolation is actually always larger than the error due to the truncation of the Fourier series.\\

	Rather than deriving the estimates of the approximation error directly, we shall use the results obtained in the previous section and then estimate the difference between the two different expansions, which we recognize as the aliasing error given by
	\begin{align*}
		\| \mathcal{A}_N \|_{L^2 [0, 2\pi]} = \left\| \displaystyle \sum_{|n|<N} \left( \sum_{\substack{|m|\leq \infty \\ m \neq 0}} \hat{u}_{n + Mm}  \right) \right\|_{L^2 [0, 2\pi]}
	\end{align*}

    As before, we first consider the behavior of the approximation in the $L^2$-norm. We will first show that the bound on the aliasing error, $\mathcal{A}_N$ , in equation above is of the same order as the truncation error. The error caused by truncating the continuous expansion is essentially the same as the error produced by using the discrete coefficients rather than the continuous coefficients.
    \begin{lemma}
    \label{estimating_aliasing_error}
    For any $u(x) \in W_p^r [0, 2\pi]$, where $r > 1/2$, the aliasing error
    \begin{align*}
        \| \mathcal{A}_N \|_{L^2 [0, 2\pi]} = \displaystyle \left(\sum_{|n| \leq \infty} |\widetilde{c}_n \widetilde{u}_n - \hat{u}_n|^2 \right)^{1/2} \leq CN^{-r} \|u^{r}\|_{L^2 [0, 2\pi]}
    \end{align*}
	\end{lemma}
	\begin{proof}
    From Lemma \ref{lemma_2.1} we have
    \begin{align*}
        |\widetilde{c}_n \widetilde{u}_n - \hat{u}_n|^2 = \displaystyle \left|\sum_{\substack{|m|\leq \infty \\ m \neq 0}} \hat{u}_{n + Mm} \right|^2
    \end{align*}
    To estimate this, we first note that
    \begin{align*}
        \displaystyle \left|\sum_{\substack{|m|\leq \infty \\ m \neq 0}} \hat{u}_{n + Mm} \right|^2 &= \left|\sum_{\substack{|m|\leq \infty \\ m \neq 0}} |n + Mm|^r \hat{u}_{n + Mm} \frac{1}{|n + Mm|^r} \right|^2 \\
        &\leq \left(\sum_{\substack{|m|\leq \infty \\ m \neq 0}} |n + Mm|^{2r} |\hat{u}_{n + Mm}|^2 \right) \left(\sum_{\substack{|m|\leq \infty \\ m \neq 0}} \frac{1}{|n + Mm|^{2r}}  \right)
    \end{align*}
    using the Cauchy-Schwartz inequality. Since $M =2N + 1$ and  $|n| \leq N$, we have to $N(2m - 1) = 2Nm - N \leq |n + Mm|$. Hence, bounding of the second term is ensured by
    \begin{align*}
        \displaystyle \sum_{\substack{|m|\leq \infty \\ m \neq 0}} \frac{1}{|n + Mm|^{2r}} \leq \frac{2}{N^{2r}} \sum^{\infty}_{m=1} \frac{1}{(2m - 1)^{2r}} = C_1 N^{-2r},
    \end{align*}
    provided $r > 1/2$. Here, the constant $C_1$ is a consequence of the fact that the power series converges, and it is independent of $N$.\\
    Summing over $n$, we have
    \begin{align*}
        \displaystyle \sum_{|n|\leq N} \left|\sum_{\substack{|m|\leq \infty \\ m \neq 0}} \hat{u}_{n + Mm} \right|^2 &\leq \sum_{|n| \leq N} C_1 N^{-2r} \sum_{\substack{|m|\leq \infty \\ m \neq 0}} |n + Mm|^{2r} |\hat{u}_{n + Mm}|^2 \\
        &\leq C_2 N^{-2r} \| u^{(r)} \|^2_{L^2[0, 2\pi]}
    \end{align*}
	\end{proof}

    We are now in a position to state the error estimate for the discrete approximation.
    \begin{teor}
    \label{estimating_error_I_N_L2}
    For any $u(x) \in W_p^r [0, 2\pi]$ with $r > 1/2$, there exists a positive constant $C$, independent of $N$ , such that
    \begin{align*}
        \| u - \mathcal{J}_{2N}u \|_{L^2 [0, 2\pi]} \leq CN^{-r} \| u^{(r)} \|_{L^2 [0, 2\pi]}
    \end{align*}
	\end{teor}
    \begin{proof}
    Let’s write the difference between the function and its discrete approximation
    \begin{align*}
        \| u - \mathcal{J}_{2N}u \|_{L^2 [0, 2\pi]} &= \| (\mathcal{P}_{2N} - \mathcal{J}_{2N})u + u - \mathcal{P}_{2N}u \|_{L^2 [0, 2\pi] } \\
        &\leq  \| (\mathcal{P}_{2N} - \mathcal{J}_{2N})u \|_{L^2 [0, 2\pi] } + \| u - \mathcal{P}_{2N}u \|_{L^2 [0, 2\pi] }
    \end{align*}
    Thus, the error has two components. The first one, which is the difference between the continuous and discrete expansion coefficients, is the aliasing error, which is bounded in Lemma \ref{estimating_aliasing_error}. The second, which is the tail of the series, is the truncation error, which is bounded by the result of Theorem \ref{estimating_error_PN_L2}. The desired result follows from these error bounds.
	\end{proof}

    Theorem above confirms that the approximation errors of the continuous expansion and the discrete expansion are of the same order, as long as $u(x)$ has at least half a derivative. Furthermore, the rate of convergence depends, in both cases, only on the smoothness of the function being approximated.


\chapter{Abstract Framework for DNN Models}

\section{An abstract framework for ANN}
\subsection{Problem: function interpolation}
What we want to do is to approximate a function $f^*: V \to U$, with some given data $\{X^i, Y^i\}_{i=1}^N$. {\bf{But the fact is that, we know only few things about $f^*$!}} If we can know more things about $f^*$, we can design some better model for interpolation such as:
\begin{itemize}
\item FEM: $f^*$ is a solution of elliptic PDE,
\item Spectral method: $f^*$ has sufficient high smoothness,
\item Deep learning: $f^*$ is produced by the human neural network.
\item $\cdots$
\end{itemize}
A general idea is to construct a parameters depended function family such $f(W; x)$, where $W \in \mathcal{W}$, so we can have the next simple problem:
\begin{problem}
Find the parameter $\theta^{*}$ such that:
\begin{equation}
\theta^* = \mathop{\arg\min}_{\theta \in \Theta} \sum_{i}^N L(f(\theta;X^i) , Y^i).
\end{equation}
Here $L(\cdot, \cdot)$ is a general metric to define the difference between the real function value $Y^i$ and the predicted out put $f(\theta;X^i)$, a simple example is $ L(f(\theta;X^i) , Y^i) = \|f(\theta;X^i) - Y^i\|^2_{U}$.
\end{problem}
Then we could choose $f(\theta^*; x)$ as the interpolation of $f$ by given data $\{X^i, Y^i\}_{i=1}^N$.

So, all problem reduced to how to construct a suitable function family $f(W;x): V \to U$. In my opinion, different kinds of machine learning models mean different function families. For example, linear regression means that:
\begin{equation}
f(\theta; x) = w x + b, \quad \theta = \{w, b \},
\end{equation}
where
\begin{equation}
w \in L(V, U) \quad \text{and} \quad b \in U .
\end{equation}

\subsection{Function family with abstract auxiliary space}
So, now we want to explain the deep learning model based on the idea of function family. The model for deep learning is a little different from the traditional models. The main idea is to imitate the human neural network structure that is the composition of many neurons because $f$ is the reaction of humans for the "input" from real world. But in fact we don't know what real human neural network connection is, so we use a simple idea that those neurons have the hierachical structure. (I will show in the latter, it seems that adding the shortcut connection into those pure hireachical structure will be better.)

From the discussion above, We can take the fully connected feedforward neural network as the composition of affine map and nonlinear map in just some abstract linear space. It means that we construct a nonlinear function from $V$ to $U$ with the auxiliary space $V_i$ with $i = 1:J$ such that we have those next functions:
\begin{equation}\label{equ:affinemap}
A_i: V_{i-1} \to V_i  \quad \text{with} \quad A_i(x) = W_i (x)  + \theta_i  \quad \text{with} \quad W_i \in L(V_{i-1}, V_i) ~~ \theta_i \in V_i.
\end{equation}
and
\begin{equation}
g_i: V_i \to V_i \quad \text{is a general nonlinear function}
\end{equation}
where $i = 0:J$ with $V_0 = V$ and $V_J = U$.

$$
A_i=(W_i, \theta_i).
$$

So the abstract nonlinear iterative form for $f$ is like:
\begin{equation}\label{equ:abstractNN1}
f = f_{out}\circ f_J,
\end{equation}
with
\begin{equation}\label{equ:abstractNN2}
f_j = g_j(A_j(f_{j-1})), \quad j = 1:J,
\end{equation}
and
\begin{equation}\label{equ:abstractNN3}
f_0 = f_{in}(x), \quad x \in V.
\end{equation}
Here in most cases, $f_0(x)=x$.  But sometimes we can also view $f_0$ as
denoising process?

We note that
$$
f_j: V\times {\cal W}^j\mapsto V_j.
$$
and
$$
f_j(x,{\mathbf A}_j)
$$
where
$$
{\mathbf A}_j=(A_1,\ldots A_j)=({\mathbf A}_{j-1},A_j).
$$

Notice that
$$
f_0=f_{\rm in}: V=V_0\mapsto V
$$
which is pre-process, and
$$
f_{J+1}=f_{\rm out}: U=V_{J}\mapsto U
$$
which is a post-process.

Generally speaking, if we fix those auxiliary space $V_i$, $g_i$, $f_{in}$ and $f_{out}$, then $f$ only depends on those affine map $A_i \in A(V_{i-1}, A_{i})$ defined in \ref{equ:affinemap}.  So in the abstract form,
\begin{equation}
{\cal W}^j = \Pi_{i=1}^j A(V_{i-1}, V_i).
\end{equation}
\begin{equation}
\mathcal{W} = \Pi_{i=1}^J A(V_{i-1}, V_i).
\end{equation}

\subsection{Loss functions and optimization problems}
Forget about "BP", we first express our problem as:
\begin{problem}Fix $V_i$, $g_i$, $f_{in}$ and $f_{out}$, then the optimization problem is:
\begin{equation}
\mathop{\arg\min}_{{\mathbf A}\in {\cal W}} \sum_{j=1}^N \| f(X^j, {\mathbf A}) - Y^j \|^2_{U}
\end{equation}
\end{problem}

We need to discuss other possible loss functions ....

\section{ANN: a fully connected model}
Once we use general vector space like $\mathbb{R}^n$, $\mathbb{R}^{n_i}$ and $\mathbb{R}^c$ to represent $V$, $V_i$ and $U$. So the affine map from $V_{i-1}$ to $V_i$ can be represented as:
\begin{equation}
A_i(x) = W_i x + \theta_i, \quad \text{with} \quad W_i \in \mathbb{R}^{n_i \times n_{i-1}} \quad \theta_i \in \mathbb{R}^{n_i}.
\end{equation}
And if the nonlinear map $g_i$ is a little special like:
\begin{equation}
g_i(x_1, \cdots, x_{n_i}) = (g(x_1), \cdots, g(x_{n_i}))^T, \quad \forall i = 1:J,
\end{equation}
where $g$ is a certain nonlinear function form $\mathbb{R}$ to $\mathbb{R}$. Some general choose for $g$ can be: Sigmiod, ReLU and etc.

So the vector representation of the \ref{equ:abstractNN1} to \ref{equ:abstractNN3} is the ANN model exactly. And $J$ is the number of hidden layers and $dim(V_i)$ is the neural numbers for the hidden layer $i$.

The model will be expressed like:
\begin{equation}\label{equ:repNN1}
f(W;x) = f_{out}\circ f_J,
\end{equation}
with
\begin{equation}\label{equ:repNN2}
f_j = g_j(W_jf_{j-1} + \theta_j), \quad j = 1:J,
\end{equation}
and
\begin{equation}\label{equ:repNN3}
f_0 = f_{in}(x), \quad x \in V.
\end{equation}
So if we fix those auxiliary space by $n_i$, $g_i$, $f_{in}$ and $f_{out}$, all parameters are:
\begin{align}
\mathcal{W} &= \{ (W_i, \theta_i) ~ :  ~ W_i \in \mathbb{R}^{n_{i} \times n_{i-1}} ~ ~\theta_i \in \mathbb{R}^{n_i}, i = 1:J \}, \\
&= \Pi_{i=1}^J ( \mathbb{R}^{n_{i} \times n_{i-1}} \times  \mathbb{R}^{n_i}).
\end{align}

\section{Classification problems}
Now we will state some observation about the classification problems. First is about what kinds of set can be split by linear model(neural network without hidden layers).
\begin{definition}(linear separate)\label{assum:linsep}
Assume that $\{A_i\}_{i=1}^c \subset \mathbb{R}^n$ can be separated by hyperplane by $\{H_i\}_{i=1}^c$ i.e for every $A_i$ there exists $H_i$ such $H_i$ will separate $A_i$ and $\{A_1, \cdots, A_{i-1}, A_{i+1}, \cdots, A_c\}$.
\end{definition}
\begin{lemma}
If the different finite sets $\{A_i\}_{i=1}^c \subset \mathbb{R}^n$ satisfy the assumption \ref{assum:linsep}, then there exists a simple neural network such that
\begin{equation}
f(x_i) = e_i \quad \forall x_i \in A_i ~~ i= 1:c.
\end{equation}
\end{lemma}
\begin{proof}We can just choose
\begin{equation}
f(x; \{W,b\})  = H(Wx + b)
\end{equation}
with $W_i$ and $b_i$ define the hyperplane $H_i$ and $H$ is the Heaviside function. Then this $f$ is what we want. So this is a simple neural network without hidden layers.
\end{proof}

%\begin{theorem}[Kernel function]For different finite set $\{A_i\}_{i=1}^c$in $\mathbb{R}^n$(may we need some other conditions), there exists a function $g: \mathbb{R}^n \to \mathbb{R}^m$ such that $\{g(A_i)\}_{i=1}^c$ will satisfy the assumption of linear separable \ref{assum:linsep}.
%\end{theorem}
%\begin{proof}Xiaodong or check literature.
%\end{proof}

\begin{definition}[Separabel]
For different finite set $\{A_i\}_{i=1}^c$in $\mathbb{R}^n$, we say they are separable if there exists a kernel function $g \in (C^0({I^n}))^m$ such that $\{g(A_i)\}_{i=1}^c$ will satisfy the assumption of linear separable \ref{assum:linsep}.
\end{definition}

\begin{remark} Here we show some some cases about the mapping $g$. We can construct a partition of $\mathbb{R}^n$ based on $\{A_i\}_{i=1}^c$, and then use partition of unity we get
$\{g_i\}_{i=1}^c$ with
\begin{equation}
g_{i}(x) = \delta_{ij} \quad \forall i ~~ \text{and}~~  x \in A_j.
\end{equation}
Then we will have:
\begin{align}
g: \mathbb{R}^n &\to \mathbb{R}^c \\
x &\mapsto (g_1, \cdots, g_c).
\end{align}
This means that, if $x\in A_i$ then
\begin{align}
g(x) = e_i \quad \forall i = 1:c.
\end{align}

This means $\{g(A_i)\}_{i=1}^c$ are just $ m = c$ points and can be linearly separated with sparse structure for $W$. Here one can just take $W = I_{m \times c} = I_{c\times c}$ and $b_i = -\frac{1}{2}$.

What's more, we can think about the binary problems $A_0$ and $A_1$, if there is a manifold $\mathcal{M}$ such that $\mathcal{M}$ can separate them.(We can fix the orientation of $\mathcal{M}$, and makes this statement in mathematics.) If
\begin{align}
\mathcal{M} = \{ x ~ | ~ \phi(x) = 0\},
\end{align}
with $\phi \in C^{\infty}(\mathbb{R}^n)$. So, here you have two ways to choose the mapping,
\begin{itemize}
\item Just choose $g = \phi$, then this satisfy this condition of the continuity of $\phi$.
\item If $\phi = \sum_{i=1}^m w_i \phi_i + b = 0$, so we can take $g = (\phi_1, \cdots, \phi_m)$, and the separation hyperplane is:
\begin{align}
\sum_{i=1}^m w_i y_i + b = 0,
\end{align}
in $\mathbb{R}^m$.
\end{itemize}

{\bf But, how to extend this binary problems into multi-classification problems in this way seems to
non-trival.}

\end{remark}

\begin{theorem}
For different finite set $\{A_i\}_{i=1}^c$in $\mathbb{R}^n$ satisfy the separable condition, then there exists a DNN model with one hidden layer such that
\begin{equation}
f_\theta(x_i) = e_i \quad \forall x_i \in A_i \quad i = 1:c,
\end{equation}
with
\begin{align}
f_{\theta}(x) = H(W_2\sigma(W_1x + b_1) + b_2).
\end{align}
What's more, we can set $W_2 \in \mathbb{R}^{c \times N}$ with
\begin{equation}
W_2 = W \cdot K,
\end{equation}
such that $W \in \mathbb{R}^{c \times m}$ and $K \in \mathbb{R}^{m \times N}$.
\end{theorem}
\begin{proof}
First, there exists a $g$ and $H(Wx + b)$ such that
\begin{align}
\tilde{f}_{\theta}(x_i) = H(W\tilde{g}(x_i) + b) = e_i \quad \forall x_i \in A_i \quad i = 1:c.
\end{align}

Then by the continuous and properties of Heaviside function, there exists $\epsilon > 0$ such that so for any $g \in  (C^0({I^n}))^m$ with
\begin{align}
\|g - \tilde{g}\|_{L^{\infty}} \le \epsilon.
\end{align}
We will have
\begin{align}
f_{\theta}(x_i) = H(Wg(x_i) + b) = e_i \quad \forall x_i \in A_i \quad i = 1:c.
\end{align}

Because of the universal approximation theory for artificial neural network, i.e. there exists a network like
\begin{align}
g(x) = K\sigma(W_1 x + b_1) + \alpha,
\end{align}
with
\begin{align}
K = \mathrm{diag}\{K_1, \cdots, K_m\}, \quad K_i \in \mathbb{R}^{1\times n_i}  \quad N = \sum_{i=1}^m n_i,
\end{align}
such that
\begin{align}
\|g - \tilde{g}\|_{L^{\infty}} \le \epsilon.
\end{align}

So
\begin{align}
f_{\theta}(x_i) = H(W_2\sigma(W_1x_i + b_1) + b_2) = e_i \quad \forall x_i \in A_i \quad i = 1:c,
\end{align}
and
\begin{equation}
W_2 = W \cdot K,
\end{equation}
with $W \in \mathbb{R}^{c \times m}$ and $K \in \mathbb{R}^{m \times N}$.

\end{proof}

\section{CNN model under this framework}
\subsection{Simple convolution introduction(without multichannel)}
There is a kind of special linear map from some general finite dimension tensor space to another is called ``convolution".  Generally, we have a tensor like $X \in \mathbb{R}^{n \times m}$ and a small kernel such as $K \in \mathbb{R}^{k\times k}$, then we have:
\begin{definition}[Original Convolution with stride 1 for CNN]
\begin{equation}\label{equ:conv}
(X\ast K)_{i,j} = \sum_{s, t = 1}^k X_{i-1 + s, j-1 + t} K_{s,t}.
\end{equation}
It's easy to see that $(X\ast K) \in \mathbb{R}^{(n-k + 1) \times (m-k + 1)}$.
\end{definition}
If we use stride $p$, we have:
\begin{definition}[Original Convolution with stride $p$ for CNN]
\begin{equation}\label{equ:convstride}
(X\ast_{p} K)_{i,j} = \sum_{s, t = 1}^k X_{(i-1)p + s, (j-1)p + t} K_{s,t}.
\end{equation}
So, we have
$
(X\ast_p K) \in \mathbb{R}^{((n-k)/p + 1) \times ((m-k)/p+1)}.
$
\end{definition}
This stride properties in some application is be used as pooling(subsampling, coarsening).

Here we can define a special pooling operator as $S^p$ which likes the $C/F$ split for choosing coarse point:
\begin{equation}
S^p(X)_{i,j} = X_{(i-1)p + 1, (j-1)p + 1},
\end{equation}
then we have:
\begin{equation}\label{equ:stride}
X \ast_p K = S^p(X\ast K),
\end{equation}
with the $\ast$ and $\ast_p$ defined by \ref{equ:conv} and \ref{equ:convstride}
\begin{proof}
\begin{align}
S^p(X \ast K)_{i,j} &= (X \ast K)_{(i-1)p+1, (j-1)p + 1}  \\
&= \sum_{s,t = 1}^k X_{(i-1)p + 1 -1 +s, (j-1)p +1 -1 +t}K_{s,t}  \\
&= (X \ast_p K)_{i,j}.
\end{align}
\end{proof}

\subsubsection{Convolution(without changing size)}
For many case kernel size is some small odd numbers such as 1, 3, 5... So we ca have those next definition for convolution:
\begin{definition}[Convolution with Padding]

We write $K\in \mathbb R^{2k+1, 2k+1}$, we then write
\begin{equation}\label{ConvPadding}
(X \hat{\ast} K)_{i,j} := (\rm{P}^k(X) \ast K)_{i,j} = \sum_{s, t = -k}^k X_{i + s, j + t} K_{s,t},
\end{equation}
with $\rm{P}^k$ means Padding.
\end{definition}
Here we see that, to make this definition well-defined, we have $X\ast K \in \mathbb{R}^{n-2k, m-2k}$, because if the index of $X$ start from $1$ to $n(m)$, then $X \ast K$ from $k+1$ to $n(m) - k$. To make the convolution don't change size, we can use Padding strategy like in image process.
%In fact, if the resolution is very high, which means the measure of the boundary of the image is close to 0, so in traditional image process, this strategy hasn't been studied very well. There are basic three strategy:
\begin{description}
\item[Zero Padding] The simplest and most common used strategy is zero padding with:
\begin{equation}
\rm{P}^k_0: \mathbb{R}^{n \times m} \to \mathbb{R}^{n+2k \times m+2k}
\end{equation}
with
\begin{equation}
\rm{P}_0^k(X)_{i,j} = \begin{cases}
0 &i(j)=-k+1:0 ~~\text{and}~~ i(j) = n(m)+1:n(m)+k \\
X_{i,j} &i(j) = 1:n(m)
\end{cases}
\end{equation}
\item[Reflection Padding] This strategy is defined by:
\begin{equation}
\rm{P}^k_r: \mathbb{R}^{n \times m} \to \mathbb{R}^{n+2k \times m+2k}
\end{equation}
with
\begin{equation}
\rm{P}_r^k(X)_{i,j} = \begin{cases}
X_{1-i, 1-j} &i(j)=-k+1:0 ~~\text{and}~~ i(j) = n(m)+1:n(m)+k \\
X_{i,j} &i(j) = 1:n(m) \\
0 &\text{others}
\end{cases}
\end{equation}
\item[Shift Padding] This strategy is defined by:
\begin{equation}
\rm{P}^k_s: \mathbb{R}^{n \times m} \to \mathbb{R}^{n+2k \times m+2k}
\end{equation}
with
\begin{equation}
\rm{P}_s^k(X)_{i,j} = \begin{cases}
X_{k-i, k-j} &i(j)=-k+1:0 ~~\text{and}~~ i(j) = n(m)+1:n(m)+k \\
X_{i,j} &i(j) = 1:n(m) \\
0 &\text{others}
\end{cases}
\end{equation}
\end{description}

So we can have:
\begin{equation}
\rm{dim}(X\hat{\ast}K) = \rm{dim}(\rm{P}^k(X) \ast K) = \rm{dim}(X),
\end{equation}
and the $i,j$ index is consistent for both side of the above equation where $\rm{P}^k = \rm{P}_0^k, \rm{P}_{r}^k$ or $\rm{P}_s^k$.

\begin{remark}We note all the later convolution as the \eqref{ConvPadding} for consistent.

\end{remark}

\subsection{Convolution with multichannel}
\subsubsection{Traditional convolution for multichannel case}
Here in real data, we have some special data like colour image $X \in \mathbb{R}^{n \times m \times3}$. Here this image is a 2D graph, with $3$ means the $RGB$ channels. So, we cannot just think $X$ as a general $3$-order tensor. We may just use this as an example, $X \in \mathbb{R}^{n\times m \times c}$, we say that $n \times m$ is the essential dimension and $c$ is the channel dimension for $X$. If we want to do convolution for $X$, here we cannot just use $K \in \mathbb{R}^{k\times k}$ because of multichannel. A simple idea is to use also different $c$ kernels and collect them together as $K \in \mathbb{R}^{k\times k \times c}$, and then we can do the general convolution for signal channel separately. This means that
\begin{equation}
(X\ast_{p} K)_{i,j,l} = \sum_{s, t = 1}^k X_{(i-1)p + s, (j-1)p + t,l} K_{s,t,l},
\end{equation}
and $(X\ast_p K) \in \mathbb{R}^{((n-k)/p + 1)\times ((m-k)/p+1) \times c}$. But we need to recall that, even for $X \in \mathbb{R}^{n\times m \times c}$ it is in fact only stand for just one 2D image(with multi-channel). So if we talk about features in AI, $(X\ast_p K) \in \mathbb{R}^{((n-k)/p + 1)\times ((m-k)/p+1) \times c}$ should be reduced to the essential dimension, one simple way is just to add all the channels value together w.r.t essential dimension(this is equal to added with weights because we can reduce those DoF into the previous $c$ kernels.):
\begin{equation}
(X \hat{\ast}_p K)_{i,j} = \sum_{l=1}^{c} \sum_{s, t = 1}^k X_{(i-1)p + s, (j-1)p + t,l} K_{s,t,l},
\end{equation}
then we have
\begin{equation}
(X \hat{\ast}_p K) \in \mathbb{R}^{((n-k)/p + 1) \times ((m-k)/p+1)} \quad \text{with} \quad K \in \mathbb{R}^{n \times m \times c}.
\end{equation}
So, for a given real data with $c$ channels like $X \in  \mathbb{R}^{n\times m \times c}$ if we use kernel $K \in \mathbb{R}^{k\times k \times c}$ we can only get a signal channel output. To get multichannel form $X$, we just need to use $d$ kernels $K \in \mathbb{R}^{k\times k \times c}$, or we say $K \in \mathbb{R}^{k\times k \times c \times b}$, then we have:
\begin{equation}\label{6.6}
(X \hat{\ast}_p K)_{i,j,g} = \sum_{l=1}^{c} \sum_{s, t = 1}^k X_{(i-1)p + s, (j-1)p + t,l} K_{s,t,l,g},
\end{equation}
with
\begin{equation}
(X \hat{\ast}_p K) \in \mathbb{R}^{((n-k)/p + 1) \times ((m-k)/p+1) \times b} \quad \text{with} \quad K \in \mathbb{R}^{n \times m \times c \times b}.
\end{equation}

In the next section, we set that all tensor is with
multichannel(signal channel is a special case, and we note it as $X
\in \mathbb{R}^{n \times m \times 1}$), and we just use $\ast_p$ no
longer $\hat{\ast}_p$.

\subsubsection{New version}
Here we consider our data like $T = (T_1, T_2, T_3)$ with $T_i \in \mathbb{R}^{n\times n}$.

Prof. Xu's idea is to say that, we need to keep the original ``channel" parallel i.e we just need three kernels like
\begin{align}
K_0 = \begin{pmatrix}
K^1 \\ K^2 \\ K^3
\end{pmatrix}.
\end{align} So we get:
\begin{equation}
T \to T\tilde{\ast} K_0 = (T_1 \ast K^1, T_2\ast K^2, T_3 \ast T^3),
\end{equation}
and then we apply the active function $a$ to $(T_1 \ast K^1, T_2\ast K^2, T_3 \ast T^3)$ element by element. And then use
\begin{align}
T^{new} = (a(T_1 \ast K^1), a(T_2\ast K^2), a(T_3 \ast T^3))
\end{align} as the next input for the CNN network.

Now we want to show that the traditional method with more kernels can reproduce the $T^{new}$ in Prof. Xu's case.

Here we set we have some more kernels like
\begin{align}
K =
\begin{pmatrix}
K_1^1 & \cdots & K_1^N \\
K_2^1 & \cdots & K_2^N \\
K_3^1 & \cdots & K_3^N
\end{pmatrix},
\end{align} with $K_i^j \in \mathbb{R}^{3\times3}$.

What we have at most are:
\begin{equation}
\tilde{T} = T\tilde{\ast} K =  \begin{pmatrix}
T_1 \ast K_1^1 & \cdots & T_1\ast K_1^N \\
T_2 \ast K_2^1 & \cdots & T_2\ast K_2^N \\
T_3 \ast K_3^1 & \cdots & T_3\ast K_3^N
\end{pmatrix}
\end{equation}

{\bf Here in fact we can have a simple question: why we don't just put $a(\tilde{T})$ as the next input for the next layer of CNN? Maybe this dimension is to large?}

Traditional method for convolution with channel is to say we need to compress the original ``channel" dimension by take
\begin{equation}
\tilde{T}  \to ( \sum_{j=1:3} T_j\ast K_{j}^1, \cdots, \sum_{j=1:3} T_j\ast K_{j}^N),
\end{equation}
and then apply the active function $a$ getting:
\begin{equation}
(a( \sum_{j=1:3} T_j\ast K_{j}^1), \cdots, a(\sum_{j=1:3} T_j\ast K_{j}^N)).
\end{equation}

So we can see that, if we take $ N = 3$ then we can reproduce the
$T^{new} = (a(T_1 \ast K^1), a(T_2\ast K^2), a(T_3 \ast T^3))$ from $(a( \sum_{j=1:3} T_j\ast K_{j}^1), \cdots, a(\sum_{j=1:3} T_j\ast K_{j}^N))$ by just take
\begin{align}
K^i = \sum_{j=1:3} K_j^i, \quad \forall i = 1:3.
\end{align}

\begin{remark}
Here we can take Prof. Xu's idea and the traditional convolution as consistent by means that because the dimension for $a(\tilde{T})$ is too large({\bf In fact, we like redundancy some times, why we just use this directly?}), we need some ways to compress $\tilde{T}$ first and then apply active function $a$. The simplest next two ways is:
\begin{itemize}
\item Compress(sum) $\tilde{T}$ in row. This leads to Prof. Xu's idea keeping the ``channel" number as 3.
\item Compress(sum) $\tilde{T}$ in column. This leads to traditional idea changing the ``channel" into $N$.
\end{itemize}

{\bf Another questions is how to compress this $\tilde{T}$ ``better"?  What dose better means?}

\end{remark}

\subsection{Convolution with vectorization and matrix tensor product}
\subsubsection{Vectorization and matrix tensor product}
First, we can define the next vectorization operator:
\begin{equation}
\rm{Vec}: \mathbb{R}^{n \times m} \to \mathbb{R}^{nm}
\end{equation}
with
\begin{equation}
(\rm{Vec}(X))_{(i-1)m + j} = X_{ij}.
\end{equation}

In this notation, for a discrete $u_{i,j}$ with $ i, j = 1:n$, we can have the next 5-points difference scheme with:
\begin{equation}
\rm{Vec}(\Delta_h u) = M_5 \rm{Vec}(u),
\end{equation}
with
\begin{equation}
M_5 = T\otimes I_{n} + I_{n} \otimes T,
\end{equation}
where $T$ is the $1-D$ mid-point difference matrix like:
\begin{equation}
T = \rm{tridiag}(-1,2,-1) \in \mathbb{R}^{n \times n}
\end{equation}

\subsubsection{5-points difference as a special convolution}
For general convolution definition, the convolution operator will reduce the dimension for the input data. Here we begin with $1-D$ case, like $K \in \mathbb{R}^{3}$, then for a vector $X \in \mathbb{R}^n$, we do the general convolution with stride 1, then we have
\begin{equation}
X \ast K \in \mathbb{R}^{n-2},
\end{equation}
and if we take $K = (-1, 2, -1)$, then
\begin{equation}
X \ast K = \Delta_h X_{2:n-1},
\end{equation}
with boundary condition $X_{\rm{left}} = X_1$ and $X_{\rm{right}} = X_n$.

To preserve the dimension, we can use the ``Padding" operator for $1-D$:
\begin{equation}
\rm{Pad}(X) = (0, X, 0),
\end{equation}
there are similar definition for $n-D$ tensor.

Then we have:
\begin{equation}
\rm{Pad}(X) \ast K = \Delta_h X = T X.
\end{equation}

So for $2-D$ case $u \in \mathbb{R}^{n \times n}$, and
\begin{equation}
K =
\begin{pmatrix}
0 & -1 & 0 \\
-1 & 4 & -1 \\
0 & -1 & 0
\end{pmatrix}
\end{equation}
then we have:
\begin{equation}
\rm{Vec}(\rm{Pad}(u) \ast K ) = \rm{Vec}(\Delta_h u) = M_5 \rm{Vec}(u).
\end{equation}

\subsection{Convolution explained by liner map with special structure}
CNN can also be recovered by the abstract form above with the represent of $V_i$ to be general tensor space. For example, $V_{i-1} = \mathbb{R}^{n \times m}$ and $V_i = \mathbb{R}^{l}$, then we can just set
\begin{equation}
A_i(X) = WX + \theta \in V_i,
\end{equation}
where
\begin{equation}
(WX)_{g} = \sum_{s,t=1}^{n,m}W_{s,t,g}X_{s,t}, \quad W \in \mathbb{R}^{l \times n \times m}
\end{equation}

Just as what we see above, the dimension of $V_i$ can not be arbitrary, it is defined by the kernel and stride size and the numbers of the kernel.

An example for this is like that, $X \in V_{i-1}$ is a signal channel 2D tensor, and we choose just one kernel $K \in \mathbb{R}^{k\times k}$ with stride p, which means that $V_i = \mathbb{R}^{(n-k)/p + 1)\times ((m-k)/p+1)}$.  Then the $W \in \mathbb{R}^{((n-k)/p + 1)\times ((m-k)/p+1) \times n \times m}$ should be like:
{\small{
\begin{equation}
W_{i,j, s,t} = \begin{cases}
K_{s-(i-1)p,t-(j-1)p} ~ &\text{if} ~ s \in (i-1)p+1:(i-1)p+k ~ \& ~ t \in (j-1)p+1:(j-1)p+k \\
&0.
\end{cases}
\end{equation}
}}
Now if we consider $X \in V_{i-1} = \mathbb{R}^{n \times m \times c }$ with $c$ channels and we choose kernel as $K \in \mathbb{R}^{k\times k \times c \times b}$ and use convolution with stride p, which means $V_{i} = \mathbb{R}^{((n-k)/p + 1)\times ((m-k)/p+1) \times b}$. So we have
\begin{equation}
W \in L(V_{i-1}, V_i) = \mathbb{R}^{(((n-k)/p + 1)\times ((m-k)/p+1) \times b) \times (n \times m \times c)},
\end{equation}
with
{\small{
\begin{equation}
W_{i,j,g, s,t,l} = \begin{cases}
K_{s-(i-1)p,t-(j-1)p,l,g} ~ &\text{if} ~ s \in (i-1)p+1:(i-1)p+k ~ \& ~ t \in (j-1)p+1:(j-1)p+k \\
&0.
\end{cases}
\end{equation}
}}
Then we have:
\begin{equation}
WX = X \ast_p K.
\end{equation}

At last, in the CNN model, the shift vector $\tilde\theta \in V_i$ is also have a special structure with
\begin{equation}
\tilde\theta_{i,j,l} = \theta_l,
\end{equation}
which means $dim(\tilde\theta) = b = $ the number of kernels in $K$. Which means that:
\begin{equation}
dim(W) + dim(\tilde\theta) = dim(K) + dim(\tilde\theta) = k\times k \times c \times b + b \neq dim(L(V_{i-1}, V_i)) + dim(V_i).
\end{equation}

\subsection{Active function}

\subsection{Pooling and convolution with stride}
Just as saying before, if $X \in V_{i-1} = \mathbb{R}^{n \times m \times c }$ with $c$ channels and we choose kernel as $K \in \mathbb{R}^{k\times k \times c \times b}$ and use convolution with stride p, then we have
\begin{equation}
V_{i} = \mathbb{R}^{((n-k)/p + 1)\times ((m-k)/p+1) \times b},
\end{equation}
which means that the essential dimension of $X$ was almost decreased to $\frac{1}{p}$, which is almost like result of the pooling(subsampling or coarsening) operation. So, in some recent popular CNN models, they often choose the convolution with stride $2$ as the a combination of convolution and pooling layers.

However, we can say that, some general pooling operation is in fact nonlinear operation but convolution with stride $p$ is still a special linear map and this may also change the channel dimensions. So we would like to added the pooling operation as a special kind of restriction, such as $X \in V_{i-1} = \mathbb{R}^{2n \times 2m \times c }$ then
\begin{equation}
R(X)_{i,j,l} = r(X_{(2i - 1:2i), (2j-1:2j),l}),
\end{equation}
with $r(\bm{x})$ can be:
\begin{itemize}
\item Maxout: $r(\bm{x}) = \max \{x_1, \cdots, x_4\}$,
\item Average: $r(\bm{x}) = \frac{\sum_{i=1}^4 x_i}{4}$,
%\item $L^2$ normal: $r(\bm{x}) = \|x\|$,
\item $\cdots$
\end{itemize}
Here we see that $R: V_i \to \hat{V}_i$ only change the essential dimension for $X$ and apply $r$ separately w.r.t channel dimension. In real application, you can also use the channel-cross pooling strategy which will a little like the convolution with stride.

\subsection{General structure for CNN}
A general CNN model is the composition of convolution, nonlinear map and pooling layers then followed with few ANN layers.
Now we still assume that data is like $\{X^i, Y^i\}_{i=1}^N$ with $X_i \in V_0$ and $Y_i \in U$. We choose some auxiliary space $V_i$, $\hat{V}_i$ with $i = 1:J$ and $U_i$ with $i = 1:L$.

We can use just $f_{ANN}(W_A; x)$ to denote the ANN model from $V_J$ to $U$ with auxiliary space $U_i$ with $i = 1:L$.

And the CNN part can be expressed as:
\begin{align}
f_{CNN} &= f_J(W_{CNN};x), \\
f_j &= R_j \circ g(W_i f_{j-1} + \tilde{\theta}_i), \quad j = 1:J \\
f_0 &= x.
\end{align}
\begin{itemize}
\item For many real CNN models, $R_j = \rm{id}$ for many $j \in 1:J$.
\item This likes a slash cycle with many smooth steps in every level.
\item The reason why they need so many smooth steps is to increase the parameter numbers, so try general cycle might be a interesting method.
\end{itemize}

Ana at last, we get a general CNN model as:
\begin{equation}
f(W;x) = f_{ANN}(W_{ANN}; f(W_{CNN};x)), \quad \text{with} \quad W = \{W_{ANN}, W_{CNN}\}.
\end{equation}

\section{RNN model under this framework}
\subsection{A simple example of RNN}
Here we consider our problem with data like $X^i = (x^{i,(1)}, \cdots, x^{i,(t)}, \cdots, x^{i,(\tau)})$ with $x^{i,(t)} \in V^{(t)} = V$ for all $t = 1:\tau$ and $Y^i = (y^{i,(1)}, \cdots, y^{i,(t)}, \cdots, y^{(i,\tau)})$ with $y^{i,(t)} \in U^{(t)} = U$ for all $t = 1:\tau$. So, we can say that:
\begin{equation}
X \in \bar{V} = \underbrace{V\times V \times\cdots \times V }_{\tau} ~\text{and}~ Y \in \bar{U} =  \underbrace{U\times U \times\cdots \times U }_{\tau}.
\end{equation}
So we can use the similar idea in ANN, by choose auxiliary space $V_i$ and construct:
\begin{equation}
\bar{V}_i = \underbrace{V_i \times V_i \times \cdots \times V_i}_{\tau}, \quad i = 1:J.
\end{equation}

So, we can also talk about the ANN with $\bar{V}_i$ $i = 0:J+1$ like the definition in \ref{equ:abstractNN1} to \ref{equ:abstractNN3}. What's more, if
\begin{equation}
\bar{W}_i = \text{diag}(W^{(1)}_i, \cdots, W_i^{(\tau)}),
\end{equation}
and $g_i$ is a nonlinear map form $V_i^{(t)}$ to $V_i^{(t)}$. Then the ANN model form $\bar{V}$ to $\bar{U}$ is just $\tau$ separate ANN model form $V$ to $U$. So, a naive idea to change this should be choose $\bar{W}_i$ not be block diagonal matrix. {\bf{But, this is not RNN!}}

Now we show a simple RNN model with:
\begin{align}
f_{1}^{(t)} &= g_1^{(t)}(\theta_1 + W_1f_1^{(t-1)} + U_1x^{(t)}), \\
f_{2}^{(t)} &= \theta_2 + U_2f_{1}^{(t)}, \\
f^{(t)} &= \rm{softmax}(f_2^{(t)}).
\end{align}

Also we can have this kind of RNN network:
\begin{align}
f_{1}^{(t)} &= g_1^{(t)}(\theta_1 + W_1f_2^{(t-1)} + U_1x^{(t)}), \\
f_{2}^{(t)} &= \theta_2 + U_2f_{1}^{(t)}, \\
f^{(t)} &= \rm{softmax}(f_2^{(t)}).
\end{align}

So it seems that, we can have the next definition for RNN:
\begin{equation}\label{equ:abstractRNN1}
f = f_{out}\circ f_J,
\end{equation}
with
\begin{equation}\label{equ:abstractRNN2}
f^{(t)}_j = g_j(\bar{A}_j(f^{(t)}_{j-1}) + \bar{W}_{j}^j f^{(t-1)}_{j}),
\end{equation}
and
\begin{equation}\label{equ:abstractRNN3}
f_0 = f_{in}(x), \quad x \in \bar{V}.
\end{equation}
Here
\begin{equation}
\bar{W}_j^{j} \in L(\bar{V}_j, \bar{V}_j), \text{is a blocked lower triangular matrix}.
\end{equation}
which is invariant through all timesteps from $t=1$ to $\tau$. \\
This formula seems the symmetric form for the next section about the shortcut connection form.
\subsection{Development of the simple RNN}
Note that $\bar{W}_j^j$ denotes the relation between $V_j^{(t-1)}$ with $V_j^{(t)}$,we suppose that the relation is the same through all the timesteps, which is similar to the model of Markov chain. \\
However, there exists some problems is the $\bar{W}_j^j$ is a simple affine transformation. With the same matrix multiplied by many times, the eigenvalues will explode or decay to zero quickly. That leads to gradient exploding or gradient vanishing, which makes the simple model hard to capture the long-term dependence. Here we introduce the LSTM model to solve the problem. \\
Here we transforms the $\bar{A}_j$ and $\bar{W}_j^J$ into the following form:
\begin{equation}
f^{(t)}_j = g_j (\bar{W}_{j}^j (f^{(t-1)}_{j},f^{(t)}_{j-1})),
\end{equation}
\begin{equation}
\bar{W}_{j}^j (f^{(t-1)}_{j},f^{(t)}_{j-1}) = \sigma_1 \ast s_j^{(t-1)} + \sigma_2 \ast q_j^{(t)}
\end{equation}
where
\begin{align}
s_j^{(t-1)} & = g_j^{-1} f_j^{(t-1)} = \bar{A}_j(f^{(t-1)}_{j-1}) + \bar{W}_{j}^{j} f^{(t-2)}_{j} \\
\sigma_1 & = g_j^{(f)}(\bar{A}_j^{(f)}(f^{(t)}_{j-1}) + \bar{W}_{j}^{j^{(f)}} f^{(t-1)}_{j}) \\
\sigma_2 & = g_j^{(g)}(\bar{A}_j^{(g)}(f^{(t)}_{j-1}) + \bar{W}_{j}^{j^{(g)}} f^{(t-1)}_{j}) \\
q_j^{(t)} & = g_j^{(n)}(\bar{A}_j^{(n)}(f^{(t)}_{j-1}) + \bar{W}_{j}^{j^{(n)}} f^{(t-1)}_{j})
\end{align}
The superscripts f and g denote the 'forget' and 'get' gating, which control which part of $f_j^{(t-1)}$ and $f_{j-1}^{(t)}$ can be transformed into $f_j^{(t)}$.Here $\bar{A}_j^{(f)} $,$\bar{W}_{j}^{j^{(f)}}$,$\bar{A}_j^{(g)}$ and $\bar{W}_{j}^{j^{(g)}}$ are of the same definition in the simple RNN model. That's exactly the LSTM model. \\
Also,the LSTM model can be simplified into the GRU model, which keeps $\sigma_1$ with the same definition, while$\ \sigma_2 = 1 - \sigma_1$.That saves some cost of the computation.

\section{Shortcut connection}
First, we would like to talk about the so called ``shortcut connection" in general neural network. It can be defined more generally by
\begin{equation}
f_j = g_j(A_j(f_{j-1} + \sum_{i=0}^{j-2}\tilde{W}_i^j f_i)) = g_j( \sum_{i=0}^{j-1}W_i^{j}f_i + \theta_j),
\end{equation}
where
\begin{equation}
W_i^j \in  L(V_i, V_j) \quad \text{and} \quad W_j = W_{j-1}^j.
\end{equation}
\subsection{ResNet}
The most important structure in ResNet model is the ``shortcut connection'' with 2-shortcusts, i.e:
\begin{align}
&f = f_{out}\circ f_J, \\
&\begin{cases}
f_j = g_j(A_jf_{j-1} + I_jf_{j-2}), \quad &\text{for} ~j~ \text{is odd} \\
f_j = g_j(A_jf_{j-1}), \quad  &\text{for} ~j~ \text{is even},
\end{cases} \\
&f_0 = x.
\end{align}

Here the convolution in ResNet adopt padding with zero such that the essential dimension will not change expect after the pooling layers. So in many case $I_j$ is just identity operator. If the are dimension reduction in essential dimension the $I_j$ is a simple restrict operator which needs to learn and just add zeros for the channel dimension.

\subsection{Two-grid method}
So, if we use the shortcut connection in a general ANN model, we can recover the simple two-grid scheme to solve $Au = b$ from $u^k$ to $u^{k+1}$ with $u^k \in V = U = \mathbb{R}^n$.

One thing I need to mention again is that, here we just use the network frame to recover the two-grid iterative scheme, all $A_i$ and $g_i$ is defined by ourself, one on need to be learnt.

Here we set $g_i = id$ for $i = 1:6$.  And $V_i = V$ for $i = 1,4,5,6$ and $V_i = V_c$ for $i = 2,3$, where $V_c$ is the coarse space. $P$ is the interpolation operator form $V_c$ to $V$ and $A_c = P^T A P$, $R$ is the smoother on the fine grid. So we have the next definition of $A_i$:
\begin{itemize}
\item $f_1 = A_1(f_0) = b - Af_0$,
\item $f_2 = A_2(f_1) = P^T f_1$,
\item $f_3 = A_3(f_2) = A^{-1}_c f_2$,
\item $f_4 = A_4(f_3) = P f_3$,
\item $f_5 = A_5(f_4) + W_1^5 f_1 =f_1 -  Af_4$,
\item $f_6 = A_6(f_5) + W_0^6 f_0 + W_4^6 f_4 = f_0 + f_4 + Rf_5$.
\end{itemize}
Then if we take $f_0 = u_k$ we will have $f_6 = u^{k+1}$ comes from a two-grid step with
\begin{equation}
u^{k+1} = u^{k} + B(b - A u^k),
\end{equation}
with $Bg$ is defined by
\begin{align}
w &= PA_c^{-1}P^{T}g, \\
Bg &= w + R(g - Aw).
\end{align}

In this point of view, it seems that adopting the shortcut connection in deep learning seems to make sense. And just as we mentioned before, we don't know what real human neural connections, and deep learning with shortcut connection will more general to approximate the human neural network. What's more, a popular CNN model ResNet used this structure and won the ImageNet classification challenge in 2015.

\section{U-Net and multigrid model in CNN}
Fro traditional CNN model, especially for classification or locallization problem, the general CNN model w.r.t essential dimension like a slash cycle very much, such as AlexNet, VGGNet or ResNet. But for segmentation problem, because of the high dimension of both input and output data, CNN model wasn't very popular for some reasons. Traditional method with image process still is the main methods. But in 2105, [ ] proposed a network to dealt with biology cell image segmentation problem, and win the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. The U-Net structure of essential dimension likes the V-cycle very much.

\begin{figure}[!htb]
	\center{\includegraphics[width=12cm] {UNetAIMedical1.png}}
	\caption{Architecture for U-Net}
	\label{U-Net-model}
\end{figure}

\newpage


\section{Rational Homology and Euler Characteristics}
\subsection{Rational Homology}
When we defined the $n$-chains of simplicial homology, we took it as the free abelian group, i.e. free $\mathbb Z$-module, generated by the $n$-simplices.
But there is no reason why we should just restrict ourself to $\mathbb Z$-modules.
\footnote{Well, you have the universal coefficient theorem.}
If we replace $\mathbb Z$ by $\mathbb Q$, we get a vector space.
And the homology theory on this is called the rational homology.
\begin{definition}
    Let $K$ be a simplicial complex.
    Define the $\mathbb Q$-vector space of rational $n$-chains to be the $\mathbb Q$-vector space with basis being the set of $n$-simplices of $K$.
    Denote this space by $C_n(K;\mathbb Q)$.\\
    The boundary map $\partial_n:C_n(K;\mathbb Q)\to C_{n-1}(K;\mathbb Q)$, the cycles $Z_n(K;\mathbb Q)$, the boundaries $B_n(K;\mathbb Q)$ and the homology groups $H_n(K;\mathbb Q)$ are defined exactly as before.
\end{definition}
Correspondingly, by viewing abelian groups as $\mathbb Z$-modules, we can think of the objects $C_n(K),Z_n(K),B_n(K),H_n(K)$ appeared in our original simplicial homology theory instead as $C_n(K;\mathbb Z),Z_n(K;\mathbb Z),B_n(K;\mathbb Z),H_n(K;\mathbb Z)$.
This base ring (e.g. $\mathbb Z$ or $\mathbb Q$ as we have seen) is called the coefficient of the simplicial homology.\\
Perhaps unsurprisingly, homology with coefficient in $\mathbb Q$ actually contains less information than homology with coefficients in $\mathbb Z$.
\begin{lemma}
    If $H_n(K;\mathbb Z)\cong\mathbb Z^b\oplus F$ where $F$ is a finite abelian group, then $H_n(K;\mathbb Q)=\mathbb Q^b$.
\end{lemma}
\begin{proof}
    There is a natural map $C_n(K;\mathbb Z)\to C_n(K;\mathbb Q)$ via the inclusion of $\mathbb Z$ in $\mathbb Q$.
    This induces a chain map $C_\bullet(K;\mathbb Z)\to C_\bullet(K;\mathbb Q)$ and hence a natural homomorphism $H_n(K;\mathbb Z)\to H_n(K;\mathbb Q)$.
    If $c\in Z_n(K;\mathbb Q)$, then there is an integer $m$ such that $mc\in Z_n(K;\mathbb Z)$ has integer coefficients.
    So $mc$ is in the image of the map $Z_n(K;\mathbb Z)\to Z_n(K;\mathbb Q)$.
    Now $H_n(K;\mathbb Q)\cong\mathbb Q^{b'}$ for some natural number $b'$ as it has to be a finite dimensional vector space.
    But the above argument then shows $b'\le b$.
    Let $[c_1],\ldots,[c_b]\in H_n(K;\mathbb Z)$ generate the $\mathbb Z^b$ factor in $H_n(K;\mathbb Z)$.
    Then it makes sense to talk about them as elements of $H_n(K;\mathbb Q)$ as well.
    Suppose there is $\lambda_1,\ldots,\lambda_b\in\mathbb Q$ not all zero such that $\sum_i\lambda_i[c_i]=0$ in $H_n(K;\mathbb Q)$, then there exists $c\in C_{n+1}(K;\mathbb Q)$ such that $\partial_{n+1}c=\sum_i\lambda_ic_i$.
    Pick integer $m>0$ such that $m\lambda_i$ are all integers, then $\partial_{n+1}(mc)=\sum_i(m\lambda_i)c_i$, therefore $\sum_i(m\lambda_i)[c_i]=0$ in $H_n(K;\mathbb Z)$.
    But $[c_i]$ are linearly independent in $H_n(K;\mathbb Z)$, so $m\lambda_i=0$ for all $i$, hence $\lambda_i=0$ and therefore $[c_i]$ are linearly independent in $H_n(K;\mathbb Q)$.
    This means $b'\ge b$.
    Combining the two gives $b'=b$.
\end{proof}
Consequently we cannot distinguish $\mathbb RP^2$ and a point with rational homology.
Then what is the point of having it?
Well, throwing away information isn't necessarily bad.
\subsection{Euler Characteristics}
\begin{definition}
    Let $K$ be a simplicial complex.
    The Euler characteristic of $K$ is
    $$\chi(K)=\sum_{n=0}^\infty(-1)^n\dim_{\mathbb Q}H_n(K;\mathbb Q)$$
    If $X$ is a topological space with $X=|K|$, then we write $\chi(X)=\chi(K)$ which is well-defined as the homology groups do not depend on specific triangulation.
\end{definition}
There is no issue about the convergence of the series as eventually $H_n=0$ and hence the series terminates.
\begin{lemma}\label{euler}
    $$\chi(K)=\sum_{n=0}^\infty(-1)^n\dim_{\mathbb Q}C_n(K;\mathbb Q)=\sum_{n=0}^\infty(-1)^n|\{\sigma\in K:\dim\sigma=n\}|$$
\end{lemma}
\begin{proof}
    Write $\dim=\dim_\mathbb Q$.
    Note that we have
    $$\dim H_n(K;\mathbb Q)=\dim Z_n(K;\mathbb Q)-\dim B_n(K;\mathbb Q)$$
    $$\dim C_n(K;\mathbb Q)=\dim\ker\partial_n+\dim\operatorname{Im}\partial_n=\dim B_{n-1}(K;\mathbb Q)+\dim Z_n(K;\mathbb Q)$$
    Thus we can just write
    \begin{align*}
        \sum_{n=0}^\infty(-1)^n\dim C_n(K;\mathbb Q)&=\sum_{n=0}^\infty (-1)^n\dim Z_n(K;\mathbb Q)+\sum_{n=1}^\infty (-1)^n\dim B_{n-1}(K;\mathbb Q)\\
        &=\sum_{n=0}^\infty (-1)^n(\dim Z_n(K;\mathbb Q)-\dim B_n(K;\mathbb Q))\\
        &=\sum_{n=0}^\infty(-1)^n\dim_{\mathbb Q}H_n(K;\mathbb Q)
    \end{align*}
    as desired.
\end{proof}
\begin{example}
    If $\dim K=2$, we get the familiar $\chi(K)=V-E+F$.
    So for example $\chi(S^2)=1-0+1=2$.
    Consequently any triangulation of the $2$-sphere, i.e. any polyhedron, has $V-E+F=2$.
\end{example}
\subsection{The Lefschetz Fixed-Point Theorem}
\begin{definition}
    Let $X$ be triangulable and $\phi:X\to X$.
    The Lefschetz number of $\phi$ is
    $$L(\phi)=\sum_{n=0}^\infty(-1)^n\operatorname{tr}(\phi_\ast:H_n(X;\mathbb Q)\to H_n(X;\mathbb Q))$$
\end{definition}
Again the sum is eventually zero.
\begin{example}
    $L(\operatorname{id}_X)=\chi(X)$.
\end{example}
\begin{lemma}
    If $f:K\to K$ is a simplicial map, then
    $$L(|f|)=\sum_{n=0}^\infty (-1)^n\operatorname{tr}(f_n:C_n(K;\mathbb Q)\to C_n(K;\mathbb Q))$$
\end{lemma}
\begin{proof}
    Given a commutative diagram of vector spaces
    \[
        \begin{tikzcd}
            0\arrow{r}&A\arrow{d}{\alpha}\arrow{r}&B\arrow{d}{\beta}\arrow{r}&C\arrow{d}{\gamma}\arrow{r}&0\\
            0\arrow{r}&A'\arrow{r}&B'\arrow{r}&C'\arrow{r}&0
        \end{tikzcd}
    \]
    with exact rows, then it is easy to check that $\operatorname{tr}\beta=\operatorname{tr}\alpha+\operatorname{tr}\gamma$.
    The proof follows almost immediately by using the same idea as we did in Lemma \ref{euler}.
\end{proof}
\begin{theorem}[Lefschetz Fixed-Point Theorem]
    Let $\phi:X\to X$ be a map where $X$ is triangulable.
    If $L(f)\neq 0$, then $\phi$ has a fixed point.
\end{theorem}
We can actually count the fixed point in the full version of the theorem, which is out of the scope of this course.
\begin{proof}
    We shall show that if $\phi$ has no fixed point, then $L(\phi)=0$.
    As $X$ is triangulable, it has to be compact, so there is some $\delta>0$ such that $\|x-\phi(x)\|>\delta$ for all $x\in X$.
    Choose $K$ such that $X=|K|$ and, possibly using barycentric subdivision, $\operatorname{mesh}K<\delta/2$.
    Then if $x\in\sigma\in K$, then $\phi(x)\notin\sigma$.
    Let $f:K^{(r)}\to K$ be a simplicial approximation to $\phi$.
    If $v\in K^{(r)}$ is a vertex with $v\in\sigma\in K$, then $\phi(v)\in\operatorname{St}_K(f(v))$, consequently $\|\phi(v)-f(v)\|<\delta/2$.
    But $\|\phi(v)-v\|>\delta$, so $\|v-f(v)\|>\delta/2$, so $f(v)\notin\sigma$.
    Let $i_\bullet:C_\bullet(K;\mathbb Q)\to C_\bullet(K^{(r)};\mathbb Q)$ be a chain map that induces the canonical isomorphism on homology, which is supposed to map an $n$-simplex in $K$ to the sum of the $n$-simplices in $K^{(r)}$ supported in it.
    Since $f$ takes vertices of $\sigma$ out of it, it follows that $f_n\circ i_n(\sigma)$ is supported on simplices disjoint from $\sigma$.\\
    Since $\phi_\ast$ is induced at the level of chains by $f_n\circ i_n$, we now have
    $$L(\phi)=\sum_{n=0}^\infty(-1)^n\operatorname{tr}f_n\circ i_n$$
    by the preceding lemma.
    But $i_n\circ i_n$ throws any $n$-simplex elsewhere, hence $\operatorname{tr}f_n\circ i_n=0$, so $L(\phi)=0$.
\end{proof}
\begin{corollary}
    If $X$ is triangulable and contractible, then any map $\phi:X\to X$ has a fixed point.
\end{corollary}
\begin{proof}
    We just need to show that it has nonzero Lefschetz number.
    $X\simeq\{\ast\}$, so essentially $H_n(X;\mathbb Q)=0$ if $n>0$ and $\dim H_0(X;\mathbb Q)=\mathbb Q$.
    So the only nonzero map $\phi_\ast$ is $\phi_\ast:H_0(X;\mathbb Q)\to H_0(X;\mathbb Q)$ which is the identity.
    Thus $L(\phi)=1$.
\end{proof}


\subsection{Orthogonal groups \(O(n, F)\)}

\subsubsection{Recap: Metric-preserving transformations}

The bilinear form is:

\(u^TMv\)

The transformations which preserve this are:

\(P^TMP=M\)

\subsubsection{The orthogonal group}

If the metic is \(M=I\) then the condition is:

\(P^TP=I\)

\(P^T=P^{-1}\)

These form the orthogonal group.

We use \(O\) instead of \(P\):

\(O^T=O^{-1}\)

\subsubsection{Rotations and reflections}

The orthogonal group is the rotations and reflections.

\subsubsection{Parameters of the orthogonal group}

The orthogonal group depends on the dimension of the vector space, and the underlying field. So we can have:
\begin{itemize}
\item \(O(n, R)\); and
\item \(O(n, C)\).
\end{itemize}

\subsubsection{We generally refer only to the reals}

\(O(n)\) means \(O(n,R)\).

The generally refer to the reals only.


\documentclass[../main.tex]{subfiles}

\begin{document}
\chapter{Derivative Formulæ}
\section{General Functions}
For any two functions $u$ and $v$:
\[ (u + v)' = u' + v' \]
When there is a constant, $c$:
\[ (c \cdot u)' = c \cdot u' \]
\section{Trigonometric Functions}
\begin{align*}
    \frac{d}{dx} \sin x &= \cos x\\
    \frac{d}{dx} \cos x &= - \sin x
\end{align*}
\section{Product Rule}
\begin{equation}
    (u \cdot v)' = v \cdot u' + u \cdot v'
    \label{eqn:product-rule}
\end{equation}
\begin{exmp}
    To differentiate $f(x) = x^3 \sin x$, we let
    $u = x^3$ and $v = \sin x$:
    \begin{align*}
        \therefore \quad u' &= 3x^2\\
                         v' &= \cos x
    \end{align*}
    From Equation \ref{eqn:product-rule}, we know:
    \begin{align*}
        f'(x)   &= vu' + uv'\\
                &= 3x^2 \sin x + x^3 \cos x
    \end{align*}
\end{exmp}
\section{Quotient Rule}
\[ \left( \frac{u}{v} \right)' = \frac{vu' - uv'}{v^2} \]
\section{Chain Rule}
The \emph{chain rule} (in Leibniz's notation) can be written
in the following way:
\[ \frac{dy}{dt} = \frac{dy}{dx} \cdot \frac{dx}{dt} \]
\begin{exmp}
    To differentiate $y = \sin^{10} t$, we let
    $x = \sin t$:
    \begin{align*}
        y &= x^{10}\\
        \therefore \quad \frac{dy}{dx} &= 10x^9
    \end{align*}
    From the chain rule, we get:
    \[ \frac{dy}{dt} = 10x^9 \cos t \quad \because \quad \frac{dx}{dt} = \cos t \]
    Finally, we need to substitute $x = \sin t$:
    \[ \frac{dy}{dt} = 10 \sin^9 t \cos t \]
\end{exmp}
\begin{exmp}
    To differentiate $\sin (10t)$, we let
    $x = 10t$ and $y = \sin x$:
    We now have:
    \[ \frac{dx}{dt} = 10 \quad , \quad \frac{dy}{dx} = \cos x \]
    \begin{align*}
        \frac{d}{dt} \sin (10t) &= \frac{dy}{dt}\\
        \frac{dy}{dt}           &= \frac{dy}{dx} \cdot \frac{dx}{dt}\\
                                &= 10 \cos x\\
                                &= 10 \cos (10t)
    \end{align*}
\end{exmp}
\end{document}


\subsection{Autoregressive Conditional Heteroskedasticity (ARCH)}

\subsubsection{Variance of the AR(\(1\)) model}

The standard AR(\(1\)) model is:

\(y_t=\alpha + \beta y_{t-1}+\epsilon_t \)

The variance is:

\(Var(y_t)=Var(\alpha + \beta y_{t-1}+\epsilon_t)\)

\(Var(y_t)(1-\beta^2)=Var(\epsilon_t)\)

Assuming the errors are IID we have:

\(Var(y_t))=\dfrac{\sigma^2 }{1-\beta^2 }\)

This is independent of historic observations, which may not be desirable.

\subsubsection{Conditional variance}

Consider the alternative formulation:

\(y_t=\epsilon_t f(y_{t-1})\)

This allows for conditional heteroskedasticity.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS671: Machine Learning
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 5}

Consider the following optimization problem.

\begin{equation}
\begin{aligned}
& \underset{\mathbf{x}}{\text{minimize}} & & x_1^2 + 2x_2^2\\
& \text{subject to} & & x_1 + x_2 - 1 \leqslant 0.\\
\end{aligned}
\label{eq51}
\end{equation}

Find a point satisfying the Kuhn-Tucker conditions.
Is that point an optimal solution?

\subsection*{Solution}

Let $\mathbf{x}(x_1, x_2)$ be a feasible solution of the optimization problem given in Eq. \ref{eq51}.
We define $f(\mathbf{x}) = x_1^2 + 2x_2^2$ and $g_1(\mathbf{x}) = x_1 + x_2 - 1$.
To verify Kuhn-Tucker conditions for any feasible solution $\mathbf{x}$, we first need to show that both functions $f$ and $g_1$ are convex and differentiable at $\mathbf{x}$.
Using the convexity theorem, $f$ is convex on $\mathbb{R}^2$ if and only if the Hessian matrix $H_f(\mathbf{x})$ is positive semidefinite for every $x\in \mathbb{R}^2$.
Hessian matrices $H_f(\mathbf{x})$ and $H_{g_1}(\mathbf{x})$ are computed in Eq. \ref{eq52} and Eq. \ref{eq53}, respectively.

\begin{equation}
H_f(\mathbf{x})
= \begin{pmatrix}
\frac{\partial^2 f}{\partial^2 x_1} & \frac{\partial^2 f}{\partial x_1 \partial x_2}\\[0.6em]
\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial^2 x_2}\\
\end{pmatrix}
= \begin{pmatrix}
2 & 0\\[0.6em]
0 & 4\\
\end{pmatrix}
\label{eq52}
\end{equation}

\begin{equation}
H_{g_1}(\mathbf{x})
= \begin{pmatrix}
\frac{\partial^2 g_1}{\partial^2 x_1} & \frac{\partial^2 g_1}{\partial x_1 \partial x_2}\\[0.6em]
\frac{\partial^2 g_1}{\partial x_2 \partial x_1} & \frac{\partial^2 g_1}{\partial^2 x_2}\\
\end{pmatrix}
= \begin{pmatrix}
0 & 0\\[0.6em]
0 & 0\\
\end{pmatrix}
\label{eq53}
\end{equation}

Since $H_f(\mathbf{x})$ is a diagonal matrix, its eigenvalues are the elements on its principle diagonal.
Also, since $H_{g_1}(\mathbf{x})$ is a zero matrix, it has only one eigenvalue $0$ of multiplicity $2$.
Since eigenvalues of both matrices are non-negative, both $H_f(\mathbf{x})$ and $H_{g_1}(\mathbf{x})$ are positive semi-definite.
Hence, we have shown both $f$ and $g_1$ are convex functions.

Therefore, according to Kuhn-Tucker theorem, any point $\mathbf{x}$ should satisfy conditions stated in Eq. \ref{eq54} and Eq. \ref{eq55} as well as the active constraint presented in Eq. \ref{eq56} to satisfy the Kuhn-Tucker Sufficient conditions.

\begin{equation}
(\nabla f)_\mathbf{x} + u_1 (\nabla g_1)_\mathbf{x} = \mathbf{0}
\label{eq54}
\end{equation}
\begin{equation}
u_1 \geq 0
\label{eq55}
\end{equation}
\begin{equation}
u_1 g_1(\mathbf{x}) = 0
\label{eq56}
\end{equation}

Given that $f(\mathbf{x}) = x_1^2 + 2x_2^2$ and $g_1(\mathbf{x}) = x_1 + x_2 - 1$, Eq. \ref{eq54} can be rewritten as shown in Eq. \ref{eq57}.
\begin{equation}
\begin{pmatrix} 2x_1 \\ 4x_2 \end{pmatrix} + u_1 \begin{pmatrix} 1 \\ 1 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}
\label{eq57}
\end{equation}
Therefore,
\begin{equation}
\left\{ \begin{array}{ll} 2x_1+u_1 = 0\\ 4x_2 + u_1 = 0\\ \end{array}\right.
\label{eq58}
\end{equation}
Hence,
\begin{equation}
2(2x_2-x_1) = 0
\label{eq59}
\end{equation}
But at the same time, Eq. \ref{eq56} requires
\begin{equation}
u_1(x_1 + x_2 - 1) = 0
\label{eq60}
\end{equation}
Therefore, either $u_1 = 0$ or $x_1 + x_2 - 1 = 0$.
In case, $u_1 \neq 0$,using Eq. \ref{eq59} $x_1$ and $x_2$ are obtained as $x_1 = 2/3$ and $x_2 = 1/3$.
However, substituting these values in Eq. \ref{eq58} would necessarily result in $u_1 \leq 0$ and this will contradict the Kuhn-Tucker condition in Eq. \ref{eq55}.
Therefore, $u_1 = 0$.
Finally, by substituting $u_1$ in Eq. \ref{eq55}, the point will be obtained as $\mathbf{x}(x_1, x_2) = (0, 0)$.
Since this point adheres to all Kuhn-Tucker conditions, it is an optimal point which minimizes the function $f = x_1^2 + 2x_2^2$ and satisfies $g(x) = -1 \leq 0$.


\label{capitulo FCINO}

\section{Introduction}

% Esta es la definición general de la FFT que habrá que definir en el futuro
% http://es.mathworks.com/help/symbolic/mupad_ref/ifourier.html

The continuous Fourier transform (CFT) of a function $f(t)$ and its inverse will be defined here as
\begin{equation}
\begin{gathered}
F(p) = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty} f(x)\,\, e^{-ipx}\,dx
\\
f(x) = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty} F(p)\,\, e^{ipx}\,dp
\end{gathered}
\end{equation}
where $i$ is the imaginary unit. The discrete Fourier Transform (DFT) and the inverse DFT of a
secuence $\mathbf{z}=\{z_1,z_2,\ldots,z_n\}$ (which may have real or complex values) will be defined as
\begin{equation}
\begin{gathered}
D_k(z) = \sum_{j=1}^n z_j e^{-2\pi i(j-1)(k-1)/n}\qquad\therefore\qquad k=1,2,\ldots,n
\\
D_k^{-1}(z) = \frac{1}{n}\sum_{j=1}^n z_j e^{2\pi i(j-1)(k-1)/n}\qquad\therefore\qquad k=1,2,\ldots,n
\end{gathered}
\end{equation}
An Fast Fourier Transform (FFT) computes the DFT and produces exactly the same result as evaluating the DFT definition directly; the most important difference
is that an FFT is much faster.

Let us assume that $f(x)$ is a zero outside the interval $(-L/2,L/2)$. Let $\Delta x=L/n$ be the interval in $x$ for the $n$ input values of $f(x)$.

Further, Fourier transform software routines require a fixed discrete grid spacing.
Then the sampling interval on grid spacing is $\Delta x=a/n$, and $\Delta p =2\pi/(n\Delta x)$ on momentum spacing.
\begin{equation}
\begin{gathered}
x_j = j\Delta x, \qquad f_j = f(x_j) \qquad\therefore\qquad j=-\frac{n}{2},\ldots,\frac{n}{2}
\\[4mm]
p_k = k\Delta p, \qquad F_k = F(p_k) \qquad\therefore\qquad k=-\frac{n}{2},\ldots,\frac{n}{2}
\end{gathered}
\end{equation}
Then, we can write CFT as follows
\begin{equation}
F_k = \frac{\Delta x}{\sqrt{2\pi}}\sum_{j=-n/2}^{n/2} f_j\,\, e^{-2\pi i kj/n}
\qquad\therefore\qquad
k=-\frac{n}{2},\ldots,\frac{n}{2}
\label{eq: CFT raw}
\end{equation}
However most of implementations use positive indices, and therefore the spatial samples inside the sum must be re-arranged.
According with the FFTW library we use the standard order (\textit{s-order}) vector. This means that the
positive values are stored in the first half of the vector and the negative frequencies
are stored in backwards order in the second half of the vector, with the zero-value component
in the first element. We will represent this format as a \textit{tilde}, for example: $\mathbf{z}$
represent the array in \textit{s-order} and $\mathbf{\tilde{z}}$ in \textit{n-order}.
\begin{equation}
\text{$s$-order:}\qquad \boldsymbol{z} = \Bigl( z \Bigr)_{J=1}^n
\qquad\qquad\qquad\qquad\qquad\qquad
\text{$n$-order:}\qquad \tilde{\boldsymbol{z}} = \Bigl( z \Bigr)_{j=-n/2}^{n/2}
\end{equation}
where both sequences are related throught the following functions,
\begin{equation}
\tilde{\boldsymbol{z}} = \text{fftshift}(\boldsymbol{z}) \qquad\qquad \boldsymbol{z} = \text{ifftshift}(\tilde{\boldsymbol{z}})
\end{equation}
The next table represents the structure corresponds in the case where $n$ is an even number
\vspace{5mm}
\begin{center}
\begin{tabular}{c|rrrrrrrr}
$n$ & \multicolumn{4}{c|}{$n/2-1$} & \multicolumn{4}{c}{$n/2+1$}
\\\hline
$j$ & $-\frac{n}{2}+1$ & $-\frac{n}{2}+2$ & $\cdots$ & $-1$ &
$0$ & $\cdots$ & $\frac{n}{2}-1$ & $\frac{n}{2}$
\\
$J$ & $\frac{n}{2}+2$ & $\frac{n}{2}+3$ & $\ldots$ & $n$ &
$1$ & $\ldots$ & $\frac{n}{2}$ &
$\frac{n}{2}+1$
\\\hline
\end{tabular}
\\[5mm]
\begin{tabular}{c|rrrrrrrr}
$n=8$& \multicolumn{3}{c|}{$3$} & \multicolumn{5}{c}{$5$}
\\\hline
$j$ & $-3$ & $-2$ & $-1$ & $0$ & $1$ & $2$ & $3$ & $4$
\\
$J$ & $6$ & $7$ & $8$ & $1$ & $2$ & $3$ & $4$ & $5$
\\\hline
\end{tabular}
\end{center}
If $n$ is odd then general structure of the table above still applies, but $n/2+1$ does not appear.
\begin{center}
\begin{tabular}{c|rrrrrrrr}
$n$ & \multicolumn{4}{c|}{$\lfloor\frac{n}{2}\rfloor$} & \multicolumn{4}{c}{$\lfloor\frac{n}{2}\rfloor+1$}
\\\hline
$j$ & $-\lfloor\frac{n}{2}\rfloor$ & $-\lfloor\frac{n}{2}\rfloor+1$ & $\cdots$ & $-1$ &
$0$ & $\cdots$ & $\lfloor\frac{n}{2}\rfloor-1$ &
$\lfloor\frac{n}{2}\rfloor$
\\
$J$ & $\lfloor\frac{n}{2}\rfloor+1$ & $\lfloor\frac{n}{2}\rfloor+2$ & $\ldots$ & $n$ &
$1$ & $\ldots$ & $\lfloor\frac{n}{2}\rfloor-1$ &
$\lfloor\frac{n}{2}\rfloor$
\\\hline
\end{tabular}
\\[5mm]
\begin{tabular}{c|rrrrrrr}
$n=7$& \multicolumn{3}{c|}{$3$} & \multicolumn{4}{c}{$4$}
\\\hline
$j$ & $-3$ & $-2$ & $-1$ & $0$ & $1$ & $2$ & $3$
\\
$J$ & $5$ & $6$ & $7$ & $1$ & $2$ & $3$ & $4$
\\\hline
\end{tabular}
\end{center}
After this transformation Equation (\ref{eq: CFT raw}) can be reexpressed as follow
\begin{equation}
F_K = \frac{\Delta x}{\sqrt{2\pi}}\sum_{J=1}^{n} f_J\,\, e^{-2\pi i (K-1)(J-1)/n}
\qquad\therefore\qquad
K=1,2,\ldots,n
\label{eq: CFT end}
\end{equation}
The reordering of spatial samples means that the frequency samples should be re-arranged as well.
Now it is clear that we can use the DFT definition to express the FT
(Denote K the new index in the frequency domain, which leads to the following expression:)
\begin{equation}
F_K = \frac{\Delta x}{\sqrt{2\pi}}D_K( \boldsymbol{f} )
\qquad\therefore\qquad
K=1,2,\ldots,n
\end{equation}

\clearpage
To be specific, the abscissas for the input data are,
\begin{equation}
x_j = \left(j-1-\frac{n}{2}\right)\Delta x \qquad\therefore\qquad j=1,2,\ldots,n
\end{equation}
The abscissas for the output data set will be defined as,
\begin{equation}
p_k = \left(k-1-\frac{n}{2}\right)\Delta p \qquad\therefore\qquad k=1,2,\ldots,n
\end{equation}
where $\Delta p = 2\pi/a = 2\pi/n\Delta x$ which conduce to the incertidumbre principle $\Delta x \Delta p = 2\pi/n$ and the important known as
``angular Nyquist frequency'' $\Omega=\pi/\Delta x$, which corresponds to the maximum and the negative mimimum value of frequency for the series.

% http://www.mvkonnik.info/2014/06/fft-ifft-and-why-do-we-need-fftshift.html
% Usually most of implementations uses positive indices, and therefore the spatial samples inside the sum must be rearranged:

\section{Convention}

To facilitate our discusion, we first define two general functions $f(x)$ and $F(p)$, which are
Fourier transforms of each other,
\begin{equation}
\begin{gathered}
\psi(x) = \frac{1}{\sqrt{2\pi\hbar}}\int_{-\infty}^{\infty} \psi(p)\,\, e^{ipx/\hbar}\,dp
\\
\psi(p) = \frac{1}{\sqrt{2\pi\hbar}}\int_{-\infty}^{\infty} \psi(x)\,\, e^{-ipx/\hbar}\,dx
\end{gathered}
\end{equation}
In general we are going to use atomic units ($\hbar=1$).

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|}
\hline
\texttt{FFT}($\mathbf{f}$, $\tilde{\mathbf{F}}^\pm\rightarrow\mathbf{f}$, $\text{sign}=+1$) &
${\displaystyle
\tilde{F}_\alpha^\pm =
\frac{1}{\sqrt{2\pi}}\sum_{\beta=1}^{n} f_\beta\,\exp\left\{\pm 2\pi i\,\,\frac{(\alpha-1)(\beta-1)}{n}\right\}, \qquad \alpha = 1, 2, \ldots, n
}$
\\\hline
\texttt{iFFT}($\mathbf{\tilde{F}}$, $\mathbf{f}^\pm\rightarrow\mathbf{\tilde{F}}$, $\text{sign}=-1$) &
${\displaystyle
f_\beta =
\frac{1}{n\sqrt{2\pi}}\sum_{\alpha=1}^{n} \tilde{F}_\alpha^\pm\,\exp\left\{\mp 2\pi i\,\,\frac{(\beta-1)(\alpha-1)}{n}\right\}, \qquad \beta = 1, 2, \ldots, n
}$
\\\hline
\end{tabular}
\caption{\footnotesize{
asdasd
}}
\label{table: fft transforms}
\end{table}

% \translation{
\section{Fast Fourier transform (FFT)}

Let $\mathbf{f}=\{f_1,f_2,\ldots,f_n\}$ and $\mathbf{F}=\{F_1,F_2,\ldots,F_n\}$ two arrays of complex numbers.
The direct Discrete Fourier Transform (dDFT) of the array $\mathbf{f}$ is defined by the formula
\begin{equation}
F_\alpha^\pm = \sum_{\beta=1}^{n} f_\beta\,\exp\left\{\pm 2\pi i\,\,\frac{(\alpha-1)(\beta-1)}{n}\right\}, \qquad \alpha = 1, 2, \ldots, n
\end{equation}
The inverse DFT (iDFT) for the array $\mathbf{F}$ is defined as
\begin{equation}
f_\beta = \frac{1}{n}\sum_{\alpha=1}^{n} F_\alpha^\pm\,\exp\left\{\mp 2\pi i\,\,\frac{(\beta-1)(\alpha-1)}{n}\right\}, \qquad \beta = 1, 2, \ldots, n
\end{equation}
It differs from the direct transform by the normalization constant $1/n$.

The forward DFT (fDFT) corresponds to a sign of $+1$ in the exponent and backward DFT (bDFT) to a sign $-1$. Traditionally
the dDFT is associated with the sign $+$ and a the iDFT with the sign $-1$, in such case dDFT is equivalent to fDFT and
iDFT is equivalent to bDFT. However, here we will keep this difference.

According with the FFTW library we use the standard order (\textit{s-order}) output. This means that the
positive frequencies are stored in the first half of the output and the negative frequencies
are stored in backwards order in the second half of the output, with the zero-frequency component
in the first element. We will represent this format as a \textit{tilde}, for example: $\mathbf{x}$
represent the array in \textit{n-order} and $\mathbf{\tilde{x}}$ in \textit{s-order}.

There are three different ways to use the available methods available in scift
to calculate the FFT.
\begin{equation}
\left[\mathbf{x},\mathbf{f}\right]
\mathrel{\substack{ \xrightarrow{\text{dFFT}(\pm)} \\ \xleftarrow{\text{iFFT}(\mp)} \\ . }}
\left[\boldsymbol{\tilde{\omega}},\mathbf{\tilde{F}}^\pm\right]
\mathrel{\substack{ \xrightarrow{\text{shift}} \\ \xleftarrow{\text{ishift}} \\ . }}
\left[\boldsymbol{\omega},\mathbf{F}^\pm\right]
\mathrel{\substack{ \xrightarrow{\text{dFFT}(\pm)} \\ \xleftarrow{\text{iFFT}(\mp)} \\ . }}
\left[\boldsymbol{\tilde{x}},\mathbf{\tilde{f}}^\pm\right]
\mathrel{\substack{ \xrightarrow{\text{shift}} \\ \xleftarrow{\text{ishift}} \\ . }}
\left[\boldsymbol{x},\mathbf{f}^\pm\right]
\end{equation}

\begin{equation}
\left[\mathbf{x},\mathbf{f}\right]
\longrightarrow
\left[\boldsymbol{\tilde{\omega}},\mathbf{\tilde{F}}\right]
\longrightarrow
\left[\boldsymbol{\omega},\mathbf{F}\right]
\end{equation}
When $n$ is even the location $n/2$ contains the most positive and negative frequencies
$\pm \frac{n/2}{nh}$ $\left( = \pm \frac{1}{2h} \right)$ which are equivalent.
This frequency is called Nyquist frequency. This is the highest frequency component that
should exist in the input series for the DFT to yield "uncorrupted" results. More
specifically if there are no frequencies above Nyquist the original signal can be exactly
reconstructed from the samples.
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\multicolumn{5}{c|}{$n/2+1$} & \multicolumn{4}{c}{$n/2-1$} \\
\hline
$\tilde{\omega}_1$ & $\tilde{\omega}_2$ & $\tilde{\omega}_3$ & $\ldots$ & $\tilde{\omega}_{n/2+1}$ & $\tilde{\omega}_{n/2-1}$ & $\ldots$ &
$\tilde{\omega}_{n-1}$ & $\tilde{\omega}_n$ \\
\hline\hline
$0$ & $\frac{1}{nh}$ & $\frac{2}{nh}$ & $\ldots$ & $\frac{n/2}{nh}$ & -$\frac{n/2-1}{nh}$ & $\ldots$ & $-\frac{2}{nh}$ & $-\frac{1}{nh}$ \\
\hline
\end{tabular}
\\[6mm]
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\multicolumn{4}{c|}{$n/2-1$} & \multicolumn{5}{c}{$n/2+1$} \\
\hline
$\omega_1$ & $\ldots$ & $\omega_{n/2}$ & $\omega_{n/2+1}$ & $\omega_{n/2}$ & $\omega_{n/2+1}$ & $\omega_{n/2+2}$ & $\ldots$ & $\omega_n$ \\
\hline\hline
$\tilde{\omega}_{n/2+1}$ & $\ldots$ & $\tilde{\omega}_{n-1}$ & $\tilde{\omega}_{n}$ & $\tilde{\omega}_{1}$ & $\tilde{\omega}_{2}$ & $\tilde{\omega}_3$ &
$\ldots$ & $\tilde{\omega}_{n/2}$ \\
\hline
-$\frac{n/2-1}{nh}$ & $\ldots$ & $-\frac{2}{nh}$ & $-\frac{1}{nh}$ & $0$ & $\frac{1}{nh}$ & $\frac{2}{nh}$ & $\ldots$ & $\frac{n/2}{nh}$ \\
\hline
\end{tabular}
\end{center}
\vspace{5mm}
If $n$ is odd then general structure of the table above still applies, but $n/2+1$ does not appear.
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\multicolumn{5}{c|}{$n/2$} & \multicolumn{4}{c}{$n/2-1$} \\
\hline
$\tilde{\omega}_1$ & $\tilde{\omega}_2$ & $\tilde{\omega}_3$ & $\ldots$ & $\tilde{\omega}_{n/2}$ & $\tilde{\omega}_{n/2+1}$ & $\ldots$ & $\tilde{\omega}_{n-1}$
& $\tilde{\omega}_n$ \\
\hline\hline
$0$ & $\frac{1}{nh}$ & $\frac{2}{nh}$ & $\ldots$ & $\frac{n/2-1}{nh}$ & -$\frac{n/2-1}{nh}$ & $\ldots$ & $-\frac{2}{nh}$ & $-\frac{1}{nh}$ \\
\hline
\end{tabular}
\\[6mm]
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\multicolumn{4}{c|}{$n/2-1$} & \multicolumn{5}{c}{$n/2$} \\
\hline
$\omega_1$ & $\ldots$ & $\omega_{n/2}$ & $\omega_{n/2+1}$ & $\omega_{n/2}$ & $\omega_{n/2+1}$ & $\omega_{n/2+2}$ & $\ldots$ & $\omega_n$ \\
\hline\hline
$\tilde{\omega}_{n/2+1}$ & $\ldots$ & $\tilde{\omega}_{n-1}$ & $\tilde{\omega}_{n}$ & $\tilde{\omega}_{1}$ & $\tilde{\omega}_{2}$ & $\tilde{\omega}_3$ &
$\ldots$ & $\tilde{\omega}_{n/2}$ \\
\hline
-$\frac{n/2-1}{nh}$ & $\ldots$ & $-\frac{2}{nh}$ & $-\frac{1}{nh}$ & $0$ & $\frac{1}{nh}$ & $\frac{2}{nh}$ & $\ldots$ & $\frac{n/2-1}{nh}$ \\
\hline
\end{tabular}
\end{center}
\vspace{5mm}

% Given a window length n and a sample spacing d:
% 	!! @returns:	f : ndarray. Array of length n containing the sample frequencies.
% 	!! f = [0, 1, ...,   n/2-1,     -n/2, ..., -1] / (d*n)   if n is even
% 	!! f = [0, 1, ..., (n-1)/2, -(n-1)/2, ..., -1] / (d*n)   if n is odd

\subsection{Directly (High level interface)}
By using this option, scift internally make the shift and normalization for the $x$ coordinate.

\begin{equation}
\Bigl[x,f(x)\Bigr]
\stackrel{\text{5}}{\longrightarrow}
\Bigl[\omega,\hat{f}(\omega)\Bigr]
\end{equation}

\begin{equation}
\Bigl[\omega,\hat{f}(\omega)\Bigr]
\stackrel{\text{8}}{\longrightarrow}
\Bigl[x,f(x)\Bigr]
\end{equation}

\lstset{language=Fortran}
\begin{lstlisting}
call xGrid.init( -3.0_8*Math_PI, 3.0_8*Math_PI, 1000 )
call funcA.init( xGrid, funcTestWithNoise )
call funcA.save("func")

funcA = FFT_fft( funcA )
call funcA.save("Ffunc")

funcA = FFT_ifft( funcA )
call funcA.save("iFFfunc")
\end{lstlisting}

This is the result

\begin{figure}[ht]
\centering
\includegraphics[scale=0.36]{images/FFT-func.eps}
\includegraphics[scale=0.36]{images/FFT-Ffunc.eps}
\caption{\footnotesize{
Geometria para s=3
}}
% \label{}
\end{figure}

\subsection{By using plans (Low level interface)}
In this case is neccesary to keep in mind the reorganization on the frequency axis and the normalization
of the Fourier transform

\begin{equation}
\Bigl[x,f(x)\Bigr]
\stackrel{\text{8}}{\longrightarrow}
\Bigl[x,\hat{f}(\tilde{\omega})\Bigr]
\stackrel{\text{9}}{\longrightarrow}
\Bigl[\tilde{\omega},\hat{f}(\tilde{\omega})\Bigr]
\stackrel{\text{10}}{\longrightarrow}
\Bigl[\omega,\hat{f}(\omega)\Bigr]
\end{equation}

\begin{equation}
\Bigl[\omega,\hat{f}(\omega)\Bigr]
\stackrel{\text{13}}{\longrightarrow}
\Bigl[\tilde{\omega},\hat{f}(\tilde{\omega})\Bigr]
\stackrel{\text{14}}{\longrightarrow}
\Bigl[\tilde{\omega},F(x)\Bigr]
\stackrel{\text{15}}{\longrightarrow}
\Bigl[x,F(x)\Bigr]
\stackrel{\text{16}}{\longrightarrow}
\Bigl[x,f(x)\Bigr]
\end{equation}

\lstset{language=Fortran}
\begin{lstlisting}
call xGrid.init( -3.0_8*Math_PI, 3.0_8*Math_PI, 1000 )
call funcA.init( xGrid, funcTestWithNoise )
call funcA.save("func")

planF = FFT_plan( funcA, FFT_FORWARD )
planB = FFT_plan( funcA, FFT_BACKWARD )

call FFT_execute( planF )
funcA.xGrid = FFT_omegaGrid( funcA.xGrid )
call FFT_shift( funcA )
call funcA.save("Ffunc")

call FFT_ishift( funcA )
call FFT_execute( planB )
funcA.xGrid = FFT_xGrid( funcA.xGrid )
funcA = funcA/real( funcA.nPoints(), 8 )
call funcA.save("iFFfunc")

call FFT_destroyPlan( planF )
\end{lstlisting}

\subsection{By using the oriented object interface}
The function
\begin{equation}
\begin{aligned}
&
\xlongleftrightarrow{\text{sync=F}\qquad\quad}
\quad\Bigl[x,\hat{f}(\tilde{\omega})\Bigr]
\\
\Bigl[x,f(x)\Bigr]\quad
&
\xlongleftrightarrow{\text{sync=T}\qquad\quad}
\quad\Bigl[\tilde{\omega},\hat{f}(\tilde{\omega})\Bigr]
\\
&
\xlongleftrightarrow{\text{sync=T,shift=T}}
\quad\Bigl[\omega,\hat{f}(\omega)\Bigr]
\end{aligned}
\end{equation}

\lstset{language=Fortran}
\begin{lstlisting}
call xGrid.init( -3.0_8*Math_PI, 3.0_8*Math_PI, 1000 )
call funcA.init( xGrid, funcTestWithNoise )
call funcA.save("func")

call fft.init( funcA, FFT_SPATIAL_DOMAIN )

call fft.execute( FFT_FORWARD, sync=.true., shift=.true. )
call funcA.save("Ffunc")

call fft.execute( FFT_BACKWARD, sync=.true., shift=.true. )
call funcA.save("iFFfunc")
\end{lstlisting}

\subsection{Calculating the derivative of second order of a signal}
The function
\begin{equation}
\frac{d^n}{dx^n}f(x) = (i\omega)^n\hat{f}(\omega)
\end{equation}

\begin{equation}
\Bigl[x,f(x)\Bigr]
\stackrel{\text{7}}{\longrightarrow}
\Bigl[x,\hat{f}(\tilde{\omega})\Bigr]
\xlongrightarrow{\text{9}}
\Bigl[x,(i\tilde{\omega})^2\hat{f}(\tilde{\omega})\Bigr]
\stackrel{\text{11}}{\longrightarrow}
\Bigl[x,\frac{d^2}{dx^2}f(x)\Bigr]
\end{equation}

\lstset{language=Fortran}
\begin{lstlisting}
call xGrid.init( -3.0_8*Math_PI, 3.0_8*Math_PI, 1000 )
call funcA.init( xGrid, funcTest )
call funcA.save("func")

call fft.init( funcA, FFT_SPATIAL_DOMAIN )

call fft.execute( FFT_FORWARD )

funcA.yArray = ( Math_I*fft.omega )**2*funcA.yArray

call fft.execute( FFT_BACKWARD )

call funcA.save("dfunc")

call funcB.init( xGrid, d2funcTest )
call funcB.save("exact")
\end{lstlisting}

This is the result

\begin{figure}[ht]
\centering
\includegraphics[scale=0.36]{images/FFT-func2.eps}
\includegraphics[scale=0.36]{images/FFT-dfunc.eps}
\caption{\footnotesize{
Left panel: Original function. Right panel: Derivative of second order of the signal calculated by using
the FFT interface, the exact solution is also included.
}}
% \label{}
\end{figure}


\subsection{Euler Characteristic}
\begin{definition}
    Let $S$ be a compact Riemann surface.
    A topological triangle in $S$ is a continuous embedding $\Delta\hookrightarrow S$ where $\Delta$ is a closed (non-degenerate) triangle in the plane $\mathbb R^2$.
    A triangulation of $S$ is a finite collection of topological triangles $\{\Delta_i\}$ on $S$ such that:\\
    1. $\bigcup_i\Delta_i=S$.\\
    2. If $i\neq j$ then $\Delta_i\cap\Delta_j$ is either empty, a common vertex, or a common edge.\\
    3. Each edge is contained in exactly two triangles.
\end{definition}
\begin{definition}
    The Euler characteristic of a triangulation of $S$ is $\chi=V-E+F$, where $V$ is the number of vertices, $E$ is the number of edges and $F$ is the number of triangles.
\end{definition}
\begin{lemma}
    1. Every compact Riemann surface $S$ has a triangulation.\\
    2. $\chi$ does not depend on the triangulation we choose.
\end{lemma}
\begin{proof}
    Omitted.
\end{proof}
\begin{definition}
    The Euler characteristic $\chi(S)$ of $S$ is the Euler characteristic of any of its triangulations.
\end{definition}
\begin{example}
    $\chi(\mathbb C_\infty)=4-6+4=2$ by identifying it with a regular tetrahedron.
    $\chi(\mathbb C/\Lambda)=0$ by attempting to triangulate its representation as a quotient space of $[0,1]^2$.
\end{example}
Turns out, every compact Riemann surface is homeomorphic to an $g$-torus $\Sigma_g$ for some $g$.
Moreover, $\chi(\Sigma_g)=2-2g$.
Therefore $\chi(S)$ determines a compact Riemann surface $S$ up to homeomorphism.