\chapter{Numerical Results and Discussion}\label{sec:results}

After various numerical models and methods for biophysical simulations of the neuromuscular system were described in the previous chapters, the remainder of this work deals with their application and discusses the newly obtained insights.
The current chapter presents numerical results and demonstrates the use of OpenDiHu for all major components of the multi-scale models.

\Cref{sec:poisson_diffusion} begins with the simulation of toy problems such as Poisson and diffusion equations, which are used as building blocks for the more advanced simulations. Subsequently, dedicated solvers for the solid mechanics problem, the CellML models, the fiber based electrophysiology and the multidomain model are presented in \cref{sec:solver_solid_mechanics,sec:results_cellml_models,sec:results_fiber_based_electrophysiology,sec:solver_multidomain_model}. Finally, \cref{sec:coupled_electrophysiology_and_solid_mechanics} combines the fiber based electrophysiology model and the multidomain model with the solid mechanics solver to yield a comprehensive multi-physics simulation of muscle contraction.

% ==================
%
% =-------------------

\section{Solution of Poisson and Diffusion Problems}\label{sec:poisson_diffusion}

Setting up a composite multi-scale simulation, where multiple equations are coupled, requires a profound understanding of the model components.
Thus, it can help to first simulate isolated models. We provide simple examples with our software, such as Laplace and Diffusion problems, as prototypes for elliptic and parabolic partial differential equations. The examples with analytic solutions are also used to validate the basic finite element solvers.

In this section, we showcase three of these simple problems. First, we consider the 1D Poisson problem $u''(x) = f$ on $\Omega=[0,1]$ with Dirichlet boundary conditions $u(0)=0$ and $u(1)=1$ and right-hand side $f(x)=6\,x$. The analytic solution is $y(x)=x^3$. \Cref{fig:poisson} shows the analytic solution and the results of the finite element computation with linear and quadratic ansatz functions for two elements. Both linear and quadratic finite element solutions cannot exactly represent the cubic function, however, yield the best possible approximation.
The first bidomain equation given in \cref{eq:bidomain1} is a 3D version of this Poisson problem and is needed in the multi-domain model to simulate EMG signals on the muscle surface.

The second example is a 2D Laplace problem $c(x)\,\Delta u(x) = 0$. The solution is given in \cref{fig:laplace_composite_1} and can be interpreted as a static electric potential field.
The discretization uses quadratic Lagrange ansatz functions and is composed of two joined rectangular parts, each given by a structured mesh. The conductivity is set as $c=1$ in the left part and as $c=2$ in the right part. Dirichlet boundary conditions prescribe the electric potential at the five upper points in the right mesh to $u = -1$ and at the center of the right mesh as $u=1$. In addition, Neumann boundary conditions $\partial u / \partial \bfn = -1$ corresponding to an outward electric current are set on the left boundary of the left mesh with the normal vector $\bfn$ pointing to the left. \Cref{fig:laplace_composite_1} visualizes the values of the degrees of freedom of the right-hand side contribution of the Neumann boundary conditions by the arrows.
A 3D Laplace problem is also part of the multi-scale model and describes volume conduction in the adipose tissue domain as formulated in \cref{eq:body}.

% poisson - laplace
\begin{figure}
  \centering%
  \begin{subfigure}[t]{0.38\textwidth}%
    \centering%
    \includegraphics[width=\textwidth]{images/results/basic/analytic.pdf}%
    \caption{Solution of a 1D Poisson problem for linear and quadratic ansatz functions.}%
    \label{fig:poisson}%
  \end{subfigure}\quad
  \begin{subfigure}[t]{0.58\textwidth}%
    \centering%
    \includegraphics[width=\textwidth]{images/results/basic/laplace_composite_1.png}%
    \caption{Finite element mesh and solution of a 2D electric conduction problem .}%
    \label{fig:laplace_composite_1}%
  \end{subfigure}
  \caption{Exemplary problems that can be solved with OpenDiHu and are part of the multi-scale problem.}%
  \label{fig:poisson_laplace_composite_1}%
\end{figure}%

The third presented example solves the 2D diffusion equation $\partial u/\partial t - \div(\bfsigma\,\grad u) = 0$ with homogeneous Neumann boundary conditions. The equation can be interpreted as a transient electric conduction problem. As shown in \cref{fig:diffusion1}, the initial charge distribution is $u=1$ in a rectangle in the inner of the domain and $u=0$ everywhere else. The anisotropic diffusion or conductivity tensor $\bfsigma$ is constant in the domain and set to %
\begin{align*}
  \bfsigma = \frac15\,\mat{1 & 1\\
              1 & 6}.
\end{align*}
A regular mesh with $40\times 40$ elements and linear finite element ansatz functions is used. \Cref{fig:diffusion2} shows the solution at time $t=5$, where the initially discontinuous charge distribution has smoothed out and has expanded mainly in $y$ direction, which is the preferential direction of electric conduction in this example. A 3D version of this equation is part of the multidomain model and given by \cref{eq:multidomain1}.

% diffusion
\begin{figure}
  \centering%
  \begin{subfigure}[t]{0.4\textwidth}%
    \centering%
    \includegraphics[width=\textwidth]{images/results/basic/diffusion1.pdf}%
    \caption{Initial charge distribution, $t=0$.}%
    \label{fig:diffusion1}%
  \end{subfigure}\quad
  \begin{subfigure}[t]{0.4\textwidth}%
    \centering%
    \includegraphics[width=\textwidth]{images/results/basic/diffusion2.pdf}%
    \caption{Solution at $t=5$.}%
    \label{fig:diffusion2}%
  \end{subfigure}
  \caption{A 2D electric conduction problem as a demonstrator for the solution of transient problems in OpenDiHu.}%
  \label{fig:diffusion}%
\end{figure}%

\begin{reproduce_no_break}
  The three presented simulations can be executed and visualized as follows:
  \begin{lstlisting}[columns=fullflexible,breaklines=true,postbreak=\mbox{\textcolor{gray}{$\hookrightarrow$}\space},language=python]
    cd $\$$OPENDIHU_HOME/examples/poisson/poisson1d_2/build_release
    ./linear ../settings_1d.py && plot out/*.py
    ./quadratic ../settings_1d.py && plot out/*.py
    ./hermite ../settings_1d.py && plot out/*.py

    cd $\$$OPENDIHU_HOME/examples/laplace/laplace_composite/build_release/
    ./laplace_composite_2d ../settings_2d_2.py && paraview paraview_state.pvsm

    cd $\$$OPENDIHU_HOME/examples/diffusion/anisotropic_diffusion/build_release
    ./anisotropic_diffusion2d ../settings2d.py && plot out/*.py
  \end{lstlisting}
\end{reproduce_no_break}

%-----
\section{Simulation of Solid Mechanics Models}\label{sec:solver_solid_mechanics}

Next, we demonstrate the solid mechanics solvers, which can be used to compute muscle contraction. In this section, we focus on the passive material behavior.
As described in \cref{sec:material_linear_model,sec:material_modeling}, the mechanics equations can be computed in linearized or in nonlinear form within OpenDiHu.
In the following, \cref{sec:comparison_linear_nonlinear} applies both model approaches in a simulation of an externally stretched muscle and compares the results. Then, \cref{sec:validation_nonlinear} validates the implementation of the nonlinear hyperelasticity solver in OpenDiHu. Finally, \cref{sec:simulation_hyperelastic_tendon} showcases, using the simulation of a tendon, how more complex material models can be computed.

%-----
\subsection{Comparison of Linear and Nonlinear Mechanics Models}\label{sec:comparison_linear_nonlinear}

We demonstrate the use of linear and nonlinear mechanics models in a simulation of an externally stretched biceps muscle. The muscle belly is fixed at its lower end and an upwards pulling force acts on the upper end, effectively stretching the muscle tissue in vertical direction.

We solve two scenarios with the same geometry and boundary conditions but different material models. The first scenario uses the linearized mechanics model given in \cref{sec:material_linear_model}. We use material parameters obtained from porcine in vitro indentor tests in literature \cite{schock1982vivo} and set the bulk modulus to $K=\SI{39}{\kilo\pascal}$ and the shear modulus to $\mu=\SI{48}{\kilo\pascal}$.

The second scenario uses the incompressible transversely isotropic hyperelastic muscle material based on the Mooney-Rivlin description without active stress, which is defined in \cref{sec:material_nonlinear_model}. The material parameters are set to the values given \cite{Heidlauf2016}.

\Cref{fig:lin_nonlin_muscle_mechanics} shows the geometric setup of the model. We discretize the biceps geometry by a 3D mesh with 252  elements, quadratic finite element ansatz functions,  and a total of $13 \times 13 \times 15 = 2535$ nodes.
The linearized material model uses this mesh to construct the stiffness matrix and to solve the linear system.
For the nonlinear model, an additional coarser linear mesh is constructed, and linear-quadratic Taylor-Hood elements are used for the discretization.

A total force of $(F_x,F_y,F_z) = (0,\SI{-0.4}{\newton},\SI{-3}{\newton})$ is applied, which points in negative $z$-direction, i.e., upwards in \cref{fig:lin_nonlin_muscle_mechanics}, and slightly in negative $y$-direction, i.e, to the left in \cref{fig:lin_nonlin_muscle_mechanics}.
Instead of a single force vector acting on a point, the equivalent constant surface load is applied on the whole top face of the muscle geometry.

We consider a static problem where no timestepping is required. In the linear model, the resulting displacements are obtained by a GMRES solver, which solves the linear system of equations \cref{eq:linearized_helper4} corresponding  to the finite element formulation.
The nonlinear model uses increasing load steps as described in \cref{sec:convergence_improvements_for_the_nonlinear_solver}, which are adaptively refined in case the solver diverges at one load step. The scheme solves a system of nonlinear equations for every load step, and the contained linear system is solved by a direct solver.

% linear and nonlinear mechanics solvers
\begin{figure}
  \centering%
  \hfill
  \begin{subfigure}[t]{0.4\textwidth}%
    \centering%
    \includegraphics[height=12cm]{images/results/basic/lin_nonlin_muscle_mechanics_b.png}%
    \caption{Solution of the linear model. The arrows at the top visualize the (negated) right-hand side of the finite element formulation, with the absolute values indicated by the arrow lengths and color. The surface of the muscle mesh is colored according to the values of the displacements.}%
    \label{fig:lin_nonlin_muscle_mechanics_b}%
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.4\textwidth}%
    \centering%
    \includegraphics[height=12cm]{images/results/basic/lin_nonlin_muscle_mechanics_a.png}%
    \caption{Solution of the nonlinear model. The arrows specify the traction vectors in current configuration.  Their absolute values are indicated by the arrow sizes and the color. The surface of the muscle mesh is colored according to the second Piola-Kirchhoff stress.}%
    \label{fig:lin_nonlin_muscle_mechanics_a}%
  \end{subfigure}
  \hfill
  \caption{Solid mechanics solver example: Comparison of linear and nonlinear mechanics models. A biceps muscle is stretched by an applied force. The yellow mesh specifies the identical reference configuration in both scenarios.}%
  \label{fig:lin_nonlin_muscle_mechanics}%
\end{figure}%

The results of the linear and nonlinear models are shown in \cref{fig:lin_nonlin_muscle_mechanics_b,fig:lin_nonlin_muscle_mechanics_a}.
In both images, the identical reference configuration is given by the yellow wireframe and the deformed muscle is given by the solid body with colored mesh.
The deformed body in the linear model in \cref{fig:lin_nonlin_muscle_mechanics_b} is colored according to the resulting vector of unknowns, which contains the displacements.  Arrows on the upper end of the geometry indicate the negative right-hand side of the linear system as formulated in \cref{eq:linearized_mechanics_rhs}. The arrows correspond to the applied Neumann boundary conditions in the weak form of the finite element formulation and point in the direction of the applied surface load.

In the visualization of the nonlinear model in \cref{fig:lin_nonlin_muscle_mechanics_a}, the deformed muscle body is colored according to the second Piola-Kirchhoff (PK2) stress. It can be seen that the stress is highest at the bottom bearing and at the top end, where the muscle cross-section is smaller. The arrows visualize the traction forces $\bft$ on virtual horizontal cuts. As a result, the arrows that can be seen on top of the muscle geometry correspond to the applied external force, and the arrows at the bottom indicate the forces on the bearing.

A comparison of the two obtained results from the linear and nonlinear models shows a qualitatively different outcome. With the linear model, the muscle bends to the left, whereas, with the nonlinear model, it bends to the right. This effect is a result of the different material behavior. The linear model is isotropic and the deformation follows the direction of the applied force, which points to the upper left. The nonlinear model has an anisotropy and is stiffer in fiber direction. As a consequence, the muscle deforms less in longitudinal direction and therefore moves to the right.
Thus, the material models influence the bending direction in this scenario.

In a second example, we compare the muscle stretches that results from different external forces acting in $z$-direction.  We use the same scenario as before and increase the applied force from 0 to \SI{15}{\newton}. We measure the displacement of one node in the top face of the muscle, for both the linear model and the nonlinear model. While the stress-strain relations  in 1D extension tests  can be derived analytically for linear and nonlinear models, our examples considers a real 3D setting where this relation is influenced by the geometry, e.g., by non-parallel fiber directions, as the force is not applied exactly in fiber direction.

\Cref{fig:linear_nonlinear_displacements} shows the resulting muscle extensions for different applied external forces for the linear and nonlinear models.
It can be seen that the stretch of the muscle increases nonlinearly for the transversely isotropic hyperelastic model, in contrast to the linear progression of the linear model.
The slopes of the two curves are qualitatively different, which is a result of the chosen material parameters from different experimental origins. It would be possible to scale the linear model to better match the nonlinear model behavior by simply reducing the value of the bulk modulus accordingly.

% study of displacements, linear  nonlinnear
\begin{figure}
  \centering%
  \includegraphics[width=0.7\textwidth]{images/results/basic/linear_nonlinear_displacements.pdf}%
  \caption{Solid mechanics example: Quantitative comparison of the relation between applied force and extension of the muscle for a linear and a nonlinear solid mechanics model.}%
  \label{fig:linear_nonlinear_displacements}%
\end{figure}

The two presented studies show that a linear isotropic material model can give significantly different results than a more accurate nonlinear transversely isotropic model. Therefore, simulations of muscle contraction that target high accuracy should use the according nonlinear models. Nevertheless, both approaches are implemented and can be used with OpenDiHu.

\begin{reproduce_no_break}
  The two simulations for \cref{fig:lin_nonlin_muscle_mechanics} can be run as follows:
  \begin{lstlisting}[columns=fullflexible,breaklines=true,postbreak=\mbox{\textcolor{gray}{$\hookrightarrow$}\space}]
    cd $\$$OPENDIHU_HOME/examples/solid_mechanics/linear_elasticity/muscle/build_release
    ./linear_elasticity ../settings_linear_elasticity.py
    cd $\$$OPENDIHU_HOME/examples/solid_mechanics/mooney_rivlin_transiso/build_release
    ./3d_hyperelasticity ../settings_3d_muscle.py --njacobi=1
  \end{lstlisting}
  The study in \cref{fig:linear_nonlinear_displacements} can be run and plotted using the scripts in the repository at \href{https://github.com/dihu-stuttgart/performance}{github.com/dihu-stuttgart/performance}
  in the directory \code{opendihu/}\\\code{23_linear_nonlinear_mechanics}.
\end{reproduce_no_break}

%-----
\subsection{Validation of the Nonlinear Solid Mechanics Solver}\label{sec:validation_nonlinear}

Next, we perform tests to validate our implementation of the nonlinear hyperelasticity solvers.
We simulate the same scenario with our software and with the nonlinear finite element analysis tool \emph{FEBio} \cite{Maas2012}.
FEBio is developed at the University of Utah and the Columbia University in the USA. FEBio contains solid mechanics solvers that can be run from the command line or a graphical user interface model. An extensive model library contains material models also from the domain of biomechanics. The mechanics solver uses the PARDISO linear solver \cite{pardiso2020}, which exploits shared memory parallelism.

An adapter in OpenDiHu exists, which can output the required configuration file for FEBio, run the solver, and parse the computed solution from the text files that are output by FEBio. Thus, we can conduct our validation studies fully in OpenDiHu by using similar Python settings files and the same meshes for the computation in OpenDiHu and the reference solution computed by FEBio.

Apart from the present study, the FEBio adapter in OpenDiHu can also be used to solve quasi-static coupled problems with the electrophysiology part solved in OpenDiHu and the mechanics part solved in FEBio. However, test have shown that the interfacing method of generating configuration files and parsing result files in every timestep leads to higher runtimes than directly using the mechanics solver of OpenDiHu.

In our validation studies, we consider a unit cube  that is discretized by $8\times 8 \times 8$ quadratic elements and 4913 degrees of freedom. \Cref{fig:tensile_shear_test_img} shows the discretized cube in yellow color. Its orientation is given by the coordinate frame in the lower left of \cref{fig:tensile_test_img}.
The following Dirichlet boundary conditions are prescribed: All points of the lower face are fixed at $z=0$. The points of the two edges ($y=0 \wedge z=0$) and ($x=0 \wedge z=0$) are additionally fixed in $y$ and $x$ directions, respectively. The corner at $x=y=z=0$ is fixed completely. Thus, the cube can freely deform in its bottom plane, but not move nor rotate as a whole.

% visualization of the scenarios
\begin{figure}
  \centering%
  \hfill
  \begin{subfigure}[t]{0.45\textwidth}%
    \centering%
    \includegraphics[height=8cm]{images/results/basic/tensile_test_img.png}%
    \caption{Tensile test scenario used in the first validation experiment.}%
    \label{fig:tensile_test_img}%
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.45\textwidth}%
    \centering%
    \includegraphics[height=8cm]{images/results/basic/shear_test_img.png}%
    \caption{Shear test scenario used in the second validation experiment.}%
    \label{fig:shear_test_img}%
  \end{subfigure}
  \hfill
  \caption{Scenarios used for validation of the solid mechanics solver. The reference and the current configuration are given by the yellow and orange meshes, respectively.}%
  \label{fig:tensile_shear_test_img}%
\end{figure}%

The first study is a tensile test, where a uniform surface load pointing in positive $z$ direction is applied on the top face of the cube. We increase the force from 1 to \SI{50}{\newton}. For the largest force, the cube deforms as shown by the orange geometry in \cref{fig:tensile_shear_test_img}. Note that the volume is preserved due to the incompressibility constraint in the material model.

We use an incompressible and isotropic Mooney-Rivlin material with parameters $c_1=c_2=1$. The material can be simulated in three different forms in OpenDiHu. In the following, we list all model formulations in OpenDiHu and the reference formulation in FEBio,
expressed by the strain energy functions $\Psi$, $\Psi_\text{iso}$ and $\Psi_\text{vol}$ introduced in the modeling chapter in \cref{sec:material_modeling}:

\begin{enumerate}[label=(\roman*)]
\item the \say{fully incompressible}, mixed $u$-$p$ formulation, which ensures incompressibility using the Lagrange multipliers,
\begin{align}
    \Psi_\text{iso}(\bar{I}_1,\bar{I}_2) &= c_1\,(\bar{I}_1 - 3) + c_2\,(\bar{I}_2 - 3), \quad J=1, \label{eq:validation_incompressible}
\end{align}
\item the \say{nearly incompressible} formulation in terms of the invariants $I_1$ to $I_3$,
\begin{subequations}\label{eq:validation_1b}
  \begin{align}
      \Psi(I_1,I_2,I_3) &= c_1\,(I_1 - 3) + c_2\,(I_2 - 3) + \kappa\,(\sqrt{I_3} - 1)^2 - d\,\log(\sqrt{I_3}),\label{eq:validation_nearly_incompressible_1} \\
         d &= 2\,(c_1 + 2\,c_2) \label{eq:validation_nearly_incompressible},
\end{align}
\end{subequations}
\item the nearly incompressible formulation given in decoupled form, in terms of the reduced invariants $\bar{I}_1$ and $\bar{I}_2$,
\begin{subequations}\label{eq:validation_1c}
  \begin{align}
      \Psi_\text{iso}(\bar{I}_1,\bar{I}_2) &= c_1\,(\bar{I}_1 - 3) + c_2\,(\bar{I}_2 - 3),\label{eq:validation_nearly_incompressible_decoupled_1} \\
      \Psi_\text{vol}(J) &= \kappa\,G(J) \quad\text{with }  G(J) = \dfrac14\big(J^2 - 1 - 2\,\log(J)\big), \label{eq:validation_nearly_incompressible_decoupled}
  \end{align}
\end{subequations}
\item and the one used in FEBio, which also describes a nearly incompressible material in decoupled form, but with a different penalty function $G(J)$,
\begin{subequations}\label{eq:validation_1d}
  \begin{align}
      \Psi_\text{iso}(\bar{I}_1,\bar{I}_2) &= c_1\,(\bar{I}_1 - 3) + c_2\,(\bar{I}_2 - 3),\label{eq:validation_nearly_incompressible_decoupled_febio_1} \\
      \Psi_\text{vol}(J) &= \kappa\,G(J) \quad\text{with } G(J) = \dfrac12\big(\log(J)\big)^2 \label{eq:validation_nearly_incompressible_decoupled_febio}.
  \end{align}
\end{subequations}
\end{enumerate}
%
For the three nearly incompressible descriptions in \cref{eq:validation_1b,eq:validation_1c,eq:validation_1d}, we set the incompressibility parameter to $\kappa=\num{1e3}$.

We compare the resulting normal stress value $S_{33}$ in $z$-direction of the second Piola-Kirchhoff stress tensor $\bfS$ for all formulations listed in \crefrange{eq:validation_incompressible}{eq:validation_1d}. For the tensile test, this stress value is constant throughout the domain. \Cref{fig:validation_tensile_test} shows the computed stresses over the computed strain values.
It can be seen that the three formulations in OpenDiHu yield approximately the same results as the reference solution given by FEBio over the whole range of applied forces.

% results tensile test
\begin{figure}
  \centering%
  \includegraphics[width=\textwidth]{images/results/basic/validation_tensile_test.pdf}%
  \caption{Solid mechanics solver validation: Results of the tensile test validation experiment. The stress-strain curve for three different formulations in OpenDiHu and the computation in FEBio match.}%
  \label{fig:validation_tensile_test}%
\end{figure}

As the previous tensile test only validates stress and strain in one direction, we additionally conduct a numerical shear experiment. A shear force $\bfF=(0.1\alpha, 0.05\alpha, 0)^\top$ is applied on the top face of the cube and $\alpha$ is again varied between 1 and $\SI{50}{\newton}$. \Cref{fig:shear_test_img} shows the deformed configuration for the highest force by the orange colored body.

% results shear test
\begin{figure}
  \centering%
  \includegraphics[width=0.7\textwidth]{images/results/basic/validation_shear_test.pdf}%
  \caption{Solid mechanics solver validation:Results of the shear test validation experiment. The values of the second Piola-Kirchhoff tensor computed by OpenDiHu (solid lines) and FEBio (dotted lines) closely match.}%
  \label{fig:validation_shear_test}%
\end{figure}

In this second study, we consider one point in the interior of the domain, which is 3 elements below the top face of the mesh. We compare all six distinct entries of the symmetric second Piola-Kirchhoff tensor $\bfS$ between the fully incompressible model in OpenDiHu and the nearly incompressible model in FEBio.

\Cref{fig:validation_shear_test} shows the computed values in a stress-strain diagram. The solutions of OpenDiHu and FEBio are given by solid and dotted lines, respectively. It can be seen that the curves coincide, which validates the implementation in OpenDiHu.

\begin{reproduce_no_break}
  The tensile test validation experiment can be reproduced by the following commands:
  \begin{lstlisting}[columns=fullflexible,breaklines=true,postbreak=\mbox{\textcolor{gray}{$\hookrightarrow$}\space}]
    cd $\$$OPENDIHU_HOME/examples/solid_mechanics/tensile_test/build_release
    ../run_force.sh
    cd $\$$OPENDIHU_HOME/examples/solid_mechanics/tensile_test
    ./plot_validation.py
  \end{lstlisting}
  The shear test can be executed analogously by replacing \code{tensile_test} by \code{shear_test} in the given paths.
\end{reproduce_no_break}

%-----
\subsection{Simulation of a Hyperelastic Tendon Material}\label{sec:simulation_hyperelastic_tendon}

Next, we demonstrate the use of a more complex constitutive material model, which represents tendon tissue. The material is formulated in \cite{Carniel2017}. The model describes microstructural interactions between collagen fibers and their matrix. It consists of a transversely isotropic model, which describes the high stiffness in fiber direction, and a coupled model for the compressive response. The model is formulated in terms of a logarithmic strain measure.

%from the paper: The high stiffness of tendons under tensile tests is handled by a transversely isotropic model while the coupled compressive response is modeled by means of a Fung-type potential in terms of Seth-Hill’s generalized strain tensors. In present study the logarithm strain measure is used instead of the usually employed Green-Lagrange strain

\Cref{fig:tendon_material_simulation} shows the geometries of the tendons of the biceps brachii and the results of the simulations. The lower tendon in \cref{fig:dynamic_mooney_rivlin_6} is fixed at its left end and a constant surface traction of \SI{1}{\newton} in total pulls to the right. The image shows the initial configuration by the wireframe mesh and the current configuration after $t=\SI{10}{\ms}$, colored according to the resulting velocity.
Similarly, the upper tendons in \cref{fig:dynamic_mooney_rivlin_7} are fixed at the right ends and stretched to the left resulting from the applied force at the left end.

\begin{figure}
  \centering%
  \begin{subfigure}[t]{\textwidth}%
    \centering%
    \includegraphics[width=0.9\textwidth]{images/results/basic/dynamic_mooney_rivlin_6.png}%
    \caption{Dynamic simulation of the lower tendon of a biceps brachii. The attachment to the ulna bone is at the left end. The free right end bends due to the applied surface traction.}%
    \label{fig:dynamic_mooney_rivlin_6}%
  \end{subfigure}\\[4mm]
  \begin{subfigure}[t]{\textwidth}%
    \centering%
    \includegraphics[width=0.9\textwidth]{images/results/basic/dynamic_mooney_rivlin_7.png}%
    \caption{Simulation of the two upper tendons of the two biceps heads.}%
    \label{fig:dynamic_mooney_rivlin_7}%
  \end{subfigure}
  \caption{Simulation of tendons as a showcase of dynamic simulations with complex material models. The color coding indicates the velocity.}%
  \label{fig:tendon_material_simulation}%
\end{figure}%

In summary, this section demonstrated the capabilities of the solid mechanics solvers in OpenDiHu. \Cref{sec:comparison_linear_nonlinear,sec:simulation_hyperelastic_tendon} simulated extension of the biceps muscle and tendons due to external forces. The comparison of results from a linear and a nonlinear model showed that a linear isotropic material cannot always accurately predict the behavior of muscle tissue and, thus, a nonlinear model is required. The validation experiments in \cref{sec:validation_nonlinear} demonstrated that OpenDiHu correctly computes deformation and stresses of incompressible materials.

The solid mechanics solvers can also be coupled to solvers of electrophysiology to simulate muscle contraction resulting from the spatially heterogeneous activation and considering the neuronal stimulation dynamics. Moreover, coupled simulations of the muscle and tendons are possible. Such simulations are described in \cref{sec:fiber_based_contraction} and \cref{sec:surface_coupling_contraction}, respectively.

% ------------
%
% f===========


\section{Options, Part 1}

\subsection*{Net option payoff}

The break-even point is given by $S_T$ at which Net Payoff is zero:

$Net payoff = max[S_T-K,0]-C(1+r)^T $

\subsection*{Option strategies}

\begin{itemize}[leftmargin=*]
	\item Protective Put: Buy stock + Buy a put
	\item Bull Call Spread: Buy a Call K1 + Write a Call K2, $K1<K2$
	\item Straddle: Buy a Call at $K$ + Buy a Put at $K$
\end{itemize}

\subsection*{Corporate securities as options}
\begin{itemize}
	\item Equity ($E$): A call option on the firm’s assets ($A$) with the exercise price
	equal to its bond’s redemption value.
	\item Debt ($D$): A portfolio combining the firm’s assets ($A$) and a short position in
	the call with the exercise price equal to its bond’s face value ($F$):

\end{itemize}

$ A = D+E \implies D=A-E$ \\
$ E \equiv max(0,A-F) $ \\
$ D=A-E=A-max[0,A-F]$

\subsection*{Put-Call parity for European options}

$ C + B \cdot K = P + S$

\subsection*{Binominal option pricing model}
Stock

	\begin{tikzpicture}[>=stealth,sloped]
	\matrix (tree) [
	matrix of nodes,
	minimum size=0cm,
	column sep=1cm,
	row sep=0.0cm,nodes={text width=2em}
	]
	{
		&  $S_u=uS_0$ 	 &  \\
		{}	&   		 &  \\
		&  $S_d=dS_0$ 	 &  \\
	};
	\node[bullet,right=0mm of tree-2-1.east,label=above:$S_0$](b-2-1){};
	\node[bullet,left=0mm of tree-1-2.west](b-1-2){};
	\node[bullet,left=0mm of tree-3-2.west](b-3-2){};
	\draw[->] (b-2-1) -- (b-1-2) node [midway,above] {};
	\draw[->] (b-2-1) -- (b-3-2) node [midway,above] {};
	\end{tikzpicture}

Bond

\begin{tikzpicture}[>=stealth,sloped]
	\matrix (tree) [
	matrix of nodes,
	minimum size=0cm,
	column sep=1cm,
	row sep=0.0cm,nodes={text width=2em}
	]
	{
		&  $1+r$ 	 &  \\
		{}	&   		 &  \\
		&  $1+r$ 	 &  \\
	};
	\node[bullet,right=0mm of tree-2-1.east,label=above:$1$](b-2-1){};
	\node[bullet,left=0mm of tree-1-2.west](b-1-2){};
	\node[bullet,left=0mm of tree-3-2.west](b-3-2){};
	\draw[->] (b-2-1) -- (b-1-2) node [midway,above] {};
	\draw[->] (b-2-1) -- (b-3-2) node [midway,above] {};
\end{tikzpicture}

Option at Expiration

\begin{tikzpicture}[>=stealth,sloped]
	\matrix (tree) [
	matrix of nodes,
	minimum size=0cm,
	column sep=1cm,
	row sep=0.0cm,nodes={text width=2em}
	]
	{
		&  $max(S_u-K,0)$ 	 &  \\
		{}	&   	     	 &  \\
		&  $max(S_d-K,0)$ 	 &  \\
	};
	\node[bullet,right=0mm of tree-2-1.east,label=above:$C_0$](b-2-1){};
	\node[bullet,left=0mm of tree-1-2.west](b-1-2){};
	\node[bullet,left=0mm of tree-3-2.west](b-3-2){};
	\draw[->] (b-2-1) -- (b-1-2) node [midway,above] {};
	\draw[->] (b-2-1) -- (b-3-2) node [midway,above] {};
\end{tikzpicture}

Replicating portfolio (call option)
$
\begin{pmatrix}
	S_u & (1+r)  \\
	S_d  & (1+r)  \\

\end{pmatrix} \cdot
\begin{pmatrix}
	a \\
	b \\
\end{pmatrix}
=
\begin{pmatrix}
	max(K-S_u,0)\\
	max(K-S_d,0) \\
\end{pmatrix}
$

solve system to form a portfolio of stock and bond that replicates the call’s payoff:
$a$ shares of the stock;
$b$ dollars in the riskless bond

\subsection*{Binominal option pricing model with multiple periods}

\begin{tikzpicture}[>=stealth,sloped]
	\matrix (tree) [
	matrix of nodes,
	minimum size=0cm,
	column sep=1cm,
	row sep=0.0cm,nodes={text width=2em}
	]
	{
		 &  		 & $C_{uu}$  & \\
		 &  $C_u$    &  		 & \\
	  	 &           & $C_{ud}$	 & \\
     {}  &           &	         & \\
		 &           & $C_{du}$	 & \\
		 &  $C_{d}$  &      	 & \\
		 &           & $C_{dd}$	 & \\
	};
	\node[bullet,right=0mm of tree-4-1.south,label=above:$C_0$](b-4-1){};
	\node[bullet,left=1mm of tree-2-2.south](b-2-2){};
	\node[bullet,left=1mm of tree-6-2.south](b-6-2){};
	\node[bullet,left=1mm of tree-1-3.south](b-1-3){};
	\node[bullet,left=1mm of tree-3-3.south](b-3-3){};
	\node[bullet,left=1mm of tree-5-3.south](b-5-3){};
	\node[bullet,left=1mm of tree-7-3.south](b-7-3){};
	\draw[->] (b-4-1) -- (b-2-2) node [midway,above] {};
	\draw[->] (b-4-1) -- (b-6-2) node [midway,above] {};
	\draw[->] (b-2-2) -- (b-1-3) node [midway,above] {};
	\draw[->] (b-2-2) -- (b-3-3) node [midway,above] {};
	\draw[->] (b-6-2) -- (b-5-3) node [midway,above] {};
	\draw[->] (b-6-2) -- (b-7-3) node [midway,above] {};
\end{tikzpicture}

Compute the time-0 value working backwards: first $C_u$ and $C_d$ then $C_0$.
In summary:
\begin{itemize}
	\item Replication strategy gives payoffs identical to those of the call.
	\item Initial cost of the replication strategy must equal the call price
\end{itemize}


\lab{GMRES}{GMRES}
\objective{The Generalized Minimal Residuals (GMRES) algorithm is an iterative Krylov subspace method for efficiently solving large linear systems.
In this lab we implement the basic GMRES algorithm, then make an improvement by using restarts.
We then discuss the convergence of the algorithm and its relationship with the eigenvalues of a linear system.
Finally, we introduce SciPy's version of GMRES.}

\begin{comment}
\section*{The Arnoldi Iteration and Approximate Solutions} % ==================

Let $A$ be an $m\times m$ matrix (real or complex), where $m$ is very large,
and let $b \in \mathbb{F}^m$ ($\mathbb{F}$ may either be the real or complex numbers).
Let $\mathcal{K}_n$ denote the order-$n$ Krylov subspace generated by $A$ and $b$.
In each iteration, we consider the least squares problem
\begin{equation}
\underset{x \in \mathcal{K}_n}{\text{minimize}}\qquad \|b-Ax\|_2.
\label{eq:GMRES_lstsq1}
\end{equation}
Now if $x \in K_n$, then $x$ can be expressed as a linear combination of basis vectors $b, Ab, \ldots, A^{n-1}b$, i.e.
\[
x = y_1b + y_2Ab + \cdots + y_nA^{n-1}b.
\]
If we let $K_n$ be the matrix whose columns are $b, Ab, A^{2}b, \cdots, A^{n-1}b$, then we can write this simply as
$x = K_n y$.
Then the solution of the least squares problem is the vector $K_{n}y$ such that $\|b-A K_{n}y\|_2$ is minimized.
\begin{figure}
\includegraphics[width=\textwidth]{figures/LeastSquares.pdf}
\caption{GMRES involves solving the least squares problem repeatedly.}
\end{figure}

The major drawback of this approach is that it relies on the matrix $K_n$, which tends to be ill-conditioned due to its columns
being far from orthogonal (as discussed in Lab \ref{lab:kry_arnoldi}).
%To see this, suppose there is an eigenbasis for $A$ with associated eigenvalues, and suppose $\lambda$ is the largest eigenvalue.
%Then $\lambda^n$ is an eigenvalue of $A^n$.
%Since $\lambda$ is bigger than the other eigenvalues of $A$, $\lambda^n$ may be much, much bigger than the other eigenvalues of $A^n$,
%which are simply the eigenvalues of $A$ raised to the $n$th power.
%Thus the eigenvector associated with $\lambda$ often begins to dominate as we progress, with the end result being that the columns of
%$K_n$ become nearly linearly dependent.
The easiest fix in this situation is to use the Arnoldi iteration so that we have an orthonormal basis for $\mathcal{K}_n$ to work with.
Not only does this alleviate the problem of ill-conditioning, it also allows us to optimize in other ways, due to the special
structure of the matrices produced.

Let $q_1,\ldots, q_n$ be the orthonormal basis for $\mathcal{K}_n$ obtained by the Arnoldi iteration, and let $Q_n$ be the matrix
having these vectors as its columns.
Recall that $q_1 = b/\|b\|_2$.
Finally, let $H_n$ be the $(n+1)\times n$ upper Hessenberg matrix generated by the Arnoldi iteration, and let $e_1=(1,0,\cdots,0)$.

In this orthonormal basis, $x \in \mathcal{K}_n$ implies that there is some vector $y$ such that $x = Q_n y$.
Further, it is not hard to check that the matrices generated by the Arnoldi iteration satisfy the equation
\[
AQ_n = Q_{n+1}H_n.
\]
We also have the identity
\[
b = \|b\|_2q_1 = \|b\|_2Q_{n+1}e_1.
\]
Putting all of this together, we can rewrite the objective function of our least squares problem as follows:
\begin{align*}
\|b - Ax\|_2 &= \|Ax - b\|_2\\
&= \|AQ_ny - b\|_2\\
&= \|Q_{n+1}H_ny - \left(\|b\|_2Q_{n+1}e_1\right)\|_2\\
&= \|Q_{n+1}\left(H_n y - \|b\|_2e_1\right)\|_2.
\end{align*}

The matrix $Q_{n+1}$ has orthonormal columns, but it does not have enough columns to be a unitary matrix.
Let us extend the set $q_1,\ldots, q_{n+1}$ to an orthonormal basis $q_1,\ldots,q_{n+1},q_{n+2},\ldots,q_m$
of our space, and let $Q'$ be the matrix whose columns are equal to these vectors.
Then $Q'$ is now a unitary matrix, and hence preserves the norm, i.e. $\|Q'z\|_2 = \|z\|_2$ for all $z \in \mathbb{F}^m$.
Given $x \in \mathbb{F}^{n+1}$, if we define $x' \in \mathbb{F}^m$ to be
\[
x' =
\begin{bmatrix}
  x\\
  0\\
  \vdots\\
  0
\end{bmatrix},
\]
then you can easily check that
\[
Q'x' = Q_{n+1}x.
\]
From this, we deduce that
\begin{align*}
\|Q_{n+1}x\|_2 &= \|Q'x'\|_2\\
& = \|x'\|_2\\
&= \|x\|_2.
\end{align*}
Hence, we conclude that
\[
\|Q_{n+1}\left(H_n y - \|b\|_2e_1\right)\|_2 = \|H_n y - \|b\|_2e_1\|_2.
\]
Thus, the least squares problem given by \ref{eq:GMRES_lstsq1} is equivalent to the problem
\begin{equation}
\underset{y \in \mathbb{F}^n}{\text{minimize}}\qquad \|H_n y - \|b\|_2e_1\|_2.
\label{eq:GMRES_lstsq2}
\end{equation}
If $y$ is the solution to this problem, then the solution to \ref{eq:GMRES_lstsq1}, and hence an approximate
solution to $Ax = b$ is given by $x=Q_n y$.

We can measure how good our approximate solution is by considering the residual, which we define to be
\[
\frac{\|Ax-b\|_2}{\|b\|_2}.
\]
We can express this residual in terms of $y$ as follows:
\begin{equation}
\frac{\|H_n y - \|b\|_2e_1\|_2}{\|b\|_2}.
\label{eq:GMRES_residual}
\end{equation}
\end{comment}

\section*{The GMRES Algorithm} % ==============================================

GMRES is an iterative method that uses Krylov subspaces to reduce a high-dimensional problem to a sequence of smaller dimensional problems.
Let $A$ be an invertible $m \times m$ matrix and let $\b$ be a vector of length $m$.
Let $\mathcal{K}_n(A, \b)$ be the order-$n$ Krylov subspace generated by $A$ and $\b$.
Instead of solving the system $A\x = \b$ directly, GMRES uses least squares to find $\x_n \in \mathcal{K}_n$ that minimizes the residual $r_n = \|\b - A\x_n\|_2$.
The algorithm terminates when this residual is smaller than some predetermined value.
In many situations, this happens when $n$ is much smaller than $m$.

The GMRES algorithm uses the Arnoldi iteration for numerical stability.
The Arnoldi iteration produces $H_n$, an $(n+1)\times n$ upper Hessenberg matrix, and $Q_n$, a matrix whose columns make up an orthonormal basis of $\mathcal{K}_n(A, \b)$, such that $AQ_n = Q_{n+1}H_n$.
The GMRES algorithm finds the vector $\x_n$ which minimizes the norm $\norm{\b - A\x_n}_2$, where $\x_n = Q_n \y_n  + \x_0$ for some $\y_n \in \mathbb{R}^n$.
Since the columns of $Q_n$ are orthonormal, the residual can be equivalently computed as
\begin{equation}
\qquad \|\b - A\x_n\|_2 = \|Q_{n+1}(\beta \e_1 - H_n \y_n)\|_2 = \|H_n \y_n - \beta \e_1\|_2.
\label{eq:GMRES_lstsq1}
\end{equation}

Here $\e_1$ is the vector $[1, 0, \ldots, 0]\trp$ of length $n+1$ and $\beta=\norm{\b - A\x_0}_2$, where $\x_0$ is an initial guess of the solution.
% (Ordinarily this guess is zero; however, a modified version of the algorithm will be discussed in a later section, in which nonzero guesses will be used.)
Thus, to minimize $\norm{\b - A\x_n}_2$, the right side of (\ref{eq:GMRES_lstsq1}) can be minimized, and $\x_n $ can be computed as $\x_n=Q_n \y_n + \x_0$.

\begin{comment}
so that instead of solving \eqref{eq:GMRES_lstsq1}, at the $n$th iteration we solve
\begin{equation}
\underset{\y \in \mathbb{F}^n}{\text{minimize}}\qquad \|H_n \y - \|\b\|_2\e_1\|_2.
\label{eq:GMRES_lstsq2}
\end{equation}
Here, $H_n$ is the $(n+1)\times n$ upper Hessenberg matrix generated by the Arnoldi iteration and $\e_1$ is the vector $(1, 0, \ldots, 0)$ of length $n+1$.
If $\y$ is the minimizer for the $n$th iteration of \eqref{eq:GMRES_lstsq2}, then the residual is
\begin{equation}
\frac{\|H_n \y - \|\b\|_2\e_1\|_2}{\|\b\|_2},
\label{eq:GMRES_residual}
\end{equation}
and the corresponding minimizer for (\ref{eq:GMRES_lstsq1}) is $Q_n\y$, where $Q_n$ is the matrix whose columns are $\q_1, \ldots, \q_n$ as defined by the Arnoldi iteration.
This algorithm is outlined in Algorithm \ref{alg:gmres}.
For a complete derivation see [TODO: ref textbook].
\end{comment}

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{GMRES}{$A$, $\b$, $\x_0$, $k$, \li{tol}}
	\State $Q \gets \allocate{\size{\b}}{k+1}$			\Comment{Initialization.}
	\State $H \gets \zeros{k+1}{ k}$
	\State $r_0 \gets \b - A(\x_0)$
	\State $Q_{:,0} = r_0/\norm{r_0}_2$
    \For{$j=0\ldots k-1$}							\Comment{Perform the Arnoldi iteration.}
        \State $Q_{:,j+1} \gets A(Q_{:,j})$
        \For{$i=0\ldots j$}
            \State $H_{i,j} \gets Q_{:,i}\trp Q_{:,j+1}$
            \State $Q_{:,j+1} \gets Q_{:,j+1} - H_{i,j} Q_{:,i}$
        \EndFor
        \State $H_{j+1,j} \gets \norm{Q_{:,j+1}}_2$
        \If{$|H_{j+1,j}|>$ \li{tol}}                           \Comment{Avoid dividing by zero.}
            \State $Q_{:,j+1} \gets Q_{:,j+1}/H_{j+1,j}$
        \EndIf
        \State $\y \gets$ least squares solution to $\norm{H_{:j+2,:j+1}\x - \beta e_1}_2$     \Comment{$\beta$ and $\e_1$ as in (\ref{eq:GMRES_lstsq1}).}
        \State \li{res} $\gets \norm{H_{:j+2,:j+1}\y - \beta e_1}_2$
        \If{\li{res} $<$ \li{tol}}
            \State \pseudoli{return} $Q_{:,:j+1}\y + \x_0$, \li{res}
        \EndIf
    \EndFor
    \State \pseudoli{return} $Q_{:,:j+1}\y + \x_0$, \li{res}
\EndProcedure
\end{algorithmic}
\caption{The GMRES algorithm. This algorithm operates on a vector $\b$ and a linear operator $A$.
It iterates $k$ times or until the residual is less than \li{tol}, returning an approximate solution to $A\x=\b$ and the error in this approximation.}
\label{alg:gmres}
\end{algorithm}

%\begin{warn}
%The Python function \li{linalg.lstsq} solves a least squares problem, returning not only the vector, $y$, but also the residual,
%the rank of the matrix, and the singular values.
%Be careful when you write your code that you access the correct results and don't just assume that \li{linalg.lstsq} returns the
%vector that you want.
%The least squares solver also returns a residual, but it's not the number we reference in this book, so be sure to take the square
%root of the residual reported by the solver and divide by $\norm{b}$ to get $res=\norm{Ax-b}/\norm{b}$.
%\end{warn}
% Hint: explain how to find lstsq to find the residual?

\begin{problem}
Write a function that accepts a matrix $A$, a vector $\b$, and an initial guess $\x_0$, a maximum number of iterations $k$ defaulting to $100$, and a stopping tolerance \li{tol} that defaults to $10^{-8}$.
Use Algorithm \ref{alg:gmres} to approximate the solution to $A\x=\b$ using the GMRES algorithm.
Return the approximate solution and the residual at the approximate solution.

You may assume that $A$ and $\b$ only have real entries.
Use \li{scipy.linalg.lstsq()} to solve the least squares problem.
Be sure to read the documentation so that you understand what the function returns.

Compare your function to the following code.
\begin{lstlisting}
>>> A = np.array([[1,0,0],[0,2,0],[0,0,3]])
>>> b = np.array([1, 4, 6])
>>> x0 = np.zeros(b.size)
>>> gmres(A, b, x0, k=100, tol=1e-8)
(array([ 1.,  2.,  2.]), 7.174555448775421e-16)
\end{lstlisting}
\label{prob:MyGMRES}
\end{problem}

\begin{comment}
\subsection*{Breakdowns in GMRES} % -------------------------------------------

One of the most important characteristics of GMRES is that it does not terminate unless it reaches an exact solution.
That is, suppose that $\q_1, \ldots, \q_n$ has already been computed, where
\[\mathcal{K}_n(A, \b) = \text{span}\{\b, A\b, \ldots, A^{n-1}\b\} = \text{span}\{\q_1, \ldots, \q_n\}.\]
The next step is to compute $A^n\b$ and orthogonalize it against $\mathcal{K}_n(A, \b)$, yielding $\q_{n+1}$.
But if $A^n\b \in \mathcal{K}_n(A,\b)$ then $\q_{n+1}=0$, and the algorithm will break when $\q_{n+1}$ is normalized.
In this situation, $\b$ is in the $\text{span}\{\q_1, \ldots, \q_n\}$ so the least squares solution to \eqref{eq:GMRES_lstsq1} is an \emph{exact} solution to $A\x=\b$.
This is why Algorithm 1.1 returns $\x_n$ from the current iteration and residual if $\norm{\q_{n+1}}_2$ is less than the given tolerance.
\end{comment}

\subsection*{Convergence of GMRES} % ------------------------------------------

One of the most important characteristics of GMRES is that it will always arrive at an exact solution (if one exists).
At the $n$-th iteration, GMRES computes the best approximate solution to $A\x = \b$ for $\x_n \in \mathcal{K}_n$.
If $A$ is full rank, then $\mathcal{K}_m = \mathbb{F}^m$, so the $m$th iteration will always return an exact answer.
Sometimes, the exact solution $\x \in \mathcal{K}_n$ for some $n<m$, in this case  $x_n$ is an exact solution.
In either case, the algorithm is convergent after $n$ steps if the $n$th residual is sufficiently small.

The rate of convergence of GMRES depends on the eigenvalues of $A$.

\begin{problem}
\label{prob:plot_gmres}
Add a keyword argument \li{plot} defaulting to \li{False} to your function from Problem \ref{prob:MyGMRES}.
If \li{plot=True}, keep track of the residuals at each step of the algorithm.
At the end of the iteration, before returning the approximate solution and its residual error, create a figure with two subplots.
\begin{enumerate}
\item Make a scatter plot of the eigenvalues of $A$ on the complex plane.
\item Plot the residuals versus the iteration counts using a log scale on the $y$-axis\\(use \li{ax.semilogy()}).
\end{enumerate}
\end{problem}

\begin{problem}
\label{prob:make_plots}
Use your function from Problem \ref{prob:plot_gmres} to investigate how the convergence of GMRES relates to the eigenvalues of a matrix as follows.
Define an $m\times m$ matrix
\[A_n = nI+P,\]
 where $I$ is the identity matrix and $P$ is an $m \times m$ matrix with entries taken from a random normal distribution with mean 0 and standard deviation $1/(2\sqrt{m})$.
 Call your function from Problem \ref{prob:plot_gmres} on $A_n$ for $n=-4,-2,0,2,4$.
 Use $m=200$, let $\b$ be an array of all ones, and let $\x_0 = \0$.

Use \li{np.random.normal()} to create the matrix $P$.
When analyzing your results, pay special attention to the clustering of the eigenvalues in relation to the origin.
Compare your results with $n=2$, $m=200$ to Figure \ref{fig:plot_gmres}.

Ideas for this problem were taken from Example 35.1 on p. 271 of \cite{Trefethen1997}.

\begin{figure}[H] % Convergence of GMRES.
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/scatter_gmres.pdf}
\end{subfigure}
%
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/gmres_convergence.pdf}
\end{subfigure}
\caption{On the left, the eigenvalues of the matrix $A_2$ defined in Problem \ref{prob:make_plots}.
On the right, the rapid convergence of the GMRES algorithm on $A_2$ with starting vector $\b = (1, 1, \ldots, 1)$.}
\label{fig:plot_gmres}
\end{figure}
\end{problem}

% This section is poorly written. It is vague, not mathematically precise.
% If we ever want to include it, then it needs a lot of work.
\begin{comment}
\subsection*{Optimizing Least-Squares for GMRES (Optional)} % -----------------

The Hessenberg structure and the Krylov subspace relations enable us to save time on the least-squares part of the problem if we use QR factorization.
Observe that if $H_n$ can be factored as $Q_n R_n$, where $Q_n$ is not the same matrix as above and $R_n$ is invertible upper triangular, we may solve the least squares problem by simply solving $R_n x_n=\norm{b} Q_{n}^{H}e_1$ via back substitution.
There are two ways in which we can speed up this process.
First, we take advantage of the Hessenberg structure by using the techniques from Problem \ref{prob:givens_hessenberg} in Lab \ref{lab:givens}.
Recall that the technique in this situation was to use Givens rotations to eliminate the subdiagonal elements one at a time.
This process, which was part of a previous lab, reduces the operation count from $O(n^3)$ to $O(n^2)$.
The second speedup comes from the fact that we already know the QR factorization for $H_{n-1}$ from the previous step of the algorithm.
This means that we can simply update the QR factorization from the previous step rather than computing it all over again.
Since $H_{n}$ has only one more column and row than $H_{n-1},$ all we need to do is update the last column by performing all previous Givens rotations on just the last column of $H_n$, which requires only $O(n)$ work.
Then we perform one final Givens rotation on $H_n$ to eliminate the new subdiagonal entry which was not present in $H_{n-1}$.
Thus, the QR factorization of $H_n$ can be reduced from an $O(n^3)$ process to only $O(n)$ using these special techniques.

The back substitution necessary to solve the least squares problem can also be reduced to an operation of $O(n)$.

The speedup from $O(n^3)$ to $O(n)$ is very good, but it can only partially alleviate the problems that come with a problem that is ill-suited for GMRES.
It may still be useful because it allows us to perform more iterations in a reasonable amount of time.
In many situations, the simple technique of the next section will keep $n$ low enough that the optimizations from this section are not critical.

 \begin{problem}
 \label{prob:GMRES2}
 (Optional) Modify MyGMRES to incorporate these optimizations, and call this program MyGMRES1.
 Run both programs on a series of five random $100\times 100$ matrices and compare the time each requires.
 Are the gains substantial?
 Try it again matrices of size $1000\times 1000$ or larger, and see how substantial the difference becomes.
 Try the same thing using the techniques of the next section.
 Explain why the difference in performance is less dramatic this time.
 \end{problem}
\end{comment}

\section*{GMRES with Restarts} % -------------------------------------------

The first few iterations of GMRES have low spatial and temporal complexity.
However, as $k$ increases, the $k$th iteration of GMRES becomes more expensive temporally and spatially.
In fact, computing the $k$th iteration of GMRES for very large $k$ can be prohibitively complex.

This issue is addressed by using GMRES(k), or GMRES with restarts.
When $k$ becomes large, this algorithm restarts GMRES with an improved initial guess.
The new initial guess is taken to be the vector that was found upon termination of the last GMRES iteration run.
\begin{comment}
GMRES with restarts is outlined in Algorithm \ref{alg:gmres_k}.

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{GMRES(k)}{$A, \b, \x_0, k, tol, restarts$}
  \State $n \gets 0$ \Comment{Initialize}
   \While{$n \leq restarts$}
	\State{ Perform the GMRES algorithm, obtaining a least squares solution $\y$}.
	\State{ If the desired tolerance was reached, return. Otherwise, continue.}
    \State $\x_0 \gets \y$
    \State $n \gets n + 1$
    \EndWhile
    \State \pseudoli{return} $\y, \,\, res$		\Comment{Return the approximate solution and the residual}
\EndProcedure
\end{algorithmic}
\caption{The GMRES(k) algorithm. This algorithm performs GMRES on a vector $\b$ and matrix $A$. It iterates $k$ times before restarting.
It terminates after $restarts$ restarts or when the residual is less than $tol$, returning an approximate solution to $A\x=\b$ and the error of this approximation. }
\label{alg:gmres_k}
\end{algorithm}
\end{comment}
The algorithm GMRES(k) will always have manageable spatial and temporal complexity, but it is less reliable than GMRES.
If the true solution $\x$ to $A\x=\b$ is nearly orthogonal to the Krylov subspaces $\mathcal{K}_n(A, \b)$ for $n\leq k$, then GMRES(k) could converge very slowly or not at all.

\begin{problem}
Write a function that implements GMRES with restarts as follows.
\begin{enumerate}
\item Perform the GMRES algorithm for a maximum of k iterations.
\item If the desired tolerance was reached, terminate the algorithm.
If not, repeat step 1 using $x_k$ from the previous GMRES algorithm as a new initial guess $x_0$.
\item Repeat step 2 until the desired tolerance has been obtained or until a given maximum number of restarts has been reached.
\end{enumerate}
Your function should accept all of the same inputs as the function you wrote in Problem 1 with the exception of $k$, which will now denote the number of iterations before restart (defaults to $5$), and an additional parameter \li{restarts} which denotes the maximum number of restarts before termination (defaults to $50$).
\label{prob:GMRESk}
\end{problem}

\section*{GMRES in SciPy} % ===================================================

The GMRES algorithm is implemented in SciPy as the function \li{scipy.sparse.linalg.gmres()}.
Here we use this function to solve $A\x=\b$ where $A$ is a random $300 \times 300$ matrix and $\b$ is a random vector.

\begin{lstlisting}
>>> import numpy as np
>>> from scipy import sparse
>>> from scipy.sparse import linalg as spla

>>> A = np.random.rand(300, 300)
>>> b = np.random(300)
>>> x, info = spla.gmres(A, b)
>>> print(info)
3000
\end{lstlisting}

The function outputs two objects: the approximate solution $\x$ and an integer \li{info} which gives information about the convergence of the algorithm.
If \li{info=0} then convergence occured; if \li{info} is positive then it equals the number of iterations performed.
In the previous case, the function performed 3000 iterations of GMRES before returning the approximate solution $\x$.
The following code verifies how close the computed value was to the exact solution.
\begin{lstlisting}
>>> la.norm((A @ x) - b)
4.744196381683801
\end{lstlisting}

A better approximation can be obtained using GMRES with restarts.
\newpage
\begin{lstlisting}
# Restart after 1000 iterations.
>>> x, info = spla.gmres(A, b, restart=1000)
>>> info
0
>>> la.norm((A @ x) - b)
1.0280404494143551e-12
\end{lstlisting}
This time, the returned approximation $\x$ is about as close to a true solution as can be expected.

\begin{problem}
Plot the runtimes of your implementations of GMRES from Problems 1 and 4 and \li{scipy.sparse.linalg.gmres()} use the default tolerance and \li{restart=1000} with different matrices.
Use the $m \times m$ matrix $P$ with $m=25,50,\dots 200$ and with entries taken from a random normal distribution with mean 0 and standard deviation $1/(2\sqrt{m})$.
Use a vector of ones for $\b$ and a vector of zeros for $\x_0$.
Use a single figure for all plots, plot the runtime on the $y$-axis and $m$ on the $x$-axis.
\end{problem}

\begin{comment}
% This problem is a potential application problem that could be included in the lab.
% The solutions to this problem are included in solutions.py.
\begin{problem}
Using the function \li{finite_difference()} from your iterative solvers lab, modify the \li{hot_plate()} function to use the GMRES algorithm instead of the SOR method.
Do you see any differences in speed?
Can you solve larger systems with GMRES?
Try the different versions of the GMRES algorithm that we have discussed, what differences do you see?
\end{problem}
\end{comment}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Written By Michael Brodskiy
% Class: Analytic Geometry & Calculus III (Math-292)
% Professor: V. Cherkassky
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[12pt]{article}
\usepackage{alphalph}
\usepackage[utf8]{inputenc}
\usepackage[russian,english]{babel}
\usepackage{titling}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{amssymb}
\usepackage[super]{nth}
\usepackage{everysel}
\usepackage{ragged2e}
\usepackage{geometry}
\usepackage{fancyhdr}
\geometry{top=1.0in,bottom=1.0in,left=1.0in,right=1.0in}
\newcommand{\subtitle}[1]{%
  \posttitle{%
    \par\end{center}
    \begin{center}\large#1\end{center}
    \vskip0.5em}%

}
\usepackage{hyperref}
\hypersetup{
colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=blue,
citecolor=blue,
}

\urlstyle{same}

\title{Lecture XVII Notes}
\date{July 13, 2020}
\author{Michael Brodskiy\\ \small Professor: V. Cherkassky}

% Mathematical Operations:

% Sum: $$\sum_{n=a}^{b} f(x) $$
% Integral: $$\int_{lower}^{upper} f(x) dx$$
% Limit: $$\lim_{x\to\infty} f(x)$$

\begin{document}

\maketitle

\section{Mass and Density $-$ 15.4}

The mass may be expressed as
$$m=\iint_D \rho(x,y)\,dA$$

Total charge is expressed as:
$$Q=\iint_D \sigma(x,y)\,dA$$

The (first) moments on the $x$ and $y$ axis, respectively, may be found by using:

$$M_x=\iint_D y\rho(x,y)\,dA\text{ and }M_y=\iint_D x\rho(x,y)\,dA$$

The center of mass, $(\overline{x},\overline{y})$ may be found by rearranging the formula:

$$m\overline{x}=M_y\text{ and } m\overline{y}=M_x$$
$$\overline{x}=\frac{M_y}{m}=\frac{\iint_D x\rho(x,y)\,dA}{\iint_D \rho(x,y)}\,dA$$
  $$\overline{y}=\frac{M_x}{m}=\frac{\iint_D y\rho(x,y)\,dA}{\iint_D \rho(x,y)}\,dA$$

  The moment of inertia (or the second moment) is $mr^2$, where $r$ is the distance from a particle to the axis. About the $x$ axis, the moment of inertia is equal to:

  $$I_x = \iint_D y^2\rho(x,y)\,dA$$

  Across the $y$ axis the moment of inertia is equal to:

  $$I_y=\iint_D x^2\rho(x,y)\,dA$$

  In addition to this, the polar moment of inertia (about the origin) is equal to:

  $$I_0\iint_D (x^2+y^2)\rho(x,y)\,dA$$

  This means that $I_0=I_x+I_y$

  \section{Surface Area $-$ 15.5}

  The surface area, $A(S)$ is defined by:

  $$A(S)=\lim_{(m,n)\to\infty}\sum_{i=1}^m\sum_{j=1}^n \Delta T_ij$$

  $T_ij$ is defined by $|a\text{ x }b|$, where:

  $$a=\Delta x\bold{\hat{i}} + f_x(x_i,y_i)\Delta x \bold{\hat{k}}$$
  $$b=\Delta y\bold{\hat{j}} + f_y(x_i,y_j)\Delta y \bold{\hat{k}}$$

  Therefore, as an iterated integral, the surface area may be expressed as:

  $$A(S)=\iint_D \sqrt{1+\left(\frac{\partial z}{\partial x}\right)^2+\left(\frac{\partial z}{\partial y}\right)^2}\,dA$$

\end{document}


\chapter{Graph}

\section{Basic}
\runinhead{Graph representation.} $V$ for a vertex set with a map, mapping from vertex to its neighbors. The mapping relationship represents the edges $E$.
\begin{python}
V = defaultdict(list)
\end{python}

\runinhead{Complexity.} Basic complexities:

\begin{tabular}{lll}
\hline\noalign{\smallskip}
\textbf{Algorithm} & \textbf{Time}  & \textbf{Space}\\
\noalign{\smallskip}\hline\noalign{\smallskip}
dfs & $O(|E|)$ & $O(|V|), O(\text{longest path})$ \\
bfs & $O(|E|)$ & $O(|V|)$ \\
\noalign{\smallskip}\hline\noalign{\smallskip}
\caption{CAPTIONS}
\end{tabular}

\section{DFS}
\rih{Number of Islands.} The most fundamental and classical problem.
\begin{python}
11000
11000
00100
00011
Answer: 3
\end{python}
\rih{Clue}:
\begin{enumerate}
\item Iterative dfs
\end{enumerate}
\begin{python}
class Solution(object):
  def __init__(self):
    self.dirs = [(-1, 0), (1, 0), (0, -1), (0, 1)]

  def numIslands(self, grid):
    cnt = 0
    visited = [[False for _ in xrange(n)]
               for _ in xrange(m)]
    for i in xrange(m):
      for j in xrange(n):
        if not visited[i][j] and grid[i][j] == "1":
          self.dfs(grid, i, j, visited)
          cnt += 1

    return cnt
\end{python}
\newpage
\begin{python}
  def dfs(self, grid, i, j, visited):
    m = len(grid)
    n = len(grid[0])
    visited[i][j] = True

    for dir in self.dirs:
      I = i+dir[0]
      J = j+dir[1]
      if (0 <= I < m and 0 <= J < n and
        not visited[I][J] and grid[I][J] == "1"):
        self.dfs(grid, I, J, visited)
\end{python}
If the islands are constantly updating and the query for number of islands is called multiple times, need to use union-find (Section \ref{section:unionFind}) to reduce each query's complexity from $O(mn)$ to $O(\log mn)$.

\section{BFS}
\subsection{BFS with Abstract Level}
Start bfs with a set of vertices in abstract level, not necessarily neighboring vertices.

Example: $-1$ obstacles, $0$ targets, calculate all other vertices' Manhattan distance to its nearest target:
$$
\begin{bmatrix}
\infty & -1 & 0 & \infty \\
\infty & \infty & \infty & -1 \\
\infty & -1 & \infty & -1 \\
0 & -1 & \infty & \infty \\
\end{bmatrix}
$$

is calculated as:
$$
\begin{bmatrix}
3 & -1 & 0 & 1 \\
2 & 2 & 1 & -1 \\
1 & -1 & 2 & -1 \\
0 & -1 & 3 & 4 \\
\end{bmatrix}
$$
\newpage
\rih{Code:}
\begin{python}
self.dirs = ((-1, 0), (1, 0), (0, -1), (0, 1))

def wallsAndGates(self, mat):
  q = [(i, j) for i, row in enumerate(mat)
     for j, val in enumerate(row) if val == 0]
  for i, j in q:  # iterator
    for d in self.dirs:
      I, J = i+d[0], j+d[1]
      if (0 <= I < m and  0 <= J < n and
        mat[I][J] > mat[i][j]+1):
        mat[I][J] = mat[i][j]+1
        q.append((I, J))
\end{python}

\section{Detect Acyclic}
\begin{enumerate}
\item \pythoninline{marked} is reset after a dfs.
\item \pythoninline{visited} should be updated only in the end of the dfs.
\item For directed graph:
\begin{enumerate}
\item Should dfs for all neighbors except for vertices in \pythoninline{visited}, to avoid revisiting. For example, avoid revisiting A, B when start from C in the graph $C \rightarrow A \rightarrow B$.
\item Excluding predecessor \pythoninline{pi} is erroneous in the case of $A \leftrightarrow B$
\end{enumerate}
\item For undirected graph:
\begin{enumerate}
\item Should dfs for all neighbors except for the predecessor \pythoninline{pi}. $A-B$.
\item Excluding neighbors in \pythoninline{visited} is redundant, due to \pyinline{pi}.
\end{enumerate}
\end{enumerate}

\subsection{Directed Graph}
Detect cycles (any) in directed graph.

\begin{python}
def dfs(self, V, v, visited, pathset):
  if v in pathset:
    return False

  pathset.add(v)
  for nbr in V[v]:
    if nbr not in visited:
      if not self.dfs(V, nbr, visited, pathset):
        return False

  pathset.remove(v)
  visited.add(v)
  return True
\end{python}

\subsection{Undirected Graph}
Detect cycles (any) in undirected graph.

\begin{python}
def dfs(self, V, v, pi, visited, pathset):
  if v in pathset:
    return False

  pathset.add(v)
  for nbr in V[v]:
    if nbr != pi:
      if not self.dfs(V, nbr, v, visited, pathset):
        return False

  pathset.remove(v)
  visited.add(v)
  return True
\end{python}

\section{Topological Sorting}
For a graph $G=\{V, E\}$, if $A \rightarrow B $, then $A$ is before $B$ in the ordered list.
\subsection{Algorithm}
\rih{Core clues}:
\begin{enumerate}
\item \textbf{Dfs neighbors first}. If the neighbors of current node is  $\neg$visited, then dfs the neighbors
\item \textbf{Process current node}. After visiting all the neighbors, then visit the current node and push it to the result queue.

\end{enumerate}
Notice:
\begin{enumerate}
\item Need to check ascending order or descending order.
\item Need to \textbf{detect cycle}; thus the dfs need to construct result queue and detect cycle simultaneously, by using two sets: $visited$ and $pathset$.
\end{enumerate}
\newpage
\begin{python}
from collections import deque

def topological_sort(self, V):
  visited = set()
  ret = deque()

  for v in V.keys():
    if v not in visited:
      if not self.dfs_topo(V, v, visited, set(), ret):
        return []  # contains cycle

  return list(ret)

def dfs_topo(self, V, v, visited, pathset, ret):
  if v in pathset:
    return False

  pathset.add(v)
  for nbr in V[v]:
    if nbr not in visited:
      if not self.dfs_topo(V, nbr, visited, pathset, ret):
        return False

  pathset.remove(v)
  visited.add(v)
  ret.appendleft(v)
  return True

\end{python}

\subsection{Applications}
\begin{enumerate}
\item Course scheduling problem with pre-requisite.
\end{enumerate}

\section{Union-Find}\label{section:unionFind}
Improvements:
\begin{enumerate}
\item Weighting: size-baladnced tree
\item Path Compression.
\end{enumerate}}
\subsection{Algorithm}
Weighted union-find with path compression.\\
\rih{Core clues.}
\begin{enumerate}
\item \textbf{$\pi$ array}:an array to store each item's predecessor pi. The predecessor are lazily updated to its ancestor. When \pyinline{x == pi[x]}, then \pyinline{x} is the ancestor (i.e. root).
\item \textbf{Size-balanced}: merge the tree according to the size to maintain balance.
\item \textbf{Path compression}: Make the ptr in $\pi$ array to point to its root rather than its immediate parent.
\end{enumerate}
\begin{figure}[]
\centering
\subfloat{\includegraphics[scale=.70]{uf}}
\caption{Weighted quick-union traces}
\label{fig:union_find}
\end{figure}

\newpage
\begin{python}
class UnionFind(object):
  def __init__(self):
    self.pi = {}  # item -> pi
    self.sz = {}  # root -> size

  def __len__(self):
    """number of unions"""
    return len(self.sz)  # only root nodes have size

  def add(self, x):
    if x not in self.pi:
      self.pi[x] = x
      self.sz[x] = 1

  def root(self, x):
    """path compression"""
    pi = self.pi[x]
    if x != pi:
      self.pi[x] = self.root(pi)
    return self.pi[x]

  def unionize(self, x, y):
    pi1 = self.root(x)
    pi2 = self.root(y)

    if pi1 != pi2:
      if self.sz[pi1] > self.sz[pi2]:
        pi1, pi2 = pi2, pi1
        # size balancing
      self.pi[pi1] = pi2
      self.sz[pi2] += self.sz[pi1]
      del self.sz[pi1]

  def isunion(self, x, y):
    if x not in self.pi or y not in self.pi:
      return False
    return self.root(x) == self.root(y)
\end{python}

\subsection{Complexity}
$m$ union-find with $n$ objects: $O(n)+m O(\lg n)$

\section{Axis Projection}
Project the mat dimension from 2D to 1D, using \textit{orthogonal axis}.

\runinhead{Smallest bounding box.} Given the location $(x, y)$ of one of the 1's, return the area of the smallest bounding box that encloses 1's.
$$
\begin{bmatrix}
0& 0& 1& 0 \\
0& 1& 1& 0 \\
0& 1& 0& 0 \\
\end{bmatrix}
$$

Clues:
\begin{enumerate}
\item Project the 1's onto x-axis, binary search for the left bound and right bound of the bounding box.
\item Do the same for y-axis.
\end{enumerate}

Time complexity: $O(m\log n + n \log m)$, where $O(m), O(n)$ is for projection complexity.


\chapter{Approximation}

\section{Error}

\index{approximation error}
\index{error!of approximation}
If the data set is \(D = \{(x_1,y_1),\ldots,(x_n,y_n)\}\) and the approximator is \(f\),
then the \emph{approximation error} is \(e = \sum_{k=1}^n f(x_k) - y_k\).

\section{Metric space}

Before we can approximate, we must have a metric space.
You must define the concept of distance.
You must have a target, an error function, and a desired accuracy.

After that, we are interested in
what we are approximating, how fast we converge, how accurate we are.

At a higher level, we can think of approximation about
transformation between metric spaces.

In a narrow sense, we can compute the approximate decimal representation of a real number.
The truncated decimal representation is the approximation of the real number.
We know the real number with certainty.
There is only one such real number.
We can define it.
We just haven't computed its decimal representation.

In a narrow sense, to approximate a real number is
to construct a sequence that converges to that real number.

\section{The meat}

Define a metric \(d\) as a function satisfying these, for all \(x\) and \(y\):
\begin{align*}
    d~x~y &\ge 0
    \\
    d~x~y &= d~y~x
    \\
    d~x~y = 0 &\iff x=y
\end{align*}

Given \(d\) and \(<\), we can verify whether a sequence approximates a value.
However, those are not enough to construct an approximation scheme.

\(\sqrt{2}\) is the positive \(x\) that satisfies \(x^2 = 2\).
We can rearrange \(x = 2/x\). If we let \(f~x = 2/x\), then \(\sqrt{2}\) is the fixed point of \(f\)?

\begin{align*}
    f~x &= x
    \\
    f~x &= 2/x
    \\
    f~y &= 2/y
    \\
    f~y - f~x &= 2/y - 2/x = 2/y - x
    \\
    \frac{f~y - f~x}{y - x} &= \frac{2(x-y)}{xy(y-x)} = - \frac{2}{xy} = m
    \\
    f~y &\approx f~x + m \cdot (y - x)
    \\
    f~x &\approx f~y - m \cdot (y - x)
    \\
    x &\approx f~y - m \cdot (y - x)
\end{align*}

Given a fixed point equation, can we construct an approximation scheme?

Newton-Raphson method: given a monotonous function with nonzero derivative, we can approximate.

\section{Motivating example for the guide and update functions}

Computing \(\sqrt{x}\) is harder than computing \(x^2\).
We can combine iteration and squaring to compute square root.

What is \(\sqrt{2}\)? It is the positive \(x\) that satisfies the equation \(x^2 - 2 = 0\).

We know that \(x\) is between 1 and 2.
We can use binary search.

We have an initial approximation that \(x \in [a,b]\) and we want to close in:
we want to show that \(x \in [a',b']\) where \([a',b'] \subset [a,b]\).
Let \(c = (a+b)/2\). If \(g~c > 0\), the midpoint is above than the target,
so we update \(a' = a\) and \(b' = c\).
If \(g~c < 0\), the midpoint is below the target,
so we update \(a' = c\) and \(b' = b\).

Look at this example approximation scheme to compute \(\sqrt{2}\).
\begin{align*}
    f~x = \begin{cases}
        x / 2 &: x^2 - 2 > 0
        \\
        x &: x^2 - 2 = 0
        \\
        x + 1 &: x^2 - 2 < 0
    \end{cases}
\end{align*}

Given \fun{raise}, \fun{lower}, \fun{guide}, we can construct this approximation scheme:
\begin{align*}
    f~x = \begin{cases}
        \fun{lower}~x &: \fun{guide}~x > 0
        \\ x &: \fun{guide}~x = 0
        \\ \fun{raise}~x &: \fun{guide}~x < 0
    \end{cases}
\end{align*}

The key fact is that \(g~x\) does not have to compute the exact error.
It only has to compute an error that is monotonous to the actual error.
For example, instead of using the error function \(e~x = x - \sqrt{2}\),
we can define the guide function \(g~x = x^2 - 2\).

\section{Guide function}

The guide function is easier to compute than the error function.

The guide function is related to the error function:
\begin{align*}
    e~x = 0 &\iff g~x = 0
    \\
    e~x < e~y &\iff g~x < g~y
\end{align*}

\section{Guided approximation scheme}

Given an update function \(u\) and a guide function \(g\),
we get a free iterative approximation scheme \(f\):
\begin{align*}
    f~x = u~x~(g~x)
\end{align*}

Simple update function:
\[
    u~x~v = x - v/2
\]

\section{Testing whether a function is an approximation scheme}

Iff \(e\) is a monotonous function
and \(|e~(f~x)| < |e~x|\) for all \(x\),
then \(f^\infty~x\) converges.

\section{The update function must satisfy these properties.}

Constraint: If the guide is zero, then the approximation is exact
and it must keep the approximation.
\[
    u~x~0 = x
\]

Constraint: For smaller error, the update function must jump smaller.
\[
    |a| < |b| \iff |u~x~a - x| < |u~x~b - x|
\]

The update function must jump against the error.
\[
    (u~x~a - x) \cdot a < 0
\]

\section{Conclusion}

Consider a circle of radius 1 and a square of side 2.
The area of the circle and the circumference of the square:
\[
    \pi^2 < 2^2
\]
\[
    \pi^3 < 2^3
\]
\[
    \pi^n < 2^n
\]

\[
    [2 \cos (\pi/4)]^2 < \pi^2
\]

\[
    x^2 + y^2 = 1
\]

\[
    \int_0^1 \sqrt{1-x^2}~dx = \pi / 4
\]

via Riemann sum:

\[
    \pi/4 = \lim_{n\to\infty} \sum_{k=0}^{n-1} \frac{1}{n} \sqrt{1 - (k/n)^2}
\]

Let
\[
    f~n = \sum_{k=0}^{n-1} \frac{1}{n} \sqrt{1 - (k/n)^2}
\]
State \(f~(2 \cdot n)\) in terms of \(f~n\).

\[
    f~(2 \cdot n) = \frac{1}{2} \cdot f~n
\]

\[
    f~(2 \cdot n) = \sum_{k=0}^{2 \cdot n - 1} \frac{1}{2 \cdot n} \sqrt{1 - (k/(2 \cdot n))^2}
\]

\[
    f~(2 \cdot n) = \sum_{k=0}^{2 \cdot n - 1} \frac{1}{2 \cdot n} \sqrt{1 - ((k/2)/n)^2}
\]

\[
    f~(2 \cdot n) =
    \sum_{k=0,2,\ldots}^{2 \cdot n - 2} \frac{1}{2 \cdot n} \sqrt{1 - ((k/2)/n)^2}
    + \sum_{k=1,3,\ldots}^{2 \cdot n - 1} \frac{1}{2 \cdot n} \sqrt{1 - ((k/2)/n)^2}
\]

\[
    f~(2 \cdot n) =
    \sum_{2 \cdot j=0,2,\ldots}^{2 \cdot n - 2} \frac{1}{2 \cdot n} \sqrt{1 - (((2 \cdot j)/2)/n)^2}
    + \sum_{2 \cdot j + 1=1,3,\ldots}^{2 \cdot n - 1} \frac{1}{2 \cdot n} \sqrt{1 - (((2 \cdot j + 1)/2)/n)^2}
\]
\[
    f~(2 \cdot n) =
    \sum_{j=0}^{n - 1} \frac{1}{2 \cdot n} \sqrt{1 - (j/n)^2}
    + \sum_{2 \cdot j=0,2,\ldots}^{2 \cdot n - 2} \frac{1}{2 \cdot n} \sqrt{1 - (((2 \cdot j + 1)/2)/n)^2}
\]
\[
    f~(2 \cdot n) =
    \sum_{j=0}^{n - 1} \frac{1}{2 \cdot n} \sqrt{1 - (j/n)^2}
    + \sum_{j=0}^{n-1} \frac{1}{2 \cdot n} \sqrt{1 - ((j+1/2)/n)^2}
\]
\[
    f~(2 \cdot n) =
    \frac{1}{2} \sum_{j=0}^{n - 1} \frac{1}{n} \sqrt{1 - (j/n)^2}
    + \sum_{j=0}^{n-1} \frac{1}{2 \cdot n} \sqrt{1 - (j/n + 1/(2n))^2}
\]
\[
    f~(2 \cdot n) =
    \frac{1}{2} \cdot f~n
    + \frac{1}{2} \sum_{j=0}^{n-1} \frac{1}{n} \sqrt{1 - ((j+1/2)/n)^2}
\]

\[
    f~d~(2 \cdot n) = \frac{f~d~n + f~(d + 1/(2\cdot n))~n}{2}
\]

\[
    f~d~n = \frac{1}{n} \cdot \sum_{k=0}^{n-1} \sqrt{1 - (k/n + d)^2}
\]

The identity:
\[
    f~d~n = \frac{f~d~(n/2) + f~(d + 1/n)~(n/2)}{2}
\]

\[
    f~0~2 = \frac{f~0~1 + f~(1/2)~1}{2}
\]

\[
    f~0~4 = \frac{f~0~2 + f~(1/4)~2}{2}
\]

\[
    f~0~4 = \frac{\frac{f~0~1 + f~(1/2)~1}{2} + \frac{f~(1/4)~1 + f~(3/4)~1}{2}}{2}
\]

\[
    f~0~4 = \frac{f~0~1 + f~(1/4)~1 + f~(2/4)~1 + f~(3/4)~1}{4}
\]

\[
    f~0~n = \frac{1}{n} \cdot \sum_{k=0}^{n-1} f~(k/n)~1
\]

That holds for all Riemann sums where \(d\) is the offset (translation).

\[
    f~d~1 = \sqrt{1 - d^2}
\]

Let \(x~n = f~0~(2^n)\).

If you have a guide function and have an interval that contains the target,
you can use the interval approximation scheme.

The equation for the error function is easy to write but hard to compute.
The key of approximation is finding a guide function.
If the number is the root of a polynomial,
then the polynomial equation can be the guide function.

\section{Incremental Riemann sum}

Let
\[
    s~f~d~n = \frac{1}{n} \sum_{k=0}^{n-1} f~(k/n + d)
\]
where \(s~f~0~n\) is the finite Riemann sum.
\[
    \int_0^1 f~(x+u) ~ dx = \lim_{n\to\infty} s~f~u~n
\]

Then we have the recurrence relation:
\[
    s~f~d~(2 \cdot n) = \frac{s~f~d~n + s~f~(d + 1/(2 \cdot n))~n}{2}
\]
The recurrence relation can be used to approximate the integral to any accuracy,
but it converges slowly.

Equivalently:
\[
    s~f~d~n = \frac{s~f~d~(n/2) + s~f~(d + 1/n)~(n/2)}{2}
\]

where
\[
    s~f~d~1 = f~d
\]

\section{Sequence from iteration}

The sequence generated by iterating \(f\) on \(x\) is:
\[
    x, ~ f~x, ~ f~(f~x), ~ \ldots, ~ f^n~x, ~ \ldots
\]

In the theory of unary algebras, that sequence is also known as
the \emph{orbit} of \(x\) in the unary algebra \((X,f)\)
where \(x : X\).

\section{Approximating sequence}

Formally we define a sequence \(x = \{ x_0, x_1, \ldots \}\) approximates \(y\) iff \(x_\infty = y\).

\section{Approximation is not estimation.}

Approximation converges.
Estimation doesn't converge because the actual value is unknown.

Approximation doesn't guess.
Estimation guesses.

Approximation has error.
Estimation has uncertainty.

Meta approximation? Approximate the approximation scheme?

\section{Network of radial basis functions}

We can approximate a high-dimensional sparse function
by a weighted sum of radial functions.
(A radial function is a function whose value at a point
depends only on the distance of that point from a certain point.)
See Bromhead and Lowe 1988, radial basis function neural network?
% https://en.wikipedia.org/wiki/Radial_basis_function

\section{Approximating unknown functions}

Of course we can't approximate a function
if we don't know anything about it at all.
We must know something.

Given a set of sample values,
what is the most likely function?
Given a set of sample bit strings,
what is the most likely algorithm?
Isn't this the idea of algorithmic complexity (Solomonoff)?

We don't know the closed form expression of the function,
and we only know a few finitely many values of the function at some points.
For example, let the input be a \(64 \times 64\) image
where each pixel is a 8-bit unsigned integer,
and let the input be a boolean that indicates
whether the input is the digit zero.

\section{There is an abstract converging approximation scheme in a metric space.}

Let there be a type \(A\).
Let there be a total order \(<\) on \(A\).
We want to approximate \(y : A\).
Start with any \(a : A\).
Apply approximation scheme \(f : A \to A\).
An approximation scheme is an endofunction.
If \(d~(f~x)~y < d~x~y\) for almost all \(x\), then \(f^\infty~a\) converges to \(y\).
To approximate \(y\), apply \(f\) repeatedly.
To approximate \(y\), iterate \(f\).
If \( f^\infty~x = y \), we say \(f\) converges to \(y\) from \(x\).

\section{Example with a continued fraction}

For example, let \(f~x = 1 + 1/x\).
It follows that
\[
    f^\infty~x = 1 + \frac{1}{1 + \frac{1}{1 + \ldots}} = \frac{1 + \sqrt{5}}{2} = \Phi
\]
For every \(x > 0\), we have \(f^\infty~x = \Phi\).
Therefore that \(f\) is an approximation scheme for \(\Phi\).

Let \(f~x = 2 + 1/x\).
\[
    2 + \frac{1}{2 + \frac{1}{2 + \ldots}} = 1 + \sqrt{2}
\]

\section{Some approximation schemes give rise to continued fractions.}

Rearranging a continued fraction gives a polynomial equation.

We can go the other way too.
We can always transform a polynomial equation into a continued fraction
by rearranging the equation so that the unknown variable appears on both sides.

You rearrange the equation and end up with the same thing you begin with.
\[
    1 + \frac{1}{1 + \frac{1}{1 + \ldots}} = x
\]
Subtract both sides by 1.
\[
    \frac{1}{1 + \frac{1}{1 + \ldots}} = x - 1
\]
Invert both sides.
\[
    1 + \frac{1}{1 + \ldots} = \frac{1}{x - 1}
\]
The left side is the same as what we started out with.
\[
    x = \frac{1}{x - 1}
\]
We have a quadratic equation.
\[
    x^2 - x - 1 = 0
\]

Let \(f~x = 1 + 1/x^2\).
\[
    1 + \frac{1}{\left(1 + \frac{1}{(1+\ldots)^2}\right)^2}
\]

\section{Continued fraction of functions}

Let's use the same equation \(f~x = 1 + 1/x\),
but now \(x : \Real \to \Real\).
Actually \(f~x~t = 1 + \frac{1}{x~t}\).

Some converges to the constant function \(t \to \Phi\).

\section{Continued fraction of square matrices}

\(f~x = I_n + x^{-1}\).

Some converge. Some oscillate.

\section{Taylor series of function space}

Given \(x : \Real\),
the Maclaurin series of \(f\) is
\[
    \sum_{k=0}^{\infty} \frac{(d^k~f)~0 \cdot x^k}{k!}
\]

Now what if \(x : \Real \to \Real\)?

\section{Derivative in function space}

Let \(x : \Real \to \Real\).
Take the limit as \(h \to 0\).
\[
    h \cdot d~f~x = f~(x + h) - f~x
\]

% https://en.wikipedia.org/wiki/Generalizations_of_the_derivative#Functional_analysis

\section{Combination of simpler functions}

\[
    g~(f_0,f_1,\ldots) \sim w_0 \cdot f_0 + w_1 \cdot f_1 + \ldots
\]

\section{Continued fraction involving discrete types}

\section{Problem statement}

Approximate a function as a combination of simpler functions.

Approximation needs distance.
Approximating \(x : A\) requires that a distance \(d~x~y\) be defined.
The value of \(d~x~y\) is not important.
The important is the ordering among those values.

A straightforward approximation scheme \(f\) satisfies:
\[
    d~(f~x)~(f~(f~x)) < d~x~(f~x)
\]

\section{Details}

We have a square integrable function \(f : \Real^\infty \to I\)
where \(I = [0,1]\) is the unit interval.
We don't know the equation for \(f\), but we have some samples.
We want to approximate it as a combination of simpler functions
\[
    f \sim g~\begin{bmatrix} c_0 & c_1 & \ldots \end{bmatrix}
\]
where each \(c_k : \Real \to \Real\)
and \(g : (\Real \to \Real)^\infty \to (I \to I)\).

If we generalize that by defining \(A = I \to I\),
we have a thing \(f : A\)
where each \(c_k : A\) and \(g : A^\infty \to A\).
\[
    f \sim g~c
\]

In the case of a perceptron, \(g\) has the form
\[
    g~c = s ~ (m~w~c)
\]
where \(W : \Real^\infty\), \(c : \Real^\infty\),
and \(s : \mathbb{R} \to I\).
The purpose of \(s\) is to keep the summation result in \(I\).

We can state it as a combination of other functions.

We can truncate its Taylor series.

We can fit a function to the points.

% QUESTION

Define $f^\infty(x)$ as $\lim_{n\to\infty} f^n(x)$, where $f^n = \underbrace{f \circ \ldots \circ f}_{n}$.

Usually the unqualified term 'continued fraction' means continued fraction in $\mathbb{R}$. For example, consider $f(x) = 1+1/x$ where $x \in \mathbb{R}$. In that case, for all $x > 0$, we have $f^\infty(x) = \Phi = \frac{1+\sqrt{5}}{2}$.

Now consider another $f(x) = 1 + 1/x$ where $x \in \mathbb{R} \to \mathbb{R}$. For some $x$, we have $f^\infty(x) = \hat\Phi$ where $\hat\Phi$ is the constant function that always gives $\Phi$. In other words, we can think of $f^\infty$ as a continued fraction in the function space $\mathbb{R} \to \mathbb{R}$.

Have anybody studied such continued fractions in function spaces? I googled:

- continued fraction in function space
- continued fraction of real function


\subsection{Monte Carlo sampling and analysis}
\input{6DL/EEn}

\begin{lemma}   \label{MC}
For any $g\in L^\infty(G)$, we have
  \begin{equation}
  \begin{split}
    \mathbb{E}_n\Big(\mathbb{E}g-\frac1n\sum_{i=1}^n
    g(\omega_i)\Big)^2
    &=\frac{1}{n}\mathbb{V}(g)=\frac{1}{n}\Big(\mathbb{E}(g^2) - \big (\mathbb{E}(g)\big )^2\Big)
    \\
    &=
    \left\{
             \begin{aligned}
    \frac{1}{n}\mathbb{V}(g)
   & \le\frac{1}{n} \sup_{\omega, \omega'\in G} |g(\omega) - g(\omega')|^2
    \\
\frac{1}{n}\Big(\mathbb{E}(g^2) - \big (\mathbb{E}(g)\big )^2\Big)
&\le\frac{1}{n} \mathbb E(g^2)\le \frac{1}{n}\|g\|^2_{L^\infty},
\end{aligned}
\right.
\end{split}
  \end{equation}
\end{lemma}

\begin{proof}%[Proof of Lemma \ref{MC}]
First note that
 \begin{equation}
    \label{eqn}
    \begin{aligned}
\left(\mathbb{E} g-\frac1N\sum_{i=1}^Ng(\omega_i)\right)^2
  &
=\frac{1}{N^2} \left(\sum_{i=1}^N(\mathbb{E} g-g(\omega_i))\right)^2
  \\
  &=\frac{1}{N^2} \sum_{i,j=1}^N(\mathbb{E} g-g(\omega_i))(\mathbb{E} g-g(\omega_j))
  \\
  &=\frac{I_1}{N^2} +\frac{I_2}{N^2}.
    \end{aligned}
  \end{equation}
with
\begin{equation}
I_1= \sum_{i=1}^N(\mathbb{E} g-g(\omega_i))^2,\quad I_2=\sum_{i\neq  j}^N\left ((\mathbb{E}g)^2-\mathbb {E}(g)(g(\omega_i)+
g(\omega_j))+g(\omega_i)g(\omega_j))\right).
\end{equation}
Consider $I_1$, for any $i$,
 $$
 \mathbb{E}_N(\mathbb{E} g-g(\omega_i))^2
 =\mathbb{E}(\mathbb{E} g-g)^2 = \mathbb{V}(g).
 $$
Thus,
$$
 \mathbb{E}_N (I_1) = n\mathbb{V}(g).
$$
For $I_2$, note that
$$
\mathbb E_N g(\omega_i)=\mathbb E_N g(\omega_j) =\mathbb E(g)
$$
and, for $i\neq j$,
\begin{equation}\label{key}
\begin{aligned}
\mathbb {E}_N ( g(\omega_i)g(\omega_j)) &=
\int_{G\times G\times\ldots\times G}
g(\omega_i) g(\omega_j) \lambda(\omega_1) \lambda(\omega_2)\ldots \lambda(\omega_N)
d\omega_1d\omega_2\cdots d\omega_N \\
&= \int_{G\times G} g(\omega_i) g(\omega_j) \lambda(\omega_1)
\lambda(\omega_1) \lambda(\omega_2)
d\omega_1d\omega_2 \\
&= \mathbb {E}_N (
g(\omega_i))\mathbb {E}_n(g(\omega_j))
=[\mathbb E(g)]^2.
\end{aligned}
\end{equation}
Thus
 \begin{equation}
\mathbb{E}_N (I_2) = \mathbb{E}_N \left( \sum_{i\neq j}^N((\mathbb{E}g)^2-\mathbb
  E(g)(\mathbb E(g(\omega_i))+ \mathbb E(g(\omega_j)))+\mathbb E(g(\omega_i)g(\omega_j))) \right)=0.
  \end{equation}
Consequently, there exist the following two formulas for $\displaystyle \mathbb{E}_N\left(\mathbb{E} g-
      \frac1N\sum_{i=1}^Ng(\omega_i)\right)^2$:
 \begin{equation}
 \mathbb{E}_N\left(\mathbb{E} g-
      \frac1N\sum_{i=1}^Ng(\omega_i)\right)^2 = \frac{1}{N^2}\mathbb{E}_N (I_1)
      =
           \left\{
             \begin{aligned}
            \frac{1}{N}\mathbb{E}\big ((\mathbb{E} g-g)^2\big )\\
            \frac{1}{N}(\mathbb{E}(g^2) - (\mathbb{E} g)^2).
            \end{aligned}
    \right.
  \end{equation}
Based on the first formula above, since
$$
|g(\omega) - \mathbb{E} g|=|\int_G \big (g(\omega) - g(\tilde \omega) \big )\lambda(\tilde \omega)d\tilde \omega|\le \sup_{\omega, \omega'\in G} |g(\omega) - g(\omega')|,
$$
it holds that
 \begin{equation}
 \mathbb{E}_N\left(\mathbb{E} g-
      {1\over N }\sum_{i=1}^Ng(\omega_i)\right)^2
            \le\frac{1}{N} \sup_{\omega, \omega\in G} |g(\omega) - g(\omega')|^2.
  \end{equation}
Due to the second formula above,
 \begin{equation}
    \label{eqn}
 \mathbb{E}_N\left(\mathbb{E} g-
      \frac1N\sum_{i=1}^Ng(\omega_i)\right)^2
            \le\frac{1}{N} \mathbb E(g^2)\le\frac{1}{N}\|g\|^2_{L^\infty}
  \end{equation}
which completes the proof.
\end{proof}

We can also generalize Lemma \ref{MC} to Hilbert spaces following a similar analysis.

\begin{lemma}
For any $g: G\rightarrow H$ where $H$ is a Hilbert space, we have
  \begin{equation}
  \begin{split}
    \mathbb{E}_n\Big(\|\mathbb{E}g-\frac1n\sum_{i=1}^n
    g(\omega_i)\|_H^2\Big)
    &=\frac{1}{n}\mathbb{V}(g)=\frac{1}{n}\Big(\mathbb{E}(\|g\|_H^2) - \big (\mathbb{E}(\|g\|_H)\big )^2\Big)
    \\
    &=
    \left\{
             \begin{aligned}
    \frac{1}{n}\mathbb{V}(g)
   & \le\frac{1}{n} \sup_{\omega, \omega'\in G} \|g(\omega) - g(\omega')\|_H^2
    \\
\frac{1}{n}\Big(\mathbb{E}(\|g\|_H^2) - \big (\mathbb{E}(\|g\|_H)\big )^2\Big)
&\le\frac{1}{n} \mathbb E(\|g\|_H^2),
\end{aligned}
\right.
\end{split}
  \end{equation}
\end{lemma}

\begin{proof}%[Proof of Lemma \ref{MC}]
First note that
 \begin{equation}
    \label{eqn}
    \begin{aligned}
\left\|\mathbb{E} g-\frac1N\sum_{i=1}^Ng(\omega_i)\right\|_H^2
  &
=\frac{1}{N^2} \left\|\sum_{i=1}^N(\mathbb{E} g-g(\omega_i))\right\|_H^2
  \\
  &=\frac{1}{N^2} \sum_{i,j=1}^N\left(\mathbb{E} g-g(\omega_i), \mathbb{E} g-g(\omega_j)\right)
  \\
  &=\frac{I_1}{N^2} +\frac{I_2}{N^2}.
    \end{aligned}
  \end{equation}
with
\begin{equation}
I_1= \sum_{i=1}^N\|\mathbb{E} g-g(\omega_i)\|_H^2,\quad I_2=\sum_{i\neq  j}^N\left(\mathbb{E} g-g(\omega_i), \mathbb{E} g-g(\omega_j)\right).
\end{equation}
Consider $I_1$, for any $i$,
 $$
 \mathbb{E}_N(\|\mathbb{E} g-g(\omega_i)\|_H^2)
 =\mathbb{E}(\|\mathbb{E} g-g\|_H^2) = \mathbb{V}(g).
 $$
Thus,
$$
 \mathbb{E}_N (I_1) = n\mathbb{V}(g).
$$
For $I_2$, note that
$$
\mathbb E_N \|g(\omega_i)\|_H=\mathbb E_N \|g(\omega_j)\|_H =\mathbb E(\|g\|_H)
$$
and, for $i\neq j$,
\begin{equation}\label{key}
\begin{aligned}
\mathbb {E}_N ( g(\omega_i), g(\omega_j)) &=
\int_{G\times G\times\ldots\times G}
g(\omega_j) g(\omega_j) \lambda(\omega_1) \lambda(\omega_2)\ldots \lambda(\omega_N)
d\omega_1d\omega_2\cdots d\omega_N \\
&= \int_{G\times G} (g(\omega_j) , g(\omega_j) )
\lambda(\omega_i) \lambda(\omega_j)
d\omega_id\omega_j \\
&=\left( \mathbb {E}_N (
g(\omega)) , \mathbb {E}_N(g(\omega))\right)
=\|\mathbb E(g)\|_H^2.
\end{aligned}
\end{equation}
Thus
 \begin{equation}
\mathbb{E}_N (I_2) = \mathbb{E}_N \left( \sum_{i\neq j}^N\big(\|\mathbb{E}g\|_H^2-
(\mathbb  E(g), \mathbb E(g(\omega_i))+ \mathbb E(g(\omega_j)))
+ (g(\omega_i), g(\omega_j))\big) \right)=0.
  \end{equation}
Consequently, there exist the following two formulas for $\displaystyle \mathbb{E}_N\left\|\mathbb{E} g-
      \frac1N\sum_{i=1}^Ng(\omega_i)\right\|_H^2$:
 \begin{equation}
 \mathbb{E}_N\left\|\mathbb{E} g-
      \frac1N\sum_{i=1}^Ng(\omega_i)\right\|_H^2 = \frac{1}{N^2}\mathbb{E}_N (I_1)
      =
           \left\{
             \begin{aligned}
            \frac{1}{N}\mathbb{E}\big (\|\mathbb{E} g-g\|_H^2\big )\\
            \frac{1}{N}(\mathbb{E}(\|g\|_H^2) - \|\mathbb{E} g\|_H^2).
            \end{aligned}
    \right.
  \end{equation}
Based on the first formula above, since
$$
|g(\omega) - \mathbb{E} g|=|\int_G \big (g(\omega) - g(\tilde \omega) \big )\lambda(\tilde \omega)d\tilde \omega|\le \sup_{\omega, \omega'\in G} |g(\omega) - g(\omega')|,
$$
it holds that
 \begin{equation}
 \mathbb{E}_N\left\|\mathbb{E} g-
      {1\over N }\sum_{i=1}^Ng(\omega_i)\right\|_H^2
            \le\frac{1}{N} \sup_{\omega, \omega'\in G} \|g(\omega) - g(\omega')\|_H^2.
  \end{equation}
Due to the second formula above,
 \begin{equation}
    \label{eqn}
 \mathbb{E}_N\left\|\mathbb{E} g-
      \frac1N\sum_{i=1}^Ng(\omega_i)\right\|_H^2
            \le\frac{1}{N} \mathbb E(\|g\|_H^2),
  \end{equation}
which completes the proof.
\end{proof}

%\input{6DL/MonteCarloProbability}
\begin{lemma}\label{lem:sample}
\textup{[Monte Carlo Sampling]}
	Consider
	\begin{equation}
	\label{uv}
	u(x)=\int_{G}g(x,\theta)\rho(\theta)d\theta   = \mathbb E (g)
	\end{equation}
	with $0\le \rho(\theta)\in L^1(G)$. For any $N\ge 1$, there exist $\theta_i^*\in G$ such that
	$$
	\|u-u_N\|_{L^2(\Omega)}^2
	\le\frac{1}{N}
	\int_G \|g(\cdot,\theta)\|_{L^2(\Omega)}^2\rho(\theta)d\theta = {\|\rho\|_{L^1(G)}\over N}\mathbb E (\|g(\cdot,\theta)\|_{L^2(\Omega)}^2)
	$$
	where
	$
	\|g(\cdot,\theta)\|_{L^2(\Omega)}^2 = \int_{\Omega} [g(x,\theta)]^2 d\mu(x),
	$
	\begin{equation}\label{fndef}
	u_N(x)=\frac{\|\rho\|_{L^1(G)}}{N}\sum_{i=1}^N g(x,\theta_i^*).
	\end{equation}

Similarly, if $g(\cdot, \theta)\in H^m(\Omega)$, for any $N\ge 1$, there exist $\theta_i^*\in G_i$ with $f_N$ given in \eqref{fndef} such that
	\begin{equation}\label{eq:hm}
	\|u-u_N\|_{H^m(\Omega)}^2
	\le
	\int_G  \|g(\cdot,\theta)\|_{H^m(\Omega)}^2\rho(\theta)d\theta
	=\frac{\|\rho\|_{L^1(G)}}{N} \mathbb E (\|g(\cdot,\theta)\|_{H^m(\Omega)}^2).
	\end{equation}
\end{lemma}

\iffalse
\begin{proof}
Note that
\begin{equation}
\label{uv}
u(x) = \|\rho\|_{L^1(G)}\mathbb E (g).
\end{equation}
By Lemma \ref{MC},
$$
\mathbb {E}_n\left(\bigg(\mathbb E(g(x,\cdot))
-\frac{1}{N}\sum_{i=1}^N g(x,\theta_i))\bigg)^2
\right)\le {1\over N} \mathbb E (g^2).
$$
By taking integration w.r.t. $x$ on both sides, we get
$$
\mathbb {E}_n\left(h(\theta_1,\theta_2, \cdots, \theta_N)
\right)\le {1\over N} \mathbb {E} \Big(\int_{\Omega} g^2 d\mu(x)\Big),
$$
where
$$
h(\theta_1,\theta_2, \cdots, \theta_N) =  \int_{\Omega} \bigg(\mathbb E(g(x,\cdot))
-\frac{1}{N}\sum_{i=1}^N g(x,\theta_i))\bigg)^2 d\mu(x).
$$
Since $\mathbb {E}_N (1) = 1$ and $\mathbb {E}_N (h) \le {1\over N} \mathbb {E} \Big(\int_{\Omega} g^2 d\mu(x)\Big)$, there exist $\theta_i^* \in G$ such that
$$
h(\theta_1^*, \theta_2^*, \cdots, \theta_N^*) \le  {1\over N}  \int_{\Omega} \mathbb {E} (g^2) d\mu(x).
$$
%Otherwise, $\mathbb {E}_n\left(h \right) >  {1\over n} \mathbb {E} \Big(\int_{\Omega} g^2 d\mu(x)\Big)$   if $h(\theta_1,\theta_2, \cdots, \theta_n) > {1\over n}  \int_{\Omega} \mathbb {E} (g^2)) d\mu(x)$.
This implies that
$$
	\mathbb{E}_n\|u-u_N\|_{L^2(\Omega)}^2
	\le\frac{\|\rho\|_{L^1(G)}}{N}
	\int_G \|g(\cdot,\theta)\|_{L^2(\Omega)}^2\lambda(\theta)d\theta.
	$$
	The proof for \eqref{eq:hm} is similar to the above analysis for the $L^2$-error analysis, which completes the proof.
\end{proof}
\fi

We also have a more general version of the above lemma.
\begin{lemma}\label{lem:sampleHk}
	Let
	\begin{equation} \label{uint}
	u(x)=\int_{G}g(x,\theta)\lambda(\theta)d\theta  = \mathbb E (g)
	\end{equation}
	with $\|\lambda(\theta)\|_{L^1(\Theta)}=1$.
	For any $N\ge 1$, there exist $\theta_i^*\in G$ such that
	$$
	\|u-u_N\|_{H^m(\Omega)}^2
	\le
	\int_G  \|g(\cdot,\theta)\|_{H^m(\Omega)}^2\lambda(\theta)d\theta
	=\frac{1}{N} \mathbb E (\|g(\cdot,\theta)\|_{H^m(\Omega)}^2)
	$$
	where
	$$
	u_N(x)=\frac{1}{N}\sum_{i=1}^N g(x,\theta_i^*)
	$$
	In particular, if
	\begin{equation}
	\label{eq:4}
	|D^\alpha g(x,\theta)|\le C, \quad\forall x, \theta, |\alpha|\le m
	\end{equation}
	Then
	$$
	\|u-u_N\|_{H^m(\Omega)}
	\le
	\begin{pmatrix}
	m+d\\
	m
	\end{pmatrix}^{1/2}
	|\Omega|^{1/2}
	N^{-1/2}.
	$$
\end{lemma}


\section*{Dictionary Learning}
Adapt the dict. to signal charact.$(\mathbf{U}^\star, \mathbf{Z}^\star) \in \argmin_\mathbf{U,Z} \| \mathbf{X} - \mathbf{U}\mathbf{Z} \|_F^2$ not jointly convex (just 1 arg)

\textbf{Matrix Factorization by Iter Greedy Minim.}
\begin{inparaenum}[\color{gray} 1.]
  \item Coding step: $\mathbf{Z}^{t+1} \in \argmin_\mathbf{Z} \| \mathbf{X} - \mathbf{U}^t \mathbf{Z} \|_F^2$ subj. to $\mathbf{Z}$ being sparse ($\mathbf{z}_n^{t+1}\in \argmin_\mathbf{z}\|\mathbf{z}\|_0$ s.t. \\ $\|\mathbf{x}_n - \mathbf{U}^t\mathbf{z}\|_2 \le \sigma \|\mathbf{x}_n\|_2$)
  \item Init: random, samples from $X$ or fixed overcomplete dictionary.
  \item Dict update step: $\mathbf{U}^{t+1} \in \argmin_\mathbf{U} \| \mathbf{X} - \mathbf{UZ}^{t+1} \|_F^2$, subj. to $\forall l\in [L]:\|\mathbf{u}_l\|_2 = 1$ One at a time: set $\mathbf{U} = [\mathbf{u}_1^t\cdots \mathbf{u}_l\cdots \mathbf{u}_L^t]$ (fix all except $\mathbf{u}_l$), isolate $\mathbf{R}_l^t$ (residual due to atom $\mathbf{u}_l$), find $\mathbf{u}_l^*$ that minimizes $\mathbf{R}_l^t$ s.t. $||\mathbf{u}_l^*||_2=1$: $\min_{u_l}\|\mathbf{R}_l^t - \mathbf{u}_l(\mathbf{z}_l^{t+1})^\top\|_F^2$ using SVD (first left-singular vector of $\mathbf{R}_l^t$).
\end{inparaenum}


\section{Projectile (vacuum, gravity)}
Associated external model: \texttt{projectile.py}

Solves the projectile motion equations
\begin{equation}
  x = x_0 + v_{x,0} t,
\end{equation}
\begin{equation}
  y = y_0 + v_{y,0} t + \frac{a}{2} t^2,
\end{equation}
with the following inputs
\begin{itemize}
  \item $x_0$, or \texttt{x0}, initial horizontal position,
  \item $y_0$, or \texttt{y0}, initial vertical position,
  \item $v_0$, or \texttt{v0}, initial speed (scalar),
  \item $\theta$, or \texttt{ang}, angle with respect to horizontal plane,
\end{itemize}
and following responses:
\begin{itemize}
  \item $r$, or \texttt{r}, the horizontal distance traveled before hitting $y=0$,
  \item $x$, or \texttt{x}, the time-dependent horizontal position,
  \item $y$, or \texttt{y}, the time-dependent vertical position,
  \item $t$, or \texttt{time}, the series of time steps taken.
\end{itemize}
The simulation takes 10 equally spaced time steps from 0 to 1 second, inclusive, and returns all four values
as vector quantities.

\subsection{Grid, $x_0,y_0$}
If a Grid sampling strategy is used and the following distributions are applied to $x_0$ and $y_0$, with three
samples equally spaced on the CDF between 0.01 and 0.99 for each input, the following are some of the samples obtained.
\begin{itemize}
  \item $x_0$ is distributed normally with mean 0 and standard deviation 1,
  \item $y_0$ is distributed normally with mean 1 and standard deviation 0.2.
\end{itemize}

\begin{table}[h!]
  \centering
  \begin{tabular}{c c|c|c c}
    $x_0$ & $y_0$ & $t$ & $x$ & $y$ \\ \hline
    -2.32634787404 & 0.53473045192 & 0 & -2.32634787404 & 0.534730425192 \\
                   &               & $1/3$ & -2.09064561365 & 0.225988241143 \\
                   &               & 1 & -1.61924109285 & -3.65816279362 \\ \hline
    1.0            & 0.0           & 0 & 1.0 & 0.0 \\
                   &               & $1/3$ & 0.235702260396 & 0.691257815951 \\
                   &               & 1 & 0.707106781187 & -3.19289321881
  \end{tabular}
\end{table}


    \filetitle{dot}{Gross rate of growth pseudofunction}{modellang/dot}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
dot(Expr)
dot(Expr,K)
\end{verbatim}

\paragraph{Description}\label{description}

If the input argument \texttt{k} is not specified, this pseudofunction
expands to

\begin{verbatim}
((Expr)/(Expr{-1}))
\end{verbatim}

If the input argument \texttt{k} is specified, it expands to

\begin{verbatim}
((Expr)/(Expr{k}))
\end{verbatim}

The two derived expressions, \texttt{Expr\{-1\}} and \texttt{Expr\{k\}},
are based on \texttt{Expr}, and have all its time subscripts shifted by
--1 or by \texttt{k} periods, respectively.

\paragraph{Example}\label{example}

The following two lines

\begin{verbatim}
dot(Z)
dot(X+Y,-2)
\end{verbatim}

will expand to

\begin{verbatim}
((Z)/(Z{-1}))
((X+Y)/(X{-2}+Y{-2}))
\end{verbatim}


\section*{Vectors and Matrices}

\subsection*{Vectors}
\begin{itemize}
  \item [vector:] $\vectorarrow*{v}=\begin{pmatrix}v_1\\v_2\\ \vdots \end{pmatrix}$
  \item [transpose:] $\vectorarrow*{v}^\transpose=(v_1 v_2 \dotsc)$
  \item [inner product:] $\vectorarrow*{a} \dotproduct \vectorarrow*{b} =
    \sum a_ib_i$
  \item [Hermitian conjugate:] $\vectorarrow*{V}^\dagger =
    \left(\vectorarrow*{V}^*\right)^\transpose =
    \left(\vectorarrow*{V}^\transpose\right)^*$
  \item [tensor product:] $%
    \vectorarrow*{a} \otimes \vectorarrow*{b} = %
    \begin{pmatrix}a_1\\a_2\end{pmatrix}%
    \otimes %
    \begin{pmatrix}b_1\\b_2\end{pmatrix}%
    = \begin{pmatrix}a_1\begin{pmatrix}b_1\\b_2\end{pmatrix}
\\a_2\begin{pmatrix}b_1\\b_2\end{pmatrix}\end{pmatrix}
    = \begin{pmatrix}
      a_1b_1\\a_1b_2\\a_2b_1\\a_2b_2\end{pmatrix}
    $
  \item [ket:] $\ket{\psi} = (z_1, z_2, \dotsc)^\transpose$
  \item [bra:] $\bra{\psi} = (z^*_1, z^*_2, \dotsc)$
\end{itemize}

\subsection*{}