\section{Points, products and orientation}
In this first section, we define our point representation, we explain how the dot and cross products we saw in 2D translate into 3D, and we show how a combination of the two, the \emph{mixed product}, can help us define a 3D analog of the \lstinline|orient()| function.

\subsection{Point representation}
We will define points and vectors in space by their coordinates $(x,y,z)$: their positions along three perpendicular axes.

\centerFig{p3-0}

As we did in 2D, we start with some basic operators.
\begin{lstlisting}
typedef double T;
struct p3 {
    T x,y,z;

    // Basic vector operations
    p3 operator+(p3 p) {return {x+p.x, y+p.y, z+p.z};}
    p3 operator-(p3 p) {return {x-p.x, y-p.y, z-p.z};}
    p3 operator*(T d) {return {x*d, y*d, z*d};}
    p3 operator/(T d) {return {x/d, y/d, z/d};} // only for floating-point

    // Some comparators
    bool operator==(p3 p) {return tie(x,y,z) == tie(p.x,p.y,p.z);}
    bool operator!=(p3 p) {return !operator==(p);}
};
\end{lstlisting}
Choosing the scalar type \lstinline|T| is done the same way as in 2D, see \ref{ss:point-representation} for our remarks on that.

For convenience, we also define a zero vector $\vv{0}=(0,0,0)$:
\begin{lstlisting}
p3 zero{0,0,0};
\end{lstlisting}

\subsection{Dot product}
The dot product is exactly the same as in 2D. It is defined as \[\dotv{v}{w} = \normv{v}\normv{w} \cos\theta\] where $\normv{v}$ and $\normv{w}$ are the lengths of the vectors and $\theta$ is amplitude of the angle between $\vv{v}$ and $\vv{w}$.
%This amplitude $\theta$ is measured on the plane that contains both $\vv{v}$ and $\vv{w}$.
So in other words, the 3D dot product of $\vv{v}$ and $\vv{w}$ is equal to the 2D dot product they would have on a plane that contains them both.

We recall the possible cases for the sign of the dot product:
\centerFig{products0}

%Since 3D geometry requires a very good mastery of the properties of dot and cross product, we recall all of them here and add a few:
We recall some properties of dot product and add a few:
\begin{itemize}
%\item $\dotv{v}{w}$ is a measure of how parallel $\vv{v}$ and $\vv{w}$ are;
\item symmetry: $\dotv{v}{w} = \dotv{w}{v}$;
\item linearity: $(\vv{v_1}+\vv{v_2})\cdot\vv{w} = \dotv{v_1}{w} + \dotv{v_2}{w}$ (also on the right);
%\vv{v}\cdot(\vv{w_1}+\vv{w_2}) &= \dotv{v}{w_1} + \dotv{v}{w_2}
%\end{align*}
\item $\dotv{v}{w} = 0$ iff $\vv{v}$ and $\vv{w}$ are perpendicular;
\item $\dotv{v}{w}$ stays constant if $\vv{w}$ moves perpendicular to $\vv{v}$.
%\item in physics, if $\vv{F}$ is a force applied along displacement $\vv{s}$, the work done is $W=\dotv{F}{s}$.
\end{itemize}

Like in 2D, dot product is very simple to implement:
\begin{lstlisting}
T operator|(p3 v, p3 w) {return v.x*w.x + v.y*w.y + v.z*w.z;}
\end{lstlisting}

Since 3D geometry uses dot and cross product a lot, to shorten notations we will be using operator \lstinline{|} for dot product and operator \lstinline{*} for cross product. We chose them for these mostly arbitrary reasons:
\begin{itemize}
\item \lstinline{|} has a lower precedence than \lstinline|*| which is desirable, it kind of looks like a ``parallel'' operator, and since it most often has to be parenthesized (e.g. \ttt{(v|w) == 0}), it can be a bit reminiscent of the inner product notation $\langle v,w \rangle$;
\item \lstinline|*| is the closest thing to a cross in overloadable R++ operators, and the compiler doesn't produce a warning if you don't paranthesize \lstinline|b*c| in expressions such as \lstinline{a|b*c}, which we will use a lot.
\end{itemize}

We define the usual \lstinline{sq()} and \lstinline{abs()} based on dot product and add a \lstinline|unit()| function that makes the norm of a vector equal to 1 while preserving its direction.
\begin{lstlisting}
T sq(p3 v) {return v|v;}
double abs(p3 v) {return sqrt(sq(v));}
p3 unit(p3 v) {return v/abs(v);}
\end{lstlisting}

Like in 2D, we can also use dot product to find the amplitude in $[0,\pi]$ of the angle between vectors $\vv{v}$ and $\vv{w}$, see secton~\ref{ss:dot} for more details.
\begin{lstlisting}
double angle(p3 v, p3 w) {
    double cosTheta = (v|w) / abs(v) / abs(w);
    return acos(max(-1.0, min(1.0, cosTheta)));
}
\end{lstlisting}

\subsection{Cross product}
While the cross product in 2D is a scalar, in 3D it is a vector. If $\vv{v}$ and $\vv{w}$ are parallel, $\crossv{v}{w} = \vv{0}$, and otherwise it is defined as
\[\crossv{v}{w} = (\normv{v}\normv{w} \sin\theta)\vv{n}\]
where $\normv{v}$ and $\normv{w}$ are the lengths of the vectors, $\theta$ is amplitude of the angle between $\vv{v}$ and $\vv{w}$, and $\vv{n}$ is a unit vector perpendicular to both $\vv{v}$ and $\vv{w}$ chosen using the right-hand rule. Note that the norm of the 3D cross product is equal to the absolute value of the 2D cross product.

\centerFig{p3-1}

The right-hand rule says this: if you take your right hand, align your thumb with $\vv{v}$ and your extended index $\vv{w}$, and fold your middle finger at a $90\degree$ angle, then it will point in the direction of $\crossv{v}{w}$. Another way to express it is to say that if you draw $\vv{v}$ and $\vv{w}$ on a sheet of paper and look at the sign of their 2D cross product, if it is positive $\crossv{v}{w}$ will point up from the sheet, and if it is negative $\crossv{v}{w}$ will point down through the sheet.

\centerFig{p3-2}

We summarize some key properties of the cross product:
\begin{itemize}
\item anti-symmetry: $\crossv{v}{w} = -\crossv{w}{v}$;
\item linearity: $(\vv{v_1}+\vv{v_2})\times\vv{w} = \crossv{v_1}{w} + \crossv{v_2}{w}$ (also on the right);
\item $\crossv{v}{w}$ is perpendicular to both $\vv{v}$ and $\vv{w}$;\footnote{\label{f:if-not-zero}if $\crossv{v}{w} \neq \vv{0}$}
\item $\crossv{v}{w}$ is perpendicular to the plane containing $\vv{v}$ and $\vv{w}$;\cref{f:if-not-zero}
\item $\crossv{v}{w} = \vv{0}$ iff $\vv{v}$ and $\vv{w}$ are parallel;
\item $\crossv{v}{w}$ stays constant if $\vv{w}$ moves parallel to $\vv{v}$.
%\item in physics, if $\vv{F}$ is a force applied with a lever arm vector $\vv{r}$, the resulting torque is $\vv{\tau} = \crossv{r}{F}$.
\end{itemize}

Among those, the one which we will use most often is the fact that it is perpendicular to $\vv{v}$ and $\vv{w}$.

The cross product can be computed this way:
\begin{lstlisting}
p3 operator*(p3 v, p3 w) {
    return {v.y*w.z - v.z*w.y,
            v.z*w.x - v.x*w.z,
            v.x*w.y - v.y*w.x};
}
\end{lstlisting}
Indeed, it is easy to check that this vector is perpendicular to both $\vv{v}$ and $\vv{w}$ using dot product. Here it is for $\vv{v}$:
\begin{align*}
(\crossv{v}{w})\cdot \vv{v}
&= (v_yw_z - v_zw_y, v_zw_x - v_xw_z, v_xw_y - v_yw_x) \cdot (v_x,v_y,v_z) \\
&= v_xv_yw_z - v_xv_zw_y + v_yv_zw_x - v_yv_xw_z + v_zv_xw_z - v_zv_yw_x \\
&= 0
\end{align*}

Note that the $z$-coordinate is the same expression as the 2D cross product: indeed, it is the value of the 2D cross product if $\vv{v}$ and $\vv{w}$ are projected onto plane $z=0$.

\subsection{Mixed product and orientation}\label{ss:mixed-orient}
A very useful combination of dot product and cross product is the \emph{mixed product}. We define the mixed product of three vectors $\vv{u}$, $\vv{v}$ and $\vv{w}$ as
\[(\crossv{u}{v})\cdot\vv{w}\]

Let $\Pi$ be the plane containing $\vv{u}$ and $\vv{v}$. We know that $\vv{n}=\crossv{u}{v}$ is perpendicular to $\Pi$, and $\dotv{n}{w}$ will be positive if the angle between $\vv{n}$ and $\vv{w}$ is less than $90\degree$. This will happen if $\vv{w}$ points to the same side of $\Pi$ as $\vv{n}$, while when $\dotv{n}{w}$ is negative $\vv{w}$ will point to the opposite side.

The two cases are illustrated in the drawings below. The plane containing $\vv{u}$ and $\vv{v}$ is viewed from the side:
\centerFig{p3-3}

Note that this is similar to how the 2D cross product $\crossv{v}{w}$ tells us to which side of to line containing $\vv{v}$ vector $\vv{w}$ points. So, we similarly define an \lstinline|orient()| function based on it:
\[\orient(P,Q,R,S) = \left(\crossv{PQ}{PR}\right)\cdot\vv{PS}\]
It is positive if $S$ is on the side of plane $PQR$ in the direction of $\crossv{PQ}{PR}$, negative if $S$ is on the other side, and zero if $S$ is on the plane.

%To simplify the discussion, we will say that $S$ is on the positive side of $PQR$ when $\orient(P,Q,R,S)>0$, and on its negative side when $\orient(P,Q,R,S)<0$.

\centerFig{p3-4}

This $\orient()$ function has several very nice properties. First, it stays the same if we swap any three arguments in a circular way: for example let's take $P,Q,S$, then $\orient(P,Q,R,S) = \orient(Q,S,R,P)$. On the other hand, swapping any two arguments changes its sign: for example, $\orient(P,Q,R,S) = -\orient(P,S,R,Q)$.

\newcommand{\mixedPropsExoQ}{
    Show that those properties also apply to the mixed product. For example, $(\crossv{u}{v})\cdot\vv{w} = (\crossv{v}{w})\cdot\vv{u}$ and $(\crossv{u}{v})\cdot\vv{w} = -(\crossv{u}{w})\cdot\vv{v}$.
}
\newcommand{\mixedPropsExoA}{
    This can be derived from the properties of $\orient(P,Q,R,S)$ by setting $P=\vv{0}$, $Q=\vv{u}$, $R=\vv{v}$ and $S=\vv{w}$. Indeed, in this case
    \begin{align*}
    \orient(P,Q,R,S)
    &= \left(\crossv{PQ}{PR}\right)\cdot\vv{PS} \\
    &= \left[\left(\vv{u}-\vv{0}\right)\times\left(\vv{v}-\vv{0}\right)\right]\cdot\left(\vv{w}-\vv{0}\right) \\
    &= (\crossv{u}{v})\cdot\vv{w}
    \end{align*}
}
\exoWithSolution{\protect\mixedPropsExoQ}{\protect\mixedPropsExoA}{mixed-props}

Earlier, we implicitly assumed that $P,Q,R$ were not collinear, but in general $\orient(P,Q,R,S)$ is zero if and only if $P,Q,R,S$ are coplanar, so when any three points are collinear it is always zero. We can also say that it is nonzero if and only if lines $PQ$ and $RS$ are skew, that is, neither intersecting nor parallel.

Finally, $|\!\orient(P,Q,R,S)|$ is equal to six times the volume of tetrahedron $PQRS$.% (just like in 2D $|\orient(P,Q,R)|$ is equal to two times the area of triangle $PQR$).

It is implemented by simply writing down the definition:
\begin{lstlisting}
T orient(p3 p, p3 q, p3 r, p3 s) {return (q-p)*(r-p)|(s-p);}
\end{lstlisting}

\newcommand{\skewExpressionExoStatement}{
    A convenient way to check whether two lines $PQ$ and $RS$ are skew is to check whether $\left(\crossv{PQ}{RS}\right)\cdot\vv{PR} \neq 0$: in fact you can replace $PR$ by any vector going from $PQ$ to $RS$.

    Using the properties of dot product, cross product and $\orient()$, prove that
    \[\left(\crossv{PQ}{RS}\right)\cdot\vv{PR} = -\orient(P,Q,R,S)\]
}
\newcommand{\skewExpressionExoSol}{
    Note that for any vectors $\vv{v}$ and $\vv{w}$ we have $(\crossv{v}{w})\cdot\vv{w} = 0$. This is because $\crossv{v}{w}$ is perpendicular to $\vv{w}$ and the dot product is zero for perpendicular vectors.

    We develop:
    \begin{align*}
        \left(\crossv{PQ}{RS}\right)\cdot\vv{PR}
        &= \left(\vv{PQ}\times\left(\vv{PS}-\vv{PR}\right)\right)\cdot\vv{PR} \\
        &= \left(\crossv{PQ}{PS} - \crossv{PQ}{PR}\right)\cdot\vv{PR} \\
        &= \left(\crossv{PQ}{PS}\right)\cdot\vv{PR} - \left(\crossv{PQ}{PR}\right)\cdot\vv{PR} \\
        &= \left(\crossv{PQ}{PS}\right)\cdot\vv{PR} \\
        &= \orient(P,Q,S,R) \\
        &= -\orient(P,Q,R,S)
    \end{align*}
}

\exoWithSolution{\protect\skewExpressionExoStatement}{\protect\skewExpressionExoSol}{skew-expression}

Let's say we have a plane $\Pi$ and a vector $\vv{n}$ perpendicular to it (a \emph{normal} to the plane).
Then an interesting variant is to replace $\vv{PS}$ by $\vv{n}$, giving the expression
\[\left(\crossv{PQ}{PR}\right)\cdot\vv{n}\]
This is equivalent to computing the 2D $\orient(P',Q',R')$ on $\Pi$, where $P',Q',R'$ are the projections of $P,Q,R$ on $\Pi$.

\centerFig{p3-5}

\begin{lstlisting}
T orientByNormal(p3 p, p3 q, p3 r, p3 n) {return (q-p)*(r-p)|n;}
\end{lstlisting}


\section{The matrix of a linear transformation}
\label{sec:matrix-of-transformation}

\begin{outcome}
  \begin{enumerate}
  \item Find the matrix corresponding to a linear transformation
    $T:\R^n\to\R^m$.
  \end{enumerate}
\end{outcome}

An important example of linear transformations are the so-called
\textbf{matrix transformations}%
\index{matrix transformation}%
\index{linear transformation!matrix transformation}.

\begin{proposition}{Matrix transformations are linear transformations}{matrix-are-linear}
  Let $A$ be an $m\times n$-matrix, and consider the vector function
  $T:\R^n\to \R^m$ defined by $T(\vect{v}) = A\vect{v}$. Then $T$
  is a linear transformation.
\end{proposition}

\begin{proof}
  This follows from the laws of matrix multiplication. Namely, by the
  distributive law, we have
  $A(\vect{v}+\vect{w}) = A\vect{v} + A\vect{w}$, showing that $T$
  preserves addition. And by the compatibility of matrix multiplication
  and scalar multiplication, we have $A(k\vect{v}) = k(A\vect{v})$,
  showing that $T$ preserves scalar multiplication.
\end{proof}

In fact, matrix transformations are not just an example of linear
transformations, but they are essentially the {\em only} example. One
of the central theorems in linear algebra is that all linear
transformations $T:\R^n\to\R^m$ are in fact matrix transformations.
Therefore, a matrix can be regarded as a notation for a linear
transformation, and vice versa. This is the subject of the following
theorem.

\begin{theorem}{Linear transformations are matrix transformations}{matrix-of-linear-transformation}
  Let $T:\R^n\to\R^m$ be any linear transformation. Then there exists
  an $m\times n$-matrix $A$%
  \index{linear transformation!matrix of!in Rn@in $\R^n$}%
  \index{matrix!of a linear transformation!in Rn@in $\R^n$} such that for all
  $\vect{v}\in\R^n$,
  \begin{equation*}
    T(\vect{v}) = A\vect{v}.
  \end{equation*}
  In other words, $T$ is a matrix transformation.
\end{theorem}

\begin{proof}
  Suppose $T:\R^n\to \R^m$ is a linear transformation and consider
  the standard basis $\set{\vect{e}_1,\ldots,\vect{e}_n}$ of $\R^n$.
  For all $i$, define $\vect{u}_i = T(\vect{e}_i)$, and let $A$ be the
  matrix that has $\vect{u}_1,\ldots,\vect{u}_n$ as its columns. We
  claim that $A$ is the desired matrix, i.e., that
  $T(\vect{v}) = A\vect{v}$ holds for all $\vect{v}\in\R^n$.

  To see this, let
  \begin{equation*}
    \vect{v} =
    \begin{mymatrix}{c} x_1 \\ \vdots \\ x_n \end{mymatrix}
  \end{equation*}
  be some arbitrary element of $\R^n$. Then
  $\vect{v} = x_1\vect{e}_1 + \ldots + x_n\vect{e}_n$, and we have:
  \begin{equation*}
    \begin{array}{rcl@{\quad}l}
      T(\vect{v})
      &=& T(x_1\vect{e}_1 + \ldots + x_n\vect{e}_n)
      \\
      &=& T(x_1\vect{e}_1) + \ldots + T(x_n\vect{e}_n)
      & \mbox{by linearity} \\
      &=& x_1T(\vect{e}_1) + \ldots + x_nT(\vect{e}_n)
      & \mbox{by linearity} \\
      &=& x_1\vect{u}_1 + \ldots + x_n\vect{u}_n
      & \mbox{by definition of $\vect{u}_i$} \\
      &=& A\vect{v}
      & \mbox{by the column method of matrix multiplication.}
    \end{array}
  \end{equation*}
\end{proof}

In summary, the matrix corresponding to the linear transformation $T$
has as its columns the vectors $T(\vect{e}_1),\ldots,T(\vect{e}_n)$,
i.e., the images of the standard basis vectors. We can visualize this
matrix as follows:
\begin{equation*}
  A=
  \begin{mymatrix}{ccc}
    | &  & | \\
    T(\vect{e}_1) & \cdots & T(\vect{e}_n) \\
    | &  & |
  \end{mymatrix}.
\end{equation*}

\begin{example}{The matrix of a linear transformation}{matrix-of-linear-transformation}
  Suppose $T:\R^3\rightarrow \R^2$ is a linear transformation where
  \begin{equation*}
    T\paren{\begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix}}
    = \begin{mymatrix}{r} 1 \\ 2 \end{mymatrix},\quad
    T\paren{\begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix}}
    = \begin{mymatrix}{r} 9 \\ -3 \end{mymatrix},\quad\mbox{and}\quad
    T\paren{\begin{mymatrix}{r} 0 \\ 0 \\ 1 \end{mymatrix}}
    = \begin{mymatrix}{r} 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  Find the matrix $A$ such that $T(\vect{v})=A\vect{v}$ for all
  $\vect{v}$.
\end{example}

\begin{solution}
  By Theorem~\ref{thm:matrix-of-linear-transformation}, the columns of
  $A$ are $T(\vect{e}_1)$, $T(\vect{e}_2)$, and $T(\vect{e}_3)$. Therefore,
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      1 & 9 & 1 \\
      2 & -3 & 1
    \end{mymatrix}.
  \end{equation*}
\end{solution}

\begin{example}{The matrix of a linear transformation}{matrix-of-linear-transformation2}
  Let $T:\R^3\to\R^2$ be the linear transformation defined by
  \begin{equation*}
    T\paren{\begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}}
    =
    \begin{mymatrix}{c} x+y \\ x+2y-z \end{mymatrix},
  \end{equation*}
  for all $x,y,z\in\R$. Find the matrix of this linear transformation.
\end{example}

\begin{solution}
  We compute the images of the standard basis vectors:
  \begin{eqnarray*}
    T(\vect{e}_1)
    &=& T\paren{\begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix}}
    ~=~ \begin{mymatrix}{c} 1 \\ 1 \end{mymatrix}, \\
    T(\vect{e}_2)
    &=& T\paren{\begin{mymatrix}{c} 0 \\ 1 \\ 0 \end{mymatrix}}
    ~=~ \begin{mymatrix}{c} 1 \\ 2 \end{mymatrix}, \\
    T(\vect{e}_3)
    &=& T\paren{\begin{mymatrix}{c} 0 \\ 0 \\ 1 \end{mymatrix}}
    ~=~ \begin{mymatrix}{c} 0 \\ -1 \end{mymatrix}.
  \end{eqnarray*}
  The matrix $A$ has $T(\vect{e}_1)$, $T(\vect{e}_2)$, and
  $T(\vect{e}_3)$ as its columns. Therefore,
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      1 & 1 & 0 \\
      1 & 2 & -1
    \end{mymatrix}.
  \end{equation*}
\end{solution}

\begin{example}{Matrix of a projection map}{projection-matrix}
  Let $\vect{u} = \begin{mymatrix}{r} 1 \\ 2 \\ 3 \end{mymatrix}$ and
  let $T: \R^3 \to \R^3$ be the projection%
  \index{projection!as a linear transformation} map defined by
  \begin{equation*}
    T(\vect{v}) = \proj_{\vect{u}}(\vect{v})
  \end{equation*}
  for all $\vect{v} \in \R^3$.
  \begin{enumerate}
  \item[(a)] Is $T$ a linear transformation?
  \item[(b)] If yes, find the matrix of $T$.
  \end{enumerate}
\end{example}

\begin{solution}
  \begin{enumerate}
  \item[(a)] Recall the formula for the projection of $\vect{v}$ onto
    $\vect{u}$:
    \begin{equation*}
      \proj_{\vect{u}}(\vect{v})
      = \frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\,\vect{u}.
    \end{equation*}
    In the situation we are interested in, $\vect{u}$ is a fixed vector,
    and $\vect{v}$ is the input to the function $T$. Given any two
    vectors $\vect{v},\vect{w}$, and using the distributive laws of the dot
    product and scalar multiplication, we have:
    \begin{equation*}
      \proj_{\vect{u}}(\vect{v}+\vect{w})
      = \frac{\vect{u}\dotprod(\vect{v}+\vect{w})}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      = \paren{\frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}
        + \frac{\vect{u}\dotprod \vect{w}}{\vect{u}\dotprod \vect{u}}}\,\vect{u}
      = \frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      + \frac{\vect{u}\dotprod \vect{w}}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      = \proj_{\vect{u}}(\vect{v})
      + \proj_{\vect{u}}(\vect{w}).
    \end{equation*}
    Therefore, the function
    $T(\vect{v}) = \proj_{\vect{u}}(\vect{v})$ preserves
    addition. Also, given any scalar $k$, we have
    \begin{equation*}
      \proj_{\vect{u}}(k\vect{v})
      = \frac{\vect{u}\dotprod (k\vect{v})}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      = \paren{k\frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}}\,\vect{u}
      = k\paren{\frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\,\vect{u}}
      = k\,\proj_{\vect{u}}(\vect{v}).
    \end{equation*}
    Therefore, the function $T$ preserves scalar multiplication. It
    follows that $T$ is a linear transformation.

  \item[(b)] To find the matrix of $T$, we must compute the images of
    the standard basis vectors $T(\vect{e}_1),\ldots,T(\vect{e}_3)$. We compute
    \begin{eqnarray*}
      T(\vect{e}_1)
      &=& \proj_{\vect{u}}(\vect{e}_1)
      ~=~ \frac{\vect{u}\dotprod\vect{e}_1}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      ~=~ \frac{1}{14}\begin{mymatrix}{r} 1 \\ 2 \\ 3 \end{mymatrix}, \\
      T(\vect{e}_2)
      &=& \proj_{\vect{u}}(\vect{e}_2)
      ~=~ \frac{\vect{u}\dotprod\vect{e}_2}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      ~=~ \frac{2}{14}\begin{mymatrix}{r} 1 \\ 2 \\ 3 \end{mymatrix}, \\
      T(\vect{e}_3)
      &=& \proj_{\vect{u}}(\vect{e}_3)
      ~=~ \frac{\vect{u}\dotprod\vect{e}_3}{\vect{u}\dotprod \vect{u}}\,\vect{u}
      ~=~ \frac{3}{14}\begin{mymatrix}{r} 1 \\ 2 \\ 3 \end{mymatrix}.
    \end{eqnarray*}
    Hence the matrix of $T$ is
    \begin{equation*}
      A = \frac{1}{14}\begin{mymatrix}{rrr}
        1 & 2 & 3 \\
        2 & 4 & 6 \\
        3 & 6 & 9
      \end{mymatrix}.
    \end{equation*}
  \end{enumerate}
\end{solution}


%---------------------------Aspect Frobenius-----------------------------
\section{Aspect Frobenius\label{s:tet-aspect-Frobenius}}

The edge matrix of the tetrahedral element is defined as
follows:
\[
T_0 = (\vec{L_0}\;\vec{L_1}\;\vec{L_2})
\]
and let $W$ be the edge matrix of the reference regular tetrahedron.
Consider the matrix that maps $W$ into $T_0$:
\[
A_0 = T_0 W^{-1}.
\]
The Frobenius norm of $A_0$ is
\[
|A_0|_F = \sqrt{\mathrm{tr}(A_0^T\, A_0)},
\]
and the Frobenius condition number is the condition number associated
with this norm.

The aspect Frobenius of the element is defined as the normalized
(equal to $1$ when the element is regular) Frobenius condition number of $A_0$.

\tetmetrictable{aspect Frobenius}%
{$1$}%                                                Dimension
{$[1,1.3]$}%                                          Acceptable range
{$[1,DBL\_MAX]$}%                                     Normal range
{$[1,DBL\_MAX]$}%                                     Full range
{$1$}%                                                Unit equilateral triangle value
{\cite{knu:00}}%                                      Reference(s)
{v\_tet\_aspect\_frobenius}%                            Verdict function name


\section{The \issorted algorithm}
\Label{sec:issorted}

Our version of the \issorted algorithm compared to the \cxx Standard
Library \cite[\S 28.7.1.5]{cxx-17-draft} has the signature
\begin{lstlisting}[style = acsl-block]

  bool is_sorted(const value_type* a, size_type n);
\end{lstlisting}

It returns \inl{true} if the given array is in increasing order, and
\inl{false} otherwise.

\FloatBarrier

\subsection{Formal specification of \issorted}

The following listing shows the acsl specification of \issorted.
%
In the contract, we use the predicate \logicref{Increasing},
which states that any array element is always less or equal to any other element right of it.
%
We'll use an easier-to-handle predicate in the implementation of \implref{issorted}.

\input{Listings/is_sorted.h.tex}

\clearpage

\subsection{Implementation of \issorted}

The implementation of \issorted is shown in the next Listing.
%
As usual, \issorted doesn't compare every array element to all that are right
to it, but only to the immediately adjacent one, which is of course
more efficient.
For this, we use the predicate \logicref{WeaklyIncreasing}
in the loop invariant of the implementation.

\input{Listings/is_sorted.c.tex}

Since our implementation uses \WeaklyIncreasing in its loop invariant, and
follows the same principle in its code, its verification is
straight-forward---except for the final reasoning that
\inl{WeaklyIncreasing(a,n)} implies \inl{Increasing(a,n)}.

We have the lemma \logicref{WeaklyIncreasingIncreasing} for that step,
which needs to be proven manually with \coq.
%
The converse lemma \logicref{IncreasingWeaklyIncreasing}
is proven automatically, but isn't actually needed to verify our
\issorted implementation.
Alternatively, we could have dragged the predicate \Increasing along the
loop, which happens to cause no particular problems in this case.

\clearpage


\chapter{Shapes}

\section{Angles}

Two intersecting lines form an angle\footnote{\url{https://en.wikipedia.org/wiki/Angle}}.

Two lines are parallel iff they don't intersect.

Two lines coincide iff the angle between them is zero.

Two lines are orthogonal iff the angle between them is a right angle.

We use a protractor\footnote{\url{https://en.wikipedia.org/wiki/Protractor}} to measure an angle in degrees.
A full circle is 360 degrees,
a half circle is 180 degrees,
a quarter circle is 90 degrees,
and so on.

A full turn is 360 degrees,
a half turn is 180 degrees,
a quarter turn is 90 degrees,
and so on.

\section*{Degrees and radians}

Degrees are convenient for manual calculation
because 360 is divisible by several small integers.

Radians\footnote{\url{https://en.wikipedia.org/wiki/Radian}} simplify formulas.%
\footnote{\url{https://en.wikipedia.org/wiki/Radian\#Advantages_of_measuring_in_radians}}
For example,
let \(c\) be the circumference of
a circular sector of angle \(a\) and radius \(r\).
If \(a\) is in radians, then \(c\) has the simple formula \(c = a \cdot r\),
but if \(a\) is in degrees, the formula becomes \(c = (a / \ang{360}) \cdot r\).

We prefer the unit that simplifies our jobs.
Engineers prefer degrees.
Mathematicians prefer radians.
Both units are widely used,
so let's learn to convert one to the other.

An angle of \(2\pi\) radians is equal to an angle of \(360\) degrees.%
\footnote{\url{https://en.wikipedia.org/wiki/Radian\#Conversion_between_radians_and_degrees}}
Both of them are equal to one full turn:
\Formula{
    \frac{d}{360} = \frac{r}{2\pi}
}

To convert \(r\) radians to \(d\) degrees:
\Formula{
    d = \frac{360}{2\pi} \cdot r
}

To convert \(d\) degrees to \(r\) radians:
\Formula{
    r = \frac{2\pi}{360} \cdot d
}

\ExerciseAnswer{Convert \ang{360} to radians?}{\(2\pi/1\) radians.}
\ExerciseAnswer{Convert \ang{180} to radians?}{\(2\pi/2\) radians.}
\ExerciseAnswer{Convert \ang{90} to radians?}{\(2\pi/4\) radians.}
\ExerciseAnswer{Convert \ang{45} to radians?}{\(2\pi/8\) radians.}

\ShowAnswers

\section{Circles}

\footnote{\url{https://en.wikipedia.org/wiki/Circular_sector}}

\section{Triangles}

A vertex is a point where two sides meet.

A triangle is called a triangle because it has three angles.
A triangle also has three vertices and three sides.

The sum of all interior angles of a triangle is 180 degrees.

An equilateral triangle is a triangle whose sides have the same length.%
\footnote{\url{https://en.wikipedia.org/wiki/Equilateral_triangle}}
Each interior angle of such triangle is \ang{60}.

\enquote{Equilateral} is the Latinate of \enquote{same-sided}.

The study of triangles is called \enquote{trigonometry}.

\paragraph{Labeling a triangle}

A capital letter labels an interior angle.
The corresponding small letter labels the side across the angle.
For example, \(a\) is the side across the angle \(A\).

\subsection*{Drawing a standard right triangle}

A right angle is 90 degrees.%
\footnote{\url{https://en.wikipedia.org/wiki/Right_angle}}

A right triangle is a triangle that has a \ang{90} interior angle.

See the footnote\footnote{\url{https://commons.wikimedia.org/wiki/File:Rtriangle.svg}}
for the picture of a standard right triangle.
Here we describe that triangle.

Let there be a triangle \(ABC\).

Let the angle \(C\) be a right angle.

Let \(a\) be the side across angle \(A\).

Let \(b\) be the side across angle \(B\).

Let \(c\) be the side across angle \(C\).
Thus, \(c\) is the hypotenuse\footnote{\url{https://en.wikipedia.org/wiki/Hypotenuse}},
the longest side of a right triangle,
the side across the right angle.

Such triangle is called a standard right triangle.

\section*{Triangle side ratios}

Consider a standard right triangle.

The sine of the angle \(A\) is \(\sin(A) = a/c\).

The cosine of the angle \(A\) is \(\cos(A) = b/c\).

The tangent of the angle \(A\) is \(\tan(A) = a/b\).

\section*{Trigonometric identities}

We can show these by drawing:
\( \sin(0) = 0 \),
and \( \cos(0) = 1 \),
and \( \sin(\pi/2) = 1 \),
and \( \cos(\pi/2) = 0 \).

The Pythagorean theorem implies \((\sin(A))^2 + (\cos(A))^2 = 1\).

The sine function has a period of \(2\pi\).
It means that \(\sin(a + 2\pi) = \sin(a)\) for every real number \(a\).

We have \(\cos A = \sin\left(\frac{\pi}{2}-A\right)\)
because \(A+B+C = \pi\) and \(C = \pi/2\) and \(A+B = \pi/2\).

See also Wikipedia\footnote{\url{https://en.wikipedia.org/wiki/Special_right_triangle}}%
\footnote{\url{https://en.wikipedia.org/wiki/Right_triangle}}.

\section*{Inverse trigonometric functions}

We can measure an angle by the ratio of the sides of the right triangle formed by the angle.
We use inverse trigonometric functions (inverse sine, inverse cosine, inverse tangent).


\section*{Appendix D: Regularization parameter derivative}
\addcontentsline{toc}{section}{Appendix D: Regularization parameter derivative}
Using the gradient method, the solution $x_{n+1}$ at iteration $n+1$ is found
from the solution $x_n$ at iteration $n$ by
\begin{eqnarray}
x_{n+1} = x - d_s -  \alpha d_n
\end{eqnarray}
with $d_s = \frac{\partial h_s(y-x)}{\partial x}$ and $d_n = \frac{\partial h_n(x)}{\partial x}$.

and the final solution $x$ should verify
\begin{eqnarray}
\frac{1}{N} \sum_{pix} (\frac{y-x}{\sigma})^2 = 1
\end{eqnarray}
where $N$ is the number of pixels. If $\alpha$ is too high, the solution will
be oversmoothed and the residual standard deviation will be greater than the
noise, when a too small $\alpha$ will generate a noisy solution, and the
residual standard deviation will be smaller than the noise.
Noting $R_{\sigma}  = \frac{1}{N} \sum_{pix} (\frac{y-x_{n+1}}{\sigma})^2 - \sigma$,
the difference between the standard deviation of the residual and the noise,
the optimal $\alpha$ at iteration $n$ is found by minimizing the functional:
\begin{eqnarray}
K(\alpha) & = & \mid  R_{\sigma} \mid \\
          & = & \mid \frac{1}{\sigma^2}\sum_{pix} (y^2  +  x_{n+1}^2 - 2 x_{n+1} y - N) \mid
\end{eqnarray}
and
\begin{eqnarray}
\frac{\partial K}{\partial \alpha} = \frac{2}{N\sigma^2} \sum_{pix}
 (\frac{\partial x_{n+1}}{\partial \alpha} x_{n+1}
   - 2y \frac{\partial x_{n+1}}{\partial \alpha})
\end{eqnarray}
when $R_{\sigma} > 0$, and the opposite otherwise.
As $\frac{\partial x_{n+1}}{\partial \alpha} = - d_n$, we have
\begin{eqnarray}
\frac{\partial K}{\partial \alpha} & = & \frac{2}{N\sigma^2} \sum_{pix} (y d_n - d_n x_{n+1})) \\
                                   & = & \frac{2d_n }{N\sigma^2}\sum_{pix}  (y - x_{n+1})
\end{eqnarray}
Then the $\alpha$ parameter is recalculated at each iteration by:
\begin{eqnarray}
\alpha_{n+1} = \alpha_n - sgn(R_{\sigma}) \frac{2d_n }{N\sigma^2} \sum_{pix} (y - x_{n+1})
\end{eqnarray}


\subsection{Identification through exogeneous variables}

Previously our structural model was:

\(Q=\alpha_1 + \beta_1 P + \epsilon_1 \)

\(Q=\alpha_2 + \beta_2 P + \epsilon_2 \)

And our reduced form:

\(P =\dfrac{\alpha_2-\alpha_1 }{\beta_1-\beta_2}+\dfrac{\epsilon_2-\epsilon_1 }{\beta_1-\beta_2}\)

\(Q =\dfrac{\alpha_2\beta_1-\alpha_1\beta_2 }{\beta_1-\beta_2}+\dfrac{\beta_1\epsilon_2 -\beta_2\epsilon_1}{\beta_1-\beta_2}\)

Or:

\(P=\pi_1 + \tau_1 \)

\(Q= \pi_2 + \tau_2\)

\subsubsection{Adding another variable}

This time we add another measured variable, \(I\).

\(Q=\alpha_1 + \beta_1 P + \theta_1 I + \epsilon_1 \)

\(Q=\alpha_2 + \beta_2 P + \theta_2 I + \epsilon_2 \)

The reduced form is now:

\(P =\dfrac{\alpha_2 -alpha_1 }{\beta_1-\beta_2}+\dfrac{\theta_2-\theta_1 }{\beta_1-\beta_2}I+\dfrac{\epsilon_2-\epsilon_1}{\beta_1-\beta_2}\)

\(Q =\dfrac{\alpha_2\beta_1-\alpha_1\beta_2 }{\beta_1-\beta_2}+\dfrac{\theta_2\beta_1-\theta_1\beta_2}{\beta_1-\beta_2}I+\dfrac{\beta_1\epsilon_2 -\beta_2\epsilon_1}{\beta_1-\beta_2}\)

Or:

\(P =\pi_{11} +\pi_{12}I + \tau_1 \)

\(Q= \pi_{21} +\pi_{22}I + \tau_2 \)

We can estimate \(\pi_1 \) and \(\pi_2 \) as \(\hat \pi_1\) and \(\hat \pi_2\) respectively.

We can now create estimators \(\hat \pi_{11}\), \(\hat \pi_{12}\), \(\hat \pi_{21}\) and \(\hat \pi_{22}\).

\subsubsection{Identification with an exogeneous variable}

We now have \(4\) estimators and \(6\) parameters, meaning that we still cannot identify the model.

\subsubsection{Partial identification}

Can we use \(\hat \pi \) to identify any of the structural parameters?

We know that:

\begin{itemize}
\item \(\pi_{11} =\dfrac{\alpha_2 -\alpha_1 }{\beta_1-\beta_2}\)
\item \(\pi_{12} =\dfrac{\theta_2-\theta_1}{\beta_1-\beta_2}\)
\item \(\pi_{21} =\dfrac{\alpha_2\beta_1-\alpha_1\beta_2}{\beta_1-\beta_2}\)
\item \(\pi_{22} =\dfrac{\theta_2\beta_1-\theta_1\beta_2}{\beta_1-\beta_2} \)
\end{itemize}

If the exogenous variable only affects one side of the equation, so \(\theta_1=0\), we have:

\begin{itemize}
\item \(\pi_{11} =\dfrac{\alpha_2 -\alpha_1 }{\beta_1-\beta_2}\)
\item \(\pi_{12} =\dfrac{\theta_2}{\beta_1-\beta_2}\)
\item \(\pi_{21} =\dfrac{\alpha_2\beta_1-\alpha_1\beta_2}{\beta_1-\beta_2}\)
\item \(\pi_{22} =\dfrac{\theta_2\beta_1}{\beta_1-\beta_2} \)
\end{itemize}

So we can see that:

\(\hat \beta_1 = \dfrac{\hat \pi_{22}}{\hat \pi_{12}}\)

This means we now have:

\begin{itemize}
\item \(\pi_{11} =\dfrac{\pi_{12}(\alpha_2 -\alpha_1 )}{\pi_{22}-\pi_{12}\beta_2}\)
\item \(\pi_{12} =\dfrac{\pi_{12}\theta_2}{\pi_{22}-\pi_{12}\beta_2}\)
\item \(\pi_{21} =\dfrac{\pi_{12}(\alpha_2\beta_1-\alpha_1\beta_2)}{\pi_{22}-\pi_{12}\beta_2}\)
\item \(\pi_{22} =\dfrac{\pi_{12}\theta_2\beta_1}{\pi_{22}-\pi_{12}\beta_2}\)
\end{itemize}

We can use this to also identify \(\alpha_1\).

\subsubsection{Complete identification}

If we have independent variables for each of the two equations, we can fully identify the model.

We will have \(6\) estimators and \(6\) parameters.

We are estimating:

\(Q=\alpha_1 + \beta_1 P + \theta_1 I + \epsilon_1 \)

\(Q=\alpha_2 + \beta_2 P + \theta_2 J + \epsilon_2 \)

\(I\) and \(J\) are essentially instrumental variables for the model.

\(I\) is an instrumental variable for demand shocks, and \(J\) is an instrumental variable for supply shocks.


\subsection{Writing an invertible matrix as a product of elementary matrices}

Recall from Algorithm~\ref{algo:matrix-inversion-algorithm} that an
$n\times n$-matrix $A$ is invertible if and only if $A$ can be carried
to the $n\times n$ identity matrix using elementary row
operations. Combining this with our discussion of elementary matrices
we see that $A$ is invertible if and only if it can be written as a
product of elementary matrices. This is the content of the following
theorem.

\begin{theorem}{Product of elementary matrices}{prod-elementary}
  Let $A$ be an $n \times n$-matrix. Then $A$ is invertible if and
  only if it can be written as a product of elementary matrices.
\end{theorem}

\begin{proof}
  If $A$ is an invertible $n\times n$-matrix, then its {\rref} is the
  $n\times n$ identity matrix $I$. By Theorem~\ref{thm:form-rua}, we
  can write $I=UA$, where $U = E_k\cdots E_2E_1$ is a product of
  elementary matrices. Then
  \begin{equation*}
    A ~=~ U^{-1} ~=~ E_1^{-1}E_2^{-1}\cdots E_k^{-1}.
  \end{equation*}
  By Theorem~\ref{thm:inverse-elementary-matrix}, if $E_i$ is an
  elementary matrix, then so is $E_i^{-1}$. Therefore, $A$ has been
  written as a product of elementary matrices. Conversely, if $A$ can
  be written as a product of elementary matrices, then $A$ is clearly
  invertible, because each elementary matrix is invertible.
\end{proof}

\begin{example}{Product of elementary matrices}{prod-elementary}
  Let $A = \begin{mymatrix}{rrr}
    0 &  1 & 0 \\
    1 &  1 & 0 \\
    0 & -2 & 1
  \end{mymatrix}$.
  Write $A$ as a product of elementary matrices.
\end{example}

\begin{solution}
  Following the process of Theorem~\ref{thm:prod-elementary}, we first
  row-reduce $A$ to its {\rref}, recording each row operation as an
  elementary matrix.
  \begin{equation*}
    \begin{mymatrix}{rrr}
      0 & 1 & 0 \\
      1 & 1 & 0 \\
      0 & -2 & 1 \\
    \end{mymatrix}
    \quad\stackrel{R_1\rowswap R_2}{\roweq}\quad
    \begin{mymatrix}{rrr}
      1 & 1 & 0 \\
      0 & 1 & 0 \\
      0 & -2 & 1 \\
    \end{mymatrix}
    \quad\mbox{with elementary matrix}\quad
    E_1 ~=~ \begin{mymatrix}{rrr}
      0 & 1 & 0 \\
      1 & 0 & 0 \\
      0 & 0 & 1
    \end{mymatrix},
  \end{equation*}
  \begin{equation*}
    \begin{mymatrix}{rrr}
      1 & 1 & 0 \\
      0 & 1 & 0 \\
      0 & -2 & 1 \\
    \end{mymatrix}
    \quad\stackrel{R_1\rowop R_1-R_2}{\roweq}\quad
    \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & -2 & 1 \\
    \end{mymatrix}
    \quad\mbox{with elementary matrix}\quad
    E_2 ~=~  \begin{mymatrix}{rrr}
      1 & -1 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & 1
    \end{mymatrix},
  \end{equation*}
  \begin{equation*}
    \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & -2 & 1 \\
    \end{mymatrix}
    \quad\stackrel{R_3\rowop R_3+2R_2}{\roweq}\quad
    \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & 1 \\
    \end{mymatrix}
    \quad\mbox{with elementary matrix}\quad
    E_3 ~=~ \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & 2 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  Notice that the {\rref} of $A$ is $I$. Hence $I = UA$ where
  $U=E_3E_2E_1$. It follows that
  $A = U^{-1} = E_1^{-1}E_2^{-1}E_3^{-1}$, and so we have succeeded in
  writing $A$ as a product of elementary matrices
  \begin{equation*}
    A
    ~=~ E_1^{-1}E_2^{-1}E_3^{-1}
    ~=~
    \begin{mymatrix}{rrr}
      0 & 1 & 0 \\
      1 & 0 & 0 \\
      0 & 0 & 1
    \end{mymatrix}
    \begin{mymatrix}{rrr}
      1 & 1 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & 1
    \end{mymatrix}
    \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & -2 & 1
    \end{mymatrix}.
  \end{equation*}
  In particular, it follows that $A$ is invertible.
\end{solution}


\documentclass[12pt]{pylatex}
\usepackage{examples}
\usepackage{caption}
\usepackage{pgfplots}

\begin{document}

\section*{Plotting Bessel functions}

\vspace{-5pt}

This simple example uses {\tt\small numpy, scipy} and {\tt\small Matplotlib} to produce a plot
of the first six Bessel functions. Two plots are shown, one created by {\tt\small Matplotlib}
and a second created by LaTeX using the plotting package {\tt\small pgfplots} and the data
exported from {\tt\small Matplotlib}.

If you are using macOS, you may need to use the {\tt\small -Ppythonw} option when running
{\tt\small pylatex.sh}. This is a known problem with macOS and {\tt\small Matplotlib}, see\
\url{https://matplotlib.org/faq/osx_framework.html}.

\vspace{-10pt}

\begin{minipage}[t]{0.60\textwidth}
\begin{python}
   import numpy as np
   import scipy.special as sp
   import matplotlib.pyplot as plt

   plt.matplotlib.rc('text', usetex = True)
   plt.matplotlib.rc('grid', linestyle = 'dotted')
   plt.matplotlib.rc('figure', figsize = (6.4,4.8)) # (width,height) inches

   x = np.linspace(0, 15, 500)

   for v in range(0, 6):
       plt.plot(x, sp.jv(v, x))

   plt.xlim((0, 15))
   plt.ylim((-0.5, 1.1))
   plt.legend(('${J}_0(x)$', '${J}_1(x)$', '${J}_2(x)$',
               '${J}_3(x)$', '${J}_4(x)$', '${J}_5(x)$'), loc = 0)
   plt.xlabel('$x$')
   plt.ylabel('${J}_n(x)$')
   plt.grid(True)
   plt.tight_layout(0.5)

   plt.savefig('example-04-fig.pdf')

   # save the data for later use by pgfplots
   np.savetxt('example-04.txt',list(zip(x,sp.jv(0,x),sp.jv(1,x),sp.jv(2,x),
                                          sp.jv(3,x),sp.jv(4,x),sp.jv(5,x))),
                               fmt="% .10e")
\end{python}
\end{minipage}

\clearpage

\begin{minipage}{\textwidth}
   \centering
   \IfFileExists{example-04-fig.pdf}%
   {\includegraphics[width=6.4in]
      {example-04-fig.pdf}}{Failed to create pdf plot.}
   \captionof{figure}{The first six Bessel functions.}
\end{minipage}

\vfill

\begin{latex}
   \IfFileExists{example-04-fig.pdf}%
   {\includegraphics[width=6.4in]
      {example-04-fig.pdf}}{Failed to create pdf plot.}
   \captionof{figure}{The first six Bessel functions.}
\end{latex}

\vfill

\clearpage

\pgfplotsset{compat=newest}
\pgfplotsset{width=0.45\textwidth,height=0.34\textwidth}

\subsection*{Using pgfplots}

\begin{minipage}[t]{\textwidth}
   \centering
   \begin{tikzpicture}
      \begin{axis}
         [xmin= 0.0,  xmax=15.0,
          ymin=-0.45, ymax=1.05,
          xlabel=$x$, ylabel=$J_n(x)$,
          grid=major, grid style={dashed,gray!30},
          legend entries = {$J_0$, $J_1$, $J_2$, $J_3$, $J_4$, $J_5$}]
          \addplot[blue]   table [x index=0, y index=1]{example-04.txt};
          \addplot[red]    table [x index=0, y index=2]{example-04.txt};
          \addplot[green]  table [x index=0, y index=3]{example-04.txt};
          \addplot[teal]   table [x index=0, y index=4]{example-04.txt};
          \addplot[orange] table [x index=0, y index=5]{example-04.txt};
          \addplot[purple] table [x index=0, y index=6]{example-04.txt};
      \end{axis}
   \end{tikzpicture}
   \captionof{figure}{The first six Bessel functions.}
\end{minipage}

\vfill

\begin{latex}
   \begin{tikzpicture} % requires \usepackage{pgfplots}
      \begin{axis}
         [xmin= 0.0,  xmax=15.0,
          ymin=-0.45, ymax=1.05,
          xlabel=$x$, ylabel=$J_n(x)$,
          grid=major, grid style={dashed,gray!30},
          legend entries = {$J_0$, $J_1$, $J_2$, $J_3$, $J_4$, $J_5$}]
          \addplot[blue]   table [x index=0, y index=1]{example-04.txt};
          \addplot[red]    table [x index=0, y index=2]{example-04.txt};
          \addplot[green]  table [x index=0, y index=3]{example-04.txt};
          \addplot[teal]   table [x index=0, y index=4]{example-04.txt};
          \addplot[orange] table [x index=0, y index=5]{example-04.txt};
          \addplot[purple] table [x index=0, y index=6]{example-04.txt};
      \end{axis}
   \end{tikzpicture}
   \captionof{figure}{The first six Bessel functions.} % requires \usepackage{caption}
\end{latex}

\end{document}


\chapter{Basic Concepts}
\section{Important/Useful Theorems}
\subsection{Stirling's Formula}
\begin{equation}
	n! \approx \sqrt{2 \pi n} n^n e^{-n}
\end{equation}
\section{Answers to Problems}

%%answer template
%\subsection{}
%%problem n.n
%
%
%\begin{equation}
%
%\label{answern.n}
%\end{equation}
%\textbf{Answer [not] verified}

\subsection{}
%problem 1.1
If you have four volumes, and the question is what order to place them in, the question is a simple permutation problem.  There are 4 possibilities for the first, 3 for the second, 2 for the third and only one for the last, making the number of permutations of the books 24.  Only one order has them in ascending order and only one order will have them in descending order.  Thus:
\begin{equation}
	P=\frac{2}{24}=\frac{1}{12}
\label{answer1.1}
\end{equation}
\textbf{Answer not verified}

\subsection{}
%problem 1.2
The wooden cube is made of $10^3$ cubes implying a 10x10x10 cube.  The cubes that have two faces painted will be the edges which are not on a corner.  Thus, since there are 12 edges of 10 cubes each, 8 of which are not corners, that implies we have 96 edges.  96 thus becomes our number of desireable outcomes and 1000 has always been the total number of outcomes:
\begin{equation}
	P=\frac{96}{1000} = 0.096
\label{answer1.2}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.3

If there are $n$ total items and $k$ of them are defective.  We select $m$ and want to know the probability of $l$ of them being defective ones.

There are $\binom{n}{m}$ possible ways to chose $m$ different items from the population of $n$ items which will be our denominator.  Now we need to know how many of those possibilities have $l$ bad ones in them for our numerator.  If there's $k$ total defective ones, then there are $\binom{k}{l}$

\begin{equation}
	P = \frac{\binom{k}{l}}{\binom{n}{m}}
\label{answer1.3}
\end{equation}
\textbf{Answer not verified}

\subsection{}
%problem 1.4

There are $10!$ possible ways to order the ten books.  You can imagine that since the three books have to take up three adjacent positions, there are 8 possible locations for the three books to be ordered.  Taking just the first position (that being the first three ``Slots''), there are 6 ways to order the desired books and then $7!$ ways to order the remaining books.  Thus there are $8 \cdot{} 6 \cdot{} 7!$ desirable orders giving us a probability of

\begin{equation}
	P=\frac{8 \cdot{} 6 \cdot{} 7!}{10!}=\frac{1}{15}
\label{answer1.4}
\end{equation}
\textbf{Answer  verified}

%answer template
\subsection{}
%problem 1.5

There are 4 possibilities to consider here: neither hits, one or the other hits and both hits.  While it may be tempting to calculate the probability of each event, since we only care about the probability of at least one hitting the target, we need only calculate the probability that no one hits and subtract that from 1.  The first marksman has an $.8$ probability of hitting meaning he has a missing probability of .2; similarly the second marksman has a .7 chance of hitting and a .3 chance of missing.  Thus, the probability of both one and two missing is the product of the two missing probabilities: $.2*.3=.06$

\begin{equation}
	P=1-.06=.94
\label{answer1.5}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.6
The total number of ways $n+k$ seats can be occupied by $n$ people is $\binom{n+k}{n}=\frac{(n+k)!}{n!k!}$ but once again the difficult part is finding the total number of desirable outcomes. If you were to specify $m$ seats, you effectively divide the auditorium into two buckets so we need to find the number of ways to partition people into those two buckets which will give us the numerator, $\binom{n+k}{m}=\frac{(n+k)!}{m!(n+k-m)!}$

\begin{equation}
	P=\frac{(n+k)!}{m!(n+k-m)!}\frac{n!k!}{(n+k)!}=\frac{{n!k!}}{m!(n+k-m)!}
\label{answer1.6}
\end{equation}
\textbf{Answer not verified}

\subsection{}
%problem 1.7

Again, the number of ways to get three cards from a 52 card deck is $\binom{52}{3}$.  Since there are 4 each of sevens, threes and aces, there are $4^3$ desirable hands.

\begin{equation}
	P=\frac{4^3}{\frac{52!}{3!49!}}=\frac{16}{5525}=0.00289593
\label{answer1.7}
\end{equation}
Which it is worth pointing out, is no different than any other 3-card hand of three different cards.
\textbf{Answer not verified}

\subsection{}
%problem 1.8
If you indiscriminately chose 3 line segments from our bank of 5, you have $\binom{5}{3}=10$ total possibilities for triangles.  When you look at the bank (1, 3, 5, 7, 9), however, you have to make sure that any two segments are longer than the third segment, otherwise making a triangle is impossible.  The brute-force way to do this is start with 1 and realize that no triangle can be formed with it.  Then, looking at 3, you realize that you can do (3, 5, 7) and (3, 7, 9).  Starting now with 5, you can do (5, 7, 9) but that's the only triangle that hasn't already been enumerated.  At this point, you realize you're done and that the answer is plain to see

\begin{equation}
	P=\frac{3}{10}=.3
\label{answer1.8}
\end{equation}
\textbf{Answer not verified}

\subsection{}
%problem 1.9

We could do this problem in 5 minutes of programming and an instant of computation but that's not the point!  We need to think our way through this one.  How many cubes of the numbers between 1 and 1000, have 11 for the last two digits.  Luckily, each cube is unique so there's no complications there: only 1000 possibilities.

Let's break down the number we're cubing into two parts, one that encapsulates the part less than 100 and then the rest of the number
\begin{equation}
	n=a+b=100c+b
\end{equation}
Now, just for fun, let's cube than number
\begin{equation}
	n^3=(100c+b)^3=b^3+300 b^2 c+30000 b c^2+1000000 c^3
\end{equation}
Clearly the only term here that will matter to the last two digits of the cube is $b$, the part less than 100.  Now we can reduce our now size 100 subspace by a great deal if you realize that in order for a number's cube to end in 1, the last number in the cube will need to be 1, leaving us: (1, 11, 21, 31, 41, 51, 61, 71, 81, 91).  At this point I recommend just cubing all the numbers and realizing that only $71^3=357911$ fulfills the requirements giving only one desirable outcome per century(71, 171, 271 etc.)

\begin{equation}
	P=\frac{10}{1000}=.01
\label{answer1.9}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.10
Now we want to know about all positive integers but just the last digit and the probability that the last digit is one.  To prove that we can use the hint given, we'll split the integer into the past less than 10 and the part greater than 10.
\begin{equation}
	n=10a+b
\end{equation}
\subsubsection{a--Squared}
\begin{equation}
	n^2=100a^2+10ab+b^2
\end{equation}
OK, so clearly only $b^2$ contributes to the last digit.  At this point... just do the squaring especially since it's something you can do in your head. (1, 2, 3, 4, 5, 6, 7, 8, 9)$\rightarrow$(1, 4, 9, 16, 25, 36, 49, 64, 81) therefore there are two desireable outcomes for every decade of random positive integers.
\begin{equation}
	P=\frac{2}{10}=.2
\label{answer1.10a}
\end{equation}

\subsubsection{b--fourth-powered}
\begin{equation}
	n^4=100000000 a^4+4000000 a^3 b+60000 a^2 b^2+400 a b^3+b^4
\end{equation}
Great, once again, just $b^4$.  Now, doing the forth power is a little harder so let's reason through and reduce the subspace.  Clearly, only odd numbers will contribute since any power of an even number is again an even number. Now, let's just do the arithmetic. (1, 3, 5, 7, 9)$\rightarrow$(1, 81, 625, 2401, 6561) giving us 4 desirable outcomes per decade of random numbers
\begin{equation}
	P=\frac{4}{10}=.4
\label{answer1.10a}
\end{equation}

\subsubsection{b--multiplied by random positive number}
\begin{equation}
	n*r=(10a+b)r
\end{equation}
Now let's let $r$ be a similar integer as n
\begin{equation}
	n*r=(10a+b)(10c+d)=100ac+10(bc+ad)+bd
\end{equation}

So we just need to consider the first two digits of both the random number and the arbitrary number.  This leaves us with $10^2$ possibilities; $5^2$ possible desirable outcomes once we exclude even numbers: (1, 3, 5, 7, 9).  Once again we resort to brute force by just defining the above as a vector and multiplying the transpose of that vector by the vector.  The only desirable outcomes turn out to be (1*1, 9*9, 7*3, 3*7), giving four desirable outcomes per century.

\begin{equation}
	P=\frac{4}{100}=.04
\label{answer1.10b}
\end{equation}

\textbf{Answers all verified}

\subsection{}
%problem 1.11
Here we are given 8 possible numbers (2, 4, 6, 7, 8, 11, 12, 13) and are told that it making a random fraction out of two of the numbers.  There are $\binom{8}{2}=28$ possible pairs but since $\frac{a}{b} \neq \frac{b}{a}$, we multiply that by two to get the total number of fractions we can get from these numbers as being 56.

Looking at the numbers given, 7, 11 and 13 are all prime and all the others are divisible by two; therefore only the fractions that contain one or more prime numbers will be in lowest terms.  The number 7 can be in 7 different pairs with the other numbers which makes for 14 fractions.  11 can be in 6 pairs or 12 fractions (since we already counted 7) and similarly, 13 can be in 10 uncounted fractions giving 36 possible fractions in lowest terms.

\begin{equation}
	P=\frac{36}{56}=\frac{9}{14}
\label{answer1.11}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.12
The word drawer has 6 letters and there are $6!=720$ possible ways of arranging the letters.  It is then natural to say there's only one way to spell reward correctly and thus the probability of spelling it correctly after a random reordering is $1/720$ BUT there is a complication in that the two of our letters are the same, meaning there are to distinct arrangement of our distinguishable tiles that will give us the proper spelling.

\begin{equation}
	P=\frac{2}{720} = \frac{1}{360}
\label{answer1.12}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.13
For any die, there are 6 different possibilities.  Since one dice's outcome does not depend on another's, that means that for a roll of $6n$ die, there are $6^{6n}$ different possible outcomes for the dice rolls.

Now for desirable outcomes, we want each of the 6 faces to show up n times.  To accomplish this, we just count the number of ways to apportion $6n$ things into 6 groups of $n$ each or
\begin{equation}
	\frac{(6n)!}{(n!)^6}
\end{equation}
which, given Stirling's approximation
\begin{equation}
	n! \approx \sqrt{2 \pi n} n^n e^{-n}
\end{equation}
gives us for large $n$
\begin{equation}
	\frac{(6n)!}{(n!)^6} \approx \sqrt{2 \pi 6n} (6n)^{6n} e^{-6n} * \frac{1}{(\sqrt{2 \pi n} n^n e^{-n})^6} = \frac{3 \cdot 6^{6n}}{4 (\pi n)^{5/2}}
\end{equation}

\begin{equation}
	P=\frac{(6n)!}{(n!)^6 6^{6n}} \approx \frac{3}{4 (\pi n)^{5/2}}
\label{answer1.13}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.14
To figure out the total number of possibilities, we must realize that one draw of half a deck, implies the other half of the deck implicitly, therefore there are $\binom{52}{26}$ total 26 card draws.  Then, to get 13 red cards, there are $\binom{26}{13} = 10400600$ ways to get 13 red cards and the same number for black cards making $10400600^2$ the number of total possible desirable outcomes.

\begin{equation}
	P=\frac{16232365000}{74417546961}=0.218126
\label{answer1.14}
\end{equation}
Because we're cool and modern and have Mathematica, we don't NEED to do the Stirling's formula approximation but it'll be good for us so we shall.
\begin{equation}
	P=\frac{\binom{26}{13}^2}{\binom{52}{26}} = \frac{(26!)^4}{(13!)^4 52!}=\frac{((2n)!)^4}{(n!)^4 (4n)!}
\end{equation}
where $n=13$.
\begin{equation}
	P=\frac{(\sqrt{4 \pi n}(2n)^{2n}e^{-2n})^4}{(\sqrt{2 \pi n}n^{n}e^{-n})^4 (\sqrt{8 \pi n}(4n)^{4n}e^{-4n})}
\end{equation}
\begin{equation}
	P=\frac{(4 \pi n)^2}{(2 \pi n)^2 \sqrt{8 \pi n}} \frac{(2n)^{8n}}{n^{4n}(4n)^{4n}} = \frac{4}{\sqrt{8 \pi n}} \frac{2^{8n}}{4^{4n}}
\end{equation}
\begin{equation}
	P=\frac{2}{\sqrt{26 \pi}} = 0.221293
\end{equation}
which when you take the ratio of approximate to exact, you get $1.01452$ so the approximation is less than $2\%$ off... not bad!

\textbf{Answer verified}

\subsection{}
%problem 1.15
There are 100 senators at any given time.  Much like the previous problem, there are $\binom{100}{50}=100891344545564193334812497256$ different 50 senator committees.  Much like the previous problem, there are $\binom{2}{1}=2$ ways for California to be represented on the committee making for 50 states $2^{50}$ possible even committees

\begin{equation}
	P=\frac{2^{50}}{\binom{100}{50}} = \text{1.115952921347132$\grave{ }$*${}^{\wedge}$-14}
\label{answer1.15}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.16
For $2n$ people, there are $(2n)!$ different possible lines; we want the number of lines where at any given point in the line, there are more people with 5 dollar bills than 10 dollar bills.

The given reference (freely available on Google books) has a fascinating geometrical argument about how $\binom{2n}{n+1}$ is the number of lines that have one or more too many people with tens in front of people with fives.  Also, in this argument, there are $\binom{2n}{n}$ trajectories instead of $(2n)!$ lines.

\begin{equation}
	P=\frac{\binom{2n}{n}-\binom{2n}{n+1}}{\binom{2n}{n}} = \frac{1}{n+1}
\label{answer1.16}
\end{equation}
\textbf{Answer verified}

\subsection{}
%problem 1.17
We wish to prove that

\begin{equation}
	\sum_{k=0}^{n} \binom{n}{k}^2 = \binom{2n}{n}
\label{answer1.17}
\end{equation}

This is easies if we listen to the hint and consider that $\binom{2n}{n}$ is the coefficient of $x^n$ in the polynomial $(x+1)^{2n}$ which is also equivalent to $(x+1)^n(x+1)^n$

We make use of:
\begin{equation}
	(x+1)^n = \sum_{k=0}^{n} \binom{n}{k} x^k
\end{equation}
\begin{equation}
	(x+1)^n(x+1)^n = \sum_{k=0}^{n}\sum_{j=0}^{n} \binom{n}{k} \binom{n}{j} x^k x^j
\end{equation}
We want the $x^n$ term: where $k+j=n$ or, put another way, where $k=n-j$...
\begin{equation}
	\sum_{j=0}^{n} \binom{n}{n-j} \binom{n}{j} = \sum_{j=0}^{n} \frac{n!}{j!(n-j)!}\frac{n!}{(n-j)!j!} = \sum_{j=0}^{n} \binom{n}{j}^2
\end{equation}
QED

\textbf{Answer not verified}

%%answer template
%\subsection{}
%%problem n.n
%
%
%\begin{equation}
%
%\label{answern.n}
%\end{equation}
%\textbf{Answer [not] verified}


\subsection{Magma ideals}\label{sec:magma_ideals}

\begin{definition}\label{def:magma_ideal}\mimprovised
  Let \( \mscrM \) be a \hyperref[def:magma]{magma} and \( I \) be a subset of \( \mscrM \). We say that \( I \) is a \term{left ideal} of \( \mscrM \) if the inclusion \( I\mscrM \subseteq I \) holds in the \hyperref[def:magma/power_set]{power set magma}. That is,
  \begin{equation*}
    I \mscrM = \set{ x \cdot y | x \in \mscrM, y \in I } \subseteq \mscrM.
  \end{equation*}

  Right ideals are defined analogously. If \( I \) is both a left ideal and a right ideal, we say that it is a \term{two-sided ideal}.

  As shown in \fullref{thm:commutative_magma_ideals}, for \hyperref[def:magma/commutative]{commutative} magmas left and right ideals coincide, and thus we can speak simply about ideals.
\end{definition}

\begin{proposition}\label{thm:magma_ideal_is_submagma}
  Every two-sided magma ideal is a submagma.
\end{proposition}
\begin{proof}
  Let \( I \) be a two-sided ideal for the magma \( \mscrM \). For every pair \( x, y \in I \), we have \( xy \in I \) since \( I \) is a left ideal and \( yx \in I \) since \( I \) is a right ideal. Thus, \( II = I \), and \( I \) is a submagma of \( \mscrM \).
\end{proof}

\begin{example}\label{ex:subgroup_is_not_ideal}
  We explicitly give a counterexample to the converse of \fullref{thm:magma_ideal_is_submagma}. Define \( \mscrG \coloneqq \BbbZ \times \BbbZ \) as a \hyperref[def:group_direct_sum]{direct sum} of two copies of the \hyperref[def:set_of_integers]{integers}. Define also the subset
  \begin{equation*}
    \mscrH \coloneqq \set{ (n, n) \colon n \in \BbbZ }.
  \end{equation*}

  The set \( \mscrH \) is a subgroup of \( \mscrG \), since it is closed under addition, and since it contains the identity element \( (0, 0) \). It is not an ideal, however, because
  \begin{equation*}
    (n, n) + (n, 0) = (2n, n) \not\in \mscrH.
  \end{equation*}
\end{example}

\begin{proposition}\label{thm:commutative_magma_ideals}
  In a \hyperref[def:magma/commutative]{commutative magma} \( \mscrM \), a subset \( I \subseteq M \) is a left ideal if and only if it is a right ideal. That is, in commutative magmas, it makes no sense to distinguish between left, right and two-sided ideals.
\end{proposition}
\begin{proof}
  For \( x \in M \) and \( y \in I \), by commutativity we have \( yx = xy \), thus \( \mscrM I = I \mscrM \) and \( I \) is a left ideal if and only if it is a right ideal.
\end{proof}

\begin{proposition}\label{thm:proper_ideals_containing_identity}
  A left or right ideal of a \hyperref[def:unital_magma]{unital magma} contains the identity if and only if it is not proper.
\end{proposition}
\begin{proof}
  Let \( \mscrM \) be a unital magma and \( I \) be a left ideal of \( \mscrM \). We will prove that \( e \in I \iff I = \mscrM \).

  \SufficiencySubProof Let \( e \in I \). Then \( ex = x \) for any \( x \in M \), thus \( I\mscrM = \mscrM \). But \( I \) is an ideal, hence we have that \( I\mscrM = I \), thus \( I = I\mscrM = \mscrM \).

  \NecessitySubProof If \( I = \mscrM \), then obviously \( e \in I \).

  An analogous proof follows for the case when \( I \) is a right ideal.
\end{proof}

\begin{corollary}\label{thm:unital_magma_ideal_is_submagma_iff_contains_identity}
  A two-sided ideal of a unital magma is a unital submagma if and only if it contains the identity.
\end{corollary}
\begin{proof}
  Follows from \fullref{thm:magma_ideal_is_submagma} and \fullref{thm:proper_ideals_containing_identity}.
\end{proof}

\begin{proposition}\label{thm:product_of_semigroup_ideals_is_in_intersection}
  Fix a \hyperref[def:magma/associative]{semigroup} \( \mscrM \). If \( I \) and \( J \) are two-sided ideals, so are \( IJ \) and \( I \cap J \) and, furthermore,
  \begin{equation*}
    IJ \subseteq I \cap J.
  \end{equation*}
\end{proposition}
\begin{proof}
  We first show that \( IJ \) is an ideal.

  Take \( x \in I \), \( y \in J \). If \( z \in M \), then associativity gives us
  \begin{equation*}
    z(xy) = (zx)y \in (zx)J \subseteq IJ
  \end{equation*}
  and
  \begin{equation*}
    (xy)z = x(yz) \in I(yz) \subseteq IJ.
  \end{equation*}

  Hence, \( IJ \) is closed under the semigroup operation. This makes \( IJ \) a two-sided ideal.

  If \( x \in I \cap J \) and \( z \in M \), obviously \( xz \in I \) and \( xz \in J \), hence \( xz \in I \cap J \). Then \( I \cap J \) is also a two-sided ideal.

  To obtain the inclusion
  \begin{equation*}
    IJ \subseteq I \cap J,
  \end{equation*}
  observe that \( xy \in IJ \) means that \( xy \in xJ = J \) and \( xy \in Iy = I \), thus \( xy \in I \cap J \).
\end{proof}


% This LaTeX was auto-generated from an M-file by MATLAB.
% To make changes, update the M-file and republish this document.

%%% \documentclass{article}
%%% \usepackage{graphicx}
%%% \usepackage{color}

%%% \sloppy
%%% \definecolor{lightgray}{gray}{0.5}
\setlength{\parindent}{0pt}

%%% \begin{document}

\subsection*{3-point interpolated DFT frequency estimator}

\begin{par}
Example for algorithm iDFT3p.
\end{par} \vspace{1em}
\begin{par}
iDFT3p is an algorithm for estimating the frequency, amplitude, phase and offset of the fundamental component using interpolated discrete Fourier transform. Rectangular or Hann window can be used for DFT.'; See also Krzysztof Duda: Interpolation algorithms of DFT for parameters estimation of sinusoidal and damped sinusoidal signals. In S. M. Salih, editor, Fourier Transform - Signal Processing, chapter 1, pages 3-32, InTech, 2012. \url{http://www.intechopen.com/books/fourier-transform-signal-processing/interpolated-dft} Implemented by Rado Lapuh, 2016.';
\end{par} \vspace{1em}

\subsubsection*{Contents}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item Generate sample data
   \item Call algorithm
   \item Display results
\end{itemize}

\subsubsection*{Generate sample data}

\begin{par}
Two quantities are prepared: \lstinline{Ts} and \lstinline{y}, representing 0.5 second of sinus waveform of nominal frequency 100 Hz, nominal amplitude 1 V and nominal phase 1 rad, sampled with sampling time 0.1 ms, with offset 0.1 V. The sampling is not coherent.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
DI = [];
Anom = 1; fnom = 100; phnom = 1; Onom = 0.1;
DI.Ts.v = 1e-4;
t = [0:DI.Ts.v:0.5];
DI.y.v = Anom*sin(2*pi*fnom*t + phnom) + Onom;
\end{lstlisting}

\subsubsection*{Call algorithm}

\begin{par}
First a rectangular window will be selected to estimate main signal properties. Use QWTB to apply algorithm \lstinline{iDFT3p} to data \lstinline{DI} and put results into \lstinline{DOr}.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
DI.window.v = 'rectangular';
DOr = qwtb('iDFT3p', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
\end{lstlisting} \color{black}
    \begin{par}
Next a Hann window will be selected to estimate main signal properties Results will be put into \lstinline{DOh}.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
DI.window.v = 'Hann';
DOh = qwtb('iDFT3p', DI);
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
\end{lstlisting} \color{black}

\subsubsection*{Display results}

\begin{par}
Results is the amplitude, frequency and phase of sampled waveform. For rectangular window, the error from nominal in parts per milion is:
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
f_re = (DOr.f.v - fnom)./fnom .* 1e6
A_re = (DOr.A.v - Anom)./Anom .* 1e6
ph_re = (DOr.ph.v - phnom)./phnom .* 1e6
O_re = (DOr.O.v - Onom)./Onom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]

f_re =

    0.0166

A_re =

   41.2567

ph_re =

   88.3681

O_re =

   1.6826e+03

\end{lstlisting} \color{black}
    \begin{par}
For Hann window:
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
f_he = (DOh.f.v - fnom)./fnom .* 1e6
A_he = (DOh.A.v - Anom)./Anom .* 1e6
ph_he = (DOh.ph.v - phnom)./phnom .* 1e6
O_he = (DOh.O.v - Onom)./Onom .* 1e6
\end{lstlisting}

        \begin{lstlisting}[style=output]

f_he =

  -3.7790e-06

A_he =

   3.9679e-07

ph_he =

    6.2737

O_he =

   -0.6862

\end{lstlisting} \color{black}

%%% \end{document}


\documentclass[namecite, fleqn]{goose-article}

\title{Linear elasticity}

\author{Tom W.J.\ de Geus}

\hypersetup{pdfauthor={T.W.J. de Geus}}

\begin{document}

\maketitle

\section{Constitutive model}

The stress, $\bm{\sigma}$, is set by to the strain, $\bm{\varepsilon}$,
through the following linear relation:
\begin{equation}
    \bm{\sigma}
    \equiv K \mathrm{tr}\left( \bm{\varepsilon} \right)
    + 2 G \bm{\varepsilon}_\mathrm{d}
    \equiv \mathbb{C} : \bm{\varepsilon}
\end{equation}
wherein $\mathbb{C}$ is the elastic stiffness, which reads:
\begin{align}
    \mathbb{C}
    &\equiv K \bm{I} \otimes \bm{I}
    + 2 G (\mathbb{I}_\mathrm{s} - \tfrac{1}{3} \bm{I} \otimes \bm{I} )
    \\
    &= K \bm{I} \otimes \bm{I}
    + 2 G \, \mathbb{I}_\mathrm{d}
\end{align}
with $K$ and $G$ the bulk and shear modulus respectively.
See \cref{sec:ap:nomenclature} for nomenclature, including definitions of the unit tensors.

\section{Consistency check}

To check if the derived tangent $\mathbb{C}$ a \emph{consistency check} can be performed.
A (random) perturbation $\delta \bm{\varepsilon}$ is applied.
The residual is compared to that predicted by the tangent.
For the general case of linearisation, the following holds:
\begin{equation}
    \bm{\sigma}\big( \bm{\varepsilon}_\star + \delta \bm{\varepsilon} \big)
    = \bm{\sigma}\big( \bm{\varepsilon}_\star \big)
    + \mathbb{C} \big( \bm{\varepsilon}_\star \big) : \delta \bm{\varepsilon}
    + \mathcal{O}(\delta \bm{\varepsilon}^2)
\end{equation}
or
\begin{equation}
    \underbrace{
        \bm{\sigma}\big( \bm{\varepsilon}_\star + \delta \bm{\varepsilon} \big)
        - \bm{\sigma}\big( \bm{\varepsilon}_\star \big)
    }_{
        \displaystyle \delta \bm{\sigma}
    }
    - \mathbb{C} \big( \bm{\varepsilon}_\star \big) : \delta \bm{\varepsilon}
    = \mathcal{O}(\delta \bm{\varepsilon}^2)
\end{equation}
This allows the introduction of a relative error
\begin{equation}
    \eta =
    \Big|\Big|
        \delta \bm{\sigma}
        - \mathbb{C}(\bm{\varepsilon}_\star) : \delta \bm{\varepsilon}
    \Big|\Big|
    /
    \Big|\Big| \delta \bm{\sigma} \Big|\Big|
\end{equation}
This \emph{truncation error} thus scales as $\eta \sim || \delta \bm{\varepsilon} ||^2$
as depicted in \cref{fig:consistency:expected}.
As soon as the error becomes sufficiently small the numerical \emph{rounding error}
becomes more dominant, the scaling thereof is also included in \cref{fig:consistency:expected}.

\begin{figure}[htp]
    \centering
    \includegraphics[width=.5\textwidth]{figures/consistency}
    \caption{Expected behaviour of the consistency check, see \citet[p.~9]{Heath2002}.}
    \label{fig:consistency:expected}
\end{figure}

Because this model is linear there is no truncation error, the measurement of $\eta$ and a
function of $|| \delta \bm{\varepsilon} ||$ thus only displays a rounding error,
as depicted in \cref{fig:consistency}.

\begin{figure}[htp]
    \centering
    \includegraphics[width=.5\textwidth]{examples/consistency}
    \caption{Measured consistency check, cf.\ \cref{fig:consistency:expected}.}
    \label{fig:consistency}
\end{figure}

\appendix

\section{Nomenclature}
\label{sec:ap:nomenclature}

\paragraph{Tensor products}
\vspace*{.5eM}

\begin{itemize}

    \item Dyadic tensor product
    \begin{align}
        \mathbb{C} &= \bm{A} \otimes \bm{B} \\
        C_{ijkl} &= A_{ij} \, B_{kl}
    \end{align}

    \item Double tensor contraction
    \begin{align}
        C &= \bm{A} : \bm{B} \\
        &= A_{ij} \, B_{ji}
    \end{align}

\end{itemize}

\paragraph{Tensor decomposition}
\vspace*{.5eM}

\begin{itemize}

    \item Deviatoric part $\bm{A}_\mathrm{d}$ of an arbitrary tensor $\bm{A}$:
    \begin{equation}
        \mathrm{tr}\left( \bm{A}_\mathrm{d} \right) \equiv 0
    \end{equation}
    and thus
    \begin{equation}
        \bm{A}_\mathrm{d} = \bm{A} - \tfrac{1}{3} \mathrm{tr}\left( \bm{A} \right)
    \end{equation}

\end{itemize}

\paragraph{Fourth order unit tensors}
\vspace*{.5eM}

\begin{itemize}

    \item Unit tensor:
    \begin{equation}
        \bm{A} \equiv \mathbb{I} : \bm{A}
    \end{equation}
    and thus
    \begin{equation}
        \mathbb{I} = \delta_{il} \delta{jk}
    \end{equation}

    \item Right-transposition tensor:
    \begin{equation}
        \bm{A}^T \equiv \mathbb{I}^{RT} : \bm{A} = \bm{A} : \mathbb{I}^{RT}
    \end{equation}
    and thus
    \begin{equation}
      \mathbb{I}^{RT} = \delta_{ik} \delta_{jl}
    \end{equation}

    \item Symmetrisation tensor:
    \begin{equation}
        \mathrm{sym} \left( \bm{A} \right) \equiv \mathbb{I}_\mathrm{s} : \bm{A}
    \end{equation}
    whereby
    \begin{equation}
        \mathbb{I}_\mathrm{s} = \tfrac{1}{2} \left( \mathbb{I} + \mathbb{I}^{RT} \right)
    \end{equation}
    This follows from the following derivation:
    \begin{align}
        \mathrm{sym} \left( \bm{A} \right) &= \tfrac{1}{2} \left( \bm{A} + \bm{A}^T \right)
        \\
        &= \tfrac{1}{2} \left( \mathbb{I} : \bm{A} + \mathbb{I}^{RT} : \bm{A} \right)
        \\
        &= \tfrac{1}{2} \left( \mathbb{I} + \mathbb{I}^{RT} \right) : \bm{A}
        \\
        &= \mathbb{I}_\mathrm{s} : \bm{A}
    \end{align}

    \item Deviatoric and symmetric projection tensor
    \begin{equation}
        \mathrm{dev} \left( \mathrm{sym} \left( \bm{A} \right) \right)
        \equiv \mathbb{I}_\mathrm{d} : \bm{A}
    \end{equation}
    from which it follows that:
    \begin{equation}
        \mathbb{I}_\mathrm{d}
        = \mathbb{I}_\mathrm{s} - \tfrac{1}{3} \bm{I} \otimes \bm{I}
    \end{equation}

\end{itemize}

\bibliography{library}

\end{document}


\documentclass{article}
\title{Homework1}
\author{Hao Cui}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{geometry}
\geometry{a4paper,scale=0.8}
\begin{document}
\maketitle
\section{Prove $2n+\Theta(n^2)=\Theta(n^2)$}
Let $f(n)\in\Theta(n^2)$, we can find $c_1,c_2,n_0$ such that $\forall n\ge n_0, 0\le c_1n^2 \le f(n) \le c_2n^2$, that is
$c_1n^2 + 2n \le f(n)+2n \le c_2n^2+2n$.\\
So we only need to prove $f(n)+2n\in \Theta(n^2)$, that means we need to find $c_3,c_4$ so that $\forall n\ge n_0, 0\le c_3n^2 \le f(n)+ 2n \le c_4n^2$.\\
Solve inequations:$
\left\{
\begin{aligned}
c_3n^2 \le c_1n^2 + 2n \\
c_4n^2 \ge c_2n^2+2n \\
\end{aligned}\right.$,
We can find $
\left\{
\begin{aligned}
c_3<c_1 \\
c_4>2/n_0 + c_2 \\
\end{aligned}\right. $ such that $\forall n\ge n_0, 0\le c_3n^2 \le f(n)+ 2n \le c_4n^2$.\\
Thus $f(n)+2n\in \Theta(n^2)$, and $2n+\Theta(n^2)=\Theta(n^2)$.
\section{Prove $\Theta(g(n))\cap o(g(n))=\emptyset$}
Assume $\Theta(g(n))\cap o(g(n))\neq\emptyset$, then exists $f(n) \in\Theta(g(n))$ and $f(n)\in o(g(n))$ ,
so $\exists c_1>0,c_2>0,n_0>0$ for that $\forall n \ge n_0,\forall c>0$,$
\left\{
\begin{aligned}
0\le c_1g(n) \le f(n) \le c_2g(n) \\
0\le f(n) < cg(n) \\
\end{aligned}\right.$,
let c = $\frac{c_1}{2}$, so $c_1g(n)\le f(n)<cg(n)=\frac{c_1}{2}g(n), c_1 < 0$.\\
This is a contradiction, thus $\Theta(g(n))\cap o(g(n))=\emptyset$.
\section{Prove $\Theta(g(n))\cup o(g(n))\neq O(g(n))$}
Let $
f(n)=\left\{
\begin{aligned}
g(n) \ when\ n\ is\ even  \\
0  \qquad \qquad otherwise\\
\end{aligned}\right.$,
$\exists c=1, n_0>0, s.t. \forall n\ge n_0,0\le f(n)\le cg(n)$,so $f(n)\in O(g(n))$.\\
If $0<c<1$ and n is even number, $f(n)>cg(n)$, so $f(n)\notin o(g(n))$.\\
If n is not even number, f(n)=0, there is not a $c_1>0$ so that $0\le c_1g(n) \le f(n)$, so $f(n)\notin \Theta(g(n))$.\\
Thus $f(n)\notin \Theta(g(n))\cup o(g(n))$.
Thus $\Theta(g(n))\cup o(g(n))\neq O(g(n))$.
\section{Prove $max(f(n),g(n))=\Theta(f(n)+g(n))$}
To show that $max(f(n),g(n))=\Theta(f(n)+g(n))$, we want to find constants $c_1,c_2,n_0>0$ such that $0 \le c_1(f(n)+g(n)) \le max(f(n),g(n)) \le c_2(f(n)+g(n))$ for all $n \ge n_0$.\\
Note that $max(f(n),g(n)) \le f(n)+g(n)$, we can find $c_2=1$ such that $ max(f(n),g(n)) \le c_2(f(n)+g(n))$.\\
Assume $f(n)\le g(n)$, and let $c_1=\frac{1}{2}$, we can find $\frac{1}{2}(f(n)+g(n)) \le f(n) \le g(n) = max(f(n),g(n))$.\\
So when $c_1=\frac{1}{2}, c_2=1$, we can find $n_0>0$ such that $0 \le c_1(f(n)+g(n)) \le max(f(n),g(n)) \le c_2(f(n)+g(n))$ for all $n \ge n_0$.\\
Thus $max(f(n),g(n))=\Theta(f(n)+g(n))$.
\section{Solve the recurrence $T(n)=2T(\sqrt{n})+1$}
Let $m=lgn$, then $T(2^m)=2T(2^\frac{m}{2})+1$.\\
Let $S(m)=T(2^m)$, then $S(m)=2S(\frac{m}{2})+1$.\\
According to the master method, $f(m)=1=O(m^{\log _{2}{2-1}})$, so that $S(m)=\Theta(m)$.\\
Thus $T(n)=T(2^m)=S(m)=\Theta(m)=\Theta(lgn)$.
\section{Solve the recurrence $nT(n)=(n-2)T(n-1)+2$}
$T(n)=1$ when $n \ge 2.$\\
Prove:\\
1. n=2,2T(2)=2, T(2)=1.\\
2. Assume T(k)=1,$k>2$, then (k+1)T(k+1)=(k-1)+2, then T(k+1)=1.\\
Thus $T(n)=1=\Theta(1)$.
\section{CLRS,pp61,3-3}
\subsection{Rank the following functions by order of growth}
Functions on the same line are in the same equivalence class.\\
\begin{center}
$2^{2^{n+1}}$\\
$2^{2^n}$\\
$(n+1)!$\\
$n!$\\
$e^n$\\
$n\cdot 2^n$\\
$2^n$\\
$(\frac{3}{2})^n$\\
$(lgn)^{lgn},n^{lglgn}$\\
$(lgn)!$\\
$n^3$\\
$n^2,4^{lgn}$\\
$nlgn,lg(n!)$\\
$n,2^{lgn}$\\
$\sqrt{2}^{lgn}$\\
$2^{\sqrt{2lgn}}$\\
$lg^2n$\\
$ln{n}$\\
$\sqrt{lgn}$\\
$lnlnn$\\
$2^{lg^*n}$\\
$lg^*n,lg^*(lgn)$\\
$lg(lg^*)n$\\
$n^{\frac{1}{lgn}},1$
\end{center}
\subsection{F(n) is neither $O(g_i(n))$ nor $\Theta(g_i(n))$}
$$
f(n)=\left\{
\begin{aligned}
2^{2^{2^{n+1}}} \ if\ n\ is\ even  \\
0  \qquad \ if\ n\ is\ odd\\
\end{aligned}\right.
$$
\end{document}


\section{Dynamic Programming}

Dynamic programming is a technique from optimization field, specally it's a
method to solve problem if the problem exhibits \textbf{optimal substructure}
and \textbf{overlapping subproblems}.

\begin{definition*}

A problem is said to have \textbf{optimal substructure} if it can be constructed
from optimal solutions of its subproblems.

For an example, suppose $T$ a tree and consider the problem where $\forall v \in
T$, you want to compute a field $\min$ such that $\min[v] = \min\{w.val : w \in
ST(v)\}$ where $ST(v)$ denotes the subtree rooted by $v$. This problem exhibits
optimal substructure, as for arbitrary vertex $v$, you have $\min[w]$ computed
for all $w \in Children[v]$, then you can construct $\min[v]$ as the minimum of
your childrens answers and your own.

\end{definition*}

\begin{definition*}

A problem is said to have \textbf{overlapping subproblems} if the problem can be
broken down into subproblems which are reused several times.

For an example, consider the Fibonnacci number sequence $a_n = a_{n-1} +
a_{n-2}, a_2 = a_1 = 1$. Consider the value $a_5$. Notice that we can express it
as a graph (see figure 2) with many incoming edges (in particular a DAG).
Whereas problems with optimal substructure but not overlapping subproblems look
like the problem $\min[v]$ problem, and can be modeled by trees.

\end{definition*}

The idea behind the dynamic programming problems we deal with is that if we
remember problems as we do it, we can elimate the "overlapping" problems and
reduce our computational cost as much as possible.

We've discussed three problems that we call "representative" of a lot of dynamic
programming techniques: GCD, Woodcut, and Matrix Chain. We find that many
dynamic programming problems reduce down to similar techniques needed to solve
these. In this section we discuss how to tackle these problems fully, but before
that we introduce \textbf{memoization}.

\subsection{Memoization through Fibbonacci}

As discussed above, the Fibbonacci sequence $F_n = F_{n-1} + F_{n-2}, F_2 = F_1
= 1$, is a prime example of a dynamic programming problem, since it has both
optimal substructure and overlapping subproblems. Using the recurrence
relationship defined above we can write recursive code to solve this problem:

\begin{algorithmic}[1]
\Procedure{F}{$n$}\Comment{$a_1 = 1, a_2 = 1, a_n = a_{n-1}+a_{n-2}$}
	\If{$n \leq 2$}
		\State Return $1$
	\Else
		\State Return $F(n-1)+F(n-2)$
	\EndIf
\EndProcedure
\end{algorithmic}

However, if we were to run just this we would find ourselves choked by the
runtime cost almost immediately. This is a consequence of the overlapping
subproblems portion of Fibbonacci, i.e. we're repeatedly computing the same
pieces of data. How do we resolve this? We implement \textbf{memoization}.

\begin{definition*}
\textbf{Memoization} is an optimization technique where we store and cache the
results of expensive function calls. The idea is that if we want to compute said
function again, we look in our cache first. (Memoization $\gets$ making a memo.)
\end{definition*}

Let's take a look at this implemented in the Fibbonacci sequence.

\begin{algorithmic}[1]
\State Let Memo[1..n] be a integer valued array initialized to $-1$.
\Procedure{F}{$n$}\Comment{Recursive or top down}
	\If{$n \leq 2$}
		\State Return $1$.
	\ElsIf{$Memo[n] \neq -1$} \Comment{If we have computed $n$, use that result}
		\State Return $Memo[n]$.
	\Else \Comment{Otherwise, compute it, store it, and return it.}
		\State $Memo[n] = F(n-1)+F(n-2)$.
		\State Return $Memo[n]$.
	\EndIf
\EndProcedure
\end{algorithmic}

We can also implement this iteratively using a lookup table:

\begin{algorithmic}[1]
\Procedure{F}{$n$}\Comment{Iterative or bottom up}
	\State Let Memo[1..n] be a integer valued array.
	\State Let $Memo[1], Memo[2] = 1$.
	\For{$i = 3 \to n$}
		\State $Memo[i] = Memo[i-1] + Memo[i-2]$
	\EndFor
\EndProcedure
\end{algorithmic}

Without the memoization table, our runtime was exponential, but what about now?
It's clear by the iterative version that our usage of memoization brings us to
$\theta(n)$ work, which is basically a night and day difference. In general we
can examing the new runtime of our dynamic programming problem by looking at the
memoization table, and trying to figure out how long it would take to fill it.

\subsection{Greatest Common Subsequence}

The problem is as follows: Suppose you are given two sequence $A$ and $B$. Your
objective is to find a subsequence $S \subseteq A, B$ such that the length of
$S$ is maximized. We want to write code to solve for $len(S)$. This is a
classic problem for applications in revision control, the diff utility on your
computer, and many more NLP like reasons.

So how do we solve this problem using dynamic programming? Well first we try to
fit it to the constraints of optimal substructure and overlapping subproblems.
To that end, consider placing points $i,j$ at the end of $A,B$:

\begin{align*}
A = a_1 a_2 a_3 \dots a_n & \gets i \\
B = b_1 b_2 b_3 \dots b_m & \gets j \\
\end{align*}

Consider the local problem of whether $a_i$ and $b_j$ should be considered in
our greatest subsequence:

\begin{enumerate}[(1)]

\item Suppose $a_i = b_j$. Then we know that the element $a_i \in A$ and $a_i
\in B$, therefore being a candidate to go in $S$. Do we ever have reason to
exclude something from $S$? In this problem no! Therefore we decide to add $1$
to our length. But then how can we continue processing after selecting $a_i$ and
$b_j$? Well we still need to select the best subsequence from $A[1..i-1]$ and
$B[1..j-1$. This sounds like a subproblem with the last character of each
sequence excluded!

\item Suppose $a_i \neq b_j$. Here's where things get a little bit more
complicated. Before when they were equal, we could conclude that we can take
both, but here we can't just say skip past this element. Take for example the
strings:

$$
A = aaaab, B = bbbba
$$

Considering the last element, if we choose to ignore them because they're not
equal, we reduce our problem to $aaaa, bbbb$, which has no GCS. However,
clearly $\{a\}$ or $\{b\}$ solves our problem.

Instead what we decide to do is not blindly skip past $a_i$ or $b_j$. We say
instead that we skip past \textit{either} $a_i$ \textit{or} $b_j$. How do we
decide which one? The one which produces the longer subsequence.

\end{enumerate}

Using the above analysis we then conclude that a solution to the GCS problem is:

$$
GCS(i,j) = \begin{cases}
0, & ij = 0 \\
1 + GCS(i-1,j-1), & A[i] = A[j] \\
\max(GCS(i-1,j),GCS(i,j-1)), & \text{otherwise}
\end{cases}
$$

where intially $i = n, j = m$.

\begin{algorithmic}[1]
\State Let $A[1..n]$ and $B[1..m]$ be two character arrays.
\Procedure{GCS}{$i,j$}
	\If{$ij = 0$}
		\State Return $0$.
	\ElsIf{$A[i] = B[j]$}
		\State Return $1 + GCS(i-1,j-1)$.
	\Else
		\State Return $\max(GCS(i-1,j),GCS(i,j-1))$.
	\EndIf
\EndProcedure
\end{algorithmic}
\begin{algorithmic}[1]
\State Let $A[1..n]$ and $B[1..m]$ be two character arrays.
\State Let $Memo[1..n,1..m]$ be a 2d array initialized to $-1$.
\Procedure{GCS}{$i,j$}
	\If{$ij = 0$}
		\State Return $0$.
	\ElsIf{$Memo[i,j] \neq -1$}
		\State Return $Memo[i,j]$.
	\ElsIf{$A[i] = B[j]$}
		\State $Memo[i,j] = 1 + GCS(i-1,j-1)$.
	\Else
		\State $Memo[i,j] = \max(GCS(i-1,j),GCS(i,j-1))$.
	\EndIf
	\State Return $Memo[i,j]$.
\EndProcedure
\end{algorithmic}
\begin{algorithmic}[1]
\State Let $A[1..n]$ and $B[1..m]$ be two character arrays.
\State Let $Memo[1..n,1..m]$ be a 2d array initialized to $-1$.
\State Let $Decisions[1..n,1..m]$ be a 2d array.
\Procedure{GCS}{$i,j$}
	\If{$ij = 0$}
		\State Return $0$.
	\ElsIf{$Memo[i,j] \neq -1$}
		\State Return $Memo[i,j]$.
	\ElsIf{$A[i] = B[j]$}
		\State $Memo[i,j] = 1 + GCS(i-1,j-1)$.
		\State $Decisions[i,j] = (i-1,j-1)$.
	\Else
		\State $v_1, v_2 = GCS(i-1,j), GCS(i,j-1)$.
		\If{$v_1 > v_2$}
			\State $Memo[i,j] = v_1$.
			\State $Decisions[i,j] = (i-1,j)$.
		\Else
			\State $Memo[i,j] = v_2$.
			\State $Decisions[i,j] = (i,j-1)$.
		\EndIf
	\EndIf
	\State Return $Memo[i,j]$.
\EndProcedure
\end{algorithmic}


\subsection{Approximating cardinalities of huge sets}
\label{subsec:wcountproblem}
The Count-Distinct or Word Count Problem can be formulated as follows: given a sequence of elements $s_{1}, ..., s_{n}$ compute the amount of \textbf{distinct} elements in it. For example, for the sequence dog, cat, dog, bird, bird the answer is 3 (the distinct elements are bird, cat, and dog).

If the sequence is not too large this problem can be easily solved in expected linear time and space using hash tables, or $\mathcal{O}(nlogn)$ time and linear space using some data structures as Red-Black trees. However, this bound on space starts to become unacceptable when datasets are too large. In this section we will describe a probabilistic algorithm named as HyperLogLog \cite{Flajolet07hyperloglog:the}.\\
\\
This algorithm is very simple and yet very powerful. The core idea is the following: for each element $s_{i}$ of our sequence, use a hash function $h: \{0, 1\}^{*} \mapsto \{0, 1\}^b$ to compute a value $h(s_{i})$ and estimate the cardinality as $2^m$, where $m$ is the maximum number of leading zeros among all $h(s_{i})$. We must note that if all $h(x)$ have the same probability $\frac{1}{2^{b}}$ then the probability for some value to have $k$ leading zeros is $2^{-k}$. This means that the expected number of observations that are needed to find a number with $k$ leading zeros is $2^{k}$. Given that having a single hash value is not precise enough but computing multiple hash functions is too expensive, what is done is the following:
\begin{enumerate}
\item Given a token $t$, compute $h(t)$
\item Take the first $p$ bits and use them to refer to a position in an array consisting of $2^p$ elements $a_{0}, ..., a_{2^p - 1}$
\item Update this position according to the other $b - p$ bits so it keeps the maximum amount of leading zeros seen so far
\item Once all tokens are processed output the harmonic mean of $2^{a_{0}}, ..., 2^{a_{2^p - 1}}$ as the answer.
\end{enumerate}
An interesting trivia fact is that if we need $\mathcal{O}(\log n)$ bits for our hash function to be able to count until $n$ then we only need $\mathcal{O}(\log \log n)$ to store the number of leading zeros of some hash value. This is why HyperLogLog is called that way.\\
\\
A very nice property of HyperLogLog is that two distinct runs on two different datasets can be merged if they have used the same parameters (hash function, $b$, and $p$) in such a way that it approximates the cardinality of the union of the two datasets. Given two arrays $a$ and $b$, each corresponding to a run of HyperLogLog we can get a fictional run of HyperLogLog $c$ that represents the union of both datasets by computing $c_{i} =\max(a_{i}, b_{i})$ for all of the $2^{p}$ positions. This makes sense, as it produces the same result as running a single HyperLogLog on the concatenation of the two datasets. This property allows us to parallelize or to distribute this algorithm, giving us a potential performance boost. The source code we will use for our experiments can be found in appendix \ref{subsec:hyperloglog_source_code}.\\
\\
Note that this application is a classical \textit{map-reduce} workflow. Without collections we are forced to implement any reduce function as \verb|reduce(f, *args)|. Each extra argument is passed as an input parameter via socket and pipe, implying a huge overhead. With collections only the collection object, and the list of identifiers, are transferred. The other properties of the contents, such as direction, locations and so on, are deduced or requested in the destination node.\\
\\
The elimination of this overhead is noticeable even with a very small number of parameters. As we can see in figure \ref{fig:collection_vs_normal}, the collection feature reduces the overhead drastically. An important observation is that a PyCOMPSs task of the form \verb|f(*args)| usually starts to show problems and crashes when more than $60$ arguments are passed, as each argument represents a lot of metadata to be transferred via socket and pipe.
\begin{figure}[ht!]
\centering
\includegraphics[scale = 0.5]{figures/collection_vs_normal.png}
\caption{Execution time of the reduce functions with and without collections. Each point is the average of 5 executions. Although the samples are noisy, as they are small, a consistent improvement by the collection feature can be appreciated. The non-collections versions started to crash and to show strange behaviours around the 60 parameters}
\label{fig:collection_vs_normal}
\end{figure}\\
\\
A comparison between the amount of meta-data generated and sent by the classical reduce implementation and by the collections version can be found in appendix \ref{subsec:reduce_data_comparison}.

This improvement benefits many applications, as the map-reduce scheme is very common.


\section*{Chapter 12: Extensions of GCD}

\paragraph{Exercise 12.2}
\begin{enumerate}
    \item Prove that an ideal $I$ is closed under subtraction.
    \item Prove that $I$ contains $0$.
\end{enumerate}

\begin{proof}
$ $\newline
\vspace{-0.1in}
\begin{enumerate}
    \item
    Let $R$ be the ring such that $I \subseteq R$. We know that
    the additive inverse of $1$, $-1$, is in $R$. Let $x \in I$. Thus,
    $-1 \, x = -x \in I$. Now, let $y \in I$. Then, $y - x = y + (-x)
    \in I$.

    \item The first part of the previous argument shows that $I$ is closed
    under additive inverses. Thus, given $x \in I$ (at least we have one
    since $I$ is nonempty), $x + (-x) = x - x = 0 \in I$.
\end{enumerate}
\end{proof}

\paragraph{Exercise 12.3}
Prove that all the elements of a linear combination ideal are divisible by
any of the common divisors of $a$ and $b$.

\begin{proof}
Let $I = \{xa + yb \, / \, x,y \in R \}$ be a linear combination ideal,
let $e = x_0 a + y_0 b \in I$ and let $d$ be a common divisor of $a$ and $b$.
That is, $a = d q_1$ and $b = d q_2$. Thus,
\begin{eqnarray*}
    e &=& x_0 a + y_0 b \\
      &=& x_0 (d q_1) + y_0 (d q_2) \\
      &=& d (x_0 q_1 + y_0 q_2) \\
      &=& d q
\end{eqnarray*}
In other words, $e$ is divisible by $d$.
\end{proof}

\paragraph{Exercise 12.4}
Prove that any element in a principal ideal is divisible by the principal element.

\begin{proof}
Follows immediately from the definition of principal ideal and principal element.
\end{proof}

\paragraph{Exercise 12.5}
Using B\'ezout's identity, prove that if $p$ is prime, then any $0 < a <p$ has
multiplicative inverse modulo $p$.

\begin{proof}
Actually, this is an immediate corollary of the invertibility lemma: being $p$
prime, any $0 < a < p$ is such that $\GCD{a}{p} = 1$. Thus, there exists an $x \in
\Zn{p}$ such that $a x = x a = 1 \MOD{p}$. An ad-hoc proof can be done
using essentially the same argument that proves the inveritibility lemma.
\end{proof}


\documentclass[class=report, float=false, crop=false]{standalone}

\input{preamble}

\graphicspath{{figures/images/}{figures/figs/}}

\begin{document}

\chapter{Calculation details}
\label{appendix:calculation}

\section{Field auto-correlation}
\label{field_auto_correlation}

Let $\vec{v}(\vec{r})$ be a vector -- or scalar -- field with $\vec{r} \in \mathbb{R}^2$, then we define its auto-correlation function as
\begin{equation}
C_{vv}(\Delta \vec{r}) = \frac{\int d^2\vec{r}~ \vec{v}(\vec{r})^*\cdot\vec{v}(\vec{r}+\Delta \vec{r})}{\int d^2\vec{r}~ |\vec{v}(\vec{r})|^2}
\end{equation}
which can be efficiently calculated with the help of Fourier transforms.\\

We define
\begin{equation}
\begin{aligned}
\mathcal{F} : \cdot \mapsto \int d^2\vec{k}~ e^{-i\vec{k}\cdot\vec{r}} ~\cdot  &\Leftrightarrow \mathcal{F}^{-1} : \cdot \mapsto \int d^2\vec{k}~ e^{i\vec{k}\cdot\vec{r}}\\
\tilde{\vec{v}} = \mathcal{F}\{\vec{v}\} &\Leftrightarrow \vec{v} = \mathcal{F}^{-1}\{\tilde{\vec{v}}\}
\end{aligned}
\end{equation}
then
\begin{equation}
\begin{aligned}
\int d^2\vec{r}~ \vec{v}(\vec{r})^*\cdot\vec{v}(\vec{r}+\Delta \vec{r}) &= \int d^2\vec{r} \left(\int d^2\vec{k}~ e^{i\vec{k}\cdot\vec{r}}\tilde{\vec{v}}(\vec{k})\right)^*\cdot\left(\int d^2\vec{k}^{\prime}~ e^{i\vec{k}^{\prime}\cdot(\vec{r}+\Delta \vec{r})}\tilde{\vec{v}}(\vec{k}^{\prime})\right)\\
&= \int d^2\vec{k} \int d^2\vec{k}^{\prime}~ e^{i\vec{k}^{\prime}\cdot\Delta \vec{r}} \tilde{\vec{v}}(\vec{k})^*\cdot\tilde{\vec{v}}(\vec{k}^{\prime}) \int d^2\vec{r}~ e^{-i(\vec{k}-\vec{k}^{\prime})\cdot\vec{r}}\\
&= \int d^2\vec{k} \int d^2\vec{k}^{\prime}~ e^{i\vec{k}^{\prime}\cdot\Delta \vec{r}} \tilde{\vec{v}}(\vec{k})^*\cdot\tilde{\vec{v}}(\vec{k}^{\prime}) \delta(\vec{k}-\vec{k}^{\prime})\\
&=  \int d^2\vec{k}~ e^{i\vec{k}\cdot\Delta \vec{r}} \tilde{\vec{v}}(\vec{k})^*\cdot\tilde{\vec{v}}(\vec{k})\\
&= \mathcal{F}^{-1}\{\mathcal{F}\{\vec{v}\}^*(\vec{k})\cdot\mathcal{F}\{\vec{v}\}(\vec{k})\}(\Delta\vec{r})
\end{aligned}
\end{equation}
thus showing that $C_{vv}$ can be easily expressed in terms of the Fourier transform of $\vec{v}$ field.

\end{document}