Fig. \ref{fig:siso-channels} illustrates the frequency response of a SISO FF and FS channel based on the same tap delays and tap gains.

\begin{figure}[ht]
  \centering
  \subfigure[FF channel]{
    \includegraphics[width=0.48\textwidth]{siso_frequency_flat_channel}\label{fig:siso-ff}}
  \subfigure[FS channel]{
    \includegraphics[width=0.48\textwidth]{siso_frequency_selective_channel}\label{fig:siso-fs}}
  \caption{Frequency response of the SISO FF and FS channels}\label{fig:siso-channels}
\end{figure}

In the following R-E plots, the rightmost point of each curve indicates the maximum achievable rate with zero harvested DC current. It corresponds to WIT that allocates all available power to the modulated information waveform by the water-filling algorithm with $\rho  = 0$. Note that the x-axis here refers to the per-subband rate and is normalized w.r.t. bandwidth. With a fixed power budget, the power received by each subband decreases as $N$ increases. Therefore, the rate achieved by each subband decreases but the total rate increases. On the other hand, the leftmost point corresponds to the maximum output DC current with zero information rate, which correspond to allocating all power to the multisine waveform with $\rho  = 1$ (WPT).

Nevertheless, the discrete rate constraint \eqref{eqn:original_rate_constraint} in the optimization problem prevent the solutions from achieving the absolute WIT points using the proposed WIPT approach. Therefore, we perform an individual WIT and combine the results to obtain closed R-E plots.

\subsection{R-E Region vs Subband}\label{sec:re-region-vs-subband}
Fig. \ref{fig:siso-subband} illustrates the R-E region against subband $N = 1,2,4,8,16$ for superposed waveform and no power waveform over FF and FS channels respectively.

\begin{figure}[ht]
  \centering
  \subfigure[FF: Superposed waveform]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_subband_superposed_waveform}\label{fig:subband-ff-superposed}}
  \subfigure[FF: No power waveform]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_subband_no_power_waveform}\label{fig:subband-ff-no-power}}
  \quad
  \subfigure[FS: Superposed waveform]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_subband_superposed_waveform}\label{fig:subband-fs-superposed}}
  \subfigure[FS: No power waveform]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_subband_no_power_waveform}\label{fig:subband-fs-no-power}}
  \caption{R-E region vs $N$ for FF and FS channels}\label{fig:siso-subband}
\end{figure}

It can be observed that the introduction of multisine power waveform boosts the harvested energy for $N > 4$ where the superposed waveform outperforms the modulated signal for WIPT. In contrast, the R-E performance of both signals are very close for $N \leqslant 4$ so that multisine is unnecessary. The reason is that the fourth order terms of power and information waveforms \eqref{eqn:power_waveform_fourth_order} and \eqref{eqn:information_waveform_fourth_order} have different contribution to the harvested DC current. Despite both posynomials consist of monomials of similar magnitude (${\prod\nolimits_{j = 0}^3 {{s_{P,{n_j},{m_j}}}{A_{{n_j},{m_j}}}} }$ and $\prod\nolimits_{j = 0,2} {{s_{I,{n_0},{m_j}}}{A_{{n_0},{m_j}}}} \prod\nolimits_{j = 1,3} {{s_{I,{n_1},{m_j}}}{A_{{n_1},{m_j}}}} $ ), the power posynomial contains $(2{N^3} + N)/3$ monomials but the information posynomial only holds ${N^2}$ monomials. Therefore, the energy benefit of multisine is amplified with a large $N$. Although it seems that a very large $N$ can significantly increases the output DC current, this is not true because each subband will receive less power such that the magnitude of monomials decreases accordingly.

A contrast of R-E plots on FF and FS channels also indicates the benefit of frequency selectivity on the harvested power. The gain is particularly obvious in the low-rate region, where all the power is allocated to the subband with the strongest amplitude. We observe from the FS channel instance in Fig. \ref{fig:siso-fs} that some edge frequencies enjoy a larger gain than the center frequency, whose advantage is exploited when more subbands are used. In comparison, the small amplitude at the center frequency accounts for the lower output DC current when $N = 1$ (indicated by the blue curves in Fig. \ref{fig:subband-fs-superposed} and \ref{fig:subband-fs-no-power}).

Without power waveform, the R-E region appears convex such that PS dominates TS over all $N$. On the other hand, the R-E region achieved by superposed waveform with PS is convex for $N = 2,4$ but concave-convex for $N = 8,16$. Therefore, the optimal strategy is using PS for a small $N$, TS for a large $N$, and a combination of PS and TS for a medium $N$. As shown in Fig. \ref{fig:siso-subband-optimal}, the best curve for medium $N$ consists of two parts. The straight part is achieved by TS between WPT (multisine only with $\rho  = 1$) that corresponds to the leftmost point and WIPT (superposed waveform with $0 < \rho  < 1$) that corresponds to the tangent point, while the convex part is the contribution of WIPT only. The characteristics of the optimal R-E region comes from the rectifier nonlinearity.

\begin{figure}[ht]
  \centering
  \subfigure[Optimal strategy for $N = 8$]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_subband_8_optimal}\label{fig:subband-8-optimal}}
  \subfigure[Optimal strategy for $N = 16$]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_subband_16_optimal}\label{fig:subband-16-optimal}}
  \caption{Optimal R-E region for FF channel with medium $N$}\label{fig:siso-subband-optimal}
\end{figure}

Moreover, the plots over the FS channel suggests that for single-carrier transmission, the modulated waveform outperforms the superposed waveform for WPT. At a zero-approaching rate, the modulated waveform produces a DC current of \SI{2.69}{\uA} while the superposed waveform only delivers \SI{2.64}{\uA}. The actual current gap is even larger since the former still guarantees a slightly higher rate. On the contrary, the superposed waveform leads to a larger harvested current for $N \geqslant 2$. It demonstrates that modulation is beneficial in single-carrier transmission but can be detrimental in multi-carrier transmission. The reason is that the modulation gain \eqref{eqn:modulation_gain} of modulated waveform outperforms the energy benefit of multisine in single-carrier transmission but is outperformed in multi-carrier transmission. The result is inline with the scaling laws proposed in \cite{Clerckx2018}.

One problem is that the leftmost point of some curves did not start from the y-axis. Although a zero rate constraint is employed in WIPT, a candidate solution may achieve a nonzero rate. It is because the current gain of the next iteration is smaller than the threshold $\varepsilon$ so that the algorithm terminates and outputs the existing R-E pair. This phenomenon occurs at the low-rate region with a small $N$, where the current gain is relatively small in each iteration and the output current is almost saturated. It can be fixed either by reducing the threshold $\varepsilon$ or developing an individual function for WPT.

\subsection{R-E Region vs SNR}\label{sec:re-region-vs-snr}
Fig. \ref{fig:siso-ff-snr} and \ref{fig:siso-fs-snr} contrast the performance of the modulated waveform, ideal superposed waveform and its lower bound for $N = 16$ and ${\text{SNR}} = 10,20,30,40$ dB over the example FF and FS channels.

\begin{figure}[ht]
  \centering
  \subfigure[FF: ${\text{SNR}} = 10$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_snr_10dB}\label{fig:snr-ff-10db}}
  \subfigure[FF: ${\text{SNR}} = 20$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_snr_20dB}\label{fig:snr-ff-20db}}
  \quad
  \subfigure[FF: ${\text{SNR}} = 30$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_snr_30dB}\label{fig:snr-ff-30db}}
  \subfigure[FF: ${\text{SNR}} = 40$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_snr_40dB}\label{fig:snr-ff-40db}}
  \caption{R-E region vs SNR for FF channel}\label{fig:siso-ff-snr}
\end{figure}

\begin{figure}[ht]
  \centering
  \subfigure[FS: ${\text{SNR}} = 10$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_snr_10dB}\label{fig:snr-fs-10db}}
  \subfigure[FS: ${\text{SNR}} = 20$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_snr_20dB}\label{fig:snr-fs-20db}}
  \quad
  \subfigure[FS: ${\text{SNR}} = 30$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_snr_30dB}\label{fig:snr-fs-30db}}
  \subfigure[FS: ${\text{SNR}} = 40$ dB]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_snr_40dB}\label{fig:snr-fs-40db}}
  \caption{R-E region vs SNR for FS channel}\label{fig:siso-fs-snr}
\end{figure}

Thanks to the contribution of multisine waveform, the R-E region of the superposed signal is enlarged in both cases. This phenomenon is especially obvious in the low-rate region, where the multisine dominates the transmission. It results from the fact that the nonlinear rectifier favors the deterministic multisine with high PAPR. On the contrary, there is some randomness involved in the modulated waveform that produces fluctuations to the rectifier and leads to some power loss \cite{Clerckx2018}. This phenomenon is in sharp contrast to the conclusion based on linear harvester model that both waveforms are equally suitable for WPT \cite{Xu2014a}. As shown in the plots, even with the assumption that the deterministic power waveform creates some interference to the information waveform, the rate loss is compensated by the power gain such that the superposed waveform strictly outperforms the modulated waveform.

Another observation is the performance gap between the ideal superposed waveform and its lower bound widens as SNR increases. This is as expected because the rate is dominated by noise at low SNR and by interference at high SNR. For ${\text{SNR}} = 10$ dB, the interference is much lower than noise even if a large amount of power is allocated to the multisine component. Hence, the curves almost overlap with each other. On the other hand, at a higher SNR, the rate loss by interference increases while the energy benefit of power waveform remains unchanged. To obtain the optimal R-E tradeoff, the transmitter tends to allocate less power to the multisine so that the harvested current drops. Therefore, the rate boost of deterministic power waveform grows as SNR increases.

The plots also demonstrate that for the superposed waveform with a sufficiently large $N$, TS is preferred at low SNR and PS is favored at high SNR, while a combination of TS and PS is generally optimal for medium SNR. On the other hand, the R-E region is strictly convex for the modulated waveform-only transmission due to its inefficiency to boost the harvested energy. It corresponds to the conventional opinion that PS always outperforms TS for no power waveform transmission. In this case, the R-E curve is approximately straight at low SNR but with large curvature at high SNR. The reason is that at a low SNR, the water-filling strategy concentrates the power to the best subband to maximize the rate, and the region boundary is obtained by varying $\rho $ only. In comparison, more subbands are utilized in the transmission as SNR increases. Therefore, only a small portion of power is required to achieve a decent rate while the remaining part can be used to boost the output current.

A comparison between the results over FF and FS channels emphasizes the benefit of frequency selectivity on the harvested current. The gain is more significant for modulated waveform (around \SI{1}{\uA}) than superposed waveform (around \SI{0.25}{\uA}). One possible reason is that the power are concentrated in few subbands such that the number of terms in \eqref{eqn:power_waveform_fourth_order} and \eqref{eqn:information_waveform_fourth_order} are comparable. In such cases, the impact of channel amplitude on each monomial is more significant on the information waveform and contributes to a larger gain in the harvested current.

\subsection{R-E Region vs PAPR}\label{sec:re-region-vs-papr}
Fig. \ref{fig:re-papr} investigates the relationship between PAPR and R-E region for $N = 8, 16$ over the FF and FS channels.

\begin{figure}[ht]
  \centering
  \subfigure[FF: $N = 8$]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_papr_8}\label{fig:re-ff-papr-8}}
  \subfigure[FF: $N = 16$]{
    \includegraphics[width=0.48\textwidth]{siso_re_ff_papr_16}\label{fig:re-ff-papr-16}}
  \quad
  \subfigure[FS: $N = 8$]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_papr_8}\label{fig:re-fs-papr-8}}
  \subfigure[FS: $N = 16$]{
    \includegraphics[width=0.48\textwidth]{siso_re_fs_papr_16}\label{fig:re-fs-papr-16}}
  \caption{R-E region vs PAPR for FF channel}
  \label{fig:re-papr}
\end{figure}

A first observation is that a large enough PAPR is required to fully exploit the power gain of the multisine waveform. For $N = 16$, the R-E region is convex for a PAPR no larger than 20 dB and is concave-convex when it increases to 30 dB. For instance, with a small PAPR of 10 dB, the use of multisine waveform is strictly constrained such that the modulated waveform dominates the transmit signal. Hence, the corresponding R-E plot is similar to the result without power waveform. On the contrary, a PAPR of 30 dB is large enough to achieve the optimal performance of the superposed signal in the low-rate region. It can be concluded that the energy benefit of the multisine waveform indeed comes from the high PAPR. In each cycle, the peak pushes the rectifier output voltage to a high level which decreases slowly in the rest of the period.

A contrast of the R-E plots of $N = 8$ and 16 also suggests a larger $N$ requires higher PAPR to improve the performance. When PAPR increases from 20 to 30 dB, the current gain is more significant for $N = 16$ than $N = 8$. On the other hand, the impact of PAPR for small $N$ is not as significant as for large $N$. This verifies the positive correlation between $N$ and PAPR discussed in Section \ref{sec:rectifier-behavior}. Although increasing $N$ in a proper range can effectively boost the harvested energy, the PAPR constraint may limit the use of a large $N$ in practice.

It is interesting to notice that the frequency selectivity helps to achieve the optimal R-E region with a lower PAPR. As shown in Fig. \ref{fig:re-fs-papr-8}, a PAPR constraint of 20 dB is enough to guarantee the best tradeoff and a larger budget is unnecessary. However, the transmission over the FF channel requires a larger PAPR of 30 dB to achieve optimum behavior. This is because frequency selective channel can further amplify the difference of frequency components such that the received signal is with enhanced PAPR.

The main disadvantage of the proposed approach is that the oversampling procedure further increases the overall computational complexity in the optimization.


\documentclass[./\jobname.tex]{subfiles}
\begin{document}

\section{Multimodality and Symmetry}
\label{chap:multimodality_and_symmetry}
The optimisation algorithm tries to find the best approximation of $N$ kernels to the solution of a differential equation. Assume, that a kernel $K$ is fully defined by a vector of parameters $\mathbf{p}$. The best fit is defined as
\begin{equation}
\mathbf{\hat{p}_{apx}} = \left[\underbrace{\left[ \mathbf{\hat{p}}_{K_0} \right] }_{\text{kernel 0}}, \cdots \underbrace{\left[ \mathbf{\hat{p}}_{K_i} \right] }_{\text{kernel i}}, \cdots \underbrace{\left[ \mathbf{\hat{p}}_{K_N} \right]}_{\text{kernel N}} \right]^T
\end{equation}
where the parameters $\mathbf{\hat{p}}$ of every kernel are chosen optimally. The optimal kernel functions $K(\mathbf{\hat{p}}_{K_i}, \mathbf{x})$ are summed up to form the optimal approximation $\hat{u}_{apx}(\mathbf{x})$.
\begin{equation}
\label{eq:uapx_kernel_sum}
\hat{u}_{apx}(\mathbf{x}) = \sum_{i=0}^{N} K(\mathbf{\hat{p}}_{K_i}, \mathbf{x})
\end{equation}
Since the order of the summation is irrelevant, any kernel-wise permutation describes an optimal solution $\mathbf{\hat{p}_{apx}}$. Thus, the fitness function $F(u_{apx}(\mathbf{x}))$ has at least $N!$ number of local optima and all of them share the same function value. Further, a symmetry in the location of the optima is observed. All optima lay on the surface of the hypersphere that is centred at the origin and has a radius of $r = || \mathbf{\hat{p}_{apx}} ||$. The following 3D plot in figure \ref{fig:optima_distribution} shows an exemplary distribution of optima on the fitness function. For the sake of simplicity, a kernel now consists of only one parameter. As an example, the vector $\mathbf{\hat{p}_{apx}} = \left[ 2, 1, -1 \right]^T$ describes an optimal solution that consists of 3 kernels. Any permutation of these three coordinates is itself a perfect fit.
\begin{figure}[h]
	\centering
	\noindent\adjustbox{max width=0.7\linewidth}{
		\includegraphics[width=\textwidth]{../img/pdf/symmetry.pdf}
	}
	\unterschrift{Distribution of exemplary optima on the fitness function in 3D space.}{}{}
	\label{fig:optima_distribution}
\end{figure}
This symmetry is independent of the kernel type. Large parts of the fitness function, such as the weighting and penalty factors or the number of collocation points, have no influence on the actual radial arrangement of the optima. The symmetry is a fundamental property of the sum of \gls{rbf}. Thus, it could even be applied in other fields that use this kind of representation. One of these could be non-linear function approximation. \\ Obviously, the order of summation is not only true for the optimum, but also for every other point in between. Thus, the entire fitness function exhibits this radial symmetry. More work investigating the structure of the fitness function must be done. With more knowledge about the features of this function, it might be possible to design algorithms specifically to that problem.

\end{document}


\documentclass[11pt, oneside]{article}   	% use "amsart" instead of "article" for AMSLaTeX format

% \usepackage{draftwatermark}
% \SetWatermarkText{Draft}
% \SetWatermarkScale{5}
% \SetWatermarkLightness {0.95}
% \SetWatermarkColor[rgb]{0.7,0,0}

\usepackage{geometry}                		% See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   		% ... or a4paper or a5paper or ...
%\geometry{landscape}                		% Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    		% Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}				% Use pdf, png, jpg, or eps� with pdflatex; use eps in DVI mode
								% TeX will automatically convert eps --> pdf in pdflatex
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{hyperref}
\usepackage{url}
\usepackage{authblk}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{fixltx2e}
\usepackage{hyperref}
\usepackage{alltt}
\usepackage{color}
\usepackage{bigints}

\newcommand{\argmax}{\operatornamewithlimits{argmax}}
\newcommand{\argmin}{\operatornamewithlimits{argmin}}

\title{Notes on MSE Gradients for Neural Networks}
\author{David Meyer \\ dmm@\{1-4-5.net,uoregon.edu,...\}}
% \date{17 Jan 2016}

\begin{document}
\maketitle

\section{Introduction}

\section{Mean Squared Error (MSE)}
\noindent
First, notation: scalars are represented in regular math font, e.g., $y_i$, where vectors are in bold, e.g., $\mathbf{x}_i$. Given these definitions we can define our \emph{labelled data} or \emph{training examples} as a set of $n$ tuples where the $i^\text{th}$ tuple has the form $(\mathbf{x}_i,y_i)$, where $\mathbf{x}_i \in \mathbb{R}^n$ is a vector of inputs and $y_i \in \mathbb{R}$ is the observed output.

\bigskip
\noindent
Ideally our neural network should output $y_i$ when given $\mathbf{x}_i$ as an input. Of course, during training this doesn't always happen so we need to define an \emph{error or cost} function that quantifies the difference between the actual observed output and the prediction of the neural network. A simple measure of the error is the Mean Squared Error, or MSE.  We define the MSE as follows:

\begin{flalign}
E := \frac{1}{m} \sum\limits_{i = 1}^{m} (h(\mathbf{x}_i) - y_i)^2
\end{flalign}

\bigskip
\noindent
where $h(\mathbf{x}_i)$ is the output of the neural network.

\section{Basic Building Blocks: Perceptrons}
\noindent
The simplest classifiers out of which we will build our neural network are perceptrons \cite{Rosenblatt1958}. In reality, a perceptron is a linear classifier.   A perceptron takes an input vector $\mathbf{x}$ which is multiplied pairwise by  a weight vector $\mathbf{w}$, then sums the products up together with a bias term $b$. This sum (the \emph{dot product}, Equation \ref{eqn:dot_product}) is then fed through  an activation function $\sigma:\mathbb{R} , \mathbb{R}$. This is depicted in Figure \ref{fig:perceptron}.  Note here that $w_0 = b$ and $a_0 = 1$;  I like Figure \ref{fig:perceptron} but I will use the more conventional $\mathbf{x}$ for the input vector  rather than $\mathbf{a}$ as is used in the figure.  The behavior of the perceptron can then be described as $\sigma(\mathbf{w} \cdot \mathbf{x})$, where $\mathbf{w}$ and $\mathbf{x}$ have the following form\footnote{Again noting that $\mathbf{a}$ in Figure \ref{fig:perceptron} is frequently
called $\mathbf{x}$, the input vector; I'll use $\mathbf{x}$ here.}:

\begin{flalign*}
\boldsymbol{w}  & = \begin{bmatrix}  w_{1}  \\ w_{2} \\  \vdots \\ w_{n}   \end{bmatrix}  \\ \\
\boldsymbol{x}   & = \begin{bmatrix}  x_{1}  \\  x_{2}   \\ \vdots \\  x_{n}   \end{bmatrix}
\end{flalign*}

\bigskip
\noindent
$\mathbf{w}^\text{T}$ ($\mathbf{w}$ transpose) is defined to be

\begin{flalign*}
\boldsymbol{w}^T  & = \begin{bmatrix}  w_{1}, & w_{2}, & \hdots, &  w_{n}  \end{bmatrix}
\end{flalign*}

\bigskip
\noindent
The  \emph{dot product} between $\mathbf{w}$ and $\mathbf{x}$, $\mathbf{w} \cdot \mathbf{x}$, is defined as

\begin{equation*}
\label{eqn:dot_product}
\mathbf{w} \cdot \mathbf{x} = \boldsymbol{w}^T\boldsymbol{x} =
\left[\begin{array}{cccc} w_{1}, & w_{2}, & \cdots, & w_{n} \end{array} \right]
\left[ \begin{array}{cccc} x_1 \\ x_2 \\ \vdots \\ x_n \end{array} \right] =
\sum\limits_{i = 1}^{n}w_{i}x_{i} = w_{1}x_1 + w_{2}x_2 +  \ldots + w_{n}x_n
\end{equation*}

\bigskip
\noindent
Note that the weight vector, $\mathbf{w}$ will be a $M \times N$ matrix if there is more than one layer of artificial neurons.

\bigskip
\noindent
The last piece of the puzzle are the kinds of activation functions\footnote{Activation functions are sometimes called \emph{link} functions in a Generalized Linear Model setting.} that $\sigma$ might be:
\begin{itemize}
\item Sigmoid: $\sigma(x) = \frac{1}{1+e^{-x}}$
\item Hyperbolic tangent: $\sigma(x) = \tanh(x)$
\item Linear: $\sigma(x) = x$
\item  Rectified Linear Unit: $\sigma(x) = \max(0,x)$
\item  Exponential Linear Unit:
$\sigma(x) =\left \{
        \begin{array}{ll}
		x   & \text{if } x \ge 0 \\
		a(e^x - 1)     & \text{otherwise }
	\end{array}
\right. $
\item ...
\end{itemize}

\begin{figure}
\center{\includegraphics[scale=0.8] {images/perceptron.png}}
\caption{Basic Perceptron/Linear Classifier}
\label{fig:perceptron}
\end{figure}

\section{Building a Single Layer Neural Network}
\noindent
So far we've defined the error $E$ as the MSE, namely, $E := \frac{1}{m} \sum\limits_{i = 1}^{m} (h(\mathbf{x}_i) - y_i)^2$. Here both the error and the output of the network
($h_{\mathbf{w}}(\mathbf{x}_i) = \sigma(\mathbf{w} \cdot \mathbf{x}_i)$) depend on the weight vector $\mathbf{w}$. We write the error function, parameterized by $\mathbf{w}$, as

\begin{flalign}
E(\mathbf{w}) := \frac{1}{m} \sum\limits_{i = 1}^{m} (h_{\mathbf{w}}(\mathbf{x}_i)  - y_i)^2
\end{flalign}

\bigskip
\noindent
Now, our goal is to find a weight vector $\mathbf{w}$ such that $E(\mathbf{w})$ is minimized. In effect this means that the perceptron will correctly predict the output for the inputs in the training set. Of course, we want the perceptron to  \emph{generalize}, so that it makes correct predictions on the test set and on new examples. But how to do this
minimization?

\begin{figure}
\center{\includegraphics[scale=0.65] {images/optimization.jpg}}
\caption{Non-Convex Error Surface}
\label{fig:non-convex}
\end{figure}

\bigskip
\noindent
We do the minimization by applying the \emph{gradient descent} algorithm.  In effect we will treat the error as a surface in $n$-dimensional space and search for the greatest downwards slope at the current point $\mathbf{w}_t$ and will go in that direction to obtain $\mathbf{w}_{t+1}$. Following this process we will hopefully find a minimum point on the error surface and we will use the coordinates of that point as the final weight vector\footnote{Consider, however, the situation in which the error surface is non-convex, such as is in Figure \ref{fig:non-convex}.}. In any event, the update rule can be stated as follows (in both \emph{partial derivative} and \emph{gradient} notations):

\begin{flalign}
\mathbf{w}_{t+1} & := \mathbf{w}_t - \eta \frac{\partial E(\mathbf{w})}{\partial{\mathbf{w}}}
\quad \qquad \qquad  \mathbin{\#} \text{partial derivative notation} \\
\mathbf{w}_{t+1} & := \mathbf{w}_t - \eta \nabla_{\mathbf{w}} E(\mathbf{w})
\:  \qquad \qquad  \mathbin{\#} \text{gradient (nabla) notation}
\end{flalign}

\bigskip
\noindent
where $\eta$ is the \emph{learning rate}. Now, notice that the \emph{gradient} of $E$ on $w$ is

\begin{flalign}
\nabla_{w} E(\mathbf{w}) = \frac{\partial E(\mathbf{w})}{\partial{\mathbf{w}}} = \Bigg[\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_0}}},
\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_1}}}, \cdots,  \frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_n}}} \Bigg ]
\end{flalign}

\bigskip
\noindent
Now we can calculate the gradient, $\nabla_{\mathbf{w}} E(\mathbf{w})$.  We start by calculating $\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_j}}}$ for each $j$.  So first....
Note that the \emph{chain rule} states that if $h(x) = f(g(x))$ then the derivative $\frac{d h(x)}{dx} = h^\prime(x) = f^\prime(g(x)) g^\prime(x)$.  We will also use the \emph{power rule}: If $y = u^n$, then $\frac{dy}{dx} = n u^{n-1} \frac{du}{dx}$. So the partial derivative $\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_j}}}$ can be computed as follows: So for example element of the gradient $0 \le j \le n$

\begin{flalign}
\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_j}}} & = \frac{\partial}{\partial w_j}
\frac{1}{m}\sum\limits_{i = 1}^m (h_{\mathbf{w}}(\mathbf{x}_i) - y_i)^2
 \qquad \qquad \qquad \qquad  \qquad \mathbin{\#} \text{definition of } E\\
&=\frac{1}{m}\ \sum\limits_{i = 1}^m 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \frac{\partial}{\partial w_j} (h_{\mathbf{w}}(\mathbf{x}_i) - y_i)  \qquad \quad \:  \: \mathbin{\#} \text{power rule} \\
&= \frac{1}{m}\ \sum\limits_{i = 1}^m 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \frac{\partial}{\partial w_j} \sigma(\mathbf{w} \cdot \mathbf{x}_i)  \; \quad \qquad \qquad  \mathbin{\#} h_{\mathbf{w}}(\mathbf{x}_i) = \sigma(\mathbf{w} \cdot \mathbf{x}_i) \\
&= \frac{1}{m}\ \sum\limits_{i = 1}^m 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \sigma^\prime (\mathbf{w} \cdot \mathbf{x}_i)  \frac{\partial}{\partial w_j} \mathbf{w} \cdot \mathbf{x}_i  \: \:  \qquad  \mathbin{\#} \text{chain rule} \\
&= \frac{1}{m}\ \sum\limits_{i = 1}^m 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \sigma^\prime (\mathbf{w}  \cdot \mathbf{x}_i)\frac{\partial}{\partial w_j} \sum\limits_{k =1}^{n} w_k x_{i,k}  \:  \mathbin{\#} \text{defn dot product} \\
&= \frac{1}{m}\ \sum\limits_{i = 1}^m 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \sigma^\prime (\mathbf{w} \cdot \mathbf{x}_i) x_{i,j}   \qquad \quad \quad \quad   \mathbin{\#} \frac{\partial w_k x_{i,k}}{\partial w_j} \ne 0 \text { when } k = j
\end{flalign}

\bigskip
\noindent
Note that going from Equation 7 to Equation 8 uses the \emph{sum rule}

\bigskip
\begin{flalign}
\frac{d}{dx} (f(x) + g(x)) &= \frac{d}{dx} f(x) + \frac{d}{dx} g(x)
\end{flalign}

\bigskip
\noindent
Here $f(x) = h_{\mathbf{w}}(\mathbf{x}_i)$ and $g(x) = -y_i$. $\frac{\partial}{\partial w_j} h_{\mathbf{w}}(\mathbf{x}_i) = \frac{\partial}{\partial w_j}  \sigma(\mathbf{w} \cdot \mathbf{x}_i)$ and $\frac{\partial y_i}{\partial w_j} = 0$, so we're left with term  $\frac{\partial}{\partial w_j} \sigma(\mathbf{w} \cdot \mathbf{x}_i)$ as we see in Equation 8.

\bigskip
\noindent
Now, using the sigmoid activation function $\sigma(x) = \frac{1}{1+e^{-x}}$, who's derivative
$\sigma^\prime(x) = \sigma(x) (1 - \sigma(x))$, gives us
\begin{flalign}
\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w_j}}}
& = \frac{2}{m} \sum\limits_{i = 1}^m (h_{\mathbf{w}}(\mathbf{x}_i) - y_i)  \sigma^\prime (\mathbf{w} \cdot \mathbf{x}_i) x_{i,j}  \\
&=  \frac{2}{m} \sum\limits_{i = 1}^m (\sigma(\mathbf{w} \cdot \mathbf{x}_i) -y_i) \sigma(\mathbf{w} \cdot \mathbf{x})  (1 - \sigma(\mathbf{w} \cdot \mathbf{x})) x_{i,j} \\
\end{flalign}

\noindent
Now, we can compute the gradient $\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w}}}$ as follows:

\begin{flalign}
\frac{\partial E(\mathbf{w})}{\partial{\mathbf{w}}} &=\frac{2}{m} \sum\limits_{i = 1}^m (\sigma(\mathbf{w} \cdot \mathbf{x}_i) -y_i) \sigma(\mathbf{w} \cdot \mathbf{x}_i)  (1 - \sigma(\mathbf{w} \cdot \mathbf{x}_i)) \mathbf{x}_i \\
\end{flalign}

\noindent
Finally, let the update rate $\eta = 0.1$. Then the update to $\mathbf{w}$ is computed as
\begin{flalign}
\mathbf{w}_{t+1} := \mathbf{w}_t - \frac{0.2}{m} \sum\limits_{i = 1}^m (h_\mathbf{w}(\mathbf{x}_i) - y_i)
h_\mathbf{w}(\mathbf{x}_i ) (1 - h_\mathbf{w}(\mathbf{x}_i ) ) \mathbf{x}_i
\end{flalign}
where $h_\mathbf{w}(\mathbf{x}_i) = \sigma(\mathbf{w}_t \cdot \mathbf{x}_i)$.

\section{What About Multilayer Networks?}

Consider a more general multilayer neural network, such as shown in Figure \ref{fig:multi-layer-perceptron}. Here we are using the notation $w_{i , j}$ to denote the weights on the connection between perceptrons (nodes) $i$ and $j$. Note that the notation  $w_{i \rightarrow j} \equiv w_{i, j}$ in Figure \ref{fig:multi-layer-perceptron}.  Now,  armed with this notation we can write the sum of the inputs to perceptron (node) $j$ as

\begin{flalign}
s_j := \sum\limits_{k} z_k w_{k, j}
\end{flalign}

\noindent
Here $k$ iterates over all the perceptrons connected to $j$. The output of $j$ is written as $z_j = \sigma (s_j)$, where $\sigma$ is $j$'s activation (link) function.

\bigskip
\noindent
Now, we can use the same error (cost) function for the multlayer network, $E(\mathbf{w})$

\begin{flalign}
E(\mathbf{w}) := \frac{1}{m} \sum\limits_{i = 1}^{m} (h_{\mathbf{w}}(\mathbf{x}_i)  - y_i)^2
\end{flalign}
\noindent
except that now $\mathbf{w}$ is a matrix that contains all the weights for the network: \\
$\mathbf{w} = [w_{i , j}] \; \forall i,j$.

 \bigskip
 \noindent
The goal is again to find the $\mathbf{w}$ that minimizes $E(\mathbf{w})$ using gradient descent. So we need to calculate $\frac{\partial E(\mathbf{w})}{\partial \mathbf{w}}$. The first step is to separate the contributions of each of the $m$ training examples using the following observation:

\begin{flalign}
\frac{\partial E(\mathbf{w})}{\partial \mathbf{w}} = \frac{1}{m}\sum\limits_{i = 1}^m \frac{\partial E_i(\mathbf{w})}{\partial \mathbf{w}}
\end{flalign}

\noindent
where $E_i(\mathbf{w}) = (h_{\mathbf{w}}(\mathbf{x}_i) - y_i)^2$. Then

\begin{flalign}
\frac{\partial E_i(\mathbf{w})}{\partial w_{j , k}}
&= \frac{\partial}{\partial w_{j , k}} (h_{\mathbf{w}}(\mathbf{x}_i) - y_i)^2  \qquad \qquad \qquad \qquad  \: \: \mathbin{\#} \text{definition of } E\\
& = 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial w_{j , k}} \\
& = 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_k} \frac{\partial s_k}{\partial w_{j , k}}  \qquad \quad \quad \quad   \mathbin{\#} \text{chain rule}
\label{eqn:cr} \\
&= 2 (h_{\mathbf{w}}(\mathbf{x}_i) - y_i) \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_k} z_j
\label{eqn:z}
\end{flalign}

\begin{figure}
\center{\includegraphics[scale=0.6] {images/multi-layer-perceptron.png}}
\caption{Multi-Layer Perceptron}
\label{fig:multi-layer-perceptron}
\end{figure}

\bigskip
\noindent
Note that in going from Equation \ref{eqn:cr} to  Equation \ref{eqn:z},  $s_k = \sum\limits_{i} z_i w_{i , k}$, so $\frac{\partial s_k}{\partial w_{j , k}} \ne 0$ where $i = j$ and 0 otherwise.

\bigskip
\noindent
Now, if the $k^\text{th}$ node is an output node,  then

\bigskip
\begin{flalign}
\frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_k}
&= \frac{\partial \sigma(s_k)}{\partial s_k}  = \sigma^\prime(s_k)
\end{flalign}

\bigskip
\noindent
so that
\begin{flalign}
\frac{\partial E_i(\mathbf{w})} {\partial w_{j , k}} = 2 (h{\mathbf{w}}(\mathbf{x}_i) -y_i) \sigma^\prime(s_k) z_j
\end{flalign}

\bigskip
\noindent
On the other hand, if $k$ is not an output node, then changes to $s_k$ can affect all the nodes which are connected to $k$'s output, as follows

\begin{flalign}
\frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_k} &= \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial z_k} \frac{\partial z_k}{\partial s_k}  \qquad \qquad \quad \qquad \qquad \qquad   \mathbin{\#} \text{chain rule again} \\
&= \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial z_k}  \sigma^\prime(s_k)
 \qquad  \qquad \qquad  \qquad \qquad  \mathbin{\#}  z_k = \sigma(s_k) \\
 &= \sum\limits_{o \in \{v \mid v \rightarrow  k\}} \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_o} \frac{\partial s_o}{\partial z_k} \sigma^\prime(s_k)  \qquad \quad \quad  \:  \mathbin{\#} v \text{ is connected to } k
\label{eqn:zo0} \\
 &= \sum\limits_{o \in \{v \mid v \rightarrow  k\}} \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_o} w_{k , o}  \sigma^\prime(s_k)  \qquad \quad \quad  \:  \mathbin{\#} s_o = \sum\limits_{i} z_i w_{i , o}
\label{eqn:zo1}
\end{flalign}

\bigskip
\noindent
Note that in going from Equation \ref{eqn:zo0} to Equation \ref{eqn:zo1} we see that
\bigskip
\begin{flalign}
\frac{\partial s_o}{\partial z_k} &= \frac{\partial}{\partial z_k} \sum\limits_i z_i w_{i , o}
\end{flalign}
\bigskip
\noindent
which is only non-zero when $i = k$, so that $\frac{\partial s_o}{\partial z_k} = w_{k , o}$ (Equation \ref{eqn:zo1}).

\noindent
So what is left is to calculate $s_k$ and $z_k$ (feeding forward) and then work backwards from the output calculating
$\frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_k}$ and back propagate the error down the network ("backprop"). The summary looks like:

\bigskip
\begin{itemize}
\item $k$ is an output node: $\frac{\partial E_i(\mathbf{w})} {\partial w_{j , k}} = 2 (h{\mathbf{w}}(\mathbf{x}_i) -y_i) \sigma^\prime(s_k) z_j$
\item otherwise: $\frac{\partial E_i(\mathbf{w})} {\partial w_{j , k}} = 2 (h{\mathbf{w}}(\mathbf{x}_i) -y_i) \sigma^\prime(s_k) z_j \sum\limits_{o \in \{v \mid v , k\}} \frac{\partial h_{\mathbf{w}}(\mathbf{x}_i)}{\partial s_o} w_{k , o}$
\end{itemize}

\bigskip
\noindent
Using these results we see that

\bigskip
\begin{flalign}
\frac{\partial E_i(\mathbf{w})}{\partial \mathbf{w}} = \Bigg [\frac{\partial E_i(\mathbf{w})} {\partial w_{j , k}} \Bigg ] \: \: \forall j,k
\end{flalign}

\bigskip
\noindent
Finally, the weights can be updated in batch mode, in which case the update rule  for batch size of $m$ is
\begin{flalign}
\mathbf{w}_{t+1} & := \mathbf{w}_t - \eta \frac{\partial E(\mathbf{w})}{\partial \mathbf{w}} \\
& := \mathbf{w}_t - \eta \sum\limits_{i = 1}^m \frac{\partial E_i(\mathbf{w})}{\partial \mathbf{w}}
\end{flalign}

\bigskip
\noindent
Or if we take a Stochastic Gradient Descent (SGD) approach (one training example at a time):
\begin{flalign}
\mathbf{w}_{t+1} & := \mathbf{w}_t - \eta \frac{\partial E(\mathbf{w})}{\partial \mathbf{w}}
\end{flalign}

\bigskip
\section{Acknowledgements}
\noindent
Thanks to Armin Wasicek for his careful reading of earlier versions of this document.

% \newpage
\bibliographystyle{plain}
\bibliography{/Users/dmm/papers/bib/ml}

\end{document}


\documentclass{beamer}
\usetheme{Warsaw}
\usepackage{nhtvslides}
\usepackage{graphicx}
\usepackage{listings}
\lstset{language=CAML,
basicstyle=\ttfamily\footnotesize,
frame=shadowbox,
breaklines=true}
\usepackage[utf8]{inputenc}

\title{Integration, rotation matrices, and quaternions}

\author{Dr. Giuseppe Maggiore}

\institute{NHTV University of Applied Sciences \\
Breda, Netherlands}

\date{}

\begin{document}
\maketitle

\begin{frame}{Table of contents}
\tableofcontents
\end{frame}

\section{Integration and rotation}
\begin{slide}{Integration and rotation}{Integration and rotation}{
\item We integrate $L$ from $\tau$
\item We integrate $w$ from $L$ and $J^-1$
\item We integrate $R$ from $w$
}\end{slide}

\begin{slide}{Integration and rotation}{Issues}{
\item \textit{Issue 1:} recomputing $J$ from every particle of the body and at every tick is too slow
\item \textit{Issue 2:} integrating $R$ slowly ``breaks it'', meaning that $[T\ N\ B]$ is not a rotation anymore
}\end{slide}

\section{Caching J}
\begin{slide}{Caching J}{Issue 1: body shape}{
\item $J$ represents the tendency of the body to resist rotation
\item But the body just moves and rotates, it does not change shape
\item We should be able to simply recompute $J$ from its initial value and the rotation
}\end{slide}

\begin{frame}{Caching J}
\begin{block}{$J$ from $J_{body}$}
\begin{eqnarray}
J &=& \sum_i(|r_i(t)|^2I - r_i(t) r_i^T(t)) \\
&=& \sum_i(|R r_i(t_0)|^2I - R r_i(t_0) r_i^T(t_0) R^T) \\
&=& \sum_i(|r_i(t_0)|^2I - R r_i(t_0) r_i^T(t_0) R^T) \\ % rotation keeps length
&=& \sum_i(|r_i(t_0)|^2 R R^T - R r_i(t_0) r_i^T(t_0) R^T) \\ % R R^T = I
&=& \sum_i(R |r_i(t_0)|^2 R^T - R r_i(t_0) r_i^T(t_0) R^T) % scalar and matrix multiplication is commutative
\end{eqnarray}
\end{block}
\end{frame}

\begin{frame}{Caching J}
\begin{block}{$J$ from $J_{body}$}
\begin{eqnarray}
J &=& \sum_i(R |r_i(t_0)|^2 R^T - R r_i(t_0) r_i^T(t_0) R^T) \\ % scalar and matrix multiplication is commutative
&=& \sum_i R(|r_i(t_0)|^2 - r_i(t_0) r_i^T(t_0)) R^T \\ % grouping
&=& R \left( \sum_i (|r_i(t_0)|^2 - r_i(t_0) r_i^T(t_0)) \right) R^T \\ % grouping
&=& R J_{body} R^T
\end{eqnarray}
\end{block}
\end{frame}

\begin{slide}{Caching J}{$J^{-1}$ from $J_{body}{-1}$}{
\item We need $J^{-1}$ more than $J$
\item We can compute it without an inversion though
\item We compute (just once) $J_{body}^{-1}$
\item $J^{-1} (R J_{body} R^T)^{-1} = R J_{body}^{-1} R^T$
}\end{slide}

\section{Orthonormalization of $R$}
\begin{slide}{Orthonormalization of $R$}{Issue 2: $R$ is a rotation matrix}{
\item $R = [T\ N\ B]$
\item The vectors $T$, $N$, and $B$ must be of unit length and orthonormal ($T \cdot N = 0, T \cdot B = 0, \dots$)
\item As we integrate $R$, this stops being true
\item $R$ stops being just a rotation matrix, and also incorporates some scale and some skew
\item This is very bad!
}\end{slide}

\begin{frame}{Boxes stop being boxes}
\center
\includegraphics[height=5cm]{Pics/Kill_me_please.png}
\end{frame}

\begin{slide}{Orthonormalization of $R$}{Gram-Schimdt method}{
\item $\hat R = [\hat T\ \hat N\ \hat B]$
\item We normalize $T = \frac{\hat T}{|\hat T|}$
\item We remove the projection of $\hat N$ over $T$: $\hat N' = \hat N - \hat N (\hat N \cdot T)$
\item We normalize $N = \frac{\hat N'}{|\hat N'|}$
\item We compute $B = T \times N$
}\end{slide}

\begin{slide}{Orthonormalization of $R$}{Gram-Schimdt method}{
\item Lot of useless computation
\item Matrix representation very redundant
\item $\hat B$ is completely unneeded and is ignored!
}\end{slide}

\section{Using quaternions}
\begin{slide}{Using quaternions}{About quaternions}{
\item We just need a single rotation around an axis (which is an arbitrary rotation in 3D)
\item Less degrees of freedom means less drift and less need for normalization
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item Let us consider a rotation matrix around the Z axis
\item $R_0 = \left[ \begin{matrix}
\cos \theta & -\sin \theta & 0 \\
\sin \theta & \cos \theta & 0 \\
0 & 0 & 1\\
\end{matrix} \right] =
\left[ \begin{matrix}
c & -s & 0 \\
s & c & 0 \\
0 & 0 & 1 \\
\end{matrix} \right]$
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item Any vector $v$ on the XY plane rotated by $R_0$ remains on XY, rotated around Z by an angle $\theta$
\item Any vector $v$ on the Z axis rotated by $R_0$ remains (identical) on Z
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item Let us transform this rotation into an arbitrary rotation $R_1$ around an axis $d$ and of angle $\theta$
\item We consider an \textit{orthonormal basis} $a,b,d$
\item We perform a \textit{basis change} of $R_0$ onto $a,b,d$
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item We need to build $R_1$ such that
\begin{itemize}
\item $R_1 a = ca + sb$
\item $R_1 b = -sa + cb$
\item $R_1 d = d$
\end{itemize}
\item That is $R_1 \underbrace{[a\ b\ d]}_P = \underbrace{[a\ b\ d]}_P R_0$
\item $P^{-1} = P^T$, so $R_1 = P R_0 P^T = c(aa^T + bb^T) + s(ba^T - ab^T) + dd^T$
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item $R_1 = c(aa^T + bb^T) + s(ba^T - ab^T) + dd^T$
\item This means that we can now construct a rotation matrix from an axis and an angle
\item Unfortunately we also need two ``dummy'' vectors $a$ and $b$
\item Any pair of those suffices, as long as they are correctly related to $d$
\item We now remove the explicit dependency on them
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item Consider a vector expressed in relationship to $a,b,d$: $v = \underbrace{\alpha}_{a \cdot v} a + \underbrace{\beta}_{b \cdot v} b + \underbrace{\delta}_{d \cdot v} d$
\item We now consider two arbitrary but useful quantities:
\item $d \times v = d \times (\alpha a + \beta b + \delta d) = \alpha d \times a + \beta d \times b + \delta d \times d = -\beta a + \alpha b = Dv = \left[ \begin{matrix}
0 & -d_z & d_y \\
d_z & 0 & -d_x \\
-d_y & d_x & 0 \\
\end{matrix} \right] v$
\item $d \times (d \times v) = d \times (-\beta a + \alpha b) = -\alpha a - \beta b = D^2v$
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item Now let us study the progression of $Iv, Dv, D^v$
\item $Iv = v = \alpha a + \beta b + \delta d = aa^Tv + bb^Tv + dd^Tv$, so $I = aa^T + bb^T + dd^T$
\item $Dv = \alpha b - \beta a = ba^Tv - ab^Tv$, so $D = ba^T-ab^T$
\item $D^2v = -\alpha a - \beta b = \delta d - v = dd^Tv - v$, so $D^2 = dd^T - I$
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item We combine those three into $R_1 = c(aa^T + bb^T) + s(ba^T - ab^T) + dd^T$
\item $I = aa^T + bb^T + dd^T$, $D = ba^T-ab^T$, $D^2 = dd^T - I$
\pause
\item $R_1 = c(I - dd^T) + sD + dd^T$
\item $R_1 = I + sD + (1-c)D^2$
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item We can now compute $R_1 = I + sD + (1-c)D^2$ from just $d, \theta$
\item Moreover, we can rotate a vector without even building $R_1$
\begin{eqnarray}
R_1v &=& (I + s D + (1-c)D^2)v \\
&=& Iv + s D v + (1-c)D^2 v \\
&=& v + s d\times v + (1-c)d \times (d \times v)
\end{eqnarray}
}\end{slide}

\begin{slide}{Using quaternions}{Quaternion primer}{
\item Any representation of $d$ and $\theta$ would be fine
\item A particularly convenient one is $[\cos \frac{\theta}{2}, \sin \frac{\theta}{2} d]$
}\end{slide}

\section{Time derivative of a quaternion}
\begin{slide}{Deriving quaternions}{Derivative of rotation matrix}{
\item We studied how to compute $\dot R$ from $R$ and $\omega$
\item If we represent rotations with quaternions, we need to compute $\dot q$
}\end{slide}

\begin{slide}{Deriving quaternions}{Derivative of quaternion}{
\item Given the current orientation $q(t_0)$ and angular velocity $\omega(t_0)$
\item Assuming very small $t-t_0$
\item The derivative of the original quaternion is $\frac{d}{dt}\underbrace{[\cos \frac{|\omega(t_0)|(t-t_0)}{2}, \frac{\omega(t_0)}{|\omega(t_0)|} \sin \frac{|\omega(t_0)|(t-t_0)}{2}]}_{\omega \text{ converted to a quaternion}} q(t_0)$
}\end{slide}

\begin{slide}{Deriving quaternions}{Derivative of quaternion}{
\item $\frac{d}{dt}[\cos \frac{|w(t_0)|(t-t_0)}{2}, \frac{w(t_0)}{|w(t_0)|} \sin \frac{|w(t_0)|(t-t_0)}{2}] q(t_0)$
\item We now derive the $w$ part ($q(t_0)$ is a constant) and replace $t$ with $t_0$ because we want to know the value of the derivative at the current time $t_0$
\item $[-\frac{|\omega(t_0)|}{2} \sin \frac{|\omega(t_0)|(t_0-t_0)}{2}, \frac{\omega(t_0)}{|\omega(t_0)|} \frac{|\omega(t_0)|}{2} \cos \frac{|\omega(t_0)|(t-t_0)}{2}] q(t_0)$
\item $[-\frac{|\omega(t_0)|}{2} \sin 0, \frac{\omega(t_0)}{|\omega(t_0)|} \frac{|\omega(t_0)|}{2} \cos 0] q(t_0)$
\item $[0, \frac{\omega(t_0)}{2}] q(t_0) = \frac{1}{2} [0, \omega(t_0)] q(t_0)$
}\end{slide}

\begin{slide}{Deriving quaternions}{Derivative of quaternion}{
\item At least remember this:
\item $\dot q = \frac{1}{2} [0, \omega] q$
}\end{slide}

\section{Kinematics source code}
\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
// initialization
RigidBody* body[n];
double t = <your choice of initial time>;
double dt = <your choice of time step>;
for (int i = 0; i < n; ++i) {
  // Set the initial state of the rigid bodies.
  body[i] = new RigidBody(...);

// Part of the physics tick.
for (i = 0; i < n; ++i) {
  body[i].Update(t, dt);
  t += dt;
}
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
struct RigidBody {
  RigidBody (double m, matrix inertia, Function force, Function torque);

  // force/torque function format
  typedef vector ( *Function ) (
    double, // time of application
    point, // position
    quaternion, // orientation
    ... // whole state of body, one var at a time
    )
  );

  // Runge-Kutta fourth order differential equation solver
  void Update (double t, double dt);
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
protected:
  // convert (Q,P,L) to (R,V,W)
  void Convert (quaternion Q, vector P, vector L, matrix& R, vector& V, vector& W) const;
  // constant quantities
  double m_mass, m_invMass;
  matrix m_inertia, m_invInertia;
  // state variables
  vector m_X; // position
  quaternion m_Q; // orientation
  vector m_P; // linear momentum
  vector m_L; // angular momentum
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
  // derived state variables
  matrix m_R; // orientation matrix
  vector m_V; // linear velocity vector
  m_W; // angular velocity
  // force and torque functions
  Function m_force; Function m_torque;
};
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
void RigidBody::Convert (quaternion Q, vector P, vector L, matrix& R, vector& V, vector& W) const {
  Q.ToRotationMatrix(R);
  V = m_invMass*P;
  W = R*m_invInertia*Transpose(R)*L;
}
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
void RigidBody::Update (double t, double dt) {
  double halfdt = 0.5 * dt, sixthdt = dt / 6.0;
  double tphalfdt = t + halfdt, tpdt = t + dt;
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
  vector XN, PN, LN, VN, WN;
  quaternion QN;
  matrix RN;
  // A1 = G(t,S0), B1 = S0 + (dt / 2) * A1
  vector A1DXDT = m_V;
  quaternion A1DQDT = 0.5 * m_W * m_Q;
  vector A1DPDT = m_force(t,m_X,m_Q,m_P,m_L,m_R,m_V,m_W);
  vector A1DLDT = m_torque(t,m_X,m_Q,m_P,m_L,m_R,m_V,m_W);
  XN = m_X + halfdt * A1DXDT;
  QN = m_Q + halfdt * A1DQDT;
  PN = m_P + halfdt * A1DPDT;
  LN = m_L + halfdt * A1DLDT;
  Convert(QN,PN,LN,RN,VN,WN);
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
  // A2 = G(t + dt / 2,B1), B2 = S0 + (dt / 2) * A2
  vector A2DXDT = VN;
  quaternion A2DQDT = 0.5 * WN * QN;
  vector A2DPDT = m_force(tphalfdt,XN,QN,PN,LN,RN,VN,WN);
  vector A2DLDT = m_torque(tphalfdt,XN,QN,PN,LN,RN,VN,WN);
  XN = m_X + halfdt * A2DXDT;
  QN = m_Q + halfdt * A2DQDT;
  PN = m_P + halfdt * A2DPDT;
  LN = m_L + halfdt * A2DLDT;
  Convert(QN,PN,LN,RN,VN,WN);
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
  // A3 = G(t + dt / 2,B2), B3 = S0 + dt * A3
  vector A3DXDT = VN;
  quaternion A3DQDT = 0.5 * WN * QN;

  vector A3DPDT = m_force(tphalfdt,XN,QN,PN,LN,RN,VN,WN);
  vector A3DLDT = m_torque(tphalfdt,XN,QN,PN,LN,RN,VN,WN);
  XN = m_X + dt * A3DXDT;
  QN = m_Q + dt * A3DQDT;
  PN = m_P + dt * A3DPDT;
  LN = m_L + dt * A3DLDT;
  Convert(QN,PN,LN,RN,VN,WN);
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kinematics source code}
\begin{lstlisting}
  // A4 = G(t + dt,B3),
  // S1 = S0 + (dt / 6) * (A1 + 2 * A2 + 2 * A3 + A4)
  vector A4DXDT = VN;
  quaternion A4DQDT = 0.5 * WN * QN;
  vector A4DPDT = m_force(tpdt,XN,QN,PN,LN,RN,VN,WN);
  vector A4DLDT = m_torque(tpdt,XN,QN,PN,LN,RN,VN,WN);
  m_X = m_X + sixthdt*(A1DXDT + 2.0*(A2DXDT + A3DXDT) + A4DXDT);
  m_Q = m_Q + sixthdt*(A1DQDT + 2.0*(A2DQDT + A3DQDT) + A4DQDT);
  m_P = m_P + sixthdt*(A1DPDT + 2.0*(A2DPDT + A3DPDT) + A4DPDT);
  m_L = m_L + sixthdt*(A1DLDT + 2.0*(A2DLDT + A3DLDT) + A4DLDT);
  Convert(m_Q,m_P,m_L,m_R,m_V,m_W);
}
\end{lstlisting}
\end{frame}

\begin{frame}{That's it}
\center
\fontsize{18pt}{7.2}\selectfont
Thank you!
\end{frame}

\end{document}