\section{Soution Methods}\label{sec:solution_methods}

\subsection{Value Function Iteration}

Three different solution methods is presented in this section. I lead with (a slightly modified) value function iteration, next deep Q-Learning is presented ending with double deep Q-learning iteration.

The value function iteration presented in this paper has certain modifications to the algorithm presented in \ref{sec:dynamic_programming}. First and foremost, I consider the Q-function instead of the value function allowing storing state-action value pairs. Second it should be noted that in this formulation some states are continuous not allowing for tabular solutions. Thirdly, the dimension of the state-space + the size of the grid made it infeasible\footnote{In my initial attempt to solve the model by value function iteration, I attempted to get the expectation of the value function using Gauss-Hermite integration, and discretizing the state space. The solution time was infeasible, which was why my approach changed.} to solve the model by classical ways of solving a model by discretizing the state space in a grid.

The model is solved using backward induction. This is due to the fact the model terminates in a deterministically when the agents reach a certain age. For each step a large random sample of states is drawn conditioning on a given age. For each of the states the agent takes each of the possible actions storing the results. This way a large sample of rewards and states can be generated. Furthermore, for each action taken (if the state is not terminal) the  Q-function can be evaluated in the new state, taken the max of each possible action, allowing for the estimation of the value function. A graphical representation of the concept is shown in figure \ref{fig:vfi_figure}.

\begin{figure}[ht]
    \centering
    \includegraphics[scale=0.15]{figures/vfi_figure.png}
    \caption{Value Function Iteration}
    \label{fig:vfi_figure}
\end{figure}

The Q-function for each action in the action space is made by mapping each $a\in \actionspace$:

\begin{equation}
 Q(S_t, a) = R_{t+1} + \gamma \underset{a'\in \actionspace}{\max} Q(S_{t+1}, a')
\end{equation}

In that sense this can be considered akin to Monte Carlo integration, however instead of approximating the expectation in a single point, rather find the distribution of the rewards + discounted value function over the entire state space. The idea is to approximate the integral by using a statistical method, in this case deep learning even though another machine learning method would be equally good. Consider $f$ to be a deep neural network, that has the property:

\begin{equation}
    f: \statespace \mapsto\R^{\mid \actionspace \mid}
\end{equation}

For a given point in state space a prediction of the value function is computed for each possible action. This implies the method only is feasible for discrete state space. By trying to reduce the mean squared error between the true values of the Q-function, and the prediction, the $\E[Q(a, s)]$ can be found, which corresponds to integration as could be done using Gauss Hermite or Monte Carlo integration. The algorithm is presented in algorithm \ref{alg:dqi}.

\begin{algorithm}[H]
\SetAlgoLined
\KwResult{Estimated Q function}
 Initialize $\tilde{Age} = Age_{max}$\;
 Initialize empty lists for storing results: $X, Y$\;
 Initialize memory counter $j=1$\;
 \While{$\tilde{Age} > Age_{min}$}{
  Draw $\{s_{i}\}_{i=1}^{N}$, where $s_i \sim Uniform(\statespace) \mid Age=\tilde{Age}$ \;
  \ForEach{$s_{i}$}{
  Create empty array $Z$ of length $\mid \actionspace \mid$\;
  \eIf{$\tilde{Age}= Age_{max}$}{
   \ForEach{$a_k \in \actionspace$}{
    $Z[k] \leftarrow R_{t+1}, \quad  R_{t+1}\sim \mathcal{E}\mid A_t = a_k, S_t = s_{i}$ \;
   }
   }{
   \ForEach{$a_k \in \actionspace$}{
    $Z[k] \leftarrow R_{t+1} + \gamma \underset{a \in \actionspace}{\max} \hat{Q}(S_{t+1}, a), \quad  R_{t+1}, S_{t+1} \sim \mathcal{E} \mid A_t = a_k, S_t = s_{i} $\;
    }
  }
  $Y[j] \leftarrow Z, X[j] \leftarrow s_i$\;
  $j = j + 1$\;
  }
  Estimate $\hat{Q}$ by training a Deep NN using samples from $X, Y$\;
  Decrease $\tilde{Age}$ be one\;
 }
 \caption{Value Function Iteration Solution Method}
 \label{alg:dqi}
 \end{algorithm}

Since I use a deep neural network to approximate the $Q$-function, the architecture and hyper parameters of the network needs to be considered. The same is true for the sampling scheme.

For each age 20.000 random samples is drawn. This is because any smaller number of draws seemed to be detrimental to the performance. This is inline with standard deep learning practices, where neural networks are known to be very data hungry. A random sample of 100.000 observations is used when training the network. If I have not yet accumulated 100.000 observations the algorithm draws all observations. The architecture of network is fairly simple being a two-layer fully connected network. First layer being 16 nodes wide, second fully connected layer being 8 nodes wide. I found that mini batching, did not seem to work well on this particular task, and instead I train on all observations, using a validation split of 30 \%, training for a maximum of 150 epochs\footnote{A epoch corresponds to a full sweep through the the data set.} and finally I allow for early stopping, that is, when the validation loss is not further decreasing, I stop the training of the network. I do allow the algorithm a ``patience'' of 5, Implying that the algorithm will try to lower its validation loss for five additional epochs before terminating the training.

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqi_model1_beta_2_solution_benchmark_paths.png}
  \caption{Simulated Paths}
  \label{fig:dqi_solution_beta2_path}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqi_model1_beta_2_solution_benchmark_variance.png}
  \caption{Variance of Paths}
  \label{fig:dqi_solution_beta2_var}
\end{subfigure}
    \caption{Value Function Iteration Solution vs. Benchmark $(\beta_L = 2)$}
    \label{fig:dqi_solution_beta2}
\end{figure}

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqi_model1_beta_4_solution_benchmark_paths.png}
  \caption{Simulated Paths}
  \label{fig:dqi_solution_beta4_path}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqi_model1_beta_4_solution_benchmark_variance.png}
  \caption{Variance of Paths}
  \label{fig:dqi_solution_beta4_var}
\end{subfigure}
    \caption{Value Function Iteration Solution vs. Benchmark $(\beta_L = 4)$}
    \label{fig:dqi_solution_beta4}
\end{figure}

Figure \ref{fig:dqi_solution_beta2} and \ref{fig:dqi_solution_beta4} shows the results of the Value Function Iteration algorithm\footnote{The plots use the name DQIteration (Deep Q-Function Iteration) instead of Value Function Iteration.} compared to 4 benchmarks. Three deterministic agents working either $0$, $37$ or $45$ hours per week and one agent taking random actions. Figure \ref{fig:dqi_solution_beta2} shows the utility for each step over the life cycle when the preference for leisure is fixed at $\beta_L = 2$. As the figure shows the VFI agent learns to navigate the environment, just as well as the best deterministic agent. Looking to the RHS plot of \ref{fig:dqi_solution_beta2} it is clear that there is a substantial overlap between the two agents. The variance of the path is represented as one standard deviation of the utility for a given age for all episodes of the given agent. Figure \ref{fig:dqi_solution_beta4} compares the benchmark agents with the VFI solution when considering a preference for leisure $\beta_L = 4$. Again the VFI solution is as good as the best benchmark.

\subsection{Deep Q-learning}

In this paper I implement\footnote{Originally I used my own implementation, however slow performance, caused me to use and modify an existing implementation allowing for a speed up of a factor of 10. The same is true for the Double Deep Q-learning algorithm. The original implementation can be found at \textcite{tabor_deep_2020}.} the deep Q-learning algorithm used by \textcite{mnih_playing_2013} for beating Atari games as a way to solve model. A few modifications is made to the original implementation, due to the fact, the environment they were navigating only returned sensory data (an RGB representation of an image).  A part of their achievement was to transform these images into features that the value function accurately could map into scores of the game. Another difference is that this paper implements a scaling module of the variables for better performance.

Just as described in section \ref{sec:rl_theory} the algorithm tries to maximize the Bellman equation. However now the value-function is estimated using Deep neural network as a function approximator. Mathematically this can be described as finding:

\begin{equation}
    Q^*(S_t,A_t) = \E [R_{t+1} + \underset{a}{\max}  Q^*(S_{t+1}, a) \mid S_t, A_t]
\end{equation}

However, here $Q^*(S, A)$ is approximated by a parametric function (in this case a Deep Neural Network) $Q(S, A ; \theta)$. Following the terminology of \textcite{mnih_playing_2013}, this function approximator is referred to as the Q-network. The Q-network can be trained using stochastic gradient descent as described in section \ref{sec:deep_learning}:

\begin{equation}
    \Loss_i(\theta_i) = \E \lsp (Y_i - Q(S_t, A_t ; \theta_i))^2 \rsp
\end{equation}

\begin{equation}
    Y_i = \E [R_{t+1} + \gamma \underset{a}\max Q(S_{t+1}, a; \theta_{i-1}) \mid S_t, A_t ]
\end{equation}

Where $i$ implies the iteration of the algorithm, such $i$ increments by one for each update of the parameters $\theta$. The equation used for updating the weight of the neural network is described below:

\begin{equation}
    \nabla_{\theta_i} \Loss_i (\theta_i)  \E \lsp \lp Y_i - Q(S_t, A_t ; \theta) \rp \nabla_{\theta_i} Q(S_t, A_t; \theta_i) \rsp
\end{equation}

Following the formulation of \textcite{mnih_playing_2013} $Q(S_{t+1}, A_{t+1}; \theta_{i-1})$ is held fixed, allowing for just writing $Y_i$, and not the parametric form of $Y_i$.

A traditional choice would be to update the weight after each step in the algorithm only using the last sample. This would correspond to the traditional Q-learning algorithm. This was the approach used by the TD Gammon agent created by Tesauro\footnote{I have not been able to get access to the original paper, so I have relied on the description made by Sutton and Barto.} as described by \textcite{sutton_reinforcement_2018}. The approach of using Q-learning with a non-linear function approximator has been shown to diverge under certain circumstances, and did not extend itself well to learning any other game than backgammon \parencite{tsitsiklis_analysis_1997}. To accommodate this problem a replay buffer is implemented. At each step an entry is made to the replay memory containing $(s_t, a_t, r_t, s_{t+1})$. This data set $\mathcal{D}$ has a capacity of $N$ entries. Using the replay memory a random mini batch is sampled used to update the weights of the Q-network. Note here that the capacity of the memory buffer should be greater, by a substantial margin than the number of samples drawn. The random sample has a couple of advantages. It decorrelates the observations used to update the weights, allowing for better training \textcite{mnih_playing_2013}. Using a replay buffer requires off-policy learning which is the reason for using Q-learning. This is due to the fact, that current parameters $\theta$ is not the same as those generating the data. Note, that experiences is drawn randomly, and no experiences (which could have important insights) is prioritized. The full algorithm is summarized in algorithm \ref{alg:dqlearning}.

\begin{algorithm}[H]
\SetAlgoLined
 Initialize replay memory $\mathcal{D}$ with capacity $N$\;
 Initialize action-value function $Q$ with random weights\;
 Initialize memory counter $j=1$\;
 \ForEach {episode $\in \{1, 2, \cdots M \}$}{
  Initialize sequence with an initial state $s_1$. This is drawn randomly.\;
    \For{$t \in \{1, 2, \cdots, T \}$}{
    With probability $\epsilon$ select a random action $a_t$\;
    Otherwise select $A_t = \underset{a}{\argmax}Q^*(S_t, a ; \theta)$\;
    Execute action $A_t$ in environment and observe reward $R_t$ and the new state $S_{t+1}$\;
    Store transition $(S_t, A_t, R_{t+1}, S_{t+1})$ in $\mathcal{D}$\;
    Sample random mini-batch of transitions $(S_t, A_t, R_{t+1}, S_{t+1})$ from $\mathcal{D}$\;
    Set $
      Y_j = \begin{cases}
        R_j & \text{if terminal states} \\
        R_j + \gamma \underset{a}{\max}Q(S_{t+1}, a; \theta) & \text{if non-terminal states}
      \end{cases}
    $ \;
    Perform gradient descent step on $(Y_j - Q(S_t, A_t ; \theta))^2$ \;
    Increment $j$\;
  }
}
\caption{Deep Q-learning}
\label{alg:dqlearning}
\end{algorithm}

Following the implementation \textcite{mnih_playing_2013} a trick is applied to reduce the number of computations needed when running the algorithm. Ordinarily the Q-function maps a state action pair to a scalar value estimate of the value function. The most obvious implementation would let the action be part of the input to the function, however, such implementation would require $\mid \actionspace \mid$ number of look ups at each step. This is due to the fact that each action in the action space must be used for evaluation. Instead, the Q-function in this implementation maps the state space to a scalar value for each action $Q: \statespace \mapsto \R^{\mid \actionspace \mid}$.

The solution used the following hyper parameters and architecture decisions: The Deep Neural network consists of an input layer of same size as the state space, followed by two fully connected layers of width $256$ using rectified linear units as activation functions. Finally, the network uses a linear output layer for its predictions. The output layer is of the same size as the action space. The learning rate, $\alpha=0.0005$ and I let epsilon be decremented after each update by: $\epsilon_i = \max (\epsilon_{i-1} \cdot 0.9999, 0.01$), where $0.01$ is the minimum exploration that will be done. The replay buffer has a capacity of 1 million rows, and the mini batches used for updating the parameters is $64$ rows. I use the Adam optimizer for the gradient descent step when updating the weights. I scale the state space so that each variable approximately has mean zero and standard deviation of 1 when doing the batch training. \textcite{goodfellow_deep_2016} argues that it increases performance, and in general accepted as being an important step for getting good performance out of neural networks. The impatience parameter is set to $\gamma=0.99$, using a standard value. I make the parameter $\beta_L$ a part of the state space, drawing a random value (uniformly) in the interval $[0.2, 6.0]$ at the beginning of each episode, letting the agent navigate through the environment with given preference for leisure. This is done, so only a single solution of the model is necessary for estimating the parameter later. I scale the rewards (so they are approximate zero mean and have a standard deviation of 1, conditional on the $\beta_L$ value. Again this is done to improve the training performance and to do introspection; it becomes possible to see if there is any trend in the training performance. The algorithm is trained for 3000 episodes. Figure \ref{fig:training_performance_simple_model}  (left plot) shows the training performance of the Deep Q-learning algorithm. The plot shows the agent's cumulative rewards over the life cycle of each episode. The performance begins at around 2.5 and ends at an average of $7.9$ as the asymptotic value. The performance seems to reach this asymptotic level at around 1000 episodes.

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqn_training_performance_simple_model.png}
  \caption{DQN Training Performance}
  \label{fig:dqn_training_performance_simple}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/ddqn_training_performance_simple_model.png}
  \caption{Double DQN Training Performance}
  \label{fig:ddqn_training_performance_simple}
\end{subfigure}
    \caption{Comparing Training Perfance of Deep Q-learning and Double Deep Q-learning}
    \label{fig:training_performance_simple_model}
\end{figure}

Figure \ref{fig:dqn_solution_beta2} and \ref{fig:dqn_solution_beta4} compares the performance of the algorithm to two different benchmarks. One benchmark is the environment with $\beta_L = 2$ another with $\beta_L=4$. These differences in preferences will yield different optimal policies conditional on the $\beta_L$. I compare the policy chosen by the agent with different deterministic policies. Figure \ref{fig:dqn_solution_beta2} (left plot) shows either using $0, 37, 45$ hours per week as comparisons and an agent picking randomly. All agents have been simulated for 300 episodes, yielding the plots being averages, and the standard deviations are calculated on a per age basis for each agent. It should also be noted that $\epsilon$ (the exploration ratio of the agent) is set to 0, such that the agent now only greedily navigates the environment. Comparing this to the VFI agent I find  it reasonable to assume that the DQN agent has learned to navigate the environment. A small dip from around age 55 can be observed, but I believe this does not make the feat any less impressive, and furthermore I believe that the solution fairly accurately approximates the optimal policy. Again the best benchmark for $\beta_L = 2$ is an agent that works 45 hours a week. The deterministic and DQN agent seem to follow the same policy until  the last periods, where they diverge a little. I still to conclude that the differences in performance is very small. Figure \ref{fig:dqn_solution_beta4} shows the same plot just for $\beta_L = 4$. Again the DQN agent fairly accurately approximates the optimal policy, with even less variance than observed in with $\beta_L = 2$. One thing to note is, that a small hump can be observed for the DQN agent at around age 28, where it has found a way to beat the deterministic policy. Again this might just be variance of the simulations.

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqn_model1_beta_2_solution_benchmark_paths.png}
  \caption{Simulated Paths}
  \label{fig:dqn_solution_beta2_path}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqn_model1_beta_2_solution_benchmark_variance.png}
  \caption{Variance of Paths}
  \label{fig:dqn_solution_beta2_var}
\end{subfigure}
    \caption{Deep Q-learning Solution vs. Benchmark $(\beta_L = 2)$}
    \label{fig:dqn_solution_beta2}
\end{figure}

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqn_model1_beta_4_solution_benchmark_paths.png}
  \caption{Simulated Paths}
  \label{fig:dqn_solution_beta4_path}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/dqn_model1_beta_4_solution_benchmark_variance.png}
  \caption{Variance of Paths}
  \label{fig:dqn_solution_beta4_var}
\end{subfigure}
    \caption{Deep Q-Learning Solution vs. Benchmark $(\beta_L = 4)$}
    \label{fig:dqn_solution_beta4}
\end{figure}

\subsection{Double Deep Q-learning}

Deep Q-learning is good starting point for a learning algorithm using non-linear function approximation, however certain properties of the algorithm is problematic. \textcite{van_hasselt_deep_2015} argues this is mainly due to the fact that Deep Q-learning can tend to be overoptimistic in a problematic way. In general there exists two ways in which overoptimism can be good or at least not detrimental to performance. 1) If the algorithm uniformly overestimates the Q-function for all actions, then this is not associated with any problems, since taking the max of the Q-function w.r.t. to the action, will lead to the same result had the estimations been correct. In other words, performance would not change, however introspection of the algorithm might be harder. 2) It can be a good thing for an algorithm to be optimistic when faced with uncertainty. If an action would lead to observing an unexplored part of the state space, it might be a good thing to be optimistic in regard to exploration, allowing for possible new policies with higher yielding returns over the episode. Deep Q-learning do not conform to any of these properties, instead it usually overestimates the value of the action it has taken due to the max operator in the value estimation: $R_{t+1} + \gamma \underset{a}{\argmax} Q(S_{t+1},a)$. A simple extension presented by \textcite{van_hasselt_deep_2015} is to decouple the prediction with the evaluation. This can be done by having two different sets of weights, where one $\theta$ is used for the policy (policy weights) and one $\tilde{\theta}$ is used for the evaluation (target weights). This can be presented the following way. The Q-network target follows the equation below:

\begin{equation}\label{eq:dqn_target}
    Y_t^{Q} = R_{t+1} + \gamma Q(S_{t+1}, \underset{a}{\argmax} (S_{t+1}, a ; \theta) ; \theta)
\end{equation}

The Double DQN target contrasts the DQN-target shown above, by using two sets of weights:

\begin{equation}\label{eq:ddqn_target}
    Y_t^{DoubleQ} = R_{t+1} + \gamma Q(S_{t+1}, \underset{a}{\argmax}Q(S_{t+1}, a; \theta); \tilde{\theta})
\end{equation}

This alleviates the problem of overestimating the values returned by the algorithm for the following reason. Consider \eqref{eq:dqn_target}, when calculating the value of the state action pair, one uses the same function to choose the action, prompting  a real risk of overestimation of the state action pair of the chosen action. The equation below \eqref{eq:ddqn_target} instead estimates the value of the position by using a different set of parameters, $\tilde{\theta}$. Following the implementation of \textcite{van_hasselt_deep_2015} instead of training 2 separate networks the target weights are inherited from the evaluation weights. This is to reduce time of training the algorithm, even though the policy and target network can not be considered perfectly decoupled. The implication of this design choice is $\tilde{\theta} = \theta^{previous}$, such that, the target estimation is made by the old weights, while the value of greedy policy uses the new weights:

\begin{equation}\label{eq:ddqn_target_final}
    Y_t^{DoubleQ} = R_{t+1} + \gamma Q(S_{t+1}, \underset{a}{\argmax}Q(S_{t+1}, a; \theta_t); \theta_t^{previous})
\end{equation}

The full algorithm, which to large extend mirrors the algorithm for Deep Q-learning is presented in algorithm \ref{alg:ddqlearning}:

\begin{algorithm}[H]
\SetAlgoLined
 Initialize replay memory $\mathcal{D}$ with capacity $N$\;
 Initialize action-value function $Q$ with random weights: $\theta, \tilde{\theta}$\;
 Initialize memory counter $j \leftarrow 1$\;
 Initialize $k$ when to replace target weights $\tilde{\theta}$\;
 \ForEach {episode $\in \{1, 2, \cdots M \}$}{
  Initialize sequence with an initial state $s_1$. This is drawn randomly.\;
    \For{$t \in \{1, 2, \cdots, T$ \}}{
    With probability $\epsilon$ select a random action $a_t$\;
    Otherwise select $A_t = \underset{a}{\argmax}Q^*(S_t, A_t ; \theta)$\;
    Execute action $A_t$ in environment and observe reward $R_t$ and the new state $S_{t+1}$\;
    Store transition $(S_t, A_t, R_{t+1}, S_{t+1})$ in $\mathcal{D}$\;
    Sample random mini-batch of transitions $(S_t, A_t, A_{t+1}, S_{t+1})$ from $\mathcal{D}$\;
    Set $
      Y_j = \begin{cases}
        R_{t+1} & \text{if terminal states} \\
        R_{t+1} + \gamma Q(S_{t+1}, \underset{a}{\argmax} Q(S_t, a; \theta_t); \tilde{\theta_t}) & \text{if non-terminal states}
      \end{cases}
    $ \;
    Perform gradient descent step on $(Y_j - Q(S_t, A_t ; \theta))^2$ w.r.t. $\theta$ \;
    If $j$ is divisible by $k$ replace target weights $\tilde{\theta} \leftarrow \theta$\;
    Increment $j$ \;
  }
}
\caption{Double Deep Q-learning}
\label{alg:ddqlearning}
\end{algorithm}

Just as with the Deep Q-learning implementation the Q-function map from: $\statespace \mapsto \R^{\mid \actionspace \mid}$, to reduce the number of computations.

For the Double DQN agent I in general use the same hyper parameters used for the DQN agent: The network architecture is again an input layer of the size of the state space, followed by two fully connected layers with Rectified Linear units as activation functions ending with an output layer with linear activation of same size as the action space. Again I let the learning rate be $\alpha = 0.0005$ and the $\epsilon_i = \max (\epsilon_{i-1} \cdot 0.9999, 0.01)$, such that exploration is performed throughout the training period. Again the replay buffer has a capacity of a million rows and I use mini batches of size 64 to perform stochastic gradient descent on the weight of the neural network approximating the Q-function. I transform the states such that they are mean zero and a standard deviation of 1. The same is true for the rewards conditional on the $\beta_L$ value. I uniformly draw $\beta_L$ from the interval $[0.2, 6]$ and sets this to be part of the environment in the beginning of each episode. I use the Adam optimizer when updating the weights using gradient descent. I let the target network inherit the old weights every 100'th iteration of the algorithm. The algorithm is trained for 3000 episodes. Figure \ref{fig:training_performance_simple_model} (right plot) shows the performance. Very similar to the DQN, I find the performance stabilizes at around 1000 episodes, and reaches an asymptotic performance of $8.22$, which is slightly higher compared to the DQN agent which had an asymptotic value of $7.92$.

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/ddqn_model1_beta_2_solution_benchmark_paths.png}
  \caption{Simulated Paths}
  \label{fig:ddqn_solution_beta2_path}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/ddqn_model1_beta_2_solution_benchmark_variance.png}
  \caption{Variance of Paths}
  \label{fig:ddqn_solution_beta2_var}
\end{subfigure}
    \caption{Double Deep Q-Learning solution vs. benchmark $(\beta_L = 2)$}
    \label{fig:ddqn_solution_beta2}
\end{figure}

\begin{figure}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/ddqn_model1_beta_4_solution_benchmark_paths.png}
  \caption{Simulated Paths}
  \label{fig:ddqn_solution_beta4_path}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=1\linewidth]{figures/ddqn_model1_beta_4_solution_benchmark_variance.png}
  \caption{Variance of Paths}
  \label{fig:ddqn_solution_beta4_var}
\end{subfigure}
    \caption{Double Deep Q-Learning solution vs. benchmark $(\beta_L = 4)$}
    \label{fig:ddqn_solution_beta4}
\end{figure}

Figure \ref{fig:ddqn_solution_beta2} and \ref{fig:ddqn_solution_beta4} shows the performance of the Double DQN agent comparing it to two different preferences of leisure: $\beta_L = 2$ and $\beta_L = 4$. Four agents is used for comparison: An agent that chooses actions randomly, and three agents that in an deterministic fashion works $0, 37$ or $45$ hours a week. The results of the Double DQN agent is comparable to those found within the other agents (VFI and DQN). Figure \ref{fig:ddqn_solution_beta2} shows the performance where the preference for leisure $\beta_L = 2$. Again the best deterministic policy is the 45 hour agent, which is very close to what the DQN agent chooses, except for the last 5 years of the agent's life cycle where the utility is slightly lower. Comparing the variance of the utility on RHS plot of figure \ref{fig:ddqn_solution_beta2}, there is overlap of the simulations of the Double DQN agent and the 45-hours a week agent. Figure \ref{fig:ddqn_solution_beta4} compares the Double DQN agent to the agents when the preference for leisure $\beta_L = 4$. Again, the optimal policy pretty closely mirrors the best deterministic agent (the agent that does not work), with a clear overlap presented in figure on RHS.


\section*{Data-Parallel Execution Model}
The execution is SPMD, with each thread using unique coordinates
to access some region of the data-structure. A grid is a 3D-array
of blocks and a block is a 3D array of threads.

Each block has a 3-vector \texttt{blockIdx}, and each thread has
a 3-vector \texttt{threadIdx}, the components having identifiers
\texttt{x}, \texttt{y}, \texttt{z}. dimensions of each block and each
grid can be specified using variables of type \texttt{dim3}. If
scalar dimension specifiers are used (e.g. \texttt{int}), then
the \texttt{x} dimension is set to the scalar value while
the \texttt{y} and \texttt{z} are set to \texttt{1}. A grid
can have higher dimensionality then its blocks.

It might be a good practice to assign indices such that the largest
of the dimensions comes first, e.g.
\begin{minted}
{c}
dim3 dimBlock(2, 2, 1);
dim3 dimGrid(4, 2, 2);
KernelFunction<<<dimGrid, dimBlock>>>(...);
\end{minted}

This works better when mapping thread coordinates into data indices
in accessing multidimensional arrays.

NVIDIA GPUs in 2012 could support up to 1024 threads per block with
flexibility in how one can index them.

\subsection*{Mapping Threads To Multidimensional Data}
The programmer needs to think about the dimensionality of the grid
and blocks based on the nature of the data in consideration. E.g.
an image is a 2D grid of values. If the image is greyscale, then
we can do with a representation that takes a scalar value per pixel.

So, if the resolution of our image is $76\times 62$, we could create
a grid using $16\times 16$ blocks, and let the grid be $5\times 4$
blocks. $16\times 16$ is a typical choice of block dimensions for
2D data structures. In this example, we are allocating $80\times 64$
threads to process $76\times 62$ pixels.

To avoid wasteful computations, there should be a conditional guard
so that spawned threads in the grid that do not have pixel values to
work with do not do any computations, like was done in the vector
addition example, wherein four 256-thread blocks were spawned to
deal with a vector of length 1000.

If you have been provided with parameters for a 2D array, e.g. the
dimensions $m\times n$ for the image, then you can use something
like this two spawn a grid for processing the image.
\begin{minted}
{c}
    dim3 dimBlock(ceil(n/16.0), ceil(m/16.0));
    dim3 dimGrid(16, 16, 1);
    pictureKernel<<<dimGrid, dimBlock>>>(d_Pin, d_Pout, n, m);
\end{minted}

So, here's the important bit. We have a global memory pointer
\texttt{d\_Pin} for our input data. As of 2012, CUDA C is based on
ANSI C, which requires the number of rows and columns in 2D arrays
to be known at compile time, which spells doom for most of our
dynamically allocated datastructures. So, the programmer has to
explicitly flatten the tensor to an equivalent 1D array.

The most up to date documents on NVIDIA are on C++. The C documents
are 2010-ish old.The CUDA 11.0 guides on programming and best
practices are exclusive to C++.

Checking C99 compatibility would require some testing. For now, all
I need to know is that C arrays are stored in row-major order.

The following listing shows an example of a CUDA code for matrix
multiplication for square matrices
\begin{code}
\inputminted[samepage=false, breaklines, linenos]{c}{../codes/matrixAdd/matrixMul.cu}
\label{lst:matrixMul}
\caption{Multiplication of square matrices}
\end{code}

\subsection*{Synchronization and Transparent Scalability}
In general one has to coordinate the execution of threads. CUDA allows
threads in the same block to coordinate using a barrier
synchronization function \texttt{\_\_syncthreads()}. All threads
in the calling block will be held until all threads reach the same
location in the program. I see potential deadlocks.

At the very least using this particular barrier synchronization
would allow different blocks to execute independent of each other.

\subsection*{Thread Scheduling and Latency Tolerance}
Thread scheduling depend on specific hardware implementations, A warp
is a unit of thread scheduling, typically 32 threads, but is not
defined in the CUDA specification. A streaming multiprocessor
executes all threads in a WARP in a SIMD fashion - all threads in a
warp will have the same execution timing.

In general, there are fewer streaming processors (SP) per SM than
the number of threads assigned to each SM; a scheduler is involved
in the execution of all the warps assigned to an SM. By having
more warps per SM than what the underlying hardware can execute at
a time, CUDA processors efficiently execute long latency operations
such as global memory accesses - context switching. Another resident
warp that is not scheduled is then scheduled and the previously
scheduled warp, that is waiting for the completion of a long latency
operation is queued.

Examples of long latency operations include pipelined FP arithmetic,
branch instructions etc. \textit{Note that warp scheduling has zero-overhead when switching to another ready warp.}

\subsection*{SIMT and Warp-Level Primitives}
I came across a nice blog post by NVIDIA on warp-level primitives.
The question that I had been trying to address was ``is it safe to
remove \texttt{\_\_syncthreads()} when the block size is the same as
the device warp size?". The answer is no. There can be divergent
control flow between threads in the same warp. The simplest
example is when checking boundary conditions when working with
tensors/tensor-like data structures.

While GPU computing is usually defined as SIMD (Single Instruction
Multiple Thread), the classification SIMD in the context of Flynn's
taxonomy is usually reserved for vector architecture, i.e. a
scalar thread issues vector instructions, and the exact same
instruction is applied across many data elements.

SIMT (Single Instruction Multiple Thread) is a more appropriate
classification for GPUs, even at the level of a single warp. Each
thread has access to its own registers and can load and store from
divergent addresses and follow different control paths.

It is possible to achieve even higher performance by using warp-level
primitives and exploiting the flexibility provided by this SIMT
architecture.

A consequence of this is that we can have cooperative groups
collectives, e.g. accumulating the sum of values in all threads
into a single thread (analogous to \texttt{MPI\_Reduce}).

\begin{minted}
    {c}
    #define FULL_MASK 0xffffffff
    for (int offset = 0x10; offset > 0; offset /= 2)
        val += __shfl_down_sync(FULL_MASK, val, offset);
\end{minted}

The above code performs a tree-reduction to compute the sum of
the \texttt{val} variable held by each thread in a warp. See figure
\ref{fig:figures-shfldown-png}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/shfldown.png}
    \caption{Warp-level parallel reduction (taken from NVIDIA's blog)}
    \label{fig:figures-shfldown-png}
\end{figure}

(TODO: document some warp-level primitives)


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Spherical top-hat model}

We start off with the discussion of curved models, which is the same as in the FAC course.

We consider an Einstein-de Sitter universe, and a sphere with \(\Omega_{p} (t_i) = 1 + \delta_{i}\).

The density profile is considered to be in the shape of a top-hat, which is flat in the center and at the boundaries, shaped like a plateau.
We want to have zero peculiar velocity at the beginning; pure Hubble flow.
Initially we have \(\delta = \delta_{+} + \delta_{-}\), and we must also have \( \frac{2}{3} \delta_{+} = \delta_{-}\).

We find
%
\begin{align}
\rho_{P} (t_m) = \rho_{c} (t_i) \Omega_{P} (t_i) \qty[ \frac{\Omega_{P}(t_i) - 1}{\Omega_{P}(t_i)}]^3
\,,
\end{align}
%
where \(t_m\) is the turnaround time.

With these hypotheses, we find that the ratio of the inner to the outer density is around \(5.6\):
%
\begin{align}
\frac{\rho_{P}}{\rho_{C}} = \qty( \frac{3 \pi }{4})^2
\,.
\end{align}

If we used linear theory instead of the correct theory we would have found \(2.07\) for the ratio: the result is very different.

The total energy of our top-hat is given by
%
\begin{align}
E _{\text{eq}} = - \frac{1}{2} \frac{3}{5} \frac{GM^2}{R _{\text{eq}}}
\,.
\end{align}

At turn-around \(T = 0\), so at that point
%
\begin{align}
E_m = - \frac{3}{5} \frac{GM^2}{R_m}
\,,
\end{align}
%
and as long as the energy is conserved, we find that \(R_m = 2 R _{\text{eq}}\).
This means that the volume at turnaround is 8 times that of equilibrium.

Therefore, we can find what the ratio of the perturbation density to the background one is of 180 at collapse.
If we still were to use linear theory at this time we would get \num{1.686}.

Now we introduce the \emph{mass function of cosmic structure}, which is defined as
%
\begin{align}
n(M) = \dv{N}{M}
\,,
\end{align}
%
the number of objects per unit volume with mass between \(M \) and \(M + \dd{M}\).

Then we take a filter, we consider the probability that at a certain point the perturbation's magnitude is larger than \num{1.686}, which is equivalent to saying that it has virialized.

Then, the ansatz is
%
\begin{align}
n(M) M \dd{M} = \rho_{m} \qty(\mathbb{P}_{> \delta_{c}} - \mathbb{P}_{ > \delta_{c}} (M + \dd{M}))
= \rho_{m} \abs{ \dv{\mathbb{P}_{> \delta_{c}}}{M}} \dd{M}
\,,
\end{align}
%
but the probability depends on the mass only through the variance of the distribution of density: then we write
%
\begin{align}
n(M) M \dd{M} = \rho_{m}
\abs{ \dv{\mathbb{P}_{> \delta_{c}}}{\sigma_{M}}}
\abs{ \dv{\sigma_{M}}{M}}
\dd{M}
\,,
\end{align}
%
and finally we get
%
\begin{align}
n(M) = \frac{2}{\sqrt{\pi }} \frac{\rho_{m}}{M_*^2}  \alpha \qty( \frac{M}{M_{*}})^{\alpha -2} \exp( - \qty( \frac{M}{M_{*}})^{2\alpha } )
\,,
\end{align}
%
which is equivalent to the Schecter luminosity function, where \(\alpha = 1/2\) --- that is, with white noise.

\end{document}


\chapter{Event Detection}
In this chapter, we will introduce the mechanism of event detection in this project. First for the audio clips we have downloaded, we need to extract relevant features from them. After getting features, a Gaussian Mixture Model is built on those features. Then for the new testing audio, we would implement a segmentation algorithm to cut the clip into pieces and calculate the score of each models to decide which events those segments are detected.
\section{Feature Extraction}
Since the audio data is downloaded from a sound website where clips are crowdsourced, the clips are in various format. We converted all audio clips to WAV format and averaged the channels into one if it has multiple channels. Moreover, clips are all downsampled to 16khz sample rate for feature extraction.\\
For the audio data, mel-frequency cepstrum coefficients (MFCCs) features are a widely-used feature. It was brought up by Davis and Mermelstein in the 1980s.
It was experimented that the mel-frequency cepstra possess a significant advantage over the linear-frequency cepstra-specifically, MFCC allow better suppression of insignificant spectral variation in the higher frequency bands \parencite{davis1980comparison}.

\section{Model Building}
From the previous feature extraction process, we could a matrix representation for an audio clip. The column number stands for the dimension of the features and each row corresponds to one observation. From these data, we need to build a model that can capture the overall feature distribution and also are convenient to use for testing data.\\

For this goal, Gaussian Mixture Model (GMM) is used. A gaussian mixture model (or density) is a weighted sum of N gaussian densities, given by the equation:
(gaussian equation).
The intuition of using GMM to model the audio events is that the individual component densities of a multi-model density may model some underlysing set of acoustic classes \parencite{reynolds1995robust}.
Like in speech, a words may consist of some vowels and consonants.
An audio event is naturally more complicated than speech, and may therefore contain more characteristics.
Therefore, we use different gaussian densities with different \mu-vector and \sigma-vector, where the \mu-vector may capture the overall value for one class and the \sigma-vector shows the variation of that class.
When these densities are added together, forming a GMM, they can have a good depiction of some unusual distributions. \\

We train our GMM models on the training data using Expecation-Maximization(EM) algorithm.
This part use the gaussian toolbox provided by Matlab software.
In the section of evaluation, we provide a experiment of controlling the number of mixtures and using k-means algorithm to evaluate the performance of our model.

\section{Audio Segmentation}
Once the models are built, we are ready to use it to test on a potential event clip. But for the problem of scene recognition, we need to detect the   events happening in a clip, therefore, a audio segmentation technique is employed to segment audio clips. \\
There are lots of research carried out on the segmentation of audios.
Some may simply use the average frame energy as the threshold to cut the clips, while other complicated methods involve more features and even use machine learning techniques.
In this project, we use a method of setting a threshold combining frame energy and spectral centroids, and this method is described in \parencite{giannakopoulos2009method}.
This method uses two features from audio:
\begin{itemize}
\item{Frame Energy: this features depicts the overall energy level of one frame. Typically frames where there are some events happening have a higher energy than silent frames.}
\item{Spectral centroid: this represents the "center of gravity" of the audio spectrum. For example, the spectral centroid of the i-th frame (denoted by $$ {C_i}$$) is given by the equation:
spectral centroid = \frac{\sum\limits_{k=1}^Nk\times Amp(k)}{\sum\limits_{k=1}^NAmp(k)}
}
In this equation, Amp(k) is the amplitude corresponding to bin k in Discrete Fourier Transform (DFT) spectrum. So a higher spectral centroid denotes a higher frequency in this frame. Because noises are often in a low frequency, we could use this feature to filter out them.


\section{Functions}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% This work is licensed under the Creative Commons Attribution 4.0 International %
% License. To view a copy of this license, visit                                 %
% http://creativecommons.org/licenses/by/4.0/.                                   %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[11pt]{article}
\usepackage[cm]{fullpage}
%%AVC PACKAGES
\usepackage{avcgreek}
\usepackage{avcfonts}
\usepackage{avcmath}
\usepackage[numberby=section]{avcthm}
\usepackage{qcmacros}
\usepackage{goldstone}
%%MACROS FOR THIS DOCUMENT
\numberwithin{equation}{section}
\usepackage[
  margin=1.5cm,
  includefoot,
  footskip=30pt,
  headsep=0.2cm,headheight=1.3cm
]{geometry}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\fancyhead[LE,RO]{Quiz 2, Handout 2: Wick's Theorem}
\fancyfoot[CE,CO]{\thepage}
\usepackage{url}

\begin{document}

\setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}

\setcounter{section}{1}
\section{Wick's theorem}

\begin{dfn}
\thmtitle{Normal ordering}
The \textit{normal ordering} of a string $q_1\cd q_n$ of particle-hole operators is the mapping $q_1\cd q_n\mapsto\no{q_1\cd q_n}\equiv\e_{\pi}q_{\pi(1)}\cd q_{\pi(n)}$ where $\pi\in\mr{S}_n$ is a permutation that puts the string in normal order.
More generally, \textit{$\F$-normal ordering} maps the string into
$\gno{q_1\cd q_n}\equiv\e_{\si}q_{\si(1)}\cd q_{\si(n)}$ where $\si$ puts the string in $\F$-normal order.
\end{dfn}

\begin{dfn}\label{dfn:contraction}
\thmtitle{Contraction}
A \textit{contraction} of two particle particle-hole operators $q_1$ and $q_2$ is the difference between their product and its normal-ordering, $\no{\ctr{}{q}{_1}{q}q_1q_2}\equiv q_1q_2 - \no{q_1q_2}$.
This associates a scalar value with every pair in $\{a_p\}\cup\{a_p\dg\}$.
\begin{align}
\begin{array}{l@{\hspace{2cm}}l}
  \no{\ctr{}{a}{_p}{a} a_pa_q}
=
  a_pa_q
-
  a_pa_q
=
  0
&
  \no{\ctr[0.5]{}{a}{_p\dg}{a} a_p\dg a_q}
=
  a_p\dg a_q
-
  a_p\dg a_q
=
  0
\\[5pt]
  \no{\ctr[0.5]{}{a}{_p\dg}{a} a_p\dg a_q\dg}
=
  a_p\dg a_q\dg
-
  a_p\dg a_q\dg
=
  0
&
  \no{\ctr{}{a}{_p}{a} a_p a_q\dg}
=
  a_p a_q\dg
+
  a_q\dg a_p
=
  \d_{pq}
\end{array}
\end{align}
More generally, we can define a \textit{$\F$-normal contraction} of two operators by subtracting their $\F$-normal-ordering instead, $\gno{\ctr{}{q}{_1}{q}q_1q_2}\equiv q_1q_2 - \gno{q_1q_2}$.
In this case, contractions of like operators still vanish but the mixed cases are more complicated.
\begin{align}
  \gno{\ctr[0.5]{}{a}{_p\dg}{a} a_p\dg a_q}
=
  \g_{pq}
&&
  \gno{\ctr{}{a}{_p}{a} a_p a_q\dg}
=
  \h_{pq}
&&
  \g_{pq}
\equiv
\left\{
\begin{array}{cc}
\d_{pq} & \text{$p,q$ occupied in $\F$} \\
0 & \text{$p,q$ virtual}
\end{array}
\right.
&&
  \h_{pq}
\equiv
\left\{
\begin{array}{cc}
0 & \text{$p,q$ occupied in $\F$} \\
\d_{pq} & \text{$p,q$ virtual}
\end{array}
\right.
\end{align}
In words, dagger-on-the-left contractions (``hole contractions'') are elements of a matrix $\bm{\g}$ which is zero everywhere but its occupied block, where $\g_{ij}=\d_{ij}$, whereas daggers-on-the-right contractions (``particle contractions'') are elements of a matrix $\bm{\h}$ which is zero everywhere but its virtual block, $\h_{ab}=\d_{ab}$.
Noting that $\gno{\ctr{}{q}{_1}{q}q_1q_2}=\ip{\F|q_1q_2 - \gno{q_1q_2}|\F}=\ip{\F|q_1q_2|\F}$, these matrices can be identified as
$\g_{pq}=\ip{\F|a_p\dg a_q|\F}$
and
$\h_{pq}=\ip{\F|a_p a_q\dg|\F}$, known as the \textit{one-particle} and \textit{one-hole density matrices} of $\F$, respectively.
\end{dfn}

\begin{ntt}\label{ntt:normal-ordering-with-contraction}
\thmtitle{Normal-ordered strings with contractions}
Let the notation $\gno{q_1\cd\ctr{}{q}{_i\cd}{q}q_i\cd q_j\cd q_n}$ stand for
\begin{align}
&&
  \gno{q_1\cd\ctr{}{q}{_i\cd}{q}q_i\cd q_j\cd q_n}
\equiv
  (-)^{j-i-1}\ctr{}{q}{_i}{q}\,q_iq_j\,\gno{q_1\cd\cancel{q_i}\cd\cancel{q_j}\cd q_n}
\end{align}
where the phase factor corresponds to the signature of the permutation required to bring $q_i$ and $q_j$ together.  The same rule applies for normal-ordered strings with multiple contraction lines.
\end{ntt}

\begin{prob}\label{prob:wick-example}
Show that the expansion of $a_p a_q a_s\dg a_r\dg$ in terms of strings that are in normal order
\begin{align}
  a_p a_q a_s\dg a_r\dg
=&\
  a_s\dg a_r\dg a_p a_q
+
  \d_{ps} a_r\dg a_q
-
  \d_{pr} a_s\dg a_q
-
  \d_{qs} a_r\dg a_p
+
  \d_{qr} a_s\dg a_p
-
  \d_{ps}\d_{qr}
+
  \d_{pr}\d_{qs}
\\
\intertext{can be expressed as follows, using \cref{ntt:normal-ordering-with-contraction}.}
\label{eq:wicks-theorem-first-example}
  a_p a_q a_s\dg a_r\dg
=&\
  \no{a_pa_qa_s\dg a_r\dg}
+
  \no{\ctr{}{a}{_pa_q}{a}          a_pa_qa_s\dg a_r\dg}
+
  \no{\ctr[0.5]{}{a}{_pa_qa_s\dg}{a}a_pa_qa_s\dg a_r\dg}
+
  \no{\ctr{a_p}{a}{_q}{a}          a_pa_qa_s\dg a_r\dg}
+
  \no{\ctr{a_p}{a}{_qa_s\dg}{a}    a_pa_qa_s\dg a_r\dg}
+
  \no{\ctr[0]{}{a}{_pa_q}{a}
      \ctr[1]{a_p}{a}{_qa_s\dg}{a} a_pa_qa_s\dg a_r\dg}
+
  \no{\ctr[1]{}{a}{_pa_qa_s\dg}{a}
      \ctr[0]{a_p}{a}{_q}{a}       a_pa_qa_s\dg a_r\dg}
\end{align}
That is, the string equals its normal-ordering plus all possible contractions.
This is one example of a general a result known as Wick's theorem, which will be proven below after we introduce some convenient notation.
\end{prob}

\begin{ntt}\label{ntt:contraction-notation}
For a particle-hole operator string $Q=q_1\cd q_n$, let the $\gno{Q(\ctr{}{q}{_i}{q}q_iq_j)}$ denote $\gno{q_1\cd\ctr{}{q}{_i\cd}{q}q_i\cd q_j\cd q_n}$.
This is well-defined as long as $i<j$.
Let $\gno{\ol{Q}}$ stand for the sum of all unique single, double, triple, etc.\ contractions of $Q$
\begin{align*}
  \gno{\ol{Q}}
\equiv
  \sum_{k=1}^{\floor{\sfr{n}{2}}}
  \sum_{(i_1j_1)\cd(i_kj_k)}^{\mr{Ctr}_k(Q)}
  \no{Q(
    \ctr{}{q}{_{i_1}}{q}
    q_{i_1}q_{j_1}
    \cd
    \ctr{}{q}{_{i_k}}{q}
    q_{i_k}q_{j_k}
  )}
\end{align*}
where $\mr{Ctr}_k(Q)$ runs over all unique sets $\{(i_1j_2)\cd(i_kj_k)\,|\,i_p<j_p\}$ of $k$ pairs of operator indices in $Q$ and $\floor{\cdot}$ is the floor function.
Let $\gno{\ol{\ol{Q}}}$ denote the sum of all \textit{complete contractions}: the terms from $\gno{\ol{Q}}$ in which every operator in $Q$ is involved in a contraction.
Finally, let $\gno{\ctr{}{Q}{}{Q} QQ'}$ denote the sum of all single, double, etc.\ \textit{cross contractions}: those in which all contractions have an operator from $Q$ on the left and one from $Q'$ on the right.
In this context, contractions involving two operators from $Q$ or two operators from $Q'$ are called \textit{internal contractions} of $Q$ or $Q'$.
To review, in this notation equation~\ref{eq:wicks-theorem-first-example} would be written as
$
  a_p a_q a_s\dg a_r\dg
=
  \no{a_pa_qa_s\dg a_r\dg}
+
  \no{\ol{a_pa_qa_s\dg a_r\dg}}
$
and the last two terms on the right equal
$
  \no{\ol{\ol{a_pa_qa_s\dg a_r\dg}}}
$.
\end{ntt}

\begin{lem}\label{lem:pre-wick-lemma}
\thmtitle{$\no{Q}q=\no{Qq}+\sum_k\no{\ctr{Q(}{q}{_k)}{q} Q(q_k)q}$}
\begin{samepage}
\thmproof{
    Let $n$ be the number of operators in $Q$ and begin by assuming that $Q$ is
    already in normal order, i.e.~$\no{Q}=Q$.
    If $q$ is an annihilation operator then $\no{Q}q=Qq=\no{Qq}$ and all
    cross-contractions vanish, so the statement holds trivially.
    If $q$ is a creation operator, note that $\no{Qq}=(-)^nqQ$.
    By the anticommutator relation derived in
    appendix~\ref{app:pull-through-relation}, we find%
\footnote{%
    Note that
    \(
        (-)^{n-k}
    \)
    is the sign factor for bringing
    \(
        q_k
    \)
    and
    \(
        q
    \)
    together,
    and that
    \(
        \ctr{}{q}{_k}{q} q_kq
        =
        [q_k,q]_+
    \)
    when
    \(
        q
    \)
    is a creation operator.
}
    \begin{align*}
      \no{Q}q
    =
      Qq
    =
      (-)^n qQ
    +
      \sum_{k=1}^n
      (-)^{n-k}
      q_1\cd [q_k,q]_+\cd q_n
    =
      \no{Qq}
    +
      \sum_{k=1}^n
      \no{\ctr{Q(}{q}{_k)}{q} Q(q_k)q}
      \,,
    \end{align*}
    which completes the proof under our starting assumption.
    When
    \(
        Q
    \)
    is not in normal order, we can write
    \(
        \no{Q}
        =
        (-)^p
        Q'
    \)
    and apply the proposition to
    \(
        Q'
    \)
    in order to confirm that it still holds.%
\footnote{%
    For this last step, note that
    \(
        \no{(\no{Q})q}
        =
        \no{Qq}
    \).
}
}
\end{samepage}
\end{lem}

\begin{thm}\label{wicks-theorem-vacuum}
\thmtitle{Wick's theorem (vacuum).} $Q=\no{Q}+\no{\ol{Q}}$\vspace{5pt}
\thmproof{
  Let $n$ be the length of $Q$.
  The result holds for $n=2$ since $q_1q_2=\no{q_1q_2}+\ctr{}{q}{_1}{q} q_1q_2$ by the definition of contraction.
  Now, assume it holds for $n$ operators and consider $Qq$.
  By our inductive assumption, $Qq=\no{Q}q + \no{\ol{Q}}q$.
  Applying \cref{lem:pre-wick-lemma} to $\no{Q}q$ gives
  $\no{Q}q=\no{Qq}+\sum_i\no{\ctr{Q(}{q}{_i)}{q} Q(q_i)q}$.
  Expanding $\no{\ol{Q}}q$ and applying \cref{lem:pre-wick-lemma} to each term gives
  \\[5pt]
  {\footnotesize
  $\begin{array}{rl}\displaystyle
    \sum_{k=1}^{\floor{\sfr{n}{2}}}
    \sum_{(i_1j_1)\cd(i_kj_k)}^{\mr{Ctr}_k(Q)}
    \no{Q(
      \ctr{}{q}{_{i_1}}{q} q_{i_1}q_{j_1}
      \ctr{}{q}{_{i_k}}{q} q_{i_k}q_{j_k}
    )}
    q
  =&\displaystyle
    \sum_{k=1}^{\floor{\sfr{n}{2}}}
    \sum_{(i_1j_1)\cd(i_kj_k)}^{\mr{Ctr}_k(Q)}
    \no{Q(
      \ctr{}{q}{_{i_1}}{q} q_{i_1}q_{j_1}
      \ctr{}{q}{_{i_k}}{q} q_{i_k}q_{j_k}
      )q
    }
  +
    \sum_{k=1}^{\floor{\sfr{n}{2}}}
    \sum_{(i_1j_1)\cd(i_kj_k)}^{\mr{Ctr}_k(Q)}
    \sum_{i\notin\{i_1,j_1,\cd,i_k,j_k\}}
    %\sum_{\substack{i\,\notin\\\{i_1,j_1,\cd,i_k,j_k\}}}
    \no{Q(
      \ctr{}{q}{_{i_1}}{q} q_{i_1}q_{j_1}
      \ctr{}{q}{_{i_k}}{q} q_{i_k}q_{j_k}
      \ctr{}{q}{_i)}{q}    q_i)q
    }
  \\=&\displaystyle
    \sum_{(i_1j_1)}^{\mr{Ctr}_1(Q)}
    \no{Q(
      \ctr{}{a}{_{i_1}}{q} q_{i_1}q_{j_1}
      )q
    }
  +
    \sum_{k=2}^{\floor{\sfr{(n+1)}{2}}}
    \sum_{(i_1j_2)\cd(i_kj_k)}^{\mr{Ctr}_k(Qq)}
    \no{Qq(
      \ctr{}{q}{_{i_1}}{q}  q_{i_1}q_{j_1}
      \cd
      \ctr{}{q}{_{i_k}}{q}  q_{i_k}q_{j_k}
    )}
  \end{array}$}\\
  and, combining these results, we find
  \\[5pt]
  {\footnotesize
  $\begin{array}{rl}\displaystyle
    Qq
  =
    \no{Q}q
  +
    \no{\ol{Q}}q
  =&\displaystyle
    \no{Qq}
  +
    \sum_{i=1}^n
    \no{Q(
      \ctr{}{q}{_i)}{q}  q_i)q
    }
  +
    \sum_{(i_1j_1)}^{\mr{Ctr}_1(Q)}
    \no{Q(
      \ctr{}{a}{_{i_1}}{q} q_{i_1}q_{j_1}
      )q
    }
  +
    \sum_{k=2}^{\floor{\sfr{(n+1)}{2}}}
    \sum_{(i_1j_2)\cd(i_kj_k)}^{\mr{Ctr}_k(Qq)}
    \no{Qq(
      \ctr{}{q}{_{i_1}}{q}  q_{i_1}q_{j_1}
      \cd
      \ctr{}{q}{_{i_k}}{q}  q_{i_k}q_{j_k}
    )}
  \\[20pt]=&\displaystyle
    \no{Qq}
  +
    \sum_{k=1}^{\floor{\sfr{(n+1)}{2}}}
    \sum_{(i_1j_2)\cd(i_kj_k)}^{\mr{Ctr}_k(Qq)}
    \no{Qq(
      \ctr{}{q}{_{i_1}}{q}  q_{i_1}q_{j_1}
      \cd
      \ctr{}{q}{_{i_k}}{q}  q_{i_k}q_{j_k}
    )}
  \end{array}$}\\
  which is $\no{Qq}+\no{\ol{Qq}}$.
  So if the statement holds for strings of length $n$ it must also hold for strings of length $n+1$.
  By induction, the theorem holds for $Q$ of arbitrary length.
}
\end{thm}

\begin{thm}\label{wicks-theorem}
\thmtitle{Wick's theorem (general)} $Q=\gno{Q}+\gno{\ol{Q}}$\vspace{5pt}
\thmproof{
    Under particle-hole isomorphism,
    \(
        \Phi
    \)-normal
    ordering is algebraically identical to vacuum normal ordering.
}
\end{thm}

\begin{cor}\label{wick-operator-product}
\thmtitle{Wick's theorem for operator products}\vspace{5pt}
$\gno{Q}\gno{Q'}=\gno{QQ'}+\gno{\ctr{}{Q}{}{Q}  QQ'}$
\thmproof{
  By Wick's theorem $\gno{Q}\gno{Q'}=\gno{QQ'}+\gno{\pr{\ol{\gno{Q}\gno{Q'}}}}$ and since $\gno{Q}$ and $\gno{Q'}$ are in normal order their contractions vanish, leaving $\gno{\pr{\ol{\gno{Q}\gno{Q'}}}}=\gno{\ctr{}{Q}{}{Q} QQ'}$.
}
\end{cor}

\begin{cor}\label{wick-expectation-value}
\thmtitle{Wick's theorem for expectation values}\vspace{5pt}
$\ip{\F|Q|\F}=\gno{\ol{\ol{Q}}}$
\thmproof{
  From Wick's theorem $\ip{\F|Q|\F}=\ip{\F|\gno{Q}|\F}+\ip{\F|\gno{\ol{Q}}|\F}$ and any incompletely contracted terms have vanishing expectation values. Therefore, $\ip{\F|\gno{Q}|\F}=0$ and $\ip{\F|\gno{\ol{Q}}|\F}=\gno{\ol{\ol{Q}}}$.
}
\end{cor}

\begin{rmk}
To recap, let's state Wick's theorem and its corollaries in words as the following three rules.
\begin{enumerate}
\item
  An operator string equals its normal ordering plus all contractions.
\item
  A product of normal-ordered operators equals the normal ordering of the product plus all cross-contractions.
\item
  The reference expectation value of a string equals the sum of its complete contractions.
\end{enumerate}
The next proposition provides a convenient rule for evaluating completely
contracted operator strings.
\end{rmk}

\begin{prop}
\thmstatement{The sign of a completely contracted string is $\pr{-}^c$ where $c$ is the number of contraction line intersections.}\vspace{5pt}
\thmproof{
  Let
  $\e_{\pi}
  \gno{
    \ctr{}{q}{_{\pi(1)}}{q}     q_{\pi(1)}q_{\pi(2)}
    \cd
    \ctr{}{q}{_{\pi(n-1)}}{q}  q_{\pi(n-1)}q_{\pi(n)}
  }$
  be the disentangled form of a complete contraction of $q_1\cd q_n$, where $n$ is an even integer.
  The phase factor for the contraction, $\e_{\pi}$, is equal to the signature of the disentangling permutation, which is equal to the signature of the inverse permutation $\pi^{-1}$, restoring the original ordering of the operators.
  $\pi^{-1}$ can be expressed as a series of transpositions swapping pairs of operators not connected by a contraction line.
  Since every operator has a contraction line overhead, each of these transpositions changes the number of line intersections by exactly $\pm1$, so $\e_{\pi}=(-)^c$ where $c$ is the number of intersections in the original contraction pattern.
}
\end{prop}

\begin{dfn}
\thmtitle{Correlation component of the Hamiltonian}
Using Wick's theorem, we can expand $\vac$-normal one- and two-particle excitations as linear combinations of $\F$-normal-ordered ones.
\begin{align}
&
  a_p\dg a_q
=
  \gno{a_p\dg a_q}
+
  \g_{pq}
&&
  a_p\dg a_q\dg a_sa_r
=
  \gno{a_p\dg a_q\dg a_sa_r}
-
  \g_{ps}\gno{a_q\dg a_r}
+
  \g_{pr}\gno{a_q\dg a_s}
+
  \g_{qs}\gno{a_p\dg a_r}
-
  \g_{qr}\gno{a_p\dg a_s}
+
  \g_{pr}\g_{qs}
-
  \g_{ps}\g_{qr}
\end{align}
Substituting these into the electronic Hamiltonian leads to an expression for $H$ in terms of $\F$-normal operators.
\begin{align}
&&
  H
=
  E_\mr{ref}
+
  \sum_{pq}^\infty
  f_{pq}
  \gno{a_p\dg a_q}
+
  \tfr{1}{4}
  \sum_{pqrs}^\infty
  \ip{pq||rs}
  \gno{a_p\dg a_q\dg a_sa_r}
&&
\begin{array}{r@{\ }l}
  E_\mr{ref}
\equiv&
\ds{
  \sum_{pq}^\infty
  h_{pq}\g_{pq}
+
  \tfr{1}{2}
  \sum_{pqrs}^\infty
  \ip{pq||rs}
  \g_{pr}\g_{qs}
}
\\[5pt]
  f_{pq}
\equiv&
  h_{pq}
+
  \ds{\sum_{rs}^{\infty}}
  \ip{pr||qs}
  \g_{rs}
\end{array}
\end{align}
Note that $E_\mr{ref}$ is another expression for the Hartree-Fock energy, $E_\mr{ref}=\ip{\F|\op{H}|\F}$, and $f_{pq}$ is the matrix representation of the Fock operator in the spin-orbital basis, $\bm{f}=[f_{pq}]$ where $f_{pq}=\ip{\y_p|\op{f}|\y_q}$.
The second and third terms in this expression together make up the \textit{correlation component} of the electronic Hamiltonian, $H_\mr{c}\equiv H - \ip{\F|H|\F}$.
\end{dfn}

\begin{ex}
\thmtitle{Derivation of CIS equations}
The configuration interaction singles (CIS) equations take the form
\begin{align}
  \sum_{jb}
  \ip{\F_i^a|H - E_\mr{ref}|\F_j^b}\,
  (\bo{c}_k)_b^j
=
  \w_k\,
  (\bo{c}_k)_a^i
\end{align}
where $\w_k$ approximates the excitation energy of the $k\eth$ state, $\w_k=E_k-E_\mr{ref}$.
In order to solve the CIS eigenvalue problem, we need to have an expression for the matrix elements
$\ip{\F_i^a|H_\mr{c}|\F_j^b}$ in terms of our known quantities, the one- and two-electron integrals.
To do this, we can evaluate
$\ip{\F_i^a|\gno{a_p\dg a_q}|\F_j^b}$
and
$\ip{\F_i^a|\gno{a_p\dg a_q\dg a_s a_r}|\F_j^b}$
using Wick's theorem.
\begin{align*}
  \ip{\F|\gno{a_i\dg a_a}\,\gno{a_p\dg a_q}\,\gno{a_b\dg a_j}|\F}
=&\
  \gno{
    \ctr[1]{}{a}{_i\dg a_a a_p\dg a_q a_b\dg }{a}
    \ctr{a_i\dg }{a}{_a}{a}
    \ctr{a_i\dg a_aa_p\dg}{a}{_q}{a}
    a_i\dg a_a a_p\dg a_q a_b\dg a_j
  }
+
  \gno{
    \ctr[1.5]{}{a}{_i\dg a_a a_p\dg }{a}
    \ctr[2.0]{a_i\dg a_a}{a}{_p\dg a_q a_b\dg}{a}
    \ctr[0.5]{a_i\dg}{a}{_a a_p\dg a_q}{a}
    a_i\dg a_a a_p\dg a_q a_b\dg a_j
  }
=
  \g_{ij}
  \h_{ap}
  \h_{qb}
-
  \g_{iq}
  \g_{pj}
  \h_{ab}
\\
  \ip{\F|\gno{a_i\dg a_a}\,\gno{a_p\dg a_q\dg a_s a_r}\,\gno{a_b\dg a_j}|\F}
=&\
  \gno{
    \ctr[1]{}{a}{_i\dg a_a a_p\dg a_q\dg }{a}
    \ctr{a_i\dg }{a}{_a}{a}
    \ctr[2]{a_i\dg a_a a_p\dg }{a}{_q\dg a_s a_r a_b\dg }{a}
    \ctr{a_i\dg a_a a_p\dg a_q\dg a_s}{a}{_r}{a}
    a_i\dg a_a a_p\dg a_q\dg a_s a_r a_b\dg a_j
  }
+
  \gno{
    \ctr[1]{}{a}{_i\dg a_a a_p\dg a_q\dg }{a}
    \ctr{a_i\dg }{a}{_a a_p\dg}{a}
    \ctr[2]{a_i\dg a_a }{a}{_p\dg a_q\dg a_s a_r a_b\dg }{a}
    \ctr{a_i\dg a_a a_p\dg a_q\dg a_s}{a}{_r}{a}
    a_i\dg a_a a_p\dg a_q\dg a_s a_r a_b\dg a_j
  }
\\&\
+
  \gno{
    \ctr[1]{}{a}{_i\dg a_a a_p\dg a_q\dg a_s}{a}
    \ctr{a_i\dg }{a}{_a}{a}
    \ctr[2]{a_i\dg a_a a_p\dg }{a}{_q\dg a_s a_r a_b\dg }{a}
    \ctr{a_i\dg a_a a_p\dg a_q\dg}{a}{_s a_r}{a}
    a_i\dg a_a a_p\dg a_q\dg a_s a_r a_b\dg a_j
  }
+
  \gno{
    \ctr[1]{}{a}{_i\dg a_a a_p\dg a_q\dg a_s}{a}
    \ctr{a_i\dg }{a}{_a a_p\dg}{a}
    \ctr[2]{a_i\dg a_a }{a}{_p\dg a_q\dg a_s a_r a_b\dg }{a}
    \ctr{a_i\dg a_a a_p\dg a_q\dg}{a}{_s a_r}{a}
    a_i\dg a_a a_p\dg a_q\dg a_s a_r a_b\dg a_j
  }
\\=&\
-
  \g_{is}
  \h_{ap}
  \g_{qj}
  \h_{rb}
+
  \g_{is}
  \h_{aq}
  \g_{pj}
  \h_{rb}
+
  \g_{ir}
  \h_{ap}
  \g_{qj}
  \h_{sb}
-
  \g_{ir}
  \h_{aq}
  \g_{pj}
  \h_{sb}
\end{align*}
Multiplying these by $f_{pq}$ and $\tfr{1}{4}\ip{pq||rs}$ and summing over Hamiltonian indices yields the following.
\begin{align*}
  \ip{\F_i^a|H_\mr{c}|\F_j^b}
=
  f_{ab}\g_{ij}
-
  f_{ji}
  \h_{ab}
-
  \ip{aj||bi}
\end{align*}
\end{ex}

\newpage
\appendix
\section{The pull-through relation}\label{app:pull-through-relation}

\begin{prop}\label{prop:pull-through-relation}
\thmtitle{Pull-through relation}
\thmstatement{
  For any non-commuting $x_1,\ld,x_n$, and $y$ for which addition, subtraction and multiplication are defined, $x_1\cd x_ny=\pr{\mp}^nyx_1\cd x_n+\sum_{k=1}^n\pr{\mp}^{n-k}x_1\cd[x_k,y]_{\pm}\cd x_n$, where $[x,y]_{\pm}\equiv xy\pm yx$.
}
\thmproof{
  The $n=1$ case follows from the definition of the commutator brackets: $xy=\mp yx+[x,y]_{\pm}$.
  Now, assume it holds for $n$ and consider the $n+1$ case.
  Since $x_1\cd x_{n+1}y=x_1\cd x_n(\mp yx_{n+1}+[x_{n+1},y]_{\pm})$, we find
  \begin{align*}
    x_1\cd x_{n+1}y
  =&\
  \mp
    \pr{
      \pr{\mp}^nyx_1\cd x_n
    +
      \sum_{k=1}^n
      \pr{\mp}^{n-k}
      x_1\cd [x_k,y]_{\pm}\cd x_n
    }
    x_{n+1}
  +
    x_1\cd x_n[x_{n+1},y]_{\pm}
  \\=&\
    \pr{\mp}^{n+1}
    yx_1\cd x_{n+1}
  +
    \sum_{k=1}^{n+1}
    \pr{\mp}^{n+1-k}
    x_1\cd[x_k,y]_{\pm}\cd x_{n+1}
  \end{align*}
  and, by induction, the result holds for all $n$.
}
\end{prop}

\end{document}


% ------------------------------------------------------------------
\chapter{Introduction to MatConvNet}\label{s:intro}
% ------------------------------------------------------------------

\vlnn is a simple MATLAB toolbox implementing Convolutional Neural Networks (CNN) for computer vision applications. This documents starts with a short overview of CNNs and how they are implemented in \vlnn. Section~\ref{s:blocks} lists all the computational building blocks implemented in \vlnn that can be combined to create CNNs and gives the technical details of each one. Finally, Section~\ref{s:wrappers} discusses more abstract CNN wrappers and example code and models.

A \emph{Convolutional Neural Network} (CNN) can be viewed as a function $f$ mapping data $\bx$, for example an image, on an output vector $\by$. The function $f$ is a composition of a sequence (or a directed acyclic graph) of simpler functions $f_1,\dots,f_L$, also called \emph{computational blocks} in this document. Furthermore, these blocks are \emph{convolutional}, in the sense that they map an input image of feature map to an output feature map by applying a translation-invariant and local operator, e.g. a linear filter. The \vlnn toolbox contains implementation for the most commonly used computational blocks (described in Section~\ref{s:blocks}) which can be used either directly, or through simple wrappers. Thanks to the modular structure, it is a simple task to create and combine new blocks with the existing ones. %New blocks are also easy to create and combine with the existing ones.

Blocks in the CNN usually contain parameters $\bw_1,\dots,\bw_L$. These are \emph{discriminatively learned from example data} such that the resulting function $f$ realizes an useful mapping. A typical example is image classification; in this case the output of the CNN is a vector $\by=f(\bx)\in\real^C$ containing the confidence that $\bx$ belong to any of $C$ possible classes. Given training data $(\bx^{(i)},\by^{(i)})$ (where $\by^{(i)}$ is the indicator vector of the class of $\bx^{(i)}$), the parameters are learned by solving
\begin{equation}\label{e:objective}
 \argmin_{\bw_1,\dots\bw_n}
 \frac{1}{n}\sum_{i=1}^n
 \ell\left(
 f(\bx^{(i)};\bw_1,\dots,\bw_L),
 \by^{(i)}
 \right)
\end{equation}
where $\ell$ is a suitable \emph{loss function} (e.g. the hinge or log loss).

The optimization problem~\eqref{e:objective} is usually non-convex and very large as complex CNN architectures need to be trained from hundred-thousands or even millions of examples. Therefore efficiency is a paramount. Optimization often uses a variant of \emph{stochastic gradient descent}. The algorithm is, conceptually, very simple: at each iteration a training point is selected at random, the derivative of the loss term for that training sample is computed resulting in a gradient vector, and parameters are incrementally updated by moving towards the local minima in the direction of the gradient. The key operation here is to compute the derivative of the objective function, which is obtained by an application of the chain rule known as \emph{back-propagation}. \vlnn can evaluate the derivatives of all the computational blocks. It also contains several examples of training small and large models using these capabilities and a default solver, although it is easy to write customized solvers on top of the library.

While CNNs are relatively efficient to compute, training requires iterating many times through vast data collections. Therefore the computation speed is very important in practice. Larger models, in particular, may require the use of GPU to be trained in a reasonable time. \vlnn has integrated GPU support based on NVIDIA CUDA and MATLAB built-in CUDA capabilities.

% ------------------------------------------------------------------
\section{\vlnn at a glance}\label{s:vlnn}
% ------------------------------------------------------------------

\vlnn has a simple design philosophy. Rather than wrapping CNNs around complex layers of software, it exposes simple functions to compute CNN building blocks, such as linear convolution and ReLU operators. These building blocks are easy to combine into a complete CNNs and can be used to implement sophisticated learning algorithms. While several real-world examples of small and large CNN architectures and training routines are provided, it is always possible to go back to the basics and build your own, using the efficiency of MATLAB in prototyping. Often no C coding is required at all to try a new architectures. As such, \vlnn is an ideal playground for research in computer vision and CNNs.

\vlnn contains the following elements:
\begin{itemize}
\item \emph{CNN computational blocks.} A set of optimized routines computing fundamental building blocks of a CNN. For example, a convolution block is implemented by \linebreak \verb!y=vl_nnconv(x,f,b)! where \verb!x! is an image, \verb!f! a filter bank, and \verb!b! a vector of biases (Section~\ref{s:convolution}). The derivatives are computed as
\verb![dzdx,dzdf,dzdb] = vl_nnconv(x,f,b,dzdy)! where \verb!dzdy! is the derivative of the CNN output w.r.t \verb!y!~(Section~\ref{s:convolution}). Section~\ref{s:blocks} describes all the blocks in detail.
\item \emph{CNN wrappers.} \vlnn provides a simple wrapper, suitably invoked by \verb!vl_simplenn!, that implements a CNN with a linear topology (a chain of blocks). This is good enough to run most of current state-of-the-art models for image classification. You are invited to look at the implementation of this function, as it is a great starting point to understand how to implement more complex CNNs.
\item \emph{Example applications.} \vlnn provides several example of learning CNNs with stochastic gradient descent and CPU or GPU, on MNIST, CIFAR10, and ImageNet data.
\item \emph{Pre-trained models.} \vlnn provides several state-of-the-art pre-trained CNN models that can be used off-the-shelf, either to classify images or to produce image encodings in the spirit of Caffe or DeCAF.
\end{itemize}

% ------------------------------------------------------------------
\subsection{The structure and evaluation of CNNs}\label{s:forward}
% ------------------------------------------------------------------

CNNs are obtained by connecting one or more \emph{computational blocks}. Each block $\by = f(\bx,\bw)$ takes an image $\bx$ and a set of parameters $\bw$ as input and produces a new image $\by$ as output. An image is a real 4D array; the first two dimensions index spatial coordinates (image rows and columns respectively), the third dimension feature channels (there can be any number), and the last dimension image instances. A computational block $f$ is therefore represented as follows:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x) [data] {$\bx$};
\node (f) [block,right of=x]{$f$};
\node (y) [data, right of=f] {$\by$};
\node (w) [data, below of=f] {$\bw$};
\draw [->] (x.east) -- (f.west) {};
\draw [->] (f.east) -- (y.west) {};
\draw [->] (w.north) -- (f.south) {};
\end{tikzpicture}
\end{center}
Formally, $\bx$ is a 4D tensor stacking $N$ 3D images
\[
   \bx \in \real^{H \times W \times D \times N}
\]
where $H$ and $W$ are the height and width of the images, $D$ its depth, and $N$ the number of images. In what follows, all operations are applied identically to each image in the stack $\bx$; hence for simplicity we will drop the last dimension in the discussion (equivalent to assuming $N=1$), but the ability to operate on image batches is very important for efficiency.

In general, a CNN can be obtained by connecting blocks in a directed acyclic graph (DAG). In the simplest case, this graph reduces to a sequence of computational blocks $(f_1,f_2,\dots,f_L)$. Let $\bx_1,\bx_2,\dots,\bx_L$ be the output of each layer in the network, and let $\bx_0$ denote the network input. Each output $\bx_l$ depends on the previous output $\bx_{l-1}$ through a function $f_l$ with parameter $\bw_l$ as $\bx_l = f_l(\bx_{l-1};\bw_l)$; schematically:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x0)  [data] {$\bx_0$};
\node (f1) [block,right of=x0]{$f_1$};
\node (f2) [block,right of=f1,node distance=3cm]{$f_2$};
\node (dots) [right of=f2]{...};
\node (fL) [block,right of=dots]{$f_L$};
\node (xL)  [data, right of=fL] {$\bx_L$};
\node (w1) [data, below of=f1] {$\bw_1$};
\node (w2) [data, below of=f2] {$\bw_2$};
\node (wL) [data, below of=fL] {$\bw_L$};
\draw [->] (x0.east) -- (f1.west) {};
\draw [->] (f1.east) -- node {$\bx_2$} (f2.west);
\draw [->] (f2.east) -- node {$\bx_3$} (dots.west) {};
\draw [->] (dots.east) -- node {$\bx_{L-1}$} (fL.west) {};
\draw [->] (fL.east) -- (xL.west) {};
\draw [->] (w1.north) -- (f1.south) {};
\draw [->] (w2.north) -- (f2.south) {};
\draw [->] (wL.north) -- (fL.south) {};
\end{tikzpicture}
\end{center}
Given an input $\bx_0$, evaluating the network is a simple matter of evaluating all the intermediate stages in order to compute an overall function $\bx_L = f(\bx_0;\bw_1,\dots,\bw_L)$.

% ------------------------------------------------------------------
\subsection{CNN derivatives}\label{s:backward}
% ------------------------------------------------------------------

In training a CNN, we are often interested in taking the derivative of a loss $\ell : f(\bx,\bw) \mapsto \real$ with respect to the parameters. This effectively amounts to extending the network with a \emph{scalar block} at the end:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x0)  [data] {$\bx_0$};
\node (f1) [block,right of=x0]{$f_1$};
\node (f2) [block,right of=f1,node distance=3cm]{$f_2$};
\node (dots) [right of=f2]{...};
\node (fL) [block,right of=dots]{$f_L$};
\node (loss) [block,right of=fL,node distance=3cm]{$\ell$};
\node (w1) [data, below of=f1] {$\bw_1$};
\node (w2) [data, below of=f2] {$\bw_2$};
\node (wL) [data, below of=fL] {$\bw_L$};
\node (z) [data, right of=loss] {$z\in\real$};
\draw [->] (x0.east) -- (f1.west) {};
\draw [->] (f1.east) -- node {$\bx_2$} (f2.west);
\draw [->] (f2.east) -- node {$\bx_3$} (dots.west) {};
\draw [->] (dots.east) -- node {$\bx_{L-1}$} (fL.west) {};
\draw [->] (fL.east) -- node {$\bx_L$} (loss.west);
\draw [->] (loss.east) -- (z) {};
\draw [->] (w1.north) -- (f1.south) {};
\draw [->] (w2.north) -- (f2.south) {};
\draw [->] (wL.north) -- (fL.south) {};
\end{tikzpicture}
\end{center}
The derivative of $\ell \circ f$ with respect to the parameters can be computed but starting from the end of the chain (or DAG) and working backwards using the chain rule, a process also known as back-propagation. For example the derivative w.r.t. $\bw_l$ is:
\begin{equation}\label{e:chain-rule}
 \frac{dz}{d(\vv\bw_l)^\top}
 =
 \frac{dz}{d(\vv\bx_{L})^\top}
 \frac{d\vv\bx_{L}}{d(\vv\bx_{L-1})^\top}
 \dots
 \frac{d\vv\bx_{l+1}}{d(\vv\bx_{l})^\top}
 \frac{d\vv\bx_{l}}{d(\vv\bw_{l})^\top}.
\end{equation}
Note that the derivatives are implicitly evaluated at the working point determined by the input $\bx_0$ during the evaluation of the network in the forward pass. The $\vv$ symbol is the vectorization operator, which simply reshape its tensor argument to a column vector. This notation for the derivatives is taken from~\cite{kinghorn96integrals} and is used throughout this document.

Computing~\eqref{e:chain-rule} requires computing the derivative of each block $\bx_l = f_l(\bx_{l-1},\bw_l)$ with respect to its parameters $\bw_l$ and input $\bx_{l-1}$. Let us know focus on computing the derivatives for one computational block. We can look at the network as follows:
\[
    \underbrace{
    \ell \circ f_{L}(\cdot,\bw_L)
     \circ f_{L-1}(\cdot,\bw_{L-1})
     \dots
     \circ f_{l+1}(\cdot,\bw_{l+1})
     }_{\displaystyle z(\cdot)}
     \circ f_{l}(\bx_l,\bw_{l})
     \circ \dots
\]
where $\circ$ denotes the composition of function. For simplicity, lump together the factors from $f_l+1$ to the loss $\ell$ into a single scalar function $z(\cdot)$ and drop the subscript $l$ from the first block. Hence, the problem is to compute the derivative of $(z \circ f)(\bx,\bw) \in \real$ with respect to the data $\bx$ and the parameters $\bw$. Graphically:
\begin{center}
\begin{tikzpicture}[auto, node distance=2cm]
\node (x) [data] {$\bx$};
\node (f) [block,right of=x ] {$f$};
\node (bz)[block,right of=f ] {$z(\cdot)$};
\node (z) [data, right of=bz] {$z$};
\node (w) [data, below of=f ] {$\bw$};
\draw [->] (x.east) -- (f.west) {};
\draw [->] (f.east) -- node {$\by$}  (bz.west) {};
\draw [->] (w.north) -- (f.south) {};
\draw [->] (bz.east) -- (z.west) {};
\end{tikzpicture}
\end{center}
The derivative of $z \circ f$ with respect to $\bx$ and $\bw$ are given by:
\[
\frac{dz}{d(\vv \bx)^\top}
=
\frac{dz}{d(\vv \by)^\top}
\frac{d\vv f}{d(\vv \bx)^\top},
\quad
\frac{dz}{d(\vv \bw)^\top}
=
\frac{dz}{d(\vv \by)^\top}
\frac{d\vv f}{d(\vv \bw)^\top},
\]
We note two facts. The first one is that, since $z$ is a scalar function, the derivatives have a number of elements equal to the number of parameters. So in particular $dz/d\vv \bx^\top$ can be reshaped into an array $dz/d\bx$ with the same shape of $\bx$, and the same applies to the `derivatives $dz/d\by$ and $dz/d\bw$. Beyond the notational convenience, this means that storage for the derivatives is not larger than the storage required for the model parameters and forward evaluation.

The second fact is that computing $dz/d\bx$ and $dz/d\bw$ requires the derivative $dz/d\by$. The latter can be obtained by applying this calculation recursively to the next block in the chain.

% ------------------------------------------------------------------
\subsection{CNN modularity}\label{s:modularity}
% ------------------------------------------------------------------

Sections~\ref{s:forward} and~\ref{s:backward} suggests a modular programming interface for the implementation of CNN modules. Abstractly, we need two functionalities:
\begin{itemize}
\item \emph{Forward messages:} Evaluation of the output $\by=f(\bx,\bw)$ given input data $\bx$ and parameters $\bw$ (forward message).
\item \emph{Backward messages:} Evaluation of the CNN derivative $dz/d\bx$ and $dz/d\bw$ with respect to the block input data $\bx$ and parameters $\bw$ given the block input data $\bx$ and paramters $\bw$ as well as the CNN derivative $dx/d\by$ with respect to the block output data $\by$.
\end{itemize}

% ------------------------------------------------------------------
\subsection{Working with DAGs}\label{s:dag}
% ------------------------------------------------------------------

CNN can also be obtained from more complex composition of functions forming a DAG. There are $n+1$ variables $\bx_i$ and $n$ functions $f_i$ with corresponding arguments $\br_{ik}$:
\begin{align*}
\bx_0 &\\
\bx_1 &= f_1(\br_{1,1},\dots,\br_{1,m_1}),\\
\bx_2 &= f_2(\br_{2,1},\dots,\br_{2,m_2}),\\
&\vdots\\
\bx_n &= f_n(\br_{n,1},\dots,\br_{n,m_n})
\end{align*}
Variables are connected to arguments by relations
\[
     \br_{ik} = \bx_{\pi_{ik}}
\]
where $\pi_{ik}$ denotes the variable $\bx_{\pi_{ik}}$  that feeds argument $\br_{ik}$. Together with the implicit fact that each argument $\br_{ik}$ feeds into the  variable $\bx_i$ through function $f_i$, this defines a bipartite DAG representing the dependencies  between variables and argument. The DAG is bipartite. This DAG must be acyclic, hence assuring that, given the value of the input $\bx_0$, all other variables can be evaluated iteratively. Due to this property, without loss of generality we will assume that variables $\bx_0,\bx_1,\dots,\bx_n$ are sorted such that $\bx_i$ depends only on variables that come before in the order (i.e. one always has $\pi_{ik} < i$).

Now assume that $\bx_n = z$ is a scalar network output (usually the learning loss). As before, we are interested in computing derivatives of this function with respect to the network variables. In order to do so, write the output of the DAG $z(\bx_i)$ as a function of the variable $\bx_i$; this has the following interpretation: the DAG is modified by removing function $f_i$ and making $\bx_i$ an input, setting $\bx_i$ to the specified value, setting all the other inputs to the working point at which the derivative should be computed, and evaluating the resulting DAG. Likewise, one can define functions $z(\br_{ij})$ for the arguments. The derivative with respect to $\bx_j$ can then be expressed as
\[
   \frac{dz}{d(\vv \bx_j)^\top}
   =
  \sum_{(i,k): \pi_{ik}=j}
  \frac{dz}{d(\vv \bx_i)^\top}
  \frac{d\vv \bx_i}{d(\vv \br_{ik})^\top}
  \frac{d\vv \br_{ik}}{d(\vv \bx_j)^\top}
   =
   \sum_{(i,k): \pi_{ik}=j}
   \frac{dz}{d(\vv \bx_i)^\top}
   \frac{d\vv f_i}{d(\vv \br_{ik})^\top}.
\]
Note that these quantities can be computer recursively, backward from the output $z$. In this case, each variable node $\bx_i$ (corresponding to $f_i$) stores the derivative  $dz/d \bx_i$. When node $\bx_i$ is processed, the derivatives $d\vv f_i /d(\vv \br_{ik})^\top$ are computed for the $m_i$ arguments of the corresponding function $f_i$ pre-multiplied by the factor $dz/d \bx_i$ (available at that node) and then accumulated to the corresponding parent variables $\bx_j$'s derivatives $dz/d \bx_j$.

% ------------------------------------------------------------------
\section{About \vlnn}
% ------------------------------------------------------------------

\vlnn main features are:
\begin{itemize}
\item \emph{Flexibility.} Neural network layers are implemented in a straightforward manner, often directly in MATLAB code, so that they are easy to modify, extend, or integrate with new ones.
\item \emph{Power.} The implementation can run the latest models such as Krizhevsky~\textit{et al.}~\cite{krizhevsky12imagenet}, including the DeCAF and Caffe variants, and variants from the Oxford Visual Geometry Group. Pre-learned features for different tasks can be easily downloaded.
\item \emph{Efficiency.} The implementation is quite efficient, supporting both CPU and GPU computation (in the latest versions of MALTAB).
\item \emph{Self contained.} The implementation is fully self-contained, requiring only MATLAB and a compatible C/C++ compiler to work (GPU code requires the freely-available CUDA DevKit). Several fully-functional image classification examples are included.\end{itemize}

\paragraph{Relation to other CNN implementations.} There are many other open-source CNN implementations. \vlnn borrows its convolution algorithms from Caffe (and is in fact capable of running most of Caffe's models). Caffe is a \cpp framework using a custom CNN definition language based on Google Protocol Buffers. Both \vlnn and Caffe are predated by Cuda-Convnet~\cite{krizhevsky12imagenet}, a \cpp-based project that allows defining a CNN architectures using configuration files. While Caffe and Cuda-Convnet can be somewhat faster than \vlnn, the latter exposes individual CNN building blocks as MATLAB functions, as well as integrating with the native MATLAB GPU support, which makes it very convenient for fast prototyping. The DeepLearningToolbox~\cite{deepltbx12} is a MATLAB toolbox implementing, among others, CNNs, but it does not seem to have been tested on large scale problems. While \vlnn specialises on CNNs and computer vision applications, there are several general-purpose machine learning frameworks which include CNN support, but none of them interfaces natively with MATLAB. For example, the Torch7 toolbox~\cite{collobert2011torch7} uses Lua and Theano~\cite{bergstra2010} uses Python.

% ------------------------------------------------------------------
\subsection{Acknowledgments}\label{s:ack}
% ------------------------------------------------------------------

The implementation of several CNN computations in this library are inspired by the Caffe library~\cite{jia13caffe} (however, Caffe is \emph{not} a dependency). Several of the example networks have been trained by Karen Simonyan as part of~\cite{chatfield14return}.

We kindly thank NVIDIA for suppling GPUs used in the creation of this software.


\documentclass[a4paper,12pt]{article}
\usepackage[T2A]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[english,russian]{babel}
\usepackage{amsmath,amsfonts,amssymb,amsthm,mathtools}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath, amsfonts, amssymb, amsthm, mathtools}
\usepackage{enumitem}
\usepackage{float}
\usepackage{graphicx}
\usepackage{tikz}
\graphicspath{ {images/} }
\usepackage{graphicx}
\graphicspath{{pictures/}}
\DeclareGraphicsExtensions{.pdf,.png,.jpg}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}

\newenvironment{theorem}[2][Theorem]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{lemma}[2][Lemma]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{exercise}[2][Exercise]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{reflection}[2][Reflection]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{proposition}[2][Proposition]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\newenvironment{corollary}[2][Corollary]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
\author{Vyacheslav Gorchakov}
\title{PCA on higher orders.}
\date{2021}
\begin{document}
\maketitle
\newpage

\smallskip

\begin{center}
{\bf Motivation }
\end{center}

Principal component analysis (PCA) is a classical
statistic technique that has been applied to many fields, such
as knowledge representation, face recognition and image
compression. The objectives of PCA are to reduce the dimensionality of the dataset and identify new meaningful underlying variables. The key idea is to project the objects to an orthogonal subspace for their compact representations. It can be noted that in the classical 1D-PCA
the 2D data sample (e.g. image) must be initially converted
to a 1D vector form. The resulting sample vector will lead
to a high dimensional vector space. It is consequently difficult to evaluate the covariance matrix accurately when the sample vector is very long and the number of training samples is small. However, advances of PCA approach have been mostly limited to vector or matrix type data despite the fact that continued advances in information and sensing technology have been making large-scale, multi-modal, and multi-relational datasets evermore commonplace. Indeed, such multimodal data sets are now commonly encountered in a hugevariety of applications including chemometrics, hyperspectral imaging, high resolution videos, neuroimaging (EEG, fMRI), biometrics and social network analysis. When applied to these higher order data sets, standard vector and matrix models such as PCA have been shown to be inadequate at capturing the cross-couplings across the different modes and burdened
by the increasing storage and computational costs. Therefore, there is a growing need for PCA type methods that can learn from tensor data while respecting its inherent multi-modal structure for multilinear dimensionality reduction and subspace estimation.

\begin{center}
{\bf Theory }
\end{center}
Let $[n] = {1,2,...,n}$ for all $n \in \mathbb{N}$. A $d-mode$ or $dth-order$ tensor is simply a d-dimensional array of complex valued data $\mathbb{A} \in \mathbb{C}^{n_1 \times n_2 \times .. \times n_d}$ for given dimensions $n_1, ..., n_d \in \mathbb{N}$.

The \textit{vectorization} of $\mathbb{A} \in \mathbb{C}^{n_1 \times n_2 \times .. \times n_d}$ will always reshape $\mathbb{A}$ into a vector denoted by $a \in \mathbb{C}^{n_1 \cdot n_2 \cdot...n_d}$

\paragraph{The Standard Inner Product Space of $d$-mode Tensors.}

Let $\mathbb{A} \in \mathbb{C}^{n_1 \times n_2 \times .. \times n_d}$ and $\mathbb{B} \in \mathbb{C}^{n_1^{'} \times n_2^{'} \times .. \times n_d^{'}}$ be two tensors. The result of tensor product is a $(d + d^{'})$-mode tensor whose entries are given by

$${(\mathbb{A} \times \mathbb{B})}_{i_1, .., i_d, {i_1}^{'},.. {i_d}^{'}} = \sum_{i_1 = 1}^{n_1} \sum_{i_2 = 1}^{n_2} ... \sum_{i_d = 1}^{n_d} a(i_1, .., i_d) \overline{b({i_1}^{'},.. {i_d}^{'})}$$
where $\overline{(\cdot)}$ means the complex conjugate operation.

This inner product then gives rise to the standart Eucledian norm

$$\|\mathbb{A} \| = \sqrt{(\mathbb{A} \times \mathbb{A})} = \sqrt{\sum_{i_1 = 1}^{n_1} \sum_{i_2 = 1}^{n_2} ... \sum_{i_d = 1}^{n_d}|a(i_1, .., i_d)|^2}$$
\paragraph{Trivial PCA for Tensors Based on Implicit Vectorization}

Here we have d-mode tensors $\mathbb{A}_1,\mathbb{A}_2, .., \mathbb{A}_n \in \mathbb{R}^{n_1 \times n_2 \times .. \times n_d}$ each of which represents an individial's images. Assume that the image data has been centered so that $\sum_{j = 1}^m \mathbb{A}_j = 0$, this problem reduces to finding a set of $R < m$ ortonormal "eigenface" basis tensors $\mathbb{B}_1,\mathbb{B}_2, .., \mathbb{B}_n \in \mathbb{R}^{n_1 \times n_2 \times .. \times n_d}$ whose span $S$:
$$S = \left[ \sum_{j = 1}^R \alpha_j \mathbb{B_j} | \alpha \in \mathbb{R}^R \right]$$
minimizes the error
$$E_{PCA} (S) = \sum_{j=1}^m \min \| \mathbb{A_j} - \mathbb{X_j} \|^2$$
where $\mathbb{X_j} \in S$ and $\mathbb{X_j}$ provides approximation for each of the original individuals image data $\mathbb{A_j}$ in compressed form.

\paragraph{Multilinear Principal Component Analysis (MPCA)}

MPCA performs feature extraction by determing a multilinear projection that captures most of the original tensorial input variations.

We have d-mode tensors $\mathbb{A}_1,\mathbb{A}_2, .., \mathbb{A}_n \in \mathbb{R}^{n_1 \times n_2 \times .. \times n_d}$ each of which represents an individual's image. MPCA aims to find $d$ low-rank orthogonal projection matrices $
\mathbb{U}^{(j)} \in \mathbb{R}^{n_j \times \mathbb{R}_j}$ of rank $\mathbb{R}_j \leq n_j$ for all $j \in [d]$  such that the projections of the tensor objects to the lower dimensional subspace $S$ minimize the error:

$$E_{MPCA} (S) = \sum_{j = 1}^{m} \min_{X_j \in S} \| \mathbb{A}_j - \mathbb{X}_j \|^2 = \min_{\mathbb{U}^{(1)}, \mathbb{U}^{(2)},..,\mathbb{U}^{(n)}} \sum_{j = 1}^{m} \|\mathbb{A}_j = \mathbb{A}_j \times U^{(1),T} \times .. \times U^{(d),T} \|^2$$

A subspace minimizing this error can be approximated using Alternating Partial Projections (APP).

We simply fix $d-1$ of the current mode projection matrices before optimizing the single remaining free mode's projection matrix. Optimizing over a single free mode cam be accomplished exactly by computing a partial $SVD$ of a matricized version of the tensor data.

\paragraph{PCA for moving objects tracking on video}

Here video is seen as the image sequence or frame sequence of moving objects and static objects and the goal is to ignore non-moving objects and pick the frames that contains the moving objects. The purpose of PCA here is to produce principal component (PC) images that contain both static and dynamic information on the interested clip. Firstly, the video images normally include three bands (R, G and B), every band in the clip is assembled to form a single-band image set. Then, PCA is performed on every image set and produce a series of Principal Components for every individual band.

The main aim is to extract the PCs that contain the tracking pattern. In a segmented PC image it can usually be observed three main regions: noise region, background region and track region. Noise regions generally have small areas, which commonly are too small to be the track of the interested moving object. On the other hand, background regions generally come with relatively large areas and turn out to be the static information. As a result, the rest of the regions will be grouped together to become the track regions.

In the end of track extraction, we compile the total area of track regions in pixel as the index of each PC image. The PC image with highest index would be the interested PC, which is able to represent the track of moving object.

\newpage
\begin{center}
{\bf Experiments}
\end{center}

\paragraph{PCA for images}

Here we use SVD, which gives us the best k-rank approximation of the image. If we take the SVD of the image I, then $I=U S V$, where $U$ and $V$ are the left and right singular vectors and $S$ is the matrix of singular values.
In application to the single image and varying the number of components, we get:

\begin{center}
\includegraphics[scale=0.4]{image5.png}
\end{center}

\paragraph{PCA for moving objects tracking on video}

The duration of whole test video images is 108 seconds, and two moving
objects with different sizes (one regular car and one motorcycle) are the main targets in the test. The result of interested clip extraction is shown below. The x axis represents the serial number of the frames and the y axis indicates the number of motion pixels. Two significant peaks can be evidently observed in the graphic, there are two periods that contain moving objects are detected.
\begin{center}
\includegraphics[scale=0.3]{image1.jpg}
\end{center}

The interested clips of two periods are respectively shown in the first picture below (regular car and motorcycle). After extracting the interested clips and converting to PC space, a series of corresponding PC image sequences are revealed in the second picture.

\begin{center}
\includegraphics[scale=0.3]{image2.jpg}
\end{center}

\begin{center}
\includegraphics[scale=0.4]{image3.jpg}
\end{center}

\end{document}