% chapter included in vclmanual.tex
\documentclass[vcl_manual.tex]{subfiles}
\begin{document}

\flushleft

\chapter{Boolean operations and per-element branches}\label{chap:BooleanOperations}
Consider this piece of C++ code:

\begin{lstlisting}[frame=none]
int a[4], b[4], c[4], d[4];
  ...
for (int i = 0; i < 4; i++) {
    d[i] = (a[i] > 0 && a[i] < 10) ? b[i] : c[i];
}
\end{lstlisting}
\vspacesmall

We can do this with vectors in the following way:

\begin{lstlisting}[frame=none]
Vec4i a, b, c, d;
  ...
d = select(a > 0 & a < 10, b, c);
\end{lstlisting}
\vspacesmall

The \codei{select} function is similar to the \codei{?:}  operator.
It has three vector parameters: The first parameter is a boolean vector that chooses between the elements of the second and the third vector parameter.
\vspacesmall

The relational operators \codei{\textgreater}, \codei{\textgreater=}, \codei{\textless}, \codei{\textless=}, \codei{==}, \codei{!=} produce boolean vectors,
which accept the boolean operations \codei{\&},
\codei{|}, \codei{$\wedge$}, \codei{$\sim$} (and, or, exclusive or, not).
\vspacesmall

In the above example, the expressions \codei{a \textgreater{} 0} and \codei{a \textless{} 10} are boolean vectors of type \codei{Vec4ib}. The boolean vectors must have a type that matches the data vectors they are used with. Table \ref{table:BooleanVectorClasses} on page \pageref{table:BooleanVectorClasses} shows which boolean vector class to use for each vector type.
\vspacesmall

The vector elements that are not selected are calculated anyway because normally all parts of a vector are calculated. For example:

\begin{lstlisting}[frame=none]
Vec4f a(-1.0f, 0.0f, 1.0f, 2.0f);
Vec4f b = select(a >= 0.0f, sqrt(a), 0.0f);
\end{lstlisting}
\vspacesmall

Here, we will be calculating the square root of -1 even though we are not using it. This will not cause problems if floating point exceptions are masked off, which they normally are. A safe solution that works even if floating point exceptions are enabled would be:

\begin{lstlisting}[frame=none]
Vec4f a(-1.0f, 0.0f, 1.0f, 2.0f);
Vec4f b = sqrt(max(a, 0.0f));
\end{lstlisting}
\vspacesmall

Likewise, the \codei{\&} and \codei{|} operators are calculating both input operands, even if the second operand is not needed. The following examples illustrates this:

\begin{lstlisting}[frame=none]
// array version:
float a[4] = {0.0f, 1.0f, 2.0f, 3.0f};
float b[4];
for (int i = 0; i < 4; i++) {
   if (a[i] > 0.0f && 1.0f/a[i] != 4.0f) {
      b[i] = a[i];
   }
   else {
      b[i] = 1.0f;
   }
}
\end{lstlisting}
\vspacesmall

and the vector version of the same:

\begin{lstlisting}[frame=none]
Vec4f a(0.0f, 1.0f, 2.0f, 3.0f);
Vec4f b = select(a > 0.0f & 1.0f/a != 4.0f, a, 1.0f);
\end{lstlisting}
\vspacesmall

In the array version, we will never divide by zero because the \codei{\&\&} operator does not evaluate the second operand when the first operand is false. But in the vector version, we are indeed dividing by zero because the \codei{\&} operator always evaluates both operands. The vector class library defines the operators \codei{\&\&} and \codei{||} as synonyms for \codei{\&} and \codei{|} for convenience, but they are still doing the bitwise AND or OR operation, so \codei{\&} and \codei{|} are actually more representative of what these operators really do. This example should be changed to:

\begin{lstlisting}[frame=none]
Vec4f a(0.0f, 1.0f, 2.0f, 3.0f);
Vec4f b = select(a > 0.0f & a != 0.25f, a, 1.0f);
\end{lstlisting}
\vspacesmall

\section{Internal representation of boolean vectors}\label{InternalRepresentationOfBoolean}

The way boolean vectors are stored depends on the instruction set and the Vector Class Library (VCL) version.
Older instruction sets have the boolean vectors stored with the same number of bits as the data vectors they are applied to (broad boolean vectors). The later instruction sets AVX512 and AVX512VL allow boolean vectors to be stored with only one bit for each element (compact boolean vectors).
\vspacesmall

Version 1.xx of the VCL is using the broad boolean vectors for the sake of backwards compatibility, while version 2.xx is prioritizing the more efficient compact boolean vectors when the appropriate instruction set is enabled. The boolean vector sizes are summarized in the following table.
\vspacesmall

\label{tableBooleanVectorSizes}
\begin{tabular}{|p{50mm}|p{40mm}|p{40mm}|}
\hline
\bfseries Data vector size \newline and instruction set & \bfseries VCL version 1 \newline Boolean vectors & \bfseries VCL version 2 \newline Boolean vectors \\ \hline
128 bits & broad & broad  \\ \hline
128 bits with AVX512VL & broad & compact \\ \hline
256 bits & broad & broad  \\ \hline
256 bits with AVX512VL & broad & compact \\ \hline
512 bits & broad & broad  \\ \hline
512 bits with AVX512F & compact & compact \\ \hline
\end{tabular}
\vspacebig

The broad boolean vectors are stored as integer vectors with the same number of bits per element as the integer or floating point vectors they are used for. For example, the broad boolean vector class \codei{Vec4fb} is stored as a vector of four 32-bit integers because it is used with vectors \codei{Vec4f} of four single precision floating point numbers, using 32 bits each. The broad boolean vector class \codei{Vec4db} is stored as a vector of four 64-bit integers because it is used with vectors \codei{Vec4d} of four double precision floating point numbers, using 64 bits each. Note that the integer representation of true in a broad boolean vector element is not 1, but  -1. The representation of false is 0. Any other values than 0 and -1 in broad boolean vectors will produce wrong and inconsistent results that depend on the instruction set.
\vspacesmall

The compact boolean vectors are stored with one bit per element (at least 8 bits).
You should make no assumption about how boolean vectors are stored if your code may be compiled for different instruction sets or different versions of VCL. For example,
\codei{Vec16ib} uses 16 bits of storage when compiling for AVX512, but 512 bits of storage when compiling for AVX2. Do not store boolean vectors directly to binary files, and do not transmit boolean vectors between different functions that may be compiled for different instruction sets or different VCL versions.
\vspacesmall

Different compact boolean vectors are mutually compatible if they have the same number of elements. Different broad boolean vectors are mutually compatible if they have the same number of elements and the same number of bits. Broad and compact boolean vectors are not compatible with each other. See page \pageref{ConversionBetweenBooleanTypes} for conversion between different types of boolean vectors.
\vspacesmall

\section{Functions for use with booleans}\label{FunctionsForBooleans}

\vspacesmall
\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & vector select(boolean vector s, vector a, vector b) \\ \hline
\bfseries Defined for & all integer and floating point vector classes \\ \hline
\bfseries Description & branch per element.\newline
result[i] = s[i] ? a[i] : b[i] \\ \hline
\bfseries Efficiency & good \\ \hline
\end{tabular}
\begin{lstlisting}[frame=none]
// Example:
Vec4i a(-1, 0, 1, 2);
Vec4i b = select(a>0, a+10, a-10); // b = (-11,-10,11,12)
\end{lstlisting}
\vspacesmall

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & if\_add(boolean vector f, vector a, vector b) \\ \hline
\bfseries Defined for & all integer and floating point vector classes \\ \hline
\bfseries Description & conditional addition \newline
result[i] = f[i] ? (a[i] + b[i]) : a[i] \\ \hline
\bfseries Efficiency & good \\ \hline
\end{tabular}
\begin{lstlisting}[frame=none]
// Example:
Vec4i a(-1, 0, 1, 2);
Vec4i b = if_add(a < 0, a, 100);  // b = (99,0,1,2)
\end{lstlisting}
\vspacesmall

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & if\_sub(boolean vector f, vector a, vector b) \\ \hline
\bfseries Defined for & all integer and floating point vector classes \\ \hline
\bfseries Description & conditional subtraction \newline
result[i] = f[i] ? (a[i] - b[i]) : a[i] \\ \hline
\bfseries Efficiency & good \\ \hline
\end{tabular}
\vspacebig

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & vector if\_mul(boolean vector f, vector a, vector b) \\ \hline
\bfseries Defined for & all integer and floating point vector classes \\ \hline
\bfseries Description & conditional multiplication\newline
result[i] = f[i] ? (a[i] * b[i]) : a[i] \\ \hline
\bfseries Efficiency & good \\ \hline
\end{tabular}
\vspacebig

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & vector if\_div(boolean vector f, vector a, vector b) \\ \hline
\bfseries Defined for & all floating point vector classes \\ \hline
\bfseries Description & conditional division\newline
result[i] = f[i] ? (a[i] / b[i]) : a[i] \\ \hline
\bfseries Efficiency & medium \\ \hline
\end{tabular}
\vspacebig

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & vector andnot(vector, vector) \\ \hline
\bfseries Defined for & all boolean vector classes \\ \hline
\bfseries Description & andnot(a,b) = a \& $\sim$ b \\ \hline
\bfseries Efficiency & good \\ \hline
\end{tabular}
\vspacebig

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & bool horizontal\_and(boolean vector) \\ \hline
\bfseries Defined for & all boolean vector classes \\ \hline
\bfseries Description & The output is the AND combination of all elements \\ \hline
\bfseries Efficiency & Medium for broad boolean vectors. Better if SSE4.1 or later. Good for compact boolean vectors \\ \hline
\end{tabular}
\begin{lstlisting}[frame=none]
// Example:
Vec4i a(-1, 0, 1, 2);
bool  b = horizontal_and(a > 0);  // b = false
\end{lstlisting}
\vspacesmall

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & bool horizontal\_or(boolean vector) \\ \hline
\bfseries Defined for & all boolean vector classes \\ \hline
\bfseries Description & The output is the OR combination of all elements \\ \hline
\bfseries Efficiency & Medium for broad boolean vectors. Better if SSE4.1 or later. Good for compact boolean vectors \\ \hline
\end{tabular}
\begin{lstlisting}[frame=none]
// Example:
Vec4i a(-1, 0, 1, 2);
bool  b = horizontal_or(a > 0);  // b = true
\end{lstlisting}
\vspacesmall

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & int horizontal\_find\_first(boolean vector) \\ \hline
\bfseries Defined for & all boolean vector classes \\ \hline
\bfseries Description & Returns an index to the first element that is true.
Returns -1 if all elements are false \\ \hline
\bfseries Efficiency & medium \\ \hline
\end{tabular}
\begin{lstlisting}[frame=none]
// Example:
Vec4i  a(1, 2, 3, 4);
Vec4i  b(0, 2, 3, 5);
int c = horizontal_find_first(a == b);  // c = 1
\end{lstlisting}
\vspacesmall

\begin{tabular}{|p{30mm}|p{120mm}|}
\hline
\bfseries Function & unsigned int horizontal\_count(boolean vector) \\ \hline
\bfseries Defined for & all boolean vector classes \\ \hline
\bfseries Description & counts the number of elements that are true \\ \hline
\bfseries Efficiency & medium if SSE4.2 or later \\ \hline
\end{tabular}
\begin{lstlisting}[frame=none]
// Example:
Vec4i  a(1, 2, 3, 4);
Vec4i  b(0, 2, 3, 5);
int c = horizontal_count(a == b);  // c = 2
\end{lstlisting}
\vspacesmall

\end{document}


\documentclass[12]{scrartcl}
\usepackage{amssymb,amsmath,gensymb,dsfont,calc,multicol,fullpage}
\makeatletter
\newcommand\Aboxed[1]{
   \@Aboxed#1\ENDDNE}
\def\@Aboxed#1&#2\ENDDNE{%
   &
   \settowidth\@tempdima{$\displaystyle#1{}$}
   \setlength\@tempdima{\@tempdima+\fboxsep+\fboxrule}
   \kern-\@tempdima
   \boxed{#1#2}
}
\makeatother

\begin{document}

\title{Homework 7, Section 1.7: 6, 10, 12, 18, 21, 29, 36, 37}
\author{Alex Gordon}
\date{\today}
\maketitle
\section*{Homework}
\subsection*{6.}
$\begin{bmatrix} -4&-3&0&0&0 \\ 0&-1&5&0 \\ 0&0&-15&0 \\ 0&0&0&0  \end{bmatrix}$\\

As we can see the three basic variables give the trivial solution and no free variable. Therefore the columns of A are linearly independent.
\subsection*{10. A)}
$v_3$ in span $\{v_1, v_2\}$\\
The augmented matrix is:\\
$\begin{bmatrix} 1&-3&2 \\ -3&9&-5 \\ -5&15&h  \end{bmatrix}$\\
The reduced matrix is:\\
$\begin{bmatrix} 1&-3&2 \\ 0&0&1 \\ 0&0&10+h  \end{bmatrix}$\\
As we can see, $0 = 1$ is not possible, therefore the system has no solution to this equation.
\subsection*{10. B)}
The reduced matrix is:\\
$\begin{bmatrix} 1&-3&2&0 \\0&0&1&0 \\ 0&0&10+h&0  \end{bmatrix}$\\
If the system has a non trivial solution then $\{v_1, v_2, v_3\}$ is linearly dependent. For non-trivial solutions, $10+h = 0$ means that $h = -10$. $x_2$ is a free variable, which means we have a non-trivial solution.
\subsection*{12.}
The system is linearly dependent. Therefore the system has a non-trivial solution and $h = -18$
\subsection*{18.}
Linearly dependent.
\subsection*{21. A)}
Since the homogeneous system $Ax = 0$ always has the trivial solution, that means that regardless of A, to possess linearly independent columns, $Ax = 0$ always has a trivial solution. This means the statement is not true.
\subsection*{21. B)}
If we consider that linear combinations of all vectors then set $S = 0$, then there exists at least one nonzero scalar to satisfy the equation. \\
Since that means every vector in the set $S$ can be left on one side of the equation and all other vectors are on the other side that shows that S is a linear combination of the other vectors, meaning the statement is true. \\
\subsection*{21. C)}
The columns of any 4x5 matrix are linearly dependent. The statement is true.
\subsection*{21. D)}
If x and y are linearly independent then z is in the span of $\{x,y\}$. \\
Therefore any one of the conditions are true (they can't be true simultaneously)
\subsection*{36.}
By theorem 7, it is true.
\subsection*{37.}
The given set of vectors $\{v_1, v_2, v_3\}$ are linearly dependent and so the set of vectors $\{v_1, v_2, v_3, v_4\}$ are linearly dependent, hence the statement is true.

\end{document}


\chapter{Assignment: \protect\\ Clustering vs. Classification}
\label{hw:clustering-classification}

\newthought{Clustering and classification are fundamentally different tasks.} The former tries to find similar data instances and puts them into groups or clusters. The latter looks for the patterns in the data and infers correlations between the features and the target variable. Clustering is a part of unsupervised learning and doesn't require a target variable. Classification is a part of supervised learning and requires a target variable. A target variable is the thing we want to predict. With clustering, we are not predicting anything, but trying to find groups of similar data instances.

\begin{figure}[h]
  \centering
  \includegraphics[scale=0.7]{workflow.png}%
  \caption{$\;$}
  \label{fig:wf1}
\end{figure}

For this task, use \textit{heart-disease} data and build a simple hierarchical clustering workflow (with Euclidean distance and Ward linkage) and a logistic regression model and inspect it in a nomogram. In clustering, cut at two clusters. Refer to the workflow above and answer the following questions:

\begin{enumerate}
    \item Explore the groups with \widget{Box Plot}. What is the characteristic of cluster 1 (C1)? What is the characteristic of cluster 2 (C2)?
    \item Which are the top three attributes distinguishing between clusters? Write them down.
    \item In \widget{Nomogram}, which are the three most important attributes for the model?
    \item Are the attributes from clustering the same as those from logistic regression? Why (not)?
    \item \textit{Major vessels colored} is the most important attribute for logistic regression. How well does it split between the clusters?
\end{enumerate}


\chapter{Experimental Concepts}
This chapter serves as a reminder of commonly used ideas in high-energy physics (HEP) experiments.

\section{Experimental possibilities}
In reality there are rather few experimental possibilities for HEP experiments:
\begin{itemize}
\item Scatter one particle off another and observe the reaction;
\item Generate a particle in a reaction and observe its decay;
\item Detect neutrinos and observe neutrino oscillations;
\item Measure a particular particle's properties such as mass, charge, spin, parity, lifetime.
\end{itemize}

\section{Cross section}
The cross section, $\sigma$, for a particular reaction is proportional to the probability for the interaction to take place. In HEP experiments, it is expressed in the unit barns, where $\SI{1}{\barn} = \SI{1e-28}{\meter^2}$.

\subsection{Beam incident on a target}
Assume the beam is comprised of bunches with $N_B$ particles per bunch. The beam is incident on a target with area $A$, length $l$ inside the luminous region, and mass density $\rho$. Then the number of target particles seen by the beam is
\begin{equation}
N_T = \frac{Al\rho N_A}{m}
\end{equation}
where $N_A$ is Avagadro's number and $m$ is the molecular mass, such that $N_A/m$ is the mass of each target particle.

Now we can define the cross section as
\begin{align}
P(\text{interaction}) &= (\text{Number of target particles per unit area}) \times \sigma \\
&= \frac{Al\rho N_A}{m} \times \frac{\sigma}{A} \nonumber \\
&= \frac{l\rho N_A \sigma}{m}.
\end{align}
Therefore the total number of interactions per bunch is given by
\begin{equation}
N_I = \frac{l\rho N_A N_B \sigma}{m}.
\end{equation}
Now take the target and bunch to contain number densities $n_B$ and $n_T$ of particles, respectively. They have relative speed $u \simeq c$. Then the number of target particles in the luminous region is $n_T V$, so the probability of interaction may now be written
\begin{equation}
P(\text{interaction}) = \frac{n_T V \sigma}{A}.
\end{equation}
There are $n_B u A$ beam particles passing through the luminous region per second, so the rate of interactions is
\begin{equation}
\frac{\dd N_I}{\dd t} = n_B n_T V u \sigma.
\end{equation}

\subsection{Beam-beam collision}
For a circular collider, assume a rotation frequency $f$ for both counter-circulating beams consisting of $n$ bunches, each with $N_B$ particles and area $A$. Then the rate of interactions is
\begin{align}
\frac{\dd N_I}{\dd t} &= P(\text{collision}) \times (\text{particles per second in one beam}) \\
&= \frac{N_B \sigma}{A} \times n f N_B \nonumber \\
&= \frac{n f N_B^2 \sigma}{A} \label{eq:collRate}.
\end{align}

\section{Luminosity}
Using the result from \eqref{eq:collRate}, we define the instantaneous luminosity
\begin{equation}\boxed{
L \equiv \frac{1}{\sigma} \frac{\dd N_I}{\dd t}
}\end{equation}
hence for beam-beam collisions
\begin{equation}
L = \frac{n f N_B^2}{A}.
\end{equation}

In an effort to increase the luminosity, focussing magnets near to the collision regions are used to decrease the bunch area. Also, the number of particles per bunch should be made as large as possible while also maintaining a stable bunch.

Spontaneous luminosity tends to decrease with run time due to collision remnants decreasing the vacuum in the beam pipe and consequently deforming the bunches. An LHC beam has a typical lifetime of around 10--15 hours (the lifetime is the time for $L$ to decay by a factor $e$). At this point the beam is dumped and a new beam is accelerated and injected into the storage ring ready for collisions.

The units of $L$ are typically \si{\per \pico\barn \per \second}. To get a measure of the total number of interactions observed, and hence the amount of data collected, $L$ may be integrated to give the integrated luminosity,
\begin{equation}\boxed{
L_\text{int} = \int L \, \dd t
}
\end{equation}
which often has the units of inverse femtobarns, \si{\per \femto \barn}.

\section{Natural units and conversion factors}
In natural units, we have $\hbar = c = 1$. $c$ is the conversion factor between space and time or mass and energy, and $\hbar$ is the conversion between energy and time. Consider the units of their product:
\begin{align}
[\hbar c] &= [ET][LT^{-1}] \nonumber \\
&= [E][L].
\end{align}
Now we have $c = \SI{3e8}{\meter\per\second}$ and $\hbar = \SI{1.05e-34}{\joule\second} = \SI{6.56e-22}{\mega\electronvolt\second}$, so
\begin{equation}\boxed{
\hbar c = \SI{197}{\mega \electronvolt \femto \meter} \equiv 1}.
\end{equation}
Therefore, we have
\begin{equation}
\SI{1}{\giga\electronvolt} = \frac{1000}{\SI{197}{\femto\meter}} = \SI{5.08e15}{\per\meter} = \SI{1.52e24}{\per\second}
\end{equation}
\begin{equation}
\Rightarrow\quad \frac{1}{\si{\giga\electronvolt^2}} = \SI{3.88e-32}{\meter^2} = \SI{0.388}{\milli\barn}.
\end{equation}


% \newappendix{Algorithmic details for estimating surplus description length} \label{sec:sdl_details}

% Recall that the SDL is defined as
% \begin{align}
%         m_{\mathrm{SDL}}(\phi, \mathcal{D}, \mathcal{A},\eps) &= \sum_{n=1}^\infty \Big[ L(\mathcal{A}_\phi, n) - \eps \Big]_+
% \end{align}
% For simplicity, we assume that $L$ is bounded in $[0,1]$. Note that this can be achieved by truncating the cross-entropy loss.

% \begin{algorithm}
% \caption{Estimate surplus error}
% \label{alg:sdl}
% \KwIn{tolerance $\eps $, max iterations $M$, number of datasets $K$, representation $ \phi$, data distribution $\mathcal{D}$, algorithm $\mathcal{A}$ }
% \KwOut{Estimate $\hat m $ of  $m(\phi, \mathcal{D}, \eps, \mathcal{A})$ and indicator $ I$ of whether this estimate is tight or lower bound}
% \vspace{1mm} \hrule \vspace{1mm}
% Sample $ K$ datasets $ D_M^{k}\sim \mathcal{D}$ of size $ M+1$\\
% \For{$n = 1$ \KwTo $M$}{
%     For each $ k \in [K]$, run $ \mathcal{A}$ on $ D_M^{k}[1:n]$ to produce a predictor $ \hat p_n^k$\\
%     Take $ K $ test samples $ (x_k, y_k) = D_M^k[M+1]$\\
%     Evaluate $ \hat L_n = \frac{1}{K}\sum_{k=1}^K \ell(\hat p_n^k, x_k , y_k) $
%     }
% Set $ \hat m = \sum_{n=1}^M [\hat L_n - \eps]_+$ \vspace{1mm}\\
% \lIf {$ \hat L_M \leq \eps/2$} {Set $I = $ \texttt{tight} \textbf{else} {Set $ I = $ \texttt{lower bound}}}
% \Return $\hat m, I$
% \end{algorithm}

% In our experiments we replace $D^k_M[1:n]$ of Algorithm \ref{alg:sdl} with sampled subsets of size $n$ from a single evaluation dataset.
% Additionally, we use between 10 and 20 values of $n$ instead of evaluating $L(\mathcal{A}_\phi, n)$ at every integer between $1$ and $M$.
% This strategy, also used by \citet{Blier2018TheDL} and \citet{Voita2020InformationTheoreticPW}, corresponds to the description length under a code which updates only periodically during transmission of the data instead of after every single point.

% \begin{theorem}
% Let the loss function $L$ be bounded in $[0,1]$ and assume that it is decreasing in $ n$. With $ (M+1)K $ datapoints, if the sample complexity is less than $ M$, the above algorithm returns an estimate $ \hat m$ such that with probability at least $ 1- \delta$
% \begin{align}
%     |\hat m - m(\phi, \mathcal{D}, \eps, \mathcal{A})| \leq  M\sqrt{ \frac{\log (2M/\delta)}{2K}}.
% \end{align}
% If $ K \geq \frac{\log(1/\delta)}{2\eps^2}$ and the algorithm returns \texttt{tight} then with probability at least $ 1-\delta$ the sample complexity is less than $ M $ and the above bound holds.
% \end{theorem}
% \begin{proof}
% First we apply a Hoeffding bound to show that each $ \hat L_n$ is estimated well. For any $ n$, we have
% \begin{align}
%     P \bigg( \big|\hat L_n   - L(\mathcal{A}_\phi,n)  \big| > \sqrt{\frac{\log(2M/\delta)}{2K}} \bigg) \leq 2 \exp\bigg(-2K  \frac{\log(2M/\delta)}{2K}\bigg) = 2 \frac{\delta}{2M} = \frac{\delta}{M}
% \end{align}
% since each $ \ell(\hat p_n^k, x_k , y_k)$ is an independent variable, bounded in [0,1] with expectation $ L(\mathcal{A}_\phi, n)$.

% Now when sample complexity is less than $ M$, we use a union bound to translate this to a high probability bound on error of $ \hat m$, so that with probability at least $ 1- \delta$:
% \begin{align}
%     |\hat m - m(\phi, \mathcal{D}, \eps, \mathcal{A})| &= \bigg|\sum_{n=1}^M [\hat L_n - \eps]_+  - [L(\mathcal{A}_\phi,n) - \eps]_+  \bigg|\\
%     &\leq \sum_{n=1}^M\bigg| [\hat L_n - \eps]_+  - [L(\mathcal{A}_\phi,n) - \eps]_+ \bigg|\\
%     &\leq \sum_{n=1}^M \bigg|\hat L_n - L(\mathcal{A}_\phi,n) \bigg|\\
%     &\leq M \sqrt{ \frac{\log (2M/\delta)}{2K}}
% \end{align}
% This gives us the first part of the claim.

% We want to know that when the algorithm returns \texttt{tight}, the estimate can be trusted (i.e. that we set $ M $ large enough). Under the assumption of large enough $K$, and by an application of Hoeffding, we have that
% \begin{align}
%     P \bigg(  L(\mathcal{A}_\phi,M) - \hat L_M  > \eps/2 \bigg) \leq  \exp\bigg(-2K \eps^2 \bigg) \leq  \exp\bigg(-2 \frac{\log(1/\delta)}{2\eps^2} \eps^2 \bigg) = \delta
% \end{align}
% If $ \hat L_M \leq \eps/2$, this means that $ L(\mathcal{A}_\phi,M) \leq \eps$ with probability at least $ 1-\delta$. By the assumption of decreasing loss, this means the sample complexity is less than $ M$, so the bound on the error of $ \hat m$ holds.
% \end{proof}

% \newappendix{Algorithmic details for estimating sample complexity} \label{sec:sc_details}
% Recall that $\eps$ sample complexity ($\eps$SC) is defined as
% \begin{align}
%      m_{\eps\mathrm{SC}}(\phi, \mathcal{D}, \mathcal{A},\eps) &= \min \Big\{ n \in \mathbb{N} : L(\mathcal{A}_\phi, n) \leq \eps \Big\}.
% \end{align}

% We estimate $m_{\eps\mathrm{SC}}$ via recursive grid search. To be more precise, we first define a search interval $[1,N]$, where $N$ is a large enough number such that $L(\mathcal{A}_\phi,N) \ll \eps$. Then, we partition the search interval in to 10 sub-intervals and estimate risk of hypothesis learned from $D^n \sim \mathcal{D}^n$ with high confidence for each sub-interval. We then find the leftmost sub-interval that potentially contains $m_{\eps\mathrm{SC}}$ and proceed recursively. This procedure is formalized in Algorithm~\ref{alg:esc} and its guarantee is given by Theorem~\ref{thm:esc}.
% \begin{algorithm}[h!]
% \caption{Estimate sample complexity via recursive grid search}
% \label{alg:esc}
% \KwIn{Search upper limit $N$, parameters $\eps$, confidence parameter $\delta$, data distribution $\mathcal{D}$, and learning algorithm $\mathcal{A}$.}
% \KwOut{Estimate $\hat{m}$ such that $m_{\eps\mathrm{SC}}(\phi,\mathcal{D},\mathcal{A},\eps) \le \hat m$ with probability $1-\delta$.}
% \vspace{1mm} \hrule \vspace{1mm}
% let $S = 2\log (20k/\delta)/\eps^2$, and let $[\ell,u]$ be the search interval initialized at $\ell = 1, u = N$.\\
% \For{$r=1$ \KwTo $k$}{
%     Partition $[\ell,u]$ into 10 equispaced bins and let $\Delta$ be the length of each bin. \\
%     \For{$j = 1$ \KwTo $10$}{
%         Set $n = \ell + j \Delta$. \\
%         Compute $\hat L_n = \frac{1}{S}\sum_{i=1}^S \ell(\mathcal{A}(D^n_i),x_i,y_i)$ for $S$ independent draws of $D^n$ and test sample $(x,y)$. \\
%         \If{$\hat L_n \le \eps/2$}{
%         Set $u = n$ and $\ell = n - \Delta$. \\
%         \textbf{break}
%         }
%         }
% }
% \Return $\hat m = u$, which satisfies $m_{\eps\mathrm{SC}}(\phi,\mathcal{D},\mathcal{A},\eps) \le \hat m$ with probability $1-\delta$, where the randomness is over independent draws of $D^n$ and test samples $(x,y)$.
% \end{algorithm}

% \begin{theorem}
% \label{thm:esc}
% Let the loss function $L$ be bounded in $[0,1]$ and assume that it is decreasing in $ n$. Then, Algorithm~\ref{alg:esc} returns an estimate $ \hat m$ that satisfies $m_{\eps\mathrm{SC}}(\phi,\mathcal{D},\mathcal{A},\eps) \le \hat m$ with probability at least $ 1- \delta$.
% \end{theorem}

% \begin{proof}
% By Hoeffding, the probability that $|\hat L_n-L(\mathcal{A}_{\phi},n)| \ge \eps/2$, where $\hat L$ is computed with $S = 2\log(20k/\delta)/\eps^2$ independent draws of $D^n \sim \mathcal{D}^n$ and $(x,y) \sim \mathcal{D}$, is less than $\delta/(10k)$. The algorithm terminates after evaluating $\hat L$ on at most $10k$ different $n$'s. By a union bound, the probability that $|\hat L_n - L(\mathcal{A}_{\phi},n)| \le \eps/2$ for all $n$ used by the algorithm is at least $1-\delta$. Hence, $\hat L_n \le \eps/2$ implies $L(\mathcal{A}_\phi,n) \le \eps$ with probability at least $1-\delta$.
% \end{proof}

\newappendix{Experimental details} \label{sec:experiment_details}

In each experiment we first estimate the loss-data curve using a fixed number of dataset sizes $n$ and multiple random seeds, then compute each measure from that curve.
Reported values of SDL correspond to the estimated area between the loss-data curve and the line $y=\eps$ using Riemann sums with the values taken from the left edge of the interval.
This is the same as the chunking procedure of \citet{Voita2020InformationTheoreticPW} and is equivalent to the code length of transmitting each chunk of data using a fixed model and switching models between intervals.
Reported values of $\eps$SC correspond to the first measured $n$ at which the loss is less than $\eps$.

All of the experiments were performed on a single server with 4 NVidia Titan X GPUs, and on this hardware no experiment took longer than an hour.
All of the code for our experiments, as well as that used to generate our plots and tables, is included in the supplement.

\subsection{MNIST experiments}

For our experiments on MNIST, we implement a highly-performant vectorized library in \hyperlink{https://jax.readthedocs.io/en/latest/}{JAX} to construct loss-data curves.
With this implementation it takes about one minute to estimate the loss-data curve with one sample at each of 20 settings of $n$.
We approximate the loss-data curves at 20 settings of $n$ log-uniformly spaced on the interval $[10, 50000]$ and evaluate loss on the test set to approximate the population loss.
At each dataset size $n$ we perform the same number of updates to the model; we experimented with early stopping for smaller $n$ but found that it made no difference on this dataset.
In order to obtain lower-variance estimates of the expected risk at each $n$, we run 8 random seeds for each representation at each dataset size, where each random seed corresponds to a random initialization of the probe network and a random subsample of the evaluation dataset.

Probes consist of two-hidden-layer MLPs with hidden dimension 512 and ReLU activations.
All probes and representations are trained with the Adam optimizer \citep{Kingma2015AdamAM} with learning rate $10^{-4}$.

Each representation is normalized to have zero mean and unit variance before probing to ensure that differences in scaling and centering do not disrupt learning.
The representations of the data we evaluate are implemented as follows.

\paragraph{Raw pixels.}
The raw MNIST pixels are provided by the Pytorch \texttt{datasets} library \citep{Paszke2019PyTorchAI}.
It has dimension $28 \times 28 = 784$.

\paragraph{CIFAR.}
The CIFAR representation is given by the last hidden layer of a convolutional neural network trained on the CIFAR-10 dataset.
This representation has dimension 784 to match the size of the raw pixels.
The network architecture is as follows:

\begin{verbatim}
    nn.Conv2d(1, 32, 3, 1),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(32, 64, 3, 1),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Flatten(),
    nn.Linear(1600, 784)
    nn.ReLU()
    nn.Linear(784, 10)
    nn.LogSoftmax()
\end{verbatim}

\paragraph{VAE.}
The VAE (variational autoencoder; \citet{Kingma2014AutoEncodingVB,Rezende2014StochasticBA}) representation is given by a variational autoencoder trained to generate the MNIST digits.
This VAE's latent variable has dimension 8.
We use the mean output of the encoder as the representation of the data.
The network architecture is as follows:
\begin{verbatim}
self.encoder_layers = nn.Sequential(
    nn.Linear(784, 400),
    nn.ReLU(),
    nn.Linear(400, 400),
    nn.ReLU(),
    nn.Linear(400, 400),
    nn.ReLU(),
)
self.mean = nn.Linear(400, 8)
self.variance = nn.Linear(400, 8)

self.decoder_layers = nn.Sequential(
    nn.Linear(8, 400),
    nn.ReLU(),
    nn.Linear(400, 400),
    nn.ReLU(),
    nn.Linear(400, 784),
)
\end{verbatim}

\subsection{Part of speech experiments}

We follow the methodology and use the official code\endnote{Code available at \url{https://github.com/lena-voita/description-length-probing}.} of \citet{Voita2020InformationTheoreticPW} for our part of speech experiments using ELMo \citep{Peters2018DeepCW} pretrained representations.
In order to obtain lower-variance estimates of the expected risk at each $n$, we run 4 random seeds for each representation at each dataset size, where each random seed corresponds to a random initialization of the probe network and a random subsample of the evaluation dataset.
We approximate the loss-data curves at 10 settings of $n$ log-uniformly spaced on the range of the available data $n \in [10, 10^6]$.
To more precisely estimate $\eps$SC, we perform one recursive grid search step: we space 10 settings over the range which in the first round saw $L(\mathcal{A}_\phi, n)$ transition from above to below $\eps$.

Probes consist of the MLP-2 model of \citet{Hewitt2019DesigningProbes,Voita2020InformationTheoreticPW} and all training parameters are the same as in those works.


\begin{appendix}
\section{Examples}

\subsection{Leech word square-freeness}
Let us consider the SLP $\slp{L}$ that derives substring of Leech square free word:

\begin{center}
$\slpterm{L}{1}{a}, \slpterm{L}{2}{b}, \slpterm{L}{3}{c}, \slpnonterm{L}{4}{2}{3}, \slpnonterm{L}{5}{2}{1},
\slpnonterm{L}{6}{1}{4}$

$\slpnonterm{L}{7}{3}{4}, \slpnonterm{L}{8}{6}{5}, \slpnonterm{L}{9}{8}{4}, \slpnonterm{L}{10}{8}{7},
\slpnonterm{L}{11}{10}{9}$
\end{center}

Parse tree for $\slp{L}$ presented at figure 4.

\begin{figure}[th]
\LeechSLP
\end{figure}

Let us check square freeness of $\slp{L}$. Since $|L| = 15$ then the algorithm check the following segments of root
length: [1, 1], [2, 3], [4, 7]. Notice that rules of $\slp{L}$ already ordered by length of derived text.

\begin{itemize}
  \item \textbf{checking square freeness of squares that root length equals to 1}

  Firstly the algorithm skip terminal rules. Next it consecutively look for squares exactly in other rules. For
  instance, let us consider how it checks $\slpnonterm{L}{10}{8}{7}$. The algorithm find $L_8[5] = a$ and $L_7[1] = c$
  using $\slp{L}_{10}$. Since $a \neq c$ it moves to $\slp{L}_{11}$.

  It is clear that $\slp{L}_4, \dots, \slp{L}_{11}$ have no squares of length 2 around its cut positions.
  \item \textbf{checking square freeness of squares that root length belong to [2, 3]}

  The algorithm skip all rules that have length less 4 (i.e. $\slp{L}_1, \dots, \slp{L}_7$). For instance, let us
  consider how it checks $\slpnonterm{L}{8}{6}{5}$. Since $|L_8| = 5$ and block length is equal to 1 then the algorithm
  construct an SLP $\slp{L}_8'$ that derives text \emph{\$~\$~\$~\$~\$~a~b~c~b~a~\$~\$~\$~\$~\$~\$}, where \emph{\$} is
  special symbol not from $\Sigma$. At figure 5 presented partition of $L_8'$ into blocks of length 1.

  \LeechWordPartitionSimple

  Remind that the algorithm necessary to check blocks $B_5, \dots, B_{12}$. It no need to check $B_5$ since it contains
  \emph{\$}. Also it no need to check $B_9$ and $B_{10}$ since the search area is out of $L_8$. For instance, let us
  consider how it checks $B_7$. The algorithm build $\slp{L}_8'\substr{6}{7}$ that derives $B_7$ and
  $\slp{L}_8'\substr{8}{11}$ that derives search area $B_9 \cdot B_{10}$. Next it runs pattern matching algorithm on
  $\slp{L}_8'\substr{6}{7}$, $\slp{L}_8'\substr{8}{11}$ and obtain occurrence of $B_7$ at position 9 of $L_8'$. Finally
  it runs \textbf{SubsExt} problem with the following parameters: $\slp{L}_8'$ and positions 6, 7, 8, 9. The algorithm
  obtains $\ell_{ex} = r_{ex} = 0$. Since $\ell_{ex} + r_{ex} = 0 = a - (k - 1) \cdot 2^{i-2} = 8 - (9 - 1) \cdot 1$ then
  algorithm moves to $L_9$.

  \item \textbf{checking square freeness of squares that root length belongs to [4, 7]}

  The algorithm skip all rules that have length less 8 (i.e. $\slp{L}_1, \dots, \slp{L}_9$). For instance, let us
  consider how it checks $\slpnonterm{L}{11}{10}{9}$. Firstly it construct an SLP $\slp{L}_{11}'$ that derives $L$
  surrounded with \emph{\$} and $|L_{11}'| = 32$. At figure 6 presented partition of $L_{11}'$ into blocks of length
  2.

  \LeechWordPartitionComplex

  Next the algorithm find no squares for $B_5, \dots, B_8$ since pattern matching algorithm returns empty set of
  results. Next the algorithm find occurrence of $B_9$ at position 22 of $L_{11}'$ and run \textbf{SubsExt}
  problem with the following parameters: $\slp{L}_{11}'$ and positions 16, 18, 20, 22. Since $\ell_{ex} = r_{ex} = 0$ the
  algorithm moves to $B_{10}$. Finally the algorithm find no squares for $B_{10}, B_{11}, B_{12}$ since correspond
  search areas contains \emph{\$}.
\end{itemize}

\subsection{Construction PS-table for $\slp{F}_7$}

Let us consider how the algorithm construct PS-table for $\slp{F}_7$ that derives text \emph{a~b~a~a~b~a~b~a~a~b~a~a~b}.
The PS-table size is equal to $(\lfloor\log |F_7|\rfloor+1)\times (|\slp{F}_7|+1) = 4 \times 8$.

\begin{itemize}
  \item \textbf{looking for squares that root length equals to 1}

  Firstly the algorithm mark cells with $\varnothing$ for rules with length less than 2. Next it consecutively look for
  squares exactly in other rules. PS(1, 3) = $\varnothing$ since $F_1[1] = a \neq b = F_2[1]$. Analogously PS(1, 4) =
  PS(1, 6) = $\varnothing$. PS(1, 5) = \{1, 3, 3\} since $F_4[3] = a = F_3[1]$. PS(1, 7) = \{1, 8, 8\} since $F_6[8] = a
  = F_5[1]$. After first step PS-table has the following view:

  \begin{figure}[h]
	  {\footnotesize\noindent
		\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline
	 	& $\slpterm{F}{1}{a}$ & $\slpterm{F}{2}{b}$ & $\slpnonterm{F}{3}{1}{2}$ & $\slpnonterm{F}{4}{3}{1}$ &
	 	$\slpnonterm{F}{5}{4}{3}$ & $\slpnonterm{F}{6}{5}{4}$ & $\slpnonterm{F}{7}{6}{5}$ \\ \hline

	 	[1, 1] & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & \{1, 3, 3\} & $\varnothing$ & \{1, 8, 8\} \\
	 	\hline

	 	[2, 3] & & & & & & & \\ \hline

	 	[4, 7] & & & & & & & \\ \hline
		\end{tabular}
	  }
  \end{figure}

  \item \textbf{looking for squares that root length belongs to [2, 3]}

  The algorithm mark cells with $\varnothing$ for rules with length less than 4. For instance, let us consider
  how the algorithm fill PS(2, 6). The algorithm construct SLP $\slp{F}_6'$ that derives text
  \emph{\$~\$~\$~a~b~a~a~b~a~b~a~\$~\$~\$~\$~\$} of length 16. At figure 7 presented partition of $F_6'$ into blocks.

  \FibonacciWordPartition

  $B_5$: using \textbf{PM} problem the algorithm obtain occurrence of $B_5$ at position 8 of $F_6'$ on
  $\slp{F}_{6}'\substr{5}{6}, \slp{F}_{6}'\substr{7}{9}$; the substring extending algorithm obtain $\ell_{ex} = r_{ex} = 1$
  on $\slp{F}_{6}'$ with parameters 5, 6, 8, 9; so family of repetitions \{3, 7, 7\} was found;

  $B_6$: using \textbf{PM} problem the algorithm obtain occurrence of $B_6$ at position 9 of $F_6'$ on
  $\slp{F}_{6}'\substr{6}{7}, \slp{F}_{6}'\substr{8}{10}$; the substring extending algorithm obtain $\ell_{ex} = 2, r_{ex}
  = 0$ on $\slp{F}_{6}'$ with parameters 6, 7, 9, 10; so family of repetitions \{3, 7, 7\} was found;

  $B_7$: using \textbf{PM} problem the algorithm obtain occurrence of $B_7$ at position 9 of $F_6'$ on
  $\slp{F}_{6}'\substr{7}{8}, \slp{F}_{6}'\substr{9}{11}$; the substring extending algorithm obtain $\ell_{ex} = 0, r_{ex}
  = 1$ on $\slp{F}_{6}'$ with parameters 7, 8, 9, 10; so no family of repetitions was found;

  $B_8$: using \textbf{PM} problem the algorithm obtain occurrence of $B_8$ at position 10 of $F_6'$ on
  $\slp{F}_{6}'\substr{8}{9}, \slp{F}_{6}'\substr{10}{12}$; the substring extending algorithm obtain $\ell_{ex} = r_{ex}
  = 1$ on $\slp{F}_{6}'$ with parameters 8, 9, 10, 11; so family of repetitions \{2, 8, 9\} was found;

  $B_9$: using \textbf{PM} problem the algorithm obtain occurrence of $B_9$ at position 11 of $F_6'$ on
  $\slp{F}_{6}'\substr{9}{10}, \slp{F}_{6}'\substr{11}{13}$; the substring extending algorithm obtain $\ell_{ex} = 1,
  r_{ex} = 0$ on $\slp{F}_{6}'$ with parameters 9, 10, 11, 12; so family of repetitions \{2, 9, 9\} was found;

  The algorithm skip blocks $B_{10}, \dots, B_{12}$ since the search area consist of \emph{\$}. After merging families
  of repetitions the algorithm have the following result: \{3, 7, 7\}, \{2, 8, 9\}. Finally the algorithm check purity
  of families and shift them form $F_{6}'$ to $F_6$.

  After second step PS-table has the following view:

  \begin{figure}[h]
	  {\footnotesize\noindent
		\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline
	 	& $\slpterm{F}{1}{a}$ & $\slpterm{F}{2}{b}$ & $\slpnonterm{F}{3}{1}{2}$ & $\slpnonterm{F}{4}{3}{1}$ &
	 	$\slpnonterm{F}{5}{4}{3}$ & $\slpnonterm{F}{6}{5}{4}$ & $\slpnonterm{F}{7}{6}{5}$ \\ \hline

	 	[1, 1] & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & \{1, 3, 3\} & $\varnothing$ & \{1, 8, 8\} \\
	 	\hline

	 	[2, 3] & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & \{3, 4, 4\}, \{2, 5, 6\} &
	 	\{3, 10, 10\} \\ 	\hline

	 	[4, 7] & & & & & & & \\ \hline
		\end{tabular}
	  }
  \end{figure}

  \item \textbf{looking for squares that root length belongs to [4, 7]}

  The algorithm mark cells with $\varnothing$ for rules with length less than 8. The algorithm construct SLP
  $\slp{F}_7'$ that surrounds with \emph{\$} and has length 32. At figure 8 presented partition of $F_7'$ into
  blocks. Let us consider how the algorithm fill PS(3, 7). It skips $B_5$ since $B_5$ contains \emph{\$}.

   \FibonacciWordPartitionComplex

  $B_6$: using \textbf{PM} problem the algorithm obtain occurrence of $B_6$ at position 15; the substring extending algorithm
  obtain $\ell_{ex} = 1, r_{ex} = 3$ on $\slp{F}_{7}'$ with parameters 10, 12, 14, 18; so family of repetitions \{5, 14,
  15\} was found;

  $B_7$: using \textbf{PM} problem the algorithm obtain occurrence of $B_7$ at position 17; the substring extending algorithm
  obtain $\ell_{ex} = 3, r_{ex} = 1$ on $\slp{F}_{7}'$ with parameters 12, 14, 16, 20; so family of repetitions \{5, 14,
  15\} was found;

  $B_8$: using \textbf{PM} problem the algorithm obtain occurrence of $B_8$ at position 20; the substring extending algorithm
  obtain $\ell_{ex} = r_{ex} = 0$ on $\slp{F}_{7}'$ with parameters 14, 16, 18, 22; so there are no families of
  repetitions;

  $B_9$: using \textbf{PM} problem the algorithm obtain no occurrence of $B_9$; so there are no families of repetitions;

  The algorithm skips $B_{10}$ and $B_{11}$ since search areas consist of \emph{\$}. It skips $B_{12}$ since $B_{12}$
  contains \emph{\$}. After merging families of repetitions the algorithm have the following result: \{5, 14, 15\}.
  Finally the algorithm check purity of families and shift them form $F_{7}'$ to $F_7$.

  Finally PS-table has the following view:

  \begin{figure}[h]
	  {\footnotesize\noindent
		\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline
	 	& $\slpterm{F}{1}{a}$ & $\slpterm{F}{2}{b}$ & $\slpnonterm{F}{3}{1}{2}$ & $\slpnonterm{F}{4}{3}{1}$ &
	 	$\slpnonterm{F}{5}{4}{3}$ & $\slpnonterm{F}{6}{5}{4}$ & $\slpnonterm{F}{7}{6}{5}$ \\ \hline

	 	[1, 1] & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & \{1, 3, 3\} & $\varnothing$ & \{1, 8, 8\} \\
	 	\hline

	 	[2, 3] & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & \{3, 4, 4\}, \{2, 5, 6\} &
	 	\{3, 10, 10\} \\ \hline

	 	[4, 7] & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & $\varnothing$ & \{5, 5, 6\}
	 	\\ \hline
		\end{tabular}
	  }
  \end{figure}
\end{itemize}

\subsection{Text with complex family of pure squares}
Let us consider an SLP $\slp{E} = \slp{E}_l \cdot \slp{E}_r$ such that $\slp{E}_l$ derives text $(a~b~a~b~a)^6~a~b~a$
and $\slp{E}_r$ derives text $b~a~a~b~a~(a~b~a~b~a)^7$. So $\slp{E}$ derives text $(a~b~a~b~a)^7~a~b~a~(a~b~a~b~a)^7$
and $|E| = 73$. Let us consider how the algorithm looking for pure squares for rule $\slp{E}$ and segment [16, 31]. The
algorithm construct SLP $\slp{E}'$ that surrounds with \emph{\$} and has length 128. The partition of $E'$ into blocks
presented at figure 9.

\ComplexExamplePartititon

Let us consider how the algorithm process block $B_8$. It find occurrence of $B_8$ in $B_9 \cdot B_{10}$ at positions
$\prog{72}{2}{5}$. Since the algorithm find more than one occurrence it extends periodicity to calculate parameters
$\alpha_L, \alpha_R, \gamma_L, \gamma_R$. So $\alpha_L = \infty, \alpha_R = 69, \gamma_L = 66, \gamma_R = \infty$.
According to lemma !!! the algorithm find complex family of pure squares associated with each of root length \{18, 23\}
and centred at positions \{66, 67, 68, 69\}. Next we write every root from the family exactly:

\begin{itemize}
  \item \textbf{roots of length 18}

  $a~b~a(a~b~a~b~a)^3$, $b~a(a~b~a~b~a)^3a$, $a(a~b~a~b~a)^3a~b$, $(a~b~a~b~a)^3a~b~a$;
  \item \textbf{roots of length 23}

  $a~b~a(a~b~a~b~a)^4$, $b~a(a~b~a~b~a)^4a$, $a(a~b~a~b~a)^4a~b$, $(a~b~a~b~a)^4a~b~a$;
\end{itemize}

\end{appendix}


\subsection{Identity matrix and the Kronecker delta}


\documentclass{article}
\usepackage[legalpaper, portrait, margin=0.99in]{geometry}

% Language setting
% Replace `english' with e.g. `spanish' to change the document language
\usepackage[english]{babel}

% Set page size and margins
% Replace `letterpaper' with`a4paper' for UK/EU standard size
\usepackage[letterpaper,top=2cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}
% Useful packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{authblk}
\usepackage{amsfonts}
\usepackage{comment}

\title{Vectorized Adjoint Sensitivity Method for Graph Convolutional Neural Ordinary Differential Equations}
\author[1]{Jack Cai}
\affil[1]{University of Toronto, Division of Engineering Science}
\date{August 9, 2021}

\begin{document}
\maketitle

\begin{abstract}
This document, as the title stated, is meant to provide a vectorized implementation of adjoint dynamics calculation for Graph Convolutional Neural Ordinary Differential Equations (GCDE). The adjoint sensitivity method is the gradient approximation method for neural ODEs that replaces the back propagation. When implemented on libraries such as PyTorch or Tensorflow, the adjoint can be calculated by autograd functions without the need for a hand-derived formula. In applications such as edge computing and in memristor crossbars, however, autograds are not available, and therefore we need a vectorized derivation of adjoint dynamics to efficiently map the system on hardware. This document will go over the basics, then move on to derive the vectorized adjoint dynamics for GCDE.
\end{abstract}

\section{Introduction and Preliminaries}

Neural Ordinary Differential Equations (ODE) is a class of residual neural networks that frames the layer-wise propagation of hidden states into an initial value problem using differential equation solvers.
\begin{equation}
h_{t+1} = h_{t} + f(h_{t}, \theta_{t})
\end{equation}
A typical residual network may have layer wise transformation of hidden states looks like Equation (1). If we add more layers and take smaller time step, then the entire problem can be framed as an ODE defined by Equation (2), and a solution at time $t_{1}$ defined by Equation (3).
\begin{equation}
\frac{dh(t)}{dt} = f(h(t), t, \theta)
\end{equation}
\begin{equation}
h(t_{1}) = h(t_{0}) + \int_{t_{0}}^{t_{1}} f(h(t), t, \theta)dt
\end{equation}
Compared to conventional deep neural networks (DNNs), neural ODE uses less parameters applied more times to the hidden states, thereby achieves the depth of DNN while having higher memory efficiency.

\subsection{Adjoint sensitivity method}
The adjoint sensitivity method is used for automatic differentiation for neural ODE. Unlike traditional backpropagation, a quantity named adjoint is calculated for each $t$ and another ODE solver is used to integrate the overall gradient. Let $L()$ be the scaler loss function for the neural ODE, the adjoint is defined as $a(t) = \frac{\partial L}{\partial h(t)}$, and its dynamic is given by Equation (4):
\begin{equation}
\frac{da(t)}{dt} = -a(t)^{T}\frac{\partial f(h(t), t, \theta)}{\partial h}
\end{equation}
The adjoint at each instant is calculated by a backward ODE solver, and the overall gradient of the parameters $\frac{\partial L}{\partial \theta}$ is given by Equation (5), which is integrated by another ODE solver:
\begin{equation}
\frac{\partial L}{\partial \theta} = -\int_{t_{1}}^{t_{0}} a(t)^{T}\frac{\partial f(h(t), t, \theta)}{\partial \theta}dt
\end{equation}
The proof and intuition behind adjoint sensitivity method is presented in the original paper. For the scope of this paper, we are only focused on how Equation (4) and (5) can be vectorized for GCDE.

\subsection{GCDE}
GCDE, in short, is the neural ODE version of the state of the art graph learning method Graph Convolutional Neural Network (GCN). For GCN and GCDE, hidden state is represented by a matrix H of dimension $\mathbb{R}^{N\times C}$, representing $N$ nodes with $C$ features. Each layer of GCN or each step of GCDE involves node-wise exchange of information and convolution on nodes. The dynamic of GCDE (which is the same as the layer-wise propagation of GCN) is given by Equation (6), where $A\in\mathbb{R}^{N\times N}$ is symmetric that represents the graph topology and $W\in\mathbb{R}^{C\times C}$ represents the convolution filters:
\begin{equation}
\frac{dH(t)}{dt} = f(H(t), A, W) = ReLU(AH(t)W)
\end{equation}
To calculate the adjoint, we need to calculate calculate the Jacobian of the function $f(H(t), A, W)$, namely $\frac{\partial f(H(t), A, W)}{\partial H(t)}$ and $\frac{\partial f(H(t), A, W)}{\partial W}$. This poses us a challenge as we will have to unroll the matrix into vectors. While this can be done easily with built-in autograd functions in PyTorch and Tensorflow, it becomes messy when we write it out to obtain an analytical vectorized solution.

\subsection{Matrix conventions}
Before we delve into deriving the adjoint dynamics, I would like to discuss some convention used in this document. All matrices are represented by capital letters, all vectors are represented by bold lower case letter, all entries within a matrix are represented by the $i$,$j$ subscripts which denotes the $i_{th}$ and $j_{th}$ entry of the matrix, and all entries within a vector are represented by a single subscript denoting the index. For example, $A$ is a matrix, $\mathbf{b}$ is a vector, $a_{1, 2}$ denotes the $1, 2$ entry of $A$, and $b_{2}$ denotes the second entry of $\mathbf{b}$. We use $``:"$ to denote the entire indices along a row or column, so that we represent the $i_{th}$ row of $A$ as $\mathbf{a}_{i,:}$, and we represent $j_{th}$ column of A as $\mathbf{a}_{:,j}$.

\subsection{Multivariable calculus conventions}

Let $f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$ where it takes in a vector $\mathbf{x} \in\mathbb{R}^{n}$ and output a vector $\mathbf{y} \in\mathbb{R}^{m}$, i.e.:

\begin{equation}
\begin{bmatrix}
    y_{1}       \\
    y_{2}      \\
    \vdots \\
    y_{m}
\end{bmatrix}
= f(
\begin{bmatrix}
    x_{1} \\
    x_{2} \\
    \vdots \\
    x_{n}
\end{bmatrix}
)
\end{equation}
Then the Jacobian matrix $J_{f} \in \mathbb{R}^{m\times n}$ is defined as:

\begin{equation}
J_{f} = f(
\begin{bmatrix}
    \frac{\partial y_{1}}{\partial x_{1}} & \frac{\partial y_{1}}{\partial x_{2}} &\hdots & \frac{\partial y_{1}}{\partial x_{n}} \\
    \frac{\partial y_{2}}{\partial x_{1}}  & \ddots &\hdots & \vdots \\
    \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial y_{m}}{\partial x_{1}}  & \hdots & \hdots & \frac{\partial y_{m}}{\partial x_{n}}
\end{bmatrix}
)
\end{equation}
Following this convention, let $f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$ and $g: \mathbb{R}^{m} \rightarrow \mathbb{R}^{p}$, then $g \circ f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{p}$. The chain rule is defined as:
\begin{equation}
    J_{g \circ f} = J_{g}\cdot J_{f}
\end{equation}

\subsection{Partial derivatives of a matrix}
Finding the Jacobian for GCDE is challenging not only because it is a composite function, but also because the input is a matrix, we need to unroll the matrix into vectors to match our matrix Jacobian convention. Let $roll()$ and $unroll()$ to be such operations:
\begin{equation}
    \mathbb{R}^{m\times n}\ni A = roll(\mathbf{a}), \mathbf{a}\in\mathbb{R}^{mn}
\end{equation}
\begin{equation}
    \mathbb{R}^{mn}\ni\mathbf{a} = unroll(A),  A\in\mathbb{R}^{m\times n}
\end{equation}
The example below illustrate how they can be used. Suppose $f: \mathbb{R}^{m\times n} \rightarrow \mathbb{R}^{p\times n}$ defined by $f(A) = BA$, where $B \in \mathbb{R}^{p\times m}$. Using Equation (10) and (11), we can convert $f$ into $g: \mathbb{R}^{mn}\rightarrow \mathbb{R}^{pn}$, such that:
\begin{equation}
    g(\mathbf{a}) = unroll(B(roll(\mathbf{a})))
\end{equation}
As a result, we can obtained a Jacobian matrix $J_{g} \in \mathbb{R}^{pn\times mn}$, which follows the convention defined in section 1.4. The way we unroll a matrix could be by rows or by columns -- it really depends on the situation.
\section{Adjoint derivation for GCDE}
The key of finding the Jacobian matrix for $f(H(t), A, W)$ is to find the Jacobian matrix for the three-matrix multiplication step, as the derivative for $ReLU$ is just an elementwise binary step function. So before we moving on solving the Jacobian matrix for the whole thing, let's consider the general case function $f(A) = XAY: \mathbb{R}^{n\times p}  \rightarrow \mathbb{R}^{m\times q}$ where $X\in\mathbb{R}^{m\times n}$, $A\in\mathbb{R}^{n\times p}$, and $Y\in\mathbb{R}^{p\times q}$. We can think of it as composite function $f(A) = h\circ g(A)$, where:
\begin{equation}
    g(A) = XA = B, B\in\mathbb{R}^{m\times p}
\end{equation}
\begin{equation}
h(B) = BY = C, C\in\mathbb{R}^{m\times q}
\end{equation}
According to our convention, we need to unroll the matrix into vectors in order to calculate the Jacobian matrix. Let $\widehat{g}$ and $\widehat{h}$ to denote them and $\mathbf{a} = unroll(A)$ and $\mathbf{b} = unroll(B)$:
\begin{equation}
\widehat{g}(\mathbf{a}) = unroll(X(roll(\mathbf{a})))
\end{equation}
\begin{equation}
\widehat{h}(\mathbf{b}) = unroll((roll(\mathbf{b}))Y)\end{equation}
\begin{center}
    $\widehat{f}(\mathbf{a}) = \widehat{h}\circ\widehat{g}(\mathbf{a})$
\end{center}
Hence, according to chain rule, the Jacobian matrix $J_{\widehat{h}\circ\widehat{g}} = J_{\widehat{h}}\cdot J_{\widehat{g}}$ and we will calculate it step by step below.

\subsection{Jacobian matrix $J_{\widehat{g}}$ for $g(A)=XA$}
We will start by breaking $X$ into rows and $A$ into columns. According to our convention outlined in section 1.3, Equation (13) becomes:

\begin{center}$g(A) = XA = \begin{bmatrix} \mathbf{x}_{1,:} \\ \vdots \\ \mathbf{x}_{1,:} \end{bmatrix} \begin{bmatrix} \mathbf{a}_{:,1} && \hdots && \mathbf{a}_{:,p} \end{bmatrix} \\$
\end{center}

\begin{center}$
= \begin{bmatrix}
    \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1} & \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,2} &\hdots & \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,p} \\
    \mathbf{x}_{2,:} \cdot \mathbf{a}_{:,1}  & \ddots &\hdots & \vdots \\
    \vdots & \vdots & \ddots & \vdots\\
    \mathbf{x}_{m,:} \cdot \mathbf{a}_{:,1}  & \hdots & \hdots & \mathbf{x}_{m,:} \cdot \mathbf{a}_{:,p}
\end{bmatrix}
= \begin{bmatrix}
    b_{1, 1} & b_{1,2} &\hdots & b_{1,p} \\
    b_{2,1}  & \ddots &\hdots & \vdots \\
    \vdots & \vdots & \ddots & \vdots\\
    b_{m,1}  & \hdots & \hdots & b_{m,p}
\end{bmatrix}
$
\end{center}
Since we used rows of $X$ and columns of $A$, for our convenience, we will unroll $B$ by rows and $A$ by columns. Therefore our Jacobian matrix $J_{\widehat{g}}$ becomes:
\begin{center}$
J_{\widehat{g}} =
\begin{bmatrix}
    \frac{\partial b_{1,1}}{\partial a_{1,1}} & \frac{\partial b_{1,1}}{\partial a_{2,1}} &\hdots &  \frac{\partial b_{1,1}}{\partial a_{1,2}} &\hdots & \frac{\partial b_{1,1}}{\partial a_{n,p}} \\
    \frac{\partial b_{1, 2}}{\partial a_{1,1}}  & \ddots &\hdots & \frac{\partial b_{1,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial b_{2,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial b_{m,p}}{\partial a_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial b_{m,p}}{\partial a_{n,p}}
\end{bmatrix}
=
\begin{bmatrix}
    \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1}}{\partial a_{1,1}} & \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1}}{\partial a_{2,1}} &\hdots &  \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1}}{\partial a_{1,2}} &\hdots & \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1}}{\partial a_{n,p}} \\
    \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,2}}{\partial a_{1,1}}  & \ddots &\hdots & \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial \mathbf{x}_{2,:} \cdot \mathbf{a}_{:,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial \mathbf{x}_{m,:} \cdot \mathbf{a}_{:,m}}{\partial a_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial \mathbf{x}_{m,:} \cdot \mathbf{a}_{:,p}}{\partial a_{n,p}}
\end{bmatrix}
$
\end{center}
Notice that for $\frac{\partial \mathbf{x}_{1, :} \cdot \mathbf{a}_{:,1}}{\partial a_{2,1}}$, the partial derivative is non-zero because $\mathbf{x}_{1, :} \cdot \mathbf{a}_{:,1}$ depends on the entry $a_{2,1}$; whereas for $\frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,2}}{\partial a_{1,1}}$, the partial derivative is zero because $a_{1,1}$ is not an entry inside the column vector $\mathbf{a}_{:, 2}$, therefore change in $a_{1,1}$ do not impact $\mathbf{x}_{1,:} \cdot \mathbf{a}_{:,2}$. \textit{The entry must be within the column vector to have a non-zero partial derivative}. Following this pattern, our Jacobian matrix $J_{\widehat{g}}$ becomes:
\begin{center}$
J_{\widehat{g}} = \begin{bmatrix}
    \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1}}{\partial a_{1,1}} & \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,1}}{\partial a_{2,1}} &\hdots &  0 &\hdots & 0 \\
    0  & \ddots &\hdots & \frac{\partial \mathbf{x}_{1,:} \cdot \mathbf{a}_{:,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial \mathbf{x}_{2,:} \cdot \mathbf{a}_{:,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    0  & \hdots & \hdots & \hdots & \hdots & \frac{\partial \mathbf{x}_{m,:} \cdot \mathbf{a}_{:,p}}{\partial a_{n,p}}
\end{bmatrix}
$
\end{center}
Taking the derivative of the dot product, vectorize it and we get the Jacobian matrix $J_{\widehat{g}}$:
\begin{equation}
J_{\widehat{g}} = \begin{bmatrix}
    x_{1,1} &\hdots & x_{1,n}&  0 &\hdots & 0 \\
    0   &\ddots & 0 &x_{1,1}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    x_{2,1}  & \vdots & x_{2,n} & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    0  & \hdots & \hdots & \hdots & \hdots & x_{m,n}
\end{bmatrix} = \begin{bmatrix}
     \mathbf{x}_{1,:}  & &   &  \\
      & \mathbf{x}_{1,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{1,:}\\
    \mathbf{x}_{2,:}  & &   &  \\
      & \mathbf{x}_{2,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{2,:}\\
     & \hdots& \hdots &\\
     \mathbf{x}_{m,:}  & &   &  \\
      & \mathbf{x}_{m,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{m,:}
\end{bmatrix}
\in \mathbb{R}^{mp \times np}
\end{equation}

\subsection{Jacobian matrix $J_{\widehat{h}}$ for $h(B)=BY$}
Similarly to section 2.1, we will start by breaking $B$ and $Y$ into rows and columns. So Equation (14) becomes:

\begin{center}$
h(B) = BY = \begin{bmatrix} \mathbf{b}_{1,:} \\ \vdots \\ \mathbf{b}_{m,:} \end{bmatrix}
\begin{bmatrix} \mathbf{y}_{:,1} && \hdots && \mathbf{y}_{:,q} \end{bmatrix} \\$
\end{center}

\begin{center}$
= \begin{bmatrix}
    \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} & \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,2} &\hdots & \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,p} \\
    \mathbf{b}_{2,:} \cdot \mathbf{y}_{:,1}  & \ddots &\hdots & \vdots \\
    \vdots & \vdots & \ddots & \vdots\\
    \mathbf{b}_{m,:} \cdot \mathbf{y}_{:,1}  & \hdots & \hdots & \mathbf{b}_{m,:} \cdot \mathbf{y}_{:,p}
\end{bmatrix}
= \begin{bmatrix}
    c_{1,1} & c_{1,2} &\hdots & c_{1,p} \\
    c_{2,1}  & \ddots &\hdots & \vdots \\
    \vdots & \vdots & \ddots & \vdots\\
    c_{m,1}  & \hdots & \hdots & c_{m,p}
\end{bmatrix}
$
\end{center}
We will unroll B by rows and C by columns to construct our Jacobian matrix:
\begin{center}$
J_{\widehat{h}} =
\begin{bmatrix}
    \frac{\partial c_{1,1}}{\partial b_{1,1}} & \frac{\partial c_{1,1}}{\partial b_{1,2}} &\hdots &  \frac{\partial c_{1,1}}{\partial b_{2,1}} &\hdots & \frac{\partial c_{1,1}}{\partial b_{m,p}} \\
    \frac{\partial c_{1,2}}{\partial b_{1,1}}  & \ddots &\hdots & \frac{\partial c_{1,2}}{\partial b_{2,1}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial c_{2,1}}{\partial b_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial c_{m,q}}{\partial b_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial c_{m,q}}{\partial b_{m,p}}
\end{bmatrix}
=
\begin{bmatrix}
    \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} }{\partial b_{1,1}} & \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} }{\partial b_{1,2}} &\hdots &  \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} }{\partial b_{2,1}} &\hdots & \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} }{\partial b_{m,p}} \\
    \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,2} }{\partial b_{1,1}}  & \ddots &\hdots & \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,2} }{\partial b_{2,1}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial \mathbf{b}_{2,:} \cdot \mathbf{y}_{:,1} }{\partial b_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial \mathbf{b}_{m,:} \cdot \mathbf{y}_{:,q} }{\partial b_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial \mathbf{b}_{m,:} \cdot \mathbf{y}_{:,q} }{\partial b_{m,p}}
\end{bmatrix}
$
\end{center}
By the pattern that non-pairing entry and row vectors have zero partial derivatives, the Jacobian matrix can be reduced to:
\begin{center}$
J_{\widehat{h}}
=
\begin{bmatrix}
    \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} }{\partial b_{1,1}} & \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,1} }{\partial b_{1,2}} &\hdots &  0 &\hdots & 0 \\
    \frac{\partial \mathbf{b}_{1,:} \cdot \mathbf{y}_{:,2} }{\partial b_{1,1}}  & \ddots &\hdots & 0 & \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    0 & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    0  & \hdots & \hdots & \hdots & \hdots & \frac{\partial \mathbf{b}_{m,:} \cdot \mathbf{y}_{:,q} }{\partial b_{m,p}}
\end{bmatrix}
$
\end{center}
Taking the partial derivatives of the dot product:
\begin{center}$
J_{\widehat{h}}
=
\begin{bmatrix}
    y_{1,1} & y_{2,1} &\hdots & y_{p,1}&  0 &\hdots & 0 \\
    y_{1,2}  & \vdots &\ddots  &y_{p,2}&0& \hdots& \vdots \\
    \vdots &\vdots& \vdots & \ddots & \hdots & \hdots & \vdots\\
    y_{1,q} & y_{2,q} & \vdots & y_{p,q}& 0 & \hdots & \vdots\\
    0& 0 & \vdots & 0& y_{1,1} & \hdots & \vdots\\
    0 & 0 & \vdots & 0& y_{1,2} & \hdots & \vdots\\
    \vdots&\vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    0 & 0 & \vdots & 0& y_{1,q} & \hdots & \vdots\\
    \vdots&\vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    0  &\hdots& \hdots & \hdots & \hdots & \hdots & y_{p,q}
\end{bmatrix}
$
\end{center}
Notice that the diagonal $q\times p$ blocks are $Y^T$, therefore:
\begin{equation}
J_{\widehat{h}}
=
\begin{bmatrix}
    Y^T \\ & Y^T \\ & & Y^T \\ & & & \ddots \\ & & & & Y^T
\end{bmatrix} \in \mathbb{R}^{mq \times mp}
\end{equation}

\subsection{Calculating $J_{\widehat{h}\circ\widehat{g}} $ from the chain rule}
By the chain rule, we calculate $J_{\widehat{h}\circ\widehat{g}} \in \mathbb{R}^{mq \times np}$ by $J_{\widehat{h}}\cdot J_{\widehat{g}}$:
\begin{center}$
J_{\widehat{h}}\cdot J_{\widehat{g}}
=
\begin{bmatrix}
    Y^T \\ & Y^T \\ & & Y^T \\ & & & \ddots \\ & & & & Y^T
\end{bmatrix} \cdot
\begin{bmatrix}
     \mathbf{x}_{1,:}  & &   &  \\
      & \mathbf{x}_{1,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{1,:}\\
     & \hdots& \hdots &\\
     \mathbf{x}_{m,:}  & &   &  \\
      & \mathbf{x}_{m,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{m,:}
\end{bmatrix}
$
\end{center}
To simplify the matrix multiplication, we break done $Y^T$ to (note that $\mathbf{y}_{i,:}^T \in \mathbb{R}^{q\times1}$ is the transpose of the first row of $Y$):
\begin{center}$
Y^T=\begin{bmatrix}
    \mathbf{y}_{1,:}^T & \mathbf{y}_{2,:}^T & \hdots & \mathbf{y}_{p,:}^T
\end{bmatrix}
$\end{center}
And we break the matrix multiplication blocks by blocks:
\begin{center}
    $J_{\widehat{h}}\cdot J_{\widehat{g}}=
    \begin{bmatrix}
    \begin{bmatrix}Y^T & \textbf{0}& \hdots& \textbf{0}\end{bmatrix} \cdot J_{\widehat{g}}\\
    \begin{bmatrix}\textbf{0}& Y^T& \hdots& \textbf{0}\end{bmatrix} \cdot J_{\widehat{g}}\\
    \hdots     \\
     \begin{bmatrix}\textbf{0} & \textbf{0}& \hdots& Y^T\end{bmatrix} \cdot J_{\widehat{g}}
    \end{bmatrix}
    $
\end{center}
Let's look at the first block:
\begin{equation}
\begin{bmatrix}Y^T & \textbf{0}& \hdots& \textbf{0}\end{bmatrix} \cdot J_{\widehat{g}} =
    \begin{bmatrix}
    Y^T & \textbf{0}& \hdots& \textbf{0}
\end{bmatrix} \cdot
\begin{bmatrix}
     \mathbf{x}_{1,:}  & &   &  \\
      & \mathbf{x}_{1,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{1,:}\\
     & \hdots& \hdots &\\
     \mathbf{x}_{m,:}  & &   &  \\
      & \mathbf{x}_{m,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{m,:}
\end{bmatrix}
\end{equation}
Note that since $Y^T$  $\in q\times p$, only the first $p$ rows of $J_{\widehat{g}}$ matters. Hence Equation (19) becomes:
\begin{center}
    $Y^T \cdot
    \begin{bmatrix}
     \mathbf{x}_{1,:}  & &   &  \\
      & \mathbf{x}_{1,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{1,:}\\
\end{bmatrix}$
\end{center}

\begin{center}
    $=\begin{bmatrix}
    \mathbf{y}_{1,:}^T & \mathbf{y}_{2,:}^T & \hdots & \mathbf{y}_{p,:}^T
\end{bmatrix} \cdot
    \begin{bmatrix}
     \mathbf{x}_{1,:}  & &   &  \\
      & \mathbf{x}_{1,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{1,:}\\
\end{bmatrix}$
\end{center}
\begin{center}
    $= \begin{bmatrix}
     \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{1,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{1,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{1,:}\end{bmatrix} \in \mathbb{R}^{q\times pn}$
\end{center}
If we repeat for other blocks, we would get the Jacobian matrix for $J_{\widehat{h}\circ\widehat{g}} $ to be:

\begin{equation}
    J_{\widehat{h}\circ\widehat{g}} = \begin{bmatrix}
    \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{1,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{1,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{1,:} \\
    \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{2,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{2,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{2,:} \\
    \vdots & \vdots & \ddots & \vdots \\
    \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{m,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{m,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{m,:} \\
     \end{bmatrix} \in \mathbb{R}^{qm \times pn}
\end{equation}
It is also important to keep in mind what each entry within $J_{\widehat{h}\circ\widehat{g}}$ mean:
\begin{center}
    $J_{\widehat{h}\circ\widehat{g}}=J_{\widehat{h}}\cdot J_{\widehat{g}} =
    \begin{bmatrix}
    \frac{\partial c_{1,1}}{\partial b_{1,1}} & \frac{\partial c_{1,1}}{\partial b_{1,2}} &\hdots &  \frac{\partial c_{1,1}}{\partial b_{2,1}} &\hdots & \frac{\partial c_{1,1}}{\partial b_{m,p}} \\
    \frac{\partial c_{1,2}}{\partial b_{1,1}}  & \ddots &\hdots & \frac{\partial c_{1,2}}{\partial b_{2,1}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial c_{2,1}}{\partial b_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial c_{m,q}}{\partial b_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial c_{m,q}}{\partial b_{m,p}}
\end{bmatrix} \cdot
    \begin{bmatrix}
    \frac{\partial b_{1,1}}{\partial a_{1,1}} & \frac{\partial b_{1,1}}{\partial a_{2,1}} &\hdots &  \frac{\partial b_{1,1}}{\partial a_{1,2}} &\hdots & \frac{\partial b_{1,1}}{\partial a_{n,p}} \\
    \frac{\partial b_{1, 2}}{\partial a_{1,1}}  & \ddots &\hdots & \frac{\partial b_{1,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial b_{2,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial b_{m,p}}{\partial a_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial b_{m,p}}{\partial a_{n,p}}
\end{bmatrix}
$
\end{center}
\begin{equation}
    = \begin{bmatrix}
    \frac{\partial c_{1,1}}{\partial a_{1,1}} & \frac{\partial c_{1,1}}{\partial a_{2,1}} &\hdots &  \frac{\partial c_{1,1}}{\partial a_{1,2}} &\hdots & \frac{\partial c_{1,1}}{\partial a_{n,p}} \\
    \frac{\partial c_{1,2}}{\partial a_{1,1}}  & \ddots &\hdots & \frac{\partial c_{1,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial c_{2,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial c_{m,q}}{\partial a_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial c_{m,q}}{\partial a_{n,p}}
\end{bmatrix}
\end{equation}

\subsection{Adjoint calculation}
Now we have successfully found the general solution of Jacobian matrix for $f(A)=XAY$ and $g(A)=XA$ (or more precisely, the unrolled version of them, namely $f(\mathbf{a})$ and $\widehat{g}(\mathbf{a})$, and it is time to apply them in the adjoint calculation for GCDE, as they are the special cases of the general solutions outlined in section 2.1-2.3. Back in section 1.2 I stated that we need to find the Jacobian matrix $\frac{\partial f(H(t), A, W)}{\partial H(t)}$ and $\frac{\partial f(H(t), A, W)}{\partial W}$; well that does not actually make sense in our convention since (1) $H(t)$ is not a vector and (2) the adjoint must be a vector, which means $\frac{\partial L}{\partial H(t)}$ should also be unrolled. Therefore, a GCDE version of adjoint dynamics for Equation (4) and (5) is:
\begin{equation}
    -a(t)^{T}\frac{\partial f(h(t), t, \theta)}{\partial h} \rightarrow
    -\frac{\partial L}{\partial \mathbf{h}(t)}^{T}  \frac{\partial \widehat{f}(\mathbf{h}(t), A, W)}{\partial \mathbf{h}(t)}
\end{equation}
\begin{equation}
    -a(t)^{T}\frac{\partial f(h(t), t, \theta)}{\partial \theta} \rightarrow
        -\frac{\partial L}{\partial \mathbf{h}(t)}^{T}  \frac{\partial \widehat{f}(\mathbf{h}(t), A, W)}{\partial \mathbf{w}}
\end{equation}
where $\mathbf{h}(t) = unroll(H(t))$, $\mathbf{w} = unroll(W)$, and $\widehat{f}(\mathbf{h}(t), A, W) = unroll(f(roll(\mathbf{h}(t)), A, W))$.
However, this does not stop us to find a vectorized equivalence for Equation (4) and (5) for GCDE implementation on hardware, as keeping the matrices unrolled greatly increases the dimensions and could not take the parallel in-memory computing advantages brought by memristor crossbars.

%The first adjoint, $\frac{\partial L}{\partial H(t)}^T\cdot\frac{\partial f(H(t), A, W)}{\partial H(t)}$

%$\frac{\partial f(H(t), A, W)}{\partial H(t)}$ and $\frac{\partial f(H(t), A, W)}{\partial W}$

\subsection{Vectorized Equation (22)}
We start the vectorization from the general case $f(A)=XAY$ we discussed in the earlier sections. Let $ \mathbb{R}^{m\times q}\ni C = f(A)$, and $\mathbb{R}^{mq}\ni\mathbf{c} = unroll(C)$, we unroll $C$ row by row so that its partial derivatives with respect to a scalar loss function $L$ has the form:
\begin{center}
    $\frac{\partial L}{\partial \mathbf{c}} =
    \begin{bmatrix}
        \frac{\partial L}{\partial c_{1,1}} \\ \vdots \\\frac{\partial L}{\partial c_{1,q}}\\ \frac{\partial L}{\partial c_{2,1}} \\ \vdots \\ \frac{\partial L}{\partial c_{m,q}}
    \end{bmatrix} =
    \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}}^T \\ \frac{\partial L}{\partial \mathbf{c}_{2,:}}^T\\ \vdots \\ \frac{\partial L}{\partial \mathbf{c}_{m,:}}^T
    \end{bmatrix} \in \mathbb{R}^{mq}, \frac{\partial L}{\partial \mathbf{c}_{i,:}} \in \mathbb{R}^{1\times q}
    $
\end{center}
We want to find (a reminder that $\widehat{f}(\mathbf{a})$ is a unrolled version of $f(A)$):
\begin{center}
$
-\frac{\partial L}{\partial \mathbf{c}}^{T} \frac{\partial \widehat{f}(\mathbf{a})}{\partial \mathbf{a}}=
-\frac{\partial L}{\partial \mathbf{c}}^{T}  J_{\widehat{f}} = -\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}$
\end{center}
Expand and plug in Equation (20):
\begin{center}
    $ -\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}} =
    -\begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} & \frac{\partial L}{\partial \mathbf{c}_{2,:}} & \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}}
    \end{bmatrix} \cdot
    \begin{bmatrix}
    \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{1,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{1,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{1,:} \\
    \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{2,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{2,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{2,:} \\
    \vdots & \vdots & \ddots & \vdots \\
    \mathbf{y}_{1,:}^T\cdot\mathbf{x}_{m,:}  &  \mathbf{y}_{2,:}^T\cdot\mathbf{x}_{m,:} & \hdots & \mathbf{y}_{p,:}^T\cdot\mathbf{x}_{m,:} \\
     \end{bmatrix}
    $
\end{center}
Since $\frac{\partial L}{\partial \mathbf{c}_{i,:}} \in \mathbb{R}^{1\times q}$ and $\mathbf{y}^T_{i,:}\cdot \mathbf{x}_{j,:} \in \mathbb{R}^{q\times n}$, the blocks defined above have matching dimensions. Hence we can take their dot product directly:
\begin{center}
    $ -\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}} = -
    \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{1,:}\cdot \mathbf{x}_{1,:}  + \hdots + \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{1,:}\cdot \mathbf{x}_{m,:}  &
        \hdots &
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{p,:}\cdot \mathbf{x}_{1,:}   + \hdots + \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{p,:}\cdot \mathbf{x}_{m,:}
    \end{bmatrix}
    $
\end{center}
Note that this dot product is a long $1\times np$ vector, and we can simplify it by rolling it into a $p\times n$ matrix:

\begin{equation}
    -roll(\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}) = -
    \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{1,:}\cdot \mathbf{x}_{1,:}   + \hdots + \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{1,:}\cdot \mathbf{x}_{m,:} \\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{2,:}\cdot \mathbf{x}_{1,:}   + \hdots + \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{2,:}\cdot \mathbf{x}_{m,:} \\
        \vdots\\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{p,:}\cdot \mathbf{x}_{1,:}   + \hdots + \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{p,:}\cdot \mathbf{x}_{m,:}
    \end{bmatrix}
\end{equation}
Equation (24) can be further simplified:
\begin{center}
    $-roll(\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}) = \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{1,:} & \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{1,:} \\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{2,:}  &\hdots & \frac{\partial L}{\partial \mathbf{c}_{m, :}} \cdot \mathbf{y}^T_{2,:}\\
        \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{p,:} &  \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{p,:}
    \end{bmatrix} \cdot \begin{bmatrix}
        \mathbf{x}_{1,:} \\ \mathbf{x}_{2,:} \\ \vdots \\ \mathbf{x}_{m,:}
    \end{bmatrix} = \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{1,:} & \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{1,:} \\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{2,:}  &\hdots & \frac{\partial L}{\partial \mathbf{c}_{m, :}} \cdot \mathbf{y}^T_{2,:}\\
        \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{p,:} &  \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{p,:}
    \end{bmatrix} \cdot X$
\end{center}
Let:
\begin{center}
    $E = \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{1,:} & \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{1,:} \\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{2,:}  &\hdots & \frac{\partial L}{\partial \mathbf{c}_{m, :}} \cdot \mathbf{y}^T_{2,:}\\
        \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{p,:} &  \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{p,:}
    \end{bmatrix}$
\end{center}
We can simplify E further by looking at it row by row. The first row of E is:
\begin{equation}
    \mathbf{e}_{1,:} = \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} \cdot \mathbf{y}^T_{1,:} & \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}} \cdot \mathbf{y}^T_{1,:}
    \end{bmatrix}
\end{equation}
Notice that each entry within Equation (25) is a scalar because $\frac{\partial L}{\partial \mathbf{c}_{i,:}} \in \mathbb{R}^{1\times q}$ and $\mathbf{y}^T_{1,:} \in \mathbb{R}^{q\times 1}$. Therefore, we can manipulate Equation (25) such that:
\begin{center}
    $\mathbf{e}_{1,:} = (\mathbf{y}^T_{1,:})^{T} \cdot \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}}^{T} & \frac{\partial L}{\partial \mathbf{c}_{2,:}}^{T} & \hdots &\frac{\partial L}{\partial \mathbf{c}_{m,:}}^{T}
    \end{bmatrix} = \mathbf{y}_{1,:} \cdot \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}}^{T} & \frac{\partial L}{\partial \mathbf{c}_{2,:}}^{T} & \hdots &\frac{\partial L}{\partial \mathbf{c}_{m,:}}^{T}
    \end{bmatrix}$
\end{center}
We define $\frac{\partial L}{\partial C}$ having matching entries to $C$:
\begin{center}
    $\frac{\partial L}{\partial C} = \begin{bmatrix}
        \frac{\partial L}{\partial c_{1,1}} & \frac{\partial L}{\partial c_{1,2}} & \hdots &\frac{\partial L}{\partial c_{1,q} } \\
        \frac{\partial L}{\partial c_{2,1}} & \ddots & \hdots &\vdots \\
        \vdots & \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial c_{m,1}} & \hdots & \hdots &\frac{\partial L}{\partial c_{m,q}}
    \end{bmatrix}$
\end{center}
Therefore:
\begin{center}
        $\mathbf{e}_{1,:} = \mathbf{y}_{1,:} \cdot \frac{\partial L}{\partial C}^T$
\end{center}
If we repeat for all rows of $E$, then:
\begin{center}
   $E = \begin{bmatrix}
        \mathbf{y}_{1,:} \cdot \frac{\partial L}{\partial C}^T \\
        \mathbf{y}_{2,:} \cdot \frac{\partial L}{\partial C}^T\\
        \vdots \\
        \mathbf{y}_{p,:} \cdot \frac{\partial L}{\partial C}^T
    \end{bmatrix}
    = \begin{bmatrix}
        \mathbf{y}_{1,:}  \\
        \mathbf{y}_{2,:} \\
        \vdots \\
        \mathbf{y}_{p,:}
    \end{bmatrix} \cdot \frac{\partial L}{\partial C}^T = Y \cdot \frac{\partial L}{\partial C}^T$
\end{center}
As a result:
\begin{equation}
    -roll(\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}) = -E \cdot X = -Y \cdot \frac{\partial L}{\partial C}^T \cdot X
\end{equation}
We need to keep in mind what this matrix actually represents. So we go back to Equation (21):
\begin{center}
    $ -roll(\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}) =
    -roll(\begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{c}_{1,:}} & \frac{\partial L}{\partial \mathbf{c}_{2,:}} & \hdots & \frac{\partial L}{\partial \mathbf{c}_{m,:}}
    \end{bmatrix} \cdot= \begin{bmatrix}
    \frac{\partial c_{1,1}}{\partial a_{1,1}} & \frac{\partial c_{1,1}}{\partial a_{2,1}} &\hdots &  \frac{\partial c_{1,1}}{\partial a_{1,2}} &\hdots & \frac{\partial c_{1,1}}{\partial a_{n,p}} \\
    \frac{\partial c_{1,2}}{\partial a_{1,1}}  & \ddots &\hdots & \frac{\partial c_{1,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial c_{2,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial c_{m,q}}{\partial a_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial c_{m,q}}{\partial a_{n,p}}
\end{bmatrix})
    $
\end{center}

\begin{center}
$=-roll(\begin{bmatrix}
        \frac{\partial L}{\partial a_{1,1}} & \frac{\partial L}{\partial a_{2,1}} & \hdots & \frac{\partial L}{\partial a_{n,1}} & \frac{\partial L}{\partial a_{1,2}} & \hdots& \frac{\partial L}{\partial a_{n,p}}
    \end{bmatrix})$
\end{center}
\begin{equation}
=-\begin{bmatrix}
        \frac{\partial L}{\partial a_{1,1}} & \frac{\partial L}{\partial a_{2,1}} & \hdots & \frac{\partial L}{\partial a_{n,1}} \\
        \frac{\partial L}{\partial a_{1,2}} & \frac{\partial L}{\partial a_{2,2}} & \hdots & \frac{\partial L}{\partial a_{n,2}} \\
        \vdots & \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial a_{1,p}} & \frac{\partial L}{\partial a_{2,p}} & \hdots & \frac{\partial L}{\partial a_{n,p}}
    \end{bmatrix}=
    -\begin{bmatrix}
        -\frac{\partial L}{\partial \mathbf{a}_{:,1}}^T- \\
        -\frac{\partial L}{\partial \mathbf{a}_{:,2}}^T- \\
        \vdots\\
        -\frac{\partial L}{\partial \mathbf{a}_{:,p}}^T- \\
    \end{bmatrix}
\end{equation}
Notice that the entries of this matrix matches the transpose of $A$. Hence, in order to match the entries of $A$, we will take Equation (27)'s transpose -- and here we found the general vectorized adjoint solution for $f(A)$:
\begin{center}
    $vectorized(-\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}) = -\begin{bmatrix}
        -\frac{\partial L}{\partial \mathbf{a}_{:,1}}^T- \\
        -\frac{\partial L}{\partial \mathbf{a}_{:,2}}^T- \\
        \vdots\\
        -\frac{\partial L}{\partial \mathbf{a}_{:,p}}^T- \\
    \end{bmatrix}^T = -(Y \cdot \frac{\partial L}{\partial C}^T \cdot X)^T $
\end{center}
\begin{equation}
    vectorized(-\frac{\partial L}{\partial \mathbf{c}}^{T} J_{\widehat{h}\circ\widehat{g}}) = -X^T \cdot \frac{\partial L}{\partial C} \cdot Y^T
\end{equation}
The adjoint dynamics shown in Equation (22) for GCDE is just a special of Equation (28). The derivative of the $ReLU()$ activation function is an element-wise binary step function:
\begin{equation}
    step(x)=
    \begin{cases}
      1 & x>0 \\
      0 & x\leq 0
   \end{cases}
\end{equation}
Hence, let $\odot$ denote the Hadamard product between matrices, by chain rule and Equation (28), the vectorized Equation (22) is:
\begin{center}
    $vectorized(-\frac{\partial L}{\partial \mathbf{h}(t)}^{T}  \frac{\partial \widehat{f}(\mathbf{h}(t), A, W)}{\partial \mathbf{h}(t)}) =
    -A^T \cdot (\frac{\partial L}{\partial H(t)} \odot step(H(t)))\cdot W^T$
\end{center}
where:
\begin{center}
    $\frac{\partial L}{\partial H(t)} = roll(\frac{\partial L}{\partial \mathbf{h}(t)})=\begin{bmatrix}
        \frac{\partial L}{\partial h_{1,1}} & \frac{\partial L}{\partial h_{1,2}} & \hdots &\frac{\partial L}{\partial h_{1,q} } \\
        \frac{\partial L}{\partial h_{2,1}} & \ddots & \hdots &\vdots \\
        \vdots & \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial h_{m,1}} & \hdots & \hdots &\frac{\partial L}{\partial h_{m,q}}
    \end{bmatrix}$
\end{center}
Since by definition of GCDE, the graph topology matrix A is always symmetric, the final vectorized Equation (22) is:
\begin{equation}
 vectorized(-\frac{\partial L}{\partial \mathbf{h}(t)}^{T}  \frac{\partial \widehat{f}(\mathbf{h}(t), A, W)}{\partial \mathbf{h}(t)}) =
    -A \cdot (\frac{\partial L}{\partial H(t)} \odot step(H(t)))\cdot W^T
\end{equation}
\subsection{Vectorized Equation (23)}
We will again vectorize Equation (23) from a general case, $g(A)=XA$. This is because we can treat $AH(t)$ in Equation (6) as a single matrix that linearly transforms $W$. Let $ \mathbb{R}^{m\times p}\ni B = g(A)$, and $\mathbb{R}^{mp}\ni\mathbf{b} = unroll(B)$, we unroll $B$ row by row so that its partial derivatives with respect to a scalar loss function $L$ has the form:
\begin{center}
    $\frac{\partial L}{\partial \mathbf{b}} =
    \begin{bmatrix}
        \frac{\partial L}{\partial b_{1,1}} \\ \vdots \\\frac{\partial L}{\partial b_{1,p}}\\ \frac{\partial L}{\partial b_{2,1}} \\ \vdots \\ \frac{\partial L}{\partial b_{m,p}}
    \end{bmatrix} =
    \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{b}_{1,:}}^T \\ \frac{\partial L}{\partial \mathbf{b}_{2,:}}^T\\ \vdots \\ \frac{\partial L}{\partial \mathbf{b}_{m,:}}^T
    \end{bmatrix} \in \mathbb{R}^{mp}, \frac{\partial L}{\partial \mathbf{b}_{i,:}} \in \mathbb{R}^{1\times p}
    $
\end{center}
In addition, we define
\begin{center}
    $\frac{\partial L}{\partial B} = \begin{bmatrix}
        \frac{\partial L}{\partial b_{1,1}} & \frac{\partial L}{\partial b_{1,2}} & \hdots &\frac{\partial L}{\partial b_{1,q} } \\
        \frac{\partial L}{\partial b_{2,1}} & \ddots & \hdots &\vdots \\
        \vdots & \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial b_{m,1}} & \hdots & \hdots &\frac{\partial L}{\partial b_{m,q}}
    \end{bmatrix}$
\end{center}
Since we are considering the general case $g(A)=XA$ and we are given an arbitury adjoint $\mathbf{b}$, we can change Equation (17) to:
\begin{center}
$-\frac{\partial L}{\partial \mathbf{h}(t)}^{T}  \frac{\partial \widehat{f}(\mathbf{h}(t), A, W)}{\partial \mathbf{w}} \rightarrow
-\frac{\partial L}{\partial \mathbf{b}}^{T} \frac{\partial \widehat{g}(\mathbf{a})}{\partial \mathbf{a}} = -\frac{\partial L}{\partial \mathbf{b}}^{T}  J_{\widehat{g}}$
\end{center}
Expand and plug in Equation (17):
\begin{center}
    $ -\frac{\partial L}{\partial \mathbf{b}}^{T} J_{\widehat{g}} =
    -\begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{b}_{1,:}} & \frac{\partial L}{\partial \mathbf{b}_{2,:}} & \hdots & \frac{\partial L}{\partial \mathbf{b}_{m,:}}
    \end{bmatrix} \cdot
    \begin{bmatrix}
     \mathbf{x}_{1,:}  & &   &  \\
      & \mathbf{x}_{1,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{1,:}\\
     & \hdots& \hdots &\\
     \mathbf{x}_{m,:}  & &   &  \\
      & \mathbf{x}_{m,:} & &  \\
     &  & \ddots &\\
     &  &  & \mathbf{x}_{m,:}
\end{bmatrix}$
\end{center}
We again break it into blocks:
\begin{center}
    $ -\frac{\partial L}{\partial \mathbf{b}}^{T} J_{\widehat{g}} =
    -\begin{bmatrix} \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{b}_{1,:}} & \frac{\partial L}{\partial \mathbf{b}_{2,:}} & \hdots & \frac{\partial L}{\partial \mathbf{b}_{m,:}}
    \end{bmatrix} \cdot
    \begin{bmatrix}
     \mathbf{x}_{1,:}  \\
       \vdots\\
    \textbf{0} \\
     \mathbf{x}_{2,:} \\
      \vdots\\
     \mathbf{x}_{m,:} \\
    \vdots\\
    \textbf{0} \\
\end{bmatrix} \hdots
    \begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{b}_{1,:}} & \frac{\partial L}{\partial \mathbf{b}_{2,:}} & \hdots & \frac{\partial L}{\partial \mathbf{b}_{m,:}}
    \end{bmatrix} \cdot
    \begin{bmatrix}
      \textbf{0} \\
       \vdots\\
    \mathbf{x}_{1,:} \\
      \textbf{0} \\
      \vdots\\
      \textbf{0}\\
    \vdots\\
    \mathbf{x}_{m,:} \\
\end{bmatrix}
\end{bmatrix}$
\end{center}
If we expand the first block:
\begin{center}
    $\begin{bmatrix}
        \frac{\partial L}{\partial \mathbf{b}_{1,:}} & \frac{\partial L}{\partial \mathbf{b}_{2,:}} & \hdots & \frac{\partial L}{\partial \mathbf{b}_{m,:}}
    \end{bmatrix} \cdot
    \begin{bmatrix}
     \mathbf{x}_{1,:}  \\
       \vdots\\
    \textbf{0} \\
     \mathbf{x}_{2,:} \\
      \vdots\\
     \mathbf{x}_{m,:} \\
    \vdots\\
    \textbf{0} \\
\end{bmatrix}  = \begin{bmatrix}
 \frac{\partial L}{\partial b_{1,1}} \cdot x_{1,1} + \frac{\partial L}{\partial b_{2,1}} \cdot x_{2,1} + \hdots + \frac{\partial L}{\partial b_{m,1}} \cdot x_{m,1} \\
 \frac{\partial L}{\partial b_{1,1}} \cdot x_{1,2} + \frac{\partial L}{\partial b_{2,1}} \cdot x_{2,2} + \hdots + \frac{\partial L}{\partial b_{m,1}} \cdot x_{m,2} \\
 \vdots\\
 \frac{\partial L}{\partial b_{1,1}} \cdot x_{1,p} + \frac{\partial L}{\partial b_{2,1}} \cdot x_{2,p} + \hdots + \frac{\partial L}{\partial b_{m,1}} \cdot x_{m,p}
\end{bmatrix}^T$
\end{center}
\begin{center}
    $= (\begin{bmatrix}
  x_{1,1} &  x_{2,1} & \hdots &  x_{m,1} \\
  x_{1,2} &  x_{2,2} & \hdots &  x_{m,2} \\
 \vdots &\vdots& \ddots &\vdots\\
  x_{1,p} &  x_{2,p} & \hdots &  x_{m,p}
\end{bmatrix} \cdot \begin{bmatrix}
 \frac{\partial L}{\partial b_{1,1}} \\ \frac{\partial L}{\partial b_{2,1}} \\ \vdots \\ \frac{\partial L}{\partial b_{m,1}}
\end{bmatrix})^T = (X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,1}})^T$
\end{center}
If we repeat for other blocks, we would get:
\begin{center}
    $ -\frac{\partial L}{\partial \mathbf{b}}^{T} J_{\widehat{g}} = \begin{bmatrix}
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,1}} \\
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,2}} \\
    \vdots\\
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,p}}
    \end{bmatrix}^T$ or $ (-\frac{\partial L}{\partial \mathbf{b}}^{T} J_{\widehat{g}})^T = \begin{bmatrix}
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,1}} \\
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,2}} \\
    \vdots\\
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,p}}
    \end{bmatrix}$
\end{center}
We will make it vectorized by rolling it into a $n\times p$ matrix:
\begin{center}
    $roll((-\frac{\partial L}{\partial \mathbf{b}}^{T} J_{\widehat{g}})^T) = -\begin{bmatrix}
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,1}} &
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,2}} &
    \hdots&
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,p}}
    \end{bmatrix}$
\end{center}
Further vectorization could be done:
\begin{center}
    $-\begin{bmatrix}
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,1}} &
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,2}} &
    \hdots&
    X^T \cdot \frac{\partial L}{\partial \mathbf{b}_{:,p}}
    \end{bmatrix} = -X^T \cdot \begin{bmatrix}
     \frac{\partial L}{\partial \mathbf{b}_{:,1}} &
     \frac{\partial L}{\partial \mathbf{b}_{:,2}} &
    \hdots&
     \frac{\partial L}{\partial \mathbf{b}_{:,p}}
    \end{bmatrix}$
\end{center}
\begin{equation}
    = -X^T \cdot \frac{\partial L}{\partial B}
\end{equation}
Of course, we need to keep in mind what $-X^T \cdot \frac{\partial L}{\partial B}$ actually represents:
\begin{center}
    $roll((-\frac{\partial L}{\partial \mathbf{b}}^{T} J_{\widehat{g}})^T) =  -(\begin{bmatrix}
        \frac{\partial L}{\partial b_{1,1}} & \hdots &\frac{\partial L}{\partial b_{1,p}}& \frac{\partial L}{\partial b_{2,1}} & \hdots & \frac{\partial L}{\partial b_{m,p}}
    \end{bmatrix} \cdot \begin{bmatrix}
    \frac{\partial b_{1,1}}{\partial a_{1,1}} & \frac{\partial b_{1,1}}{\partial a_{2,1}} &\hdots &  \frac{\partial b_{1,1}}{\partial a_{1,2}} &\hdots & \frac{\partial b_{1,1}}{\partial a_{n,p}} \\
    \frac{\partial b_{1, 2}}{\partial a_{1,1}}  & \ddots &\hdots & \frac{\partial b_{1,2}}{\partial a_{1,2}}& \hdots& \vdots \\
    \vdots & \vdots & \ddots & \vdots & \hdots & \vdots\\
    \frac{\partial b_{2,1}}{\partial a_{1,1}} & \vdots & \vdots & \ddots & \hdots & \vdots\\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\
    \frac{\partial b_{m,p}}{\partial a_{1,1}}  & \hdots & \hdots & \hdots & \hdots & \frac{\partial b_{m,p}}{\partial a_{n,p}}
\end{bmatrix})^T$
\end{center}
\begin{center}
$=-roll(\begin{bmatrix}
        \frac{\partial L}{\partial a_{1,1}} & \frac{\partial L}{\partial a_{2,1}} & \hdots & \frac{\partial L}{\partial a_{n,1}} & \frac{\partial L}{\partial a_{1,2}} & \hdots& \frac{\partial L}{\partial a_{n,p}}
    \end{bmatrix}^T)$
\end{center}
\begin{equation}
=-\begin{bmatrix}
        \frac{\partial L}{\partial a_{1,1}} & \frac{\partial L}{\partial a_{1,2}} & \hdots & \frac{\partial L}{\partial a_{1,p}} \\
        \frac{\partial L}{\partial a_{2,1}} & \frac{\partial L}{\partial a_{2,2}} & \hdots & \frac{\partial L}{\partial a_{2,p}} \\
        \vdots & \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial a_{n,1}} & \frac{\partial L}{\partial a_{n,2}} & \hdots & \frac{\partial L}{\partial a_{n,p}}
    \end{bmatrix}
\end{equation}
\begin{center}
    $ = -X^T \cdot \frac{\partial L}{\partial B}$
\end{center}
Since the entries of $-X^T\cdot \frac{\partial L}{\partial B}$ matches $A$, Equation (31) is a good vectorized solution. Therefore:
\begin{center}
    $vectorized(-\frac{\partial L}{\partial \mathbf{b}}^{T}  J_{\widehat{g}}) = -X^T\cdot \frac{\partial L}{\partial B}$
\end{center}
Now if we plug Equation (31) into our GCDE special case, and if we define:
\begin{center}
    $\frac{\partial L}{\partial W} = roll(\frac{\partial L}{\partial \mathbf{w}})=\begin{bmatrix}
        \frac{\partial L}{\partial w_{1,1}} & \frac{\partial L}{\partial w_{1,2}} & \hdots &\frac{\partial L}{\partial w_{1,q} } \\
        \frac{\partial L}{\partial w_{2,1}} & \ddots & \hdots &\vdots \\
        \vdots & \vdots & \ddots & \vdots\\
        \frac{\partial L}{\partial w_{m,1}} & \hdots & \hdots &\frac{\partial L}{\partial w_{m,q}}
    \end{bmatrix}$
\end{center}
Then the vectorized Equation (23) is:
\begin{equation}
 vectorized(-\frac{\partial L}{\partial \mathbf{w}}^{T}  \frac{\partial \widehat{f}(\mathbf{h}(t), A, W)}{\partial \mathbf{w}}) =
    -(AH(t))^T \cdot (\frac{\partial L}{\partial W} \odot step(H(t)))
\end{equation}
\subsection{Summary}
In conclusion, we have found the vectorized adjoint dynamics for GCDE. Using the vectorized adjoint dynamics, we do not need to unroll the hidden states and parameters, and we could take the advantages of the in-memory matrices programmed on memristor crossbar.
\section*{Acknowledgement}
I thank Louis Primeau for his thorough tutorial on multivariable calculus and matrix unrolling for Jacobian calculation. I thank Nafiseh Ghoroghchian for her guide on matrix differentiation and proofread of the work.
\bibliographystyle{alpha}
\bibliography{sample}
Neural ODE: https://arxiv.org/pdf/1806.07366.pdf \\
GCDE: https://arxiv.org/pdf/1911.07532.pdf \\
GCN: https://arxiv.org/pdf/1609.02907.pdf \\
Multivariable calculus crash course, Louis Primeau \\
Neural ODE for Memristor Crossbar, Louis Primeau
\end{document}


\subsection{Integer caching}

if we set x =2  we can either create 2 in memory, or simply point x to 2, which is already in memory

that means if we do x=2 y=2 they have the same pointer

can also cache some other common data values, eg empty lists

makes sense if pointer is smaller in memory than value