% Copyright 2018-2021 Melvin Eloy Irizarry-Gelpí
\setcounter{chapter}{2}
\chapter{Circuits in Series and Parallel}
%
In this experiment you will learn about circuits connected in series or in parallel.
%
\section{Preliminary}
%
A \textbf{resistor} is any component in an electric circuit with \textbf{electrical resistance}. The SI unit for resistance is the ohm:
\begin{equation}
	1 \ \text{ohm} = 1 \ \text{V/A}
\end{equation}
Ohm's law is a statement that relates the voltage $V$ across a resistor to the amount of current $I$ leaving (or equivalently, entering) a resistor:
\begin{equation}
	V = R I
\end{equation}
Here, $R$ is the amount of electrical resistance. Solving for $R$ gives
\begin{equation} \label{eq.03.ROhmLaw}
	R = \frac{V}{I}
\end{equation}
For \textbf{ohmic resistors}, the ratio of voltage and current is fixed (i.e. constant) and \textbf{always} gives the same value. This is \textbf{not true} for \textbf{non-ohmic resistors}.

An electric circuit can contain many components that are connected with wires. The electric properties of the circuit will depend on \textbf{how} these components are connected. You are going to look at two ways of connecting two components in a circuit to a battery: \textbf{series} and \textbf{parallel}.
%
\subsection{Circuits in Series}
%
A collection of two resistors in \textbf{series} can be replaced by an \textbf{equivalent resistor} with a resistance given by the sum of the individual resistances:
\begin{equation} \label{eq.03.RSeries}
	R_{\text{eq}} = R_{1} + R_{2}
\end{equation}
When two resistors are connected in \textbf{series} to a battery, the \textbf{current} $I_{0}$ leaving the battery is the \textbf{same} as the \textbf{current} $I_{1}$ leaving resistor 1, and also the same as the \textbf{current} $I_{2}$ leaving resistor 2:
\begin{equation} \label{eq.03.ISeries}
	I_{0} = I_{1} = I_{2}
\end{equation}
When two resistors are connected in \textbf{series} to a battery, the \textbf{voltage} $V_{0}$ across the battery is \textbf{split} between the \textbf{voltage} $V_{1}$ across resistor 1, and the \textbf{voltage} $V_{2}$ across resistor 2:
\begin{equation} \label{eq.03.VSeries}
	V_{0} = \left| V_{1} + V_{2} \right|
\end{equation}
The absolute value is needed because the voltage across each resistor is \textbf{negative} since it correspond to an \textbf{electric potential drop}.
%
\subsection{Circuits in Parallel}
%
A collection of two resistors in \textbf{parallel} can be replaced by an \textbf{equivalent resistor} with a resistance that satisfies the relation
\begin{equation}
	\frac{1}{R_{\text{eq}}} = \frac{1}{R_{1}} + \frac{1}{R_{2}}
\end{equation}
Solving for the equivalent resistance gives
\begin{equation} \label{eq.03.RParallel}
	R_{\text{eq}} = \frac{R_{1} R_{2}}{R_{1} + R_{2}}
\end{equation}
When two resistors are connected in \textbf{parallel} to a battery, the \textbf{current} $I_{0}$ leaving the battery is split between the \textbf{current} $I_{1}$ leaving resistor 1, and the \textbf{current} $I_{2}$ leaving resistor 2:
\begin{equation} \label{eq.03.IParallel}
	I_{0} = I_{1} + I_{2}
\end{equation}
When two resistors are connected in \textbf{parallel} to a battery, the \textbf{voltage} $V_{0}$ across the battery has the same magnitude as the \textbf{voltage} $V_{1}$ across resistor 1, and also the same magnitude as the \textbf{voltage} $V_{2}$ across resistor 2:
\begin{equation} \label{eq.03.VParallel}
	V_{0} = \left| V_{1} \right| = \left| V_{2} \right|
\end{equation}
Again, remember that the voltage across a resistor (measured in the direction along the current) is negative because it correspond to a drop in electric potential.
%
\subsection{Power}
%
The power $P$ input or output of a component in an electric circuit can be found by multiplying the voltage $V$ across that component, and the amount of current $I$ leaving that component:
\begin{equation}
	P = VI
\end{equation}
The SI unit for power is the \textbf{watt} (W). One watt is equivalent to one joule of energy consumed or generated per second:
\begin{equation}
	1 \ \text{W} = 1 \ \text{J/s}
\end{equation}
Since power equals voltage multiplied by current, you also have
\begin{equation}
	1 \ \text{W} = 1 \ \text{V {\textperiodcentered} A}
\end{equation}
%
\section{Experiment}
%
There are four experiments. The first two experiments involve using \textbf{two ohmic} resistors. The other two experiments use \textbf{two non-ohmic} resistors (light bulbs).
%
\subsection{Part 1: Resistors in Series}
%
In this part, you connect two \textbf{resistors} in \textbf{series} to a battery.
%
\subsection{Part 2: Resistors in Parallel}
%
In this part, you connect two \textbf{resistors} in \textbf{parallel} to a battery.
%
\subsection{Part 3: Light Bulbs in Series}
%
In this part, you connect two \textbf{light bulbs} in \textbf{series} to a battery.
%
\subsection{Part 4: Light Bulbs in Parallel}
%
In this part, you connect two \textbf{light bulbs} in \textbf{parallel} to a battery.
%
\section{Analysis}
%
Each run is a measurement of either \textbf{voltage} (electric potential difference between two points) or \textbf{current}. If the voltage and current sensors were zeroed correctly, then the measurement over time should correspond to the measurement of the desired quantity, and no offset or baseline needs to be taken into account. One way to determine a single value is to take the \textbf{time-average}.

Note that each \textbf{voltage measurement has three decimal figures}, and each \textbf{current measurement has four decimal figures}. You should round your time-averaged values appropriately.

In each part there are three components: one battery and two resistors (or two light bulbs). There are \textbf{six quantities} that describe the system:
\begin{itemize}
	\item Time-averaged voltage across the battery: $V_{0}$
	\item Time-averaged voltage across resistor/bulb 1: $V_{1}$
	\item Time-averaged voltage across resistor/bulb 2: $V_{2}$
	\item Time-averaged current leaving the battery: $I_{0}$
	\item Time-averaged current leaving resistor/bulb 1: $I_{1}$
	\item Time-averaged current leaving resistor/bulb 2: $I_{2}$
\end{itemize}
These six quantities satisfy certain mathematical relations. Note that $V_{1}$ and $V_{2}$ are measured to be negative.
%
\subsection{Part 1: Two Resistors in Series}
%
For the two \textbf{resistors in series}, you find the time average for all six quantities above and fill a table similar to Table \ref{table.03.resistors.series}.

You should check that relations Equation (\ref{eq.03.ISeries}) and Equation (\ref{eq.03.VSeries}) hold true. The percent difference between the labeled resistance and the prediction Equation (\ref{eq.03.ROhmLaw}) from Ohm's law should be very small. The resistance of the battery should be very close to the \textbf{series equivalent} resistance in Equation (\ref{eq.03.RSeries}).
%
\subsection{Part 2: Two Resistors in Parallel}
%
For the two \textbf{resistors in parallel}, you find the time average for all six quantities above and fill a table similar to Table \ref{table.03.resistors.parallel}.

You should check that relations Equation (\ref{eq.03.IParallel}) and Equation (\ref{eq.03.VParallel}) hold true. The percent difference between the labeled resistance and the prediction Equation (\ref{eq.03.ROhmLaw}) from Ohm's law should be very small. The resistance of the battery should be very close to the \textbf{parallel equivalent} resistance in Equation (\ref{eq.03.RParallel}).
%
\subsection{Part 3: Two Light Bulbs in Series}
%
For two \textbf{light bulbs in series}, you can find the time average for all six quantities mentioned above, and fill a table similar to Table \ref{table.03.bulbs.series}. Note that the fourth column consist of the ratio of voltage and current for each component. This is a quantity with units of resistance (ohms) but it does not correspond to the actual resistance of the light bulbs. As you will find, the light bulbs are not ohmic, and thus the ratio of the voltage across the light bulb and the current leaving the light bulb is not constant. However, you should find that the relations Equation (\ref{eq.03.ISeries}) and Equation (\ref{eq.03.VSeries}) again hold.
%
\subsection{Part 4: Two Light Bulbs in Parallel}
%
For two \textbf{light bulbs in parallel}, you can find the time average for all six quantities mentioned above, and table similar to Table \ref{table.03.bulbs.parallel}. Note that the fourth column consist of the ratio of voltage and current for each component. You can verify that the light bulbs are not ohmic by showing that the values in the fourth column in Tables \ref{table.03.bulbs.series} and \ref{table.03.bulbs.parallel} are very different. However, you should find that the relations Equation (\ref{eq.03.IParallel}) and Equation (\ref{eq.03.VParallel}) again hold.
%
\section{My Data}
%
For parts 1 and 2, I used \textbf{three} resistors, instead of two. The resistances are given by
\begin{align}
	R_{1} = 10 \text{ ohm} && R_{2} = 51 \text{ ohm} && R_{3} = 68 \text{ ohm}
\end{align}
The equivalent resistance in \textbf{series} with three resistors is given by
\begin{equation}
	R_{\text{eq}} = R_{1} + R_{2} + R_{3}
\end{equation}
In parallel, the equivalent resistant satisfies the following relation:
\begin{equation}
	\frac{1}{R_{\text{eq}}} = \frac{1}{R_{1}} + \frac{1}{R_{2}} + \frac{1}{R_{3}}
\end{equation}
Solving for the equivalent resistance in \textbf{parallel} gives
\begin{equation}
	R_{\text{eq}} = \frac{R_{1} R_{2} R_{3}}{R_{1} R_{2} + R_{1} R_{3} + R_{2} R_{3}}
\end{equation}
As you can see from my spreadsheet, the results are in good agreement with expectations.
%
\subsection{Part 1: Three Resistors in Series}
%
Table \ref{table.03.3resistors.series} has the results for three resistors in series. You can check that the values for the current in the third column are all very close. Indeed,
\begin{equation}
	I_{0} \approx I_{1} \approx I_{2} \approx I_{3}
\end{equation}
This is the three-resistor analog of Equation (\ref{eq.03.ISeries}). Also, you can check that
\begin{equation}
	| V_{1} + V_{2} + V_{3} | = 3.075 \text{ V}
\end{equation}
which is very close to the value of $V_{0}$ (the voltage across the battery). This is the three-resistor analog of Equation (\ref{eq.03.VSeries}).

The last column in Table \ref{table.03.3resistors.series} contains the power generated/consumed by the corresponding component. As you can see, the rate of energy generation for the battery is very close to the sum of the rate of consumption for each resistor. Note that, in \textbf{series}, the resistor with the \textbf{largest} resistance consumes the most energy.
%
\subsection{Part 2: Three Resistors in Parallel}
%
Table \ref{table.03.3resistors.parallel} has the results for three resistors in parallel. You can check that, ignoring the signs, the values for the voltage in the second column are all very close. Indeed,
\begin{equation}
	V_{0} \approx |V_{1}| \approx |V_{2}| \approx |V_{3}|
\end{equation}
This is the three-resistor analog of (\ref{eq.03.VParallel}). Also, you can check that
\begin{equation}
	I_{1} + I_{2} + I_{3}  = 0.3532 \text{ A}
\end{equation}
which is very close to the value of $I_{0}$ (the current leaving the battery). This is the three-resistor analog of Equation (\ref{eq.03.IParallel}).

The last column in Table \ref{table.03.3resistors.parallel} contains the power generated/consumed by the corresponding component. As you can see, the rate of energy generation for the battery is very close to the sum of the rate of consumption for each resistor. Note that, in \textbf{parallel}, the resistor with the \textbf{smallest} resistance consumes the most energy.

One last thing about resistors: proof that they \textbf{are ohmic} (at least for the amount of voltage and current that we used) is that the values in the fourth column in Tables \ref{table.03.3resistors.series} and \ref{table.03.3resistors.parallel} are consistent (except for the first value, because it corresponds to the equivalent resistance). That is, it does not matter if they are connected in series or parallel, the amount of current going through the resistor is proportional to the amount of voltage across the resistor.
%
\subsection{Part 3: Two Light Bulbs in Series}
%
The six runs in the text file \texttt{bulbs-serial.txt} are for two light bulbs connected in series.
\begin{itemize}
	\item Run 1: measurement of voltage across battery ($V_{0}$).
	\item Run 2: measurement of voltage across long light bulb ($V_{L}$).
	\item Run 3: measurement of voltage across round light bulb ($V_{R}$).
	\item Run 4: measurement of current leaving battery ($I_{0}$).
	\item Run 5: measurement of current leaving long light bulb ($I_{L}$).
	\item Run 6: measurement of current leaving round light bulb ($I_{R}$).
\end{itemize}
The \textbf{long} light bulb is connected \textbf{first}, and the \textbf{round} light bulb is connected \textbf{second}.
%
\subsection{Part 4: Two Light Bulbs in Parallel}
%
The six runs in the text file \texttt{bulbs-parallel.txt} are for two light bulbs connected in parallel.
\begin{itemize}
	\item Run 1: measurement of voltage across battery ($V_{0}$).
	\item Run 2: measurement of voltage across long light bulb ($V_{L}$).
	\item Run 3: measurement of voltage across round light bulb ($V_{R}$).
	\item Run 4: measurement of current leaving battery ($I_{0}$).
	\item Run 5: measurement of current leaving long light bulb ($I_{L}$).
	\item Run 6: measurement of current leaving round light bulb ($I_{R}$).
\end{itemize}
%
\section{Your Data}
%
If your data for resistors in series and/or parallel is not good, let me know via email and I will make data available to you.
%
% \newpage
% \section{Your Lab Report}
% %
% In your lab report you should include:
% \begin{enumerate}
% 	\item One table like Table \ref{table.03.resistors.series} for two resistors in series.
% 	\item One table like Table \ref{table.03.resistors.parallel} for two resistors in parallel.
% 	\item One table like Table \ref{table.03.bulbs.series} for two light bulbs in series.
% 	\item One table like Table \ref{table.03.bulbs.parallel} for two light bulbs in parallel.
% \end{enumerate}
% You should also:
% \begin{enumerate}
% 	\item Verify that equations (\ref{eq.03.ISeries}) and (\ref{eq.03.VSeries}) hold for two resistors, and also for two light bulbs, connected in series.
% 	\item Verify that the voltage across the battery divided by the current leaving the battery is close to (\ref{eq.03.RSeries}) for two resistors in series.
% 	\item Verify that equations (\ref{eq.03.IParallel}) and (\ref{eq.03.VParallel}) hold for two resistors, and also for two light bulbs, connected in parallel.
% 	\item Verify that the voltage across the battery divided by the current leaving the battery is close to (\ref{eq.03.RParallel}) for two resistors in parallel.
% 	\item Confirm that the long light bulb is non-ohmic by comparing the ratio of the voltage and the current $V_{1} / I_{1}$ in series and in parallel.
% 	\item Confirm that the round light bulb is non-ohmic by comparing the ratio of the voltage and the current $V_{2} / I_{2}$ in series and in parallel.
% 	\item Find which resistor consumes more power when two resistors are connected in series.
% 	\item Find which resistor consumes more power when two resistors are connected in parallel.
% 	\item Find which light bulb consumes more power when two light bulbs are connected in series.
% 	\item Find which light bulb consumes more power when two light bulbs are connected in parallel.
% \end{enumerate}
%
\newpage
\section{Tables}
%
\begin{table}[ht]
	\begin{center}
		\begin{tabular}{l|r|r|r}
            \textbf{Quantity} & \textbf{Battery} & \textbf{Resistor 1} & \textbf{Resistor 2} \\
            \hline
            Voltage (V) & $V_{0}$ & $V_{1}$ & $V_{2}$ \\
            Current (A) & $I_{0}$ & $I_{1}$ & $I_{2}$ \\
            \hline
            Observed Resistance (ohm) & $V_{0} / I_{0}$ & $V_{1} / I_{1}$ & $V_{2} / I_{2}$ \\
            Expected Resistance (ohm) & $R_{1} + R_{2}$ & $R_{1}$ & $R_{2}$ \\
            Percent Difference (\%) & & & \\
            \hline
            Power (W) & $V_{0} I_{0}$ & $V_{1} I_{1}$ & $V_{2} I_{2}$ \\
            \hline
        \end{tabular}
	\end{center}
	\caption{Time-averages and resistance for two resistors in series.}
	\label{table.03.resistors.series}
\end{table}
%
\begin{table}[ht]
	\begin{center}
		\begin{tabular}{l|r|r|r}
            \textbf{Quantity} & \textbf{Battery} & \textbf{Resistor 1} & \textbf{Resistor 2} \\
            \hline
            Voltage (V) & $V_{0}$ & $V_{1}$ & $V_{2}$ \\
            Current (A) & $I_{0}$ & $I_{1}$ & $I_{2}$ \\
            \hline
            Observed Resistance (ohm) & $V_{0} / I_{0}$ & $V_{1} / I_{1}$ & $V_{2} / I_{2}$ \\
            Expected Resistance (ohm) & $R_{1} R_{2} / (R_{1} + R_{2})$ & $R_{1}$ & $R_{2}$ \\
            Percent Difference (\%) & & & \\
            \hline
            Power (W) & $V_{0} I_{0}$ & $V_{1} I_{1}$ & $V_{2} I_{2}$ \\
            \hline
        \end{tabular}
	\end{center}
	\caption{Time-averages and resistance for two resistors in parallel.}
	\label{table.03.resistors.parallel}
\end{table}
%
\begin{table}[ht]
	\begin{center}
		\begin{tabular}{l|r|r|r}
            \textbf{Quantity} & \textbf{Battery} & \textbf{Long bulb} & \textbf{Round bulb} \\
            \hline
            Voltage (V) & $V_{0}$ & $V_{L}$ & $V_{R}$ \\
            Current (A) & $I_{0}$ & $I_{L}$ & $I_{R}$ \\
            \hline
            Observed Resistance (ohm) & $V_{0} / I_{0}$ & $V_{L} / I_{L}$ & $V_{R} / I_{R}$ \\
            Power (W) & $V_{0} I_{0}$ & $V_{L} I_{L}$ & $V_{R} I_{R}$ \\
            \hline
        \end{tabular}
	\end{center}
	\caption{Time-averages and power for two light bulbs in series.}
	\label{table.03.bulbs.series}
\end{table}
%
\begin{table}[ht]
	\begin{center}
		\begin{tabular}{l|r|r|r}
            \textbf{Quantity} & \textbf{Battery} & \textbf{Long bulb} & \textbf{Round bulb} \\
            \hline
            Voltage (V) & $V_{0}$ & $V_{L}$ & $V_{R}$ \\
            Current (A) & $I_{0}$ & $I_{L}$ & $I_{R}$ \\
            \hline
            Observed Resistance (ohm) & $V_{0} / I_{0}$ & $V_{L} / I_{L}$ & $V_{R} / I_{R}$ \\
            Power (W) & $V_{0} I_{0}$ & $V_{L} I_{L}$ & $V_{R} I_{R}$ \\
            \hline
        \end{tabular}
	\end{center}
	\caption{Time-averages and power for two light bulbs in parallel.}
	\label{table.03.bulbs.parallel}
\end{table}
%
\begin{table}[ht]
	\begin{center}
		\begin{tabular}{l|r|r|r|r}
			\textbf{Quantity} & \textbf{Battery} & \textbf{Resistor 1} & \textbf{Resistor 2} & \textbf{Resistor 3} \\
			\hline
			Voltage (V) & 4.430 & \textminus 0.338 & \textminus 1.757 & \textminus 2.345 \\
			Current (A) & 0.0346 & 0.0346 & 0.0347 & 0.0347 \\
			\hline
			Observed Resistance (ohm) & 128.05 & 9.76 & 50.70 & 67.68 \\
			Expected Resistance (ohm) & 129 & 10 & 51 & 68 \\
			Percent Difference (\%) & \textminus 0.73 & \textminus 2.36 & \textminus 0.58 & \textminus 0.46 \\
			\hline
			Power (W) & 0.15 & \textminus 0.01 & \textminus 0.06 & \textminus 0.08 \\
			\hline
		\end{tabular}
	\end{center}
	\caption{Time-averages and resistance for three resistors in series.}
	\label{table.03.3resistors.series}
\end{table}
%
\begin{table}[ht]
	\begin{center}
		\begin{tabular}{l|r|r|r|r}
			\textbf{Quantity} & \textbf{Battery} & \textbf{Resistor 1} & \textbf{Resistor 2} & \textbf{Resistor 3} \\
			\hline
			Voltage (V) & 3.731 & \textminus 3.707 & \textminus 3.736 & \textminus 3.734 \\
			Current (A) & 0.4904 & 0.3722 & 0.0739 & 0.0554 \\
			\hline
			Observed Resistance (ohm) & 7.61 & 9.96 & 50.55 & 67.37 \\
			Expected Resistance (ohm) & 7.45 & 10 & 51 & 68 \\
			Percent Difference (\%) & 2.20 & \textminus 0.39 & \textminus 0.88 & \textminus 0.92 \\
			\hline
			Power (W) & 1.83 & \textminus 1.38 & \textminus 0.28 & \textminus 0.21 \\
			\hline
		\end{tabular}
	\end{center}
	\caption{Time-averages and resistance for three resistors in parallel.}
	\label{table.03.3resistors.parallel}
\end{table}
%
\FloatBarrier
\newpage
\section{Figures}
%
\begin{figure}[ht]
	\centering
	\includegraphics[scale=0.74]{image/03-serial-parallel/resistor-V.pdf}
	\caption{Voltage across ohmic resistor}
	\label{figure.03.resistor.v}
\end{figure}
%
\begin{figure}[ht]
	\centering
	\includegraphics[scale=0.74]{image/03-serial-parallel/resistor-I.pdf}
	\caption{Current through ohmic resistor}
	\label{figure.03.resistor.i}
\end{figure}
%
\begin{figure}[ht]
	\centering
	\includegraphics[scale=0.74]{image/03-serial-parallel/bulb-V.pdf}
	\caption{Voltage across non-ohmic light bulb}
	\label{figure.03.bulb.v}
\end{figure}
%
\begin{figure}[ht]
	\centering
	\includegraphics[scale=0.74]{image/03-serial-parallel/bulb-I.pdf}
	\caption{Current through non-ohmic light bulb}
	\label{figure.03.bulb.i}
\end{figure}


\subsection{Theoretical Results}

\hogwild's original article \cite{2011NRRW} manages to prove that the
asynchronous noise generated by \hogwild, under certain sparsity assumptions of
the $f_i$'s, converged at the same rate as seen in the stochastic gradient
method. However, a newer article from 2015 by De Sa et al. \cite{2015SZOR}
presented a new framework in the context of martingale theory, which drops said
sparsity assumptions and generalizes to certain non-convex formualations. Given
that I just learned about martingale theory essentially two weeks ago in Basic
Probability, and that much of my synthetic tests are on dense datasets, I feel
compelled to present this argument instead.

The following is a cleaned up summary of the arguments presented in the article,
except I focus on just demonstrating how martingale theory can be used to
generalize convergence arguments for a sequential stochastic gradient method to
the asynchronous case, and cut out the generalization required for the
non-convex situation.

\subsubsection{Initial Machinery}

First, recall the definition of a martingale:
\begin{definition} \label{def:martingale}
  \cite{2004JP} A sequence of random variables $(X_n)_{n \geq 0}$ is called
  a martingale, or an $(\mathcal{F}_n)$-martingale, if
  \begin{enumerate}[(i)]
    \item $\E{|X_n|} < \infty, \forall n$.
    \item $X_n$ is $\mathcal{F}_n$ measurable, $\forall n$.
    \item $\E{X_n \mid \mathcal{F}_m} = X_m$ a.s., $\forall m \leq n$.
  \end{enumerate}
  furthermore a supermartingale (submartingale) is one satisfying (i), (ii)
  exactly, and (iii) with $\leq\ (\geq)$ instead.
\end{definition}
Using this, the idea is to model our convergence with a non-negative
supermartingale $W_t(x_t, \dots, x_0)$ which is a function of the previous
stochastic gradient iterates. These $W_t$, when used in the theory later, will
be associated with specific stochastic algorithms, as an example below we'll see
it applied to serial stochastic gradient. However, given such a supermartingale,
and given a bounded stopping time $B$ (in literature known as a horizon), if our
stochastic gradient iterates are written as $x_{t+1} = x_t - \eta \nabla
f_t(x_t)$, where $\eta$ is the learning rate, and $f_t$ denotes the random
function chosen at time step $t$, then we see that condition (iii) in the above
definition implies that:
\[
  W_{t+1}(x_t - \eta \nabla f_t(x_t), x_t, \dots, x_0)
  \leq
  W_t(x_t, \dots, x_0), \forall t \leq B
\]
which certainly makes sense if our $-\nabla f_t(x_t)$ is a sufficient search
direction. Furthermore, letting our success region be denoted as $S
= B_\e(x^*)$, where $x^*$ is the minimizer of our optimization problem, if we
impose that if $x_t \not\in S, \forall t \leq T$ then:
\[
  W_T(x_T, \dots, x_0) \geq T
\]
then we call $W_t$ a {\it rate supermartingale}. To simplify notation, let $F_t$
be the event where $\nexists t \leq T$ such that $x_t \in S$.

A good example of the power of this machinery is proving a convergence bound on
the serial version of stochastic gradient: using (iii) of definition
\ref{def:martingale}, one can see that considering $F_T$:
\[
  \E{W_0(x_0)}
  \underbrace{\geq}
  \E{W_T}
  =
  \Prob{F_t}
  \underbrace{\Prob{F_T}\E{W_T | F_T}}_{W_T \geq T} +
  \underbrace{\Prob{F_T^c}\E{W_T | F_T^c}}_{W_T \geq 0}
  \geq
  \Prob{F_T}T
\]
where the first inequality is by Doob's optional sampling theorem. Thus for
a simple serial stochastic gradient method: $\Prob{F_t} = \E{W_0} / T$.

Now that we can characterize serial stochastic gradient in this model, we need
a method to analyze asynchronous noise. Recall that since we've guaranteed in
the description of \hogwild\ that writes to the iteration variable $x_t$ are
done atomically, the only race condition possible is when updates to entries of
$x_t$ are interleaved with either the other thread's updates or its reads on
$x_t$.

Indeed, when going to update the $i$th component of $x_{t+1}$, the variable used
to compute the gradient may have long since changed, making our iteration look
more like $x_{t+1} = x_t - \nabla f_t(v_t)$ where $v_t$'s entries were the
entries of some previous iterate $x$. Let $\tau_{i,t}$ denote the lag for the
update of the $i$th component of $x_{t+1}$, i.e. $(v_t)_i = ( x_{t-\tau_{i,t}}
)_i$. Then we recognize that this lag for each component $i$ (supposing the
computer hasn't crashed) must be bounded; let $\tau'$ be the the maximum over
$i$ of such bounds and let $\tau = \E{\tau'}$. This is known in literature as
the {\it worst-case expected delay}.

Finally, we need one last definition, mainly one of convinience, to proceed
onward. This describes the main conditions upon a rate supermartingale necessary
to prove that the asynchronous noise error is irrelevant to the convergence
rate.
\begin{definition}
  An algorithm with associated rate supermartingale $W$ is $(H, R, \xi)$-bounded
  if the following conditions hold.
  \begin{enumerate}[(1)]
    \item $W$ must be Lipschitz continuous in the current iterate with parameter
      $H$, i.e.
      \[
        \norm{W_t(u, x_{t-1}, \dots, x_0) - W_t(v, x_{t-1}, \dots, x_0)}
        \leq
        H\norm{u-v}, \forall t, u, v, x_t, \dots, x_0.
      \]
    \item $\nabla f$ must be Lipschitz continuous in expectation with parameter
      $R$, i.e.
      \[
        \E{||\nabla f(x) - \nabla f(y)||} \leq R\norm{u-v}
      \]
    \item The expected magnitude of the update must be bounded by $\xi$, i.e.
      \[
        \E{||\nabla f(x)||} \leq \xi
      \]
  \end{enumerate}
  Note that these look very familiar to the conditions in the stochastic
  gradient method convergence theorem (theorem \ref{thm:sgd}).
\end{definition}

\subsubsection{Convergence of \hogwild}

Finally, now that all of the machinery has been defined, we can get to the main
result:
\begin{theorem} \label{thm:convergence}
  Suppose we have an asynchronous stochastic algorithm with associated rate
  supermartingale $W$ which is $(H,R,\xi)$ bounded with horizon $B$.
  Furthermore, assume that $HR\xi \tau < 1$; then $\forall T \leq B$:
  \[
    \Prob{F_T} \leq \frac{\E{W(0, x_0)}}{(1-HR\xi\tau)T}
  \]
\end{theorem}
Before we prove it, we briefly discuss how to use this theorem in practice.
Suppose we have a loss function $f$ which we want to use \hogwild\ on to
minimize. Then first we need to obtain a rate supermartingale proving the
problem%
\footnote{
  In the article, De Sa et al. describes this as being no more difficult than
  proving serial convergence, but the proof (in the convex case) doesn't seem to
  have any similarity between it and serial convergence, so I can't validate
  this claim.
}, determine $H, R, \xi$ such that $W$ is $(H,R,\xi)$-bounded, and then apply
this theorem to get a proper rate of convergence. This is a very powerful
theorem, as we'll be able to state later, given strongly convex $f$ and with the
other required bounds, we can blanket prove that \hogwild\ works on them.
Furthermore, for nicer non-convex problems, such as the low-rank least-squared
matrix completion problem presented in the paper, it's easy to derive a proper
rate supermartingale as well.

Now, with respect to the proof, we only outline it because it's mainly just
a repeated application of the above bounds we have in order to lower bound away
noise terms, and then it follows the path outlined in the serial stochastic
gradient method proof.
\begin{proof}
  \begin{enumerate}[(i)]
    \item With $W$ defined exactly as in the serial case, it's not a rate
      supermartingale. Instead, from it we construct rate supermartingale $V_t$,
      where $\forall t, x$ where $x_u$ not converged $\forall u < t$:
      \[
        V_t(x_t, \dots, x_0) = W_t(x_t, \dots, x_0)
        - \underbrace{HR\xi \tau t}_{(1)}
        + \underbrace{
          HR \sum_{k=1}^\infty \norm{x_{t-k+1}-x_{t-k}} \sum_{m=k}^\infty
          \Prob{\tau' \geq m}
        }_{(2)}
      \]
      where (1) allows for longer iteration counts (as HOGWILD needs to allow
      for given noise corruption), and (2) measures distance between recent
      iterates. Otherwise if $x_u$ is converged, then we let $V_t(x_t, \dots,
      x_0) = V_u(x_u, \dots, x_0)$. Basically, we've defined $V$ a stopped
      process.
    \item Show $V_t$ is a rate supermartingale for \hogwild
    \item Using a similar process to the serial stochastic gradient proof, show
      that $\E{V_T} \leq \E{V_0} = \E{W_0}$, then using the law of total
      expectation on this with $F_T$, and recalling that $\E{W_T | F_T} \geq T$,
      we recieve the desired result.
  \end{enumerate}
\end{proof}
Given this, the general theorem for the convex case is:
\begin{theorem} \label{thm:convexconv}
  Consider trying to minimize $f$, which is:
  \begin{itemize}
    \item Strongly convex with parameter $c$.
    \item $\nabla f_k$ Continuously differentiable in $||\cdot||_1$ with
      Lipschitz constant $L$.
    \item Upper bounded second moment of the gradient by $M^2$
    \item Success criteria $\norm{x-x^*}^2 \leq \e$, for some $\e > 0$.
  \end{itemize}
  Then we can construct rate supermartingale $W_t$ such that it's $(H,R,\xi)$
  bounded with $H = 2\sqrt{\e}(2\eta c\e - \eta^2 M^2)^{-1}, R = \eta L,
  \xi = \eta M$. Then choosing step size $\eta$ (for some $\nu \in (0,1)$):
  \[
    \eta = \frac{c\e \nu}{M^2 + 2LM\tau\sqrt{\e}}
  \]
  then we recieve:
  \[
    \Prob{F_T} \leq \frac{M^2 + 2LM\tau\sqrt{\e}}{c^2\e\nu T} \log
    \left(
      e \norm{x_0 -x^*}^2 \e^{-1}
    \right)
  \]
\end{theorem}