\documentclass[12pt]{article}

\usepackage[option1, option2]{package}

\begin{document}

% Here is a comment. Commands in tex are called by backslash: \command
\section{Section head}
\subsection{Subsection head}

This is text, and the following is in-line mathematics:
$\mathrm{e}^{-\pi\matrhm{i}} = -1$.
A display-style equation is shown below:

\begin{equation}
  E = mc^2
\end{equation}

\end{document}


\newpage
\chapter{Plug flow reactor}

\section{Introduction}
The plug flow reactor model of Camflow simulates a one dimensional plug flow reactor with gas-phase chemistry. The model can handle a number of temperature conditions such as isothermal, non-isothermal, or user defined temperature profiles.

\section{Fundamentals}
The plug flow reactor model solves the governing equations for continuity
\begin{equation}
 \frac{\rho u A_c}{dz} = 0,
\end{equation}
species continuity
\begin{equation}
 \rho u A_c\frac{dY_k}{dz} = W_kA_c\dot{\omega}_k \quad k=1\ldots K_g,
\end{equation}
energy equation
\begin{equation}
 \rho u A_c\frac{c_pdT}{dz} + \sum_{k=1}^{K_g} \dot{\omega}_kh_kW_kA_c = UA_s(T_w-T),
\end{equation}
and the equation of state
\begin{equation}
 p\bar{W}=\rho R T.
\end{equation}
In the above equations, $\rho$ is the density in kg/m$^3$, $u$ is the velocity in m/s, $A_c$ is the area of cross section in m$^2$, $Y_k$ is the mass fraction of the k\'th chemical species, $W_k$ is the molecular mass in kg/mol of the k\'th chemical species, $\dot{\omega}_k$ is the molar production rate in mol/m$^3$-s of the k\'th chemical species, $c_p$ the specific heat at constant pressure in J/mol-K, $T$ the temperature in K, $T_w$ is the temperature of the wall in K, $A_s$ is the surface area per unit volume, and $U$ is the over all heat transfer coefficient in J/m$^2$sK.

\section{Input file}
An example of ``camflow.xml'' is shown below
{\scriptsize{\begin{verbatim}
<?xml version="1.0" encoding="ISO-8859-1"?>
<camflow>
   <reactor model="plug">
    <diameter unit="m">0.015</diameter>
    <length unit="cm">5</length>
  </reactor>
  <op_condition>
     <step_ignite>10</step_ignite>
     <temperature>isothermal</temperature>
     <twall unit="K">1073</twall>
    <pressure unit="Pa">1e5</pressure>
  </op_condition>
  <inlet>
     <fuel>
       <velocity unit="m/s">0.1</velocity>
       <temperature unit="C">800</temperature>
       <!--flowrate unit="cgs">4.63e-3</flowrate-->
       <molefrac>
        <species name="NO2">0.1</species>
        <species name="N2">*</species>
       </molefrac>
     </fuel>
  </inlet>
  <solver mode="coupled" solver="cvode">
     <tols>
        <species>
           <aTol>1.e-06</aTol>
           <rTol>1.e-06</rTol>
        </species>
        <temperature>
           <aTol>1.e-03</aTol>
           <rTol>1.e-03</rTol>
        </temperature>
        <flow>
           <aTol>1.e-03</aTol>
           <rTol>1.e-03</rTol>
        </flow>
     </tols>
  </solver>
  <initialize>
    <Tprofile unit_L="cm" unit_T="K">
      <position x="0.0">373.7</position>
      <position x="0.125">484.5</position>
      <position x="0.25">583.7</position>
      <position x="0.375">672.2</position>
      <position x="0.5">753.5</position>
      <position x="0.75">901.4</position>
      <position x="1.0">1027.0</position>
      <position x="1.25">1120.0</position>
      <position x="1.5">1184.0</position>
      <position x="2.0">1260.0</position>
      <position x="3.0">1348.0</position>
      <position x="6.0">1475</position>
      <position x="10.0">1524.0</position>
    </Tprofile>
 </initialize>
 <report species="mole">
 </report>
</camflow>

\end{verbatim}}

}

The input file follows xml specification with a number of child elements. Each child element is described in detail below.

\begin{itemize}
 \item \textbf{rector} : The reactor element specifies which reactor models is to be simulated and for a plug flow reactor camflow expects plug as the model attribute value. The reactor element also holds child element for specifying reactor diameter and the reactor length, and each child element is given with the unit attribute. The unit of the value specified can be in ``cm'', ``m'', or in ``in''ches. Appropriate attribute must be specified.

\item \textbf{op\_conditions} : The element op\_conditions describes the operating conditions for the plug flow reactor. This includes the specification of the pressure and the condition applied to the solution of energy equation. The reactor pressure may be specified in the units of ``Pa'', ``atm'', or ``bar''. The temperature unit can be either in ``K'' or in ``C''. The temperature element can take the values of ``isothermal'', ``adiabatic'', ``userdefined'', or ``nonisothermal''. In the case of isothermal calculation, the energy equation is not solved and the reactor is assumed to be at the same temperature as the incoming fuel. The user may also perform the integration for a pre-calculated or measured temperature profile. In this case the temperature child element must be assigned with the value ``userdefined'' and the user defined temperature profile cane specified (explained later). For adiabatic calculations, provide the temperature element with the value ``adiabatic''. Radiation heat losses from the reactor are completely neglected. For non-isothermal calculation it is mandatory to specify the reactor wall temperature. The heat transfer coefficient is calculated internally as a function of reactor position using the following correlation.
\begin{equation}
 Nu= \frac{hD}{k},
\end{equation}
where $Nu$ is the Nusselt number, $h$ the heat transfer coefficient, $D$ the diameter and $k$ the thermal conductivity. The Nusselt number is defined as
\begin{equation}
 Nu = 3.657+8.827\bigg(\frac{1000}{\mathrm{Gz}}\bigg)^{-0.545} \exp\bigg(\frac{-48.2}{\mathrm{Gz}}\bigg),
\end{equation}
and the Greatz number Gz is defined as
\begin{equation}
 \mathrm{Gz} = \frac{D Re Pr}{z}
\end{equation}
with D the diameter, $Re$ the Reynolds\'s number, $Pr$ the Prandtl number, and $z$ the axial position of the reactor. In this version, the overall heat transfer coefficient is replaced with the heat transfer coefficient calculated from Nusselt number. Strictly speaking the correlation presented above are valid only for a non-reacting multi-component gas mixture. However, the above correlations are used for reacting case as well due to the non-availability of better formulations.\\

\textbf{step\_ignite} is an option to evaluate the minimum temperature required to ignite the gas-mixture and is optional. However, if the user is not interested in the ignition temperature, this element should not be present in the inputfile. If its present then \textbf{step\_ignite} must specify the step for temperature increment, and the \textbf{temprature} specification must be ``adiabatic''. The ignition temperature calculated is not printed to the output file, rather only a screen output is generated.\\

\item \textbf{inlet} : The inlet element holds the information on reactants and the reactant temperature at axial position at z=0, and the flow rate or velocity at z=0; Either the velocity or the flow rate needs to be specified. The velocity may be specified in m/s or in cm/s, while the flow rate may be specified in ``cgs'' units or in ``si'' units. The temperature of the reactants must be specified using the temperature element with the appropriate units. The mass or mole fraction of the reactant species need to be specified within the element molefrac or massfrac. The sum of mass fractions or mole fraction of the reactant species must sum up to 1. Instead of specifying the mass/mole fractions of all species, the last species can be assigned with *. In this case the mole/mass fraction of the last species will be 1-sum of others.

\item \textbf{solver}: The solver element holds the solver control specifications. The attributes ``mode'' should always be specified as ``coupled'' for plug flow reactor simulation. The solver name is essentially provided to switch from one solver to another. However, the present version of Camflow uses only CVode as the numerical integrator, and therefore accepts only ``cvode'' as the solver name. The element ``tols'' hold the various tolarences that can be applied to the species, energy, and continuity equations. For species a relative tolarence of at least 10$^{-6}$ should be used. The user may need to adjust the tolarence values for the species in case of solution difficulties.

\item \textbf{initialize} The initialize element can be used to specify various initial conditions. However, for plug flow reactor model, the only initial property that can be specified is the user defined temperature profile. The temperature profile can be specified by using the ``Tprofile'' element with two attributes namely ``unit\_L'' for length unit and ``unit\_T'' for temperature unit. The length unit can be in ``cm'' or in ``m'', where as the temperature unit can be either in ``K'' or in ``C''. The actual temperature as a function of reactor position is specified with the child elements position with the attribute ``x'', which stands for the position with the reactor. If the length unit is specified as ``cm'' then ``x'' is the position from the reactor inlet in ``cm'', and the value for the position element is the temperature at position ``x''.

\item \textbf{report}: The desired output for the species composition must be specified in this element using the species attribute. ``mole'' or ``mass'' may be used as the attribute values, and correspondingly the output will be produced either in mole fraction or mass fractions.

\end{itemize}

\section{Executing the binary}
The plug reactor model of Camflow expects three input files namely, ``camflow.xml'', ``therm.dat'',  ``chem.inp''. For performing non-isothermal calculation an additional file ``tran.dat'' specifying the transport data of all chemical species present in the system need to be specified. All these files must be present in the working directory. Upon succesful execution the output file ``profile.dat'' containing the axial position (m), density (kg/m$^3$), velocity (m/s), massflow rate (kg/m$^2$s), residence time (1/s) temperature (K), and the species compositions in mass or mole fractions.

\section{Results}
The following figure shows the species profiles and temperature for Hydrogen oxidation reaction
\begin{figure*}[h]
 \centering
\includegraphics[scale=0.6]{plug_profile.eps}
\caption{Species profiles hydrogen oxidation with user defined temperature profile}
\end{figure*}

%===============================================================================================
%
%
%
%===============================================================================================


\subsection{Cross-price elasticity of demand}

We have our Marshallian demand function:

\(x_i=x_{di}(I, \mathbf p)\)

The derivative of this with respect to price is the additional amount consumed after prices increase.

\(\dfrac{\delta }{p_i}x_{di}(I, \mathbf p)\)

\subsection{Complements}

\subsection{Substitutes}


\documentclass{article}
\usepackage{graphicx}
\usepackage[margin=2cm]{geometry}
\usepackage[english]{babel}
\usepackage{float}
\usepackage{amsmath}
\usepackage{cite}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{hyperref}
\usepackage{multicol}
\usepackage{parskip}
\title{Context Attacks on CBSTM-IoT}
\author{Cody Lewis}
\date{\today}

\begin{document}
    \maketitle

    We implemented a simulation of the trust model proposed in \cite{rafey2016cbstm},
    this trust model managed to mitigate the context attacks.

    \section{Configurations of the Simulation}
    Our simulation contained 100 nodes where 30 were adversaries, that is, to
    show the effects of the attack while still below the Byzantine threshold
    \cite{lamport2019byzantine}. Friendships between nodes was determined
    randomly such that the average amount of friends that each node had was
    approximately 50, then $frlow$ and $frhigh$ were calculated as a standard
    deviation below and above that mean respectively.

    We defined the relationship factor with the following function,
    \begin{equation}
        R = \frac{1}{2^n}
    \end{equation}
    where $n$ is the degree of separation between nodes $i$ and $k$. Where
    nodes belonging to the same owner have $0$ degrees of separation, and
    those belonging to different owners have $1$ degree of separation.
    Owners have at most 7 nodes belonging to them.

    We set the computing power, $CP$, to $1$ in order to maximize malevolent abilities of
    the adversaries, also to show this trust model's greatest amount of resistance
    against the attacks.

    We decided that in the case where the sum of direct trust and indirect trust was greater
    than 1, that only direct trust was used, resulting in following trust weight factors
    to be calculated as,
    \begin{equation}
        \alpha = \begin{cases}
            1 & : DT + IndT > 1 \\
            \frac{DT}{DT + IndT} & : 0 < DT + IndT \leq 1 \\
            0 & : DT + IndT = 0
        \end{cases}
    \end{equation}
    and,
    \begin{equation}
        \beta = 1 - \alpha
    \end{equation}

    where the trust of node $j$ calculated by node $i$ under context $c$ is,
    \begin{equation}
        T_{i,j}^c = \alpha \cdot DT_{i,j} + \beta \cdot IndT_{i,j}^c,
    \end{equation}
    such that $DT$ is the aggregation of direct trusts for all contexts, and
    $IndT$ is the aggregation of indirect trusts under context $c$ for recommendations
    from each other node.

    The effects of this can be seen Figure \ref{fig:trust_plot}, where after approximately 20
    transactions, the trust jumps from using a combination of direct and indirect to
    only using direct.

    Nodes in this simulation were implemented to act benevolent with any context
    value less than or equal to the their randomly assigned context potential,
    under contexts above that they will act malevolent. A context, $c_i$ is less than or
    equal to another context, $c_j$, iff $i \leq j$. The context setting
    adversaries attack by always reporting to the other nodes with a bad
    mouthed recommendation with the target context.

    \begin{table}
        \begin{tabularx}{\textwidth}{X X}
            \toprule
            \textbf{Parameter} & \textbf{Value} \\
            \midrule
            $w$ & $\{0.25, 0.25, 0.25, 0.25\}$ \\
            \midrule
            $w_h$ & $0.50$ \\
            \midrule
            $w_d$ & $0.50$ \\
            \midrule
            $R$ & $0.5$ \\
            \midrule
            $CP$ & $1$ \\
            \midrule
            $fb_{max}$ & $10$ \\
            \midrule
            $MD$ & $0.1$ \\
            \midrule
            Contexts & $\{c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9, c_{10}\}$ \\
            \midrule
            Total nodes & $ 100 $ \\
            \midrule
            Transactions & $ 50 $ \\
            \midrule
            Adversaries & $ 30\% $ \\
            \bottomrule
        \end{tabularx}
        \caption{The parameters of CBSTM IoT trust model implementation}
        \label{table:sim-params}
    \end{table}

    \newpage

    \section{Results}
    This trust model was resistant to the context attacks, due to the split of direct and
    indirect trusts, where direct trust is calculated as an aggregation of
    experiences in all contexts. Another factor that allowed for the mitigation
    of context attacks was the filtration of recommendations based on context,
    where the contexts themselves do not have any other effect on resulting
    trust calculation. Since the contexts were only used for the filtration of
    recommendations, standard bad mouthing mitigation techniques were effective
    against the context setting attack. The comparison of the context setting
    attack to a completely benevolent network is shown in Figure \ref{fig:trust_plot}, where the red
    line is the trust values where 30\% of network are context setting adversaries, and
    the blue line is when there are no adversaries in the network.

    \begin{figure}[H]
        \centering
        \includegraphics[width=\textwidth]{../1-trust-eval-on-8-in-context-3.png}
        \caption{Comparison of the context setting attack to a benevolent network}
        \label{fig:trust_plot}
    \end{figure}

    \bibliographystyle{plain}
    \bibliography{CBSTM-IoT}
\end{document}


\documentclass[letterpaper,11pt]{article}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage[hmargin=1.25in,vmargin=1in]{geometry}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{lmodern}
\usepackage{microtype}
\usepackage{minted}

\DeclareMathOperator*{\argmin}{arg\,min}

\author{Philip Pham}
\date{\today}
\title{CSE 547 - Assignment 1}

\begin{document}
\maketitle

\section*{Problem 0}

\begin{description}
\item[List of collaborators:] I have not collaborated with anyone.
\item[List of acknowledgements:] None.
\item[Certify that you have read the instructions:] Yes.
\item[Terms and Conditions to use the dataset:] I accept the terms and
  conditions to use the COCO dataset.
\end{description}

\section*{Problem 1}

Read the course website, up until ``Lecture Notes and Readings'', so that you
understand the course policies on grading, late policies, projects, requirements
to pass etc. Write ``I have read and understood these policies'' to certify
this. If you have questions, please contact the instructors.

\subsection*{Solution}

I have read and understood these policies.

\section*{Problem 2}

Consider the function from class (and the notes):
\begin{equation}
  f\left(w_1,w_2\right) = \left[\sin\left(2\pi\frac{w_1}{w_2}\right) + 3\frac{w_1}{w_2} - \exp\left(2w_2\right)\right]\left[3\frac{w_1}{w_2} - \exp\left(2w_2\right)\right].
\end{equation}

Suppose our program for this function uses the following evaluation trace:

\begin{description}
\item[input:] $z_0 = \left(w_1,w_2\right)$
  \begin{enumerate}
  \item $z_1 = w_1/w_2$
  \item $z_2 = \sin\left(2\pi z_1\right)$
  \item $z_3 = \exp\left(2w_2\right)$
  \item $z_4 = 3z_1 - z_3$
  \item $z_5 = z_2 + z_4$
  \item $z_6 = z_4z_5$
  \end{enumerate}
\item[return:] $z_6$
\end{description}

\subsection*{The Forward Mode of AutoDiff (AD)}

The forward mode for auto-differentiation is a conceptually simpler
way to compute the derivative. Let us examine the forward mode to
compute the derivative of one variable, $\frac{df}{dw_1}$. In the
forward mode, we sequentially compute both $z_t$ and its derviative
$\frac{dz_t}{dw_1}$ using the previous variables $z_1,\ldots,z_{t-1}$
and the previous derivatives
$\frac{dz_1}{dw_1},\ldots,\frac{dz_{t-1}}{dw_1}$.

Explicitly write out the forward mode in our example.

\subsubsection*{Solution}

In the forward mode, we sequentially compute the values $z_t$ and derivates of
$z_t$ with respect to $w_1$ in order for $t = 1,2,\ldots,6$.

Suppose we want to calculate $\frac{df}{dw_1}(u,v)$. Fix $w_1 = u$ and
$w_2 = v$.

\begin{enumerate}
\item Compute $z_1 = u/v$.
\item Compute $\frac{dz_1}{w_1} = 1/v$.
\item Compute $z_2 = \sin\left(2\pi z_1\right)$.
\item Compute
  $\displaystyle\frac{dz_2}{dw_1} = \frac{\partial z_2}{\partial
    z_1}\frac{dz_1}{w_1} = 2\pi\cos\left(2\pi z_1\right)\frac{dz_1}{w_1}$.
\item Compute $z_3 = \exp\left(2v\right)$.
\item Compute $\displaystyle \frac{dz_3}{dw_1} = 0$.
\item Compute $z_4 = 3z_1 - z_3$.
\item Compute
  $\displaystyle \frac{dz_4}{dw_1}= \frac{\partial z_4}{\partial z_1}\frac{dz_1}{dw_1} + \frac{\partial z_4}{\partial z_3}\frac{dz_3}{dw_1} = 3\frac{dz_1}{dw_1} - \frac{dz_3}{dw_1}$.
\item Compute $z_5 = z_2 + z_4$.
\item Compute $\displaystyle \frac{dz_5}{dw_1} = \frac{\partial z_5}{\partial z_2}\frac{dz_2}{dw_1} + \frac{\partial z_5}{\partial z_4}\frac{dz_4}{dw_1} = \frac{dz_2}{dw_1} + \frac{dz_4}{dw_1}$.
\item Compute $z_6 = z_4z_5$.
\item Compute $\displaystyle\frac{dz_6}{dw_1} = \frac{\partial z_6}{\partial z_4}\frac{dz_4}{dw_1} + \frac{\partial z_6}{\partial z_5}\frac{dz_5}{dw_1} = z_5\frac{dz_4}{dw_1} + z_4\frac{dz_5}{dw_1}$.
\end{enumerate}

Each step can be computed by substituting values from the previous steps. The
output is $\displaystyle\boxed{\frac{df}{dw_1}(u,v) = \frac{dz_6}{dw_1}.}$

\subsection*{The reverse mode of AD}

Now let use consider the reverse mode to compute the derivative
$\frac{df}{dw}$, which is a two-dimensional vector.

Explicitly write out the reverse mode in our example with the assumption that
you have evaluated the trace and already stored all the $z_t$s in memory
already.

\subsubsection*{Solution}

In the reverse mode, we compute $\frac{df}{dz_t} = \frac{dz_6}{dz_t}$ in order
$t = 6,5,\ldots,0$. The general algorithm is
\begin{equation}
  \frac{dz_6}{dz_t} = \sum_{\text{$c$ is a child of $t$}}
  \frac{dz_6}{dz_c}
  \frac{\partial z_c}{\partial z_t}.
\end{equation}

\begin{enumerate}
\item Seed $\displaystyle \frac{dz_6}{dz_6} = 1$.
\item Compute
  $\displaystyle \frac{dz_6}{dz_5} =
  \frac{dz_6}{dz_6}\frac{\partial z_6}{\partial z_5} = z_4$.
\item Compute
  $\displaystyle \frac{dz_6}{dz_4} = \frac{dz_6}{d z_5}\frac{\partial
    z_5}{\partial z_4} + \frac{d z_6}{d z_6}\frac{\partial z_6}{\partial z_4} =
  \frac{dz_6}{d z_5} + \frac{d z_6}{d z_6}z_5
  =
  z_4 + z_5
  $.
\item Compute
  $\displaystyle \frac{dz_6}{dz_3} = \frac{dz_6}{d z_4}\frac{\partial
    z_4}{\partial z_3} = -\frac{dz_6}{d z_4} = -z_4 - z_5$.
\item Compute
  $\displaystyle \frac{dz_6}{dz_2} = \frac{dz_6}{d z_5}\frac{\partial
    z_5}{\partial z_2} = \frac{dz_6}{d z_5} = z_4$.
\item Compute
  $\displaystyle \frac{dz_6}{dz_1} =
  \frac{dz_6}{d z_2}\frac{\partial  z_2}{\partial z_1} +
  \frac{dz_6}{d z_4}\frac{\partial  z_4}{\partial z_1} =
  2\pi\cos\left(2\pi z_1\right)\frac{dz_6}{d z_2} +
  3\frac{dz_6}{d z_4}$.
\item Compute
  $\displaystyle
  \frac{dz_6}{dw_1} = \frac{dz_6}{dz_1}\frac{\partial z_2}{\partial w_1}
  = \frac{1}{w_2}\frac{dz_6}{dz_1}$.
\item Compute
  $\displaystyle
  \frac{dz_6}{dw_2} = \frac{dz_6}{dz_1}\frac{\partial z_1}{\partial w_2} +
  \frac{dz_6}{dz_3}\frac{\partial z_3}{\partial w_2}
  = -\frac{w_1}{w_2^2}\frac{dz_6}{dz_1} + 2\exp\left(2w_2\right)\frac{dz_6}{dz_3}$.
\end{enumerate}

Each step can be computed by substituting the output of one of the previous
steps. From the last two steps, we obtain our desired result
\begin{equation}
  \boxed{
    \frac{df}{dw} = \begin{pmatrix}
      \displaystyle\frac{dz_6}{dw_1} \\ \displaystyle\frac{dz_6}{dw_2}
    \end{pmatrix}.
  }
\end{equation}

\section*{Problem 3: Computation and Memory in AD}

Suppose we seek to compute the derivative with respect to a real valued function
$f\left(w\right) : \mathbb{R}^d \rightarrow \mathbb{R}$. Let us examine some of
the computational and memory issues involved in AD.

\subsection*{Computation}

Let $T$ be the computation time to compute $f(w)$ using our program.

\begin{enumerate}
\item Suppose we want to find the derivative of one variable $\frac{df(w)}{dw_1}$
  with respect to the variable $w_1$. In order notation, how does the
  computational complexity (the runtime) of the forward mode compare to the
  reverse mode $\frac{df(w)}{dw_1}$?
  \subsubsection*{Solution}

  For the forward mode, we just do one sweep through our computation graph to
  compute $\frac{df(w)}{dw_1}$, so the running time is $O(T)$.

  In the reverse mode, we also just do one sweep through the graph, so the
  running time is also $O(T)$.

  Both running times are of the same complexity in this case.

\item Suppose we want to find the derivative $\frac{df(w)}{dw}$,which is a
  $d$-dimensional vector. How would we do this with the forward mode and, in
  order notation, what is the computational complexity? Again, in order
  notation, how does the computational complexity (the runtime) of the forward
  mode compare to the reverse mode to compute $\frac{df(w)}{dw}$.
  \subsubsection*{Solution}

  For the forward mode, we have to do a sweep for each
  $\frac{df(w)}{dw_1},\ldots,\frac{df(w)}{dw_d}$ that we want to compute, so the
  runtime complexity is $O(Td)$.

  In the reverse mode, we have to do one sweep. Then, we have enough
  information to compute each $\frac{df(w)}{dw_j}$, so the runtime complexity is
  $O(T + d)$.

  The computational complexity of the reverse mode is smaller than that of the
  forward mode.

  \item If we could easily parallelize our computation (not worrying about
    communication), do you see a way to speed up the reverse mode? If so, what
    would be the new serial runtime? If not, why?
    \subsubsection*{Solution}

    Only one sweep needs to be done, and sweeping though the graph must be done
    in a certain order, so there is no easy way to parallelize that part. Once
    the sweep is done, one could use the computed intermediate derivatives to to
    calculate each $\frac{df(w)}{dw_j}$ in a parallel manner. In that case the
    new runtime complexity is $O(T)$.

  \item If we could easily parallelize our computation (not worrying about
    communication), do you see a way to speed up the forward mode? If so, how so
    and what would be the new serial runtime? If not, why?
    \subsubsection*{Solution}

    Yes. The obvious way is to parallize the forward mode is to have $d$ workers
    each doing a separate sweep and computing a separate
    $\frac{df(w)}{dw_j}$. If we do that, the runtime complexity is also $O(T)$.
\end{enumerate}

\subsection*{Memory}

Memory is often a bottleneck in practice (e.g. we load as much as we can on a
GPU when doing our computation in batches of data). Let us understand some of
these issues.  Suppose our input $w$ is $d$-dimensional and our evaluation trace
is $T$ steps. Assume one unit of memory is required to store a real
number. Also, assume that we are free to delete variables at any time to free up
memory (in practice, this often occurs by overwriting variables).  In this
question, when memory is used to refer to how much ``scratch space'' we need to
utilize in order to run our program. In this question, order notation is
sufficient. When stating your answers do not include the memory required to
store the parameter $w$ or the program itself. Also, when in your answers, do
not include the memory required to write out the programs output (assume we have
already allocated this memory). We are interested in how much excess memory the
program needs to have free in order to store its intermediate variables and do
all its computations.

\begin{enumerate}
\item Suppose that we were just interested in computing $f(w)$. Is it often the
  case that we can get away with less than $T$ units of memory? How so?

  \subsection*{Solution}
  Yes, we can get away with less than $T$ units of memory by discarding
  unnecessary intermediate variables.

  For example suppose the computation graph was a perfectly balanced tree and
  $z_T = z_{T - 1} + z_{T-2}$ where the computation traces of
  $z_{T - 1} + z_{T-2}$ are each $(T - 1)/2$. After computing $z_{T - 1}$, we
  can free any memory used to compute it.

\item Suppose we only need to use $m$ units of memory to compute $f(w)$. If we
  only wanted to compute $\frac{df}{dw_1}$, how much memory would we require to
  compute this using the forward mode? Explain.

  \subsection*{Solution}
  We also only need $O(m)$ units of memory to compute $\frac{df}{dw_1}$. Suppose
  that we are trying to compute $z_t$. We only need to know the value of its
  parents. To compute $\frac{dz_t}{dw_1}$, we only need to know the value of its
  parents and the derivatives of the of the parents with respect to $w_1$.
\item Suppose we only need to use $m$ units of memory to compute $f(w)$. If we
  only wanted to compute $\frac{df}{dw_1}$, how much memory would we require to
  compute this using the reverse mode? Explain.

  \subsection*{Solution}

  Since we want to avoid blowing up the computation, we need $O(T)$ units of
  memory. One may think that since the computation of $\frac{dz_T}{dz_t}$ only
  requires the children of $z_t$, less units of memory are needed. But since
  we're working backwards, we may be deep in the graph, so those children have
  to be computed ahead of time. The first steps require knowing the values near
  the leaves for instance. If we discarded the intermediate values, we would may
  to do expensive calculations to recover them when deep in the graph.

\item Suppose we only need to use $m$ units of memory to compute $f(w)$ and
  suppose we want to compute $\frac{df}{dw}$. If we don't care about runtime,
  what is the most memory efficient algorithm you can come up with? How much
  memory is needed?

  \subsection*{Solution}

  We could just repeat the forward pass to compute each $\frac{df}{dw_j}$. Each
  pass takes $O(m)$. After we're done the pass, we would discard any
  intermediate variables and just keep $\frac{df}{dw_j}$. If $w$ is
  $d$-dimensional, we only need $O(m + d)$ units of memory.

\end{enumerate}

\section*{Problem 4: PyTorch Can Give Us Some Crazy Answers}

Now you will construct an example in which PyTorch provides derivatives that
make no sense. The issue is in understanding when it is ok and when it is not ok
to use dynamic computation graphs. You are going to find a way code up the same
function in two different ways so that PyTorch will return different derivatives
at the same point. The purpose of this exercise is to better understand how
dynamic computation graphs work and to understand what you are doing when you
using various AD softwares.

\subsection*{Two Identical Non-differentiable Functions with Different ``Derivatives''}

\begin{enumerate}
\item Define \emph{and} plot a one dimensional, real valued function
  which is not continuous.

  \subsubsection*{Solution}
  Consider the function
  \begin{equation}
    f(x) = \begin{cases}
      2x, - 1& x < 0; \\
      0, & x = 0; \\
      2x + 1, & x > 0.
    \end{cases}
    \label{eqn:problem4}
  \end{equation}

  It is implemented in Listings \ref{lst:f} and \ref{lst:g} and plotted in
  Figure \ref{fig:problem4}.

  \begin{figure}[h]
    \centering
    \includegraphics{problem4/problem4.pdf}
    \caption{The functions from Listings \ref{lst:f} and \ref{lst:g} are plotted
      along with their PyTorch-computed derivatives.}
    \label{fig:problem4}
  \end{figure}
\item Now write out your function in PyTorch. You should be able to
  define this function so that PyTorch returns a derivative of $0$ at
  some point $x_0$.
  \subsubsection*{Solution}
  Equation \ref{eqn:problem4} is implemented as $f$ in Listing
  \ref{lst:f}. In Figure \ref{fig:problem4}, you can see that PyTorch
  calculates the derivative as $0$ at $x_0 = 0$ (left plot, blue
  \texttt{x}).

  \begin{listing}
\begin{minted}{python}
def f(x: Variable) -> Variable:
    assert x.requires_grad
    return (2*x*torch.sign(x) + 1)*torch.sign(x)
\end{minted}
  \caption{Equation \ref{eqn:problem4} defined with the sign function factored out.}
\label{lst:f}
\end{listing}

\item Now find another way to write out your function in PyTorch; do
  this so that it is exactly the same function. Do this in a way so
  that PyTorch now returns a derivative of $2$ at exactly the same
  point $x_0$ that you obtained a derivative of $0$ in the previous
  question.
  \subsubsection*{Solution}
  Equation \ref{eqn:problem4} is implemented as $g$ in Listing
  \ref{lst:g}. In Figure \ref{fig:problem4}, you can see that PyTorch
  calculates the derivative as $2$ at $x_0 = 0$ (right plot, blue
  \texttt{x}).

\begin{listing}
\begin{minted}{python}
def g(x: Variable) -> Variable:
    def g1d(x: Variable) -> Variable:
        if x.data[0] > 0:
            return 2*x + 1
        elif x.data[0] < 0:
            return 2*x - 1
        else:
            return 2*x

    if x.dim() == 0:
        return 1*x
    if x.size() == torch.Size([1]):
        return g1d(x)

    return torch.stack([g(sub_x) for sub_x in x])
\end{minted}
  \caption{Equation \ref{eqn:problem4} defined element-wise by recursing into the tensor.}
\label{lst:g}
\end{listing}
\item There is a no sane definition of the derivative for your
  function. Yet, you should not only found a way to get PyTorch to
  provide a derivative, you should have also found a way to give you
  two \emph{different} derivatives at exactly the same point (on the same
  function). What went wrong?

  \subsubsection*{Solution}

  PyTorch is just keeping track of the operations and does a pointwise
  calculation. It's not paying attention to any local behvaior like
  continuity. In Listing \ref{lst:g}, by changing the \texttt{else}
  arm, we can have it return any arbitrary number for the
  derivative. For instance, having it return \texttt{-3*x} would have
  PyTorch calculating the derivative as $-3$ at $x_0 = 0$.
\end{enumerate}

\subsection*{Extra Credit: Differentiable Functions with Different ``Derivatives''}

Provide a differentiable function, where you can code it up in PyTorch
in two different ways and where you can get two different derivatives
at the same point $x_0$.

\subsubsection*{Solution}

We can reuse the same idea. PyTorch doesn't correctly compute the
derivative when squaring the sign function, even though, the sign
function squared is just the identity.

Using this, I implement the simple, differentiable line $f(x) = 2x$ in
two ways: (1) verbosely using the sign function as $f$ and (2) the
canonical way as $g$ in Listing \ref{lst:fg_diff}.

\begin{listing}
\begin{minted}{python}
def f(x: Variable) -> Variable:
    assert x.requires_grad
    return 2*x*torch.sign(x)*torch.sign(x)

def g(x: Variable) -> Variable:
    assert x.requires_grad
    return 2*x
\end{minted}
  \caption{The function $f(x) = 2x$ defined in two different ways.}
\label{lst:fg_diff}
\end{listing}

This results in Figure \ref{fig:problem4_differentiable}. PyTorch
computes $f^\prime(0) = 0$ despite the true value being $2$.

\begin{figure}[h]
  \centering
  \includegraphics{problem4/problem4_differentiable.pdf}
  \caption{$f$ and $g$ from Listing \ref{lst:fg_diff} plotted along with
    their PyTorch-computed derivatives.}
  \label{fig:problem4_differentiable}
\end{figure}

The code for this exercise can be found at \url{https://gitlab.cs.washington.edu/pmp10/cse547/tree/master/hw1/problem4}.

\section*{Problem 5: Elementary properties of $l_2$ regularized logisitic regression}

\subsection*{The binary case}

Consider minimizing
\begin{equation}
  J(\mathbf{w}) = -l\left(\mathbf{w},\mathcal{D}_\mathrm{train}\right) + \lambda\left\lVert\mathbf{w}\right\rVert_2^2,
\end{equation}
where
\begin{equation}
  l\left(\mathbf{w},\mathcal{D}\right) = \sum_j \log \mathbf{P}\left(
    y^j \mid \mathbf{x}^j, \mathbf{w}
  \right)
\end{equation}
is the log-likelihood on the data set $\mathcal{D}$ for
$y^j \in \{ \pm 1 \}$

State if the following are true or false. Briefly explain your reasoning.

\begin{enumerate}
\item With $\lambda > 0$ and the features $x^j_k$ linearly separable,
  $J\left(\mathbf{w}\right)$ has multiple locally optimal solution.
  \subsubsection*{Solution}

  \emph{False}. When the features are linearly separable, we can push loss
  unregularized loss arbitrarily close to $0$ but the the loss is
  still convex. The sum of two convex functions is convex, so there
  will be global optimum.

\item Let
  $\hat{\mathbf{w}} = \argmin_{\mathbf{w}} J\left(\mathbf{w}\right)$ be
    a global optimum. $\hat{\mathbf{w}}$ is typically sparse.

    \subsubsection*{Solution}
    \emph{False}. This may be true with $l_1$ regularization, but is not
    usually the case with $l_2$ regularization. If one considers the
    dual Lagrangian problem, $\hat{\mathbf{w}}$ will lie on some
    hypersphere at a point which will not generally have $0$ values.
  \item If the training data is linearly separable, then some weights
    $w_j$ might become infinite if $\lambda = 0$.
    \subsubsection*{Solution}

    \emph{True}. By making the weights larger and larger we can push the loss
    to be arbitrarily close to $0$ if $\lambda = 0.$

  \item $l\left(\hat{\mathbf{w}}, \mathcal{D}_\mathrm{train}\right)$
    always increases as we increase $\lambda$.
    \subsubsection*{Solution}

    \emph{True}. If one thinks in term of the Lagrangian dual problem
    increasing $\lambda$ is contraining the weights further away from
    the global optimum by restricting them to a smaller hypersphere.

  \item $l\left(\hat{\mathbf{w}}, \mathcal{D}_\mathrm{test}\right)$
    always increases as we increase $\lambda$.
    \subsubsection*{Solution}

    \emph{False}. While the training loss may increase, we could be
    overfitting. Thus, sometimes increasing $\lambda$ may decrease
    test loss.
\end{enumerate}

\subsection*{Multi-class Logistic Regression}

In multi-class logistic regression, suppose
$Y \in \left\{y_1,\ldots,y_R\right\}$. A simplified version (with no
bias term) is as follows. When $k < R$ the posterior probability is given by:
\begin{equation}
  P\left( Y = y_k \mid X\right) =
  \frac{\exp\left(\left\langle w_k, X\right\rangle\right)}
  {1 + \sum_{j=1}^{R-1}\exp\left(\left\langle w_j, X\right\rangle\right)}.
  \label{eqn:multiclass_log_reg}
\end{equation}

For $k = R$, the posterior is
\begin{equation}
  P\left( Y = y_R \mid X\right) =
  \frac{1}
  {1 + \sum_{j=1}^{R-1}\exp\left(\left\langle w_j, X\right\rangle\right)}.
\end{equation}

To simplify notation, we can define $w_R = \mathbf{0}$ as a vector of
all $0$s. This gives us Equation \ref{eqn:multiclass_log_reg} for all
$k$.

\begin{enumerate}
\item How many parameters do we need to estimate? What are these
  parameters?

  \subsubsection*{Solution}

  Assume the data is $D$-dimensional. Our parameters are the weights
  $w_k$ each which is a $D$-dimensional vector. There are
  $\boxed{(R - 1)D}$ parameters to estimate.
\item Given $N$ training samples
  $\left\{\left(x^1, y^1\right),\left(x^2,
      y^2\right),\ldots,\left(x^N, y^N\right)\right\}$, write down
  explicitly the log-likelihood function and simplify it as much as
  you can:
  \begin{equation}
    L\left(w_1,\ldots,w_{R-1}\right) = \sum_{j=1}^N \log\left(
      P\left(y^j \mid x^j,w\right)
    \right).
    \label{eqn:log_likelihood}
  \end{equation}

  \subsubsection*{Solution}

  Let $y^j = y_{l^j}$, that is, let $l^j$ be the class label of observation
  $j$. If we use the posterior probability in Equation
  \ref{eqn:multiclass_log_reg}, then, we have that
  \begin{equation}
    \log\left(
      P\left(y^j \mid x^j,w\right)
    \right)
    = \left\langle w_{l^j}, x^j\right\rangle -
    \log\left(
      1 + \sum_{k=1}^{R - 1}\exp\left(
        \left\langle w_k, x^j\right\rangle
      \right)
    \right).
    \label{eqn:log_p}
  \end{equation}

  Substituting Equation \ref{eqn:log_p} into Equation \ref{eqn:log_likelihood}, we have
  \begin{equation}
    L\left(w_1,\ldots,w_{R-1}\right)
    = \sum_{j=1}^N\left(
      \left\langle w_{l^j}, x^j\right\rangle
      -
      \log\left(
      1 + \sum_{k=1}^{R-1}\exp\left(\left\langle w_k,x^j\right\rangle\right)
      \right)
    \right).
    \label{eqn:log_p_simple}
  \end{equation}
\item Compute the gradient of $L$ with respect to each $w_k$ and simplify it.
  \subsubsection*{Solution}

  $w_k$ is a $D$-dimensional vector so, $\frac{\partial L}{\partial w_k}$ will
  also be $D$-dimensional. Denote the features for the observations with class
  label $k$ by $\mathcal{X}_k = \left\{x^j : l^j = k\right\}$.

  We have that for $k = 1,2,\ldots,R-1$:
  \begin{align}
    \frac{\partial L}{\partial w_k}
    &= \sum_{x \in \mathcal{X}_k} x
      - \sum_{j=1}^Nx^j\frac{
      \exp\left(\left\langle w_k,x^j\right\rangle\right)
      }
      {1 + \sum_{m=1}^{R - 1}\exp\left(\left\langle w_m,x^j\right\rangle\right)}
      \label{eqn:log_likelihood_gradient}\\
    &= \sum_{x \in \mathcal{X}_k} x -
      \sum_{j=1}^N x^j P\left(Y = y_k \mid X = x^j\right).
      \nonumber
  \end{align}
\item Now add the regularization term $\lambda$ and define a new objective
  function:
  \begin{equation}
    L\left(w_1,\ldots,w_{R-1}\right) = \sum_{j=1}^N\log\left(
      P\left(y^j \mid x^j, w\right)
    \right)
    +
    \frac{\lambda}{2}\sum_{l=1}^{R-1}\left\lVert w_l\right\rVert_2^2.
    \label{eqn:log_likelihood_with_penalty}
  \end{equation}

  Compute the gradient of this new $L$ with respect to each $w_k$.

  \subsubsection*{Solution}

  We can just differentiate term by term. The gradeint of the first term comes
  from Equation \ref{eqn:log_likelihood_gradient}. We can write
  $\left\lVert w_l\right\rVert_2^2 = w_{l1}^2 + w_{l2}^2 + \cdots + w_{lD}^2$,
  so we have that
  \begin{equation}
    \frac{\partial L}{\partial w_l}
    = \sum_{x \in \mathcal{X}_l} x
      - \sum_{j=1}^Nx^j\frac{
      \exp\left(\left\langle w_l,x^j\right\rangle\right)
      }
      {1 + \sum_{m=1}^{R - 1}\exp\left(\left\langle w_m,x^j\right\rangle\right)}
      + \lambda w_l.
  \end{equation}
\end{enumerate}

\section*{Problem 6: Getting Familiar with Our Dataset}

Let us consider solving a binary classification problem (labels being
$\{-1, 1\}$) with the dataset provided. We will use the square loss and consider
training two models, namely $(i)$ a linear model and $(ii)$ a multi-layer
perceptron. Note that this the same dataset that we will start branching out on,
for the purposes of later assignments and the (default) course project.  The
dataset contains two supercategories: vehicle and animal, and a number of
categories for each supercategory. In the small dataset provided, each image
contains objects of a single supercategory, say vehicle, and potentially
multiple objects from the supercategory, such as car, boat, etc. In this
exercise we shall build a classifier that learns to classify between these
supercategories, by optimizing a square loss objective with the above
models. For the purposes of learning these classifiers, we shall use features
from a convolutional neural network (as opposed to the raw pixels from these
images). We have provided starter code to read these features.

\subsection*{SGD and Linear Regression}

Here, the objective function we choose to optimize is:
\begin{equation}
  L(w) = \frac{\lambda}{2}\left\lVert w \right\rVert_2^2  + \frac{1}{n}\sum_{i=1}^n\frac{1}{2}\left(y_i - \left\langle w, x_i\right\rangle\right)^2,
\end{equation}
where, $y_i \in \{-1, 1\}$ is the label, $x_i \in \mathbb{R}^d$ are the
features, $w \in \mathbb{R}^d$ is the linear model that we wish to optimize for
and $\lambda > 0$ is the strength of $l_2$ regularization. Now consider running
stochastic gradient descent on $L(w)$, where the stochastic gradient is computed
using a single sample (i.e. a batch size of $1$).

\begin{enumerate}
\item Report the stepsize at which SGD starts to diverge (specified up to, say a
  factor of $2$ from the actual value). Why would you expect the algorithm to
  diverge at too large a learning rate?

  \subsubsection*{Solution}

  I found that my model began to diverge with a learning rate of
  $2 \times 10^{-4}$. The misclassification became 50\% with is essentially a
  coin flip.

  I'd expect the algorithm to diverge since it's taking too large steps in the
  parameter space and missing the optima.

\item After every $500$ updates (index the first update at $0$), compute $L(w)$
  evaluated over the training/development/test dataset and make a plot with
  these values in the $y$-axis and the iteration on the $x$-axis. All three
  curves should be on one same plot. What value of $\lambda$ did you use?

  \subsubsection*{Solution}

  This is plotted in Figure \ref{fig:linear_loss}. I used $\lambda = 128$.

  \begin{figure}
    \centering
    \includegraphics{problem6/linear_loss.pdf}
    \caption{Loss for the linear classifier.}
    \label{fig:linear_loss}
  \end{figure}

\item Compute the misclassification error every $500$ updates and plot these
  quantities (in a single plot) evaluated over the training, development and
  test dataset. Here, make sure to start your $x$-axis at a slightly later
  iteration to make the behavior of the $0/1$ error more easy to view (it is
  difficult to view the long run behavior if the y-axis is over too large a
  range). Report the lowest test error.

  \subsubsection*{Solution}

  This is plotted in Figure \ref{fig:linear_misclassification}. The linear model
  performed the best, better than the multi-layer perceptron. From Table
  \ref{tab:model_results}, the lowest misclassification rate was 7.2\% for the
  test data and 6.4\% for the validation data.

  \begin{figure}
    \centering
    \includegraphics{problem6/linear_misclassification.pdf}
    \caption{Misclassification rate for the linear classifier.}
    \label{fig:linear_misclassification}
  \end{figure}
\end{enumerate}

\subsection*{Implement a Multi-Layer Perceptron (MLP)}

Here, the objective function we choose to optimize is:
\begin{equation}
  L(w) = \frac{\lambda}{2}\left\lVert w \right\rVert_2^2  + \frac{1}{n}\sum_{i=1}^n\frac{1}{2}\left(y_i - f_i(w)\right)^2,~\text{where}~
  f_i(w) = \left\langle w_2, \operatorname{relu}\left(w_1^\intercal x_i\right)\right\rangle.
\end{equation}
$y_i \in \{-1, 1\}$ is the label, $x_i \in \mathbb{R}^d$ are the features,
$w_1 \in \mathbb{R}^d \times \mathbb{R}^h$, $w_2 \in \mathbb{R}^h$, and
$\operatorname{relu}(x) = \max\left\{x, 0\right\}$, where the max is applied
element wise, and $h$ is the number of hidden nodes in our MLP.  Now consider
running stochastic gradient descent on $L(w)$, where the stochastic gradient is
computed using a single sample. In this exercise, answer the questions below
with the number of hidden nodes being (a) $10$, (b) $100$, (c) $500$.

\begin{enumerate}
\item Report the stepsize at which SGD starts to diverge (specified up to, say a
  factor of $2$ from the actual value).

  \subsubsection*{Solution}

  I started seeing divergence with a learning rate of $0.001$. The algorithm
  didn't make any progress and got stuck with a 50\% misclassification rate.

  \item Compute $L(w)$ evaluated over the training, development, and, test
    dataset every $500$ updates and plot these values in a single plot. Specify
    your learning rate scheme if you chose to decay your learning rate.

    \subsubsection*{Solution}

    The plots are shown in Figures \ref{fig:mlp10_loss}, \ref{fig:mlp100_loss},
    and \ref{fig:mlp500_loss}. $\lambda = 0.0004$ was used as the $l_2$
    penalty. There appears to overfitting as the test and validation loss are
    much higher than the training loss. A fixed learning rate was used.

    \begin{table}
      \centering
      \begin{tabular}{lrrr|rrr}
        \toprule
        & \multicolumn{3}{c}{Loss} & \multicolumn{3}{c}{Accuracy} \\
        \cmidrule{2-4} \cmidrule{5-7}
        Model & Training & Test & Validation & Training & Test & Validation \\
        \midrule
        Linear & 0.141587 & 0.342109 & 0.340544 & 0.9945 & 0.928 & 0.936 \\
        MLP (10 units) & 0.202833 & 0.421297 & 0.331022 & 0.9620 & 0.862 & 0.928 \\
        MLP (100 units) & 0.182172&0.394072&0.289352 & 0.9725&0.876&0.926 \\
        MLP (500 units) & 0.160434&0.417597&0.313770&0.9810&0.876&0.936 \\
        \bottomrule
      \end{tabular}
      \caption{The smallest loss and highest accuracy obtained by each model.}
      \label{tab:model_results}
    \end{table}

    \begin{figure}
      \centering
      \includegraphics{problem6/mlp10_loss.pdf}
      \caption{Loss for the multi-layer perceptron with 10 hidden units.}
      \label{fig:mlp10_loss}
    \end{figure}

    \begin{figure}
      \centering
      \includegraphics{problem6/mlp100_loss.pdf}
      \caption{Loss for the multi-layer perceptron with 100 hidden units.}
      \label{fig:mlp100_loss}
    \end{figure}

    \begin{figure}
      \centering
      \includegraphics{problem6/mlp500_loss.pdf}
      \caption{Loss for the multi-layer perceptron with 500 hidden units.}
      \label{fig:mlp500_loss}
    \end{figure}

  \item Compute the misclassification error every $500$ updates and plot these
    quantities (in a single plot) evaluated over the training, development and
    test dataset. As in the case of the linear model, make sure to start your
    $x$-axis at a slightly later iteration so as to make the behavior of the
    $0/1$ error more easy to view. Report the lowest test error.

    \subsubsection*{Solution}

    The plots are shown in Figures \ref{fig:mlp10_misclassification},
    \ref{fig:mlp100_misclassification}, and
    \ref{fig:mlp500_misclassification}. These plots mirror the those for loss
    and show evidence of overfitting. The lowest misclassification rate was was
    the 12.4\% achieved by using $100$ and $500$ hidden units. On the validation
    dataset, the $500$ hidden units model does the best misclassification rate
    of 6.4\%.

    \begin{figure}
      \centering
      \includegraphics{problem6/mlp10_misclassification.pdf}
      \caption{Misclassification rate for the multi-layer perceptron with 10 hidden units.}
      \label{fig:mlp10_misclassification}
    \end{figure}

    \begin{figure}
      \centering
      \includegraphics{problem6/mlp100_misclassification.pdf}
      \caption{Misclassification rate for the multi-layer perceptron with 100 hidden units.}
      \label{fig:mlp100_misclassification}
    \end{figure}

    \begin{figure}
      \centering
      \includegraphics{problem6/mlp500_misclassification.pdf}
      \caption{Misclassification rate for the multi-layer perceptron with 500 hidden units.}
      \label{fig:mlp500_misclassification}
    \end{figure}

\end{enumerate}

\subsection*{Get Familiar with Amazon Web Services (AWS)}

Use Amazon AWS to run the code developed in previous sections. Attach a
screenshot showing the progress of the algorithm (and the result) as viewed in
the remote machine. The purpose of this exercise is to get familiar with AWS
and set yourselves up for future assignments and the course project.

\subsubsection*{Solution}

I trained the models as AWS Batch jobs. You can see the result of a linear model
in Figure \ref{fig:linear_batch} and the result of the multi-layer perceptron
model in Figure \ref{fig:mlp_batch}.

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{linear_screenshot.png}
  \caption{Training the linear classifier as an AWS batch job with $\lambda = 0.0004$.}
  \label{fig:linear_batch}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{mlp_10_screenshot.png}
  \caption{Training the multi-layer perceptron as an AWS batch job with 10 hidden units.}
  \label{fig:mlp_batch}
\end{figure}

Code to run this exercise can be found at
\url{https://gitlab.cs.washington.edu/pmp10/cse547/blob/master/hw1/run.py}. The
Jupyter Notebook for the plots is in
\url{https://gitlab.cs.washington.edu/pmp10/cse547/blob/master/hw1/problem6}.

\end{document}
% Local Variables:
% TeX-command-extra-options: "-shell-escape"
% End:


\documentclass[Thesis.tex]{subfiles}
\begin{document}
\chapter{Automatic Differentiation}
\label{chp:auto-diff}

\glsresetall

One of the central parts of the success of neural networks in machine learning
is the backpropagation algorithm of
\cref{eq:backprop-delta-recursive,eq:backprop-grads-recursive}. The algorithm is able to
describe how parameters should be updated in arbitrary networks by propagating
the error estimates backwards through the network. While we could derive these
equations relatively simply in the case of a pure feed-forward neural network
such as the ones described here, there is a whole world of variants for which
we would need to derive different equations, such as recurrent networks,
residual networks, convolutional networks etc. Nevertheless, they all still rely
on the basic premise of applying the chain rule.

One of the perhaps most appealing attributes of \gls{ml} frameworks such as TensorFlow~\cite{tensorflow2015-whitepaper}
is how we can design arbitrary networks \emph{without} having to manually work
out the associated derivatives. The way this is done is through \gls{ad}, an
algorithmic application of the chain rule that can compute derivatives of any
function specified by a computer program.

This is possible by exploiting the fact that any computer program must be built
up of elementary operations (\(+,-,*,/\)) and perhaps some elementary functions
(\(\exp, \sin\) etc.). We know how to handle derivatives for all of these, and
importantly we know how to combine them in arbitrary ways through the chain
rule. This allows us to construct a computation for any function which
will compute its derivative.

\Gls{ad} promises a general purpose implementation that can
calculate derivatives exactly (within numerical precision) in a manner that uses
at most a constant factor more operations than the original function. We can
also generalize to higher order derivatives by repeated application of the
algorithm. That is, viewing the derivative computation as a function in its own
right we can apply the same trick to construct a computation for the second
derivative.

\section{Example}

Let the function we wish to differentiate be the following:\footnote{This example is in large part similar to one found
    on the \gls{ad} Wikipedia page, simply because I quite like this explanation.}

\begin{align}
  f(x_1, x_2) &= x_1x_2 + \sin(x_1)\\
              &= w_1w_2 + \sin(w_1)\\
              &= w_3 + w_4\\
              &= w_5 = y,
\end{align}
where we have introduced the temporary variables $w_i$ for later notational brevity.
Assume that we are interested in $\pdv*{y}{x_1}$ and $\pdv*{y}{x_2}$. There are
two main approaches in use: \emph{forward} and \emph{reverse} mode.

\subsection{Forward Mode}

We first decide on one independent variable to differentiate with respect to,
say $x_1$. We then compute $\dot w_i = \pdv*{w_i}{x_1}$ in order of increasing
$i$ (forward through the computation).

When we step through the computation of $f(x_1, x_2)$, we can simultaneously
compute the derivative we want. By normal rules of differentiation:

\begin{center}
  \begin{tabular}{ll}
    Value & Derivative (w.r.t. $x_1$)\\\hline\\
    $w_1 = x_1$ & $\dot w_1 = 1$\\
    $w_2 = x_2$ & $\dot w_2 = 0$\\
    $w_3 = w_1w_2$ & $\dot w_2 = \dot w_1 w_2 + w_1\dot w_2$\\
    $w_4 = \sin(w_1)$ & $\dot w_4 = \cos(w_1)\dot w_1$\\
    $w_5 = w_3 + w_4$ & $\dot w_5 = \dot w_3 +\dot w_4$\\
  \end{tabular}
\end{center}
Substituting in the values from top to bottom we see that we indeed get
$\pdv{y}{x_1}=\cos(x_1)+x_2$ as one would expect. We can now repeat the process
once more for $x_2$. Evidently, we only need to change the initial values $\dot
w_1$ and $\dot w_2$, while the rest remains the same. Because of this, these
\say{trivial} first derivatives are sometimes referred to as \emph{seed} values.

\subsection{Reverse Mode}

Instead of first fixing the independent variable, we instead fix the dependent
variable which should be differentiated (only one in this case, \(y\)), and then
compute its derivative w.r.t. the different sub expressions $w_i$. We define
\(\bar w_i = \pdv*{y}{w_i}\), and construct the following table of operations
(backwards through the computation):

\begin{align*}
  \bar w_5 &= \pdv{y}{w_5} = 1\\
  \bar w_4 &= \pdv{y}{w_4} = \bar w_5\pdv{w_5}{w_4} = \bar w_5\\
  \bar w_3 &= \pdv{y}{w_3} = \bar w_5\pdv{w_5}{w_3} = \bar w_5\\
  \bar w_2 &= \pdv{y}{w_2} = \bar w_3\pdv{w_3}{w_2} = \bar w_3 w_1\\
  \bar w_1 &= \pdv{y}{w_1} = \bar w_4\pdv{w_4}{w_1} + \bar w_3 \pdv{w_3}{w_1}= \bar w_4 \cos(w_1) + \bar w_3w_2.
\end{align*}
Notice how reverse mode required only one pass though the algorithm in order to
find the derivatives of both independent variables, compared to two passes for
forward mode.

Keen readers might also recognize backpropagation as a special case of reverse mode \gls{ad}.

\section{Forward Mode vs. Reverse Mode}

While the two described approaches both obtain the same result, one is generally
preferable to the other depending on the structure of the computation. Let the
computation be a function $f: \mathbb{R}^{m}\to\mathbb{R}^n$. Forward mode
requires $m$ passes (one per input), while reverse mode requires $n$ passes (one
per output). So in general, if $m \gg n$ we prefer reverse mode and when $m\ll
n$ we prefer forward.

Because of this, reverse mode automatic differentiation is by far the most
prevailing choice in \gls{ml} frameworks. This is because models almost always have
more inputs than outputs.

\section{Automatic Differentiation for VMC}

As \gls{vmc} also needs quite a lot of derivatives it is natural to wonder if we
can have \gls{ad} tools work to our advantage also in this area.

\subsection{First Order Derivatives}

To see how this could work, we should revisit the list of required operations
for wave functions in \cref{sec:arbitrary-models-as-trial-wave-functions}. In
the list we see the need for first order derivatives w.r.t. both the inputs to
the wave function and its parameters. Both of these are straightforward to
obtain in an efficient manner using reverse mode \gls{ad}. To illustrate why, we can
consider the \emph{Jacobian} matrix, $\mat J$, which for a function
$f:\mathbb{R}^m\to\mathbb{R}^n$ looks like:

\begin{align}
  \mat J &=
  \begin{pmatrix}
    \pdv{f_1}{x_1} & \pdv{f_1}{x_2} & \dots & \pdv{f_1}{x_m}\\
    \pdv{f_2}{x_2} & \pdv{f_2}{x_2} & \dots & \pdv{f_2}{x_m}\\
    \vdots & \vdots & \ddots & \vdots\\
    \pdv{f_n}{x_1} & \pdv{f_n}{x_2} & \dots & \pdv{f_n}{x_m}\\
  \end{pmatrix}\,.
\end{align}
A forward mode sweep calculates a column of $\mat J$, while a reverse mode sweep
calculates a row~\cite{auto-diff-Berland}. In our case, $f$ is a wave function
which has a scalar output, and $\mat J$ is really a row vector. Reverse mode is
clearly superior in this case, requiring only a single pass.

\subsection{Higher Orders}

Returning to the list of operations, we still have the ominous Laplace
operator, $\laplacian$. To see why this will be harder, we can consider the
\emph{Hessian} matrix, $\mat H$:

\begin{align}
  \mat H &=
  \begin{pmatrix}
    \pdv[2]{f}{x_1} & \pdv{f}{x_1}{x_2} & \dots & \pdv{f}{x_1}{x_m}\\
    \pdv{f}{x_2}{x_1} & \pdv[2]{f}{x_2} & \dots & \pdv{f}{x_2}{x_m}\\
    \vdots & \vdots & \ddots & \vdots\\
    \pdv{f}{x_M}{x_1} & \pdv{f}{x_M}{x_2} & \dots & \pdv[2]{f}{x_m}\\
  \end{pmatrix}\,.
\end{align}
We can compute these second order derivatives by applying \gls{ad} twice. First we do
a reverse sweep and obtain the Jacobian (the row vector). The act of computing
the Jacobian is it self a function, $f: \mathbb{R}^m\to\mathbb{R}^m$, and as
such we can apply \gls{ad} to it an obtain the elements of $\mat H$.

Computing the required Laplace operator equates to computing the trace (sum of
diagonal) of the Hessian matrix. Remember that a forward/reverse sweep
calculates a column/row of the matrix. That means that in order to get the
diagonal elements, we actually need $m$ sweeps, regardless of what direction we
go. Sadly, the complexity of this operation is so large that we have found it
unfeasible to use for even a moderate number of inputs.

\section{Why Automatic Differentiation Failed}

Because \gls{ad} is so central to most \gls{ml} frameworks, like TensorFlow and
PyTorch, a great deal of effort was put in to attempt to make a viable \gls{vmc}
implementation within these. That would give out-of-the-box functionality for
neural networks, optimization strategies, \acrshort{gpu} acceleration etc.
Sadly, the inefficiency of the calculation of the Laplacian operator put a stop
to such a marriage. The time spent by \gls{ad} to compute $\laplacian$ was
orders of magnitude worse than the manually coded analytically expressions, and
scaled significantly worse with increasing number of particles. This severe
deficiency of \gls{ad} resulted in the unfortunate need to depart from the
existing \gls{ml} frameworks and implement everything manually (i.e.
analytically expressions for derivatives). This decision was not made lightly,
and only after a painstaking effort to somehow attempt to defy the above
mathematical argument. Unfortunately, mathematics was right.

We sincerely hope that \gls{ad} can somehow be efficiently incorporated in
future works. Research into how \gls{ad} can be efficiently applied to higher
order derivatives is ongoing, see for instance~\textcite{wang2017}. If nothing
else, we could potentially use \gls{ad} for Hamiltonians which do not include
the Laplacian operator.

\end{document}