% Chapter 2
%
\chapter{Technology and Basic Principles} % Main chapter title
\label{Chapter2} % For referencing the chapter elsewhere, use \ref{Chapter2}
%
%2. Technology and basic principles
    %
    %Public Key Cryptography

        %Key management

        %Certificates
        %\label{sec:cert} %Discussion of what data certificates usually contain

        %\label{sec:pgp_enc} %Demmonstration/Example of PGP encryption

    %Web of Trust
        %Overview

        %Ceritifcates

        %Certificate sharing

        %Sylbil attack
        %\label{sec:trust_syl} %Sylbil attack theory

    %WebRTC
    %\label{sec:webrtc}

        %Connection setuo
            %ICE, STUN \& TURN
            %\label{sec:webrtc_icetri} % WebRTC ICE trickling
            %SDP and DTLS-SCTP
            %Offer and Answer
            %\label{sec:webrtc_off}

        %Signaling server

    %WS
    %\label{sec:ws}

%
% THIS SHOULD GO THROUGH THE TECHNOLOGY OBJECTIVELY AND HOW IT WORKS!
%
This chapter will go over the technologies proposed and used in SendIt and give an introduction to their functionality, as well as the reasons for choosing said technology. It will give a basic understanding of the technology, which the rest of the thesis is built upon.
%
\section{Cryptography}
\label{sec:pkc}
%
    %
    %Key pairs
    The encryption and authentication scheme used in SendIt is based on public-key cryptography and symmetric cryptography. Public-key cryptography is a technology that creates a public and a private key pair, where the public key is used for encryption. The encrypted data can \emph{only be decrypted by the corresponding private key}. The inverse is also true. Anything encrypted with the private key, can be decrypted by the public key. The common usage is to spread the public key around, hence the name public. The private key (also called secret key) is kept safe, and only the identity that is the owner of the key pair should have access to it \cite{trcekManagingInformationSystems2006}.

    In contrast, symmetric cryptography utilizes the same key for encryption and decryption. The advantage of symmetric cryptography is that it can encrypt larger amounts of data. The disadvantage is that anyone with the key, can read any message. Symmetric keys usually also needs a value called IV (Initialization Vector), that is used to randomize the encrypted values. It allows for two identical messages to have different ciphers. There are two forms of IV schemes. One is using completely random number(s) each time, called a randomized scheme. The other scheme requires each IV to be unique, and is called a stateful scheme \cite{bellareIntroductionModernCryptography2005}.

    SendIt uses public-key cryptography to exchange encrypted messages that only the person having the correct key can decrypt. This is the same principles as used in Public Key Infrastructure. The difference lies in that PKI relies on a central authority for signing and verifying the keys used, in the form of certificates. SendIt tries to avoid any central form of governance, and as such, implements the web of trust model instead, in order to avoid this central management. This is discussed more in \Cref{sec:trustmodel}.

    \subsection{Key management}
    This means that the keys have to be the same every time, which requires each user to store their keys for later. This should be done with some consideration, since a stolen private key means anyone can now take your identity and successfully authenticate as you. As such the need for secure storage arises. This can be solved by encrypting the keys with a password, then storing them on the local machine. Only when a key is in use, will it be read and decrypted. This decreases the chance of the key being stolen, since it minimizes the attack surface by only having the key available when it is in use.

    When storing keys it is necessary to keep more information than just the key. For example, it is also necessary to know who it belongs to. For the proposed system, information relevant to the trust-level of each key also needs to be stored. The easiest way to do this is to store the certificate of each key with the key. That means the trust in a key will have to be re-calculated every time it is loaded. This is costly when it comes to computational resources, but allows the system to re-calculate the trust  in a key before each use. This gives a flexible and up-to-date environment where trust is evaluated, similar to in the real world.

    \subsection{Certificates}
    \label{sec:cert}
    %
    SendIt also proposes to use digital certificates like the ones used in the web of trust model \cite{ar_pgp}. Certificates traditionally contain various information. The certificate is then signed by any number of “introducers”, by encrypting it using their private key. An introducer is an identity who trusts another identity and vouches for the legitimacy of that identity \cite{ar_pgp}.

    Certificates used in PKI are required to contain enough information to provide a third party with the subject's public key. Usually certificates used in PKI contain more data than required. The minimum required data can be summed up in four points \cite{vaccaPublicKeyInfrastructure2004}:\\
    \begin{itemize}
        \item CA identification information
        \item Subject identification information
        \item Subject public key
        \item Validity (time)
    \end{itemize}
    More information on introducers and certificates can be found in \Cref{sec:trustmodel}.

    \subsection{Authentication}
    \label{sec:pgp_enc}
    The last part of the puzzle is \emph{how can a user be authenticated based on this information?} There are many different schemes, but SendIt focuses on the one suggested by PGP for securely exchanging messages \cite{b_pgp}. The system is based on public key encryption. It also uses a session key, which is a random, one time generated symmetric key. The system guarantees the authenticity, confidentiality, and integrity of the data. Encryption and decryption using the PGP system is shown in \Cref{fig:enc} and \Cref{fig:decr}.

    The way PGP works is by first generating a symmetric key and encrypting the data to be transferred with this symmetric key. This results in a cipher. Let us call this \emph{Cipher 1}. Then the symmetric key is encrypted with the recipients public key. This also results in a cipher (\emph{Cipher 2}). The data transferred to the recipient is \emph{Cipher 1} and \emph{Cipher 2}.

    The recipient then retrieves the symmetric key, by decrypting \emph{Cipher 2}, using their private key. Then, using the retrieved symmetric key, \emph{Cipher 1} is decrypted and the original data is available.

    This solution is made with a one-directional exchange of data in mind. This system will be used to authenticate both endpoints, which means it will have bi-directional communication. The way to solve this issue, is by taking advantage of the way symmetric keys encrypt data. The initial exchange from Sender to Receiver gives proof to the Sender that the recipient is as intended. This is not the case if the same IV is re-used for the reply. As such, it should be changed.

    Usually the IV is shared in cleartext, since knowledge of the key is also required to decrypt it, therefore it is not sensitive data. In SendIt, the IV used in the reply is encrypted with the Sender's public key. That means that in order to be able to access the data, the private key is needed, and as such, the Sender is also authenticated. This solves the issues of mutual, bi-directional authentication.

%
\section{Web of trust}
\label{sec:trustmodel}
%
    This section is a suggestion and evaluation of the system only. This has not been implemented in SendIt due to time constraints, but is considered a part of the design.

    \subsection{Overview}
    The model suggested is based on the web of trust. The web of trust model says that trust should be split into two groups: \textbf{Trustworthiness of public-key certificate} - \emph{How willing are you to trust this key pair when it is being used to communicate?} and \textbf{Trustworthiness of an introducer} - \emph{How willing are you to trust this key pair when it is being used to introduce another key?}.

    It also suggests that the trust level given to a key should be based on points given to a certificate, based on the sum of introducers and their respective trust level. There is no concrete policy or algorithm suggested for evaluating which level of trust is assigned. The model does define levels of trust, but they are different for the two groups previously mentioned.

    There is also no recommendation of which trust level should be considered safe or unsafe \cite{ar_pgp}. This is left up to each individual user, since there exist many ways to rate and evaluate trust. For this system, a simple point-based system is likely to be the best fit. The web of trust model has also been recommended by other researchers, albeit for a more traditional WebRTC authentication system \cite{ar_whois}. For a look into more options, J\o sang et al. (2007) \cite{a_trustserv} gives a good overview over possible alternatives.

    \subsection{Certificates}
    %
    The way one infers, evaluates, and determines trust is by using certificates. The trust is evaluated based on interactions with others and the interactions between trusted introducers and others. This means that information from identities one does not trust will be disregarded (their introductions mean nothing), while still valuing and receiving updated information from identities one does trust. In other words, many untrusted identities introducing someone does not hold any value, while introductions from a few trusted identities has a significant impact on the trust level assigned to a certificate.

    \subsection{Certificate sharing}
    The exchange of keys and the trust each identity assigns these keys are central to the web of trust model, as described previously (\Cref{sec:trustmodel}). Since the connection used for communication is already secure, and each party authenticated, this data can be transferred without additional encryption. This section will describe how this is intended to be implemented in SendIt. This is included as part of this chapter since it has not been implemented in the prototype.
    %
        \subsubsection*{Request certificates:}
        SendIt will have an automated, bi-directional sharing of certificates, that happens on every exchange. This sharing will be part of the file transfer protocol and will happen after the file transfer has completed (before terminating the connection). This automatic sharing will abstract the system away from the user, to make the system easy to use and understand, while still updating the trust-level for different identities according to recent changes in the system. The choice to make this part mandatory also allows for the system to avoid being biased by only receiving updates from certain identities.
        %
        \subsubsection*{Evaluate certificates:}
        While exchanging certificates with all communication-partners is important to avoid bias, it is also important to let the user influence and differentiate the level of trust given to each partner. This is where settings and evaluation comes into play. Each user can change the level of trust in each partner manually, based on his own preferences.

        As an extension of the trust given to each partner, the information they share is also evaluated based on how trusted the partner is. The trust evaluation of each key, and by extension each identity, will vary depending on user settings and preferences. It will however always follow the scheme as described above. The effect of this evaluation will result in a corresponding trust-level associated with the identity. Identities that have too low a trust level to be used can either raise an \emph{untrusted user warning} before communication occurs, or a \emph{connection terminated message}, depending on the user's settings. These keys will still be stored in the user's key-chain and stay available for future evaluation, even if their current level of trust is low.

    \subsection{Sybil attack}
    \label{sec:trust_syb}
    A common attack in decentralized reputation systems is the Sybil attack. It is an attack that takes advantage of the fact that the system can be manipulated by having a large number of identities share the same, false information, to innocent identities. In SendIt's case, that a certain key pair belongs to a certain identity, while in fact it does not. Through sheer numbers, the attack manipulates identities in the system to believe that the attacker should be the trusted identity, not the original identity. This misleads users to connect to the wrong identity, which makes them vulnerable to attacks. It can also lead to being unable to connect to the legitimate identity, since all traffic for that identity is instead routed to the attacker \cite{lncs_sybil}.

%
\section{WebRTC}
\label{sec:webrtc}
%
    WebRTC is a fairly new technology developed for real-time communication between web entities. It focuses on allowing video, audio, and data transfers over P2P, through an easy to use API. It does this without the need to install plugins or having to download native applications, since it is already implemented in most of today's popular browsers.

    %Connection setup
    \subsection{Connection setup}
    \label{sec:conn_setup}
    %
    To set up a connection using WebRTC, there are different variables one has to take into account. First, \emph{what are the network conditions for each endpoint?} To be able to create a connection, a way to address each endpoint directly is needed. To do this WebRTC utilizes STUN \cite{ar_stun}, and/or TURN \cite{ar_turn}, and ICE \cite{ar_ice}.
    %
    \subsubsection*{ICE, STUN, and TURN}
    \label{sec:webrtc_icetri}
    %
    \begin{figure}[th]
      \centering
      \includegraphics[width=\textwidth]{Figures/ICE_trickling}
      \decoRule
      \caption[ICE trickling]{This illustration displays how ICE trickling works. Do note that for ICE trickling, process number 3 in the illustration can happen at any time, and any number of times.}
      \label{fig:ice_trick}
    \end{figure}
    %
    The ICE protocol works by trying to connect directly to the endpoint. If an endpoint is behind symmetrical NAT, however, a TURN server is required. A TURN server acts as a relay-server that both endpoints connect to, and there is no longer a direct P2P connection. Instead each endpoint is connected to a server by a P2P connection, and the server forwards information to the other end. It is the ICE protocol that detects and manages this information, and tries to find the best possible connection \cite{ar_webrtc_sign}.

    ICE is a protocol for NAT traversal used in Offer/Answer protocols. In regular WebRTC-applications, something called ICE trickling is used. This is because generating the Offer and the Answer is quick, but gathering ICE candidates takes time. When using ICE trickling, the Offer/Answer is generated and shared independently of the ICE candidates. Once the Offer/Answer is generated, ICE candidate gathering begins. Once an ICE candidate is found, it is immediately shared. This allows the connection setup to happen first, then ICE candidates to be shared afterwards \cite{ar_ice_trickle}.  This is illustrated in \Cref{fig:ice_trick}. In SendIt, this functionality is utilized in the ACS mode.

    In Serverless mode, all the information is gathered, then attached to the Offer/Answer, and then shared. ICE is used in order to be able to address and reach computers behind NAT \cite{ar_ice}. The easiest way to describe the whole process is to imagine ICE as the decision maker, utilizing STUN and TURN to find information about the endpoint. Once all the information from STUN and TURN is gathered, ICE prioritizes the information, and shares it as separate possible ways to connect to the endpoint. These connection alternatives are called ICE candidates.
    %
    \subsubsection*{SDP and DTLS-SCTP}
    %
    Session Description Protocol (SDP) is a format for session descriptions. This is a protocol for formatting data, for example discovered through the ICE protocol, to be shared with the other endpoint \cite{ar_sdp}. For the Serverless mode, it is up to the users how they share the Offer and Answer, and as such it does not use a transfer-protocol. In the ACS mode, secure WebSockets are used for sharing the information.

    The WebRTC DataChannel used for direct communication uses DTLS-SCTP as it's transport-protocol \cite{ar_webrtc_dc}.
    %
    \begin{quotation}
        \enquote{DTLS itself is modelled upon the stream-orientated TLS, a protocol which offers full encryption with asymmetric cryptography methods, data authentication, and message authentication \cite{StudyWebRTCSecurity}.}
    \end{quotation}
    %
    In other words, the communication is securely transported between the endpoints, once the connection setup is done.

    %WRITE XEXPLICITLY! TODO
    Encapsulation of SCTP over DTLS, as well as the ICE and SDP protocols are complex. As such, an in-depth explanation is available in Holmberg, H.-C. (2015)\cite{url_webrtc_data}, for those unfamiliar with this technology. In summary: each endpoint gathers the necessary connection data and then formats it according to the SDP protocol, for session-negotiation and initiation. Then the DataChannel is created and the connection is made.
    %
    \subsubsection*{Offer and Answer}
    \label{sec:webrtc_off}
    %
    Following, is how the relevant RFC describes how endpoint guarantee is done:
    %
    \begin{quotation}
        \enquote{A certificate fingerprint is a secure one-way hash of the DER (distinguished encoding rules) form of the certificate. If the X.509 certificate presented for the TLS connection matches the fingerprint presented in the SDP, the endpoint can be confident that the author of the SDP is indeed the initiator of the connection \cite{ar_tls_sdp}.}
    \end{quotation}
    %
    This explanation can be a little hard to understand, so let us break it down. Both the Offer and the Answer generated by the WebRTC API has a fingerprint of a certificate attached. The endpoint generates a X.509 certificate and a corresponding private key. By matching the fingerprint received during signaling with the certificate used during the setup of the DTLS-SCTP connection, one can be certain that the endpoint is the same. This is illustrated in \Cref{fig:DTLS_setup}.
    %
    \begin{figure}[t]
      \centering
      \includegraphics[width=\textwidth]{Figures/DTLS_Exchange}
      \decoRule
      \caption[DTLS setup]{This illustration shows and describes how DTLS is setup between two endpoints. The final channel in the illustration is of the type SRTP, but the same procedure is true for SCTP \cite{WebRTCManMiddle2015}}
      \label{fig:DTLS_setup}
    \end{figure}
    %
    \subsection{Signaling server}
    \label{sec:sign_serv}
    %
    In the usual use case of WebRTC, a signaling server is used to share SDP Offers and Answers, as well as continuously sharing ICE candidates. This separate channel allows for re-negotiation of communication, if the connection breaks down. An Offer is the exchange of SDP information from the sender, to the receiver. An Answer is the exchange of SDP information from the receiver to the sender.

    There are several reasons for the usage of a signaling server. One is the ability to reconnect in the event of a failure in the established connection. Another is that it delays the Offer and Answer from being created until both endpoints are online, after which it is immediately created and shared. This is the case when using ACS. When using Serverless mode, however, this is not the case. This means it is affected by the fact that network conditions can change rapidly and, as such, ICE candidates may no longer be viable, leaving endpoints with no way to connect or renegotiate the connection. For more information on the lifetime of the exchange, see \Cref{Chapter5}.

    Another use of the signaling server is that it allows for encrypted communication between the parties, via the server. As long as you trust the signaling server, your communication is confidential. This is important in regards to the exchange of the Offer and Answer, as they contain information that allows a secure connection to be set up, as described in the previous section.

    Serverless mode does not have a secure channel over which the Offer and Answer is exchanged. That means it would be vulnerable to a MITM-attack. To combat this, the Offer and Answer will be encrypted by each endpoints public key for all connections after the first (See \Cref{fig:enc} for illustration). With this protection in place, an attacker can not get a hold of the Offer and/or Answer without stealing the private key that can decrypt the cipher.

%
\section{WebSockets}
\label{sec:ws}
%
To understand why WebSockets are necessary, let us first examine how normal communication between clients and a server is done. The client sends a request and the server sends a response. If the data contained in the response is time critical, it will, in many cases, already be outdated by the time it is rendered by the client. A manual way to combat this is to refresh the page. A more elegant way is the use of polling. Polling is a regularly timed, synchronous call the client makes to the server, to look for new information. This works well if you can predict when new data will be available. The problem is that this is often not the case. There are other alternatives as well, for example, long polling or streaming. However, they all come with certain issues regarding real-time, two-way communication. Especially latency and overhead are problematic.

This is where WebSockets come into play. WebSockets offers real-time, duplex, bi-directional connections. It makes communication between a client and a server easier and faster. It also supports real-time communication, which is an added bonus. WebSockets are commonly used in a lot of real-time applications in today's online environment. From transferring game data to simple chat applications. It is also commonly used for signaling in WebRTC applications. This is also how it is used in SendIt, specifically in the ACS mode.

Another great thing about WebSockets, is that they support the traditional use of TLS. This means that the WebSocket protocol can use the same security mechanisms as traditional HTTPS traffic. This makes it easy to protect the confidentiality, integrity, and availability of network communications. Finally, all modern browsers have native support for the WebSocket protocol and as such it is incredibly easy to deploy \cite{wangDefinitiveGuideHTML52013}. For these reasons, the WebSocket protocol was chosen to take care of the communication between endpoints and the server, required by the ACS mode.


\documentclass{article}

\usepackage[final]{style}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{verbatim}
\usepackage{graphicx}       % for figures
\usepackage[]{algorithm2e}

\title{Lecture \#20: Convolutional Neural Networks}

\author{
Kendall Beache, Sammy Mohammed, Hannah Zhang \\
  Department of Computer Science\\
  Stanford University\\
  Stanford, CA 94305 \\
  \texttt{\{kbeache, sammym, hzhang16\}@stanford.edu} \\
}

\begin{document}

\maketitle

\section{Introduction}
Throughout this class, we have explored several different computer vision techniques, including edge detection, clustering methods, classifiers, and feature detectors/descriptors. These techniques still have one main drawback, in that they require humans to hand-design features. In this lecture, we will go over backpropagation in neural networks, a method to recursively find the ideal weights for the neural network, and convolutional neural networks, a new method designed to solve any image processing problem the network is trained for.

A convolutional neural network is an algorithm that performs \textit{end-to-end learning}, directly mapping raw inputs (images) to a desired output, such as labels or predictions. The history of convolutional neural networks spans decades, but the biggest breakthroughs were in 2010 and 2012, with the publication of Mohamed et al's  Acoustic Modeling
using Deep Belief Networks, and Dahel et al's Context-Dependent
Pre-trained Deep Neural
Networks for Large Vocabulary
Speech Recognition. Now, convolutional neural networks and deep learning have led to rapid progress in computer vision, which we will explore. Deep learning approaches usually involve combining a small set of simple tools to build a network and then training that network on data for the specific problem you're trying to solve; they can often be adapted to different problems by simply swapping out the training data.

\section{Backpropagation in Neural Networks}

\subsection{Fundamentals of Backpropagation}

\textit{Backpropagation} is an algorithm used to build arbitrarily large neural networks, and gradients only require local information to calculate gradients, via recursive implementation of the chain rule. It allows any gradient in the network to be computed by solving the gradients in later layers. Backprop is necessary, as calculating the gradients for intermediate variables manually is difficult even with just a few layers, and does not scale well to neural networks with many layers.
\\
\begin{figure}[h]
\includegraphics[width=6cm]{Capture.png}
\centering
\caption{1D example (CS 131 lecture slide 20-10)}
\end{figure}
 \\
In order to demonstrate backprop, we will use a 1-d neural network as an example. In this case, $a = wx$, and $\hat{y} =$ max($0,a$). Our goal is to calculate $dL/dW$. Using the chain rule, we can see that $dL/dw_1 = dL/da * da/dw_1$, and $dL/da = dL/d\hat{y} * d\hat{y}/da$. We can then apply the chain rule to solve.

\subsection{Rules for Calculating Gradients}

There are some general rules that can help when calculating gradients of input features. If the operation is addition, then the gradients of the input features are distributed in that $\frac{dL}{dx_i} = \frac{dL}{dy}$ for all input features $x_i$. If the operation is multiplication, then each gradient will be proportional to the values of the other input features. For example, if there are two input features $w$ and $x$, then $\frac{dL}{dw} = x\frac{dL}{dy}$ and $\frac{dL}{dx} = w\frac{dL}{dy}$. If the operation is a maximum function, like $y = $ max$(x, w)$, then $\frac{dL}{dw} = \frac{dL}{dy}$ if $w \geq x$ and 0 otherwise; $\frac{dL}{dx} = \frac{dL}{dy}$ if $x \geq w$ and 0 otherwise. If the function is $y = e^x$, then $\frac{dL}{dx} = y \frac{dL}{dy}$, since $\frac{dy}{dx} = e^x$.

\section{Convolutional Neural Networks}

Convolutional neural networks are a type of deep learning models that are often used for object recognition and classification. At a high level, CNNs start with an original image, and convolve it with multiple filters. The parameters for convolution are also chosen via backpropagation, ensuring that we do not have to hand-choose features.

Convolutional neural networks can be broken down into 3 steps
1) Convolution, 2) Non Linearity,
3) Pooling

\subsection{Convolution:} We use convolution in this step as a means to extract features from the original input image. A CNN learns the values of the filters or kernels during the training process. Typically we use 1 2D convolution layer on black and white images; however, if you wanted to extract features from a color image you can use 3 2D convolution layers (one for each channel).

The size of the convolved feature is controlled by three parameters: depth, stride, and zero-padding that we need to decide before the convolution step is performed. Depth corresponds to the number of filters we use for the convolution operation. Stride is the number of pixels by which we slide our filter matrix over the input matrix. We use zero-padding to apply the filter to bordering elements of our input image matrix. Zero padding also allows us to control the size of the feature maps.

Introduce Non-Linearity (ReLu Function):
After every convolution, the ReLu function is applied to introduce non-linearity in our model (because most real-world data we would want our ConvNet to learn would be non-linear.) ReLU stands for Rectified Linear Unit and its output is given by the following:
\begin{figure}[h]
\includegraphics[width=6cm]{relu.png}
\centering
\caption{The ReLU function. (Kanchan Sarkar, Medium))}
\end{figure}

Other non-linear functions that can be applied to the convolved images include the sigmoid and tanh function; however, the ReLu function tends to give the best results.

\subsection{Pooling}
Modern convolutional neural networks incorporate pooling layers -- layers in the neural network that downsize the dimensionality but retain the most important information. This works by defining a spatial neighborhood, such as a 2x2 window, and then taking different data from the neighborhood depending upon the pooling algorithm. One example of this sort of pooling is MaxPool, where the max value in the window is the value that is retained. There are other pooling algorithms, such as AveragePool and SumPool, where the average is taken and the sum of the window is taken.

\begin{figure}[h]
\includegraphics[width=6cm]{maxpooling.png}
\centering
\caption{Max pooling reduces dimensionality by a factor of 2. (CS 131 lecture slide 20-38)}
\end{figure}

\subsection{Architecture}
These different layers can be stacked, feeding the input of one layer into the other in order to help improve output quality. These layers extract relevant information, and then classify.

\begin{figure}[h]
\includegraphics[width=6cm]{structure.png}
\centering
\caption{CNN Architecture (CS 131 lecture slide 20-41)}
\end{figure}

This basic structure can be expanded -- consider GoogLeNet, a CNN with an Inception layer, a hand-designed network within a network.

\begin{figure}[h]
\includegraphics[width=6cm]{Inception.png}
\centering
\caption{GoogLeNet architecture and Inception module. (CS 131 lecture slide 19-101)}
\end{figure}

However, this layering process comes with a drawback. Convolutional neural networks are not shift-invariant, due to the pooling layer -- minor shifts in the image can result in dramatically different classification outputs. (Best demonstrated here: https://richzhang.github.io/antialiased-cnns/). One potential solution would be the BlurPool algorithm, where the window is blurred and shifted before MaxPool is ran, resulting in better outcomes. However, this does not solve the shift-invariance problem, so results may still be unpredictable.

\section{Conclusion}
While CNNs meet the basic architecture described above, there are still many variations between different networks. CNNs can differ by the amount of layers or values of the hyperparameters used. If you are interested in learning more about different CNNs, read more on  GoogLeNet, VGGNet, or ZF Net.

% References
\small
\bibliographystyle{plain}
\bibliography{bibliography}
\end{document}


\documentclass[a4paper,10pt]{article}
%\documentclass[a4paper,10pt]{scrartcl}

\usepackage[utf8]{inputenc}
\usepackage{dsfont}

\title{Coupled Probabilistic Latent Semantic Analysis}
\author{John McCrae}

\pdfinfo{%
  /Title    (Coupled Probabilistic Latent Semantic Analysis)
  /Author   (John McCrae)
  /Creator  ()
  /Producer ()
  /Subject  ()
  /Keywords ()
}

\begin{document}
\maketitle

\section{Preliminaries}

Assume the observed variables are

\[
 X = ( w^l_{jn} ) ; w^l_{jn} \in \{ 1,\ldots V \}
\]

Where there are $J$ documents each of length $N^l_j$ and $l \in \{1,2\}$. The
latent variables are

\[
 Y = ( z^l_{jn} ) ; z^l_{jn} \in \{ 1,\ldots K \}
\]

\section{Model definition}

Assume we have the following model variables

\[
 \Theta = ( \phi^l_{wk}, \theta^l_{kj} )
\]

And that the model is given as
\[
 p_\Theta(X,Y) = \prod_j^J\prod_l^2\prod_n^{N^l_j}
\phi^l_{w^l_{jn}z^l_{jn}}\theta^l_{z^l_{jn}j}
\]

And we have the marginal distribution as follows:
\[
 p_\Theta(X) = \prod_j^J\prod_l^2\prod_n^{N^l_j}\prod_k^K
\phi^l_{w^l_{jn}k}\theta^l_{kj}
\]

Following \cite{platt} we define the posterior condition as

\[
 \pi_{jk}(w_j,z_j) = \frac{\sum_n^{N^2_j} \mathds{1}(z^1_{jn} = k)}{N^1_j} -
  \frac{\sum_n^{N_j^2} \mathds{1}(z^2_{jn} = k)}{N^2_j}
\]
\[
  = \frac{N^1_{jk}}{N^1_j} - \frac{N^2_{jk}}{N^2_j}
\]

And that

\[
 \pi(X,Y) = \sum_j^J\sum_k^K \pi_{jk}(w_j,z_j)
\]

Hence, the problem is to find $\Theta^*$ for some $\epsilon$ such that

\[
 \Theta^* = \arg\max_\Theta p_\Theta(Y|X)
\]

Such that

\[
 E_\Theta[\pi(X,Y)] \leq \epsilon
\]

\section{Solution}

By \cite{ganchev}, this can be solved by an E-M procedure where

\[
 \textrm{\bf{E}:}~~~~ q^{t+1} = \arg\min_q KL(q(Y) || p_{\Theta^t} (Y | X))
\]
\[
 \textrm{\bf{M}:}~~~~ \Theta^{t+1} = \arg\max_\Theta
E_{q^{t+1}}[log_\Theta(p(X,Y))]
\]

Where KL is as usual:
\[
 KL(p(X)||q(X)) = \sum_X p(x) \log\left(\frac{p(x)}{q(x)}\right)
\]

And $q(Y)$ is constrained by
\[
 q(Y) : E_q[\pi(X,Y)] \leq \epsilon
\]

By Lagrangian duality, the problem of solving
\[
 \arg\min_q KL(q(Y) || p_{\Theta} (Y | X))
\]
Subject to
\[
 E_q[\pi(X,Y)] \leq \xi ; ||\xi|| < \epsilon
\]

Is equivalent to finding
\[
 q^*(z_j) = \frac{p_\Theta(z_j|X) \exp(-\lambda^*
\theta(X,Y))}{\zeta(\lambda^*)}
\]
Where
\[
 \lambda^* = \arg\max_{\lambda \geq 0} - log(\zeta(\lambda)) - \epsilon\lambda
\]
And
\[
 \zeta(\lambda) = \sum_Y p_\Theta(Y|X) \exp(-\lambda \pi(X,Y))
 \]
 \[
 = \sum_j^J\sum_l^2 p_\Theta(z^l_j|X) e^{-
\lambda \pi_j(X,z_j)}
\]
It follows by differentiation
\[
 -\frac{\zeta'(\lambda^*)}{\zeta(\lambda^*)} - \epsilon = 0
\]
And the solution to this can be found by Newtonian gradient descent

Given we have $\lambda^*$ then we can apply

\[
 q^*(z_j) \propto p_\Theta(z_j|X) exp(-\lambda^*\pi(X,Y))
\]

And hence we can use a Gibbs-like sampling

\[
 p_\Theta(z^{tl}_{jn} = k|X) \propto
\phi^{tl}_{x_{jn}k}\theta^{tl}_{kj}exp(-\lambda^*\pi_{jk}(w_j,z_j))
\]

Finally the maximization step is trivial

\[
 \phi^l_{wk} = \frac{N^l_{wk}}{N^l_k}
\]

\[
 \theta^l_{kj} = \frac{N^l_{kj}}{N^l_j}
\]

\section{Dirichlet assumption}

The Dirichlet assumption is that

\[
 \theta^l_j \sim Dir(\alpha)
\]
\[
 \phi^l_k \sim Dir(\beta)
\]

The result of which is that the maximization step is generalized
to~\cite{porteous}

\[
 \theta^l_{kj} = \frac{N^l_{kj} + \alpha}{N^l_k + K\alpha}
\]
\[
 \phi^l_{wk} = \frac{N^l_{wk} + \beta}{N^l_k + V \beta}
\]

\section{Initialization}

\begin{enumerate}
 \item Sort words by frequency $\{w'_1,\ldots w'_V\}$
 \item Find $K-1$ values, $\kappa_i$, such that $\kappa_i < \kappa_{i+1}$ and
   $\sum_{j = \kappa_i}^{\kappa_{i+1}} w'_j \leq \frac{N}{K}$
 \item Initialize $z^l_{jn} = k$ where $w^l_{jn} = w'_i$ and $\kappa_k \leq i <
\kappa_{i+1}$
 \item Calculate initial $\Theta$

\section{Notes}

\cite{platt} used 500 iterations at $\alpha=1.1$, $\beta=1.01$.

\end{enumerate}

\begin{thebibliography}{9}

\bibitem{platt}
John C. Platt and Kristina Toutanova and Wen-tau Yih (2010).
Translingual Document Representations from Discriminative Projections.

\bibitem{ganchev}
Kuzman Ganchev and João Graça and Jennifer Gillenwater and Ben Taskar (2010).
Posterior Regularization for Structured Latent Variable Models.

\bibitem{porteous}
Ian Porteous and David Newman and Alexander Ihler and Arthur Asuncion
and Padhraic Smyth and Max Welling (2008).
Fast Collapsed Gibbs Sampling For Latent Dirichlet Allocation.

\end{thebibliography}

\end{document}


%%!TEX TS-program = latex
\documentclass[]{beamer}
%\usepackage{helvet}
%\usepackage{pstricks,pst-node,pst-tree}
\usepackage{graphicx}
\hypersetup{pdfpagemode=FullScreen}
\usetheme{Singapore}
%\usetheme{copenhagen}
%\usetheme{Boadilla}
%\usetheme{Warsaw}
\usecolortheme{seagull}
\setbeamercovered{transparent}
\beamertemplatenavigationsymbolsempty
\setbeamertemplate{footline}[frame number]

\usepackage{hyperref}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{dsfont}
\usepackage{multicol}
\usepackage{multirow}

\newcommand{\R}{\ensuremath{\mathds{R}}}
\newcommand{\C}{\ensuremath{\mathds{C}}}
\newcommand{\Q}{\ensuremath{\mathds{Q}}}
\newcommand{\N}{\ensuremath{\mathcal{N}}}
\newcommand{\Z}{\ensuremath{\mathds{Z}}}

\setbeamerfont{bib}{size*={4.00}{4.00}}
\usepackage[square, sort, numbers, authoryear]{natbib}
\renewcommand{\bibsection}{\subsubsection*{\bibname } }

\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\Corr}{Corr}
\DeclareMathOperator*{\E}{E}
\DeclareMathOperator*{\sign}{sign}
\renewcommand{\vec}[1]{\mathbf{#1}}
\usepackage{hyperref}

\usepackage{multicol}
\usepackage{multirow}
\usepackage{pbox}

\institute[]{}
%\logo{\pgfimage[width=.8cm,height=.8cm]{../KU_logo}}
\title[Active Manifesto]{
{
Speeding up the Manifesto Project: \\ Active learning strategies for \\efficient automated political annotations
}}
\author{
Felix Biessmann\thanks{felix.biessmann@gmail.com},~
Philipp Schmidt\thanks{schmidtiphil@gmail.com}
}
\date{}
\begin{document}

\begin{frame}
\titlepage
\end{frame}

%
\section{Intro}
\subsection{}

\begin{frame}\frametitle{Disclaimers}
\small
\begin{itemize}
\item (For us) This open source project is a hobby
\item It has nothing to do with our job
\item Apologies if we missed to cite somebody in this room
\item We'd be excited to hear about more related work
\end{itemize}
\end{frame}

\begin{frame}\frametitle{Motivation}
\begin{itemize}[<+->]
\item Automated political analysis required for
\begin{itemize}
\item Political scientists
\item Journalists
\item Average media consumer
\end{itemize}
\item ML models need in-domain training data \footnote{\cite{Biessmann16}}
\item But annotation budget is often limited:
\begin{itemize}
\item Temporal constraints (before elections) \footnote{\cite{merz2017}, \cite{bronline}}
\item Online news media (too much content)
\end{itemize}
\item[$\rightarrow$] If you cannot annotate all texts:\\
\centering How to choose which texts to annotate (first)?
\end{itemize}
\end{frame}

\begin{frame}\frametitle{Active Learning}
\begin{itemize}[<+->]
\item Given limited annotation budget, find the best model
\item How? \\
\begin{itemize}
\item Annotate difficult ones\footnote{For which model is most uncertain.} first
\end{itemize}
\item Why?
\begin{itemize}
\item Intuition: \\
\textit{ Model learns most from difficult examples}\\
\item Math:\\
\textit{ Gradient of loss function is larger for difficult examples }\\
\end{itemize}
\end{itemize}
\end{frame}

\section{Methods}
\subsection{}

\begin{frame}\frametitle{Data}

\begin{itemize}
\item All annotated German texts from:\\
\url{https://manifestoproject.wzb.eu/}
\item Custom python tooling for manifesto API:\\
{\footnotesize \url{https://github.com/felixbiessmann/active-manifesto} }\\
\item Only texts with more than 1000 observed labels
\end{itemize}
\end{frame}

\begin{frame}\frametitle{Model}
\begin{itemize}
\item Preprocessing
\begin{itemize}
\item Unigram Bag-of-Words features
\item Hashing Vectorizer
\end{itemize}
\item Classification Model: Multinomial Logistic Regression
\begin{eqnarray}\label{eq:logreg_multiclass}
p(y=k|\vec{x}) = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}  \textrm{ with }  z_k=\vec{w}_k^{\top}\vec{x}.
\end{eqnarray}
With
\begin{itemize}
\item Labels $y\in\{1,2,\dots,K\}$ (manifesto code)\\
\item $\vec{w}_1,\dots,\vec{w}_K\in\R^{d}$ weight vectors of $k$th manifesto code\\
%\item $L_2$ norm regularization of weights
\end{itemize}
\end{itemize}
\end{frame}

\begin{frame}\frametitle{Offline Experiments}

\begin{itemize}
\item Train model on 1\%, 10\%, 20\%, \dots, 100\% of training data
\item Vary sampling strategies to select from unlabelled texts
\item Compute accuracy on hold-out data
\end{itemize}
\end{frame}

\begin{frame}\frametitle{Active Learning Strategies}

\begin{itemize}
\item Random Baseline: Uniform random sampling
\item Uncertainty Sampling: Only top-prediction counts
\begin{align}\label{eq:uncertainty_sampling}
\vec{x}_i = \argmax_{i,k} \left(1- p(y=k|\vec{x}_i,\vec{W})\right)
\end{align}
\item Entropy Sampling: All predictions count
\begin{align}\label{eq:entropy_sampling}
\vec{x}_i = \argmax_{i} \sum_k p(y=k|\vec{x}_i,\vec{W}) \log(p(y=k|\vec{x}_i,\vec{W}))
\end{align}
\item Margin Sampling: Top 2 predictions count
\begin{align}\label{eq:entropy_sampling}
\vec{x}_i = \argmin_{i} \left(p(y=k_1|\vec{x}_i,\vec{W}) - p(y=k_2|\vec{x}_i,\vec{W}) \right)
\end{align}
\end{itemize}
\end{frame}

\section{Results}
\subsection{}

\begin{frame}\frametitle{Results: 'Perfect' Reference Model}
\footnotesize
\begin{table}
\centering
\begin{tabular}{cccccc}
\toprule
&  manifesto code & precision  &  recall&  f1-score &  support\\
\midrule
&   107&  0.60& 0.48& 0.53&  774\\
&   201&  0.51& 0.55& 0.53& 1194\\
&   202&  0.63& 0.57& 0.60&  983\\
&   305&  0.46& 0.59& 0.52&  783\\
&   403&  0.52& 0.48& 0.50& 1281\\
&   411&  0.39& 0.60& 0.47& 1535\\
&   501&  0.61& 0.55& 0.58& 1380\\
&   502&  0.65& 0.41& 0.50&  587\\
&   503&  0.46& 0.52& 0.49& 2083\\
&   506&  0.63& 0.48& 0.54& 1026\\
&   605&  0.56& 0.44& 0.49&  576\\
&   701&  0.59& 0.39& 0.47& 1123\\
\bottomrule
& avg / total&  0.50& 0.48& 0.48&17559\\
\end{tabular}
\caption{Precision, recall, F1 score and number of instances per class. }
\label{tab:baseline_model_report}
\end{table}

\end{frame}

\begin{frame}\frametitle{Active Learning Results}
\begin{center}
\includegraphics[width=\textwidth]{images/active_learning_manifesto.pdf} \\
Median accuracy and the 5th/95th percentile across 100 repetitions \\
\end{center}

\end{frame}

\section{Conclusion}
\subsection{}

\begin{frame}\frametitle{Conclusion}
\begin{itemize}
\item Automated political analysis requires annotations
\item \textbf{Limited budged} for annotations of political texts
\item Active Learning
\begin{itemize}
\item Helps to select which texts to annotate
\item Perfect model with 80\% of data
\item Almost perfect (over 90\%) with 50\% of data
\end{itemize}
\item[$\rightarrow$] Active learning can speed up political annotations.
\item Code:\\
\footnotesize
 \url{https://github.com/felixbiessmann/active-manifesto}
\end{itemize}
\end{frame}

\begin{frame}\frametitle{Limitations}
\begin{itemize}
\item We used a simple model
\item Random sampling is an unrealistic baseline
\item We only performed offline experiments
\item[$\rightarrow$] More convincing: online experiments
\end{itemize}
\end{frame}

\section{Demo}
\subsection{}

\begin{frame}\frametitle{Demo \url{http://rightornot.info}}
\begin{itemize}
\item Goal: Collect Annotations with Active Learning
\begin{enumerate}
\item For political analysis of non-manifesto texts
\item For comparing manifesto annotations with laymen judgements
\end{enumerate}
\item Incentive for users:
\begin{enumerate}
\item Estimate your political bias
\item Escape your political filter bubble
\end{enumerate}
\end{itemize}
\centering
\end{frame}

\begin{frame}\frametitle{Demo \url{http://rightornot.info}}
\centering
\includegraphics[width=.9\textwidth]{images/active_learning_manifesto-left-neutral-right}\\
Labels: Left, Neutral, Right
\end{frame}

\begin{frame}\frametitle{Demo \url{http://rightornot.info}}
\centering
\includegraphics[width=.6\textwidth]{images/web-demo}\\
\end{frame}

\begin{frame}\frametitle{References}
\usebeamerfont{bib}

\bibliographystyle{abbrvnat}
\def\newblock{}
\vspace{2em}
\bibliography{active-manifesto}
\end{frame}

\end{document}


%% LyX 2.2.2 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english]{beamer}
\usepackage{mathptmx}
\usepackage{eulervm}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}

\makeatletter

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
%% A simple dot to overcome graphicx limitations
\newcommand{\lyxdot}{.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Textclass specific LaTeX commands.
 % this default might be overridden by plain title style
 \newcommand\makebeamertitle{\frame{\maketitle}}%
 % (ERT) argument for the TOC
 \AtBeginDocument{%
   \let\origtableofcontents=\tableofcontents
   \def\tableofcontents{\@ifnextchar[{\origtableofcontents}{\gobbletableofcontents}}
   \def\gobbletableofcontents#1{\origtableofcontents}
 }

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
\usetheme{CambridgeUS}
\beamertemplatenavigationsymbolsempty

\AtBeginSection[]{
  \begin{frame}
  \vfill
  \centering
  \begin{beamercolorbox}[sep=8pt,center,shadow=true,rounded=true]{title}
    \usebeamerfont{title}\insertsectionhead\par%
  \end{beamercolorbox}
  \vfill
  \end{frame}
}

\makeatother

\usepackage{babel}
\begin{document}

\global\long\def\reals{\mathbf{R}}
\global\long\def\RR{\mathbf{R}}
 \global\long\def\integers{\mathbf{Z}}
 \global\long\def\naturals{\mathbf{N}}
 \global\long\def\rationals{\mathbf{Q}}
 \global\long\def\ca{\mathcal{A}}
 \global\long\def\cb{\mathcal{B}}
 \global\long\def\cc{\mathcal{C}}
 \global\long\def\cd{\mathcal{D}}
 \global\long\def\ce{\mathcal{E}}
 \global\long\def\cf{\mathcal{F}}
 \global\long\def\cg{\mathcal{G}}
 \global\long\def\ch{\mathcal{H}}
 \global\long\def\ci{\mathcal{I}}
 \global\long\def\cj{\mathcal{J}}
 \global\long\def\ck{\mathcal{K}}
 \global\long\def\cl{\mathcal{L}}
 \global\long\def\cm{\mathcal{M}}
 \global\long\def\cn{\mathcal{N}}
 \global\long\def\co{\mathcal{O}}
 \global\long\def\cp{\mathcal{P}}
 \global\long\def\cq{\mathcal{Q}}
 \global\long\def\calr{\mathcal{R}}
 \global\long\def\cs{\mathcal{S}}
 \global\long\def\ct{\mathcal{T}}
 \global\long\def\cu{\mathcal{U}}
 \global\long\def\cv{\mathcal{V}}
 \global\long\def\cw{\mathcal{W}}
 \global\long\def\cx{\mathcal{X}}
 \global\long\def\cy{\mathcal{Y}}
 \global\long\def\cz{\mathcal{Z}}
 \global\long\def\ind#1{1(#1)}
 %\newcommand{\pr}{P}
\global\long\def\pr{\mathbb{P}}
 \global\long\def\predsp{\cy}
 %{\hat{\cy}}
\global\long\def\outsp{\cy}

\global\long\def\prxy{P_{\cx\times\cy}}
 \global\long\def\prx{P_{\cx}}
 \global\long\def\prygivenx{P_{\cy\mid\cx}}
 %\newcommand{\ex}{E}
\global\long\def\ex{\mathbb{E}}
 \global\long\def\var{\textrm{Var}}
 \global\long\def\Ind{\mathbf{1}}
 \global\long\def\cov{\textrm{Cov}}
 \global\long\def\sgn{\textrm{sgn}}
 \global\long\def\sign{\textrm{sign}}
 \global\long\def\kl{\textrm{KL}}
 \global\long\def\law{\mathcal{L}}
 \global\long\def\eps{\varepsilon}
 \global\long\def\as{\textrm{ a.s.}}
 \global\long\def\io{\textrm{ i.o.}}
 \global\long\def\ev{\textrm{ ev.}}
 \global\long\def\convd{\stackrel{d}{\to}}
 \global\long\def\eqd{\stackrel{d}{=}}
 \global\long\def\del{\nabla}
 \global\long\def\loss{\ell}
 \global\long\def\risk{R}
 \global\long\def\emprisk{\hat{R}}
 \global\long\def\lossfnl{L}
 \global\long\def\emplossfnl{\hat{L}}
 \global\long\def\empminimizer#1{\hat{#1}^{*}}
 \global\long\def\minimizer#1{#1^{*}}
\global\long\def\optimizer#1{#1^{*}}
 \global\long\def\etal{\textrm{et. al.}}
 \global\long\def\tr{\operatorname{tr}}

\global\long\def\trace{\operatorname{trace}}
 \global\long\def\diag{\text{diag}}
 \global\long\def\rank{\text{rank}}
 \global\long\def\linspan{\text{span}}
 \global\long\def\spn{\text{span}}
 \global\long\def\proj{\text{Proj}}
 \global\long\def\argmax{\operatornamewithlimits{arg\, max}}
 \global\long\def\argmin{\operatornamewithlimits{arg\, min}}

\global\long\def\bfx{\mathbf{x}}
 \global\long\def\bfy{\mathbf{y}}
 \global\long\def\bfl{\mathbf{\lambda}}
 \global\long\def\bfm{\mathbf{\mu}}
 \global\long\def\calL{\mathcal{L}}

\global\long\def\vw{\boldsymbol{w}}
 \global\long\def\vx{\boldsymbol{x}}
 \global\long\def\vxi{\boldsymbol{\xi}}
 \global\long\def\valpha{\boldsymbol{\alpha}}
 \global\long\def\vbeta{\boldsymbol{\beta}}
 \global\long\def\vsigma{\boldsymbol{\sigma}}
\global\long\def\vtheta{\boldsymbol{\theta}}
 \global\long\def\vd{\boldsymbol{d}}
 \global\long\def\vs{\boldsymbol{s}}
 \global\long\def\vt{\boldsymbol{t}}
 \global\long\def\vh{\boldsymbol{h}}
 \global\long\def\ve{\boldsymbol{e}}
 \global\long\def\vf{\boldsymbol{f}}
 \global\long\def\vg{\boldsymbol{g}}
 \global\long\def\vz{\boldsymbol{z}}
 \global\long\def\vk{\boldsymbol{k}}
 \global\long\def\va{\boldsymbol{a}}
 \global\long\def\vb{\boldsymbol{b}}
 \global\long\def\vv{\boldsymbol{v}}
 \global\long\def\vy{\boldsymbol{y}}

\global\long\def\hil{\ch}
 \global\long\def\rkhs{\hil}
 \global\long\def\ber{\text{Ber}}

\title[DS-GA 1003 ]{Gaussian Mixture Models}

\author{David Rosenberg, Brett Bernstein}

\date{\today}

\institute{New York University}

\makebeamertitle

\section{Intro Question}
\begin{frame}{Intro Question}
  Suppose we begin with a dataset $\cd = \{x_1,\ldots,x_n\}\subseteq\RR^2$
  and we run $k$-means (or $k$-means$++$) to obtain $k$ cluster
  centers.  Below we have drawn the cluster centers.  If we are given
  a new $x\in\RR^2$, we can assign it a label based on which cluster
  center is closest.  What regions of the plane below correspond to
  each possible labeling?
  \begin{center}
    %trim={<left> <lower> <right> <upper>}
    \includegraphics[height=.7\textheight,trim={0 6cm 0 8cm}]{Figures/clustering/vorpoints.pdf}
  \end{center}
\end{frame}
\begin{frame}{Intro Solution}
  \begin{itemize}
  \item Note that each cell is disjoint (except for the boarders), and
    convex.
  \item This can be thought of as a limitation of $k$-means: neither
    will be true for GMMs.
  \end{itemize}
  \begin{center}
    \includegraphics[height=.9\textheight,trim={0 5cm 0 8cm}]{Figures/clustering/voronoi.pdf}
  \end{center}
\end{frame}
\section{Gaussian Mixture Models}
\begin{frame}{Yesterday's Intro Question}
  Consider the following probability model for generating data.
  \begin{enumerate}
  \item Roll a weighted $k$-sided die to choose a label
    $z\in\{1,\ldots,k\}$.  Let $\pi$ denote the PMF for the die.
  \item Draw $x\in\reals^d$ randomly from the multivariate normal
    distribution $\cn(\mu_z,\Sigma_z)$.
  \end{enumerate}
  Solve the following questions.
  \begin{enumerate}
  \item What is the joint distribution of $x,z$ given $\pi$ and the
    $\mu_z,\Sigma_z$ values?
  \item Suppose you were given the dataset
    $\cd=\{(x_1,z_1),\ldots,(x_n,z_n)\}$.  How would you estimate the
    die weightings, and the $\mu_z,\Sigma_z$ values?
  \item How would you determine the label for a new datapoint $x$?
  \end{enumerate}
\end{frame}
\begin{frame}{Yesterday's Intro Solution}
  \begin{enumerate}
  \item The joint PDF/PMF is given by
    $$p(x,z) = \pi(z)f(x;\mu_z,\Sigma_z)$$
    where
    $$f(x;\mu_z,\Sigma_z) =
    \frac{1}{\sqrt{|2\pi\Sigma_z|}}\exp\left(-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right).$$
  \item We could use maximum likelihood estimation.  Our estimates are
    $$\begin{array}{rcl}
    n_z  & = & \sum_{i=1}^n \Ind(z_i=z) \\
    \hat{\pi}(z) & = & \frac{n_z}{n}\\
    \hat{\mu}_z & = & \frac{1}{n_z}\sum_{i:z_i=z} x_i\\
    \hat{\Sigma}_z & = & \frac{1}{n_z}\sum_{i:z_i=z} (x_i-\hat{\mu}_z)(x_i-\hat{\mu}_z)^T.
    \end{array}$$
  \item $\argmax_z p(x,z)$
  \end{enumerate}
\end{frame}
\begin{frame}{Probabilistic Model for Clustering}

\begin{itemize}
\item Let's consider a \textbf{generative model} for the data.
\item Suppose

\begin{enumerate}
\item There are $k$ clusters.
\item We have a probability density for each cluster.
\end{enumerate}
\item Generate a point as follows

\begin{enumerate}
\item Choose a random cluster $z\in\left\{ 1,2,\ldots,k\right\} $.
\item Choose a point from the distribution for cluster $Z$.
\end{enumerate}
\item The clustering algorithm is then:
  \begin{enumerate}
  \item Use training data to fit the parameters of the generative
    model.
  \item For each point, choose the cluster with the highest likelihood
    based on model.
  \end{enumerate}
\end{itemize}
\end{frame}

\begin{frame}{Gaussian Mixture Model ($k=3$)}

\begin{enumerate}
\item Choose $z\in\left\{ 1,2,3\right\}$
\item Choose $x\mid z\sim\cn\left(X\mid\mu_{z},\Sigma_{z}\right)$.
\end{enumerate}
\begin{center}
\includegraphics[width=0.7\textwidth]{../Figures/clustering/mixture-3-gaussians}
\par\end{center}

\end{frame}

\begin{frame}{Gaussian Mixture Model Parameters ($k$ Components)}

\begin{center}
\begin{eqnarray*}
\text{Cluster probabilities}: &  & \pi=\left(\pi_{1},\ldots,\pi_{k}\right)\\
\text{Cluster means}: &  & \mu=\left(\mu_{1},\ldots,\mu_{k}\right)\\
\text{Cluster covariance matrices:} &  & \Sigma=\left(\Sigma_{1},\ldots\Sigma_{k}\right)
\end{eqnarray*}
\begin{itemize}
\item What if one cluster had many more points than another cluster?
\end{itemize}
\end{center}
\end{frame}

\begin{frame}{Gaussian Mixture Model: Joint Distribution }

\begin{itemize}
\item Factorize the joint distribution:
\begin{eqnarray*}
p(x,z) & = & p(z)p(x\mid z)\\
& = & \pi_{z}\cn\left(x\mid\mu_{z},\Sigma_{z}\right)
\end{eqnarray*}

\begin{itemize}
\item $\pi_{z}$ is probability of choosing cluster $z$.

\item $x\mid z$ has distribution $\cn(\mu_{z},\Sigma_{z})$.

\item $z$ corresponding to $x$ is the true cluster assignment.
\end{itemize}
\end{itemize}

\begin{itemize}
\item Suppose we know all the parameters of the model.
\item Then we can easily compute the joint $p(x,z)$, and the
  conditional $p(z\mid x)$.
\end{itemize}
\end{frame}

\begin{frame}{Latent Variable Model }

\begin{itemize}
\item We observe $x$.
\item In the intro problem we had labeled data, but here we don't observe
  $z$, the cluster assignment.

\item Cluster assignment $z$ is called a \textbf{hidden} \textbf{variable}
or \textbf{latent variable}.

\end{itemize}
\begin{definition}
A\textbf{ latent variable model }is a probability model for which
certain variables are never observed.

\end{definition}

e.g. The Gaussian mixture model is a latent variable model.
\end{frame}

\begin{frame}{The GMM ``Inference'' Problem}

\begin{itemize}
\item We observe $x$. We want to know $z$.

\item The conditional distribution of the cluster $z$ given $x$ is
\[
p(z\mid x)=p(x,z)/p(x)
\]

\item The conditional distribution is a\textbf{ soft assignment }to clusters.

\item A \textbf{hard assignment }is
\[
z^{*}=\argmax_{z\in\left\{ 1,\ldots,k\right\} }p(z\mid x).
\]

\item So if we have the model, clustering is trivial.
\end{itemize}
\end{frame}

\section{Mixture Models}
\begin{frame}{Gaussian Mixture Model: Marginal Distribution}

\begin{itemize}
\item The \textbf{marginal distribution} for a single observation $x$ is\textbf{
\begin{eqnarray*}
p(x) & = & \sum_{z=1}^{k}p(x,z)\\
 & = & \sum_{z=1}^{k}\pi_{z}\cn\left(x\mid\mu_{z},\Sigma_{z}\right)
\end{eqnarray*}
}

\item Note that $p(x)$ is a convex combination of probability densities.

\item This is a common form for a probability model...
\end{itemize}
\end{frame}

\begin{frame}{Mixture Distributions (or Mixture Models)}

\begin{definition}
A probability density $p(x)$ represents a \textbf{mixture distribution
}or \textbf{mixture model, }if we can write it as a \textbf{convex
combination} of probability densities. That is,
\[
p(x)=\sum_{i=1}^{k}w_{i}p_{i}(x),
\]
where $w_{i}\ge0$, $\sum_{i=1}^{k}w_{i}=1$, and each $p_{i}$ is
a probability density.

\end{definition}

\begin{itemize}
\item In our Gaussian mixture model, $x$ has a \textbf{mixture distribution}.

\item More constructively, let $S$ be a set of probability distributions:

\begin{enumerate}
\item Choose a distribution randomly from $S$.
\item Sample $x$ from the chosen distribution.
\end{enumerate}
\item Then $x$ has a mixture distribution.
\end{itemize}
\end{frame}

\section{Learning in Gaussian Mixture Models}

\begin{frame}{The GMM ``Learning'' Problem}

\begin{itemize}
\item Given data $x_{1},\ldots,x_{n}$ drawn from a GMM,
\item Estimate the parameters:
\begin{eqnarray*}
\text{Cluster probabilities}: &  & \pi=\left(\pi_{1},\ldots,\pi_{k}\right)\\
\text{Cluster means}: &  & \mu=\left(\mu_{1},\ldots,\mu_{k}\right)\\
\text{Cluster covariance matrices:} &  & \Sigma=\left(\Sigma_{1},\ldots\Sigma_{k}\right)
\end{eqnarray*}

\item Once we have the parameters, we're done.

\item Just do ``inference'' to get cluster assignments.
\end{itemize}
\end{frame}

\begin{frame}{Estimating/Learning the Gaussian Mixture Model}

\begin{itemize}
\item One approach to learning is \textbf{maximum likelihood}

\begin{itemize}
\item find parameter values that give \textbf{observed data }the\textbf{
highest likelihood}.
\end{itemize}
\item The model likelihood for $\cd=\left\{ x_{1},\ldots,x_{n}\right\} $
is
\begin{eqnarray*}
L(\pi,\mu,\Sigma) & = & \prod_{i=1}^{n}p(x_{i})\\
 & = & \prod_{i=1}^{n}\sum_{z=1}^{k}\pi_{z}\cn\left(x_{i}\mid\mu_{z},\Sigma_{z}\right).
\end{eqnarray*}

\item As usual, we'll take our objective function to be the log of this:
\begin{eqnarray*}
J(\pi,\mu,\Sigma) & = & \sum_{i=1}^{n}\log\left\{ \sum_{z=1}^{k}\pi_{z}\cn\left(x_{i}\mid\mu_{z},\Sigma_{z}\right)\right\}
\end{eqnarray*}

\end{itemize}
\end{frame}

\begin{frame}{Properties of the GMM Log-Likelihood}

\begin{itemize}
\item GMM log-likelihood:
\begin{eqnarray*}
J(\pi,\mu,\Sigma) & = & \sum_{i=1}^{n}\log\left\{ \sum_{z=1}^{k}\pi_{z}\cn\left(x_{i}\mid\mu_{z},\Sigma_{z}\right)\right\}
\end{eqnarray*}

\item Let's compare to the log-likelihood for a single Gaussian:
\begin{eqnarray*}
 &  & \sum_{i=1}^{n}\log\cn\left(x_{i}\mid\mu,\Sigma\right)\\
 & = & -\frac{nd}{2}\log\left(2\pi\right)-\frac{n}{2}\log\left|\Sigma\right|-\frac{1}{2}\sum_{i=1}^{n}(x_{i}-\mu)'\Sigma^{-1}(x_{i}-\mu)
\end{eqnarray*}

\item For a single Gaussian, the $\log$ cancels the $\exp$ in the Gaussian
density.

\begin{itemize}
\item $\implies$ Things simplify a lot.
\end{itemize}

\item For the GMM, the sum inside the $\log$ prevents this cancellation.

\begin{itemize}
\item $\implies$ Expression more complicated. No closed form expression
for MLE.
\end{itemize}
\end{itemize}
\end{frame}

\section{Issues with MLE for GMM}
\begin{frame}{Identifiability Issues for GMM}

\begin{itemize}
\item Suppose we have found parameters
\begin{eqnarray*}
\text{Cluster probabilities}: &  & \pi=\left(\pi_{1},\ldots,\pi_{k}\right)\\
\text{Cluster means}: &  & \mu=\left(\mu_{1},\ldots,\mu_{k}\right)\\
\text{Cluster covariance matrices:} &  & \Sigma=\left(\Sigma_{1},\ldots\Sigma_{k}\right)
\end{eqnarray*}
 that are at a local minimum.
\end{itemize}

\begin{itemize}
\item What happens if we shuffle the clusters? e.g. Switch the labels for
clusters 1 and 2.
\end{itemize}

\begin{itemize}
\item We'll get the same likelihood. How many such equivalent settings are
there?
\end{itemize}

\begin{itemize}
\item Assuming all clusters are distinct, there are $k!$ equivalent solutions.
\end{itemize}

\begin{itemize}
\item Not a problem per se, but something to be aware of.
\end{itemize}
\end{frame}

\begin{frame}{Singularities for GMM}

\begin{itemize}
\item Consider the following GMM for 7 data points:
\end{itemize}
\begin{center}
\includegraphics[height=0.4\textheight]{../Figures/clustering/gmm-singularity}
\par\end{center}

\begin{itemize}
\item Let $\sigma^{2}$ be the variance of the skinny component.
\item What happens to the likelihood as $\sigma^{2}\to0$?

\item In practice, we end up in local minima that do not have this problem.

\begin{itemize}
\item Or keep restarting optimization until we do.
\end{itemize}

\item Bayesian approach or regularization will also solve the problem.
\end{itemize}
\let\thefootnote\relax\footnotetext{\tiny{From Bishop's \emph{Pattern recognition and machine learning}, Figure 9.7.}}
\end{frame}

\begin{frame}{Gradient Descent / SGD for GMM}

\begin{itemize}
\item What about running gradient descent or SGD on
\begin{eqnarray*}
J(\pi,\mu,\Sigma) & = & -\sum_{i=1}^{n}\log\left\{ \sum_{z=1}^{k}\pi_{z}\cn\left(x_{i}\mid\mu_{z},\Sigma_{z}\right)\right\} ?
\end{eqnarray*}

\item Can be done \textendash{} but need to be clever about it.

\item Each matrix $\Sigma_{1},\ldots,\Sigma_{k}$ has to be positive semidefinite.

\item How to maintain that constraint?

\begin{itemize}
\item Rewrite $\Sigma_{i}=M_{i}M_{i}^{T}$, where $M_{i}$ is an unconstrained
matrix.

\item Then $\Sigma_{i}$ is positive semidefinite.

\end{itemize}
\end{itemize}
\end{frame}

\section{The EM Algorithm for GMM}
\begin{frame}{MLE for GMM}

\begin{itemize}
\item From yesterday's intro questions, we know that we can solve the MLE
  problem if the cluster assignments $z_i$ are known
  \begin{eqnarray*}
    n_z  & = & \sum_{i=1}^n \Ind(z_i=z)\\
    \hat{\pi}(z) & = & \frac{n_z}{n}\\
    \hat{\mu}_z & = & \frac{1}{n_z}\sum_{i:z_i=z} x_i\\
    \hat{\Sigma}_z & = & \frac{1}{n_z}\sum_{i:z_i=z} (x_i-\hat{\mu}_z)(x_i-\hat{\mu}_z)^T.
  \end{eqnarray*}
\item In the EM algorithm we will modify the equations to handle
  our evolving \textbf{soft assignments}, which we will call \textbf{responsibilities}.
\end{itemize}
\end{frame}

\begin{frame}{Cluster Responsibilities: Some New Notation}

\begin{itemize}
\item Denote the probability that observed value $x_{i}$ comes from cluster
$j$ by
\[
\gamma_{i}^{j}=\pr\left(Z=j\mid X=x_{i}\right).
\]

\item The \textbf{responsibility }that cluster $j$ takes for observation
$x_{i}$.

\item Computationally,
\begin{eqnarray*}
\gamma_{i}^{j} & = & \pr\left(Z=j\mid X=x_{i}\right).\\
 & = & p\left(Z=j,X=x_{i}\right)/p(x)\\
 & = & \frac{\pi_{j}\cn\left(x_{i}\mid\mu_{j},\Sigma_{j}\right)}{\sum_{c=1}^{k}\pi_{c}\cn\left(x_{i}\mid\mu_{c},\Sigma_{c}\right)}
\end{eqnarray*}

\item The vector $\left(\gamma_{i}^{1},\ldots,\gamma_{i}^{k}\right)$ is
exactly the \textbf{soft assignment} for $x_{i}$.

\item Let $n_{c}=\sum_{i=1}^{n}\gamma_{i}^{c}$ be the ``number'' of points
\textbf{soft assigned} to cluster $c$.
\end{itemize}
\end{frame}
\begin{frame}{EM Algorithm for GMM: Overview}
\begin{itemize}
  \item If we know $\pi$ and $\mu_j,\Sigma_j$ for all $j$ then we can
    easily find $\gamma_i^j=\pr(Z=j\mid X=x_i)$.
  \item If we know the (soft) assignments, we can easily find
    estimates for $\pi$, $\mu_j,\Sigma_j$ for all $j$.
  \item Repeatedly alternate the previous 2 steps.
\end{itemize}
\end{frame}
\begin{frame}{EM Algorithm for GMM: Overview}

\begin{enumerate}
\item Initialize parameters $\mu,\Sigma,\pi$.

\item ``E step''. Evaluate the responsibilities using current parameters:
\[
\gamma_{i}^{j}=\frac{\pi_{j}\cn\left(x_{i}\mid\mu_{j},\Sigma_{j}\right)}{\sum_{c=1}^{k}\pi_{c}\cn\left(x_{i}\mid\mu_{c},\Sigma_{c}\right)},\quad \text{for $i=1,\ldots,n$ and $j=1,\ldots,k$.}
\]

\item ``M step''. Re-estimate the parameters using responsibilities.
  [Compare with intro question.]
\begin{eqnarray*}
\mu_{c}^{\text{new}} & = & \frac{1}{n_{c}}\sum_{i=1}^{n}\gamma_{i}^{c}x_{i}\\
\Sigma_{c}^{\text{new}} & = & \frac{1}{n_{c}}\sum_{i=1}^{n}\gamma_{i}^{c}\left(x_{i}-\mu_c^{\text{new}}\right)\left(x_{i}-\mu_c^{\text{new}}\right)^{T}\\
\pi_{c}^{\text{new}} & = & \frac{n_{c}}{n},
\end{eqnarray*}
\item Repeat from Step 2, until log-likelihood converges.
\end{enumerate}
\end{frame}

\begin{frame}{EM for GMM}

\begin{itemize}
\item Initialization
\end{itemize}
\begin{center}
\includegraphics[height=0.55\textheight]{../Figures/clustering/9\lyxdot 8a}
\par\end{center}

\let\thefootnote\relax\footnotetext{\tiny{From Bishop's \emph{Pattern recognition and machine learning}, Figure 9.8.}}

\end{frame}

\begin{frame}{EM for GMM}

\begin{itemize}
\item First soft assignment:
\end{itemize}
\begin{center}
\includegraphics[height=0.55\textheight]{../Figures/clustering/9\lyxdot 8b}
\par\end{center}

\let\thefootnote\relax\footnotetext{\tiny{From Bishop's \emph{Pattern recognition and machine learning}, Figure 9.8.}}

\end{frame}

\begin{frame}{EM for GMM}

\begin{itemize}
\item First soft assignment:
\end{itemize}
\begin{center}
\includegraphics[height=0.55\textheight]{../Figures/clustering/9\lyxdot 8c}
\par\end{center}

\let\thefootnote\relax\footnotetext{\tiny{From Bishop's \emph{Pattern recognition and machine learning}, Figure 9.8.}}

\end{frame}

\begin{frame}{EM for GMM}

\begin{itemize}
\item After 5 rounds of EM:
\end{itemize}
\begin{center}
\includegraphics[height=0.55\textheight]{../Figures/clustering/9\lyxdot 8e}
\par\end{center}

\let\thefootnote\relax\footnotetext{\tiny{From Bishop's \emph{Pattern recognition and machine learning}, Figure 9.8.}}

\end{frame}

\begin{frame}{EM for GMM}

\begin{itemize}
\item After 20 rounds of EM:
\end{itemize}
\begin{center}
\includegraphics[height=0.55\textheight]{../Figures/clustering/9\lyxdot 8f}
\par\end{center}

\let\thefootnote\relax\footnotetext{\tiny{From Bishop's \emph{Pattern recognition and machine learning}, Figure 9.8.}}

\end{frame}

\begin{frame}{Relation to $K$-Means}

\begin{itemize}
\item EM for GMM seems a little like $k$-means.

\item In fact, there is a precise correspondence.

\item First, fix each cluster covariance matrix to be $\sigma^{2}I$.
\item Then the density for each Gausian only depends on distance to
  the mean.

\item As we take $\sigma^{2}\to0$, the update equations converge to doing
$k$-means.

\item If you do a quick experiment yourself, you'll find

\begin{itemize}
\item Soft assignments converge to hard assignments.
\item Has to do with the tail behavior (exponential decay) of
  Gaussian.
\end{itemize}
\item Can use $k$-means$++$ to initialize parameters of EM algorithm.
\end{itemize}
\end{frame}
\section{Math Prerequisites for General EM Algorithm}
\begin{frame}{Jensen's Inequality}
  \begin{itemize}
  \item Which is larger: $\ex[X^2]$ or $\ex[X]^2$?
    \pause
  \item Must be $\ex[X^2]$ since $\var[X] = \ex[X^2]-\ex[X]^2\geq0$.
  \item More general result is true:
  \end{itemize}
  \begin{theorem}{Jensen's Inequality}
    If $f:\RR\to\RR$ is convex and $X$ is a random variable then $\ex[f(X)] \geq
    f(\ex[X])$. If $f$ is strictly convex then we have equality iff
    $X=\ex[X]$ with probability 1 (i.e., $X$ is constant).
  \end{theorem}
\end{frame}
\begin{frame}{Proof of Jensen}
  \begin{block}{Exercise}
    Suppose $X$ can take exactly two value: $x_1$ with probability
    $\pi_1$ and $x_2$ with probability $\pi_2$.  Then prove Jensen's inequality.
  \end{block}
  \pause
  \begin{itemize}
  \item Let's compute $\ex[f(X)]$:
    $$\ex[f(X)] = \pi_1f(x_1) + \pi_2f(x_2) \leq f(\pi_1x_1+\pi_2x_2)
    = f(\ex[X]).$$
  \item For the general proof, what do we know is true about all
    convex functions $f:\RR\to\RR$?
  \end{itemize}
\end{frame}
\begin{frame}{Proof of Jensen}
  \begin{enumerate}
  \item Let $e = \ex[X]$. (Remember $e$ is just a number.)
  \item Since $f$ has a subgradient at $e$, there is an
    underestimating line $g(x)=ax+b$ that passes through the point
    $(e,f(e))$.
  \item Then we have
    \begin{eqnarray*}
      \ex[f(X)]  & \geq & \ex[g(X)] \\
      & = & \ex[aX+b]\\
      & = & a\ex[X]+b\\
      & = & ae+b\\
      & = & f(e)\\
      & = & f(\ex[X]).
    \end{eqnarray*}
  \item If $f$ is strictly convex then $f=g$ at exactly 1 point, so
    equality iff $X$ is constant.
  \end{enumerate}
\end{frame}
\begin{frame}{KL-Divergence}
\begin{itemize}
\item Let $p(x)$ and $q(x)$ be probability mass functions (PMFs) on
  $\cx$.
\item We want to measure how different they are.
\item The \textbf{Kullback-Leibler} or \textbf{``KL'' Divergence} is
  define by
  $$\kl(p\|q) = \sum_{x\in\cx} p(x)\log\frac{p(x)}{q(x)}.$$
  (Assumes \textit{absolute continuity}: $q(x)=0$ implies $p(x)=0$.)
\item Can also write
  $$\kl(p\|q) = \ex_{x\sim p}\log\frac{p(x)}{q(x)}.$$
\item Note, the KL-divergence is not symmetric and doesn't satisfy the
  triangle inequality.
\end{itemize}
\end{frame}
\begin{frame}{Gibbs' Inequality}
  \begin{theorem}{Gibbs' Inequality}
    Let $p(x)$ and $q(x)$ be PMFs on $\cx$.  Then
    $$\kl(p\|q)\geq 0,$$
    with equality iff $p(x)=q(x)$ for all $x\in\cx$.
  \end{theorem}
  \begin{itemize}
  \item Since
    $$\kl(p\|q)=\ex_p\left[-\log\left(\frac{q(x)}{p(x)}\right)\right],$$
    this is screaming for Jensen's inequality.
  \end{itemize}
\end{frame}
\begin{frame}{Gibbs' Inequality: Proof}
  \begin{eqnarray*}
    \kl(p\|q)
    & = & \ex_p\left[-\log\left(\frac{q(x)}{p(x)}\right)\right]\\
    & \geq & -\log\left(\ex_p\left[\frac{q(x)}{p(x)}\right]\right)\\
    & = & -\log\left(\sum_{x:p(x)>0}p(x)\frac{q(x)}{p(x)}\right)\\
    & = & -\log\left(\sum_x q(x)\right)\\
    & = & -\log1=0.
  \end{eqnarray*}
  \begin{itemize}
  \item Since $-\log$ is strictly convex, we have equality iff $q/p$
    is constant, i.e., $q=p$.
  \end{itemize}
\end{frame}
\end{document}


\chapter{Implementation}
\label{chp:implementation}

\nomad is a \verb|C++| implementation of reverse mode automatic
differentiation that uses an \textit{operator overloading} strategy.
More precisely, we introduce a dual number type and then overload
common functions to accept this type.

A key difference of \nomad to other implementations is the generalization
to higher-orders.  Most higher-order automatic differentiation
implementations leverage the recursive nature of higher-order dual
numbers directly, automatically differentiating through a first-order
implementation to compute higher-order partial derivatives and
the subsequent linear differential operators.  This approach allows for
arbitrary-order automatic differentiation, but only at the cost of inefficient and
sometimes numerically unstable results.  \nomad,
on the other hand, explicitly implements second and third-order operators
with a focus on performance and numerical stability.

In this chapter we introduce the architecture behind the \nomad automatic
differentiation implementation.  We first review the internal representation
of the expression graph and how it interfaces with function overloads before
discussing the user interface itself.

\section{Internal Representation of the Expression Graph}
\label{sec:exp_graph_rep}

While the expression graph represents the entire composite function, each
node in the expression graph represents just a single component function.
Consequently each node must be responsible for both implementing the
local pushforward and pullback of the corresponding function and storing
the dual numbers, input nodes, and any auxiliary storage convenient for
the operator implementations, such as partial derivatives.

Like many other automatic differentiation implementations, \nomad
topologically sorts the expression graph into a linear stack of nodes,
sometimes known as a tape (Figure \ref{fig:topologicalSort}).  This ordering
preserves the structure of the expression graph, ensuring that a sweep
through the stack executes a valid forward or reverse sweep through the
expression graph.

\begin{figure}
\setlength{\unitlength}{0.1in}
\centering
\begin{picture}(50, 20)
%
%\put(0, 0) { \framebox(50, 20){} }
%\put(25, 0) { \framebox(25, 30){} }
%\put(25, 0) { \framebox(6.25, 30){} }
%\put(25, 0) { \framebox(12.5, 30){} }
%\put(25, 0) { \framebox(18.75, 30){} }
%
%\put(25, 0) { \framebox(3.125, 30){} }
%\put(25, 0) { \framebox(9.375, 30){} }
%\put(25, 0) { \framebox(15.625, 30){} }
%
% Expression Graph
%
\put(6.25, 2.5) { \circle{4} }
\put(6.25, 2.5) { \makebox(0, 0) {$ x_{1} $} }
%
\put(12.5, 2.5) { \circle{4} }
\put(12.5, 2.5) { \makebox(0, 0) { $ x_{2} $ } }
%
\put(18.75, 2.5) { \circle{4} }
\put(18.75, 2.5) { \makebox(0, 0) { $ x_{3} $ } }
%
\put(6.25, 4.5) { \vector(3, 4){2.75} }
\put(12.5, 4.5) { \vector(-3, 4){2.75} }
\put(12.5, 4.5) { \vector(3, 4){2.75} }
\put(18.75, 4.5) { \vector(-3, 4){2.75} }
%
\put(10, 10) {\circle{4} } % Tweaked to the right
\put(9.375, 10) { \makebox(0, 0) { $f_{1}$ } }
%
\put(16.25, 10) {\circle{4} } % Tweaked to the right
\put(15.625, 10) { \makebox(0, 0) { $f_{2}$ } }
%
\put(9.375, 12) { \vector(3, 4){2.75} }
\put(15.625, 12) { \vector(-3, 4){2.75} }
%
\put(13, 17.5) {\circle{4} } % Tweaked to the right
\put(12.5, 17.5) { \makebox(0, 0) { $ g $ } }
%
% Middle Arrow
%
\put(21.875, 10) { \thicklines \vector(1, 0){6.25} }
%
% Stack
%
\put(33.5, 4) { \framebox(8, 2){ $x_{1}$} }
\put(33.5, 6) { \framebox(8, 2){ $x_{2}$ } }
\put(33.5, 8) { \framebox(8, 2){ $x_{3}$ } }
\put(33.5, 10) { \framebox(8, 2){ $f_{1}$ } }
\put(33.5, 12) { \framebox(8, 2){ $f_{2}$ } }
\put(33.5, 14) { \framebox(8, 2){ $g$ } }
%
\end{picture}
\caption{
A topological sort of the expression graph yields a linear stack of nodes,
sometimes known as a \textit{tape}, ordered such that a pass through the
stack yields a valid forward or reverse sweep of the expression graph.
}
\label{fig:topologicalSort}
\end{figure}

Nodes are implemented with the \verb|var_node| class which defines
default pushforward and pullback methods that can be specialized
for functions with structured Jacobians.  Because the amount of storage for
each node can vary widely depending on the corresponding function and the
order of the desired linear differential operator, storage is decoupled from
each node.  Instead the necessary data are stored in three global stacks, the
inputs stack, dual numbers stack, and partials stack, with \verb|var_node|
containing only an address to the relevant data in each
(Figure \ref{fig:architecture}), as well as accessor and mutator methods that
abstract the indirect storage.

Storage for each stack is pre-allocated and expanded only as necessary with
an arena-based allocation pattern.

\begin{figure}
\setlength{\unitlength}{0.1in}
\centering
\begin{picture}(50, 30)
%
%\put(0, 0) { \framebox(50, 30){} }
%
%\put(0, 0) { \framebox(12.5, 30){} }
%\put(0, 0) { \framebox(25, 30){} }
%\put(0, 0) { \framebox(37.5, 30){} }
%
%\put(0, 0) { \framebox(50, 7.5){} }
%\put(0, 0) { \framebox(50, 15){} }
%\put(0, 0) { \framebox(50, 22.5){} }
%
% Var Body
%
\put(8.5, 2) { \makebox(8, 2){Var Node} }
\put(8.5, 6) { \framebox(8, 2){} }
\put(8.5, 8) { \framebox(8, 2){} }
\put(8.5, 10) { \framebox(8, 2){} }
\put(8.5, 12) { \framebox(8, 2){} }
\put(8.5, 14) { \framebox(8, 2){} }
\put(8.5, 16) { \framebox(8, 2){} }
\put(8.5, 18) { \framebox(8, 2){} }
\put(8.5, 20) { \framebox(8, 2){} }
\put(8.5, 22) { \framebox(8, 2){} }
\put(8.5, 24) { \framebox(8, 2){} }
%
% Inputs
%
\put(39, 24) { \makebox(8, 2){Inputs} }
\put(25, 22) { \framebox(2, 6){} }
\put(27, 22) { \framebox(2, 6){} }
\put(29, 22) { \framebox(2, 6){} }
\put(31, 22) { \framebox(2, 6){} }
\put(33, 22) { \framebox(2, 6){} }
\put(35, 22) { \framebox(2, 6){} }
\put(37, 22) { \framebox(2, 6){} }
%
% Dual Numbers
%
\put(39, 15) { \makebox(8, 2){Dual} }
\put(39, 13) { \makebox(8, 2){Numbers} }
\put(25, 12) { \framebox(2, 6){} }
\put(27, 12) { \framebox(2, 6){} }
\put(29, 12) { \framebox(2, 6){} }
\put(31, 12) { \framebox(2, 6){} }
\put(33, 12) { \framebox(2, 6){} }
\put(35, 12) { \framebox(2, 6){} }
\put(37, 12) { \framebox(2, 6){} }
%
% Partials
%
\put(39, 4) { \makebox(8, 2){Partials} }
\put(25, 2) { \framebox(2, 6){} }
\put(27, 2) { \framebox(2, 6){} }
\put(29, 2) { \framebox(2, 6){} }
\put(31, 2) { \framebox(2, 6){} }
\put(33, 2) { \framebox(2, 6){} }
\put(35, 2) { \framebox(2, 6){} }
\put(37, 2) { \framebox(2, 6){} }
%
% Arrows
%
\thicklines
\put(32, 22) { \vector(-1, -1){4} }
\put(34, 22) { \vector(0, -1){4} }
%
\put(16.5, 15) { \vector(4, 3){15.5} }
\put(16.5, 15) { \vector(1, 0){19.5} }
\put(16.5, 15) { \vector(2, -1){17.5} }
%
\end{picture}
\caption{
Upon a topological sort (Figure \ref{fig:topologicalSort}), the expression graph
is represented by a stack of \texttt{var\_node} objects.  The input nodes,
dual numbers, and partial derivatives are stored in external stacks, with each
\texttt{var\_node} storing only addresses to each.  Note that the inputs stack
address not the \texttt{var\_node} objects directly, but rather only the dual numbers
of those nodes needed for implementing the pushforward and pullback operators.
}
\label{fig:architecture}
\end{figure}

\subsubsection{The Inputs Stack}

The inputs stack represents the edges in the expression graph
needed for directing the forward and reverse sweeps.  Because
the pushforward and pullback operators need to read and write only
the dual numbers at each node, edges index the dual numbers of any
dependencies directly (Figure \ref{fig:architecture}) and avoid the overhead
that would be acquired for indirect access through the \verb|var_node|
objects.

\subsubsection{Dual Number Stack}

The dual number stack stores the $2^{k}$ components of the $k$th-order
dual numbers at each node.  For example, a first-order expression graph
will need only two elements for each node while a third-order expression
graph will need eight (Figure \ref{fig:dualNumberStorage}).

\begin{figure}
\setlength{\unitlength}{0.1in}
\centering
\begin{picture}(50, 20)
%
%\put(0, 0) { \framebox(50, 20){} }
%
% First-Order
%
\put(0, 15) { \makebox(8, 2){First-} }
\put(0, 13) { \makebox(8, 2){Order} }
\put(9, 17) { \makebox(2, 2){ $\ldots$ } }
\put(9, 11) { \makebox(2, 2){ $\ldots$ } }
\put(11, 12) { \framebox(4, 6){ $x $ } }
\put(15, 12) { \framebox(4, 6){ $ \delta x $ } }
\put(19, 17) { \makebox(2, 2){ $\ldots$ } }
\put(19, 11) { \makebox(2, 2){ $\ldots$ } }
%
{ \thicklines \put(13, 20) { \vector(0, -1){2} } }
%
% Second-Order
%
\put(0, 5) { \makebox(8, 2){Third-} }
\put(0, 3) { \makebox(8, 2){Order} }
\put(9, 7) { \makebox(2, 2){ $\ldots$ } }
\put(9, 1) { \makebox(2, 2){ $\ldots$ } }
\put(11, 2) { \framebox(4, 6){ $s$ } }
\put(15, 2) { \framebox(4, 6){ $\delta s$ } }
\put(19, 2) { \framebox(4, 6){ $\delta t$ } }
\put(23, 2) { \framebox(4, 6){ $\delta^{2} t$ } }
\put(27, 2) { \framebox(4, 6){ $\delta u$ } }
\put(31, 2) { \framebox(4, 6){ $\delta^{2} u$ } }
\put(35, 2) { \framebox(4, 6){ $\delta^{2} v$ } }
\put(39, 2) { \framebox(4, 6){ $\delta^{3} v$ } }
\put(43, 7) { \makebox(2, 2){ $\ldots$ } }
\put(43, 1) { \makebox(2, 2){ $\ldots$ } }
%
{ \thicklines \put(13, 10) { \vector(0, -1){2} } }
%
\end{picture}
\caption{
The dual number stack is able to accommodate storage of dual numbers
of any order without reallocating memory.  For example, first-order dual
numbers require only two elements while third-order dual numbers require
eight elements.
}
\label{fig:dualNumberStorage}
\end{figure}

\subsubsection{Partials Stack}

A common approach in many automatic differentiation implementations
is to compute partial derivatives online as necessary and avoid storing them
in the expression graph.  This strategy is sound for first-order reverse
mode calculations where the partials are used only once, but higher-order
calculations require multiple sweeps that reuse the partials.  Recomputing
the partials for each sweep then becomes a significant computational burden.

With a focus on implementing efficient higher-order methods, \nomad
explicitly stores partial derivatives in a dedicated partials stack.  When
constructing an $k$th-order expression graph only the $k$th-order partial
derivatives and lower are stored, and only if the partial derivatives are
non-zero.  For example, a second-order expression graph will calculate
and store only the first and second-order partial derivatives.

To avoid redundant calculations and storage, \nomad stores only the
unique higher-order values.  In a second-order calculation a node
representing the component function
$f: \mathbb{R}^{N} \rightarrow \mathbb{R}$ will store
%
\begin{equation*}
\frac{ \partial^{2} f }{ \partial x_{i} \partial x_{j} }, \, i \in 1, \ldots, N, j \in 1, \ldots i,
\end{equation*}
%
while a third-order calculation will store
%
\begin{equation*}
\frac{ \partial^{3} f }{ \partial x_{i} \partial x_{j} \partial x_{k} }, \,
i \in 1, \ldots, N, j \in 1, \ldots i, k \in 1, \ldots, j.
\end{equation*}
%
For example, when executing a third-order calculation the node representing
a binary function, $f : \mathbb{R}^{2} \rightarrow \mathbb{R}$, utilizes the
compact storage demonstrated in Figure \ref{fig:partialsStorage}.

\begin{figure}
\setlength{\unitlength}{0.1in}
\centering
\begin{picture}(50, 10)
%
%\put(0, 0) { \framebox(50, 10){} }
%
% Second-Order
%
\put(5, 7) { \makebox(2, 2){ $\ldots$ } }
\put(5, 1) { \makebox(2, 2){ $\ldots$ } }
\put(7, 2) { \framebox(4, 6){ $ \frac{ \partial f }{ \partial x} $ } }
\put(11, 2) { \framebox(4, 6){ $ \frac{ \partial f }{ \partial y} $ } }
\put(15, 2) { \framebox(4, 6){ $ \frac{ \partial^{2} f }{ \partial x^{2}} $ } }
\put(19, 2) { \framebox(4, 6){ $ \frac{ \partial^{2} f }{ \partial x \partial y} $ } }
\put(23, 2) { \framebox(4, 6){ $ \frac{ \partial^{2} f }{ \partial y^{2}} $ } }
\put(27, 2) { \framebox(4, 6){ $ \frac{ \partial^{3} f }{ \partial x^{3}} $ } }
\put(31, 2) { \framebox(4, 6){ $ \frac{ \partial^{3} f }{ \partial x^{2} \partial y} $ } }
\put(35, 2) { \framebox(4, 6){ $ \frac{ \partial^{3} f }{ \partial x \partial y^{3}} $ } }
\put(39, 2) { \framebox(4, 6){ $ \frac{ \partial^{3} f }{ \partial y^{3}} $ } }
\put(43, 7) { \makebox(2, 2){ $\ldots$ } }
\put(43, 1) { \makebox(2, 2){ $\ldots$ } }
%
{ \thicklines \put(9, 10) { \vector(0, -1){2} } }
%
\end{picture}
\caption{
Only unique partial derivatives are stored in the partials stack,
as demonstrated here for node representing a binary function,
$f: \mathbb{R}^{2} \rightarrow \mathbb{R}$, in a third-order
expression graph that requires first, second, and third-order
partial derivatives.
}
\label{fig:partialsStorage}
\end{figure}

In general a function with $N$ inputs and non-vanishing derivatives at
all orders will require $\binom{N + M}{M}$ elements to store the unique
$M$th-order partial derivatives.

\section{Extending Functions to Accept Dual Numbers}

In order to implement automatic differentiation calculations, each dual-number
valued-function is responsible for not only computing the function
value but also expanding the expression graph with the addition of
a node with the desired pushforward and pullback methods and
addresses to the top of the inputs, dual numbers, and partials stacks.
The evaluation of a composite dual number-valued function will then
build the complete expression graph primed for the application of
linear differential operators.

Dual numbers in \nomad are exposed to the user by the \verb|var|
class with \verb|var|-valued functions responsible for the management
of the expression graph.  In this section we present the details of
the \verb|var| class and the implementation of \verb|var|-valued functions.

\subsection{The var Class}

The \verb|var| class itself is only a lightweight wrapper for an underlying
\verb|var_node| in the expression graph, storing an address to the
corresponding node with a variety of member functions that expose the
node's data.  Most importantly, \verb|var| is templated to allow for the
compile-time configuration of the expression graph,
%
\begin{verbatim}
template <short AutodiffOrder, bool StrictSmoothness, bool ValidateIO>
class var { ...
\end{verbatim}
%
\nomad also defines a variety of type definitions for the most commonly
used template configurations (Table \ref{tab:typedefs}).

\begin{table*}[t!]
	\centering
	\renewcommand{\arraystretch}{2}
	\begin{tabular}{cccc}
	\rowcolor[gray]{0.9} Type Definition & \verb|AutodiffOrder|
	& \verb|StrictSmoothness| & \verb|ValidateIO| \\
	\verb|var1| & \verb|1| & \verb|true| & \verb|false| \\
	\rowcolor[gray]{0.9} \verb|var2| & \verb|2| & \verb|true| & \verb|false| \\
	\verb|var3| & \verb|3| & \verb|true| & \verb|false| \\
	\rowcolor[gray]{0.9} \verb|var1_debug| & \verb|1| & \verb|true| & \verb|true| \\
	\verb|var2_debug| & \verb|2| & \verb|true| & \verb|true| \\
	\rowcolor[gray]{0.9} \verb|var3_debug| & \verb|3| & \verb|true| & \verb|true| \\
	\verb|var1_wild| & \verb|1| & \verb|false| & \verb|false| \\
	\rowcolor[gray]{0.9} \verb|var2_wild| & \verb|2| & \verb|false| & \verb|false| \\
	\verb|var3_wild| & \verb|3| & \verb|false| & \verb|false| \\
	\end{tabular}
	\caption{The \texttt{nomad} namespace includes helpful type definitions for
	the most common \texttt{var} configurations.}
	\label{tab:typedefs}
\end{table*}

\subsubsection{AutodiffOrder}

The template parameter \verb|AutodiffOrder| defines the maximum order
linear differential operator that the corresponding expression graph will
admit.  Attempting to apply a second-order linear differential operator
to a graph built from \verb|var|s with \verb|AutodiffOrder = 1|, for example,
will induce a compile-time error.

Note that lower-order graphs require the computation and storage of
fewer dual numbers and partial derivatives, not to mention faster pushforward
and pullback implementations.  Consequently, although a higher-order
graph will admit lower-order operators, it is much more efficient to match
the order of the expression graph, via the choice of \verb|AutodiffOrder|,
to the linear differential operators of interest.

\subsubsection{StrictSmoothness}

\verb|StrictSmoothness| defines the smoothness requirements of
component functions.

Formally, automatic differentiation requires only that each component
function have well-behaved derivatives defined in some neighborhood
containing the point at which the function is being evaluated.  For
example, a linear differential operator applied to the absolute value
function is well defined everywhere except at the origin.  Many
algorithms that use these operators, however, actually require that
the derivatives be well-defined \textit{everywhere} because even
functions with only point discontinuities in their derivatives will
manifest undesired pathologies.

With these algorithms in mind, when \verb|StrictSmoothness = true|
\nomad will disable functions and operators that are not everywhere
smooth, as well as those that admit comparisons that may introduce cusps.

\subsubsection{ValidateIO}

\verb|ValidateIO| enables strict validation of the inputs and outputs
to each component function.

Identifying the source of potential floating-point pathologies like
underflow, overflow and \verb|NaN| in a large, composite function
is often a challenging debugging task.  When \verb|ValidateIO = true|,
\nomad assists users by checking all input values, output values,
and output partial derivatives for \verb|NaN| and throws an
exception identifying the responsible component function if any
are found.  Domain validation will also be done if implemented in
the function.  These checks, however, are a nontrivial computational
burden and should be disabled for high-performance applications.

\subsection{var-valued Functions}

Extending a function to take \verb|var| arguments requires not just propagating the
value of the function but also creating a new node in the expression
graph.  In this section we review the steps necessary for implementing
a \verb|var|-valued function and present some specific examples.

\subsubsection{Input Validation}

Firstly, a \verb|var|-valued function has to validate its inputs if
\verb|ValidateIO| is \verb|true|.  \nomad provides helper functions
that validate double values and throws \nomad-specific exceptions
if \verb|NaN|s are encountered,
%
\begin{verbatim}
inline void validate_input(double val, std::string f_name)
\end{verbatim}
%
Here \verb|f_name| is the name of the function being implemented
and is used to trace the exception to the origin of the violation.
Domain constraints may also be validated here with helper functions
such as
%
\begin{verbatim}
inline void validate_lower_bound(double val, double lower, std::string f_name)
\end{verbatim}

\subsubsection{Expression Graph Node Creation}

With the inputs validated we can now append a new node to the
expression graph with the \verb|create_node| function.  This function
is templated to accept the type of node being used by the function
and will require the number of inputs to the function unless the
specific \verb|var_node| implementation does not require it.

For example, if the function is utilizing the default \verb|var_node|
implementation then the call would be
%
\begin{verbatim}
create_node<var_node<AutodiffOrder, PartialsOrder>>(n_inputs);
\end{verbatim}
%
On the other hand, a function utilizing a \verb|var_node| specialization
with a predefined number of inputs would not need to specify an
argument,
%
\begin{verbatim}
create_node<binary_var_node<AutodiffOrder, PartialsOrder>>();
\end{verbatim}

\subsubsection{Pushing Inputs}

Once the node representing the component function has been created
we can now being to push the necessary data onto the external stacks.
First are the addresses of the inputs to the function,
%
\begin{verbatim}
inline void push_inputs(nomad_idx_t input)
\end{verbatim}
%
which are readily accessed from the input \verb|var| arguments.

\subsubsection{Pushing Dual Numbers}

Next are the dual numbers.  The function
%
\begin{verbatim}
template<short AutodiffOrder, bool ValidateIO>
inline void push_dual_numbers(double val)
\end{verbatim}
%
pushes $2^{\mathrm{AutodiffOrder}}$ components onto the stack, with
the first component set to the value of the function, \verb|val|, and the
rest set to zero.

When \verb|ValidateIO = true| the function \verb|push_dual_numbers|
will also check the output \verb|val| for a \verb|NaN| and throw an
exception if necessary.  Consequently the call to \verb|push_dual_numbers|
must be wrapped in a \verb|try/catch| block, preferable one that throws
a new exception identifying the current function, for example
%
\begin{verbatim}
try {
  push_dual_numbers<AutodiffOrder, ValidateIO>(binary_function(x, y));
} catch(nomad_error) {
  throw nomad_output_value_error("binary_function");
}
\end{verbatim}

\subsubsection{Pushing Partials}

Finally we can compute the partial derivatives and push them onto the
partials stack with
%
\begin{verbatim}
template<bool ValidateIO>
inline void push_partials(double partial)
\end{verbatim}
%
As with \verb|push_dual_numbers|, \verb|push_partials| will optionally
validate the values of the partial derivatives and must be wrapped in
a \verb|try/catch| block,
%
\begin{verbatim}
try {
  if (AutodiffOrder >= 1) {
    push_partials<ValidateIO>(df_dx);
    push_partials<ValidateIO>(df_dy);
  }
  ...Push higher-order partial derivatives...
} catch(nomad_error) {
  throw nomad_output_partial_error("binary_function");
}
\end{verbatim}

Because partial derivatives only up to \verb|AutodiffOrder| need to be
computed and stored, efficient implementations of a \verb|var|-valued
function should evaluate partial derivatives only conditionally on
\verb|AutodiffOrder|.

\subsubsection{Returning a New Var}

Finally we return a new \verb|var| that wraps the freshly-created node.
%
\begin{verbatim}
return var<AutodiffOrder, StrictSmoothness, ValidateIO>(next_node_idx_ - 1);
\end{verbatim}
%
\verb|next_node_idx_| addresses the top of the \verb|var_node| stack, hence
\verb|next_node_idx_ - 1| addresses the last node pushed to the stack.

\subsubsection{Example Implemention of a Smooth Function}

The logic of a \verb|var|-valued function implementation is more clear
when each step is presented together.  Here is an example implementation
of a smooth, binary function using the default \verb|var_node| implementation.

\begin{verbatim}
template <short AutodiffOrder, bool StrictSmoothness, bool ValidateIO>
inline var<AutodiffOrder, StrictSmoothness, ValidateIO>
  binary_function(const var<AutodiffOrder, StrictSmoothness, ValidateIO>& v1,
                  const var<AutodiffOrder, StrictSmoothness, ValidateIO>& v2) {

  // Validate input values if ValidateIO is true
  if (ValidateIO) {
    validate_input(v1.first_val(), "binary_function");
    validate_input(v2.first_val(), "binary_function");
  }

  // Create a var_node on the top of the stack
  const short partials_order = 3;
  const unsigned int n_inputs = 2;

  create_node<var_node<AutodiffOrder, partials_order>>(n_inputs);

  // Push dependencies to top of the inputs stack
  push_inputs(v1.dual_numbers());
  push_inputs(v2.dual_numbers());

  // Push dual numbers to the the top of the dual numbers stack
  double x = v1.first_val();
  double y = v2.first_val();

  try {
    push_dual_numbers<AutodiffOrder, ValidateIO>(binary_function(x, y));
  } catch(nomad_error) {
    throw nomad_output_value_error("binary_function");
  }

  // Push partial derivatives to the top of the partials stack
  try {
    if (AutodiffOrder >= 1) {
      ...Compute df_dx and df_dy...
      push_partials<ValidateIO>(df_dx);
      push_partials<ValidateIO>(df_dy);
    }
    if (AutodiffOrder >= 2) {
      ...Compute df2_dx2, df2_dxy, and df2_dy2...
      push_partials<ValidateIO>(df2_dx2);
      push_partials<ValidateIO>(df2_dxdy);
      push_partials<ValidateIO>(df2_dy2);
    }
    if (AutodiffOrder >= 3) {
      ...Compute df3_dx3, df3_dx2dy, df3_dxdy2, and df3_dy3...
      push_partials<ValidateIO>(df3_dx3);
      push_partials<ValidateIO>(df3_dx2dy);
      push_partials<ValidateIO>(df3_dxdy2);
      push_partials<ValidateIO>(df3_dy3);
    }
  } catch(nomad_error) {
    throw nomad_output_partial_error("binary_function");
  }

  // Return a new var that wraps the newly created node
  return var<AutodiffOrder, StrictSmoothness, ValidateIO>(next_node_idx_ - 1);

}
\end{verbatim}
%
Note that expensive calculations, such as when validating inputs and computing
and storing partial derivatives, are conditioned on template parameters wherever
possible to avoid unnecessary computation.  Because these template parameters
are known at compile-time these checks incur no run-time penalties.

When the partial derivatives of a function are structured then we can achieve
much higher performance by specializing the underlying \verb|var_node|.
For example, the only non-zero partial derivatives of the addition operator
are at first-order, and they are all equal to $1$.  Consequently we can speed
up the automatic differentiation computations by specializing \verb|var_node|
to avoid unnecessary multiplications by $1$.

This is done in the \verb|binary_sum_var_node| and the implementation of
the addition operator then becomes
%
\begin{verbatim}
template <short AutodiffOrder, bool StrictSmoothness, bool ValidateIO>
inline var<AutodiffOrder, StrictSmoothness, ValidateIO>
  operator+(const var<AutodiffOrder, StrictSmoothness, ValidateIO>& v1,
            const var<AutodiffOrder, StrictSmoothness, ValidateIO>& v2) {

  if (ValidateIO) {
    validate_input(v1.first_val(), "operator+");
    validate_input(v2.first_val(), "operator+");
  }

  // Create a specialized var_node on the stack
  create_node<binary_sum_var_node<AutodiffOrder>>();

  push_inputs(v1.dual_numbers());
  push_inputs(v2.dual_numbers());

  try {
    push_dual_numbers<AutodiffOrder, ValidateIO>(v1.first_val() + v2.first_val());
  } catch(nomad_error) {
    throw nomad_output_value_error("operator+");
  }

  // The binary_sum_var_node pushforward and pullback implementations
  // use hardcoded partial derivatives and don't require partial derivatives to
  // be pushed onto the partials stack

  return var<AutodiffOrder, StrictSmoothness, ValidateIO>(next_node_idx_ - 1);

}
\end{verbatim}

\subsubsection{Example Implemention of a Non-Smooth Function}

Non-smooth functions are implemented almost exactly the same as smooth functions.
The only difference is in the function signature which uses the \verb|enable_if|
template metaprogram to disable the function unless \verb|StrictSmoothness = 1|.

For example, the absolute value function is implemented as
%
\begin{verbatim}
template <short AutodiffOrder, bool StrictSmoothness, bool ValidateIO>
inline typename
  std::enable_if<!StrictSmoothness,
                 var<AutodiffOrder, StrictSmoothness, ValidateIO> >::type
  fabs(const var<AutodiffOrder, StrictSmoothness, ValidateIO>& input) {

  if (ValidateIO) validate_input(input.first_val(), "fabs");

  const short partials_order = 1;
  const unsigned int n_inputs = 1;

  create_node<unary_var_node<AutodiffOrder, partials_order>>(n_inputs);

  double x = input.first_val();

  try {
    push_dual_numbers<AutodiffOrder, ValidateIO>(fabs(x));
  } catch(nomad_error) {
    throw nomad_output_value_error("fabs");
  }

  push_inputs(input.dual_numbers());

  try {
    if (AutodiffOrder >= 1) {
      if (x < 0)
        push_partials<ValidateIO>(-1);
      else
        push_partials<ValidateIO>(1);
    }
  } catch(nomad_error) {
    throw nomad_output_partial_error("fabs");
  }

  return var<AutodiffOrder, StrictSmoothness, ValidateIO>(next_node_idx_ - 1);

}
\end{verbatim}

\section{The \nomad User Interface}
\label{sec:user_interface}

In \nomad, linear differential operators are implemented as functionals acting
on \verb|var|-valued functors.

Specifically, functors are classes deriving from \verb|base_functor|
%
\begin{verbatim}
template<class T>
class base_functor {
public:
  virtual T operator()(const Eigen::VectorXd& x) const;
  typedef T var_type;
};
\end{verbatim}
%
that implement a \verb|var|-valued \verb|operator()| taking an
\verb|Eigen::VectorXd&| of input values.  For example, at
first-order the composite function
%
\begin{equation*}
f \! \left( x, y, z \right) = \cos \! \left( e^{x} + e^{y} \right) / z
\end{equation*}
%
would be defined as
%
\begin{verbatim}
class example_functor: public base_functor<var1> {
  var1 operator()(const Eigen::VectorXd& x) const {
    var1 v1 = x[0];
    var1 v2 = x[1];
    var1 v3 = x[2];
    return cos( exp(v1) + exp(v2) ) / v3;
  }
};
\end{verbatim}

The functionals implementing linear differential operators then
take a functor instance as an argument, first calling \verb|operator()|
to build the expression graph and then the various sweeps
necessary to compute the differential operator itself.  For example,
the gradient is defined with the signature
%
\begin{verbatim}
template <typename F>
void gradient(const F& f,
              const Eigen::VectorXd& x,
              Eigen::VectorXd& g)
\end{verbatim}
%
Consequently computing a gradient is straightforward,
%
\begin{verbatim}
int n = 100;
Eigen::VectorXd x = Eigen::VectorXd::Ones(n);

double f;
Eigen::VectorXd grad(x.size());

example_functor example;
gradient(example, x, f, grad);

std::cout << grad.transpose() << std::endl;
\end{verbatim}

For the complete list of \verb|var|-valued functions and linear differential
operators implemented in \nomad please consult Chapter \ref{chap:reference_guide}.