%!TEX root = ../main.tex

In this chapter, we will shift gears and discuss optical hardware platforms for machine learning applications.
The field of machine learning has seen an impressive revival in the past decade, with several applications from machine translation \cite{brown1993mathematics}, computer vision \cite{faugeras1993three}, health and medicine \cite{esteva2017dermatologist}, and game playing \cite{silver2017mastering}.
While the basic ideas and models of machine learning, and \textit{deep learning} \cite{goodfellow2016deep,lecun2015deep} have existed for several decades, this recent success may be largely attributed to (1) the widespread availability of hardware platforms capable of running these models efficiently, and (2) the accessibility of large datasets needed to train these models.

Indeed, with the new applications and models introduced in recent years, the hardware demand continues to rise exponentially over time.
One common approach to satisfying this demand involves running the computation on graphical processing units (GPUs), which creates benefits from massive parallelization.
Along this line, companies are beginning to invest in hardware platforms that are specially designed to run machine learning software platforms.
One such example is the \textit{tensor processing unit} (TPU), which runs Google's Tensorflow package.

Alternatively, technologies are being considered that would replace the typical digital electronics hardware architectures with those more suited for machine learning computation.
This is evidenced by the renewed interest in analog computing and quantum machine learning  \cite{biamonte_quantum_2017}, in which a physical system is used to emulate the machine learning model.
Other groups have attempted to create \textit{neuromorphic} hardware, which is designed to vaguely mimick the circuitry of the human brain \cite{tait_neuromorphic_2017}.

Of great interest among these new platforms are so-called \textit{linear nanophotonic processors} \cite{harris2018linear}.
The idea here is to replace the linear elements common to most machine learning models with linear optical devices that may perform linear operations via transmission of light through their components.
While this idea has existed for quite some time \cite{miller_sorting_2015,Miller2015,Miller2013a,Miller2015,Miller2013a}, its implementation for deep learning was recently demonstrated in a photonic integrated circuit \cite{shen2017deep} and has been used in recent years for quantum information processing \cite{obrien_photonic_2009}.
The main advantage of this technique is that it can, in principle, perform these crucial operations faster, with lower latency, and with a lower energy cost when compared to digital electronics.
A further benefit is that the operations, themselves, may be reconfigured using standard optical components, such as phase shifters.
This ability will be necessary for the training of these platforms, as we will discuss.

\section{Optical Neural Networks}

% Here we briefly review the basics of feedforward artificial neural networks (ANNs) and describe their implementation in a reconfigurable optical circuit, as proposed in Ref. \citenum{shen_deep_2017}.
% An ANN is a function which accepts an input vector, $x_0$ and returns an output vector, $x_L$.
% This is accomplished in a layer-by-layer fashion, with each layer consisting of a linear matrix-vector multiplication followed by the application of an element-wise nonlinear function, or \textit{activation}, on the result.
% For a layer with index $i$, containing a weight matrix $\hat{W}_i$ and activation function $f_i(\cdot)$, its operation is described mathematically as
% \begin{equation}
%    x_i = f_i{\left( \hat{W}_i \cdot x_{i-1} \right)}
% \end{equation}
% for $i$ from 1 to $L$.

% Before they are able to perform a given machine learning task, ANNs must be trained.
% The training process is typically accomplished by minimizing the prediction error of the ANN on a set of training examples, which come in the form of input and target output pairs.
% For a given ANN, a loss function is defined to quantify the difference between the target output and output predicted by the network.
% During training, this loss function is minimized with respect to tunable degrees of freedom, namely the elements of the weight matrix $\hat{W}_i$ within each layer.
% In general, although less common, it is also possible to train the parameters of the activation functions \cite{trentin_networks_2001}.

There are significant efforts in constructing artificial neural network architectures using various electronic solid-state platforms \cite{Merolla2014,Prezioso2015}, but ever since the conception of ANNs, a hardware implementation using optical signals has also been considered \cite{Abu-Mostafa1987, Jutamulia1996}. In this domain, some of the recent work has been devoted to photonic spike processing \cite{Rosenbluth2009, Tait2014} and photonic reservoir computing \cite{Brunner2013, Vandoorne2014}, as well as to devising universal, chip-integrated photonic platforms that can implement any arbitrary ANN \cite{Shainline2017, shen2017deep}.  Photonic implementations benefit from the fact that, due to the non-interacting nature of photons, linear operations -- like the repeated matrix multiplications found in every neural network algorithm -- can be performed in parallel, and at a lower energy cost, when using light as opposed to electrons.

A key requirement for the utility of any ANN platform is the ability to train the network using algorithms such as error backpropagation \cite{Rumelhart1986}. Such training typically demands significant computational time and resources and it is generally desirable for error backpropagation to be implemented on the same platform. This is indeed possible for the technologies of Refs. \cite{Merolla2014, Graves2016, Hermans2015} and has also been demonstrated e.g. in memristive devices \cite{alibart2013pattern, Prezioso2015}. In optics, as early as three decades ago, an adaptive platform that could approximately implement the backpropagation algorithm experimentally was proposed \cite{wagner1987multilayer,psaltis1988adaptive}. However, this algorithm requires a number of complex optical operations that are difficult to implement, particularly in integrated optics platforms. Thus, the current implementation of a photonic neural network using integrated optics has been trained using a model of the system simulated on a regular computer \cite{shen2017deep}. This is inefficient for two reasons. First, this strategy depends entirely on the accuracy of the model representation of the physical system. Second, unless one is interested in deploying a large number of identical, fixed copies of the ANN, any advantage in speed or energy associated with using the photonic circuit is lost if the training must be done on a regular computer.  Alternatively, training using a brute force, \textit{in situ} computation of the gradient of the objective function has been proposed \cite{shen2017deep}. However, this strategy involves sequentially perturbing each individual parameter of the circuit, which is highly inefficient for large systems.

To overcome this, we proposed \cite{hughes2018training} a procedure to compute the gradient of the cost function of a photonic ANN by use of only \textit{in situ} intensity measurements.    Our procedure works by \textit{physically} implementing the adjoint method in the device \cite{Georgieva2002, Veronis2004, hughes_method_2017}.  As discussed, the adjoint method scales in constant time with respect to the number of parameters, which allows for backpropagation to be efficiently implemented in a hybrid opto-electronic network.  Although we focus our discussion on a particular hardware implementation of a photonic ANN, our conclusions are derived starting from Maxwell’s equations, and may therefore be extended to other photonic platforms.

First, we introduce the operation and gradient computation of a feed-forward photonic ANN.  In its most general case, a feed-forward ANN maps an input vector to an output vector via a sequence of linear operations and element-wise nonlinear functions of the vectors, also called `activations'.  A cost function, $\mathcal{L}$, is defined over the outputs of the ANN and the matrix elements involved in the linear operations are tuned to minimize $\mathcal{L}$ over a number of training examples via gradient-based optimization.  The \textit{backpropagation algorithm} is typically used to compute these gradients analytically by sequentially utilizing the chain rule from the output layer backwards to the input layer.

Here, we will outline these steps mathematically for a single training example, with the procedure diagrammed in Fig. \ref{fig:backprop}a.  We focus our discussion on the photonic hardware platform presented in \cite{shen2017deep}, which performs the linear operations using optical interference units (OIUs).  The OIU is a mesh of controllable Mach-Zehnder interferometers (MZIs) integrated in a silicon photonic circuit. It is worth noting that this is the same component proposed in the previous Chapter to control power delivery to the accelerator on a chip.
By tuning the phase shifters integrated in the MZIs, any unitary $N \times N$ operation on the input can be implemented \cite{Reck1994,Clements2016}, which finds applications both in classical and quantum photonics \cite{Carolan2015, Harris2017}.  In the photonic ANN implementation from Ref. \cite{shen2017deep}, an OIU is used for each linear matrix-vector multiplication, whereas the nonlinear activations are performed using an electronic circuit, which involves measuring the optical state before activation, performing the nonlinear activation function on an electronic circuit such as a digital computer, and preparing the resulting optical state to be injected to the next stage of the ANN.

We first introduce the notation used to describe the OIU, which consists of a number, $N$, of single-mode waveguide input ports coupled to the same number of single-mode output ports through a linear and lossless device. In principle, the device may also be extended to operate on a different number of inputs and outputs. We further assume directional propagation such that all power flows exclusively from the input ports to the output ports, which is a typical assumption for the devices of Refs. \cite{Miller2013a, shen2017deep, Harris2017, Carolan2015, Reck1994,miller_selfconfiguring_2013,Clements2016}. In its most general form, the device implements the linear operation
\begin{equation}
\hat{W}\mathbf{X}_\textrm{in} = \mathbf{Z}_\textrm{out},
\label{eq:original_linear_system}
\end{equation}
where $\mathbf{X}_\textrm{in}$ and $\mathbf{Z}_\textrm{out}$ are the modal amplitudes at the input and output ports, respectively, and $\hat{W}$, which we will refer to as the transfer matrix, is the off-diagonal block of the system's full scattering matrix,
\begin{equation}
\begin{pmatrix}
\mathbf{X}_\textrm{out} \\
\mathbf{Z}_\textrm{out}
\end{pmatrix} = \begin{pmatrix}
0 & \hat{W}^T \\
\hat{W} & 0
\end{pmatrix}
\begin{pmatrix}
\mathbf{X}_\textrm{in} \\
\mathbf{Z}_\textrm{in}
\end{pmatrix}.
\label{eq:smatrix}
\end{equation}
Here, the diagonal blocks are zero because we assume forward-only propagation, while the off-diagonal blocks are the transpose of each other because we assume a reciprocal system. $\mathbf{Z}_\textrm{in}$ and $\mathbf{X}_\textrm{out}$ correspond to the input and output modal amplitudes, respectively, if we were to run this device in reverse, i.e. sending a signal in from the output ports.

%\begin{figure}[H]
%\centering
%\includegraphics[width=.7\columnwidth]{figures/insitu_W_diagram}
%\caption{\label{fig:W} Notation describing the $\hat{W}$ operator}
%\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/insitu_backprop}
\caption{\label{fig:backprop} (a) A schematic of the ANN architecture demonstrated in Ref \cite{shen2017deep}.  The boxed regions correspond to OIUs that perform a linear operation represented by the matrix $\hat{W}_l$.  Integrated phase shifters (blue) are used to control the OIU and train the network.  The red regions correspond to nonlinear activations $\mathbf{f}_l(\cdot)$.  (b) Illustration of operation and gradient computation in an ANN.  The top and bottom rows correspond to the forward and backward propagation steps, respectively.  Propagation through a square cell corresponds to matrix multiplication.  Propagation through a rounded region corresponds to activation.  $\odot$ is element-wise vector multiplication.}
\end{figure}

Now we may use this notation to describe the forward and backward propagation steps in a photonic ANN.  In the forward propagation step, we start with an initial input to the system, $\mathbf{X}_0$, and perform a linear operation on this input using an OIU represented by the matrix $\hat{W}_{1}$.  This is followed by the application of a element-wise nonlinear activation, $\mathbf{f}_1(\cdot)$, on the outputs, giving the input to the next layer.  This process repeats for the each layer $l$ until the output layer, $L$.  Written compactly, for $l = 1~...~L$
\begin{equation}
\mathbf{X}_l = \mathbf{f}_l(\hat{W}_l\mathbf{X}_{l-1}) \equiv \mathbf{f}_l(\mathbf{Z}_l).
\label{eq:NN_recursive}
\end{equation}
Finally, our cost function $\mathcal{L}$ is an explicit function of the outputs from the last layer, $\mathcal{L} = \mathcal{L}(\mathbf{X}_L)$.  This process is shown in Fig. \ref{fig:backprop}(a).

The nonlinear activation functions, $\mathbf{f}_l()$, play a key role in ANNs by enabling them to learn complex mappings between their inputs and outputs.
Whereas digital processors have the expressiveness to trivially apply nonlinearities such as the widely-used \texttt{sigmoid}, \texttt{ReLU}, and \texttt{tanh} functions, the realization of nonlinearities in optical hardware platforms is more challenging.
One reason for this is that optical nonlinearities are relatively weak, necessitating a combination of large interaction lengths and high signal powers, which impose lower bounds on the physical footprint and the energy consumption, respectively.
Although it is possible to resonantly enhance optical nonlinearities, this comes with an unavoidable trade-off in reducing the operating bandwidth, thereby limiting the information processing capacity of an ONN.
Additionally, maintaining uniform resonant responses across many elements of an optical circuit necessitates additional control circuitry for calibrating each element \cite{radulaski_thermally_2018}.

A more fundamental limitation of optical nonlinearities is that their responses tend to be fixed during device fabrication.
This limited tunability of the nonlinear optical response prevents an ONN from being reprogrammed to realize different forms of nonlinear activation functions, which may be important for tailoring ONNs for different machine learning tasks.
Similarly, a fixed nonlinear response may also limit the performance of very deep ONNs with many layers of activation functions since the optical signal power drops below the activation threshold, where nonlinearity is strongest, in later layers due to loss in previous layers.
For example, with optical saturable absorption from 2D materials in waveguides, the activation threshold is on the order of 1-10 mW \cite{bao_monolayer_2011, park_monolayer_2015, jiang_low-dimensional_2018}, meaning that the strength of the nonlinearity in each subsequent layer will be successively weaker as the transmitted power falls below the threshold.

In light of these challenges, the ONN demonstrated in Ref. \citenum{shen2017deep} implemented its activation functions by detecting each optical signal, feeding them through a conventional digital computer to apply the nonlinearity, and then modulating new optical signals for the subsequent layer.
Although this approach benefits from the flexibility of digital signal processing, conventional processors have a limited number of input and output channels, which make it challenging to scale this approach to very large matrix dimensions, which corresponds to a large number of optical inputs.
Moreover, digitally applied nonlinearities add latency from the analog-to-digital conversion process and constrain the computational speed of the neural network to the same GHz-scale clock rates which ONNs seek to overcome.
Thus, a hardware nonlinear optical activation, which doesn't require repeated bidirectional optical-electronic signal conversion, is of fundamental interest for making integrated ONNs a viable machine learning platform.

For this purpose,  as diagrammed in Fig. \ref{fig:overview}, we proposed an electro-optic architecture for synthesizing optical-to-optical nonlinearities which alleviates the issues discussed above.
Our architecture features complete \textit{on}-\textit{off} contrast in signal transmission, a variety of nonlinear response curves, and a low activation threshold.
Rather than using traditional optical nonlinearities, our scheme operates by measuring a small portion of the incoming optical signal power and using electro-optic modulators to modulate the original optical signal, without any reduction in operating bandwidth or computational speed.
Additionally, our scheme allows for the possibility of performing additional nonlinear transformations on the signal using analog electrical components.
Related electro-optical architectures for generating optical nonlinearities have been previously considered \cite{lentine_evolution_1993, majumdar_cavityenabled_2014, tait_silicon_2018}.
While we focused on the application of our architecture as an element-wise activation in a feedforward ONN, but the synthesis of low-threshold optical nonlinearities could be of broader interest to optical computing and information processing.  For more information on this work, we refer the reader to Ref.~\citenum{williamson_reprogrammable_2019}.

\begin{figure*}
  \centering
  \includegraphics{figures/insitu_activation}
  \caption{(a) Block diagram of a feedforward neural network of $L$ layers.
  Each layer consists of a $\hat{W}_i$ block representing a linear matrix which multiplies vector inputs $x_{i-1}$.
  The $f_i$ block in each layer represents an element-wise nonlinear activation function operating on vectors $z_i$ to produce outputs $x_{i}$.
  (b) Schematic of the optical interferometer mesh implementation of a single layer of the feedforward neural network.
  (c) Schematic of the proposed optical-to-optical activation function which achieves a nonlinear response by converting a small portion of the optical input, $z$ into an electrical signal, and then intensity modulating the remaining portion of the original optical signal as it passes through an interferometer.}
  \label{fig:overview}
\end{figure*}

\section{Training of Optical Hardware}

To train the network, we must minimize the cost function with respect to the linear operators, $\hat{W}_l$, which may be adjusted by tuning the integrated phase shifters within the OIUs.  While a number of recent papers have clarified how an individual OIU can be tuned by sequential,  \textit{in situ} methods to perform an arbitrary, pre-defined operation \cite{miller_selfconfiguring_2013, Miller2013a, Miller2015, Annoni2017}, these strategies do not straightforwardly apply to the training of ANNs, where nonlinearities and several layers of computation are present.  In particular, the training of ANN requires gradient information which is not provided directly in the methods of Ref. \cite{miller_selfconfiguring_2013, Miller2013a, Miller2015, Annoni2017}.

In Ref. \cite{shen2017deep}, the training of the ANN was done \textit{ex situ} on a computer model of the system, which was used to find the optimal weight matrices $\hat{W}_l$ for a given cost function. Then, the final weights were recreated in the physical device, using an idealized model that relates the matrix elements to the phase shifters. Ref. \cite{shen2017deep} also discusses a possible \textit{in situ} method for computing the gradient of the ANN cost function through a serial perturbation of every individual phase shifter (`brute force' gradient computation). However, this gradient computation has an unavoidable linear scaling with the number of parameters of the system.  The training method that we propose here operates without resorting to an external model of the system, while allowing for the tuning of each parameter to be done in parallel, therefore scaling significantly better with respect to the number of parameters when compared to the brute force gradient computation.

To introduce our training method we first use the backpropagation algorithm to derive an expression for the gradient of the cost function with respect to the permittivities of the phase shifters in the OIUs. In the following, we denote $\epsilon_l$ as the permittivity of a single, arbitrarily chosen phase shifter in layer $l$, as the same derivation holds for each of the phase shifters present in that layer.  Note that $\hat{W}_l$ has an explicit dependence on $\epsilon_l$, but all field components in the subsequent layers also depend implicitly on $\epsilon_l$.

As a demonstration, we take a mean squared cost function
\begin{align}
\mathcal{L} &= \frac{1}{2}\big(\mathbf{X}_L -\mathbf{T} \big)^\dagger \big( \mathbf{X}_L -\mathbf{T} \big),
\label{eq:NN_forward}
\end{align}
where $\mathbf{T}$ is a complex-valued target vector corresponding to the desired output of our system given input $\mathbf{X}_0$.

Starting from the last layer in the circuit, the derivative of the cost function with respect to the permittivity $\epsilon_L$ of one of the phase shifters in the last layer is given by

\begin{align}
\frac{d\mathcal{L}}{d\epsilon_L} &= \mathcal{R}\left\{\big(\mathbf{X}_L - \mathbf{T} \big)^\dagger \frac{d\mathbf{X}_L}{d\epsilon_L} \right\}\\
    &= \mathcal{R}\left\{ \left(  \bm{\Gamma}_L \odot {\mathbf{f}_L}^{'}(\mathbf{Z}_{L}) \right)^T \frac{d \hat{W}_L}{d\epsilon_L}\mathbf{X}_{L-1} \right\}\\
    &\equiv \mathcal{R}\left\{ \boldsymbol{\delta}_L^T \frac{d \hat{W}_L}{d\epsilon_L} \mathbf{X}_{L-1} \right\},
\label{eq:backprop_L}
\end{align}
where $\odot$ is element-wise vector multiplication, defined such that, for vectors $\mathbf{a}$ and $\mathbf{b}$, the $i$-th element of the vector $\mathbf{a} \odot \mathbf{b}$ is given by $a_i b_i$. $\mathcal{R}\{\cdot\}$ gives the real part, ${\mathbf{f}_l}^{'}(\cdot)$ is the derivative of the $l$th layer activation function with respect to its (complex) argument.  We define the vector $\bm{\delta}_L \equiv \bm{\Gamma}_L \odot {\mathbf{f}_L}^{\,'}$ in terms of the error vector $\bm{\Gamma}_L \equiv \big(\mathbf{X}_L - \mathbf{T} \big)^*$.

For any layer $l < L$, we may use the chain rule to perform a recursive calculation of the gradients
\begin{align}
\bm{\Gamma}_l &= \hat{W}^T_{l+1} \bm{\delta}_{l+1} \label{eq:backprop_general_Gamma}
\\
\bm{\delta}_l &= \bm{\Gamma}_l \odot {\mathbf{f}_l}^{'}(\mathbf{Z}_{l}) \label{eq:backprop_general2}
\\
\frac{d\mathcal{L}}{d\epsilon_l} &= \mathcal{R}\left\{ \bm{\delta}_l^T \frac{d \hat{W}_l}{d\epsilon_l} \mathbf{X}_{l-1} \right\}.
\label{eq:backprop_general}
\end{align}
Figure \ref{fig:backprop}(b) diagrams this process, which computes the $\bm{\delta}_l$ vectors sequentially from the output layer to the input layer.  A treatment for non-holomorphic activations is derived in Appendix II.

We note that the computation of $\bm{\delta}_l$ requires performing the operation $\bm{\Gamma}_l = \hat{W}^T_{l+1} \bm{\delta}_{l+1}$, which corresponds physically to sending $\bm{\delta}_{l+1}$ into the output end of the OIU in layer $l+1$.  In this way, our procedure `backpropagates' the vectors $\bm{\delta}_l$ and $\bm{\Gamma}_l$ physically through the entire circuit.

%We note two interesting features present in Eq. (\ref{eq:backprop_general}).  First, this expression is the derivative of $\mathcal{R}\left\{\boldsymbol{\delta}_l^T \hat{W}_l \mathbf{X}_{l-1} \right\} = \mathcal{R}\left\{\boldsymbol{\delta}_l^T \mathbf{Z}_{l} \right\}$ with respect to $\epsilon_l$.  Thus, the minimization of this loss function requires minimizing the modal overlap between the output $\mathbf{Z}_l$ and the vector $\boldsymbol{\delta}_l^*$ for each layer in the network.  However, we note that during training, both $\mathbf{Z}_l$ and $\boldsymbol{\delta}_l$ are not static as changes to the surrounding OIUs will affect their values.  \textcolor{purple}{add some clarification to the previous paragraphs}.  Secondly, w

\section{Protocols for Experimental Measurement of Adjoint Gradient}

In the previous Section, we showed that the crucial step in training the ANN is computing gradient terms of the form  $\mathcal{R}\left\{\boldsymbol{\delta}_l^T \frac{d \hat{W}_l}{d\epsilon_l} \mathbf{X}_{l-1}\right\}$, which contain derivatives with respect to the permittivity of the phase shifters in the OIUs. In this Section, we show how this gradient may be expressed as the solution to an electromagnetic adjoint problem.

The OIU used to implement the matrix $\hat{W}_l$, relating the complex mode amplitudes of input and output ports, can be described using first-principles electrodynamics.  This will allow us to compute its gradient with respect to each $\epsilon_l$, as these are the physically adjustable parameters in the system. Assuming a source at frequency $\omega$, at steady state Maxwell's equations take the form
\begin{equation}
\Big[ \hat{\nabla} \times \hat{\nabla} \times ~ -k_0^2 \hat{\epsilon}_r \Big]\mathbf{e} = -i\omega \mu_0 \mathbf{j},
\label{eq:maxwells_equations_physics}
\end{equation}
which can be written more succinctly as
\begin{equation}
\hat{A}(\epsilon_r) \mathbf{e} = \mathbf{b}
\label{eq:maxwells_equations_A}.
\end{equation}
Here, $\hat{\epsilon}_r$ describes the spatial distribution of the relative permittivity ($\epsilon_r$), $k_0=\omega^2/c^2$ is the free-space wavenumber, $\mathbf{e}$ is the electric field distribution, $\mathbf{j}$ is the electric current density, and $\hat{A} = \hat{A}^T$ due to Lorentz reciprocity. Eq. (\ref{eq:maxwells_equations_A}) is the starting point of the finite-difference frequency-domain (FDFD) simulation technique \cite{shin2012choice}, where it is discretized on a spatial grid, and the electric field $\mathbf{e}$ is solved given a particular permittivity distribution, $\boldsymbol{\epsilon}_r$, and source, $\mathbf{b}$.

To relate this formulation to the transfer matrix $\hat{W}$, we now define source terms $\mathbf{b}_i$, $i \in 1 \dots 2N$, that correspond to a source placed in one of the input or output ports.  Here we assume a total of $N$  input and $N$ output waveguides.  The spatial distribution of the source term, $\mathbf{b}_i$, matches the mode of the $i$-th single-mode waveguide. Thus, the electric field amplitude in port $i$ is given by $\mathbf{b}_i^{\ T} \mathbf{e}$, and we may establish a relationship between $\mathbf{e}$ and $\mathbf{X}_\textrm{in}$, as
\begin{equation}
X_{\textrm{in},i} = \mathbf{b}_i^{\ T} \mathbf{e}
\end{equation}
for $i = 1~...~N$ over the input port indices, where $X_{\textrm{in},i}$ is the $i$-th component of $\mathbf{X}_\textrm{in}$. Or more compactly,
\begin{equation}
\mathbf{X}_\textrm{in} \equiv \hat{P}_{\textrm{in}} \mathbf{e},
\label{eq:basis_conversion_in}
\end{equation}
Similarly, we can define
\begin{equation}
Z_{\textrm{out},i} = \mathbf{b}_{i+N}^{\ T} \mathbf{e}
\end{equation}
for $i+N = (N+1)~...~2N$ over the output port indices, or,
\begin{equation}
\mathbf{Z}_\textrm{out} \equiv \hat{P}_{\textrm{out}} \mathbf{e},
\label{eq:basis_conversion_in}
\end{equation}
and, with this notation, Eq. (\ref{eq:original_linear_system}) becomes
\begin{equation}
\hat{W}\hat{P}_{\textrm{in}}\mathbf{e} = \hat{P}_{\textrm{out}}\mathbf{e}
\label{eq:summary_of_basis_change}
\end{equation}

%We can now apply this to the gradient of the ANN loss function from the previous Section.
%For layer $l$, the gradient of the loss function with respect to the $i$-th phase shifter in this layer, labeled $\epsilon_l$, was given by
%\begin{equation}
%\frac{d\mathcal{L}}{d\epsilon_l} = \mathcal{R}\left\{ \boldsymbol{\delta}_l^T \frac{d %\hat{W}_l}{d\epsilon_l}\mathbf{X}_{l-1} \right\},
%\label{eq:backprop_general_copy}
%\end{equation}
%where the $\boldsymbol{\delta}_l$ vectors were computed by propagating backwards through the system.  It is simple to show that this term is the gradient of a sub-problem with an objective function of the form
%\begin{equation}
%\mathcal{L}(\phi) = \mathcal{R}\left\{ \boldsymbol{\delta}^T \hat{W}(\phi)\mathbf{X} %\right\} = \mathcal{R}\left\{ \boldsymbol{\delta}^T \mathbf{Z}(\phi) \right\}.
%\label{eq:objective_function_definition}
%\end{equation}
%for a feed-forward system.  Here the layer subscript, $l$, and the phase shifter %number $i$ are implicit.  Thus, training the ANN corresponds to minimizing the modal %overlap between the output of the system at each layer, $\mathbf{Z}$, and the %complex conjugate of the delta vector for that layer, $\boldsymbol{\delta}^*$.

%
%Using Eq. (\ref{eq:maxwells_equations_A}) and Eq. (\ref{eq:summary_of_basis_change}), combined with the fact that $\frac{d\mathbf{X}_{l-1}}{d\epsilon_l}=0$, we may compute an expression for $ \frac{d\hat{W}_l}{d\epsilon_l}\mathbf{X}_{l-1}$ in terms of the Maxwell operator, $\hat{A}$.  Then, this can be inserted into Eq. (\ref{eq:backprop_general}) to obtain

We now use the above definitions to evaluate the cost function gradient in Eq. (\ref{eq:backprop_general}). In particular, with Eqs. (\ref{eq:backprop_general}) and (\ref{eq:summary_of_basis_change}), we arrive at
\begin{equation}
\frac{d\mathcal{L}}{d\epsilon_l} = -\mathcal{R}\left\{\boldsymbol{\delta_l}^T \hat{P}_{\textrm{out}} \hat{A}^{-1} \frac{d\hat{A}}{d\epsilon_l} \hat{A}^{-1} \mathbf{b}_{x,l-1} \right\}.
\label{eq:objective_function_derivative_EM_basis}
\end{equation}
Here $\mathbf{b}_{x,l-1}$ is the modal source profile that creates the input field amplitudes $\mathbf{X}_{l-1}$ at the input ports.

The key insight of the adjoint variable method is that we may interpret this expression as an operation involving the field solutions of two electromagnetic simulations, which we refer to as the `original' (og) and the `adjoint' (aj)
\begin{align}
\hat{A} \mathbf{e}_{\textrm{og}} &= \mathbf{b}_{x,l-1} \\
\hat{A} \mathbf{e}_{\textrm{aj}} &= \hat{P}_{\textrm{out}}^T \boldsymbol{\delta},
\label{eq:adjoint}
\end{align}
where we have made use of the symmetric property of $\hat{A}$.

Eq. (\ref{eq:objective_function_derivative_EM_basis}) can now be expressed in a compact form as
\begin{equation}
\frac{d\mathcal{L}}{d\epsilon_l} = -\mathcal{R}\left\{\mathbf{e}_{\textrm{aj}}^T\frac{d\hat{A}}{d\epsilon_l}\mathbf{e}_{\textrm{og}} \right\}.
\label{eq:objective_function_derivative_simple}
\end{equation}

If we assume that this phase shifter spans a set of points, $\mathbf{r}_\phi$ in our system, then, from Eq. (\ref{eq:maxwells_equations_physics}), we obtain
\begin{equation}
\frac{d\hat{A}}{d\epsilon_l} = -k_0^2 \sum_{{\mathbf{r}\,}' \in \mathbf{r}_\phi}\hat{\delta}_{\mathbf{r},{\mathbf{r}\,}'},
\label{eq:dAdphi}
\end{equation}
where $\hat{\delta}_{\mathbf{r},{\mathbf{r}\,}'}$ is the Kronecker delta.

Inserting this into Eq. (\ref{eq:objective_function_derivative_simple}), we thus find that the gradient is given by the overlap of the two fields over the phase-shifter positions
%
\begin{equation}
\frac{d\mathcal{L}}{d\epsilon_l} = k_0^2 \mathcal{R}\left\{ \sum_{\mathbf{r} \in \mathbf{r}_\phi} \mathbf{e}_{\textrm{aj}}(\mathbf{r}) \mathbf{e}_{\textrm{og}}(\mathbf{r}) \right\}.
\label{eq:sensitivity_sum}
\end{equation}
%
This result now allows for the computation in parallel of the gradient of the loss function with respect to \textit{all} phase shifters in the system, given knowledge of the original and adjoint fields.

\begin{figure}[ht!]
\centering
\includegraphics[width=0.7\textwidth]{figures/insitu_TR}
\caption{\label{fig:TR_schematic} Schematic illustration of our proposed method for experimental measurement of gradient information. The box region represents the OIU.  The colored ovals represent tunable phase shifters, and we illustrate computing the gradient with respect to the red and the yellow phase shifters, labeled $1$ and $2$, respectively. (a): We send the original set of amplitudes $\mathbf{X}_{l-1}$ and measure the constant intensity terms at each phase shifter. (b): We send the adjoint mode amplitudes, given by $\boldsymbol{\delta}_l$, through the output side of our device, recording $\mathbf{X}_{TR}^*$ from the opposite side, as well as $|\mathbf{e}_\textrm{aj}|^2$ in each phase-shifter. (c): We send in $\mathbf{X}_{l-1}$ + $\mathbf{X}_{TR}$, interfering $\mathbf{e}_{\textrm{og}}$ and $\mathbf{e}_\textrm{aj}^*$ inside the device and recovering the gradient information for all phase shifters simultaneously.}
\end{figure}

We now propose a method to compute the gradient from the previous section through \textit{in situ} intensity measurements.  This represents the most significant result of this chapter.  Specifically, we wish to generate an intensity pattern with the form $\mathcal{R}\big\{\mathbf{e}_\textrm{og} \mathbf{e}_\textrm{aj} \big\}$, matching that of Eq. (\ref{eq:sensitivity_sum}).  We note that interfering $\mathbf{e}_{\textrm{og}}$ and $\mathbf{e}_\textrm{aj}^{\,*}$ directly in the system results in the intensity pattern:
\begin{equation}
I = |\mathbf{e}_\textrm{og}|^2 + |\mathbf{e}_\textrm{aj}|^2 + 2\mathcal{R}\big\{\mathbf{e}_\textrm{og}\mathbf{e}_{\textrm{aj}} \big\},
\label{eq:intensity_correct}
\end{equation}
the last term of which matches Eq. (\ref{eq:sensitivity_sum}). Thus, the gradient can be computed purely through intensity measurements if the field $\mathbf{e}_\textrm{aj}^{\,*}$ can be generated in the OIU.

\begin{figure*}[t]
\includegraphics[width=\textwidth]{figures/insitu_FDFD}
\caption{\label{fig:fdfd} Numerical demonstration of the time-reversal procedure.  (a): Relative permittivity distribution for three MZIs arranged to perform a 3x3 linear operation. Blue boxes represent where phase shifters would be placed in this system. As an example, we compute the gradient information for a layer with $\mathbf{X}_{l-1} = [0~0~1]^T$ and $\boldsymbol{\delta}_l = [0~1~0]^T$, corresponding to the bottom left and middle right port, respectively. (b): Real part of the simulated electric field $E_z$ corresponding to injection from the bottom left port. (c): Real part of the adjoint $E_z$, corresponding to injection from the middle right port. (d): Time-reversed adjoint field as constructed by our method, fed in through all three ports on the left. (e): The gradient information $\frac{d\mathcal{L}}{d\epsilon_l}(x,y)$ as obtained directly by the adjoint method, normalized by its maximum absolute value. (f): The gradient information as obtained by the method introduced in this work, normalized by its maximum absolute value. Namely, the field pattern from (b) is interfered with the time-reversed adjoint field of (d) and the constant intensity terms are subtracted from the resulting intensity pattern. Panels (e) and (f) match with high precision.}
\end{figure*}

The adjoint field for our problem, $\mathbf{e}_\textrm{aj}$, as defined in Eq. (\ref{eq:adjoint}), is sourced by $\hat{P}_{\textrm{out}}^T \boldsymbol{\delta}_l$, meaning that it physically corresponds to a mode sent into the system from the output ports.  As complex conjugation in the frequency domain corresponds to time-reversal of the fields, we expect $\mathbf{e}^{\,*}_{\textrm{aj}}$ to be sent in from the input ports. Formally, to generate $\mathbf{e}^{\,*}_{\textrm{aj}}$, we wish to find a set of input source amplitudes, $\mathbf{X}_{TR}$, such that the output port source amplitudes, $\mathbf{Z}_{TR} = \hat{W}\mathbf{X}_{TR}$, are equal to the complex conjugate of the adjoint amplitudes, or $\boldsymbol{\delta}_l^*$. Using the unitarity property of transfer matrix $\hat{W}_l$ for a lossless system, along with the fact that $\hat{P}_{\textrm{out}} \hat{P}_{\textrm{out}}^T = \hat{I}$ for output modes, the input mode amplitudes for the time-reversed adjoint can be computed as
\begin{equation}
\mathbf{X}_{TR}^* = \hat{W}_l^T\boldsymbol{\delta}_l.
\label{eq:TR_adjoint}
\end{equation}
As discussed earlier, $\hat{W}_l^T$ is the transfer matrix from output ports to input ports. Thus, we can experimentally determine $\mathbf{X}_{TR}$ by sending $\boldsymbol{\delta}_l$ into the device output ports, measuring the output at the input ports, and taking the complex conjugate of the result.

We now summarize the procedure for experimentally measuring the gradient of an OIU layer in the ANN with respect to the permittivities of this layer's integrated phase shifters:
\begin{enumerate}
  \itemsep0em
  \item Send in the original field amplitudes $\mathbf{X}_{l-1}$ and measure and store the intensities at each phase shifter.
  \item Send $\boldsymbol{\delta}_l$ into the output ports and measure and store the intensities at each phase shifter.
  \item Compute the time-reversed adjoint input field amplitudes as in Eq. (\ref{eq:TR_adjoint}).
  \item Interfere the original and the time-reversed adjoint fields in the device, measuring again the resulting intensities at each phase shifter.
  \item Subtract the constant intensity terms from steps 1 and 2 and multiply by $k_0^2$ to recover the gradient as in Eq. (\ref{eq:sensitivity_sum}).
\end{enumerate}

%It is interesting to note that our approach operates in a similar fashion to the time-domain formalism of the adjoint variable method \cite{nikolova2004sensitivity, abenius2006solving}, in which the sensitivity is obtained by integrating the original field with the time-reversed adjoint field. In the frequency domain, time reversal of a field mathematically corresponds to complex conjugation.

%Generating a time-reversed, physical copy of a wave is a difficult task in general, requiring phase conjugation \cite{feinberg1995phase, cronin1995theory}, time-reversal mirrors \cite{fink1992time, lerosey2004time}, or other techniques \cite{bacot2016time}.  However, given the feed-forward nature of our system, combined with the fact that we have full access to all input and output channels in the form of single mode waveguides, this process can be greatly simplified -- specifically, we can define sources at the input ports that generate the time-reversed adjoint $\mathbf{e}_{\textrm{aj}}^*$.

This procedure is also illustrated in Fig. \ref{fig:TR_schematic}.

\subsection{Numerical Demonstrations}

We numerically demonstrate this procedure in Fig. \ref{fig:fdfd} with a series of FDFD simulations of an OIU implementing a $3 \times 3$ unitary matrix \cite{Reck1994}. These simulations are intended to represent the gradient computation corresponding to one OIU in a single layer, $l$, of a neural network with input $\mathbf{X}_{l-1}$ and delta vector $\boldsymbol{\delta}_l$. In these simulations, we use absorbing boundary conditions on the outer edges of the system to eliminate back-reflections.  The relative permittivity distribution is shown in Fig. \ref{fig:fdfd}(a) with the positions of the variable phase shifters in blue.  For demonstration, we simulate a specific case where $\mathbf{X}_{l-1} = [0~0~1]^T$, with unit amplitude in the bottom port and we choose $\boldsymbol{\delta}_l = [0~1~0]^T$.  In Fig. \ref{fig:fdfd}(b), we display the real part of $\textbf{e}_\textrm{og}$, corresponding to the original, forward field.

The real part of the adjoint field, $\textbf{e}_\textrm{aj}$, corresponding to the cost function $\mathcal{L} = \mathcal{R}\left\{\boldsymbol{\delta}_l^T \hat{W}_l \mathbf{X}_{l-1} \right\}$ is shown in Fig. \ref{fig:fdfd}(c).  In Fig. \ref{fig:fdfd}(d) we show the real part of the time-reversed copy of $\textbf{e}_\textrm{aj}$ as computed by the method described in the previous section, in which $\mathbf{X}^*_{TR}$ is sent in through the input ports.  There is excellent agreement, up to a constant, between the complex conjugate of the field pattern of (c) and the field pattern of (d), as expected.

In Fig. \ref{fig:fdfd}(e), we display the gradient of the objective function with respect to the permittivity of each point of space in the system, as computed with the adjoint method, described in Eq. (\ref{eq:sensitivity_sum}).  In Fig. \ref{fig:fdfd}(f), we show the same gradient information, but instead computed with the method described in the previous section.  Namely, we interfere the field pattern from panel (b) with the field pattern from panel (d), subtract constant intensity terms, and multiply by the appropriate constants.  Again, (b) and (d) agree with good precision.

We note that in a realistic system, the gradient must be constant for any stretch of waveguide between waveguide couplers because the interfering fields are at the same frequency and are traveling in the same direction.  Thus, there should be no distance dependence in the corresponding intensity distribution. This is largely observed in our simulation, although small fluctuations are visible because of the proximity of the waveguides and the sharp bends, which were needed to make the structure compact enough for simulation within a reasonable time. In practice, the importance of this constant intensity is that it can be detected \textit{after} each phase shifter, instead of inside of it.

Finally, we note that this numerically generated system experiences a total power loss of 41\% due to scattering caused by very sharp bends and stair-casing of the structure in the simulation.  We also observe approximately 5-10\% mode-dependent loss, as determined by measuring the difference in total transmitted power corresponding to injection at different input ports.  Minimal amounts of reflection are also visible in the field plots.  Nevertheless, the time-reversal interference procedure still reconstructs the adjoint sensitivity with very good fidelity.

\begin{figure}
\centering
\includegraphics[width=\columnwidth]{figures/insitu_ANN_demo}
\caption{Numerical demonstration of a photonic ANN implementing an XOR gate using the backpropagation algorithm and adjoint method described in this work. (a) The architecture of the ANN.  Two layers of $3 \times 3$ OIUs with $z^2$ activations.  (b) The mean-squared error (MSE) between the predictions and targets as a function of training iterations. (c) The absolute value of the network predictions (blue circles) and targets (black crosses) before training.  (d) The absolute value of the network predictions after training, showing that the network has successfully learned the XOR function. \label{fig:demo} }
\end{figure}

%\subsection{Numerical Demonsrtation}

Finally, we use the techniques from the previous Sections to numerically demonstrate the training of a photonic ANN to implement a logical XOR gate, defined by the following input to target ($\mathbf{X}_0 \to \mathbf{T}$) pairs
\begin{equation}
[0~0]^T \to 0,  ~~~ [0~1]^T \to 1,  ~~~ [1~0]^T \to 1, ~~~ [1~1]^T \to 0.
\label{eq:XOR_definition}
\end{equation}
This problem was chosen as demonstration of learning a nonlinear mapping from input to output \cite{Vandoorne2014} and is simple enough to be solved with a small network with only four training examples.

As diagrammed in Fig. \ref{fig:demo}a, we choose a network architecture consisting of two $3 \times 3$ unitary OIUs.  On the forward propagation step, the binary representation of the inputs, $\textbf{X}_0$, is sent into the first two input elements of the ANN and a constant value of $1$ is sent into the third input element, which serves to introduce artificial bias terms into the network.  These inputs are sent through a $3 \times 3$ unitary OIU and then the element-wise activation $f(z) = z^2$ is applied.  The output of this step is sent to another $3 \times 3$ OIU and sent through another activation of the same form.  Finally, the first output element is taken to be our prediction, $\textbf{X}_L$, ignoring the last two output elements.  Our network is repeatedly trained on the four training examples defined in Eq. (\ref{eq:XOR_definition}) and using the mean-squared cost function presented in Eq. (\ref{eq:NN_forward}).

For this demonstration, we utilized a matrix model of the system, as described in \cite{Reck1994,Clements2016}.  This model allows us to compute an output of the system given an input mode and the settings of each phase shifter.  Although this is not a first-principle electromagnetic simulation of the system, it provides information about the complex fields at specific reference points within the circuit, which enables us to implement training using the backpropagation method, combined with the adjoint gradient calculation.  Using these methods, at each iteration of training we compute the gradient of  our cost function with respect to the phases of each of the integrated phase shifters, and sum them over the four training examples.  Then, we perform a simple steepest-descent update to the phase shifters, in accordance with the gradient information. This is consistent with the standard training protocol for an ANN implemented on a conventional computer.  More details on this formalism are presented in Appendix I.  Our network successfully learned the XOR gate in around 400 iterations. The results of the training are shown in Fig. \ref{fig:demo}b-d.

Next, we demonstrate training on a more complex problem. Specifically, we generate a set of one thousand training examples represented by input and target $(\mathbf{X}_0 \to \mathbf{T})$ pairs. Here, $\mathbf{X}_0 = [x_1, x_2, P, 0]^T$ where $x_1$ and $x_2$ are the independent inputs, which we constrain to be real for simplicity, and $P(x_1, x_2)= \sqrt{P_0-x_1^2-x_2^2}$ represents a mode added to the third port to make the norm of $\mathbf{X}_0$ the same for each training example.  In this case, we choose $P_0 = 10$.  Each training example has a corresponding label, $y \in \{0,1\}$ which is encoded in the desired output, $\mathbf{T}$, as $[1, 0, 0, 0].^T$ and $[0, 1, 0, 0].^T$ for $y = 0$ and $y=1$ respectively.

For a given $x_1$ and $x_2$, we define $r$ and $\phi$ as the magnitude and phase of the vector $(x_1, x_2)$ in the 2D-plane, respectively. To generate the corresponding class label, we first generate a uniform random variable between 0 and 1, labeled $\mathcal{U}$, and then set $y=1$ if
\begin{equation}
\exp{\left(-\frac{(r-r_0-\Delta \sin(2\phi))^2}{2\sigma^2}\right)} + 0.1~\mathcal{U} > 0.5.
\end{equation}
Otherwise, we set $y=0$.  For the demonstration, $r_0 = 0.6$, $\Delta = 0.15$, and $\sigma = 0.2$. The underlying distribution thus resembles an oblong ring centered around $x_1 = x_2 = 0$, with added noise.

As diagrammed in Fig. \ref{fig:demo}(a), we use a network architecture consisting of six $4 \times 4$ layers of unitary OIUs, with an element-wise activation $f(z) = |z|$ after each unitary transformation except for the last in the series, which has an activation of $f(z) = |z|^2$. After the final activation, we apply an additional `softmax' activation, which gives a normalized probability distribution corresponding to the predicted class of $\mathbf{X}_0$. Specifically, these are given by $s(z_i) = \exp{(z_i)}/\left( \sum_j \exp{(z_j)} \right)$, where $z_{i = 1, 2}$ is the first/second element of the output vector of the last activation (the other two elements are ignored). The ANN prediction for the input $\mathbf{X}_0$ is set as the larger one of these two outputs, while the total cost function is defined in the cross-entropy form
%
\begin{equation}
\mathcal{\mathcal{L}} = \frac{1}{M}\sum_{m=1}^M \mathcal{L}^{(m)} = \frac{1}{M}\sum_{m=1}^M -\log(s(z_{m, t})),
\label{eq:cross_entropy}
\end{equation}
%
where $\mathcal{L}^{(m)}$ is the cost function of the $m$-th example, the summation is over all training examples, and $z_{m, t}$ is the output from the target port, $t$, as defined by the target output $\mathbf{T}^{(m)}$ of the $m$-th example. We randomly split our generated examples into a training set containing 75\% of the originally generated training examples, while the remaining 25\% are used as a test set to evaluate the performance of our network on unseen examples.

\begin{figure}
\includegraphics[width=\columnwidth]{figures/insitu_ring}
\caption{Numerical demonstration of a photonic ANN learning to classify an oblong ring. (a) The architecture of the ANN.  Six layers of $4 \times 4$ OIUs with $|z|$ activations.  A final softmax activation is applied at the very end.  (b) The loss function of Eq. (\ref{eq:cross_entropy}) over training iterations. (c) The training examples, blue and red dots correspond to $y=0$ and $y=1$ labels on a given $x_1$ and $x_2$ input.  The background shows the prediction of the network on a continuum of $x_1$ and $x_2$ pairs, with colors representing the corresponding predictions.  One can see that the ring was learned successfully without overfitting.  \label{fig:demo} }
\end{figure}

At each iteration of training we compute the gradient of the cost function with respect to the phases of each of the integrated phase shifters, and sum this over each of the training examples. For the backpropagation through the activation functions, since $|z|$ and $|z|^2$ are non-holomorphic, we use the notes from Appendix II to obtain
%
\begin{align}
\bm{\delta}_L &= 2 \mathbf{Z}_L^* \odot \mathcal{R}\{\bm{\Gamma}_L\} \\
\bm{\delta}_l &= \exp(-i\bm{\phi}_l) \odot \mathcal{R}\{\bm{\Gamma}_l\},
\end{align}
where $\bm{\phi}_l$ is a vector containing the phases of $\mathbf{Z}_l$ and $\bm{\Gamma}_L$ is given by the derivative of the cross-entropy loss function for a single training example
%
\begin{equation}
\bm{\Gamma}_L = \frac{\partial \mathcal{L}^{(m)}}{\partial z_{m, i}} = s(z_{m, i}) - \delta_{i, t},
\end{equation}
%
where $\delta_{i, t}$ is the Kronecker delta.

With this, we can now compute the gradient of the loss function of eq. \ref{eq:cross_entropy} with respect to all trainable parameters, and perform a parallel, steepest-descent update to the phase shifters, in accordance with the gradient information. Our network successfully learned the this task in around 4000 iterations. The results of the training are shown in Fig. \ref{fig:demo}(b). We achieved a training and test accuracy of 91\% on both the training and test sets, indicating that the network was not overfitting to the dataset.  This can also be confirmed visually from Fig. \ref{fig:demo}(c).  The lack of perfect predictions is likely due to the inclusion of noise.

%Here we first discuss some of the practical details involved in implementing our method.  First, although we allow both the linear operations and nonlinear activations to be complex, we have enforced that the final cost function be real-valued and \textcolor{purple}{that the activations be complex differentiable, or holomorphic}.  A comprehensive overview of the use of complex values in neural networks is given in \cite{trabelsi2017deep}.

%\textcolor{purple}{Secondly, we have used the digital units to perform the nonlinear activations and their derivatives.}  We have additionally assumed that we may store the inputs $\mathbf{X}_{l-1}$ and delta vectors $\boldsymbol{\delta}_l$ from the forward and backward propagation steps in the digital units to make the gradient computation more efficient.  \textcolor{purple}{Although these assumptions are consistent with the demonstration in \cite{shen2017deep}, it would be ideal to find an optical system to replace these digital units for these functionalities}.

\subsection{Discussion}

Here, we justify some of the assumptions made in this work.  Our strategy for training a photonic ANN relies on the ability to create arbitrary complex inputs.  We note that a device for accomplishing this has been proposed and discussed in \cite{Miller2017}.  Our recovery technique further requires an integrated intensity detection scheme to occur in parallel and with virtually no loss.  This may be implemented by integrated, transparent photo-detectors, which have already been demonstrated in similar systems \cite{Annoni2017}.  Furthermore, as discussed, this measurement may occur in the waveguide regions directly after the phase shifters, which eliminates the need for phase shifter and photodetector components at the same location.  Finally, in our procedure for experimentally measuring the gradient information, we suggested running isolated forward and adjoint steps, storing the intensities at each phase shifter for each step, and then subtracting this information from the final interference intensity. Alternatively, one may bypass the need to store these constant intensities by introducing a low-frequency modulation on top of one of the two interfering fields in Fig. \ref{fig:TR_schematic}(c), such that the product term of Eq. (\ref{eq:intensity_correct}) can be directly measured from the low-frequency signal.  A similar technique was used in \cite{Annoni2017}.

%% New section

We now discuss some of the limitations of our method.  In the derivation, we had assumed the $\hat{W}$ operator to be unitary, which corresponds to a lossless OIU.  In fact, we note that our procedure is exact in the limit of a lossless, feed-forward, and reciprocal system.  However, with the addition of any amount of uniform loss, $\hat{W}$ is still unitary up to a constant, and our procedure may still be performed with the added step of scaling the measured gradients depending on this loss (see a related discussion in Ref. \cite{Miller2017}).  Uniform loss conditions are satisfied in the OIUs experimentally demonstrated in Refs. \cite{shen2017deep, miller_selfconfiguring_2013}.  Mode-dependent loss, such as asymmetry in the MZI mesh layout or fabrication errors, should be avoided as its presence limits the ability to accurately reconstruct the time-reversed adjoint field.  Nevertheless, our simulation in Fig. \ref{fig:fdfd} indicates that an accurate gradient can be obtained even in the presence of significant mode-dependent loss.  In the experimental structures of  Refs. \cite{shen2017deep, miller_selfconfiguring_2013}, the mode-dependent loss is made much lower due to the choice of the MZI mesh.  Thus we expect our protocol to work in practical systems.  Our method, in principle, computes gradients in parallel and scales in constant time.  In practice, to get this scaling would require careful design of the circuits controlling the OIUs.

Conveniently, since our method does not directly assume any specific model for the linear operations, it may gracefully handle imperfections in the OIUs, such as deviations from perfect 50-50 splits in the MZIs.  Lastly, while we chose to make an explicit distinction between the input ports and the output ports, i.e. we assume no backscattering in the system, this requirement is not strictly necessary. Our formalism can be extended to the full scattering matrix.  However, this would require special treatment for subtracting the backscattering.

The problem of overfitting is one that must be addressed by `regularization' in any practical realization of a neural network.  Photonic ANNs of this class provide a convenient approach to regularization based on `dropout' \cite{srivastava2014dropout}.  In the dropout procedure, certain nodes are probabilistically and temporarily `deleted’ from the network during train time, which has the effect of forcing the network to find alternative paths to solve the problem at hand.  This has a strong regularization effect and has become popular in conventional ANNs.  Dropout may be implemented simply in the photonic ANN by `shutting off’ channels in the activation functions during training.  Specifically, at each time step and for each layer $l$ and element $i$, one may set $f_l(Z_i) = 0$ with some fixed probability.

%% New section

\section{Conclusion}

In this Chapter, we have demonstrated a method for performing backpropagation in an ANN based on a photonic circuit.  This method works by physically propagating the adjoint field and interfering its time-reversed copy with the original field.  The gradient information can then be directly measured out as an \textit{in-situ} intensity measurement. While we chose to demonstrate this procedure in the context of ANNs, it is broadly applicable to any reconfigurable photonic system.  One could imagine this setup being used to tune phased arrays \cite{sun2013large}, optical delivery systems for dielectric laser accelerators \cite{hughes_-chip_2018}, or other systems that rely on large meshes of integrated optical phase shifters.  Furthermore, it may be applied to sensitivity analysis of photonic devices, enabling spatial sensitivity information to be measured as an intensity in the device.

Our work should enhance the appeal of photonic circuits in deep learning applications, allowing for training to happen directly inside the device in an efficient and scalable manner.  Furthermore, this method is broadly applicable to integrated and adaptive optical systems, enabling the possibility for automatic self-configuration and optimization without resorting to brute force gradient computation or model-based methods, which often do not perfectly represent the physical system.


\subsection{P-hacking}

Likely to see some significant results from random chance.


\documentclass[12]{scrartcl}
\usepackage{amssymb,amsmath,gensymb,dsfont,calc,multicol,fullpage}
\makeatletter
\newcommand\Aboxed[1]{
   \@Aboxed#1\ENDDNE}
\def\@Aboxed#1&#2\ENDDNE{%
   &
   \settowidth\@tempdima{$\displaystyle#1{}$}
   \setlength\@tempdima{\@tempdima+\fboxsep+\fboxrule}
   \kern-\@tempdima
   \boxed{#1#2}
}
\makeatother

\begin{document}

\title{Homework 4, Section 1.4: 4, 5, 14, 17, 19, 21, 22, 24, 31, 35}
\author{Alex Gordon}
\date{\today}
\maketitle
\section*{Homework}
\subsection*{4. A)}
$  \begin{bmatrix} 3 \\ 8 \end{bmatrix} $
\subsection*{4. B)}
$  \begin{bmatrix} 3 \\ 8 \end{bmatrix} $
\subsection*{5.}
$  \begin{bmatrix} -4\\1  \end{bmatrix} $
\subsection*{14. A)}
$ x_1  \begin{bmatrix} 2\\0\\1  \end{bmatrix} + x_2 \begin{bmatrix} 5\\1\\2  \end{bmatrix} + x_3 \begin{bmatrix} -1\\-1\\0  \end{bmatrix} = \begin{bmatrix} 4\\-1\\4  \end{bmatrix}$
This, this is a matrix equation.
\\
\\
Converting this to REF gives
\\
\\
$\begin{bmatrix} 2& 0& 4& 9 \\ 0& 1& -1& -1 \\ 0 & 0 & 0 & 3  \end{bmatrix}$
\\
\\
Once this is converted to a linear system we can see that the final row results in $0 = 3$, which is not possible so the system is inconsistent, meaning u does not span the columns of A.
\subsection*{17.}
Since there is no pivot position in the 4th row of A, then by theorem 4, the equation $Ax = b$ does not have a solution for each b in the domain of $R^4$.

This goes on to show that statement D has to be false
\subsection*{19.}
Since statement D is false, then all 4 statements have to be false. Therefore, since not all vectors in $R^4$ can be written as a linear combination of the columns of A.
\subsection*{21.}
The matrix $  \begin{bmatrix} v_1&v_2&v_3  \end{bmatrix} $ does not have a pivot in every row, thus by Theorem 4, $  \begin{bmatrix} v_1&v_2&v_3  \end{bmatrix} $ does not span $R^4$.
\subsection*{22.}
Since the system is consistent  $  \begin{bmatrix} v_1&v_2&v_3  \end{bmatrix} $ does span $R^3$.
\subsection*{24. A)}
 $Ax = b$
Since this is a vector equation and  $Ax = b$, then the matrix equation and the vector equation have the same x values. Therefore, they have the same solution set.
\subsection*{24. B)}
Since the system is consistent, then the x values exists. Therefore, any b can be expressed by $b = a_1x_1 + a_2x_2.....$. This means that b is spanned by the columns of A.
\subsection*{24. C)}
False
\subsection*{24. D)}
False
\subsection*{24. E)}
True
\subsection*{24. F)}
False
\subsection*{31. }
Since a does not have a pivot position in every row, the equation  $Ax = b$ cannot be consistent for all b in $R^M$
\subsection*{35.}
Suppose that y and z satisfy  $Ay = z$. This means that $4z = 4Ay$. Since that fits the definition of a scalar in an $mn$ matrix, then if we replace the constant scalar we get $4(Ay) = A(4y)$. This shows that 4y is a solution, thus making it consistent.

\end{document}


  \subsection{Increasing house price risk decreases risky share}

  % \figName allows generic definition of hypertargets based on title of figure
  \renewcommand{\figName}{shareFuncByRisk}
  \renewcommand{\figFile}{\figName}
  \input{\FigDir/\figName} % Read in the tex to generate the figure

  The volatility of the housing market can have strong implications for the portfolio decisions of households who own their houses. A higher standard deviation in house prices implies a larger implicit holding of a risky asset (the house), regardless of house size. For this reason, households would optimally choose to avoid risk in other risky assets. As we see in  figure~\ref{fig:shareFuncByRisk} for 2 households who have a house of equal size, the risky share of a household in a more volatile market is lower than that of a household in a less volatile market, except at low levels of market resources. Households that experience high price volatility in the housing market reduce their exposure to risk elsewhere, leading to lower risky portfolio shares.


% !TEX root=../main.tex
\chapter{Background}
\renewcommand{\baselinestretch}{\mystretch}
%\setlength{\parindent}{0pt}
\PARstart{T}{his} chapter is a summary of previous research based on audio data. Three main aspects of work were focused on where the performance has been improved. Firstly, different feature extract methods based on audio data are compared in previous studies. Two classical representation will be introduced, namely STFT and MFCCs. Afterwards, statistical learning and deep neuronal network models applied to audio data mining are compared in Section 2.2. Finally, studies of data pre-processing methods including noise reduction and augmentation are discussed.
\section{Audio Feature Extraction}
When Welch~\cite{welch1967use} proposed a fast Fourier transform algorithm, namely short-time Fourier transform (STFT), efficient estimation of power spectra could be used on record analysis. He suggested segmental processing of the whole record and moving the window with possible overlapping, which can not only modify periodogram by averaging but also reduce computational complexity. Afterwards, Davis and Mermelstein~\cite{davis1980comparison} introduced Mel-frequency cepstral coefficients (MFCC) based on STFT. Due to the linear characteristic of Fourier transform, MFCCs are perceptual features which mimic received signal by the cochlea, aiming to enhance signal with significant contribution in phonetic detection. They firstly applied triangular mel-filters to scale spectrogram, then taking logarithm and discrete cosine transform (DCT) to calculate significant cepstral coefficients. \par
Lippends et al.~\cite{lippens2004comparison} compared performances of an auditory model and MFCC features in music genre classification task. They extracted 4 features from STFT and first 5 Mel-frequency cepstral coefficients classified by Gaussian Mixture Model (GMM), which shows that the auditory model has no outstanding accuracy than standard MFCC. Moreover, 12 identified MFCCs are used on recognition of anuran species~\cite{bedoya2014automatic}, achieving average accuracy in 99.61\% of all six species. However, MFCCs are not robust for the noisy environment, resulting in utilizing mel-spectrogram as representations. Choi et al.~\cite{choi2016automatic} compared STFT, MFCC and Mel-spectrogram (with accuracy 84.6\%, 86.2\% and 89.4\% respectively) using four layers of fully convolutional neural networks (FCN) on automatic tagging MagnaTagATune dataset. Similar work was studied in~\cite{huzaifah2017comparison} with additional fast
wavelet transform (FWT) and continuous wavelet transform
(CWT) as comparisons. He proved that mel-spectrogram still achieved significant performance on UrbanSound8K dataset among variations of settings, although STFT and CWT had some outstanding results on some models.\par
MFCC and mel-spectrogram are broadly used on automated species detection as well. Clemin et al.~\cite{clemins2003application} studied in recognizing African elephants sound based on GMM in 2003, resulting accuracies of 83.8\% for call and 88.1\% for identification.
Feature comparison of log-magnitude spectrogram using CNN and log-mel band energy using CRNN were studied in \cite{cakir2017convolutional}, improving the accuracy in 3\%. They stated that log-mel band energy makes bird sound more distinguishable. Another application on insect sound recognition was studied by~\cite{le2011insect}, using Probabilistic Neural Network (PNN) based on Bayesian classifiers. Although the identification accuracy is over 92\%, the method of selecting noise-free sections still introduced limitations in generalisation.

In summary, approaches of feature extraction significantly determine the performance of audio detection problems. By transforming time-series into cepstrum coefficients or spectrogram, the feature dimensions are reduced to a large extent. MFCCs present distinction on compressive representations which uses 12-20 coefficients as features~\cite{lippens2004comparison,le2011insect,clemins2003application}. Hence, it is beneficial to the linear statistical model, such as hidden Markov models and Gaussian Mixture Model. Mel-spectrogram is still an alternative with the advantage of noisy robustness. Therefore, mel-spectrogram is a suitable representation for deep learning, containing more distinguishable features.
\section{Automated Recognition Models}
In the past decade, hidden Markov models (HMM) and Gaussian Mixture model (GMM) were based on mathematical statistics. HMM aims to use observable data to predict the hidden feature based on the Markov process, for example predict event label with giving time-series signal. The property of the Markov process is that hidden variables of current time $s(t)$ only depends on previous time $s(t-1)$. Therefore, this property determined the suitability of speech recognition~\cite{rabiner1989tutorial} whose signals have relevance among context. Research based on HMM in antbirds recognition was studied by Vlad et al.~\cite{trifa2008automated}, who used optional MFCC features including energy, delta coefficients, acceleration. They found that varying estimated states from 5 to 15 could degrade performance dropped from 94.6\% to 82.5\%.\par
GMM is another general model since most of data in real life are subject to Gaussian distribution. Thus, using GMM by setting several mixtures can estimate the data distribution. Data can be considered as same label if they have a similar distribution. Brown \& Smaragdis~\cite{brown2009hidden} compared both GMM and HMM model on marine call types classification. The results proved that both these statistical models are powerful for acoustic classification of killer whales.
Zhao~\cite{zhao2017automated} used a sophisticated method of modelling log-band energy in order to automatically segment activities. The GMM-based model with two mixtures can capture both high-energy frames and low-energy frames, selecting the cross point as the threshold. Therefore, discriminative features of bird were selected, resulting in improved robustness of SVM to classify.

Both HMM and GMM are based on expectation maximum algorithm to estimate the probability of unknown parameters. Nevertheless, HMM takes temporal context into account, whereas GMM treats the whole signal as a unique spectral individual~\cite{brown2009hidden}. Comparing studies in~\cite{clemins2003application,trifa2008automated,zhao2017automated}, MFCCs were often used as features of GMM and HMM model. The reason may be that MFCCs largely compress the feature representations which make the statistical model easy to train. However, the mel-spectrogram can be consider as an effective alternative when using powerful models like CNN.

In recent years, deep learning has been rapidly developed on audio recognition since it can learn more effective features from input. Since VGGNet has outstanding performance on Large Vocabulary Continuous Speech Recognition (LVCSR)~\cite{sercu2016very}, convolutional neural network successfully applied on Acoustic Event Detection (AED) 80.3\% with accuracy~\cite{takahashi2016deep}.
Moreover, Cakir et al.~\cite{cakir2017convolutional} proposed that using converlutional recurrent neural network (CRNN) with log-mel band energy as features improved the accuracy up to 88.5\% on bird event detection. Continue to bird detection, Grill et al.~\cite{8081512} designed two types of CNN architectures to detect the presence of bird call. General architecture designed with each convolutional layer following a pooling layer and using three dense layers, while local architecture used convolutional layers instead of dense layer transforming into a one-dimensional sequence and selected the maximum. Table \ref{tabel:summarymodel} makes a summary of these researches with different models and feature extraction methods. The types of recognition tasks and overall accuracy are recorded as well.
\begin{table}[htp]
\centering
\caption{Summary of models and feature extraction methods on species detection}
\label{tabel:summarymodel}
\begin{tabular}{ccccc}
\hline
Model& Species& Task& Feature& Accuracy(\%)\\
\hline
GMM& Elephant (2003)& Detection&MFCC& 83.8~\cite{clemins2003application}\\
HMM& Antbird (2008)& Detection&MFCC& 94.6~\cite{trifa2008automated}\\
GMM+HMM& Whales (2009)&Classification& MFCC & 92~\cite{brown2009hidden}\\
GMM+SVM& Bird (2017)& Classification& MFCC& $>$ 90~\cite{zhao2017automated}\\
PNN& Insect (2011)& Identification& MFCC& 92.44~\cite{le2011insect}\\
CNN& Bird (2017)&Detection& Log-spectrogram& 85.5~\cite{cakir2017convolutional}\\
CRNN& Bird (2017)& Detection& Log-mel energy
& 88.5 \cite{cakir2017convolutional}\\
CNN& Bat (2018)& Detection& STFT
& 89.6 \cite{batdetect18}\\
\hline
\end{tabular}
\end{table}
\section{Processing on Audio data}
For obtaining higher performance for species detection, several data processing method can be applied in order to improve the feature representation instead of inputting raw data. There are two main method that significant improve the performance of CNN model, specifically noise reduction and data augmentation. Noise reduction can enhance the region of interests (ROIs), making the call more distinct. Augmentation can increase the variation for training if only small number of data can be collected.

Generally, the acoustic recording is unavoidably introduced disturbance by the high-level noise, which related to the habitats that recording devices installed. Thus, environmental sounds have significant effects on the estimation of species' activity, introducing detection bias. As the denoising method implemented in~\cite{aide2013real,batdetect18}, regions of interests were enhanced with reducing the noise level. They applied spectral subtraction method which removed the mean amplitude in each frequency band as shown in \ref{fig:batnoise}. The level of background noise was suppressed to a large extent and removing the additive noise. Thus, the CNN model can effectively classify the foreground echo and the background noise with a high output probability of the presence of call.
\begin{figure}[htp]
\centering
\includegraphics[scale=0.8]{Figs/chap2/batnoise.png}
\caption[Noise reduction of bat call]{Noise reduction of bat call in spectrogram representation~\cite{batdetect18}}
\label{fig:batnoise}
\end{figure}

Minimum mean-square error log-spectral amplitude (MMSE-LSA) estimator is another effective algorithm, which has been extensively implemented in automatic speech recognition (ASR) systems. Kim \& Rose~\cite{kim2002cepstrum} proposed a way of combining model based on MMSE-LSA to decompose noise and speech signal in the cepstral domain. Via combining MMSE-LSA technique with HMM model, the robustness of MFCCs is significantly increased. The MMSE-LSA estimator highly depends on a prior signal to noise ratio (SNR), which is estimated in the presence of non-speech region. Hence, \textit{a prior} SNR estimator based on CNN is introduced by \cite{nicolson2019deep}. The experiments by using deep learning showed the quality and intelligibility scores are much better than masking and mapping methods. However, there are fewer studies applied the MMSE-LSA for species detection tasks. In this project, I applied MMSE-LSA on 'whinny' calls to evaluate the performance.

Augmenting data is a necessary approach for model training especially on a small dataset. Takahashi et al.~\cite{takahashi2016deep} introduced a new augmentation method for audio event classification since the number of audio in each class is small. It is called Equalized Mixture Data Augmentation (EMDA) which mixes two different audio segments in the same class to synthesize new data. Moreover, he further modified the frequency characteristics by enhancing or disturbing a particular frequency band. By using augmentation, the accuracy was improved in 10\% compared with the original dataset. Other augmentation methods are compared in \cite{sprengel2016audio} as well for bird sounds identification, namely adding white noise, pitch shift, time shift and combining same class. He proved that time-shift took a large impact on performance, resulting in accuracy decreasing 3\% without applying time-shift.


\documentclass[12pt,a4paper]{article}
\usepackage{bussproofs}
\usepackage{amsmath}

\title{Proof Trees}
\author{Steven Libby}
\date{\vspace{-5ex}}

% these are several symbols that I redefined to make formulas easier to read
\def\land{\wedge}           % /\
\def\lor{\vee}              % \/
\def\lnot{\neg}             % -
\def\liff{\leftrightarrow}  % <->
\def\xor{\oplus}            % (+)
\def\T{\top}                % T
\def\F{\bot}                % _|_
\def\proves{\vdash}         % |-

% these are commands that use the bussproofs package.
% they let you write tree style proofs.
% you can get the package here:  https://www.math.ucsd.edu/~sbuss/ResearchWeb/bussproofs/bussproofs.sty

\newcommand{\premise}    [1]{\AxiomC{#1}}
\newcommand{\assumption} [1]{\AxiomC{[#1]}}
\newcommand{\andI}       [1]{\RightLabel{$\land I$} \BinaryInfC{#1} }
\newcommand{\andEL}      [1]{\RightLabel{$\land E1$}\UnaryInfC{#1}  }
\newcommand{\andER}      [1]{\RightLabel{$\land E2$}\UnaryInfC{#1}  }
\newcommand{\orIL}       [1]{\RightLabel{$\lor I1$} \UnaryInfC{#1}  }
\newcommand{\orIR}       [1]{\RightLabel{$\lor I2$} \UnaryInfC{#1}  }
\newcommand{\orE}        [1]{\RightLabel{$\lor E$}  \TrinaryInfC{#1}}
\newcommand{\arrowI}     [1]{\RightLabel{$\to I$}   \BinaryInfC{#1} }
\newcommand{\arrowE}     [1]{\RightLabel{$\to E$}   \BinaryInfC{#1} }
\newcommand{\notE}       [1]{\RightLabel{$\lnot E$} \BinaryInfC{#1} }
\newcommand{\notI}       [1]{\RightLabel{$\lnot I$} \UnaryInfC{#1}  }
\newcommand{\FE}         [1]{\RightLabel{$\F E$}    \UnaryInfC{#1}  }
\newcommand{\TI}         [1]{\RightLabel{$\T I$}    \UnaryInfC{#1}  }
\newcommand{\LEM}        [1]{\RightLabel{$LEM$}     \UnaryInfC{#1}  }

\begin{document}
\maketitle

This is meant to be a more complete explanation of proof trees.
Proof Trees are a graphical format for formal proofs that eliminate the need to worry about getting references correct.
I'll introduce the concept with several examples.

The basic idea of a proof tree is pretty straightforward.
If you can match all of the premises, then you can derive the conclusion.
This leads to structures that look like the following.
\begin{prooftree}
    \premise{A}
    \UnaryInfC{B}

    \premise{C}
    \premise{D}
    \BinaryInfC{E}

    \premise{A}
    \UnaryInfC{B}
    \premise{A}
    \BinaryInfC{B}

    \TrinaryInfC{F}
\end{prooftree}

Before we go any further, I need to explain a few terms.
A \textbf{premise} is the expression above the bar in an inference rule.
It is either something we were given, something we assumed, or something we derived.
A \textbf{conclusion} is the expression below the bar in an inference rule.
It was derived from premises.
A \textbf{proof tree} is a tree made out of inference rules.
The \textbf{leaves} of the proof tree are anything that isn't under a bar.
A proof tree is \textbf{valid} if all of the leaves are given or assumed, and the premises of inference rules match the definition of the inference rules.
Finally, a formula $A$ \textbf{matches} another formula $B$, if we can replace the variables in $B$ in such a way that $A = B$.

\pagebreak
\noindent
Let's look at an example.
Given the rule
\begin{prooftree}
  \premise{$A$}
  \premise{$A \to B$}
  \arrowE{$B$}
\end{prooftree}
prove:
$A, A \to B, B \to C \proves C$.\\
We can do this with the following proof
\begin{prooftree}
  \premise{$A$}
  \premise{$A \to B$}
  \arrowE{$B$}
  \premise{$B \to C$}
  \arrowE{$C$}
\end{prooftree}

Here $A$, $A \to B$, and $B \to C$ are \textit{premises},
$C$ is the \textit{conclusion},
The entire structure is the \textit{proof tree}, and $A$, $A \to B$, and $B \to C$ are \textit{leaves} in the tree
(notice that there's no bar above them).
It's clear that the upper left part of the proof tree \textit{matched} our $\to E$ rule.
It's not as clear, however, that the lower use of $\to E$ matched.\\

\noindent
Matching is very important here, so I'll go over it with some examples.
The simplest example is that any formula matches itself.\\
$A \text{ matches } A$.\\

\noindent
Also any formula can be matched by a variable.\\
$A \text{ matches } V$.\\
$A \land B \text{ matches } V$.\\
$A \to (B \land C) \text{ matches } V$.\\

\noindent
And finally, any formula $A$ matches a formula $B$ if their top level operators are the same, and their subformulas match.
Again, this is easier to see by example.\\
$(A \land B) \to C \text{ matches } D \to C$\\
$A \land B \text{ matches } C \land D$\\
$B \land A \text{ matches } A \land B$\\
$(A \land B) \lor (C \land D) \text{ matches } (A \land B) \lor C$\\

\noindent
It's also important to know when a formula \textit{doesn't} match another one.\\
$A \lor B \text{ doesn't match } A \land B$, because the $\land$ doesn't match $\lor$.\\
$(A \to B) \to C \text{ doesn't match } A \to (B \to C)$.  \\
While $A\to B \text{ matches } A$, $C \text{ doesn't match } B \to C$.\\

\noindent
Wrapping this all up: since we only used the $\to E$ rule,
all of our leaves were premises, and we matched the definition for every rule, our proof tree is \textit{valid}.

\section{Proof Rules}
Now that we have definitions, we need the inference rules.
These are the rules I've been using for this class, although the ideas work with any set of inference rules.

There are two types of rules here:
Introduction rules, which add an operator (a symbol) into the formula,
and Elimination rules, which take a formula with an operator and remove the operator.
Introduction rules introduce something, and elimination rules eliminate something.\\

\begin{tabular}{|ccc|}
  \hline

 & & \\
  \premise{$A \land B$}
  \andEL{$A$}
  \DisplayProof &

  \premise{$A \land B$}
  \andER{$B$}
  \DisplayProof &

  \premise{$A$}
  \premise{$B$}
  \andI{$A \land B$}
  \DisplayProof \\

 & & \\

  \premise{$A$}
  \orIL{$A \lor B$}
  \DisplayProof &

  \premise{$B$}
  \orIR{$A \lor B$}
  \DisplayProof &

  \premise{$A \lor B$}
  \premise{$A \to C$}
  \premise{$B \to C$}
  \orE{$C$}
  \DisplayProof \\

 & & \\

  \premise{$[A]$}
  \premise{$[A]$}
  \noLine
  \UnaryInfC{\vdots}
  \noLine
  \UnaryInfC{$B$}
  \arrowI{$A \to B$}
  \DisplayProof &

  &

  \premise{$A$}
  \premise{$A \to B$}
  \arrowE{$B$}
  \DisplayProof \\

 & & \\

  \premise{$A \to \F$}
  \notI{$\lnot A$}
  \DisplayProof &

  \premise{$A$}
  \premise{$\lnot A$}
  \notE{$\F$}
  \DisplayProof & \\

 & & \\

  \premise{$\ $}
  \TI{$\T$}
  \DisplayProof &

  \premise{$\F$}
  \FE{$A$}
  \DisplayProof &

  \premise{$\ $}
  \LEM{$A \lor \lnot A$}
  \DisplayProof \\
 & & \\
  \hline
\end{tabular}
\\

It is worth convincing yourself that these rules are correct.
It's also worth finding the informal proof strategies that correcpond to these rules.
For example: $\lnot I$ is a proof by contradiction.  We assumed $A$, and proved $\F$, so we know that our assumption of $A$ was wrong.

\section{Proofs}
A proof in propositional logic is a sequence of steps, where each step is an inference rule using conclusions from the previous steps.
In the normal 3-column proofs, this means that you have to keep track of what line on which you reached a conclusion.
Tree proofs eliminate this reference tracking.

Now we'll work through a few simple proofs.
I'll explain how the inference rules are used,
why they're valid proofs,
and some general strategies for coming up with proofs.
Unfortunately, this is not an algorithm for coming up with proofs.
It can still be tricky to figure out which rules will be the most useful to check.

\section{And Introduction and Elimination}
We'll start off with one of the simpler examples of a proof,
the commutativity of $\land$.
The first thing to notice here is that we always have a \textit{goal} that we're trying to prove.
Here the goal is $B \land A$, and it's good to keep this in mind.
In fact, before we even start, we can write $B \land A$ at the bottom of the proof, since we know we have to prove it.

\begin{prooftree}
  \premise{$?$}
  \premise{$?$}
  \BinaryInfC{$B \land A$}
\end{prooftree}

At this point we could use one of five rules to get to $B \land A$.  $\land I, \land E1, \land E2, \lor E, \to E, \text{and} \F E$.
We can not use any of the others, because their conclusions don't \textit{match} $B \land A$.
We'll start with $\land I$, because that one looks promising.
I know this because I've already done the proof.  Deciding which rules are good rules to check comes with practice.

\begin{prooftree}
  \premise{$B$}
  \premise{$A$}
  \andI{$B \land A$}
\end{prooftree}

Now, we're actually ready to finish the proof.
It turns out that we have everything we need.
All that's left is to get $A$ and $B$ from $A \land B$.
We can do this with $\land E$.

\begin{prooftree}
  \premise{$A \land B$}
  \andER{$B$}
  \premise{$A \land B$}
  \andEL{$A$}
  \andI{$B \land A$}
\end{prooftree}

In this proof we used a couple of strategies that are good to keep in mind.\\
\textbf{destruction-construction:} it's often the case that we can pull apart a formula into component parts with elimination rules.
Then we can construct our new formula with introduction rules.\\
\textbf{working backwards:} Since we know what the end result should be, it's often easier to start with that, and find which rules can apply.\\
\textbf{eliminating bad paths:} Finding a proof for a theorem is a lot like solving a puzzle.  We need to find where all of the pieces fit.
We can also get a lot of information by ruling out several pieces from our search.

\section{Arrow Introduction}
Arrow introduction is usually the rule people have problems with.
It's really not that hard, but you have to be okay with assumptions.
Assumptions work just like premises, but you can only use them in the arrow introduction branch.
That is, if you have,

\begin{prooftree}
  \assumption{$A$}
  \premise{\vdots}
  \noLine
  \UnaryInfC{$B$}
  \arrowI{$A \to B$}
\end{prooftree}

Then you can use the $[A]$ anywhere above the $\to I$ bar.
At this point $[A]$ works just like a premise.
Also note that $[A]$ is the only thing on the left hand side.
This is just marking that $A$ is the variable that we're assuming to be true.
To show how we can use this rule, let's revisit the proof we did earlier.\\
$A, A \to B, B \to C \proves C$.\\
However, let's make this an implication.\\
$A \to B, B \to C \proves A \to C$.\\

Since, we want to prove $A \to C$, then $\to I$ is probably our best choice.

\begin{prooftree}
  \assumption{$A$}
  \premise{$?$}
  \arrowI{$A \to C$}
\end{prooftree}

Now that we have assumed $A$, we can actually complete the proof just like we did before.
We can use $A$ just like any other premise.
We just need the extra step of assume $A$, and derive $C$.

\begin{prooftree}
  \assumption{$A$}

    \assumption{$A$}
    \premise{$A \to B$}
    \arrowE{$B$}

    \premise{$B \to C$}
    \arrowE{$C$}

  \arrowI{$A \to C$}
\end{prooftree}

\pagebreak
\section{Or Elimination}

The Or Elimination rule can be a little bit tricky.
The rule itself says.
\begin{prooftree}
  \premise{$A \lor B$}
  \premise{$A \to C$}
  \premise{$B \to C$}
  \orE{$C$}
\end{prooftree}

Where did $C$ come from?
What does it mean?
Well, Or Elimination is actually a common proof technique.
It's proof by cases.\\
If we have $A \lor B$, then we have two cases.\\
Case 1: $A$ is true, so we show $A \to C$.\\
Case 2: $B$ is true, so we show $B \to C$.\\
Now, no matter which case is true, we know that we can conclude $C$.

We use this to split up our proof into cases.
We can use this on the following theorem.
$A \lor B, B \to C, (A \land D) \to B, D \proves C$.

We know immediately that our conclusion is going to have to be $C$, so we can start with that.
We also already have $B \to C$, and $A \lor B$, so proof by cases looks like a good idea.

\begin{prooftree}
  \premise{$A \lor B$}

  \premise{$?$}
  \UnaryInfC{$A \to C$}
  \premise{$B \to C$}

  \orE{$C$}
\end{prooftree}

Now we need to fill in the $A \to C$ branch.
Remember, to get something of the form $A \to C$, we have to assume $A$, and prove $C$.

\begin{prooftree}
  \premise{$A \lor B$}

  \assumption{$A$}
  \premise{$?$}
  \arrowI{$A \to C$}
  \premise{$B \to C$}

  \orE{$C$}
\end{prooftree}

\pagebreak
We don't have anything in our premises that lets us go from $A$ to $C$, so let's look for another route.
We have a $B \to C$.  That look promising; let's try that.

\begin{prooftree}
  \premise{$A \lor B$}

  \assumption{$A$}
  \premise{$?$}
  \UnaryInfC{$B$}
  \premise{$B \to C$}
  \arrowE{$C$}
  \arrowI{$A \to C$}

  \premise{$B \to C$}

  \orE{$C$}
\end{prooftree}

We're almost done here.  We just need to prove $B$.
This isn't too hard, since we have $A, D,$ and $A \land D \to B$.

\begin{prooftree}
  \premise{$A \lor B$}

  \assumption{$A$}

  \assumption{$A$}
  \premise{$D$}
  \andI{$A\land D$}
  \premise{$(A \land D) \to B$}
  \arrowE{$B$}

  \premise{$B \to C$}
  \arrowE{$C$}
  \arrowI{$A \to C$}

  \premise{$B \to C$}

  \orE{$C$}
\end{prooftree}

You'll notice that proof by cases can grow pretty large.
They also tend to be uncommon outside of contrived situations.
The rules for $\lor I$ introduce a new variable (or expression) that is unrelated to the premise.
This is unlikely to be useful in a proof unless you can introduce a variable related to other premises.
However, proof by cases is very useful for dealing with the law of the excluded middle.

\section{Law of the Excluded Middle}

This is an odd rule to get a feel for.
It can be used to prove some odd results, and it seems like we get an expression coming out of nowhere.
LEM proofs also tend to be some of the trickiest proofs to come up with.
I'd recommend trying this if you can't find another way to prove the result.
It has no premise, and the $A$ in the conclusion can be anything, including $\F$.
The reason that this ends up being consistent is because we actually get a pretty weak result.

\begin{prooftree}
  \premise{$\ $}
  \LEM{$A \lor \lnot A$}
\end{prooftree}

Suppose we tried having the $A$ be $\F$.  Then we would have $\F \lor \lnot \F$, and $\lnot \F$ is $\T$, so it's still fine.
In order to use this rule at all, we need to do a proof by cases.
Let's try the following proof that double negation cancels out.

$\lnot \lnot A \proves A$.

This can be a tricky proof.
It would be entirely reasonable to try to prove this with $\lnot E$ or $\F E$.
However, you'll end up stuck trying to prove it either of these ways.
So let's try a LEM proof.
We know we have to use proof by cases with LEM, so let's get that set up.

\begin{prooftree}

  % LEM branch
  \premise{$\ $}
  \LEM{$A \lor \lnot A$}

  % A -> A branch
  \assumption{$A$}
  \premise{$?$}
  \arrowI{$A \to A$}

  % start !A -> A branch
  \assumption{$\lnot A$}
  \premise{$?$}
  \arrowI{$\lnot A \to A$}

  \orE{$A$}
\end{prooftree}

Ok, so one of these branches is going to be pretty easy to fill in.
We assumed $A$, so we can use that anywhere in that subproof.

\begin{prooftree}

  % LEM branch
  \premise{$\ $}
  \LEM{$A \lor \lnot A$}

  % A -> A branch
  \assumption{$A$}
  \assumption{$A$}
  \arrowI{$A \to A$}

  % start !A -> A branch
  \assumption{$\lnot A$}
  \premise{$?$}
  \arrowI{$\lnot A \to A$}

  \orE{$A$}
\end{prooftree}

The rest of this isn't as clear, but it's not too bad.
Remember we have $\lnot \lnot A$ as a premise, and we just assumed $\lnot A$.
Or, to put that another way, we have $(\lnot A)$ and $\lnot (\lnot A)$.
At this point we can use proof by contradiction ($\lnot E$) to finish the proof.

\begin{prooftree}

  % LEM branch
  \premise{$\ $}
  \LEM{$A \lor \lnot A$}

  % A -> A branch
  \assumption{$A$}
    \assumption{$A$}
  \arrowI{$A \to A$}

  % start !A -> A branch
  \assumption{$\lnot A$}

    \assumption{$\lnot A$}
    \premise{$\lnot \lnot A$}
    \notE{$\F$}

    \FE{$A$}

  \arrowI{$\lnot A \to A$}

  \orE{$A$}
\end{prooftree}

\end{document}


\input{mixed/fig/tikzsettings}
\maketitle

\section*{Preface}
%\input{../preface}

These notes compile information from various sources, which should be
consulted for more details on particular subjects. These are in
particular the books by Daniele Boffi, Franco Brezzi, and Michel
Fortin~\cite{BoffiBrezziFortin13} and by Alexandre Ern and Jean-Luc
Guermond~\cite{ErnGuermond04} on the general theory of mixed finite
elements, the books by Dietrich Braess~\cite{Braess97,Braess13} and
Philippe Ciarlet~\cite{Ciarlet88} on elasticity, the book by Vivette
Girault and Pièrre-Arnaud Raviart~\cite{GiraultRaviart86} on
incompressible fluids, the book by Peter Monk~\cite{Monk03}, and the
works by Douglas Arnold, Richard Falk, and Ragnar
Winther~\cite{ArnoldFalkWinther06acta,ArnoldFalkWinther10} on the
finite element exterior calculus.

Special thanks go to Dr.~Daniel Arndt for contributing problems,
solutions, and improving the notes at many points. Furthermore, I
thank Vladislav Olkhovskiy and Saurabh Mehta for valuable corrections
and additions.
\thispagestyle{empty}
\setcounter{page}{0}

\tableofcontents

\chapter{Introduction: from elliptic to mixed problems}
We begin our course of mixed finite element methods by studying a
vector-valued elliptic problem. Then, we study its dependence on its
parameters and naturally arrive at a mixed formulation. We derive a
few properties of mixed systems and then turn our attention to the
first example: the Stokes equations of incompressible flow. We close
the chapter by considering mixed systems as first order conditions
for constrained minimization problems.

\input{mixed/mixedintro}

\section{Abstract saddle-point systems}
\input{mixed/abstract-saddle}

\section{Relation to constrained minimization}
\input{mixed/lagrange}

\section{Additional problems and experiments}

\begin{Problem}{elasticity-locking-convergence}
  The graph for $\lambda=100$ in \cref{fig:elasticity-locking}
  suggests the following behavior: on very coarse meshes, the solution
  is badly approximated. Eventually, the approximation cathes up and
  is even faster than orer $h^{-1}$ until it approximates
  asymptotically the straight line for $\lambda=1$. The curve for
  $\lambda=1000$ seems to show the beginning of this behavior.

  \begin{enumerate}
  \item Compute on finer meshes to see if this is actually true. If
    such a behavior is confirmed, this implies that the part of the
    bilinear form with $\lambda$ is less important on finer meshes,
    something which cannot be covered by Céa's lemma from the elliptic
    toolbox.
  \item Compute the energy norm of the error as in the symmetric
    finite element toolbox. What do those curves look like? There, the
    constant in Céa's lemma is 1.
  \item Can we draw any conclusions on the best approximation in the
    energy norm?
  \end{enumerate}
\end{Problem}

\chapter{Conditions for well-posedness}
\label{sec:mixed-wellposedness}
In this chapter, we will first modify conditions for well-posedness in
finite dimensions from positive definiteness to the general case. In
particular, we will derive a quantitative formulation, which we will
study for infinite dimensional problems in the second section. In the
third section, we derive the inf-sup condition for mixed problems as a
special case.
\input{mixed/infsup}

\chapter{The Stokes problem}
\label{cha:stokes}
\input{mixed/stokes}

\chapter{Mixed formulation of elliptic problems}
\label{cha:darcy}
\input{mixed/darcy}

\chapter{Maxwell's equations and the de Rham complex}
\section{Maxwell's equations}
\label{cha:maxwell}
\input{mixed/maxwell}

\section{The de Rham complex}
\label{cha:derham}
\input{mixed/derham}

\chapter{Divergence conforming discontinuous Galerkin methods}
\label{cha:hdivdg}
\input{mixed/hdivdg}

% \chapter{Almost incompressible elasticity}
% \section{Augmented saddle point problems}
% \input{augmented}
% \section{Discretization}
% \input{lame-navier-discrete}

\appendix
\chapter{Appendix}
\section{Notation}

\input{mixed/notation}

\section{Useful tools}
\input{mixed/tools}

\section{Derivation of the Lamé-Navier equations}
\label{sec:lame-navier}
\input{mixed/lame-navier}

\bibliographystyle{apalike}
\bibliography{all}
\printindex

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End: