\section{Audio Impedance Measurement}
\label{sect:ZMeas}
%
Audio Vector Network Analysis (AVNA) measurements break into two main types,  reflection and transmission.  This section covers the first case, reflection.  The major use of reflection measurements is to determine the impedance
which creates the reflection, or, put another way, a quantity directly associated with impedance, such as inductance. This section covers AVNA impedance measurements. The next section covers transmission measurements.

The term "Vector" refers to the fact that we always measure both the amplitude and the relative phase of impedance.  This enables us to measure a combination of resistive and reactive components.
%
\subsection{Description}
\label{subsect:ZDescr}
Before getting to the details of the measurements, we need to explore a few background items.

\subsection{Complex Impedance}Using the AVNA1 impedance \q{T} terminals , the basic measurement quantity that is often displayed is impedance described by a resistance in series with a reactance. The reactance can be either a lossless inductance or a capacitance. Mathematically, this is represented as a complex number $\vec{Z}$=R+jX where R is the series resistance and X is the series reactance, both in Ohms. The AVNA1 uses complex arithmetic notation for $\vec{Z}$ which allows mathematically sound computation to be done on circuit models; the user can think of it as a way to keep the resistance separate from the reactance.

If the reactance is positive, the series component represents an inductor. Likewise, if the reactance is negative, the series component is a capacitor.  Note that this is only one of many representations. As an example, a series configuration can be converted to a parallel pair of components, neither being the same value as the series pair.  We will not attempt to cover all the possibilities here, but a few of them will be described below, as they appear in the display.

\subsection{Reference Resistance}Two different reference resistances can be used with the AVNA, 50 and 5,000 Ohms. The best value to use is generally the one that is in the range of the measured values. It is easy to pick 50 Ohms if the measured value is, say, 10 + j100. If the series reactance is much larger than the series resistance, we have a high-Q component that is not the easiest to measure with an AVNA. For that case we are probably going to uses the reference resistance that is closest to \(X\) since that determines the component value. If the user is trying to measure a resistance value, it is best to pick a reference resistance closest to the resistor value.

A more elaborate procedure for finding the best reference resistance for a general impedance type is to first measure with 50 Ohms. Then find the geometric mean of the resistance and the reactance, that is \(\sqrt{R X}\). Then remeasure with 5,000 Ohm reference. Choose the ratio of the geometric mean to the reference resistance (50 or 5,000) which is closest to 1.0.

The best reference resistance decision is calculated by ratio, not by subtraction. For instance 4,000 Ohms has a ratio with 50 Ohms of  \( \frac{4000}{50} = 80\) whereas the ratio with 5,000 Ohms is \( \frac{5000}{4000} = 1.25\). The latter is closer to 1.0 than the former.  Note that we put the larger value on top of the ratio for convenience in comparing the two results.
%
\subsection{Instructions}
\label{subsect:ZInstr}
\textbf{Getting Started - }Here is a step-by-step description of a single frequency impedance measurement followed by a description of a swept impedance measurement. This will all be done directly from the touch screen. Everything covered here, and more, can be done via the USB-Serial link, as is described in the "AVNA1 Serial Interface" section.

When the AVNA1 is powered up, you have a choice of four Audio Test Instruments along with Service and Calibration functions.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_000.pdf}
\caption{Audio Test Instrument Home  Screen.}
\label{AVNA_000-label}
\end{center}
\end{figure}
%
Referring to Figure \ref{AVNA_000-label}, we are now at the main AVNA1 Audio Test Instrument home screen.  For our impedance measurements, we will select the menu item, "\textsf{AVNA}" by touching that box at the bottom of the screen leading to the AVNA main screen.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_001.pdf}
\caption{AVNA Main  Screen.}
\label{AVNA_001-label}
\end{center}
\end{figure}
%
We now have the choice  of single frequency or swept measurements as well as changing of the Reference Resistance described above. A last choice is the "\textsf{What?}" function for exploring components without knowing much about them. To continue our impedance measurement, we select "\textsf{Set Ref R}".
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_003.pdf}
\caption{AVNA reference resistor selection  Screen.}
\label{AVNA_003-label}
\end{center}
\end{figure}
%
The current reference, as described above, is shown on the screen. It is also shown, during AVNA measurements, at the top of the screen in small type. Selecting the current value has no effect, and so in this example we choose the menu item, "\textsf{5k Ohms}".
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_004.pdf}
\caption{AVNA reference resistor selection of 5000 (5K) Ohms.}
\label{AVNA_004-label}
\end{center}
\end{figure}
At this point we have changed the reference resistance and can return to the AVNA main screen, Figure \ref{AVNA_001-label}, by tapping the menu item, "\textsf{Back}".

\textbf{Measuring an Impedance - }As an example, suppose we put together a series combination of a 10 Ohm resistor and a 0.22 $\mu$F capacitor and put this across the \q{Z} terminals. To get started, we can measure this at a single frequency by tapping on the menu item, "\textsf{Single Freq}". The frequency can be selected from a list of 13 ranging from 10 Hz to 40 kHz by using the menu items, "\textsf{Freq Down}" and "\textsf{Freq Up}".  For this sample measurement set the frequency to "1000 Hz".  Next, the menu item, "\textsf{Meas Z}" is tapped to start a continuing series of impedance measurement, as shown next.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_006.pdf}
\caption{Impedance measurement of a series combination of 10 Ohms and a 0.22 $\mu$F capacitor.}
\label{AVNA_006-label}
\end{center}
\end{figure}
A couple of items to note:  First, this measurement repeats every second or so.
This is useful if you are measuring multiple components, or if you want to do mental averaging of the component values.

Next, we see the series combination of  $\vec{Z}$=11.07 -j700.7, which is the resistance and reactance. I measured the 10 Ohm resistor, without the capacitor and saw 9.81 Ohms with a 50 Ohm reference and 8.23 Ohms with a 5000 Ohm reference, so the 11.07 Ohm value which is probably a combination of the difficulty in measuring the resistor with 700 Ohms reactance in series as well as some loss in the capacitor.  The 9.81 Ohms is very close to the measured DC value. Then we see the translation of the reactance value to a capacity of 227.1 nF (or 0.2271 $\mu$F).  Often this is the value we are looking for.

The Q value shown of 63.3 is the ratio of the reactance magnitude to the resistance value.  We are accustomed to measuring Q of inductors, and that would be shown here if the reactance were positive.  The Q of a capacitor has a physical meaning that parallels that of an inductor.

Lastly, the same screen shows the parallel RC values. At the one frequency where we measure the series connected components, we could have connected this parallel combination, and we would have measured the same quantities.
For more details on the series/parallel equivalence, see the original QEX article.\footnote{\textbf{\texttt{http://www.janbob.com/electron/AVNA1/Larkin-QEX-2018-May-Jun.pdf}}, pg. 12} In particular, note that in general, both the resistance and reactance values will change between the series and parallel representations.

\textbf{Swept Impedance Measurements - }  We do not have a graphical presentation of frequency swept impedance data.  Instead we have a tabular listing. To do this measurement from the single frequency version, we tap on "\textsf{Back}", bringing back the screen of Figure \ref{AVNA_001-label}. At this point we could change the reference resistor to 50 Ohms, but not for this exercise. Instead, we tap on "\textsf{Sweep}", bringing up the following screen.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_007.pdf}
\caption{Main Screen for swept measurements.}
\label{AVNA_007-label}
\end{center}
\end{figure}
We can change the 7 screen displayed frequencies shown in Figure \ref{AVNA_009-label} with the two menu items at the bottom.  In all cases, the measurements are made at all 13 frequencies, but there is not room to display them all.  For now, we will leave the frequency settings at 50 to 5000 Hz.  We command the impedance measurement by tapping on, "\textsf{Meas Z}".  Here is the display.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_009.pdf}
\caption{Swept impedance measurements.}
\label{AVNA_009-label}
\end{center}
\end{figure}
%
The display shows the impedance for seven frequencies.
The menu items, "\textsf{Disp Frq Down}" and "\textsf{Disp Frq Up}" allow the displayed frequencies to be shifted down or up to cover the entire 10 to 40,000 Hz range.  The impedance is shown in  \(R+jX\) format. It can be seen that negative reactance values display with a minus sign in front of the value. Next in the display line is a translation of the  \(jX\) value to either a capacitance or inductor value. The units, in this case,  "\textsf{nF}" change with component value.  In the right hand column is a rough measure of measurement quality. A single letter \textsf{E, G, P} corresponds to (\textsf{E})xcellent, (\textsf{G})ood, or (\textsf{P})oor. These indicate the difference between the impedance and the reference resistance.
The \textsf{E}, \textsf{G}, and \textsf{P} designations should not be taken too literally, but if the letter shows "\textsf{P}" or even "\textsf{G}",  you might consider the values and whether to re-measure with the other reference resistance.

Going back to the AVNA main menu, there is a menu item, "\textsf{What?}".  This is handy if the component type or value is unknown, or maybe a quick answer is wanted.  This starts with the following menu that describes what to do.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_012.pdf}
\caption{AVNA What? Screen.}
\label{AVNA_012-label}
\end{center}
\end{figure}
%
The only option available is to omit measuring at 10, 20 and 50 Hz that speeds up the measurement. Otherwise, tapping on "\textsf{Search Value}" produces the estimate:
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_013.pdf}
\caption{AVNA What? Screen after measuring.}
\label{AVNA_013-label}
\end{center}
\end{figure}
%
Two different values are shown, corresponding to the two reference resistor values.  The "\textsf{What?}" process chooses the frequency that gives an "E" for excellent measurement, if possible.  Often, this differs in frequency between the two reference resistor values.  For our 10 Ohm and 0.22 $\mu$F series combination, we find the 50 Ohm reference measurement at 20,000 Hz and the 5000 Ohm measurement down at 200 Hz.  In either case, the capacitive value works out well, but measuring the 10 Ohm resistor in series with a 3492 Ohm reactance (at 200 Hz) is looking less successful with a 19.28 indicated \(R\) value.  It would be interesting to explore this further by measuring the two components individually with swept measurements. (We won't do that here to save the fun for others.)
%
\subsection{Discussion}
%
\textbf{Measurement Technique - } There are two commonly used methods of measuring impedance: the unbalanced Wheatstone bridge.\footnote{The classic Wheatstone bridge,\textbf{\texttt{https://en.wikipedia.org/wiki/Wheatstone\_bridge}}} \footnote{The Wheatstone bridge in various AC/RF forms,\textbf{\texttt{http://g3ynh.info/zdocs/bridges/part\_1.html}}}
%
 and the series resistor method.   The latter method is used in the AVNA.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.75]{./images/AVNA_900.pdf}
\caption{Functional schematic of impedance measuring circuit.}
\label{AVNA_900-label}
\end{center}
\end{figure}
%
Figure \ref{AVNA_900-label} shows the deceptively simple measurement circuit.  A software Direct Digital Synthesis (DDS) in the Teensy DSP creates a sine wave signal at a frequency between 10 and 40,000 Hz.  This is converted to an analog voltage by the left DAC on the Teensy Audio Adapter board which then goes to an audio amplifier producing the analog signal  $\vec{V_R}$ seen on the diagram.
The overhead bar indicates that this is a complex voltage where we care about the phase of the sine wave as well as the amplitude.  \vecr{$\mathbf{V_R}$} is applied to the series combination of our reference resistor, $R_R$ and the unknown impedance, $\vec{\mathbf{Z_u}}$.  This results in a current through the pair  of \(\vec{\mathbf{V_R}}  /  (R_R +   \vec{\mathbf{Z_u}})\).  By measuring the complex voltage at the top of the unknown, $\vec{\mathbf{v_Z}}$ we can determine this current as
\begin{equation}
 \vec{\mathbf{i_u}}=\vec{\mathbf{V_R}}-\vec{\mathbf{V_Z}}/R_R
\end{equation}
 This in turn allows calculating the unknown impedance as
\begin{equation}
\vec{\mathbf{Z_u}} =\vec{\mathbf{V_Z}}/\vec{\mathbf{i_u}}               =R_R \times\vec{\mathbf{V_Z}}/\vec{\mathbf{V_R}}-\vec{\mathbf{V_Z}}
\end{equation}
As one would expect, this requires knowledge of the reference resistor value, but the absolute voltages are not used, but rather the ratio of absolute voltages.  This makes it important to have the two voltmeters, $\vec{\mathbf{V_R}}$ and $\vec{\mathbf{V_Z}}$ track in both amplitude and phase.  There is separate analog circuitry in the two voltmeter paths, so errors will occur.  These errors are removed by the calibration procedure where the two voltmeters are connected to the signal source and the ratio  of voltages and the difference in phase of the two is recorded. These are then applied as corrections at each impedance measurement.

\textbf{Measuring Amplitude and Phase - }The discussion of Measuring Technique, above, refers to the two voltmeters that are needed.  The  implementation of voltmeters is discussed in the QEX article referenced above, pages 8-9,
and won't be repeated here.  The basic process is to generate a pair of software sine waves that are 90 degrees apart in phase but at the exact measurement frequency.  Each of these is multiplied (mixed) with the signal being measured and then low-pass filtered.  This produces the two complex voltages, needed for the impedance calculation.  All of the mixing and filtering occurs in the Teensy DSP.


\chapter{Related Work}
\label{text:related}
Socially-aware and safe navigation among human pedestrians poses a problem which is far from being solved and applicable to a wide range of real-world scenarios. One of the challenging factors of this field lies in the broad palette of building blocks that a possible solution may depend on, each being a challenging subproblem. In the following, some of the related subproblems tackled within project \project are introduced.

\section{Pedestrian Prediction}
\label{text:related/prediction}
The problem of predicting human motions has been tackled by many different approaches. Some approaches try to model the human dynamics, including the dynamics of their interaction with their environment, or try to come up with a cost function that the human is trying to maximize (ontological methods). Other approaches model the pedestrian behavior merely by observing and (afterward) replicating it, without the use of inherent assumptions about the structure of the interaction or the human mindset itself (phenomenological methods).

\subsection{Ontological Approaches}
As described above, ontological approaches tackle the problem of predicting human movement by describing it as guided by an underlying "physical" model or some cost function. One popular example of a "physics-based" model, i.e., algorithms that model the pedestrian dynamics using first-order physical principles, is the social forces model \cite{Helbing1995}. The Social Forces model is one of the most commonly used pedestrian predictions models in the field, due to its combined capabilities to accurately predict the behavior of human movement, its interpretability, and its small computational complexity. The model uses first-order mechanical principles for forecasting the human motions in accordance with several interaction forces, that describe the impacts the pedestrian is exposed to:

\begin{align}
\vec{F} &= \underbrace{\frac{1}{\tau_{\alpha}} (v^0_{\alpha} \vec{e}_{\alpha} - \vec{v}_{\alpha})}_{F_{goal}} - \underbrace{\nabla_{\vec{r}_{\alpha \beta}} V_{\alpha \beta}[b(\vec{r}_{\alpha \beta})]}_{F_{interaction}}
\label{eq:social_forces} \\
2b &= \sqrt{(||\vec{r}_{\alpha \beta}|| + ||\vec{r}_{\alpha \beta} - v_{\beta} \dt \vec{e}_{\beta}||)^2 - (v_{\beta} \dt)^2}
\end{align}

For every pedestrian $\alpha$, Social Forces describes an attractive force to reach an imaginary goal position with desired velocity $v^0_{\alpha} \vec{e}_{\alpha}$, with $\vec{e}_{\alpha}$ being the unit direction vector pointing from the pedestrian's current to its goal position and $v^0_{\alpha}$ being the desired speed, e.g.,\ the maximal speed. The second force describes the repulsive forces the pedestrians exert among each other, modeled by the gradient of some potential field $V_{\alpha \beta} = V_{\alpha \beta}^0 \exp(-b / \sigma)$ in the direction pointing from pedestrian $\alpha$ to pedestrian $\beta$. Thereby, $b$ takes the relative speed and directions of the regarded pedestrian pair into account. While Social Forces is a mainly deterministic model, similar approaches use recursive Bayesian filtering to deal with uncertainty, such as \cite{Schneider2013}\cite{Rehder2015}\cite{Guo2016}. Due to the small computational complexity and the high interpretability of these models, they tend to be very useful for large scale simulations but are prone to modeling errors in small scale predictions, as they widely hinge on a set of estimated parameters which is assigned individually to each pedestrian, might not grasp the exact dynamical model of the pedestrian and do not take into account knowledge about past interactions, which is likely to be important for the up-coming interactions.
\newline
Another way of explicitly formulating the humans' interaction with their environment is by deriving a cost function that the human maximizes during the interaction. While some works describe the cost function as a payoff of a cooperative or adversarial game and apply game theory to predict the behavior of the other agents as well as of the agent itself, such as \cite{Bouzat2014}\cite{Nikolaidis2017}, other works use \ac{IRL} \cite{Ng2000} to characterize the interaction as a value function assigned to each state-action-pair, such as \cite{Fahad2018}\cite{Fernando2019}\cite{Saleh2018}, or as an expansion to probabilistic predictions using maximum entropy \ac{IRL} \cite{Ziebart2008}. Since IRL-based approaches rely on the Markovian property, they are not capable of including the interaction history while losing the property of interpretability by using a sufficiently complex reward function for modeling human interaction. Consequently, these approaches might work well under the constraints of limited data due to the possibility of encoding prior knowledge and the extrapolation capabilities of \ac{IRL}-based approaches but lack behind phenomenological in the presence of many available data.

\subsection{Phenomenological Approaches}
Phenomenological methods incorporate data-driven techniques to observe and imitate human behavior. As a consequence, they make merely minimal assumptions about the inherent interaction process, especially in comparison to most of the ontological methods discussed above. Therefore, the availability of a reasonably large set of observable data is the key to developing a well-performing prediction model. However, since increasingly many pedestrian datasets became available over recent years, such as \cite{Pellegrini2009}\cite{Rasouli2019}\cite{Caesar2020}, phenomenological models became increasingly accurate for a wide range of scenarios.
\newline
In contrast to previously described methods, phenomenological approaches do not assume Markovian state transitions and are therefore capable of taking the state histories of each agent into account, a presumably crucial feature for a precise prediction over future states. \ac{LSTM} modules \cite{Hochreiter1997} have been developed to model temporal sequences and are thus perfectly suitable for the purpose of mining past human trajectories for predicting future states, as done in \cite{Chen2019a}\cite{Hug2018}\cite{Zhang2019}\cite{Jain2016} (similarly using GRUs \cite{Liu2020}). However, the resulting deterministic trajectory output is not able to account for the uncertainty assigned to each prediction of future states, particularly due to the dynamic nature of human gait. To fully account for each possible future outcome, however, unimodal predictions would either miss some eventualities or be overly conservative estimates.
\newline
Generative models proved to be well applicable to solve this dilemma by producing general (including multi-modal) probability distribution over future pedestrian trajectories, learned from data. A \ac{GAN} \cite{Goodfellow2014}, a type of (deep) generative models, has been widely used in the field, as in \cite{Gupta2018}\cite{Kosaraju2019}\cite{Ouyang2018} but is especially hard to train due to the internal conflict between generator and discriminator and generally output empirical distributions. Related to \ac{GAN}s are \ac{VAE}, a class of generative models that make use of a latent space. Conditional VAEs (CVAEs) have been recently used to predict distributions over pedestrian trajectories. CVAEs can learn the probability distribution of their latent space variables and map samples of these distribution to obtain a desired output distribution, allowing empirical as well as analytical distributions, and being easier to train in comparison to \ac{GAN}s. Especially, this only allows to formulate the desired output distribution to be processable for online usage. Examples of works using (C)\ac{VAE}s in the area of pedestrian prediction are \cite{Ivanovic2018}\cite{Salzmann2020}\cite{Poibrenski2020}\cite{Lee2017}.

\subsection{Trajectron}
The Trajectron++ model \cite{Salzmann2020} by Salzmann et alt., an enhancement of the original Trajectron model \cite{Ivanovic2018}\footnote{For the sake of brevity in the following the Trajectron++ model will be addressed by the Trajectron model.}, is a generative, graph-structured, pedestrian prediction model based on a C\ac{VAE}. By incorporating the state histories of various agents in the scene as well as their environment, it can efficiently predict a distribution over a full trajectory of future states, each being multi-modal and represented by a \ac{GMM}. Especially, to the best of the authors' knowledge, the Trajectron model is the only model within the field of pedestrian prediction that is able to condition the prediction on a robot's planned trajectory.

\begin{figure}[!ht]
\begin{center}
\includegraphics[width=\imgwidth]{images/trajectron++.png}
\captionof{figure}{Spatiotemporal network architecture of Trajectron++ model \cite{Salzmann2020}}
\label{img:trajectron_model}
\end{center}
\end{figure}

\begin{equation}
\max_{\phi, \theta, \psi} \sum_{i=1}^N \mathbb{E}_{z \sim q_{\phi}(z | x_i, y_i)} [\log p_\psi (y_i | x_i, z)] - \beta D_{KL} (q_{\phi}(z | x_i, y_i) || p_{\theta}(z | x_i)) + \alpha I_q(\boldsymbol{x}; z)
\label{eq:trajectron_loss}
\end{equation}

Figure \ref{img:trajectron_model} shows the architecture of the Trajectron model. Each agent in the scene is modeled as a node in a spatio-temporal graph, with assigned state history and properties shared by nodes of the same type, e.g.,\ pedestrian or vehicle. Equation \ref{eq:trajectron_loss} shows the loss function the model is being trained with. In the first step, the inputs, which are the state history of each node, including the robot, the environment obstacle map, as well as the planned robot trajectory (inputs $\boldsymbol{x}$), are encoded to a 25 dimensional latent space $\boldsymbol{z}$. The latent space representation is then sampled to generate the output distribution $\boldsymbol{y}$, unrolled over the full prediction horizon. Formally, the model is described as estimating the conditional probability distribution $p(\boldsymbol{y}|\boldsymbol{x})$ by marginalizing over the latent variable $\boldsymbol{z}$:

\begin{equation}
p(\boldsymbol{y}|\boldsymbol{x}) = \sum_{\boldsymbol{z}} p_{\psi} (\boldsymbol{y} | \boldsymbol{x}, z) p_{\theta}(z | \boldsymbol{x})
\end{equation}

The optimization can be regarded as maximizing the $\beta$-weighted evidence lower-bound of the conditional distribution $p(\boldsymbol{y}|\boldsymbol{x})$ \cite{Ivanovic2018}. Although the model is able to deal with the agent's environment as well as several types of agents project, \project focuses on the pedestrian type in a free-space environment, which is described in Section \ref{text:approach/formulation} and its implications in Section \ref{text:experiments/integration}.

\section{Safe Planning under Uncertainty}
\label{text:related/uncertainty}
Trajectory optimization itself is a wide field with many different methodologies. Though the area can be roughly broken down into two categories, shooting and collocation \cite{Kelly2017}.\footnote{There are several other directions of separation possible such as the direct vs indirect methods, but I will focus on shooting vs collocation here. Further information about the taxonomy of trajectory optimization can be found in \cite{Kelly2017}  and \cite{Chai2020}.} Shooting optimizes for the control inputs and unrolls them using a simulation environment to compute objective and constraint. Thereby, the dynamics constraint $x = \f(x, u)$ is intrinsically enforced. In comparison, collocation uses all controls and states as decision variables while constraining the system dynamics and tries to solve the \ac{NLP} by approximating some function (e.g.\ Lagrange polynomials in orthogonal collocation).
\newline
While traditionally, the problem of trajectory optimization under uncertainty has been tackled by using fixed error bounds on all uncertainties in the system, which is also known as robust control \cite{Bemporad1999}. Consequently, solutions obtained were largely sub-optimal due to widely overestimating the actual system uncertainties, or the optimization formulation became infeasible and thus impossible to solve. Subsequently, many different directions of research have evolved, such as formulating the problem as a \ac{POMDP} and (online) \ac{POMDP}-solvers to find feasible trajectories \cite{Chen2016}, using Monte-Carlo planning \cite{Janson2015} \cite{Silver2010}, signal temporal logic \cite{Sadigh2016}, (probabilistic) decision graphs \cite{Koenig1994}, sampling-based methods, as well as constraint formulations that estimate and bound the risk (or chance) of collision \cite{Ono2015}\cite{Lew2019}\cite{Chow2015a}\cite{Chow2013}\cite{Ono2012}\cite{Ludersa}\cite{Luders2011}\cite{Otte2014} or reachability-based concepts for guaranteeing the impossibility of collision solely on the base of the dynamical properties and initial conditions of interacting agents \cite{Leung2020}\cite{Dhinakaran2017}\cite{Margellos2009}\cite{Chen2017b} \cite{Althoff2009}\cite{Althoff2010}. Since discussing all of these different techniques surely is out of the scope of this work, only three approaches, which are most relevant to this work, will be examined in the following:

\subsection{Sampling-based Path Planning}
Sampling-based path planning methods, such as RRT \cite{LaValle1998}, have been shown to be an efficient solution for path planning in continuous-space, static environments, while often guaranteeing asymptotic optimality and feasibility of the derived solution, such as \cite{Karaman2011}\cite{Luders}. These algorithms are iteratively sampling random points from the configuration space, while retaining those in the free space (i.e., free from obstacles), storing them as milestones in a roadmap, and connecting those milestones, which can be connected completely in the free space. The algorithm repeats until either the goal state has been incorporated in the roadmap or the maximal computation time has exceeded. To account for dynamic and probabilistic obstacles, a temporal dimension has to be added to the search space, making the problem computationally challenging to solve in real-time. For addressing this issue, rapid replanning and repairing \cite{Otte2014} are often used. To implicitly take into account system uncertainties, the occupancy cost can be augmented by risk level sets that assign a certain risk to each robot state, depending on the obstacle's state, e.g., its orientation and velocity, as shown in \cite{Pierson2018}\cite{Pierson2019}. However, despite the success of sampling-based methods in online path planning, they do not explicitly leverage future predictions of the environment and are therefore restricted to only (temporally) locally optimal solutions.

\subsection{Chance Constraints}
Chance constraints are a specific way of formulating a constraint under uncertainty, such that it holds for at least a pre-defined probability level $p$. With a general inequality constraint $g(x) \leq 0$ it is:

\begin{equation}
Pr(g(x) \leq 0) \geq p
\label{eq:chance_constraint}
\end{equation}

Usually, constraint \ref{eq:chance_constraint} is reformulated as a set of deterministic constraints to be manageable for optimization. Therefore, several methods have been purposed: Sampling-based methods relying on the Bernstein approximation \cite{Calafiore2005}, on scenario planning approach \cite{Bemporad1999}, or Monte Carlo sampling \cite{Hong2011}\cite{Janson2015} as well as dynamic-programming based approaches for discrete space \cite{Chow2013}\cite{Ono2015} \cite{Chow2015a}. While these methods work for general distributions but are usually too slow for an online optimization approach, shrinking the space of distributions can improve the performance, such as in \cite{Chen2018}\cite{Calafiore2006}\cite{Carvalho2014}\cite{Blackmore2009}\cite{Blackmore2011}. Thereby, handling multi-modality of the risk distribution efficiently is a widely untouched ground, to the best of the authors' knowledge, which only has been solved for a specific scenario, such as in \cite{Hu2018}. To conclude, solving a chance constraints program efficiently is quite hard to do, especially when dealing with general, such as multi-modal and non-convex risk distributions.

\subsection{Hamilton-Jacobi Reachability}
Reachability analysis deals with the problem of a two-person, zero-sum differential game. Specifically, it tries to solve the question how to react if there is another player that interferes with the fulfillment of the robot's objective by optimizing the joint system state, assuming that the counter-player always counters the robot's action optimally, i.e., assuming the worst case disturbance $\d = \beta[\u]$ \cite{Pavone2020}. Under these conditions, the optimal control strategy can be derived by maximizing: \\

\begin{equation}
V(x(t), t) = \min_{\beta[\u](\cdot)} \max_{\u(\cdot)} \left[ \int_t^0 l(\x(\tau), \u(\tau), \boldsymbol{d}(\tau)) \, d\tau + l_f(\x(0)) \right]
\label{eq:j_reachability}
\end{equation}

with $(\x(\tau), \u(\tau), \d(\tau))$ describing the joint system with dynamics $f(\boldsymbol{x}, \u, \d)$. While solving the robot's optimal control strategy $\u_{opt}$ in open-loop (also \textit{forward} reachability) is computationally inexpensive but leads to overly conservative and hence un-realistic estimates, since the counter-player knows the entire policy $\u_{opt}$ in advance and can re-act accordingly, \ac{HJR} (also \textit{backward} reachability) addresses the problem of maximizing Equation \ref{eq:j_reachability} in closed-loop, i.e., by allowing the robot to adapt its control policy at each time-step. As proven in \cite{Pavone2020}, this is equivalent to solving the Hamilton-Jacobi-Isaacs differential equation with boundary condition: \\

\begin{problem}{General \ac{HJR} problem}
\begin{align}
\pd{V}{t} + \max_{\u}  \min_{\d} &\left[l(\x, \u, \d) + \nabla V^T f(\x, \u, \d) \right] = 0 \\
V(\x, 0) &= l_f(\x) \\
\Rightarrow \u_{opt} &= \arg \max_{\u}  \min_{\d} \nabla V^T f(\x, \u, \d)
\label{eq:hjr_problem_u}
\end{align}
\label{eq:hjr_problem}
\end{problem}

Solving Problem \ref{eq:hjr_problem} has been examined for several scenarios and applications over the recent years, as in \cite{Dhinakaran2017}\cite{Margellos2009}\cite{Chen2017b}, and has shown to be especially useful in the field of autonomous driving \cite{Althoff2009}\cite{Althoff2014}\cite{Althoff2010}. In opposite to forward reachability, \ac{HJR} finds the non-overly conservative avoidance maneuvers, "stemming from its equivalence to an exhaustive search over the joint system dynamics" while still being flexible with respect to the system dynamics, as depicted by \cite{Leung2020}. To solve \ref{eq:hjr_problem}, the value function $V(\cdot)$ as well as its gradient are computed for every cell of a $n$-dimensional grid that discretizes the joint state $\x(\tau)$ and for some discrete time horizon (including $\tau \rightarrow \infty$).

\section{Navigation in Human Crowds}
\label{text:related/crowd_navigation}
Navigation in human crowds is especially important for vastly integrating autonomous ground vehicles in our daily life but poses a hard problem due to a priori unknown human intentions and hard to quantify objectives, e.g.,\ the notion of socially-acceptable behavior. Existing work in this field can be roughly divided into two main directions, model- and learning-based approaches:

\subsection{Model-Based Approaches}
Traditionally, the pedestrians are regarded as dynamic obstacles, with the collision avoidance algorithm being used for robot navigation \cite{vandenBerg2011}\cite{Fox1997}\cite{Luo2018a}\cite{Phillips2011}. \ac{ORCA} \cite{vandenBerg2011}, for example, guarantees a collision-free pre-defined time horizon, under the assumption of perfect knowledge about the state of each interacting agents as well as, more importantly, a shared, deterministic policy over all agents. Although these models have been advanced to relax some of the assumptions such as perfect perception \cite{Hennes2012}, they still fail to capture a human-like behavior and, consequently, may lead to unsafe and socially unacceptable robot actions. To solve this issue, it is crucial to directly include the predicted impact of the robot's planned trajectories into the optimization itself. For this reason, the robot's motion has been incorporated in some of the (model-based) pedestrian prediction models discussed in Section \ref{text:related/prediction}, e.g.,\ the Social Forces model \cite{Helbing1995}, usually by augmenting the model by robot-specific parameters while preserving the general concept of interaction \cite{Ferrer2013}\cite{Luo2018a}. Nonetheless, as discussed, these models usually are largely hinging on parameters, making them hard to tune and hence inaccurate in prediction and thus planning as a consequence of modeling errors. Additionally, model-based approaches are usually deterministic, regarding only one out of assumably many possible outcomes, and therefore might lead to un-safe behavior.

\subsection{Learning-based Approaches}
Learning-based approaches try to replicate human behavior that has been learned from human demonstration in simulations or by accumulating features over possible trajectories of interacting agents while minimizing some interactive cost function (e.g.,\ the separation distance) \cite{Kim2016}\cite{Kretzschmar2016}. While these models tend to plan trajectories that are more socially acceptable and natural, they are much larger in computational cost and very hard to train due to the dynamic and stochastic nature of human gait. Other approaches come up with a set of socially compliant rules, such as "passing on the right" and use deep reinforcement learning to apply them  \cite{Knepper2012}\cite{Chen2017}\cite{Everett2018}. However, it can be doubted whether these methods can be generalized to unseen scenarios, how to certify a safe interaction, and whether these rules hold at all.
\newline\newline
While model-based approaches are computationally efficient and traceable but lack an accurate capturing of human behavior, learning-based methods do model the human motion more naturally and are therefore capable of planning socially-aware trajectories. However, they tend to be computationally expensive, are neither interpretable nor able to give any safety guarantees, which is crucial for secure human-robot interaction. Therefore, this works wants to combine both worlds, the predictive abilities of deep learned models with the safety guarantees of model-based safety guaranteeing methods, such as \ac{HJR}, which was discussed above. To truly leverage the full "knowledge" of the prediction model, concepts of model-gradient informed concepts, which have been already shown to be successful in control theory \cite{Chen2019}\cite{Fan2020}, are thereby re-formulated for trajectory optimization. While most of the described approaches deal with deterministic and uni-modal predictions, due to the highly stochastic human motion, anticipatory and efficient, but safe interactions can only be derived when the system can reason about many possible future outcomes of the scenario so that multi-modality should be taken into account.

% sequential-action control \cite{Nishimura2020}\cite{Nishimura2020a}


\documentclass[10pt,portrait]{article}
\usepackage[portrait]{geometry}
\input{math_header}

% Format inherited from <MA1101R Cheatsheet 17/18 Sem 1 Finals>
% Original document is by Lee Yiyuan and Eugene Lim
%
% All the theorems are numbered according to <LINEAR ALGEBRA - Concepts and
% Techniques on Euclidean Space>, ISBN 978-981-3152-88-5, Second Edition (2016)
% -----------------------------------------------------------------------

\title{MA1101R Cheatsheet 19/20 Semester 1 Final}

\begin{document}

\begin{center}
{\large MA1101R Cheatsheet 19/20 Semester 1 Final}\\{by Howard Liu}
\end{center}

\footnotesize

\begin{multicols}{2}
\begin{justifying}

\setlength{\premulticols}{1pt}
\setlength{\postmulticols}{1pt}
\setlength{\multicolsep}{1pt}
\setlength{\columnsep}{2pt}

\section{Matrices}

\begin{namedthm*}{Theorem 1.2.7}
	If \textbf{augmented matrices} of two systems of linear equations are row equivalent, then the two systems have the same set of solutions. (\(\ast\) Even for two homogeneous linear systems, we still need to say that \(\begin{pmatrix}[c|c]\matr{A} & \matr{0}\end{pmatrix}\) is row equivalent to \(\begin{pmatrix}[c|c]\matr{B} & \matr{0}\end{pmatrix}\), not that \(\matr{A}\) is row equivalent to \(\matr{B}\).)
\end{namedthm*}

\begin{namedthm*}{Example 1.4.10}
	Suppose augmented matrix \(\matr{R}\) is in (R)REF:
	\begin{enumerate}
		\item LS has no solution \\
		    \(\Leftrightarrow\) Last column of \(\matr{R}\) is pivot.
		\item LS has one unique solution \\
		    \(\Leftrightarrow\) \textbf{Only} last column of \(\matr{R}\) is non-pivot.
		\item LS has infinite number of solution \\
		    \(\Leftrightarrow\) At least one column other than the last one is non-pivot \\
		    \(\Leftrightarrow\) Number of variables $>$ Number of non-zero rows in \(\matr{R}\) \\
		(\(\ast\) \# non-pivot columns in (R)REF \(- 1 =\) \# unique solutions)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Theorem 6.1.8}
	\(\matr{A}\) is invertible when:
	\begin{enumerate}
		% T2.4.7
		\item \(\exists \matr{B}\) s.t. \(\matr{AB} = \matr{I} \lor \matr{BA} = \matr{I}\)
		\item Refer to \(\textbf{Theorem 2.4.7.2}\) below
		\item \(\rref(\matr{A}) = \matr{I}\)
		\item \(\matr{A}\) is a product of elementary matrices
		% T2.5.19
		\item \(\det(\matr{A}) \ne 0\)
		% T3.6.11
		\item Rows of \(\matr{A}\) is a basis of \(\mathbb{R}^n\)
		\item Columns of \(\matr{A}\) is a basis of \(\mathbb{R}^n\)
		% T6.1.8
		\item 0 is not an eigenvalue of \(\matr{A}\)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Remark 2.3.4 (Cancellation Laws for Matrices)}
	Let \(\matr{A}\) be an invertible \(m \times m\) matrix,
	\begin{enumerate}[label=(\alph*)]
		\item If \(\matr{B}_1\) and \(\matr{B}_2\) are \(m \times n\) matrices with \(\matr{AB_1} = \matr{AB_2}\), then \(\matr{B}_1 = \matr{B}_2\)
		\item If \(\matr{C}_1\) and \(\matr{C}_2\) are \(n \times m\) matrices with \(\matr{C_1A} = \matr{C_2A}\), then \(\matr{C}_1 = \matr{C}_2\)
	\end{enumerate}
\end{namedthm*}
\begin{namedthm*}{Theorem 2.4.7.2 (generalised)}
	Relationship between singularity of \(\matr{A}\) and the number of solutions of a linear system \(\matr{Ax} = \matr{b}:\)
	\begin{enumerate}
		\item \(\matr{A}\) is singular \(\Leftrightarrow \matr{Ax} = \matr{b}:\) has $\infty$ solutions (only case for homogeneous LS) or no solutions
		\item \(\matr{A}\) is invertible \(\Leftrightarrow \matr{Ax} = \matr{b}:\) has one unique solution (trivial solution for homogeneous LS)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Definition 2.5.2}
    Let \(\matr{A} = \left(a_{ij}\right)\) be an \(n \times n\) matrix. Let \(\matr{M}_{ij}\) be an \(\nobreak{(n - 1)\times (n - 1)}\) matrix obtained from \(\matr{A}\) by deleting the \(i\)th row and the \(j\)th column. Then the \textit{determinant} of \(\matr{A}\) is defined as
    \[
        \det(\matr{A}) =
            \begin{cases}
                a_{11} & \text{if \(n = 1\)} \\
                a_{11}A_{11} + \cdots + a_{1n}A_{1n} & \text{if \(n > 1\)}
            \end{cases}
    \]
    where
    \[
        A_{ij} = (-1)^{i + j} \det\left(\matr{M_{ij}}\right)
    \]
    The number \(A_{ij}\) is called the \((i, j)\)\textit{-cofactor} of \(\matr{A}\).
\end{namedthm*}

\begin{namedthm*}{Theorem 2.5.8}
    The determinant of a triangular matrix is equal to the product of its diagonal entries.
\end{namedthm*}

\begin{namedthm*}{Theorem 2.5.12 (added-on)}
	The determinant of a square matrix is 0 when:
	\begin{enumerate}
		\item it has two identical rows, or
		\item it has two identical columns
		\item any row/column of its (R)REF is zero
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Theorem 2.5.15}
    Let \(\matr{A}\) be a square matrix. \(k\) is a non-zero constant.
    \begin{enumerate}
    	\item \(\matr{A} \xrightarrow{k\vect{R}_i} \matr{B} \Rightarrow \det(\matr{B}) = k\det(\matr{A})\)
    	\item \(\matr{A} \xrightarrow{\vect{R}_i \leftrightarrow \vect{R}_j} \matr{B} \Rightarrow \det(\matr{B}) = -\det(\matr{A})\)
    	\item \(\matr{A} \xrightarrow{\vect{R}_i + k\vect{R}_j} \matr{B} \Rightarrow \det(\matr{B}) = \det(\matr{A})\)
        \item Let \(\matr{E}\) be an elementary matrix of the same size as \(\matr{A}\). Then \(\det(\matr{EA}) = \det(\matr{E})\det(\matr{A})\).
    \end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Remark 2.5.18}
	Since \(\det(\matr{A}) = \det(A^T)\), theorem 2.5.15 holds if ``rows" are changed to ``columns".
\end{namedthm*}

\begin{namedthm*}{Theorem 2.5.22}
	Let \(\matr{A}\) and \(\matr{B}\) are two square matrices of order \(n\) and \(c\) is a scalar. Then
	\begin{enumerate}
		\item \(\det(c\matr{A}) = c^n\det(\matr{A})\)
		\item \(\det(\matr{AB}) = \det(\matr{A})\det(\matr{B})\)
		\item if \(\matr{A}\) is invertible, \(\det(\matr{A}^-1) = \frac{1}{\det(\matr{A})}\)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Definition 2.5.24}
	Let \(\matr{A}\) be a square matrix of order \(n\). Then the \textit{(classical) adjoint} of \(\matr{A}\) is the \(n \times n\) matrix
	\[
	\adj(\matr{A}) = \left(A_{ij}\right)_{n \times n}^T
	\]
	where \(A_{ij}\) is the \((i, j)\)-cofactor of \(\matr{A}\).
\end{namedthm*}

\begin{namedthm*}{Theorem 2.5.27 (Cramer's Rule)}
    Suppose \(\matr{A}\vect{x} = \vect{b}\) is a linear system where \(\matr{A}\) is an \(n \times n\) matrix. Let \(\matr{A_i}\) be the matrix obtained from \(\matr{A}\) be replacing the \(i\)th column of \(\matr{A}\) by \(\vect{b}\). If \(\matr{A}\) is invertible, then the system has only one solution
    \[
        \vect{x} = \frac{1}{\det(\matr{A})}\begin{pmatrix}\det\left(\matr{A_1}\right) \\ \vdots \\ \det\left(\matr{A_n}\right) \end{pmatrix}
    \]
\end{namedthm*}

\begin{namedthm*}{Mixed Notes 1}
	\(\matr{A}^{-1}\) is able to be computed by:
	\begin{enumerate}
		\item Find \(\matr{B}\) s.t. \(\matr{AB} = \matr{I} \lor \matr{BA} = \matr{I}\)
		\item Find using \textbf{Theorem 2.5.25}: \(\matr{A}^{-1} = \frac{1}{\det(\matr{A})}\adj(\matr{A})\)
		\item Find using: \(\begin{pmatrix}[c|c] \matr{A} & \matr{I}\end{pmatrix} \xrightarrow{GJE} \begin{pmatrix}[c|c] \matr{I} & \matr{A}^{-1}\end{pmatrix}\)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Mixed Notes 2}
	\(\det(\matr{A})\) is able to be computed by:
	\begin{enumerate}
		\item Using \textbf{Theorem 2.5.2}
		\item Using cross multiplication (for \(2 \times 2\) and \(3 \times 3\) matrices only)
		\item Doing some ERO (e.g. GE, consider \textbf{Thoerem 2.5.15}) and making it triangular  then using \textbf{Theorem 2.5.8} or making it have properties in \textbf{Theorem 2.5.12}
		\item Using \textbf{Theorem 2.5.22}
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Mixed Notes 3}
	Some random notes:
	\begin{enumerate}
		\item In \(\mathbb{R}^n\) where \(n \ge 2\), a set with 1 parameter is a line and that with 2 parameters is a space.
		\item \(\matr{M}^2 + \matr{M} = \matr{I} \Rightarrow \matr{M}(\matr{M} + \textcolor{red}{\matr{I}}) = \matr{I}\) (Don't put that \(\matr{I}\) to be scalar 1!)
		\item Two matrices have same RREF \(\Leftrightarrow\) They are row equivalent
		\item In exam, express a matrix in the form \(\matr{A} = (a_{ij})_{m \times n}\). \textbf{DO NOT} use dots form
		\item When using ERO \(\vect{R}_i = \frac{1}{k}\vect{R}_j\), discuss whether \(k\) is 0 when necessary
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Mixed Notes 4}
	Generally, for (square) matrices \(\matr{A}\) and \(\matr{B}\),
	\begin{enumerate}
		\item \(\matr{AB} \ne \matr{BA}\)
		\item \((\matr{AB})^2 \ne \matr{A}^2\matr{B}^2\)
		\item \(\matr{AB} = 0 \nRightarrow \matr{A} = 0 \lor \matr{B} = 0\)
		\item \(\matr{A}^2 = I \nRightarrow \matr{A} = \pm \matr{I}\) (For example: 2 EMs of 2nd type ERO)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Mixed Notes 5}
	When expanding a row/column with cofactors of the other row/column, 0 will be yielded:
	\[
	    \sum_{m=1}^n a_{im}A_{jm} = \sum_{m=1}^n a_{mi}A_{mj} = 0, \text{ for some } i \ne j
	\]
\end{namedthm*}

\section{Euclidean Spaces}

\begin{namedthm*}{Discussion 3.2.5}
	Given \(S = \{\vect{v_1}, \vect{v_2}, \dots, \vect{v_m}\} \subseteq \mathbb{R}^n\}\), show \(\lspan(S) = \mathbb{R}^n\):

	\medskip
	\noindent
	Consider \(\vect{v_i} = \left(v_{i1}, \dots, v_{in}\right)\),
	\[
	\begin{pmatrix}
	\vect{v_{11}} & \dots & \vect{v_{m1}}\\
	\vdots & \ddots & \vdots\\
	\vect{v_{1n}} & \dots & \vect{v_{mn}}
	\end{pmatrix} \xrightarrow{GE} \matr{R}
	\]
	\(\lspan(S) = \mathbb{R}^n \Leftrightarrow \matr{R}\) has no zero rows
\end{namedthm*}

\begin{namedthm*}{Theorem 3.2.7}
	If \(|S| < n\), \(\lspan(S) \ne \mathbb{R}^n\).
\end{namedthm*}

\begin{namedthm*}{Theorem 3.2.10}
	Let \(S_1 = \{\vect{u_1}, \dots, \vect{u_k}\}\) and \(S_2 = \{\vect{v_1}, \dots, \vect{v_m}\}\) be subsets of \(\mathbb{R}^n\). Then, \(\lspan(S_1) \subseteq \lspan(S_2) \Leftrightarrow \forall i=1, 2, \dots, k\), \(u_i \in \lspan\{\vect{v_1}, \dots, \vect{v_m}\}\).
\end{namedthm*}

\begin{namedthm*}{Definition 3.3.2}
     Let \(V\) be a subset of \(\mathbb{R}^n\). Then \(V\) is called a \textit{subspace} of \(\mathbb{R}^n\) if \(V = \lspan(S)\) where \(S = \{\vect{u_1}, \dots, \vect{u_k}\}\) for some vectors \(\vect{u_1}, \dots, \vect{u_k} \in \mathbb{R}^n \).

     \medskip
     \noindent
     More precisely, \(V\) is called the \textit{subspace spanned} by \(S\) (or the \textit{subspace spanned} by \( \vect{u_1}, \dots, \vect{u_k} \)). We also say that \(S\) \textit{spans} (or \(\vect{u_1}, \dots, \vect{u_k}\) \textit{span}) the subspace \(V\).

     \medskip
     \noindent
     By contraposition, \(V = \lspan(S) \Rightarrow \vect{0} \in V \equiv \vect{0} \notin V \Rightarrow V \ne \lspan(S)\). (\(\ast\) i.e., If \(\vect{0}\) is not in \(V\), \(V\) is not a subspace of \(\mathbb{R}^n\))
\end{namedthm*}

\begin{namedthm*}{Theorem 3.3.6}
	If \(V = \{\matr{x} | \matr{Ax} = \matr{0}\}\), \(V\) is a subspace of \(\mathbb{R}^n\).
\end{namedthm*}

\begin{namedthm*}{Remark 3.3.8}
    Let \(V\) be a non-empty subset of \(\mathbb{R}^n\). Then \(V\) is a subspace of \(\mathbb{R}^n\) if and only if
    \[
        \text{for all } \vect{u}, \vect{v} \in V \text{ and } c, d\in \mathbb{R},\enspace c\vect{u} + d\vect{v} \in V
    \]
    (\(\ast\) This checks whether V is \textbf{closed} under addition and scalar multiplication)
\end{namedthm*}

\begin{namedthm*}{Definition 3.4.2/4}
	Consider \(\vect{u_1}, \vect{u_2}, ..., \vect{u_k}\) which are column vectors, set \(S = {\vect{u_1}, \vect{u_2}, ..., \vect{u_k}}\) is \textbf{Linear Indepedent} iff. any of:
	\begin{enumerate}
		% T3.4.2
		\item \((\vect{u_1} \vect{u_2} ... \vect{u_k})\vect{x} = \matr{0}\) has only trivial solution.
		% T3.4.4
		\item No vectors in \(S\) can be written as a linear combination of other vectors in \(S\).
		% From revision note of tutor
		\item \(S\) is a subset of a \textbf{Linear Independent} set.
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Definition 3.5.4/Theorem 3.6.7}
	A set \(S\) is a basis of a vector space if:
	\begin{enumerate}[label*=\arabic*.]
		\item \(S \subseteq V\)
		\item Any 2 of the 3 below:
		\begin{enumerate}[label*=\arabic*.]
			\item \(S\) is Linear Independent
			\item \(S\) spans \(V\)
			\item \(|S| = \dim(V)\)
		\end{enumerate}
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Definition 3.5.8}
	Let \(S = {\vect{u_1}, \vect{u_2}, ..., \vect{u_k}}\) be a basis for a vector space \(V\) and \(\vect{v}\) is a vector in \(V\). By T3.5.7, \(\vect{v}\) is expressed uniquely as a LC:
	\[
	    \vect{v} = c_1\vect{u_1} + c_2\vect{u_2} + \dots + c_k\vect{u_k}
	\]
	Then we shall have the \textbf{coordinate vector} of \(\vect{v}\) relative to the basis \(S\) : \((\vect{v})_S = (c_1, c_2, \dots, c_k) \in \mathbb{R}^k\) (assuming vectors in \(S\) are in fixed order).
\end{namedthm*}

\begin{namedthm*}{Remark 3.5.10/Theorem 3.5.11}
	Let \(S\) be a basis for a vector space \(V\),
	\begin{enumerate}
		\item \(\forall \vect{u}, \vect{v} \in V, \vect{u} = \vect{v} \Leftrightarrow (\vect{u})_S = (\vect{v})_S\)
		\item Coordinate vectors are closed under scalar multiplication and addition
		\item Let \(\vect{v_1}, \vect{v_2}, \dots, \vect{v_r} \in V\), they are LI iff. \((\vect{v_1})_S, (\vect{v_2})_S, \dots, (\vect{v_k})_S\) are LI
		\item \(\lspan {\vect{v_1}, \vect{v_2}, \dots, \vect{v_r}} = V \Leftrightarrow \lspan {(\vect{v_1})_S, (\vect{v_2})_S, \dots, (\vect{v_k})_S} = \mathbb{R}^{|S|}\)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Theorem 3.6.9}
	Let \(U\) be a subspace of \(V\), then \(\dim(U) \le \dim(V)\). Furthermore, if \(U \ne V\), then \(\dim(U) < \dim(V)\).
\end{namedthm*}

\begin{namedthm*}{Definition 3.7.3}
	Let \(S = {\vect{u_1}, \vect{u_2}, ..., \vect{u_k}}\) and \(T\) be two bases for a vector space. The square matrix \(\matr{P} =\begin{pmatrix}{[\vect{u_1}]}_T & {[\vect{u_2}]}_T & \dots & {[\vect{u_k}]}_T\end{pmatrix}\) is called the \textbf{transition matrix} from \(S\) to \(T\).
\end{namedthm*}

\begin{namedthm*}{Mixed Theorem 6}
	Consider \(S\) and \(T\) are two bases for vector space \(V\) and \(\matr{P}\) is the transition matrix from \(S\) to \(T\). If \(\matr{A}\) and \(\matr{B}\) are matrices with elements of \(S\) and \(T\) respectively as columns, we have \(\matr{BP} = \matr{A}\).
\end{namedthm*}

\begin{namedthm*}{Mixed Theorem 7}
	ERO preserves row space \textbf{(T4.1.7)}, and we have:
	\begin{itemize}
		\item \textbf{(R4.1.9)} \(\matr{R}\) is RREF of \(\matr{A}\). Non-empty rows in \(\matr{R}\) forms the basis of row space of \(\matr{A}\).
		\item \textbf{(T4.2.1)} Row space and column space of a matrix have the same dimension.
	\end{itemize}
\end{namedthm*}

\begin{namedthm*}{Remark 4.2.5}
	Regarding rank(\(\matr{A}\)):
	\begin{enumerate}
		\item For \(m * n\) matrix \(\matr{A}\), \(\rank(\matr{A}) \le \min{m, n}\). If \(\rank(\matr{A}) = \min{m, n}\), \(\matr{A}\) is said to have \textbf{full rank}.
		\item A square matrix \(\matr{A}\) have full rank iff. it is invertible.
		\item \(\rank(\matr{A}) = \rank(\matr{A^T})\).
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Theorem 4.3.6}
	Suppose linear system \(\matr{A}\vect{x} = \vect{b}\) has solution \(\vect{v}\), then the solution set of this system is given by:
	\[
	    M = \{ \vect{u} + \vect{v} | \vect{u} \in \text{nullspace\((\matr{A})\)}\}
	\]
\end{namedthm*}

\section{Orthogonality}

\begin{namedthm*}{Definition 5.1.2.3/4}
	For two vectors \(\vect{u}\) and \(\vect{v}\):

	\(d(\vect{u}, \vect{v}) = \norm{\vect{u} - \vect{v}}\).

	Angle between \(\vect{u}\) and \(\vect{v}\) is:
	\[
	    \cos^{-1}(\frac{\vect{u}\cdot\vect{v}}{\norm{\vect{u}}\norm{\vect{v}}})
	\]
\end{namedthm*}

\begin{namedthm*}{Theorem 5.2.4}
	If \(S\) is an orthogonal set of non-zero vectors in a vector space, \(S\) is \textbf{LI}.
\end{namedthm*}

\begin{namedthm*}{Theorem 5.2.8}
	Consider \(S = \{\vect{u_1}, \vect{u_2}, \dots, \vect{u_k}\}\) is a basis for a vector space \(V\), then for any vector \(\vect{w}\) in \(V\):
	\begin{enumerate}
		\item If \(S\) is orthogonal, we have
		\[
		    (\vect{w})_S = (\frac{\vect{w}\cdot\vect{u_1}}{\vect{u_1}\cdot\vect{u_1}}\vect{u_1}, \frac{\vect{w}\cdot\vect{u_2}}{\vect{u_2}\cdot\vect{u_2}}\vect{u_2}, \dots, \frac{\vect{w}\cdot\vect{u_k}}{\vect{u_k}\cdot\vect{u_k}}\vect{u_k})
		\]
		\item If \(S\) is orthonomal, we have
		\[
		    (\vect{w})_S = (\vect{w}\cdot\vect{u_1}, \vect{w}\cdot\vect{u_2}, \dots, \vect{w}\cdot\vect{u_k})
		\]
	\end{enumerate}
    \textbf{T5.2.15}: \((\vect{w})_S\) is the projection of \(\vect{w}\) onto \(V\) if \(\vect{w} \in \mathbb{R}^n \land V\) is a subspace of \(\mathbb{R}^n\) (condition of \(\vect{w}\) changed but same formula applies).
\end{namedthm*}

\begin{namedthm*}{Theorem 5.2.19 (Gram-Schmidt Process)}
	Let {\(\vect{u_1}, \vect{u_2}, \dots, \vect{u_k}\)} be a basis for a vector space \(V\). Let
	\begin{gather*}
	    \vect{v_1} = \vect{u_1},\\
	    \vect{v_2} = \vect{u_2} - \frac{\vect{u_2}\cdot\vect{v_1}}{\vect{v_1}\cdot\vect{v_1}}\vect{v_1},\\
	    \vect{u_3} = \vect{u_3} - \frac{\vect{u_3}\cdot\vect{v_1}}{\vect{v_1}\cdot\vect{v_1}}\vect{v_1} - \frac{\vect{u_3}\cdot\vect{v_2}}{\vect{v_2}\cdot\vect{v_2}}\vect{v_2},\\
	    \vdots
	\end{gather*}
	Then {\(\vect{v_1}, \vect{v_2}, \dots, \vect{v_k}\)} is an orthogonal basis for \(V\). Normalize all vectors in it then we have a orthonormal basis for \(V\).
\end{namedthm*}

\begin{namedthm*}{Definition 5.3.6}
	Let \(\matr{A}\vect{x} = \vect{b}\) be a linear system where \(\matr{A}\) is an \(m * n\) matrix. A vector \(\vect{u} \in \mathbb{R}^n\) is called a \textbf{least squares solution} to the linear system if \(\forall \vect{u} \in \mathbb{R}^n, \norm{\vect{b} - \matr{A}\vect{u}} \le \norm{\vect{b} - \matr{A}\vect{v}}\).
\end{namedthm*}

\begin{namedthm*}{Theorem 5.3.8}
	Continuing \textbf{D5.3.6}, let \(\vect{p}\) be the projection of \(\vect{b}\) onto the column space of \(\matr{A}\). \(\vect{u}\) is the least squares solution iff. \(\matr{A}\vect{u} = \vect{p}\).
\end{namedthm*}

\begin{namedthm*}{Theorem 5.3.10}
	Continuing \textbf{D5.3.6}, \(\vect{u}\) is the least squares solution iff. \(\vect{u}\) is a solution to \(\matr{A}^T\matr{A}\vect{x} = \matr{A}^T\vect{b}\).
\end{namedthm*}

\begin{namedthm*}{D5.4.3/R5.4.4/T5.4.6}
	\(\matr{A}\) is a square matrix of order \(n\). The following are equivalent:
	\begin{enumerate}
		\item \(\matr{A}\) is orthogonal
		\item \(\matr{A}^{-1} = \matr{A}^T\)
		\item \(\matr{A}\matr{A}^T = \matr{A}^T\matr{A} = \matr{I}\)
		\item The rows of \(\matr{A}\) form an \textbf{orthonormal} basis for \(\mathbb{R}^n\)
		\item The columns of \(\matr{A}\) form an \textbf{orthonormal} basis for \(\mathbb{R}^n\)
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Theorem 5.4.7}
	Let \(S\) and \(T\) be two \textbf{orthonormal} bases for a vector space and let \(\matr{P}\) be the transition matrix from \(S\) to \(T\). Then \(\matr{P}\) is orthogonal and \(\matr{P}^T\) is the transition matrix from \(T\) to \(S\).
\end{namedthm*}

\section{Diagonalization}

\begin{namedthm*}{Definition 6.1.3}
	\(\matr{A}\) is a square matrix of order \(n\). \(\vect{u} \in \mathbb{R}^n\) is an non-zero column vector that satisfies:
	\[
	    \matr{A}\vect{u} = \lambda \vect{u}
	\]
	for some scalar \(\lambda\). \(\lambda\) is called an \textbf{eigenvalue} of \(\matr{A}\). \(\vect{u}\) is said to be an \textbf{eigenvector} of \(\matr{A}\) \textbf{associated} with the eigenvalue \(\lambda\).
\end{namedthm*}

\begin{namedthm*}{Theorem 6.1.9}
	If \(\matr{A}\) is triangular, the eigenvalues of \(\matr{A}\) are the diagonal entries of \(\matr{A}\).
\end{namedthm*}

\begin{namedthm*}{Remark 6.2.5}
	Suppose the characteristic polynomial of the matrix \(\matr{A}\) can be factorized as
	\[
	    \det(\lambda\matr{I} - \matr{A}) = (\lambda - \lambda_1)^{r_1}(\lambda - \lambda_2)^{r_2}\dots(\lambda - \lambda_k)^{r_k}
	\]
	where \(\lambda_1, \lambda_2, \dots, \lambda_k\) are distinct eigenvalues of \(\matr{A}\). Then for each eigenvalue \(\lambda_i\),
	\[
	    \dim(E_{\lambda_i}) \le r_i
	\]
	Furthermore, \(\matr{A}\) is diagonalizable iff. \(\forall 1 \le i \le k, \dim(E_{\lambda_i}) = r_i\).
\end{namedthm*}

\begin{namedthm*}{Definition 6.3.2/T*.4}
	A square matrix \(\matr{A}\) is said to be orthogonally diagonalizable iff. there exists an orthogonal matrix \(\matr{P}\) such that \(\matr{P}^T\matr{A}\matr{P}\) is diagonal.

	A square matrix is orthogonally diagonalizable iff. it is \textbf{symmetric}.
\end{namedthm*}

\begin{namedthm*}{Algorithm 6.3.5}
	Similar to the process for the normal matrix, orthogonal matrix \(\matr{P}\) can be found by using vectors of \(T\) as \textbf{its columns} where \(T = T_{\lambda_1} \cup T_{\lambda_2} \cup \dots \cup T_{\lambda_k}\) and \(T_{\lambda_i}\) is transformed from \(S_{\lambda_1}\) using Gram-Schmidt Process.
\end{namedthm*}

\section{Linear Transformation}

\begin{namedthm*}{Theorem 7.1.4}
	Let \(T\) be a linear transformation, we have:
	\begin{enumerate}
		\item \(T(\vect{0}) = \vect{0}\)
		\item \(T\) is closed under scalar multiplication and addition
	\end{enumerate}
\end{namedthm*}

\begin{namedthm*}{Discussion 7.1.8}
	Let \(T: \mathbb{R}^n \rightarrow \mathbb{R}^m\) be a linear transformation with the standard matrix \(\matr{A}\). Let \(\{\vect{e_1}, \vect{e_2}, \dots, \vect{e_n}\}\) be the standard basis for \(\mathbb{R}^n\). We then have:
    \[
        \matr{A} =
        \begin{pmatrix}
        T(\vect{e_1}) & T(\vect{e_2}) & \dots & T(\vect{e_n})
        \end{pmatrix}
    \]
\end{namedthm*}

\begin{namedthm*}{Theorem 7.2.4}
	Continuing \textbf{D7.1.8}. We have:
	\[
	    R(T) = \lspan\{T(\vect{e_1}), T(\vect{e_2}), \dots, T(\vect{e_n})\} = \text{the column space of }\vect{A}
	\]
	which is a subspace of \(\mathbb{R}^m\)
\end{namedthm*}

\begin{namedthm*}{D7.2.5/T7.2.9/D7.2.10/T7.2.12}
	Continuing \textbf{T7.2.4}. We have:
	\begin{itemize}
		\item \(\rank(T) = \dim(R(T)) = \rank(\matr{A})\)
		\item \(\nullity(T) = \nullity(\matr{A})\)
		\item \(\rank(T) + \nullity(T) = n\)
		\item \(\ker(T) = \text{the nullspace of }\matr{A}\)
	\end{itemize}
\end{namedthm*}
\end{justifying}
\end{multicols}

\end{document}


\subsection{Correctness}

An algorithm is correct if it produces the expected output for each input.

\subsection{Partial and total correctness}

An algorithm is only partially correct if may not terminate. Otherwise it is totally correct.


\documentclass[11pt]{article}
\usepackage{fullpage}
\usepackage{url}
\usepackage{color}
\usepackage{amsmath, amssymb, bm}
\usepackage{ulem}
\usepackage{framed}
\usepackage{xcolor}

\textheight=8.85in

\pagestyle{myheadings}

\setlength{\tabcolsep}{0in}
\begin{document}

\thispagestyle {empty}

\newcommand{\lsp}[1]{\large\renewcommand{\baselinestretch}{#1}\normalsize}
\newcommand{\hsp}{\hspace{.2in}}
\newcommand{\comment}[1]{}
\newtheorem{thm}{Theorem}[section]
\newtheorem{lem}{Lemma}[section]
\newtheorem{cor}{Corollary}[section]
\newtheorem{prop}{Proposition}[section]
\newtheorem{problem}{Problem}[section]

\newcommand{\R}{{\rm\hbox{I\kern-.15em R}}}
\newcommand{\IR}{{\rm\hbox{I\kern-.15em R}}}
\newcommand{\IN}{{\rm\hbox{I\kern-.15em N}}}
\newcommand{\IZ}{{\sf\hbox{Z\kern-.40em Z}}}
\newcommand{\IS}{{\rm\hbox{S\kern-.45em S}}}
\newcommand{\Real}{I\!\!R}

\newcommand{\bPhi}{\bm{\Phi}}
\newcommand{\bphi}{\bm{\phi}}
\newcommand{\bt}{\mathbf{t}}
\newcommand{\bw}{\mathbf{w}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\by}{\mathbf{y}}
\newcommand{\bX}{\mathbf{X}}
\newcommand{\bmm}{\mathbf{m}}
\newcommand{\bS}{\mathbf{S}}
\newcommand{\bL}{\mathbf{L}}
\newcommand{\bM}{\mathbf{M}}

\newcommand{\linesep}{\vspace{.2cm}\hrule\vspace{0.2cm}}
\newcommand{\categorysep}{\vspace{0.5cm}}
\newcommand{\entrysep}{\vspace{0cm}}

\newcommand{\category}[1]{\categorysep
                  \noindent {\bf \large #1}
              \linesep}

\pagestyle{empty}

\begin{center}
{\large \textbf{CSE 847 (Spring 2021): Machine Learning--- Homework 2}} \\
 Instructor: Jiayu Zhou \quad
 Due on Wednesday, Feb 24 11:59 PM Easter Time. \\
 All submissions should be uploaded to D2L.
\end{center}

\section{Linear Algebra II}

\begin{enumerate}
\item (20 points) Compute (by hand) the eigenvalues and the eigenvectors of the following matrix:
$$A = \begin{pmatrix} 2 & 1 & 0 \\ 1 & 2& 0\\ 0 & 0 & 1 \end{pmatrix}.$$
For the eigenvalues:
\begin{center}
      $\det[A - \lambda \mathbf{I} ] = 0$\\
      $\det[\begin{pmatrix} 2-\lambda & 1 & 0 \\ 1 & 2-\lambda& 0\\ 0 & 0 & 1-\lambda \end{pmatrix}] = 0$\\
      $(2-\lambda)[(2-\lambda)(1-\lambda) - 0*0] - 1*[1*(1-\lambda) - 0*0] + 0*[1*0 - 0*(2-\lambda)] = 0$\\
      $(2-\lambda)[(2-\lambda)(1-\lambda)] - (1-\lambda) = 0$\\
      $3 - 7\lambda + 5\lambda^2 - \lambda^3 = 0$\\
      $-(x-3)(x-1)(x-1) = 0$\\
      $\lambda = 3, \lambda = 1$ , with multiplicity of 2\\
\end{center}
For the eigenvectors. First, $\lambda = 3$.
\begin{center}
      $\begin{pmatrix} -1 & 1 & 0 \\ 1 & -1 & 0\\ 0 & 0 & -2 \end{pmatrix}  \begin{pmatrix} u_1  \\ u_2 \\ u_3 \end{pmatrix}= \begin{pmatrix} 0  \\ 0 \\ 0 \end{pmatrix}$\\
      $-u_1 + u_2 = 0, u_1-u_2 = 0, -2u_3 = 0$\\
      $u_1 = u_2 = 1, u_3 = 0$\\
      Eigenvector for $\lambda = 3 = \begin{pmatrix} 1  \\ 1 \\ 0 \end{pmatrix}$
\end{center}
Second and third for $\lambda = 1$.
\begin{center}
      $\begin{pmatrix} 1 & 1 & 0 \\ 1 & 1 & 0\\ 0 & 0 & 0 \end{pmatrix}  \begin{pmatrix} u_1  \\ u_2 \\ u_3 \end{pmatrix}= \begin{pmatrix} 0  \\ 0 \\ 0 \end{pmatrix}$\\
      $u_1 + u_2 = 0, u_1+u_2 = 0, u_3 = ?$\\
      $u_1 = -u_2 and u_2 = u_2 , u_3 = u_3$\\
      Eigenvectors for $\lambda = 1$ is the set $\{ u_2*\begin{pmatrix} -1  \\ 1 \\ 0 \end{pmatrix} + u_3 *\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} \}$
\end{center}
\item Given the three vectors $v_1 = (2, 0, -1), v_2 = (0, -1, 0)$ and $v_3 = (2, 0, 4)$ in $\mathbb R^3$.
\begin{itemize}
\item (10 points) Show that they form an orthogonal set under the standard
      Euclidean inner product for $\mathbb R^3$ but not an orthonormal set. \\
      A) This means that each vector in the set must be perpendicular to each other,
      i.e. $v_i^Tv_j = 0$, but they are not normalized, i.e. $\left\lVert v_i \right\rVert  \neq 0$.
      \begin{center}
            $v_1^Tv_2 = \begin{pmatrix} 2  \\ 0 \\ -1 \end{pmatrix} \begin{pmatrix} 0 , -1, 0 \end{pmatrix} = 2*0 + 0*-1 + 0*-1 = 0$
            $v_2^Tv_3 = \begin{pmatrix} 0  \\ -1 \\ 0 \end{pmatrix} \begin{pmatrix} 2 , 0, 4 \end{pmatrix} = 2*0 + 0*-1 + 0*4 = 0$
            $v_3^Tv_1 = \begin{pmatrix} 2  \\ 0 \\ 4 \end{pmatrix} \begin{pmatrix} 2 , 0, -1 \end{pmatrix} = 2*2 + 0*0 + -1*4 = 0$
            $\left\lVert v_1 \right\rVert = \sqrt{2^2 + 0^2 + (-1)^2 } = \sqrt{5}$
      \end{center}
      So, because the different combinations of the vectors in the set are perpendicular, they are orthogonal. But
      because at least one of the set has a length greater than 1, then the set doesn't form an orthonormal basis.
\item (10 points) Turn them into a set of vectors that will form an orthonormal set of
      vectors under the standard Euclidean inner product for $\mathbb R^3$. \\
      A) This means we need to normalize each of the vectors.
      \begin{center}
            $\left\lVert v_1 \right\rVert = \sqrt{2^2 + 0^2 + (-1)^2 } = \sqrt{5} \Rightarrow v_1 = ( \frac{2\sqrt{5}}{5}, 0, \frac{-\sqrt{5}}{5})$
            $\left\lVert v_2 \right\rVert = \sqrt{0^2 + (-1)^2 + 0^2 } = 1$
            $\left\lVert v_3 \right\rVert = \sqrt{2^2 + 0^2 + 4^2 } = 2\sqrt{5}\Rightarrow v_3 = ( \frac{\sqrt{5}}{5}, 0, \frac{2\sqrt{5}}{5})$
      \end{center}
\end{itemize}

\item (10 points) Suppose that $A$ is an $n \times m$ matrix with linearly independent columns.
      Show that $A^T A$ is an invertible matrix. \\
      A) $A^T A$ creates a new $m \times m$ matrix. In order to be invertible, the created matrix must be
      non-singular, which means that the matrix must be a square, full-rank matrix. Because $A$ has linearly independent columns,
      if the created matrix is linearly independent, then it will be full-rank and non-singular, thus invertible.
      However, this means that linear independence must be transferable through multiplication of two, linearly independent
      matrices. To show this, consider $(A^TA)x = 0$. Matrix multiplication is associative, so $A^T(Ax) = 0$. Let $(Ax) = y$, and
      $A^Ty = 0$. Because $A^T$ is a linearly independent matrix, $A^Ty = 0$ only has the trivial solution of $y = 0$. Substituting
      that into $(Ax) = y$, then $(Ax) = 0$, and because $A$ is linearly independent, the only trivial solution is $x = 0$.
      This means that the combination $A^TAx=0$ has only the trivial solution where $x = 0$, and so, the create matrix is linearly
      independent, non-singular, and therefore, invertible.

\item (10 points) Suppose that $A$ is an $n \times m$ matrix with linearly independent columns.
      Let $\bar x$ be a least squares solution to the system of equations $Ax = b$ (the solution of $\min_x \|Ax - b\|_2^2$).
      Show that $\bar x$ is the \textbf{unique} solution to the associated normal system
      $A^T A \bar x = A^T b$. \\
      A) For this question, we know that $\bar x$ is the least squares solution to the system of equations $Ax = b$. In order for
      the least squares to be associated with the normal system, we need to show that the gradient of the least squares equation
      results in the normal system, and then, we can show uniqueness through the fact that $A^TA$ is non-singular and invertible,
      as discussed above. First, let's derive that normal system from the least squares equation:
      \begin{center}
           $\frac{1}{2}\|Ax - b\|_2^2$ \\
           $\frac{\partial }{\partial x_{j_0}} (\sum_{j=1}^{n} A_{ij}x_j - b_i )^2$ Definition of least squares in element form for one element\\
           $2A_{ij_0}(\sum_{j=1}^{n} A_{ij}x_j - b_i )$ \\
           $\sum_{i=1}^{n}A_{ij_0}(\sum_{j=1}^{n} A_{ij}x_j - b_i )$ Sum across all the elements for least squares \\
           $A^T(Ax - b)$ Back to matrix notation \\
           $A^TAx = A^Tb)$ Distribute and set equal \\
      \end{center}
      So, the gradient of the least squares is the normal system, and so any value $\bar x$ that solves the least
      squares solves the normal system and vice versa. To ensure that the solution is unique, we must ask when
      a solution is unique. We know this happens when $A^TA$ is non-singular, and so, because we know from the
      previous problem that $A^TA$ is non-singular, we can say that this solution is unique.

\end{enumerate}

\section{Linear Regression I}

Questions in the textbook Pattern Recognition and Machine Learning:
\begin{enumerate}
\item (10 points) Page 174, Question 3.2 \\
A) Let $w$ be some vector in the column space of $\Phi$, and $w = \Phi v$. Then, the projection $Pw$ is
\begin{center}
      $Pw = \Phi (\Phi^T\Phi)^{-1}\Phi^T (\Phi v)$\\
      $\Phi (\Phi^T\Phi)^{-1}(\Phi^T \Phi v)$\\
      $\Phi v = w$ Canceling like terms
\end{center}
This means that $\Phi (\Phi^T\Phi)^{-1}\Phi^T$ can take any vector an project it onto the column space of $\Phi$.
Similarly, for least squares, the optimal position for the least squares equation is the projection closest to $t$.
Let $t = t_\Phi + t_{\Phi \perp}$ be the orthogonal decomposition with respect to space $\mathbb{S}$. By definition,
we know that $t_\Phi$ lies in the space $\mathbb{S}$ and there is a coresponding vector $\Phi y$ in $\mathbb{S}$ where
$\Phi y = t_\Phi$ (the projection). We can then say that $t - t_\Phi = t - \Phi y = x_{\Phi \perp}$ (subbing it into the first equation),
which lies in the space $\mathbb{S}^\perp$. Note that the $Col(\Phi)^perp = Nul(\Phi^T)$, which means we can say that
$0 = \Phi^T(t - \Phi y) \Rightarrow \Phi^T t - \Phi^T \Phi y$. Switching some things around, we can see that
$y = (\Phi^T \Phi)^{-1}\Phi^T t$, or the orthogonal projection of $t$.

\item (10 points) Page 175, Question 3.7
From the hint, we know from Bayes' theorem that
$$
p(\bw | \bt) \propto p(\bt | \bw) p(\bw)
$$
where the factors on the r.h.s are given by (3.10) and (3.48), respectively.
\begin{align*}
p(\bt | \mathbf X, \bw, \beta ) &= \prod_{n=1}^N \mathcal N (t_n | \bw^T \bphi(\bx_n), \beta^{-1} )  \tag {3.10}\\
p(\bw) &= \mathcal N (\bw| \mathbf m_0, \bS_0) \tag {3.48}
\end{align*}
then plugin:
\begin{align*}
p(\bw | \bt)
&\propto
{\color{black}\left[ \prod_{n=1}^N \mathcal N(t_n | \bw^T \bphi(\bx_n), \beta^{-1}) \right]}
{\color{black}\mathcal N (\bw | \mathbf m_0, \bS_0)}\\
&\propto
{\color{black}\exp\left( -\frac{\beta}{2} (\bt - \bPhi \bw)^T (\bt - \bPhi \bw) \right) }
{\color{black}\exp\left( -\frac{1}{2}(\bw - \mathbf m_0)^T \bS_0^{-1} (\bw - \mathbf m_0) \right)}
\end{align*}
And then, we can expand out the equations and combine like terms.
\begin{align*}
{\color{black}\exp\left( -\frac{\beta}{2} (\bt^T - \bPhi^T \bw^T) (\bt - \bPhi \bw) -\frac{1}{2}(\bw^T - \mathbf m_0^T) \bS_0^{-1} (\bw - \mathbf m_0) \right)}\\
{\color{black}\exp\left( -\frac{\beta}{2} (\bt^T \bt - 2\bt^T \bPhi \bw + \bw^T \bPhi^T \bPhi \bw) -\frac{1}{2}(\bS_0^{-1} \bw^T \bw - 2\bS_0^{-1} \mathbf m_0^T \bw + \bS_0^{-1} \mathbf m_0^T \mathbf m_0) \right)}\\
{\color{black}\exp\left( -\frac{1}{2} ( \bw^T( \bS_0^{-1} + \beta \bPhi^T \bPhi)\bw  - 2 \bw( \bS_0^{-1} \mathbf m_0^T + \beta \bt^T \bPhi ) + (\beta \bt^T \bt + \bS_0^{-1} \mathbf m_0^T \mathbf m_0) \right)}\\
\end{align*}
Finally, we have to complete the square. First, we start with the initial term, which turns out to be our $\bS_N^{-1}$.
Then, we need to select our new $\bM_N$. There are some additional parameters $C$, but those can be ignored as we will have shown
that this is proportional to (3.49).
\begin{align*}
\bS_N^{-1} &= \bS_0^{-1} + \beta \bPhi^T \bPhi \\
\bM_N &= \bS_N( \bS_0^{-1} \mathbf m_0 + \beta \bt \bPhi^T )\\
{\color{black}\exp\left( -\frac{1}{2} (w - \bM_N)^T \bS_N (w - \bM_N) + C\right)}\\
\end{align*}
From this, you can see that the different equations are proportional.

\item (10 points) Page 175, Question 3.10
With the help of the hint, we can use (3.3), (3.8) and (3.49),
\begin{align*}
y(\bx, \bw) &= \bw^T \bphi(\bx) \tag{3.3}\\
p(t|\bx, \bw, \beta) &= \mathcal N (t|y(\bx, \bw), \beta^{-1}) \tag{3.8}\\
p(\bw | \bt) &= \mathcal N(\bw | \bmm_N, \bS_N) \tag{3.49}
\end{align*}
we can re-write (3.57) as
\begin{align*}
p(t|\bx, \bt, \alpha, \beta)
&= \int p(t| \bx, \bw, \beta) p(\bw | \bt, \alpha, \beta) d \bw
= \int \mathcal N (t | \bphi(\bx)^T \bw, \beta^{-1})
\mathcal N (\bw | \bmm_N, \bS_N) d \bw
\end{align*}
From here, we can see that the first factor of the integrand in rewritten equation is equivalent to the more
generic form described in (2.114):
\begin{align*}
      p(\mathbf y | \bx) = \mathcal N (\by | \mathbf A \bx + \mathbf b, \mathbf L^{-1}) \tag{2.114}
\end{align*}
where $Ax+b = \bw^T \phi x$ and the precision matrix $\mathbf L^{-1} = \beta^{-1}$, or the inverse covariance matrix.
We can then also update the second factor with the more generic form of the distribution described in (2.113):
\begin{align*}
      p(\bx)             = \mathcal N (\bx | \bm{\mu}, \bm \Lambda^{-1}) \tag{2.113}\\
\end{align*}
where $\mu = \bmm_N$, or the mean of the distribution is equivalent to mean of the posterior and $\Lambda^{-1} = \bS_N$
or the precision is equal to the covariance of the posterior. In form below:
\begin{align*}
   \int \mathcal N (y | Ax+b, \bL^{-1})
      \mathcal N (x | \mu, \Lambda^{-1}) d \bx
\end{align*}
Pulling everything together, we can apply the Bayesian process described in 2.3.3 and the mean and covariance of
the distribution $p(y)$ (2.109 and 2.110) to create equivalent of (2.115).
\begin{align*}
   \int \mathcal N (y | Ax+b, \bL^{-1})
      \mathcal N (x | \mu, \Lambda^{-1}) d \bx = \mathcal N (y | \mathbb{E} [y], COV[y]) =
      \mathcal N (y | \mathbf{A}\mu + b, \bL^{-1} + \mathbf{A} \Lambda^{-1} \mathbf{A}^T)
\end{align*}

\item (10 points) Page 175, Question 3.11
We can show this by looking at functions, (3.59) and (3.54), after both are adjusted for the next iteration (Thanks hint!).
The two functions are below:
\begin{align*}
      \bS_{N+1}^{-1} &= \bS_N^{-1} + \beta \bphi_{N+1} \bphi_{N+1}^T. \tag{3.54}\\
      \sigma_{N+1}^2 (\bx) &= \frac{1}{\beta} + \bphi(\bx)^T \bS_{N+1} \bphi(\bx) \tag{3.59}
\end{align*}
We can then substitute (3.54) into (3.59) and simplify to show the relation between $\sigma_{N+1}^2 (\bx)$ and $\sigma_{N}^2 (\bx)$.
After we substitute $\bS_{N+1}^{-1}$, we can use (3.110) to help with the simplification. Finally, we distribute $\bphi(\bx)$
to finish creating the relation.
\begin{align*}
      \sigma_{N+1}^2 (\bx) = \frac{1}{\beta} + \bphi(\bx)^T \bS_{N+1} \bphi(\bx)\\
      \frac{1}{\beta} + \bphi(\bx)^T (\bS_{N}^{-1} + \beta \bphi_{N+1}(\bx) \bphi_{N+1}(\bx)^T)^{-1} \bphi(\bx)\\
      \frac{1}{\beta} + \bphi(\bx)^T (\bS_{N} - \frac{(\bS_{N} \beta \bphi_{N+1}(\bx))(\bphi_{N+1}(\bx)^T \bS_{N})}
      {1+\bphi_{N+1}(\bx)^T\bS_{N}\beta \bphi_{N+1}(\bx)}) \bphi(\bx)\\
      \frac{1}{\beta} + \bphi(\bx)^T \bS_{N} \bphi(\bx) - \bphi(\bx) \frac{(\bS_{N} \beta \bphi_{N+1}(\bx))(\bphi_{N+1}(\bx)^T \bS_{N})}
      {1+\bphi_{N+1}(\bx)^T\bS_{N}\beta \bphi_{N+1}(\bx)}\\
      \sigma_{N+1}^2 (\bx) \leq \sigma_{N}^2 (\bx) - \bphi(\bx) \frac{(\bS_{N} \beta \bphi_{N+1}(\bx))(\bphi_{N+1}(\bx)^T \bS_{N})}
      {1+\bphi_{N+1}(\bx)^T\bS_{N}\beta \bphi_{N+1}(\bx)}
\end{align*}
So, because the next iteration reduces the current iteration, the next iteration will always be smaller than the current.
\end{enumerate}

\end{document}


\section{Sorting the Input}
\label{sec:sorting}
As mentioned in section \ref{sec:intro_real}, the simplest solution seems to be
ignoring the dynamic nature of the problem. We could simply add each input pair
$(j,r)$ to a list, at $O(1)$ cost per input, and then when queried:
\begin{enumerate}
	\item sort the list by movie $j$ at cost $O(|S| \log |S|)$
	\item calculate the averages at cost $O(|S|)$ (as the values relevant to
		each average are now adjacent)
	\item sort the resulting list by the averages at cost $O(|U| \log |U|)$
\end{enumerate}

This is obviously bad, as queries will take $O(|S| \log |S| + |U| \log|U|)$
time.
\subsection{Sorting dynamically}
Reassuming the dynamic approach, we can trivially improve this algorithm
by maintaining a hash table of running averages, updating the relevant value for
each input. Now queries are just $O(|U| \log |U|)$, and processing each input
element is $O(1)$. This is the lower bound for comparison based sorting, but we
may choose to prioritize a faster query time, in exchange for slower per-element
processing. This can be done by maintaining an always-ordered data structure of
$(j, \overline r)$ pairs, ordered by $\overline r$, being the running average of
ratings for $j$. While we can obviously do this using insertion sort, we can
improve on the resulting $O(|U|)$ per-element cost, without degrading the $O(|U|)$
query time it provides. This can be achieved by maintaining a binary search-tree
instead of a list, taking care not to delete nodes with equal average, if they
pertain to different movies, and to never keep more than one node for each
movie.

For efficient querying we need the tree to be ordered by the
running averages, but for updating the averages we need an efficient lookup
based on the movie id (the stored average cannot be known solely from the input
element). We additionally maintain a hash table, mapping movies to stored
averages or, equivalently, to pointers to nodes in the tree. This provides a
solution with $O(\log |U|)$ per-element processing time, $O(|U|)$ query
time, and $(|U|)$ memory consumption.

%\section{Heap and Min-Max Heap}
%
%From \texttt{Heap, Wikipedia}:
%\begin{quote}Heap is a specialized tree-based data structure that satisfies the heap property:
%If A is a parent node of B then the key of node A is ordered with respect to the key of node
%B with the same ordering applying across the heap.
%\end{quote}
%
%Ordering items using Heap takes $O(n\log{}n)$. Where creation of a heap takes
%$O(n)$. Then we are traversing through heap n times using fuction which takes
%$O(\log{}n)$. Which results in $O(n + n\log{}n)$, which leads to our prevoiusly
%stated complexity.
%
%Also Binary Heap can offer better average insertion time than BST which may
%be important factor in some scenarions when the data are often updated but we
%still need to able to acess the sorting results.
%
%In scenario when we are only interested in Top and Low k elements of a set we might takes
%different heap based approach called Min-Max Heap.
%
%Provides properties of regular heap like a:
%\begin{itemize}
%\item $O(n)$ built time,
%\item $O(\log{}n)$ for insertion and deletion
%\end{itemize}
%
%From \texttt{Min-max heap, Wikipedia}:
%\begin{quote}Heap is a specialized tree-based data structure that satisfies the heap property:
%If A is a parent node of B then the key of node A is ordered with respect to the key of node
%B with the same ordering applying across the heap.
%\end{quote}
%
%But even more important are properties carried by Min-Max variation. Which are
%guaranteeing us a constant time for Min and Max operations. The Top/Low(k)
%finding operation can be performed in $O(k\log{}k)$ time.
%
%Using this approach requires to store all input elements. It takes $O(n)$ for
%large sets.
%
%Based on : \texttt{M. D. Atkinson, J.R. Sack, N. Santoro, and T. Strothotte,
%Communications of the ACM,
%October, 1986,
%Min-Max Heaps and Generalized Priority Queues}


%% LyX 2.2.3 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[12pt,english]{article}
\usepackage{lmodern}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage[a4paper]{geometry}
\geometry{verbose,tmargin=1.5cm,bmargin=1.5cm,lmargin=1.5cm,rmargin=1.5cm,headheight=1.5cm,headsep=1.5cm,footskip=1.5cm}
\usepackage{amsmath}
\usepackage{graphicx}

\makeatletter

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
%% Because html converters don't know tabularnewline
\providecommand{\tabularnewline}{\\}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
\usepackage{lmodern}
\usepackage[T1]{fontenc}

\makeatother

\usepackage{babel}
\begin{document}

\title{EE6132: Deep Learning for Image Processing}

\author{\textbf{Assignment 1: MNIST Classification using Multilayer Perceptron}}

\date{Adarsh B (MM14B001)}

\maketitle
\newpage{}

\tableofcontents{}

\newpage{}

\section{Overview}

Following is a pictorial description of the Multilayer Perceptron
model used for training and classifying on MNIST data:
\begin{center}
\includegraphics[width=16cm]{/home/adarsh/mlp}
\par\end{center}

\begin{center}
\begin{tabular}{|c|c|}
\hline
Minibatch size & 64\tabularnewline
\hline
Regularization & l2\tabularnewline
\hline
Regularization parameter ($\lambda$) & 0.005\tabularnewline
\hline
No. of training iterations & 8000\tabularnewline
\hline
Update algorithm & SGD with Momentum acceleration\tabularnewline
\hline
\end{tabular}
\par\end{center}

\rule[0.5ex]{0.9\columnwidth}{1pt}

\bigskip{}

\section{Submissions}

\subsection{Backpropagation equations}

Feedforwarding in the neural network takes place as follows (f can
be ReLU or sigmoid):

\begin{align*}
a_{i}^{1} & =x_{i}\\
z_{i}^{2} & =(w_{ij}^{1}a_{j}^{1}+b_{i}^{1})\\
a_{i}^{2} & =f(z_{i}^{2})\\
z_{i}^{3} & =(w_{ij}^{2}a_{j}^{2}+b_{i}^{2})\\
a_{i}^{3} & =f(z_{i}^{3})\\
z_{i}^{4} & =(w_{ij}^{3}a_{j}^{3}+b_{i}^{3})\\
a_{i}^{4} & =f(z_{i}^{4})\\
z_{i}^{5} & =w_{ij}^{4}a_{j}^{4}+b_{i}^{4}\\
a_{i}^{5} & =linear(z_{i}^{2})\\
\hat{y}_{i} & =softmax(a_{i}^{5})
\end{align*}

For taking momentum into account, we initialize $v^{i}$ and $u^{i}$
as the velocity terms for $w^{i}$ and $b^{i}$ respectively. $\text{\ensuremath{\delta}}^{i}$
is the error derivative with respect to $a^{i}$. \ensuremath{\alpha}
is the learning rate, $\lambda$ is the regularization parameter,
and \ensuremath{\mu} is the momentum parameter.

\bigskip{}

\noindent \textbf{Final layer:}

\noindent Since we are using cross-entropy as the loss function, the
derivative for cross entropy loss with respect to the final activation
can be evaluated as:

\begin{align*}
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}z_{i}^{5}} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}\hat{y}_{i}}\frac{\text{\ensuremath{\partial}}\hat{y}_{i}}{\text{\ensuremath{\partial}}z_{i}^{5}}\\
 & =-\frac{(y_{i}-\hat{y_{i}})}{\hat{y_{i}}(1-\hat{y}_{i})}\hat{y_{i}}(1-\hat{y}_{i})\\
 & =\boldsymbol{(\hat{y_{i}}-y_{i})}
\end{align*}

\noindent
\begin{align*}
\text{\ensuremath{\delta}}_{i}^{5} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}z_{i}^{5}}\\
 & =(\hat{y_{i}}-y_{i})\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{4}} & =\text{\ensuremath{\delta}}_{i}^{5}\frac{\text{\ensuremath{\partial}}z_{i}^{5}}{\text{\ensuremath{\partial}}w_{ij}^{4}}\\
 & =\boldsymbol{(\hat{y_{i}}-y_{i})a_{j}^{4}+\lambda w_{ij}^{4}}\\
v_{ij}^{4} & =\text{\ensuremath{\mu}}v_{ij}^{4}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{4}}\\
w_{ij}^{4} & =\boldsymbol{w_{ij}^{4}+v_{ij}^{4}}\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{4}} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}z_{i}^{5}}\frac{\text{\ensuremath{\partial}}z_{i}^{5}}{\text{\ensuremath{\partial}}b_{i}^{4}}\\
 & =\boldsymbol{(\hat{y_{i}}-y_{i})}\\
u_{i}^{4} & =\text{\ensuremath{\mu}}u_{ij}^{4}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{4}}\\
b_{i}^{4} & =\boldsymbol{b_{i}^{4}+u_{i}^{4}}
\end{align*}

\noindent \textbf{Hidden layer 3:}
\begin{align*}
\text{\ensuremath{\delta}}_{i}^{4} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}a_{i}^{4}}\\
 & =(\text{\ensuremath{\delta}}_{j}^{5}w_{ji}^{4})f'(z_{i}^{4})\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{3}} & =\text{\ensuremath{\delta}}_{i}^{4}\frac{\text{\ensuremath{\partial}}a_{i}^{4}}{\text{\ensuremath{\partial}}w_{ij}^{3}}\\
 & =\boldsymbol{((\text{\ensuremath{\delta}}_{j}^{5}w_{ji}^{4})f'(z_{i}^{4}))a_{j}^{3}+\lambda w_{ij}^{3}}\\
v_{ij}^{3} & =\text{\ensuremath{\mu}}v_{ij}^{3}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{3}}\\
w_{ij}^{3} & =\boldsymbol{w_{ij}^{3}+v_{ij}^{3}}\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{4}} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}a_{i}^{5}}\frac{\text{\ensuremath{\partial}}a_{i}^{5}}{\text{\ensuremath{\partial}}b_{i}^{4}}\\
 & =\boldsymbol{((\text{\ensuremath{\delta}}_{j}^{5}w_{ji}^{4})f'(z_{i}^{4}))}\\
u_{i}^{3} & =\text{\ensuremath{\mu}}u_{ij}^{3}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{3}}\\
b_{i}^{3} & =\boldsymbol{b_{i}^{3}+u_{i}^{3}}
\end{align*}

\noindent \textbf{Hidden layer 2:}
\begin{align*}
\text{\ensuremath{\delta}}_{i}^{3} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}a_{i}^{3}}\\
 & =(\text{\ensuremath{\delta}}_{j}^{4}w_{ji}^{3})f'(z_{i}^{3})\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{2}} & =\text{\ensuremath{\delta}}_{i}^{3}\frac{\text{\ensuremath{\partial}}a_{i}^{3}}{\text{\ensuremath{\partial}}w_{ij}^{2}}\\
 & =\boldsymbol{((\text{\ensuremath{\delta}}_{j}^{4}w_{ji}^{3})f'(z_{i}^{3}))a_{j}^{2}+\lambda w_{ij}^{2}}\\
v_{ij}^{2} & =\text{\ensuremath{\mu}}v_{ij}^{2}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{2}}\\
w_{ij}^{2} & =\boldsymbol{w_{ij}^{2}+v_{ij}^{2}}\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{3}} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}a_{i}^{4}}\frac{\text{\ensuremath{\partial}}a_{i}^{4}}{\text{\ensuremath{\partial}}b_{i}^{3}}\\
 & =\boldsymbol{((\text{\ensuremath{\delta}}_{j}^{4}w_{ji}^{3})f'(z_{i}^{3}))}\\
u_{i}^{2} & =\text{\ensuremath{\mu}}u_{ij}^{2}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{2}}\\
b_{i}^{2} & =\boldsymbol{b_{i}^{2}+u_{i}^{2}}
\end{align*}
\textbf{Hidden layer 1:}
\begin{align*}
\text{\ensuremath{\delta}}_{i}^{2} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}a_{i}^{2}}\\
 & =(\text{\ensuremath{\delta}}_{j}^{3}w_{ji}^{2})f'(z_{i}^{2})\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{1}} & =\text{\ensuremath{\delta}}_{i}^{2}\frac{\text{\ensuremath{\partial}}a_{i}^{2}}{\text{\ensuremath{\partial}}w_{ij}^{1}}\\
 & =\boldsymbol{((\text{\ensuremath{\delta}}_{j}^{3}w_{ji}^{2})f'(z_{i}^{2}))a_{j}^{1}+\lambda w_{ij}^{1}}\\
v_{ij}^{1} & =\text{\ensuremath{\mu}}v_{ij}^{1}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}w_{ij}^{1}}\\
w_{ij}^{1} & =\boldsymbol{w_{ij}^{1}+v_{ij}^{1}}\\
\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{2}} & =\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}a_{i}^{3}}\frac{\text{\ensuremath{\partial}}a_{i}^{3}}{\text{\ensuremath{\partial}}b_{i}^{2}}\\
 & =\boldsymbol{((\text{\ensuremath{\delta}}_{j}^{3}w_{ji}^{2})f'(z_{i}^{2}))}\\
u_{i}^{1} & =\text{\ensuremath{\mu}}u_{ij}^{1}-\text{\ensuremath{\alpha}}\frac{\text{\ensuremath{\partial}}E}{\text{\ensuremath{\partial}}b_{i}^{1}}\\
b_{i}^{1} & =\boldsymbol{b_{i}^{1}+u_{i}^{1}}
\end{align*}

\pagebreak{}

\subsection{Learning curve plots}

Following is the plot of train and test loss vs iterations for a network
with Sigmoid activations:
\begin{center}
\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/train_test_loss_sigmoid_1e-2}
\par\end{center}

Following plots show the comparison of loss evolution between different
learning rates:
\begin{center}
\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/train_loss_sigmoid_lr_comparison}\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/test_loss_sigmoid_lr_comparison}
\par\end{center}

The test accuracies for various cases is tabulated as shown. Clearly,
it can be observed that convergence is faster in case of higher learning
rates.
\begin{center}
\begin{tabular}{|c|c|}
\hline
\textbf{alpha} & \textbf{Accuracy}\tabularnewline
\hline
\hline
1e-2 & 94.89\%\tabularnewline
\hline
1e-3 & 90.49\%\tabularnewline
\hline
1e-4 & 84.36\%\tabularnewline
\hline
\end{tabular}
\par\end{center}

\pagebreak{}

\subsection{Learning rate scheduling}

Learning rate has been decayed by a factor of 0.85 for every 250 iterations.
The comparison between scheduled and unscheduled learning rates is
turned in below:
\begin{center}
\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/train_loss_sigmoid_decay_comparison}\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/test_loss_sigmoid_decay_comparison}
\par\end{center}

The accuracies obtained in both the cases are tabulated bwlow:
\begin{center}
\begin{tabular}{|c|c|}
\hline
Experiment & Accuracy\tabularnewline
\hline
\hline
Constant learning rate & 94.89\%\tabularnewline
\hline
Scheduled learning rate & 90.60\%\tabularnewline
\hline
\end{tabular}
\par\end{center}

Despite having a lower accuracy in case of scheduled learning rate,
the convergence is smooth. With a high fixed learning rate, the system
can be thought of having too much \textbf{kinetic energy} and the
parameters oscillate around rapidly, unable to settle down into deeper,
but narrower parts of the loss function. This can be seen in the unsteady
green line from the test loss plot. Sometimes, the loss function sometimes
can even get stagnated at a particular value. But with a decaying
learning rate, \textbf{deeper} along with \textbf{smoother convergence}
over long iterations helps the network to learn better.

\subsection{Experimenting with ReLU activation function}

Following is the plot of train and test loss vs iterations for a network
with ReLU activations:
\begin{center}
\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/train_test_loss_relu_1e-2}
\par\end{center}

\pagebreak{}

Following plots show the comparison of loss evolution between different
learning rates:
\begin{center}
\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/train_loss_relu_lr_comparison}\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/test_loss_relu_lr_comparison}
\par\end{center}

The test accuracies for various cases is tabulated as shown. Again,
it can be observed that convergence is faster in case of higher learning
rates.
\begin{center}
\begin{tabular}{|c|c|}
\hline
\textbf{alpha} & \textbf{Accuracy}\tabularnewline
\hline
\hline
1e-2 & 97.79\%\tabularnewline
\hline
1e-3 & 96.87\%\tabularnewline
\hline
1e-4 & 94.28\%\tabularnewline
\hline
\end{tabular}
\par\end{center}

\noindent \textbf{Comparison between ReLU and Sigmoid:}

Following plots depict the convergence comparison between ReLU and
sigmoid activations for alpha=1e-2.
\begin{center}
\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/train_sigmoid_relu_comparison}\includegraphics[width=9cm]{/home/adarsh/PA1_MM14B001/outputs/test_sigmoid_relu_comparison}
\par\end{center}

Clearly, ReLU converges faster as compared to sigmoid activation.
Test accuracies are also higher for ReLU than sigmoid (97.79\% for
ReLU and 94.89\% for Sigmoid).

\subsection{Sample predictions}
\begin{center}
\begin{tabular}{|c|c|}
\hline
\textbf{Sigmoid} & \textbf{ReLU}\tabularnewline
\hline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_1} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_1}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_2} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_2}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_3} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_3}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_4} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_4}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_5} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_5}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_6} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_6}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_7} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_7}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_8} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_8}\tabularnewline
\hline
\end{tabular}%
\begin{tabular}{|c|c|}
\hline
\textbf{Sigmoid} & \textbf{ReLU}\tabularnewline
\hline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_9} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_9}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_10} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_10}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_11} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_11}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_12} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_12}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_13} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_13}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_14} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_14}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_15} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_15}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_16} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_16}\tabularnewline
\hline
\end{tabular}
\par\end{center}

\begin{center}
\begin{tabular}{|c|c|}
\hline
\textbf{Sigmoid} & \textbf{ReLU}\tabularnewline
\hline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_17} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_17}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_18} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_18}\tabularnewline
\hline
\end{tabular}%
\begin{tabular}{|c|c|}
\hline
\textbf{Sigmoid} & \textbf{ReLU}\tabularnewline
\hline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_19} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_19}\tabularnewline
\hline
\includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/sigmoid/sample_20} & \includegraphics[width=4cm]{/home/adarsh/PA1_MM14B001/outputs/relu/sample_20}\tabularnewline
\hline
\end{tabular}
\par\end{center}
\end{document}