\documentclass[a4paper, 12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage[]{amsfonts}
\usepackage[]{graphicx}

\title{CS231A Course Notes 1: Camera Models}
\author{Kenji Hata and Silvio Savarese}
\date{}

\renewcommand\emph{\textbf}

\begin{document}

\maketitle

\section{Introduction}
The camera is one of the most essential tools in computer vision. It is the mechanism by which we can record the world around us and use its output - photographs - for various applications. Therefore, one question we must ask in introductory computer vision is: how do we model a camera?

\section{Pinhole cameras}
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-1.pdf}
\caption{A simple working camera model: the pinhole camera model.}
\label{fig:simpleCamera}
\end{figure}
Let's design a simple camera system -- a system that can record an image of an object or scene in the 3D world. This camera system can be designed by placing a barrier with a small aperture between the 3D object and a photographic film or sensor. As Figure~\ref{fig:simpleCamera} shows, each point on the 3D object emits multiple rays of light outwards. Without a barrier in place, every point on the film will be influenced by light rays emitted from every point on the 3D object. Due to the barrier, only one (or a few) of these rays of light passes through the aperture and hits the film. Therefore, we can establish a one-to-one mapping between points on the 3D object and the film. The result is that the film gets exposed by an ``image" of the 3D object by means of this mapping. This simple model is known as the \emph{pinhole camera model}.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-2.pdf}
\caption{A formal construction of the pinhole camera model.}
\label{fig:pinholeCamera}
\end{figure}
A more formal construction of the pinhole camera is shown in Figure~\ref{fig:pinholeCamera}. In this construction, the film is commonly called the \emph{image or retinal plane}. The aperture is referred to as the \emph{pinhole} $O$ or \emph{center of the camera}. The distance between the image plane and the pinhole $O$ is the \emph{focal length} $f$. Sometimes, the retinal plane is placed between $O$ and the 3D object at a distance $f$ from $O$. In this case, it is called the \emph{virtual image} or \emph{virtual retinal plane}. Note that the projection of the object in the image plane and the image of the object in the virtual image plane are identical up to a scale (similarity) transformation.

Now, how do we use pinhole cameras? Let $P = \begin{bmatrix}x & y & z\end{bmatrix}^T$ be a point on some 3D object visible to the pinhole camera. $P$ will be mapped or \textbf{projected} onto the image plane $\Pi'$, resulting in point\footnote{Throughout the course notes, let the prime superscript (e.g. $P'$) indicate that this point is a projected or complementary point to the non-superscript version. For example, $P'$ is the projected version of $P$.} $P' = \begin{bmatrix}x' & y'\end{bmatrix}^T$. Similarly, the pinhole itself can be projected onto the image plane, giving a new point $C'$.

Here, we can define a coordinate system $\begin{bmatrix}i & j & k\end{bmatrix}$ centered at the pinhole $O$ such that the axis $k$ is perpendicular to the image plane and points toward it. This coordinate system is often known as the \emph{camera reference system} or \emph{camera coordinate system}. The line defined by $C'$ and $O$ is called the \emph{optical axis} of the camera system.

Recall that point $P'$ is derived from the projection of 3D point $P$ on the image plane $\Pi'$. Therefore, if we derive the relationship between 3D point $P$ and image plane point $P'$, we can understand how the 3D world imprints itself upon the image taken by a pinhole camera. Notice that triangle $P'C'O$ is similar to the triangle formed by $P$, $O$ and $(0,0,z)$. Therefore, using the law of similar triangles we find that:

\begin{equation}
    P' = \begin{bmatrix}x'&y'\end{bmatrix}^T = \begin{bmatrix}f\frac{x}{z} & f\frac{y}{z}\end{bmatrix}^T
\end{equation}

Notice that one large assumption we make in this pinhole model is that the aperture is a single point. In most real world scenarios, however, we cannot assume the aperture can be infinitely small. Thus, what is the effect of varying aperture size?

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-3.pdf}
\caption{The effects of aperture size on the image. As the aperture size decreases, the image gets sharper, but darker.}
\label{fig:apertureSize}
\end{figure}

As the aperture size increases, the number of light rays that passes through the barrier consequently increases. With more light rays passing through, then each point on the film may be affected by light rays from multiple points in 3D space, blurring the image. Although we may be inclined to try to make the aperture as small as possible, recall that a smaller aperture size causes less light rays to pass through, resulting in crisper but darker images. Therefore, we arrive at the fundamental problem presented by the pinhole formulation: can we develop cameras that take crisp and bright images?

\section{Cameras and lenses}
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-4.pdf}
\caption{A setup of a simple lens model. Notice how the rays of the top point on the tree converge nicely on the film. However, a point at a different distance away from the lens results in rays not converging perfectly on the film.}
\label{fig:lens}
\end{figure}

In modern cameras, the above conflict between crispness and brightness is mitigated by using \emph{lenses}, devices that can focus or disperse light. If we replace the pinhole with a lens that is both properly placed and sized, then it satisfies the following property: all rays of light that are emitted by some point $P$ are refracted by the lens such that they converge to a single point $P'$ in the image plane. Therefore, the problem of the majority of the light rays blocked due to a small aperture is removed (Figure~\ref{fig:lens}). However, please note that this property does not hold for all 3D points, but only for some specific point $P$. Take another point $Q$ which is closer or further from the image plane than $P$. The corresponding projection into the image will be blurred or out of focus. Thus, lenses have a specific distance for which objects are ``in focus". This property is also related to a photography and computer graphics concept known as depth of field, which is the effective range at which cameras can take clear photos.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-5.pdf}
\caption{Lenses focus light rays parallel to the optical axis into the focal point. Furthermore, this setup illustrates the paraxial refraction model, which helps us find the relationship between points in the image plane and the 3D world in cameras with lenses.}
\label{fig:paraxial}
\end{figure}

Camera lenses have another interesting property: they focus all light rays traveling parallel to the optical axis to one point known as the \emph{focal point} (Figure~\ref{fig:paraxial}). The distance between the focal point and the center of the lens is commonly referred to as the \emph{focal length} $f$. Furthermore, light rays passing through the center of the lens are not deviated. We thus can arrive at a similar construction to the pinhole model that relates a point $P$ in 3D space with its corresponding point $P'$ in the image plane.

\begin{equation}
    P' = \begin{bmatrix}x'\\ y'\end{bmatrix} = \begin{bmatrix}z'\frac{x}{z} \\ z'\frac{y}{z}\end{bmatrix}
\end{equation}

The derivation for this model is outside the scope of the class. However, please notice that in the pinhole model $z' = f$, while in this lens-based model, $z' = f+z_0$.  Additionally, since this derivation takes advantage of the paraxial or ``thin lens" assumption\footnote{For the angle $\theta$ that incoming light rays make with the optical axis of the lens, the paraxial assumption substitutes $\theta$ for any place $\sin(\theta)$ is used. This approximation of $\theta$ for $\sin\theta$ holds as $\theta$ approaches 0.}, it is called the \textbf{paraxial refraction model}.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-6.pdf}
\caption{Demonstrating how pincushion and barrel distortions affect images.}
\label{fig:distortion}
\end{figure}

Because the paraxial refraction model approximates using the thin lens assumption, a number of aberrations can occur. The most common one is referred to as \emph{radial distortion}, which causes the image magnification to decrease or increase as a function of the distance to the optical axis. We classify the radial distortion as \emph{pincushion distortion} when the magnification increases and \emph{barrel distortion}\footnote{Barrel distortion typically occurs when one uses fish-eye lenses.} when the magnification decreases. Radial distortion is caused by the fact that different portions of the lens have differing focal lengths.

\section{Going to digital image space}
In this section, we will discuss the details of the parameters we must account for when modeling the projection from 3D space to the digital images we know. All the results derived will use the pinhole model, but they also hold for the paraxial refraction model.

As discussed earlier, a point $P$ in 3D space can be mapped (or projected) into a 2D point $P'$ in the image plane $\Pi'$. This $\mathbb{R}^3 \rightarrow \mathbb{R}^2$ mapping is referred to as a \emph{projective transformation}. This projection of 3D points into the image plane does not directly correspond to what we see in actual digital images for several reasons. First, points in the digital images are, in general, in a different reference system than those in the image plane. Second, digital images are divided into discrete pixels, whereas points in the image plane are continuous. Finally, the physical sensors can introduce non-linearity such as distortion to the mapping. To account for these differences, we will introduce a number of additional transformations that allow us to map any point from the 3D world to pixel coordinates.

\subsection{The Camera Matrix Model and Homogeneous Coordinates}

\subsubsection{Introduction to the Camera Matrix Model}
The camera matrix model describes a set of important parameters that affect how a world point $P$ is mapped to image coordinates $P'$. As the name suggests, these parameters will be represented in matrix form. First, let's introduce some of those parameters.

The first parameters, $c_x$ and $c_y$, describe how image plane and digital image coordinates can differ by a translation. Image plane coordinates have their origin $C'$ at the image center where the $k$ axis intersects the image plane. On the other hand, digital image coordinates typically have their origin at the lower-left corner of the image. Thus, 2D points in the image plane and 2D points in the image are offset by a translation vector $\begin{bmatrix}c_x, c_y\end{bmatrix}^T$. To accommodate this change of coordinate systems, the mapping now becomes:

\begin{equation}
    P' = \begin{bmatrix}x'\\y'\end{bmatrix} = \begin{bmatrix}f\frac{x}{z}+c_x \\ f\frac{y}{z}+c_y\end{bmatrix}
\end{equation}

The second effect we must account for that the points in digital images are expressed in pixels, while points in image plane are represented in physical measurements (e.g. centimeters). In order to accommodate this change of units, we must introduce two new parameters $k$ and $l$. These parameters, whose units would be something like $\frac{\mathrm{pixels}}{\mathrm{cm}}$, correspond to the change of units in the two axes of the image plane. Note that $k$ and $l$ may be different because the aspect ratio of a pixel is not guaranteed to be one. If $k=l$, we often say that the camera has \emph{square pixels}. We adjust our previous mapping to be

\begin{equation}
    P' = \begin{bmatrix}x'\\y'\end{bmatrix} = \begin{bmatrix}fk\frac{x}{z}+c_x \\ fl\frac{y}{z}+c_y\end{bmatrix} = \begin{bmatrix}\alpha\frac{x}{z}+c_x \\ \beta\frac{y}{z}+c_y\end{bmatrix}
    \label{eq:finalImage}
\end{equation}

Is there a better way to represent this projection from $P\rightarrow P'$? If this projection is a linear transformation, then it can be represented as a product of a matrix and the input vector (in this case, it would be $P$. However, from Equation~\ref{eq:finalImage}, we see that this projection $P\rightarrow P'$ is not linear, as the operation divides one of the input parameters (namely $z$). Still, representing this projection as a matrix-vector product would be useful for future derivations. Therefore, can we represent our transformation as a matrix-vector product despite its nonlinearity? Homogeneous coordinates are the solution.

\subsubsection{Homogeneous Coordinates}

One way to solve this problem is to change the coordinate systems. For example, we introduce a new coordinate, such that any point $P' =(x',y')$ becomes $(x',y',1)$. Similarly, any point $P =(x,y,z)$ becomes $(x,y,z,1)$. This augmented space is referred to as the \emph{homogeneous coordinate system}. As demonstrated previously, to convert a Euclidean vector $(v_1,...,v_n)$ to homogeneous coordinates, we simply append a 1 in a new dimension to get $(v_1,...,v_n,1)$. Note that the equality between a vector and its homogeneous coordinates only occurs when the final coordinate equals one. Therefore, when converting back from arbitrary homogeneous coordinates $(v_1,  ... , v_n , w)$, we get Euclidean coordinates $(\frac{v_1}{w},...,\frac{v_n}{w})$. Using homogeneous coordinates, we can formulate
\begin{equation}
    P_h' = \begin{bmatrix}\alpha x + c_xz\\\beta y + c_yz \\ z\end{bmatrix} =
    \begin{bmatrix}
    \alpha & 0 & c_x & 0\\
    0 & \beta & c_y & 0 \\
    0 & 0 & 1 & 0
    \end{bmatrix}
    \begin{bmatrix}x\\y\\z\\1\end{bmatrix} =     \begin{bmatrix}
    \alpha & 0 & c_x & 0\\
    0 & \beta & c_y & 0 \\
    0 & 0 & 1 & 0
    \end{bmatrix} P_h
    \label{eq:homogeneous}
\end{equation}

From this point on, assume that we will work in homogeneous coordinates, unless stated otherwise. We will drop the $h$ index, so any point $P$ or $P'$ can be assumed to be in homogeneous coordinates. As seen from Equation~\ref{eq:homogeneous}, we can represent the relationship between a point in 3D space and its image coordinates by a matrix vector relationship:
\begin{equation}
    P' = \begin{bmatrix}x'\\y'\\ z\end{bmatrix}=\begin{bmatrix}
    \alpha & 0 & c_x & 0\\
    0 & \beta & c_y & 0 \\
    0 & 0 & 1 & 0
    \end{bmatrix}\begin{bmatrix}x\\y\\z\\1\end{bmatrix}=
    \begin{bmatrix}
    \alpha & 0 & c_x & 0\\
    0 & \beta & c_y & 0 \\
    0 & 0 & 1 & 0
    \end{bmatrix}P = MP
    \label{eq:canonical}
    % = K\begin{bmatrix} I&0\end{bmatrix}P
\end{equation}
We can decompose this transformation a bit further into
\begin{equation}
P' = MP = \begin{bmatrix}
    \alpha & 0 & c_x \\
    0 & \beta & c_y  \\
    0 & 0 & 1
    \end{bmatrix}\begin{bmatrix}I & 0\end{bmatrix}P = K\begin{bmatrix}I & 0\end{bmatrix}P
    \label{eq:decomposedHomogenousTransform}
\end{equation}

The matrix $K$ is often referred to as the \emph{camera matrix}.

\subsubsection{The Complete Camera Matrix Model}

The camera matrix $K$ contains some of the critical parameters that describes a camera's characteristics and its model, including the $c_x, c_y, k,$ and $l$ parameters as discussed above. Two parameters are currently missing this formulation: \emph{skewness} and \emph{distortion}. We often say that an image is skewed when the camera coordinate system is skewed, meaning that the angle between the two axes is slightly larger or smaller than 90 degrees. Most cameras have zero-skew, but some degree of skewness may occur because of sensor manufacturing errors. Deriving the new camera matrix accounting for skewness is outside the scope of this class and we give it to you below:
\begin{equation}
K = \begin{bmatrix}x'\\y'\\ z\end{bmatrix}=\begin{bmatrix}
    \alpha & -\alpha\cot\theta & c_x \\
    0 & \frac{\beta}{\sin\theta} & c_y  \\
    0 & 0 & 1
    \end{bmatrix}
\end{equation}
Most methods that we introduce in this class ignore distortion effects, therefore our class camera matrix $K$ has 5 degrees of freedom: 2 for focal length, 2 for offset, and 1 for skewness. These parameters are collectively known as the \emph{intrinsic parameters}, as they are unique and inherent to a given camera and relate to essential properties of the camera, such as its manufacturing.

\subsection{Extrinsic Parameters}

So far, we have described a mapping between a point $P$ in the 3D camera reference system to a point $P'$ in the 2D image plane using the intrinsic parameters of a camera described in matrix form. But what if the information about the 3D world is available in a different coordinate system? Then, we need to include an additional transformation that relates points from the world reference system to the camera reference system. This transformation is captured by a rotation matrix $R$ and translation vector $T$. Therefore, given a point in a world reference system $P_w$, we can compute its camera coordinates as follows:
\begin{equation}
P = \begin{bmatrix}R&T\\0&1\end{bmatrix} P_w
\end{equation}
Substituting this in equation (\ref{eq:decomposedHomogenousTransform}) and simplifying gives
\begin{equation}
P' = K\begin{bmatrix}R&T\end{bmatrix}P_w = MP_w
\label{eq:cameramatrix}
\end{equation}

These parameters $R$ and $T$ are known as the \emph{extrinsic parameters} because they are external to and do not depend on the camera.

\vspace{12pt}

This completes the mapping from a 3D point $P$ in an arbitrary world reference system to the image plane. To reiterate, we see that the full projection matrix $M$ consists of the two types of parameters introduced above: \emph{intrinsic} and \emph{extrinsic} parameters. All parameters contained in the camera matrix $K$ are the intrinsic parameters, which change as the type of camera changes. The extrinsic paramters include the rotation and translation, which do not depend on the camera's build. Overall, we find that the $3\times4$ projection matrix $M$ has 11 degrees of freedom: 5 from the intrinsic camera matrix, 3 from extrinsic rotation, and 3 from extrinsic translation.

\section{Camera Calibration}
To precisely know the transformation from the real, 3D world into digital images requires prior knowledge of many of the camera's intrinsic parameters. If given an arbitrary camera, we may or may not have access to these parameters. We do, however, have access to the images the camera takes. Therefore, can we find a way to deduce them from images? This problem of estimating the extrinsic and intrinsic camera parameters is known as \emph{camera calibration}.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/1-7.pdf}
\caption{The setup of an example calibration rig.}
\label{fig:calibration}
\end{figure}

Specifically, we do this by solving for the intrinsic camera matrix $K$ and the extrinsic parameters $R,T$ from Equation~\ref{eq:cameramatrix}. We can describe this problem in the context of a calibration rig, such as the one show in Figure~\ref{fig:calibration}. The rig usually consists of a simple pattern (i.e. checkerboard) with known dimensions. Furthermore, the rig defines our world reference frame with origin $O_w$ and axes $i_w, j_w, k_w$. From the rig's known pattern, we have known points in the world reference frame $P_1,...,P_n$. Finding these points in the image we take from the camera gives corresponding points in the image $p_1,...,p_n$.

We set up a linear system of equations from $n$ correspondences such that for each correspondence $P_i, p_i$ and camera matrix $M$ whose rows are $m_1, m_2, m_3$:
\begin{equation}
p_i = \begin{bmatrix}u_i\\v_i\end{bmatrix} = MP_i = \begin{bmatrix}\frac{m_1 P_i}{m_3P_i}\\\frac{m_2 P_i}{m_3P_i}\end{bmatrix}
\end{equation}

As we see from the above equation, each correspondence gives us two equations and, consequently, two constraints for solving the unknown parameters contained in $m$. From before, we know that the camera matrix has $11$ unknown parameters. This means that we need at least $6$ correspondences to solve this. However, in the real world, we often use more, as our measurements are often noisy. To explicitly see this, we can derive a pair of equations that relate $u_i$ and $v_i$ with $P_i$.
\begin{align*}
u_i(m_3P_i) - m_1P_i = 0\\
v_i(m_3P_i) - m_2P_i = 0
\end{align*}

Given $n$ of these corresponding points, the entire linear system of equations becomes
\begin{align*}
u_1(m_3P_1) -& m_1P_1 = 0\\
v_1(m_3P_1) -& m_2P_1 = 0\\
&\vdots\\
u_n(m_3P_n) -& m_1P_n = 0\\
v_n(m_3P_n) -& m_2P_n = 0\\
\end{align*}

This can be formatted as a matrix-vector product shown below:
\begin{equation}
\begin{bmatrix}
P_1^T & 0^T & -u_1P_1^T \\
0^T & P_1^T & -v_1P_1^T \\
& \vdots &\\
P_n^T & 0^T & -u_nP_n^T \\
0^T & P_n^T & -v_nP_n^T
\end{bmatrix}
\begin{bmatrix}
m_1^T \\ m_2^T \\m_3^T
\end{bmatrix} = \mathbf{P}m = 0
\label{eq:linearsystem}
\end{equation}

When $2n > 11$, our homogeneous linear system is overdetermined. For such a system $m=0$ is always a trivial solution. Furthemore, even if there were some other $m$ that were a nonzero solution, then $\forall k\in \mathbb{R},km$ is also a solution. Therefore, to constrain our solution, we complete the following minimization:
\begin{equation}
\begin{aligned}
    & \underset{m}{\text{minimize}}
    & & \|\mathbf{P}m\|^2 \\
    & \text{subject to}
    & & \|m\|^2 = 1
\end{aligned}
\end{equation}
To solve this minimization problem, we simply use singular value decomposition. If we let $P = UDV^T$, then the solution to the above minimization is to set $m$ equal to the last column of $V$. The derivation for this solution is outside the scope of this class and you may refer to Section 5.3 of Hartley \& Zisserman on pages 592-593 for more details.

After reformatting the vector $m$ into the matrix $M$, we now want to explicitly solve for the extrinsic and intrinsic parameters. We know our SVD-solved $M$ is known up to scale, which means that the true values of the camera matrix are some scalar multiple of $M$:
\begin{equation}
\rho M = \begin{bmatrix}
\alpha r_1^T - \alpha\cot \theta r_2^T + c_xr_3^T & \alpha t_x - \alpha \cot \theta t_y + c_x t_z \\
\frac{\beta}{\sin\theta}r_2^T + c_yr_3^T & \frac{\beta}{\sin\theta}t_y + c_yt_z \\ r_3^T & t_z
\end{bmatrix}
\end{equation}

Here, $r_1^T$, $r_2^T$, and $r_3^T$ are the three rows of $R$. Dividing by the scaling parameter gives
\[M = \frac{1}{\rho}\begin{bmatrix}
\alpha r_1^T - \alpha\cot \theta r_2^T + c_xr_3^T & \alpha t_x - \alpha \cot \theta t_y + c_x t_z \\
\frac{\beta}{\sin\theta}r_2^T + c_yr_3^T & \frac{\beta}{\sin\theta}t_y + c_yt_z \\ r_3^T & t_z
\end{bmatrix} = \begin{bmatrix}A & b\end{bmatrix} =\begin{bmatrix}
a_1^T \\ a_2^T \\ a_3^T
\end{bmatrix} \begin{bmatrix}
b_1\\b_2\\b_3
\end{bmatrix}
\]

Solving for the intrinsics gives
\begin{equation}\begin{aligned}
    \rho &= \pm \frac{1}{\|a_3\|}\\
    c_x &= \rho^2(a_1\cdot a_3)\\
    c_y &=\rho^2(a_2\cdot a_3)\\
    \theta &= \cos ^{-1} \left(-\frac{(a_1 \times a_3)\cdot(a_2\times a_3)}{\|a_1\times a_3\|\cdot\|a_2\times a_3\|}\right)\\
    \alpha &= \rho^2 \|a_1 \times a_3\| \sin \theta\\
    \beta &= \rho^2 \|a_2 \times a_3\| \sin \theta
\end{aligned}\end{equation}

The extrinsics are
\begin{equation}\begin{aligned}
    r_1 &= \frac{a_2\times a_3}{\|a_2\times a_3\|}\\
    r_2 &= r_3\times r_1\\
    r_3 &= \rho a_3\\
    T &= \rho K^{-1} b
\end{aligned}\end{equation}
We leave the derivations as a class exercise or you can refer to Section 5.3.1 of the Forsyth \& Ponce textbook.

With the calibration procedure complete, we warn against degenerate cases. Not all sets of $n$ correspondences will work. For example, if the points $P_i$ lie on the same plane, then the system will not be able to be solved. These unsolvable configurations of points are known as \emph{degenerate configurations}. More generally, degenerate configurations have points that lie on the intersection curve of two quadric surfaces. Although this outside the scope of the class, you can find more information in Section 1.3 of the Forsyth \& Ponce textbook.

\section{Handling Distortion in Camera Calibration}
So far, we have been working with ideal lenses which are free from any distortion. However, as seen before, real lenses can deviate from rectilinear projection, which require more advanced methods. This section provides just a brief introduction to handling distortions.

Often, distortions are radially symmetric because of the physical symmetry of the lens. We model the radial distortion with an isotropic transformation:
\begin{equation}
    QP_i = \begin{bmatrix}
    \frac{1}{\lambda}&0&0\\0 &    \frac{1}{\lambda} & 0 \\ 0 & 0 &1\end{bmatrix} M P_i = \begin{bmatrix}
    u_i\\v_i\end{bmatrix}
    = p_i
\end{equation}
If we try to rewrite this into a system of equations as before, we get

\begin{align*}
    u_iq_3P_i  = q_1P_i\\
    v_iq_3P_i  = q_2P_i
\end{align*}

This system, however, is no longer linear, and we require the use of nonlinear optimization techniques, which are covered in Section 22.2 of Forsyth \& Ponce. We can simplify the nonlinear optimization of the calibration problem if we make certain assumptions. In radial distortion, we note that the ratio between two coordinates $u_i$ and $v_i$ is not affected. We can compute this ratio as
\begin{equation}
    \frac{u_i}{v_i} = \frac{\frac{m_1P_i}{m_3P_i}}{\frac{m_2P_i}{m_3P_i}} = \frac{m_1P_i}{m_2P_i}
\end{equation}

Assuming that $n$ correspondences are available, we can set up the system of linear equations:
\begin{align*}
    v_1(m_1P_1)-&u_1(m_2P_1) = 0\\
    &\vdots\\
    v_n(m_1P_n)-&u_n(m_2P_n) = 0
\end{align*}
Similar to before, this gives a matrix-vector product that we can solve via SVD:
\begin{equation}
    Ln = \begin{bmatrix}
    v_1P_1^T & -u_1P_1^T\\
    \vdots & \vdots\\
    v_nP_n^T & -u_nP_n^T\\
    \end{bmatrix}
    \begin{bmatrix}
    m_1^T\\m_2^T
    \end{bmatrix}
\end{equation}

Once $m_1$ and $m_2$ are estimated, $m_3$ can be expressed as a nonlinear function of $m_1,m_2,$ and $\lambda$. This requires to solve a nonlinear optimization problem whose complexity is much simpler than the original one.

\section{Appendix A: Rigid Transformations}
The basic rigid transformations are rotation, translation, and scaling. This appendix will cover them for the 3D case, as they are common type in this class.

Rotating a point in 3D space can be represented by rotating around each of the three coordinate axes respectively. When rotating around the coordinate axes, common convention is to rotate in a counter-clockwise direction. One intuitive way to think of rotations is how much we rotate around each degree of freedom, which is often referred to as \emph{Euler angles}. However, this methodology can result in what is known as \emph{singularities}, or \emph{gimbal lock}, in which certain configurations result in a loss of a degree of freedom for the rotation.

One way to prevent this is to use rotation matrices, which are a more general form of representing rotations. Rotation matrices are square, orthogonal matrices with determinant one. Given a rotation matrix $R$ and a vector $v$, we can compute the resulting vector $v'$ as
\[v' = Rv\]

Since rotation matrices are a very general representation of matrices, we can represent a rotation $\alpha, \beta, \gamma$ around each of the respective axes as follows:
\[R_x(\alpha) = \begin{bmatrix}1 & 0 & 0\\ 0 &\cos\alpha &-\sin\alpha \\ 0 &\sin\alpha &\cos\alpha \end{bmatrix}\]
\[R_y(\beta) = \begin{bmatrix}\cos\beta & 0 & \sin\beta \\ 0 & 1 & 0 \\ -\sin\beta & 0 &\cos\beta \end{bmatrix}\]
\[R_z(\gamma) = \begin{bmatrix}\cos \gamma & -\sin\gamma &0 \\ \sin\gamma &\cos\gamma & 0 \\ 0 & 0 & 1 \end{bmatrix}\]

Due to the convention of matrix multiplication, the rotation achieved by first rotating around the z-axis, then y-axis, then x-axis is given by the matrix product $R_xR_yR_z$.

Translations, or displacements, are used to describe the movement in a certain direction. In 3D space, we define a translation vector $t$ with 3 values: the displacements in each of the 3 axes, often denoted as $t_x, t_y, t_z$. Thus, given some point $P$ which is translated to some other point $P'$ by $t$, we can write it as:
\[ P' = P + t = \begin{bmatrix}P_x\\P_y\\P_z\end{bmatrix} + \begin{bmatrix}t_x\\t_y\\t_z\end{bmatrix}\]

In matrix form, translations can be written using homogeneous coordinates. If we construct a translation matrix as
\[ T = \begin{bmatrix}
1 & 0 & 0 & t_x \\0 & 1 & 0 & t_y \\0 & 0 & 1 & t_z \\ 0 & 0 & 0 & 1
\end{bmatrix}\]
then we see that $P'=TP$ is equivalent to $P' = P+t$.

If we want to combine translation with our rotation matrix multiplication, we can again use homogeneous coordinates to our advantage. If we want to rotate a vector $v$ by $R$ and then translate it by $t$, we can write the resulting vector $v'$ as:
\[ \begin{bmatrix}v'\\1\end{bmatrix} = \begin{bmatrix}
R & t \\ 0 & 1
\end{bmatrix}\begin{bmatrix}
v \\ 1
\end{bmatrix}\]

Finally, if we want to scale the vector in certain directions by some amount $S_x, S_y, S_z$, we can construct a scaling matrix
\[S = \begin{bmatrix}
S_x & 0 & 0 \\ 0 & S_y & 0 \\ 0 & 0 & S_z
\end{bmatrix}\]

Therefore, if we want to scale a vector, then rotate, then translate, our final transformation matrix would be:
\[T = \begin{bmatrix}
RS & t \\ 0 & 1
\end{bmatrix}\]

Note that all of these types of transformations would be examples of affine transformations. Recall that projective transformations occur when the final row of $T$ is not $\begin{bmatrix}
0 & 0 &0 &1
\end{bmatrix}$.

\section{Appendix B: Different Camera Models}
We will now describe a simple model known as the \emph{weak perspective model}. In the weak perspective model, points are first projected to the reference plane using orthogonal projection and then projected to the image plane using a projective transformation.
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/weak_perspective.png}
\caption{The weak perspective model: orthogonal projection onto reference plane}
\label{fig:weak_perspective}
\end{figure}

As Figure~\ref{fig:weak_perspective} shows, given a reference plane $\Pi$ at a distance $z_o$ from the center of the camera, the points $P,Q,R$ are first projected to the plane $\Pi$ using an orthogonal projection, generating points $P\_, Q\_, R\_$.  This is a reasonable approximation when deviations in depth from the plane are small compared to the distance of the camera.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/weak_perspective2.png}
\caption{The weak perspective model: projection onto the image plane}
\label{fig:weak_perspective2}
\end{figure}

Figure~\ref{fig:weak_perspective2} illustrates how points $P\_, Q\_, R\_$ are then projected to the image plane using a regular projective transformation to produce the points $p', q', r'$. Notice, however, that because we have approximated the depth of each point to $z_o$ the projection has been reduced to a simple, constant magnification. The magnification is equal to the focal length $f'$ divided by $z_o$, leading to
\begin{align*}
x'=\frac{f'}{z_0}x\ \ \ \ \
y'=\frac{f'}{z_0}y
\end{align*}
This model also simplifies the projection matrix
\[M = \begin{bmatrix}
A & b \\ 0 & 1
\end{bmatrix}\]
As we see, the last row of $M$ is $\begin{bmatrix}
0 & 0& 0 &1
\end{bmatrix}$ in the weak perspective model, compared to $\begin{bmatrix}
v&1
\end{bmatrix}$ in the normal camera model. We do not prove this result and leave it to you as an exercise. The simplification is clearly demonstrated when mapping the 3D points to the image plane.
\begin{equation}
    P' = MP = \begin{bmatrix}
    m_1 \\ m_2 \\ m_3
    \end{bmatrix}P = \begin{bmatrix}
    m_1P \\ m_2P \\ 1
    \end{bmatrix}
\end{equation}
Thus, we see that the image plane point ultimately becomes a magnification of the original 3D point, irrespective of depth. The nonlinearity of the projective transformation disappears, making the weak perspective transformation a mere magnifier.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/orthographic.png}
\caption{The orthographic projection model}
\label{fig:orthographic}
\end{figure}

Further simplification leads to the \emph{orthographic (or affine) projection model}. In this case, the optical center is located at infinity. The projection rays are now perpendicular to the retinal plane. As a result, this model ignores depth altogether.  Therefore,
\begin{align*}
    x' = x\\
    y' = y
\end{align*}
Orthographic projection models are often used for architecture and industrial design.

Overall, weak perspective models result in much simpler math, at the cost of being somewhat imprecise. However, it often yields results that are very accurate when the object is small and distant from the camera.
\end{document}


\documentclass[a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb,siunitx,cancel}

\def\mhe{m_{\mathrm{He}}}

\title{Interstellar Propulsion}
\author{Matthias Raives}
\date{\today}
\begin{document}

  \maketitle

  \section{Bussard Ramjet}
    This can be derived multiple ways both relativistic and non-relativistic.  The terminal velocity is large enough that the relativistic derivation is preferred, but the non-relativistic answers are correct to an order-of-magnitude level (within a factor of 3).  Three derivations that I have found are presented here.

    \subsection{Relativistic Monentum Conservation}
      The spacecraft generates thrust from the proton-proton chain:
      \begin{equation}
          6\mathrm{p} \rightarrow \mathrm{He}^{4} + 2\mathrm{p} + \nu_{e} + \Delta E
      \end{equation}
      Since our reactor is perfectly efficient, we'll assume that all the $\Delta E$ goes into launching the neutrino (which we'll assume is negligible) and helium nucleus out of the spaceship for propulsion, and that we capture the two extra protons produced (so that, in the steady state, we only need to capture 4 protons per reaction).

      At terminal velocity, the momentum of the ejected helium nuclei is equal to the total momentum of the protons we're collecting.  The momentum of the protons is:
      \begin{equation}
          p_{1} = \sum p_{p} = 4\gamma m_{p}v_{T} = \frac{4m_{p}v_{T}}{\sqrt{1-\frac{v_{T}^{2}}{c^{2}}}},
      \end{equation}
      and the momentum of the exhaust can be determined by:
      \begin{align}
          E = \mhe c^{2} + \Delta E &= (\mhe^{2}c^{4} + p_{2}^{2}c^{2})^{1/2}\\
          (\mhe+\Delta m)^{2}c^{4} &= \mhe^{2}c^{4} + p_{2}^{2}c^{2}\\
          \cancel{\Delta m^{2}} + \mhe\Delta m &= p_{2}^{2}c^{-2}\\
          p_{2} &= c\sqrt{\mhe\Delta m}.
      \end{align}
      The helium nucleus mass is about $\mhe\sim 4 m_{p}$, and the energy produced by the proton-proton chain converts to a mass:
      \begin{equation}
          \Delta m = \frac{\Delta E}{c^{2}} \sim 10^{-2}m_{p}
      \end{equation}
      Setting these momenta equal:
      \begin{align}
          p_{1} &= p_{2}\\
          \frac{4m_{p}v_{T}}{\sqrt{1-\frac{v_{T}^{2}}{c^{2}}}} &= c\sqrt{\mhe\Delta m}\\
          \frac{\frac{v_{T}}{c}}{\sqrt{1-\frac{v_{T}^{2}}{c^{2}}}} &= \frac{\sqrt{\mhe\Delta m}}{4m_{p}}\\
          \frac{v_{T}}{c} &= \left(\frac{1}{\frac{16m_{p}^{2}}{\mhe\Delta m}+1}\right)^{1/2}\\
          \frac{v_{T}}{c} &= \left(\frac{1}{400+1}\right)^{1/2} \sim \frac{1}{20} = 0.05
      \end{align}

    \subsection{Non-relativistic Momentum Conservation}
      This method is similar to the first, but it uses the non-relativistic formulas.  We have the total momentum of the protons:
      \begin{equation}
          p_{1} = \sum p_{p} = 4m_{p}v_{T},
      \end{equation}
      and the exhaust momentum:
      \begin{equation}
          p_{2} = \sqrt{2\mhe\Delta E} = c\sqrt{2\mhe\Delta m}.
      \end{equation}
      Setting these equal:
      \begin{align}
        p_{1} &= p_{2}\\
        4m_{p}v_{T} &= c\sqrt{2\mhe\Delta m}\\
        \frac{v_{T}}{c} &= \left(\frac{\mhe\Delta m}{8m_{p}^{2}}\right)^{1/2}\\
        \frac{v_{T}}{c} &= \frac{\sqrt{2}}{20} \sim 0.07
      \end{align}

    \subsection{Non-relativistic Drag}
      The generalized drag formula is:
      \begin{equation}
          \frac{1}{2}\rho v_{T}^{2}A = F
      \end{equation}
      where $A$ is the effective collecting area of our craft.  In this case, $F$ is the thrust generated by our reactor.  We can approximate $F$ by:
      \begin{equation}
          F = \frac{\mathrm{d}p}{\mathrm{d}t} \sim \frac{p}{4\tau} = \frac{pv_{T}}{4\lambda}
      \end{equation}
      where $\tau = \frac{\lambda}{v_{T}}$ is the mean time between interactions with ISM protons, and $\lambda$ is the mean free path of our spaceship in the ISM, given by
      \begin{equation}
          \lambda = \frac{m_{p}}{\rho A}.
      \end{equation}
      Thus, we have:
      \begin{equation}
          \frac{1}{2}\rho v_{T}^{2}A = \frac{p\rho Av_{T}}{4m_{p}}.
      \end{equation}
      The exhaust momentum $p$ is the same as in the earlier non-relativistic derivation.  Solving for $v_{T}$:
      \begin{equation}
          \frac{v_{T}}{c} = \frac{1}{2c}\frac{p}{m_{p}} = \left(\frac{\mhe\Delta m}{2m_{p}^{2}}\right)^{1/2} = \frac{\sqrt{2}}{20} \sim 0.14
      \end{equation}

  \section{Project Orion}
    \subsection{Rocket Equation}
    Integral to this question is the Rocket Equation:
    \begin{equation}
      \Delta v = v_{e}\ln\left(\frac{m_{0}}{m_{f}}\right)
    \end{equation}
    The derivation is as follows.  At time $t=0$, the rocket has mass $m+\Delta m$ and velocity $v$.  At time $t=\Delta t$, the rocket has velocity $v+\Delta v$, mass $m$, and the spent fuel has mass $\Delta m$ and velocity $v-v_{e}$, where $v_{e}$ is the rest frame ejecta velocity.  By conservation of momentum, then:
    \begin{align}
      (m+\Delta m)v &= m(v+\Delta v) + \Delta m (v-v_{e})\\
      mv + \Delta m v &= mv + m\Delta v + \Delta m v - \Delta mv_{e}\\
      \Delta m v_{e} = m\Delta v\\
      \Delta v &= v_{e}\frac{\Delta m}{m}
    \end{align}
    Integrating this yields:
    \begin{equation}
     \Delta v = v_{e}\ln\left(\frac{m_{0}}{m_{f}}\right)
    \end{equation}
    where $m_0$ is the initial mass of the spacecraft (fuel + payload) and $m_f$ is the final mass (just payload).

    \subsection{Exhaust Velocity}
      In the case of pulse propulsion, the ``ejecta velocity'' is actually the velocity of the explosion, when it pushes against your ship.  A nuclear bomb of mass $M$ has yield $E = \epsilon Mc^{2}$; the ejecta then has velocity:
      \begin{equation}
        v_{e} = \sqrt{\frac{2E}{M}} = \sqrt{2\epsilon c^{2}}
      \end{equation}
      However, only a fraction $f\leq \frac{1}{2}$ of the mass is intercepted, so the effective velocity is
      \begin{equation}
       v_{e} = f\sqrt{2\epsilon} c.
      \end{equation}

    \subsection{Mass Ratio}
      We can calculate $\epsilon$ as:
      \begin{equation}
        \epsilon = \frac{\Delta E}{mc^2} \approx 10^{-3}
      \end{equation}
      Adopting a fiducial $f=\frac{1}{2}$, we see:
      \begin{equation}
        v_e \approx 0.02c
      \end{equation}
      and thus
      \begin{equation}
       \frac{m_{0}}{m_{f}} = \exp\left(\frac{\Delta v}{v_{e}}\right) = e^{2.5} \approx 10.
      \end{equation}
      That is, about 90\% of the mass of the spacecraft must be fuel (i.e., nukes).  The answer here will vary considerably because of the exponential, however.  Also, modern thermonuclear weapons would increase the efficiency of this design.

\end{document}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{float}
\usepackage{subcaption}
\usepackage{bm}
\graphicspath{{./}}
\usepackage[left=1cm,right=1cm,top=1cm,bottom=1cm]{geometry}
\begin{document}
\title{Rotation Invariance using Second-order Convolutional Neural Networks}

\section{Introduction}
We propose a non-trivial filter filter for a Convolutional Neural Network (CNN) that is invariant to rotations of the input image around the center of the filter.
In particular we show how to construct such non-trivial rotationally-invariant filter by using two convolutional neural network layers with a nonlinear ReLu after each convolution.
Our layer has the following two important properties that cannot both be attained by multiplication by a single filter.
One, it is rotationally invariant, and two, it is non-trivial.
What we mean by non-trivial is that our single filter is able to distinguigsh a dumbbell from a donut, even if the donut looks exactly like the surface generated by a rotated dumbbell.
No actiong by a single linear filter can have both those properties since linearity implies the second property cannot hold given the first.
Furthermore, the filter does not need to be trained on rotated views of its input image.
In \cite{chengzhouhan}, Cheng et al. build a neural network that is essentially rotation invariant.
In \cite{jaderberg}, Jaderberg builds a neural network that is invariant to rotations (more generally, to affine transformations of the input).
However, both of those papers require that the network be trained on several transformed input representations in order for the network to learn to be rotationally invariant.
Relatedly, neither of those networks is guaranteed mathematically to be invariant to rotations of the input image (even for square images rotated a quarter turn, for example).
We construct a non-trivial rotationally invariant CNN architecture that does not need to be trained on multiple rotated views on an object.
Our filter is exactly rotationally invariant for rotations by a quarter turn, and the only reason it is only approximately invariant for other rotations is because of the rectangular nature of image pixels, which we compensate for by using gaussian blurring.
For sufficiently high-resolution pixels (or for pixels arranged finely enough along grid lines in polar coordinates) our method is guaranteed to be rotationally invariant to arbitrary precision.
Na\"ively, one might expect that rotationally invariant filters satisfy our goal.
However, as pointed out above, since a single filter is linear in its inputs, any rotationally invariant filter will return the same value for an image and for any weighted average of various rotations of that image.
We find this problematic because we would like our rotationally invariant filter to be able to distinguish objects like a donut from a dumbbell, even though if you rotate a dumbbell you get an object that looks like a donut.
We show that two layers of rotationally invariant second-order filters have the desired properties.

\section{Related Work}
In \cite{marcos}, they generate exact rotation invariance over a finite number of rotations by taking the maxpool over a set of rotated filters.

\section{Defining a rotationally invariant filter}
No standard first-order filter is able to accomplish that task because first-order filters are linear in their inputs, so if they are invariant to rotation then they are also necessarily equally triggered by an image of that object and by a weighted average of images of rotations of that object.

We construct an (approximately) rotationally-symmetric filter as follows.
We assume a square filter of side length $2h + 1$.
The pixels are defined by $\mathbf x = (x,y),\text{ for } x,y\in [-h,h]$ we define the radius $r$ of a pixel as $r_{\mathbf{x}} = ||x||_2$.
We can therefore construct an approximately rotationally-invariant basis filter by setting the filter value $f_n(\mathbf{x})$ to be equal to
$f(\mathbf{x}) = P(n,\sigma,r_x)$
where $n$ is the radius parameter for the desired filter, $\sigma$ is the standard deviation of the blurring gaussian for the filter, and $P(n, \sigma, r_x)$ is the gaussian probability density value of the point $r_x$ for the gaussian centered at $n$ with standard deviation $\sigma$.

\begin{figure}[h]
\centering
\begin{subfigure}[t]{.3\textwidth}
 \centering
 \includegraphics[width=1in]{basis}
 \caption{Our rotationally symmetric basis filter of radius 5 pixels}
\end{subfigure}
\begin{subfigure}[t]{.3\textwidth}
 \centering
 \includegraphics[width=1in]{rotbasis45}
 \caption{That basis filter rotated by 45 degrees counterclockwise (using bilinear interpolation to fit rotated pixels)}
\end{subfigure}
\begin{subfigure}[t]{.3\textwidth}
 \centering
 \includegraphics[width=1in]{diffbasisrot45}
 \caption{The difference in pixel intensity between the basis filter and the rotated basis filter (hardly discernable)}
\end{subfigure}
\end{figure}

\bibliography{traversr}{}
\bibliographystyle{plain}
\end{document}


\input{permve-ntnu-latex-assignment.tex}

\title{
\normalfont \normalsize
\textsc{Norwegian University of Science and Technology\\IT3708 -- Subsymbolic Methods in AI}
}

\author{Per Magnus Veierland\\permve@stud.ntnu.no}

\date{\normalsize\today}

\begin{document}

\fancyfoot[C]{}
\maketitle

\section*{Intelligence Emerging: EA Rank Errata Suggestion}

In the description of the rank selection in evolutionary algorithms on page 434 in section A.5.2 I've found a small error concerning the derivation in Equation A.6 and A.7.

Equation A.6 shows that Equation~\ref{eq:a6a} is equal to Equation~\ref{eq:a6b}.

\begin{equation}
\label{eq:a6a}
\sum_{i=1}^{N}\Bigg[ \text{min} + (\text{max} - \text{min})\frac{\textit{rank}(i, g) - 1}{N - 1}\Bigg]
\end{equation}

\begin{equation}
\label{eq:a6b}
N \cdot \text{min} + (\text{max} - \text{min}) \sum_{i=1}^{N}\frac{i}{N - 1}
\end{equation}

However these are not equivalent, and Equation~\ref{eq:a6b} should instead be:

\begin{equation}
N \cdot \text{min} + (\text{max} - \text{min}) \sum_{i=1}^{N}\frac{i - 1}{N - 1}
\end{equation}

This results in the following derivation, which differs from the first two steps of Equation A.7:

\begin{equation}
\sum_{i=1}^{N}\frac{i - 1}{N - 1} =
\frac{1}{N - 1}\sum_{i=1}^{N}(i - 1) =
\frac{1}{N - 1}\sum_{i=1}^{N - 1}i =
\frac{1}{N - 1} \cdot \frac{(N - 1)N}{2} =
\frac{N}{2}
\end{equation}

\end{document}


% This LaTeX was auto-generated from MATLAB code.
% To make changes, update the MATLAB code and republish this document.

\documentclass{article}
\usepackage{graphicx}
\usepackage{color}

\sloppy
\definecolor{lightgray}{gray}{0.5}
\setlength{\parindent}{0pt}

\begin{document}

\section*{Author: DAVID LI}

\begin{verbatim}FinalPrep.m --- ELEC 403 Final exam, this matlab file is used to do the
various problems in the ELEC 403 textbook that are related to the final
exam, of course the focus is on chapter 5 and chapter 7.\end{verbatim}

\subsection*{Contents}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item Prob 5.2 with jacobian f(x) = x\^{}2+2*y\^{}2+4*x+4*y
   \item Prob 5.2 with Newton
   \item Prob 5.4 -- Find a good starting point besides x0 = [1 1]
   \item 5.4 Using Newton
   \item 5.4 Using Gauss Newton
   \item Prob 5.5 -- use a closer point so that it can be done manually
   \item Chapter 7, using Prob 5.2 with DFP
   \item Chapter 7, using Prob 5.2 with BFGS
   \item Examples from the Internet --- Using DFP
   \item USING BFGS
\end{itemize}

\subsection*{Prob 5.2 with jacobian f(x) = x\^{}2+2*y\^{}2+4*x+4*y}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item REMEMBER THAT CHAPTER 5 only has SDM, Newton and Gauss-Newton*
\end{itemize}
\begin{verbatim}
syms x_1 x_2
f = x_1^2+2*x_2^2+4*x_1+4*x_2;
latex(f);
g=gradient(f);
latex(g);
f1 = x_1+2;
f2 = sqrt(2)*(x_2+1);
j=jacobian([f1,f2],[x_1,x_2]);
h = hessian(f);
%steep_desc3('func52','grad52',[-1; 1],1e-3);
gauss_newton('func52','grad52','jacob52',[-1; 1],1e-3);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program gauss_newton.m

F_k =

     3

gk =

     2
     8

Jk =

    1.0000         0
         0    1.4142

Hk =

    2.0000         0
         0    4.0000

dk =

   -1.0000
   -2.0000

ak =

    1.0000

xk =

   -2.0000
   -1.0000

F_k1 =

    -6

gk =

   1.0e-12 *

    0.4450
   -0.2185

Jk =

    1.0000         0
         0    1.4142

dk =

   1.0e-12 *

   -0.2225
    0.0546

ak =

     1

adk =

   1.0e-12 *

   -0.2225
    0.0546

er =

     9

xk =

    -2
    -1

F_k1 =

    -6

gk =

     0
     0

Jk =

    1.0000         0
         0    1.4142

dk =

     0
     0

ak =

     1

adk =

     0
     0

er =

     0

Solution point:

xs =

    -2
    -1

Objective function at the solution point:

fs =

    -6

Number of iterations performed:

k =

     3

\end{verbatim} \color{black}

\subsection*{Prob 5.2 with Newton}

\begin{verbatim}
newton('func52','grad52','hess52',[0; 0],0.1,1e-3);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program newton.m

gk =

     4
     4

Hk =

     2     0
     0     4

dk =

    -2
    -1

Alpha: 1.000000000

er =

    2.2361

gk =

     0
     0

Hk =

     2     0
     0     4

ak =

     1

Alpha: 1.000000000
Solution point:

xs =

    -2
    -1

Objective function at the solution point:

fs =

    -6

Number of iterations performed:

k =

     2

\end{verbatim} \color{black}

\subsection*{Prob 5.4 -- Find a good starting point besides x0 = [1 1]}

\begin{verbatim}
syms x_1 x_2
f = 5*x_1^2-9*x_1*x_2+4.075*x_2^2+x_1;
latex(f);
g = gradient(f);
latex(g);
steep_desc3('func54','grad54',[-16; -17.9],1e-3);
h = hessian(f);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program steep_desc3.m
Alpha: 0.055180698
	 xk+1: -16.115879466
	 xk+1: -17.795984384
f(xk) = -8.148958e+00

Alpha: 36.073499423
	 xk+1: -16.298584133
	 xk+1: -17.999528044
f(xk) = -8.149995e+00
Norm Value: 2.735160e-01

Alpha: 0.055180697
	 xk+1: -16.299131033
	 xk+1: -17.999037137
f(xk) = -8.150000e+00
Norm Value: 7.349077e-04

Solution point:

xs =

 -16.299131032840879
 -17.999037136909219

Objective function at the solution point:

fs =

  -8.149999976794053

Number of iterations performed:

k =

     3

\end{verbatim} \color{black}

\subsection*{5.4 Using Newton}

\begin{verbatim}
h = hessian(f);
latex(h)
newton('func54','grad54','hess54',[-10; -5],0.1,1e-6);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
ans =

\left(\begin{array}{cc} 10 & -9\\ -9 & \frac{163}{20} \end{array}\right)

Program newton.m

gk =

  -54.0000
   49.2500

Hk =

   10.0000   -9.0000
   -9.0000    8.1500

dk =

   -6.3000
  -13.0000

Alpha: 0.945675721

er =

   13.6613

gk =

   -2.9335
    2.6755

Hk =

   10.0000   -9.0000
   -9.0000    8.1500

ak =

     1

Alpha: 1.000000000

gk =

   1.0e-13 *

    0.5684
   -0.5684

Hk =

   10.0000   -9.0000
   -9.0000    8.1500

ak =

     1

Alpha: 1.000000000
Solution point:

xs =

 -16.300000000000054
 -18.000000000000060

Objective function at the solution point:

fs =

  -8.149999999999963

Number of iterations performed:

k =

     3

\end{verbatim} \color{black}

\subsection*{5.4 Using Gauss Newton}

\begin{verbatim}
j = jacobian(f,[x_1,x_2]);
gauss_newton('func54','grad54','jacob54',[-16; -18],1e-3);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program gauss_newton.m

F_k =

   -7.7000

gk =

    3.0000
   -2.7000

Jk =

    3.0000   -2.7000

Hk =

   18.0000  -16.2000
  -16.2000   14.5800

dk =

   -0.0920
    0.0828

ak =

    1.7993

xk =

  -16.1656
  -17.8511

F_k1 =

   -8.1494

gk =

    0.0037
    0.0041

Jk =

    0.0037    0.0041

dk =

  -60.4078
  -67.1792

ak =

    0.0022

adk =

   -0.1338
   -0.1488

er =

    0.4494

xk =

  -16.2994
  -17.9999

F_k1 =

   -8.1500

gk =

    0.0049
   -0.0044

Jk =

    0.0049   -0.0044

dk =

  -56.5957
   50.8910

ak =

   1.0000e-05

adk =

   1.0e-03 *

   -0.5660
    0.5089

er =

   5.5388e-04

Solution point:

xs =

 -16.300007501945657
 -17.999413346528229

Objective function at the solution point:

fs =

  -8.149998557647706

Number of iterations performed:

k =

     3

\end{verbatim} \color{black}

\subsection*{Prob 5.5 -- use a closer point so that it can be done manually}

\begin{par}
Produce the latex needed for final exam review document
\end{par} \vspace{1em}
\begin{verbatim}
syms x_1 x_2 x_3
f = (x_1+5)^2+(x_2+8)^2+(x_3+7)^2+2*x_1^2*x_2^2+4*x_1^2*x_3^2;
latex(f)
g = gradient(f);
latex(g)

steep_desc3('func55','grad55',[0; -8; -7],1e-3)
\end{verbatim}

        \color{lightgray} \begin{verbatim}
ans =

2\, {x_{1}}^2\, {x_{2}}^2 + 4\, {x_{1}}^2\, {x_{3}}^2 + {\left(x_{1} + 5\right)}^2 + {\left(x_{2} + 8\right)}^2 + {\left(x_{3} + 7\right)}^2

ans =

\left(\begin{array}{c} 4\, x_{1}\, {x_{2}}^2 + 8\, x_{1}\, {x_{3}}^2 + 2\, x_{1} + 10\\ 4\, x_{2}\, {x_{1}}^2 + 2\, x_{2} + 16\\ 8\, x_{3}\, {x_{1}}^2 + 2\, x_{3} + 14 \end{array}\right)

Program steep_desc3.m
Alpha: 0.001538462
	 xk+1: -0.015384615
	 xk+1: -8.000000000
	 xk+1: -7.000000000
f(xk) = 2.492308e+01

Alpha: 0.499585233
	 xk+1: -0.015384615
	 xk+1: -7.996216159
	 xk+1: -6.993378279
f(xk) = 2.492302e+01
Norm Value: 7.626575e-03

Alpha: 0.001540793
	 xk+1: -0.015407926
	 xk+1: -7.996216155
	 xk+1: -6.993378281
f(xk) = 2.492302e+01
Norm Value: 2.331078e-05

Solution point:

xs =

  -0.015407926169005
  -7.996216155022361
  -6.993378280948492

Objective function at the solution point:

fs =

  24.923018533805848

Number of iterations performed:

k =

     3

ans =

   -0.0154
   -7.9962
   -6.9934

\end{verbatim} \color{black}

\subsection*{Chapter 7, using Prob 5.2 with DFP}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item REMEMBER THAT CHAPTER 7 only has DFP and BFGS*
   \item DFP
   \item BFGS
\end{itemize}
\begin{verbatim}
dfp('func52','grad52',[-1; 1],1e-6);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program dfp.m
------- ITERATION 1 ------
STEP 1

xk =

    -1
     1

Sk =

     1     0
     0     1

gk =

     2
     8

STEP 2

dk =

    -2
    -8

ak =

    0.2576

dtk =

   -0.5152
   -2.0606

xk_new =

   -1.5152
   -1.0606

STEP 3
Norm of delta is 8.76 and epsi is 1.000000e-06
STEP 4

gk_new =

    0.9697
   -0.2424

gmk =

   -1.0303
   -8.2424

sg =

   -1.0303
   -8.2424

sw1 =

    0.2654    1.0615
    1.0615    4.2461

sw2 =

    1.0615    8.4922
    8.4922   67.9376

sw3 =

   68.9991

Sk =

    0.9998   -0.0625
   -0.0625    0.2578

gk =

    0.9697
   -0.2424

------- ITERATION 2 ------
STEP 2

dk =

   -0.9846
    0.1231

ak =

    0.4924

dtk =

   -0.4848
    0.0606

STEP 3
Norm of delta is 0.49 and epsi is 1.000000e-06
STEP 4

gk_new =

     0
     0

gmk =

   -0.9697
    0.2424

sg =

   -0.9846
    0.1231

sw1 =

    0.2351   -0.0294
   -0.0294    0.0037

sw2 =

    0.9695   -0.1212
   -0.1212    0.0151

sw3 =

    0.9846

Sk =

    0.5000   -0.0000
   -0.0000    0.2500

gk =

     0
     0

------- ITERATION 3 ------
STEP 2

dk =

     0
     0

ak =

     1

dtk =

     0
     0

STEP 3
Norm of delta is 0.00 and epsi is 1.000000e-06
solution point:

xs =

    -2
    -1

objective function at the solution point:

fs =

    -6

number of iterations at convergence:

k =

     3

\end{verbatim} \color{black}

\subsection*{Chapter 7, using Prob 5.2 with BFGS}

\begin{verbatim}
bfgs('func52','grad52',[-1; 1],1e-6);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program bfgs.m
------- ITERATION 1 ------
STEP 1

xk =

    -1
     1

Sk =

     1     0
     0     1

gk =

     2
     8

STEP 2

dk =

    -2
    -8

ak =

    0.2576

dtk =

   -0.5152
   -2.0606

xk_new =

   -1.5152
   -1.0606

STEP 3
Norm of delta is 8.76 and epsi is 1.000000e-06
STEP 4

gk_new =

    0.9697
   -0.2424

gmk =

   -1.0303
   -8.2424

D =

   17.5152

sg =

   -1.0303
   -8.2424

sw1 =

    0.2654    1.0615
    1.0615    4.2461

sw2 =

    0.5308    2.1230
    4.2461   16.9844

Sk =

    1.0142   -0.0643
   -0.0643    0.2580

fk =

   -5.7576

gk =

    0.9697
   -0.2424

------- Iteration 2 ------
STEP 2

dk =

   -0.9991
    0.1249

ak =

    0.4853

dtk =

   -0.4848
    0.0606

STEP 3
Norm of delta is 0.49 and epsi is 1.000000e-06
STEP 4

gk_new =

     0
     0

gmk =

   -0.9697
    0.2424

D =

    0.4848

sg =

   -0.9991
    0.1249

sw1 =

    0.2351   -0.0294
   -0.0294    0.0037

sw2 =

    0.4844   -0.0606
   -0.0606    0.0076

Sk =

    0.5000    0.0000
    0.0000    0.2500

fk =

    -6

gk =

     0
     0

------- Iteration 3 ------
STEP 2

dk =

     0
     0

ak =

     1

dtk =

     0
     0

STEP 3
Norm of delta is 0.00 and epsi is 1.000000e-06
solution point:

xs =

    -2
    -1

objective function at the solution point:

fs =

    -6

number of iterations at convergence:

k =

     3

\end{verbatim} \color{black}

\subsection*{Examples from the Internet --- Using DFP}

\begin{par}
$f(x)=-2*x_1^2-10*x_2^2$
\end{par} \vspace{1em}
\begin{verbatim}
dfp('funcNet1','gradNet1',[1; -1],1e-3);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program dfp.m
------- ITERATION 1 ------
STEP 1

xk =

     1
    -1

Sk =

     1     0
     0     1

gk =

     4
   -20

STEP 2

dk =

    -4
    20

ak =

    0.0577

dtk =

   -0.2308
    1.1538

xk_new =

    0.7692
    0.1538

STEP 3
Norm of delta is 10.58 and epsi is 1.000000e-03
STEP 4

gk_new =

    3.0769
    3.0769

gmk =

   -0.9231
   23.0769

sg =

   -0.9231
   23.0769

sw1 =

    0.0533   -0.2663
   -0.2663    1.3314

sw2 =

    0.8521  -21.3018
  -21.3018  532.5444

sw3 =

  533.3964

Sk =

    1.0004    0.0300
    0.0300    0.0512

gk =

    3.0769
    3.0769

------- ITERATION 2 ------
STEP 2

dk =

   -3.1705
   -0.2499

ak =

    0.2699

dtk =

   -0.8556
   -0.0674

STEP 3
Norm of delta is 1.33 and epsi is 1.000000e-03
STEP 4

gk_new =

   -0.3456
    1.7281

gmk =

   -3.4225
   -1.3488

sg =

   -3.4644
   -0.1718

sw1 =

    0.7321    0.0577
    0.0577    0.0045

sw2 =

   12.0017    0.5951
    0.5951    0.0295

sw3 =

   12.0886

Sk =

    0.2500   -0.0001
   -0.0001    0.0503

gk =

   -0.3456
    1.7281

------- ITERATION 3 ------
STEP 2

dk =

    0.0866
   -0.0869

ak =

    0.9949

dtk =

    0.0862
   -0.0865

STEP 3
Norm of delta is 0.12 and epsi is 1.000000e-03
STEP 4

gk_new =

   1.0e-03 *

   -0.9953
   -0.9953

gmk =

    0.3446
   -1.7291

sg =

    0.0864
   -0.0870

sw1 =

    0.0074   -0.0074
   -0.0074    0.0075

sw2 =

    0.0075   -0.0075
   -0.0075    0.0076

sw3 =

    0.1801

Sk =

    0.2501    0.0000
    0.0000    0.0500

gk =

   1.0e-03 *

   -0.9953
   -0.9953

------- ITERATION 4 ------
STEP 2

dk =

   1.0e-03 *

    0.2489
    0.0498

ak =

    0.9997

dtk =

   1.0e-03 *

    0.2488
    0.0498

STEP 3
Norm of delta is 0.00 and epsi is 1.000000e-03
solution point:

xs =

   1.0e-10 *

   0.458617078233185
  -0.458617078910812

objective function at the solution point:

fs =

     2.523955499581143e-20

number of iterations at convergence:

k =

     4

\end{verbatim} \color{black}

\subsection*{USING BFGS}

\begin{verbatim}
bfgs('func52','grad52',[-1; 1],1e-6);
\end{verbatim}

        \color{lightgray} \begin{verbatim}
Program bfgs.m
------- ITERATION 1 ------
STEP 1

xk =

    -1
     1

Sk =

     1     0
     0     1

gk =

     2
     8

STEP 2

dk =

    -2
    -8

ak =

    0.2576

dtk =

   -0.5152
   -2.0606

xk_new =

   -1.5152
   -1.0606

STEP 3
Norm of delta is 8.76 and epsi is 1.000000e-06
STEP 4

gk_new =

    0.9697
   -0.2424

gmk =

   -1.0303
   -8.2424

D =

   17.5152

sg =

   -1.0303
   -8.2424

sw1 =

    0.2654    1.0615
    1.0615    4.2461

sw2 =

    0.5308    2.1230
    4.2461   16.9844

Sk =

    1.0142   -0.0643
   -0.0643    0.2580

fk =

   -5.7576

gk =

    0.9697
   -0.2424

------- Iteration 2 ------
STEP 2

dk =

   -0.9991
    0.1249

ak =

    0.4853

dtk =

   -0.4848
    0.0606

STEP 3
Norm of delta is 0.49 and epsi is 1.000000e-06
STEP 4

gk_new =

     0
     0

gmk =

   -0.9697
    0.2424

D =

    0.4848

sg =

   -0.9991
    0.1249

sw1 =

    0.2351   -0.0294
   -0.0294    0.0037

sw2 =

    0.4844   -0.0606
   -0.0606    0.0076

Sk =

    0.5000    0.0000
    0.0000    0.2500

fk =

    -6

gk =

     0
     0

------- Iteration 3 ------
STEP 2

dk =

     0
     0

ak =

     1

dtk =

     0
     0

STEP 3
Norm of delta is 0.00 and epsi is 1.000000e-06
solution point:

xs =

    -2
    -1

objective function at the solution point:

fs =

    -6

number of iterations at convergence:

k =

     3

\end{verbatim} \color{black}

\end{document}


\section{The Real and Complex Number Systems}

\subsection{Exercise 1}
If $rx = q$ or $r+x = q$ for some rational $q$, then substracting $r$ from $q$ or dividing $q$ by $r$
yields $x$ rational, which is a contradiction.

\subsection{Exercise 2}
We can first show that $\sqrt{3}$ is irrational by seeing that $\frac{a^2}{b^2} = 3 \implies 3 | a, 3 | b$.
Then, since $12 = 3 * 2^2$, we have that $\sqrt{12}$ is irrational as well.

\subsection{Exercise 4}
If $\alpha > \beta$ then $\alpha$ would be an upper bound as well.

\subsection{Exercise 5}
$\forall x \in A, \: -x \leq \sup{-A}$ and $\forall \epsilon \in \mathbb{R}, \: \exists x \in A \: | \sup{-A} + \epsilon < -x \leq \sup{-A}$. Negating the last inequality gives $\inf{A} = -\sup{-A}$.

\subsection{Exercise 6}
(a) Follows from $m = \frac{np}{q}$.

(b) Put $r = \frac{m}{n}, \: s = \frac{p}{q}$. Then $b^r b^s = b^{\frac{mq}{nq}} b^{\frac{np}{nq}}$.
Pulling out $\frac{1}{nq}$ gives the desired result.

(c) $b^r$ is an upper bound since $b > 1$, and if it were not the supremum we could choose
$t < r$ such that $b^t > b^r$. This is not possible since again, $b > 1$.

(d) Every element in $B(x + y)$ can be expressed as $b^{s + t} = b^s b^t \: s \leq x, \: t \leq y$.
If $\sup{B(x+y)} = \alpha < \sup{B(x)}\sup{B(y)}$, then $b^s b^t \leq \alpha \implies B(x) \leq \alpha b^{-t}
\implies B(y) \leq \frac{\alpha}{B(x)} \implies B(x) B(y) \leq \alpha$.

\subsection{Exercise 7}
(a) $b^n - 1 = (b - 1) (b^{n-1} + b^{n-2} + ... + 1) \geq n (b-1)$ since $b > 1$.

(b) Plug $b^{\frac{1}{n}}$ into (a).

(c) Plug $n > \frac{b_-1}{t-1}$ into (b).

(d) Using (c) gives that we can choose $n$ such that
$b^{\frac{1}{n}} < y \dot b^{-w} \implies b^{w + \frac{1}{n}} < y$.

(e) We can take the reciprocal of (c) and do the same as in (d).

(f) If $b^x > y$ we can apply (e) for a contradiction, if $b^x < y$ we can apply (d)
for a contradiction.

(g) Supremum is unique.

\subsection{Exercise 8}
Suppose $(0, 1) < (0, 0)$. Then $(0, -1) < (0, 0)$ after multiplying by  $(0, 1)$ twice yields a
contradiction. Similarly, assuming the opposite yields $(-1, 0) > (0, 0)$.

\subsection{Exercise 9}
Does exhibit least upper-bound property since you can take $(\sup{a_i}, \sup{b_i})$.

\subsection{Exercise 10}
Exception is 0.

\subsection{Exercise 11}
Take $w = \frac{1}{\abs{z}}z$ and $r = \abs{z}$ when $\abs{z} \neq 0$.
$w$ and $r$ are not uniquely determined; take $z = 0$ for example.

\subsection{Exercise 12}
By strong induction:
\begin{align*}
        \abs{z_1 + ... + z_{n+1}} &\leq \abs{z_1 + ... + z_n} + \abs{z_{n+1}} \\
                                  &\leq \abs{z_1} + ... + \abs{z_{n+1}}
\end{align*}

\subsection{Exercise 13}
\begin{align*}
        \abs{x - y}^2 &= x\bar{x} - 2\abs{x}\abs{y} + y \bar{y} \\
                      &\geq (\abs{x} - \abs{y})^2
\end{align*}


\section{More about $\Tor$}
Where is everybody? Looks like nobody wants to hear about $\Tor$. You're the select few.

On Monday I gave ``axioms'' for $\Tor$, basically by saying that $\Tor^R_n(M,-):\mathbf{Mod}_R\to\mathbf{Mod}_R$ is like a homology theory. Today I'm going to show a construction of $\Tor$, and verify the axioms. Or at least the lexseq business. I also tried to show that it's a reasonable idea to study free resolutions, namely:
\begin{equation*}
\xymatrix{\cdots\ar@{-->}[rr] & & F_2\ar[dr]\ar@{-->}[rr]^d & & F_1\ar[dr]\ar@{-->}[rr]^d & & F_0\ar[dr]\\
& K_2\ar[ur]\ar[dr] & & K_1\ar[ur]\ar[dr] & & K_0\ar[ur]\ar[dr] & & N\ar[dr]\\
0\ar[ur] & & 0\ar[ur] & & 0\ar[ur] & & 0\ar[ur] & & 0}
\end{equation*}
Where $F_{i+1}$ surjects onto $K_i$ and the $F_i$ are free $R$-modules. Splicing these exact sequences gives you a exact sequence in the top row, which is a free resolution of $N$. Of course there are a lot of choices involved, so free resolutions aren't unique. The resolution $F_\bullet$ does \emph{not} include $N$, and in the following diagram, the top row isn't exact but $\cdots\to F_0\to N\to 0$ is exact.
\begin{equation*}
\xymatrix{\cdots\ar[r] & F_2\ar[r] & F_1\ar[r] & F_0\ar[r]\ar[d]^\epsilon & 0\\
 & & & N & }
\end{equation*}
Then, note that:
\begin{equation*}
 H_q(F_\bullet)=\begin{cases}
N & q=0\\
0 & q>0
\end{cases}
\end{equation*}
\begin{construction}
We construct $\Tor^R_n(M,N)$ via $ H_n(M\otimes_R F_\bullet)$ where $F_\bullet$ is a free resolution of $N$.
\end{construction}
I have to check that this is well-defined, that it's functorial, and that it satisfies the lexseq. Maybe I should also check what $ H_0(M\otimes_R F_\bullet)$ is. I do get $M\otimes_R F_\bullet$, because tensoring with $M$ is right exact, i.e., you have an exact sequence $M\otimes_R F_1\xrightarrow{p} M\otimes_R F_0\to M\otimes_R N\to 0$, and the zeroth homology is the cokernel of $p$, which is $M\otimes_R N$.

The check that it's well-defined goes like this. I call this the fundamental theorem of homological algebra.
\begin{theorem}[Fundamental theorem of homological algebra]
Let $f:M\to N$ be an $R$-module homomorphism. Let $\cdots\to E_1\to E_0\to M\to 0$ be such that each $E_n$ is free, and $\cdots\to F_1\to F_0\to N\to 0$ is exact. The fundamental theorem says that I can lift the map $f:M\to N$ to a chain map $E_\bullet\to F_\bullet$ (i.e. they commute with the differentials and the augmentations $\epsilon_N:F_0\to N$ and $\epsilon_M:E_0\to M$), that is unique up to chain homotopy. I.e., they sit in the following commutative diagram:
\begin{equation*}
\xymatrix{\cdots\ar[r] & E_2\ar[r]\ar@{-->}[d]^{f_2} & E_1\ar[r]\ar@{-->}[d]^{f_1} & E_0\ar[r]^{\epsilon_M}\ar@{-->}[d]^{f_0} & M\ar[d]^f\ar[r] & 0\\
\cdots\ar[r] & F_2\ar[r] & F_1\ar[r] & F_0\ar[r]^{\epsilon_N} & N\ar[r] & 0}
\end{equation*}
\end{theorem}
I'm making a big deal about homological algebra in this course on algebraic topology, because this really is a homotopy-theoretic statement. It's part of the homotopy theory of chain complexes. I just want to mention something.
\begin{definition}
A projective $R$-module $P$ is something such that there's a lift:
\begin{equation*}
\xymatrix{ & M\ar@{->>}[d]\\
P\ar@{-->}[ur]\ar[r] & N}
\end{equation*}
\end{definition}
Every free module is projective, clearly. Anything that's a direct summand in a projective is also projective. Any projective module is a direct summand of a free module.
\begin{example}
Let $k$ be a field. Then $k\times k$ acts on $k$ via $(a,b)c=ac$. This is an example of a projective that isn't free.
\end{example}
\begin{remark}
This proof uses only that $E_n$ is projective. But if you have a PID, there's no difference between projective and free.
\end{remark}
\begin{proof}[Proof of the fundamental theorem of homological algebra]
Let's try to construct $f_0$. Consider:
\begin{equation*}
\xymatrix{0\ar[r] & K_0\ar[r]\ar@{-->}[d]^{g_0} & E_0\ar[r]^{\epsilon_M}\ar@{-->}[d]^{f_0} & M\ar[d]^f &\\
0\ar[r] & L_0=\ker(\epsilon_N)\ar[r] & F_0\ar@{->>}[r]^{\epsilon_N} & N\ar[r] & 0}
\end{equation*}
We know that $E_0=R\langle S\rangle$. What we do is push forward the generators of $E$ via $\epsilon_M$, push it forward to $f$, and pull it back via $\epsilon_N$ which makes sense because it's surjective. This gives us $f_0$. You can restrict it to get $g_0$. Now I'm in exactly the same situation.
\begin{equation*}
\xymatrix{0\ar[r] & K_1\ar[r]\ar@{-->}[d]^{g_1} & E_1\ar[r]^{\epsilon_M}\ar@{-->}[d]^{f_1} & K_0\ar[d]^{g_0} &\\
0\ar[r] & L_1\ar[r] & F_1\ar[r] & L_0\ar[r] & 0}
\end{equation*}
And $g_1$ exists. Now we need to prove the chain homotopy claim. Suppose I have $f_\bullet:E_\bullet\to F_\bullet$ and $f^{\prime}_\bullet:E_\bullet\to F_\bullet$. Then $f^\prime_n-f_n$ (which we'll rename $\ell_n$) is a chain map lifting $0:M\to N$. Let's rename things, so I have:
\begin{equation*}
\xymatrix{\cdots\ar[r] & E_2\ar[r]\ar[d]^{\ell_2} & E_1\ar[r]\ar[d]^{\ell_1} & E_0\ar[r]^{\epsilon_M}\ar[d]^{\ell_0} & M\ar[d]^0\ar[r] & 0\\
\cdots\ar[r] & F_2\ar[r] & F_1\ar[r] & F_0\ar[r]^{\epsilon_N} & N\ar[r] & 0}
\end{equation*}
We want that $\ell_\bullet\simeq 0$. That is, we want $h:E_n\to F_{n+1}$ such that $dh+hd=\ell$. To begin with, we consider:
\begin{equation*}
\xymatrix{ & E_0\ar[d]^{\ell_0}\ar@{-->}[dl]^h & \\
F_1\ar[r]^d & F_0}
\end{equation*}
At the beginning, we want $dh=0$. Well, we consider:
\begin{equation*}
\xymatrix{ & & E_0\ar[d]^{\ell_0}\ar[dl]\ar@{-->}[dll] & \\
F_1\ar@{->>}[r] & L_0\ar[r] & F_0}
\end{equation*}
Because $F_1\to L_0$ is a surjection, the lift exists, and we have $dh=\ell_0$. For the next step, we have:
\begin{equation*}
\xymatrix{ & & E_1\ar[r]\ar[d]^{\ell_1}\ar@{-->}[dll] & E_0\ar[d]^{\ell_0}\ar[dl]^h & \\
F_2\ar@{->>}[r] & L_1\ar[r] & F_1\ar[r]^d & F_0}
\end{equation*}
So what do I want to do here? Ultimately what I want is that $dh=\ell_1-hd$. Well, $d(\ell_1-hd)=d\ell_1-dhd=d\ell_1-\ell_0d=0$ where the last equality comes because $\ell$ is a chain map. So now we can use exactness of $E_\bullet$ to define $h$. Exactly the same process continues.
\end{proof}
That's it. We're going to use this several times. I'm glad to have mentioned the notion of projectivity, because we'll use it later. Now, apply $M\otimes_R -$ to that resolution (???). Suppose I have $f:N\to N^\prime$, and get a map $f_\bullet:F_0\to F^\prime_\bullet$. Apply $M\otimes_R -$ to this, to get a chain map $M\otimes_R F_\bullet\to M\otimes_R F^\prime_\bullet$ to get a map in homology $ H_\ast(M\otimes_R F_\bullet)\to H_\ast(M\otimes_R F^\prime_\bullet)$. How independent is this of the lifting that I chose? Suppose I have two chain maps $1\otimes f_0,f\otimes f_0^\prime:M\otimes_R F_\bullet\to M\otimes_R F^\prime_\bullet$. I can certainly form $1\otimes h:M\otimes_R F_n\to M\otimes_R F^\prime_{n+1}$. I know that $dh+hd=f-f^\prime$. When I tensor, I get $1\otimes(hd+dh)=1\otimes(f-f^\prime)$. But I want that $(1\otimes h)(1\otimes d)+(1\otimes d)(1\otimes h)=1\otimes f-1\otimes f^\prime$. We use a further property of the tensor product:
\begin{enumerate}
\setcounter{enumi}{7}
\item If $f,f^\prime:N\to N^\prime$, then $1\otimes(f+f^\prime)=1\otimes f+1\otimes f^\prime:M\otimes_R N\to M\otimes_R N^\prime$, and $1\otimes(rf)=r(1\otimes f)$. And $M\otimes_R -$ is an $R$-linear functor.
\end{enumerate}
There's things called derived functors. In more general cases, you can't use chain complexes, but rather you use simplicial resolutions. There's non-additive homological algebra. Anyway, you check that $1\otimes f$ and $1\otimes f^\prime$ are indeed chain homotopic, and so you're done.

I think I've verified that it's well-defined and functorial. What about the lexseq? Now, I start with a sexseq and want to get an lexseq. Suppose I have a sexseq $0\to A\to B\to C\to 0$. I should first come up with an sexseq of resolutions. Consider:
\begin{equation*}
\xymatrix{0\ar[r] & A\ar[r] & B\ar[r] & C\ar[r] & 0\\
 & F^\prime_0\ar[u] & & F^{\prime\prime}_0\ar[u] & \\
 & F^\prime_1\ar[u] & & F^{\prime\prime}_1\ar[u] & \\
 & \vdots\ar[u] & & \vdots\ar[u] & }
\end{equation*}
I want to get a free resolution in the middle. The only thing that I can think of doing is constructing:
\begin{equation*}
\xymatrix{0\ar[r] & A\ar[r]^i & B\ar[r] & C\ar[r] & 0\\
0\ar[r] & F^\prime_0\ar[u]^{\epsilon_A}\ar[r] & F^\prime_0\oplus F^{\prime\prime}_0\ar@{-->}[u]^{\epsilon_B}\ar[r] & F^{\prime\prime}_0\ar[u]^{\epsilon_B}\ar[r] & 0 \\
 & F^\prime_1\ar[u] & & F^{\prime\prime}_1\ar[u] & \\
 & \vdots\ar[u] & & \vdots\ar[u] & }
\end{equation*}
In fact, it's the only choice because you have a free module and the sexseq splits. If I'm going to make this work, this is the only thing I can do. I need the augmentation, though. We can think of $\epsilon_B$ as a row vector. The first entry obviously has to be $i\epsilon$. And, well, there's a lift\footnote{There's some ambiguity here. I have to make a choice. It might seem that the boundary map is made up of the choices, but it \emph{isn't}! I haven't proved that. It still needs to be proved.} because $F^{\prime\prime}_0$ is a free resolution:
\begin{equation*}
\xymatrix{0\ar[r] & A\ar[r]^i & B\ar[r] & C\ar[r] & 0\\
0\ar[r] & F^\prime_0\ar[u]^{\epsilon_A}\ar[r] & F^\prime_0\oplus F^{\prime\prime}_0\ar@{-->}[u]^{\epsilon_B}\ar[r] & F^{\prime\prime}_0\ar[u]^{\epsilon_B}\ar@{-->}[ul]^{\overline{\epsilon}}\ar[r] & 0 \\
 & F^\prime_1\ar[u] & & F^{\prime\prime}_1\ar[u] & \\
 & \vdots\ar[u] & & \vdots\ar[u] & }
\end{equation*}
So that $\epsilon_B=[i\epsilon,\overline{\epsilon}]$. This is surjective by the Snake lemma. Consider:
\begin{equation*}
\xymatrix{& 0 & 0 & 0 & \\
0\ar[r] & A\ar[r]^i\ar[u] & B\ar[r]\ar[u] & C\ar[r]\ar[u] & 0\\
0\ar[r] & F^\prime_0\ar[u]^{\epsilon_A}\ar[r] & F^\prime_0\oplus F^{\prime\prime}_0\ar@{-->}[u]^{\epsilon_B}\ar[r] & F^{\prime\prime}_0\ar[u]^{\epsilon_B}\ar@{-->}[ul]^{\overline{\epsilon}}\ar[r] & 0\\
 0\ar[r] & K^\prime_0=\ker\epsilon_A\ar[u]\ar[r] & K_0\ar[r]\ar[u] & K^{\prime\prime}_0=\ker\epsilon_B\ar[u]\ar[r] & 0\\
 & 0\ar[u] & 0\ar[u] & 0\ar[u] & }
\end{equation*}
The bottom row is exact by the $3\times 3$-lemma. It's why I gave it to you for homework! Anyway, I now have a sexseq of free resolutions $0\to F^\prime_\bullet\to F_\bullet\to F^{\prime\prime}_\bullet\to 0$. Now I want a sexseq $0\to M\otimes_R F^\prime_\bullet\to M\otimes_R F_\bullet\to M\otimes_R F^{\prime\prime}_\bullet\to 0$, but I don't know this because $M\otimes_R -$ isn't exact. It's why we got into the business in the whole place. But $0\to F^\prime_\bullet\to F_\bullet\to F^{\prime\prime}_\bullet\to 0$ is split. And applying any functor gives a splitting of the sexseq, i.e., $M\otimes_R -$ sends split sexseqs to (split) sexseq. This means that $0\to M\otimes_R F^\prime_\bullet\to M\otimes_R F_\bullet\to M\otimes_R F^{\prime\prime}_\bullet\to 0$ also splits. Thus we're done proving the lexseq in $\Tor$.

The key idea in homological algebra is that free modules are good.


\graphicspath{{Chapter2/Figs/}}

\section{Natural Gradient descent}

%REPRODUCED WITH SOME MODIFCIATIONS FROM:
%	https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/
%	Hoffman2013
%	Amari1998

The application of standard gradient descent optimisation to functions that depend on probability distributions (as the KL divergence in in variational inference) has an important limitation, as we will show below. \\

Consider the gradient descent optimisation of an ELBO $\Lagr(\bfX|\theta)$, where $\bfX$ are the hidden variables and $\theta$ the corresponding parameters. From the definition of a gradient:
\[
\nabla \Lagr(\bfX|\theta) = \lim_{h\to0} \frac{\Lagr(\theta + h) - \Lagr(\theta)}{\|h\|}
\]
where $h$ represents an infinitesimally small positive step in the space of $\theta$.\\

To find the steepest gradient, one would need to search over all possible directions $d$ in an infinitely small distance $h$, and select the $\hat{d}$ with the largest gradient:
\[
\nabla \Lagr(\bfX|\theta) = \lim_{h\to0} \frac{1}{\|h\|}\argmax_{d \, s.t. \|d\|=h} \Lagr(\theta+d) - \Lagr(\theta)
\]
Notice that the neighborhood of $\theta$ is measured in terms of its Euclidean norm, and the direction of steepest descent is hence dependent on the Euclidean geometry of the $\theta$ space. This becomes problematic in variational inference. A small step from $\theta_t$ to $\theta^{(t+1)}$ does not guarantee a similar small change from $\Lagr(\bfX|\theta^t)$ to $\Lagr(\bfX|\theta^{(t+1)})$. As an example, consider four random variables:
\begin{equation}
	\begin{split}
		\psi_1 &\sim \Ndist{0}{5} \\
		\psi_2 &\sim \Ndist{10}{5}
	\end{split}
	\qquad
	\begin{split}
		\psi_3 &\sim \Ndist{0}{1} \\
		\psi_4 &\sim \Ndist{10}{1}
	\end{split}
\end{equation}

Using the euclidean metric, the distance between $\psi_1$ and $\psi_2$ is the same as the distance between $\psi_3$ and $\psi_4$. However, the distance in distribution space (measured for example by the KL divergence) is clearly much larger between $\psi_1$ and $\psi_2$ than between $\psi_3$ and $\psi_4$ (Figure X).

\begin{figure}[]
	\begin{center}
		\includegraphics[width=0.75\textwidth]{rnorm.pdf}
		\caption{}
		\label{fig:XX}
	\end{center}
\end{figure}

Hence, rather than using the euclidean distance, it is more appropriate to use a KL divergence as a distance metric:
\[
	\nabla \Lagr(\bfX|\theta)_{KL} = \lim_{h\to0} \frac{1}{\|h\|}\argmax_{d \, s.t. KL[p_\theta||p_{\theta+d}]=\|h\|} \Lagr(\theta+d) - \Lagr(\theta)
\]
The direction of steepest ascent measured by the KL divergence is called the natural gradient.

To find the optimal $\hat{d}_{KL}$, one needs to solve the following optimisation problem:
\[
	\hat{d}_{KL} = \argmin_{d} \Lagr(\theta+d)
\]
with the constrain $KL[p_\theta||p_{\theta+d}]=c$, where $c$ is an arbitrary constant. Introducing Lagrange multipliers $\lambda$: \cite{??}:
\[
	\Lagr(\theta+d) + \lambda(KL[p_\theta||p_{\theta+d}] - c)
\]
Before taking its derivative, we can simplify the equation by approximating the  term $ \Lagr(\theta+d)$ with a First-order Taylor series and the term $KL[p_\theta||p_{\theta+d}]$ with a second-order Taylor series around $\theta+d$:
TO FINMISH
\[
KL[p_\theta||p_{\theta+d}]  \approx
KL[p_\theta||p_{\theta}] +
%d \d nabla_{\theta+d} KL[(\bfX|\theta)||(\bfX|\theta+d)] | _{\theta} +
%\frac{1}{2} d^T \d nabla^2_{\theta+d} KL[(\bfX|\theta)||(\bfX|\theta+d)] | _{\theta}d
\]
The first two terms vanish. The third term corresponds to the negative of the Hessian matrix with respect to $\theta+d$ (the Fisher Information Matrix $F$), evaluated at $\theta$. Hence, the term $KL[p_\theta||p_{\theta+d}]$ simplifies to $\frac{1}{2} d^T F d$

The equation to optimise becomes:
\[
\Lagr(\theta) + \nabla_{\theta} \Lagr(\theta)^T d + \frac{1}{2} \lambda d^T F d - \lambda c
\]

Finally, by taking the derivative of Equation X with respect to $d$, setting to zero and solving, we obtain the direction of the steepest natural gradient. It corresponds to the standard (euclidean) gradient pre-multiplied by the inverse of the Fisher Information Matrix:
\[
%	\frac{\partial \Lagr(\theta+d)}{\partial d} + \lambda\frac{\partial(KL[p_\theta||p_{\theta+d}]}{\partial d} = 0
\hat{d} \propto F^{-1} \nabla_{\theta} \Lagr(\theta)
\]

FINISH HERE. CHECK IF THE BOTTOM IS MEANINGFUL:::

Plugging this expression into [[EQUATION XX]], we find that the optimal $d$ is:
\[
\hat{d} = -\frac{1}{\lambda} F(\theta)^{-1} \nabla_{\theta} \Lagr(\theta)
\]
The constatn factor can be absorbed into the learning rate. The natural gradient is then defined
\[
\nabla \Lagr(\bfX|\theta)_{KL} = F(\theta)^{-1} \nabla_{\theta} \Lagr(\theta)
\]

% \cite{[[GITHUBPAGE]]} shows how, in a simple logistic regression problem, the use of natural gradients lead to improved convergence over standard gradient descent.


\documentclass[../../main.tex]{subfiles}
\begin{document}
\subsubsection{At Index}
The operation $atIndex$ will return the Value at a
specified numeric index within a Collection or
an empty Collection if there is no value at the specified index.
\begin{schema}{AtIndex[Collection, \nat]}
  idx? : \nat\\
  coll? : Collection \\
  atIndex~\_ : Collection \cross \nat \surj V
  \where
  \# ~ idx? = 1 \\
  coll! = atIndex(coll?, idx?) = (head ~(~idx? \extract coll?)) \iff idx? \in coll? \\
  coll! = atIndex(coll?, idx?) = \langle  \rangle \iff idx? \not \in coll?
\end{schema}
Given the definition of the $Collection$ and $V$ free types
\begin{zed}
  Collection :== emptyColl ~| ~append \ldata Collection \cross Scalar ~\lor Collection ~\lor KV \cross \nat \rdata \\
  V ::= Scalar ~| ~Collection ~| ~KV
\end{zed}
The collection member $coll?_{idx?} : V$ is implied from $append$ accepting the argument of type $Scalar ~\lor Collection ~\lor KV \equiv V$ which means each Collection member is of type $V$. Given that extraction ($~\_~\extract~\_$) returns a Collection,
\begin{axdef}
  \seq X : Collection
  \where
  \_~\extract~\_ : \power \nat_1 \cross \seq X \fun \seq X
\end{axdef}
in order for $atIndex$ to return the collection member without altering its type,
the first member of $atIdx'$ must be returned, not $atIdx'$ itself.
\begin{axdef}
  atIdx' : Collection \\
  coll!, coll?_{idx?} : V \\
  \where
  atIdx' = (~idx? \extract coll?) \implies \langle coll?_{idx?} \rangle \\
  coll! = head(atIdx') = coll?_{idx?}
\end{axdef}
The $head$ call is made possible by restricting $idx?$ to be a single numeric value.
\begin{argue}
  idx?, idx' : \nat \\
  \t1 \# ~ idx? = 1 @ (~idx? \extract coll?) = \langle coll?_{idx?} \rangle @ \\
  \t2 (head (~idx? \extract coll?)) = coll?_{idx?} & expected return given $idx?$ \\
  \t1 \# ~ idx' \geq 2 @ (~idx' \extract coll?) = \langle coll?_{idx'_{i}}~..~~coll?_{idx'_{j}} \rangle @ \\
  \t2 (head (~idx' \extract coll?)) = coll?_{idx'_{i}} & unexpected return given $idx'$
\end{argue}
Additionally, if the provided $idx? \not \in coll?$ then an empty Collection will be returned
given that $head$ must be passed a non-empty Collection.
\begin{axdef}
  head : \seq_1 X \fun X
  \where
  idx? \not \in coll? \implies (~idx? \extract coll?) = \langle  \rangle ~ \lnot ~\seq_1
\end{axdef}
The properties of $atIndex$ are illustrated in the following examples.
\begin{argue}
  X = \langle x_{0}, x_{1}, x_{2} \rangle \\
  \t1 x_{0} = 0 \\
  \t1 x_{1} = foo \\
  \t1 x_{2} = \langle a, b, c \rangle \\
  atIndex(X, 0) = 0 & $head~(\langle~x_{0}~\rangle)$\\
  atIndex(X, 1) = foo & $head~(\langle~x_{1}~\rangle)$\\
  atIndex(X, 2) = \langle a, b, c \rangle & $head~(\langle~x_{2}~\rangle)$\\
  atIndex(X, 3) = \langle  \rangle & $3 \not \in X \implies x_{3} \not \in X$
\end{argue}
\end{document}


\SubSecDef{verify}{Verifying Algebraic Security}

\PropRef{composability} shows that we can compose algebraically secure circuits. Large circuits can be constructed from a set of \emph{gadgets} - small algebraically secure circuits with some useful semantics. In order to design new gadgets we need to be able to check their algebraic security. The simplest way to get a bound on the absolute correlation is based on the algebraic degree of computed functions: the minimum weight of a \emph{nonzero} function of $n$ bits of degree $d$ is equal to $2^{n-d}$ (see e.g. \cite{Carlet}). Therefore, we can think about the following algorithm for checking a circuit $C(x,r_C)$: for any fixed input $x$ compute the ANFs of the functions computed in $C(x,\cdot)$ (functions of $r_C$) and return the maximum observed degree. The degree does not grow when functions are combined linearly. Therefore, the absolute correlation bound can not grow as well, except when the resulting function is constant in which case the absolute correlation is maximal and the gadget may be insecure. As a result, our method for verifying algebraic security splits into two parts:

\begin{enumerate}
    \item verify that there is no absolute correlation equal to 1 among restrictions of functions from $\FUNCSD{1}(C)$ except the constant functions and affine functions of $x$;
    \item compute the maximum degree among all restrictions of the intermediate functions and compute the corresponding correlation bound.
\end{enumerate}

The second step is straight-forward. We describe an algorithm that solves the first step.

Consider a circuit $C(x,r): \field{N}\times\field{R}\to\field{M}$.
For all $c \in \field{N}$ let $L_c$ be the linear map that returns the restriction $x=c$ of a function $f$ from $\FUNCSD{1}(C)$ (e.g. if functions are represented as truth table vectors then $L_c$ returns the truth table entries corresponding to the case $x = c$). Note that the domain of $L_c$ is defined to be the subspace $\FUNCSD{1}(C)$.

We now give an equivalent condition for the first part of the verification. It serves as a basis for the verification algorithm given in \AlgRef{verify}.

\begin{proposition}
The circuit $C$ is $\maxcor$-1-AS for some $\maxcor < 1$ if and only if for all $c$ the following holds:
\begin{equation}
    \dim{\ker{L_c}} = N.
\end{equation}
\end{proposition}
\begin{proof}
For any $c \in \field{N}$ let $F_c$ be the subspace of $\FUNCSD{1}(C)$ containing functions that are constant when $x$ is fixed to $c$. Also let $F = \bigcup_c F_c$.
$\maxcor < 1$ requires that any $f\in \FUNCSD{1}(C)$ either belongs to $\XFUNCSD{1}(C)$ or is non-constant for any fixed $x$. It is equivalent to require that $F$ is equal to $\XFUNCSD{1}(C)$. Note that each $F_c$ includes $\XFUNCSD{1}(C)$ as a subset. Therefore, $F = \bigcup_c F_c$ is equal to $\XFUNCSD{1}(C)$ if and only if for all $c$ $F_c = \XFUNCSD{1}(C)$. Since these are linear subspaces then we can compare their dimensions.

$\XFUNCSD{1}(C)$ is spanned by all $x_i$ and the constant-1 function:
\eql{prop-algo-dimX}{
    \dim{\XFUNCSD{1}(C)} = N + 1;
}

The constant-1 function always belongs to $\FUNCSD{1}(C)$ and to any of the $F_c$. The subspace of functions that are constant on the restriction can be obtained by adding the constant-1 function to the subspace of functions that are equal to zero on the restriction:
\eql{xxx}{
    F_c &= \ker{L_c} \oplus \pset{\fzero,\fone}, \\
    \EqLabel{prop-algo-dimFc}
    \dim{F_c} &= \dim{\ker{L_c}} + 1.
}

By comparing the dimensions obtained in \EqRef{prop-algo-dimX},\EqRef{prop-algo-dimFc} we prove the proposition.
\end{proof}

\newcommand\tbas{\bas_c}

The algorithm operates on functions using their truth tables. The truth tables are obtained by evaluating the circuit on all possible inputs and recording the values computed in each node. The set of computed truth tables corresponds to $\FUNCS(C)$. By removing redundant vectors we can compute a basis $\bas$ of $\FUNCSD{1}(C)$ (and also ensure presence of the constant-1 vector). Then, for each $c$ we take the part of each basis vector that corresponds to the fixed $x=c$ (and $r$ taking all possible values). These parts form the subspace $\Ima{L_c}$. We compute a basis $\tbas$ of these parts. Finally, we verify that
\begin{equation}
    \dim{\ker{L_c}} = \dim{\FUNCSD{1}(C)} - \dim{\Ima{L_c}} = |\bas| - |\tbas| = N.
\end{equation}

\FigTex{algo-verify.tex}

The algorithm is implemented in SageMath~\cite{sage} and is publicly available in~\cite{OurWhiteboxCode}.

\paragraph{Complexity analysis.}
The truth tables have size $2^{N+R}$ bits. Computing the basis of $\FUNCSD{1}(C)$ takes time $\OO(min(2^{N+R}, |C|)^\matexp)$. The same holds for $\Ima{L_c}$ except that the vectors have size $2^{R}$ and for small $R$ this can be done more efficiently. The total complexity is $\OO(min(2^{N+R}, |C|)^\matexp + 2^N min(2^{R}, |C|)^\matexp)$. Recall that by Proposition~\ref{wba.prop:nonlinear-gates} we should consider only the nonlinear nodes of the circuit.