\documentclass{article}
\usepackage{graphicx}
\usepackage{amssymb,amsmath}
\usepackage{breqn}
\usepackage{fancyhdr}
\pagestyle{fancyplain}
\rhead{Prepared by Adam Beardsley on \today}

\begin{document}

\section{Image to PS Notes}

We start with an image of the sky, $I_{Jy}(\theta_x,\theta_y,f)$, in units of Jy / str. We can convert to temperature units using the Rayleigh-Jeans law.
\begin{equation}
I_T(\theta_x,\theta_y,f) = 10^{-23}\frac{c^2 \;\text{str}}{2 f^2 k_B} I_{Jy}(\theta_x,\theta_y,f)
\end{equation}
$A_b$ is the array beam in steradians. The factor of $10^{-23}$ in front is for the conversion from Jy to SI units ($10^{-26}$), and from K to mK ($10^{3}$). In this space we can easily convert angles and frequency to physical space (Mpc) using the following transformations (Morales and Hewitt, 2004).
\begin{subequations}\label{eq:transformations}
\begin{align}
r_x & = D_M(z) \theta_x \\
r_y & = D_M(z) \theta_y \\
r_z & = \frac{c(1+z)^2}{H_0 f_{21} E(z)} \Delta f
\end{align}
\end{subequations}
Where $r_z$ is measured relative to the closest distance line-of-sight of the observation, and $\Delta f$ is the channel frequency relative to the top of the observing band.

The next step is to Fourier transform the image to $k-$space. This is done in practice with a two dimensional discrete transform in the perpendicular direction and a FFT in the parallel direction. Mathematically all three steps can be written with a sum over all three dimensions of $\mathbf{r}$, where the indices $i,j$ indicate that $\mathbf{r}$ and $\mathbf{k}$ are discrete.
\begin{equation}
\tilde{I}(\mathbf{k}_j) = \sum_{\mathbf{r}_i} I(\mathbf{r}_i)e^{-i \mathbf{k}_j \cdot \mathbf{r}_i} \Delta r_x \Delta r_y \Delta r_z
\end{equation}
This expression has units of mK Mpc$^3$. I will drop the indices to simplify notation.

Next we square and relate to the power spectrum (e.g. Morales and Wyithe, 2010).
\begin{equation}
\left<\left|\tilde{I}(\mathbf{k})\right|^2\right> = \frac{1}{(2\pi)^3}\int P(\mathbf{k}')\left|W(\mathbf{k}-\mathbf{k}')\right|^2 d^3\mathbf{k}'
\end{equation}
Note in previous versions of this memo, the $2\pi$'s were omitted. Correspondence with Matt McQuinn confirmed they should be there.
The window function is compact and sharply peaked in $k-$space, so we approximate the power spectrum to be constant over the integral (e.g. Bowman, 2006). We then arrive at our estimate of $P(\mathbf{k})$.
\begin{equation}\label{eq:PS}
\hat{P}(\mathbf{k}) = \frac{\left<\left|\tilde{I}(\mathbf{k}\right|^2\right> }{ \frac{1}{(2\pi)^3}\int \left|\widetilde{W}(\mathbf{k}-\mathbf{k}')\right|^2 d^3\mathbf{k}'}
\end{equation}

The integral of the window function is determined by the bandwidth and field of view of the instrument (See e.g. Morales, 2005; Morales and Hewitt, 2004; and Bowman, 2006). In our units system this is given by
\begin{equation}
\frac{1}{(2\pi)^3} \int \left|\widetilde{W}(\mathbf{k}-\mathbf{k}')\right|^2 d^3\mathbf{k}' = \int \left|W(\mathbf{r}-\mathbf{r}')\right|^2 d^3 \mathbf{r}' \approx D_M(z)^2 \Omega \Delta D
\end{equation}
where $\Omega$ is the solid angle of the observation, and $\Delta D$ is the extent of the observation in the line-of-sight direction. This integral has units of Mpc$^3$. Thus our units for the power spectrum in Eq. \ref{eq:PS} are mK$^2$ Mpc$^3$ as expected.

\section{Detailed Window Function Calculation}
We can calculate the integrated window function for zenith pointing analytically using a MWA beam model. We treat a tile as a set of short dipoles sitting above a ground plane. The dipoles will have relative phases with respect to one another based on the direction of some source. We will calculate the response for an individual dipole at location $(x_i,y_i,h)$ relative to the center of the tile. The vertical component, $h$, is the height of the dipole, and we will assume it is the same for all dipoles on a single tile.

The phase difference for the dipole is found using simple geometry.
\begin{equation}
\Delta\phi_i = \frac{2\pi}{\lambda} (x_i \sin\theta_x+y_i \sin\theta_y)
\end{equation}

In addition to the phase differences, the ground plane also serves as an amplitude modulation. If the reflected wave is in phase with the incoming wave, the dipole will have twice the response as if the plane was not there. On the other hand, if the reflection is completely out of phase, the dipole will have zero response. We can derive the form of the amplitude modulation using geometry again. This time we find the phase difference between the incident wave and the reflected wave. The reflected part can be found by modeling the path difference between the true dipole above the plane and a virtual dipole below the plane.

\begin{equation}
\Delta D = 2 h \cos\theta_z
\end{equation}
Here $\theta_z$ is the zenith angle. The phase will be given by this path difference as well as a $\pi$ flip at the ground plane interface.
\begin{equation}
\Delta\phi_{r,i} = \pi + \frac{2 \pi}{\lambda} (2h \cos\theta_z)
\end{equation}

Because this phase is independent of dipole (assumed $h$ is the same for all dipoles), we can solve for the amplitude of the incident plus reflected wave.
\begin{subequations}
\begin{align}
A & = \left|1 + e^{i \Delta \phi_r} \right| \\
& = \left( (1 + \cos(\Delta \phi_r))^2 + (\sin(\Delta \phi_r))^2\right)^{1/2} \\
& = \left(1 + 2\cos(\Delta \phi_r) + \cos^2(\Delta\phi_r) + \sin^2(\Delta\phi_r) \right)^{1/2} \\
& = \left(2(1+\cos(\Delta \phi_r))\right)^{1/2} \\
& = 2 \left|\cos(\Delta \phi_r/2)\right| \\
& = 2 \left|\cos\left(\pi/2 + \frac{2 \pi}{\lambda}h \cos\theta_z\right))\right| \\
& = 2 \sin\left(\frac{2 \pi}{\lambda}h \cos\theta_z\right)
\end{align}
\end{subequations}

The dipole will be sensitive only to radiation with polarization aligned along its axis. Due to the orthogonality between direction of propagation and polarization, a non polarized source will appear polarized to the instrument if it is off zenith. For a dipole with single polarization, the amplitude attenuation is given by
\begin{equation}
P = 1-\sin(\theta_z)\cos(\theta_{az}),
\end{equation}
where $\theta_{az}$ is the azimuth angle.

The response for the dipole is then given by
\begin{subequations}
\begin{align}
R_i &= APe^{i \Delta \phi_i} \\
& = 2 \sin\left(\frac{2\pi}{\lambda}h\cos\theta_z\right) \mathrm{exp}\left[\frac{2\pi i}{\lambda}(x_i \sin\theta_x + y_i cos\theta_y)\right]\left(1-\sin(\theta_z)\cos(\theta_{az})\right)
\end{align}
\end{subequations}

Next we construct the response for the full tile by summing over the dipoles. We can leverage the symmetry of the tile by defining a horizontal space, $\delta_x$, which we will assume to be uniform.
\begin{subequations}
\begin{align}
R_{tile} &= \sum_i R_i \\
& = \sum_i 2 \sin\left(\frac{2\pi}{\lambda}h\cos\theta_z\right) \mathrm{exp}\left[\frac{2\pi i}{\lambda}(x_i \sin\theta_x + y_i cos\theta_y)\right]\left(1-\sin(\theta_z)\cos(\theta_{az})\right) \\
& = 2 \sin\left(\frac{2\pi}{\lambda}h\cos\theta_z\right) \sum_j \sum_k  \mathrm{exp}\left[\frac{2\pi i}{\lambda}(j \delta_x \sin\theta_x + k \delta_x cos\theta_y)\right]\left(1-\sin(\theta_z)\cos(\theta_{az})\right)
 \end{align}
\end{subequations}
The $j$ and $k$ sums are over the half integers -3/2, -1/2, 1/2, and 3/2.

In the approximation that the tiles are identical (and thus have the same $R_{tile}$), the window function is simply the magnitude squared of the response. We also normalize the window so that at phase center it is equal to unity. Further, the window function in general is a function of the frequency, but we will assume it is constant within a channel.
\begin{equation}
W(\theta_x,\theta_y,f) = \frac{\left|R_{tile}(\theta_x,\theta_y)\right|^2}{\left|R_{tile}(0,0)\right|^2}
\end{equation}

Next we can use the transformations from Eq. \ref{eq:transformations} to rewrite the window function in terms of cosmological axes ($\mathbf{r}$). With some simplification we arrive at the following for the window function squared.
\begin{dmath}
\left| W(\mathbf{r})\right|^2 = \frac{1}{256 \left|\sin\left(\frac{2\pi h}{\lambda}\right)\right|^4} \left| \sin\left(\frac{2\pi h \cos(\frac{\sqrt{r_x^2+r_y^2}}{D_m})}{\lambda}\right) \times \\ \\
\left(\cos\left(\frac{\pi \delta_x \sin(r_x/D_m)}{\lambda}\right)+\cos\left(\frac{3\pi \delta_x \sin(r_x/D_m)}{\lambda}\right) \right) \times \\ \\
\left(\cos\left(\frac{\pi \delta_x \sin(r_y/D_m)}{\lambda}\right)+\cos\left(\frac{3\pi \delta_x \sin(r_y/D_m)}{\lambda}\right) \right)\right|^4 \times \\ \\
\left(1-\sin\left(\frac{\sqrt{r_x^2+r_y^2}}{D_m}\right)\frac{r_y}{\sqrt{r_x^2+r_y^2}}\right)^4
\end{dmath}

This can then be integrated (probably numerically) to arrive at our estimate of the integrated window function (with the assumptions of identical tiles phased at zenith). As an example I compute this integral for observing parameters shown in Table \ref{tbl:obs_params}, and find
\begin{equation}
\int |W(\mathbf{r})|^2d^3 \mathbf{r} \approx 7.0 \times 10^8 \text{ Mpc}^3
\end{equation}

\begin{table}
\begin{center}
\begin{tabular}{l c c}
\hline
Parameter & Symbol & Value \\
\hline
Redshift & $z$ & 8 \\
Wavelength & $\lambda$ & 1.90 m \\
Dipole spacing & $\delta_x$ & 1.1 m \\
Dipole height & $h$ & 0.35 m \\
Bandwidth & $B$ & 8 MHz
\end{tabular}
\caption{Observing and instrument parameters for example calculation}
\label{tbl:obs_params}
\end{center}
\end{table}

\section{Calculating Window Function from Ian's Data Cubes}

We now aim to calculate the above integral from the data cubes that Ian produces with FHD. In particular, these are cubes of the weights and the variance of each pixel (in image space). Let the symbols for these cubes be $w$ and $\sigma^2$ respectively. I will continue to use a capital $W$ for the window function.

The first step is to Fourier transform the weights and variances to $(k_x,k_y,r_z)$ space. This is the space in which the weights and variances are non-covariant and easiest to manipulate.

We begin by relating the weights cube $w$ to the sampled window function. There may be a normalization difference, which we will solve for, and the weights account for all the gridded visibilities ($N_{\text{vis}}$) rather than a single beam. The same factors come into the variance cube. So we can write,
\begin{subequations} \label{eq:relate_w_to_W}
\begin{align}
\sum_{\text{beam}} W(k_{\perp},r_z) & = \frac{1}{N_{\text{vis}}} \sum_{\text{cube}} W_0 w(k_{\perp},r_z) \\
\sum_{\text{beam}} |W(k_{\perp},r_z)|^2 & = \frac{1}{N_{\text{vis}}} \sum_{\text{cube}} |W_0|^2 \sigma^2(k_{\perp},r_z)
\end{align}
\end{subequations}
where $W_0$ is the normalization constant.

Next we solve for $W_0$ by forcing $W(r_x=0,r_y=0,r_z) = 1$ (as in the previous section). The window function is $\mathbf{r}$ space is related to the window function in $(k_{\perp},r_z)$ space by a two dimensional inverse Fourier transform. Our convention for the Fourier transform did not have $2\pi$ in going from $k$ to $r$, so now we must include it.
\begin{equation}
W(\mathbf{r}) = \frac{1}{(2\pi)^2} \int W(k_{\perp},r_z) e^{i \mathbf{k}_{\perp} \cdot \mathbf{r}_{\perp}} d^2 k_{\perp}
\end{equation}
We can then evaluate when $r_{\perp}=0$ and apply our normalization.
\begin{equation}
W(r_{\perp}=0,r_z) =  \frac{1}{(2\pi)^2} \int W(k_{\perp},r_z) d^2 k_{\perp} = 1
\end{equation}
This integral can be approximated with a sum and related to the weights using Eq. \ref{eq:relate_w_to_W} to solve for $W_0$.
\begin{subequations} \label{eq:solve_W0}
\begin{align}
1 & = \frac{1}{(2\pi)^2} \int W(k_{\perp},r_z) d^2 k_{\perp} \\
& \approx  \frac{1}{(2\pi)^2} \sum_{\text{beam}} W(k_{\perp},r_z) (\Delta k_{\perp})^2 \\
& =  \frac{(\Delta k_{\perp})^2}{N_{\text{vis}}(2\pi)^2} \sum_{\text{cube}} W_0  w(k_{\perp},r_z) \\
\rightarrow W_0 & = \frac{N_{\text{vis}}(2\pi)^2}{(\Delta k_{\perp})^2} \frac{1}{\sum_{\text{cube}} w(k_{\perp},r_z)}
\end{align}
\end{subequations}
Note that this result is dependent on our normalization choice which is only true at zenith pointing and phase center. So the sum over the weights should be done for that particular case, and used for other pointings and phase centers in the following work.

Armed with a solution for $W_0$ we now seek to solve the integral in question.
\begin{subequations}
\begin{align}
\int |W(\mathbf{r})|^2 d^3 \mathbf{r} & = \frac{1}{(2\pi)^2} \int |W(k_{\perp},r_z)|^2 d^2k_{\perp}d r_z \\
& \approx \frac{\Delta D}{(2\pi)^2} \sum_{\text{beam}} |W(k_{\perp},r_z)|^2 (\Delta k_{\perp})^2 \\
& = \frac{\Delta D (\Delta k_{\perp})^2}{(2\pi)^2} \frac{1}{N_{\text{vis}}} \sum_{\text{cube}} |W_0|^2 \sigma^2(k_{\perp},r_z) \\
& = \frac{\Delta D (\Delta k_{\perp})^2}{(2\pi)^2 N_{\text{vis}}} \frac{N_{\text{vis}}^2(2\pi)^4}{(\Delta k_{\perp})^4} \frac{\sum_{\text{cube}} \sigma^2(k_{\perp},r_z)}{\left|\sum_{\text{cube}} w(k_{\perp},r_z)\right|^2}  \\
& = \frac{(2\pi)^2 \Delta D N_{\text{vis}}}{(\Delta k_{\perp})^2} \frac{\sum_{\text{cube}} \sigma^2(k_{\perp},r_z)}{\left|\sum_{\text{cube}} w(k_{\perp},r_z)\right|^2}
\end{align}
\end{subequations}
In the first step we used Parseval's theorem to relate the integral of the function squared in one space to the integral of the Fourier transform squared. The second step is approximating the integral as a sum, and doing the trivial integral over $r_z$ (the window is assumed to be constant over the band). Next we make use of Eq. \ref{eq:relate_w_to_W} to replace the sum over the window function. The fourth step is using the result of Eq. \ref{eq:solve_W0} to plug in for $W_0$. Finally we simplify in the last step. It is worth noting that the units of our final expression are Mpc$^3$, as expected.

This result should allow for a check on the data pipeline by comparing this calculation to that of the previous section. There are still a few outstanding questions (in my mind anyway):
\begin{itemize}
  \item Is the normalization correct? I believe there are assumptions in the noise calculations that require the window function to be normalized to the collecting area. How exactly is that done? \\ \\
 { \bf Answer:} The normalization is fixed by how we do our calibration. By saying that our apparent flux of a source on phase center is equal to the true flux of that source, we have set the normalization of the window function to be unity at phase center. So the assumption above is correct.
  \item In what space should the integral be computed? In the past we have said it doesn't matter if the integral is done in $r$ or $k$ space because of Parseval's theorem. However, there are $2\pi$'s relating the two spaces, and it is important to use the correct version.
  \item There are probably more\ldots will add when I think of them.
\end{itemize}

\section{A couple notes on W-projection}
For computational reasons, we seek to estimate the effect W-projection has on our longest baselines. Let us consider a single baseline, projected onto a plane perpendicular to the line of pointing.
\begin{figure}[h!]
\includegraphics[width=\columnwidth]{W_projection.png}
\caption{A projected baseline with non zero pointing.}
\label{fig:proj_base}
\end{figure}
Let $b$ represent the longest baseline, and $b'$ represent the projected baseline. We wish to find the ratio $b'/b$.

By inspection we can see that the projected baseline length can be determined from a pure geometric projection combined with the diffraction opening angle, $\alpha$. Mathematically,
\begin{equation}
b' = b\cos(\theta) + x = b\cos(\theta) + b\sin(\theta)\tan(\alpha).
\end{equation}
Alpha is approximately half of the primary beam width. So our ratio becomes
\begin{equation} \label{eq:bprimeb}
\frac{b'}{b} = \cos(\theta) + \sin(\theta)\tan(\mathrm{FoV}/2)
\end{equation}
We can plot this ratio as a function of $\theta$ for a conservative field of view of 40$^{\circ}$ (Fig. \ref{fig:bprimeb}).
\begin{figure}[h!]
\includegraphics[width=\columnwidth]{bprimeb.png}
\caption{Projected baseline ratio as a function of pointing angle.}
\label{fig:bprimeb}
\end{figure}

We are mostly interested in the maximum of this ratio. We set the derivative with respect to theta equal to zero and solve.
\begin{subequations}
\begin{align}
\frac{\partial}{\partial\theta}(b'/b)  = -\sin(\theta) + \cos(\theta)\tan(\mathrm{FoV}/2) &= 0 \\
 \tan(\theta_{max}) & = \tan(\mathrm{FoV}/2) \\
 \rightarrow \theta_{max} &= \mathrm{FoV}/2
\end{align}
\end{subequations}
Plugging this result back into Eq. \ref{eq:bprimeb}, we have the maximum ratio for a given field of view.
\begin{subequations}
\begin{align}
\left.\frac{b'}{b}\right\vert_{\theta_{max}} & = \cos(\mathrm{FoV}/2) + \sin(\mathrm{FoV}/2)\tan(\mathrm{FoV}/2) \\
& = \sec(\mathrm{FoV}/2)
\end{align}
\end{subequations}

\end{document}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{geometry}
\usepackage[utf8]{inputenc}
\usepackage{enumitem}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1em}

\begin{document}

\section{Functions in number theory}
\subsection{Rules}
\begin{enumerate}
\item
\textbf{Fermat's little theorem.} \\
For prime $p$ and integer $a$
$$a^p \equiv a \mod p$$

\item
\textbf{Wilson's theorem.} \\
$$(p-1)! \equiv -1 \mod p$$
if and only if when $p$ is prime number.

\item
\textbf{Number of factors} \\
The number of positive factors of $n=p_1^{\alpha_1} \dots p_k^{a_k}$
$$d(n)= (\alpha_1+1)(\alpha_2+1)\dots (\alpha_k+1)$$
\item
\textbf{Sum of factors} \\
The sum of positive factors of $n=p_1^{\alpha_1} \dots p_k^{a_k}$

$$\sigma(n)= \frac{p_1^{\alpha_1+1}-1}{p_1-1} \dots  \frac{p_k^{\alpha_k+1}-1}{p_k-1}$$

\item
\textbf{Euler's function} \\
Euler’s function or totient function $\varphi(n)$ is defined for $n=p_1^{\alpha_1} \dots p_k^{a_k}$ as the number
of positive integers less than $n$ and coprime to $n$. Then
$$\varphi(n) = n \left(1-\frac{1}{p_1}\right) \dots  \left(1-\frac{1}{p_k}\right)$$

\item
\textbf{Euler's theorem}  (Generalisation of Fermat's theorem)\\
Let $n$ be a natural number and $a$ an integer such that $\gcd(a,n)=1$. Then
$$a^{\varphi(n)} \equiv 1 \mod n $$

\end{enumerate}

\subsection{Problems}
\begin{enumerate}

\item
Find all primes $p$, for which the sum of all positive factors of $p^4$ is a perfect square.

\item
Prove that for positive integer $n$
$$\sum_{d|n} \varphi (d) =n $$

\item
Prove that for positive integers $a$ and $b$
$$ \varphi(ab) = \varphi (a)\varphi(b)\frac{\gcd(a,b)}{\varphi(\gcd(a,b))} $$

\item % https://artofproblemsolving.com/wiki/index.php?title=Fermat%27s_Little_Theorem#Problems
One of Euler's conjectures was disproved in the 1960s by three American mathematicians when they showed there was a positive integer such that $133^5+110^5+84^5+27^5=n^5$. Find the value of ${n}$.

\item  %p46 http://s3.amazonaws.com/aops-cdn.artofproblemsolving.com/resources/articles/olympiad-number-theory.pdf
How many prime numbers $p$ are there, such that $29^p+1$ is a multiple of $p$?

\item
Prove that if $\gcd(a,n)=1$, then
$$a^b \equiv a^{b \mod \varphi(n)}  \mod n$$

\item % http://s3.amazonaws.com/aops-cdn.artofproblemsolving.com/resources/articles/olympiad-number-theory.pdf
Find the last three digits of $2008^{2007^{2006^{\dots^{^{}2^{1}}}}}$.

\item
Prove that there exists no positive integer for which $n! + 19^{93}$ is a perfect square.

\item
Find all pairs of positive prime numbers $(p_1,p_2)$ for which the equation $$\phi(n^2) = n + p_1p_2$$
has a solution for $n$ in positive integers.

\item % http://www.math.olympiaadid.ut.ee/eng/archive/bw/bw05sol.pdf

Let $p$ be a prime number and let $n$ be a positive integer. Let $q$ be a positive divisor of $(n+1)^p-n^p$. Show that $q-1$ is divisible by $p$.

\end{enumerate}

\end{document}


%!TEX root =  ../main.tex

\subsection{Indeterminate Forms}

\objective{Classify difficult limits problems and solve via certain techniques}

Just as we used limits to define the derive six chapters ago, now they return the favor
and derivatives themselves allow us to find certain limits.  We have seen how removable
discontinuities take the form $\frac{0}{0}$, but what are we to do if a derivative
produce that form?  In fact, there are many forms which mathematicians classify
as \textbf{indeterminate}

$$
\frac{0}{0} , \frac{\infty}{\infty} , 0\times\infty , \infty-\infty , 0^0 , ^\infty \text{, and } \infty^0
$$

These are all indicative of a variety of answers, some finite, some infinite.  There is
no way to know what answer will predominate in the end without more work.
Consider this limit
\begin{equation}
\lim_{x\rightarrow0}\frac{5x}{\ln(x+1)}
\end{equation}
Obviously, if we attempt a direct evaluation, we get a meaningless answer.  But
consider what we know about this quotient function.  Every ``nudge'' in the $x$
direction (dx) will produce a small change in the top half of the fraction: 5dx, to
be precise.  Similarly, the change in the denominator (it's implicit differential)
is $\frac{1}{x+1}$dx.  As a fraction, this is $\cfrac{5dx}{\frac{1}{x+1}dx}$ or
$5(x+1)$.  Notice this is not like the maneuvering we did with removable
discontinuities; we have not constructed an identical but hole-less version
of the equation.  Our new equation is useful because it goes through the hole.

This technique for solving limits at indeterminate value is call \textbf{L'Hopital's}
Rule (LO-pee-talz), even though it was found by L'Hopital's collegue Bernoulli.

\personfeature[0in]{\chapdir/pics/Leopold_Kronecker_1865}{Leopold Kronecker}{1823 -
1891}{was a German mathematician who worked on number theory, algebra and logic.
He criticized Cantor's work on set theory, and said, ``God made the integers,
all else is the work of man.''  He studied elliptic functions and conjectured his
``dearest dream of youth", a generalization that was later put forward by
Hilbert in a modified form as his twelfth problem.
\href{https://en.wikipedia.org/wiki/Leopold_Kronecker}{Wikipedia}}

\subsection{Infinite Forms}
Not every form with a zero in the denominator or an infinity in it is indeterminate.
The following forms resolve in other ways, and cannot be solved with L'Hopital's
Rule:
$$
\frac{1}{0} , \frac{\infty}{1} , \frac{\infty}{0} , \infty + \infty , \infty \times\infty
$$

Some other fractions you should recognize from algebra and/or experience as zero:
$$
\frac{0}{1} = \frac{1}{\infty} = \frac{0}{\infty}
$$

To return to our earlier equation of $\frac{5x}{\ln(x+1)}$, what is the limit as $x$
approaches infinity?  Logarithmic graphs \emph{seem} to taper off, and direct
evaluation is useless (plugging in infinity for $x$) because we get $\frac{\infty}{\infty}$.
But L'Hopital's Rule works here too, because that is an indeterminate form.
Our limit will be the same as
\begin{equation}
\lim_{x\rightarrow0}5(x+1)
\end{equation}
which is infinite.


\section{Previous work}
\label{sec:previous:work}

First of all, we recall some modular arithmetic fundamentals. Let
$a,b\in\mathbb{Z}$ and $n\in\mathbb{N}$, \textit{$a$ is congruent to $b$ modulo
$n$}, written as $a \equiv_{n} b$, if $a \mod n = b \mod n$; from this, follows
that $n \left| (a-b)\right.$ and $a - b = nk$, for some $k\in\mathbb{Z}$, are
equivalent definitions.

%\subsection{Lucas theorem exposed}

In \cite{fine:1947}, Fine exposes the following theorem which is a fundamental
result for our work,

\begin{theorem}[Lucas] Let $p$ be a prime and $m,n\in\mathbb{N}$, then
    \begin{displaymath}
        {{m}\choose{n}} \equiv_{p}
            {{m_{0}}\choose{n_{0}}}
            {{m_{1}}\choose{n_{1}}}
            \cdots
            {{m_{k}}\choose{n_{k}}}
    \end{displaymath}
    representing $m=\left(m_{0},m_{1},\ldots,m_{k}\right)_{p}$ and
    $n=\left(n_{0},n_{1},\ldots,n_{k}\right)_{p}$ in base $p$, for some $k\in\mathbb{N}$.
    \iffalse
    So both $m_{j}$ and $n_{j}$ are in $\lbrace 0,\ldots, p-1 \rbrace$, for
    $j\in\lbrace 0,\ldots,k\rbrace$.
    \fi
    \label{thm:lucas:theorem}
\end{theorem}

\iffalse
\begin{proof}
    Consider the sequence
    $\vect{\alpha}_{m}=\left({{m}\choose{n}}\right)_{n\in\mathbb{N}}$ of
    binomial coefficients which count the number of $n$-elements subsets out of
    a set composed by $m$ elements. We use $\vect{\alpha}_{m}$ to
    define a formal power series on the indeterminate variable $t$ and its generating
    function
    \begin{displaymath}
        %\begin{split}
            \sum_{n=0}^{m}{{{m}\choose{n}}\,t^{n}} %&
                = \left(1+t\right)^{m}
                = \left(1+t\right)^{m_{0}+m_{1}p+\cdots+m_{k}p^{k}}%\\
                %&
                = \prod_{r=0}^{k}{{\left(1+t\right)^{m_{r}p^{r}}}}
                = \prod_{r=0}^{k}{\left(\left(1+t\right)^{p^{r}}\right)^{m_{r}}}
        %\end{split}
    \end{displaymath}
    where $m=\left(m_{0},m_{1},\ldots,m_{k}\right)_{p}$ is represented in
    base $p$. The expansion of $\left(1+t\right)^{p^{r}}$ yields
    \begin{displaymath}
            \left(1+t\right)^{p^{r}} = \sum_{s=0}^{p^{r}}{{{p^{r}}\choose{s}}\,t^{s}}
                = 1+\sum_{s=1}^{p^{r}-1}{{{p^{r}}\choose{s}}\,t^{s}}+t^{p^{r}}
    \end{displaymath}
    and the application of the congruence relation $\equiv_{p}$ makes the last sum
    vanish because $\displaystyle p\left|{{p^{r}}\choose{s}}\right.$, therefore
    $\left(1+t\right)^{p^{r}} \equiv_{p} 1+t^{p^{r}}$ holds and it allows us to rewrite
    the products as
    \begin{displaymath}
            \prod_{r=0}^{k}{\left(\left(1+t\right)^{p^{r}}\right)^{m_{r}}}
                \equiv_{p} \prod_{r=0}^{k}{\left(1+t^{p^{r}}\right)^{m_{r}}}
                \equiv_{p} \prod_{r=0}^{k}{\sum_{s_{r}=0}^{m_{r}}{{{m_{r}}\choose{s_{r}}}t^{s_{r}p^{r}}}}.
    \end{displaymath}
    there is a product of $k+1$ formal power series, properly shifted according to
    $p_{0},p_{1},\ldots,p_{k}$: such product yields a new formal power series which satisfies
    \begin{displaymath}
        \sum_{n=0}^{m}{{{m}\choose{n}}\,t^{n}}
        \equiv_{p}
        \sum_{n=0}^{m}{\left(\sum_{\vect{\omega}\in\Omega_{n}}{\prod_{i=0}^{k}{{{m_{i}}\choose{\omega_{i}}}}}\right)\,t^{n}}
    \end{displaymath}
    where $\Omega_{n}=\left\lbrace\vect{\omega}=\left(\omega_{0},\ldots,\omega_{k}\right):
        \sum_{j=0}^{k}{\omega_{j}p^{j}}=n\right\rbrace$.
    There exists a \emph{unique} set of coefficients
    $\lbrace n_{0},n_{1},\ldots,n_{k}\rbrace$ such that $n=\sum_{j=0}^{k}{n_{j}p^{j}}$, therefore
    each set $\Omega_{n}$ contains only one element, namely $\vect{\omega}=(n_{0},\ldots,n_{k})$, so
    \begin{displaymath}
        \sum_{n=0}^{m}{{{m}\choose{n}}\,t^{n}}
        \equiv_{p} \sum_{n=0}^{m}{\left({\prod_{i=0}^{k}{{{m_{i}}\choose{n_{i}}}}}\right)\,t^{n}}
    \end{displaymath}
    holds and the argument follows by equating coefficients attached to $t^{n}$,
    %for $n\in\lbrace0,\ldots,m\rbrace$,
    as required.

\end{proof}
\fi

%\subsection{Divisibility -- with visibility}

In \cite{sved:1988}, Marta Sved gives solid bases to understand
\textit{divisibility properties} of some \emph{counting numbers}, namely
coefficients occurring in combinatorics such as binomials and Stirling numbers,
that shows divisibility structures of remarkable design.

She starts working on the Pascal triangle $\mathcal{P}$, whose coefficients
are defined by the recurrence
\begin{equation}
    {{n}\choose{k}}={{n-1}\choose{k-1}}+{{n-1}\choose{k}},
    \label{eq:binomial:recurrence}
\end{equation}
and studies a new triangle $\mathcal{P}_{\equiv_{p}}$ obtained from
$\mathcal{P}$ where each coefficient is taken modulo $p$, for some small prime
number $p$. Besides, she introduces a little language that allows her to
characterize these "modular" triangles and we readly adopt it as well.

Let $\mathcal{M}=\left(m_{n,k}\right)_{n,k\in\mathbb{N}}$ be an infinite, lower
triangular array so that $m_{n,k}=0$ if $n<k$. Choose a prime $p$, then the
array $\displaystyle\mathcal{M}_{\equiv_{p}} = \left(m_{n,k}\mod p :
m_{n,k} \in \mathcal{M}\right)_{n,k\in\mathbb{N}}$ contains the following regions
\begin{itemize}

\item a \emph{principal cell} $\mathcal{M}_{\equiv_{p}}^{\bigtriangleup} =
        \left(m_{n,k}\in \mathcal{M}_{\equiv_{p}}\right)_{n,k\in\lbrace 0,\ldots,p-1\rbrace}$;

\item a \emph{cell} $c_{p,\beta}=\left( \beta\,m_{n,k}\mod p:m_{n,k}\in
\mathcal{M}_{\equiv_{p}}^{\bigtriangleup}\right)_{n,k\in\mathbb{N}}$ which is a
copy of the \emph{principal cell} with each entry multiplied by the same
constant $\beta\in\mathbb{N}$;

\item a \emph{principal cluster} $\mathcal{M}_{\equiv_{p}}^{(\alpha)} =
\left(m_{n,k}\in \mathcal{M}_{\equiv_{p}}\right)_{n,k\in\lbrace
0,\ldots,p^{\alpha}-1\rbrace}$ of order $\alpha\in\mathbb{N}$, so that for
$\alpha=1$ it denotes the \emph{principal cell};

\item a \emph{cluster} $\mathcal{C}_{p,\beta}^{(\alpha)}=\left(
\beta\,m_{n,k}\mod p:m_{n,k}\in
\mathcal{M}_{\equiv_{p}}^{(\alpha)}\right)_{n,k\in\mathbb{N}}$ of order
$\alpha\in\mathbb{N}$ which is a copy of the \emph{principal cluster} of order
$\alpha$ with each entry multiplied by the same constant $\beta\in\mathbb{N}$;

\item finally, a \emph{zero-hole} of order $\alpha\in\mathbb{N}$ which is an
\emph{upside-down} triangular array of coefficients where each one is a
multiple of $p$, from here the name, distributed according to the scheme
(i)~$p^{\alpha}-1$ coefficients lie on the first row, (ii)~$p^{\alpha}-2$
coefficients lie on the second row down to (iii)~a single coefficient that lies
on the $(p^{\alpha}-1)$-th row, eventually.  Observe that a \emph{zero-hole} of
order $\alpha$ has $p^{\alpha}-1$ rows, one less than a \emph{principal
cluster} of the same order.

\end{itemize}

Although a formal proof can be stated by induction using
\autoref{eq:binomial:recurrence}, Marta shows that the following  theorem  has
an easier formulation using the proposed language.

\begin{theorem}[Sved]
    Let $p$ be a prime and $\mathcal{P}_{\equiv_{p}}$ be the
    modular Pascal array. The principal cluster $\mathcal{P}_{\equiv_{p}}^{(\alpha+1)}$
    of order $\alpha + 1$ consists of $p$ layers of clusters of order $\alpha$ alternating
    with zero-holes of order $\alpha$, while the first layer contains the principal
    cluster of order $\alpha$ only.
\end{theorem}


%================================
\section{Continuous Maps}
%================================

%--------------------------------
\begin{definition}
	Let $(X, \mathcal T_X)$ and $(Y, \mathcal T_Y)$ be topological spaces. A function $f: X \to Y$ is \textit{continuous} iff
	$$
	\forall U \in \mathcal T_Y, \quad f^{-1}[U] \in \mathcal T_X.
	$$
\end{definition}
%--------------------------------

%--------------------------------
\begin{note} \
	\begin{enumerate}
		\item Let $(X, \mathcal T)$ and $(X, \mathcal T')$ be topological spaces, let identity map $i:(X, \rho) \to (X, \rho')$ be surjective. By definition, $i$ is continuous over $X$ iff $\mathcal T \supseteq \mathcal T'$.
		\item Let $(X, \mathcal T_X)$ and $(Y, \mathcal T_Y)$ be topological spaces if $\mathcal T_X$ is the discrete topology, then any function $f: X \to Y$ is continuous over $X$, because any sets in $X$ is open. Naturally, for any $U \in \mathcal T_Y$, $f^{-1}[U]$.
		\item By the previous conditions, if $\mathcal T_X$ is indiscrete topology but $\mathcal T_Y$ is not, then no function $f: X \to Y$ is continuous. (Check again!)
		\item TO DO: Find valued example by point sets, for example let $\mathcal T = \{a, b, c, d, e\}$.
	\end{enumerate}
\end{note}
%--------------------------------

%--------------------------------
\begin{lemma}
	Let $(X, \mathcal T_X)$ and $(Y, \mathcal T_X)$ be topological space. A function: $f:X\to Y$ is continuous iff for any closed subset $V$ of $Y$, the preimage $f^{-1}[V]$ is closed in $X$.
\end{lemma}
%--------------------------------

%--------------------------------
\begin{proof}
	If $V$ is closed in $Y$, then there is a $U \in \mathcal T_Y$, such that $V = Y \setminus U$.
	$$
	f^{-1}[V] = f^{-1}[Y \setminus U] = X \setminus f^{-1}[U] \in \mathcal T_Y.
	$$
\end{proof}
%--------------------------------

%--------------------------------
\begin{lemma}
	Continuous maps preserve convergence of sequence..
\end{lemma}
%--------------------------------

%--------------------------------
\begin{note}
	This is not vise versa, i.e., there exists discontinuous map preserve convergence of sequence either.

	For example, let $(X, \mathcal T)$ and $(X, \mathcal T')$ be topological space. Let $\mathcal T'$ be discrete topology, and let
	$$
	\mathcal T = \{ \text{monotones of $X$} \} \cup \{ \emptyset, X \}.
	$$
	Then any $\{x_n\} \subseteq X$ converges iff it is in the form of
	$$
	x, x, x, \ldots
	$$
	Clearly, for all $f:(X, \mathcal T) \to (X, \mathcal T')$, for any $\{x_n\}$ converges to $x \in (X, \mathcal T)$, $\{f(x_n)\}$ also converges to $f(x) \in (X, \mathcal T')$. It is in the form of
	$$
	f(x), f(x), f(x), \ldots
	$$
	But, $f$ is not continuous over $(X, \mathcal T)$. (For any $U \subsetneq X$ with $|U|>1$, $f^{-1}[U] \notin \mathcal T$).
\end{note}
%--------------------------------

%


\documentclass{pset_template}

\title{Boolean Algebra}
\date{February 1, 2019}
\editorOne{Alexander Sun}
\editorTwo{Sanjit Bhat}
\lectureNum{1}
\contestMonth{February}

\begin{document}
\maketitle

\section{Introduction}
Boolean algebra is the branch of algebra in which the variables store truth value. All variables are true(1) or false(0).
There are 3 main operations that create the base for boolean algebra: $\textbf{AND(conjunction)}$, $\textbf{OR(disjunction)}$, and $\textbf{NOT(negation)}$.

\bigskip
\noindent
\section{Basic Operations}
\textbf{AND}, denoted x $\land$ y or x AND y or x $\cdot$ y, satisfies x $\land$ y = 1 if x = y = 1 , else x $\land$ y = 0

\noindent
\textbf{OR}, denoted x $\lor$ y or x OR y or x + y, satisfies x $\lor$ y = 0 if x = y = 0 , else x $\lor$ y = 1

\noindent
\textbf{NOT}, denoted $\neg$x or NOT x or $\sim$x or $\bar{x}$, satisfies $\neg$x = 0 if x = 1 and $\neg$x = 1 if x = 0, reverses truth values of operation

\begin{center}
\begin{tabular}{ |c|c|c|c| }
 \hline
 x & y & x$\land$y & x$\lor$y \\
 \hline
  0 & 0 & 0 & 0 \\
 \hline
 1 & 0 & 0 & 1 \\
 \hline
 0 & 1 & 0 & 1 \\
 \hline
 1 & 1 & 1 & 1 \\
 \hline
\end{tabular}
\end{center}

\begin{center}
\begin{tabular}{ |c|c| }
 \hline
 x & $\neg$x \\
 \hline
  0 & 1  \\
 \hline
 1 & 0 \\
 \hline
\end{tabular}
\end{center}

\section{Secondary Operations}

\textbf{Material Implication}, denoted  x $\rightarrow$ y = $\neg$x$\lor$y, if x = 1, then x $\rightarrow$ y = y, if x = 0, then x $\rightarrow$ y = 1

\noindent
\textbf{Exclusive Or}, denoted x $\oplus$ y or x XOR y, x $\oplus$ y = 1 if x = 1 $\&$ y = 0 or x = 0 $\&$ y = 1, else x $\oplus$ y = 0, true when values are different

\noindent
\textbf{Equivalence}, denoted x $\equiv$ y, x $\equiv$ y = 1 if x =1 $\&$ y =1, or if x = 0 $\&$ y = 0, complement of XOR, true when values are the same
\noindent
\textbf{Dual}, the dual is found by replacing all OR's with AND's and all AND's with OR's, and all 1's with 0's and all 0's with 1's

\noindent
\textbf{Complement}, found by negating each individual value and replaving all OR's with AND's and all AND's with OR's and all 1's with 0's and all 0's with 1's
\bigskip

\noindent
Memorizing boolean algebra laws is extremely beneficial to being able to solve problems quickly and efficiently. Here is a link to page with almost every law. Most are derivable, but should still be memorized.  http://www.uiltexas.org/files/academics/UILCS-BooleanIdentities.pdf

Demorgan's Rule is crucial to simplifying boolean algebra problems. It states: $$ \bar{A} + \bar{B} = \overline{AB}$$ or $$ \bar{A} * \bar{B} = \overline{A+B}$$

\bigskip

\noindent
With these basic rules memorized, all boolean algebra problems should be simple to work through

\section{Exercises}
\begin{enumerate}
\item Simplify completely:(ACSL 2001-2002)
$( A + B )\oplus A B$

\item Simplify the following expression: F = BC + $\overline{BC}$ + BA

\item Simplify the Boolean expression (A+B+C)$\overline{(D+E)}$ + (A+B+C)(D+E):
\end{enumerate}

For more practice resources on boolean algebra, read through
\href{http://www.categories.acsl.org/wiki/index.php?title=Boolean_Algebra}{the ACSL Wiki page}.
\end{document}


%
% Copyright 2018 Parakram Majumdar
%
% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at
%
%     http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.
%

\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{tikz}
\usetikzlibrary{shapes.geometric, arrows,positioning}
\title{Multi Layered Perceptron}
\author{Parakram Majumdar}

\begin{document}
  \maketitle

\newcommand{\R}{\mathbb{R}}
\newcommand{\Prob}[2]{\mathbb{#1}\left[ #2 \right]}
\newcommand{\sigmoid}[1]{\text{sigmoid}\left(#1\right)}
\newcommand{\map}[2]{\text{map}\left(#1,#2\right)}

\section{Introduction}
  We are given a training set $T$ of $t$ points in $\R^d$,
  a set $L$ of $l$ labels,
  and a \emph{classification} $C: T \rightarrow L$.

  Machine learning is interested in learning classifiers of the form
  $f: \R^d \rightarrow L$ that can closely approximate $C$.

\section{Single Perceptron, Single Point, Single Label}
  Having truly one label is an uninteresting case,
  as basically all points would have the same label,
  and there is nothing to ``learn''.
  Instead we assume two labels, $L_1$ and $L_2$.
  We then train a classifier for $L_1$,
  and assign to $L_2$ whatever is not in $L_1$.

  Given a single point $p \in \R^d$,
  a single perceptron $c$
  with \emph{weight} vector $w_c \in \R^d$,
  and a \emph{bias} $b_c \in \R$,
  for classifying a single Label $L_1$,
  accepts $p$ with the probability
  \[ \hat y_{p,L_1} = \Prob{\hat P}{p \in L_1} =  \sigmoid{w_c \cdot p + b_c}\]
  where
  \[ \sigmoid{x} = \frac{e^x}{e^{x} + 1} \]
  If the actual classification of the training point
  is given by $y_{p,L_1} = \Prob{I}{p \in L_1}$
  then training criterion is to minimize
  the cross entropy:
  \begin{eqnarray*}
    \mathbb{H} & = & \Prob{H}{\mathbb{P}, \mathbb{\hat P}} \\
               & = & \Prob{E}{\log{ \frac{1}{\hat y}}} \\
               & = & \sum_{L_i \in L}{\Prob{H}{y_{p, L_i}, \hat y_{p, L_i}}}\\
               & = & -\sum_{L_i \in L}{y_{p, L_i}\log{\hat y_{p, L_i}}}\\
               & = & -y_{p,L_1}\log(\hat y_{p,L_1}) - (1 - y_{p,L_1})\log(1 - \hat y_{p,L_1})
   \end{eqnarray*}

\section{Multiple Points}
  Given a set of $t$ points
  represented as a 2 dimensional tensor $T \in \R^{t\times d}$,
  the perceptron $c$ can then be \emph{applied}
  to all the points in $T$ to get
  \[ \hat y_{T,c} = \map{T \cdot w_c + b_c\overrightarrow{u_t}}{\sigmoid \cdot} \]
  where
  \begin{itemize}
    \item $\map{\overrightarrow x}{f}$ is
            the element-wise application of $f$ on $\overrightarrow x$
    \item $\overrightarrow{u_n} = [1, 1, 1, ... n \text{ times}]$
  \end{itemize}
  The $i^{th}$ element of $\hat y_{T, c}$ then gives us
  the probability of the $i^{th}$ point being in $L_1$.
  \[ \hat y_{T, c}[i] = \Prob{\hat P}{T[i] \in L_1} = \hat y_{T[i], c}\]
  And therefore, if the actual classification
  of all training points is given by
  \[ y_{T,c} = \map{T}{\Prob{I}{\cdot \in L_1}} \]
  then the training criterion is to minimize the cross entropy
  \begin{eqnarray*}
    \mathbb{H} & = & \sum_{i=1}^t \Prob{H}{y_{T, c}[i], \hat y_{T, c}[i]} \\
               & = & \text{sum}(\text{map}(y_{T, c},
                                            \hat y_{T, c},
                                            \Prob{H}{\cdot, \cdot}
                                           )
                               )
  \end{eqnarray*}
  where
  \begin{itemize}
    \item $\text{sum}(x)$ gives the sum of all the elements
          in the tensor $x$
    \item $\text{map}(x, y, f(\cdot, \cdot))$
          is the element-wise application of
          the binary function $f$ to the elements of $x$ and $y$.
          In other words,
          \[ \text{map}(x, y, f(\cdot, \cdot))[i] = f(x[i], y[i]) \]
  \end{itemize}

\section{Multiple Labels}
  Classifying a point across multiple labels may be achieved
  using multiple perceptrons.
  However, if we simply use the multiple perceptrons independently,
  we end up with unconstrained probabilities
  for a point belonging to each label.
  Instead, we would ideally want the probabilities to add up to 1,
  since a point can only belong to one label.

  Suppose we have $l$ labels, and corresponding $l$ perceptrons.
  Each perceptron produces an output $x_i \in \R^t$,
  where $t$ is the number of points.
  Then the probability of point $T_j$
  belonging to label $L_k$
  is
  \[ \hat y_{T_j, L_k} =
        \frac{e^{x_k[j]}}
             {\sum_{m = 1}^{l} e^{x_m[j]} }
  \]

  To simplify this notation, we introduce the softmax function:
  \begin{eqnarray*}
    \text{softmax}(\overrightarrow z)
      & = & \text{softmax}(z_1, z_2, ... z_n)
    \\
      & = & \left[ \frac{e^{z_1}}{\sum_{i=1}^n e^{z_i}},
                   \frac{e^{z_2}}{\sum_{i=1}^n e^{z_i}},
                   ...,
                   \frac{e^{z_n}}{\sum_{i=1}^n e^{z_i}}
            \right]
    \\
    \text{softmax}(\overrightarrow z; j)
      & = & \frac{e^{z_j}}{\sum_{i=1}^n e^{z_i}}
  \end{eqnarray*}

  And concisely represent
  all the probabilities as:
  \[\hat y_{T, L} = \text{map}(x_1, x_2, ..., x_l, \text{softmax})\]

  Therefore, the training criteria is to minimize the cross-entropy:
  \[ \mathbb{H}(\hat y_{T, L}, y_{T, L})
       = \sum_{T_i \in T, L_j \in L} \mathbb{H}(\hat y_{T_i, L_j},
                                                y_{T_i, L_j})
  \]

\section{Multiple Layers of Perceptrons}
  Multiple layers of perceptrons,
  with non-linear activation functions, are necessary
  for classifying data that is not linearly separable.
  An activation function is applied to the output of a neuron
  before feeding it to the next layer.

\section{Sample 2-Layer Perceptron}
  Let the input data be organized
  into a 2-dimensional structure
  $T \in \R^{t \times d}$,
  where $t$ is the number of points
  and $d$ is the dimensionality of each point.

  We feed these points to the first layer
  consisting of, say, $m$ perceptrons.
  Thus, we have $m$ weight vectors:
  \[ w_{1, 1}, w_{1, 2}, ..., w_{1, m} \in \R^d \]
  and $m$ biases:
  \[ b_{1, 1}, b_{1, 2}, ..., b_{1, m} \in \R \]
  to give the unactivated output
  $p_{1, i} \in \R^{t}$
  of the $i^{th}$ perceptron:
  \[ p_{1, i} = T \cdot w_{1, i} + b_{1, i}\overrightarrow u_{t} \]

  These unactivated outputs $p_{1, \cdot}$ are passed
  through a sigmoid activation function
  to get the final activated outputs
  $z_{1, \cdot} \in R^{t}$
  of this layer:
  \[ z_{1, i} = \text{map}(p_{1, i}, \sigmoid{\cdot})
  \]

  These outputs $z_{1, \cdot}$ then become
  the inputs for the next and final layer.
  For convenience, we define
  $z_1 \in \R^{t \times m}$
  as the output vector of the first layer,
  created by concatenating all the individual outputs.

  Since se have $l$ labels for classification,
  the final layer should have $l$ perceptrons.
  Thus, we have $l$ weight vectors:
  \[ w_{2, 1}, w_{2, 2}, ..., w_{2, l} \in R^m \]
  and $l$ biases:
  \[ b_{2, 1}, b_{2, 2}, ..., b_{2, l} \in R \]
  to give the unactivated outputs $p_{2, \cdot} \in \R^t$:
  \[ p_{2, i} = z_1 \cdot w_{2, i} + b_{2, i} \overrightarrow u_{t} \]

  We leave the outputs unactivated for this final layer,
  and concatenate the outputs to get $p_2 \in {R^{t \times l}}$
  Thus, the probabilities $\hat y \in \R^{t \times l}$
  of the points belonging to classes is given by:
  \[ \hat y = \text{map}(p_2, \text{softmax}(\cdot)) \]

  If the real probabilities are given by $y \in R^{t \times l}$
  then the training criterion is to minimize the cross entropy:
  \[
    \mathbb{H}
    = \Prob{H}{\hat y, y}
    = \sum_{i \in T, j \in L} \Prob{H}{\hat y_{i, j}, y_{i, j}}
  \]

  \tikzstyle{io} = [circle,
                    text centered,
                    draw=black]
  \tikzstyle{perceptron} = [rectangle,
                            text centered,
                            draw=black,
                            minimum width=1.5cm,
                            minimum height=.8cm,
                            fill=blue!20]
  \tikzstyle{weight} = [rectangle,
                        text centered,
                        draw=black,
                        fill=yellow!30,
                        rounded corners]
  \tikzstyle{arrow} = [thick,->,>=stealth]

  \begin{tikzpicture}[node distance=2cm]
  \node (input) [io, fill=green!20] {$T$};
  \node (inputlabel) [above of=input] {Input};

  \node (p12) [perceptron, right=1cm of input] {$z_{12}$};
  \node (p11) [perceptron, above=2cm of p12] {$z_{11}$};
  \node (p13) [perceptron, below=2cm of p12] {$z_{13}$};
  \node (w11) [weight, below=.5cm of p11] {$w_{11}, b_{11}$};
  \node (w12) [weight, below=.5cm of p12] {$w_{12}, b_{12}$};
  \node (w13) [weight, below=.5cm of p13] {$w_{13}, b_{13}$};
  \node (p1label) [above=0.2cm of p11] {Layer 1};

  \node (p22) [perceptron, right=1cm of p12] {$p_{22}$};
  \node (p21) [perceptron, above=2cm of p22] {$p_{21}$};
  \node (p23) [perceptron, below=2cm of p22] {$p_{23}$};
  \node (w21) [weight, below=.5cm of p21] {$w_{21}, b_{21}$};
  \node (w22) [weight, below=.5cm of p22] {$w_{22}, b_{22}$};
  \node (w23) [weight, below=.5cm of p23] {$w_{23}, b_{23}$};
  \node (p2label) [above=0.2cm of p21] {Layer 2};

  \node (output) [perceptron, right=1cm of p22] {$\hat y$};
  \node (softmaxlabel) [above of=output] {Prediction};

  \node (error) [io, right=1cm of output, fill=red!20] {$\mathbb{H}$};
  \node (errorlabel) [above of=error] {Error};
  \node (y) [io, below=.5cm of error, fill=green!20] {$y$};

  \draw [arrow] (input) -- (p11.west);
  \draw [arrow] (input) -- (p12.west);
  \draw [arrow] (input) -- (p13.west);

  \draw [arrow] (p11.east) -- (p21.west);
  \draw [arrow] (p11.east) -- (p22.west);
  \draw [arrow] (p11.east) -- (p23.west);

  \draw [arrow] (p12.east) -- (p21.west);
  \draw [arrow] (p12.east) -- (p22.west);
  \draw [arrow] (p12.east) -- (p23.west);

  \draw [arrow] (p13.east) -- (p21.west);
  \draw [arrow] (p13.east) -- (p22.west);
  \draw [arrow] (p13.east) -- (p23.west);

  \draw [arrow] (p21.east) -- (output);
  \draw [arrow] (p22.east) -- (output);
  \draw [arrow] (p23.east) -- (output);

  \draw [arrow] (w11) -- (p11);
  \draw [arrow] (w12) -- (p12);
  \draw [arrow] (w13) -- (p13);
  \draw [arrow] (w21) -- (p21);
  \draw [arrow] (w22) -- (p22);
  \draw [arrow] (w23) -- (p23);

  \draw [arrow] (output) -- (error);
  \draw [arrow] (y) -- (error);
  \end{tikzpicture}

\end{document}


\subsection{Geometric description of the cross product}

The following is the geometric description of the cross
product. Recall that the dot product of two vectors results in a
scalar. In contrast, the cross product results in a vector, as the
cross product gives a direction as well as a magnitude.%
\index{cross product!geometric description}

\begin{definition}{Geometric definition of cross product}{cross-product-geometric}
  Let\/ $\vect{u}$ and $\vect{v}$ be two vectors in $\R^3$. Their
  \textbf{cross product}%
  \index{cross product}, written $\vect{u}\times \vect{v}$, is the
  vector defined by the following three rules.%
  \index{cross product!geometric description}

  \begin{enumerate}
  \item Its length is $\norm{\vect{u}\times \vect{v}} =\norm{\vect{u}} \norm{
      \vect{v}} \sin \theta$,
    where $\theta$ is the included angle between $\vect{u}$ and $\vect{v}$.

  \item It is orthogonal to both $\vect{u}$ and $\vect{v}$.

  \item The vectors $\vect{u}$, $\vect{v}$, and $\vect{u}\times
    \vect{v}$, in that order, form a right-handed system.
  \end{enumerate}
\end{definition}

We note that the length of the cross product,
$\norm{\vect{u}\times\vect{v}}$, given by the formula
$\norm{\vect{u}}\norm{\vect{v}}\sin\theta$, is the area of the
parallelogram determined by $\vect{u}$ and $\vect{v}$, as shown in the
following picture.%
\index{cross product!area of parallelogram}%
\index{area!of parallelogram}%
\index{parallelogram!area of}%
\begin{center}
  \begin{tikzpicture}[scale=1.3]
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (0:8mm) arc (0:45:8mm) -- cycle;
    \draw[->,thick,red] (0,0) -- node[left] {$\vect{v}$} (2,2);
    \draw[->,thick,blue] (0,0) -- node[below] {$\vect{u}$} (3,0);
    \draw[thick] (2,2)--(5,2);
    \draw[thick] (3,0)--(5,2);
    \draw[dashed] (2,2)--(2,0);
    \node[green!30!black] at (22.5:5.5mm) {$\theta$};
    \node[right] at (2,1){$\norm{\vect{v}} \sin(\theta)$};
  \end{tikzpicture}
\end{center}


\section{Function extensionality}
\label{chap:funext}

\subsection{Equivalent forms of function extensionality}

\begin{axiom}[Function Extensionality]\label{axiom:funext}
  \index{function extensionality}\index{identity type!of a Pi-type@{of a $\Pi$-type}}\index{extensionality principle!for functions}%
  For any type family $B$ over $A$, and any two dependent functions $f,g:\prd{x:A}B(x)$, the canonical map\index{htpy-eq@{$\htpyeq$}}\index{htpy-eq@{$\htpyeq$}!is an equivalence}\index{is an equivalence!htpy-eq@{$\htpyeq$}}
  \begin{equation*}
    \htpyeq:(f=g)\to (f\htpy g)
  \end{equation*}
  that sends $\refl{f}$ to $\reflhtpy_{f}$ is an equivalence. We will write $\eqhtpy$\index{eq-htpy@{$\eqhtpy$}} for its inverse.
\end{axiom}

In other words, the axiom of function extensionality asserts that for any two dependent functions $f,g:\prd{x:A}B(x)$, the type of identifications $f=g$ is equivalent to the type of homotopies $f\htpy g$ from $f$ to $g$. By the fundamental theorem of identity types (\cref{thm:id_fundamental})\index{fundamental theorem of identity types} there are three equivalent ways of asserting function extensionality. In the following theorem we state one further equivalent condition.

\begin{thm}\label{thm:funext_wkfunext}
The following are equivalent:\index{function extensionality}
\begin{enumerate}
\item The axiom of function extensionality.
\item For any type family $B$ over $A$ and any dependent function $f:\prd{x:A}B(x)$, the total space
  \begin{equation*}
    \sm{g:\prd{x:A}B(x)}f\htpy g
  \end{equation*}
  is contractible.
\item
  The principle of \define{homotopy induction}\index{homotopy induction}\index{induction principle!for homotopies}:
  for any type family $B$ over $A$, any dependent function $f:\prd{x:A}B(x)$, and any family of types $P(g,H)$ indexed by $g:\prd{x:A}B(x)$ and $H:f\htpy g$, the evaluation function
\begin{equation*}
\Big(\prd{g:\prd{x:A}B(x)}{H:f\htpy g}P(g,H)\Big)\to P(f,\reflhtpy_f)
\end{equation*}
given by $s\mapsto s(f,\reflhtpy_f)$ has a section.
\item The \define{weak function extensionality principle}\index{weak function extensionality} holds: For every type family $B$ over $A$ one has\index{contractible!weak function extensionality}
\begin{equation*}
\Big(\prd{x:A}\iscontr(B(x))\Big)\to\iscontr\Big(\prd{x:A}B(x)\Big).
\end{equation*}
\end{enumerate}
\end{thm}

\begin{proof}
The fact that function extensionality is equivalent to (ii) and (iii) follows directly from \cref{thm:id_fundamental}.

To show that function extensionality implies weak function extensionality, suppose that each $B(a)$ is contractible with center of contraction $c(a)$ and contraction $C_a:\prd{y:B(a)}c(a)=y$. Then we take $c\defeq \lam{a}c(a)$ to be the center of contraction of $\prd{x:A}B(x)$. To construct the contraction we have to define a term of type
\begin{equation*}
\prd{f:\prd{x:A}B(x)}c=f.
\end{equation*}
Let $f:\prd{x:A}B(x)$. By function extensionality we have a map $(c\htpy f)\to (c=f)$, so it suffices to construct a term of type $c\htpy f$. Here we take $\lam{a}C_a(f(a))$. This completes the proof that function extensionality implies weak function extensionality.

In the remaining part of the proof, we will show that weak function extensionality implies that the type
\begin{equation*}
\sm{g:\prd{x:A}B(x)}f\htpy g
\end{equation*}
is contractible for any $f:\prd{x:A}B(x)$. In order to do this, we first note that we have a section-retraction pair
\begin{equation*}
  \begin{tikzcd}[column sep=small]
    \Big(\sm{g:\prd{x:A}B(x)}f\htpy g\Big) \arrow[r,"i"] & \Big(\prd{x:A}\sm{b:B(x)}f(x)=b\Big) \arrow[r,"r"] & \Big(\sm{g:\prd{x:A}B(x)}f\htpy g\Big).
  \end{tikzcd}
\end{equation*}
Here we have the functions
\begin{align*}
i & \defeq \lam{(g,H)}{x}(g(x),H(x)) \\
r & \defeq \lam{p}\pairr{\lam{x}\proj 1(p(x)),\lam{x}\proj 2(p(x))}.
\end{align*}
Their composite is homotopic to the identity function by the computation rule for $\Sigma$-types and the $\eta$-rule for $\Pi$-types:
\begin{align*}
r(i(g,H)) & \jdeq r(\lam{x}\pairr{g(x),H(x)}) \\
& \jdeq \pairr{\lam{x}g(x),\lam{x}H(x)} \\
& \jdeq \pairr{g,H}.
\end{align*}
Now we observe that the type $\prd{x:A}\sm{b:B(x)}f(x)=b$ is a product of contractible types, so it is contractible by our assumption of the weak function extensionality principle. The claim therefore follows, since retracts of contractible types are contractible by \cref{ex:contr_retr}\index{contractible!retract of}.
\end{proof}

For the remainder of this chapter we will assume that the function extensionality axiom holds\index{axiom!function extensionality}. In \cref{thm:funext-univalence} we will derive function extensionality from the univalence axiom.

As a first application of the function extensionality axiom we generalize the weak function extensionality axiom to $k$-types.

\begin{thm}\label{thm:trunc_pi}\index{k-type@{$k$-type}}
For any type family $B$ over $A$ one has\index{truncated type!closed under Pi@{closed under $\Pi$}}
\begin{equation*}
\Big(\prd{x:A}\istrunc{k}(B(x))\Big)\to \istrunc{k}\Big(\prd{x:A}B(x)\Big).
\end{equation*}
\end{thm}

\begin{proof}
The theorem is proven by induction on $k\geq -2$. The base case is just the weak function extensionality principle\index{weak function extensionality}, which was shown to follow from function extensionality in \cref{thm:funext_wkfunext}.

For the inductive hypothesis, assume that the $k$-types are closed under dependent function types. Assume that $B$ is a family of $(k+1)$-types. By function extensionality, the type $f=g$ is equivalent to $f\htpy g$ for any two dependent functions $f,g:\prd{x:A}B(x)$. Now observe that $f\htpy g$ is a dependent product of $k$-types, and therefore it is an $k$-type by our inductive hypotheses. Therefore, it follows by \cref{thm:ktype_eqv} that $f=g$ is an $k$-type, and hence that $\prd{x:A}B(x)$ is an $(k+1)$-type.
\end{proof}

\begin{cor}\label{cor:funtype_trunc}\index{truncated type!closed under exponentials}
Suppose $B$ is a $k$-type. Then $A\to B$ is also a $k$-type, for any type $A$.
\end{cor}

\subsection{The type theoretic principle of choice}

The type theoretic principle of choice\index{type theoretic choice} asserts that $\Pi$ distributes over $\Sigma$\index{distributivity!of Pi over Sigma@{of $\Pi$ over $\Sigma$}}. More precisely, it asserts that the canonical map\index{choice@{$\choice$}}
\begin{equation*}
  \choice:\Big(\prd{x:A}\sm{y:B(x)}C(x,y)\Big)\to \Big(\sm{f:\prd{x:A}B(x)}\prd{x:A}C(x,f(x))\Big)
\end{equation*}
given by $\lam{h}(\proj 1(h(x)),\proj 2(h(x)))$, is an equivalence. In order to see this as a principle of choice, one can view the left hand side as the type of functions $h$ that pick for every $x:A$ a term $y:B(x)$ equipped with a term of type $C(x,y)$. The function $\choice$ then constructs a dependent function $f:\prd{x:A}B(x)$ equipped with a term of type $\prd{x:A}C(x,f(x))$. In this section we show that the map $\choice$ is an equivalence, and we use this to characterize the identity of any dependent function type $\prd{x:A}B(x)$ in terms of any characterization of the identity types of the individual types $B(x)$.

\begin{thm}\label{thm:choice}
Consider a family of types $C(x,y)$ indexed by $x:A$ and $y:B(x)$. Then the map
\begin{equation*}
  \choice:\Big(\prd{x:A}\sm{y:B(x)}C(x,y)\Big)\to \Big(\sm{f:\prd{x:A}B(x)}\prd{x:A}C(x,f(x))\Big)
\end{equation*}
given by $\lam{h}(\proj 1(h(x)),\proj 2(h(x)))$ is an equivalence.
\end{thm}

\begin{proof}
  We define the map\index{choice-1@{$\choice^{-1}$}}
  \begin{equation*}
    \choice^{-1}:\Big(\sm{f:\prd{x:A}B(x)}\prd{x:A}C(x,f(x))\Big)\to\Big(\prd{x:A}\sm{y:B(x)}C(x,y)\Big)
  \end{equation*}
  by $\lam{(f,g)}{x}(f(x),g(x))$. Then we have to construct homotopies
  \begin{equation*}
    \choice\circ\choice^{-1}\htpy\idfunc,\qquad\text{and}\qquad
    \choice^{-1}\circ\choice\htpy\idfunc.
  \end{equation*}
  For the first homotopy it suffices to construct an identification
  \begin{equation*}
    \choice(\choice^{-1}(f,g))=(f,g)
  \end{equation*}
  for any $f:\prd{x:A}B(x)$ and any $g:\prd{x:A}C(x,f(x))$. We compute the left-hand side as follows:
  \begin{align*}
    \choice(\choice^{-1}(f,g))
    & \jdeq \choice(\lam{x}(f(x),g(x))) \\
    & \jdeq (\lam{x}f(x),\lam{x}g(x)).
  \end{align*}
  By the $\eta$-rule it follows that $f\jdeq \lam{x}f(x)$ and $g\jdeq\lam{x}g(x)$. Therefore we have the identification
  \begin{equation*}
    \refl{(f,g)}:\choice(\choice^{-1}(f,g))=(f,g).
  \end{equation*}
  This completes the construction of the first homotopy.

  For the second homotopy we have to construct an identification
  \begin{equation*}
    \choice^{-1}(\choice(h))=h
  \end{equation*}
  for any $h:\prd{x:A}\sm{y:B(x)}C(x,y)$. We compute the left-hand side as follows:
  \begin{align*}
    \choice^{-1}(\choice(h))
    & \jdeq \choice^{-1}(\lam{x}\proj 1(h(x)),(\lam{x}\proj 2(h(x)))) \\
    & \jdeq \lam{x}(\proj 1(h(x)),\proj 2(h(x)))
  \end{align*}
  However, it is \emph{not} the case that $(\proj 1(h(x)),\proj 2(h(x)))\jdeq h(x)$ for any $h:\prd{x:A}\sm{y:B(x)}C(x,y)$. Nevertheless, we have the identification
  \begin{equation*}
    \eqpair(\refl{},\refl{}):(\proj 1(h(x)),\proj 2(h(x)))= h(x).
  \end{equation*}
  Therefore we obtain the required homotopy by function extensionality:
  \begin{equation*}
    \lam{h}\eqhtpy(\lam{x}\eqpair(\refl{\proj 1(h(x))},\refl{\proj 2(h(x))})):\choice^{-1}\circ\choice\htpy\idfunc.\qedhere
  \end{equation*}
\end{proof}

\begin{cor}
For type $A$ and any type family $C$ over $B$, the map
\begin{equation*}
\Big(\sm{f:A\to B} \prd{x:A}C(f(x))\Big)\to\Big(A\to\sm{y:B}C(x)\Big)
\end{equation*}
given by $\lam{(f,g)}{x}(f(x),g(x))$ is an equivalence.
\end{cor}

\begin{rmk}
  The type theoretic choice principle can be used to derive the binomial theorem\index{binomial theorem}. We give an informal argument of how this goes. Recall that the binomial theorem asserts that
  \begin{equation*}
    (n+m)^k=\sum_{l=0}^k\binom{k}{l}n^l m^{k-l}
  \end{equation*}
  for any three natural numbers $k,m,n$.

  Consider the types $A\defeq\Fin(k)$\index{Fin@{$\Fin$}}, $B\defeq\Fin(n)$ and $C\defeq\Fin(m)$. Then we can define the type family $P:\bool\to\UU$ given by
  \begin{align*}
    P(\btrue) & \defeq B \\
    P(\bfalse) & \defeq C.
  \end{align*}
  Now, the type theoretic principle of choice gives us an equivalence
  \begin{equation*}
    \Big(\prd{x:A}\sm{t:\bool}P(t)\Big)\simeq \Big(\sm{f:A\to\bool}\prd{x:A}P(f(x))\Big).
  \end{equation*}
  Now we note that the type $(f(x)=1)+(f(x)=0)$ is contractible for any $f:A\to\bool$ and $x:A$. Therefore we have equivalences
  \begin{align*}
    \sm{f:A\to\bool}\prd{x:A}P(f(x)) & \simeq
    \sm{f:A\to\bool}\prd{x:A}{t:(f(x)=1)+(f(x)=0)}P(f(x)) \\
    & \simeq \sm{f:A\to\bool}(\fib{f}{1}\to B)\times (\fib{f}{0}\to C)
  \end{align*}
  Now we note that, because there are $\binom{k}{l}$\index{binomial coefficient} ways to choose a subset of $l$ elements of $A$, there are
  \begin{equation*}
    \sum_{l=0}^k\binom{k}{l}n^l m^{k-l}
  \end{equation*}
  elements in the above type.
\end{rmk}

\subsection{Universal properties}
The function extensionality principle allows us to prove \emph{universal properties}. Universal properties are characterizations of all maps out of or into a given type, so they are very important. Among other applications, universal properties characterize a type up to equivalence. In the following theorem we prove the universal property of dependent pair types.

\begin{thm}\index{universal property!Sigma-types@{$\Sigma$-types}}\index{Sigma-type@{$\Sigma$-type}!universal property}
Let $B$ be a type family over $A$, and let $X$ be a type. Then the map\index{ev-pair@{$\evpair$}}
\begin{equation*}
\evpair:\Big(\Big(\sm{x:A}B(x)\Big)\to X\Big)\to \Big(\prd{x:A}(B(x)\to X)\Big)
\end{equation*}
given by $f\mapsto\lam{a}{b}f(a,b)$ is an equivalence.
\end{thm}

\begin{proof}
The map in the converse direction is simply
\begin{equation*}
\ind{\Sigma} : \Big(\prd{x:A}(B(x)\to X)\Big)\to \Big(\Big(\sm{x:A}B(x)\Big)\to X\Big).
\end{equation*}
By the computation rules for $\Sigma$-types we have
\begin{equation*}
\lam{f}\refl{f}:\evpair\circ\ind{\Sigma}\htpy\idfunc
\end{equation*}

To show that $\ind{\Sigma}\circ\evpair\htpy\idfunc$ we will also apply function extensionality. Thus, it suffices to show that $\ind{\Sigma}(\lam{x}{y}f((x,y)))=f$. We apply function extensionality again, so it suffices to show that
\begin{equation*}
\prd{t:\sm{x:A}B(x)}\ind{\Sigma}\big(\lam{x}{y}f((x,y))\big)(t)=f(t).
\end{equation*}
We obtain this homotopy by another application of $\Sigma$-induction.
\end{proof}

\begin{cor}\label{cor:times_up_out}\index{universal property!cartesian product}\index{cartesian product!universal property}
Let $A$, $B$, and $X$ be types. Then the map\index{ev-pair@{$\evpair$}}
\begin{equation*}
\evpair: (A\times B \to X)\to (A\to (B\to X))
\end{equation*}
given by $f\mapsto\lam{a}{b}f((a,b))$ is an equivalence.
\end{cor}

The universal property of identity types is sometimes called the \emph{type theoretical Yoneda lemma}\index{Yoneda lemma (type theoretical)}: families of maps out of the identity type are uniquely determined by their action on the reflexivity identification.

\begin{thm}\label{thm:yoneda}\index{universal property!identity type}\index{identity type!universal property}
Let $B$ be a type family over $A$, and let $a:A$. Then the map\index{ev-refl@{$\evrefl$}}
\begin{equation*}
\evrefl:\Big(\prd{x:A} (a=x)\to B(x)\Big)\to B(a)
\end{equation*}
given by $\lam{f} f(a,\refl{a})$ is an equivalence.
\end{thm}

\begin{proof}
The inverse $\varphi$ is defined by path induction, taking $b:B(a)$ to the function $f$ satisfying $f(a,\refl{a})\jdeq b$. It is immediate that $\evrefl\circ\varphi\htpy \idfunc$.

To see that $\varphi\circ \evrefl\htpy\idfunc$, let $f:\prd{x:A}(a=x)\to B(x)$. To show that $\varphi(f(a,\refl{a}))=f$ we use function extensionality (twice), so it suffices to show that
\begin{equation*}
\prd{x:A}{p:a=x} \varphi(f(a,\refl{a}),x,p)=f(x,p).
\end{equation*}
This follows by path induction on $p$, since $\varphi(f(a,\refl{a}),a,\refl{a})\jdeq f(a,\refl{a})$.
\end{proof}

\subsection{Composing with equivalences}

We show in this section that a map $f:A\to B$ is an equivalence if and only if for any type $X$ the precomposition map\index{precomposition map}
\begin{equation*}
\blank\circ f: (B\to X)\to (A\to X)
\end{equation*}
is an equivalence. Moreover, we will show in \cref{ex:equiv_precomp} that the `dependent version' of this statement also holds: a map $f:A\to B$ is an equivalence if and only if for any type family $P$ over $B$, the precomposition map
\begin{equation*}
\blank\circ f: \Big(\prd{y:B}P(y)\Big)\to\Big(\prd{x:A}P(f(x))\Big)
\end{equation*}
is an equivalence.

\begin{thm}\label{ex:equiv_precomp}\index{equivalence!precomposition}
For any map $f:A\to B$, the following are equivalent:
\begin{enumerate}
\item $f$ is an equivalence.
\item For any type family $P$ over $B$ the map
\begin{equation*}
\Big(\prd{y:B}P(y)\Big)\to\Big(\prd{x:A}P(f(x))\Big)
\end{equation*}
given by $h\mapsto h\circ f$ is an equivalence.
\item For any type $X$ the map
\begin{equation*}
(B\to X)\to (A\to X)
\end{equation*}
given by $g\mapsto g\circ f$ is an equivalence.
\end{enumerate}
\end{thm}

\begin{proof}
To show that (i) implies (ii), we first recall from \cref{lem:coherently-invertible} that any equivalence is also coherently invertible\index{coherently invertible}. Therefore $f$ comes equipped with
\begin{align*}
g & : B \to A\\
G & : f\circ g \htpy \idfunc[B] \\
H & : g\circ f \htpy \idfunc[A] \\
K & : G\cdot f \htpy f\cdot H.
\end{align*}
Then we define the inverse of $\blank\circ f$ to be the map
\begin{equation*}
\varphi:\Big(\prd{x:A}P(f(x))\Big)\to\Big(\prd{y:B}P(y)\Big)
\end{equation*}
given by $h\mapsto \lam{y}\tr_P(G(y),h(g(y)))$.

To see that $\varphi$ is a section of $\blank\circ f$, let $h:\prd{x:A}P(f(x))$. By function extensionality it suffices to construct a homotopy $\varphi(h)\circ f\htpy h$. In other words, we have to show that
\begin{equation*}
\tr_P(G(f(x)),h(g(f(x)))=h(x)
\end{equation*}
for any $x:A$. Now we use the additional homotopy $K$ from our assumption that $f$ is coherently invertible. Since we have $K(x):G(f(x))=\ap{f}{H(x)}$ it suffices to show that
\begin{equation*}
\tr_P(\ap{f}{H(x)},hgf(x))=h(x).
\end{equation*}
A simple path-induction argument yields that
\begin{equation*}
\tr_P(\ap{f}{p})\htpy \tr_{P\circ f}(p)
\end{equation*}
for any path $p:x=y$ in $A$, so it suffices to construct an identification
\begin{equation*}
\tr_{P\circ f}(H(x),hgf(x))=h(x).
\end{equation*}
We have such an identification by $\apd{h}{H(x)}$.

To see that $\varphi$ is a retraction of $\blank\circ f$, let $h:\prd{y:B}P(y)$. By function extensionality it suffices to construct a homotopy $\varphi(h\circ f)\htpy h$. In other words, we have to show that
\begin{equation*}
\tr_P(G(y),hfg(y))=h(y)
\end{equation*}
for any $y:B$. We have such an identification by $\apd{h}{G(y)}$. This completes the proof that (i) implies (ii).

Note that (iii) is an immediate consequence of (ii), since we can just choose $P$ to be the constant family $X$.

It remains to show that (iii) implies (i). Suppose that
\begin{equation*}
\blank\circ f:(B\to X)\to (A\to X)
\end{equation*}
is an equivalence for every type $X$. Then its fibers are contractible by \cref{thm:contr_equiv}. In particular, choosing $X\jdeq A$ we see that the fiber
\begin{equation*}
\fib{\blank\circ f}{\idfunc[A]}\jdeq \sm{h:B\to A}h\circ f=\idfunc[A]
\end{equation*}
is contractible. Thus we obtain a function $h:B\to A$ and a homotopy $H:h\circ f\htpy\idfunc[A]$ showing that $h$ is a retraction of $f$. We will show that $h$ is also a section of $f$. To see this, we use that the fiber
\begin{equation*}
\fib{\blank\circ f}{f}\jdeq \sm{i:B\to B} i\circ f=f
\end{equation*}
is contractible (choosing $X\jdeq B$).
Of course we have $(\idfunc[B],\refl{f})$ in this fiber. However we claim that there also is an identification $p:(f\circ h)\circ f=f$, showing that $(f\circ h,p)$ is in this fiber, because
\begin{align*}
(f\circ h)\circ f & \jdeq f\circ (h\circ f) \\
& = f\circ \idfunc[A] \\
& \jdeq f
\end{align*}
Now we conclude by the contractibility of the fiber that $(\idfunc[B],\refl{f})=(f\circ h,p)$. In particular we obtain that $\idfunc[B]=f\circ h$, showing that $h$ is a section of $f$.
\end{proof}

\begin{exercises}
\exercise Show that the functions\index{inv-htpy@{$\invhtpy$}!is an equivalence}\index{concat-htpy@{$\concathtpy$}!is a family of equivalences}\index{concat-htpy'@{$\concathtpy'$}!is a family of equivalences}\index{is an equivalence!inv-htpy@{$\invhtpy$}}\index{is an equivalence!concat-htpy(H)@{$\concathtpy(H)$}}\index{is an equivalence!concat-htpy'(K)@{$\concathtpy'(K)$}}
\begin{align*}
\invhtpy & : (f \htpy g) \to (g \htpy f) \\
\concathtpy(H) & : (g \htpy h) \to (f \htpy h) \\
\concathtpy'(K) & : (f \htpy g) \to (f \htpy h)
\end{align*}
are equivalences for every $f,g,h : \prd{x:A}B(x)$. Here, $\concathtpy'(K)$ is the function defined by $H\mapsto \ct{H}{K}$.
\exercise \label{ex:isprop_istrunc}
\begin{subexenum}
\item Show that for any type $A$ the type $\iscontr(A)$ is a proposition\index{is-cont(A)r@{$\iscontr(A)$}!is a proposition}\index{is contractible!is a property}. %There's an easy proof using double singleton induction. This is a nice application of weak funext.
\item Show that for any type $A$ and any $k\geq-2$, the type $\istrunc{k}(A)$ is a proposition.\index{istrunc@{$\istrunc{k}$}!is a proposition}
\end{subexenum}
\exercise \label{lem:postcomp_equiv}
Let $f:X\to Y$ be a map. Show that the following are equivalent:
\begin{enumerate}
\item $f$ is an equivalence.\index{equivalence!postcomposition}
\item The map $f\circ\blank : X^A\to Y^A$ is an equivalence for every type $A$.
\end{enumerate}
\exercise \label{ex:isprop_isequiv}Let $f:A\to B$ be a function.
\begin{subexenum}
\item Show that if $f$ is an equivalence, then the type $\sm{g:B\to A}f\circ g\htpy \idfunc$ of sections of $f$ is contractible.
\item Show that if $f$ is an equivalence, then the type $\sm{h:B\to A}h\circ f\htpy \idfunc$ of retractions of $f$ is contractible.
\item Show that $\isequiv(f)$ is a proposition.\index{is-equiv(f)@{$\isequiv(f)$}!is a proposition}
\item Use \cref{ex:prop_equiv,ex:isprop_istrunc} to show that $\isequiv(f)\eqvsym \iscontr(f)$.\index{is-equiv(f)@{$\isequiv(f)$}!is-equiv(f) is-contr(f)@{$\isequiv(f)\eqvsym\iscontr(f)$}}
\end{subexenum}
Conclude that $\eqv{A}{B}$ is a subtype of $A\to B$, and in particular that the map $\proj 1 : (\eqv{A}{B})\to (A\to B)$ is an embedding.
\exercise \label{ex:prop_equiv}
\begin{subexenum}
\item \label{ex:equiv-bi-implication}Let $P$ and $Q$ be propositions. Show that\index{bi-implication}
\begin{equation*}
\eqv{(P\leftrightarrow Q)}{(\eqv{P}{Q})}.
\end{equation*}
\item Show that $P$ is a proposition if and only if $P\to P$ is contractible.
\end{subexenum}
\exercise Show that $\pathsplit(f)$\index{path-split!is a proposition} and $\iscohinvertible(f)$\index{coherently invertible!is a proposition} are propositions for any map $f:A\to B$. Conclude that we have equivalences\index{is-equiv(f)@{$\isequiv(f)$}!is-equiv(f) path-split(f)@{$\isequiv(f)\eqvsym\pathsplit(f)$}}\index{is-equiv(f)@{$\isequiv(f)$}!is-equiv(f) is-coh-invertible(f)@{$\isequiv(f)\eqvsym\iscohinvertible(f)$}}
  \begin{equation*}
    \isequiv(f) \eqvsym \pathsplit(f) \eqvsym \iscohinvertible(f).
  \end{equation*}
%\item Let $B$ and $C$ be type families over $A$, suppose that $p:\id{a}{a'}$ in $A$, and consider two functions $f:B(a)\to C(a)$ and $g:B(a')\to C(a')$.
%\begin{subexenum}
%\item Show that the square
%\begin{equation*}
%\begin{tikzcd}
%B(a) \arrow[r,"f"] \arrow[d,swap,"\tr_B(p)"] & C(a) \arrow[d,"\tr_C(p)"] \\
%B(a') \arrow[r,swap,"g"] & C(a')
%\end{tikzcd}
%\end{equation*}
%commutes for every homotopy $H:\tr_{B(x)\to C(x)}(p,f)\htpy g$. In other words, construct a function of type
%\begin{equation*}
%\Big(\tr_{B(x)\to C(x)}(p,f)\htpy g\Big)\to \Big(\tr_C(p)\circ f\htpy g\circ \tr_B(p)\Big)
%\end{equation*}
%\item Show that this map is an equivalence.
%\end{subexenum}
\exercise \label{ex:idfunc_autohtpy}Construct for any type $A$ an equivalence\index{has-inverse(f)@{$\hasinverse(f)$}!has-inverse(id) id htpy id@{$\hasinverse(\idfunc)\simeq (\idfunc\htpy\idfunc)$}}
\begin{equation*}
\eqv{\hasinverse(\idfunc[A])}{\Big(\idfunc[A]\htpy\idfunc[A]\Big)}.
\end{equation*}
Note: We will use this fact in \cref{ex:is_invertible_id_S1} to show that there
are types for which $\isinvertible(\idfunc[A])\not\eqvsym\isequiv(\idfunc[A])$.
\exercise
\begin{subexenum}
\item Show that the type\index{universal property!empty type}\index{empty type!universal property}
\begin{equation*}
\prd{t:\emptyt}P(t)
\end{equation*}
is contractible for any $P:\emptyt\to \UU$.
\item Show that for any type $X$ the following are equivalent:
  \begin{enumerate}
  \item the unique map $\emptyt \to X$ is an equivalence.
  \item The type $Y^X$ is contractible for any type $Y$.
  \end{enumerate}
\end{subexenum}
\exercise Consider two types $A$ and $B$.\index{universal property!coproduct}\index{coproduct!universal property}
\begin{subexenum}
\item Show that the map\index{ev-inl-inr@{$\evinlinr$}}
\begin{equation*}
  \evinlinr: \Big(\prd{t:A+B}P(t)\Big) \to
  \Big(\prd{x:A}P(\inl(x))\Big)\times\Big(\prd{y:B}P(\inr(y))\Big)
\end{equation*}
given by $f\mapsto (f\circ\inl,f\circ\inr)$ is an equivalence.
\item Show that the following are equivalent for any type $X$ equipped with maps $i:A\to X$ and $j:B\to X$:
  \begin{enumerate}
  \item The map $\indcoprod(i,j) :A+B\to X$ is an equivalence.
  \item For any type $Y$, the map
    \begin{equation*}
      \lam{f}(f\circ i,f\circ j):(X\to Y)\to (A\to Y)\times (B \to Y)
    \end{equation*}
    is an equivalence.
  \end{enumerate}
\end{subexenum}
\exercise
\begin{subexenum}
\item Show that the map\index{universal property!unit type}\index{unit type!universal property}\index{ev-pt@{$\evpt$}}
\begin{equation*}
\Big(\prd{t:\unit}P(t)\Big)\to P(\ttt)
\end{equation*}
given by $\lam{f}f(\ttt)$ is an equivalence.
\item Consider a type $X$ equipped with a point $x:X$. Show that the following are equivalent:
\begin{enumerate}
\item The map $\indunit(x):\unit\to X$ is an equivalence (i.e., $X$ is contractible).
\item For any type $Y$ the map
\begin{equation*}
\lam{f}f(x) : (X\to Y)\to Y
\end{equation*}
is an equivalence.
\end{enumerate}
\end{subexenum}
\exercise \label{ex:sec_retr}Consider a commuting triangle
\begin{equation*}
\begin{tikzcd}[column sep=tiny]
A \arrow[rr,"h"] \arrow[dr,swap,"f"] & & B \arrow[dl,"g"] \\
& X
\end{tikzcd}
\end{equation*}
with $H:f\htpy g\circ h$.
\begin{subexenum}
\item Show that if $h$ has a section, then $\sections(g)$ is a retract of $\sections(f)$.
\item Show that if $g$ has a retraction, then $\retractions(h)$ is a retract of $\sections(f)$.
\end{subexenum}
\exercise \label{ex:equiv_pi} Consider a family $f_i:A_i\to B_i$ of $k$-truncated maps, indexed by $i:I$. Show that the map
  \begin{equation*}
    \lam{h}{i}f_i(h(i)):\Big(\prd{i:I}A_i\Big)\to\Big(\prd{i:I}B_i\Big)
  \end{equation*}
  is again $k$-truncated. Conclude that if each $f_i$ is an equivalence, then so is the above map.
  \exercise Consider a map $f:X\to Y$. Show that the following are equivalent:
  \begin{enumerate}
  \item The map $f$ is $k$-truncated.
  \item For every type $A$, the postcomposition function
    \begin{equation*}
      f\circ\blank : (A\to X)\to (A\to Y)
    \end{equation*}
    is $k$-truncated.
  \end{enumerate}
  In particular it follows that $f$ is an embedding if and only if $f\circ\blank$ is an embedding.

  Hint: Show that the square
  \begin{equation*}
    \begin{tikzcd}[column sep=huge]
      (f=g) \arrow[r,"\apfunc{m}"] \arrow[d,swap,"\htpyeq"] & (m\circ f=m\circ g) \arrow[d,"\htpyeq"] \\
      (f\htpy g) \arrow[r,swap,"H\mapsto m\cdot H"] & (m\circ f\htpy m\circ g)
    \end{tikzcd}
  \end{equation*}
  commutes, and apply \cref{ex:equiv_pi}.
  \exercise \label{ex:pi-fib}Consider a function $f:A\to B$, and let $P$ be a family of types over $B$. Show that the map
  \begin{equation*}
    \Big(\prd{b:B}\fib{f}{b}\to P(b)\Big)\to\Big(\prd{a:A}P(f(a))\Big)
  \end{equation*}
  given by $h\mapsto h_{f(a)}(a,\refl{f(a)})$ is an equivalence.
  \exercise \label{ex:triangle_fib}Consider a diagram of the form
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[dr,swap,"f"] & & B \arrow[dl,"g"] \\
      & X
    \end{tikzcd}
  \end{equation*}
  \begin{subexenum}
  \item Show that the type $\sm{h:A\to B} f\htpy g\circ h$ is equivalent to the type of families of maps
    \begin{equation*}
      \prd{x:X}\fib{f}{x}\to\fib{g}{x}.
    \end{equation*}
  \item Show that the type $\sm{h:\eqv{A}{B}} f\htpy g\circ h$ is equivalent to the type of families of equivalences
    \begin{equation*}
      \prd{x:X}\fib{f}{x}\eqvsym\fib{g}{x}.
    \end{equation*}
  \end{subexenum}
  \exercise \label{ex:sq_fib}Consider a diagram of the form
  \begin{equation*}
    \begin{tikzcd}
      A \arrow[d,swap,"f"] & B \arrow[d,"g"] \\
      X \arrow[r,swap,"h"] & Y.
    \end{tikzcd}
  \end{equation*}
  Show that the type $\sm{i:A\to B}h\circ f\htpy g\circ i$ is equivalent to the type of families of maps
  \begin{equation*}
    \prd{x:X}\fib{f}{x}\to\fib{g}{h(x)}.
  \end{equation*}
  % Note: In \cref{thm:pb_fibequiv_complete} we will characterize the type of families of equivalences $\prd{x:X}\fib{f}{x}\simeq\fib{g}{x}$.
  % \item Show that the type $\sm{i:\eqv{A}{B}}h\circ f\htpy g\circ i$ is equivalent to the type of families of equivalences
  %   \begin{equation*}
  %     \prd{x:X}\fib{f}{x}\eqvsym\fib{g}{h(x)}.
  %   \end{equation*}
  \exercise \label{ex:iso_equiv}Let $A$ and $B$ be sets. Show that type type $\eqv{A}{B}$ of equivalences from $A$ to $B$ is equivalent to the type $A\cong B$ of \define{isomorphisms}\index{isomorphism}\index{set!isomorphism} from $A$ to $B$, i.e., the type of quadruples $(f,g,H,K)$ consisting of
  \begin{align*}
    f & : A\to B \\
    g & : B\to A \\
    H & : f\circ g = \idfunc[B] \\
    K & : g\circ f = \idfunc[A].
  \end{align*}
  \exercise \label{ex:pi_sec}Let $B$ be a type family over $A$, and consider the postcomposition function
  \begin{equation*}
    \proj1 \circ \blank : \Big(\sm{x:A} B(x)\Big)^A \to A^A.
  \end{equation*}
  Construct equivalences
  \begin{equation*}
    \Big(\prd{x:A}B(x)\Big) \eqvsym \sections(\proj 1) \eqvsym \fib{\proj 1 \circ\blank}{\idfunc[A]}.
  \end{equation*}
  \exercise Construct equivalences
  \begin{align*}
    \Fin(n^m) & \simeq (\Fin(m)\to\Fin(n)) \\
    \Fin(n!) & \simeq (\Fin(n)\simeq\Fin(n)).
  \end{align*}
\end{exercises}


\section*{Exercises}

\begin{ex} Find the determinants of the following matrices.
  \begin{equation*}
    (a)~\begin{mymatrix}{rr}
      1 & 3 \\
      0 & 2
    \end{mymatrix}
    \quad
    (b)~\begin{mymatrix}{rr}
      0 & 3 \\
      0 & 2
    \end{mymatrix}
    \quad
    (c)~\begin{mymatrix}{rr}
      4 & 3 \\
      6 & 2
    \end{mymatrix}
    \quad
    (d)~\begin{mymatrix}{rr}
      -3 & 4 \\
      -1 & 2
    \end{mymatrix}
  \end{equation*}
\end{ex}

\begin{ex}
  Find the following determinants.
  \begin{equation*}
    (a)~\begin{absmatrix}{rrr}
      1 & 2 & 3 \\
      4 & 5 & 6 \\
      7 & 8 & 9 \\
    \end{absmatrix}
    \quad
    (b)~\begin{absmatrix}{rrr}
      1  & 0 & 2 \\
      2  & 5 & 3 \\
      -1 & 0 & 0 \\
    \end{absmatrix}
    \quad
    (c)~\begin{absmatrix}{rrr}
      3 &  4 & 1 \\
      0 & -1 & 1 \\
      1 &  2 & 1 \\
    \end{absmatrix}
    \quad
    (d)~\begin{absmatrix}{rrr}
      0  & -2 &  1 \\
      4  & 1  & -3 \\
      -1 & 3  &  1 \\
    \end{absmatrix}
  \end{equation*}
\end{ex}


%% Magic command to compile root document
% !TEX root = ../../thesis.tex

%% Reset glossary to show long gls names
\glsresetall

%% Set path to look for the images
\graphicspath{{./Sections/Basics/Resources/}}

So far we have explained how \glspl{ann} works assuming that we feed them with vectors of fixed length. Even though we could take a multichannel image and transform it into a vector, in practice this would be computationally very expensive. For instance, assuming that we have a 3 channel image of size 224 by 224, this would result into an input vector of length $3 \cdot 224 \cdot 224=150'528$. Then, if the first layer of our network has 100 units, this would mean more than 15 millions of parameters only for the first layer. Furthermore, the transformation of our image into a vector would mean a loss of spatial information. This means that the \gls{ann} would not be able to capture or use the spatial relationship between pixels and shapes within the image.

A \gls{cnn} is a type of \gls{ann} widely used to analyze data in the form of images. The intuition behind a \gls{cnn} is that instead of just looking at an image and trying to predict the target value directly, first learn some \hl{features} within the image, and then make the predict base on this features.
To achieve this, \glspl{cnn} mainly use \hl{convolution} and \hl{pooling} layers.

\subsubsection{Convolution layer}

The only difference a

A convolution layer is very similar to a regular layer described in section \ref{sec:basics:ANN}. Basically, they only differ in the way the layer input is multiplied by the the layer weights.
Recall that in a regular layer, the input of a unit is the dot product between the layer input and its corresponding weight vector (i.e., $z=\bs{w}^T\bs{x}$).
This means that for each element in the input vector $\bs{x}$, there is a corresponding element in the weight vector $\bs{w}$. However, for a convolution layer this is not the case.
Convolution layers are based on the shared-weight architecture of the convolution \hl{kernels} or \hl{filters} that slide along the input and returns a translation known as \hl{feature maps} \cite{zhang1988shift}. This means that the \hl{kernels} weights will be used for multiple elements of the layer input. Figure \ref{fig:basics:conv_layer} shows the convolution process with a 2 by 2 kernel over a RGB image (3 channels) of size 4 by 4. Each entrance of the returned feature map $z_i$ is the dot product between the kernel weights $\bs{w}$ and the $\bs{x}_i-th$ chunk of the image.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=\linewidth]{Diagrams/Conv_layer.png}
  \caption{Convolution process steps. In red, green and blue the input image, in orange the convolution kernel (size 2 by 2 and stride of 1) and in gray the convolution output (feature map).}
  \label{fig:basics:conv_layer}
\end{figure}

Mathematically this looks as follow

\begin{equation}
  z_i = \bs{w}^T \bs{x}_i + b
\end{equation}

where $\bs{w}\in\mathbb{R}^{2 \times 2 \times 3}$, $\bs{x_i}\in\mathbb{R}^{2 \times 2 \times 3}$ and $b\in\mathbb{R}$ is the bias (not shown in the images).

Like the kernel size, the number of pixels we shift the kernel each time along side the input (\hl{Stride}) is also a hyperparameter of convolution layers. IN figure \ref{fig:basics:conv_layer}, the stride size is 1.

Figure \ref{fig:basics:conv_layer} also shows that size (width and height) of the returned feature map is smaller than the input image. If we want to keep the input and output size the same (\hl{Same convolution}), then we must add zeros at the edges of the input features (zero-padding). This is shown on figure \ref{fig:basics:conv_layer_pad}.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=0.7\linewidth]{Diagrams/Conv_layer_pad.png}
  \caption{Convolution with padding. In blue a single-channel input features, in orange the convolution kernel (size 3 by 3 and stride of 1) and in gray the convolution output (feature map).}
  \label{fig:basics:conv_layer_pad}
\end{figure}

So far we have seen that a convolution projects a multi-channel input feature (image) into a single-channel feature map. Therefore, if we want our output feature map to have $n$ channels, then our convolution must have $n$ different kernels.

Normally, a non-linear activation function is applied to the output of convolution layers (and normally also after batch normalization) to enable the \gls{cnn} to learn non-linear relations.

\subsubsection{Pooling layer}

Unlike convolution layers, the goal of Pooling layers is to reduce the feature image (height and width, but not depth) rather than learn features.
However, Pooling layers work in a similar way to convolution layers in the way that they also slide a kernel along the input. However, in this case the kernel works independently on each feature map (that is, each channel) and has no weights to learn.
This means that the pooling layers maintain the same number of input and output channels.
There are several ways to do this downsampling, but the most common are Max Polling and Average Polling. As the name suggests, Average pooling shrinks the feature image by averaging sections of it, while Max pooling takes the maximum value. Figure \ref{fig:basics:pooling} shows an example of a max and average pooling layer on a single-channel feature image using a 2 by 2 kernel and a stride of 2.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=0.7\linewidth]{Diagrams/Pooling.png}
  \caption{Max and average pooling with a 2 by 2 kernel and stride 2. The color denotes the kernel position.}
  \label{fig:basics:pooling}
\end{figure}

Normally, Pooling layers are applied over the output of the activation functions.

\subsubsection{Global Average Pooling layer}

As we mentioned at the beginning of this section, the idea of a \gls{cnn} is to first learn the features within the input images and then make a prediction based on these features. To do this, the \hl{Global Average Pooling layer} transforms the channels of the last feature map into a vector (by averaging each of its channels), so that this can be used as input in a regular \gls{ann} to make the final prediction. Figure \ref{fig:basics:global_avg_pool} shows an example of this, when it is applied into a feature map with 7 channels.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=0.7\linewidth]{Diagrams/Global_avg_pooling.png}
  \caption{Global Average Pooling layer.}
  \label{fig:basics:global_avg_pool}
\end{figure}

\subsubsection{Inception module}

Recall that a convolution layer is meant to learn features from a 3D object with 2 spatial dimensions (width and height) and a channel dimension. This means that each kernel in the convolution needs to learn simultaneously cross-channel and spatial correlations.
The intuition behind the \hl{Inception module} is to improve this process by separating this two tasks, so that the cross-channel correlations and the spatial correlations can be learned separately and independently \cite{chollet2017xception}.

A normal inception model looks at the cross-channel correlations first through a set of 3 or 4 \hl{pointwise convolutions}\footnote{A \hl{pointwise convolution} is a convolution with 1 by 1 kernels and stride 1.}, and then learns the spacial information in the downsampled feature image (in depth, not height and width), by means of regular convolution (usually with 3 by 3 or 5 by 5 kernels). Figure \ref{fig:basics:inception_module} shows a diagram of an Inveption V3 module.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=0.6\linewidth]{Inception_module.png}
  \caption{A regular Inception module (Inveption V3). Image source \cite{chollet2017xception}.}
  \label{fig:basics:inception_module}
\end{figure}

François Chollet \cite{chollet2017xception}, used the inception module as reference to propose the \hl{depthwise separable convolution}, which is something between a normal convolution and a normal convolution combined/followed by a pointwise convolution.
Figure \ref{fig:basics:extreme_inception_module} shows an \hl{extreme} version of the inception module shown in figure \ref{fig:basics:inception_module}. The \hl{depthwise separable convolution} is very similar to the one shown in figure \ref{fig:basics:extreme_inception_module}, the only difference is that the pointwise convolution is applied before the 3 by 3 convolutions instead of after.

\begin{figure}[!ht]
  \centering
  \includegraphics[width=0.6\linewidth]{Extreme_Inception_module.png}
  \caption{An extreme version of our Inception module. Image source \cite{chollet2017xception}.}
  \label{fig:basics:extreme_inception_module}
\end{figure}

Even though the \hl{depthwise separable convolution} is a simplified version of the inception module, the idea and motivation behind it is the same. The \hl{depthwise separable convolution}, and the residual block, are the main components of the \hl{Xception} architecture \cite{chollet2017xception}.


\section{Robust PCA}
\begin{compactitem}
	\item Idea: Approximate $\mathbf{X}$ with $\mathbf{L} + \mathbf{S}$, $\mathbf{L}$ is low-rank, $\mathbf{S}$ is sparse.
	\item $\min_{\mathbf{L},\mathbf{S}}\mathsf{rank}(\mathbf{L}) + \mu \lVert \mathbf{S}\rVert_0$, s. t. $\mathbf{L} + \mathbf{S} = \mathbf{X}$. As non-convex, change to $\min_{\mathbf{L},\mathbf{S}} \|\mathbf{L}\|_\star + \lambda \lVert\mathbf{S}\rVert_1$ (\emph{not} the same in general) Deal w/ missing values: subj. to $L_{ij} + S_{ij} = X_{ij}, \forall(i,j) \in \Omega_{observed}$
	\item Perfect reconstruction is \emph{not} possible if $\mathbf{S}$ is low-rank, $\mathbf{L}$ is sparse, or $\mathbf{X}$ is low-rank \textit{and} sparse. Formally coherence: $\|\mathbf{U}^\top \mathbf{e}_i\|^2 \leq \frac{\nu r}{n}$, $\|\mathbf{V}^\top \mathbf{e}_i\|^2 \leq \frac{\nu r}{n}$, $\|\mathbf{UV}^\top\|^2_{ij} \leq \frac{\nu r}{n^2}$ : $\mathbf{L}=\mathbf{U}\mathbf{D}\mathbf{V}^\top$
\end{compactitem}

\subsection*{Dual Ascent (Gradient Method for Dual Problem)}
$\boldsymbol{\lambda}^{t+1} = \boldsymbol{\lambda}^{t} + \eta \nabla D(\boldsymbol{\lambda}^t)$,
$ \nabla D (\boldsymbol{\lambda}) = \mathbf{A}\mathbf{x}^*-\mathbf{b}$ for $\mathbf{x}^* \in \arg\min_\mathbf{x} \mathcal{L}(\mathbf{x},\boldsymbol{\lambda})$
\textbf{Dual Decomposition for Dual Ascent}:
\\$\mathbf{x}_i^{t+1} := \arg\min_{\mathbf{x}_i} \mathcal{L}_i(\mathbf{x}_i,\lambda^t)$;
$\boldsymbol{\lambda}^{t+1} := \boldsymbol{\lambda}^t + \eta^t \left( \sum_{i=1}^{N} \mathbf{A}_i \mathbf{x}_i^{t+1} -\mathbf{b} \right) $

\subsection*{Alternating Direction Method of Multipliers (ADMM)}
$\min_{\mathbf{x}_1, \mathbf{x}_2} f_1(\mathbf{x}_1) + f_2(\mathbf{x}_2)$ s. t. $\mathbf{A}_1 \mathbf{x}_1 + \mathbf{A}_2 \mathbf{x}_2 = \mathbf{b}$, $f_1, f_2$ convex
\begin{inparaitem}[\color{red}\textbullet]
	\item Augmented Lagrangian: $L_p(\mathbf{x}_1, \mathbf{x}_2, \boldsymbol{\nu}) = f_1(\mathbf{x}_1) + f_2(\mathbf{x}_2) + \boldsymbol{\nu}^\top (\mathbf{A}_1 \mathbf{x}_1 + \mathbf{A}_2 \mathbf{x}_2 - \mathbf{b}) + \frac{p}{2}\| \mathbf{A}_1 \mathbf{x}_1 + \mathbf{A}_2 \mathbf{x}_2 - \mathbf{b} \|_2^2$
	\item ADMM: $\mathbf{x}_1^{(t+1)} := \argmin_{\mathbf{x}_1} L_p(\mathbf{x}_1, \mathbf{x}_2^{(t)}, \boldsymbol{\nu}^{(t)})$, $\mathbf{x}_2^{(t+1)} := \argmin_{\mathbf{x}_2} L_p(\mathbf{x}_1^{(t+1)}, \mathbf{x}_2, \boldsymbol{\nu}^{(t)})$, $\boldsymbol{\nu}^{(t+1)} := \boldsymbol{\nu}^{(t)} + p(\mathbf{A}_1 \mathbf{x}_1^{(t+1)} + \mathbf{A}_2 \mathbf{x}_2^{(t+1)} - \mathbf{b})$
  \item ADMM for RPCA: $f_1(\mathbf{L}) = \|\mathbf{L}\|_\star$, $f_2(\mathbf{S}) = \lambda \| \mathbf{S} \|_1$, $\mathbf{A}_1 \mathbf{x}_1 + \mathbf{A}_2 \mathbf{x}_2 = \mathbf{b} \text{ becomes } \mathbf{L} + \mathbf{S} = \mathbf{X}$, therefore $L_p(\mathbf{L}, \mathbf{S}, \boldsymbol{\nu}) = \|\mathbf{L}\|_* + \nu \|\mathbf{S}\|_1 + \left< \nu, \mathrm{vec}(\mathbf{L}+\mathbf{S}-\mathbf{X}) \right> + \frac{P}{2} \| \mathbf{L}+ \mathbf{S} - \mathbf{X} \|_F^2$
  %updates: $\mathbf{L}^{t+1} = \mathcal{D}_{\rho^{-1}}(X-S-\rho^{-1}\text{mat}(\lambda))$
\end{inparaitem}


\section{MAP Operator}
The operator MAP applies a uniform evaluation pattern
to all members of a composite structure: a matrix, a list or the arguments
of an operator expression. The evaluation pattern can be a
unary procedure, an operator, or an algebraic expression with
one free variable. MAP is used with the syntax:
\begin{verbatim}
  MAP(EXPRN1:algebraic,EXPRN2:expression)
\end{verbatim}
{\tt EXPRN2} is a list, a matrix or an operator expression.

{\tt EXPRN1} is
\begin{itemize}
\item the name of an operator for a single argument: the operator
 is evaluated once with each element of {\tt EXPRN1} as its single argument,
\item an algebraic expression with exactly one free variable, that is
a variable preceded by the tilde symbol: the expression
 is evaluated for each element of {\tt EXPRN1} where the element is
 substituted for the free variable,
\item a replacement rule of the form
\begin{verbatim}
  VAR => EXPRN3
\end{verbatim}
  where {\tt VAR} is a variable and {\tt EXPRN3} is an expression
  which contains {\tt VAR}.
  Here {\tt EXPRN3} is evaluated for each element of {\tt EXPRN1} where
  the element is substituted for  {\tt VAR}. {\tt VAR} may be
  optionally preceded by a tilde.
\end{itemize}
The rule form for {\tt EXPRN2} is needed when more than
one free variable occurs.

{\it Examples:}
\begin{verbatim}
     % collect absolute values
   map(abs,{1,-2,a,-a});

                      ->

   {1,2,abs(a),abs(a)}

     % integrate a matrix
   map(int(~w,x), mat((x^2,x^5),(x^4,x^5)));

                      ->

        [  3     6 ]
        [ x     x  ]
        [----  ----]
        [ 3     6  ]
        [          ]
        [  5     6 ]
        [ x     x  ]
        [----  ----]
        [ 5     6  ]

     % multiply an equation
   map(~w*6 , x^2/3 = y^3/2 -1);

                      ->

         2     3
      2*x =3*(y  - 2)

\end{verbatim}
{\tt MAP} can be applied in nested mode:
\begin{verbatim}
   map(sub(x=y,~q)-sub(x=0,~q),map(mat(int(~w,x),(x^2-y,x^3))));
                      ->

       [  2             4 ]
       [ y *(y - 3)    y  ]
       [------------  ----]
       [     3         4  ]

    % The following example needs the rule form because there are two
    % free variables in the rightmost expression.

   map(~w=>map(int(~r,x),w),{mat((x^2,x^4)),mat((x^3,x^5))});

                      ->
       {

        [  3     5 ]
        [ x     x  ]
        [----  ----]
        [ 3     5  ]

        ,

        [  4     6 ]
        [ x     x  ]
        [----  ----]
        [ 4     6  ]

        }
\end{verbatim}

\section{SELECT Operator}
The operator SELECT extracts from a list
or from the arguments of an n--ary operator elements corresponding
to a boolean predicate. The predicate pattern can be a
unary procedure, an operator or an algebraic expression with
one free variable. SELECT is used with the syntax:
\begin{verbatim}
  SELECT(EXPRN1:expression,EXPRN2:list)
\end{verbatim}

{\tt EXPRN1} is
\begin{itemize}
\item the name of an operator for a single argument: the operator
 is evaluated once with each element of {\tt EXPRN2} as its single argument,
\item an algebraic expression with exactly one free variable, that is
a variable preceded by the tilde symbol at least once: the expression
 is evaluated for each element of {\tt EXPRN2} where the element is
 substituted for the free variable,
\item a replacement rule of the form
\begin{verbatim}
  VAR => EXPRN3
\end{verbatim}
  where {\tt VAR} is a variable and {\tt EXPRN3} is an expression
  which contains {\tt VAR}.
  Here {\tt EXPRN3} is evaluated for each element of {\tt EXPRN2} where
  the element is substituted for  {\tt VAR}. {\tt VAR} may be
  optionally preceded by a tilde.
\end{itemize}
The rule form for {\tt EXPRN1} is needed when more than
one free variable occurs. The result of the operation is built
from those elements of {\tt EXPRN2} which let {\tt EXPRN1} evaluate
to a value different from {\tt 0} or {\tt nil}, using the
leading operator of  {\tt EXPRN2}.

Examples:

\begin{verbatim}
  select( ~w>0, {1,-1,2,-3,3})
                      ->
                    {1,2,3}

   % select the tersm with even powers of y
  select(evenp deg(~w,y), part((x+y)^5,0):=list);

                      ->
          5     3  2      4
        {x ,10*x *y ,5*x*y }

   % select elements of a sum directly
  select(evenp deg(~w,x), 2x^2+3x^3+4x^4);

                      ->
          2    4
        2x  +4x
\end{verbatim}