\section{Results}
All the implemented transforms where tested for correctness by comparing the output
of my implementation to other reference-implementations, like matlab's or scipy's.
The implementation was also tested for speed depending on the length of the input.
For a given input length the FFT algorithm was performed multiple times with random numbers
and the
runtime was measured and averaged. The number of times the algorithm was evaluated
was chosen, such that the uncertainty of the average was below \SI{10}{\nano\second}.
While the actual numbers will be more or less uninteresting, it is possible to see
the $\symcal{O}(N\log N)$ scaling for powers of 2, as shown in \autoref{fig:timesp2}.
Especially for higher-length input the scaling is fulfilled, but for short length
input there is too much overhead, and thus the numbers are not following $N\log N$ perfectly there.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{build/plots/times_p2.pdf}
    \caption{Runtime of my FFT implementation in double-logarithmic-scaling depending on the length of the input sequence.
        Each random-number input is transformed multiple time and the results were averaged.
        The function $t(N)=c\times N\log N$ is fitted to the measurements, as we expect this scaling for
        input lengths in the form of $2^p$. }
    \label{fig:timesp2}
\end{figure}

For arbitrary sized input, the actual runtime of this algorithm depends heavily on
the number of two's in the prime-factor decomposition of the input-length.
In easier words: the FFT of 48 numbers should be roughly twice as fast as 56, even though
56 is the higher number.
The reason is 48 is divisible by 2 4 times ($48=2^4\cdot 3$), where 56 is only divisble by
2 3 times ($56=2^3\cdot 7$).
This is also clearly seen in the data, as \autoref{fig:times} shows.
\begin{figure}[t]
    \centering
    \includegraphics[width=.5\textwidth]{build/plots/times_lin_c.pdf}
    \caption{Averaged runtime of the implemented FFT algorithm on random numbers depending on the input length.
        A clear separation of the runtime scaling is seen. The data-points are colored
        depending on the number of times the input length is divisible by 2.}
    \label{fig:times}
\end{figure}

\subsection{Depicting the Fourier Space}
A digital image is nothing more than $N=h\cdot w\cdot C$ numbers $x_{k,l,c}$
\footnote{$k=0,\dots, h-1\quad l=0,\dots,w-1\quad c=0,\dots,C-1$}
between $0$ and $1$.
Where $w$ is the width, $h$ is the height and $C$ is the number of channels of the image.
There are a lot of different types of images out there, with different channel numbers and interpretation
of them (different color encodings, transparent images, etc.), but we will focus on the
case of grayscale $C=1$ or colored RGB $C=3$ pictures.
The interpretation of the number in grayscale images is the illumination of the pixel ranging from $0=$~off~$=$~black
to $1=$~on~$=$~white.
In colored RGB images, we have three channels: Red (R), Green (G), Blue (B). Where we can light up each color separately.
Because of how the eye works, we are able to trick the brain into perceiving most of the colors by just
mixing these three.
The numbers are usually not stored as floating point numbers, but as unsigned integers, most commonly 8-bit unsigned integers.
This means that 1 is stored as 255 and 0 as 0 and all the numbers are mapped in an equally spaced fashion.
If we have multiple channels, we will Fourier transform each channel separately and
equation \eqref{eqn:dft2d} gets the form
\begin{equation*}
    X_{k,l,c} = \frac{1}{\sqrt{hw}}\sum_{n=0}^{h-1}\sum_{m=0}^{w-1}x_{n,m,c} \ e^{-i{2\pi}\ \left(\!\frac{kn}{h}+\frac{lm}{w}\right)}.
\end{equation*}
In the Fourier space each pixel is a complex number.
We usually visualize the magnitude and the phase (the argument) of each number.
Because the magnitude of the Fourier transformed numbers is not guaranteed
to be between 0 and 255, we scale the Fourier transformed numbers in such a way,
that the highest is 255 and the lowest is 0
\begin{equation*}
    \abs {X'_{k,l,c}} = 255\frac{\abs {X_{k,l,c}}-\min_{n,m} \abs {X_{n,m,c}}}{\max_{n,m} \abs {X_w{n,m,c}}-\min_{n,m} \abs {X_{n,m,c}}}.
\end{equation*}
We also scale the phases to the range of $(0, 255)$
\begin{equation*}
    \arg {X'_{k,l,c}} = 255\frac{\arg X_{k,l,c}}{2\pi}.
\end{equation*}
This mode of visualization can be seen in \autoref{subfig:ftm_n} and \autoref{subfig:ftp_n}.
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.9\linewidth}
        \centering
        \includegraphics[width=\linewidth]{images/webb.png}
        \caption{Original}
    \end{subfigure}
    \begin{subfigure}[h]{.9\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/output/webb_mag_n.png}
        \caption{Fourier Transform Magnitude}
        \label{subfig:ftm_n}
    \end{subfigure}
    \begin{subfigure}[h]{.9\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/output/webb_phase_n.png}
        \caption{Fourier Transform Phase}
        \label{subfig:ftp_n}
    \end{subfigure}
    \begin{subfigure}[h]{.9\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/output/webb_mag_log.png}
        \caption{Fourier Transform magnitude in logarithmic space}
        \label{subfig:ftm_log}
    \end{subfigure}
    \begin{subfigure}[h]{.9\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/output/webb_mag.png}
        \caption{FT Magnitude in logarithmic space and rearranged}
        \label{subfig:ftm}
    \end{subfigure}
    \caption{An example of the Fourier transform. The image is the first calibration picture of the new James-Webb space telescope\cite{webbimg}. }
    \label{fig:fourier_example_n}
\end{figure}

As you can see in the example image, the Fourier magnitude is mostly black, and the phases appear to be noise to the eye.
To better distinguish small features, we will now switch to logarithmic color-space for the magnitude
\begin{equation*}
    \abs {X''_{k,l,c}} = 255\frac{\log \abs {X_{k,l,c}}-\min_{n,m} \log\abs {X_{n,m,c}} } {\max_{n,m} \log\abs {X_{n,m,c}}-\min_{n,m} \log\abs {X_{n,m,c}}}.
\end{equation*}
This is depicted in \autoref{subfig:ftm_log}.
Finally, to better resemble the symmetries in the Fourier space and
to have low frequencies in the middle of the picture, and high frequencies outside,
the indices are shifted
\begin{equation}
    k' = \lfloor k+h/2\rfloor \mod h \qquad l'=\lfloor l+w/2\rfloor\mod w.
    \label{eqn:shift}
\end{equation}
This will be our method of visualization from now on and can be seen in \autoref{subfig:ftm}.

\subsection{Editing in Fourier Space}
If we now modify the numbers in the Fourier space and transform the modified numbers back,
we can achieve complex-looking outcome.
For example, if we take the high frequencies in the image away, we are only left with the
"long-range" information, but the "short-range" information is lost.
This means the image will appear to be blurred.
Taking away the high frequencies can be done in multiple ways.
We can just remove the highest frequencies combining the axes by calculating
$(k'-h/2)^2+(l'-w/2)^2 < r^2$ for some $r$ and setting the pixel in Fourier space to 0, if
this evaluates to false. We can also do this for each axis independently
($(k'-h/2)^2<r_y^2$, $(l'-w/2)^2<r_x^2$).
With these hard cut-offs it is easy to see periodic artifacts in the image.
If we do not want that, we can apply a smooth filter.
A selection of different blurring techniques can be seen
in \autoref{fig:blur_2} to \autoref{fig:blur_smooth}.

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_mag.png}
        \caption{Fourier space}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{images/dune.png}
        \caption{Original}
    \end{subfigure}\
    \caption{A movie poster of Dune (2021) which serves as an example image being blurred in this section \cite{dune}.}
    \label{fig:dune_orig}
\end{figure}

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur2_mask.png}
        \caption{Edit in Fourier space }
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur2.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The poster above blurred by only keeping the innermost frequencies with a radius of $20\%$ of the width.}
    \label{fig:blur_2}
\end{figure}
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur1_mask.png}
        \caption{Edit in Fourier space }
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur1.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The poster blurred by only keeping the innermost frequencies with a radius of $10\%$ of the width. This is a circular filter.}
    \label{fig:blur_1}
\end{figure}
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_rect_mask.png}
        \caption{Edit in Fourier space }
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur_rect.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The poster blurred by only keeping the $10\%$ innermost frequencies along each axis independently. This is a rectangular filter.}
    \label{fig:blur_rect}
\end{figure}
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur_smooth_mask.png}
        \caption{Edit in Fourier space }
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/dune_blur_smooth.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The poster blurred by only multipliying each pixel in the Fourier space with $\exp(-0.001 ((k'-h/2)^2+(l'-w/2)^2))$, where $k', l'$ are the shifted indices from above. This is a Gaussian filter.}
    \label{fig:blur_smooth}
\end{figure}

\FloatBarrier

If we do the opposite and take away some low frequencies, we are "sharpening" the image.
We must not understand "sharpening" in the literal meaning, like focusing the lens of a camera,
since we can not generate new information in the image. We can only enhance the short range information.
There are a couple of examples below in \autoref{fig:blackhole} to \autoref{fig:blackhole_sharp_g_g}.
You will notice the colors in the images are distorted, since we completely treat the
color channels separately. In \autoref{fig:blackhole_sharp_g_g} you can see a grayscale version
of the edit, where the colors might be less distracting.

If we take it to the extreme and remove most low frequencies, we can seethe edges in the image, this
is demonstrated in \autoref{fig:edges}.
\begin{figure}
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp_smooth_mask.png}
        \caption{Fourier space}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{images/blackhole.png}
        \caption{Original}
    \end{subfigure}\
    \caption{The first image of a black hole, taken in 2021.
        This appears not to be sharp and is taken as an example to experiment with sharpening in Fourier space in this section. }
    \label{fig:blackhole}
\end{figure}
\begin{figure}
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp1_mask.png}
        \caption{Edit in Fourier space}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp1.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The black hole image sharpened by applying a circular filter with a radius of 1\% of the height.}
    \label{fig:blackhole_sharp}
\end{figure}
\begin{figure}
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp3_mask.png}
        \caption{Edit in Fourier space}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp3.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The black hole image sharpened by applying a circular filter with a radius of 3\% of the height.}
    \label{fig:blackhole_sharp3}
\end{figure}
\begin{figure}
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp_smooth_mask.png}
        \caption{Edit in Fourier space}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp_smooth.png}
        \caption{Transformed back to real space}
    \end{subfigure}\
    \caption{The black hole image sharpened by applying a Gaussian filter of $1-\exp(-0.01 ((k'-h/2)^2+(l'-w/2)^2))$.}
    \label{fig:blackhole_sharp_g}
\end{figure}
\begin{figure}
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp_smooth_mask_g.png}
        \caption{Edit in Fourier space}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/blackhole_sharp_smooth_g.png}
        \caption{Original}
    \end{subfigure}\
    \caption{The black hole image sharpened by applying a Gaussian filter of $1-\exp(-0.01 ((k'-h/2)^2+(l'-w/2)^2))$ in black and white.}
    \label{fig:blackhole_sharp_g_g}
\end{figure}
\begin{figure}
    \centering
    \begin{subfigure}[h]{.32\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{images/A.png}
        \caption{Original }
    \end{subfigure}
    \begin{subfigure}[h]{.32\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/A_sharp_smooth_mask.png}
        \caption{Edit}
    \end{subfigure}
    \begin{subfigure}[h]{.32\linewidth}
        \centering
        \includegraphics[width=.9\linewidth]{build/output/A_sharp_smooth.png}
        \caption{Back to real space}
    \end{subfigure}
    \caption{An image sharpened with a Gaussian filter of $1-\exp(-0.001 ((k'-h/2)^2+(l'-w/2)^2))$ to demonstrate edge-detecting capabilities of Fourier edits.}
    \label{fig:edges}
\end{figure}
\FloatBarrier

\subsection{Diffraction}
As already mentioned, we can
"simulate" Fraunhofer diffraction by displaying the magnitude of the Fourier transformed image.
In the images in this section, you will be able to realize a couple of known features
of diffraction images.
Lines in the diffraction images will be perpendicular to lines in the aperture (the original image).
For two double slits of width $b$ and distance $a$ the analytical expression for Fraunhofer diffraction
is
\begin{equation*}
    I(\theta) = I_0 \left(\frac{\sin(c b \sin\theta)}{c b \sin\theta} \cos(c a \sin\theta)\right)^2.
\end{equation*}
Which is calculated by solving equation \eqref{eqn:fraunhofer}.
With some constant $c$. From this we can read that the smaller the slits are, the wider the
encapsulating intensity is (since $b < a$) and the closer the slits are, the smaller the distance between
the peaks is. This can be seen in \autoref{fig:double_slit}.

This calculation is rather simple, but also more complex apertures can be investigated, like a circular
aperture, seen in \autoref{fig:circular}. The analytical solution to this is a Bessel function of the radius.

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{\linewidth}
        \centering
        \includegraphics[width=.49\linewidth]{images/hole.png}
        \includegraphics[width=.49\linewidth]{build/output/hole_mag.png}
        \caption{Circular aperture.}
    \end{subfigure}
    \begin{subfigure}[h]{\linewidth}
        \centering
        \includegraphics[width=.49\linewidth]{images/hole_tiny.png}
        \includegraphics[width=.49\linewidth]{build/output/hole_tiny_mag.png}
        \caption{Circular aperture with smaller radius.}
    \end{subfigure}
    \caption{Fraunhofer diffraction image (right) for two different circular apertures (left).}
    \label{fig:circular}
\end{figure}

\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{\linewidth}
        \centering
        \includegraphics[width=.49\linewidth]{images/double_slit_small_norm.png}
        \includegraphics[width=.49\linewidth]{build/output/double_slit_small_norm_mag.png}
        \caption{Double slit.}
    \end{subfigure}
    \begin{subfigure}[h]{\linewidth}
        \centering
        \includegraphics[width=.49\linewidth]{images/double_slit_small_close.png}
        \includegraphics[width=.49\linewidth]{build/output/double_slit_small_close_mag.png}
        \caption{Double slit where the slits are closer.}
    \end{subfigure}
    % \begin{subfigure}[h]{\linewidth}
    %     \centering
    %     \includegraphics[width=.49\linewidth]{images/double_slit_small_wide.png}
    %     \includegraphics[width=.49\linewidth]{build/output/double_slit_small_wide_mag.png}
    %     \caption{Simulated diffraction image (right) of aperture image (left).}
    % \end{subfigure}
    \begin{subfigure}[h]{\linewidth}
        \centering
        \includegraphics[width=.49\linewidth]{images/double_slit_small_widest.png}
        \includegraphics[width=.49\linewidth]{build/output/double_slit_small_widest_mag.png}
        \caption{Double slit where the slits are further apart.}
    \end{subfigure}
    \begin{subfigure}[h]{\linewidth}
        \centering
        \includegraphics[width=.49\linewidth]{images/double_slit_norm_widest.png}
        \includegraphics[width=.49\linewidth]{build/output/double_slit_norm_widest_mag.png}
        \caption{Double slit with wider slits, but same distance as last image.}
    \end{subfigure}
    \caption{Different images of double slits with different features. The aperture image is on the left and the diffraction image is on the right.}
    \label{fig:double_slit}
\end{figure}

\subsection{Compression}
In this section we will very quickly review a naive compression algorithm, based on the discrete cosine transform.
Here the images will not be displayed in the transformed space, since they were very uninformative in my testing.
They appeared to look like mostly a single color with very subtle variation.
The idea of our naive compression is to sort the cosine-space coefficient by magnitude and just
set the lowest of them to zero. Then we only need to save the nonzero coefficients.
The results of removing the lowest coefficients in this way can be seen in \autoref{fig:compr}.

It is fairly obvious, that this compression is not looking very good, except at low compression rates.
But would it save space on disk? To answer this question I implemented a very simple binary format,
saving the cosine coefficients and their indices in a binary, compressed format. Sadly, except
at absurdly high compressions, this did not save disk space.
On the contrary: The file size actually increased, since now that we are not saving every
pixel, we also have to save the indices, which takes additional space.
The actual \texttt{.jpg} format is a little bit more complicated.
It performs the DCT on each $8\times 8$ tile in the image and instead of
setting small amplitudes to 0, it tries to find a smaller, finite set of amplitudes,
so that setting the real amplitudes to the one closest to it in the smaller set of amplitudes, does
not change too much.
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/plots/A9.png}
        \caption{10\% compression}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/plots/A5.png}
        \caption{50\% compression}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/plots/A2.png}
        \caption{80\% compression}
    \end{subfigure}
    \begin{subfigure}[h]{.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{build/plots/A1.png}
        \caption{90\% compression}
    \end{subfigure}
    \caption{The same image with different "compression levels". 10\% compression means that the lowest 10\% of the
        cosine coefficients have been set to 0.}
    \label{fig:compr}
\end{figure}


Galois theory involves studying polynomials over a field, which are ubiquitous in algebra and number theory. So we
start with the basics of fields.
\begin{defn}
Recall that a \term{field} $k$ is a commutative ring with $1$ such that $1\ne 0$ and every nonzero element is
invertible, and $1\ne 0$.
\end{defn}
That is: in a field we can add, subtract, multiply, and divide, though as usual we can't divide by zero. A ``field
homomorphism'' (meaning a structure-preserving map) is just a ring homomorphism; we ask to preserve addition and
multiplication, and subtraction and division come for free.
\begin{comp}{exm}{enumerate}
	\item The rational numbers $\Q$, the real numbers $\R$, and the complex numbers $\C$ are all fields.
	\item Let $p$ be prime. Then, $\Z/p$ is a field, called the \term{finite field of order $p$} and denoted
	$\F_p$.
	\begin{proof}
	First, we show $\Z/p$ is an integral domain: if $m\cdot n\equiv 0\bmod p$, then $m\cdot n = xp$ for some $x$,
	so $p$ divides either $m$ or $n$. Thus, either $m$ or $n$ is $0$ in $\Z/p$.

	Now, for any nonzero $m\in\Z/p$, consider the multiplication map $\vp_m\cdot:\Z/p\to\Z/p$ sending $n\mapsto
	mn$. If $a,b\in\Z/p$ are such that $\vp_m(a) = \vp_m(b)$, then $\vp_m(a-b) = m\cdot(a-b) = 0$. Since $m$ is
	nonzero and $\Z/p$ is an integral domain, then $a = b$, and so $\vp_m$ is injective. Since $\Z/p$ is finite, an
	injective map from $\Z/p$ to itself is a bijection. Thus, there's a unique $n\in\Z/p$ such that $\vp_m(n) = 1$,
	or $1 = mn$. That is, every nonzero element is invertible.
	\end{proof}
	\item If $k$ is any field, we can form the \term{field of rational functions} in $k$, denoted $k(x)$, to be
	ratios $p/q$ for polynomials $p,q\in k[x]$ with $q\ne 0$. We'd like them to be ``in lowest terms,'' but this is
	a clunky definition and it's simpler to just say that two rational functions $p/q$ and $p'/q'$ are the same if
	we can cross-multiply: $pq' = p'q$.
\end{comp}
There's not a whole lot we can say about a field $k$ in total generality, without knowing more about it, but we
know $1\in k$, and therefore $1+1\in k$, and $1+1+1\in k$, and so on. These numbers might all be distinct, like for
$\Q$, or might not be, like for $\F_p$.
\begin{defn}
The \term{characteristic} $\chr(k)$ of a field $k$ is the smallest multiple of $1$ that is equal to $0$ in $k$, or
is $0$ if no such multiple exists.
\end{defn}
For example, $\chr(\F_p) = p$, and $\chr(\Q) = \chr(\R) = 0$.
\begin{ex}
Show that for any field $k$, $\chr(k)$ is either $0$ or a prime number.
\end{ex}
The characteristic is an important property of a field: many important things in Galois theory are different in the
characteristic $0$ case and the characteristic $p$ case.

One of the consistent lessons of algebra is to study objects by looking at their homomorphisms. In the case of
fields, morphisms are the setting of Galois theory.
\begin{lem}
Let $\vp:k\to K$ be a homomorphism of fields. Then, $\vp$ is injective.
\end{lem}
\begin{proof}
The kernel $\ker(\vp)\subset k$ is an ideal of $k$. Since $k$ is a field, its only ideals are $0$ and $k$ itself.
If $\ker(\vp) = 0$, then $\vp$ is injective, as desired; if $\ker(\vp) = k$, then $\vp(1) = 0$, which is
impossible, because ring morphisms must send $1$ to $1$.
\end{proof}
This is very different than for other kinds of algebraic objects: no interesting kernels and no interesting
quotients.
\begin{defn}
If $k$ is a field, a \term{field extension} of $k$ is a homomorphism $i:k\inj L$, often written
$L/k$.\footnote{This notation looks like a quotient, but since we will never take the quotient by a field
extension, this is not ambiguous. We will take quotients of rings, however.}
\end{defn}
Galois theory is the study of field extensions and relations between them.
\begin{defn}
Let $k$ be a field and $i_K:k\inj K$ and $i_L:k\inj L$ be field extensions. An \term{embedding}
(sometimes said to be an \term{embedding over $k$}) is a field homomorphism $j:K\inj L$ such that the following
diagram commutes:
\[\xymatrix{
	K\ar[rr]^j && L\\
	& k.\ar[ul]_{i_K}\ar[ur]^{i_L}
}\]
That is, $i_L = j\circ i_K$.
\end{defn}
Embeddings keep track of how one field lies as a subfield of another.
\begin{defn}
If $k$ is a field, its \term{prime subfield} is the subfield of $k$ generated by $1$.
\end{defn}
That is, the prime subfield is the smallest field containing $1$ inside $k$, meaning it must contain $1+1$,
$1+1+1$, and so forth. If $\chr(k) = 0$, this generates a copy of $\Z$ inside $k$, and therefore $\Q$ also, since
we can invert all nonzero elements of $\Z$. That is, if $\chr(k) = 0$, then the prime subfield of $k$ is $\Q$.
In the same way, if $\chr(k) = p$, then its prime subfield is $\F_p$.
\begin{cor}
If $K/k$ is a field extension, then $\chr(k) = \chr(K)$.
\end{cor}
This is because the prime subfield of $K$ contains the prime subfield of $k$.
\begin{comp}{exm}{enumerate}
\label{fieldextexm}
	\item Since $\Q$ is a subfield of $\R$, the inclusion $\Q\inj\R$ is a field extension. In the same way,
	$\R\inj\C$ is a field extension; this fixes $\Q$, so it's an embedding over $\Q$.
	\item\label{Gaussrat} The \term{field of Gaussian rationals} is $\Q(i) = \set{a+bi\mid a,b\in\Q}$.
	\begin{ex}
	Show that $\Q(i)$ is a field.
	\end{ex}
	If $a\in\Q$, $a = a+0i\in\Q(i)$, so $\Q\inj\Q(i)$ is another example of a field extension. One can form a
	similar definition for, e.g.\ $\Q(\sqrt 3)$ or $\Q(\sqrt{-2})$, but we'll soon define something much more
	general.
\end{comp}
\begin{aside}[Categorical language in field theory]
The modern formulation of abstract algebra tends to use categorical language, defining categories of algebraic
objects such that useful constructions satisfy universal properties. However, this is uncommon for field
theory, as the category of fields is poorly behaved: few products exist (e.g.\ the ring $\Q\times\Q$ is not a
field), there are no initial or final objects, and the category is disconnected, since a map of fields must
preserve the characteristic. Specializing to the category of fields of a given characteristic fixes some, but not
all, of these problems.

Nonetheless, there are a few places where words from category theory will simplify things, and I'll try to mention
them as they happen. Since they may require knowledge beyond what I assume for these notes, they will also be
marked as asides.

Fixing a base field $k$, we can define the \term{category of field extensions} $\FExt_k$ to be the category whose
objects are field extensions $K/k$ and whose morphisms are embeddings $K\inj L$ over $k$.

{\color{red}TODO}: talk about the poset? Eventually will be a lattice.
\end{aside}
\begin{lem}
If $i:k\inj K$ is a field extension, then $K$ is a $k$-vector space.
\end{lem}
\begin{proof}
We need to define an action of $k$ on $K$, which will just be multiplication: if $\lambda\in k$ and $x\in K$, let
$\lambda\cdot x = i(\lambda)x$. The field axioms of $K$ imply (since $k$ is realized as a subfield of $K$) that
multiplication satiafies the axioms for a vector space.
\end{proof}
\begin{defn}
\label{finite_ext}
If $K/k$ is a field extension, its \term{degree}, written $[K:k]$, is the dimension of $K$ as a $k$-vector space.
If this is a finite number, $K/k$ is said to be a \term{finite extension}; otherwise, it's an \term{infinite
extension}.
\end{defn}
This is analogous to the index of a subgroup $[G:H]$. For example, $[\C:\R] = 2$, so $\C/\R$ is a finite extension,
but $\R/\Q$ is an infinite extension.

%The first important reason we care about field extensions is to discuss roots of polynomials. For example, $x^2-2$
%doesn't have a root in $\Q$, but it has $2$ roots in $\Q(\sqrt 2)$. This enables questions about polynomials to be
%turned into questions about field extensions, which can be solved with algebraic methods.
% \begin{thm}
% \label{rootinext}
% Let $f\in k[x]$ be a nonconstant polynomial. Then, there's an extension $K/k$ such that $f$ has a root in $K$.
% \end{thm}
% Algebraically, what does it actually mean that $f$ has a root in $K$, given that $f$ is only defined over $K$? The
% extension $i:k\inj K$ induces a map $i_*:k[x]\inj K[x]$ by applying $i$ to each coefficient of a polynomial.
% Technically, one says that $i_*f$ has a root in $K$, but there's rarely if ever a need to keep the two separate, so
% we identify a polynomial with its image over an extension field.
%
% Another way of thinking of this is that a polynomial defines a function: $x^2+1$, for example, is a function
% $\Q\to\Q$ that doesn't vanish. However, we can define the same function with the same coefficients on $\Q(i)$, and
% there it vanishes for $x = \pm i$.
% \begin{proof}
% Without loss of generality, assume $f$ is irreducible: if not, replace $f$ with one of its irreducible factors. A
% field containing a root of a factor of $f$ must then contain a root of $f$.
%
% Since $f$ is nonconstant and irreducible, the ideal $(f)\subset k[x]$ is maximal, and therefore $K = k[x]/(f)$ is a
% field. Let $j:k\inj k[x]$ send $a\in k$ to the constant polynomial $a$, and $\pi:k[x]\surj K$ be the canonical
% projection, taking everything mod $(f)$. Then, $\pi\circ j: k\to K$ is a field homomorphism, so it's an extension.
%
% Finally, we need to produce a root. Consider $\alpha = \pi(x)\in K$. Since $\pi$ is a ring homomorphism, it
% commutes with polynomials, and therefore $p(\alpha) = \pi(p(x))$, and this is $p(x)\bmod p(x) = 0$. Hence, $\alpha$
% is a root of $f$.
% \end{proof}
% This is a very useful way of producing field extensions, as we'll see later. In particular, it generalizes
% Example~\ref{fieldextexm}\eqref{Gaussrat}: $\Q(i)$ is isomorphic to $\Q[x]/(x^2+1)$, and similarly $\Q(\sqrt 3)
% \cong \Q[x]/(x^2-3)$. Similarly, $\C \cong \R[x]/(x^2+1)$ under the isomorphism $a+bi\mapsto a+bx$. We could use
% this to \emph{define} the complex numbers concretely: rather than supposing that we had a square root of $-1$,
% we've constructed the field $\R[x]/(x^2+1)$, where $x^2 = -1$. In the same way, any polynomial in $\Q[x]$ has a
% root in an extension of $\Q$, which motivates the existence of algebraic numbers.

{\color{red}TODO}: treat this in a unified way with adjoining an element; explain what it means to ``adjoin a
square root of $2$,'' and that this is algebraically indistinguishable (maybe this should be another section, and
then irreducibility criteria is a third section).

\subsection*{Irreducibility criteria}
The key of Theorem~\ref{rootinext} is that $f$ is irreducible; if we take $k[x]/(f)$ for a reducible $f$,
$(f)$ isn't even prime, so the resulting quotient isn't a field, or even an integral domain! So in practice, we
need to know when a polynomial is irreducible. Here are a few criteria.
\begin{lem}
\label{cubirred}
If $f\in k[x]$ has degree $2$ or $3$, then $f$ is irreducible iff it has no roots.
\end{lem}
Of course, the hypothesis is necessary: $(x+2)^2\in\Q[x]$ is reducible, but has no roots.
\begin{thm}[Rational root theorem]
{\color{red}TODO}
\end{thm}
\begin{lem}[Gauss' lemma]
\label{gausslem}
Let $R$ be a UFD and $k$ be its field of fractions. If $f\in R[x]$, then if $f$ is reducible in $k[x]$, then it's
reducible in $R[x]$.
\end{lem}
This is most often used when $R = \Z$, so $k = \Q$: then, it says that \emph{an $f\in\Z[x]$ is irreducible iff it's
irreducible in $\Q[x]$}.
\begin{prop}
Let $R$ be an integral domain and $I\subset R$ be a proper ideal. If $f\in R[x]$ and $f\bmod I$ is irreducible in
$(R/I)[x]$, then $f$ is irreducible over $R$.\footnote{The converse is untrue: $x^4+1$ is irreducible over $\Z$,
but reducible modulo every prime.}
\end{prop}
This is most often used when $R = \Z$ and $I = (p)$ for some prime $p$.
\begin{prop}[Eisenstein's criterion]
Let $R$ be an integral domain and $\p\subset R$ be a prime ideal. If $f(x) = x^n + a_{n-1}x^{n-1} + \dotsb + a_1x +
a_0$ is such that $a_{n-1},\dotsc,a_0\in\p$ and $a_0\not\in\p^2$, then $f$ is irreducible in $R[x]$.
\end{prop}
This is generally used for $R = \Z$ and $\p = (p)$ for a prime number $p$. In this case, it says the following.
\begin{cor}
Let $f\in\Z[x]$ be given by $f(x) = x^2 + a_{n-1}x^{n-1} + \dotsc + a_1x + a_0$, and suppose there's a prime number
$p\in\Z$ such that $p\mid a_i$ for $i = 0,\dotsc,n-1$, and $p^2\nmid a_0$. Then, $f$ is irreducible in $\Z[x]$.
\end{cor}
By Lemma~\ref{gausslem}, this also implies $f$ is irreducible in $\Q[x]$. The most common application of
Eisenstein's criterion is to show that a given polynomial in $\Z[x]$ is irreducible over $\Q$.

{\color{red}TODO} fill in these proofs.

Doing a few exercises using these results will probably be more helpful than reading their proofs.
\begin{proof}[Proof of Lemma~\ref{cubirred}]
If $f$ is reducible, then $f = gh$, where $g$ and $h$ are polynomials of degree at least $1$. Since $\deg(g) +
\deg(h)\le 3$, this means at least one of $g$ or $h$ has degree exactly $1$. Without loss of generality, assume
it's $g$, so $g(x) = ax + b$ for $a,b\in k$ and $a\ne 0$; then, $-b/a$ is a root of $g$, and therefore of $f$.

Conversely, if $f$ has a root, then it's reducible.
\end{proof}

In the next few sections, we'll develop the theory of a few nice kinds of field extensions.


\subsection{Normal form}

This is where a formula is shown using only:

\begin{itemize}
\item And / Conjunction- \(\land \)
\item Or / Disjunction   - \(\lor \)
\item Negation - \(\neg \)
\end{itemize}

The conjunctive normal form (CNF) is where a formula is converted to a normal form with the following layout:

$a \land b \land c \land d$

These letters can represent complex sub-formulae, in normal form.

Statements in this form are easier to evaluate, as each subformula can be evaluated separately. The statement is true only if all formulaes within are also true.

The disjunctive normal form (DNF) is similar for \(\lor \).

$a \lor b \lor c \lor d$

\subsection{Properties of the normal form}

The normal binary operators are commutitive - \(A\land B\Leftrightarrow B\land A\) and \(A\lor B\Leftrightarrow B\lor A\)

Both binary operators are associative - \((A\land B)\land C\Leftrightarrow A\land (B\land C)\) and \((A\lor B)\lor C\Leftrightarrow A\lor (B\lor C)\)

Negation is complementary.

\(A\land \neg A\Leftrightarrow F \)

\(A\lor \neg A\Leftrightarrow T\)

Normal binary operators are absorbative.

\(A\land (A\lor B)\Leftrightarrow A\)
\(A\lor (A\land B)\Leftrightarrow A\)

Identity.

\(A\land T\Leftrightarrow A\)

\(A\lor F \Leftrightarrow A\)

Distributivity.

\(A\land (B\lor C)\Leftrightarrow (A\land B)\lor (A\land C)\)

\(A\lor (B\land C)\Leftrightarrow (A\lor B)\land(A\lor C)\)


\subsection{Likelihood function}

We want to estimate parameters. One way of looking into this is to look at the likelihood function:

\(L(\theta ; X)=P(X|\theta )\)

The likelihood function shows the chance of the observed data being generated, given specific parameters.

If this has high peaks then it provides information that \(\theta \) is located in this region.

\subsection{IID}

For multiple events, the likelihood function is:

\(L(\theta ; X)=P(X|\theta )\)

\(L(\theta ; X)=P(A_1 \land B_2 \land C_3 \land D_4…|\theta )\)

If the events are independent, that is the chance of a flip doesn’t depend on any other outcomes, then:

\(L(\theta ; X)=P(A_1|\theta ).P(B_2|\theta ).P(C_3|\theta ).P(D_4|\theta )...\)

If the events are identically distributed, the chance of flipping a head doesn’t change across flips (for example the heads side doesn’t get heavier over time) then:

\(L(\theta ; X)=P(A|\theta ).P(B|\theta ).P(C|\theta ).P(D|\theta )...\)

\(L(\theta ; X)=\prod_{i=1}^n P(X_i|\theta )\)


\section{Methods}
\subsection{Using Conditional Mutual Information}
\begin{itemize}
\item Following \cite{Liu} in order to determine whether methylation $X$ and expression $Y$ of a gene exhibit an L--shape, the conditional Mutual Information $cMI(t)$ for different choices of threshold $t$ is computed.
\[
\mathit{cMI}(t)=I(X,Y|X>t)P(X>t) + I(X,Y|X\le t)P(X\le t)
\]
\item If the relation between methylation and expression shows an L-shape  as $t$ moves from 0 to 1, $\mathit{cMI}(t)$ first decreases and then increases, its value approaching zero when $t$ coincides with the reflection point.
\end{itemize}


\subsection{Normal matrices}

\(M^*M=MM^*\)

All symmetrix matrices are normal

All hermetitian matrices (inc subset symmetric) are normal

Normal matrix never defective


\lab{Algorithm}{Pseudorandom Number Generators}{Pseudorandom Number Generators}
\label{Ch:PRNG}

\objective{This section teaches about how to build pseudorandom number generators and the differences between the different kinds of PRNGs}

\section*{Linear congruential genterators}

%Lab \ref{PRNG}

Lotteries, most board games, and statistics need random numbers. In real life, we roll dice, take balls out of a bag, or spin a wheel. in order to rpresent random number on computer we use  psuedorandom number generators (PRNGs). The numbers are not truly random, but they are based on a complex formula that makes them look ``random." Another thing is that these generators have to be fast for their aplication. So you have to have something that is fast and looks random.

One of the oldest of theses is the Linear Congruiental Generator (LCG). The Generator is defined by the recurrence relation:
$X_{n+1}=(a*X_n + c)$ mod $m$ where

\begin{itemize}
\item[] $X$ is a sequence of pseudorandom values, and
\item[] $m$, $0<m$ is the modulus
\item[] $a$, $0<a<m$ is the multiplier
\item[] $c$, $0\leq c<m$ is the increment
\item[] $X_0$, $0\leq X_0 <m$ is the seed
\end{itemize}

each of these are interger constansts.

\begin{problem}
Write a LCG that produces a number an array of psuedorandom numbers between $0.0$ and $1.0$. Let the arguments be size of array and let a,c,mod,and seed be optional arguments. I recommend $a=1103515245$, $c=12345$, $m=2^{31}-1$, and $seed=4329$ as the defualt values.
\end{problem}

\begin{problem}
Write a LCG that produces a number an array of psuedorandom numbers of integers between two input arguments. Do it by calling your agorithm from problem one and mutiplying it by the values and casting the array as as int using the .astype() function. Let the arguments be size of array and the two intergers. Let a,c,mod,and seed continue to be optional arguments.
\end{problem}

One easy way to ``see" if your generator is random is to look at a bitmap of the output. In python, use the plt.imshow() function to see a bitmap of the array produced by your LCG. Resize your output to be $512 \times 512$.

\begin{problem}
For what values of $a$, $c$, and $m$ does your LCG have a definite pattern.
\end{problem}

This method is not rigorist. There are several ways to test the randomness of your output.

The length over with your random number generator repeats is called the period. The period is at most m, but it may be shorter based on the values of a and c.

According to the Hull-Dobell Theorem (find a source, used wikipedia), The LCG will only have a full period if and only if,
1. $c$ and $m$ are relatively prime,
2. $a-1$ is divisible by all prime factors of $m$,
3. $a-1$ is a multiple of 4 if $m$ is a multiple of 4

\begin{problem}
Test values of $a$,$c$, and $m$ that fit these requirements.
\end{problem}

This algorithm is used to generate random numbers in Java, C++ and other places.

\section*{Mersenne Twister and Bitwise operations}

All numbers can be represented in bits as a base two number. Computers are optimized to work with numbers in that manner. There are operations that work on the the bit representation of the two number. These are XOR, OR, and AND.

AND - if both numbers have a 1 in the ith place then the ith place is 1. Otherwise the ith place is 0.

OR - if one or both numbers have a 1 in the ith place then the ith place is 1. Otherwise the ith place is 0.

XOR - if only one of the two numbers has a 1 in the ith place then the ith place is 1. If both or neither of the numbers has a 1 in the ith place, the ith place is 0.

In addition you can shift the bitwise number over anumber of values. For example, shifting 10100 to the right by one yields 1010 and shifting it to the left by one yields 101000. This can be done by $\ll$ and $\gg$ in python.

The mersenne twister PRNG does a series of these operations to generate random numbers. the Random class in python uses the mersenne twister algorithm.

\begin{problem}
Look at the bitmap of output of sp.rand(512,512). Can you see any patterns?
\end{problem}

\begin{problem}
Analyse the first of 10,000 outputs of the random class. Can you see a period?
\end{problem}


\subsection{Multiplying a matrix and a vector}

One of the most important uses of a matrix is to multiply a matrix by
a vector. In fact, this is one of the reasons matrices were invented.
Let us start by considering an alternative way of writing a system of
linear equations.

\begin{definition}{The vector form of a system of linear equations}{vector-form}
  Suppose we have a system of equations given by
  \begin{equation*}
    \begin{array}{ccc}
      a_{11}x_1 + \ldots + a_{1n}x_n &=& b_1, \\
      a_{21}x_1 + \ldots + a_{2n}x_n &=& b_2, \\
      \vdots \\
      a_{m1}x_1 + \ldots + a_{mn}x_n &=& b_m.
    \end{array}
  \end{equation*}
  We can express this system in \textbf{vector form}%
  \index{system of linear equations!vector form}%
  \index{vector form!of a system of equations}, which is as follows:
  \begin{equation*}
    x_1 \begin{mymatrix}{c} a_{11} \\ a_{21} \\ \vdots \\ a_{m1} \end{mymatrix}
    + x_2 \begin{mymatrix}{c} a_{12} \\ a_{22} \\ \vdots \\ a_{m2} \end{mymatrix}
    + \ldots
    + x_n \begin{mymatrix}{c} a_{1n} \\ a_{2n} \\ \vdots \\ a_{mn} \end{mymatrix}
    ~=~
    \begin{mymatrix}{c} b_1 \\ b_2 \\ \vdots \\ b_m \end{mymatrix}.
  \end{equation*}
\end{definition}

Notice that each vector used here is one column from the corresponding
augmented matrix. There is one vector for each variable in the system,
along with the constant vector. The left-hand side is a linear
combination of column vectors. Linear combinations of column vectors
are so important that we introduce a special notation for them.

\begin{definition}{The product of a matrix and a vector, by columns}{matrix-times-vector-columns}
  \index{multiplication!matrix times vector}%
  \index{matrix!multiplication!matrix times vector}%
  The product of an $m\times n$-matrix $A$ and an $n$-dimensional
  column vector $\vect{x}$ is an $m$-dimensional column vector,
  defined as a linear combination of the columns of $A$ as follows:
  \begin{equation*}
    A\vect{x} \,~=~
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      \vdots & \vdots & \ddots & \vdots \\
      a_{m1} & a_{m2} & \cdots & a_{mn} \\
    \end{mymatrix}
    \begin{mymatrix}{c}
      x_1 \\
      x_2 \\
      \vdots \\
      x_n
    \end{mymatrix}
    ~=~
    x_1 \begin{mymatrix}{c} a_{11} \\ a_{21} \\ \vdots \\ a_{m1} \end{mymatrix}
    + x_2 \begin{mymatrix}{c} a_{12} \\ a_{22} \\ \vdots \\ a_{m2} \end{mymatrix}
    + \ldots
    + x_n \begin{mymatrix}{c} a_{1n} \\ a_{2n} \\ \vdots \\ a_{mn} \end{mymatrix}.
  \end{equation*}
\end{definition}

In other words, we can think of the vector $\vect{x}$ as encoding
instructions for how to take a linear combination of the columns of
$A$. The product $A\vect{x}$ is computed by taking $x_1$ times the
first column of $A$, plus $x_2$ times the second column of $A$, and so
on. For this to work, $A$ must have the same number of columns as
$\vect{x}$ has components.

\begin{example}{Multiplying a matrix and a vector, by columns}{matrix-times-vector-columns}
  Compute the product
  \begin{equation*}
    \begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 5 & 6
    \end{mymatrix} \begin{mymatrix}{r}
      7 \\
      8 \\
      9
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    \begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 5 & 6
    \end{mymatrix} \begin{mymatrix}{r}
      7 \\
      8 \\
      9
    \end{mymatrix}
    ~=~
  7\begin{mymatrix}{c}
    1 \\
    4
  \end{mymatrix}
  + 8 \begin{mymatrix}{c}
    2 \\
    5
  \end{mymatrix}
  + 9 \begin{mymatrix}{c}
    3 \\
    6
  \end{mymatrix}
  = \begin{mymatrix}{c}
    50 \\
    122
  \end{mymatrix}.
\end{equation*}
\end{solution}

There is another way of looking at the product of a matrix and a
vector. Instead of looking at the columns of $A$, we can look at the
rows.

\begin{proposition}{The product of a matrix and a vector, by rows}{matrix-times-vector-rows}
  Let $A$ be an $m\times n$-matrix and let $\vect{x}$ be an $n$-dimensional
  column vector. The product $A\vect{x}$ can also be written like this:
  \begin{equation*}
    A\vect{x} \,~=~
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      \vdots & \vdots & \ddots & \vdots \\
      a_{m1} & a_{m2} & \cdots & a_{mn} \\
    \end{mymatrix}
    \begin{mymatrix}{c}
      x_1 \\
      x_2 \\
      \vdots \\
      x_n
    \end{mymatrix}
    ~=~
    \begin{mymatrix}{c}
      a_{11}x_1+a_{12}x_2+\ldots+a_{1n}x_n \\
      a_{21}x_1+a_{22}x_2+\ldots+a_{2n}x_n \\
      \vdots \\
      a_{m1}x_1+a_{m2}x_2+\ldots+a_{mn}x_n \\
    \end{mymatrix}.
  \end{equation*}
\end{proposition}

\begin{example}{Multiplying a matrix and a vector, by rows}{matrix-times-vector-rows}
  Compute the product
  \begin{equation*}
    \begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 5 & 6
    \end{mymatrix} \begin{mymatrix}{r}
      7 \\
      8 \\
      9
    \end{mymatrix}
  \end{equation*}
  by rows.
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    \begin{mymatrix}{rrr}
      1 & 2 & 3 \\
      4 & 5 & 6
    \end{mymatrix} \begin{mymatrix}{r}
      7 \\
      8 \\
      9
    \end{mymatrix}
    ~=~
    \begin{mymatrix}{c}
      1\cdot 7 + 2\cdot 8 + 3\cdot 9 \\
      4\cdot 7 + 5\cdot 8 + 6\cdot 9
    \end{mymatrix}
    ~=~
    \begin{mymatrix}{c}
      50 \\
      122
    \end{mymatrix}.
  \end{equation*}
  Note that the is exactly the same answer as before.
\end{solution}

When we use Definition~\ref{def:matrix-times-vector-columns}, we
calculate the product by looking at one column of $A$ at a time. When
we use Proposition~\ref{prop:matrix-times-vector-rows}, we calculate the
product by looking at one row of $A$ at a time.  As the above examples
show, both methods give exactly the same answer. Please convince
yourself that this is true in general. This ability to switch back and
forth between a column-based viewpoint and a row-based viewpoint is
one of the central tools of linear algebra.

Using the above operation, we can also write a system of linear
equations in \textbf{matrix form}. In this form, we express the system
as a matrix multiplied by a vector.

\begin{definition}{The matrix form of a system of linear equations}{matrix-form}
  Suppose we have a system of equations given by
  \begin{equation*}
    \begin{array}{c}
      a_{11}x_1 + \ldots + a_{1n}x_n = b_1, \\
      a_{21}x_1 + \ldots + a_{2n}x_n = b_2, \\
      \vdots \\
      a_{m1}x_1 + \ldots + a_{mn}x_n = b_m.
    \end{array}
  \end{equation*}
  Then we can express this system in \textbf{matrix form}%
  \index{system of linear equations!matrix form}%
  \index{matrix form!of a system of equations}, which is as follows:
  \begin{equation*}
    \begin{mymatrix}{cccc}
      a_{11} & a_{12} & \cdots & a_{1n} \\
      a_{21} & a_{22} & \cdots & a_{2n} \\
      \vdots & \vdots & \ddots & \vdots \\
      a_{m1} & a_{m2} & \cdots & a_{mn}
    \end{mymatrix}
    \begin{mymatrix}{c}
      x_1 \\
      x_2 \\
      \vdots \\
      x_n
    \end{mymatrix}
    =
    \begin{mymatrix}{c}
      b_1 \\
      b_2 \\
      \vdots \\
      b_m
    \end{mymatrix}.
  \end{equation*}
\end{definition}

The matrix form of a system of equations is therefore written as
$A\vect{x}=\vect{b}$, where $A$ is the coefficient matrix%
\index{coefficient matrix}%
\index{matrix!coefficient matrix} of the system, $\vect{x}$ is an
$n$-dimensional column vector constructed from the variables of the
system, and $\vect{b}$ is an $m$-dimensional column vector constructed
from the constant terms of the system. Any system of linear equations
can be written in this form.


\subsection{Span}

\subsubsection{Span function}

We can take a subset \(S\) of \(V\). We can then make linear combinations of these elements.

This is called the linear span - \(span (S)\).


%!TEX program = xelatex

\documentclass[a4paper,12pt]{article}
\usepackage{fontspec}
\usepackage{polyglossia}
\usepackage{csquotes}
\setdefaultlanguage{english}
\usepackage[titletoc,toc,title]{appendix}
\usepackage{mathpazo}
\setmainfont
     [ BoldFont       = texgyrepagella-bold.otf ,
       ItalicFont     = texgyrepagella-italic.otf ,
       BoldItalicFont = texgyrepagella-bolditalic.otf ]
     {texgyrepagella-regular.otf}
\usepackage{setspace}

\setstretch{1.10}
\usepackage{amsmath, amsfonts, amssymb, amsthm, dsfont,cancel}
\usepackage{graphicx, float}
\usepackage{tikz, color, xcolor} % For artists
\usepackage[hmargin=3.5cm,vmargin=3cm]{geometry}
\usepackage{setspace}
\usepackage{enumerate} % For (i) (ii) items in enumerate

\newcommand\bhat[1]{\hat{b}_#1}
\newcommand\sumin{\sum_{i=1}^N}

\title{Quick proof on univariate OLS estimator}
\author{Antoine Mayerowitz}

\begin{document}

\maketitle

We suppose that the data consist of $N$ observations $\{y_i, x_i\}_{i=1}^N$ where each observation $i$ consist of a dependent variable $y_i$ and an explanatory variable $x_i$. We suppose a linear relationship in our variables such that
\begin{equation}
	y_i = b_0 + b_1 x_i + \varepsilon_i
\end{equation}
Where $b_0$ and $b_1$ are scalars, shared by all observations and $\varepsilon_i$ is the error term.

Our goal is to find $b_0$ and $b_1$ such that the \emph{sum of squared of residuals}(SSR) is minimized. Where the SSR is defined by
\begin{align*}
	SRR &= \sumin e^2 \\
			&= \sumin (y_i-\bhat{0}-\bhat{1}x_i)^2
\end{align*}

Mathematicaly we seek

\begin{equation}
	\min_{\bhat{0},\bhat{1}} \sumin (y_i-\bhat{0}-\bhat{1}x_i)^2
\end{equation}

As we need to minimize a quadratic function, we take the derivative with respect to both argument and equalize to 0.

\[
\begin{cases}
	\dfrac{\partial \; SSR}{\partial \bhat{0}} = -2 \sumin (y_i-\bhat{0}-\bhat{1}x_i) \\
	~\\
	\dfrac{\partial \; SSR}{\partial \bhat{1}} = -2 \sumin x_i (y_i-\bhat{0}-\bhat{1}x_i) \\
\end{cases}
\]

Then we equalize both equations to 0
\[
\begin{cases}
	-2 \sumin (y_i-\bhat{0}-\bhat{1}x_i) &= 0 \\
	-2 \sumin x_i (y_i-\bhat{0}-\bhat{1}x_i) &= 0 \\
\end{cases}
\]

We now have our system of 2 equations with 2 unknowns that we need to solve for $\bhat{0}$ and $\bhat{1}$

We simplify
\[
\begin{cases}
	\cancel{-2} \sumin (y_i-\bhat{0}-\bhat{1}x_i) &= 0 \\
	\cancel{-2} \sumin x_i (y_i-\bhat{0}-\bhat{1}x_i) &= 0 \\
\end{cases}
\]

Rewriting

\[
\begin{cases}
	\sumin y_i &= \sumin \bhat{0} + \bhat{1} \sumin x_i\\
	\sumin x_i y_i &= \bhat{0} \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

Using the fact that $\sum_{k=1}^K a = K\times a$ and that $\bhat{0}$ do not depends on $i$ we get

\[
\begin{cases}
	\sumin y_i &= N \bhat{0} + \bhat{1} \sumin x_i\\
	\sumin x_i y_i &= \bhat{0} \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

We multiply the first equation by $\frac{1}{N}$

\[
\begin{cases}
	\frac{1}{N}\sumin y_i &= \frac{1}{N} N \bhat{0} + \bhat{1} \frac{1}{N} \sumin x_i\\
	\sumin x_i y_i &= \bhat{0} \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

We define the average $z$ by $\bar{z} := \frac{1}{N}\sumin z_i$.

\[
\begin{cases}
	\bar{y} &= \bhat{0} + \bhat{1} \bar{x}\\
	\sumin x_i y_i &= \bhat{0} \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

We rewrite to get an expression of $\bhat{0}$
\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	\sumin x_i y_i &= \bhat{0} \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

We substitute the first equation in the second
\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	\sumin x_i y_i &= (\bar{y} - \bhat{1} \bar{x}) \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

Developing

\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	\sumin x_i y_i &= \bar{y} \sumin x_i - \bhat{1} \bar{x} \sumin x_i  + \bhat{1} \sumin x_i^2 \\
\end{cases}
\]

We factor by $\bhat{1}$
\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	\sumin x_i y_i &= \bar{y} \sumin x_i + \bhat{1} \left(\sumin x_i^2 - \bar{x} \sumin x_i \right)\
\end{cases}
\]

Isolating $\bhat{1}$

\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	~ \\
	\bhat{1} &= \dfrac{\sumin x_i y_i - \bar{y} \sumin x_i}{\sumin x_i^2 - \bar{x} \sumin x_i}
\end{cases}
\]

After a bit of algebra (see appendix), one can simplify to

\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	~ \\
	\bhat{1} &= \dfrac{\sumin (y_i - \bar{y}) (x_i - \bar{x})}{\sumin (x_i - \bar{x})^2}
\end{cases}
\]

Which is strictly equivalent to

\[
\begin{cases}
	\bhat{0} &= \bar{y} - \bhat{1} \bar{x}\\
	~ \\
	\bhat{1} &= \dfrac{Cov(x,y)}{Var(x)}
\end{cases}
\]

\clearpage
\begin{appendices}
\section{Simplifying $\bhat{1}$}
\subsection{Finding the covariance}
\[
\sumin x_i y_i - \bar{y} \sumin x_i =  N \times Cov(x,y)
\]
This proof need some tricks and heavily play with the fact that $\sumin x_i = \bar{x}$ and that $\sumin \bar{y} = N \bar{y}$. For example, one could write
\[
	\bar{y} \sumin x_i = N\bar{y}\bar{x} = \sumin \bar{x}\bar{y} =\sumin y_i \bar{x}
\]

\begin{proof}
	\begin{align*}
		\sumin x_i y_i - \bar{y} \sumin x_i &= \sumin x_i y_i - N\bar{y}\bar{x}\\
		&= \sumin x_i y_i - N\bar{y}\bar{x} - N\bar{y}\bar{x} +  N\bar{y}\bar{x}\\
		&= \sumin x_i y_i - \sumin x_i \bar{y} - \sumin \bar{x} y_i +  \sumin \bar{y}\bar{x} \\
		& = \sumin (x_i y_i -  x_i \bar{y} - \bar{x} y_i + \bar{y}\bar{x}) \\
		&= \sumin (y_i - \bar{y}) (x_i - \bar{x}) \\
		&= N \times Cov(x,y)
	\end{align*}
\end{proof}

\subsection{Finding the variance}
\[
\sumin x_i^2 - \bar{x} \sumin x_i = N \times Var(x)
\]
This proof is very similar to the one for covariance.

\begin{proof}
	\begin{align*}
		\sumin x_i^2 - \bar{x} \sumin x_i &= \sumin x_i^2 - N\bar{x}\bar{x}\\
		&= \sumin x_i^2 - 2N\bar{x}\bar{x} + N\bar{x}\bar{x} \\
		&= \sumin x_i^2 - 2 \bar{x} \sumin x_i + N\bar{x}^2 \\
		&= \sumin (x_i^2 - 2\bar{x}x_i + \bar{x}^2) \\
		&= \sumin (x - \bar{x})^2 \\
		&= N \times Var(x)
	\end{align*}
\end{proof}

\end{appendices}

\end{document}


\subsection{Substitution}

If we have a tautology, then we can substitute the formula of any propositional variable with any formula to arrive at any other tautology.

For example, we know that \(\theta \lor \neg \theta \) is a tautology. This means that an arbitrary formula for \(\theta \) is also a tautology.

An example is \((\gamma \land \alpha )\lor \neg (\gamma \land \alpha )\), which we know is a tautology, without having to examine each variable.

\subsection{Syntactic consequence}

Let us call the first formula \(A\) and the second \(B\). We can then say:

\(A\vdash B\)

This says that: if \(A\) is true, then we can deduce that \(B\) is true using steps such as substitution.

\subsection{Modus Ponens}

Modus Ponens is a deduction rule. This allows us to use stpes other than substitution to derive new tautologies.

If \(A\) implies \(B\), and \(A\) is true, then \(B\) is also true.

\((\theta \rightarrow \gamma )\land \theta \Rightarrow \gamma \)

That is, if we can show that the following are true:

\(\theta \rightarrow \gamma \)

\(\theta \)

We can infer that the following is also true:

\(\gamma \)

\subsection{Theory}

Results derived from substitution or induction are called theorems. Theorems often divided into:

\begin{itemize}
\item Theorems - important results
\item Lemmas - results used for later theorems
\item Corollaries - readily deduced from a theorem
\end{itemize}

We take a set of axioms, as true, and a deduction rule which enables us to derive additional formulae, or theorems. The collection of axioms and theorems is known as the theory.


\lab{QR Decomposition}{QR decomposition}
\label{lab:QRdecomp}
\objective{Use the Gram-Schmidt algorithm and orthonormal transformations to perform the QR decomposition.}

The QR decomposition of a matrix $A$ is a factorization $A=QR$, where $Q$ has orthonormal columns and $R$ is upper triangular.
This decomposition is useful for computing least squares and finding eigenvalues.
As stated in the following theorem, the QR decomposition of $A$ always exists when the rank of $A$ equals the number of columns of $A$.
\begin{theorem}
Let $A$ be an $m\times n$ matrix of rank $n$.  Then $A$ can be
factored into a product $Q R$, where $Q$ is an $m\times n$ matrix
with orthonormal columns and $R$ is a nonsingular $n \times n$ upper
triangular matrix.
\end{theorem}

In this lab we will only discuss real matrices.
All of these results can be extended to the complex numbers by replacing ``orthogonal matrix'' with ``unitary matrix,'' ``transpose'' with ``hermitian conjugate,'' and ``symmetric matrix'' with ``hermitian matrix.''

\section*{Modified Gram-Schmidt}
Let $A$ be an $m \times n$ matrix of rank $n$.
There are many methods for computing the QR decomposition.
First we will discuss the algorithm that computes $Q$ by applying Gram-Schmidt to the columns of $A$.

Let $\{\x_i\}_{i=1}^n$ be the columns of $A$ (the rank hypothesis implies that these are linearly independent vectors).
Then the Gram-Schmidt algorithm computes an orthonormal basis $\{\q_i\}_{i=1}^n$ for the span of the $\x_i$.
The Gram-Schmidt algorithm defines  \[ \q_1 = \frac{\x_1}{\norm{\x_1}}.\]
It recursively defines $\q_k$ for $k>1$ by
\[
\q_{k} = \frac{\x_k - \p_{k-1}}{\|\x_k - \p_{k-1}\|}, \,\,\,\, k=2,\ldots,n,
\]
where
\[
\p_{k-1} = \sum_{i=1}^{k-1} \langle \q_i, \x_k\rangle \q_i, \,\,\,\, k=2, \ldots, n
\]
and $\p_0 = 0$.

In the QR factorization of $A$, the matrix $Q$ has columns $\{\q_i\}_{i=1}^n$.
The matrix $R$ is then the upper triangular matrix with entries $r_{jk}$ where $r_{kk} = \|\x_k-\p_{k-1}\|$ and $r_{j k} = \langle \q_j, \x_k\rangle$ when $j < k$.
See [TODO: ref textbook] for a proof that $QR=A$ and $R$ is nonsingular.

When implemented with a computer, the Gram-Schmidt algorithm may produce a matrix $Q$ with columns that are not even close to orthonormal due to rounding errors.
We now introduce the modified Gram-Schmidt algorithm, which consistently produces matrices $Q$ whose columns are ``very close'' to orthonormal.

In the modified Gram-Schmidt algorithm, $\q_1$ is the normalization of $\x_1$ as before.
We then make each of the vectors $\x_2, \ldots, \x_n$ orthogonal to $\q_1$ by defining
\[
\x_k := \x_k - \langle \q_1,\x_{k} \rangle \q_1,\quad k=2,\ldots,n.
\]
(Compare this to the usual Gram-Schmidt algorithm, where we only made $\x_2$ orthogonal to $\q_1$.)
Next we define $\q_2 = \frac{\x_2}{\|\x_2\|}.$ Once again we make $\x_3, \ldots, \x_n$ orthogonal to $\q_2$ by defining
\[
\x_k := \x_k - \langle \q_2,\x_{k} \rangle \q_2,\quad k=3,\ldots,n.
\]
(Each of these new vectors is a linear combination of vectors orthogonal to $\q_1$, and hence is orthogonal to $\q_1$ as well.)
We continue this process until we have an orthonormal set $q_1, \ldots, q_k$.
The entire modified Gram-Schmidt algorithm is described in Algorithm \ref{Alg:gram_schmidt}.

%TODO: proofread this, explain notation, and make the LU decomposition alg. like this
\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Modified Gram-Schmidt}{$A$}
\State $m, n \gets \text{shape} \left( A \right)$
\State $Q \gets \text{copy} \left( A \right)$
\State $R \gets \text{zeros}((n,n))$
\For{$0 \leq i < n$}
    \State $R_{i,i} \gets \norm{Q_{:,i}}$
    \State $Q_{:,i} \gets Q_{:,i}/R_{i,i}$
    \For{$i+1 \leq j < n$}
        \State $R_{i,j} \gets Q_{:,j}^\mathsf{T}Q_{:,i}$
        \State $Q_{:,j} \gets Q_{:,j}-R_{i,j}Q_{:,i}$
	\EndFor
\EndFor
\State \pseudoli{return} $Q, R$
\EndProcedure
\end{algorithmic}
\caption{The modified Gram-Schmidt. This algorithm returns orthogonal $Q$ and upper triangular $R$ such that $A = QR$.}
\label{Alg:gram_schmidt}
\end{algorithm}

%TODO: put this section after the next? What algorithm does SciPy use?
\section*{QR decomposition in SciPy}
The linear algebra library in SciPy calculates the QR decomposition using a software package called LAPACK (Linear Algebra PACKage), which is incredibly efficient.
Here is an example of using SciPy to compute the QR decomposition of a matrix.

\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la

>>> A = np.random.rand(4,3)
>>> Q, R = la.qr(A)
>>> Q.dot(R) == A
array([[ True, False, False],
       [ True, False, False],
       [ True,  True, False],
       [ True,  True, False]], dtype=bool)
\end{lstlisting}
 Note that \li{Q.dot(R)} does not equal \li{A} exactly because of rounding errors.
 However, we can check that the entries of \li{Q.dot(R)} are ``close'' to the entries of \li{A} with the NumPy method \li{np.allclose()}.
 This method checks that the elements of two arrays differ by less than a given tolerance, a tolerance specified by two keyword arguments \li{rtol} and \li{atol} that default to $10^{-5}$ and $10^{-8}$ respectively.
 You can read the documentation to learn how the tolerance is computed from these numbers.
\begin{lstlisting}
>>> np.allclose(Q.dot(R), A)
True
\end{lstlisting}
We can use the same method to check that \li{Q} is ``very close'' to an orthogonal matrix.
\begin{lstlisting}
>>> np.allclose(Q.T.dot(Q), np.eye(4))
True
\end{lstlisting}

\begin{problem}
\label{prob:QR}
Write a function that accepts as input a $m \times n$ matrix of rank $n$ and computes its QR decomposition, returning the matrices $Q$ and $R$.
Your function should use Algorithm \ref{Alg:gram_schmidt}.
Hint: Read about the function \li{np.inner()}.
Another hint: Check that your function works by using \li{np.allclose()}.
\end{problem}

\begin{problem}
Write a function that accepts a square matrix $A$ of full rank and returns $\abs{\det(A)}$.
Use the QR decomposition of $A$ to perform this calculation.
Hint: What is the determinant of an orthonormal matrix?
\end{problem}

\section*{Householder triangularization}
Another way to compute the QR decomposition of a matrix is with a series of orthonormal transformations.
Like the Modified Gram-Schmidt algorithm, orthonormal transformations are numerically stable, meaning that they are less susceptible to rounding errors.

\subsection*{Householder transformations}
The \emph{hyperplane} in $\mathbb{R}^n$ with normal vector $\mathbf{v}$ is the set $\{ \x \in \mathbb{R}^n \mid \langle \x, \mathbf{v} \rangle = 0 \}$.
Equivalently, the hyperplane defined by $\mathbf{v}$ is just the orthogonal complement $\mathbf{v}^{\perp}$.
See Figure \ref{fig:Householder_reflector}.

A \emph{Householder transformation} of $\mathbb{R}^n$ is a linear transformation that reflects about a hyperplane.
If a hyperplane $H$ has normal vector $\mathbf{v}$, let $\mathbf{u} = \mathbf{v}/\|\mathbf{v}\|$.
Then the Householder transformation that reflects about $H$ corresponds to the matrix $H_{\mathbf{u}} = I - 2 \mathbf{u}\mathbf{u}^T$.
You can check that $(I - 2 \mathbf{u}\mathbf{u}^T)^T(I - 2 \mathbf{u}\mathbf{u}^T)=I$, so Householder transformations are orthogonal.

%\newcommand{\ipt}[2]{\ensuremath{\left\langle #1,#2 \right\rangle}}
\begin{figure}
\begin{center}
\begin{tikzpicture}
%\draw[-, thick](0,-1.5)--(0,3); %x-axis
%\draw[-,thick](-1,0)--(4,0); %y-axis
%\node[draw, thick, minimum height=4.5cm, minimum width=5cm]()at(1.5,.75){};
\draw[-,dashed, gray](-2,-1.333)--(3, 2); % hyperplane
\draw[->, gray, >=stealth,ultra thick](0,0)--(.8,-1.2); % v
\draw[->, >=stealth, thick](0,0)--(2.815, .777); % H x
\draw[->, >=stealth, thick](0,0)--(1.8,2.3); % x
\node[draw=none](v)at(.65,-.6){$\v$};
\node[draw=none](x)at(.75,1.5){$\x$};
\node[draw=none](Hx)at(3, .5){$H_{\v}\x$};
\node[draw=none](H)at(-1.5,-.6){$H$};
%\node[draw=none](bullet)at(2.31, 1.53){\textbullet};
%\node[draw=none](vandx)at(4.0,1.5){$\x -
%\left \langle \dfrac{\v}{\|\v\|}, \x \right \rangle \dfrac{\v}{\|\v\|}$};
\end{tikzpicture}
\end{center}
\caption{This is a picture of a Householder transformation.
The normal vector $\mathbf{v}$ defines the hyperplane $H$.
The Householder transformation $H_{\mathbf{v}}$ of $\mathbf{x}$ is just the reflection of $\mathbf{x}$ across $H$.}
\label{fig:Householder_reflector}
\end{figure}

\subsection*{Householder triangularization}
The QR decomposition of an $m \times n$ matrix $A$ can also be computed with Householder transformations via the Householder triangularization.
Whereas Gram-Schmidt makes $A$ \emph{orthonormal} using a series of transformations stored in an \emph{upper triangular} matrix, Householder triangularization makes $A$ \emph{triangular} by a series of \emph{orthonormal} transformations.
More precisely, the Householder triangularization finds an $m \times m$ orthogonal matrix $Q$ and and $m \times n$ upper triangular matrix $R$ such that $QA = R$.
Since $Q$ is orthogonal, $Q^{-1}=Q^T$ so $A = QR$.
(Contrast this with the QR decomposition where $A = QR$, the matrix $Q$ is $m \times n$, and $R$ is $n \times n$).
If $A$ is square, then $A = Q^TR$ is the QR decomposition.

Let's demonstrate the idea behind Householder triangularization on a $4 \times 3$ matrix $A$.
Let $e_1, \ldots, e_4$ be the standard basis of $\mathbb{R}^4$.
First we find an orthonormal transformation $Q_1$ that maps the first column of A into the span of $e_1$.
This is diagrammed below, where $*$ represents an arbitrary entry.

\def\mc#1{\multicolumn{1}{c|}{#1}}
\def\lc#1{\multicolumn{1}{|c}{#1}}
\begin{equation*}
\begin{pmatrix}
* & * & * \\
* & * & * \\
* & * & * \\
* & * & *
\end{pmatrix}
\underrightarrow{Q_1}
\begin{pmatrix}

* & * & * & \\ \cline{2-3}
\mc{0} & * & \mc{*}& \\
\mc{0} & * & \mc{*} & \\
\mc{0}& * & \mc{*} & \\ \cline{2-3}
\end{pmatrix}
\end{equation*}
Let $A_2 = Q_1A$ be the matrix on the right above.
Now find an orthonormal transformation $Q_2$ that fixes $e_1$ and maps the second column of $A_2$ into the span of $e_1$ and $e_2$.
Notice that since $Q_2$ fixes $e_1$, the top row of $A_2$ will be fixed by $Q_2$, and only entries in the boxed submatrix will change.

%\def\mc#1{\multicolumn{1}{c|}{#1}}
%\begin{equation*}
%\begin{pmatrix}
%* & * & * & \\ \cline{2-3}
%\mc{0} & * & \mc{*}& \\
%\mc{0} & * & \mc{*} & \\
%\mc{0}& * & \mc{*} & \\ \cline{2-3}
%\end{pmatrix}
%
%\underrightarrow{Q_1}
%
%\begin{pmatrix}
%* & * & * & \\ \cline{2-3}
%\mc{0} & 0 & \mc{*}& \\
%\mc{0} & 0 & \mc{*} & \\
%\mc{0}& 0 & \mc{*} & \\ \cline{2-3}
%\end{pmatrix}
%\end{equation*}

Let $A_3 = Q_2Q_1A$ be the matrix on the right above.
Finally, find an orthonormal transformation $Q_3$ that fixes $e_1$ and $e_2$ and maps the third column of $A_3$ into the span of $e_1$, $e_2$, and $e_3$.
The diagram below summarizes this process, where boxed entries indicate those affected by the operation just performed.

\begin{equation*}
Q_3 Q_2 Q_1
\begin{pmatrix}
* & * & * \\
* & * & * \\
* & * & * \\
* & * & *
\end{pmatrix}
= Q_3 Q_2
\begin{pmatrix}  \cline{2-4}
&\lc{*} & * & \mc{*} & \\
&\lc{0} & * & \mc{*}& \\
&\lc{0} & * & \mc{*} & \\
&\lc{0}& * & \mc{*} & \\ \cline{2-4}
\end{pmatrix}
= Q_3
\begin{pmatrix}
* & * & * \\ \cline{2-3}
\mc{0} & * & \mc{*} \\
\mc{0} & 0 & \mc{*} \\
\mc{0} & 0 & \mc{*} & \\ \cline{2-3}
\end{pmatrix}
=
\begin{pmatrix}
* & * & * \\
0 & * & * \\ \cline{3-3}
0 & \mc{0} & \mc{*} \\
0 & \mc{0} & \mc{0} & \\ \cline{3-3}
\end{pmatrix}.
\end{equation*}

We've accomplished our goal, which was to triangularize $A$ using orthonormal transformations.
But how do we construct the matrices $Q_k$?

It turns out that we can choose each $Q_k$ to be a Householder transformation.
Suppose we have found Householder transformations $Q_1, \ldots, Q_{k-1}$ such that $Q_{k-1}\ldots Q_2Q_1 = A_k$ where
\[
A_k = \begin{pmatrix}
T & X' \\
0 & X''
\end{pmatrix}.
\]
Here, $T$ is a $(k-1) \times (k-1)$ upper triangular matrix.
Let $\x$ be the $k^{th}$ column of $A_k$.
Write $\x = \x' + \x''$ where $\x'$ is in the span of $e_1, \ldots, e_{k-1}$.
So $\x''$ looks like $k-1$ zeros followed by the first column of $X''$.
The idea is to reflect $\x''$ about a hyperplane into the span of $e_k$.
It turns out that there are two choices that will work (see Figure \ref{fig:two reflectors}).
These hyperplanes have normal vectors $\x'' + \| \x'' \|e_k$ and $\x'' - \| \x'' \|e_k$.
In fact, the more numerically stable transformation is to reflect about the hyperplane with normal vector $\mathbf{v}_k = \x'' +\sign(x_k) \| \x'' \|e_k$, where $x_k$ is the $k^{th}$ entry of $\x''$ (or the top left entry of $X''$).
(You can check that $\mathbf{v}_k$ is orthonormal to $e_1, \ldots, e_{k-1}$, so the plane orthonormal to $\mathbf{v}_k$ contains $e_1, \ldots, e_{k-1}$, and reflecting about it fixes $e_1, \ldots, e_{k-1}$.)
Thus, $Q_k$ is the Householder transformation $H_{\mathbf{v}_k}$.

The final question is to find an efficient algorithm for computing $Q = Q_nQ_{n-1} \ldots Q_1$ and $R = Q_nQ_{n-1} \ldots Q_1A$.
The idea is to start with $R=A$ and $Q = I$. Then we compute $Q_1$ and modify $R$ to be $Q_1A$ and $Q$ to be $Q_1$.
Next we compute $Q_2$ and modify $R$ to be $Q_2Q_1A$ and $Q$ to be $Q_2Q_1$, and so forth.
As we have already discussed, $Q_k$ fixes the first $k-1$ rows and columns of any matrix it acts on.
In fact, if $\x'' = (0, \ldots, 0, x_k, x_{k+1}, \ldots, x_n)$ as above, then $\mathbf{v}_k = (0, \ldots, 0, v_{k_0}, x_{k+1}, \ldots, x_n)$ where $v_{k_0} = x_k + \sign(x_k) \| \x'' \|$.
If $\mathbf{u}_k$ is the normalization of $(v_{k_0}, x_{k+1}, \ldots, x_n) \in \mathbb{R}^{n-(k-1)}$, then
\[
Q_k = I-\frac{2\x''(\x'')^T}{\|\x''\|^2} =  \begin{pmatrix}
I & 0 \\
0 & I-2\mathbf{u}_k\mathbf{u}_k^T
\end{pmatrix}.
\]
This means that, using block multiplication,
\[
Q_kA_k =  \begin{pmatrix}
I & 0 \\
0 & I-2\mathbf{u}_k\mathbf{u}_k^T
\end{pmatrix}\begin{pmatrix}
T & X' \\
0 & X''
\end{pmatrix} = \begin{pmatrix}
T & X' \\
0 & ( I-2\mathbf{u}_k\mathbf{u}_k^T)X''
\end{pmatrix}.
\]
So at each stage of the algorithm, we only need to update the entries in the bottom right submatrix of $A_k$, and these change via matrix multiplication by $ I-2\mathbf{u}_k\mathbf{u}_k^T$. Similarly,
\[
Q_kQ_{k-1}\ldots Q_1 = Q_k \begin{pmatrix}
A\\
B
\end{pmatrix} = \begin{pmatrix}
I & 0 \\
0 & I-2\mathbf{u}_k\mathbf{u}_k^T
\end{pmatrix}\begin{pmatrix}
A\\
B
\end{pmatrix} = \begin{pmatrix}
A\\
(I-2\mathbf{u}_k\mathbf{u}_k^T)B
\end{pmatrix},
\]
so to update $\prod Q_i$, we need only modify the bottom rows.
These also change via matrix multiplication by $I-2\mathbf{u}_k\mathbf{u}_k^T$.

These arguments produce Algorithm \ref{Alg:Householder}.

%To find $Q_1$, we first identify an appropriate hyperplane to reflect $x$ into the span of $e_1$.
%It turns out there are two hyperplanes that will work, as shown in figure \ref{fig:two reflectors}.
%(In the complex case, there are infinitely many such hyperplanes.)
%Between the two, the one that reflects $x$ further will be more numerically stable.
%This is the hyperplane perpendicular to $v = sign(x_1)\norm{x}_2 e_1 + x$.
%
%To see how this works, let $x$ be the first column of the submatrix that we want to project onto the span of $e_1$.
%In order for this to be a unitary operation, this will need to preserve the norm of $x$.
%This means that $\left( I - 2 v v^\mathsf{H} \right) x = \pm \norm{x} e_1$, or, in other words,
%
%\[ 2 v v^\mathsf{H} x =
%\begin{pmatrix}
%x_1 \pm \norm{x} \\
%x_2 \\
%x_3 \\
%\vdots \\
%x_n
%\end{pmatrix}\]
%
%Let $u$ be the vector on the right hand side of this expression.
%It can be shown that the vector  $\frac{u}{\norm{u}}$ is the proper choice for $v$.
%We will show that the vector $\frac{u}{\norm{u}}$ is the proper choice for $v$.
%Notice that:
%
%\[\norm{u}^2 = \norm{x}^2 \pm 2 \norm{x} x_1 + x_1^2 + x_2 + \dots + x_n^2 = 2 \norm{x}^2 \pm 2 \norm{x} x_1 \]
%
%and that
%
%\[\norm{x}^2 \pm \norm{x} x_1 = u^\mathsf{H} x \]
%
%So we have
%
%\begin{align*}
%2 v v^\mathsf{H} x &= 2 u \frac{\norm{x}^2 \pm x_1 \norm{x}}{\norm{u}^2} \\
%		&= 2 u \frac{u^\mathsf{H} x}{\norm{u}^2} \\
%		&= 2 \frac{u}{\norm{u}} \left( \frac{u}{\norm{u}} \right)^\mathsf{H} x
%\end{align*}
%
%So $\frac{u}{\norm{u}}$ is a proper choice of $v$ that will project $x$ into the span of $e_1$.

\begin{figure}
\begin{tikzpicture}

\draw[-, dashed, gray](-3,-1)--(3,1);
\draw[-, dashed, gray](-.8,2.4)--(.6,-1.8);
\draw[-, gray, thick](-4,0)--(4,0);
\draw[->, thick, >=stealth'](0,0)--(2,1.6);
\draw[<->, thick, >=stealth'](-2.6,0)--(2.6,0);
\draw[->, gray,  ultra thick, >=stealth](0,0)--(1.5,.5);
\draw[->, gray, ultra thick, >=stealth](0,0)--(-.5,1.5);

\node[draw=none, node distance=3.5cm]
	(dummy)at(2.5,.2){};
\node[draw=none, node distance=.5cm](Hvx)
	[below of=dummy]{$H_{\v_1}\x$};
\node[draw=none, node distance=2cm](x)
	[above left of=Hvx]{$\x$};
\node[draw=none](v1)[left of=x]{$\v_1$};
\node[draw=none, node distance=.55cm](v2)[below of=x]{$\v_2$};
\node[draw=none, node distance=4cm](hvx2)[left of=dummy]{$H_{\v_2}\x$};

\end{tikzpicture}
\caption{If we want to reflect $\x$ about a hyperplane into the span of $e_1$, there are two hyperplanes that will work.
The two choices are defined by the normal vectors $\v_1$ and $\v_2$.
Reflecting about the hyperplane defined by $\v_i$ produces  $H_{\v_i}\x$.}
\label{fig:two reflectors}
\end{figure}

%TODO: explain notation, fix algorithm
\begin{algorithm}
\caption{Householder triangularization.
This algorithm returns orthonormal $Q$ and upper triangular $R$ satisfying $A = QR$.}
\label{Alg:Householder}
\begin{algorithmic}[1]
\Procedure{Householder}{$A$}
\State $m, n \gets \text{shape} \left( A \right)$
\State $R \gets \text{copy} \left( A \right)$
\State $Q \gets I_m$
\For{$0 \leq k < n-1$}
    \State $u_k \gets \text{copy} \left( R_{k:,k} \right)$
    \State $u_{k_0} \gets u_{k_0} + \text{sign} \left( u_{k_0} \right) \norm{u_k}$
    \State $u_k \gets u_k / \norm{u_k}$
    \State $R_{k:,k:} \gets R_{k:,k:} - 2 u_k \left( u_k^\mathsf{T} R_{k:,k:} \right)$
    \State $Q_{k:} \gets Q_{k:} - 2 u_k \left( u_k^\mathsf{T} Q_{k:} \right)$
\EndFor
\State \pseudoli{return} $Q^\mathsf{H}, R$
\EndProcedure
\end{algorithmic}
\end{algorithm}

\begin{problem}
\label{prob:HouseholderQR}
Write a function that accepts as input a $m \times n$ matrix of rank $n$ and computes its QR decomposition, returning the matrices $Q$ and $R$.
Your function should use Algorithm \ref{Alg:Householder}.
Hint: Read about the function \li{np.outer()}.
\end{problem}

\begin{comment}
\subsection*{Stability of the Householder QR algorithm}
We will now examine the stability of the Householder QR algorithm.
We will use SciPy's built in QR factorization which uses Householder reflections internally.

Try the following.

\begin{lstlisting}
>>> Q, X = la.qr(np.random.rand(500,500)) # create a random orthonormal matrix:
>>> R = np.triu(np.random.rand(500,500)) # create a random upper triangular matrix
>>> A = np.dot(Q,R) # Q and R are the exact QR decomposition of A
>>> Q1, R1 = la.qr(A) # compute QR decomposition of A
\end{lstlisting}

Observe:

\begin{lstlisting}
>>> la.norm(Q1-Q)/la.norm(Q) # check error in Q
0.282842955725
>>> la.norm(R1-R)/la.norm(R) # check error in R
0.0428922016647
\end{lstlisting}

This is terrible!
This algorithm works in $16$ decimal points of precision, but $Q_1$ and $R_1$ are only accurate to $0$ and $1$ decimal points, respectively.
We've lost $16$ decimal points of precision!

Don't lose hope.
Check how close the product $Q_1 R_1$ is to $A$.
\begin{lstlisting}
>>> A1 = Q1.dot(R1)
>>> np.absolute(A1 - A).max()
3.9968028886505635e-15
\end{lstlisting}
We've now recovered $15$ digits of accuracy.
Considering the error relative to the norm of $A$ (using the 2-norm for matrices), we see that this relative error is even smaller.
\begin{lstlisting}
>>> la.norm(A1 - A, ord=2) / la.norm(A, ord=2)
8.8655568331889288e-16
\end{lstlisting}
The errors in $Q_1$ and $R_1$ were somehow ``correlated," so that they canceled out in the product.
The errors in $Q_1$ and $R_1$ are called \emph{forward errors}.
The error in $A_1$ is the \emph{backward error}.

In fact, the large errors in \li{Q1} and \li{R1} were not because the algorithm was bad, it was because $A$ was poorly conditioned.
The condition number for randomly generated upper triangular matrices is generally very high, and this was the case here.
This has, in turn, made the condition number of $A$ extremely large.

Try the following to compute the condition number of $A$.
In this case the condition number of $A$ and $R$ are computed to be different, though, in theory, they should be exactly the same.
\begin{lstlisting}
>>> from numpy.linalg import cond
>>> cond(A)
4.1426075832870472e+18
>>> cond(R)
3.1767577244363792e+19
\end{lstlisting}

Householder QR factorization is more numerically stable than Gram-Schmidt or even Modified Gram-Schmidt (MGS).
However, MGS is still useful for some types of iterative methods because it finds the orthonormal basis one vector at a time instead of all at once (for an example see Lab \ref{lab:EigSolve}).
\end{comment}

\section*{Upper Hessenberg form}
An upper Hessenberg matrix is a square matrix with zeros below the first subdiagonal.
Every  $n \times n$ matrix $A$ can be written $A = Q^THQ$ where $Q$ is orthonormal and $H$ is an upper Hessenberg matrix, called the Hessenberg form of $A$.

A fast algorithm for computing the QR decomposition of a Hessenberg matrix will be taught in Lab \ref{lab:givens}. This algorithm in turn leads to a fast algorithm for finding eigenvalues of a matrix, which will be discussed in Lab \ref{lab:EigSolve}.

For now, we will outline an algorithm for computing the upper Hessenberg form of any matrix.
Like Householder triangularization, this algorithm uses Householder transformations.
To find orthogonal $Q$ and upper Hessenberg $H$ such that $A = Q^THQ$, it suffices to find such matrices that satisfy $Q^TAQ=H$.
Thus, our strategy is to multiply $A$ on the right and left by a series of orthonormal matrices until it is in Hessenberg form.
If we use the same $Q_1$ as in the first step of the Householder algorithm, then with $Q_1 A$ we introduce zeros in the first column of $A$.
However, since we now have to multiply $Q_1 A$ on the left by $Q_1^T$, all those zeros are destroyed.

Instead, let's try choosing a $Q_1$ that fixes $e_1$ and reflects the first column of $A$ into the span of $e_1$ and $e_2$.
Because $Q_1$ fixes $e_1$, the product $Q_1A$ leaves the first row of $A$ alone, and $(Q_1A)Q_1^T$ leaves the first column of $(Q_1A)$ alone.
If $A$ is a $5 \times 5$ matrix, this looks like
\[
\begin{array}{ccccc}
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
* & * & * & * & * \\
* & * & * & * & * \\
* & * & * & * & *
\end{pmatrix}
&\underrightarrow{Q_1 \cdot }&
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & *
\end{pmatrix}
&\underrightarrow{\cdot Q_1^T }&
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & *
\end{pmatrix}
\\
A & & Q_1A & & (Q_1 A) Q_1^T
  \end{array}
\]
We now iterate through the matrix until we obtain
\begin{equation*}
Q_3 Q_2 Q_1 A Q_1^T Q_2 ^T Q_3^T =
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
0 & * & * & * & * \\
0 & 0 & * & * & * \\
0 & 0 & 0 & * & *
\end{pmatrix}.
\end{equation*}

The pseudocode for computing the Hessenberg form of a matrix with Householder transformations is shown in Algorithm \ref{Alg:Hessenberg}.
Although the Hessenberg form exists for any square matrix, this algorithm only works for full-rank square matrices.
Notice that this algorithm is very similar to Algorithm \ref{Alg:Householder}.

\begin{algorithm}
\caption{Reduction to Hessenberg form for a nonsingular matrix.
This algorithm returns orthogonal $Q$ and upper Hessenberg $H$ such that $A = Q^THQ$.}
\label{Alg:Hessenberg}
\begin{algorithmic}[1]
\Procedure{Hessenberg}{$A$}
\State $m, n \gets \text{shape}(A)$
\State $H \gets \text{copy}(A)$
\State $Q \gets I_m$
\For{$0 \leq k < n-2$}
    \State $u_k \gets \text{copy}\left(H_{k+1:, k}\right)$
    \State $u_{k_0} \gets u_{k_0} + \text{sign}(u_{k_0}) \norm{u_k}$
    \State $u_k \gets u_k/\norm{u_k}$
    \State $H_{k+1:,k:} \gets H_{k+1:,k:} - 2u_k(u_k^\mathsf{T} H_{k+1:,k:})$
    \State $H_{:,k+1:} \gets H_{:,k+1:} - 2(H_{:,k+1:} u_k) u_k^\mathsf{T}$
    \State $Q_{k+1:} \gets Q_{k+1:} - 2u_k(u_k^\mathsf{T} Q_{k+1:})$
\EndFor
\State \pseudoli{return} $Q, H$
\EndProcedure
\end{algorithmic}
\end{algorithm}

When $A$ is symmetric, its upper Hessenberg form is a tridiagonal matrix.
This is because the $Q_i$'s zero out everything below the first subdiagonal of $A$ and the $Q_i^T$'s zero out everything above the first superdiagonal.
Thus, the Hessenberg form of a symmetric matrix is especially useful, since as we saw in Lab \ref{lab:complexity}, tridiagonal matrices make computations fast.

%TODO: This doesn't work if $A$ is singular
\begin{problem}
\label{prob:hessenberg}
Write a function that accepts as input a nonsingular square matrix $A$ and computes its Hessenberg form, returning orthogonal $Q$ and upper Hessenberg $H$ satisfying $A = Q^THQ$.
Your function should use Algorithm \ref{Alg:hessenberg}.
What happens when you compute the Hessenberg factorization of a symmetric matrix?
\end{problem}

%Sources: http://www.cs.unc.edu/~krishnas/eigen/node5.html
% http://en.wikipedia.org/wiki/Givens_rotation
%http://en.wikipedia.org/wiki/QR_decomposition
%	Note the Operation count: Householder is 2/3 n^3, MGS is 2 n^3
%http://en.wikipedia.org/wiki/QR_algorithm
%Applied Numerical methods using MATLAB by Yang has some code written for this
%http://www.math.kent.edu/~reichel/courses/intr.num.comp.2/lecture21/evmeth.pdf
%	These are eigenvalue algorithms explained carefully
%http://en.wikipedia.org/wiki/Householder_transformation
%Numerical Linear Algebra, by Lloyd N. Trefethen and David Bau III, Chapters 10 and 16


\subsection{Definitions}
\begin{enumerate}
	\item $f(x_0, y_0)$ is a local maximum of $f$ if for some $\delta > 0$, $f(x_0, y_0) \geq f(x, y) \forall (x,y) \in N((x,y),\delta)$.
		That is, you can draw a circle in the domain of $f$ centered at $(x_0, y_0)$ such that the value of $f$ at every point in the circle besides $(x_0, y_0)$ is less than $(x_0, y_0)$.
	\item $f(x_0, y_0)$ is a local minimum of $f$ if for some $\delta > 0$, $f(x_0, y_0) \leq f(x,y) \forall (x,y) \in N((x,y),\delta)$.
		That is, you can draw a circle in the domain of $f$ centered at $(x_0, y_0)$ such that the value of $f$ at every point in the circle besides $(x_0, y_0)$ is greater than $(x_0, y_0)$.
	\item $f(x_0, y_0)$ is a global max of $f$ if $f(x_0, y_0) \geq f(x,y) \forall (x,y) \in D(f)$ where $D(f)$ is the domain of $f$.
	\item $f(x_0, y_0)$ is a global min of $f$ if $f(x_0, y_0) \leq f(x,y) \forall (x,y) \in D(f)$ where $D(f)$ is the domain of $f$.
\end{enumerate}
\begin{theorem}
	If $(x_0, y_0)$ is in the domain of $f$ and a local extrema of $f(x,y)$, then $f_x(x_0, y_0)$ and $f_y(x_0, y_0)$ is either 0 or undefined.
\end{theorem}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\textwidth]{./Images/differentialMultivariableCalculus/optimization.png}
	\caption{Critical points appear when the partial derivatives are 0 or undefined.}
\end{figure}


\section*{Exercises}

\begin{ex}
  Let $A = \begin{mymatrix}{rr}
    1 & 2 \\
    2 & 1
  \end{mymatrix}$. Find $A^{10}$ by diagonalization.
  \begin{sol}
    First we write $A = PDP^{-1}$.
    \begin{equation*}
      \def\arraystretch{1.3}
      \begin{mymatrix}{rr}
        1 & 2 \\
        2 & 1
      \end{mymatrix}
      =
      \begin{mymatrix}{rr}
        -1 & 1 \\
        1 & 1
      \end{mymatrix}
      \begin{mymatrix}{rr}
        -1 & 0 \\
        0 & 3
      \end{mymatrix}
      \begin{mymatrix}{rr}
        -\frac{1}{2} & \frac{1}{2} \\
        \frac{1}{2} & \frac{1}{2}
      \end{mymatrix}
    \end{equation*}
    Therefore $A^{10} = PD^{10}P^{-1}$.
    \begin{equation*}
      \def\arraystretch{1.3}
      \begin{mymatrix}{rr}
        1 & 2 \\
        2 & 1
      \end{mymatrix}^{10}
      ~=~
      \begin{mymatrix}{rr}
        -1 & 1 \\
        1 & 1
      \end{mymatrix}
      \begin{mymatrix}{rr}
        (-1)^{10} & 0 \\
        0 & 3^{10}
      \end{mymatrix}
      \begin{mymatrix}{rr}
        -\frac{1}{2} & \frac{1}{2} \\
        \frac{1}{2} & \frac{1}{2}
      \end{mymatrix} \\
      ~=~
      \frac{1}{2}\begin{mymatrix}{cc}
        3^{10}+1 & 3^{10}-1 \\
        3^{10}-1 & 3^{10}+1
      \end{mymatrix}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rrr}
    1 & 4 & 1 \\
    0 & 2 & 5 \\
    0 & 0 & 5
  \end{mymatrix}$. Find $A^{50}$ by diagonalization.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rrr}
    1 & -2 & -1 \\
    2 & -1 & 1 \\
    -2 & 3 & 1
  \end{mymatrix}$. Find $A^{100}$ by diagonalization.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rr}
    -5 & -6 \\
    9 & 10 \\
  \end{mymatrix}$. Find a square root of $A$, i.e., find a matrix $B$
  such that $B^2=A$.
  \begin{sol}
    $B=\begin{mymatrix}{rr}
      -1 & -2 \\
      3  & 4 \\
  \end{mymatrix}$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rrr}
    -2 & 0 & 6 \\
    -3 & 1 & 6 \\
    -3 & 0 & 7 \\
  \end{mymatrix}$. Find a square root of $A$.
  \begin{sol}
    $A = \begin{mymatrix}{rrr}
      0  &  0 & 2 \\
      -1 &  1 & 2 \\
      -1 &  0 & 3 \\
    \end{mymatrix}$.
  \end{sol}
\end{ex}


\section{Scalar multiplication}

\begin{outcome}
  \begin{enumerate}
  \item Multiply a scalar by a vector algebraically and geometrically.
  \item Use the laws of scalar multiplication to prove equalities
    between vector expressions.
  \end{enumerate}
\end{outcome}

Scalar multiplication of vectors in $\R^n$ is defined as
follows.

\begin{definition}{Scalar multiplication of vectors in $\R^n$}{vector-scalar-multiplication}
  If $k\in\R$ is a scalar and $\vect{u}\in \R^n$ is a vector, then
  their \textbf{scalar multiplication}%
  \index{vector!scalar multiplication}%
  \index{scalar multiplication!of a vector} $k\vect{u}\in \R^n$ is
  defined by
  \begin{equation*}
    k\vect{u}=k\begin{mymatrix}{c}
      u_1 \\
      \vdots \\
      u_n
    \end{mymatrix} = \begin{mymatrix}{c}
      ku_1 \\
      \vdots \\
      ku_n
    \end{mymatrix}.
  \end{equation*}
\end{definition}

For example $3 \mat{1, 2, 3}^T = \mat{3, 6, 9}^T$ and
$-2\mat{1, 2, 3}^T = \mat{-2, -4, -6}^T$.

\begin{example}{Geometric meaning of scalar multiplication}{geometric-scalar-multiplication}
  Let $\vect{u}=\mat{2,1}^T$, and draw the following vectors to scale:
  $2\vect{u}$, $\vect{u}$, $\frac{1}{2}\vect{u}$, $0\vect{u}$,
  $-\frac{1}{2}\vect{u}$, $-\vect{u}$, and $-2\vect{u}$.  What is the
  geometric meaning of scalar multiplication?
\end{example}

\begin{solution}
  Here is a picture of the seven vectors. We draw their tails in
  different places to make their relationship easier to see.
  \begin{center}
    \begin{tikzpicture}[scale=0.7]
      \draw[->, thick, blue] (-10.5,-1)    -- node[above]{$2\vect{u}$} +(4,2);
      \draw[->, thick, blue] (-6,-0.5)  -- node[above]{$\vect{u}$} +(2,1);
      \draw[->, thick, blue] (-3,-0.25) -- node[above]{$\frac{1}{2}\vect{u}$} +(1,0.5);
      \draw[fill, blue](0,0) circle [radius=3pt] node[above]{$0\vect{u}$};
      \draw[->, thick, blue] (3,0.25)   -- node[above]{$-\frac{1}{2}\vect{u}$} +(-1,-0.5);
      \draw[->, thick, blue] (6,0.5)    -- node[above]{$-\vect{u}$} +(-2,-1);
      \draw[->, thick, blue] (10.5,1)      -- node[above]{$-2\vect{u}$} +(-4,-2);
    \end{tikzpicture}
  \end{center}
  We see that the vector $k\vect{u}$ has the same direction as
  $\vect{u}$ when $k$ is positive, and the opposite direction when $k$
  is negative. Further, the length of the vector is scaled by a factor
  of $\abs{k}$. It increases if $\abs{k}>1$ and decreases if
  $\abs{k}<1$. For example, the vector $2\vect{u}$ is exactly twice as
  long as $\vect{u}$.  (It is because of this scaling property that
  scalars are called scalars%
  \index{scalar!for scaling}).
\end{solution}

Just as with addition, scalar multiplication of vectors satisfies
several important properties. These are outlined in the following
proposition.

\begin{proposition}{Properties of scalar multiplication}{vector-scalar-multiplication}
  The following properties hold for vectors
  $\vect{u},\vect{v}\in\R^n$ and $k,\ell$ scalars.%
  \index{vector!properties of scalar multiplication}%
  \index{vector!scalar multiplication!properties}%
  \index{properties of scalar multiplication!vector}%
  \begin{itemize}
  \item The distributive law over vector addition
    \index{distributive law!over vector addition}%
    \index{vector!distributive law}%
    \begin{equation*}
      k(\vect{u} + \vect{v}) = k\vect{u} + k\vect{v}.
    \end{equation*}
  \item The distributive law over scalar addition
    \index{distributive law!over scalar addition}%
    \begin{equation*}
      (k + \ell) \vect{u} = k\vect{u} + \ell\vect{u}.
    \end{equation*}
  \item The associative law for scalar multiplication
    \index{associative law!of scalar multiplication}%
    \index{vector!associative law of scalar multiplication}%
    \begin{equation*}
      k(\ell\vect{u}) = (k\ell)\vect{u}.
    \end{equation*}
  \item The rule for multiplication by $1$
    \index{rule for multiplication by 1}%
    \index{vector!rule for multiplication by 1}%
    \begin{equation*}
      1\vect{u}=\vect{u}.
    \end{equation*}
  \end{itemize}
\end{proposition}

\begin{proof}
  We will show the proof of:
  \begin{equation*}
    k(\vect{u} + \vect{v}) = k\vect{u} + k\vect{v}.
  \end{equation*}
  Assume $\vect{u}=\mat{u_1,\ldots,u_n}^T$ and
  $\vect{v}=\mat{v_1,\ldots,v_n}^T$. We have:
  \begin{equation*}
    \begin{array}{ll}
      k(\vect{u} + \vect{v}) & = k\mat{u_1 + v_1,\ldots,u_n + v_n}^T \\
                             & = \mat{k(u_1 + v_1), \ldots, k(u_n + v_n)}^T \\
                             & = \mat{ku_1 + k v_1, \ldots, ku_n + kv_n}^T \\
                             & = \mat{ku_1, \ldots, ku_n}^T + \mat{kv_1, \ldots, kv_n}^T \\
                             & = k\vect{u} + k\vect{v}.
    \end{array}
  \end{equation*}
\end{proof}


%
% Chapter 3.8
%

\section*{3.8 Newton's Method}

Newton's method (Newton-Raphson method) is a way for finding successively better approximations for roots of real-valued functions. The idea behind Newton's method is that the tangent line is close to the curve and so its \(x\)-intercept is close to the \(x\)-intercept of the curve (namely, root \(r\) that we are seeking).

\subsection*{Using Newton's Method}

To find the formula for \(x_2\), we use the fact that the slope of \(L\) is \(f'(x_1)\), so its equation is
$$ y-f(x_1)=f'(x_1)(x-x_1) $$
Since the \(x\)-intercept of \(L\) is \(x_2\), we know that the point \((x_2, 0)\) is on the line, and so
$$ 0-f(x_1)=f'(x_1)(x_2-x_1) $$
If \(f'(x_1) \neq 0\), we can solve this equation for \(x_2\):
We use \(x_2\) as the second approximation to \(r\).
Repeat this procedure with \(x_1\) being replaced by the second approximation \(x_2\) giving \(x_3\) and so on.
\\\\\\
In general if the \(n\)th approximation is \(x_n\) and \(f'(x) \neq 0\), then the next approximation is given by
$$ x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)} $$
If the numbers \(x_n\) become closer and closer to \(r\) as \(n\) becomes large, then we say that the sequence converges to \(r\) and we write
$$ \lim_{n \to \infty}x_n=r $$


\chapter{Complex numbers}
\label{app:complex}

Throughout history, mankind has invented more and more complicated
number systems in an effort to make algebra easier.
\begin{itemize}
\item In the beginning, there were the \textbf{natural numbers}%
  \index{natural number}%
  \index{number!natural} $\N = \set{1, 2, 3, \ldots}$. However,
  after a while, it became a problem that certain equations, such as
  $x+5=3$, do not have a solution in the natural numbers.
\item To solve this problem, {\em zero} and {\em negative numbers}
  were invented, resulting in the set of \textbf{integers}%
  \index{integer}%
  \index{number!integer} $\Z=\set{\ldots,-3,-2,-1,0,1,2,3,\ldots}$. In
  the integers, the equation $x+5=3$ has a solution, namely
  $x=-2$. But some other equations, such as $2x=1$, still do not have
  a solution in the integers.
\item To solve this problem, the \textbf{rational numbers}%
  \index{rational number}%
  \index{number!rational} $\Q$ were invented. In the rational
  numbers, the equation $2x=1$ has a solution, namely
  $x=\frac{1}{2}$. However, some other equations, such as $x^2=2$,
  still do not have a solution in the rational numbers.
\item To solve this problem, the \textbf{real numbers}%
  \index{real number}%
  \index{number!real} $\R$ were invented. In the real numbers,
  the equation $x^2=2$ has a solution, namely $x=\sqrt{2}$. However,
  some other equations, such as $x^2=-1$, still don't have a solution
  in the real numbers.
\item To solve this problem, the \textbf{complex numbers}%
  \index{complex number}%
  \index{number!complex|see{complex number}} were invented.
\end{itemize}
The purpose of this section is to summarize the most important facts
about the complex numbers. We will also see that the above process
does not continue. In the complex numbers, all non-trivial polynomial
equations have a solution, and therefore, no additional numbers are
``missing''.  This property is called the \textbf{fundamental theorem
  of algebra}%
\index{fundamental theorem of algebra}%
\index{algebra!fundamental theorem of}. Gauss is usually credited with
giving a proof of this theorem in 1797 but many others worked on it
and the first completely correct proof was due to Argand in 1806.


    \documentclass{article}
    \usepackage[utf8]{inputenc}
    \usepackage{amsmath}
    \begin{document}
 %Title
 \section{Collatz Sequence for \(n=25\) }
    %Collatz Function
    \[
        f(n)=
        \begin{cases}
        \frac{n}{2}, & n \mod 2=0
        \\
        3n+1, &n \mod 2=1
        \end{cases} \\
    \]
    % Path for given N
    \(\textbf{Path for f(25)}\\[3mm]f(n), n=25
 \\
 \Rightarrow 3(25) + 1
 \\
 \Rightarrow n=76
 \\[3mm]
f(n), n=76
 \\
 \Rightarrow \frac{76}{2}
 \\
 \Rightarrow n=38
 \\[3mm]
f(n), n=38
 \\
 \Rightarrow \frac{38}{2}
 \\
 \Rightarrow n=19
 \\[3mm]
f(n), n=19
 \\
 \Rightarrow 3(19) + 1
 \\
 \Rightarrow n=58
 \\[3mm]
f(n), n=58
 \\
 \Rightarrow \frac{58}{2}
 \\
 \Rightarrow n=29
 \\[3mm]
f(n), n=29
 \\
 \Rightarrow 3(29) + 1
 \\
 \Rightarrow n=88
 \\[3mm]
f(n), n=88
 \\
 \Rightarrow \frac{88}{2}
 \\
 \Rightarrow n=44
 \\[3mm]
f(n), n=44
 \\
 \Rightarrow \frac{44}{2}
 \\
 \Rightarrow n=22
 \\[3mm]
f(n), n=22
 \\
 \Rightarrow \frac{22}{2}
 \\
 \Rightarrow n=11
 \\[3mm]
f(n), n=11
 \\
 \Rightarrow 3(11) + 1
 \\
 \Rightarrow n=34
 \\[3mm]
f(n), n=34
 \\
 \Rightarrow \frac{34}{2}
 \\
 \Rightarrow n=17
 \\[3mm]
f(n), n=17
 \\
 \Rightarrow 3(17) + 1
 \\
 \Rightarrow n=52
 \\[3mm]
f(n), n=52
 \\
 \Rightarrow \frac{52}{2}
 \\
 \Rightarrow n=26
 \\[3mm]
f(n), n=26
 \\
 \Rightarrow \frac{26}{2}
 \\
 \Rightarrow n=13
 \\[3mm]
f(n), n=13
 \\
 \Rightarrow 3(13) + 1
 \\
 \Rightarrow n=40
 \\[3mm]
f(n), n=40
 \\
 \Rightarrow \frac{40}{2}
 \\
 \Rightarrow n=20
 \\[3mm]
f(n), n=20
 \\
 \Rightarrow \frac{20}{2}
 \\
 \Rightarrow n=10
 \\[3mm]
f(n), n=10
 \\
 \Rightarrow \frac{10}{2}
 \\
 \Rightarrow n=5
 \\[3mm]
f(n), n=5
 \\
 \Rightarrow 3(5) + 1
 \\
 \Rightarrow n=16
 \\[3mm]
f(n), n=16
 \\
 \Rightarrow \frac{16}{2}
 \\
 \Rightarrow n=8
 \\[3mm]
f(n), n=8
 \\
 \Rightarrow \frac{8}{2}
 \\
 \Rightarrow n=4
 \\[3mm]
f(n), n=4
 \\
 \Rightarrow \frac{4}{2}
 \\
 \Rightarrow n=2
 \\[3mm]
f(n), n=2
 \\
 \Rightarrow \frac{2}{2}
 \\
 \Rightarrow n=1
 \\[3mm]

    \section{Credits}
    %Maybe add link to gitrepo and other shit, idk
    This is created using Collatzer (https://github.com/Z1aaan/Collatzer).
    Created By: Z1aaan

    README:
    A program created to visualize and simulate a user-given value for \textit{N}
    and see what happens when it is put under the Collatz function.
    \end{document}