\chapter{Basic Operations} \label{chap:operations}

The basic types have been introduced in the previous chapter, so we move on with the description of the basic BLAS operations.
Almost all operations supported by {\ublas} are available, including element-wise operations on vectors. Thus, consider the
\href{http://www.boost.org/doc/libs/1_52_0/libs/numeric/ublas/doc/operations_overview.htm}{ublas-documentation} as a reference as well.

\section{Vector-Vector Operations (BLAS Level 1)}

{\ViennaCL} provides all vector-vector operations defined at level 1 of BLAS. Tab.~\ref{tab:blas-level-1} shows how these operations can be carried
out in \ViennaCL. The function interface is compatible with {\ublas},
thus allowing quick code migration for {\ublas} users.

\TIP{For full details on level 1 functions, refer to the reference documentation
located in \texttt{doc/doxygen/}}

\begin{table}[tb]
\begin{center}
\begin{tabular}{l|l|p{6cm}}
Verbal & Mathematics & ViennaCL\\
\hline
swap    & $x \leftrightarrow y$ & \lstinline|swap(x,y);| \\
stretch    & $x \leftarrow \alpha x$ & \lstinline|x *= alpha;| \\
assignment & $y \leftarrow x$ & \lstinline|y = x;| \\
multiply add & $y \leftarrow \alpha x + y$ & \lstinline|y += alpha * x;| \\
multiply subtract & $y \leftarrow \alpha x - y$ & \lstinline|y -= alpha * x;| \\
elementwise product & $y_i \leftarrow x_i \cdot z_i$ & \lstinline|y = element_prod(x,z);| \\
elementwise division & $y_i \leftarrow x_i \cdot z_i$ & \lstinline|y = element_div(x,z);| \\
inner dot product & $\alpha \leftarrow x^{\mathrm{T}} y$ & \lstinline|inner_prod(x,y);| \\
$L^1$ norm & $\alpha \leftarrow \Vert x \Vert_1$ & \lstinline|alpha = norm_1(x);| \\
$L^2$ norm & $\alpha \leftarrow \Vert x \Vert_2$ & \lstinline|alpha = norm_2(x);| \\
$L^\infty$ norm & $\alpha \leftarrow \Vert x \Vert_\infty$ & \lstinline|alpha = norm_inf(x);| \\
$L^\infty$ norm index& $i \leftarrow \max_i \vert x_i \vert$ & \lstinline|i = index_norm_inf(x);| \\
plane rotation & $(x,y) \leftarrow (\alpha x + \beta y, -\beta x + \alpha y)$ & \lstinline|plane_rotation(alpha, beta, x, y);| \\
\end{tabular}
\caption{BLAS level 1 routines mapped to {\ViennaCL}. Note that the free functions reside in namespace \texttt{viennacl::linalg}}
\label{tab:blas-level-1}
\end{center}
\end{table}

\section{Matrix-Vector Operations (BLAS Level 2)}
The interface for level 2 BLAS functions in {\ViennaCL} is similar to that of
{\ublas} and shown in Tab.~\ref{tab:blas-level-2}.

\TIP{For full details on level 2 functions, refer to the reference documentation
located in \texttt{doc/doxygen/}}

\begin{table}[tb]
\begin{center}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{p{4cm}|l|p{7cm}}
Verbal & Mathematics & ViennaCL\\
\hline
matrix vector product & $y \leftarrow A x$ & \lstinline|y = prod(A, x);| \\
matrix vector product & $y \leftarrow A^\mathrm{T} x$ & \lstinline|y = prod(trans(A), x);| \\
inplace mv product & $x \leftarrow A x$ & \lstinline|x = prod(A, x);| \\
inplace mv product & $x \leftarrow A^\mathrm{T} x$ & \lstinline|x = prod(trans(A), x);| \\
\hline
scaled product add & $y \leftarrow \alpha A x + \beta y$ & \lstinline|y = alpha * prod(A, x) + beta * y| \\
scaled product add & $y \leftarrow \alpha A^{\mathrm T} x + \beta y$ & \lstinline|y = alpha * prod(trans(A), x) + beta * y| \\
\hline
tri. matrix solve & $y \leftarrow A^{-1} x$ & \lstinline|y = solve(A, x, tag);| \\
tri. matrix solve & $y \leftarrow A^\mathrm{T^{-1}} x$ & \lstinline|y = solve(trans(A), x, tag);| \\
inplace solve & $x \leftarrow A^{-1} x$ & \lstinline|inplace_solve(A, x, tag);| \\
inplace solve & $x \leftarrow A^\mathrm{T^{-1}} x$ & \lstinline|inplace_solve(trans(A), x, tag);| \\
\hline
rank 1 update & $A \leftarrow \alpha x y^{\mathrm T} + A$ & \lstinline|A += alpha * outer_prod(x,y);| \\
symm. rank 1 update & $A \leftarrow \alpha x x^{\mathrm T} + A$ & \lstinline|A += alpha * outer_prod(x,x);| \\
rank 2 update & $A \leftarrow \alpha (x y^{\mathrm T} + y x^{\mathrm T}) + A$ & \lstinline|A += alpha * outer_prod(x,y);| \lstinline|A += alpha * outer_prod(y,x);| \\
\end{tabular}
\caption{BLAS level 2 routines mapped to \ViennaCL. Note that the free functions reside in namespace \texttt{viennacl::linalg}. \lstinline|tag| is one out of \lstinline|lower_tag|, \lstinline|unit_lower_tag|, \lstinline|upper_tag|, and \lstinline|unit_upper_tag|.}
\label{tab:blas-level-2}
\end{center}
\end{table}

\section{Matrix-Matrix Operations (BLAS Level 3)}
Full BLAS level 3 support is since {\ViennaCL} 1.1.0, cf.~Tab.~\ref{tab:blas-level-3}. While BLAS
levels 1 and 2 are mostly memory-bandwidth-limited, BLAS level 3 is mostly
limited by the available computational power of the respective device. Hence,
matrix-matrix products regularly show impressive performance gains on mid-
to high-end GPUs when compared to a single CPU core.

Again, the {\ViennaCL} API is identical to that of {\ublas} and comparisons can
be carried out immediately, as is shown in the tutorial located in
\texttt{examples/tutorial/blas3.cpp}.

As for performance, {\ViennaCL} yields decent performance gains at BLAS level
3 on mid- to high-end GPUs compared to CPU implementations using a single core
only. Nevertheless, one must not expect to obtain the reported peak performance
of hundreds of GFLOPs for the multiplication of arbitrary matrices. These rates
can typically only obtained when tailoring the compute kernel(s) to a
particular device and certain matrix dimensions, while {\ViennaCL} provides
kernels that represent a good compromise between efficiency and portability
among a large number of different devices and device types.

\begin{table}[tb]
\begin{center}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{p{4cm}|l|p{7.5cm}}
Verbal & Mathematics & ViennaCL\\
\hline
matrix-matrix product & $C \leftarrow A \times B$ & \lstinline|C = prod(A, B);| \\
matrix-matrix product & $C \leftarrow A \times B^\mathrm{T}$ & \lstinline|C = prod(A, trans(B));| \\
matrix-matrix product & $C \leftarrow A^\mathrm{T} \times B$ & \lstinline|C = prod(trans(A), B);| \\
matrix-matrix product & $C \leftarrow A^\mathrm{T} \times B^\mathrm{T}$ & \lstinline|C = prod(trans(A), trans(B));| \\
\hline
tri. matrix solve & $C \leftarrow A^{-1} B$ & \lstinline|C = solve(A, B, tag);| \\
tri. matrix solve & $C \leftarrow A^\mathrm{T^{-1}} B$ & \lstinline|C = solve(trans(A), B, tag);| \\
tri. matrix solve & $C \leftarrow A^{-1} B^\mathrm{T}$ & \lstinline|C = solve(A, trans(B), tag);| \\
tri. matrix solve & $C \leftarrow A^\mathrm{T^{-1}} B^\mathrm{T}$ & \lstinline|C = solve(trans(A), trans(B), tag);| \\
%
inplace solve & $B \leftarrow A^{-1} B$ & \lstinline|inplace\_solve(A, trans(B), tag);| \\
inplace solve & $B \leftarrow A^\mathrm{T^{-1}} B$ & \lstinline|inplace\_solve(trans(A), x, tag);| \\
inplace solve & $B \leftarrow A^{-1} B^\mathrm{T}$ & \lstinline|inplace\_solve(A, trans(B), tag);| \\
inplace solve & $B \leftarrow A^\mathrm{T^{-1}} B^\mathrm{T}$ & \lstinline|inplace\_solve(trans(A), x, tag);| \\
\end{tabular}
\caption{BLAS level 3 routines mapped to \ViennaCL. Note that the free functions
reside in namespace \texttt{viennacl::linalg}}
\label{tab:blas-level-3}
\end{center}
\end{table}

\section{Initializer Types}

\NOTE{Initializer types in {\ViennaCLversion} can only be used for initializing vectors and matrices, not for computations!}

In order to initialize vectors, the following initializer types are provided, again similar to {\ublas}:
\begin{center}
\begin{tabular}{|l|p{10cm}|}
 \hline
 \lstinline|unit_vector<T>(s, i)| & Unit vector of size $s$ with entry $1$ at index $i$, zero elsewhere. \\
 \hline
 \lstinline|zero_vector<T>(s)| & Vector of size $s$ with all entries being zero. \\
 \hline
 \lstinline|scalar_vector<T>(s, v)| & Vector of size $s$ with all entries equal to $v$. \\
 \hline
\end{tabular}
\end{center}
For example, to initialize a vector \lstinline|v1| with all $42$ entries being $42.0$, use
\begin{lstlisting}
 viennacl::vector<float> v1 = viennacl::scalar_vector<float>(42, 42.0f);
\end{lstlisting}

Similarly the following initializer types are available for matrices:
\begin{center}
\begin{tabular}{|l|p{10cm}|}
 \hline
 \lstinline|identity_matrix<T>(s, i)| & Identity matrix of dimension $s \times s$. \\
 \hline
 \lstinline|zero_matrix<T>(s1, s2)| & Matrix of size $s_1 \times s_2$ with all entries being zero. \\
 \hline
 \lstinline|scalar_matrix<T>(s1, s2, v)| & Matrix of size $s_1 \times s_2$ with all entries equal to $v$. \\
 \hline
\end{tabular}
\end{center}


%!TEX root =  ../main.tex

\mychapters{Regressions}{regression}{\chapdir/pics/CMS_higgs-event}
Many situations in science are about taking actual fuzzy numbers and attempting to
derive equations from them.    Sometimes these numbers are completely obvious,
practically screaming what kind of equation they come from.  On the other hand,
it can be deeply obscure and require enough care to regress from numerical
to algebraic information.

But the consequences can be high.  Later generations may attempt to take your
equation and extrapolate well beyond the bounds you ever intended.  If you have
the wrong model, results can be disastrous, or at least, very upsetting (like the change
from Newtonian to Einsteinian physics).

\newpage
\chapterminitoc

%									13 - 1
\newpage
\invisiblesection{Graphical Patterns}
\subsection{Problems}
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch13/1301p.pdf}}
\newpage
\subsection{Increasing/Decreasing}
\subsection{Concavity}
\subsection{Asymptotes}
\subsection{Origin Question}
\subsection{Repeating}
\subsection{End Behavior}
\newpage
\subsection{Exercises}

%									13 - 2
\newpage
\invisiblesection{Numerical Patterns}
\subsection{Problems}
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch13/1302p.pdf}}
\newpage
\subsection{Add-Add}
\subsection{Add-Multiply}
logs are multiply add
\subsection{Multiply-Multiply}
roots are divide divide
\subsection{Second Difference}
cubes are add third diff.

periodic the same data reoccurs at a constant interval
\newpage
\subsection{Exercises}

%									13 - 3
\newpage
\invisiblesection{Error}
\subsection{Problems}
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch13/1303p.pdf}}
\newpage
\subsection{SSres}
\subsection{SSdev}
\subsection{r vs $r^2$}
\newpage
\subsection{Exercises}

%									13 - 4
\newpage
\invisiblesection{Linearization}
\subsection{Problems}
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch13/1304p.pdf}}
\newpage
\subsection{Who gets the log?}
\subsubsection{Lin-Log Graphs}
\subsubsection{Log-Log Graphs}
\newpage
\subsection{Exercises}

%									13 - 5
\newpage
\invisiblesection{Differential Equations}
\subsection{Problems}
\noindent\makebox[\textwidth]{\includegraphics[width=\paperwidth]{ch13/1305p.pdf}}
\newpage
\subsection{Slope Fields}
\subsection{Euler's Method}
\subsection{Logistic Curves}
\newpage
\subsection{Exercises}

%									13 - 6
\newpage
\section{Review}
\subsection{Chapter Review}
\subsection{Chapter Test}


%Note for this test case:
%Simple water flow example using ANUGA: Water flowing down a channel.
%It was called "steep_slope" in an old validation test.

\section{Shallow flow down a mild slope with coarsely resolved topography}
This case simulates very shallow flow running down a mild slope, with coarsely resolved topography, and a very low discharge. It represents a numerically more challenging idealisation of the rainfall-runoff problem, as compared to the case with higher discharge and resolution. This case has an analytical solution, and in particular, we consider the steady-uniform solution with the values of bed slope and friction slope are the same.

Suppose that we are given a one dimensional domain. The steady state conditions with a contant water depth everywhere make the shallow water equations to the single identity
\begin{equation}
z_x = - S_f.
\end{equation}
Here $q=uh$ is the momentum or water discharge and $S_f$ is the symbol for the force of bottom friction involving Manning's coefficient $n$. We take
\begin{equation}
S_f = n^2 \frac{q|q|}{h^{10/3}}.
\end{equation}
If $q$, $n$, and $z_x$ are given, then the analytical solution is
\begin{equation}
u(x)= \left[- n^{-2} q^{4/3} z_x\right]^{3/10},
\end{equation}
\begin{equation}
h(x)= \frac{q}{u}\,.
\end{equation}

\subsection{Results}
For our test, we consider a square dimensional domain with length and width 100. We take $q=1/1000.$, $n=0.03$, and $z_x=-0.1$.
The topography is
\begin{equation}
z(x, y)= -0.1 x\,.
\end{equation}
The initial condition is $u=v=0$ and
\begin{equation}
w(x,y,0)= -0.1 x + 0.01\,.
\end{equation}

Some simulation results are as follows.
Figures~\ref{fig:depthdownchan} shows the steady state depth in the downstream direction. There should be a good agreement with the analytical solution in the middle of the domain (the boundaries will not reach steady-uniform flow).
Figures~\ref{fig:xvelscrosschan} and~\ref{fig:yvelscroschan} show the steady state $x$- and $y$-velocities, along a slice in the cross slope direction (near $x=50$). In the middle of the domain the $x$-velocities should agree well with the analytical solution, and the $y$-velocities should be zero.

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{depth_x.png}
\caption{Depth in the downstream direction}
\label{fig:depthdownchan}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{xvelocity_x.png}
\caption{X velocity in the downstream direction}
\label{fig:xveldownchan}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{depth_y.png}
\caption{Depth in the downstream direction}
\label{fig:depthacrosschan}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{x_velocity.png}
\caption{$x$-velocity along the cross-section $x=50$ (i.e. a cross-section with constant bed elevation)}
\label{fig:xvelscrosschan}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.8\textwidth]{y_velocity.png}
\caption{$y$-velocity along the cross-section $x=50$ (i.e. a cross-section with constant bed elevation)}
\label{fig:yvelscroschan}
\end{center}
\end{figure}

\endinput


\section{Scale invariance of balances}
\begin{align}
\log \frac{\prod_{x_j \in i_L}{(x_j)^{1/|i_L|}}}{\prod_{x_k \in i_R}{(x_j)^{1/|i_R|}}} =
\log \frac{\prod_{p_j \in i_L}{(\textcolor{red}{n}p_j)^{1/|i_L|}}}{\prod_{p_k \in i_R}{(\textcolor{red}{n}p_j)^{1/|i_R|}}}=
\log \frac{\prod_{p_j \in i_L}{(p_j)^{1/|i_L|}}}{\prod_{p_k \in i_R}{(p_j)^{1/|i_R|}}}
\end{align}
$n$  is the true sequencing count, $x_j$ is the true abundances of species $j$ and $p_j$ is the proportion of species $j$. $i_L$ is the set of all species proportions contained in the left sub-tree at internal node $i$, $i_R$ is the set of all species proportions contained in the right sub-tree at the internal node $i$, and, $g(x)$ is the geometric mean of all of the proportions contained in $x$, $|i_R|$ is the number of species contained  $i_R$ and  $|i_L|$ is the number of species contained  $i_L$.  As shown above, the sequencing depth constant gets effectively canceled out.  Thus, log ratios are a natural normalization for sequencing depth, especially if there are no zero abundances present and the samples have sufficient coverage.

%
\section{Benchmark of compositional coherence}
\textbf{Supplemental Figure 1}

The simulation consisted of a uniform population of 1000 individuals.  A blooming was simulated across 9 time points, where a single organism eventually grew 100,000x fold.  At each time point, 30 compositions were simulated using multinomial sampling with replacement.  At each time point, a statistical test was performed comparing the sample at that time point to the original time point.  Since we know beforehand that only 1 species is changing, any other tests that don’t involve the first set of proportions that is determined to be significant with p-value $<$ 0.05 is a false positive Figure S1a-b).  In fact, if the bloom of a given species is high enough, all of the other individuals can be detected to change.  In Figure S1e, if 1 species has changed by 100,000x, then all of the pvalues will be less than 10-10, giving a false positive rate close to 100\%.  The same procedure is performed using balances, any balance that does not contain x1 that is determined to be significant from a t-test is considered a false positive (Figure S1c).  Note that this is highly dependent on the choice of the tree.  In this case, we used a tree where the blooming species x1 was to the far right of the tree (Figure S1f).  In this way, only the balance between x1, and x2 through x1000 should be changing.  But if we were to flip the tree, and place x1 to the far right of the tree, every balance in the tree will contain x1 (Figure S1g).  So as x1 blooms, the number of significant balances will increase (Figure S1d).

\begin{figure}[H]
        \centering
        \includegraphics[width=1\textwidth]{appendix_b/sup_figure1.png}
        \caption[A benchmark of statistical tests on compositional data.]{Simulations illustration the occurence of false positives in
          traditional statistical tests}
        \label{figbS1}
\end{figure}
While it may be deemed biologically irrelevant, blooms do happen frequently in microbial studies, with individual species sometimes blooming 5 orders of magnitude within a short period of time.  And the 10,000x fold growth of a single species will have the exact same effect as the 1,000x fold growth change of 10 species.  This suggests that there could be many subtle scenarios where we could be misinterpreting biologically relevant signals by testing individual proportions of microbes.\newpage
\textbf{Supplemental Figure 2}

\begin{figure}[H]
        \centering
        \includegraphics[width=1\textwidth]{appendix_b/sup_figure2.png}
        \caption[An ecological intrepretation of balances]{A simulation of 4 species, where each species is normally distributed along some environmental gradient.  Each species has a normal distribution with a variance of 3 and a mean of 3, 6, 9 and 12 respectively as shown in Figure S2a.  The resulting balances can be calculated as follows.}
        \label{figbS2}
\end{figure}
\[
abcd = \log \frac{\sqrt{ab}}{\sqrt{cd}} \qquad
ab = \log \frac{\sqrt{a}}{\sqrt{b}}
cd = \log \frac{\sqrt{c}}{\sqrt{d}}
\]
Note, it is not possible to take a logarithm of zero.  A commonly used approach around this problem is to add a pseudocount.  Here we add a pseudocount of 1 after multiplying all of the species probabilities by 10000.  These species abundances are transformed into balances as shown in Figure S2b.  Because of the zero phenomenon, the balances yield something reminiscent to a triangular wave when applied to a pair of unimodal distributions.  Take balance ab for example.  At the far left around -5, neither a or b are present, so both of their abundances are zero.  But since we are adding pseudocounts, the resulting balance is given by log(1/1)=0 . When the gradient value increases to 0, the abundance of a approaches the peak of the distribution, while the abundance of b is still zero, causing the ab balance to increase.  By the time the gradient value is around 4, the abundances of b starts to appear, causing the ab to peak.  When the gradient value is around 8, the abundance of a starts disappearing while the abundance of b starts approaching the maximum peak in Figure S2a.  At a gradient value of 10, the abundance of b also begins to dwindle, and the ab balance spirals towards zero.  This same triangular wave pattern appears in all 3 of these balances, and portions of this also appear in the 88 soils study as shown in Figure S3.

It is also important to note that a balance of zero also indicates that the abundances between the ratios are equal.  So if a balance is zero, and a pseudocount scheme was used, either the proportions between the numerator and denominator are truly equal, or both the numerator and the denominator are zero.\newpage
\section{Analysis of balances in soils}
\textbf{Supplemental Figure 3}

\begin{figure}[H]
        \centering
        \includegraphics[width=1\textwidth]{appendix_b/sup_figure3.png}
        \caption[Other balances from the 88 soils study.]{A perspective of different balances in the 88 soils study.}
        \label{figbS3}
\end{figure}

If there is truly a unimodal species distribution along pH, we’d expect to see the same sort of triangular wave pattern as shown in Figure 2S.  If this is the case, then the top balance y0 is likely to be resulting from the midsection of the triangular wave between the minimum and the maximum (Figure S3a).  The peaks of the triangular wave are a bit more apparent in the Figure S3b-d.  In Figure S3b, the lower subtree in y1 is probably reaching a maximum in the true abundance around a pH of 7.  In Figure S3c, the upper subtree in y2 is also likely approaching a maximum in the true abundance around a pH of 6.  The same sort pattern could be happening in Figure S3d with the lower subtree in y3.  These glimpses of triangular waves in these graphs suggest that there could be unimodel distributions of OTUs across the pH gradient.

As daunting as the zeros problem is, the zeros present in data sets such as the 88 soils follow predictable patterns.  Even with a simple pseudo count strategy, we can still extract sensible information about balances of microbes across different pH values.


\section{Damage Plasticty Model For Quasi-Brittle Materials}

Damage plasticity model based on
Strain can be decomposed into elastic and plastic components:
\begin{equation}
\label{eqn:const1}
\boldsymbol{\epsilon} = \boldsymbol{\epsilon}^{el} + \boldsymbol{\epsilon}^{pl}
\end{equation}

Taking the time derivative gives the decomposition of the strain rate, $\boldsymbol{\dot{\epsilon}}$ :
\begin{equation}
\label{eqn:const2}
\boldsymbol{\dot{\epsilon}} = \boldsymbol{\dot{\epsilon}}^{el} + \boldsymbol{\dot{\epsilon}}^{pl}
\end{equation}

The constitutive stress-strain relationship including a scalar damage parameter, $\mathbf{D}$ can be written as follows:
\begin{equation}
\label{eqn:const3}
\boldsymbol{\sigma} = (1-\mathbf{D})\mathbf{E}:\boldsymbol{\epsilon^{el}}
\end{equation}

For simplicity, the damaged elastic stiffness is described as the reduced stiffness due to the damage:
\begin{equation}
\label{eqn:const4}
\mathbf{E^d} = (1-\mathbf{D})\mathbf{E}
\end{equation}

Substituting \ref{eqn:const1} and \ref{eqn:const4} into \ref{eqn:const3} results in the following:
\begin{equation}
\label{eqn:const5}
\boldsymbol{\sigma} = \mathbf{E^d}:(\boldsymbol{\epsilon}-\boldsymbol{\epsilon}^{pl})
\end{equation}

Using the "usual notions of CDM" (find reference), the effective stress. $\boldsymbol{\bar{\sigma}}$, can be defined as:
\begin{equation}
\label{eqn:const6}
\boldsymbol{\bar{\sigma}} = \mathbf{E}:(\boldsymbol{\epsilon}-\boldsymbol{\epsilon}^{pl})
\end{equation}

Such that the cauchy stress tensor can be realted to the effective stress tensor as follows:
\begin{equation}
\label{eqn:const7}
\boldsymbol{\sigma} = (1-\mathbf{D})\boldsymbol{\bar{\sigma}}
\end{equation}

The nature of the damage evolution is assumed to be a function of the effective stress and the equivalent plastic strain, $\boldsymbol{\bar{\epsilon}^{pl}}$:
\begin{equation}
\label{eqn:const8}
\mathbf{D} = \mathbf{D}(\boldsymbol{\bar{\sigma}}, \boldsymbol{\bar{\epsilon}^{pl}})
\end{equation}

In this formulation, the brittle nature of rock neccessitates seperate characterization of tensile and compressive damage. In the case where a rock sample fails completely in tension, (i.e. the tensile stiffness becomes effectively 0), the compressive strength remains intacts to a fairly high degree such that two seperate damage variables for tensile damage and compressive damage. As such, the equivalent plastic strain is also considered seperately for tension and compression and is represented as follows:
\begin{equation}
\label{eqn:const9}
\boldsymbol{\bar{\epsilon}^{pl}} = \begin{bmatrix}
	\boldsymbol{\bar{\epsilon}_t^{pl}} \\
	\boldsymbol{\bar{\epsilon}_c^{pl}} \end{bmatrix}
\end{equation}

The evolution of the equivalent plastic strains are described by the time derivative of the equivalent plastic strain, which can be considered to be related to the time derivative of the plastic strain through a hardenbing rule, $\mathbf{h}$ such that:
\begin{equation}
\label{eqn:const10}
\boldsymbol{\dot{\bar{\epsilon}}^{pl}} = \mathbf{h}(\boldsymbol{\bar{\sigma}},
	\boldsymbol{\bar{\epsilon}^{pl}})\bullet \boldsymbol{\dot{\epsilon}}
\end{equation}

The flow rule can be written in terms of the flow potential function, $G(\boldsymbol{\bar{\sigma}})$, and a plastic mulitplier $\dot{\lambda}$:
\begin{equation}
\label{eqn:const11}
\boldsymbol{\dot{\epsilon}} = \dot{\lambda} \dfrac{\partial G(\boldsymbol{\bar{\sigma}})}{\partial \boldsymbol{\bar{\sigma}}}
\end{equation}

Non-associated plasticity  is used, which required the solution of non-symetric equations.


\subsection{Multiplicative Expression}

\begin{grammar}
\nonterminal{multiplicative-expression}
    \produces
	\nonterminal{power-expression} \\
    \produces
	\nonterminal{multiplicative-expression}
	\lexkeyword{*}
	\nonterminal{power-expression} \\
    \produces
	\nonterminal{multiplicative-expression}
	\lexkeyword{/}
	\nonterminal{power-expression} \\
\end{grammar}

\subsubsection{Eliminating left recursion (in BNF Form)}

Eliminating left recursion leads to
\begin{grammar}
\underbrace{\nonterminal{multiplicative-expression}}_{=: M}
    \produces
	\underbrace{\nonterminal{power-expression}}_{=: P}
	\underbrace{\nonterminal{multiplicative-expression'}}_{=:M'}\\
\nonterminal{multiplicative-expression'}
    \produces
	\lexkeyword{*}
	\nonterminal{power-expression} \nonterminal{multiplicative-expression'}
	\\
    \produces
	\lexkeyword{/}
	\nonterminal{power-expression} \nonterminal{multiplicative-expression'}
	\\
    \produces
\end{grammar}\\[-0.5cm]
\noindent

\paragraph{Item automata for M}

\[
\begin{tikzpicture}[
    every text node part/.style={align=center},
    initial text =
]
    \node[state,initial]
	(S)[rounded rectangle, draw]
	{{$[M\to .PM']$}};
    \node[state]
	(P)[rounded rectangle, draw,right=of S]
	{{$[M\to P.M']$}};
    \node[state,accepting]
	(M_)[rounded rectangle, draw,right=of P]
	{{$[M\to PM'.]$}};
    \path[->] (S) edge  node [above] {P} (P);
    \path[->] (P) edge  node [above] {M'} (M_);
\end{tikzpicture}
\]

\paragraph{Item automata for M'}
\[
\begin{tikzpicture}[
    every text node part/.style={align=center},
    initial text =
]
    \node[state,initial,accepting]
	(S)[rounded rectangle, draw]
	{
	    {$[M'\to .*PM']$}\\
	    {$[M'\to ./PM']$}\\
	    {$[M'\to .]$}
	};
    \node[state]
	(Mul)[rounded rectangle, draw,right=of S]
	{{$[M'\to *.PM']$}};
    \node[state]
	(Div)[rounded rectangle, draw,below=of Mul]
	{{$[M'\to /.PM']$}};
    \node[state]
	(P)[rounded rectangle, draw,right=of Mul]
	{
	    {$[M'\to *P.M']$} \\
	    {$[M'\to /P.M']$}
	};
    \node[state,accepting]
	(E)[rounded rectangle, draw,right=of P]
	{
	    {$[M'\to *PM'.]$} \\
	    {$[M'\to /PM'.]$}
	};
    \path[->] (S) edge  node [above] {\lexkeyword{*}} (Mul);
    \path[->] (S) edge  node [above] {\lexkeyword{/}} (Div);
    \path[->] (Mul) edge  node [above] {P} (P);
    \path[->] (Div) edge  node [above] {P} (P);
    \path[->] (P) edge  node [above] {M'} (E);
\end{tikzpicture}
\]

\subsubsection{Eliminating left recursion (in EBNF Form)}

Expressing the BNF notation in EBNF reveals how the item automata for $M$ and
$M'$ can be combined in an implementation using a while loop
checking if the
current token is in $\mathrm{First}(M') = \{ \lexkeyword{*}, \lexkeyword{/}\}$:
\begin{grammar}
\underbrace{\nonterminal{multiplicative-expression}}_{= M}
    \produces
	\underbrace{\nonterminal{power-expression}}_{= P}
	\{\;
	\underbrace{
	    \nonterminal{multiplicative-op}
	    \nonterminal{power-expression}
	    }_{=M'}
	\}
	\\
\nonterminal{multiplicative-op}
    \produces
    \lexkeyword{*} \\
    \produces
    \lexkeyword{/} \\
\end{grammar}\\[-0.5cm]


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[margin=1in]{geometry}
\usepackage{graphicx}
\usepackage[export]{adjustbox}
\usepackage{amsmath}
\usepackage[ruled,vlined]{algorithm2e}

\title{Machine Learning Overview}
\author{Hanchung Lee}
\date{February 2020}

\begin{document}

\maketitle

\section{Linear Regression}

\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    Linear Regression, in essence, is trying to fit a hyperplane through the data points that minimizes the sum of the distances of all the points to that hyperplane. That hyperplane is then used to predict the location of new data points. In other words, linear regression is a model $\emph{f} : \mathbb{R}^{d+1} \rightarrow \mathbb{R}$ for a given data pair $(x, y)$ where $x \in \mathbb{R}^{d+1}$ and $y \in \mathbb{R}$. It is if if the function \emph{f} is linear, with weights $w$ of the model is the parameters to be learned. Since regression is a mapping to $\mathbb{R}$, it can be used to estimate a value given a set of inputs $x$.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    Regression assumes that the output $y$ is linearly dependent on $x$. However, it will be linear dependent as long as $w$ is linear. It also assumes error term is a normally distributed random variable.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    We model linear regression problem $\mathbf{y} = \mathbf{X}\mathbf{w} + \mathbf{b}$ using both least squares (LS) or maximum likelihood (ML) solutions:
    $$\mathbf{LS}: \; arg \min_{w} \left\|y - \mathbf{X}w\right\|_{2}^{2}$$
    $$\mathbf{ML}: \; arg \max_{w} -\frac{1}{2\sigma}\left\|y - \mathbf{X}w\right\|_{2}^{2}$$
    Or, alternatively, solving it using vector calculus approach, we solve $\frac{\partial}{\partial{X}}\left\|\mathbf{X}\mathbf{W} + \mathbf{b}\right\|_{2}^{2} = 0$, and we get
    $$\mathbf{X} = (\mathbf{W}^T\mathbf{W})^{-1}\mathbf{W}^T\mathbf{b}$$

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    Regression can be done using loss functions that spans the $\mathbb{R}$ space. Mean square error and mean absolute error are the common cost function of choice. We used mean squared error for our formulation.\\
    \smallbreak
    Mean squared error (L2 error)  $\left\|y - y'\right\|_{2}^2 = (y - \mathbf{X}w)^2$ \\

    \item \textbf{Whare are the advantages/disadvantages?}
    \noindent
    \smallbreak
    Linear regression is one of the most fundamental machine learning model. It is easy to implement and train. The trained model is also highly explainable. The modeling framework can be extended by different tricks to make it more robust, such as feature engineering/extension/selection, kernel tricks, and adding regularization.

    On the flip side, linear regression only model linear relationships between dependent and independent variables (which can be polynomials), which is not enough for more complex relationships. It is also sensitive to anomalies and requires data normalization to prevent that. Also, it requires more samples than the number of parameters as a rule of thumb, which is common across most machine learning models.

\end{enumerate}

\section{Ridge Regression}

\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    Ridge regression is an extension of base regression with an \verb|L2 norm| regularization term $\mathbf{\lambda}\left\| \mathbf{W}\right\|_{2}^{2}$ where $\mathbf{\lambda}$ is the learning rate that adjusts the regularization of parameters $\mathbf{W_{i}}$. \cite{1} The $\mathbf{\lambda}$ term is also called the \emph{shrinkage}. And when $\mathbf{\lambda} = 0$, it is just regular linear regression.

    In other words, we use ridge regression to fix if a linear regressor has high variance to improve the out of the sample fit of the model. We can find the learning rate parameter $\lambda$ using cross validation.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    As with linear regression, Ridge regression assumes that the output $y$ is linearly dependent on $x$. However, it will be linear dependent as long as $w$ is linear. It also assumes error term is a normally distributed random variable.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    From the descriptions above, we can model ridge regression as linear regression problem with a regularization term.
    $$\mathbf{y} = \mathbf{X}\mathbf{W} + \mathbf{b} + \lambda\left\|\mathbf{W}\right\|_{2}^{2}$$
    Thus, the loss function using least squared solution would be:
    $$\mathbb{L}_{W_{ridge}} = \left\|\mathbf{y} - (mathbf{X}\mathbf{W} + \mathbf{b})\right\|_{2}^{2} + \lambda\left\|mathbf{W}\right\|_{2}^{2}$$
    Pulling in the bias term into W and solving it for \mathbf{W}, we get:
    $$\mathbf{W} = (\lambda\mathbb{I} + \mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}\mathbf{y},$$
    where the $\lambda$ term gets cancelled out as $\lambda$ goes to $\infty$.

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    Regression can be done using loss functions that spans the $\mathbb{R}$ space. Mean square error and mean absolute error are the common cost function of choice. We used mean squared error for our formulation.\\
    \smallbreak
    Mean squared error (L2 error)  $\left\|y - y'\right\|_{2}^2 = (y - \mathbf{X}w)^2$ \\
    \smallbreak

    \item \textbf{Whare are the advantages/disadvantages?}
    \noindent
    \smallbreak
    Ridge regression extends from Linear regression with a regularization term. Thus it shares similar properties as linear regression: easy to implement, easy to train. The trained model is also highly explainable. The modeling framework can be extended by different tricks to make it more robust, such as feature engineering/extension/selection.

    On the flip side, it also shares some of the problems with linear regression only model, where it expects linear relationships between dependent and independent variables (which can be polynomials), which is not enough for more complex relationships. It is also sensitive to anomalies and requires data normalization to prevent that. Also, it requires more samples than the number of parameters as a rule of thumb, which is common across most machine learning models.

\end{enumerate}

\section{Lasso Regression}

\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    Lasso regression is an extension of basic regression with an \verb|L1 norm| regularization term $\mathbf{\lambda}\left\| \mathbf{W}\right\|_{1}$ where $\mathbf{\lambda}$ is the learning rate vector that adjusts the regularization of parameters $\mathbf{W_{i}}$. \cite{1} The $\mathbf{\lambda}$ term is also called the \emph{shrinkage}

    In other words, Lasso is similar to Ridge where both helps adding biases to the model to improve the out of the sample fit of the model.
    Lasso is especially useful for dealing with sparse data.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    The same as regression, assuming \verb|i.i.d.| of sampled data.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    Same as regression except adding an L1 norm term to penalize Weight coefficients of some variables.

    Cross validation is used to select the regularization hyperparameter $\mathbf{\lambda}$.

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    $$\mathbf{W}^{lasso} = {argmin}_\mathbf{W}} \left\{\frac{1}{2} \sum_{i = 1}^{N} (y_i - \mathbf{W}_0 - \sum_{j=1}^{P} x_{ij}\mathbf{W}_{j})^2 + \mathbf{\lambda}\sum_{j=1}^{P}\left\|\mathbf{W}_{i}\right\|_{1}\right\}$$ \cite{1}, where the first term within the $argmin$ is the mean squared error and the second term is the Lasso regularization. This can be solved with convex optimization.

    \item \textbf{Whare are the advantages/disadvantages?}
    \noindent
    \smallbreak
    The prime difference between Lasso (L1) vs Ridge (L2) regularization of regression is that Lasso has the capability of forcing coefficients to $0$, and in effect, zeros out an feature variable. In essence, Lasso regression is doing feature selection.

    In a geometric sense, in high dimeisions, L1 norm will have many sharped edges and corners. During convex optimization, if the optimization hits the edges or corners, the coefficient become zero.  ON the other hand, L2 norm is a smooth surface thus it won't hit exactly model

    \includegraphics[width=0.5\textwidth, center]{ridge_vs_lasso.png}

    Thus, we can say that Lasso Regression yields a \emph{sparse} model where only a subset of the variables is in play.

    Ridge is more suited for models with dense features while Lasso is more suited for models with sparse features. Use cross validation to determine the most suitable regularization.
\end{enumerate}

\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak
\section{Gradient Boosting}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    Gradient boost decision tree models works by training many trees in sequence and ensemble the trees together at the end. When training sequentially, each of the newer trees are trained on the residual error observed by the previous tree. GBDT models tends to fit the training data well, with low bias and high variances. Because of that, it uses a learning rate $\lambda$ to scale contribution from a new tree. The idea is to take many small steps towards the right direction and ensemble them together result in a better prediction.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    Decision tree based models makes no assumption about the distribution of the underlying data. In other words, decision trees are non-parametric.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    Gradient boost initializes by building \verb|depth=1| node. It builds a tree depending on the depth and splits parameters. Then for the following trials it kept on building new trees to minimize the residual errors $(y - \gamma)$from the previous tree, and scale the contribution with a learning rate $\lambda$. The whole process is repeated $M$ times and the maximum depth of the individual trees is $J$.

    A formal definition is as follows \cite{1}
    \bigbreak
    \noindent
    \hline
    \textbf{Algorithm} Gradient Tree Boosting Algorithm
    \hline
    \begin{enumerate}
        \item Initialize $f_o(x) = argmin_{\gamma} \sum_{i=1}^{N} L(y_i, \gamma)$
        \item For \emph{m} = 1 to \emph{M}:
            \begin{enumerate}

                \item For \emph{i} = 1, 2, . . . , \emph{N} compute the pseudo residual term
                $r_{im} = -\left[ \frac{\partial L(y_i, f(x_i))}{\partial f(x_i)}\right]_{f = f_{m-1}}$
                \item For a regresion tree to the target $\emph{r}_{im}$ given terminal regions $R_{jm}, j = 1, 2, ..., J_m$
                \item For $j = 1, 2, ..., J_m$, find the new predicted value $\gamma$ \\
                $\gamma_{jm} = argmin_{\gamma} \displaystyle\sum_{x_i \in R_{jm}} L(y_i, f_{m-1}(x_i) + \gamma)$
                \item Update $f_m(x) = f_{m-1}(x) + \lambda \sum_{j=1}^{J_m} \gamma_{jm} I(x \in R_{jm})$
            \end{enumerate}
        \item Output $\hat f(x) = f_M(x)$
    \hline
    \end{enumerate}

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    Given the standard loss function setup, $L(f) = \sum_{i=1}^{N} L(y_i, f(x_i))$, proper loss function $L(f)$ can be used for different settings. A summary table is as follows \cite{1}:
    \begin{center}
      \begin{tabular}{ l | l | l }
        \hline
        Setting & Loss Function & $ \frac{-\partial L(y_i, f(x_i))}{\partial f(x_i)}$ \\ \hline
        Regression & $\frac{1}{2}[y_i - f(x_i)]^2$ & $y_i - f(x_i)$ \\ \hline
        Regression & $\left\||y_i - f(x_i)|\right\|$ & sign[y_i - f(x_i)] \\ \hline
        Regression & Huber & y_i - f(x_i) for |y_i - f(x_i)| \leq \delta_m \\
         & & \delta_m sign[y_i- f(x_i)] for |y_i - f(x_i)| > \delta_m \\
         & & where \delta_m \eq \alpha th-quantile \{|y_i - f(x_i)|\} \\ \hline
        Classification & Multinominal Deviance & \emph{k}th component: $I(y_i \eq G_k) - p_k(x_i)$ \\
        \hline
      \end{tabular}
    \end{center}

    \item \textbf{What are the advantages/disadvantages?}
    \noindent
    \smallbreak
    Gradient boost is very similar to AdaBoost. The key difference is that for Gradient Boost, it can have a tree size that is larger than a stomp (depth of 1). Because gradient boosting is a boosting model that ensembles sequentially trained models, it is a naturally a low bias model. However, the downside is to a low bias model is the variance can be high and has to be reduced by adding regularization. Also, because the model is trained sequentially, it is compute intensive to train.

\end{enumerate}

\section{K-Nearest Neighbors}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    K-Nearest Neighbors can be used for either regression or classification. It finds the K nearest points next to a new input data and determines the value or class by taking the mean value of its K nearest neighbors, or in the case of classification, mode. It captures the point similarity with its neighbours.

    The smaller the K, smaller training error; the larger the K, the more smooth the decision boundary due to majority voting of neighbors. To find the optimal K, cross validate or grid search.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    KNN is a non parametric model. The basic assumption is that the data points close to it will have the same class labels for classification, or similar values in terms of regression.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    Given data $(x_1, y_1), (x_2, y_2), ..., (x_i, y_i)$, construct KNN as follows:
    For a new input $x$:
    \begin{enumerate}
        \item Return $k$ nearest points closest to $x$, indexed as $x_{i1}, x_{i2}, ..., x_{ij}$.
        \item Return the majority vote of $y_{i1}, y_{i2}, ..., y_{ij}$.
    \end{enumerate}

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    KNN does not use a loss function to train the model. Instead, it uses distance functions, where the most common distance function is \verb|L2 norm| $= \left\|u - v\right\|_2 = (\sum_{i=1}^{d}(u_i - v_i)^{2})^\frac{1}{2}$ for dimension $\mathbb{R}^d$. Or, in a more generalized form,\\
    $$\left\|u - v\right\|_p = (\sum_{i=1}^{d}(u_i - v_i)^{p})^\frac{1}{p}$$

    Other distances can be used as well, including edit distances (how many edits away to transform from one string to another), or correlation distance (for signal detection).

    \item \textbf{What are the advantages/disadvantages?}
    \noindent
    \smallbreak
    KNN has an easy to interpret output and it does not make any assumption about the underlying data. Further, it can naturally extend to multi-class classifier and can do well in practice with enough data. For example, KNN can achieve 90\%+ with MNIST dataset.

    Om the other hand, it requires data to in or relatively close to memory, thus when the dataset is big is big enough it becomes a big search problem. Also, finding a proper distance function requires some expert knowledge, heuristics, or a grid search.
\end{enumerate}
\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak\bigbreak

\section{K-means Clustering}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    K-means is an unsupervised learning method where it tries to separate data into $K$ clusters by finding the $K$ centroids within the dataset.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    K-means is a non-parametric unsupervised learning model. It assumes the clusters separated/calculated has a "circular" shape with a centroid each  as it attempts to assign each sample $(x_1, ..., x_n)$ to one of the $K$ clusters $\{C_1, ..., C_k\}$

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    To run K-means clustering, we use the following algorithm:

    \begin{algorithm}
        Initialize randomly centeroids $\mu_1, ..., \mu_k$\;
        \Repeat{convergence}{
            assign each point $x_i$ to the cluster with the closet $\mu_j$;\\
            Calculate the new mean for each cluster as follows:;\\
            $\mu_j = \frac{1}{\|C_j\|} \sum_{x_i \in C_j} x_i$\\}
        \caption{K-means algorithms}
        Convergence: means no change in the clusters OR maximum number of iterations reached.
    \end{algorithm}

    To find the optimal $k$ that clusters the data, we can use PG-means algorithm \cite{5}. Other alternatives are G-means and P-means.
    \begin{algorithm}
        Initialize $k$ to be a small number;
        \Repeat{no more cluster center is created.}{
            Run K-means with those cluster centers, and store the resulting center as C\\
            Assign each point to its nearest cluster\\
            Determine if points in each cluster fit a Gaussian distribution (using Anderson-Darling test).\\
            For each cluster, if the points seem to be normally distributed, keep the cluster center. Otherwise replace it with two cluster centers.}
        \caption{PG-means algorithm}
    \end{algorithm}

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    There's no cost functions for K-means clustering per se, but the algorithm uses $\mu_j = \frac{1}{\|C_j\|} \sum_{x_i \in C_j} x_i$ to calculate the mean of each centroid.

    \item \textbf{What are the advantages/disadvantages?}
    \noindent
    \smallbreak
    The biggest benefit for K-means Clustering as a clustering algorithm is that its super easy to implement.

    However there are many downsides. First, it needs to know $K$, which we can 'solve' using PG-means algorithm\cite{5}, but there still exists no trivial way to evaluate the model similar to, say, comparing to counting the number of errors in a classification. K-means also suffers from the curse of dimensionality and has no theoretical foundation.

    There are other methods to cluster non-circular shapes, such as spectral clustering, DBScan, BIRCH, etc that handles other shapes. Alternatively, we can use PCA and TSNE for clustering higher dimensional data.
\end{enumerate}

\section{Naive Bayes}

\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    Naive Bayes classifier is a probabilistic model based on the Bayes rule. It is Naive because it makes an assumption that inputs are conditionally independent of each output class, which is not realistic. It is also known as "Idiot's Bayes"\cite{1}. Despite that, naive Bayes classifiers often outperform far more sophisticated alternatives. It's a simple and strong method that can be comparable to decision trees and neural networks in certain cases.

    It is most commonly seen and used in text classification models.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak

    It is Naive because it makes an assumption that inputs are conditionally independent of each output class, which is not realistic. It is also known as "Idiot's Bayes"\cite{1}.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    \textbf{Learning:} Based on the frequency counts in the dataset:
    \begin{enumerate}
        \item Estimate all $p(y), \forall y \in \mathbb{Y}$
        \item Estimate all $p(x_i|y), \forall y \in \mathbb{Y}, \forall x_i$
    \end{enumerate}
    \textbf{Classification:} For a new example, use:\\
    \begin{align*}
        y_{new} = argmax_{y \in \mathbb{Y}} \mathbf{\Pi}_{i} p(x_i|y)
    \end{align*}

    No model per se or hyperplane, just count the frequencies of various data combinations within the training examples.

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    "Cost" function for naive Bayes is not the same as algorithms, where other algorithms try to adjust the model weights with regards to the value that the cost function outputs. Naive Bayes is simply doing
    \begin{align*}
        y_{new} = argmax_{y \in \mathbb{Y}} \mathbf{\Pi}_{i} p(x_i|y)
    \end{align*}

    \item \textbf{Whare are the advantages/disadvantages?}
    \noindent
    \smallbreak
    Naive Bayes is a linear classifier, an variant of Linear Discrimanent Analysis. It is incredibly simple and easy to implement and works wonderfully for text data. It is especially appropriate when the dimension of the feature space is high. The model is light weight and runs very fast for real time inferencing. It is especially useful when inference time budget is small.

    The naive assumption of feature Independence is a double edged sword. While the assumption made naive Bayes a simple and powerful model, the assumption also prevent it to learn further intricacies of the underlying relationships. One other major disadvantage of naive Bayes is that it does not generalize well to unseen inputs.

\end{enumerate}
\pagebreak

\section{Support Vector Machines}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    Support Vector Machines finds the hyperplane that separates the data with the greatest margin. Two key concepts of Support Vector Machine is it uses soft margins instead of max margins to reduce variances and it uses the kernel trick to implicitly increase the feature space to reduce bias.

    It can be used for both classification and regression problems. It uses a kernel function to compute the inner product of two variables in a higher dimensional space. The data that resides in the soft margin are called support vectors, with distance to the soft margin line $\epsilon$. Hyperparameter \textbf{C} is learning rate and is found using cross validation.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    Support Vector Machine treats all the data points as equals so it is important to standardize the data. Otherwise it does not make assumptions about the underlying data distribution.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    Support Vector Machine is a convex optimization problem where it \textbf{Solve:} $\min_{w, b, \epsilon}\frac{1}{2}\left\|w\right\|_{2}^{2} + C\sum_{i=1}^{n} \epsilon_i$ such that $y_i(w^{T} x_i - w_0) \geq (1 - \epsilon_i)$ and $\epsilon_i \geq 0, \quad \forall i$. After derivation, Support Vector Machine function is:
    $$f(x) = w_o + \sum \hat{\alpha_i}K(x, x_i)$$
    where $\hat{\alpha_i}$ are zero for data not in the support set and $K$ is the Kernel function. The common kernels are polynomial, radio basis kernel (behaves similar to nearest neighbor), and neural network kernel. \cite{1}
    \begin{center}
      \begin{tabular}{ l l }
        \hline
        nth-Degree polynomial & $K(x,x') = (1 + (x, x'))^d$ \\
        Radial basis function (RBF) & $K(x, x') = exp(-\gamma\left\|x - x'\right\|_{2}^{2}$ \\
        Neural network & $K(x, x') = tanh(\kappa_1(x, x') + \kappa_2)$ \\
        \hline
      \end{tabular}
    \end{center}

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    Support Vector Machine uses hinge loss, which has the form of loss plus penalty. $L[y, f(x)] = [1 - yf(x)]_{+}$The following compares the negative log likelihood loss to hinge loss. Note that sharp hinge corner, that is when $\aplha$ are set to zero.

    \includegraphics[width=0.3\textwidth, center]{hinge_loss_han.png}

    \item \textbf{Whare are the advantages/disadvantages?}
    \noindent
    \smallbreak
    When classes are nearly separable, SVM does better than logistic regression and LDA. When classes are not, then SVM does similarly well as logistic regression. We can use kernel tricks with LDA and logistic regression as well, but more compute intensive.

    The downside with SVM is it does not provide a probability estimate, low interpretability, and does not do any feature selection.

\end{enumerate}

\section{Neural Networks}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts? What problem does it solve?}
    \noindent
    \smallbreak
    According to the Universal Approximator Theorum, neural networks is an universal function approximator that can be used approximate any functions that map $X \rightarrow Y$. It can be used abstracts away some of the feature engineering steps in classical machine learning tasks. For example, instead of designing image filters, CNN learns the filters itself, or instead of tagging word semantic labels, RNNs learns the relationships between words.

    The most fundamental neural network layer is a linear layer where $f(\mathbf{X}) = \sum_{m=1}^{M} g_m(\mathbf{W_m}\intercal\mathbf{X})$, where the input values go through an affine transformation and scaled by an activation function $g_m$. Activation functions usually tend to scale the outputs to pseudo range bound, such as Sigmoid and Tanh.

    \item \textbf{What are the assumptions?}
    \noindent
    \smallbreak
    The assumptions are highly dependent on the structure of neural networks. For example, some neural network architectures such as convolution neural networks or recurrent neural nets assumes that the underlying data are \emph{i.i.d} while some other architectures such as Bayesian neural networks or Graph neural nets could have different assumptions about the relationships between feature spaces.

    \item \textbf{What are the steps of the algorithm?}
    \noindent
    \smallbreak
    Neural networks is predominant solved using stochastic gradient descent with an auto differentiation library. First, inputs are feed forward through the network for an output $\hat y$, and apply the loss function. The algorithm then take the gradients of the loss function and back propagate the network to obtain the differences in parameters between target and output. The parameters are then updated with the differences. One small batch at a time.
    \bigbreak
    \hline
    \noindent
    \textbf{Algorithm} Stochastic gradient descent (SGD) update at training iteration k \cite{2}
    \hline
    \smallbreak
    \textbf{Require:} Learning rate $\epsilon_k$

    \textbf{Require:} Initial parameter $\theta$

    \qquad \textbf{while} stopping criterion not met \textbf{do}

    \qquad \qquad Sample a minibatch of \emph{m} examples from the training set $\{x^{(1)}, ..., x^{(m)}\}$ with corresponding targets $y^{(i)}$

    \qquad \qquad Compute gradent estimate: $\hat g \leftarrow + \frac{1}{m}\nabla_{\theta}\sum_{i} L(f(x^{i};\theta),y_{(i)})$

    \qquad \qquad Apply update: $\theta \leftarrow \theta - \epsisolon\hat g$

    \qquad \textbf{end while}
    \smallbreak
    \hline

    \item \textbf{What is the cost function?}
    \noindent
    \smallbreak
    Neural networks is an flexible architecture thus different loss functions can be used for different tasks. Typical loss functions includes cross entropy and negative log likelihood for classification and mean absolute error (L1) and mean squared error (L2) for regressions.

    \item \textbf{Whare are the advantages/disadvantages?}
    \noindent
    \smallbreak
    Rather than considering neural networks as a kind of model, it is closer to a highly customizable building blocks for machine learning models. Depends on the complexity of the network constructed, it can be capable of learning a large amount of information from data. At the same time it excels in learning implicit relationships.

    However, it is an black box model that is quite difficult to interpret and usually has to rely on large data sets and large amount of compute power to train. It is also difficult to debug and fine tune.

\end{enumerate}

\section{Generative vs Discriminative}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts?}
    \noindent
    \smallbreak
    \textbf{Generative classifiers} learns a model of the join probability $p(x, y)$, of the input $x$ and the label $y$, and make their predictions by using Bayes rules to calculate $p(y|x)$, then pick the most likely label $y$.\cite{4} In learning joint probability $p(x, y)$ it learns both $p(x|y)$ (likelihood) and $p(y)$ (class prior) and make inference using Bayes rule. Generative models can use the joint probability $p(x, y)$ to generate new data similar to existing data. It also requires not as much data samples to train and easy to implement.
    \begin{itemize}
        \item[--] Idea: Build a model for what positive examples look like. Build a different model for what negative example look like.
        \item[--] To predict a new example, match it with each of the models and see which match is best.
        \item[--] Model $p(x|y)$ and $p(y)$
        \item[--] Use Bayes rule to obtain $p(y|x) = \frac{p(x|y)p(y)}{p(x)}$
        \item[--] To make a prediction: \\
        \begin{align*}
            argmax_{y} p(y|x) &= argmax_{y} \frac{p(x|y)p(y)}{p(x)} \\
              &\approx argmax_{y} p(x|y)p(y)  \\
        \end{align*}

    \end{itemize}

    \noindent
    \smallbreak
    \textbf{Discriminative classifiers} models the posterior $p(y|x)$ directly, or learn a direct map from input $x$ to class labels.\cite{4} Discriminative models generally offers better performance in classification tasks.
    \begin{itemize}
        \item[--] Idea: model $p(y|x)$, conditional distribution of $y$ given $x$.
        \item[--] In discriminative algorithms: find a decision boundary that separates positive from negative examples
        \item[--] To predict a new example, check which side of the decision boundary it falls
        \item[--] Model $p(y|x)$ directly.
    \end{itemize}

    \noindent
    \smallbreak
    Here are some examples of generative and discriminative models:
    \begin{center}
      \begin{tabular}{ l l }
        \hline
        \textbf{Generative} & \textbf{Discriminative} \\
        \hline
        Naive Bayes & Logistic regression \\
        Hidden Markov model & K-nearest neighbors \\
        Gaussian mixture model & Support vector machines \\
        Generative adversarial networks & Neural networks \\
        \hline
      \end{tabular}
    \end{center}

\end{enumerate}
\pagebreak

\section{Bias-Variance Trade off}
\noindent
\begin{enumerate}
    \item \textbf{What are the basic concepts?}
    \noindent
    \smallbreak

\end{enumerate}

\begin{thebibliography}{}
\bibitem{1}
Hastie, Tibshirani, and Friedman,
\emph{The Elements of Statistical Learning}.
2nd Edition,
2009.

\bibitem{2}
Goodfellow, Bengio, Courville,
\emph{Deep Learning},
2016

\bibitem{3}
Nasirany, Thomas, Wei, and Yang,
\emph{A Comprehensive Guide to Machine Learning},
2018

\bibitem{4}
Ng and Jordan,
\emph{On Discriminative vs Generative Classifiers: A comparison of logistic regression and naive Bayes}
2002

\bibitem{5}
Hamerly and Elkan
\emph{PG-means: learning the number of clusters in data}
2003

\end{thebibliography}
\end{document}


\documentclass{standalone}
\begin{document}
		\chapter{Logarithms}
	\section{Definition}
	\quad In mathematics, the logarithm is the inverse function to exponentiation. That means the logarithm of a given number $x$ is the exponent to which another fixed number, the base $b$, must be raised, to produce that number $x$.\\

	\emph{Consider: }\\$$ 2^3 = 8$$\\

	3 is the exponent by which 2 must be raised to obtain 8. This statement can also be reversed:\\ 3 is the logarithm by which with a base of 2, results in 8. Thus:$$ 3 = \log_28$$\\

	\emph{In general: }\\$$a^b = c \iff \log_ac = b ,\quad a \in \mathbb{R^+}$$\\

	Furthermore, it is standard to represent $\log_{10}(x)$ as $\log(10) $ and $ \log_e(x) $ as $ \ln(x) $

	\bigskip
	\bigskip
	\bigskip
	\bigskip

	\tcbset{
		enhanced,
		colback=red!5!white,
		boxrule=0.1pt,
		colframe=black!75!black,
		fonttitle=\bfseries
	}
	\begin{center}
		\begin{tcolorbox}[center title,hbox,    %%<<---- here
			lifted shadow={1mm}{-2mm}{3mm}{0.1mm}%
			{black!50!white}]
			\begin{varwidth}{\textwidth}
				\begin{center}
					$	\log_a1                         = 0                    $ \\
					\bigskip
					$\log_aa                          = 1                    $ \\
					\bigskip
					$	\log_c(ab)                       \equiv \log_ca + log_cb $\\
					\bigskip
					$	\log_c\left (\frac{a}{b}\right)  \equiv \log_ca - log_cb$ \\
					\bigskip
					$	n\log_ca                         \equiv \log_ca^n   $
				\end{center}
			\end{varwidth}
		\end{tcolorbox}
	\end{center}
	\section{Proofs}

	\paragraph{Proof 1: }$\log_aa  = 1$
	\begin{center}
		$	\text{Let} \log_aa = x$
		$$   a^x      = a $$
		$$x        =1 $$
	\end{center}
	\paragraph{Proof 2: }$\log_aa  = 1$
	\begin{center}
		$	\text{Let} \log_a1 = x$
		$$   a^x      = 1 $$
		$$x        = 0$$
	\end{center}
	\paragraph{Proof 3: }$\log_cab  = \log_ca + \log_cb$
	\begin{center}
		$	\text{Let} \log_ca = x$ ; $\text{Let} \log_cb = y$
		$$\Rightarrow  c^{x} = a  \text{ ; } c^{y} = b$$
		$$\Rightarrow ab = c $$
		$$\Rightarrow\log_c(ab) = \log_c(x) + \log_c(y)$$
		$$\therefore\quad \log_c(ab) = \log_c(a) + \log_c(b)$$
	\end{center}
	\paragraph{Proof 4: }$\log_c\frac{a}{b}  = \log_ca - \log_cb$
	\begin{center}
		$	\text{Let} \log_ca = x$ ; $\text{Let} \log_cb = y$
		$$  c^{x} = a  \text{ ; } c^{y} = b$$
		$$\frac{a}{b} = c^x \times c^{-y} $$
		$$\log_c\frac{a}{b} = \log_c(x) - \log_c(y)$$
	\end{center}
	\paragraph{Proof 5: }$\log_ca^n = n\log_ca$
	\begin{center}
		$	\text{Let} \log_ca^n = x$
		$$ c^x = a^n$$
		$$c^{\frac{x}{n}} = a  $$
		$$\log_ca = \frac{x}{n}$$
		$$x = n\log_ca $$
	\end{center}
\end{document}


\subsection{Differentials}\label{sec:differentials}
Very much related to linear approximations are the {\em differentials} $dx$ and $dy$, used not to approximate values of $f$, but instead the change (or rise) in the values of $f$.

\begin{definition}{Differentials dx and dy}{dx}
Let $y=f(x)$ be a differentiable function. We define a new
  independent variable $dx$, and a new dependent variable
  $dy=f'(x)\,dx$. Notice that $dy$ is a function both of $x$ (since
  $f'(x)$ is a function of $x$) and of $dx$.  We call both $dx$ and
  $dy$ \deffont{differentials}.
\end{definition}

Now fix a point $a$ and let $\Delta x =x-a$ and $\Delta y= f(x)-f(a)$.
If $x$ is near $a$ then $\Delta x$ is clearly small. If we set $dx=\Delta x$ then we obtain
\[ dy = f'(a)\,dx \approx \frac{\Delta y}{\Delta x}\Delta x = \Delta y.\]
Thus, $dy$ can be used to approximate $\Delta y$, the actual change in
the function $f$ between $a$ and $x$. This is exactly the
approximation given by the tangent line:
\[ dy = f'(a)(x-a) = f'(a)(x-a)+f(a)-f(a)=L(x)-f(a).\]
While $L(x)$ approximates $f(x)$, $dy$ approximates how $f(x)$ has
changed from $f(a)$.
Figure~\ref{fig:differentials} illustrates the relationships.

\figure[!ht]
\centerline{\vbox{\beginpicture
\normalgraphs
%\sevenpoint
\setcoordinatesystem units <2truecm,2truecm>
\setplotarea x from 0 to 4.5, y from 0 to 2.5
\axis left /
\axis bottom ticks withvalues {$a$} {$x$} / at 1 3 / /
\plot 0.500 0.707 0.588 0.766 0.675 0.822 0.762 0.873 0.850 0.922
0.938 0.968 1.025 1.012 1.112 1.055 1.200 1.095 1.288 1.135
1.375 1.173 1.462 1.209 1.550 1.245 1.638 1.280 1.725 1.313
1.812 1.346 1.900 1.378 1.988 1.410 2.075 1.440 2.162 1.471
2.250 1.500 2.338 1.529 2.425 1.557 2.512 1.585 2.600 1.612
2.688 1.639 2.775 1.666 2.862 1.692 2.950 1.718 3.038 1.743
3.125 1.768 3.212 1.792 3.300 1.817 3.388 1.841 3.475 1.864
3.562 1.887 3.650 1.910 3.738 1.933 3.825 1.956 3.912 1.978
4.000 2.000 /
\setlinear
\plot 1 1 3 2 3 1 1 1 /
\betweenarrows {$dx=\Delta x$} from 1 0.8 to 3 0.8
\betweenarrows {$\Delta y$} from 3.2 1 to 3.2 1.73
\betweenarrows {$dy$} from 3.6 1 to 3.6 2
\setdashes <2pt>
\putrule from 3 2 to 3.6 2
\putrule from 3 1 to 3.6 1
\putrule from 3 1.73 to 3.2 1.73
\endpicture}}
\caption{Differentials.\label{fig:differentials}}
\endfigure

Here is a concrete example.

\begin{example}{Rise of Natural Logarithm}{rise of natural logarithm}
Approximate the rise of $f(x)=\ln x$ from $x=1$ to $x=1.1$, using linear approximation.
\end{example}

\begin{solution}
Note that $\ln (1.1)$ is not readily calculated (without a calculator) hence why we wish to use linear approximation to approximate $f(1.1)-f(1)$.

We fix $a=1$ and as above we have $\Delta x=x-1$ and $\Delta y=f(x)-f(1)=\ln x$, and obtain
\[ dy=f'(1)dx\approx \frac{\Delta y}{\Delta x}\Delta x=\Delta y. \]
But $f'(x)=1/x$ and thus $f'(1)=1/1=1$, we obtain in this case
\[ dy=dx\approx\Delta y. \]
Finally for $x=1.1$, we can easily approximate the rise of $f$ as
\[ f(1.1)-f(1)=\Delta y\approx dy=1.1-1=0.1. \]
The correct value of $\ln (1.1)=\ln 1$ is 0.0953\ldots and thus we were relatively close.
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:differentials}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
Let $\ds f(x) = x^4$. If $a=1$ and $dx= \Delta x =1/2$,
what are $\Delta y$ and $dy$?
\begin{sol}
	$\Delta y=65/16$, $dy=2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Let $\ds f(x) = \sqrt{x}$. If $a=1$ and $dx= \Delta x
=1/10$, what are $\Delta y$ and $dy$?
\begin{sol}
	$\ds \Delta y=\sqrt{11/10}-1$, $dy=0.05$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Let $f(x) = \sin (2x)$. If $a=\pi$ and $dx= \Delta x
=\pi/100$, what are $\Delta y$ and $dy$?
\begin{sol}
	$\ds \Delta y=\sin(\pi/50)$, $dy=\pi/50$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Use differentials to estimate the amount of paint needed to
 apply a coat of paint 0.02 cm thick to a sphere with diameter $40$
 meters. (Recall that the volume of a sphere of radius $r$ is $V
 =(4/3)\pi r^3$. Notice that you are given that $dr=0.02$.)
\begin{sol}
	$dV=8\pi/25$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\documentclass[a4paper]{article}

\def\ntitle{Field Theory}

\input{header}

\begin{document}

\maketitle

\tableofcontents

\section{Field Extensions}

\begin{definition}
  Given a field $F$, there is a unique ring homomorphism $\phi: \mathbb{Z} \rightarrow F, 1 \mapsto 1$. Let $\ker \phi = n\mathbb{Z}$, then the \emph{characteristic} of $F$ $\ch(F) = n$.
\end{definition}

\begin{definition}
  Fiven a field $F$, the \emph{prime subfield} of $F$ is the subfield generated by $1_F$.
\end{definition}

The prime subfield of any field is isomorphic to either $\mathbb{Q}$ or $\mathbb{F}_p$ for some prime $p$.

\begin{definition}
  If $K$ is a field containing a subfield $F$, then $K$ is an extension of $F$, denoted $K/F$.
\end{definition}

\begin{definition}
  The \emph{degree of the extension} $K/F$, $[K:F] := \dim_FK$. The extension is finite if $[K:F]$ is finite, otherwise infinite.
\end{definition}

\begin{proposition}
  Let $p(x) \in F[x]$ be irreducible. Then $K:=F[x]/(p(x))\supseteq F$ and $p$ has a root in $K$.
\end{proposition}

\begin{proposition}
  Let $p(x) \in F[x]$ be irreducible of degree $n$, $K:=F[x]/(p(x)),\: \theta= x \mod{p(x)} \in K$. Then $\{\theta^i\}_{i=0}^{n-1}$ is a basis for $K/F$, so $[K:F]=n$ and $K=\{\sum_{i=0}^{n-1} a_i \theta^i:a_i \in F\}$.
\end{proposition}

\begin{proof}\leavevmode
  \begin{itemize}
  \item $F[x]$ is a Euclidean domain
  \item $p$ is irreducible
  \end{itemize}
\end{proof}

\begin{definition}
  Suppose $K/F$ and $\alpha,\beta,\ldots \in K$. Then the \emph{smallest subfield} containing both $F$ and $\alpha,\beta,\ldots$, $F(\alpha,\beta,\ldots)$ is called the field \emph{generated} by $\alpha,\beta,\ldots$ over $F$.
\end{definition}

\begin{definition}
  If $K=F(\alpha)$ then $K$ is a \emph{simple extension} of $F$ and $\alpha$ is a \emph{primitive element} for $K/F$.
\end{definition}

\begin{proposition}
  Suppose $p(x) \in F[x]$ is irreducible. Suppose $K/F$ and $\alpha\in K$ such that $p(\alpha)=0$. Then $F(\alpha) \cong F[x]/(p(x))$.
\end{proposition}

\begin{proof}
  Since $p(\alpha)=0$, we have the following induced homomorphism
  \[
    \begin{tikzcd}
      F[x] \arrow{r}{\phi} \arrow{d}[swap]{\pi} & F(\alpha)\\
      F[x]/(p(x)) \arrow[dotted]{ru}[swap]{\bar\phi}
    \end{tikzcd}
  \]

  As $F \leq \im(\bar\phi) \leq F(\alpha)$ and $\alpha\in \im(\bar\phi)$, by definition $\im(\bar\phi)=F(\alpha)$.
\end{proof}

\begin{remark}
  Roots of an irreducible $p(x)$ are algebraically indistinguishable.
\end{remark}

\begin{theorem}\label{thm:uniqueness of simple extension}
  Suppose $\phi: F\xrightarrow{\sim} F',\: p(x)\in F[x]$ be irreducible and $p'(x):=\phi(p(x))$. Suppose $\alpha,\beta$ are roots of $p$ and $p'$ respectively. Then $\sigma: F(\alpha) \rightarrow F(\beta),\:\alpha \mapsto \beta$ is an isomorphism and
  \[
    \begin{tikzcd}
      F \arrow{r}{\sim}[swap]{\phi} \arrow[hook]{d} & F' \arrow[hook]{d}\\
      F(\alpha) \arrow{r}{\sim}[swap]{\sigma} & F'(\beta)
    \end{tikzcd}
  \]
\end{theorem}

\begin{proof}
  $\phi$ as a map $F[x] \rightarrow F'[x]$ maps irreducible to irreducible so
  \[
    \begin{tikzcd}
      F[x]/(p(x)) \arrow{r}{\sim} \arrow{d} & F'[x]/(p'(x)) \arrow{d}\\
      F(\alpha) \arrow{r} &F(\beta)
    \end{tikzcd}
  \]
\end{proof}

\section{Algebraic Extensions}

Let $F$ be a field and $K$ be an extension of $F$.

\begin{definition}
  The element $\alpha\in K$ is said to be \emph{algebraic} over $F$ is $\alpha$ is a root of some nonzero polynomial $f(x)\in F[x]$. If $\alpha$ is not algebraic over $F$ then $\alpha$ is said to be \emph{transcendental} over $F$. The extension $K/F$ is said to be \emph{algebraic} if every element of $K$ is algebraic over $F$.
\end{definition}

\begin{proposition}
  Let $\alpha$ be algebraic over $F$. Then there is a unique irreducible polynomial $m_{\alpha,F}(x)\in F[x]$ which has $\alpha$ as a root. A polynomial $f(x)\in F[x]$ has $\alpha$ as a root iff $m_{\alpha,F}(x)$ divides $f(x)$ in $F[x]$.
\end{proposition}

\begin{proof}\leavevmode
  \begin{itemize}
  \item Let $g(x)\in F[x]$ be a monic polynomial of minimal degree having $\alpha$ as a root.
  \item $F[x]$ is a Euclidean domain
  \end{itemize}
\end{proof}

\begin{definition}
  The polynomial $m_{\alpha,F}$ is called the \emph{minimal polynomial} for $\alpha$ over $F$. The \emph{degree} of $m_\alpha$ is called the \emph{degree} of $\alpha$.
\end{definition}

\begin{proposition}
  Let $\alpha$ be algebraic over $F$, then
  \[
    F(\alpha) \cong F[x]/(m_\alpha(x))
  \]
  so in particular
  \[
    [F(\alpha):F] = \deg m_\alpha(x) = \deg \alpha.
  \]
\end{proposition}

\begin{proposition}
  The element $\alpha$ is algebraic over $F$ iff the simple extension $F(\alpha)/F$ is finite.
\end{proposition}

\begin{proof}
  Use the fact that $\{\alpha^i\}_{i=0}^{\deg \alpha}$ is a basis for $F(\alpha)/F$ when the extension is finite.
\end{proof}

\begin{corollary}
  A finite extension is algebraic.
\end{corollary}

\begin{eg}[Quadratic Extensions over Fields of Characteristic $\neq 2$]

  \texttt{to be filled in}
\end{eg}

\begin{theorem}[Tower Law]
  Let $F\subseteq K\subseteq L$ be fields. Then
  \[
    [L:F] = [L:K][K:F]
  \]
\end{theorem}

\begin{definition}
  An extension $K/F$ is \emph{finitely generated} if there are elements $\{\alpha_i\}_{i=1}^k$ in $K$ such that $K=F(\alpha_1,\ldots,\alpha_k)$.
\end{definition}

\begin{lemma}
  $F(\alpha,\beta) = (F(\alpha))(\beta)$
\end{lemma}

\begin{theorem}
  The extension $K/F$ is finite iff $K$ is generated by a finite number of algebraic elements over $F$. More precisely,
  \[
    [F(\alpha_1,\ldots,\alpha_k):F] \leq \prod_{i=1}^{k} \deg(\alpha_i)
  \]
\end{theorem}

\begin{corollary}
  Suppose $\alpha,\beta$ are algebraic over $F$. Then $\alpha\pm\beta,\alpha\beta,\alpha/\beta$ are all algebraic as they are elements of the extenion $F(\alpha,\beta)$.
\end{corollary}

\begin{corollary}
  Let $L/F$ be an arbitrary extension. Then the collection of elements of $L$ that are algebraic over $F$ form a subfield $K$ of $L$.
\end{corollary}

\begin{theorem}[Transitivity of Algebraic Extension]
  IF $K$ is algebraic over $F$ and $L$ is algebraic over $K$ then $L$ is algebraic over $F$.
\end{theorem}

\begin{proof}
  Let $\alpha\in L$, which is algebraic over $K$ so $\alpha$ satisfies some polynomial
  \[
    \sum_{i=0}^{n} a_i \alpha^i = 0
  \]

  with $a_i\in K$. So
  \[
    [F(\alpha,a_0,\ldots,a_n):F] = [F(\alpha,a_0,\ldots,a_n):F(a_0,\ldots,a_n)][F(a_o,\ldots,a_n):F]
  \]
  which is finite.
\end{proof}

\begin{definition}
  Let $K_1$ and $K_2$ be two subfields of a field $K$. The \emph{composite field} of $K_1$ and $K_2$, denoted $K_1K_2$ is the smallest subfield of $K$ containing both $K_1$ and $K_2$.
\end{definition}

\begin{proposition}\label{prop:composite}
  Let $K_1$ and $K_2$ be two finite extensions of a field $F$ contained in $K$. Then
  \[
    [K_1K_2:F] \leq [K_1:F][K_2:F]
  \]
\end{proposition}

\section{Classical Straightedge and Compass Constructions}

Let $1$ denote a fixed given unit distance, then any distance is determined by its length $a\in \mathbb{R}$. The collection of lengths that can be obtained by compass and straightedge constructions from a unit distance is called the \emph{constructible} elements of $\mathbb{R}$.

\begin{proposition}
  The collection of constructible elements is a \emph{subfield} of $\mathbb{R}$ strictly larger than $\mathbb{Q}$.
\end{proposition}

We can construct $\sqrt a$ for any $a > 0$. We can also show, by obtaining the coordinates of lines and circles by solving their equations, that any operation on elements of $F$ produces elements in at most a \emph{quadratic} extension of $F$. Thus

\begin{proposition}
  If the element $\alpha\in\mathbb{R}$ is obtained from a field $F \subset \mathbb{R}$ by a series of operations then $[F(\alpha):F]=2^k$ for some integer $k \geq 0$.
\end{proposition}

\begin{proposition}
  Trisecting an angle is impossible since $\cos 20^{\circ}$ is not contructible.
\end{proposition}

\begin{proof}
  Let $\beta=\cos20^\circ$. By triple angle formula we have
  \[
    4\beta^3 -3\beta - 1/2 = 0
  \]
  Let $\alpha=2\beta$, then $\alpha$ satisfies the equation
  \[
    \alpha^3-3\alpha-1=0
  \]
  which can be shown to have no rational roots by Rational Root Theorem. Thus $[\mathbb{Q}(\alpha):\mathbb{Q}]=3$.
\end{proof}

\begin{remark}
  The angles $1^\circ$ and $2^\circ$ are not constructible as otherwise addition formula show that $20^\circ$ would be constructible. The regular pentagon gives $72^\circ$ and the equilateral triangle gives $60^\circ$ so $3^\circ$ is constructible.
\end{remark}

\section{Splitting Fields and Algebraic Closures}

\begin{definition}
  The extension field $K$ over $F$ is called a \emph{splitting field} for the polynomial $f(x)\in F[x]$ if $f(x)$ factors completely into linear factors in $K[x]$ and $f(x)$ does not factor completely into linear factors over any proper subfield of $K$ containing $F$.
\end{definition}

\begin{theorem}
  For any field $F$, if $f(x)\in F[x]$ then there exists an extension $K$ of $F$ which is a splitting field for $f(x)$.
\end{theorem}

\begin{proposition}
  A splitting field of polynomial of degree $n$ over $F$ is of degree at most $n!$ over $F$.
\end{proposition}

\begin{definition}
  If $K$ is an algebraic extension of $F$ which is the splitting field over $F$ for a collection of polynomials $f(x)\in F[x]$ then $K$ is called a \emph{normal} extension of $F$.
\end{definition}

\begin{eg}[Splitting Field of $x^n-1$: Cyclotomic Fields]
  Consider the splitting field of the polynomial $x^n-1$ over $\mathbb{Q}$. The roots of the polynomial are called the $n$th \emph{roots of unity}.

  Since any finite group of the multiplicative group of a field is cyclic, we call a generator of the cyclic group of all $n$th roots of unity a \emph{primitive} $n$th root of unity, denoted $\zeta_n$. Given a primitive root $\zeta_n$, the other primitive roots are then $\zeta_n^i$ where $1 \leq i < n$ is an integer relatively prime to $n$. Thus there are $\varphi(n)$ primitive roots of unity. Over $\mathbb{C}$ we can see this directly be letting $\zeta_n:=e^{2\pi i/n}$.

  The field $\mathbb{Q}(\zeta_n)$ is called the \emph{cyclotomic field of $n$th roots of unity}.

  When $n=p$ is a prime, we have the factorisation
  \[
    x^p-1 = (x-1)(x^{p-1}+\cdots+x+1)
  \]
  Since $\zeta_p \neq 1$ it is a root of the polynomial
  \[
    \Phi_p(x) = \frac{x^p-1}{x-1} = (x-1)(x^{p-1}+\cdots+x+1)
  \]
  which is irreducible. Thus $\Phi_p$ is the irreducible polynomial of $\zeta_p$ over $\mathbb{Q}$ so
  \[[\mathbb{Q}(\zeta_p):\mathbb{Q}] = p-1\]

  See more discussion of in Section \ref{sec:cyclotomic}.
\end{eg}

\begin{eg}[Splitting Field of $x^p-2$, $p$ is a prime]
  If $\alpha$ is a root of the equation $x^p-2$ then $\zeta\alpha$ is alos a root where $\zeta$ is a $p$th root of unity. Denote the positive real $p$th root of $2$ $\sqrt[p]2$. We can show that the splitting field is exactly $\mathbb{Q}(\sqrt[p]2, \zeta_p)$. The extension field has $\mathbb{Q}(\sqrt[p]2)$ and $\mathbb{Q}(\zeta_p)$ as subfields so by Proposition \ref{prop:composite}
  \[ [\mathbb{Q}(\sqrt[p]2,\zeta_p):\mathbb{Q}] = p(p-1) \]
\end{eg}

\begin{theorem}
  Let $\phi: F\xrightarrow{\sim} F'$ be an isomorphism of fields. Let $f(x)\in F[x]$ be a polynomial and $f'(x):=\phi(f(x))\in F'[x]$. Let $E$ be a splitting field for $f(x)$ over $F$ and let $E'$ be a splitting field for $f'(x)$ over $F'$. Then the isomorpism $\phi$ extends to an isomorphism $\sigma:E\xrightarrow{\sim} E'$:
  \[
    \begin{tikzcd}
      F \arrow{r}{\sim}[swap]{\phi} \arrow[hook]{d} & F' \arrow[hook]{d}\\
      E \arrow{r}{\sim}[swap]{\sigma} & E'
    \end{tikzcd}
  \]
\end{theorem}

\begin{proof}
  Induction on the degree $n$ of $f(x)$. The base case where $f(x)$ splits completely in $F[x]$ is easy. Suppose now that $p(x)$ is an irreducible factor of $f(x)$ of degree at least $2$. Let $\alpha\in E$ be a root of $p(x)$ and $\beta\in E'$ be a root of $p'(x)$. Then by Theorem \ref{thm:uniqueness of simple extension} we have
  \[
    \begin{tikzcd}
      F \arrow{r}{\sim}[swap]{\phi} \arrow[hook]{d} & F' \arrow[hook]{d}\\
      F(\alpha) \arrow{r}{\sim}[swap]{\sigma'} & F(\beta)
    \end{tikzcd}
  \]

  Now we have $f(x) = (x-\alpha)f_1(x)$ over $F(\alpha)[x]$ where $f_1(x)$ ahs degree $n-1$. Note that the field $E$ is a splitting field for $f_1(x)$ over $F(\alpha)[x]$: all the roots of $f_1(x)$ are in $E$ and if they were contained in any smaller extension $L$ containing $F(\alpha)[x]$, then since $F(\alpha)$ contains $\alpha$, $L$ would also contain all roots of $f(x)$, which would contradict the minimality of $E$ as the splitting field of $f(x)$ over $F$. Similar for $E'$. Since $f_1(x)$ has degree less than $n$, by induction there exist $\sigma: E\xrightarrow{\sim} E'$ such that
  \[
    \begin{tikzcd}
      F \arrow{r}{\sim}[swap]{\phi} \arrow[hook]{d} & F' \arrow[hook]{d}\\
      F(\alpha) \arrow{r}{\sim}[swap]{\sigma'} \arrow[hook]{d} & F(\beta) \arrow[hook]{d}\\
      E \arrow{r}{\sim}[swap]{\sigma} & E'
    \end{tikzcd}
  \]
\end{proof}

\begin{corollary}[Uniqueness of Splitting Fields]
  Any two splitting fields for a polynomial $f(x)\in F[x]$ over a field $F$ are isomorphic.
\end{corollary}

\begin{definition}
  The field $\cl F$ is called an \emph{algebraic closure} of $F$ if $\cl F$ is algebraic over $F$ and if every polynomial $f(x)\in F[x]$ splits completely over $\cl F$.
\end{definition}

\begin{definition}
  A field $K$ is said to be \emph{algebraically closed} if every polynomial with coefficients in $K$ has a root in $K$.
\end{definition}

$K=\cl K$ iff $K$ is algebraically closed.

\begin{proposition}
  Let $\cl F$ be an algebraic closure of $F$. Then $\cl F$ is algebraically closed.
\end{proposition}

\begin{proof}
  Let $f(x)\in\cl F[x]$ and $\alpha$ be a root of $f(x)$. Then
  \[ F \subseteq \cl F[x] \subseteq \cl F[x] \]
  is an algebraic extension by transitivity so $\alpha\in\cl F$.
\end{proof}

\begin{proposition}
  For any field $F$ there exists an algebraically closed field $K$ containing $F$.
\end{proposition}

\begin{proof}
  For every nonconstant monic polynomial $f$ with coefficients in $F$, let $x_f$ denote an indeterminate and consider the polynomial ring $F[\ldots,x_f,\ldots]$. Consider the ideal $I$ generated by the polynomials $f(x_f)$. Suppose this ideal is not proper, then there is a relation
  \[ \sum_{i=1}^n g_i f_i(x_{f_i}) = 1 \]
  where $g_i$ are elements of $F[\ldots,x_f,\ldots]$. For $i=1,2,\ldots,n$ let $x_{f_i}=x_i$ and let $x_{n+1},\ldots,x_m$ be the remaining variables occurring in the polynomials $g_j$. Then the relation becomes
  \[ \sum_{i=1}^n g_i(x_1,\ldots,x_m) f_i(x_i) = 1 \]

  Let $F'$ be a finite extension of $F$ containing a root $\alpha_i$ of $f_i$. Letting $x_i = \alpha_i$ and setting $x_{n+1}=\cdots=x_m=0$. Then the relation above reads $0=1$. Absurd.

  Since $I$ is a proper ideal, by Zorn's Lemma it is contained in a maximal ideal $M$. Then
  \[ F \subseteq K_1:= F[\ldots,x_f,\ldots]/M \]
  and each polynomial $f$ has a root in $K_1$ by construction. Iterate the process and we obtain a chain of fields
  \[ F= K_0 \subseteq K_1 \subseteq \cdots \]
  Let
  \[ K = \bigcup_{i\geq0} K_i \]
  which is a field containing $F$ and $K$ is algebraically closed.
\end{proof}

\begin{proposition}
  Let $K$ be an algebraically closed field and let $F$ be a subfield of $K$. Then the collection of elements $\cl F$ of $K$ that are algebraic over $F$ is an algebraic closure of $F$. An algebraic closure of $F$ is unique up to isomorphism.
\end{proposition}

\begin{theorem}[Fundamental Theorem of Algebra]
  $\mathbb{C}$ is algebraically closed.
\end{theorem}

\begin{corollary}
  $\mathbb{C}$ contains an algebraic closure for any of its subfields. In particular, $\cl{\mathbb{Q}}$, the collection of complex numbers algebraic over $\mathbb{Q}$, is an algebraic closure of $\mathbb{Q}$.
\end{corollary}

\section{Separable Extensions}

\begin{definition}
  A polynomial over $F$ is called \emph{separable} if it has no multiple roots. A polynomial which is not separable is called \emph{inseparable}.
\end{definition}

\begin{definition}
  The \emph{derivative} of the polynomial
  \[ f(x)=a_n x^n+a_{n-1}+x^{n-1}+\cdots + a_1 x + a_0 \in F[x] \]
  is defined to be the polynomial
  \[ D_x f(x) := n a_n x^{n-1} + (n-1) a_{n-1} x^{n-2}+\cdots + 2a_2 x + a_1 \in F[x] \]
\end{definition}

\begin{proposition}
  A polynomial $f(x)$ has a multiple root $\alpha$ iff $\alpha$ is also a root of $D_xf(x)$, i.e. both $f(x)$ and $D_xf(x)$ are divisible by the minimal polynomial for $\alpha$. In particular $f(x)$ is separable iff it is relatively prime to its derivative.
\end{proposition}

\begin{eg}
  The polynomial $x^{p^n}-x$ over $\mathbb{F}_p$ has derivative $-1$ so it is separable.
\end{eg}

\begin{eg}
  The polynomial $x^n-1$ has derivative $nx^{n-1}$. Over any field of characteristic not dividing $n$ this polynomial has the only root $0$, which is not a root of $x^n-1$. Thus $x^n-1$ is separable and there are $n$ distinct $n$th root of unity.
\end{eg}

\begin{corollary}
  Every irreducible polynomial over a field of characteristic $0$ is separable. A polynomial over such a field is separable iff it is the product of distinct irreducible polynomials.
\end{corollary}

\begin{proof}
  Suppose $F$ is a field of characteristic $0$ and $p(x)\in F[x]$ is irreducible of degree $n$. Then $D_xf(x)$ has degree $n-1$ which is coprime to $p(x)$.
\end{proof}

\begin{remark}
  In characteristic $p$ the derivative of any power of $x^p$ is identically $0$:
  \[ D_x(x^{pm}) = pmx^{pm-1} = 0 \]
  so it is possible for the degree of the derivative to decrease by more than one. Thus the proof in the corollary may fail.
\end{remark}

\begin{proposition}\label{prop:frobenius}
  Let $F$ be a field of characteristic $p$. Then for any $a,b\in F$,
  \begin{align*}
    (a+b)^p &= a^p+b^p\\
    (ab)^p &= a^p b^p
  \end{align*}
  so the map $a \mapsto a^p$ is an injective field endomorphism of $F$.
\end{proposition}

\begin{definition}
  The map in Proposition \ref{prop:frobenius} is called the \emph{Frobenius endomorphism} of $F$.
\end{definition}

\begin{corollary}\label{cor:perfect field}
  Suppose $\mathbb{F}$ is a finite field of characteristic $p$. Then every element is a $p$th power in $\mathbb F$, i.e. $\mathbb F = \mathbb F^p$.
\end{corollary}

\begin{proposition}
  Every irreducible polynomial over a finite field $\mathbb F$ is separable. A polynomial in $\mathbb F$ is separable iff it is the product of distinct irreducible polynomials in $\mathbb F[x]$.
\end{proposition}

\begin{proof}
  Let $\mathbb F$ be a finite field and suppose that $p(x)\in\mathbb F$ is an irreducible. If $p(x)$ were inseparable then $p(x)=q(x^p)$ for some polynomial $q(x)\in \mathbb F$. Let
  \[ q(x) = \sum_{i=0}^m a_i x^i \]
  By Corollary \ref{cor:perfect field}, there exist $b_i\in \mathbb F$ such that $a_i= b_i^p$. Then by Proposition \ref{prop:frobenius}
  \begin{align*}
    p(x)= q(x^p) &= \sum_{i=0}^m a_i (x^p)^i\\
                 &= \sum_{i=0}^m b_i^p (x^p)^i\\
                 &= \sum_{i=0}^m (b_i x^i)^p\\
                 &= (\sum_{i=0}^m b_i x^i)^p
  \end{align*}
  which show that $p$ is the $p$th power of a polynomial, a contradiction to the irreducibility of $f$.
\end{proof}

\begin{definition}
  A field $K$ of characteristic $p$ is called \emph{perfect} if every element of $K$ is a $p$th power in $K$, i.e. $K = K^p$. Any field of characteristic $0$ is also called perfect.
\end{definition}

\begin{eg}[Existence and Uniqueness of Finite Fields]
  Let $n$ be a positive integer and consider the splitting field of $x^{p^n}-x$ over $\mathbb F_p$. We have shown that it is separable, thus having $p^n$ roots. For any two roots we can show their sum, product and quotient are also a root by Proposition \ref{prop:frobenius}. Let $\mathbb F$ be the \emph{field} of all such roots, which must be the splitting field. Thus we have $[\mathbb F:\mathbb F_p] = n$.

  For the uniqueness, suppose $\mathbb F$ is any finite field of characteristic $p$. Suppose $[\mathbb F:\mathbb F_p] = n$. Since the multiplicative group $\mathbb F^\times$ has order $p^n - 1$, we have $\alpha^{p^n}-\alpha=0$ for every $\alpha\in\mathbb F$, so by counting argument $\mathbb F$ is a splitting field of $x^{p^n}-x$. Since splitting fields are unique up to isomorphism, we have proven that \emph{finite fields of any order $p^n$ exist and are unique up to isomorphism.} We denote the field $\mathbb F_{p^n}$.
\end{eg}

\begin{proposition}\label{prop:sep}
  Let $p(x)$ be an irreducible polynomial over a field $F$ of characteristic $p$. Then there is a unique integer $k \geq 0$ and a unique irreducible separable polynomial $p_{sep}(x)\in F[x]$ such that
  \[ p(x) = p_{sep}(x^{p^k}) \].
\end{proposition}

\begin{definition}
  Use the notation in Proposition \ref{prop:sep}, the degree of $p_{sep}(x)$ is called the \emph{separable degree} of $p(x)$, denoted $\deg_s p(x)$ while $p^k$ is called the \emph{inseparable degree} of $p(x)$, denoted $\deg_i p(x)$.
\end{definition}

From the relation $p(x) = p_{sep}(x^{p^k})$ we have
\[ \deg p(x) = \deg_s p(x) \deg_i p(x). \]

\begin{definition}
  The field $K$ is said to be \emph{separable} (or \emph{separably algebraic}) over $F$ if every element of $K$ is the root of a separable polynomial over $F$. A field which is not separable is \emph{inseparable}.
\end{definition}

\begin{corollary}
  Every finite extension of a perfect field is separable.
\end{corollary}

\section{Cyclotomic Polynomials and Extensions} \label{sec:cyclotomic}

\begin{definition}
  Let $\mu_n$ denote the \emph{group of $n$th roots of unity over $\mathbb Q$}.
\end{definition}

If $d$ is a divisor of $n$ and $\zeta$ is a $d$th root of unity then $\zeta$ is also a n $n$th root of unity so
\[ \mu_d \subseteq \mu_n \quad \text{for all } d|n. \]

Conversely, any element of $\mu_n$ which is also a $d$th root of unity has order $d$ which divides $|\mu_n|=n$.

\begin{definition}
  The \emph{$n$th cyclotomic polynomial $\Phi_n(x)$} is the polynomial whose roots are the primitive $n$th roots of unity:
  \[ \Phi_n(x) := \prod_{\text{primitive }\zeta\in \mu_n} (x - \zeta) = \prod_{\substack{1\leq a <n\\ (a,n)=1}} (x-\zeta_n^a) \]
\end{definition}

Since the roots of $x^n-1$ are precisely the $n$th roots of unity, we have the factorisaton
\begin{align*}
  x^n-1 &= \prod_{\zeta \in \mu_n} (x-\zeta)\\
        &= \prod_{d|n} \prod_{\substack{\zeta \in \mu_d\\\zeta\: \text{primitive}}} (x - \zeta) \quad \text{group together $\zeta$ by order $d$}\\
        &= \prod_{d|n} \Phi_d(x) \quad \text{definition of $\Phi_n(x)$}
\end{align*}

which allows us to compute $\Phi_n(x)$ recursively. Note comparing the degree gives the identity
\[ n = \sum_{d|n} \varphi(d). \]

\begin{lemma}
  The cyclotomic polynomial $\Phi_n(x)$ is a monic polynomial in $\mathbb Z[x]$ of degree $\varphi(n)$.
\end{lemma}

\begin{proof}
  It is clear that $\Phi_n(x)$ is monic and has degree $\varphi(n)$. To show the coefficients lie in $\mathbb Z$ we use induction on $n$. The base case is easy. Asumme it is true for all $1 \leq d < n$. Then $x^n-1 = f(x) \Phi_n(x)$ where $f(x) = \prod_{d|n,d\neq n} \Phi_d(x)$ is monic and has coefficients in $\mathbb Z$. Since $f(x)$ divides $x^n - 1$ in $F[x]$ where $F=\mathbb Q(\zeta_n)$ and both $f(x)$ and $x^n - 1$ have coefficients in $\mathbb Q$, $f(x)$ divides $x^n - 1$ in $\mathbb Q[x]$ by division algorithm (divisibility is independent of the ring). By Gauss' Lemma, $f(x)$ divides $x^n -1 $ in $\mathbb Z[x]$.
\end{proof}

\begin{theorem}
  The cyclotomic polynomial $\Phi_n(x)$ is an irreducible monic polynomial in $\mathbb Z[x]$ of degree $\varphi(n)$.
\end{theorem}

\begin{proof}
  To show that $\Phi_n(x)$ is irreducible, suppose we have a factorisation in $\mathbb Z[x]$
  \[ \Phi_n(x) = f(x)g(x), \]
  where we take $f(x)$ to be an irreducible factor of $\Phi_n(x)$. Let $\zeta$ be a primitive $n$th root of unity for which $f(x)$ is the minimal polynomial. Let $p$ be any prime not dividing $n$. Then $\zeta^p$ is also a primitive $n$th root of unity. Suppose it is a root of $g(x)$. Then $\zeta$ is a root of $g(x^p)$ and so
  \[ g(x^p) = f(x)h(x), \: h(x) \in \mathbb Z[x] \]

  Reduce modulo $p$, we obtain
  \[ (\bar g(x))^p = \bar g(x^p) = \bar f(x) \bar g(x) \quad \text{in } \mathbb F_p[x]. \]
  Since $\mathbb F_p[x]$ is a U.F.D., it folows that $\bar f(x)$ and $\bar g(x)$ have a factor in common.

  Now reduce $\Phi_n(x)$ modulo $p$, it follows that $\bar \Phi_n(x) \in \mathbb F_p[x]$ has a multiple root. But then $x^n-1$ is a multiple of $\bar \Phi_n(x)$ so would have a multiple root over $\mathbb F_p$, which is a contradiction since $p \nmid n$.

  So it must be the case that $\zeta^p$ is a root of $f(x)$, for every root $\zeta$ of $f(x)$. Write every integer $a$ coprime to $n$ as a product of primes $a = p_1p_2\cdots p_k$, then $\zeta^{p_1}$ is a root of $f(x)$, $\zeta^{p_1p_2} = (\zeta^{p_1})^{p_2}$ is a root of $f(x)$, etc. Then every primitive $n$th root of unity is a root of $f(x)$. So $f(x) = \Phi_n(x)$.
\end{proof}

\begin{corollary}
  \[ [\mathbb Q(\zeta_n): \mathbb Q] = \varphi(n) \]
\end{corollary}

\begin{proof}
  $\Phi_n(x)$ is the minimal polynomial for any primitive $n$th root of unity $\zeta_n$.
\end{proof}
\end{document}