\section{Binary classification}
\textbf{Binary Classification: Assign each data point to one of two classes.}

Examples:
\begin{itemize}
 \item Is there a face in this image?
\item Will this neuron spike in response to this stimulus?
\item Based on this brain-scan, does this patient have a given disease or not?
\item  Will this customer buy this product or not?
\item Is this person likely to be a democrat/republican?
\end{itemize}
Notation: we have data $D=\{(x_1, t_1),\ldots, (x_N, t_N)\}$, with $t_n=1$ if $x_n$ belongs to class $1$ and $t_n=-1$ if $x_n$ belongs to class $-1$.

\textbf{We focus on linear decision rules, also known as `linear discriminant functions'.}
\begin{figure}
	\centering
	\begin{subfigure}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=\textwidth]{./lecture6/LinClassification.pdf}
		\caption{Linear classification.}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=\textwidth]{./lecture6/NonLinClassification.pdf}
		\caption{Non-linear classification.}
	\end{subfigure}
	\caption{A classical dichotomous classification problem where the blue crosses belong to the positive class and the red circles belong to the negative class. We want to find a function which separates the two classes.}
\end{figure}

Of course, linear algorithms can be used together with \emph{nonlinear feature spaces} or \emph{nonlinear basis functions} in order to solve nonlinear classification problems!

\textbf{Linear discriminants separate the space by a hyperplane, and the parameters define its normal vector.}

\begin{itemize}
\item Decision function: $y(\xx)=\omega^\top \xx + \omega_o$
\item  Classification: \begin{align}
\mbox{if~}y(\xx)>0 & \mbox{~say $\xx$ belongs to class 1}\\
\mbox{if~}y(\xx)<0 & \mbox{~say $\xx$ belongs to class -1}\\
 \end{align}
\item The decision-surface has equation $y(\xx)=0$, and is a hyperplane of dimensionality $D-1$ (codimension).
\item  $\omega$ is the normal vector to the plane, and points into the positive class.
\item  $\omega_o$ determines the location of the decision-surface \\
	   changing $\omega_o$ changes the ratio between FPs and FNs
\item  $|y(\xx)|$ is proproptional to the perpendicular distance to the decision-surface (with factor $1$ if $|| \omega ||=1$).
\item the distance of a data point from the hyperplane reflects the certainty about the class membership.
\end{itemize}

\textbf{Multiple algorithms and methods exist for finding a good $\omega$.}
\begin{itemize}
\item Mis-classification rate $C(\omega)= \frac{1}{N} \sum_n \overbrace{\delta\left[y(\xx_n) =t_n\right]}^{\delta(x=0) = 1; \; \delta(x \not= 0) = 0}$ (i.e. average number of errors) difficult to optimize over $\omega$, and might have multiple solutions.
	\begin{itemize}
		\item This error function is not continuous and therfore not differentiable
		\item No convergence if there is no solution with zero errors
		\item Compare: Rosenblatt perceptron. The perceptron learning rule utilizes a 0-1-loss function and is guaranteed to converge in a finite number of steps as long as the data is linearly seperable. It will however not converge for overlapping data.
	\end{itemize}
\item  Many algorithms can be derived by replacing $C$ by another cost-function which can be optimized.
\item  Linear classification algorithms include Least-square classification, Fisher's linear Discriminant, Logistic regression, Support Vector Machines and Rosenblatts' perceptron.
\end{itemize}

\subsection{Least Square Classification}
\textbf{You already know one algorithm for linear classification: least square classification.}
\begin{itemize}
\item We have to fit the function $y(\xx)= \omega^\top \xx+ \omega_o $ to data.
\item  Simply do a linear regression from $\xx$ to $t$ by minimizing the sum-of-squared errors $\sum_n (y(\xx_n)-t_n)^2$.
\item  $\omega_{reg}=  \left(\sum_n x_n x_n^\top  \right)^{-1} \sum_n x_n t_n$
\item  Q: In what situations might this be a bad idea?
\end{itemize}

\begin{bbbox}{Q: In what situations might this be a bad idea?}
	In the case of a least-squares classification the hyperplane is mainly constrained by data points far away from the decision boundary. However if those data points are not perfectly aligned with the rest of the data points from the same class this can lead to a strong estimation error for the decision boundary as can be seen in figure \ref{fig:least_sq_class} left. A preferable measure contrains the hyperplane based on data points close to the decision boundary and neglects outliers. Compare: Support Vector Machine.
\end{bbbox}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure44a.pdf}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure44b.pdf}
	\end{subfigure}
	\caption{Bishop Figure 4.4}
	\label{fig:least_sq_class}
\end{figure}

\subsection{Fisher's linear discriminant}
\textbf{'Fisher's linear discriminant' is a classical and simple algorithm for linear classification}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure46a.pdf}
		\caption{Separartion of the projected class means}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure46b.pdf}
		\caption{Fisher linear discriminant.}
	\end{subfigure}
	\caption{Bishop Figure 4.6}
\end{figure}

\begin{itemize}
\item $\mathbf{m_+}= \frac{1}{N_+}\sum_{n \in C_+} x_n$ \hspace{.5cm} $\mathbf{m_{-}}= \frac{1}{N_-}\sum_{n \in C_{-}} x_n$
\item  Maximize projection-distance of class means
[projected mean/variance: on board]
$\omega_{simple} \propto \mathbf{m}_+-\mathbf{m}_-$
\end{itemize}

\begin{bbbox}{Maximize projection-distance of class means. The simple rule.}
	\begin{flalign*}
		&\text{Find } \omega \text{ such that} \\
		&d(m_+,m_-) \text{ is minimized} \\
		&\text{subject to } \|w\| = 1 \\
		&\text{where } d = \left( \omega^{\top} m_+ - \omega^{\top} m_- \right)^2 \\
		&\Rightarrow \omega_{opt} = \alpha \left(m_+ - m_- \right)
	\end{flalign*}
\end{bbbox}

\begin{itemize}
\item  Maximizing distance between means ignores that the projected variances might also be big.
\item  Fix:  Maximize the ratio of between-class variance to within-class variance ('signal to noise'). Fisher criterion
\begin{align}
	J_\omega = 2\frac{(m_+-m_-)^2}{s_+^2+s_-^2}
\end{align}
[Details and solution: on board]

\begin{bbbox}{Fisher's linear discriminant: Maximize the ratio of between-class variance and within-class variance.}
	\begin{align*}
		\text{Numerator: } N &= d\left( \omega^{\top}m_+, \omega^{\top}m_- \right) \\
		                     &= \left( \omega^{\top} m_+ - \omega^{\top} m_- \right)^2 \\
%		                     &=: (m_+-m_-)^2 \\
%		N &= \left( \omega^{\top} m_+ - \omega^{\top} m_- \right)^2 \\
		  &= \left( \omega^{\top} m_+ - \omega^{\top} m_- \right) \left( \omega^{\top} m_+ - \omega^{\top} m_- \right)^{\top} \\
		  &= \omega^{\top} (m_+-m_-) (m_+-m_-)^{\top} \omega \\
		  &= \omega^{\top} \underbrace{S_B}_{\text{between-class variance}} \omega \\
		S_+^2 &= \mbox{Var}	\left( \omega^{\top} x | x \in C_+ \right) \\
			  &= \omega^{\top} \mbox{Cov}	\left( x | x \in C_+ \right) \omega \\
        S_-^2 &= \omega^{\top} \mbox{Cov}	\underbrace{\left( x | x \in C_- \right)}_{\Sigma_-} \omega \\
        D &= \omega^{\top} \left[ \frac{1}{2} \Sigma_+ + \frac{1}{2} \Sigma_- \right] \omega \\
          &= \omega^{\top} \underbrace{S_W}_{\text{within-class variance}} \omega \\
        J(\omega) &= \frac{N}{D} = \frac{\omega^{\top} S_B \omega}{\omega^{\top} S_W \omega} \\
        \Rightarrow \omega_{opt} \propto S_W^{-1} \left(m_+ - m_- \right)
	\end{align*}
\end{bbbox}

$\omega_{lda}= \Sigma_w^{-1} (\mathbf{m}_+-\mathbf{m}_-)$ \\
However Fisher's linear discriminant is still sensitive to outliers.
\end{itemize}

%http://users.informatik.uni-halle.de/~hinnebur/Lehre/BN_seminar_web/bn_05_ag.pdf

\textbf{Aside: The multivariate Gaussian}
\begin{itemize}
\item Probability density function of $D$ dimensional Gaussian with mean $\mu$ an covariance $\Sigma$: \begin{align}p(x| \mu, \Sigma)&= (2\pi)^{-D/2}|\Sigma|^{-1/2} \exp \left(-\frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x-\mu) \right) \\
 \end{align}
\item  Maximum likelihood estimation of parameters: \begin{align}
\hat\mu= &\frac{1}{N}\sum_n x_n\mbox{~~(empirical mean)}\\
\hat \Sigma= &\frac{1}{N} \sum_n x_n x_n^\top- \hat\mu \hat \mu^\top\mbox{~~(empirical covariance)}
 \end{align}
\end{itemize}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.3\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure28a.pdf}
		\caption{Positive correlation.}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.3\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure28b.pdf}
		\caption{Independence but different variances.}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.3\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure28c.pdf}
		\caption{Independence and uni-variance.}
	\end{subfigure}
	\caption{Multivariate Gaussians with different covariance matrices. Bishop Figure 2.8}
\end{figure}

\textbf{A (super brief) primer on covariance matrices. (more details/intuition in second half of course?)}

\begin{figure}
	\includegraphics[width=.45\textwidth]{./lecture6/Figure27.pdf}
	\caption{Bishop Figure 2.7}
\end{figure}

\begin{itemize}
	\item  Covariance matrices are symmetric.
	\item  Diagonal entries: variances along coordinate-axes
	\item  Eigenvectors: principal axes of ellipsoid
	\item Eigenvalues:  variances along eigen-vectors
	\item Eigenvector with maximal/minimal eigen-value: Direction of maximal/minimal variance
	\item  Covariance matrices are `positive definite', i.e. all their eigenvalues are non-negative.
	\item  Most of this can be derived from $a^\top \mbox{Cov}(X) a= \mbox{Var}(a^\top X)$
\end{itemize}

\subsection{A generative model: Class-conditional Gaussians}
\textbf{A tale of two Gaussian: We can use a probablistic model of the data for classification}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure410a.pdf}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure410b.pdf}
	\end{subfigure}
	\caption{Bishop 4.10}
\end{figure}

\begin{itemize}
	\item Suppose that each of the two classes is modelled by a Gaussian: $x | x \in C_+ \sim \mathcal{N}\left(\mu_+, \Sigma_+\right)$, $x | x \in C_- \sim \mathcal{N}\left(\mu_-, \Sigma_-\right)$,
	\item ~[On board] Calculation of posterior class probabilities and decision criterion
	\item  If we assume $\Sigma_+=\Sigma_-$, we get $\omega_{gauss} \propto \Sigma_+^{-1} (\mathbf{m}_+-\mathbf{m}_-)$
	\item Note: We take the $t_n$ as given and built a model of $x_n | t_n$, contrast with linear regression, where we took $x_n$ as given and modelled $t_n |x_n$.
\end{itemize}

\begin{bbbox}{Probabilistic generative model: Gaussians}
We assume that data from each of the two classes is normally distributed  such that: \\
	\begin{flalign*}
	& P(x|t=1) = \mathcal{N}\left( x| \mu_+,\Sigma_+ \right) &\\
	& P(x|t=-1) = \mathcal{N}\left( x| \mu_-,\Sigma_- \right) &\\
	\end{flalign*}

If we know the prior probabilities $\pi$ for observing data from any of the two classes we can caculate the posterior probability of a data point belonging to each of the classes. This is by simply utilizing Bayes' theorem. \\
	\begin{flalign*}
	& P(t=1|x) = \frac{1}{Z} P(x|t=1) \underbrace{P(t=1)}_{\pi_+} = \frac{\pi_+}{Z} P(x|t=1) &\\
	& \text{where} &\\
	& Z = \pi_+ P(x | t = 1) + \pi_{-} P(x | t = -1) \text{ is a normalization constant} &\\
	& \text{and} &\\
	& P(t = 1 | x) + P(t = -1|x) = 1	&\\
	\end{flalign*}

We now use the log-odds to find the decision boundary for our classification problem. We assign data point $x$ to the positive class if $d(x) > 0$ and we will assign it to the negative class otherwise: \\
	\begin{flalign*}
		d(x) &=  \log \left[ \frac{P(t=1|x)}{P(t=-1|x)}\right] \\
		     &= C - \frac{1}{2} \left(x-\mu_+\right)^{\top} \Sigma_+^{-1} \left(x-\mu_+\right) \\
		     	 &\qquad {} + \frac{1}{2} \left(x-\mu_-\right)^{\top} \Sigma_-^{-1} \left(x-\mu_-\right) \\
		\textbf{Assume } \Sigma_+ = \Sigma_- = \Sigma  \\
		     &= C - \frac{1}{2} x^{\top} \Sigma^{-1} x + \frac{1}{2} \mu_+^{\top} \Sigma^{-1} x + \frac{1}{2} x^{\top} \Sigma^{-1} \mu_+ - \frac{1}{2} \mu_+^{\top} \Sigma^{-1} \mu_+ \\
		        &\qquad {} + \frac{1}{2} x^{\top} \Sigma^{-1} x - \frac{1}{2} \mu_-^{\top} \Sigma^{-1} x -\frac{1}{2} x^{\top} \Sigma^{-1} \mu_- + \frac{1}{2} \mu_-^{\top} \Sigma^{-1} \mu_- \\
		     &= C_2 + \mu_+^{\top} \Sigma^{-1} x - \mu_-^{\top} \Sigma^{-1} x \\
		     &= C_2 + \left( \mu_+ - \mu_- \right)^{\top} \Sigma^{-1} x \\
		     &= C_2 + \left[ \Sigma^{-1} \left( \mu_+ - \mu_- \right) \right]^{\top} x \\
		     &= \omega_0 + \omega^{\top}x
	\end{flalign*}
If we compare this result with our results for FDA and linear regression for classification we realize that all three approach lead to equivalent decision boundaries.
\end{bbbox}

\textbf{This approach directly generalizes to classification with unequal covariances and multi-class classification.}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure411a.pdf}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture6/Figure411b.pdf}
	\end{subfigure}
	\caption{Bishop 4.11}
\end{figure}

\begin{itemize}
\item Quadratic discriminant analysis: $\Sigma_+ \neq \Sigma_o$, decisison boundary is of form $y(\xx) = \xx^\top A \xx+\omega^\top \xx +\omega_o$
\item Multi-class: Assign each data-point to class with highest posterior probability (or calculate best assignment from cost-function).
\end{itemize}

\textbf{A simple nonlinear classifier can be constructed from kernel density estimates of the probability densities.}

\includegraphics[width=\textwidth]{./lecture6/NonLinClassification.pdf}
\begin{itemize}
	\item Idea: Once we have an estimate of the class-conditional densities $P(x| t=\pm 1)$, we can construct a rule from $d(x)=P(x| t=+ 1)-P(x| t=- 1)$.
	\item  Use \emph{kernel density estimation} to estimate  $P(x| t=\pm 1)$, i.e. place a `Gaussian bump' on each data-point:
	\begin{align}
		P(x| t=1) =\frac{1}{Z}\sum_{n_+} \exp\left( \frac{x-x_n}{\sigma}\right)^2
	\end{align}
\end{itemize}

This leads to a classifier of the form
\begin{align}
	d(x)= \sum_{n} \alpha  t_n \exp\left( \frac{x-x_n}{\sigma}\right)^2
\end{align}

\emph{Support vector machine with radial basis functions} has decision rule
\begin{align}
	d(x)=\sum_{n}  \alpha_n \exp\left( \frac{x-x_n}{\sigma}\right)^2
\end{align}

\subsection{One for the price of three.}
\begin{itemize}
	\item Today, you learned about three different algorithms for binary classification with linear decision rules.
	\item  One was based on a hack, the second one on a plausible (but ad-hoc) criterion, and the third one an a probababilistic model of the data.
	\item  All three algorithms are equivalent.
	\item  We showed that the Fisher discriminant and the probabilistic model based on two Gaussians have the same  decision criterion. In fact, it can be shown that linear regression has the same weights (Bishop 4.1.5)
%\item  The moral: Great motivations are great, but the actual algorithm matters, and it is important to check connections with other algorithms.
	\item The third motivation had immediate extensions to nonlinear algorithms and multi-class classification, and posterior probabilities.
	\item  Next week, we will learn an algorithm which actually is different, and usually better than the ones discussed today.
\end{itemize}


\chapter{Basic Statistics}

You live near a freeway, and someone asks you, ``How fast do cars on that freeway drive?''

You say ``Pretty fast.''

And they say, ``Can you be more specific?''

And you point your radar gun at a car, and say ``That one is going 32.131 meters per second.''

And they say, ``I don't want to know about that specific car. I want to know about all the cars.''

So, you spend the day beside the freeway measuring the speed of every
car that goes by. And you get a list of a thousand numbers. Here is part of the
list:

\begin{tabular}{c | c | c}
30.462 m/s  & 29.550 m/s & 29.227 m/s \\
37.661 m/s  & 27.899 m/s & 28.113 m/s \\
24.382 m/s & 35.668 m/s & 43.797 m/s \\
31.312 m/s & 37.637 m/s & 30.891 m/s
\end {tabular}

There are 12 numbers here. We say that there are 12 \textit{samples}.\index{samples}

\section{Mean}

We often talk about the \textit{average} of a set of samples, which is the
same as the \textit{mean}. To get the mean, sum up the
samples and divide that number by the number of samples.\index{mean}

The numbers in that table sum to $388.599$.  If you divide that by 12,
you find that the mean of those samples is 32.217 m/s.

We typically use the greek letter $\mu$ (``mu'') to represent the mean.

\begin{mdframed}[style=important, frametitle={Definition of Mean}]

If you have a set of samples $x_1, x_2, \ldots, x_n$, the mean is:

$$ \mu = \frac{1}{n} \sum_{i=1}^n x_i$$

\end{mdframed}

This may be the first time you are seeing a summation ($\sum$). The equation above is equivalent to:\index{summation symbol}

$$ \mu = \frac{1}{n} \left(x_1 + x_2 + \ldots + x_n\right)$$

\begin{Exercise}[title={Mean Grade}, label=grades_mean]

  Teachers often use the mean for grading. For example, if you took
  six quizzes in a class, your final grade might be the mean of the six
  scores. Find the mean of these six grades: 87, 91, 98, 65, 87, 100.

\end{Exercise}
\begin{Answer}[ref=grades_mean]

  $$\mu =\frac{1}{6} \left(87 + 91 + 98 + 65 + 87 + 100 \right) = 88$$

\end{Answer}

If you tell your friend ``I measured the speed of 1000 cars, and the
mean is 31.71 m/s'', your friend will wonder ``Are most of the speeds
clustered around 31.71? Or are they all over the place and just happen
to have a mean of 31.71?'' To answer this question we use variance.

\section{Variance}

\begin{mdframed}[style=important, frametitle={Definition of Variance}]

If you have $n$ samples $x_1, x_2, \ldots, x_n$ that have a mean of $\mu$, the \textit{variance} is defined to be:\index{variance}

$$v = \frac{1}{n}\sum_{i = 1}^{n} \left(x_i - \mu\right)^2$$
% ADD: Maybe connect to Chi-squared test
\end{mdframed}

That is, you figure out how far each sample is from the median, you
square that, and then you take the mean of all those squared
distances.

\begin{tabular} {c | c | c}

  $x$ & $x - \mu$ & $(x - \mu)^2$\\
  \hline
30.462 & -1.755 & 3.079 \\
29.550 & -2.667 & 7.111\\
29.227 & -2.990 & 8.938\\
37.661 & 5.444 & 29.642\\
27.899 & -4.318 & 18.642\\
28.113 & -4.104 & 16.839 \\
24.382 & -7.835 & 61.381 \\
35.668 & 3.451 & 11.912 \\
43.797 & 11.580 & 134.106\\
31.312 & -0.905 & 0.818\\
37.637 & 5.420 & 29.381\\
30.891 & -1.326 & 1.757\\
\hline
$\sum x = 386.599$ & & $\sum (x - \mu)^2 = 323.605$\\
mean = 32.217 & & variance = 26.967
\end{tabular}

Thus, the variance of the 12 samples is 26.967. The bigger the variances,
the farther the samples are spread apart; the smaller the variances, the closer
samples are clustered around the mean.

Notice that most of the data points deviate from the mu by 1 to 5
m/s. Isn't it odd that the variance is a big number like 26.967?
Remember that it represents the average of the squares. Sometimes, to
get a better feel for how far the samples are from the mean, we use
the square root of the variance, which is called \textit{the standard
  deviation}.

The standard deviation of your 12 samples would be $\sqrt{26.9677} =
  5.193$ m/s.
% ADD: Bell curve, KA: https://www.khanacademy.org/computer-programming/spin-off-of-galton-board-exploration/1930953307/embedded?embed=yes&article=yes&editor=no&buttons=no&author=no&width=400&height=400

The standard deviation is used to figure out a data point is an
outlier. For example, if you are asked``That car that just sped
past. Was it going freakishly fast?'' You might respond, ``No, it was
within a standard deviation of the mean.'' or ``Yes, it's speed was 2
standard deviations more than the mean. They will probably get a ticket.''
% ADD: Box and whiskers plot?

A singular $\mu$ usually represents the mean. $\sigma$ usually represents
the standard deviation. So $\sigma^2$ represents the variance.

\begin{Exercise}[title={Variance of Grades}, label=grades_variance]

  Now find the variance for your six grades. As a reminder, they were: 87, 91, 98, 65, 87, 100.

  What is your standard deviation?

\end{Exercise}
\begin{Answer}[ref=grades_variance]

  The mean of your grades is $88$.

  The variance, then is

  $$\sigma^2 = \frac{1}{6} \left((87 - 88)^2 + (91 - 88)^2 + (98 - 88)^2 + (61 - 88)^2 + (87 - 88)^2 + (100 - 88)^2 \right) = \frac{784}{6} = 65 \frac{1}{3}$$

  The standard deviation is the square root of that: $\sigma = 8.083$ points.

\end{Answer}

\section{Median}

Sometimes you want to know where the middle is. For example, you want
to know the speed at which half the cars are going faster and half are
going slower. To get the median, you sort your samples from smallest
to largest. If you have an odd number of samples, the one in the
middle is the median. If you have an even number of samples, we take
the mean of the two numbers in the middle.\index{median}
% KA: https://www.khanacademy.org/math/cc-sixth-grade-math/cc-6th-data-statistics/mean-and-median/v/statistics-intro-mean-median-and-mode

In our example, you would sort your numbers and find the two in the middle:

\begin{tabular}{c}
24.382\\
27.899\\
28.113\\
29.227\\
29.550\\
\hline
\textbf{30.462}\\
\textbf{30.891}\\
\hline
31.312\\
35.668\\
37.637\\
37.661\\
43.797\\
\end{tabular}

You take the mean of the two middle numbers: $(30.462 + 30.891)/2 =
30.692$.  The median speed would be 30.692 m/s.

Medians are often used when a small number of outliers majorly skew the
mean. For example, income statistics usually use the median income
because a few hundred billionares raise the mean a lot.

\begin{Exercise}[title={Median Grade}, label=grades_median]

  Find the median of your six grades: 87, 91, 98, 65, 87, 100.

\end{Exercise}
\begin{Answer}[ref=grades_median]

  In order the grades are 65, 87, 87, 91, 98, 100.  The middle two are 87
  and 91. The mean of those is 89. (Speed trick: The mean of two numbers is the
  number that is half-way between.)

 \end{Answer}

\section{Histograms}

A histogram is a bar chart that shows how many samples are in each
group. In our example, we group cars by speed. Maybe we count the
number of cars going between 30 and 32 m/s.  And then we count the
cars going between 32 and 34 m/2. And then we make a bar chart from
that data.\index{histograms}
% ADD: Have not explained histograms yet

Your 1000 cars would break up into these groups:

\begin{tabular}{ c | c }
0 - 2 m/s & 0 cars \\
2 - 4 m/s & 0 cars \\
4 - 6 m/s & 0 cars \\
\ldots & \ldots \\
20 - 22 m/s & 0 cars \\
22 - 24 m/s & 0 cars \\
24 - 26 m/s & 65 cars \\
26 - 28 m/s & 160 cars \\
28 - 30 m/s & 175 cars \\
30 - 32 m/s & 168 cars \\
32 - 34 m/s & 150 cars \\
34 - 36 m/s & 114 cars \\
36 - 38 m/s & 79 cars \\
38 - 40 m/s & 52 cars \\
40 - 42 m/s & 20 cars \\
42 - 44 m/s & 12 cars \\
44 - 46 m/s & 4 cars \\
46 - 48 m/s & 1 cars \\
48 - 50 m/s & 0 cars \\
\end{tabular}

Now we make a bar chart from that:

\includegraphics[width=\textwidth] {speed_histo.png}

Often a histogram will tell the story of the data. Here, you can see
that no one is going less than 24 m/s, but a lot of people travel at
30 m/s. There are a few people who travel over 40 m/s, but there are also a
couple of people who drive a lot faster than anyone else.

\section{Root-Mean-Squared}

Scientists have a mean-like statistic that they love. It is named
quadratic mean, but most just calls it Root-Mean-Squared or
RMS.

\begin{mdframed}[style=important, frametitle={Definition of RMS}]

If you have a list of numbers $x_1, x_2, \ldots, x_n$, their RMS
is \index{quadratic mean} \index{root-mean-squared} \index{RMS}

$$\sqrt{\frac{1}{n}\left( x_1^2 + x_2^2 + \ldots + x_n^2 \right)}$$

\end{mdframed}

You are taking the square root of the mean of squares of the samples,
thus the name Root-Mean-Squared.

Using your 12 samples:

\begin{tabular}{c |  c}
  $x$ & $x^2$ \\
  \hline
30.462 & 927.933 \\
29.550 & 873.203\\
29.227 & 854.218\\
37.661 & 1418.351\\
27.899 & 778.354\\
28.113 & 790.341\\
24.382 & 594.482\\
35.668 & 1272.206\\
43.797 & 1918.177\\
31.312 & 980.441\\
37.637 & 1416.544\\
30.891 & 954.254\\
\hline
\multicolumn{1}{r}{Mean of $x^2$} & {1064.875}\\
\multicolumn{1}{r}{RMS} & {32.632}
  \end{tabular}

Why is RMS useful? Let's say that all cars had the same mass $m$, and
you need to know what the average kinetic energy per car is. If you
know the RMS of the speeds of the cars is $v_{rms}$, the average kinetic energy for
each car is

$$k = \frac{1}{2}m v_{rms}^2$$

(You don't believe me? Let's prove it. Substitute in the RMS:

$$k = \frac{1}{2}m \sqrt{\frac{1}{n}\left( x_1^2 + x_2^2 + \ldots + x_n^2 \right)}^2$$

The square root and the square cancel each other out:

$$k = \frac{1}{2}m \frac{1}{n}\left( x_1^2 + x_2^2 + \ldots + x_n^2 \right)$$

Use the distributive property:

$$k = \frac{1}{n} \left( \frac{1}{2} m x_1^2 + \frac{1}{2}m x_2^2 + \ldots + \frac{1}{2}m x_n^2 \right)$$

That is all the kinetic energy divided by the number of cars, which is
the mean kinetic enegy per car. Quod erat demonstrandum! (That is a
Latin phrase that means ``which is what I was trying to
demonstrate''. You will sometimes see ``QED'' at the end of a long
mathematic proof.))

Now you are ready for the punchline: kinetic energy and heat are the
same thing. Instead of cars, heat is the kinetic energy of molecules
moving around. More on this soon.

Video: Mean, Median, Mode: https://www.youtube.com/watch?v=5C9LBF3b65s


%!TEX root=ClassNotes.tex

\section{Computing Indefinite Integrals}

In this section, we're interested in computing {\it indefinite integrals} or {\it antiderivatives}.
Indefinite integrals cannot be computed for all functions, for example, there is no closed form for the {\bf Gaussian Integral}
\begin{align*}
	\int \exp\left(-\dfrac{x^2}{2} \right) \: dx
\end{align*}
which is widely used in statistics, and the only way to compute it is using numerical methods.

There is no general method of computing indefinite integrals.
Instead, in this section we'll learn a few common tricks which you can combine in various ways to solve complicated problems.
There are two basic tools for computing indefinite integrals
\begin{enumerate}
	\item u-substitution and
	\item integration by parts.
\end{enumerate}
These are often combined with various algebraic and trig identities.

One thing that makes the computation of indefinite integrals far more complicated than that of derivatives, in practice, is that similar looking functions can have wildly different integrals. For example,
\begin{align*}
	\int 1/x \: dx              & = \ln x  + c                          \\
	\int 1/x^2 \: dx            & = -1/x + c                            \\
	\int 1/{(1+x^2)} \: dx      & = \tan^{-1} x + c                     \\
	\int 1/(1 - x^2) \: dx      & = 1/2 \cdot \ln ((1 + x)/(1 - x)) + c \\
	\int 1/\sqrt{1 - x^2} \: dx & = \sin^{-1} x + c
\end{align*}
So just by looking at a problem you cannot easily {\it guess} what it's integral is going to look like.

\subsection{u-substitution}
In terms of indefinite integrals, u-substitution takes the form
\begin{align}
	\label{eq:u-sub}
	\int f\left(g(x)\right) g'(x)\: dx = \int f(u) \: du + \mathrm{constant}
\end{align}
Ideally we want the integral on the right-hand side to be simpler than the one on the left-hand side.
We interpret this identity as having obtained the right-hand from the left-hand side by making the ``substitution''
\begin{align*}
	u  & = g(x)        \\
	du & = g'(x) \: dx
\end{align*}
This is just a {\it mnemonic} and doesn't have a concrete mathematical meaning (at least not a simple one).

\begin{remark}
	After finding the integral $\int f(u) \: du$ you should ALWAYS plug back $u = g(x)$ to get your final answer in terms of the original variable.
\end{remark}

\begin{exercise}
	Find the following indefinite integrals using u-substitution(s).
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int x e^{x^2} \: dx $
			\item $\int \dfrac{1}{x \ln x} \: dx$
			\item $\int \dfrac{e^{\sqrt{x}}}{\sqrt{x}} \: dx$
			\item $\int \dfrac{x}{4x^2+5} \: dx$
			      % \item $\int \dfrac{x}{2x^2 - 3} \: dx$
			\item $\int \dfrac{1}{x^2 + 2} \: dx$
			\item $\int \sec^2 x \tan^5 x \: dx$
			\item $\int \tan x \: dx$
			\item $\int \dfrac{e^x}{e^{2x} + 2e^x + 1} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

% \begin{exercise}
%   Sometimes there's no obvious reason to use u-substitution. Instead, we use u-substitution in the hope that integral simplifies.
%   \begin{enumerate}
% \item $\dfrac{e^x}{1 - e^{2x}}$
%     \item $\dfrac{1 - e^x}{1 + e^{x}}$
%     \item $\dfrac{e^{2x}}{\sqrt{1 + e^x}}$
%   \end{enumerate}
% \end{exercise}

\subsection{Integration by Parts}
Integration by parts is {\it sometimes} used to integrate products of functions (you should always try u-substitution first, unless the use of integration by parts is extremely obvious).
There are various ways of memorizing integration by parts, you should pick one that you find easy to remember.

One possible way to express it is as follows.
\begin{align}
	\int fg  =  fG - \int f' G  + \mathrm{constant}
\end{align}
where $G$ is any antiderivative of $g$.
In most problems, integration by parts is useful only if the function $f'G$ is simpler that the function $fg$.

\begin{example}
	For computing $\int x e^x \: dx$ by parts, we have (at least) two possible choices:
	\begin{enumerate}
		\item $f = x \mbox{ and } g = e^x$.
		      In this case,
		      \begin{align*}
			      f' G = e^x
		      \end{align*}
		\item $f = e^x \mbox{ and } g = x$.
		      In this case,
		      \begin{align*}
			      f' G = e^x \cdot x^2/2
		      \end{align*}
		      In the first case, the function $f' G$ is simpler than $fg$ and hence this is the decomposition we should use for applying integration by parts.
	\end{enumerate}
\end{example}

\begin{exercise}
	Find the following indefinite integrals using integration by parts.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int x e^x\: dx$
			\item $\int x^2 e^{x}\: dx$
			\item $\int x \sin x\: dx$
			\item $\int \ln x \: dx$ \hint{ Use $f = \ln x$ and $g = 1$.}
			\item $\int x \ln x\: dx$
			\item $\int x (\ln x)^2\: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\subsection{Trigonometric Integrals}
There are a LOT of tricks for computing integrals of functions involving trigonometric functions.
We'll only compute integrals of two kinds of functions:
\begin{align*}
	\sin^ m x \cos^n x \mbox{ \qquad and \qquad } e^{ax} \sin bx
\end{align*}

Let $m$ and $n$ be non-negative integers.
For integrating $\sin^ m x \cos^n x$ if
	{\bf either $m$ or $n$ is odd}
then we get the answer by a simple u-substitution. For example,
\begin{align*}
	\sin^ {2k+1} x \cdot \cos^n x
	 & = \sin^ {2k} x \cdot  \sin x \cdot  \cos^n x                 \\
	 & = \left( \sin^2 x \right)^k \cdot \sin x \cdot \cos^n x      \\
	 & = \left( 1 - \cos^2 x \right)^k \cdot \sin x \cdot  \cos^n x
\end{align*}
after which we can integrate using the u-substitution $ u = \cos x$.
\begin{exercise}
	Compute the following indefinite integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int \sin^3 x \cos^2 x \: dx$
			\item $\int \sin^2 x \cos^5 x \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

Let $m$ and $n$ be non-negative integers.
For integrating $\sin^ m x \cos^n x$ if
	{\bf both $m$ and $n$ are even}
then we use the double angle formulae,
\begin{align*}
	\sin^ {2k} x
	 & = \left(\sin^{2} x \right)^k
	 &                                           &  &
	\cos^ {2l} x
	 & = \left(\cos^{2} x \right)^l                   \\
	\\
	 & = \left(\dfrac{1 - \cos(2x)}{2} \right)^k
	 &                                           &  &
	 & = \left(\dfrac{1 + \cos(2x)}{2} \right)^l
\end{align*}
\begin{exercise}
	Compute the following indefinite integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int \sin^2 x \: dx $
			\item $\int \sin^2 x \cos^2 x \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

Let $a$, $b$ be real numbers.
Using the Euler's identity
\begin{align*}
	e^{ibx} = \cos bx + i \sin bx
\end{align*}
we get
\begin{align*}
	\int e^{ax} \cos bx \: dx & = \mbox{real part of } \int e^{ax} e^{ibx} \: dx      \\
	\int e^{ax} \sin bx \: dx & = \mbox{imaginary part of } \int e^{ax} e^{ibx} \: dx\end{align*}
\begin{exercise}
	Compute the following indefinite integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int e^{ax} \cos bx \: dx$
			\item $\int e^{ax} \sin bx \: dx $
		\end{enumerate}
	\end{multicols}
\end{exercise}

\subsection{Trigonometric Substitutions}

Trigonometric substitutions are extremely useful when {\it eliminating radicals} (among other things) because of the identities
\begin{align}
	\begin{split}
		\label{eq:trig_identities}
		\sin^2 x + \cos^2 x &= 1 \\
		\sec^2 x &= 1 + \tan^2 x
	\end{split}
\end{align}

\begin{example}
	If there is a term
	\begin{align*}
		\sqrt{1 + x^2}
	\end{align*}
	in our integral, then we can substitute $ x = \tan u$ so that
	\begin{align*}
		\sqrt{1 + x^2}
		 & = \sqrt{1 + \tan^2 u}                                         \\
		 & = \sec u              & \mbox{ by \eqref{eq:trig_identities}}
	\end{align*}
\end{example}

We'll need a few preliminary computations.
\begin{exercise}
	\begin{enumerate}
		\item Compute $(\sec x)'$.
		\item Show using the fundamental theorem of calculus that
		      \begin{align*}
			      \int \sec x \: dx & = \ln (\sec x + \tan x) + c  \\
			      \int \csc x \: dx & = -\ln (\csc x + \cot x) + c
		      \end{align*}
		\item {\bf Optional: } Compute the integrals $\int \sec x \: dx$ and $\int \csc x \: dx$ directly (without using the fundamental theorem).
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Compute the following integrals using trig substitutions.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int \dfrac{1}{\sqrt{1-x^2}}\: dx$
			\item $\int \dfrac{1}{\sqrt{1+x^2}} \: dx$
			\item $\int \dfrac{1}{\sqrt{x^2-1}} \: dx$
			\item $\int \dfrac{1}{x\sqrt{x^2-1}} \: dx$
			\item $\int \dfrac{1}{x\sqrt{1-x^2}} \: dx$
			\item $\int \dfrac{1}{x\sqrt{1+x^2}} \: dx$
			\item $\int \sqrt{1-x^2}\: dx$
			\item $\int x^3 \sqrt{1-x^2}\: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\begin{exercise}{{\bf (Optional)}}
	\begin{enumerate}
		\item Compute the integral
		      \begin{align*}
			      \int \sec^3 x \: dx
		      \end{align*}
		\item Compute the integral
		      \begin{align*}
			      \int \sqrt{1 + x^2} \: dx
		      \end{align*}
	\end{enumerate}
\end{exercise}

\subsection{Partial Fractions}
Partial fractions is a technique used to compute integrals of the form
\begin{align*}
	\int \dfrac{P(x)}{Q(x)} \: dx
\end{align*}
where $P(x)$ and $Q(x)$ are polynomials.
The higher the degree of the denominator $Q(x)$ the harder it is to compute the integral.\\

\subsubsection*{Linear Polynomials}
When the denominator $Q(x)$ is linear the integral can be computed easily using u-substitution.
\begin{exercise}
	Compute the following integrals
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{x^2+1}{x} \: dx$
			\item $ \int \dfrac{x^2 + 1}{x+1} \: dx$
			\item $ \int \dfrac{x}{2x-3} \: dx$
			\item $ \int \dfrac{x^2}{2x+2} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\subsubsection*{Quadratic with Repeated Roots}
When the denominator is a quadratic $Q(x) = ax^2 + bx + c$ there are 3 different methods for finding the integral, depending on what the roots of $Q(x)$ are.
\begin{align*}
	x^2 + ax + b \mbox{ has }
	\begin{cases}
		\mbox{ repeated roots } \\
		\mbox{ complex roots }  \\
		\mbox{ real non-repeated roots }
	\end{cases}
\end{align*}

The first case of {\bf repeated roots} is the easiest. In this case, our goal is to find a simple u-substitution to reduce the problem to
\begin{align*}
	\int \dfrac{R(u)}{u^2} \: du
\end{align*}
where $R(u)$ is some polynomial.
\begin{exercise}
	For each of the following problems, verify that the denominator has repeated roots. Then
	compute the integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{x}{4x^2 - 4x + 1} \: dx$
			\item $ \int \dfrac{x^2}{x^2 + 2x + 1} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\subsubsection*{Quadratic with Complex Roots}
When the denominator $Q(x) = ax^2 + bx + c$ has complex roots our goal is to find a u-substitution to reduce the problem to the integrals
\begin{align*}
	\int \dfrac{1}{u^2 + 1} \: du
	 &  & \mbox{ and }
	 &  &
	\int \dfrac{u}{u^2 + 1} \: du
\end{align*}

\begin{exercise}
	For each of the following problems, verify that the denominator has complex roots. Then
	compute the integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{1}{x^2 + 4} \: dx$
			\item $ \int \dfrac{3}{x^2 + 2x + 2} \: dx$
			\item $ \int \dfrac{3x}{x^2 + 2x + 2} \: dx$
			\item $ \int \dfrac{x}{4x^2 - 4x + 3} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\begin{exercise}If the degree of the numerator is $ \ge 2$ then we first have to do a long division to simplify the numerator. Compute the following integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{x^2}{x^2 + 4} \: dx$
			\item $ \int \dfrac{3x^3}{x^2 + 2x + 2} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\subsubsection*{Quadratic with Real Non-repeated Roots}
Finally, when the denominator $Q(x)$ has real non-repeated roots we need to use the method of partial fractions.\\

In this method we first need to find a factorization of $Q(x)$ as a product of linear terms, say $Q(x) = Q_1(x) \cdot Q_2(x)$ where both $Q_1(x)$ and $Q_2(x)$ are linear. If the degree of the numerator $P(x)$ is $\le 1$ then we can always write
\begin{align*}
	\dfrac{P(x)}{Q(x)}
	 & =
	\dfrac{A}{Q_1(x)} + \dfrac{B}{Q_2(x)}
\end{align*}
for some constants $A$ and $B$.
This is called the {\bf partial fraction decomposition} of $\frac{P(x)}{Q(x)}$. We find $A$ and $B$ by multiplying both sides by $Q(x)$ and comparing the coefficients on the left and right hand sides.

\begin{exercise}
	For each of the following problems, verify that the denominator has real non-repeated roots. Then
	compute the integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{1}{x^2 - 1} \: dx$
			\item $ \int \dfrac{4x}{x^2 - 4} \: dx$
			\item $ \int \dfrac{5}{x^2 - 2x} \: dx$
			\item $ \int \dfrac{3 x}{x^2 + x - 2} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\begin{exercise}As before if the degree of the numerator is $ \ge 2$ then we first need to do long division to simplify the numerator. Compute the following integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{3 x^2}{x^2 + x - 2} \: dx$
			\item $ \int \dfrac{x^2 - 1}{x^2 - 4} \: dx$
			\item $ \int \dfrac{4x^3 - 3x + 5}{x^2 - 2x} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\subsubsection*{Degree $\ge 3$}
When degree of the denominator $Q(x)$ is $\ge 3$, one can show the existence of a partial fraction decomposition using {\it abstract algebra}.
But the details are much more complicated and hard to do by hand.

Instead, feel free to use the internet to compute partial fraction decompositions. For example, go to \url{http://www.wolframalpha.com} and input
\begin{verbatim}
	partial fractions (x^2 - 2)/(x+1)^3
\end{verbatim}
for computing the partial fraction decomposition of $\dfrac{x^2 - 2}{(x+1)^3}$.

Once you have the partial fraction decomposition you can use u-substitution to compute the integral.

\begin{exercise}
	Compute the following integrals using partial fraction decompositions.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $ \int \dfrac{x^2 - 2}{(x+1)^3} \: dx$
			\item $ \int \dfrac{8}{3x^3 + 7x^2 + 4x} \: dx$
			\item $ \int \dfrac{x^3 + 8}{(x^2 - 1)(x - 2)} \: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}

\newpage
\subsection{Practice Problems}
We've learned several techniques for computing indefinite integrals. Of these
\begin{enumerate}
	\item Basic u-substitution
	\item Trigonometric substitutions (+ trig identities)
\end{enumerate}
are the tricky ones as there are a lot of possible substitutions to choose from. The other three
\begin{enumerate}[resume]
	\item Integration by Parts
	\item Trigonometric integrals
	\item Partial fractions
\end{enumerate}
are much easier to use.\\

 The following problems will require you to use all of the above techniques. You should not expect to {\it see} the solution right away, instead, systemically try different things until you reduce the problem to something that looks familiar.

\begin{exercise}
	Compute the following integrals.
	\begin{multicols}{2}
		\begin{enumerate}
			\item $\int \dfrac{e^x}{(e^x - 1)(e^x - 3)}\: dx$
			\item $\int \dfrac{1}{1 + e^x} \: dx$
			\item $\int e^{\sqrt{x}} \: dx$
			\item $\int \sin \sqrt{x}\: dx$
			\item $\int \dfrac{\sin^3 x}{\cos^2 x} \: dx$
			\item $\int \dfrac{1 - \sin x}{\cos^2 x}\: dx$
			\item $\int \dfrac{1}{1 + \sin x}\: dx$
			\item $\int \sqrt{1 + \cos 2x}\: dx$
			\item $\int \sec^3 x \tan x\: dx$
			\item $\int x \tan^{-1} x\: dx$
			\item $\int x^2 \tan^{-1} x\: dx$
			\item $\int \tan^{-1} {\sqrt{x}}\: dx$
			\item $\int \dfrac{x}{\sqrt{2 + 2x + x^2}} \: dx$
			\item $\int \dfrac{1}{\sqrt{2x - x^2}}\: dx$
			\item $\int \ln(1 + x^2)\: dx$
			\item $\int \tan^{2} x\: dx$
		\end{enumerate}
	\end{multicols}
\end{exercise}


\section{Linear Equations}

  A linear equation over with unknowns is represented by $\forall \{a_1, \ldots, a_n\}, b \in K,\ \ a_1 x_1 + a_2 x_2 + \cdots + a_n x_n = b$.
  A system of linear equations is a set of $L$'s, where $L$ is a linear equation as above.
  Linear equations form a vector space that can be scaled.

  Given a system of linear equations with unknowns, a solution vector $u = (a_1,\ldots, a_n)$ is something that simultaneously satisfies all of the linear equations.

  \subsection{Definitions}

    \begin{description}
      \item[Augmented Matrix]  Given a system of linear equations defined above, the augmented matrix is:\\
      $M = \left[\begin{array}{cc|c}
        a_{11} & \ldots & b_1 \\
        \vdots & \ddots & \vdots \\
        a_{m1} & \ldots & b_m
      \end{array}\right]$

      \item[Equivalent Systems of Linear Equations] Two system of linear equations are equivalent if they have the same system.
      More precisely, $M_1$ and $M_2$ are equivalent when the rows of one are linear combinations of the rows of the other.
      This means that systems of linear equations are equivalen when they are row equivalent.

      \item[Elementary Row Operation (ERO)] An ERO is an operation of either interchanging two rows, replacing a row with a multiple of itself, or replacing a row with the addition of itself and another row.

      \item[Row Echelon Form] Row echelon form consists of a matrix such that each leftmost nonzero entry of a row is to the right of the leftmost nonzero entry of each proceeding row, all zero rows are at the bottom of the matrix.

      \item[Row Canonical Form (Reduced Row Echelon) (Fully Reduced)] Row canonical form or full reduced matrix form is a matrix that satisfies row echelon form where every first entry in each row is 1.

      \item[Pivot] A pivot is the first nonzero entry in a row of a matrix in row ecehlon form.

      \item[Gaussian Elimination] A matrix can be converted into reduced row echelon form by performing row separations to put it into echelon form, dividing throughout so that every pivot is 1, and then using the 1 to zero out every space above pivots.

    \end{description}

  \subsection{Theorems}

    \subsubsection{Solution to Linear Equation}

      If $u$ is the solution to a linear equation where $L_1: x_1 + x_2 = 0; L_2 = x_1 - x_2 = 3; L_1 + L_2: 2x_1 = 3$ then $x_1 = 3/2, x_2 = -3/2$ by back substitution.

    \subsubsection{Non-Invertible Matrix}

      A matrix is not invertible if there exists some nonzero vector such that the matrix multiplied by the vector is 0.

    \subsubsection{Products of Invertible Matrices}

      A product of matrices is invertible if and only if each of the individual matrices are invertible.

    \subsubsection{A matrix is row equivalent to a unique matrix in RCF}

      Wellp... the subsection name sums it up.
      A proof is further below.

  \subsection{Proofs}

    \subsubsection{Every ERO has an inverse operation}

      BLURRY PAGE, GO BACK!

    \subsubsection{A product of a set of matrices is invertible if and only if each of the set of matrices is invertible}

      \begin{align}
        \forall invertible[A], \exists v \neq 0\ \ s.t.\ \ Av = 0 \\
        B = A^{-1} \\
        B(Av) = B \cdot [0] = [0] \\
        (BA)v = (A^{-1}A)v = Iv = v \implies v = 0
      \end{align}

      In words, since this proof can be somewhat confusing (or at least it confused me): if A is invertible there must be some nonzero vector such that Av is 0.
      Creating some matrix B that is the inverse of the vector A, should you multiply Av (0) by B it should be 0... or it could be v, depending on how you frame the problem.
      So v must be 0 and thus the matrix must be invertible.

      Another proof.
      Suppose a set of matrices A are all invertible and their product is B.
      If you repeatedly multiply both sides of this equality by a corresponding inverse of A to replace the set of matrices with the identity matrix, you will get $I = (A_n^{-1} \ldots A_1^{-1})B$.
      This can be reduced to $(A_n^{-1} \ldots A_1^{-1}) = B^{-1}$ by multiplying both sides by the inverse of B.
      Therefore, the inverse of B exists and the product of a set of invertible matrices is invertible.

      We can use this to show each element is invertible.
      If $A_1A_2\ldots A_n$ is invertible so is $(A_1\ldots A_{n-1})A_n$ by associativity.
      The expression has two terms and the term in paranetheses is invertible and $A_n$ is invertible if we know the second term.

    \subsubsection{A matrix is row equivalent to a unique matrix in RCF}

      We know that $M_1$ and $M_2$ represent equivalent linear systems if and only if these matrices are row equivalent, thus there is a sequence of EROs that can convert matrices to each other.
      $M_2$ can be transformed into a matrix $N$ in RCF by some sequence of EROS.
      $M_1$ can be transformed into a matrix $N$ in RCF by some sequence of EROS.
      All three of these matrices are equivalent.


\subsection{Euclidian division}

Euclidian division is the theory for any pair of natural numbers, we can divide one by the other and have a remainder less than the divisor. Formally:
\(\forall a \in \mathbb{N} ,\forall b \in \mathbb{N}^+ ,\exists q \in \mathbb{N},\exists r\in \mathbb{N} [(a=bq+r)\land (0\le r < b)]\)

Where \(\mathbb{N}^+\) refers to natural numbers excluding \(0\).

That is, every natural number \(a\) is a multiple \(q\) of any other natural number \(b\), plus another natural number \(r\) less than the other natural number \(b\).

These are unique. For each jump in \(q\), \(r\) falls by \(b\). As the range of \(r\) is \(b\) there is only one solution.

\(17=2.8+1\)

\(9=3.3+0\)