\subsection{How many unique operators are there?}

An arbitrary operator takes \(n\) inputs are returns \(T\) or \(F\).

With \(0\) inputs there is one posible permutation. For every additional input the number of possible permutations doubles. Therefore there are \(2^n\) possible permutations.

For the operator with one permutation there are two operators. For every additional permutation the number of operator doubles. Therefore there are \(2^{(2^n)}\) possible operations.

With \(0\) inputs, we need \(2\) different operators to cover all outputs. For \(1\) input we need \(4\) and for \(2\) inputs we need \(16\).

\subsection{We don't need \(0\)-ary operators}

There are two unique \(0\)-ary operators. One always returns \(T\) and the other always returns \(F\). These are already described.

\subsection{We need one unary operator}

For the operators with \(1\) input we have:

\begin{itemize}
\item one which always returns \(T\)
\item one which always returns \(F\)
\item one which always returns the same as the input
\item one which returns the opposite of the input
\end{itemize}

It is this last one, negation, shown as \(\neg \) and is of most interest.

\subsection{We can use a subset of binary operators}

The full list of binary operators are included below.

Of these, the first two are \(0\)-ary operators, and so are not needed. The next four are unary operators, and so are not needed.

The non-implications can be rewritten using negation.

\subsection{Brackets replace the need for n-ary operators}

N-ary operators contain \(3\) or more inputs.

N-ary operators can be defined in terms of binary operators.

As an example if we want an operator to return positive if all inputs are true, we can use:

\((\theta \land \gamma )\land \beta \)


\section{Function Reference}

\subsection*{abs($x$)}

Returns the absolute value or vector length of $x$.

{\color{blue}
\begin{verbatim}
X = (x,y,z)
abs(X)
\end{verbatim}
}

\noindent
$\left(x^2+y^2+z^2\right)^{1/2}$

\subsection*{adj($m$)}

Returns the adjunct of matrix $m$.
Adjunct is equal to determinant times inverse.

{\color{blue}
\begin{verbatim}
A = ((a,b),(c,d))
adj(A) == det(A) inv(A)
\end{verbatim}
}

\noindent
$1$

\subsection*{and($a,b,\ldots$)}

Returns 1 if all arguments are true (nonzero).
Returns 0 otherwise.

{\color{blue}
\begin{verbatim}
and(1=1,2=2)
\end{verbatim}
}

\noindent
$1$

\subsection*{arccos($x$)}

Returns the arc cosine of $x$.

{\color{blue}
\begin{verbatim}
arccos(1/2)
\end{verbatim}
}

\noindent
$\tfrac{1}{3}\pi$

\subsection*{arccosh($x$)}

Returns the arc hyperbolic cosine of $x$.

\subsection*{arcsin($x$)}

Returns the arc sine of $x$.

{\color{blue}
\begin{verbatim}
arcsin(1/2)
\end{verbatim}
}

\noindent
$\tfrac{1}{6}\pi$

\subsection*{arcsinh($x$)}

Returns the arc hyperbolic sine of $x$.

\subsection*{arctan($y,x$)}

Returns the arc tangent of $y$ over $x$.
If $x$ is omitted then $x=1$ is used.

{\color{blue}
\begin{verbatim}
arctan(1,0)
\end{verbatim}
}

\noindent
$\tfrac{1}{2}\pi$

\subsection*{arctanh($x$)}

Returns the arc hyperbolic tangent of $x$.

\subsection*{arg($z$)}

Returns the angle of complex $z$.

{\color{blue}
\begin{verbatim}
arg(2 - 3i)
\end{verbatim}
}

\noindent
$\arctan(-3,2)$

\subsection*{binding($s$)}

The result of evaluating a symbol can differ from the symbol's binding.
For example, the result may be expanded.
The {\tt binding} function returns the actual binding of a symbol.

{\color{blue}
\begin{verbatim}
p = quote((x + 1)^2)
p
\end{verbatim}
}

\noindent
$p=x^2+2x+1$

{\color{blue}
\begin{verbatim}
binding(p)
\end{verbatim}
}

\noindent
$(x+1)^2$

\subsection*{ceiling($x$)}

Returns the smallest integer greater than or equal to $x$.

{\color{blue}
\begin{verbatim}
ceiling(1/2)
\end{verbatim}
}

\noindent
$1$

\subsection*{check($x$)}

If $x$ is true (nonzero) then continue, else stop.
Expression $x$ can include the relational operators
\verb$=$,
\verb$==$,
\verb$<$,
\verb$<=$,
\verb$>$,
\verb$>=$.
Use the
\verb$not$
function to test for inequality.

{\color{blue}
\begin{verbatim}
A = 1
B = 1
check(A=B) -- stop here if A not equal to B
\end{verbatim}
}

\subsection*{circexp($x$)}

Returns expression $x$ with circular and hyperbolic functions
converted to exponentials.

{\color{blue}
\begin{verbatim}
circexp(cos(x) + i sin(x))
\end{verbatim}
}

\noindent
$\exp(ix)$

\subsection*{clear}

Clears all symbol definitions.

\subsection*{clock($z$)}

Returns complex $z$ in polar form with base of negative 1 instead of $e$.

{\color{blue}
\begin{verbatim}
clock(2 - 3i)
\end{verbatim}
}

\noindent
$13^{1/2}\,(-1)^{\arctan(-3,2)/\pi}$

\subsection*{conj($z$)}

Returns the complex conjugate of $z$.

{\color{blue}
\begin{verbatim}
conj(2 - 3i)
\end{verbatim}
}

\noindent
$2 + 3 i$

\subsection*{contract($a,i,j$)}

Returns tensor $a$ summed over indices $i$ and $j$.
If $i$ and $j$ are omitted then 1 and 2 are used.
The expression {\tt contract(m)} computes the trace of matrix $m$.

{\color{blue}
\begin{verbatim}
A = ((a,b),(c,d))
contract(A)
\end{verbatim}
}

\noindent
$a + d$

\subsection*{cos($x$)}

Returns the cosine of $x$.

{\color{blue}
\begin{verbatim}
cos(pi/4)
\end{verbatim}
}

\noindent
$\displaystyle \frac{1}{2^{1/2}}$

\subsection*{cosh($x$)}

Returns the hyperbolic cosine of $x$.

{\color{blue}
\begin{verbatim}
circexp(cosh(x))
\end{verbatim}
}

\noindent
$\tfrac{1}{2}\exp(-x)+\tfrac{1}{2}\exp(x)$

\subsection*{d($f,x$)}

Returns the partial derivative of $f$ with respect to $x$.

{\color{blue}
\begin{verbatim}
d(x^2,x)
\end{verbatim}
}

\noindent
$2x$

\bigskip
\noindent
Argument $f$ can be a tensor of any rank.
Argument $x$ can be a vector.
When $x$ is a vector the result is the gradient of $f$.

{\color{blue}
\begin{verbatim}
F = (f(),g(),h())
X = (x,y,z)
d(F,X)
\end{verbatim}
}

\noindent
$\displaystyle \begin{bmatrix}
\operatorname{d}(f(),x) & \operatorname{d}(f(),y) &  \operatorname{d}(f(),z)\\
\operatorname{d}(g(),x) & \operatorname{d}(g(),y) &  \operatorname{d}(g(),z)\\
\operatorname{d}(h(),x) & \operatorname{d}(h(),y) &  \operatorname{d}(h(),z)
\end{bmatrix}
$

\bigskip
\noindent
It is OK to use {\tt d} as a variable name.
It will not conflict with function {\tt d}.

\bigskip
\noindent
It is OK to redefine {\tt d} as a different function.
The function {\tt derivative}, a synonym for {\tt d},
can still be used to obtain a partial derivative.

\subsection*{defint($f,x,a,b$)}

Returns the definite integral of $f$ with respect to $x$
evaluated from $a$ to $b$.
The argument list can be extended for multiple integrals
as shown in the following example.

{\color{blue}
\begin{verbatim}
f = (1 + cos(theta)^2) sin(theta)
defint(f, theta, 0, pi, phi, 0, 2pi) -- integrate over theta then over phi
\end{verbatim}
}

\noindent
$\tfrac{16}{3}\pi$

\subsection*{denominator($x$)}

Returns the denominator of expression $x$.

{\color{blue}
\begin{verbatim}
denominator(a/b)
\end{verbatim}
}

\noindent
$b$

\subsection*{det($m$)}

Returns the determinant of matrix $m$.

{\color{blue}
\begin{verbatim}
A = ((a,b),(c,d))
det(A)
\end{verbatim}
}

\noindent
$a d - b c$

\subsection*{dim($a,n$)}

Returns the dimension of the $n$th index of tensor $a$.
Index numbering starts with 1.

{\color{blue}
\begin{verbatim}
A = ((1,2),(3,4),(5,6))
dim(A,1)
\end{verbatim}
}

\noindent
$3$

\subsection*{do($a,b,\ldots$)}

Evaluates each argument from left to right.
Returns the result of the final argument.

{\color{blue}
\begin{verbatim}
do(A=1,B=2,A+B)
\end{verbatim}
}

\noindent
$3$

\subsection*{dot($a,b,\ldots$)}

Returns the dot product of vectors, matrices, and tensors.
Also known as the matrix product.

{\color{blue}
\begin{verbatim}
-- solve for X in AX=B
A = ((1,2),(3,4))
B = (5,6)
X = dot(inv(A),B)
X
\end{verbatim}
}

\noindent
$\displaystyle \begin{bmatrix}-4\\ \tfrac{9}{2}\end{bmatrix}$

\subsection*{draw($f,x$)}

Draws a graph of $f(x)$.
Drawing ranges can be set with {\tt xrange} and {\tt yrange}.

{\color{blue}
\begin{verbatim}
xrange = (0,1)
yrange = (0,1)
draw(x^2,x)
\end{verbatim}
}

\subsection*{eval($f,x,a$)}

Returns expression $f$ evaluated at $x$ equals $a$.
The argument list can be extended for multivariate expressions.
For example,
\verb$eval(f,x,a,y,b)$
is equivalent to
\verb$eval(eval(f,x,a),y,b)$.

{\color{blue}
\begin{verbatim}
eval(x + y,x,a,y,b)
\end{verbatim}
}

\noindent
$a+b$

\subsection*{exp($x$)}

Returns the exponential of $x$.

{\color{blue}
\begin{verbatim}
exp(i pi)
\end{verbatim}
}

\noindent
$-1$

\subsection*{expcos($z$)}

Returns the cosine of $z$ in exponential form.

{\color{blue}
\begin{verbatim}
expcos(z)
\end{verbatim}
}

\noindent
$\displaystyle \tfrac{1}{2}\exp(iz)+\tfrac{1}{2}\exp(-iz)$

\subsection*{expcosh($z$)}

Returns the hyperbolic cosine of $z$ in exponential form.

{\color{blue}
\begin{verbatim}
expcosh(z)
\end{verbatim}
}

\noindent
$\displaystyle \tfrac{1}{2}\exp(-z)+\tfrac{1}{2}\exp(z)$

\subsection*{expsin($z$)}

Returns the sine of $z$ in exponential form.

{\color{blue}
\begin{verbatim}
expsin(z)
\end{verbatim}
}

\noindent
$\displaystyle -\tfrac{1}{2}i\exp(iz)+\tfrac{1}{2}i\exp(-iz)$

\subsection*{expsinh($z$)}

Returns the hyperbolic sine of $z$ in exponential form.

{\color{blue}
\begin{verbatim}
expsinh(z)
\end{verbatim}
}

\noindent
$\displaystyle -\tfrac{1}{2}\exp(-z)+\tfrac{1}{2}\exp(z)$

\subsection*{exptan($z$)}

Returns the tangent of $z$ in exponential form.

{\color{blue}
\begin{verbatim}
exptan(z)
\end{verbatim}
}

\noindent
$\displaystyle \frac{i}{\exp(2iz)+1}-\frac{i\exp(2iz)}{\exp(2iz)+1}$

\subsection*{exptanh($z$)}

Returns the hyperbolic tangent of $z$ in exponential form.

{\color{blue}
\begin{verbatim}
exptanh(z)
\end{verbatim}
}

\noindent
$\displaystyle -\frac{1}{\exp(2z)+1}+\frac{\exp(2z)}{\exp(2z)+1}$

\subsection*{factorial($n$)}

Returns the factorial of $n$.
The expression {\tt n!} can also be used.

{\color{blue}
\begin{verbatim}
20!
\end{verbatim}
}

\noindent
$2432902008176640000$

\subsection*{float($x$)}

Returns expression $x$ with rational numbers and integers converted to
floating point values.
The symbol {\tt pi} and the natural number are also converted.

{\color{blue}
\begin{verbatim}
float(212^17)
\end{verbatim}
}

\noindent
$\displaystyle 3.52947\times 10^{39}$

\subsection*{floor($x$)}

Returns the largest integer less than or equal to $x$.

{\color{blue}
\begin{verbatim}
floor(1/2)
\end{verbatim}
}

\noindent
$0$

\subsection*{for($i,j,k,a,b,\ldots$)}

For $i$ equals $j$ through $k$ evaluate $a$, $b$, etc.

{\color{blue}
\begin{verbatim}
for(k,1,3,A=k,print(A))
\end{verbatim}
}

\noindent
$A=1$\\
$A=2$\\
$A=3$

\bigskip
\noindent
Note: The original value of $i$ is restored after {\tt for} completes.
If symbol {\tt i} is used for index variable $i$
then the imaginary unit is overridden in the scope of {\tt for}.

\subsection*{i}

Symbol {\tt i} is initialized to the imaginary unit $\sqrt{-1}$.

{\color{blue}
\begin{verbatim}
exp(i pi)
\end{verbatim}
}

\noindent
$-1$

\bigskip
\noindent
Note: It is OK to clear or redefine {\tt i} and use the symbol for something else.

\subsection*{imag($z$)}

Returns the imaginary part of complex $z$.

{\color{blue}
\begin{verbatim}
imag(2 - 3i)
\end{verbatim}
}

\noindent
$-3$

\subsection*{inner($a,b,\ldots$)}

Returns the inner product of vectors, matrices, and tensors.
Also known as the matrix product.

{\color{blue}
\begin{verbatim}
A = ((a,b),(c,d))
B = (x,y)
inner(A,B)
\end{verbatim}
}

\noindent
$\displaystyle
\begin{bmatrix}
ax+by\\
cx+dy
\end{bmatrix}
$

\bigskip
\noindent
Note: {\tt inner} and {\tt dot} are the same function.

\subsection*{integral($f,x$)}

Returns the integral of $f$ with respect to $x$.

{\color{blue}
\begin{verbatim}
integral(x^2,x)
\end{verbatim}
}

\noindent
$\displaystyle \tfrac{1}{3}x^3$

\subsection*{inv($m$)}

Returns the inverse of matrix $m$.

{\color{blue}
\begin{verbatim}
A = ((1,2),(3,4))
inv(A)
\end{verbatim}
}

\noindent
$\displaystyle
\begin{bmatrix}
-2 & 1\\
\tfrac{3}{2} & -\tfrac{1}{2}
\end{bmatrix}
$

\subsection*{j}

Set {\tt j=sqrt(-1)} to use {\tt j} for the imaginary unit instead of {\tt i}.

{\color{blue}
\begin{verbatim}
j = sqrt(-1)
1/sqrt(-1)
\end{verbatim}
}

\noindent
$-j$

\subsection*{last}

The result of the previous calculation is stored in {\tt last}.

{\color{blue}
\begin{verbatim}
212^17
\end{verbatim}
}

\noindent
$3529471145760275132301897342055866171392$

{\color{blue}
\begin{verbatim}
last
\end{verbatim}
}

\noindent
$last=3529471145760275132301897342055866171392$

\bigskip
\noindent
Symbol {\tt last} is an implied argument when a function has no argument list.

{\color{blue}
\begin{verbatim}
float
\end{verbatim}
}

\noindent
$\displaystyle 3.52947\times10^{39}$

\subsection*{log($x$)}

Returns the natural logarithm of $x$.

{\color{blue}
\begin{verbatim}
log(x^y)
\end{verbatim}
}

\noindent
$y\log(x)$

\subsection*{mag($z$)}

Returns the magnitude of complex $z$.
Function {\tt mag} treats undefined symbols as real while {\tt abs} does not.

{\color{blue}
\begin{verbatim}
mag(x + i y)
\end{verbatim}
}

\noindent
$\displaystyle (x^2+y^2)^{1/2}$

\subsection*{not($x$)}

Returns 0 if $x$ is true (nonzero).
Returns 1 otherwise.

{\color{blue}
\begin{verbatim}
not(1=1)
\end{verbatim}
}

\noindent
$0$

\subsection*{numerator($x$)}

Returns the numerator of expression $x$.

{\color{blue}
\begin{verbatim}
numerator(a/b)
\end{verbatim}
}

\noindent
$a$

\subsection*{or($a,b,\ldots$)}

Returns 1 if at least one argument is true (nonzero).
Returns 0 otherwise.

{\color{blue}
\begin{verbatim}
or(1=1,2=2)
\end{verbatim}
}

\noindent
$1$

\subsection*{outer($a,b,\ldots$)}

Returns the outer product of vectors, matrices, and tensors.
Also known as the tensor product.

{\color{blue}
\begin{verbatim}
A = (a,b,c)
B = (x,y,z)
outer(A,B)
\end{verbatim}
}

\noindent
$\displaystyle
\begin{bmatrix}
a x & a y & a z\\
b x & b y & b z\\
c x & c y & c z
\end{bmatrix}
$

\subsection*{pi}

Symbol for $\pi$.

{\color{blue}
\begin{verbatim}
exp(i pi)
\end{verbatim}
}

\noindent
$-1$

\subsection*{polar($z$)}

Returns complex $z$ in polar form.

{\color{blue}
\begin{verbatim}
polar(x - i y)
\end{verbatim}
}

\noindent
$\displaystyle (x^2+y^2)^{1/2}\exp(i\arctan(-y,x))$

\subsection*{power}

Use \verb$^$ to raise something to a power.
Use parentheses for negative powers.

{\color{blue}
\begin{verbatim}
x^(-2)
\end{verbatim}
}

\noindent
$\displaystyle \frac{1}{x^2}$

\subsection*{print($a,b,\ldots$)}

Evaluate expressions and print the results.
Useful for printing from inside a {\tt for} loop.

{\color{blue}
\begin{verbatim}
for(j,1,3,print(j))
\end{verbatim}
}

\noindent
$j=1$\newline
$j=2$\newline
$j=3$

\section*{product($i,j,k,f$)}

For $i$ equals $j$ through $k$ evaluate $f$.
Returns the product of all $f$.

{\color{blue}
\begin{verbatim}
product(j,1,3,x + j)
\end{verbatim}
}

\noindent
$\displaystyle x^3+6x^2+11x+6$

\bigskip
\noindent
Note: The original value of $i$ is restored after {\tt product} completes.
If symbol {\tt i} is used for index variable $i$
then the imaginary unit is overridden in the scope of {\tt product}.

\subsection*{quote($x$)}

Returns expression $x$ without evaluating it first.

{\color{blue}
\begin{verbatim}
quote((x + 1)^2)
\end{verbatim}
}

\noindent
$\displaystyle (x+1)^2$

\subsection*{rank($a$)}

Returns the number of indices that tensor $a$ has.

{\color{blue}
\begin{verbatim}
A = ((a,b),(c,d))
rank(A)
\end{verbatim}
}

\noindent
2

\subsection*{rationalize($x$)}

Returns expression $x$ with everything over a common denominator.

{\color{blue}
\begin{verbatim}
rationalize(1/a + 1/b + 1/2)
\end{verbatim}
}

\noindent
$\displaystyle \frac{2a+ab+2b}{2ab}$

\bigskip
\noindent
Note:
\verb$rationalize$
returns an unexpanded expression.
If the result is assigned to a symbol, evaluating the symbol will expand the result.
Use
\verb$binding$
to retrieve the unexpanded expression.

{\color{blue}
\begin{verbatim}
f = rationalize(1/a + 1/b + 1/2)
binding(f)
\end{verbatim}
}

\noindent
$\displaystyle \frac{2a+ab+2b}{2ab}$

\subsection*{real($z$)}

Returns the real part of complex $z$.

{\color{blue}
\begin{verbatim}
real(2 - 3i)
\end{verbatim}
}

\noindent
2

\subsection*{rect($z$)}

Returns complex $z$ in rectangular form.

{\color{blue}
\begin{verbatim}
rect(exp(i x))
\end{verbatim}
}

\noindent
$\displaystyle \cos(x)+i\sin(x)$

\subsection*{run({\it file})}

Run script {\it file}.
Useful for importing function libraries.

{\color{blue}
\begin{verbatim}
run("Downloads/EVA.txt")
\end{verbatim}
}

\noindent
Note: {\it file} must be in the Downloads folder due to security requirements for apps distributed on the Mac App Store.

\subsection*{simplify($x$)}

Returns expression $x$ in a simpler form.

{\color{blue}
\begin{verbatim}
simplify(sin(x)^2 + cos(x)^2)
\end{verbatim}
}

\noindent
1

\subsection*{sin($x$)}

Returns the sine of $x$.

{\color{blue}
\begin{verbatim}
sin(pi/4)
\end{verbatim}
}

\noindent
$\displaystyle \frac{1}{2^{1/2}}$

\subsection*{sinh($x$)}

Returns the hyperbolic sine of $x$.

{\color{blue}
\begin{verbatim}
circexp(sinh(x))
\end{verbatim}
}

\noindent
$\displaystyle -\tfrac{1}{2}\exp(-x)+\tfrac{1}{2}\exp(x)$

\subsection*{sqrt($x$)}

Returns the square root of $x$.

{\color{blue}
\begin{verbatim}
sqrt(10!)
\end{verbatim}
}

\noindent
$\displaystyle 720\; 7^{1/2}$

\subsection*{stop}

In a script, it does what it says.

\subsection*{sum($i,j,k,f$)}

For $i$ equals $j$ through $k$ evaluate $f$.
Returns the sum of all $f$.

{\color{blue}
\begin{verbatim}
sum(j,1,5,x^j)
\end{verbatim}
}

\noindent
$\displaystyle x^5+x^4+x^3+x^2+x$

\bigskip
\noindent
Note: The original value of $i$ is restored after {\tt sum} completes.
If symbol {\tt i} is used for index variable $i$
then the imaginary unit is overridden in the scope of {\tt sum}.

\subsection*{tan($x$)}

Returns the tangent of $x$.

{\color{blue}
\begin{verbatim}
simplify(tan(x) - sin(x)/cos(x))
\end{verbatim}
}

\noindent
0

\subsection*{tanh($x$)}

Returns the hyperbolic tangent of $x$.

{\color{blue}
\begin{verbatim}
circexp(tanh(x))
\end{verbatim}
}

\noindent
$\displaystyle -\frac{1}{\exp(2x)+1}+\frac{\exp(2x)}{\exp(2x)+1}$

\subsection*{test($a,b,c,d,\ldots$)}

If argument $a$ is true (nonzero) then $b$ is returned, else if $c$ is true then $d$ is returned, etc.
If the number of arguments is odd then the final argument is returned if all else fails.
Expressions can include the relational operators
\verb$=$,
\verb$==$,
\verb$<$,
\verb$<=$,
\verb$>$,
\verb$>=$.
Use the
\verb$not$
function to test for inequality.
(The equality operator
\verb$==$
is available for contexts in which
\verb$=$
is the assignment operator.)

{\color{blue}
\begin{verbatim}
A = 1
B = 1
test(A=B,"yes","no")
\end{verbatim}
}

\noindent
yes

\subsection*{trace}

Set {\tt trace=1} in a script to print the script as it is evaluated.
Useful for debugging.

{\color{blue}
\begin{verbatim}
trace = 1
\end{verbatim}
}

\noindent
Note:
The
\verb$contract$
function is used to obtain the trace of a matrix.

\subsection*{transpose($a,i,j$)}

Returns the transpose of tensor $a$ with respect to indices $i$ and $j$.
If $i$ and $j$ are omitted then 1 and 2 are used.
Hence a matrix can be transposed with a single argument.

{\color{blue}
\begin{verbatim}
A = ((a,b),(c,d))
transpose(A)
\end{verbatim}
}

\noindent
$\displaystyle
\begin{bmatrix}
a & c\\
b & d
\end{bmatrix}
$

\bigskip
\noindent
Note:
The argument list can be extended for multiple transpose operations.
The arguments are evaluated from left to right.
For example,
\verb$transpose(A,1,2,2,3)$
is equivalent to
\verb$transpose(transpose(A,1,2),2,3)$.

\subsection*{unit($n$)}

Returns an $n$ by $n$ identity matrix.

{\color{blue}
\begin{verbatim}
unit(3)
\end{verbatim}
}

\noindent
$\displaystyle
\begin{bmatrix}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{bmatrix}
$

\subsection*{zero($i,j,\ldots$)}

Returns a null tensor with dimensions $i$, $j$, etc.
Useful for creating a tensor and then setting component values.

{\color{blue}
\begin{verbatim}
A = zero(3,3)
for(k,1,3,A[k,k]=k)
A
\end{verbatim}
}

\noindent
$\displaystyle
A=
\begin{bmatrix}
1 & 0 & 0\\
0 & 2 & 0\\
0 & 0 & 3
\end{bmatrix}
$


\subsection{Prime numbers and composite numbers}

\subsubsection{Definition}

A prime number is a number which does not have any divisors other than \(1\) and itself.

By convention we do not refer to \(0\) or \(1\) as prime numbers.

\subsubsection{Identifying prime numbers}

Divisors must be smaller than the number. As a result it is easy to identify early prime numbers, as we can try to divide by all preceding numbers.

\subsubsection{Examples of prime numbers}

\([2, 3 5, 7, 11, 13,...]\)

\subsubsection{Composite numbers}

Composite numbers are numbers that are  made up through the multiplication of other numbers.

They are not prime.


\subsection{Finite state Markov chains}

\subsubsection{Transition matrices}

This shows the probability for moving between discrete states.

We can show the probability of being in a state by multiplying the vector state by the transition matrix.

\(Mv\)

\subsubsection{Time-homogenous Markov chains}

For time-homogenous Markov chains the transition matrix is independent of time.

For these we can calculate the probability of being in any given state in the future:

\(M^nv\)

This becomes independent of v as we tend to infinity. The initial starting state does not matter for long term probabilities.

How to find steady state probability?

\(Mv=v\)

The eigenvectors! With associated eigenvector \(1\). There is only one eigenvector. We can find it by iteratively multiplying any vector by \(M\).


\subsection{Herfindahl–Hirschman Index (HHI)}

The index is the sum of each firm’s market share squared.

For a monopolist this is \(1\), for a completely competitive market it is \(0\), or \(\dfrac{1}{n}\).

\(H=\sum_{i=1}^ns^2_i\)

To normalise this between \(0\) and \(1\) we can use:

\(H*=\dfrac{H-\dfrac{1}{n}}{1-\dfrac{1}{n}}\)


\subsection{Green's theorem}

We have a curve \(C\) on a plane.

Inside this is region \(D\).

We have two functions: \(L(x,y)\) and \(M(x,y)\) defined on the region and curve.

\(\oint_C (L dx + M dy)=\int \int_D (\dfrac{\delta M}{\delta x}-\dfrac{\delta L}{\delta y})dx dy\)


\subsection{Multinomial distribution}

The mass function for the binomial case is:

\(f(x)=\dfrac{n!}{x!(n-x)!}p^k(1-p)^{n-k}\)

\subsection{The multinomial distribution}

This generalises the binomial distribution where there are more than \(2\) outcomes.

\(f(x_1,...,x_n)=\dfrac{n!}{\prod_i x_i!}\prod_i p_i^{x_i}\)


\section{Page Rank}
	\textbf{Page Rank} is a link analysis algorithm based on the Web graph (pages as nodes and hyper-links as edges), taking the rank value to indicate an importance of a particular page, which is defined recursively and	depends on the number of all pages that link to it (in-links).

	\subsection{The Simplest Model}
	The page in web can be regard as a node $A$ in a digraph $\mathcal G=(\mathcal V,\mathcal E)$. If the page $A$ have a hyper-link to page $B$, then there is a edge $A\rightarrow B$ in $\mathcal E$. For example, the web show in Figure \eqref{fig:fournodesweb} can be represented by a digraph $\mathcal G=(\mathcal V,\mathcal E)$ with
	\begin{gather*}
	\mathcal{V} =\{A,B,C,D\}\\
	\mathcal{E} =\{A\rightarrow B, A\rightarrow C,A\rightarrow D,B\rightarrow A,\\ B\rightarrow D,C\rightarrow A,D\rightarrow B,D\rightarrow C\}
	\end{gather*}
	\begin{figure}[h]
		\centering
		\includegraphics[width=0.7\linewidth]{fournodesweb}
		\caption{A web witch has 4 nodes}
		\label{fig:fournodesweb}
	\end{figure}

	Suppose there is a person Jack who is suffering the Internet, and at timing $t$, he is browsing page $A$. When he wants to leave page $A$, he will randomly click a hyper-link in page $A$ and go to browse next page and the possibility of clicking each page is the same. As an example in Figure \eqref{fig:fournodesweb}, each possibility from $A$ to $B,C,D$ are $\dfrac 13$. (If the page have no out-links, we can think it have a out-link to itself.) If we do that for each nodes, then we will have a transform matrix $M(\mathcal G)$. The transform matrix of the \eqref{fig:fournodesweb} is
	\begin{equation*}
		\begin{matrix}
		 to\backslash from&A&B&C&D\\
		A& &1/2 &1 &\\
		B&1/3 & & &1/2\\
		C&1/3 & & &1/2\\
		D&1/3 &1/2 & &
		\end{matrix}
	\end{equation*}
	i.e.
	\begin{equation*}
	\mathbf M=\mathbf M(\mathcal{G})=
	\begin{pmatrix}
	 &1/2 &1 &\\
1/3 & & &1/2\\
1/3 & & &1/2\\
1/3 &1/2 & &
	\end{pmatrix}.
	\end{equation*}

	Now, we assume there are a large number of people, and at the timing $t=0$, each page has the same amount of people browsing it, i.e. for every page there are $\dfrac 1{|\mathcal V|}$ people browsing it at timing $t=0$. We use a vector $\mathbf v_0 =(\dfrac 1{|\mathcal V|},...,\dfrac 1{|\mathcal V|})^T$ to represent the initial state.

	In the next timing $t=1$, every one randomly clicking a hyper-link in the page which he or she is browsing. Then the distribution of the people at $t=1$ is $$\mathbf v_1=\mathbf M\mathbf v_0.$$ By parity of reasoning, we can obtain the distribution of the people at $t=\mu$ is  $$\mathbf v_\mu=\mathbf M^\mu\mathbf v_0.$$

	We can regard this process is a Markov process, so the convergence of $\mathbf v_\mu$ is equivalent to the digraph $\mathcal G$ is \textbf{strongly connected}. So we have the following theorem:

	\begin{theorem}
		 $\mathbf v_\mu$ is convergent $\Leftrightarrow$ $\mathcal G$ is strongly connected
	\end{theorem}
	If we get the convergence result $\mathbf v_\infty$, the $i$th component $v_{\infty i}$ measures the importance of the $i$th page. Because the bigger $v_{\infty i}$ is, the more people will browse this page.
	Besides, by the theory of power method, we know if $\mathbf v_\mu$ is convergent, then the $\mathbf v_\infty$ is the corresponding eigenvector of the max eigenvalue of $\mathbf M$.
	\subsection{More General Models}
	However the $\mathcal G$ of a real Internet web is impossible to be strongly connected, because there is always some pages have no out-links. Besides, if a page only have the out-link to itself, the result $\|\mathbf v_\infty\|$ may have no use value.

	Now we suppose Jack randomly click a hyper-link in the page with probability $\alpha$ or randomly input a new URL of a page with probability $1-\alpha$, so the iteration relation becomes:
	\begin{equation}\label{alpha}
	\mathbf v_{\mu+1}=(1-\alpha)\mathbf 1 +\alpha \mathbf M \mathbf v_\mu
	\end{equation}
	Because of $\|\mathbf v_0\|_1=1$ and the sum of each column of $\mathbf M$ is equal to 1, we can obtain $\|\mathbf v_\mu\|_1=1$ for any $\mu$. And the spectral radius of $\alpha \mathbf M$ is $$\rho(\alpha \mathbf M)=\alpha\rho(\mathbf M)\leq\alpha ||\mathbf M||_1=\alpha <1.$$ So the iteration \eqref{alpha} must be convergent, and $\mathbf v_\infty$ satisfy$$ 	\mathbf v_{\infty}=(1-\alpha)\mathbf 1 +\alpha \mathbf M \mathbf v_\infty.$$
	i.e.
	$$\mathbf v_\infty = (1-\alpha)(1-\alpha \mathbf M)^{-1}\mathbf 1 $$


\section{Physical Interpretations of the Derivative}
As we've seen, the idea derivative is fundamentally about continuous change.
This idea makes the derivative very useful for describing physical situations.

\begin{example}
		Find the rate of change of the area of a circle with respect to its radius in meters.
		Find the rate of change of the volume of a sphere with respect to its radius in meters.
		What are these quantities (with appropriate units) when $r=5\text{m}$?
\end{example}
\begin{answer}
	Starting with the area of a circle,
	\begin{align*}
		A &= \pi r^2
		\dd{A}{r} = 2\pi r.
	\end{align*}

	You might recognize this as the formula for the circumference of a circle.\\

	Starting with the volume of a sphere,
	\begin{align*}
		V &= \frac{4}{3}\pi r^3 \\
		\dd{V}{r} &= 4\pi r^2.
	\end{align*}

	You might recognize this as the formula for the surface area of a sphere.\\

	When $r=5\text{m}$,
	\begin{align*}
		\dd{A}{r}\biggr\rvert_{r=5\text{m}} &= 2\pi\left(5\text{m}\right) = 10\pi\text{m}. \\
		\dd{V}{r}\biggr\rvert_{r=5\text{m}} &= 4\pi\left(5\text{m}\right)^2 = 100\pi\text{m}^2.
	\end{align*}
\end{answer}

\subsection{Displacement, Velocity, and Acceleration}
If we have an object whose position is determined by a single variable, like time $t$, then we can model the position as a function.
\begin{equation*}
	s = f(t).
\end{equation*}
The displacement over some interval of length $\Delta t$, would be
\begin{equation*}
	\Delta s = f(t + \Delta t) - f(t).
\end{equation*}
The average velocity over this interval would be
\begin{equation*}
	\bar{v} = \frac{f(t + \Delta t) - f(t)}{\Delta t}.
\end{equation*}
As $\Delta t$ approaches 0, we see that $\bar{v}$ is exactly the definition of the derivative of $f$ with respect to $t$: the instantaneous velocity.
\begin{equation*}
	v(t) = \dd{s}{t} = \lim_{\Delta t \to 0}{\frac{f(t + \Delta t) - f(t)}{\Delta t}}.
\end{equation*}
We can go through the same steps to derive that instantaneous acceleration is the derivative of instantaneous velocity with respect to $t$.
\begin{equation*}
	a(t) = \dd{v}{t} = \dd{{}^2s}{t^2} = \lim_{\Delta t \to 0}{\frac{v(t + \Delta t) - v(t)}{\Delta t}}.
\end{equation*}
Although in everyday language we might use the terms speed and velocity interchangeably, speed is defined as the absolute value of velocity, meaning it is a scalar quantity while velocity is a vector quantity.
\begin{equation*}
	\text{Speed} = \abs{v(t)} = \biggr\lvert \dd{s}{t} \biggr\rvert.
\end{equation*}

Modeling the motion of free-falling bodies was one of the earliest motivations for discovering calculus.
Through experiments and applications of physics theory, we know that the height of a falling body when dropped from initial height $h_0$ meters is modeled by
\begin{equation*}
	s(t) = h_0 - \frac{1}{2}gt^2,
\end{equation*}
where $t$ is the time in seconds since the object was released and $g = 9.81m/s^2$ is the acceleration due to gravity near Earth.

\begin{example}
	A ball is dropped from an initial height of 100 meters.
	How long does it take for the ball to hit the ground?
	What speed is the ball traveling when it hits the ground?
\end{example}
\begin{answer}
	In this case, $h_0 = 100\text{m}$.
	So,
	\begin{equation*}
		s(t) = 100\text{m} - \frac{1}{2}gt^2.
	\end{equation*}

	We want to solve for $t$ where $s(t)=0$.
	\begin{equation*}
		t = \sqrt{\frac{200\text{m}}{g}} \approx 4.515\text{s}.
	\end{equation*}

	We take take the derivative of $s$ with respect to $t$ and then take the absolute value to get the speed.
	\begin{align*}
		v(t) &= \dd{}{t}s(t) = -gt. \\
		\text{Speed}(t) &= \abs{v(t)} = gt.
	\end{align*}

	Plugging in the time we got when the ball hits the ground,
	\begin{equation*}
		\text{Speed}_{\text{Ground}} \approx g(4.515\text{s}) \approx 44.294\text{m/s}.
	\end{equation*}
\end{answer}


\section{Elementary Equations and Elementary Matrices}
\begin{definition}
    An elementary column operation on an $m\times n$ matrix $A$ are one of the followings:\\
    (i) Swap columns $i,j$ with $i\neq j$.\\
    (ii) Multiply the entire column $i$ by $\lambda\in F\setminus\{0\}$.\\
    (iii) Add $\lambda$ times column $i$ to column $j$ where $\lambda\in F$.
\end{definition}
We can do row operations in a analogous (transposed) way.
Something remarkable is that these operations are invertible.
Instead of find the inverses one-by-one, we realise these operations via the action of elementary matrices.
Let $E_{ij}$ be the matrix with $1$ on the $i,j$ entry and zero anywhere else, we have:
\begin{definition}[Elementary Matrices]
    The elementary matrices are $T_{ij}=I-E_{ii}-E_{jj}+E_{ij}+E_{ji}$ for $i\neq j$, $M_{i,\lambda}=I+(\lambda-1)E_{ij}$ for $\lambda\neq 0$ and $C_{i,j,\lambda}=I+\lambda E_{ij}$.
\end{definition}
Then, we easily see that $T_{ij}$ corresponds to column (row) operation (i), $M_{i,\lambda}$ to operation (ii) and $C_{i,j,\lambda}$ to operation (iii) via the operation of multiplying $A$ with the correspondinng matrix from the right (left).
\begin{example}
    $$\begin{pmatrix}
        1&2\\
        3&4
    \end{pmatrix}\begin{pmatrix}
        0&1\\
        1&0
    \end{pmatrix}=\begin{pmatrix}
        2&1\\
        4&3
    \end{pmatrix}$$
\end{example}
\begin{proof}[Constructive Proof of Proposition \ref{eqv_form}]
    It suffices to show that we can get from any matrix to
    $$\left( \begin{array}{c|c}
        I_r&0\\
        \hline
        0&0
    \end{array} \right)$$
    via elementary column and row operations.
    Start with a matrix $A$.
    If $A=0$ then we are done.
    Otherwise pick $a_{ij}=\lambda\neq 0$ and swap rows $i$ and $1$ and then columns $j$ and $1$, after which $\lambda$ is at position $1,1$.
    Then multiply column $1$ by $\lambda^{-1}$.
    So we get $1$ at position $1,1$.
    Now we clean up row $1$ and column $1$ via operation (iii) (both row and column).
    Afterwards we can perform the same procedure on the submatrix by removing the first row and column.
    By induction we can get the desired form at the end.
\end{proof}
There are a few variations on these row and column operations.
The first one is Gauss' pivot algorithm.
If one use only row operations, then one will reach the ``row echelon form'' (which we will define later) in the following way:
Assume $a_{i1}\neq 0$ for some $i$.
Otherwise just move on by deleting the first zero columns.
Then swap rows $i$ and $1$ and divide row $1$ by $\lambda=a_{i1}$ to get $1$ at position $1,1$ and use (iii) to clean up the first column.
And one can move on with the same method to the submatrix removing the first row and first column.
Do this repeatedly and at the end we can get a matrix satisfying:\\
1. For any $i$ here exists $k(i)\ge i$ such that $a_{ij}=0$ for any $j<k(i)$.\\
2. $k(i)$ is increasing in $i$.\\
3. Row $k(i)$ equals $e_{k(i)}$.\\
And matrices satisfying these conditions are called matrices in row echelon form.
Note that the operations above is exactly what we will get when solving a linear system of equations.
So this can be an algorithm of doing that, which is now known as Gauss' pivot algorithm (or Gaussian elimination).\\
Another variation is the following:
\begin{lemma}
    We can obtain the identity matrix from any invertible square matrix via column operations only.
\end{lemma}
By transpose, we can replace column operations by row operations.
\begin{proof}
    We argue by induction on the $k$ where we can guarantee to transform $A$ to the form
    $$\begin{pmatrix}
        I_k&0\\
        \ast&\ast
    \end{pmatrix}$$
    The initial case is obvious.
    Suppose we can do this for some $k$, we shall show that we can do this for $k+1$.
    Now there must be some $j>k$ such that $a_{k+1,j}=\lambda\neq0$.
    Otherwise the vector $e_{k+1}$ is not in the span of the column vectors of $A$, contradiction.
    So we swap columns $k+1$ and $j$ and then divide column $k+1$ by $\lambda$.
    This gets us $1$ at position $k+1,k+1$.
    Then we can clean up the row $k+1$ by this $1$, which completes the induction process.
\end{proof}
This immediately provides an algorithm (a quite cost-effective one) for computing the inverse.
As one can see, this algorithm is analogous to the algorithm of solving a nonsingular linear system.
Also,
\begin{proposition}
    Any invertible matrix is a product of elementary matrices.
\end{proposition}
\begin{proof}
    Writing the operations in the preceding lemma as a product of elementary matrices representing the operations gives the inverse of that matrix.
    But any invertible matrix is the inverse of its own inverse.
\end{proof}


\section{Related Rates}
In related rates problems, we generally have two related functions and want to know and answer questions about the rate of change of one function given that we know the rate of change of the other. The same problem solving steps as in modeling and optimization apply, but we'll usually be taking derivatives using implicit differentiation with respect to some variable like time.

\begin{example}
	Let $A$ be the area of a square with side length $x$.
	Assume that $x$ varies with time.
	How are $\dd{A}{t}$ and $\dd{x}{t}$ related?
	At a certain instant, the sides are 3 feet and growing at a rate of 2 feet per minute.
	How quickly is the area changing at this instant?
\end{example}
\begin{answer}
	Starting with the area of the square and implicitly differentiating with respect to $t$,
	\begin{align*}
		A &= x^2 \\
		\dd{A}{t} = 2x\dd{x}{t}.
	\end{align*}

	When $x=3\text{ft}$ and $\dd{x}{t}=3\text{ft/min}$,
	\begin{equation*}
		\dd{A}{t} = 2(3\text{ft})(3\text{ft/min}) = 12\text{ft$^2$/min}.
	\end{equation*}
\end{answer}

\begin{example}
	The top of a 13 foot ladder propped against a vertical wall begins falling towards the ground at 12ft/s.
	When the top of the ladder is 5 feet off the ground, how quickly is the bottom of the ladder moving away from the wall?
	How is the angle between the ladder and the ground changing?
\end{example}
\begin{answer}
	Let $h$ be the height of the top of the ladder of the ground.
	Then $\dd{h}{t} = -12\text{ft/s}$.
	Let $b$ the the distance from the base of the ladder to the wall.
	We can relate $b$ and $h$ using the Pythagorean Theorem, where the 13-foot long ladder is the hypotenuse.
	\begin{equation*}
		b^2 + h^2 = 13^2.
	\end{equation*}

	We can also use this relationship to see that when $h=5\text{ft}$, $b=12\text{ft}$.
	Implicitly differentiating,
	\begin{equation*}
		2b\dd{b}{t} + 2h\dd{h}{t} = 0.
	\end{equation*}

	Plugging in what we know and solving for $\dd{b}{t}$,
	\begin{align*}
		2(12\text{ft})\dd{b}{t} + 2(5\text{ft})(-12\text{ft/s}) &= 0 \\
		24\text{ft}\dd{b}{t} &= 120\text{ft$^2$/s} \\
		\dd{b}{t} &= 5\text{ft/s}.
	\end{align*}

	So, the base of the ladder is moving away from the wall at a rate of 5ft/s.
	Let $\theta$ be the angle between the ladder and the ground.
	We can use $\sin$ to relate $\theta$ to $b$.
	\begin{equation*}
		13\sin{\theta} = b.
	\end{equation*}

	Implicitly differentiating,
	\begin{equation*}
		13\text{ft}\cos{(\theta)}\dd{\theta}{t} = \dd{b}{t}.
	\end{equation*}

	When $\cos$ is adjacent divided by hypotenuse, so $\cos{\theta} = 12/13$.
	Plugging in what we know and solving for $\dd{\theta}{t}$,
	\begin{align*}
		13\text{ft}(12/13)\dd{\theta}{t} &= -12\text{ft/s} \\
		\dd{\theta}{t} &= -1\text{/s}.
	\end{align*}

	So, the angle between the ladder and ground is decreasing at at rate of 1 rad/s.
\end{answer}

\begin{example}
	Grain is is poured at a rate of 10ft$^3$/min and falls into a cone-shaped pile whose bottom radius is half its altitude.
	How fast will the circumference of the base be increasing when the pile is 8 ft tall?
\end{example}
\begin{answer}
	Let $h$ the the altitude of the cone.
	At the instant we care about $h=8\text{ft}$.
	Let $r$ be the bottom radius of the cone.
	At the instant we care about, $r=h/2=4\text{ft}$.
	Let $V$ be the volume of the cone.
	We know that $\dd{V}{t}=10\text{ft$^3$/s}$.
	We can relate these three quantites using the formula for the volume of a cone.
	\begin{equation*}
		V = \frac{1}{3}\pi r^2 h.
	\end{equation*}

	Since we know that $2r = h$, we can simplify to get rid of $h$.
	\begin{equation*}
		V = \frac{2}{3}\pi r^3
	\end{equation*}

	Implicitly differentiating,
	\begin{equation*}
		\dd{V}{t} = 2\pi r^2 \dd{r}{t}.
	\end{equation*}

	We know the formula for the circumference $C$ of the circular base.
	\begin{equation*}
		C = 2\pi r.
	\end{equation*}

	Implicitly differentiating,
	\begin{equation*}
		\dd{C}{t} = 2\pi \dd{r}{t}.
	\end{equation*}

	We can substitute into our equation involving $\dd{V}{t}$.
	\begin{equation*}
		\dd{V}{t} = r^2\dd{C}{t}.
	\end{equation*}

	Plugging in what we know,
	\begin{align*}
		10\text{ft}^3\text{/s} &= (4\text{ft})^2\dd{C}{t} \\
		\dd{C}{t} &= \frac{5}{8}\text{ft/s}.
	\end{align*}
\end{answer}


\chapter{Linear Maps, Kernals and Range}

\begin{definition}
	Let $U$ and $V$ be vector spaces over the same field $F$. A mapping $T:U \rightarrow V$ is linear if 	$ \forall u_1, u_2 \in U, \forall \lambda \in F$
	\begin{center}
 		$T(u_1 + u_2) = T(u_1) + T(u_2)$\\
 		$T(\lambda u_1) = \lambda T(u_1)$
	\end{center}

\end{definition}

The set of all linear mappings from $U \rightarrow V$ is denoted by $\mathcal{L}(U,V)$. If $U=V$, then linear mappings are denoted by $\mathcal{L}(U)$, which is a set of linear mappings from a set onto itself. \\

\textbf{Examples}
\begin{itemize}
	\item $T: \mathcal{C}[a, b] \rightarrow \Re$, $f \rightarrow \int_{a}^{b} f(x) dx$. Integration is linear operator since it does not matter whether you add two functions and then integrate or vice versa. Also, we can pull out scalar multiples out of the integrals.
	\item $D: \mathcal{C}^{\infty} [a, b] \rightarrow \mathcal{C}^{\infty} [a, b]$, $f \rightarrow f'$. The derivative of the sum of functions is the same as taking the sum and then derivatves. \\
\end{itemize}

\begin{definition}
	$T \in \mathcal{L}(U, V)$. Then the kernal of $T$ (null space) is defined as :
	\begin{center}
		$ker(T) := null(T) := \{u \in U | Tu = 0\}$
	\end{center}

\end{definition}

\begin{proposition}
	$ker(T)$ is a subspace of $U$.
\end{proposition}

$T$ is injective iff $ker(T) = \{0\}$. Means if we take two input points in the input space then they are always mapped to different points in the output space.
So 0 is always an element in the kernal since a linear map always maps 0 to 0. Thus the kernal is never empty. The smallest kernal of any vector space is just the 0 vector.

\begin{definition}
	The range of $T$ (image of $T$) is defined as:
	\begin{center}
		$range(T) := Image(T) := \{Tu | u \in U\}$
	\end{center}
\end{definition}

\begin{proposition}
	The range is always a subspace of $V$
\end{proposition}

$T$ is surjective iff $range(T) = V$

\begin{definition}
	$V' \subset V$ where $V'$ is any set. The pre-image of $V'$ is defined as:
	\begin{center}
		$T^{-1} (V') := \{u \in U | Tu \in V'\}$
	\end{center}
\end{definition}

\begin{proposition}
	If $V' \subset V$ is a subspace of V, then $T^{-1}(V')$ is a subspace of $U$.
\end{proposition}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\textheight]{ch-3-img-1}
	\caption{Kernal and range of a $T: U \rightarrow V$}
\end{figure}

We can see in the figure above that a part of $U$ which is mapped to 0 is known as the kernal of tha map $T$. The range lives in $V$ and the kernal lives in $U$.

\begin{theorem}
	Let $V$ be finite dimensionalm $W$ be any vector space and $T \in \mathcal{L}(V,W)$. Let $u_1,...,u_n$ be a basis of $ker(T) \subset V$. Let $w_1,...,w_m$ be a basis of the $range(T) \subset W$. Then $u_1,...,u_n$, $T^{-1}(w_1),...,T^{-1}(w_m)$ forms a basis of $V$. In particular , $dim(V) = dim(ker(T)) + dim(range(T))$
\end{theorem}

\begin{proof}
	Denote $T^{-1}(w_1) := z_1,...,T^{-1}(w_m) := z_m$. If $\{u_1,...u_n,z_1,...z_m\}$ for a basis of $V$, then they must span the whole space $V$ and be linearly independent. Let's do each step by step. \\
	Step 1: Prove that $V \subset span\{u_1,...u_n,z_1,...z_m\}$ \\
	Let $v \in V$, and $Tv \in range(T)$. Since we know the basis of $range(T)$,
	\begin{align*}
		\exists \lambda_1,...,\lambda_m: Tv &= \lambda_1 w_1 + ... + \lambda_m w_m \\
											&= \lambda_1 T(z_1) + ... + \lambda_m T(z_m) \\
											&= T(\lambda_1 z_1) + ... + T(\lambda_m z_m)
	\end{align*}
$ \implies Tv - T(\lambda_1z_1 + ... + \lambda_mz_m) = T(v - (\lambda_1z_1 + ... + \lambda_mz_m)) =  0$  \\
$\implies (v - (\lambda_1z_1 + ... + \lambda_mz_m)) \in kernal(T)$ since this vector is mapped to 0 by the linear mapping. Since we also know the basis of the kernal any vector in the kernal can be written as:
\begin{align*}
(v - (\lambda_1z_1 + ... + \lambda_mz_m)) &= \mu_1 u_1 + ... + \mu_n u_n \\
v &= \lambda_1 z_1 + ... + \lambda_m z_m + \mu_1 u_1 + ... + \mu_n u_n
\end{align*}
Step 2 : Prove that $u_1,...,u_n, z_1,...,z_m$ are linearly independent. \\
Assume that $\mu_1 u_1 + ... \mu_n u_n + \lambda_1 z_1 + ... + \lambda_m z_m  = 0$ \\
If these vectors are linearly independent then all the coefficients will simultaneously go to zero.
\begin{align*}
	\lambda_1 w_1 + ... + \lambda_m w_m &= \lambda_1 T(z_1) + ... + \lambda_m T(z_m) \\
	&= \lambda_1 T(z_1) + ... + \lambda_m T(z_m) + \mu_1 T(u_1) + ... + \mu_n T(u_n) \\
	&= T(\lambda_1 z_1 + ... \lambda_m z_m + \mu_1 u_1 + ... + \mu_n u_n) = T(0) = 0
\end{align*}
We can add $ \mu_1 T(u_1) + ... + \mu_n T(u_n)$ to the expression since it goes to zero by definition of a kernal. Everything inside the operator is zero due to the assumption at the beginning of step 2.
Since $\lambda_1 w_1 + ... + \lambda_m w_m = 0$ and $w_1,...,w_m$ is a basis
$\implies \lambda_1 = ... = \lambda_m = 0$ \\
Now $\mu_1 u_1 + ... + \mu_n u_n = 0$ since we have already proved the other half of the assumption to be zero. Since $u_1,...,u_n$ is the basis of the kernal $\implies \mu_1=...\mu_n=0$.
\end{proof}

\begin{proposition}
	Let $T \in \mathcal{L}(V, W)$ and $V, W$ be finite dimensional, then the following statements are equivalent.
	\begin{itemize}
		\item T is injective
		\item T is surjective
		\item T is bijective
	\end{itemize}
\end{proposition}

This proposition does not hold in infinite dimensional spaces.


\section{Point Mass Distributions}

If the entire probability is concentrated at a single point $\alpha$, then that can be expressed in terms of the Kronecker Delta in the discrete case and the Dirac Delta function in the continuous case.

\subsection{Kronecker Delta $\delta_{\alpha}$}
In the discrete case:

\begin{equation}
\delta_{\alpha} = \left\{\begin{array}{l} 1\mathrm{\ if\ }x = \alpha \\ 0\mathrm{\ else}\end{array} \right.
\end{equation}

\subsection{Dirac Delta Function $\delta(x-\alpha)$}
In the continuous case:

\begin{equation}
\int_A \delta(x-\alpha) \mathrm{d}x = \left\{\begin{array}{l} 1\mathrm{\ if\ }\alpha \in A \\ 0\mathrm{\ else}\end{array} \right.
\end{equation}

The Dirac Delta function is actually a distribution with extremely useful properties that I ought to write up.

\section{Uniform Distributions}

\subsection{Discrete Uniform $\mathrm{Uniform(1,k)}$}
With $k>0$ be some integer, the PMF is:

\begin{equation}
f(x) = \left\{\begin{array}{l} 1/k \mathrm{\ if\ }x \in \{1,...,k\} \\ 0\mathrm{\ else}\end{array} \right.
\end{equation}

The CDF is:

\begin{equation}
F(x)= \left\{\begin{array}{cl}
0& x < 1\\
\frac{x}{k}& 1\leq x \leq k\\
1& x>k
\end{array} \right.
\end{equation}

% continuous uniform
\subsection{Continuous Uniform $\mathrm{Uniform(a,b)}$}

With $a,b\in\mathbb{R}$ and $b>a$, the PDF is:

\begin{equation}
f(x) = \left\{\begin{array}{l} 1/(b-a) \mathrm{\ if\ }x \in [a,b] \\ 0\mathrm{\ else}\end{array} \right.
\end{equation}

The CDF is:

\begin{equation}
F(x)= \left\{\begin{array}{cl}
0& x < a\\
\frac{x-a}{b-a}& a\leq x \leq b\\
1& x>b
\end{array} \right.
\end{equation}

% bernoulli
\section{Bernoulli Processes}
A Bernoulli process is a number of discrete trials with binary outcome, for example a series of coinflips. This is typically framed in terms of success $X=1$ with probability $p$ and failure $X=0$ with probability $1-p$. The trials are independent from each other. Thinking of the process as a sequence in time, this means that the process is memoryless: the number of successes or time since the last success have no bearing on the future. Thinking about the process as a sequence in space, the successes are independently scattered over a grid like randomly flipped bits in a string of bits. That is, the events are uniformly distributed.

\subsection{Bernoulli $\mathrm{Bernoulli(p)}$}
The PMF of a single binary outcome $X=1$ with probability $p$ and $X=0$ with probability $1-p$. The PMF is:

\begin{equation}
f(x) = p^x (1-p)^{1-x} =  \left\{\begin{array}{cl}
p& x = 1\\
(1-p)& x=0
\end{array} \right.
\end{equation}

For $x\in\{0,1\}$ and $p\in[0,1]$.

% binomial
\subsection{Binomial $\mathrm{Binomial}(n,p)$}
The binomial distribution is the PMF for the number of successes with probability $p$ among $n$ trials. That is, if $X_i \sim \mathrm{Bernoulli}(p)$, then $X = \sum_i^n X_i$ has distribution:

\begin{equation}
f(x) = {n \choose x} p^x (1-p)^{n-x}
\end{equation}

It follows that if $X_1 \sim \mathrm{Binomial}(n_1,p)$ and $X_2 \sim \mathrm{Binomial}(n_2,p)$, then $X_1 + X_2 \sim \mathrm{Binomial}(n_1+n_2,p)$. The binomial distribution can be interpreted of the probability that there will be $x$ successes among $n$ draws with replacement.

% geometric distribution
\subsection{Geometric $\mathrm{Geom}(p)$}
The PMF for the number of Bernoulli trials with parameter $p\in(0,1)$ is given by:

\begin{equation}
f(x) = p(1-p)^{x-1}
\end{equation}

In the time picture, it models the number of intervals $x$ until the first success occurs, or, equivalently, the number of trials between two successes. In the space picture, it models the distance between successes that are independently scattered on a grid.

% negative binomial distribution
\subsection{Pascal, Negative Binomial $\mathrm{NB}(r,p)$}
The negative binomial distribution with parameters $r$ and $p$ gives the sum of $r$ geometric random variables with parameter $p$. In the time picture, if $X \sim \mathrm{NB}(r,p)$, $X$ gives the probability for the number of failures in a sequence of Bernoulli trials until there are $r$ successes. In the space picture, it the probability for the width of the interval of a grid between $r$ successes (not counting the spots taken up by the successes). Its PMF is given by:

\begin{equation}
f(x) = {x+r-1\choose x} p^r (1-p)^x
\end{equation}

For $r=1$, the distribution is the same as the geometric distribution with parameter $p\rightarrow 1-p$. The negative binomial distribution gives the probability that, when drawing with replacement, it will take $x$ failures until there have been $r$ successes, where each success has probability $p$.

% bernoulli without replacement
\section{Bernoulli Processes 	Without Replacement}
Bernoulli Processes were a sequence of independent trials, which can be thought of as a sequence of samples with replacement. The "hyper"- distributions treat the analogous case where samples are not replacemed.

% hypergeometric distribution
\subsection{Hypergeometric $\mathrm{Hypergeom(N,K,n)}$}
The hypergeometric distribution gives the probability for the number of successes when drawing $n$ times from a population of $N$, of which $K$ correspond to successes.

\begin{equation}
f(k) = \frac{{K \choose k}{ N-K \choose n-k}}{{N \choose k}}
\end{equation}

It is the analogue to the binomial distribution for sampling without replacement.

% negative hypergeometric distribution
\subsection{Negative Hypergeometric $\mathrm{NH(N,K,n)}$}
The negative hypergeometric distribution gives the probability that, when sampling without replacement, it will take $r$ failures until there have been $k$ successes, if the total population is $N$ and the number of elements corresponding to a success is $K$.

\begin{equation}
f(k) = \frac{{k+r-1\choose k}{N-r-k \choose K-k}}{{N\choose K}}
\end{equation}

The negative hypergeometric distribution is the analogue to the negative binomial distribution for sampling without replacement.

% poisson processes
\section{Poisson Point Processes}
Poisson point processes are the continuous analogue to Bernoulli processes. Rather than looking at the outcome of a number of binary trials, they look at the number or spacing of independent point events over a continuous interval. In one dimension, Poisson Processes very often model rare events happening over a given time interval. In higher dimensions, the Poisson process can be thought of as independently scattered points in some volume. The process is typically characterized by the parameter $\Lambda$, called the \textit{rate} or \textit{intensity} of the process. It can be written $\Lambda = \nu \lambda$ where $\nu$ is a Lebesgue measure (assigning a length or volume to a set) and $\lambda$ is a constant. For a temporal process, $\nu$ would be the time interval. For a spatial process, $\nu$ would be a volume. The events have uniform distribution over the measure. This is also called a homogenous Poisson process.

% poisson distribution
\subsection{Poisson $\mathrm{Poisson}(\Lambda)$}
The poisson distribution is the continuous analogue to the Binomial distribution. It measures the number of events for a process with intensity $\Lambda$.

\begin{equation}
f(x) = e^{-\Lambda}\frac{\Lambda^x}{x!}\ \ \mathrm{x\geq0}
\end{equation}

Just like for the Binomial distribution, if $X_1 \sim \mathrm{Poisson}(\Lambda_1)$ and $X_2 \sim \mathrm{Poisson}(\Lambda_2)$, then $X_1 + X_2 \sim \mathrm{Poisson}(\Lambda_1 + \Lambda_2)$.

The mean and variance of the poisson distribution are both $\lambda$. This link means that Poisson distributed data is inherently heteroskedastic. The Poisson distribution can be obtained from the Bernoulli distribution by considering a very large number of trials with a small probability of success in each trial. That is, if $Y \sim B(n,\pi)$ then letting $n\rightarrow \infty$ with $\pi \rightarrow 0$ and $\mu \rightarrow n\pi$ you recover the Poisson distribution. In this sense, the Poisson distribution is an approximation to a Binomial process with rare events with large $n$ and small $\pi$.

% exponential distribution
\subsection{Exponential $\mathrm{Exp}(\lambda)$}
The exponential distribution is the continuous analogue to the Geometric distribution. In one dimension, if $X \sim \mathrm{Exp}(\Lambda)$, then $X$ can be interpreted as the distance between two independently scattered points on the real line. In the time picture, that would be the time that elapses between two events, or, equivalently, the time until the next event, where $\lambda$ is the rate or intensity.

\begin{equation}
f(x) = \lambda e^{-\lambda}
\end{equation}

% gamma distribution
\subsection{Gamma $\mathrm{Gamma(\alpha,\beta)}$}
The Gamma Distribution is the continuous analogue to the negative binomial distribution. For integer $\alpha$, it gives the probability of the value of the sum of $\alpha$ exponentially distributed random variables with parameter $\beta=\frac{1}{\lambda}$. That is, if $X_i \sim Exp(\frac{1}{\beta})$ then $X = \sum_i^\alpha X_i \sim \mathrm{Gamma}(\alpha,\beta)$. The PDF is given by:

\begin{equation}
f_x = \frac{1}{\beta^\alpha \Gamma(\alpha)}x^{\alpha-1}e^{-x/\beta}
\end{equation}

With $\alpha,\beta > 0$.

As one would expect, if $X_1 \sim \Gamma(\alpha_1,\beta)$ and $X_2 \sim \Gamma(\alpha_2,\beta)$ then $X_1 + X_2 \sim \Gamma(\alpha_1+\alpha_2,\beta)$.

% t distribution
\section{$t$-Distribution $\mathrm{t_\nu\ }$}

\subsection{Cauchy Distribution $t_1$}
The Cauchy Distribution does not have a mean. It is infinite.

\section{Chi$^2$-Distribution $\chi^2_p$}
The $\chi^2$ distribution is the probability distribution of the sum of squares of standard normally distributed random variables. If $Z_i$ has standard normal distribution, then $X = \sum_i^p Z_i \sim \chi^2_p$.

% normal
\section{Normal $\mathscr{N}(\mu,\sigma^2)$}

\subsection{Standard Normal Distribution $\mathscr{N}(0,1)$ }

The normal distribution is so ubiquitous that there is notation specifically for the \textit{standard normal distribution}, which is the normal distribution with parameters $\mu = 0$ and $\sigma = 1$. It is always possible to transform a random variable $X$ to have standard normal distribution by performing a coordinate transform. The PDF of the standard normal distribution is written $\phi(x)$ and the CDF $\Phi(x)$. The PDF is given by:

\begin{equation}
\phi(x) = \frac{1}{\sqrt{2\pi}}\exp{-x^2}
\end{equation}

The CDF has no closed form expression.

By convention, $Z$ is the random variable that has standard normal distribution. If $X \sim \mathscr{N}(\mu,\sigma^2)$, then $Z = \frac{X-\mu}{\sigma} \sim \mathscr{N}(0,1)$. This standardizing transformation allows for calculations using lookup tables.

\subsubsection{Example: Interval $\mathbb{P}(a<X<b)$}

\begin{equation}
\mathbb{P}(a<X<b) =  \mathbb{P}(a' < Z < b') = \Phi(b') - \Phi(a')
\end{equation}

with $a' = \frac{a-\mu}{\sigma}$ and $b' = \frac{b-\mu}{\sigma}$.

\subsubsection{Example: Quantile $x = F^{-1}(q)$}

\begin{equation}
z = \Phi^{-1}(q)
\end{equation}

with $z = \frac{x-\mu}{\sigma}$, so that $x = \sigma \Phi^{-1}(q)+\mu$. The interquartile range is calculated by transforming $(z_{1}=\Phi^{-1}(0.25),z_{3}=\Phi^{-1}(0.75))$, and so on.

% log normal
\section{Log Normal Distribution}
The log normal distribution has the odd property that is ``thin tailed" for low variance and ``fat tailed" for high variance.

% Multinoulli Processes
\section{Categorial Processes, Multinoulli Processes}
A Multnoulli process is a number of discrete trials that may assume a number of different categorical outcomes, for example a series of dice throws or a random sequence of letters. It is the generalization of the Bernoulli process to processes with more than two possible outcomes. The trials are independent from each other. Thinking of the process as a sequence in time, this means that the process is memoryless: the sequence of future outcomes is independent of past outcomes. Thinking about the process as a sequence in space, it is a uniformly random assignment of one of a set number of outcomes to grid points.

% Multinoulli
\subsection{Categorical, Multinoulli $\mathrm{Categorical(\mathbf{p})}$}
The categorial or Multinoulli distribution is the generalization of the Bernoulli distribution to more than two possible outcomes. The PMF is given by:

\begin{equation}
f(\mathbf{x}) = \prod_i^k p_i^{[x=k]}
\end{equation}

With $p_i \in \mathbf{p}$ a discrete probability distribution and $[x=k]$ is the Iverson bracket.

% Multinomial
\subsection{Multinomial $\mathrm{Multinomial(n,\mathbf{p})}$}
The multinomial distribution is the multivariate generalization of the binomial process. Rather than a binary outcome, there are $k$ possible outcomes. If $\mathbf{X}=(X_1,X_2,...,X_k)$ has multinomial distribution with parameter vector $\mathbf{p} = (p_1,p_2,...,p_k)$, then it gives the probability that out of $n$ trials, $x_j$ will be of type $j$, where the probabilities of the different classes is given by the discrete probability distribution $\mathbf{p}$.

\begin{equation}
f(\mathbf{x}) = {n \choose x_1,x_2,...,x_k} p_1^{x_1}p_2^{x_2}...p_k^{x_k} = {|\mathbf{x}|\choose \mathbf{x}}\mathbf{p}^{\mathbf{x}}
\end{equation}

Where the final expression uses multi-index notation. The binomial process corresponds to $k=2$ classes (success and failure). The marginal distribution of $X_j$ is $Binomial(n,p_j)$.

% beta distribution
\section{Beta $\mathrm{Beta}(\alpha,\beta)$}

The PDF for the beta distribution is:

\begin{equation}
f(x) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}\ \ 0 < x < 1
\end{equation}

With $\alpha, \beta > 0$.

The Beta distribution is the conjugate prior to the Bernoulli and Binomial distributions.

% dirichlet distribution
\section{Dirichlet $\mathrm{Dirichlet}(\mathbf{\alpha})$}
The Dirichlet Distribution is the multivariate generalization of the Beta Distribution. Given $\mathbf{X} = (X_1,...,X_K)$ and the parameter vector $\mathbf{\alpha} = (\alpha_1,...,\alpha_K),\ \alpha_j > 0$, the PDF is:

\begin{equation}
f(\mathbf{x},\mathbf{\alpha}) = \frac{1}{B(\mathbf{\alpha})} \prod^K_{i=1} x_i^{\alpha_i -1}
\end{equation}

Where $B(\mathbf{\alpha})$ is the multivariate Beta function:

\begin{equation}
B(\mathbf{\alpha}) = \frac{\prod^K_{i=1}\Gamma(\alpha_i)}{\Gamma(\sum_{i=1}^K\alpha_i)}
\end{equation}

The Dirichlet distribution is the conjugate prior to the Multinoulli/Categorical and Multinomial distributions. The marginal distributions are Beta distributions, i.e. $X_i \sim \mathbf{Beta}(\alpha_i,\alpha_0-\alpha_1)$ where $\alpha_0 = \sum_j \alpha_j$.

% multivariate normal distribution
\section{Multivariate Normal $\mathscr{N}(\mathbf{\mu},\mathbf{\Sigma})$}
The multivariate normal distribution is the multivariate generalization of the normal distribution. The PDF is given by:

\begin{equation}
f{\mathbf{x}} = \frac{1}{(2\pi)^{\frac{k}{2}}|\Sigma|^{\frac{1}{2}}}\exp\{-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^T \mathbf{\Sigma}^{-1}(\mathbf{x}-\mathbf{\mu})\}
\end{equation}

% covariance matrix
\subsection{Covariance Matrix $\mathbf{\Sigma}$}
The covariance matrix $\mathbf{\Sigma}$ is symmetric and positive definite. Therefore, $\mathbf{\Sigma}^{\frac{1}{2}}$ exists and is real, symmetric, and

\begin{itemize}
\item $\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{\Sigma}^{\frac{1}{2}} = \mathbf{\Sigma}$
\item $\mathbf{\Sigma}^{\frac{1}{2}}\mathbf{\Sigma}^{-\frac{1}{2}}=\mathbf{\Sigma}^{-\frac{1}{2}}\mathbf{\Sigma}^{\frac{1}{2}}=\mathbf{I}$
\end{itemize}

% marginal distribution
\subsection{Marginal Distribution}
If $\mathbf{X} \sim \mathscr{N}(\mathbf{0},\mathbf{\Sigma})$, then the marginal distribution of $X_a$ is:

\begin{equation}
X_a \sim \mathscr{\mu_a,\Sigma_{aa}}
\end{equation}

% conditional distribution
\subsection{Conditional Distribution}
If $\mathbf{X} \sim \mathscr{N}(\mathbf{0},\mathbf{\Sigma})$, then the conditional distribution of $X_b$ given $X_a = x_a$ is:

\begin{equation}
X_b|X_a = x_a \sim \mathscr{N}(\mu_b + \Sigma_{ba}\Sigma_{aa}^{-1}(x_a - \mu_a), \Sigma_{bb}-\Sigma_{ba}\Sigma_{aa}^{-1}\Sigma_{ab})
\end{equation}

% vector multiplication
\subsection{Vector Multiplication}
If $\mathbf{a}$ is a vector, then:

\begin{equation}
\mathbf{a}^T \mathbf{X} \sim \mathscr{N}(\mathbf{a}^T \mathbf{\mu}, \mathbf{a}^T\mathbf{\Sigma}\mathbf{a})
\end{equation}

% relationship to chi2
\subsection{Relationship to Chi$^2$}
\begin{equation}
V = \mathbf{Z}^T\mathbf{Z} = (\mathbf{X}-\mathbf{\mu})^T\mathbf{\Sigma}^{-1}(\mathbf{X}-\mathbf{\mu}) \sim \chi^2_k
\end{equation}

\subsection{Multivariate Standard Normal Distribution $\mathscr{N}(\mathbf{0},\mathbf{\Sigma})$}
The standard multivariate normal distribution for $\mathbf{Z} = (Z_1,...,Z_k)$ with $\mathbf{Z} \sim \mathscr{N}(\mathbf{0},\mathbf{I})$ is:

\begin{equation}
f(z) = \prod^k_{i=1} f(z_i) = \frac{1}{(2\pi)^{\frac{k}{2}}}\exp \{ \frac{1}{2} \sum^{k}_{j=1} z_j\} = \frac{1}{(2\pi)^{\frac{k}{2}}}\exp \{ \frac{1}{2} \mathbf{z^T z}\}
\end{equation}

The transformation to a general multivariate normal random variable $\mathbf{X}$ is that if $\mathbf{Z} \sim \mathscr{0,\mathbf{I}}$, then $\mathbf{X} = \mathbf{\mu} + \mathbf{\Sigma}^{\frac{1}{2}}\mathbf{Z}$. Conversely, if $\mathbf{X}\sim \mathscr{\mathbf{\mu},\mathbf{\Sigma}}$ then $\mathbf{\Sigma}^{-\frac{1}{2}}(\mathbf{X}-\mathbf{\mu}) \sim \mathscr{N}(0,\mathbf{I})$.


\section{Clustering}

Two Views:

Clustering input: $S_n = \{x^{(i)}|n=1,\cdots,n\}$

Clustering output are indexes for the data that partition the data: $C_1,\cdots,C_k$; where $C_1 \cup C_2 \cup ... \cup C_ K = \big \{  1, 2, ..., n \big \}$ and the union of all  $C_j$ 's is the original set and the intersection of any  $C_i$  and  $C_j$  is an empty set.

Representatives of clusters: $z^{(1)},\cdots,z^{(k)}$.\\

Cost of partitioning is the sum of costs of individual clusters: $cost(C_1,\cdots,C_k) = \sum_{j=1}^k cost (C_j)$.

Cost of cluster is sum of distances from data points to the representative of the cluster:$Cost(C,z) = \sum_{i \in C} = distance(x^{(i)},z)$

Cosine similarity: $cos(x^{(i)},x^{(j)}) = \frac{x^{(i)} \cdot x^{(j)}}{||x^{(i)}|| ||x^{(j)}||}$ is not sensitive of magnitude of vector (will not react to length).\\

Euclidean square distance: $dist(x^{(i)},x^{(j)})= ||x^{(i)}-x^{(j)}||^2$. Will react to length.\\

$cost(C_1,\cdots,C_k; z^{(1)},\cdots,z^{(k)}) = \sum_{j=1}^k \sum_{c \in C_j} ||x^{(i)}-z^{(j)}||^2 $

\subsection{The K-Means Algorithm}

Only works with Euclidean square distance.

Given a set of feature vectors $S_ n = \big \{ x^{(i)}| i = 1,...,n\big \}$ and the number of clusters  $K$ we can find cluster assignments  $C_1,\cdots,C_K$ and the representatives of each of the $K$ clusters $z_1,\cdots,z_K$:\\

\begin{enumerate}
\item Randomly select $z_1,\cdots,z_K$
\item Iterate
\begin{enumerate}
\item Given $z_1,\cdots,z_K$, assign each data point $x^{(i)}$ to the closest $z_j$, so that $\text {Cost}(z_1, ... z_ K) = \sum _{i=1}^{n} \min _{j=1,...,K} \left\|  x^{(i)} - z_ j \right\| ^2$
\item Given $C_1,\cdots,C_K$ find the best representatives $z_1,\cdots,z_K$, i.e. find $z_1,\cdots,z_K$ such that $\displaystyle z_ j=\operatorname {argmin}_{z} \sum _{i \in C_ j} \| x^{(i)} - z \| ^2$
\end{enumerate}
\end{enumerate}

The best representative is found by optimization (gradient with respect to $z^{(j)}$, setting to zero and solving for $z^{(j)}$). It is the centroid of the cluster: $z^{(j)}=\displaystyle \frac{\sum _{i \in C_ j} x^{(i)}}{|C_ j|}$\\

The clustering output that the K-Means algorithm converges to depends on the intialization.
In KM the fact that the z's are actually not guaranteed to be the members of the original set of points x

\subsection{K-Medoids Algorithm}

Finds the cost-minimizing representatives  $z_1,\cdots,z_K$ for any distance measure. Uses real data points for initialization.

\begin{enumerate}
\item Randomly select $\big \{  z_1, ..., z_ K \big \}  \subseteq \big \{  x_1, ..., x_ n \big \}$
\item Iterate
\begin{enumerate}
\item Given $z_1,\cdots,z_K$, assign each data point $x^{(i)}$ to the closest $z_j$, so that $\text {Cost}(z_1, ... z_ K) = \sum _{i=1}^{n} \min _{j=1,...,K} \left\|  x^{(i)} - z_ j \right\| ^2$
\item Given $C_ j \in \big \{ C_1,...,C_ K\big \}$ find the best representative $z_ j \in \big \{ x_1,...,x_ n\big \}$ such that $\sum _{x^{(i)} \in C_ j} \text {dist}(x^{(i)}, z_ j)$ is minimal
\end{enumerate}
\end{enumerate}

\subsection{some useful notes:}
measure of the cluster heterogeneity: cluster diameter (the distance
between the most extreme feature vectors, i.e. the outliers),the average distance,
 the sum of the distances between every member and  $z_j$, the
representative vector of cluster $C_j$.


\section{Lengths in metric spaces}

On $S^2 \subset \mathbb{R}^3$, we define $\mathrm{d}(a,b) = \text{inf}\left( \{ \text{length}(\gamma) | \gamma \text{ is a path on } S^2 \text{ from } a \text{ to } b \} \right)$, where $a$ and $b$ are points of $S^2$.

\noindent Now we try to generalize this construction. We assume that $X$ is a metric space.

\begin{itemize}
  \item We will define \emph{lengths of paths in} $X$.
  \item Given lengths of paths, we will define a new distance $\hat{d}$ on $X$ as follows:
  $$\hat{d}(a,b) = \text{inf}\left( \{ \text{length}(\gamma) \ | \ \gamma \text{ is a path from } a \text{ to } b \} \right)$$
  where $a$ and $b$ are points of $X$.
\end{itemize}


\lab{Applications}{Correlation and Covariance}{Correlation and Covariance}
\label{Stats1}

\objective{This section will teach about using Python to solve problems in statistics. This will include finding means, correlation matrices, and solving least squares problems.}

\section*{Shifting Data by the Mean}

Consider the table below representing students scores in a class.\\

\begin{figure}[h!]
\begin{center}
\begin{tabular}{|c|r|r|r|r|}
	\hline
Student & Homework & Exam 1  & Exam 2 & Final \\
\hline
S1  & 89 & 91 & 77 & 75 \\
S2  & 67 & 72 & 76 & 66 \\
S3  & 72 & 77 & 69 & 70 \\
S4  & 56 & 60 & 55 & 61 \\
S5  & 92 & 98 & 89 & 86 \\
S6  & 83 & 88 & 90 & 84 \\
S7  & 45 & 60 & 55 & 48 \\
\hline
Average  & 72 & 78 & 73 & 70\\
\hline
\end{tabular}\\
\end{center}
\end{figure}

We can shift our data set by subtracting each column by its average value.  This makes it so that the average of each column in the matrix below is zero.  If $W$ represents the matrix of scores, the following Python command will subtract out the average.  Why does this work?
\begin{lstlisting}[style=python]
:X = W - sp.dot(sp.ones((7,1),dtype=sp.float_),sp.mean(W,axis=0).reshape(1,4));X
array([[ 17.,  13.,   4.,   5.],
       [ -5.,  -6.,   3.,  -4.],
       [  0.,  -1.,  -4.,   0.],
       [-16., -18., -18.,  -9.],
       [ 20.,  20.,  16.,  16.],
       [ 11.,  10.,  17.,  14.],
       [-27., -18., -18., -22.]])
\end{lstlisting}

\section*{Angles Between Vectors}

Inner products give information about lengths of vectors and angles between vectors.  Recall that the angle $\theta$ between two vectors is given by
\[
\cos{\theta} = \frac{\ipt{x}{y}}{\norm{x}\norm{y}}
\]
where the norm (or length) of a vector is $\norm{x} = \sqrt{\ipt{x}{x}}$.  Note that the usual inner product in $\mathbb{R}^n$ is given by
\[
\ipt{x}{y} = x^T y
\]
Alternatively, if we can first divide our vectors by their length (these are called unit vectors) and then take the inner product
\[
\cos{\theta} = \left\langle\frac{x}{\norm{x}},\frac{y}{\norm{y}}\right\rangle
\]
Hence, we can find the cosine of the angles between our data columns in $X$ by dividing each column by its length.  There are a couple of ways of doing this.  The following is an interesting way of doing it.  Be sure to explore why it works:
\begin{lstlisting}[style=python]
: Y = X / sp.dot(sp.ones((7,1),dtype=sp.float_),sp.sqrt(sp.diag(sp.dot(X.T,X)).T).reshape(1,4));Y
array([[ 0.39848615,  0.35329218,  0.11386819,  0.15371887],
       [-0.11720181, -0.16305793,  0.08540114, -0.12297509],
       [ 0.        , -0.02717632, -0.11386819,  0.        ],
       [-0.37504578, -0.48917378, -0.51240685, -0.27669396],
       [ 0.46880723,  0.54352643,  0.45547275,  0.49190037],
       [ 0.25784398,  0.27176321,  0.4839398 ,  0.43041282],
       [-0.63288976, -0.48917378, -0.51240685, -0.67636301]])
\end{lstlisting}
Hence we have:
\[
Y=
\begin{bmatrix}
0.3985 & 0.3533 & 0.1139 & 0.1537\\
-0.1172 & -0.1631 & 0.0854 & -0.1230\\
0 &-0.0272 &-0.1139 & 0\\
-0.3750 & -0.4892 & -0.5124 & -0.2767\\
0.4688 & 0.5435 & 0.4555 & 0.4919\\
0.2578 & 0.2718 & 0.4839 & 0.4304\\
-0.6329 & -0.4892 & -0.5124 & -0.6764
\end{bmatrix}
\]
Finally, we get the cosines of the angles between columns by computing $Y^T Y$.  In Python, that's just
\begin{lstlisting}[style=python]
: sp.dot(Y.T,Y)
array([[ 1.        ,  0.97782999,  0.89014869,  0.94908967],
       [ 0.97782999,  1.        ,  0.90978843,  0.92490144],
       [ 0.89014869,  0.90978843,  1.        ,  0.9276955 ],
       [ 0.94908967,  0.92490144,  0.9276955 ,  1.        ]])
\end{lstlisting}
This yields
\[
Y^T Y =
\begin{bmatrix}
1.0000 & 0.9778 & 0.8901 & 0.9491\\
0.9778 & 1.0000 & 0.9098 & 0.9249\\
0.8901 & 0.9098 & 1.0000 & 0.9277\\
0.9491 & 0.9249 & 0.9277 & 1.0000
\end{bmatrix}.
\]
Note that the $(j,k)$ entry of $Y^T Y$ corresponds to the cosine of the angle between the $j^{th}$ and $k^{th}$ columns.  We remark that the diagonals are always equal to one because the angle between a vector and itself is zero and the cosine of zero is one.

It is also worth noting that the matrix resulting from the operation $A \cdot A^T$ is symmetric and positive definite, so it can be solved with the cholesky decomposition that was discussed at the end of Lab \ref{lab:LUdecomp}.

\section*{Correlation}

We remark that the cosine of the angle between two vectors is sometimes called the correlation coefficient.  Two columns are said to be
\begin{itemize}
\item Perfectly correlated if the cosine of the angle between them is one.
\item Positively correlated if the cosine of the angle between them is between zero and one.
\item Uncorrelated if the cosine of the angle between them is zero.
\item Negatively correlated if the cosine of the angle between them is between negative one and zero.
\item Perfectly anticorrelated if the cosine of the angle between them is negative one.
\end{itemize}

The notion of correlation is important in establishing the relationships between measurements.  For example, there is a high correlation between those who smoke and those who get lung cancer.  It's important to understand that the high correlation alone does not necessarily imply that smoking causes lung cancer, it only links them statistically.  This is the famous issue of causation versus correlation.  For example, there is a high correlation between crime rates and sales of ice cream.  This doesn't mean that ice cream causes crime or that increases in crime makes people want to eat more ice cream, but both rates do go up in the summer.

\begin{problem}
Load the data file \texttt{data.npy}.
The dataset consists of two columns.  Find the correlation coefficient of the two columns.  Then plot the original data and see if the value that you got is reasonable.
\end{problem}


\section{Areas in the Plane}
\begin{lemma}
	If $f(x) \geq g(x)$ on $[a,b]$ then the area between $f(x)$ and $g(x)$ on $[a,b]$ is given by
	\begin{equation*}
		A = \int_{a}^{b}{\left(f(x)-g(x)\right)\d{x}}.
	\end{equation*}
\end{lemma}

You can easily visualize why this is true.
We can break the integral into two parts: adding the area under $f$ and the other subtracting the area under $g$.
The first integral will over count the area between $f$ and $g$, counting all area between $f$ and the $x$-axis.
The second integral will subtract exactly the amount of area that is over counted: the area that is also between $g$ and the $x$-axis.

\begin{figure}[H]
	\label{area_between_curves}
	\centering
	\includegraphics[width = 0.33\textwidth]{./applications_integrals/curve1.png}
	\includegraphics[width = 0.33\textwidth]{./applications_integrals/curve2.png}
	\caption{\hyperref{}{}{}{Subtracting two areas to get the area between them}}
\end{figure}

Even if both of the curves has more area below the $x$-axis, the same idea applies.
$f$ will give a smaller-magnitude negative area, while $g$ will give a larger magnitude negative area.
Subtracting a larger-magnitude negative number from a smaller magnitude negative number will give a positive area.

\begin{example}
	Find the area enclosed between the two curves $y=2-x^2$ and $y=-x$.
\end{example}
\begin{answer}
	These two curves intersect at $x=-1$ and $x=2$.
	Applying the formula,
	\begin{align*}
		A &= \int_{-1}^{2}{((2-x^2)-(-x))\d{x}} \\
		&= 2x-\frac{x^3}{3} + \frac{x^2}{2}\biggr\rvert_{-1}^{2} \\
		&= \frac{9}{2}.
	\end{align*}
\end{answer}

\subsection{Subregions}
Sometimes it might be useful to break the enclosed regions into subregions and find the area of each separately.
\begin{example}
	Find the area above the $x$-axis, below $y=\sqrt{x}$, and above $y=x-2$.
\end{example}
\begin{answer}
	If we simply took the area between the two curves, we'd count some area we don't want beneath the $x$-axis\footnote{Although this extra region is just a triangle and not too hard to find the area of, you'd effectively be applying the same strategy of two subregions, they'd just overlap. However, using geometry to your advantage is certainly a valid approach.}.
	Instead, we can break the area into two subregions: one between the $x$-axis and $\sqrt{x}$ and the other between $x-2$ and $\sqrt{x}$.
	\begin{figure}[H]
		\label{subregions}
		\centering
		\includegraphics[width = 0.33\textwidth]{./applications_integrals/two_regions.png}
		\caption{\hyperref{}{}{}{Break complicated areas into simpler subregions}}
	\end{figure}

	Finding the area of the blue region,
	\begin{equation*}
		A_1 = \int_{0}^{2}{\sqrt{x}\d{x}} = \frac{2x\sqrt{x}}{3}\biggr\rvert_0^2 = \frac{4\sqrt{2}}{3}.
	\end{equation*}

	Finding the area of the green region,
	\begin{equation*}
		A_2 = \int_{2}^{4}{(\sqrt{x}-(x-2))\d{x}} = \frac{2x\sqrt{x}}{3} - \frac{x^2}{2} + 2x \biggr\rvert_2^4 = \frac{10}{3} - \frac{4\sqrt{2}}{3}.
	\end{equation*}

	Adding the areas of the two regions,
	\begin{equation*}
		A = A_1 + A_2 = \frac{10}{3}.
	\end{equation*}
\end{answer}

\subsection{Integrating with Respect to $y$}
Another strategy when finding the area of more complex regions is to see if they become easier to deal with if we were to instead integrate with respect to $y$.
In the above example, it is indeed easier to work with respect to $y$ because both bounding curves are between the same $y$ values.
\begin{example}
	Find the area above the $x$-axis, below $y=\sqrt{x}$, and above $y=x-2$.
\end{example}
\begin{answer}
	We need to rearrange our equations to be of the form $x=\ldots$ instead of $y=\ldots$ by simply solving for $x$.
	\begin{equation*}
		x = y^2, y\geq 0 \text{ and } x = y+2.
	\end{equation*}

	Since $x=y+2$ is further from the $x$-axis, it becomes our top curve.
	\begin{align*}
		A &= \int_{0}^{2}{((y+2)-y^2)\d{y}} \\
		&= \frac{y^2}{2} + 2y - \frac{y^3}{3} \biggr\rvert_0^2 \\
		&= \frac{10}{3}.
	\end{align*}
\end{answer}


\subsection{Polar co-ordinates}

\subsubsection{All complex numbers can be shown in polar form}

Consider a complex number

\(z=a+bi\)

We can write this as:

\(z=r\cos(\theta ) + ir\sin(\theta )\)

\subsubsection{Polar forms are not unique}

Because the functions loop:

\(ae^{i\theta }=a(\cos(\theta )+i\sin(\theta ))\)

\(ae^{i\theta }=a(\cos(\theta +n\tau )+i\sin(\theta +n\tau ))\)

\(ae^{i\theta }= ae^{i\theta +n\tau}\)

Additionally:

\(ae^{i\theta }=a(\cos(\theta )+i\sin(\theta ))\)

\(ae^{i\theta }=a(\cos(\theta )+i\sin(\theta ))\)

\(ae^{i\theta }=-a(\cos(\theta )-i\sin(\theta ))\)

\(ae^{i\theta }=-a(\cos(\theta +\dfrac{\pi }{2})+i\sin(\theta +\dfrac{\pi }{2}))\)

\subsubsection{Real and imaginary parts of a complex number in polar form}

We can extract the real and imaginary parts of this number.

\(Re(z):=r\cos (\theta )\)

\(Im(z):=r\sin (\theta )\)

Alternatively:

\(Re(z)=r\dfrac{e^{i\theta }+e^{-i\theta }}{2}\)

\(Im(z)=r\dfrac{e^{i\theta }-e^{-i\theta }}{2i}\)


\section{Probability Theory}

\begin{theorem}[Laws of Set Algebra]
    For any three sets $A$, $B$, $C$ all subsets of $S$, we have that the operators $\cup$ and $\cap$ are distributive, commutative, associative and satisfy DeMorgan's Laws:
    \begin{align*}
        \comp{(A \cup B)} &= \comp{A} \cap \comp{B} \\
        \comp{(A \cap B)} &= \comp{A} \cup \comp{B}
    \end{align*}
\end{theorem}

\begin{definition}[Sigma Algebra]
    A collections of subsets S is calles a \emph{sigma algebra} (or \emph{Borel Field}), denoted by $\B{}$, if it has the following three properties:
    \begin{enumerate}[a.]
        \item $\emptyset \in \B{}$
        \item $A \in \B{} \implies \comp{A} \in \B{}$
        \item If $A_i \in \B{}$ for $i$ in $\mathcal{I}$ then $\cup_{i\in\mathcal{I}}A_i \in \B{}$, where $\mathcal{I}$ is countable. So $\B{}$ is closed under countable union.
    \end{enumerate}
\end{definition}

Note that from DeMorgan's laws we have
\begin{equation}
    \comp{\left(\bigcup_{i=1}^\infty \comp{A_i}\right)} = \bigcap_{i=1}^\infty A_i
\end{equation}
which means that, using b. we get $\B{}$ is also closed under countable intersections $\cap_{i=1}^\infty A_i \in \B{}$.

\begin{definition}[Kolmogorov Axioms]
    Given a sample space $S$ and an associated sigma algebra $\B{}$, a \emph{probability function} is a function $\P{}$ with domain $\B{}$ that satisfies
    \begin{enumerate}
        \item $\P{}(A) \geq 0 \quad \forall A \in \B{}$
        \item $\P{}(S) = 1$
        \item If $A_1, A_2, \dots \in \B{}$ are pairwise disjoint, then $\P{}(\cup_{i=1}^\infty A_i = \sum_{i=1}^{\infty})\P{}(A_i)$
    \end{enumerate}
\end{definition}

The following result makes it a bit easier to find probability functions.

\begin{theorem}
    Let $S=\{s_1, s_2, \dots\}$ be a countable set. Let $\B{}$ be any sigma algebra of subsets of $S$. Let $p_1, p_2, \dots$ be nonnegative numbers that sum to 1. For any $A \in \B{}$, define $\P{}(A)$ by
    \[
        \P{}(A) = \sum_{\{i: s_i \in A\}} p_i.
    \]
Then $\P{}$ is a probability function on $\B{}$.
\end{theorem}

\begin{definition}[Random Variable]
    A \emph{random variable} is a function from a sample space $S$ into the real numbers.
\end{definition}

\begin{definition}[cdf]
    The \emph{cumulative distribution function} of \emph{cdf} of a random variable $X$ is defined by
    \[
        F_X(x) = \P{}_X(X \leq x) \quad \forall x
    \]
\end{definition}

\begin{theorem}
    The function $F(x)$ is a cdf if and only if:
    \begin{enumerate}
        \item $\lim_{x \to - \infty} F(x) = 0$ and $\lim_{x\to\infty} F(x) = 1$.
        \item $F(x)$ is a non-decreasing function of x.
        \item $F(x)$ is right-continuous, that is, for every number $x_0$, $\lim_{x\downarrow x_0} F(x) = F(x_0)$
    \end{enumerate}
\end{theorem}

We say that a random variable is continuous if its cdf is continuous, and we say that it is discrete if its cdf is a step function.

\begin{theorem}
    The following statements are equivalent:
    \begin{enumerate}
        \item $X$ and $Y$ are identically distributed
        \item $F_X(x) = F_Y(x) \quad \forall x$
    \end{enumerate}
\end{theorem}

\begin{definition}[probability mass function]
    The \emph{probability mass function} or \emph{pmf} of a discrete random variable $X$ is
    \[
        f_X(x) = P(X=x) \quad \forall x
    \]
\end{definition}

\begin{definition}[probability density function]
    The \emph{probability density function} or \emph{pdf} of a continuous random variable $X$ is the function $f_X(x)$ that satisfies
    \[
        F_X(x) = \int_{-\infty}^{x}f_X(t) \d{} t \quad \forall x
    \]
\end{definition}

\begin{theorem}
    A function $f_X(x)$ is a pdf (of pmf) of a random variable $X$ if and only if
    \begin{enumerate}[a.]
        \item $f_X(x) \geq 0 \quad \forall x$
        \item $\sum_x f_X(x) = 1$ (pmf) or $\int_x f_X(x)\d{}x = 1$ (pdf).
    \end{enumerate}
\end{theorem}


\subsection{Length of a curve}

We have a curve from \(a\) to \(b\) in \(\mathbf R^n\).

\(f:[a,b]\rightarrow \mathbf R^n\)

We divide this into \(n\) segments.

The \(i\)th cut is at:

\(t_i=a+\dfrac{i}{n}(b-a)\)

So the first cut is at:

\(t_0=a\)

\(t_n=b\)

The distance between two sequential cuts is:

\(||f(t_i)-f(t_{i-1}||\)

The sum of all these differences is:

\(L=\sum_{i=1}^n ||f(t_{i})-f(t_{i-1})||\)

The limit is:

\(L=\lim_{n\rightarrow \infty }\sum_{i=1}^n ||f(t_{i})-f(t_{i-1})||\)
\subsubsection{Method 1}

\(L=\lim_{n\rightarrow \infty }\sum_{i=1}^n ||f(t_{i})-f(t_{i-1})||\)

\(L=\lim_{n\rightarrow \infty }\sum_{i=1}^n ||\dfrac{f(t_{i})-f(t_{i-1})}{\Delta t}||\Delta t\)

\(L=\lim_{n\rightarrow \infty }\sum_{i=1}^n ||f'(t)||\Delta t\)

\(L=\int_a^b ||f'(t)||dt\)

\subsubsection{Method 2}

\(L=\lim_{n\rightarrow \infty }\sum_{i=1}^n ||f(t_{i})-f(t_{i-1})||\)

\(L=\lim_{n\rightarrow \infty }\sum_{i=1}^n \sqrt {(f(t_{i})-f(t_{i-1}))^*M(f(t_{i})-f(t_{i-1}))}\)

\(L=\int_a^b \sqrt {(dt)^TM(dt)}\)


One way of visualizing some relations is by using a directed graph, which we define below.

\begin{definition}
A \textbf{directed graph} consists of two sets $V$ and $E \subset V \times V$. The set $V$ is called the set of vertices and the set $E$ is called the set of (directed) edges.
\end{definition}

We can visualize a directed graph as follows. For visual purposes and a concrete example, we will consider the directed graph $V = \{a, b, c\}$ and $E = \{(a, a), (a, b), (b, c), (c, c)\}$. First, we imagine the objects of $V$ as dots.

\begin{figure}[ht]
	\centering
	\includegraphics{Ch3/vertices.png}
\end{figure}

We can imagine each object $(x, y)$ of $E$ as a ``directed arrow'' (ie an arrow) where the back end of the arrow is $x$ and the front end of the arrow is $y$. For the example graph pictured this is visualized as follows.

\begin{figure}[ht]
	\centering
	\includegraphics{Ch3/edges.png}
\end{figure}

In this way a graph can be visualized. One minor thing to observe is that any edge $(x, x)$ from an object $x$ to itself is drawn as a ``loop'' from the vertex starting at $x$ and pointing back at itself. The final thing to be observed is that any relation $R$ on a set $A$ can be seen as a graph. In this case, we take the vertices to be the set $A$ and we take the set of edges to be the set $R \subset A \times A$. We call this graph the \textbf{associated graph} of the relation $R$.

\begin{example}\label{gp1}
Suppose that a relation $R$ is transitive. Recall that a relation $R$ is transitive if for all $a, b, c \in A$, $(a, b) \in R, (b, c) \in R \implies (a, c) \in R$. What does this mean for its associated graph? Suppose then we have $a, b, c \in A$ with $(a, b) \in R, (b, c) \in R$. Then $(a, c) \in R$. A consequence of this is that if we have a chain of arrows from $a$ to $c$ then by transitivity there must be an arrow from $a$ to $c$ itself. This is illustrated in the picture below.
\end{example}
\begin{figure}[ht]
\centering
\includegraphics{Ch3/transitivity.png}
\end{figure}

%remark. For any a in R transitivity implies that any vertex that can be reached by ``travelling'' through $a$ can be reached through a in one step (this might require induction `formally').

\subsubsection{Exercises}

\include{Ch3/rgraph_exercises}


\chapter{Linear Maps}
\section{Vector spaces}
A vector space over a field \(F\) is a set \(V\) that is closed under vector addition (+) and scalar multiplication (\( \cdot \)). A vecotr space must fulfill following axioms:
\begin{align}
    \begin{split}
        v, w, u \in V \\
        \alpha, \beta \in F \\
        v + w = u  \\
        \alpha \cdot v = u \\
        (\alpha \cdot \beta) \cdot v = \alpha \cdot (\beta \cdot v)      \\
        \alpha (v + w) = \alpha \cdot v + \alpha \cdot w              \\
        (\alpha + \beta) \cdot w = \alpha \cdot v + \beta \cdot w \\
        1 \cdot v = w
    \end{split}
\end{align}
The vector addition \((V,+)\) form a commutative group.
\begin{align*}
    (v + w) + u = v + (w + z) \tag*{Associativity} \\
    v + 0 = V \tag*{Identity element: zero vector} \\
    v + w = 0 \tag*{Inverse element}               \\
    v + w  = w + v \tag*{Commutativity}            \\
\end{align*}
\( (F, +, \cdot)\) form a field.
\begin{align*}
    a, a^{-1}, b, c \in F                                                         \\
    (a + b) + c = a + (b + c) \tag*{Additive associativity}                       \\
    a + a^{-1} = 0 \tag*{Additive inverse}                                        \\
    a + 0 = a \tag*{Additive identity}                                            \\
    a + b = b + a \tag*{Additive commutativity}                                   \\
    (a \cdot b) \cdot c = a \cdot (b \cdot c)  \tag*{Mulitlicative associativity} \\
    a \cdot a^{-1} = 1, a^{-1} \neq 0 \tag*{Mulitlicative inverse}                \\
    a  \cdot 1 = a \tag*{Multiplicative identity}                                 \\
    a \cdot b = b \cdot a \tag*{Mulitplicative commutativity}                     \\
    a \cdot (b + c) = a\cdot b + \cdot c \tag*{Distibutivity}
\end{align*}
Example for fields: \((\mathbb{R}, +, \cdot)\) \( (\mathbb{C}, +, \cdot)\) \\
Example for non fields: \((\mathbb{N}, +, \cdot)\) \( (\mathbb{Z}, +, \cdot)\) \\
\subsection{Subspace}
Let \(V\) be a vector space ofer a field \(F\). A subspace \(W\) is a subset of \(V\) that also form a vector space over \(F\).
\begin{example}
    \(\mathbb{R}^2\) is a vector space over \(\mathbb{R}\).
    \begin{align*}
        W = \setb{c \cdot \begin{bmatrix}
                1 \\2
            \end{bmatrix}
        }{c \in \mathbb{R}} = \{ \dots \begin{bmatrix}
            -1 \\ -2
        \end{bmatrix}, \begin{bmatrix}
            1 \\2
        \end{bmatrix}, \begin{bmatrix}
            2 \\4
        \end{bmatrix},\begin{bmatrix}
            2.4 \\ 4.8
        \end{bmatrix} \dots
        \}
    \end{align*}
\end{example}
\subsubsection{Direct sum}
Let \(V\) be a vector space ofer a field \(F\)  and \(W\) and \(U\) subspaces of \(V\)
When:
\begin{align}
    W \cap U & = \{ 0 \} \\
    W \cup U & = V
\end{align}
Then \(U\) and \(W\) are a direct sum of \(V\). It is denoted by
\begin{equation}
    V = U \oplus W
\end{equation}
\subsection{Span, (lineare Hülle)}
Let \(V\) be a vector space ofer a field \(F\) and \(S\) a finite subset of \(V\) wiht length \(n\). The span of \(S\) is the set of vectors that can be created by linear combinations with the vectors in \(S\).
\begin{equation}
    span(S) = \setb{
        \sum_{i=1}^{n} a_i \cdot s_i
    }{n \in \mathbb{N}, a_i \in F, s_i \in S}
\end{equation}
\begin{example}
    \begin{align*}
        S        & = \{
        \begin{bmatrix}
            1 \\ 0 \\ 0
        \end{bmatrix},
        \begin{bmatrix}
            0 \\ 1 \\ 0
        \end{bmatrix}
        \}                   \\
        \vspan S & =  \setb{
            a_1
            \begin{bmatrix}
                1 \\ 0 \\ 0
            \end{bmatrix} + a_2
            \begin{bmatrix}
                0 \\ 1 \\ 0
            \end{bmatrix}
            \
        }{a_1, a_2 \in \mathbb{R}}
    \end{align*}
\end{example}
\subsubsection{Spanning set}
Let \(V\) be a vector space ofer a field \(F\) and \(S\) a finite subset of \(V\). \(S\) is a spanning set of if
\begin{equation}
    \vspan S = V
\end{equation}
\begin{example}
    Let \(V\) be \(\mathbb{R}^2\) over the field \(\mathbb{R}\). Following subsets are spanning set of \(V\):
    \begin{align*}
        S_1 & = \{ \begin{bmatrix}
            1 \\ 0
        \end{bmatrix},
        \begin{bmatrix} 0 \\ 1 \end{bmatrix}
        \}                                     \\
        S_2 & = \{ \begin{bmatrix}
            3 \\ 0
        \end{bmatrix},
        \begin{bmatrix} 0 \\ 2 \end{bmatrix}
        \}                                     \\
        S_3 & = \{ \begin{bmatrix}
            3 \\ 0
        \end{bmatrix},
        \begin{bmatrix} 0 \\ 2 \end{bmatrix},
        \begin{bmatrix} 1 \\ 2 \end{bmatrix}
        \}
    \end{align*}
\end{example}
\subsection{Base}
Let \(V\) be a vector space ofer a field \(F\) and \(B\) a spanning set of \(V\). If the elements of \(B\) are lineary independent then \(B\) is called a basis.
The coefficients of the linear combination are referred to as components or coordinates of the vector with respect to \(B\).
The elements of \(B\) are called basis vectors.
\begin{example}
    Let \(V\) be \(\mathbb{R}^2\) over the field \(\mathbb{R}\). Following subsets are spanning set of \(V\):
    \begin{align*}
        B_1 & = \{ \begin{bmatrix}
            1 \\ 0
        \end{bmatrix},
        \begin{bmatrix} 0 \\ 1 \end{bmatrix}
        \}                                     \\
        B_2 & = \{ \begin{bmatrix}
            3 \\ 0
        \end{bmatrix},
        \begin{bmatrix} 0 \\ 2 \end{bmatrix}
        \}                                     \\
    \end{align*}
    In previous example \(S_3\) is not a valid base, because its elements are lineary dependent.
\end{example}
\subsubsection{Standard base}
A base \(B\) is called a Standard base if the vecotors of \(B\) are all zero, except one that equals 1. The vectors
of the standard base are called unit vectors.
\begin{example}
    The standard base for \(\mathbb{R}^n\) is
    \begin{align*}
        \hat{i} & = \begin{bmatrix}
            1 \\ 0 \\ 0
        \end{bmatrix}, \hat{j} = \begin{bmatrix}
            0 \\ 1 \\ 0
        \end{bmatrix}, \hat{k} = \begin{bmatrix}
            0 \\ 0 \\ 1
        \end{bmatrix} \\
        B       & = \{ \hat{i}, \hat{j}, \hat{k} \}
    \end{align*}
    A vector \(v\) expressed in the standard basis \(B\).
    \begin{align*}
        v = \begin{bmatrix}
            4 \\ 5 \\ 6
        \end{bmatrix}= 4 \hat{i} + 5  \hat{j} + 6  \hat{k}
    \end{align*}
\end{example}
\subsection{Dimension}\label{dimension}
The dimension \( \dim \) of a vector space is the size of its base \(B\). The dimension
is equal to the rank (see \ref{rank}) of the tranformation matrix.
\begin{example}
    The vector space \(\mathbb{R}^n\)
    \begin{align*}
        \dim \mathbb{R}^n = n
    \end{align*}
\end{example}
\begin{example}
    \begin{align*}
        V      & = \setb{
            c_1
            \begin{bmatrix}
                1 \\ 0 \\ 0 \\ 0
            \end{bmatrix},
            c_2
            \begin{bmatrix}
                0 \\ 1 \\ 0 \\ 0
            \end{bmatrix}
        }{c_1, c_2 \in \mathbb{R}} \\
        \dim V & = 2
    \end{align*}
\end{example}
\begin{example}
    \begin{align*}
        V      & = \setb{
            c_1
            \begin{bmatrix}
                1 \\ 1 \\ 0 \\ 0
            \end{bmatrix},
            c_2
            \begin{bmatrix}
                0 \\ 1 \\ 0 \\ 0
            \end{bmatrix}
        }{c_1, c_2 \in \mathbb{R}} \\
        \dim V & = 1
    \end{align*}
\end{example}
\begin{example}
    The only vector space with dimension 0 is where V contains only the zero vector.
    \begin{align*}
        \dim \{ \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}\} = 0
    \end{align*}

\end{example}
section{Linear maps}
Let \(V\) \(W\) be vector spaces over the same field \(F\).
A function \(f: V \rightarrow W\) is said to be a linear map if for any two vectors \(v, u \in V\) and any scalar \(c \in F\) the following two conditions are satisfied:
\begin{align}
    f(u + v)     & = f(u) + f(v) \tag{Additvity} \\
    f(c \cdot u) & = c \cdot u \tag{Homogenity}
\end{align}
\subsection{Transformation matrix}
Each linear transformation can be represented as a matrix vector multiplication.
\begin{align*}
    f      & : W \rightarrow V  \\
    \dim W & = n, \dim V = m    \\
    f(x)   & = A^{m \times n} x \\
\end{align*}
\begin{example}
    \begin{align*}
        f(x)                                       &
        = \begin{bmatrix}
            1 & 2 & 3 \\ 5 & 7 & 9
        \end{bmatrix} \begin{bmatrix}
            x_1 \\ x_2 \\ x_3
        \end{bmatrix}                   \\
        f \left(\begin{bmatrix}
            5 \\ 6 \\ 7
        \end{bmatrix} \right) & = \begin{bmatrix}
            38 \\ 130
        \end{bmatrix}
    \end{align*}
\end{example}
\subsubsection{Composition}
If there are two linear maps \(f, g\) wiht tranformation matrices:
\begin{align*}
    f : V \rightarrow W  = Ax \\
    g : U \rightarrow V = Bx
\end{align*}
then the compostition is:
\begin{align*}
    h : U \rightarrow W = f \circ g \\
    h(x) = A(Bx) = (A \cdot B)x
\end{align*}
\begin{example}
    \begin{align*}
        f(x)                                        & = \begin{bmatrix}
            1 & 2 & 3 \\ 5 & 7 & 9
        \end{bmatrix} \cdot x                                                                             \\
        g(x)                                        & = \begin{bmatrix}
            -1 & -2 \\ -7 & -9 \\ 13 & 17
        \end{bmatrix} \cdot x                                                                             \\
        h(x)                                        & = f \circ g = \begin{bmatrix}
            1 & 2 & 3 \\ 5 & 7 & 9
        \end{bmatrix} \begin{bmatrix}
            -1 & -2 \\ -7 & -9 \\ 13 & 17
        \end{bmatrix} \cdot x = \begin{bmatrix}
            24 & 31 \\ 64  & 80
        \end{bmatrix} \cdot x \\
        g \left( \begin{bmatrix}
            3 \\ 4
        \end{bmatrix} \right) & = \begin{bmatrix}
            -11 \\ -57 \\ 107
        \end{bmatrix}, f \left( \begin{bmatrix}
            -11 \\ -57 \\ 107
        \end{bmatrix} \right) = \begin{bmatrix}
            196 \\ 509
        \end{bmatrix}           \\
        h(\begin{bmatrix}
            3 \\ 4
        \end{bmatrix})               & = \begin{bmatrix}
            196 \\ 509
        \end{bmatrix}
    \end{align*}
\end{example}
\subsection{Image (Bild)}
The image \(f^{\rightarrow}\) of a tranformation \(L: V \rightarrow W \) is the set of vectors that the tranformation can produce.
\begin{equation}
    f^{\rightarrow} (L)  = \setb{L(x)}{x \in V}
\end{equation}
The image is the columnspan of the tranformation matrix. The dimenson of the image ist called \textbf{rank}, and is the
same as the rank of the transformation matrix.
\begin{example}
    \begin{align*}
        L(x)                & = \begin{bmatrix}
            1 & 0 \\
            0 & 1 \\
        \end{bmatrix}x \\
        f^{\rightarrow} (L) & = \mathbb{R}^2
    \end{align*}
\end{example}
\begin{example}
    \begin{align*}
        L(x)                & = \begin{bmatrix}
            1 & 0 \\
            2 & 0 \\
            1 & 1 \\
            0 & 2 \\
        \end{bmatrix}x      \\
        f^{\rightarrow} (L) & = \setb{\begin{bmatrix}
                c_1 \\ 2c_1 \\ c_1 + c_2 \\ 2 c_2
            \end{bmatrix}
        }{c_1, c_2 \in \mathbb{R}}
    \end{align*}
\end{example}
\subsection{Kernel, Null Space (Kern)}\label{kernel}
The kernel of a linear map \(L: V \rightarrow W \) is the linear subspace of the domain of the map which is mapped to the zero vector.
\begin{equation}
    \ker L = \{ v \in V | L(v) = 0 \} \\
\end{equation}

The vecots of the kernel are the set of vectors that yield the zero vector after multiplication with the tranformation matrx.
\begin{align*}
    L  & : Ax = y                          \\
    x' & \in \ker L  \textbf{ if } Ax' = 0
\end{align*}
The kernel forms a subspace of \( V \):
\begin{align*}
    v, u \in \ker L, \alpha \in \mathbb{F} \\
    \alpha v \in \ker L                    \\
    v + u \in \ker L                       \\
\end{align*}
The dimension of the kernel is called the \textbf{nullity}.
\begin{example}
    \begin{align*}
        L & : Ax = y                        \\
        A & =  \begin{bmatrix}
            1 & -1, & 0 \\
            0 & -2  & 4 \\
        \end{bmatrix} & \\
    \end{align*}
    To calculate \( \ker A\) simply set \(y\) to the zero vector and solve for \(x\).
    \begin{align*}
        \begin{bmatrix}
            1 & -1, & 0 \\
            0 & -2  & 4 \\
        \end{bmatrix} \begin{bmatrix}
            x_1 \\
            x_2 \\
            x_3 \\
        \end{bmatrix} = \begin{bmatrix}
            0 \\ 0 \\ 0
        \end{bmatrix} \\
        x_1 - x_2       = 0 \rightarrow x_1 = x_2                                          \\
        -2 x_2 + 4 x_3  = 0 \rightarrow x_2 = 2 x_3                                        \\
    \end{align*}
    The kernel is:
    \begin{align*}
        \ker L & = \setb{
            c \cdot \begin{bmatrix}
                2 \\ 2 \\ 1
            \end{bmatrix}
        }{c \in \mathbb{C} }
    \end{align*}
    A concrete example:
    \begin{align*}
        \begin{bmatrix}
            4 \\ 4 \\ 2
        \end{bmatrix}                 & \in \ker L                                                \\
        L \left(\begin{bmatrix}
            4 \\ 4 \\ 2
        \end{bmatrix} \right) & = \begin{bmatrix}
            1 \cdot 4  -1 \cdot 4  + 0 \cdot 2 \\
            0 \cdot 4  -2 \cdot 4 + 4 \cdot 2  \\
        \end{bmatrix} = \begin{bmatrix}
            0 \\ 0
        \end{bmatrix} \\
    \end{align*}
    \begin{example}

    \end{example}
    \section{Rank-Nullity theorem}
    If \(L : V \rightarrow W\) is a linear tranformation then it.
    \begin{equation}
        \rank L + \nullity T = \dim (image T) + \dim(\ker(T)) = \dim(V)
    \end{equation}
    \begin{example}
        \begin{align*}
            L          & : \begin{bmatrix}
                1 & 0 \\ 0 & 1
            \end{bmatrix} x = y \\
            image T    & = \mathbb{R}^2                     \\
            \rank L    & = 2                                \\
            \ker T     & = \{ \begin{bmatrix}
                0 \\ 0
            \end{bmatrix} \} \\
            \nullity L & = 0                                \\
            \dim V     & = \rank L + \nullity T = 2 +0 = 2
        \end{align*}
    \end{example}
    \begin{example}
        \begin{align*}
            L      & : \begin{bmatrix}
                1 & 2 & 4 \\ 2 & 4 & 8
            \end{bmatrix} x = y \\
            imag L & = \setb{
                \begin{bmatrix}
                    c \\ 2 c
                \end{bmatrix}
            }{c \in \mathbb{R}}
        \end{align*}
        \begin{align*}
            \rank L = 1 \\
            \ker L = \setb{
                \begin{bmatrix}
                    -2 c_1 - 4 c_2 \\ c_1 \\  c_2
                \end{bmatrix}
            }{ c_1, c_2 \in \mathbb{R}}
            \nullity A = 2
        \end{align*}
    \end{example}
\end{example}