\section*{Exercise 26.1-7}
\subsubsection*{Show that a flow network $G$ with vertex capacities can be transformed into a normal flow network $G'$. How many vertices and edges does $G'$ have?}
Let the $G$, $G'$, $v$ and $l$ be defined as in the exercise.

First we describes how a network $G_x$ where only one of the vertices, $x$, have a capacity constraint $l(x)$ and how it can be transformed into a new network without the vertex capacity constraint.

We define the set $E(x)$ of edges that have an edge going into the vertex with a capacity constraint. That is $E(x)  = \{(v,x) \in E: l(x)>0\}$.

To transform the network, we can augment $G_x$ with a new vertex $x'$ such that  $G_x' = (V',E')$ where $V' = V \cup \{x'\}$. We define $E'$
as $E' = E\setminus E(x) \cup E'(x) \cup \{(x',x)\}'$ where $E'(x)$ is the set of edges in $E(x)$, but with entrance point into the new vertex $x'$. So the set of edges $E'$ in the new transformed network consist of all edges in the original network, that did not have an edge going into the vertex with a capacity constraint. And all the edges that did go into the vertex with a capacity constraint have been replaced with new edges that goes into the new vertex $x'$. And a new edge have been introduced that goes from the new vertex $x'$ to the old vertex $x$. Hence in total one new vertex and one new edge have been introduced and none have been removed.

\begin{figure}[H]
\caption{\textit{Augmenting the network $G_x$ to $G_x'$}}
\vskip .2cm
\centering
\subfloat[Introducing the new vertex $x'$]{\includegraphics[angle=0,width=12cm]{26-1-7}}
\end{figure}

The capacity of the edges in the transformed network, where no vertex have a capacity is
\begin{align*}
c_{G_x'}(u,v) =
\begin{cases}
c_{G_x}(u,x), & (u,v) = (u,x')\\
l(x), & (u,v) = (x',x)\\
c_{G_x}(u,v), & \text{else}
\end{cases}
\end{align*}
That is, the capacity for an edge going into $x'$ is the same as the capacity of going into $x$ in the $G_x$ network. The new edge have a capacity of $l(x)$ that is the capacity of the vertex in the $G_x$ network. In all other cases the capacity in the transformed network is the same as in the original network.

\begin{figure}[H]
\caption{\textit{Augmenting the network $G_x$ to $G_x'$}}
\vskip .2cm
\centering
\subfloat[Setting the edge capacities]{\includegraphics[angle=0,width=12cm]{26-1-7-withCapacity}}
\end{figure}

We have now shown how a network with a single vertex capacity constraint can be transformed into a new network without a capacity constraint. This can easily be generalized to an arbitrary number of vertex capacity constraint. If $x$ and $x'$ is replaced with $x_i$ and $x'_i$ in the above, all of what we have shown still holds, and for each vertex with a capacity constraint, there will be one new vertex and one new edge. Hence if all of the vertices, including the source and sink, had a constraint there would be $\abs{V}$ extra vertices and edges. So the total number of vertices in $G'$ would be $2\abs{V}$ and the total number of edges in $G'$ would be $\abs{E}+\abs{V}$.


\section{Relative Homology}
First, let's recall a little about homology. We showed that homology factors as a functor $\mathbf{Top}\to\mathrm{h}\mathbf{Top}\to\mathbf{GrAb}$. As a corollary:
\begin{corollary}
If $f:X\to Y$ is a homotopy equivalence (i.e., an isomorphism in the homotopy category), the induced map on homology $ H_\ast(f): H_\ast(X)\to H_\ast(Y)$ is an isomorphism. This is the same thing as saying that if $f:X\to Y$ does not induce an isomorphism on homology, then $f$ can't be a homotopy equivalence. Homology's therefore often used to distinguish spaces.
\end{corollary}
\begin{example}
If $X\times\mathbf{R}^n\to X$ is the projection map, this is a homotopy equivalence because $\mathbf{R}^n$ is contractible. Therefore, $ H_\ast(X\times\mathbf{R}^n)\cong H_\ast(X)$.
\end{example}
\subsection{Towards computing homology}
Fix a space $W$, and consider the functor $X\mapsto [W,X]$. This is basically uncomputable - if you could, you'll win a Fields medal! But there's something more that homology has going for it is that it's ``local''. What do we mean by this? Homology is a bit like a measure:
	\begin{enumerate}
	\item If $A\subseteq X$ is a subspace, then $ H_\ast(X)$ is related to $ H_\ast(A)+ H_\ast(X-A)$. Called the lexseq of a pair.
	\item The homology $ H_\ast(A\cup B)$ is like $ H_\ast(A)+ H_\ast(B)- H_\ast(A\cap B)$. Called the Mayer-Vietoris sequence.
	\end{enumerate}
The thing we use is the notion of exact sequences. Let me tell you about exact sequences.
\subsection{Exact sequences}
This is a story about abelian groups.
\begin{definition}
Let $A\xrightarrow{i} B\xrightarrow{p} C$ be a sequence of abelian groups and homomorphisms. We say that the sequence is exact (at $B$) if $\ker p=\img i$, i.e., $p\circ i=0$ with no room for error.
\end{definition}
\begin{example}
If you have a chain complex $\cdots\to C_{n+1}\to C_n\to C_{n-1}\to\cdots$, it's exact at $C_n$ if the homology $ H_n(C)$ in dimension $n$ is zero. So homology is the obstruction to exactness.
\end{example}
\begin{example}
$0\to A\xrightarrow{i}B$ is exact iff $i$ is injective, and $B\xrightarrow{p}C\to 0$ is exact iff $p$ is surjective.
\end{example}
There's a beautiful book by Eilenberg and Steenrod, published in 1952, which was the founding of algebraic topology.
\begin{example}
If you have a sequence that's exact at every point, it's called a long exact sequence (henceforth called lexseq in these notes). If you have a sequence like $0\to A\xrightarrow{i} B\xrightarrow{p} C\to 0$ that's exact, then this is called a short exact sequence (henceforth called sexseq in these notes). This means that $p\circ i=0$, $i$ is injective, $p$ is surjective. Also, this sequence factors like:
	\begin{equation*}
	\xymatrix{\ker(p) \ar[dr] & & \\
	A\ar[u]\ar[r]^i & B\ar[r]^p\ar[dr] & C\\
	 & & \mathrm{coker}(i)\ar[u]}
	\end{equation*}
So $A\cong \ker p$ and $B\cong \mathrm{coker}(i)$. These things are equivalence to short exactness.
\end{example}
Let's see how this appears in algebraic topology.
\begin{definition}
A pair of spaces is a space $X$ together with subspaces $A\subseteq X$, denoted $(X,A)$. We have a new category, called $\mathbf{Top}_2$ where morphisms $(X,A)\to (Y,B)$ are maps $f:X\to Y$ that take $A\to B$. There are two functors $\mathbf{Top}\to \mathbf{Top}_2$, sending $X\mapsto (X,\emptyset)$ and $X\mapsto (X,X)$. There are also two functors back to $\mathbf{Top}$, sending $(X,A)\mapsto A$ or $(X,A)\mapsto X$.
\end{definition}
How does this behave on the level of chain complexes? If I have a pair $(X,A)$, I get a map $\Sin_n(A)\to \Sin_n(X)$ that's clearly injective. Is this a split monomorphism? Yes, unless $A=\empty$, because you can choose a point in $A$ and send everything not in $A$ to that point. Let's now apply the free abelian group functor to get $S_n(A)\to S_n(X)$. Is this a monomorphism? Yes, because monomorphisms are preserved by this functor. This is also split because being a split mono is a categorical property. This is split even when $A=\empty$ because then $S_n(\empty)=0$.
\begin{definition}
The relative $n$-chains is defined as $S_n(X,A):= S_n(X)/S_n(A)$. So we have a sexseq (ses) $0\to S_n(A)\to S_n(X)\to S_n(X,A)\to 0$. Is $S_n(X,A)$ free abelian if $S_n(X)$ and $S_n(A)$ are? If you have a ses $0\to A\to B\to C\to 0$ such that $A\to B$ is split, then it's homework to show that $B\cong A\oplus C$. So $C$ must also be free abelian if $B$ and $A$ are; i.e., $S_n(X,A)$ is free abelian.
\end{definition}
\begin{example}
Consider $\Delta^n$, which contains its boundary $\partial\Delta^n:=\bigcup \img d_i\simeq S^{n-1}$. We have the identity map $\iota_n:\Delta^n\to \Delta^n$, the universal $n$-simplex, which is in $\Sin_n(\Delta^n)\subseteq S_n(\Delta^n)$. Its boundary $\partial\iota_n\in S_{n-1}(\Delta^n)$, but it actually lands in $S_{n-1}(\partial\Delta^n)$. So $\partial\iota_n$ is \emph{not} a boundary in $\partial\Delta^n$, as we'll see, but it certainly is a cycle. So it determines a homology class, $[\partial\iota_n]$, which, it turns out, generates $ H_{n-1}(\partial\Delta^n)\simeq H_{n-1}(S^{n-1})\cong\mathbf{Z}$.

I can think of $\iota_n\in S_n(\Delta^n,\partial\Delta^n)$, or rather the class of it mod $S_n(\partial\Delta^n)$. It's a relative chain. Is it a cycle? Let's branch off a bit.
\end{example}
Consider the ses $0\to S_n(A)\to S_n(X)\to S_n(X,A)\to 0$. A $c\in S_n(X)$ determines a relative cycle if $\partial c\in S_{n-1}(A)$. I'm sorry, I've messed this up a little bit. There's so much to say here. I'm getting ahead of myself a little bit here. What I meant to say is, let's think of what $\partial$ does. We have a map of ses:
\begin{equation*}
\xymatrix{0\ar[d]\ar[r] & S_n(A)\ar[d]\ar[r] & S_n(X)\ar[d]\ar[r] & S_n(X,A)\ar@{-->}[d]\ar[r] & 0\ar[d]\\
0\ar[r] & S_{n-1}(A)\ar[r] & S_{n-1}(X)\ar[r] & S_{n-1}(X,A)\ar[r] & 0}
\end{equation*}
Does the dotted map exist? We can pull $\overline{c}\in S_n(X,A)$ to some $c\in S_n(X)$, and then define $\partial\overline{c}$ to be the pushforward of $\partial c$. Is this well-defined? If $c,c^\prime$ both map to $\overline{c}$, then $c-c^\prime=0$, so there's some $a$ in $S_n(A)$ that is sent to $c-c^\prime$, and $\partial$ pushes this forward to say that $\partial a$ maps to $\partial(c-c^\prime)=\partial c-\partial c^\prime$. Since we're quotienting out by $S_{n-1}(A)$, this means that the pushforwards of $\partial c$ and $\partial c^\prime$ are the same. I'll leave it to you (although Professor Miller explained this in detail) to show that $\partial^2=0$.

Now let's continue. A class $c\in S_n(X)$ gives a relative cycle if and only if $\partial c\in S_{n-1}(A)$ because we want $\partial c$ in $S_n(X,A)$ to be zero. So $\iota_n\in S_n(\Delta^n,\partial\Delta^n)$ is indeed a relative cycle since $\partial\iota_n\in S_{n-1}(\partial\Delta^n)$. Similarly, a class $c\in S_n(X)$ is a relative boundary if and only if there is a $b\in S_{n+1}(X)$ such that $\partial b=c\bmod S_n(A)$, i.e., $\partial b-c\in S_n(A)$. So $\iota_n\in S_n(\Delta^n,\partial\Delta^n)$ isn't a relative boundary. Therefore $ H_n(\Delta^n,\partial\Delta^n)\cong\mathbf{Z}=\langle[\iota_n]\rangle$ where $ H_n(X,A)$ denotes the relative homology. This stuff takes a little bit of time to get used to.


\section*{Clustering}
\subsection*{k-mean}
$\hat{R}(\mu) = \hat{R}(\mu_1,...,\mu_k) = \sum_{i=1}^n \underset{j\in\{1,...k\}}{\operatorname{min}}||x_i-\mu_j||_2^2$\\
$\hat{\mu} = arg \underset{\mu}{\operatorname{min}}\hat{R}(\mu)$\\
not convex! $\rightarrow$ only local optimum!\\
choosing k is difficult (plot)!\\
\textbf{Algorithm (Lloyd's heuristic):}\\
Initialize cluster centers $\mu^{(0)} = [\mu_1^{(0)},...,\mu_k^{(0)}]$\\
While not converged\\
$z_i \leftarrow arg \underset{j\in\{1,...,k\}}{min}||x_i - \mu_j^{(t-1)}||_2^2$; $\mu_j^{(t)} \leftarrow \frac{1}{n_j}\sum_{i:z_i=j}x_i$

\subsection*{k-mean++}
- Start with random data point as center\\
- Add centers 2 to k randomly, proportionally to squared distance to closest selected center\\
for $j=2$ to $k$:
$i_j$ sampled with prob.\\
$P(i_j=i) = \frac{1}{z} \underset{1\leq l<j}{min}||x_i-\mu_l||_2^2$; $\mu_j \leftarrow x_{i_j}$


\documentclass[10pt]{article}

\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm,amssymb, graphicx, multicol, array, enumerate, gensymb}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}

\newenvironment{problem}[2][Problem]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}

\begin{document}

\title{Mathematics problems}
\date{}
\maketitle

 \section{Elementary algebra}

\begin{problem}{1.1}
Simplify $$\frac{x^{n+2}}{x^{n-2}}$$
\end{problem}

\begin{problem}{1.2}
Solve for $x$:
$$x^{-1}*8=2$$
\end{problem}

\begin{problem}{1.3}
Calculate the missing value. If $a=5$ and $b=10$ then $(a^b)^0=\dots$
\end{problem}

\begin{problem}{1.4}
Calculate
$$\frac{\sqrt{4x}}{\sqrt{x}}$$
\end{problem}

\begin{problem}{1.5}
Solve for $x$:
$$x^2+(x+1)^2=(x+2)^2$$
\end{problem}

\begin{problem}{1.6}
Find the solution set for the inequality below:
$$2^x>1024$$
\end{problem}

\section{Functions of one variable}

\begin{problem}{2.1 (Based on SYD 2.5.6)}
The relationship between temperatures measured in Celsius and Fahrenheit is linear. 0\degree C is equivalent to 32\degree F and 100\degree C is the same as 212\degree F.
 Which temperature is measured by the same number on both scales?
\end{problem}

\begin{problem}{2.2}
Take the following function $f(x)=5x+4$. Find y if $f(3)=y$.
\end{problem}

\begin{problem}{2.3}
Find all values of x that satisfy:
$$x^2-4x+3=0$$
\end{problem}

\begin{problem}{2.4}
Assume that you invest 10 HUF for 90 years with a yearly compound interest of 2\%. How much money do you receive 90 years later?
\end{problem}

\begin{problem}{2.5}
Calculate the following value
$$e^{\ln 5}$$
\end{problem}

\section{Calculus}

\begin{problem}{3.1}
Calculate the following sum
$$\sum\limits_{i=1}^{\infty} \frac{12}{6^i}$$
\end{problem}

\begin{problem}{3.2}
Find the following limit
$$\lim\limits_{x \rightarrow 1}\frac{6^{1-x}}{x}$$
\end{problem}

\begin{problem}{3.3}
Find the slope of the function $f(x)=x^5-8$ at $x=-3$.
\end{problem}

\begin{problem}{3.4}
Find the following derivative
$$\frac{\mathrm{d}}{\mathrm{d}\, x} \frac{x^3+2x-1}{x-2}$$
\end{problem}

\begin{problem}{3.5}
Find the following second derivative
 $$\frac{\mathrm{d^2}}{\mathrm{d}\, x^2} 4x^4+4x^2$$
\end{problem}

\begin{problem}{3.6}
Find the following derivative:
$$\frac{\mathrm{d}}{\mathrm{d}\, x} \frac{\ln x}{e^x}$$
\end{problem}

\begin{problem}{3.7}
Consider the following function. Find all of its stationary points and classify them as local minima, local maxima or inflection points. Also decide whether it is convex or concave. If it has one or more inflection points then define where it is locally concave or locally convex. (You should create a table like we did in class)
$$f(x)=3x^2-5x+2$$
\end{problem}

\begin{problem}{3.8}
Let $f(x,y)=x^2+y^3$. Calculate $f(2,3)$
\end{problem}

\begin{problem}{3.9}
Consider the following function: $f(x,y)=\ln(x-y)$. For what combinations of $x$ and $y$ is this function defined?
\end{problem}

\begin{problem}{3.10}
Find the following partial derivative:
$$\frac{\partial}{\partial \, x} x^5+xy^3$$
\end{problem}

\begin{problem}{3.11}
Find the local maxima or minima of the following function:
$$f(x,y)=x^2y^2+10$$
\end{problem}

\begin{problem}{3.12}
Solve the following constrained optimization problem using Lagrange's method:
$\max x^2y^2$ s.t. $x+y=10$
\end{problem}

\section{Linear algebra}

\begin{problem}{4.1}
Take the following matrices:
$$A=\begin{bmatrix} 2 & 6\\ 5 & 1 \\ 1 & 9\end{bmatrix}$$
$$B=\begin{bmatrix} 1 & 1 & 7\\2 & 8 & 2\end{bmatrix}$$
What is $A \cdot B$?
\end{problem}

\begin{problem}{4.2}
Take the following matrices:
$$A=\begin{bmatrix} 2 & 2\\ 4 & 6 \\ 1 & 3\end{bmatrix}$$
$$B=\begin{bmatrix} 1 & 9 & 1\\2 & 1 & 2\end{bmatrix}$$
What is $B \cdot A$?
\end{problem}

\begin{problem}{4.3}
What is the transpose of the following matrix?
$$\begin{bmatrix}7.1 & 9.1 & 4.7\\ 2 & 7.8 & 1.1 \\ 4 & 4.44 & 0\end{bmatrix}$$
\end{problem}

\begin{problem}{4.4}
Calculate the determinant of
$$\begin{bmatrix}1 & 9 \\ 2 & 8 \end{bmatrix} $$
\end{problem}

\section{Probability theory}

\begin{problem}{5.1}
You run an experiment where you throw a (regular, 6 sided) dice twice. The first number you get will be the first digit of a two-digit number, while the second number you get will be the second digit of the same two-digit number. What is the sample space of your experiment?
\end{problem}

\begin{problem}{5.2}
Assume that in a certain country 1\% of the population uses a certain drug. You have a way to test drug use, which will give you a positive result in 99\% of the cases where the individual is indeed a drug user and a negative result in 99.5\% of the cases where the individual doesn't use the drug. What is the probability that a randomly selected citizen will have a positive drug test?
\end{problem}

\begin{problem}{5.3}
Assume that in a certain country 1\% of the population uses a certain drug. You have a way to test drug use, which will give you a positive result in 99\% of the cases where the individual is indeed a drug user and a negative result in 99.5\% of the cases where the individual doesn't use the drug. What is the probability that someone with a positive drug test is indeed a drug user?
\end{problem}
\end{document}


\section{Mathematics}

To perform mathematical operations such as {\tt max}, and {\tt rsin}, you may use this Add-on. It provides random number generations and contains a large amount of Mathematical primitives.

\subsection*{abs} \index{\verb+abs+}

{\tt abs} {\it number}
\newline\newline
Output the absolute value of the input
\begin{verbatimtab}
@> print abs -4.78
4.78
\end{verbatimtab}

\subsection*{av} \index{\verb+av+}

{\tt av} {\it thing1 thing2 thing3 thing4 ...}
\newline\newline
Output the average value of the input. The input may be a {\it number} or a {\it list}. Any input of other types will be ignored.
\begin{verbatimtab}
@> print av 3 6 2 9.6 [3 2 4 0.5]
3.762
\end{verbatimtab}

\subsection*{ceil} \index{\verb+ceil+}

{\tt ceil} {\it number}
\newline\newline
Output the smallest integer that is not less than the input.
\begin{verbatimtab}
@> print ceil 4.576
5
\end{verbatimtab}

\subsection*{deg2rad} \index{\verb+deg2rad+}

{\tt deg2rad} {\it number}
\newline\newline
Output the equivalent angle in radians for the degree given (converting degrees to radians)
\begin{verbatimtab}
@> print deg2rad 120
2.094
\end{verbatimtab}

\subsection*{difference} \index{\verb+difference+}

{\tt difference} {\it number1 number2}
\newline\newline
Output the difference between {\it number1} and {\it number2}.
\begin{verbatimtab}
@> print difference 4 5
1
\end{verbatimtab}

\subsection*{dacos} \index{\verb+dacos+}

{\tt dacos} {\it number}
\newline\newline
Output the arccosine of the input given in degrees
\begin{verbatimtab}
@> print dacos 45
0.9033
\end{verbatimtab}

\subsection*{dasin} \index{\verb+dasin+}

{\tt dasin} {\it number}
\newline\newline
Output the arcsine of the input given in degrees
\begin{verbatimtab}
@> print dasin 180
nan
\end{verbatimtab}

\subsection*{datan} \index{\verb+datan+}

{\tt datan} {\it number}
\newline\newline
Output the arctangent of the input given in degrees
\begin{verbatimtab}
@> print datan 45
0.6658
\end{verbatimtab}

\subsection*{dcos} \index{\verb+dcos+}

{\tt dcos} {\it number}
\newline\newline
Output the cosine of the input given in degrees
\begin{verbatimtab}
@> print dcos 180
-1
\end{verbatimtab}

\subsection*{dcosh} \index{\verb+dcosh+}

{\tt rcosh} {\it number}
\newline\newline
Output the hyperbolic cosine of the input given in degrees
\begin{verbatimtab}
@> print dcosh 180
11.59
\end{verbatimtab}

\subsection*{dsin} \index{\verb+dsin+}

{\tt dsin} {\it number}
\newline\newline
Output the sine of the input given in degrees
\begin{verbatimtab}
@> print dsin 60
0.866
\end{verbatimtab}

\subsection*{dsinh} \index{\verb+dsinh+}

{\tt dsinh} {\it number}
\newline\newline
Output the hyperbolic sine of the input given in degrees
\begin{verbatimtab}
@> print dsinh 180
11.55
\end{verbatimtab}

\subsection*{dtan} \index{\verb+dtan+}

{\tt dtan} {\it number}
\newline\newline
Output the tangent of the input given in degrees
\begin{verbatimtab}
@> print dtan 140
-0.8391
\end{verbatimtab}

\subsection*{dtanh} \index{\verb+dtanh+}

{\tt dtanh} {\it number}
\newline\newline
Output the hyperbolic-tangent of the input given in degrees
\begin{verbatimtab}
@> print dtanh 78
0.8767
\end{verbatimtab}

\subsection*{erf} \index{\verb+erf+}

{\tt erf} {\it number}
\newline\newline
Output the error function of the input.
\begin{verbatimtab}
@> print erf 4.5
1
\end{verbatimtab}

\subsection*{erfc} \index{\verb+erfc+}

{\tt erfc} {\it number}
\newline\newline
Output the complementary error function of the input.
\begin{verbatimtab}
@> print erfc 4.5
1.96616e-10
\end{verbatimtab}

\subsection*{exp} \index{\verb+exp+}

{\tt exp} {\it number}
\newline\newline
Output the exponential value of the input.
\begin{verbatimtab}
@> print exp 4.7
109.95
\end{verbatimtab}

\subsection*{floor} \index{\verb+floor+}

{\tt floor} {\it number}
\newline\newline
Output the largest integer that is not greater than the input.
\begin{verbatimtab}
@> print floor 4.576
4
\end{verbatimtab}

\subsection*{gamma} \index{\verb+gamma+}

{\tt gamma} {\it number}
\newline\newline
Output the log gamma function of the input.
\begin{verbatimtab}
@> print gamma 4.5
2.45374
\end{verbatimtab}

\subsection*{hypot} \index{\verb+hypot+}

{\tt hypot} {\it number1 number2}
\newline\newline
Output the Euclidean distance function of the input
\begin{verbatimtab}
@> print hypot 2.5 6.8
7.245
\end{verbatimtab}

\subsection*{incr} \index{\verb+incr+}

{\tt incr} {\it word (number)}
\newline\newline
Increment the value stored in a variable by 1 or by the number specified as the second input. The first input must be a known variable name.
\begin{verbatimtab}
@> make "a 1
@> incr "a -1
@> print :a
0
\end{verbatimtab}

\subsection*{int} \index{\verb+int+}

{\tt int} {\it number}
\newline\newline
Output a cast of the input into an integer.
\begin{verbatimtab}
@> print int 4.576
4
\end{verbatimtab}

\subsection*{log10} \index{\verb+log10+}

{\tt log10} {\it number}
\newline\newline
Output the logarithm (base 10) of the input.
\begin{verbatimtab}
@> print log10 5000
3.699
\end{verbatimtab}

\subsection*{ln} \index{\verb+ln+}

{\tt ln} {\it number}
\newline\newline
Output the natural logarithm of the input.
\begin{verbatimtab}
@> print ln 5000
8.517
\end{verbatimtab}

\subsection*{max} \index{\verb+max+}

{\tt max} {\it thing1 thing2 thing3 thing4 ...}
\newline\newline
Output the maximum value of the inputs.  The inputs may be {\it numbers} or {\it lists}. Any input of other types
will be ignored.
\begin{verbatimtab}
@> print max 3 6 2 9.6 [3 2 4 0.5]
9.6
\end{verbatimtab}

\subsection*{min} \index{\verb+min+}

{\tt min} {\it thing1 thing2 thing3 thing4 ...}
\newline\newline
Output the minimum value of the inputs.  The inputs may be {\it numbers} or {\it lists}. Any input of other types
will be ignored.
\begin{verbatimtab}
@> print min 3 6 2 9.6 [3 2 4 0.5]
0.5
\end{verbatimtab}

\subsection*{minus} \index{\verb+minus+}

{\tt minus} {\it number}
\newline\newline
Output the negative value of the input.
\begin{verbatimtab}
@> print minus -5
5
@> print minus 45.78
-45.78
\end{verbatimtab}

\subsection*{product} \index{\verb+product+}

{\tt product} {\it thing1 thing2 thing3 thing4 ...}
\newline\newline
Output the product of all the inputs.  The inputs may be {\it numbers} or {\it lists}. Any input of other types
will be ignored.
\begin{verbatimtab}
@> print product 3 6 2 9.6 [3 2 4 0.5]
4147.2
\end{verbatimtab}

\subsection*{quotient} \index{\verb+quotient+}

{\tt quotient} {\it number1 number2 }
\newline\newline
Output the quotient of {\it number1} over {\it number2.}
\begin{verbatimtab}
@> print quotient 5 7
0.714
\end{verbatimtab}

\subsection*{modulo} \index{\verb+modulo+}

{\tt modulo} {\it number1 number2 }
\newline\newline
Output the remainder from performing the integer quotient of {\it number1} over {\it number2}. Both numbers must be integers.
\begin{verbatimtab}
@> print modulo 10 2
0
\end{verbatimtab}

\subsection*{power} \index{\verb+power+}

{\tt power} {\it number1 number2}
\newline\newline
Output the power of {\it number1} by {\it number2}
\begin{verbatimtab}
@> print power 3.4 5
454.354
\end{verbatimtab}

\subsection*{racos} \index{\verb+racos+}

{\tt racos} {\it number}
\newline\newline
Output the arccosine of the input given in radians
\begin{verbatimtab}
@> print racos 0.4
1.159
\end{verbatimtab}

\subsection*{rad2deg} \index{\verb+rad2deg+}

{\tt rad2deg} {\it number}
\newline\newline
Output the corresponding angle in degrees, given the angle in radians  (converting from radians into degrees)
\begin{verbatimtab}
@> print rad2deg 2.094
120
\end{verbatimtab}

\subsection*{random} \index{\verb+random+}

{\tt random} {\it (min) max}
\newline\newline
Output a random number between the given min and max.  If only one input is given, it will be interpreted as the max and the output will be a number between 0 and max.
\begin{verbatimtab}
@> print random 40
15
\end{verbatimtab}

\subsection*{rcos} \index{\verb+rcos+}

{\tt rcos} {\it number}
\newline\newline
Output the cosine of the input given in radians
\begin{verbatimtab}
@> print rcos 60
-0.952
\end{verbatimtab}

\subsection*{rcosh} \index{\verb+rcosh+}

{\tt rcosh} {\it number}
\newline\newline
Output the hyperbolic cosine of the input given in radians
\begin{verbatimtab}
@> print rcosh 0.56
1.160
\end{verbatimtab}

\subsection*{rasin} \index{\verb+rasin+}

{\tt rasin} {\it number}
\newline\newline
Output the arcsine of the input given in radians
\begin{verbatimtab}
@> print rasin 0.4
0.411
\end{verbatimtab}

\subsection*{rsin} \index{\verb+rsin+}

{\tt rsin} {\it number}
\newline\newline
Output the sine of the input given in radians
\begin{verbatimtab}
@> print rsin 60
-0.304
\end{verbatimtab}

\subsection*{rsinh} \index{\verb+rsinh+}

{\tt rsinh} {\it number}
\newline\newline
Output the hyperbolic sine of the input given in radians
\begin{verbatimtab}
@> print rsinh 0.56
0.589
\end{verbatimtab}

\subsection*{ratan} \index{\verb+ratan+}

{\tt ratan} {\it number}
\newline\newline
Output the arctangent of the input given in radians
\begin{verbatimtab}
@> print ratan 0.6
0.540
\end{verbatimtab}

\subsection*{rtan} \index{\verb+rtan+}

{\tt rtan} {\it number}
\newline\newline
Output the tangent of the input given in radians
\begin{verbatimtab}
@> print rtan 60
0.32
\end{verbatimtab}

\subsection*{rtanh} \index{\verb+rtanh+}

{\tt rtanh} {\it number}
\newline\newline
Output the hyperbolic tangent of the input given in radians
\begin{verbatimtab}
@> print rtanh 0.7
0.604
\end{verbatimtab}

\subsection*{sqrt} \index{\verb+sqrt+}

{\tt sqrt} {\it number}
\newline\newline
Output the square root of the input.
\begin{verbatimtab}
@> print sqrt 25
5
\end{verbatimtab}

\subsection*{sum} \index{\verb+sum+}

{\tt sum} {\it thing1 thing2 thing3 thing4 ...}
\newline\newline
Output the sum of the all the inputs.  The inputs may be {\it numbers} or {\it lists}. Any input of other types
will be ignored.
\begin{verbatimtab}
@> print sum 3 6 2 9.6 [3 2 4 0.5]
30.01
\end{verbatimtab}


\newpage
\section{Introduction to logistic regression}
In this section, we introduce techniques related to basic logistic regression, see \cite{gelman2006data} for more details.
\subsection{Logistic regression}\label{sec:LR}
%We first introduce the next definition of the set of linearly classifiable weights.

Assume that we are given $k$ linearly separable sets $A_1,A_2,\cdots,A_k\in \mathbb{R}^d$, we define the set of classifiable weights as
	\begin{equation}
	\bm\Theta = \{\bm\theta = (W,b): w_ix+b_i>w_jx+b_j,~\forall x\in A_i, j\neq i, i= 1,\cdots,k\}
	\end{equation}
	which means those $(W,b)$ can separate $A_1,A_2,\cdots,A_k$ correctly.

Our linearly separable assumption implies that $\bm\Theta\neq \emptyset$.
Now we know the existence of linearly classifiable weights. But how can we find one element in $\bm\Theta$?

\begin{definition}[soft-max]\label{softmax}
 Given $s = (s_1,s_2,\cdots,s_k)^T\in \mathbb{R}^k$, we define the soft-max mapping $\sigma: \mathbb{R}^k \rightarrow\mathbb{R}^k $ as
 \begin{equation}
 \sigma(s)  = \frac{e^{s}}{e^{s}\cdot \bm{1}} = \frac{1}{\sum\limits_{i=1}^k e^{s_i}}
 \begin{pmatrix}
 e^{s_1}\\
 e^{s_2}\\
 \vdots\\
 e^{s_k}
 \end{pmatrix}
 \end{equation}
 where $e^{s} =
 \begin{pmatrix}
 e^{s_1}\\
 e^{s_2}\\
 \vdots\\
 e^{s_k}
 \end{pmatrix}
 $, $\bm{1} =
 \begin{pmatrix}
 1\\
 1 \\
 \vdots \\
 1
 \end{pmatrix} \in\mathbb{R}^k$.
\end{definition}

\begin{definition}
 Given parameter $\bm\theta = (W,b)$, we define a feature mapping $\bm p: \mathbb{R}^d \rightarrow \mathbb{R}^k$ as
 \begin{equation}
 \bm p(x; \bm\theta)  = \sigma(Wx+b) = \frac{1}{\sum\limits_{i=1}^k e^{w_i x+b_i}}
 \begin{pmatrix}
 e^{w_1 x+b_1}\\
 e^{w_2 x+b_2}\\
 \vdots\\
 e^{w_k x+b_k}
 \end{pmatrix}
 = \begin{pmatrix}
 p_1(x; \bm\theta) \\
 p_2(x; \bm\theta) \\
 \vdots \\
 p_k(x; \bm\theta)
 \end{pmatrix}
 \end{equation}
 where the $i$-th component
 \begin{equation}\label{key}
 p_i(x; \bm\theta) = \frac{e^{w_i x+b_i}}{\sum\limits_{i=1}^k e^{w_i x+b_i}}.
 \end{equation}
\end{definition}

The soft-max mapping have several important properties.
\begin{enumerate}
	\item
	$\displaystyle 0< p_i(x; \bm\theta) <1,~\sum_i p_i(x; \bm\theta) = 1$.

	This implies that $\bm p(x; \bm\theta)$ can be regarded as a probability distribution of data points which means that given $x\in \mathbb{R}^d$, we have $x\in A_i$ with probability $p_i(x; \bm{\theta})$, $i = 1,\cdots,k$.
	\item
	$p_i(x; \bm\theta)>p_j(x; \bm\theta)\Leftrightarrow w_ix+b_i>w_j x+b_j.$

	This implies that the linearly classifiable weights have an equivalent description as
	\begin{equation}
	\bm{\Theta} = \left\{\bm\theta: p_i(x; \bm\theta)>p_j(x; \bm\theta),~\forall x\in A_i, j\neq i, i= 1,\cdots,k\right\}
	\end{equation}
	\item We usually use the max-out method to do classification. For a given data point $x$, we first use a soft-max mapping to map it to $\bm p(x; \bm\theta)$, then we attach $x$ to the class $i= \arg\max_j p_i(x; \bm\theta)$.

	This means that we pick the label $i$ as the class of $x$ such that $x\in A_i$ has the biggest probability $p_i(x; \bm\theta)$.
\end{enumerate}
More detailed discussion of logistic regression from the probability perspective will be presented in the nearly future.

From the above properties, we can define the following likelihood function to help find elements in $\bm{\Theta}$:
\begin{equation}
P (\bm\theta)=
\prod\limits_{i = 1}^k \prod\limits_{x\in A_i} p_i(x; \bm\theta).
\end{equation}
%The next lemma shows that we can use this function to help us find some linearly classifiable weights.
%\begin{theorem}\label{thm1-H}
%	If $ \bm \theta \notin \Theta$, then there exists $ \bm \theta^* \in\bm \Theta$ such that
%	\begin{equation}\label{key}
%	H( \bm \theta^*) > H(\bm \theta).
%	\end{equation}
%\end{theorem}
Based on the property that
\begin{equation}\label{key}
p_i (x; \bm \theta) = \max_{1\le j \le k} p_j(x; \bm \theta), \quad\forall x \in A_i,\ \bm \theta \in \Theta,
\end{equation}
we may use the next optimization problem
\begin{equation}\label{key}
\max_{\bm \theta\in \bm{\Theta}} P(\bm \theta).
\end{equation}
to find an element in $\bm{\Theta}$.
More precisely, let us introduce the next lemmas (properties) of $P(\bm \theta)$.
\begin{lemma}\label{lemm:H1/2}
	Assume that the sets $A_1,A_2,\cdots,A_k$ are linearly separable. Then we have
	\begin{equation}
	\left\{\bm \theta:~P(\bm\theta)>\frac{1}{2}\right\}\subset \bm{\Theta}.
	\end{equation}
\end{lemma}
\begin{proof}
	It suffices to show that if $\bm\theta \not\in \bm\Theta$, we must have $P(\bm\theta)\leq\frac{1}{2}$.
	For any $\bm\theta \not\in
	\bm\Theta$, there must exist an $i_0$ ,an $x_0\in A_{i_0}$ and a
	$j_0\neq i_0$ such that
	\begin{equation}
	w_{i_0} x_0 + b_{i_0} \leq w_{j_0}x_0 + b_{j_0}.
	\end{equation}
	Then we have
	\begin{equation}
	p_{i_0}(x_0; \bm\theta) \leq \frac{e^{w_{i_0} x_0 + b_{i_0}}}{e^{w_{i_0} x_0+b_{i_0}}+e^{w_{j_0} x_0+b_{j_0}}} \leq\frac{1}{2}.
	\end{equation}
	Notice that $p_i(x; \bm \theta) < 1$ for all $i = 1,\cdots,k$, $x\in A$.
	So
	\begin{equation}
	P(\bm\theta) <  p_{i_0}(x_0; \bm\theta) \leq \frac{1}{2}.
	\end{equation}
\end{proof}

\begin{lemma}
	If $A_1,A_2,\cdots,A_k$ are linearly separable and $\bm\theta \in \bm\Theta$, we have
	\begin{equation}
	\lim_{\alpha\rightarrow +\infty}p_i(x; \alpha\bm\theta) = 1\Leftrightarrow x\in A_i.
	\end{equation}
\end{lemma}
\begin{proof}
	We first note that if  $x\in A_i$,
	\begin{equation}
	 p_i(x,\bm \theta) = \frac{1}{1+\sum\limits_{j\neq i}e^{\alpha[(w_j x+ b_j)-(w_i x+b_i)]}} \to 1, \quad \text{as} \quad \alpha \to \infty.
	\end{equation}
	On the other hand, if $x\not\in A_i$,
	\begin{equation}
	p_i(x; \bm\alpha\bm\theta) = \frac{1}{1+\sum\limits_{j\neq i}e^{\alpha[(w_j x+ b_j)-(w_i x+b_i)]}} \leq \frac{1}{2}.
	\end{equation}
	This implies that if $x\not\in A_i$, $\lim_{\alpha\rightarrow \infty}p_i(x; \alpha\bm \theta)\neq 1$ which is equivalent to the proposition that if $\lim_{\alpha\rightarrow \infty}p_i(x; \alpha\bm \theta)= 1$, then $x\in A_i$.
\end{proof}

\begin{lemma}{\label{thm2}} If $A_1,A_2,\cdots,A_k$ are linearly separable,
	\begin{equation}
	\bm\Theta = \left\{\bm\theta: \lim_{\alpha\rightarrow +\infty}P(\alpha\bm\theta) = 1\right\}.
	\end{equation}
\end{lemma}

\begin{proof}
 	We first note that if $\bm\theta \in\bm\Theta$, we have $\displaystyle\lim_{\alpha\rightarrow +\infty}p_i(x; \alpha\bm\theta) = 1$ for all $x\in A_i$. So
	\begin{equation}
	\lim\limits_{\alpha\rightarrow +\infty} P(\alpha\bm\theta) = \lim\limits_{\alpha\rightarrow +\infty} \prod\limits_{i = 1}^k \prod\limits_{x\in A_i} p_i(x; \alpha\bm\theta) = \prod\limits_{i = 1}^k \prod\limits_{x\in A_i} \lim\limits_{\alpha\rightarrow +\infty}p_i(x; \alpha\bm\theta) = 1.
	\end{equation}
	On the other hand, if $\lim\limits_{\alpha\rightarrow +\infty} P(\alpha\bm\theta) = 1$, there must exist one $\alpha_0>0$ such that $P(\alpha_0\bm\theta) >\frac{1}{2}$. From Lemma~\ref{lemm:H1/2}, we have $\alpha_0\bm\theta\in\bm\Theta$, which means $\bm\theta\in\bm\Theta$.
\end{proof}

These properties above imply that if we can obtain a classifiable weight through maximizing $P(\bm\theta)$, while lemma \ref{thm2} tells us that $P(\bm\theta)$ will not have a global minimum actually.

More specifically, we just need to find some $\bm \theta \in \Theta$ such that
\begin{equation}\label{key}
P(\bm \theta) > \frac{1}{2} \Leftrightarrow  L(\bm \theta) : = -\log P(\bm \theta )  < \log(2).
\end{equation}
%{\bf Question:} how to find these element?

\blankpage
\break
\subsection{Regularized logistic regression}
Here, we start from the regularization term $e^{-\lambda R(\|\bm\theta\|)}$ with
these next properties:
\begin{enumerate}
	\item $\lambda > 0$.
	\item $R(t)$ is a strictly increasing function on $\mathbb{R}^+$ with $R(0) = 0$, $\lim\limits_{t\rightarrow +\infty} R(t) = +\infty$.
	For example, $R(t) = t^2$.
	\item $\|\cdot\|$ is a norm on $R^{k\times(d+1)}$, a commonly used norm  is the
	following Frobenius norm:
	\begin{equation}\label{key}
	\|\bm \theta\|_F = \sqrt{\sum_{i,j}W_{ij}^2 + \sum_i b_i^2}.
	\end{equation}
\end{enumerate}

Based on this regularization term,  we may consider the following
regularized likelihood function $P_\lambda(\bm\theta)$ as
\begin{equation}\label{key}
 P_\lambda(\bm\theta) = P(\bm\theta)e^{-\lambda R(\|\bm\theta\|)}.
\end{equation}

Here, let us define
\begin{equation}\label{key}
\bm\Theta_{\lambda} = \mathop{{\arg\max}}_{\bm\theta}  P_\lambda(\bm\theta),
\end{equation}
where
\begin{equation}\label{key}
\mathop{\arg\max}_{\bm\theta}  P_\lambda(\bm\theta) = \left\{\bm \theta ~:~ P_\lambda(\bm \theta) = \max_{\bm \theta} P_\lambda(\bm \theta) \right\}.
\end{equation}

The next lemma show that the maximal set of modified objective is not empty.
\begin{lemma}Suppose that $A_1,A_2, \cdots, A_k$ are linearly separable, then
	\begin{enumerate}
		\item if $\lambda = 0$, $\bm\Theta_0 = \emptyset$,
		\item $\bm\Theta_{\lambda}$ must be nonempty for all $\lambda>0$.
	\end{enumerate}
\end{lemma}
\begin{proof}
	Lemma~\ref{thm2} shows the first proposition. For the second proposition,
	we notice that
	\begin{enumerate}
		\item $ P_\lambda(\bm 0) = \frac{1}{k^N}$.
		\item $\exists\ M_{\lambda}>0$ such that $e^{-\lambda R(\|\bm\theta\|)}<\frac{1}{k^N}$ whenever $\|\bm\theta\|> M_{\lambda}$ because of the properties of $R(\|\bm\theta\|)$.
	\end{enumerate}
	So a maxima on $\{\bm\theta: \|\bm\theta\| \le M_{\lambda}\}$ must be a global maxima. Then we can easily obtain the result in the lemma from the boundedness and closeness of $\{\bm\theta: \|\bm\theta\| \le M_{\lambda}\}$.
\end{proof}

Furthermore, we have the next theorem which shows that we can indeed get $\Theta$ by maximizing $P_\lambda(\bm \theta)$.
\begin{theorem}\label{thm-L-Theta} If $A_1,A_2,\cdots,A_k$ are linearly separable,
	\begin{equation}\label{key}
	\bm\Theta_{\lambda} \subset \bm\Theta,
	\end{equation}
when $\lambda>0$ and sufficiently small.
\end{theorem}
\begin{proof}
By Lemma~\ref{lemm:H1/2}, we can take $\bm\theta_0\in \bm\Theta$ such that $P(\bm\theta_0)> \frac{3}{4}$.
Then, for any $\lambda < \frac{\log \frac{3}{2}}{R(\|\bm\theta_0\|)}$, $\bm\theta_{\lambda}\in \bm\Theta_{\lambda}$, we have
		\[
		P(\bm\theta_{\lambda}) \geq  P_\lambda(\bm\theta_{\lambda})  \geq P_\lambda(\bm \theta_0) = P(\bm\theta_0)e^{-\lambda R(\|\bm\theta_0\|)} > \frac{3}{4}\cdot \frac{2}{3} = \frac{1}{2},
		\]
		which implies that $\bm \theta_{\lambda} \in \Theta$.
		Thus, for any $ 0< \lambda < \frac{\log \frac{3}{2}}{R(\|\bm\theta_0\|)}$, $\bm\Theta_{\lambda} \subset \bm\Theta$.

\end{proof}

The design of logistic regression is that
maximize $P_\lambda(\bm\theta)$ is equivalent to minimize $-\log P_\lambda(\bm\theta)$, i.e.,
\begin{equation}\label{key}
\max_{\bm \theta} \left\{ P_\lambda(\bm\theta) \right\} \Leftrightarrow \min_{\bm \theta} \left\{ -\log   P_\lambda(\bm\theta)\right\},
\end{equation}
while the second one is more convenient to evaluate the gradient. Meanwhile, we add a regularization term $R(\bm\theta)$ to the objective function which makes the optimization problem has a unique solution.

 Mathematically, we can formulate Logistic regression as
\begin{equation}
\min_{\bm\theta} L_\lambda(\bm \theta),
\end{equation}
where
\begin{equation}\label{eq:logisticlambda}
L_\lambda(\bm \theta)  := -\log P_\lambda(\bm\theta) = -\log P(\bm\theta) + \lambda R(\|\bm\theta\|) = L(\bm\theta) + \lambda R(\|\bm\theta\|),
\end{equation}
with
\begin{equation}\label{logistic}
L(\bm \theta) = - \sum_{i=1}^k \sum_{x\in A_i} \log p_{i}(x;\bm \theta).
\end{equation}

Then we have the next logistic regression algorithm.
\begin{algorithm}[H]
	\caption{Logistic Regression}
	\label{alg:LR-R}
	Given data $A_1, A_2, \cdots, A_k$, find
	\begin{equation}\label{key}
	\bm \theta^* = \mathop{\arg\min}_{\bm \theta}  L_\lambda(\bm \theta),
	\end{equation}
	for some sufficient small $\lambda > 0$.
\end{algorithm}

\begin{remark}
	Here
	\begin{equation}\label{key}
	L(\bm \theta)  = -\log P(\bm\theta),
	\end{equation}
	is known as the loss function of logistic regression model.
	The next reasons may show that why $L(\bm \theta)$ is popular.
	\begin{enumerate}
		\item It is more convenient to take gradient for $L(\bm \theta)$ than $P(\bm \theta)$.
		\item $L(\bm \theta)$ is related the so-called cross-entropy
		loss function which will be discussed in the next section.
		\item $L(\bm \theta)$ is a convex function which will be discussed later.
	\end{enumerate}
\end{remark}

\endinput
\begin{lemma}
	If $A_1,A_2,\cdots,A_k$ are linearly separable, $-\log H(\bm\theta)$ has no global minimum.
\end{lemma}

\begin{theorem}
	$-\log P(\bm\theta) + \lambda R(\bm\theta)$ has a global minimizer for sufficiently small $\lambda>0$.
\end{theorem}

Denote the above objective function as
\begin{equation}
L(\theta,\alpha) = \alpha (\max_i \|w_i\|_2) + \displaystyle\sum_{i=1}^k \displaystyle\sum_{x\in A_i}
\left(\log(\mathbbm{1}^T\exp(W x + b)) - (W x + b)_i\right)
\end{equation}
where $\theta = (W,b)$. And we denote a linear separable parameter set $\Theta$ as
\begin{equation}
\Theta = \{\theta~|w_ix + b_i > w_jx + b_j, x\in A_i,~j\neq i\}
\end{equation}
Set
\begin{equation}
p_i(\theta,x) = \frac{e^{w_i x+b_i}}{\sum\limits_{j=1}^k e^{w_j x+b_j}},   \forall i = 1,\cdots,k.
\end{equation}
and define the LR classifiable parameter set $\bar{\Theta}$ as
\begin{equation}
\bar{\Theta} = \{\theta~|p_i(\theta,x)>p_j(\theta,x), ~x\in A_i,~j\neq i\}
\end{equation}
Easy to observe that
\begin{equation}
\Theta = \bar{\Theta}
\end{equation}

\begin{theorem}
	Assume that $R(\theta)\geq 0$ is an non-negative function such that
	\begin{equation}
	\Theta^*(\lambda)={\rm argmax}\ P(\theta)e^{-\lambda R(\theta)} \neq \emptyset, \quad \forall \lambda>0
	\end{equation}
	then for sufficiently small $\lambda>0$,
	$$ \Theta^*(\lambda)\subset \left\{ \theta: P(\theta)\geq \frac{1}{2} \right\}\subset \Theta.$$
\end{theorem}

\begin{proof}
	Given $\theta_0\in\Theta$. Let $\alpha >0$ be such that
	$$P(\alpha \theta_0)\geq \frac{2}{3}$$
	Let $\lambda>0$ be sufficiently small such that
	$$e^{-\lambda R(\theta)}>\frac{3}{4}$$
	For any $\theta \in \Theta^*$
	$$
	P(\theta)e^{-\lambda R(\theta)} \geq P(\alpha \theta_0) e^{-\lambda R(\alpha\theta_0)} > \frac{2}{3}\cdot \frac{3}{4}=\frac{1}{2}$$
	Then $P(\theta)>\frac{1}{2}$.
\end{proof}

497-6DL/2 Linear Machine Learning Models/2.2-IntroLR.tex


\section{The \upperbound algorithm}
\Label{sec:upperbound}

The \upperbound algorithm of the \cxx Standard Library \cite[\S 28.7.3.2]{cxx-17-draft} is a
variant of binary search and closely related to \specref{lowerbound}.
The signature reads:

\begin{lstlisting}[style = acsl-block]

  size_type
  upper_bound(const value_type* a, size_type n, value_type v)
\end{lstlisting}

As with the other binary search algorithms, \upperbound requires that
its input array is in increasing order.
The index~\inl{ub} returned by \upperbound satisfies the inequality

\begin{align}
\Label{eq:upper-bound-result}
0 \leq \mathtt{ub} \leq n
\end{align}

and is involved in the following implications for a valid index~\inl{k} of the array
under consideration

\begin{alignat}{3}
\Label{eq:upper-bound-left}
0 &\leq k < \mathtt{ub} &&\qquad\Longrightarrow\qquad && a[k] \leq \mathtt{v} \\
\Label{eq:upper-bound-right}
\mathtt{ub} &\leq k < n &&\qquad\Longrightarrow\qquad &&\mathtt{v} < a[k]
\end{alignat}

Conditions~\eqref{eq:upper-bound-left} and~\eqref{eq:upper-bound-right} imply that~\inl{v}
can only occur in the array section \inl{a[0..ub-1]}.
In this sense \upperbound returns a \emph{upper bound} for the
potential indices where \inl{v} can occur.
It also means that the searched-for value \inl{v} can
\emph{never} be located at the index~\inl{ub}.

Figure~\ref{fig:upperbound} is a variant of Figure~\ref{fig:lowerbound} for the case
of \upperbound and the same example array.
The arrows indicate which indices will be returned by \upperbound for a given value.
Note how, compared to Figure~\ref{fig:lowerbound}, only the arrows from values
that \emph{are present} in the array change their target index.

\begin{figure}[hbt]
\centering
\includegraphics[width=0.60\textwidth]{Figures/upper_bound.pdf}
\caption{\Label{fig:upperbound}Some examples for \upperbound}
\end{figure}

\FloatBarrier

\subsection{Formal specification of \upperbound}

The following listing shows the specification of \specref{upperbound} which
is quite similar to the specification of \specref{lowerbound}.
The precondition \inl{increasing} expresses
that the array values need to be in increasing order.

The postconditions reflect the conditions listed above and can be expressed
using predicates \logicref{UpperBound} and \logicref{StrictLowerBound}, namely,

\begin{itemize}
\item Condition~\eqref{eq:upper-bound-result} becomes postcondition \inl{result}
\item Condition~\eqref{eq:upper-bound-left} becomes postcondition \inl{left}
\item Condition~\eqref{eq:upper-bound-right} becomes postcondition \inl{right}
\end{itemize}

\input{Listings/upper_bound.h.tex}

\subsection{Implementation of \upperbound}

Our implementation of \implref{upperbound} is shown in the following listing.
The loop invariants express that for each iteration step all indices less than
the temporary left bound \inl{left} contain values not greater than \inl{v}
and all indices not less than the temporary right bound \inl{right} contain
values greater than \inl{v}.

\input{Listings/upper_bound.c.tex}

\clearpage


\subsection{Vector Autoregression (VAR)}

We consider a vector of observables, not just one

Autoregressive (AR) model for a vector.

VAR(p) looks \(p\) back.

The AR(\(p\)) model is:

\(y_t=\alpha + \sum_{i=1}^p\beta y_{t-i}+\epsilon_t\)

VAR(\(p\)) generalises this to where \(y_t\) is a vector. We define VAR(\(p\)) as:

\(y_t\)

\(y_t=c + \sum_{i=1}^pA_i y_{t-i}+\epsilon_t\)


\documentclass[11pt]{article}
\usepackage{amsmath,textcomp,amssymb,geometry,graphicx,enumerate}
\usepackage{hyperref}
\title{An Analysis of the Balance Problem}
\date{December 27, 2014}
\author{Ted Xiao\\ UC Berkeley EECS, 2017}

\begin{document}

\maketitle

This analysis was inspired by \href{https://www.youtube.com/watch?v=wVPCT1VjySA}{a problem involving Santa Claus, 1024 wrapped gifts (with one slightly heavier), and a balance.}\\

\section*{1. The Balance Problem}
\textit{The problem:} We have a balance that can compare two different weights and tell us which is heavier or if they weigh the same. If we have 1024 boxes, with one box being slightly heavier than the others, what is the fewest number of weighs we'll need to find the heavier box?\\\
\indent\textit{The video answer:} A simple binary divide and conquer algorithm computes the solution in 10 comparisons every time. \\
\section*{2. The Binary Divide and Conquer Algorithm:}
def findBox(int[] weights):\\
\indent mid = len(weights)/2\\
\indent if totalWeight(weights[0:mid]) $<$ totalWeight(weights[mid:]):\\
\indent \indent	return findBox(weights[mid:])\\
\indent	return findBox(weights[0:mid])\\\\
\newline
\indent Our recurrence relation is then $T(n) = T(n/2) + O(1)$, giving us a runtime of $log_2{n}$ comparisons. This is obviously a much better approach than a naïve approach, which makes $O(n)$ comparisons to see when two boxes weigh different.
\newline
\section*{3. The Ternary Divide and Conquer Algorithm:}
\indent However, my intuition was that instead of dividing the boxes into TWO groups every time, we could instead divide it into THREE. Then, we’d compare two of these thirds, and then if they were equal, we’d know the heavier box MUST be in the third we didn’t weigh, by simple parity. If they weren’t equal, we’d know the heavier box is in the third that was heavier. Thus, our problem size goes down by a factor of 3 each time. There was one problem that I thought about (that the original problem managed to circumvent since 1024 is a power of 2): what if the sample size can’t be divided into 3 groups? Well, I reasoned that at most you could have 2 leftover; we’d just put these in the third we aren’t putting on the balance. Then, if the two we’re weighing are equal, we’d have the other third (with the extras) be our new set. This still cuts the amount of boxes we look at asymptotically by 3 each iteration. \\\\
def heavierBoxByThirds(int[] weights):\\
\indent	third1, third2 = len(weights)/3, 2*len(weights)/3\\
\indent	if totalWeight(weights[0:third1]) $>$ totalWeight(weights[third1:third2]):\\
\indent\indent return heavierBoxByThirds(weights[0:third1])\\
\indent	elif totalWeight(weights[0:third1]) $<$ totalWeight(weights[third1:third2]):\\
\indent\indent return heavierBoxByThirds(weights[third1:third2])\\
\indent	return heavierBoxByThirds(weights[third2:])\\
\\
\\
\indent The recurrence relation is thus $T(n) = T(n/3) + O(1)$, leading to a runtime of $log_3{n}$, asymptotically faster than the binary divide and conquer algorithm. \\
\indent One potential issue that came up: What if at each splitting into thirds, we find that the first two thirds are always equal in weight (so the heavier box is in the third with ‘extras’)? This would mean that our problem size wouldn’t be decreasing from n to n/3 at each iteration, but rather $n$ to $n/3 + 2$ (at worst case). Could this ever be worse than the binary divide and conquer, which is a decrease from $n$ to $n/2$?
\indent Well, $n/3 + 2 < n/2$ for all $n > 12$. And the problems we are looking at are all over a size of 12! So we are golden. Fun note: even for sizes under 12, it turns out binary and ternary splits are the SAME number of comparisons. Try it out!\\
\indent So this ternary divide and conquer algorithm should be the most optimal (based on worst case time) way of finding the heavier box. I confirmed my intuition with a simpler version of this \href{http://en.wikipedia.org/wiki/Balance_puzzle}{problem}.
\\\\
\textbf{Bonus:} \\
\indent I thought of a way to optimize the Binary Divide and Conquer Algorithm. It turns out a similar quarternary divide and conquer algorithm would have the same worst case run-time, but a MUCH better expected runtime! The thing about our binary divide and conquer algorithm is that is MUST go through $log_2{n}$ comparisons. There’s no way to ‘skip’ steps. If we instead to a quarternary divide, we could get lucky sometimes and cut the sample size from n to n/4 in just one comparison instead of two! Even if the two quarters are even, we will still have eliminated half of the boxes, and we still get a decrease from n to n/2 in one comparison, just as we did with the binary divide and conquer. \newline
\indent So half of the time, we’ll get lucky and get a size reduction to $1/4$ and half the time we’ll just have to settle four a size reduction to $1/2$. This averages out to a size reduction to $1/2*1/4 + 1/2*1/2 = 3/8$ Which means the quarternary divide and conquer actually has an AVERAGE running time that is better than the binary divide and conquer but worse than ternary divide and conquer.
\indent Unfortunately, it seems like further splits wouldn’t be optimal. After you compare less than half of the original boxes each time $$(1/5 + 1/5 = 2/5 < 1/2)$$ you can’t really take advantage of divide and conquer anymore.
\indent To put my solution in context with the binary divide and conquer: $log_21024 = 10$ comparisons. $log_3{1024} = 6.3$ comparisons! That’s 37\% faster. \newline
\indent Anyways, it was fun thinking about the problem\! It was fun using some of my Algorithms class knowledge, but most importantly, it’s cool that my solution works.
\end{document}


\lab{Algorithms}{Line-Search Algorithms}{Line-Search Algorithms}
\objective{Investigate various Line-Search algorithms for one-dimensional optimization.}

Line search procedures are an integral part of many nonlinear optimization techniques.
In some sense, they represent the simplest nontrivial case in general optimization, as
we only have to worry about one parameter. And yet far more sophisticated optimization
algorithms really crucially on the effectiveness and efficiency of line searches, since
higher-dimensional problems are often broken down into one-dimensional optimizations.
There are many different line search methods, and their effectiveness depends very much
on the nature of the optimization problem. Although the line search procedure is often
only a subroutine of the optimization algorithm at hand, understanding the basics of
the line search is necessary for understanding the robustness of the entire algorithm.

\subsection*{Derivative versus Derivative-Free Methods}
As you have seen in calculus classes, the derivative of a function gives information
about how the value of the function changes at each point, and can be used to determine
local optima. However, not all objective functions are differentiable, so we need other
techniques at our disposal. Line search methods may be broadly separated into two groups
based on whether they use the derivative of the objective function. We discuss two
simple examples to illustrate this distinction.

\subsection*{Golden Section Search}
This method is appropriate when minimizing a real-valued function on the reals over a
closed interval. The function must further satisfy the \emph{unimodal} property, i.e.
it has just one local minimum, and in fact decreases monotonically to the minimum from
the left, and increases monotonically to the right. The goal, of course, find the
global minimum. We do this by making a sequence of ``guesses" that we hope will converge
quickly to the minimum. Although we may not end up with the exact minimum, this method
will allow us to pin down the true minimum within an interval of any given width in a
finite number of steps.

For the Golden Section Search, each step consists of evaluating the function at two
points within the current interval, comparing these values, and then reducing the size
of the interval for the next step. Let us consider typical step in the algorithm. At
the outset, we have our function $f$ and a closed interval $[a, b]$ over which we seek
to minimize $f$. Choose two points $a'$ and $b'$ within the interval, and assume that
$a' < b'$. Now calculate $f(a')$ and $f(b')$, and assume that $f(a') \geq f(b')$.
Because of the unimodal condition, we now know that the minimizer must be in the
interval $[a', b]$, for otherwise the function $f$ would have a local minimum in
both $[a, a']$ and $[a', b]$. In the next step, we repeat the process over the interval
$[a', b]$. If instead we had $f(b') \geq f(a')$, then we choose the interval $[a, b']$
for the next step, and if the two values are equal, then it does not matter which
interval is chosen.

We now have the basic description of the algorithm, but how do we choose the two test
points $a'$ and $b'$? There is in fact an optimal choice, which reduces the amount of
work we have to do. Given an interval $[a, b]$, choose $a'$ and $b'$ satisfying
\begin{equation*}
a' = a + \rho(b - a) //
b' = a + (1 - \rho)(b - a),
\end{equation*}
where $\rho = \frac{1}{2}(3 - \sqrt{5}) \approx 0.382$. By choosing these particular
points, we need to only evaluate the function at one additional point in the next step.
To demonstrate this fact, the reader may verify that, within the interval $[a, b']$,
the point $a'$ already satisfies the equation
\begin{equation*}
a' = a + (1 - \rho)(b' - a),
\end{equation*}
and so we need only evaluate the function at the point $c$ satisfying
\begin{equation*}
c = a + \rho(b' - a).
\end{equation*}
(The constant $\rho$ is not difficult to derive, and is related to the famous Golden Ratio, hence the name of this algorithm.)

At each step, the interval is reduced by a factor of $1-\rho$, which means that after
$n$ steps, we have pinned down the minimizer to within an interval approximately
$(0.61803)^n$ times the length of the original interval. Note that this convergence is
independent of the objective function.

\begin{problem}
Implement Golden Section Search as described above. Use this to minimize $\exp{x} - 4x$
on the interval $\lbrack 0, 3 \rbrack$. How many steps do you need to take to get
within $.001$ of the true minimizer? Check that with the sentence preceding this
problem.
\end{problem}

\subsection*{Newton's Method}
To use this method, we need a real-valued function of a real variable that is twice
differentiable. The idea is to approximate the function with a quadratic polynomial and
then solve the trivial problem of minimizing the polynomial. Doing so in an iterative
manner can lead us to the actual minimizer. Let $f$ be a function satisfying the
appropriate conditions, and let us make an initial guess, $x_0$. The relevant quadratic
approximation to $f$ is
\begin{equation*}
q(x) = f(x_0) + f'(x_0)(x-x_0) + \frac{1}{2}f''(x_0)(x-x_0)^2,
\end{equation*}
or just the second-degree Taylor polynomial for $f$ centered at $x_0$. The minimum
for this quadratic function is easily found by solving $q'(x) = 0$, and we take the
obtained $x$-value as our new approximation. The formula for the $(n+1)$-th
approximation, which the reader can verify, is
\begin{equation*}
x_{n+1} = x_n - \frac{f'(x_n)}{f''(x_n)}.
\end{equation*}

As is typical with optimization algorithms, Newton's Method generates a sequence of
points or successive approximations to the minimizer. However, the convergence
properties of this sequence depend heavily on the initial guess $x_0$ and the function
$f$. Roughly speaking, if $x_0$ is sufficiently close to the actual minimizer, and if
$f$ is well-approximated by parabolas, then one can expect the sequence to converge
quickly. However, there are cases when the sequence converges slowly or not at all.

\begin{problem}
Implement Newton's Method as described above. Use this to again minimize $\exp{x} - 4x$
with an initial guess of $x_0 = 0$. Run the algorithm for as many steps as used for the
Golden Section Search. Which method gives the better result?
\end{problem}

\subsection*{General Line Search Methods}
We now discuss some important conditions for line search methods in a more general
setting. We will deal here only with methods that use the derivative.

Line-search methods require choosing a direction and a step size at each step in the
algorithm. The basic iteration is simply
\begin{equation*}
x_{k+1} = x_k + \alpha_kp_k,
\end{equation*}
where $p_k$
is the search direction, and $alpha_k$ is the step size. We will not discuss how the
search direction is selected, but only the step size. A common approach involves the
\emph{Wolfe conditions}:

\begin{align*}
&f(x_k + \alpha_kp_k) \leq f(x_k) + c_1\alpha_k\nabla f_k^Tp_k, &(0 < c_1 < 1)
\\ &\nabla f(x_k + \alpha_kp_k)^Tp_k \geq c_2\nabla f_k^Tp_k, &(c_1 < c_2 < 1)
\end{align*}

Here, $f$ is the objective function, and we use the shorthand notation $\nabla f_k$ to
mean the gradient of $f$ evaluated at the point $x_k$. The search direction $p_k$ is
often required to satisfy $p_k^T \nabla f_k < 0$, in which case it is called a
\emph{descent direction}, since in this case the function is guaranteed to decrease in
this direction. Generally speaking, choosing a step size satisfying these conditions
ensures that we achieve sufficient decrease in the function and also that we do not
terminate the search at a point of steep decrease (since then we could achieve even
better results by choosing a slightly larger step size). The first condition is known
as the \emph{Armijo} condition. The second of the two conditions can be replaced by
\begin{equation*}
| \nabla f(x_k + \alpha_kp_k)^Tp_k| \leq c_2 | \nabla f_k^Tp_k|,
\end{equation*}
and in this case we have the \emph{strong Wolfe conditions}.

The \emph{Goldstein conditions} for choosing step size:
\begin{equation*}
f(x_k) + (1-c)\alpha_k\nabla f_k^Tp_k \leq f(x_k + \alpha_kp_k) \leq f(x_k) +
c\alpha_k\nabla f_k^Tp_k,
\end{equation*}
where $0 < c < 0.5$. Similar to the Wolfe conditions, the Goldstein conditions ensure
sufficient decrease in the function and prevent the step size from being too small. In
the area of quasi-Newton optimization methods, however, the Wolfe conditions are
preferred.

\emph{Sufficient Decrease and Backtracking:}

Another way to require a sufficient decrease without taking too short of a step
involves the idea of backtracking: start with an assumed step size, and if it does not
satisfy a sufficient decrease requirement (which is to say, the step is too long),
scale the step down repeatedly until the Armijo condition is satisfied.

\begin{problem}
Implement the backtracking algorithm by writing a function that accepts as input an
objective function, the gradient of the function, the point at which to evaluate the
function, and a search direction. Note that you also need to choose the scaling factor
and the constant $c_1$ in the Armijo condition, as well as an initial step length.
\end{problem}


\documentclass[11pt,letterpaper]{article}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}

\usepackage[charter]{mathdesign}
\usepackage{fullpage}
\pagestyle{empty}

\input{../fncextra}

\addtolength{\textheight}{2em}

%
% This project can be used after linear system solution (Chapter 2) has been covered.
%

\begin{document}

\begin{center}
  \bf
  Project: In the octopus' garden
\end{center}

We've done a little one-dimensional interpolation, and we will do more later in the course. We won't be going beyond one dimension, because things can get a little tricky out there. But there is a method known as \textbf{radial basis functions} (RBF) that is not hard to get started with. RBFs have applications in machine learning, computer graphics, and differential equations.

Let $\phi(r)$ be a \emph{shape function} defined for $r\ge 0$. Popular choices include $\phi(r)=r^3$ and $\phi(r)=e^{-r^2}$. We'll limit ourselves to two dimensions, so let $(x_i,y_i)$, for $i=1,\ldots,n$, be points in a plane region $R$. Each RBF has the form
\begin{equation}
  \label{eq:rbf}
  u_i(x,y) = \phi\bigl( \sqrt{(x-x_i)^2+(y-y_i)^2} \bigr).
\end{equation}
The level curves of $u_i$ are circles centered at $(x_i,y_i)$, which is the reason for the adjective ``radial.''

Given a function $f(x,y)$ on the domain $R$, its RBF interpolant on the point set $\{(x_i,y_i)\}$ is
\begin{equation}
  \label{eq:interp}
  s(x,y) = \sum_{i=1}^n c_i u_i(x,y),
\end{equation}
where the $c_i$ values are chosen so that
\begin{equation}
  \label{eq:constraint}
  s(x_i,y_i) = f(x_i,y_i), \qquad i=1,\ldots,n.
\end{equation}
These conditions take the form of a linear system:
\begin{equation}
  \label{eq:linsys}
  \mA \bfc = \bfz, \qquad A_{ij}=u_j(x_i,y_i), \quad z_i=f(x_i,y_i).
\end{equation}
Once $\bfc$ has been obtained, the interpolant can be computed anywhere using~\eqref{eq:interp}.

One of the earliest applications of RBFs was to draw elevation contour lines on maps. Imagine you are given height data $z_i$ at each location instead of a function $f(x,y)$. You can replace the constraint~\eqref{eq:constraint} with $s(x_i,y_i)=z_i$ for all $i$, and then $s$ is defined throughout the plane as an approximation to the true elevation.

\subsection*{Project assignment}
\label{sec:project-assignment}

You should submit only M-files. Four of them are scripts required to produce output as described in detail below.
 \textbf{Each of these scripts should start with the lines}
\begin{verbatim}
    close all, clear all, rng(foo)
\end{verbatim}
where \texttt{foo} is a 4-digit integer made up by your team. (The call to \texttt{rng} makes your ``random'' results reproducible.)
You may also use any function built into core MATLAB (not any toolboxes), any of the book functions, and any of your own helper functions. In your submission, \textbf{include all other files needed, including those from the book.} You can locate all dependencies for file \texttt{p1.m}, for example, with
\begin{verbatim}
    matlab.codetools.requiredFilesAndProducts('p1')
\end{verbatim}
\textbf{If a script does not run successfully, you may receive no credit on that part of the assignment. If any code is found to be plagiarized from the internet or another group, you may receive a zero on the entire assignment.}

\begin{description}
\item[script \texttt{p1.m}:] This part is in one dimension, with $x_i=(i-1)/10$ and $y_i=0$ for $i=1,\ldots,11$, $\phi(r)=r^3$, and $f(x,y)=1/(20x^2+1)$. On one graph, plot the basis functions $u_2(x,0)$ and $u_7(x,0)$ for $0\le x \le 1$. Create and solve the linear system~\eqref{eq:linsys}. Evaluate and print out the values of $s(x,0)$ and $f(x,0)-s(x,0)$ at $x=0.25$, $x=0.5$, and $x=0.75$. On a new graph, plot $f(x,0)$ and $s(x,0)$ together.
\item[script \texttt{p2.m}:] Now you go into two dimensions, for the function
  \begin{equation}
    \label{eq:func}
    f(x,y) = e^{ -x^2 + 2xy + 2y - y^2 }, \quad 0\le x \le 1, \: 0\le y \le 1,
  \end{equation}
 and with random points selected by
\begin{verbatim}
  x = rand(n,1);  y = rand(n,1);
\end{verbatim}
for the values $n=100,200,\ldots,1000$. Make a table showing the values of $n$, the error $f-s$ at the points $(0.5,0.5)$, $(0,0.6)$, and $(1,1)$, and $\kappa(\mA)$.
\item[script \texttt{p3.m}:] Same as for \texttt{p2}, except using $\phi(r)=e^{-100r^2}$.
\item[script \texttt{p4.m}:] MATLAB ships with a data set of elevation readings of an undersea mountain. You can load and see the data using
\begin{verbatim}
load seamount
plot3(x,y,z,'o')
\end{verbatim}
(You do not need to include this plot in your submitted script.) Using the method described in the introduction, find an interpolating $s(x,y)$ using whatever you like for $\phi(r)$. Then make a contour plot of the elevation as follows:
\begin{verbatim}
xc = linspace(min(x),max(x),100);
yc = linspace(min(y),max(y),100);
[X,Y] = meshgrid(xc,yc);
% ... you write some code here to define Z ...
contourf(X,Y,Z,32), axis equal
\end{verbatim}
You need to compute \texttt{Z} such that its $(i,j)$ element is the value of $s$ at (\texttt{X(i,j)},\texttt{Y(i,j)}).

\end{description}

\end{document}


\subsection{Fourier-Collocation}
	\label{Collocation}

	The main idea of this method that we will see next is very similar to the Fourier-Galerkin method, except that we will use the interpolation operator described in (\ref{Interpolation_operator_odd}) for an odd number of points in the grid. For this, we must use another polynomial space, which we already defined in chapter \ref{Chapter_2} as $\widetilde{B}_N$ given by
	\begin{align*}
		\widetilde{B}_N = span\left\{\left(cos(nx), \hspace{0.2cm} 0 \leq n \leq \frac{N}{2} \right)\cup  \left(sin(nx), \hspace{0.2cm} 1 \leq n \leq \frac{N}{2} - 1 \right)\right\}.
	\end{align*}

	Now we will look for a solution to the problem (\ref{Bugers_Lineal}) in the space given by $S_N = \widetilde{B}_N \cap H^2_p [0, 2 \ pi]$, using the discrete expansion for the function $\varphi$ as follows
	\begin{align}
	\label{Discrete_phi}
		\mathcal{J}_N \varphi (x, t) =  \displaystyle \sum_{|n| \leq \frac{N}{2}} \widetilde{\varphi}_n (t) e^{inx}, \hspace{2mm}
			\widetilde{\varphi}_n (t) =  \displaystyle \sum_{j=0}^{2N} \varphi (x_j, t)  e^{-in x_j}
	\end{align}
	where $x_j$ are given by
	\begin{align*}
		xj = \frac{2 \pi j}{2N + 1}, \hspace{2mm} j = 0, 1, \dots, 2N.
	\end{align*}

	Remember that by (\ref{Lagrange_Odd}) the previous expansion can be written equivalently and also conveniently as
	\begin{align*}
		\mathcal{J}_N \varphi (x, t) =  \displaystyle \sum_{j=0}^{2N} \varphi (x_j, t) \psi_j (x)
	\end{align*}
	where $\psi_j (x_i) = \delta_{ij}$ for $i, j = 0, 1, \dots, 2N$, and satisfies $\mathcal{J}_N \varphi (x_j, t) = \varphi (x_j, t)$ for each $j$.

	Using the previous expansion in the equation (\ref{Bugers_Lineal}) we can obtain a residual function as we obtained it in the Fourier-Galerkin method given as follows
	\begin{align*}
		R_N (x, t) = \frac{\partial \mathcal{J}_N \varphi (x, t)}{\partial t} - \alpha \frac{\partial}{\partial x^2} \mathcal{J}_N \varphi (x, t),
	\end{align*}
	and similarly, we must force the orthogonality, for which we must remember by (\ref{Coincidence_Inner}) that in this case, the discrete product coincides with the continuum, therefore, we have to
	\begin{align*}
		\langle R_N, \psi_j \rangle_N = \int_{I} R_N (x, t) \overline{\psi}_j (x) dx = 0, \hspace{2mm} \text{for} \hspace{2mm} j = 0, 1, \dots, 2N.
	\end{align*}

	Since $\mathcal{J}_N \varphi (x_j, t) = \varphi (x_j, t)$ for every $j= 0, 1 \dots, 2N$, the orthogonality can be satisfied by solving the following problem
	\begin{align}
	\label{Collocation_Linear}
		\frac{d \mathcal{J}_N \varphi (x_j, t)}{dt} = \alpha \mathcal{J}_N \frac{\partial}{\partial x^2} \mathcal{J}_N \varphi (x_j, t), \hspace{2mm} j = 0, 1 \dots, 2N
	\end{align}
	which is a system of $2N + 1$ ordinary differential equations that can be solved using the initial condition given by
	\begin{align*}
		\mathcal{J}_N \varphi (x_j, t) =  \varphi_0 (x_j), , \hspace{2mm} j = 0, 1 \dots, 2N
	\end{align*}

	A convenient way to solve the above problem is to solve for each fixed $ j $ the following system of differential equations for the coefficients $ \widetilde{\varphi}_n$ given by
	\begin{align*}
		\frac{d \widetilde{\varphi}_n (t)}{dt} =  \alpha n^2 \widetilde{\varphi}_n (t), \hspace{2mm} |n| \leq 2N
	\end{align*}
	which is exactly the same that we obtained in the Fourier-Galerkin method, with the solution given by
	\begin{align*}
		\widetilde{\varphi}_n (t) = \widetilde{\varphi}_n (0) e^{- \alpha n^2 t}, \hspace{2mm} |n| \leq 2N
	\end{align*}

	Finally, after solving the previous problem for each $j$ we can express the solution with the expansion given by (\ref{Discrete_phi}), which is basically the same solution that we have found using the Fourier-Galerkin method. \\

	When we need to use some numerical method to solve in the variable $ t $, it is better to express the system of differential equations by configuring the following vector
	\begin{align*}
		\varphi_N (t) = \left[ \varphi(x_0, t), \varphi(x_1, t), \dots, \varphi(x_{2N}, t) \right]^T
	\end{align*}
	and using the differentiation matrix given by (\ref{matrix_DN_odd}) to obtain
	\begin{align*}
		\frac{d}{dt} \varphi_N (t) = \alpha D^{(2)}_{2N} \varphi_N (t)
	\end{align*}
	In this way we can calculate derivatives directly in real space, being a great advantage that we will discuss in the next section when implementing numerical methods based on the previous representation.


\section{Determinant and Trace}

\subsection{Exercise 1}
(a) The discriminant already has ordered versions of all the $(i, j)$ difference terms.
Applying a permutation only changes the signs of some of the difference terms, hence
$\sigma(p) = 1, -1$.

(b) $\sigma(p_1 \circ p_2) = \text{sign}(P(p_1 \circ p_2 (x_1, ..., x_n)))  = \sigma(p_1) \text{sign}(P(p_2 (x_1, ..., x_n)))$.

\subsection{Exercise 2}
(c) A transposition swaps two indices, and hence flips the sign of their associated
difference term in the discriminant.

(d) If $p(i) = j$, then we can start with the permutation $(i \: j)$.
Next, if $p(j) = k$, we can compose with  $(i \: k)$ to get $(i \: k) \circ (i \: j)$.
We can do this until we have completely reconstructed the permutation using
transpositions.

\subsection{Exercise 3}
By starting with a different $i$ in Exercise 2 (d), we can obtain a different
decomposition of transpositions. However, the parity of the decomposition must
be the same, as otherwise $\sigma(p)$ will take on two different values for
the same $p$.

\subsection{Exercise 4}
(Property II): Each term in $D(a_1, ..., a_n)$ contains exactly one element
from each of the $a_i$. Thus, scaling any of the $a_i$ by $k$ scales the entire
determinant by $k$. Similar logic for vector addition.

(Property III): The only non-zero term in $D(e_1, ..., e_n)$ is associated with
the identity permutation, hence $D(e_1, ..., e_n) = 1$.

(Property IV): Swapping two arguments is the same as applying a transposition to
each of the terms in $D(a_1, ..., a_n)$, which flips the sign of $D$.

\subsection{Exercise 5}
Suppose $a_1 = a_2$. Then:
\begin{align*}
        D(a_1, a_2, ..., a_n) &= -D(a_2, a_1, ..., a_n) \\
        D(a_1, a_2, ..., a_n) + D(a_1, a_2, ..., a_n) &= 0
\end{align*}

\subsection{Exercise 6}
We can swap rows and columns until $A$ is in the same form as in Lemma 2.
Since each row and column swap is equivalent to applying a transposition,
we get that $\det A = (-1)^{i + j} \det A_{ij}$.

\subsection{Exercise 7}
Each term in the sum $D(a_1, ..., a_n) = \sum \sigma(p) a_{p_1 1} ... a_{p_n n}$ consists of
exactly one element from each column and each row; swapping rows and columns does not
change the terms in the sum. However, the permutation associated with each term is changed.
The permutation $p$ that sends $1 \to p_1$ becomes $p'$ sending $p_1 \to 1$. This $p'$
is exactly $p^{-1} $. Since $\sigma(1) = \sigma(p^{-1}  \circ p) = \sigma(p^{-1}) \sigma(p)$,
$\sigma(p^{-1}) = \sigma(p)$ we are done.

\subsection{Exercise 8}
$P$ is the linear transformation such that $P(e_j) = e_i$; in other words,
$P$ rearranges the representation of $x$ by applying $p$ to the components of $x$.
We also have that $PQx = P q(x) = p \circ q(x)$, since $Qx$ permutes the components
of $x$ to produce $q(x)$, and $P q(x)$ permutes the components of $q(x)$ to produce
$p \circ q(x)$.

\subsection{Exercise 9}
\begin{align*}
        &\Tr AB = \sum_{i = 1}^m (AB)_{ii} = \sum_{i=1}^m \sum_{j=1}^n a_{ij}b_{ji} \\
        &\Tr BA = \sum_{j=1}^n (BA)_{jj} = \sum_{j=1}^n \sum_{i=1}^m b_{ji} a_{ij}
\end{align*}

\subsection{Exercise 10}
\begin{align*}
        \Tr A A^{\top} &= \sum (A A^{\top})_{ii} = \sum \sum a_{ij} a^{\top}_{ji} \\
                       &= \sum \sum a_{ij}^2
\end{align*}


In project 1 we used linear regression to predict a continuous output from a set of inputs \cite{project1-eirik-joakim,project1-vege}. We used ordinary least squares regression (OLS), Ridge regression and Lasso regression, where the two latter impose a penalty to the OLS. In this project we will reuse the ideas and code of project 1, but we will also use neural networks to predict continuous variables. In addition we study situations where the outcome is discrete rather than continuous. This is a classification problem, and we will use logistic regression to model the probabilities of the classes.

\subsection{Logistic regression}
Just like a linear regression model, a logistic regression model computes a weighted sum of the predictor variables, written in matrix notation as $\bm{X}^T\beta$. However, the logistic regression returns the logistic of this weighted sum as the probabilities. For a classification problem with $K$ classes, the model has the following form \citep[p.119]{james2013introduction},
\begin{equation}\label{eqT:logreg_def}
\begin{split}
\log\frac{Pr(G=1|X=x)}{Pr(G=K|X=x)} &= \beta_{10}\ +\ \beta_{1}^Tx\\
\log\frac{Pr(G=2|X=x)}{Pr(G=K|X=x)} &= \beta_{20}\ +\ \beta_{2}^Tx\\
&\vdots\\
\log\frac{Pr(G=K-1|X=x)}{Pr(G=K|X=x)} &= \beta_{(K-1)0}\ +\ \beta_{K-1}^Tx\\
\end{split}
\end{equation}
It is arbitrary which class is used in the denominator for the log-odds above. Taking the exponential on both sides and solving for $Pr(G=k|X=x)$ gives the following probabilities,
\begin{equation}\label{eqT:logreg_prob}
\begin{split}
Pr(G=k|X=x) &= \frac{exp(\beta_{k0}\ +\ \beta_{k}^Tx)}{1+\sum_{l=1}^{K-1}exp(\beta_{l0}\ +\ \beta_{l}^Tx)},\ k=1,\dots,K-1,\\
Pr(G=K|X=x) &= \frac{1}{1+\sum_{l=1}^{K-1}exp(\beta_{l0}\ +\ \beta_{l}^Tx)},
\end{split}
\end{equation}
and the probabilities sum to one. To classify the output we choose the class with the highest probability.

\subsubsection{Fitting logistic regression model}
The usual way of fitting logistic regression models is by maximum likelihood. The log-likelihood for N observations is defined as,
\begin{equation}\label{eqT:likelihood_def}
l(\theta)\ =\ \sum_{i=1}^{N}\log p_{g_i}(x_i;\theta),\\
\end{equation}
where $p_k(x_i;\theta)\ =\ Pr(G=k|X=x_i;\theta)$ and $\theta\ =\ \{\beta_{10}, \beta_1^T,\dots, \beta_{(K-1)0}, \beta_{K-1}^T\}$.

One very common classification problem is a situation with binary outcomes, either it happens or it does not. As we see from Equation \ref{eqT:logreg_def} above, setting K=2 simplifies the model considerable, since there will now be only a single linear function. $\theta$ in Equation \ref{eqT:likelihood_def} will also be simplified: $\theta = \beta = \{\beta_{10}, \beta_1^T\}$. The two-class case is what is used in this project, and the following discussion will assume the outcome has two classes.

We start by coding the two-class $g_i$ with a 0/1 response $y_i$, where $y_i$ = 1 when $g_i$ = 1, and $y_i$ = 0 when $g_i$ = 2. Next, we let $p_1(x;\theta)\ =\ p(x;\beta)$, and $p_2(x;\theta)\ =\ 1\ -\ p(x;\beta)$. The log-likelihood can then be written
\begin{equation}\label{eqT:loglike_binary}
\begin{split}
l(\beta) &= \sum_{i=1}^N\{y_i\log p(x_i;\beta)+(1-y_i)\log(1-p(x_i;\beta))\}\\
 &= \sum_{i=1}^N\{y_i\log\frac{p(x_i;\beta)}{1-p(x_i;\beta)}+\log(1-p(x_i;\beta))\}\\
 &= \sum_{i=1}^N\{y_i\beta^Tx_i + \log(1-\frac{1}{1+exp(-\beta^Tx_i)}\}\\
 &= \sum_{i=1}^N\{y_i\beta^Tx_i + \log(\frac{exp(1}{1+exp(\beta^Tx_i)}\}\\
 &= \sum_{i=1}^N\{y_i\beta^Tx_i - \log(1+exp(\beta^Tx_i))\}.
\end{split}
\end{equation}
This is the equation we want to maximize to find the best fit. Following the approach in Géron's book \citep{Geron}, we chose the equivalent approach of minimizing the following,
\begin{equation}\label{eq:Geron_cost}
J(\beta) = -\frac{1}{N}\sum_{i=1}^N\{y_i\beta^Tx_i - \log(1+exp(\beta^Tx_i)).\}
\end{equation}
This is just the negative of Equation \ref{eqT:loglike_binary}, divided by the number of samples. This is our cost function, and dividing by the number of training samples finds the mean cost.

As with all cost functions, the goal is to minimize it. This introduces it's own potential pitfalls such as overfitting. In order to avoid that, we can introduce regularizations. Let us begin by looking at the optimization problem.


%!TEX root =  ../main.tex

\subsection{Numerically}

\objective{Calculate and graph reciprocal functions.}

Calculating the reciprocal of a function is very straightforward.  For any given $x$,
take the output $f(x)$, and then reciprocate that number.  What that leads to is a
complicated picture.  However, a pattern emerges.  The reciprocal of 1 is 1, and the
reciprocal of -1 is -1.  That means points on the graph with a $y$ of 1 will not
be effected by reciprocation.  Other integer outputs will reciprocate to smaller and
smaller fractions, e.g. 2 to $\frac{1}{2}$, 3 to $\frac{1}{3}$, etc.

What is unusual is the behavior around 0.  Zero has no reciprocal, so there will
typically be a vertical asymptote.

Especially in trigonometry, reciprocals can have names.

\begin{figure}
\begin{centering}
\begin{tikzpicture}[scale=0.5]
	\draw [help lines] (-10, -10) grid (10, 10);
	\draw [thick,<->] (-10.2, 0) -- (10.2, 0);
	\draw [thick,<->] (0, -10.2) -- (0, 10.2);
        % x-axis label
        \node at (10.5, 0) {x};
        % y-axis label
        \node at (0, 10.5) {y};
        \draw[domain=-10:8.21,<->,ultra thick,samples=100,smooth] plot (\x,{1/60*(\x-4)*(\x+1)*(\x+8)});
        \draw[domain=-10.1:-8.065,<->,ultra thick,samples=100,smooth,dashed] plot (\x,{60/((\x-4)*(\x+1)*(\x+8))});
        \draw[domain=-7.93:-1.162,<->,ultra thick,samples=100,smooth,dashed] plot (\x,{60/((\x-4)*(\x+1)*(\x+8))});
        \draw[domain=-.83:3.9,<->,ultra thick,samples=100,smooth,dashed] plot (\x,{60/((\x-4)*(\x+1)*(\x+8))});
        \draw[domain=4.095:10.1,<->,ultra thick,samples=100,smooth,dashed] plot (\x,{60/((\x-4)*(\x+1)*(\x+8))});
        \draw[dashed, -] (-1,10) -- (-1,-10);
        \draw[dashed, -] (-8,10) -- (-8,-10);
        \draw[dashed, -] (4,10) -- (4,-10);
\end{tikzpicture}
\caption[Example of graphical reciprocation]{An example of a function and its reciprocal, $f(x)=\frac{1}{60}(x-4)(x+1)(x+8)$.}
\end{centering}
\end{figure}


\documentclass{amsart}

\usepackage{amsmath}
\usepackage{amssymb}

\title{PDE: Assignment 2}
\author{Mark Ditsworth}

\begin{document}
	\maketitle

	\section{Problem 1}
	Consider the inner product defined as $<x,y> = x^*By$, where vectors $x,y \in \mathbb{C}^n$, and $B$ is positive definite Hermitian $n\times n$ matrix.

	\subsection{Part 1}
	Show that $<x,y> = \overline{<y,x>}$.
	\\\\
	$\overline{<y,x>} = \overline{y^*Bx} = \overline{Bx}y = x^*B^*y$.
	\\\\
	Since $B$ is Hermitian, $B^* = B$, so $\overline{<y,x>} = x^*By = {<x,y>}$.
	\\\\
	\subsection{Part 2}
	If $M$ is an arbitrary $n\times n$ matrix, define the adjoint $M^\dag$ by $<x,My> = <M^\dag x,y>$. Give an explicit formula for $M^\dag$ in terms of $M$ and $B$.
	\\\\
	\[ <x,My> = <M^\dag x,y>
	\]
	\[ x^*BMy = (M^\dag x)^*By
	\]
	\[ x^*BM = (M^\dagger x)^*B
	\]
	\[ x^*BMB^{-1} = (M^\dagger x)^* = x^*M^{\dagger *}
	\]
	\[ BMB^{-1} = M^{\dagger *}
	\]
	\[ M^\dagger = (BMB^{-1})^*
	\]
	\\\\
	\subsection{Part 3}
	Show that if for some $A=A^*$, and $M=B^{-1}A$, then $M=M^\dagger$.
	\\\\
	\[ M^\dagger = (BMB^{-1})^*
	\]
	\[ M^\dagger = (BB^{-1}AB^{-1})^*
	\]
	\[ M^\dagger = (AB^{-1})^*
	\]
	\[ M^\dagger = B^{-1*}A^*
	\]
	Since $B$ is Hermitian and $A=A^*$,
	\[ M^\dagger = B^{-1}A = M
	\]
	\\\\
	\section{Problem 2}
	Consider the following system:
	\[ \hat{A}u = \frac{d}{dx}\left[c\frac{du}{dx}\right]
	\]
	\subsection{Part 1}
	Using center difference operations, approximate $\hat{A}u$ at $m\Delta x$.
	\\\\
	Let $u_m$ be the value of $u$ at point $m$, and $c_m$ be the value of $c$ at point $m$. $u'$ can then be approximated at point $m+0.5$ with center difference by\\
	\[u'_{m} \approx \frac{u_{m+0.5} - u_{m-0.5}}{\Delta x}
	\]\\
	where $\Delta x$ is the difference between $m_n$ and $m_{n-1}$ The derivative of $c\frac{du}{dx}$ can then be approximated by
	\\
	\[\frac{d}{dx}\left[c_m\frac{du}{dx}\right] \approx
	  \frac{c_{m+0.5}u'_{m+0.5} - c_{m-0.5}u'_{m-0.5}}{\Delta x}
	\]\\
	and by substituting $u'(m)$ in\\
	\[ \frac{d}{dx}\left[c_m\frac{du}{dx}\right] \approx
	   \frac{c_{m+0.5}\left( \frac{u_{m+1}- u_m}{\Delta x}\right) - c_{m-0.5}\left( \frac{u_m - u_{m-1}}{\Delta x}\right)}{\Delta x}
	\]
	\\\\
	\subsection{Part 2}
	Show that the finite-difference estimation from Part 1 corresponds to approximating $\hat{A}u$ by $A\mathbf{u}$, where $\mathbf{u}$ is the column vector of $M$ points $u_m$ and $A$ is the real-symmetric matrix $A = -D^TCD$.
	\\\\
	\[ D = \frac{1}{\Delta x}
	\begin{bmatrix}
		1 		& 0		&\dots	& 0\\
		-1 		& 1 	&\ddots	& 0\\
		0 		& -1 	&\ddots & 0\\
		\vdots 	&\ddots	&\ddots	& 1\\
		0		& 0		& 0		&-1\\
	\end{bmatrix}
	\quad , \quad C =
	\begin{bmatrix}
		c_{0.5}	& 0		& \dots	& 0 \\
		0		&c_{1.5}&\ddots	& 0 \\
		\vdots	&\ddots & \ddots& 0		\\
		0		& 0		& 0		&c_{M+0.5}\\
	\end{bmatrix}
	\]
	\\
	\[
	A\mathbf{u} = -D^TCD = \frac{1}{\Delta x^2}
	\begin{bmatrix}
		-c_{0.5} - c_{1.5} & c_{1.5} & 0 & \dots\\
		c_{1.5} & -c_{1.5} - c_{2.5} & c_{2.5} & 0\\
		0 & c_{2.5} &\ddots &\ddots \\
		\vdots & 0 & \ddots &\ddots \\
	\end{bmatrix}
	\begin{bmatrix}
		u_0\\ u_1\\ \vdots \\u_{M+1}
	\end{bmatrix}
	\]
	\\
	\[ =
	\begin{bmatrix}
		\frac{c_{1.5}(u_{2} - u_1) - c_{0.5}(u_1 - u_0)}{\Delta x^2}\\
		\frac{c_{2.5}(u_{3} - u_2) - c_{1.5}(u_2 - u_{1})}{\Delta x^2}\\
		\vdots\\
		\frac{c_{m+0.5}(u_{m+1} - u_m) - c_{m-0.5}(u_m - u_{m-1})}{\Delta x^2}
	\end{bmatrix}
	\]
	\\\\
	\subsection{Part 3}
	Let $c(x) = e^{3x}$. Use the above methods to attain the eigenvalues and eigenvectors. Plot the eigenvectors for the four smallest-magnitude eigenvalues. Verify that the first two eigenfunctions are orthogonal. Verify that you are getting second order convergence of the eigenvalues. ($L=1$, $M=100$)
	\\\\
	\textit{see notebook}
	\pagebreak
	\section{Problem 3}
	Consider a metal bar with length $L$, cross sectional area $a$, and varying temperature $T$ along the rod. The rod is conceptually divided into $N$ pieces of length $\Delta x = L/N$. Each piece has a uniform temperature $T_n$, giving a vector $\mathbf{T}$ of $N$ temperatures. The rate at which heat flows from piece $n$ to piece $n+1$ is given by $q = \frac{\kappa a}{\Delta x}(T_n - T_{n+1})$ ($\kappa$ is the thermal conductivity of the rod). If an amount of heat $\Delta Q$ flows into a piece, the temperature changes by $\Delta T = \Delta Q / (c\rho a\Delta x)$ where $c$ is the specific heat capacity, and $\rho$ is the density of the metal. Assume the rod is ideally insulated.
	\\
	\subsection{Part 1}
	Show that $\frac{dT_n}{dt} = \alpha(T_{n+1}-T_n) + \alpha(T_{n-1}-T_n)$. (This is Newton's Law of Cooling).
	\\\\
	\[ \frac{\Delta T_n}{\Delta t} = \frac{1}{c\rho a\Delta x}\frac{\Delta Q_n}{\Delta t}
	\]\\
	$\frac{\Delta Q_n}{\Delta t}$ is the rate of heat flow into piece $n$, which is the sum of the heat flow from it's neighbors.\\
	\[ \frac{dT_n}{dt} = \frac{1}{c\rho a\Delta x}\left[
	\frac{\kappa a}{\Delta x}(T_{n+1}-T_n) + 	\frac{\kappa a}{\Delta x}(T_{n-1}-T_n) \right]
	\]\\
	\[ \frac{dT_n}{dt} = \frac{\kappa}{c\rho \Delta x^2}\left[ (T_{n+1}-T_n)+(T_{n-1}-T_n) \right]
	\]\\
	which matches Newton's Law of Cooling, with $\alpha = \frac{\kappa}{c\rho\Delta x}$. At the end points,\\
	\[ \frac{dT_1}{dt} = \frac{\kappa}{c\rho\Delta x^2}(T_2 - T_1) \quad , \quad \frac{dT_N}{dt} = \frac{\kappa}{c\rho\Delta x^2}(T_{N-1} - T_N)
	\]
	\\
	\subsection{Part 2}
	Write the equation from Part 1 in matrix form: $\frac{d\mathbf{T}}{dt} = A\mathbf{T}$, for some matrix $A$.
	\\\\
	\[\frac{d\mathbf{T}}{dt}=A\mathbf{T} = \frac{\kappa}{c\rho\Delta x^2}
	\begin{bmatrix}
		-1     & 1    & 0    &\dots & 0 \\
		 1     & -2   & 1    & 0    & \vdots \\
		 0     &\ddots&\ddots&\ddots& 0 \\
		 \vdots&  0   &   1  &  -2  & 1\\
		 0     &\dots &  0   &   1  & -1 \\
	\end{bmatrix}
	\begin{bmatrix}
		T_1\\T_2\\ \vdots \\ T_{N-1} \\ T_N
	\end{bmatrix}
	\]\\\\

	\subsection{Part 3}
	Let $T(x,t)$ be the temperature along the rod. Suppose $T_n(t) = T([n-0.5]\Delta x,t)$ (the temperature at the center of the $n^{th}$ piece). Take the limit $N \rightarrow \infty$ (with $L$ fixed), and derive the PDE $\frac{\partial T}{\partial t}= \hat{A}T$. What is $\hat{A}$ (ignore the ends)?
	\\\\
	As $N\rightarrow\infty$, $\Delta x$ can be thought of as $\partial x$, and $T_{n+1} - T_n$ can be thought of as $\partial T(n,t)$.
	\\\\
	$(T_{n+1}-T_n) + (T_{n-1}-T_n)$ can also be expressed as  $(T_{n+1}-T_n) - (T_{n}-T_{n-1})$, which as $N \rightarrow \infty$ is $\partial^2T$.
	\\\\
	Thus
	\[ \lim\limits_{N \rightarrow \infty} \left\{ \frac{dT_n}{dt} = \frac{\kappa}{c\rho \Delta x^2}\left[ (T_{n+1}-T_n)+(T_{n-1}-T_n) \right] \right\}
	\]\\
	becomes
	\[\frac{\partial T}{\partial t} = \frac{\kappa}{c\rho}\frac{\partial^2T}{\partial x^2}
	\]
	\\\\
	So, ignoring the ends,
	\[\hat{A} = \frac{\kappa}{c\rho}\frac{\partial^2}{\partial x^2}
	\]\\
	\subsection{Part 4}
	What are the boundary conditions on $T(x,t)$ at $x=0$ and $L$? Check that if you go backwards and form a center-difference approximation of $\hat{A}$ with these boundary conditions, you will recover matrix $A$.
	\\\\
	The boundary condition is the $\frac{\partial T}{\partial x} = 0$ at $x=0,L$ since there is 0 heat flow at the ends.
	\\\\
	The center-difference approximation of $\frac{\partial T}{\partial x}|_{n\Delta x} \approx T'_{n+0.5} = \frac{T_{n+1} - T_n}{\Delta x}$
	\\\\
	The center difference approximation of $T''_n = \frac{T'_{n+0.5} - T'_{n-0.5}}{\Delta x} =
	\frac{T_{n+1}-T_n - (T_n - T_{n-1})}{\Delta x^2}$
	\\\\
	This is equivalent to $\frac{T_{n+1} - 2T_n + T_{n-1}}{\Delta x^2}$, which is the same expression from Part 1 (with $\frac{\kappa}{c\rho}=0$), and will thus result in the same $A$.
	\\
	\subsection{Part 5}
	How does $\hat{A}$ change in the $N \rightarrow \infty$ limit if the conductivity is a function $\kappa(x)$ of $x$?
	\\\\
	\[ \frac{\partial T}{\partial t} = \frac{1}{c\rho}\frac{\partial \kappa \partial T}{\partial x^2}
	\]
	\\
	Thus,
	\[A = \frac{1}{c\rho} \frac{\partial}{\partial x}\kappa\frac{\partial}{\partial x}\]\\
	\subsection{Part 6}
	Suppose that instead of a thin bar (1D), you have a metal plate (2D) with a temperature $T(x,y,t)$ and constant conductivity $\kappa$. If you go through the steps above and divide it into $N\times N$ squares of size $\Delta x \times \Delta y$, what PDE do you get for $T$ in the limit $N \rightarrow \infty$.
	\\\\
	The PDE will be the sum of the partial derivatives in the $x$ and $y$ directions.\\
	\[ \frac{dT_{m,n}}{dt} = \frac{\kappa}{c\rho}
	\left[ \frac{T_{m+1,n} - 2T_{m,n} + T_{m-1,n}}{\Delta x^2} + \frac{T_{m,n+1} - 2T_{m,n} + T_{m,n-1}}{\Delta y^2} \right]
	\]
	And as $N \rightarrow \infty$,\\
	\[ \frac{\partial T}{\partial t} = \hat{A}T =  \frac{1}{c\rho}\bigtriangledown \cdot \kappa \bigtriangledown T
	\]

\end{document}


\subsection{Composition}

Can define the following (linear) transformation:

\[\boxed{T_C(\tb{x})=T_A(T_B(\tb{x}))=(T_A\circ T_B)(\tb{x})}\]

Following diagram represents the composition:

\begin{center}
    \includegraphics[scale=0.5]{comp-trans.png}
\end{center}

Would imply that $A$ is $n\times p$, $B$ is $p\times m$, and $AB$ is $n\times m$.
Can define the following:

\[\boxed{\text{The $i^{th}$ column of the matrix $AB$ is the matrix-vector product $A$($i^{th}$ column of the matrix $B$)}}\]

\subsection{Proofs}

\textbf{\textit{Claim:}} The product of 2 invertible matrices must be an invertible matrix.\newline
\textbf{\textit{Proof:}} Given that $(AB)(AB)^{-1}=I_n$:

\begin{align*}
    &(AB)(AB)^{-1}=I_n\\
    &A(B(AB)^{-1})=I_n\\
    &A^{-1}A(B(AB)^{-1})=A^{-1}I_n\\
    &B(AB)^{-1}=A^{-1}\\
    &B^{-1}B(AB)^{-1}=B^{-1}A^{-1}\\
    &(AB)^{-1}=B^{-1}A^{-1}
\end{align*}

\noindent\textbf{\textit{Claim:}} If $(AB)^{-1}$ exists, then $A$ and $B$ are both invertible.\newline
\textbf{\textit{Proof:}} Given that $(AB)(AB)^{-1}=I_n$ and $(AB)^{-1}(AB)=I_n$:

\begin{align*}
    &A(B(AB)^{-1})=I_n\\
    &((AB)^{-1}A)B=I_n\\
    &\boxed{\therefore\;\exists\;A^{-1},\:B^{-1}\in \R^n}
\end{align*}

\subsection{Properties}

\begin{itemize}
    \item Associativity: $(AB)C=A(BC)$
    \item Distribution: $A(B+C)=AB+BC$
    \item Respects scalar multiplication: $(kA)B=k(AB)=A(kB)$
\end{itemize}