\subsection{Implicit Differentiation}
Although we normally have equations of the form $y = f(x)$, some equations might be given in are are more convenient to write in different forms.
Using our understanding of the chain rule, we can still work with these forms and find $y^\prime$. There are generally three steps to finding $y^\prime$ when equations are given in these different forms.
\begin{enumerate}
	\item Get all terms involving $y$ to one side of the equation. This step is technically optional but usually makes the step 3 more convenient.
	\item Take the derivatives of both sides of the equation, using the chain rule.
	\item Rearrange to solve for $y^\prime$, making substitutions for $y$ to get the answer in terms of input parameters (e.g $x$).
\end{enumerate}
It's possible that you can get multiple solutions for $y^\prime$.
You'll need to check if

\begin{example}
	Given that $y^2 = x$, find $y^\prime$ using implicit differentiation.
\end{example}
\begin{answer}
	Step 1 is already complete by what's given.
	Taking the derivative of both sides, remembering to use the chain rule,
	\begin{equation*}
		2yy^\prime = 1.
	\end{equation*}

	Solving for $y^\prime$,
	\begin{equation*}
		y^\prime = \frac{1}{2y}.
	\end{equation*}

	Looking back at our original equation, we see $y = \pm\sqrt{x}$.
	Substituting back into our work,
	\begin{equation*}
		y^\prime = \pm\frac{1}{2\sqrt{x}}.
	\end{equation*}
\end{answer}

Sometimes, it's not possible or is not necessary for what you're working on to complete step 3, meaning you leave your answer for $y^\prime$ in terms of $y$ and input parameters.
\begin{example}
	Find the derivative for $y$ with respect to $x$ at $(0,1)$ if $y^4 = x^3 + x + y$.
\end{example}
\begin{answer}
	Rearranging to complete step 1,
	\begin{equation*}
		y^4 - y = x^3 + x.
	\end{equation*}

	Taking the derivative of both sides,
	\begin{equation*}
		(4y^3 - 1)y^\prime = 3x^2 + 1.
	\end{equation*}

	Solving for $y^\prime$ and completing step 2,
	\begin{equation*}
		y^\prime = \frac{3x^2 + 1}{4y^3 - 1}.
	\end{equation*}

	Since we have both the $x$ and $y$ coordinates of where we're looking for the slope, we don't need to complete step 3.
	We can simply substitute to find our numerical answer for $y^\prime$.
	\begin{equation*}
		y^\prime_{(0,1)} = \frac{0 + 1}{4 - 1} = \frac{1}{3}.
	\end{equation*}
\end{answer}

Sometimes, we might already be given a formula for $y$, but rearranging and then doing implicit differentiation is easier.
\begin{example}
	Find the derivative with respect to $x$ of $y = \ln{x}$.
\end{example}
\begin{answer}
	We don't have any way to find the derivative of $\ln$ directly, but we do know how $\ln$ relates to exponential functions, a form we do know how to differentiate.
	Exponentiating both sides with base $e$,
	\begin{equation*}
		e^y = e^{\ln{x}} = x.
	\end{equation*}

	Implicitly differentiating,
	\begin{equation*}
		e^{y}y^\prime = 1.
	\end{equation*}

	Solving for $y^\prime$,
	\begin{equation*}
		y^\prime = \frac{1}{e^y}.
	\end{equation*}

	Substituting what we were given for $y$,
	\begin{equation*}
		y^\prime = \frac{1}{e^{\ln{x}}} = \frac{1}{x}.
	\end{equation*}
\end{answer}

\subsubsection{Logarithmic Differentiation}
Logarithmic differentiation is a certain type of differentiation where you take to natural log of both sides and then implicitly differentiate.
It's especially useful when input parameters appear in both the base and exponent.
\begin{example}
	Find the derivative with respect to $x$ of $y = x^{\ln{x}}$.
\end{example}
\begin{answer}
	Taking the natural log of both sides,
	\begin{equation*}
		\ln{y} = \ln^2{x}.
	\end{equation*}

	Implicitly differentiating,
	\begin{align*}
		\frac{1}{y}y^\prime &= \frac{2\ln{x}}{x} \\
		y^\prime &= y\frac{2\ln{x}}{x} \\
		&= x^{\ln{x}}\frac{2\ln{x}}{x}.
	\end{align*}
\end{answer}

We now have the tools to prove the power rule for all real exponents.
\begin{proof}
	Let $n$ be an real number.
	\begin{align*}
		y &= x^n \\
		\ln{y} &= n\ln{x} \\
		\frac{1}{y}y^\prime &= n\frac{1}{x} \\
		y^\prime &= \frac{ny}{x} \\
		&= \frac{nx^n}{x} \\
		&= nx^{n-1}.
	\end{align*}
\end{proof}

\subsubsection{Combining Power and Exponential Rule}
We can also find the derivative of $f(x)^{g(x)}$, which combines the power and exponential rules and is something you'll likely won't see in a standard calculus course.
\begin{align*}
	y &= f^g \\
	\ln{y} &= g\ln{f} \\
	\frac{1}{y}y^\prime &= g\frac{1}{f}f^\prime + \ln{f}g^\prime \\
	y^\prime &= f^g\left(g\frac{1}{f}f^\prime + \ln{f}g^\prime\right) \\
	&= f^{g-1}\left(gf^\prime + fg^\prime\ln{f}\right).
\end{align*}


\subsection{Definitions}
\begin{align*}
b&=\sqrt{a} : b \geq 0 \wedge b^2=a \\
\sqrt{a^2}&= \vert a \vert \\
\text{If } a&\geq \text{ then } \sqrt{a^2}=a
\end{align*}
\subsection{Distributing}
$(a\geq 0 \wedge b\geq 0)$
\begin{align*}
\sqrt{ab}&=\sqrt{a}\sqrt{b} \\
\sqrt{\frac{a}{b}}&= \frac{\sqrt{a}}{\sqrt{b}}\qquad(b\neq 0) \\
\sqrt{a}\sqrt{a}&=a\\
\sqrt{a^n}&=\left( \sqrt{a} \right) ^n \vee a^{\frac{n}{2}}
\end{align*}
\subsection{Rationalizing the Denominator}
$(a>0 \wedge b>0 \wedge c>0)$
\begin{align*}
\frac{a}{\sqrt{b}}=\frac{a}{\sqrt{b}}\cdot\frac{\sqrt{b}}{\sqrt{b}}&=\frac{a\sqrt{b}}{b}
&
\frac{4}{\sqrt{2}}=\frac{4}{\sqrt{2}}\cdot\frac{\sqrt{2}}{\sqrt{2}}&=\frac{4\sqrt{2}}{2}=2\sqrt{2} \\ \\
\frac{a}{b+\sqrt{c}}=\frac{a}{b+\sqrt{c}}\cdot \frac{b-\sqrt{c}}{b-\sqrt{c}}&=\frac{ab-a\sqrt{c}}{b^2-c}
&
\frac{6}{3+\sqrt{7}}=\frac{6}{3+\sqrt{7}}\cdot\frac{3-\sqrt{7}}{3-\sqrt{7}}&=\frac{18-6\sqrt{7}}{9-7} \\
& & = \frac{18-6\sqrt{7}}{9-7}=\frac{18-6\sqrt{7}}{2}&=9-3\sqrt{7} \\ \\
\frac{a}{b-\sqrt{c}}=\frac{a}{b-\sqrt{c}} \cdot \frac{b+\sqrt{c}}{b+\sqrt{c}} &= \frac{ab+a\sqrt{c}}{b^2-c}
&
\frac{5}{2-\sqrt{3}}=\frac{5}{2-\sqrt{3}}\cdot\frac{2+\sqrt{3}}{2+\sqrt{3}}&=\frac{10+5\sqrt{3}}{4-3} \\
& & =\frac{10+5\sqrt{3}}{1}&=10+5\sqrt{3} \\ \\
\frac{a}{\sqrt{b}+\sqrt{c}}=\frac{a}{\sqrt{b}+\sqrt{c}}\cdot \frac{\sqrt{b}-\sqrt{c}}{\sqrt{b}-\sqrt{c}}&=\frac{a\sqrt{b}-a\sqrt{c}}{b-c}
&
\frac{14}{\sqrt{13}+\sqrt{11}}=\frac{14}{\sqrt{13}+\sqrt{11}}\cdot \frac{\sqrt{13}-\sqrt{11}}{\sqrt{13}-\sqrt{11}}&=\frac{14\sqrt{13}-14\sqrt{11}}{13-11}\\
& & = \frac{14\sqrt{13}-14\sqrt{11}}{2}&=7\sqrt{13}-7\sqrt{11} \\ \\
\frac{a}{\sqrt{b}-\sqrt{c}}=\frac{a}{\sqrt{b}-\sqrt{c}}\cdot \frac{\sqrt{b}+\sqrt{c}}{\sqrt{b}+\sqrt{c}}&=\frac{a\sqrt{b}+a\sqrt{c}}{b-c}
&
\frac{12}{\sqrt{15}-\sqrt{7}}=\frac{12}{\sqrt{15}-\sqrt{7}} \cdot \frac{\sqrt{15}+\sqrt{7}}{\sqrt{15}+\sqrt{7}}&=\frac{12\sqrt{15}+12\sqrt{7}}{15-7}\\
& & = \frac{12\sqrt{15}+12\sqrt{7}}{8}&=\frac{3\sqrt{15}+2\sqrt{7}}{2}
\end{align*}
\subsection{OBS!}
\begin{align*}
\sqrt{a+b}&\neq \sqrt{a}+\sqrt{b}\\
\sqrt{a-b}&\neq \sqrt{a}-\sqrt{b}\\
\sqrt{a^2+b^2}&\neq a+b
\end{align*}

Algorithms and Data Structures - Reference/root.tex


\documentclass{article}

\usepackage{amsmath}
\usepackage{graphicx}

\graphicspath{ {./assets/} }

\setlength\paperwidth{20.999cm}\setlength\paperheight{29.699cm}\setlength\voffset{-1in}\setlength\hoffset{-1in}\setlength\topmargin{1.499cm}\setlength\headheight{12pt}\setlength\headsep{0cm}\setlength\footskip{1.131cm}\setlength\textheight{25cm}\setlength\oddsidemargin{2.499cm}\setlength\textwidth{15.999cm}

\begin{document}
\begin{center}
\hrule

\vspace{.4cm}
{\bf {\Huge Assignment 7}} \\
\vspace{.2cm}
{\bf Computer Graphics}
\vspace{.2cm}
\end{center}{\bf Edoardo Riggio } (edoardo.riggio@usi.ch) \hspace{\fill}  \today \\
\hrule
\vspace{.2cm}

\section{Exercise 1}
In order to compute the pixels that need to be coloured, we first need to transform the global coordinates into screen coordinates. To do so, we use the following formula:

\[ p_{ixs} = p_{ix} \cdot \frac{8}{\tan{\frac{3}{8}} \cdot \pi} \]
\[ p_{iys} = p_{iy} \cdot \frac{8}{\tan{\frac{3}{8}} \cdot \pi} \] \\
Where $p_{ixs}$ and $p_{iys}$ are the points in screen coordinates, $p_{ix}$ and $p_{iy}$ are the points in global coordinates, ${\frac{3}{8}\cdot \pi}$ is the camera opening divided by 2, and 8 is the width of the screen divided by 2 -- rounded up to the nearest integer. By using this function, we can finally compute the values of the points.\\ \\
In the following computations, the results are floored. Moreover, in the case that the $z$ value of the global point is not 1, then $p_{ix}$ and $p_{iy}$ are divided by $z$. Finally, I assume the point $(0,0)$ to be at the center of the screen, rather than at the top-left corner.

\[ p_{1xs} = \frac{1}{4} \cdot \frac{8}{\tan{\frac{3}{8}} \cdot \pi} = 0 \]
\[ p_{1ys} = \frac{1}{4} \cdot \frac{8}{\tan{\frac{3}{8}} \cdot \pi} = 0 \]
\[ p_{1s} = (0,0) \] \\
\[ p_{2xs} = 2 \cdot \frac{8}{\tan{\frac{3}{8}} \cdot \pi} = 6 \]
\[ p_{2ys} = 1 \cdot \frac{8}{\tan{\frac{3}{8}} \cdot \pi} = 3 \]
\[ p_{2s} = (6,3) \] \\
After converting the points, we can use the following pseudocode in order to compute the pixels of the line.

\begin{verbatim}
  x = p1.x
  y = p1.y
  dx = p2.x - p1.x
  dy = p2.y - p1.y
  f = -2 * dy + dx

  for i in range(0, dx+1):
    setPoints(x,y)
    x += 1
    if f < 0:
      y += 1
      f += 2 * dx
    f -= 2 * dy
\end{verbatim}
Which will output the following pixels:

\[ (0, 0), (1, 0), (2, 1), (3, 1), (4, 2), (5, 2), (6, 3) \]\\
In the case of the segment between $p_1$ and $p'2$, we have that $p'2$ has a negative $z$ value. Because the screen has $z=1$, this means that the segment intersects the screen and goes behind it. For this reason we need to find the intersection between the segment and the plane $z = 1$. By using the calculator, I've found out the intersection to be the point $(2,1,1)$. This means that the points to be coloured are the same as the ones in the previous part of the exercise.

\section{Exercise 2}
The pseudocode used to compute the midpoint algorithm for the circle is the following:

\begin{verbatim}
  function midpoint_circle(r):
    x = 0
    y = -r

    f = 1 - r

    while y < x:
      setPoint(x,y)
      setPoint(-x,y)
      setPoint(x,-y)
      setPoint(-x,-y)
      setPoint(y,x)
      setPoint(-y,x)
      setPoint(y,-x)
      setPoint(-y,-x)
      x += 1

      if f > 0:
        y += 1
        f += 2 * x - 2 * y + 1

      f += 2 * x + 1

      if (|y| < x):
        exit loop
\end{verbatim}
Here we use a similar approach as the one used in the midpoint algorithm for the line. Inside of the while loop -- which computes only the points of $\frac{1}{8}$ of the circle, and then uses symmetric properties of the circle in order to compute the points in the other $\frac{1}{8}$s of the circle -- we check if $f$ is greater, equal or smaller than 0. This is because we can recognize three different cases:

\begin{enumerate}
	\item f $>$ 0: In this case the midpoint is outside of the circle;
	\item f = 0: In this case the midpoint is on the circle;
	\item f $<$ 0: In this case the midpoint is inside of the circle.
\end{enumerate}
In this algorithm the decision value ($f$) is computed by using the formula of the circle:
	\[ x^2 + y^2 - r^2 \]

\end{document}


\documentclass[../main.tex]{subfiles}
\begin{document}
\subsection{Accessible Set-Systems}
\begin{defn}
An \textit{accessible set-system} is a pair $M=(E,\mathcal{S})$ where $E$ is a finite ground set and $\mathcal{S}$ is a non-empty subset of the power set of $E.$ Elements of $\mathcal{S}$ are called the \textit{feasible sets} of $M.$ Maximal feasible sets are also called \textit{bases}.\\
$\mathcal{S}$ satisfies the following \textit{accessibility axiom}:\\
(A) For any non-empty feasible set $X \in \mathcal{S}$ there exists an element $e \in X$ such that $X \setminus \{e\} \in \mathcal{S}.$
\end{defn}

\begin{rem}
Matroid $\supseteq$ Independence System $\supseteq$ Accessible Set-System.\\
All matroids are independence systems and all independence systems are accessible set-systems but the converse is not true in general.
\end{rem}

\noindent The axiom $(A)$ is needed due to the process of the greedy algorithm. We require the ability to sequentially select a single element and then union it to our constructed solution at each step so an arbitrary set system cannot guarantee this. Axiom (A) however ensures that at each step such an element does in fact exist, although the subset of possible choices for this element may not be the same at each iteration of the process, which leads to further possible issues.

\noindent Our general problem can now be restated as:
\begin{prop}
For any accessible set-system $M$ and any weight function $\omega:E \longrightarrow \mathbb{R^+},$ the optimisation problem is:\\
 Maximise $\omega(B)$ such that $B$ is a basis of $M.$
\end{prop}
\noindent We can now apply a modified greedy algorithm in order to find a solution to this generalised version of our matroid/independence system problem.

\begin{algorithm}[H]
\caption{Greedy algorithm for accessible set-systems}\label{modified_greedy}
Let $M=(E,\mathcal{S})$ be an accessible set-system and $\omega:E \longrightarrow \mathbb{R^+}$ a weight function.
\begin{algorithmic}[1]
\Procedure{GREEDY}{$E, \mathcal{S}, \omega,T$}
\State $T \gets \emptyset$, $X \gets E$
\While{there $\exists x \in X$ with $T \cup \{x\} \in \mathcal{S}$}
\State choose some $x \in X$ with $T \cup \{x\} \in \mathcal{S}$ and
\State $\omega(x) \geq \omega(y)$ $\forall y \in X$ with $T \cup \{y\} \in \mathcal{S}$
\State $T \gets T \cup \{x\}$, $X \gets X \setminus \{x\}$
\EndWhile
\EndProcedure
\end{algorithmic}
\end{algorithm}

\noindent Using our above definitions and algorithm we can now begin finding solutions to our problem for any accessible set-system. However, we are interested in learning about the characterisations of these set-systems that lead to optimal solutions through the use of our greedy algorithm as described in \textit{section 5.2}.\\
One such characterisation is the concept of a \textit{greedoid}. A greedoid is an accessible set-system satisfying (I3). Formally this means,
\begin{defn}
A \textit{greedoid} is a pair $(E,\mathcal{S})$ where $E$ is a finite ground set and $\mathcal{S}$ is a collection of the feasible subsets of $E$ satisfying the following conditions:\\
(I1): $\mathcal{S}$ is non-empty, $\emptyset \in \mathcal{S}.$\\
(A):  For any non-empty feasible set $X \in \mathcal{S}$ there exists an element $e \in X$ such that $X \setminus \{e\} \in \mathcal{S}.$\\
(I3): If $ A $ and $ B $ are two independent sets of $\mathcal{S}$ and $|A|>|B|$, then there exists $x \in A \setminus B$ such that $B \cup \{ x \}$ is in $\mathcal{S}.$
\end{defn}
Unfortunately the greedy algorithm while providing solutions does not gaurantee optimal solutions for all greedoids. To characterise the greedoids that do result in optimal solutions when the greedy algorithm is applied we must add to our existing machinery an additional axiom. This is called the \textit{strong exchange axiom}. This axiom is a strong version of (I3): the exchange axiom of a matroid.
\begin{prop}
Let $M=(E,\mathcal{S})$ be a greedoid. Then the above modified greedy algorithm  finds an optimal solution to our problem for any weight function $\omega:E \longrightarrow \mathbb{R^+}$ if and only if $M$ satisfies the following axiom:\\
(SE): For $A,B \in \mathcal{S}$ with $|A|=|B|+1$, there always exists some $e \in A \setminus B$ such that $B \cup \{e\}$ and $A \setminus \{e\}$ are contained in $\mathcal{S}.$ Which is very reminiscent of \textit{lemma 1.3}.
\end{prop}
\begin{rem}
It can be seen that this axiom holds trivially for matroids due to the hereditary condition.
\end{rem}
\begin{rem}
The amount of the possible greedoids which can yield optimal solutions can be further characterised using further abstracted properties of matroids, similar to axiom (SE), for this I refer you to Jungnickel's text\cite{jungnickel} and the papers cited above.
\end{rem}
\end{document}


% This is a template for doing homework assignments in LaTeX, cribbed from M. Frenkel (NYU) and A. Hanhart (UW-Madison)

\documentclass{article} % This command is used to set the type of document you are working on such as an article, book, or presenation

\usepackage[margin=1in]{geometry} % This package allows the editing of the page layout. I've set the margins to be 1inch.
\usepackage{datetime}

\usepackage{amsmath, amsfonts}  % The first package allows the use of a large range of mathematical formula, commands, and symbols.  The second gives some useful mathematical fonts.

\usepackage{graphicx}  % This package allows the importing of images

%Custom commands
\newcommand{\der}[2]{\frac{\mathrm{d} #1}{\mathrm{d} #2}}

%Custom symbols
\newcommand{\linediv}{\noindent\rule{6.5in}{2pt}}
\newcommand{\linedivm}{\noindent\rule{6.5in}{1pt}}

\begin{document}

\begin{center}
    \Large{
        \textbf{Assignment \#1}

        UW-Madison MATH 421
    }

    \vspace{5pt}

    \normalsize{
        GEOFF YOERGER

        \usdate
        \formatdate{2}{2}{2021}
    }

    \vspace{15pt}
\end{center}

\subsection*{Exercise One:}

a)

\linediv

\begin{verbatim}
If $f(x) = x^n$,  then
$$
f^\prime(x) = n x^{n-1}.
$$
\end{verbatim}

\linedivm

If $f(x) = x^n$,  then
$$
f^\prime(x) = n x^{n-1}.
$$

\linediv

\noindent b)

\linediv

\begin{verbatim}
If $n \neq -1$,  then
$$
\int x^{n} dx = \frac{1}{n+1} x^{n+1} + C.
$$
\end{verbatim}

\linedivm

If $n \neq -1$,  then
$$
\int x^{n} dx = \frac{1}{n+1} x^{n+1} + C.
$$

\linediv

\noindent c)

\linediv

\begin{verbatim}
The derivative of a function $f$ at $x=a$ is
$$
f'(a) = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h}.
$$
\end{verbatim}

\linedivm

The derivative of a function $f$ at $x=a$ is
$$
f'(a) = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h}.
$$

\linediv

\subsection*{Exercises Two:}

\noindent a)

\linediv

\begin{verbatim}
The number $e$ is defined by
$$
e = \lim_{n \to \inf} \left( 1 + \frac{1}{n} \right) ^ {n}.
$$
\end{verbatim}

\linedivm

The number $e$ is defined by
$$
e = \lim_{n \to \infty} \left( 1 + \frac{1}{n} \right) ^ {n}.
$$

\linediv

\noindent b)

\linediv

\begin{verbatim}
If $f$ is a continuous function, then
$$
\frac{d}{dx} \left[ \int_{a}^{x} f(t) dt \right] = f(x).
$$
\end{verbatim}

\linedivm

If $f$ is a continuous function, then
$$
\frac{d}{dx} \left[ \int_{a}^{x} f(t) dt \right] = f(x).
$$

\linediv

\subsection*{Exercise Three:}

\linediv

\begin{verbatim}
\begin{center}
    \begin{tabular}{|rl|cc|}
    \hline
    First Name & Last Name & Ice Cream Flavor & Number of Scoops \\
    \hline
    Alexa & Leal & Vanilla & $4$ \\
    Julia & Maschi & Chocolate & $2$ \\
    Johnny & Tran & Strawberry & $18$ \\
    Geoff & Yoerger & Chocolate Malt & $\infty$ \\
    \hline
    \end{tabular}
\end{center}
\end{verbatim}

\linedivm

\begin{center}
    \begin{tabular}{|rl|cc|}
    \hline
    First Name & Last Name & Ice Cream Flavor & Number of Scoops \\
    \hline
    Alexa & Leal & Vanilla & $4$ \\
    Julia & Maschi & Chocolate & $2$ \\
    Johnny & Tran & Strawberry & $18$ \\
    Geoff & Yoerger & Chocolate Malt & $\infty$ \\
    \hline
    \end{tabular}
\end{center}

\linediv

\subsection*{Exercise Four:}

\linediv

\begin{verbatim}
$$
\det
\begin{pmatrix}
    a & b \\
    c & d \\
\end{pmatrix}
= ad - bc.
$$
\end{verbatim}

\linedivm

$$
\det
\begin{pmatrix}
    a & b \\
    c & d \\
\end{pmatrix}
= ad - bc.
$$

\linediv

\subsection*{Exercise Five:}

\noindent 1)

\linediv

\begin{verbatim}
\newcommand{\Rb}{\mathbb{R}}
\newcommand{\Nb}{\mathbb{N}}
\newcommand{\Zb}{\mathbb{Z}}
\newcommand{\Qb}{\mathbb{Q}}
\newcommand{\Cb}{\mathbb{C}}

$$
\Nb \subset \Zb \subset \Qb \subset \Rb \subset \Cb
$$
\end{verbatim}

\linedivm

\newcommand{\Rb}{\mathbb{R}}
\newcommand{\Nb}{\mathbb{N}}
\newcommand{\Zb}{\mathbb{Z}}
\newcommand{\Qb}{\mathbb{Q}}
\newcommand{\Cb}{\mathbb{C}}

$$
\Nb \subset \Zb \subset \Qb \subset \Rb \subset \Cb
$$

\linediv

\noindent 2)

\linediv

\begin{verbatim}
\newcommand{\pder}[2]{\frac{\partial #1}{\partial #2}}

If $z = x^2 + xy + y^2$, then
$$
\pder{z}{x} = 2x + y
$$
\end{verbatim}

\linedivm

\newcommand{\pder}[2]{\frac{\partial #1}{\partial #2}}

If $z = x^2 + xy + y^2$, then
$$
\pder{z}{x} = 2x + y
$$

\linediv

\subsection*{Exercise Six:}

\linediv

\begin{verbatim}
If $f(x) = x^2$, then
\begin{align*}
    f'(a) & = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h} \\
    & = \lim_{h \to 0} \frac{(a+h)^2 - a^2}{h} \\
    & = \lim_{h \to 0} \frac{a^2 + 2ah + h^2 - a^2}{h} \\
    & = \lim_{h \to 0} 2a + h = 2a. \\
\end{align*}
\end{verbatim}

\linedivm

If $f(x) = x^2$, then
\begin{align*}
    f'(a) & = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h} \\
    & = \lim_{h \to 0} \frac{(a+h)^2 - a^2}{h} \\
    & = \lim_{h \to 0} \frac{a^2 + 2ah + h^2 - a^2}{h} \\
    & = \lim_{h \to 0} 2a + h = 2a. \\
\end{align*}

\linediv

\end{document}


\section{Abstract}

A \emph{Support Vector Machine} is a learning model used both for \emph{classification} and \emph{regression} tasks whose goal is to construct a \emph{maximum margin separator}, i.e., a decision boundary with the largest distance from the nearest training data points.

The aim of this report is to compare the \emph{primal}, the \emph{Wolfe dual}~\cite{fletcher2009support} and the \emph{Lagrangian dual} formulations of this model in terms of \emph{numerical precision}, \emph{accuracy} and \emph{complexity}.

Firstly, I will provide a detailed mathematical derivation of the model for all these formulations, then I will propose two algorithms to solve the optimization problem in case of \emph{constrained} or \emph{unconstrained} formulation of the problem, explaining their theoretical properties, i.e., \emph{convergence} and \emph{complexity}.

Finally, I will show some experiments for \emph{linearly} and \emph{nonlinearly} separable generated datasets to compare the performance of different \emph{kernels}, also by comparing the \emph{custom} results with \emph{sklearn} SVM implementations, i.e., \emph{liblinear}~\cite{fan2008liblinear} and \emph{libsvm}~\cite{chang2011libsvm} implementations, and \emph{cvxopt}~\cite{vandenberghe2010cvxopt} QP solver.


\subsection*{Ex.29.1-5}

We have the following linear program
$$
\begin{aligned}
 & \max & 2x_1 && && && - && 6x_3 &
\\
 & \ \text{s.t.} & x_1 && + && x_2 && - && x_3 & \leq 7
\\
 & & 3x_1 && - && x_2 && && & \geq 8
\\
 & & -x_1 && + && 2x_2 && + && 2x_3 & \geq 0
\\
 & & && && && && x_1,x_2,x_3 & \geq 0
\end{aligned}
$$
We first put this on standard form, by reversing two of the inequalities:
$$
\begin{aligned}
 & \max & 2x_1 && && && - && 6x_3 &
\\
 & \ \text{s.t.} & x_1 && + && x_2 && - && x_3 & \leq 7
\\
 & & -3x_1 && + && x_2 && && & \leq -8
\\
 & & x_1 && - && 2x_2 && - && 2x_3 & \leq 0
\\
 & & && && && && x_1,x_2,x_3 & \geq 0
\end{aligned}
$$
We can now convert the linear program into slack form, by introducing the slack variables $x_4$, $x_5$ and $x_6$, which measures the slack between the left-hand and right-hand sides of each of the inequalities, besides the nonnegativity constraints.
\\
The slack form will then have equality constraints, besides the nonnegativity constraints, as follows:
$$
\begin{aligned}
 & \max & && && && 2x_1 && && && - && 6x_3
\\
 & \ \text{s.t.} & x_4 = && 7 && - && x_1 && - && x_2 && + && x_3 &
\\
 & & x_5 = && -8 && + && 3x_1 && - && x_2 && &&
\\
 & & x_6 = &&  && - && x_1 && + && 2x_2 && + && 2x_3
\\
 & & && && x_1,x_2,x_3,x_4,x_5,x_6 \geq 0 \span\span\span\span\span\span\span\span\span\span\span\span
\end{aligned}
$$
And the basic and nonbasic variables are:
\\
Basic: $x_4,x_5,x_6$
\\
Nonbasic: $x_1,x_2,x_3$


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}

\title{Square root of 321}
\author{Dave Neary}
\date{January 2021}

\begin{document}

\section{Approximation of square roots}

We can estimate square roots of numbers that are close to perfect squares. For example, $18^2 = 324$
and
\begin{eqnarray*}
(18-x)^2& =& 321\\
324 - 36x + x^2 & =& 321\\
3 &=& 36x - x^2 \\
\end{eqnarray*}

Now, since $x$ will be small, $x^2$ will be negligible for a reasonable estimate. We can thus
discard the $x^2$ term, giving:

\[ x \approx \frac{3}{36} = \frac{1}{12}\]
for small values of $x$. So $\sqrt{321} \approx 18 - \frac{1}{12}$.

In general, if a number is close to a perfect square, we can write it as $n^2 + m$ for
$m\in \mathbb{Z}$, and you can approximate the square root with:
\[\sqrt{n^2+m} \approx n+\frac{m}{2n}\]

This also works for higher powers! If a number is close to $n^k$, we write it as $n^k+m$, and:
\[ \sqrt[k]{n^k+m} \approx n+\frac{m}{kn^{k-1}} \]

This is the method that Richard Feynmann used to approximate the cube root of $1729$ when
competing against an abacus salesman. $12^3 = 1728$ (which he knew from cubic inches in a cubic foot), so
\[ \sqrt[3]{1729} \approx 12 + \frac{1}{3\times12^2} = 12+\frac{1}{432} \]

We can also use the Binomial theorem extended to real exponents to get even closer approximations.
\begin{eqnarray*}
 \sqrt{321}& =& \sqrt{324 - 3} \\
& = & 18 \left(1-\frac{1}{108}\right)^{\frac{1}{2}} \\
& = &18 \left( 1 + \binom{\frac{1}{2}}{1}\left(\frac{-1}{108}\right) + \binom{\frac{1}{2}}{2}\left(\frac{1}{108^2}\right) + \cdots ) \right) \\
& \approx & 18 \left(1 - \frac{1}{216} - \frac{1}{93312} - \cdots\right) \\
& \approx & 18 - \frac{1}{12} - \frac{1}{5184} - \cdots
\end{eqnarray*}
where $\binom{x}{r} = \frac{(x)(x-1)(\cdots)(x-r+1)}{r!}$ for $x \in \mathbb{R}$.

 \end{document}


\subsection{Example}

Let $A=\begin{bmatrix}1&4&7\\ 2&5&8\\3&6&9\end{bmatrix}$. Finding the basis:

\[
    \mathrm{rref}(A)=
     \begin{bmatrix}1&0&-1\\ 0&1&2\\0&0&0\end{bmatrix}
\]

Since there is redundance, a possible expression of the basis of $A$:

\[\mathfrak{B}=\left\{\begin{bmatrix}1\\2\\3\end{bmatrix},\begin{bmatrix}4\\5\\6\end{bmatrix}\right\}\]

When finding kernel, must solve $A\tb{x}=\tb{0}$. So with $[\mathrm{rref}(A)|\tb{0}]$:

\[\tb{x}=\begin{bmatrix}t\\ -2t\\t\end{bmatrix}=t \begin{bmatrix}1\\-2\\1\end{bmatrix}\]

Thus,

\[\mathrm{ker}(A)=\left\{t\begin{bmatrix}1\\-2\\1\end{bmatrix}\bigg|\;t\in \R\right\}\]

\noindent
Subsequently, the basis of the kernel of $A$ can be represented as $\mathfrak{B}=\left\{\begin{bmatrix}1\\-2\\1\end{bmatrix}\right\}$.

\noindent
Equivalent statements for $\left\{\tb{v}_{1}, \tb{v}_{2}, \tb{v}_{3}, \ldots, \tb{v}_{m}\right\}$ being linearly independent:
\begin{itemize}
    \item None of the vectors are redundant
    \item Only relation is trivial
    \item Kernel of $\left(\begin{array}{cccc}
        \mid & \mid & & \mid \\
        \tb{v}_{1} & \tb{v}_{2} & \ldots & \tb{v}_{m} \\
        \mid & \mid & & \mid
        \end{array}\right)$ is trivial
    \item Rank of $\left(\begin{array}{cccc}
        \mid & \mid & & \mid \\
        \tb{v}_{1} & \tb{v}_{2} & \ldots & \tb{v}_{m} \\
        \mid & \mid & & \mid
        \end{array}\right)$ is $m$
    \item If $m=n$ then $\left(\begin{array}{cccc}
        \mid & \mid & & \mid \\
        \tb{v}_{1} & \tb{v}_{2} & \ldots & \tb{v}_{m} \\
        \mid & \mid & & \mid
        \end{array}\right)$ reduces to $I_n$
\end{itemize}


\section{Introduction and Background}
The FW is a classic dynamic programming algorithm that solves the \emph{all-pairs shortest path (APSP)} problem on a directed weighted
graph $G(V, E, w)$, where $V = \{1, \dots, n\}$ is a set of nodes, $E \subseteq V \times V$ are the edges and $w$ is a weight function $E \rightarrow  \mathbb{R}$
that expresses the cost of crossing two nodes. The number of nodes is denoted by $n$ and the number of edges by $m$ . \par
The output of the algorithm is typically in matrix form: the entry in the $i^{th}$ row and $j^{th}$ column is the weight of the shortest path between
nodes $i$ and $j$. FW runs in $\Theta(|V|^3)$ time and for this reason is a good choiche when working with dense graph: even though there
may be up to $\Omega(|E|^2)$ edges, the computational time is independent from the number of edges. \par
The FW algorithm is shown in \textbf{Algorithm \ref*{alg:fw1}}.

\begin{algorithm}[h!]

\SetAlgoLined

\For{$(u, v) \in E$}{
    $M_{u, v} \leftarrow w(u, v)$
}
\For{$v = 1 \rightarrow n$}{
    $M_{v, v} \leftarrow 0$
}
 \For{$k = 1 \rightarrow n$}{
  \For{$i = 1 \rightarrow n$}{
  \For{$j = 1 \rightarrow n$}{
  \If{$M_{i, j} > M_{i, k} + M_{k, j}$}{

    $M_{i, j} \leftarrow M_{i, k} + M_{k, j}$
 }
 }
 }
 }

\caption{The Floyd-Warshall (FW) algorithm}\label{alg:fw1}
\end{algorithm}

A C implementation of this algorithm can be found \href{https://github.com/firaja/Parallel-FloydWarshall/blob/master/sequential.c}{here};
this version is referred in this document as \emph{serial} FW and it is used as base for comparison. \par


\chapter{The hydrogen atom}\label{c6}
\section{The Hamiltonian}\label{c6s1}
The quantum mechanical treatment of the hydrogen atom resolved the mystery of
Bohr's second postulate. It also demonstrated the convenience of the
Schr\"{o}dinger's approach. His equation can be solved exactly for the hydrogen
atom. The hydrogen atom consists of a single electron in the electric field od a
single proton. Since the electron is almost $1836$ times lighter than the proton
we can, as a first approximation, assume that the proton is at rest. Therefore,
the energy of the system is
\begin{equation}\label{c6s1e1}
H = \frac{p^2}{2m} - \frac{1}{4\pi\epsilon_0}\frac{e^2}{r},
\end{equation}
where $e$ is the electronic charge and $r$ is the distance between the two. The
operator representation of this Hamiltonian is
\begin{equation}\label{c6s1e2}
\hat{H} = -\frac{\hslash^2}{2m}\nabla^2 - \frac{1}{4\pi\epsilon_0}\frac{e^2}{r}
\end{equation}
and the Schr\"{o}dinger's time independent equation is
\begin{equation}\label{c6s1e3}
-\frac{\hslash^2}{2m}\nabla^2\psi(\vec{r}) -
\frac{1}{4\pi\epsilon_0}\frac{e^2}{r}\psi(\vec{r}) = E\psi(\vec{r}).
\end{equation}
The spherical symmetry of the problem suggests that we should work in the
spherical polar coordinate system. The Laplacian in these coordinates is
\begin{equation}\label{c6s1e4}
\nabla^2 = \frac{1}{r^2}\frac{\partial}{\partial r}
\left(r^2\frac{\partial}{\partial r}\right) +
\frac{1}{r^2\sin\theta}\frac{\partial}{\partial\theta}
\left(\sin\theta\frac{\partial}{\partial\theta}\right) +
\frac{1}{r^2\sin^2\theta}\frac{\partial^2}{\partial\phi^2}
\end{equation}
The equation \eqref{c6s1e3} is solved using the method of `separation of
variables'. We write
\begin{equation}\label{c6s1e5}
\psi(\vec{r}) = \psi(r, \theta, \phi) = R(r)\Theta(\theta)\Phi(\phi)
\end{equation}
so that equation \eqref{c6s1e3} becomes
\begin{eqnarray}
-\frac{\hslash^2}{2m}\left[
\frac{\Theta\Phi}{r^2}\frac{d}{dr}\left(r^2\frac{dR}{dr}\right) +
\frac{R\Phi}{r^2\sin\theta}\frac{d}{d\theta}
\left(\sin\theta\frac{d\Theta}{d\theta}\right) +
\frac{R\Theta}{r^2\sin^2\theta}\frac{d^2\Phi}{d\phi^2}\right] &=& \nonumber \\
\left(E + \frac{1}{4\pi\epsilon_0}\frac{e^2}{r}\right)R\Theta\Phi & &
\label{c6s1e6}
\end{eqnarray}
Multiplying both sides by
\[
\frac{r^2\sin^2\theta}{R\Theta\Phi}
\]
and rearranging a bit, we get
\begin{equation}\label{c6s1e7}
\frac{\sin^2\theta}{R}\frac{d}{dr}\left(r^2\frac{dR}{dr}\right) +
\frac{\sin\theta}{\Theta}\frac{d}{d\theta}
\left(\sin\theta\frac{d\Theta}{d\theta}\right) +
\frac{2m}{\hslash^2}\left(E + \frac{e^2}{4\pi\epsilon_0 r}\right)
r^2\sin^2\theta =
-\frac{1}{\Phi}\frac{d^2\Phi}{d\phi^2}.
\end{equation}
The left hand side of this equation is a function of $r$ and $\theta$ while
the right hand side depends on $\phi$ alone. Therefore, each side must be equal
to a constant, say $\alpha$. In particular, we get
\begin{equation}\label{c6s1e8}
\frac{1}{\Phi}\frac{d^2\Phi}{d\phi^2} = -\alpha
\end{equation}
or
\begin{equation}\label{c6s1e9}
\frac{d^2\Phi}{d\phi^2} + \alpha\Phi = 0.
\end{equation}
This equation will have a periodic solution if $\alpha > 0$ and an exponential
one if $\alpha < 0$. The hydrogen atom is clearly periodic in $\phi$. Therefore,
we can write
\begin{equation}\label{c6s1e10}
\alpha = m_l^2
\end{equation}
so that we get
\begin{equation}\label{c6s1e11}
\frac{d^2\Phi}{d\phi^2} + m_l^2\Phi = 0
\end{equation}
and
\begin{equation}\label{c6s1e12}
\frac{\sin^2\theta}{R}\frac{d}{dr}\left(r^2\frac{dR}{dr}\right) +
\frac{\sin\theta}{\Theta}\frac{d}{d\theta}
\left(\sin\theta\frac{d\Theta}{d\theta}\right) +
\frac{2m}{\hslash^2}\left(E + \frac{e^2}{4\pi\epsilon_0r}\right)
r^2\sin^2\theta = m_l^2.
\end{equation}
The solution of the equation \eqref{c6s1e11} is
\begin{equation}\label{c6s1e13}
\Phi(\phi) = \alpha_1 e^{im_l\phi} + \alpha_2 e^{-im_l\phi},
\end{equation}
where $\alpha_1$ and $\alpha_2$ are constants of integration. Since $\Phi(\phi)
= \Phi(\phi + 2\pi)$ $m_l$ must be an integer. We can therefore write the
solution \eqref{c6s1e13} as just
\begin{equation}\label{c6s1e14}
\Phi_{m_l}(\phi) = \alpha_1 e^{im_l\phi}
\end{equation}
as the other factor is automatically included in the first one. $m_l$ is called
the \emph{magnetic quantum number} and it is an integer. In order to separate
equation \eqref{c6s1e12}, divide both sides by $\sin^2\theta$ and rearrange a
bit to get
\begin{equation}\label{c6s1e15}
\frac{1}{R}\frac{d}{dr}\left(r^2\frac{dR}{dr}\right) +
\frac{2m}{\hslash^2}\left(E + \frac{1}{4\pi\epsilon_0}\frac{e^2}{r}\right)r^2 =
-\frac{1}{\Theta\sin\theta}\left(\sin\theta\frac{d\Theta}{d\theta}\right) +
\frac{m_l^2}{\sin^2\theta}.
\end{equation}
Once again we have a situation in which the left hand side of the equation
depends on one variable, $r$, and the right hand side on another, $\theta$
so that each side is a constant, say $\beta$. Thus, we have
\begin{equation}\label{c6s1e16}
-\frac{1}{\Theta\sin\theta}\frac{d}{d\theta}
\left(\sin\theta\frac{d\Theta}{d\theta}\right) +\frac{m_l^2}{\sin^2\theta} =
\beta
\end{equation}
and
\begin{equation}\label{c6s1e17}
\frac{1}{R}\frac{d}{dr}\left(r^2\frac{dR}{dr}\right) +
\frac{2m}{\hslash^2}\left(E + \frac{1}{4\pi\epsilon_0}\frac{e^2}{r}\right)r^2 =
\beta.
\end{equation}
We will solve these equations in the following sections. And yet, before
proceeding we notice that the energy $E$ is part of the $r$-equation. This
indicates that the energy of an electron in the Hydrogen atom depends on $r$
alone, in agreement with the Bohr model.

\section{The $\theta$ and $r$ equations}\label{c6s2}
Equations \eqref{c6s1e6} and \eqref{c6s1e7} can be solved using the Frobenius
series solution method however the analysis is a bit labourious. We will,
therefore, only summarise it here. The solutions of \eqref{c6s1e16} remain
finite as $\theta \rightarrow 0$ and $\theta \rightarrow \pi$ only if $\beta$
is of the form $l(l+1)$ for a positive integer $l$. When $m_l = 0$, the
solutions of \eqref{c6s1e16} are called Legendre polynomials. When $m_l \le 0$,
they are required to be such that $-l \le m_l \le l$ and the corresponding
solutions are called associated Legendre polynomials. The product of the
solution of the $\theta$ equation and \eqref{c6s1e14} is called a spherical
harmonic of order $l$ and $m_l$. Spherical harmonics were studied as solutions
of Laplace equation since a century prior to the advent of quantum mechanics.
The Legendre polynomials for are denoted by $P_l(\cos\theta)$ and the associated
Legendre polynomials by $P_l^{m_l}(\cos\theta)$. The spherical harmonics are
denoted by
\begin{equation}\label{c6s1e18}
Y_{l, m_l}(\theta, \phi) = N_{lm_l}P_l^{m_l}(\cos\theta)\Phi_{m_l}(\phi).
\end{equation}
$l$ is called the \emph{orbital quantum number} and it restricts the values
$m_l$ can take. Likewise, the solution of the $r$ equation can be written in
terms of the associated Laguerre polynomials. The solutions exist for all
$E \ge 0$ but for $E < 0$, they exists only for the discrete set
\begin{equation}\label{c6s1e19}
E_n = -\frac{me^4}{32\pi^2 n^2\hslash^2\epsilon_0^2} =
-\frac{me^4}{8n^2h^2\epsilon_0^2},
\end{equation}
where $n \ge 0$ is an integer and it is always greater than $l$. It is called
the \emph{principal quantum number}.

Thus the three equations, one each for $r$, $\theta$ and $\phi$ give rise to the
quantum numbers $n, l, m_l$ such that
\begin{eqnarray}
n &\ge& 0 \label{c6s2e3} \\
l &<& n \label{c6s2e4} \\
|m_l| &\le& l \label{c6s2e5}
\end{eqnarray}
Equation \eqref{c6s2e5} can be written in an equivalent form
\begin{equation}\label{c6s2e6}
-l \le m_l \le l.
\end{equation}

\subsection{Problem set 1}
\begin{enumerate}
\item Consider a thought experiment in which the electrostatic potential was of
the form $e^{-r}$. Which of the three quantum numbers introduced above would
have remained unchanged.
\item Show that there are $n^2$ electrons with principal quantum number $n$.
You may remember having learnt that this number is $2n^2$. The factor of $2$
is missing in our analysis because we still have not take into account the spin
of an electron.
\item Although the proton is $1836$ times heavier than an electron, it does get
affected by the reaction of the electron. What would change in our analysis if
we want to take the proton's motion into account? (Hint: problem 4 in chapter
\ref{c3}.)
\end{enumerate}

\section{Quantization of angular momentum}\label{c6s3}
Bohr's second postulate was about the quantization of the electron's angular
momentum. We will now show that it is possible to understand it in the framework
of quantum mechanics. The angular momentum is an important dynamical variable
in quantum mechanics with a rich theory which explains its quantization in a
very natural way. We will not discuss it in this course and instead restrict
our analysis to hydrogen atom alone.

Since $\beta = l(l+1)$, we can rearrange equations \eqref{c6s1e6} and
\eqref{c6s1e7} as
\begin{eqnarray}
\frac{1}{\sin\theta}\frac{d}{d\theta}
\left(\sin\theta\frac{d\Theta}{d\theta}\right) +
\left(\frac{m_l^2}{\sin^2\theta} - l(l+1)\right)\Theta &=& 0 \label{c6s3e1} \\
\frac{1}{r^2}\frac{d}{dr}\left(r\frac{dR}{dr}\right) + \frac{2m}{\hslash^2}
\left[\left(E + \frac{1}{4\pi\epsilon_0}\frac{e^2}{r}\right)
- \frac{l(l+1)\hslash^2}{2mr^2}\right]R &=& 0 \label{c6s3e2}
\end{eqnarray}
The $r$-equation should ideally not have anything to do with the orbital motion.
Therefore, the presence of the term $l(l+1)\hslash^2/(2mr^2)$ is a bit
disconcerting. The total energy $E$ is the sum of the electron's potential
energy and kinetic energy. The former is $-e^2/(4\pi\epsilon_0 r)$, while the
latter can be written as the sum of $T_r$, the radial, and $T_o$, the orbital
kinetic energies. We conjecture that the orbital kinetic energy is
\begin{equation}\label{c6s3e3}
T_o = \frac{l(l+1)\hslash^2}{2mr^2}
\end{equation}
so that there is no dependence of the orbital motion on the $r$-equation. If
$v_o$ is the orbital velocity then
\begin{equation}\label{c6s3e4}
T_o = \frac{mv_o^2}{2}
\end{equation}
so that from \eqref{c6s3e3} and \eqref{c6s3e4} we get
\begin{equation}\label{c6s3e5}
mv_or = \sqrt{l(l+1)}\hslash.
\end{equation}
Although the angular momentum is not an integral multiple of $\hslash$ as Bohr
had proposed, it is quantized nevertheless.

\section{Electron density}\label{c6s4}
The squared modulus solution of \eqref{c6s1e6} is the probability density
function of finding an electron in the atom. Although the mathematical forms
of the wave-function and its modulus are quite formidable their plots are quite
familiar to us. The figure \ref{c6f1} labels the probability densities by the
triple $(n, l, m_l)$ of the quantum numbers. Wave functions with different
values of $l$ are called orbitals. The orbitals with $l = 0$ have a spherical
symmetry.  They are called s-orbitals. Observe the one with $n=2$. The orbital
has minumum, between two maxima of unequal sizes. The surfaces in which the
orbitals are zero are called nodal surfaces. Thus, the $2s$ orbital has a nodal
surface between the two maxima. The orbitals with $l = 1$ are called the p-
orbitals. They have two lobes separated by a nodal plane. The orbitals with
$l = 2$ are called d-orbitals and those with $l = 3$ are called f-orbitals.
They have far more complicated shapes. Their shapes play an important role in
inorganic chemistry, especially of the transition elements.

The diagram in figure \ref{c6f1} also gives an expression for the wavefunction.
The variable $\rho$ in it is the ratio $r/a_0$, where $a_0$ is the radius of the
first Bohr orbit. The factor $e^{-\rho}$ ensures that the wavefunction goes to
zero away from the nucleus. The angular distribution of the orbitals is
determined solely by the spherical harmonics and the radial distribution by the
associated Laguerre polynomials (and $e^{-\rho}$).
\begin{figure}
\begin{center}
\includegraphics[scale=0.20]{Hydrogen_Density_Plots}
\caption{Hydrogen orbitals credit: Wikipedia}
\label{c6f1}
\end{center}
\end{figure}

\subsection{Problem set 2}
\begin{enumerate}
\item Based on your understanding of the solutions of the $r, \theta$ and
$\phi$ equations show that the $|\psi|^2$ is independent of the azimuthal angle
$\phi$. How do you decide where the $z$-axis is?
\item The wavefunction for the $n=1, l=0, m_l=0$ is
\begin{equation}\label{c6s3e6}
\psi_{1,0,0}(r,\theta,\phi) = \frac{e^{-r/a_0}}{\sqrt{\pi}a_0^{3/2}}.
\end{equation}
Let $P(r)$ be the probability density of finding an electron in the thin
spherical shell with radii $r$ and $r + dr$. Then,
\begin{equation}\label{c6s3e7}
P(r) = \int_0^{\pi}\int_0^{2\pi}|\psi_{1,0,0}^2|^2r^2\sin\theta d\theta d\phi.
\end{equation}
Show that
\begin{equation}\label{c6s3e8}
P(r) = \frac{4r^2 e^{-r/a_0}}{a_0^3}.
\end{equation}
Hence show that $P$ has a maximum at $r = a_0$, where $a_0$ is the radius of the
first Bohr orbit.
\end{enumerate}


\subsection{Matrix square roots}

For a matrix \(M\), the square root \(M^{\dfrac{1}{2}}\) is \(A\) where \(AA=M\).

This does not necessarily exist.

Square roots may not be unique.

Real matrices may have no real square root.


\lab{Lorenz Equations}{Lorenz Equations}
% \epigraph{\textit{Chaos: When the present determines the future, but the approximate present does not approximately determine the future.}}{Edward Lorenz}
\objective{Investigate the behavior of a system that exhibits chaotic behavior.
Demonstrate methods for visualizing the evolution of a system.}

Chaos is everywhere.
It can crop up in unexpected places and in remarkably simple systems, and a great deal of work has been done to describe the behavior of chaotic systems.
One primary characteristic of chaos is that small changes in initial conditions result in large changes over time in the solution curves.

\section*{The Lorenz System}
One of the earlier examples of chaotic behavior was discovered by Edward Lorenz.
In 1963, while working to study atmospheric dynamics he derived the simple system of equations
\begin{align*}
\frac{\partial x}{\partial t} &= \sigma \left(y - x\right) \\
\frac{\partial y}{\partial t} &= \rho x - y - x z \\
\frac{\partial z}{\partial t} &= x y - \beta z
\end{align*}
where $\sigma$, $\rho$, and $\beta$ are all constants.
After deriving these equations, he plotted the solutions and observed some unexpected behavior.
For appropriately chosen values of $\sigma$, $\rho$, and $\beta$, the solutions did not tend toward any steady fixed points, nor did the system permit any stable cycles.
The solutions did not tend off toward infinity either.
With further work, he began the study of what was called a strange attractor.
This system, though relatively simple, exhibits chaotic behavior.

\subsection*{Plotting and Interacting}
We will be making interactive 3D plots using \li{matplotlib}.
Please refer back to the \textit{Introduction to Matplotlib} lab for tips and tricks to graphing 3D.
Below is some sample code to help you out.

\begin{lstlisting}
from matplotlib import rcParams, pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
rcParams['figure.figsize'] = (16,10)     #Affects output size of graphs.
'''
Code up your X, Y, Z values
'''
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot( X, Y, Z )    #Make sure X, Y, Z are same length.
                      #Connect points (X[i], Y[i], Z[i]) for i in len(X)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')

ax.set_xlim3d([min(X), max(X)])    #Bounds the axes nicely
ax.set_ylim3d([min(Y), max(Y)])
ax.set_zlim3d([min(Z), max(Z)])

plt.show()
\end{lstlisting}

Because we are plotting in 3D, we are also interested in being able to interact with the plot and rotate it to analyze what our solutions are actually doing.
To facilitate this, \li{matplotlib} has different backends which allow for various levels of control over what our graphs do.
These backends are accessed using \li{matplotlib}'s \li{switch_backend()} function with a call to whichever backend we want.
Your computer may default to a backend which you can already interact with (try clicking your plot and rotating it to see).
If not, consider using 'qt5agg', 'qt4agg', nbagg', or 'tkagg' in the function call.

\begin{lstlisting}
plt.switch_backend('qt5agg') # This backend opens the graph in a new window
\end{lstlisting}

\begin{warn}
Do not run \li{\%matplotlib inline} with this code. You will crash your kernel.
\end{warn}

\begin{problem}
Solve and graph the Lorenz equation by completing the code.
Initialize the initial conditions with random values between $-15$ and $15$.
For this exercise, let $\sigma=10$, $\rho=28$, $\beta=\frac{8}{3}$.
Compare to Figure \ref{fig:Single_Lorenz}
\begin{lstlisting}
import numpy as np
from scipy.integrate import odeint

def lorenz_ode(inputs, T):
	'''
	Code up the sytem of equations given
	'''
	return Xprime, Yprime, Zprime

def solve_lorenx(init_cond, time=10):
	T = np.linspace(0, time, time*100)	#initialize time interval for ode
	'''
	Use odeint in conjuction with lorenz_ode and the time interval T
	To get the X, Y, and Z values for this system.
	You will need to transpose the output of odeint to graph it correctly.
	'''
 	return X, Y, Z

sigma = 'value'
rho = 'value'
beta = 'value'
init_cond = [x0, y0, z0]

X, Y, Z = solve_lorenz(init_cond, 50)
'''
Code to graph
'''
\end{lstlisting}

\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{figures/Single_Lorenz.pdf}
\caption{Approximate solution to the Lorenz equation with random initial conditions}
\label{fig:Single_Lorenz}
\end{figure}

\section*{Basin of Attraction}
Notice in the first problem that the solution tended to a 'nice' region.
This region is a basin of attraction, and the set of numerical values towards which a system will converge to is an \textbf{attractor}.
Consider what happens when we change up the initial conditions.
\begin{problem}
Change your code to plot $n$ different solution using different random initial conditions.
Produce a plot with $n=3$ different solutions.
Compare to Figure \ref{fig:Multiple_Lorenz}
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{figures/Multiple_Lorenz.pdf}
\caption{Multiple solutions to the Lorenz equation with random initial conditions}
\label{fig:Multiple_Lorenz}
\end{figure}

\section*{Chaos}

Chaos in dynamical systems connotes a high sensitivity to initial conditions.
Small differences in initial conditions can yield widely diverging outcomes for these systems.
Despite having equations that completely determine the future behavior of systems, there is no way to determine where a system will be time $t$ down the road without running through the full interval.
In other words, these systems are unpredictable.

\begin{problem}
Change the code above so that it initializes one set of initial conditions and creates a second set of initial conditions by adding the array \li{np.random.randn(3)*(1E-10)}.
This will represent a small perturbation in the initial conditions. Make sure the \li{time} variable is large enough to notice a difference in the two solutions.
Plot both solutions together.
Refer to Figure \ref{fig:perturbed_lorenz}
\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{figures/perturbed_lorenz.pdf}
\caption{Two solutions to the Lorenz equation with perturbed initial conditions}
\label{fig:perturbed_lorenz}
\end{figure}

You may be wondering if your code was even correct.
You could distinctly see the two curves because there was some separation of color, but who is to say that the two solutions moved away from each other only slightly as time went on.
Our next task will be to \textit{animate} this beast.

\section*{Intro to Animation}
Let's get you a degree in Computer Animation.
You will be provided with some code that helps with animation.
Please feel free to play around and try to animate different things.
Again, refer to the \textit{Introduction to Matplotlib} lab for additional help.

We will call on \li{matplotlib.animate} to assist in animation. Import the following:
\begin{lstlisting}
from matplotlib.animation import FuncAnimation
\end{lstlisting}
Follow these steps for setting up an animation.
\begin{itemize}
\item Calculate all data needed for the animation (not necessary in some cases, but it simplifies things).
\item Define a figure explicitly with \li{plt.figure()} and set its window boundaries.
\item Draw empty objects that can be altered dynamically.
\item Define a function to update the drawing objects.
\item make a call to \li{FuncAnimation()} which will start the animation.
\end{itemize}
\li{FuncAnimation()} accepts the figure to be animated, the function that updates the figure, the number of frames to show before repeating, and how fast to run the animation (lower number $=$ faster).

Check out the sample code for animating two 2D objects simultaneously.
\begin{lstlisting}
from matplotlib.animation import FuncAnimation

def sine_cos_animation():
	#Calculate the data to be animated
	x = np.linspace(0, 2*np.pi, 200)[:-1]
	y1, y2 = np.sin(x), np.cos(x)

	#Create a figure and set the window boundaries
	fig = plt.figure()
	plt.xlim(0, 2*np.pi)
	plt.ylim(-1.2, 1.2)

	#Initiate empty lines of the correct dimension
	sin_drawing, = plt.plot([], [])
	cos_drawing, = plt.plot([], [])	#note the comma after the variable name

	#Define a function that updates each line
	def update(index):
		sin_drawing.set_data(x[:index], y1[:index])
		cos_drawing.set_data(x[:index], y2[:index])
		return sin_drawing, cos_drawing,

	a = FuncAnimation(fig, update, frames=len(x), interval=10)
	plt.show()
\end{lstlisting}

\begin{warn}
The above animation code works great for 2D animations, but \li{set_data()} is only good for 2D. When you want to animate in 3D, you will need an extra empty list in \li{plt.plot()} and a call to \li{set_3d_properties()} in the \li{update(index)} function.
\end{warn}

\section*{Animate}
You now know how to plot the Lorenz equation, how to plot multiple equations, and even how to animate (yay you, you're a star) simple plots.
It's time to put all your know-how to good use.

\begin{problem}
Animate the solutions of the Lorenz equation for an initial set of conditions and the perturbed conditions on the same plot.
To make the animation go faster, decrease the \li{interval} value in the \li{FuncAnimation()} call.
It will take several seconds before the curves split, so be patient.
\end{problem}

If our system is very chaotic, then even small round off errors in your computer can lead to drastically different solutions.

\begin{problem}
Now set one initial condition.
Use \li{odeint} to solve the system, but use the arguments \li{atol=1E-14}, \li{rtol=1E-12}, and then again with \li{atol=1E-15}, \li{rtol=1E-13}.
Animate both solutions on the same plot.
\end{problem}

\section*{Lyapunov Exponents}
The Lyapunov exponent of a dynamical system is one measure of how chaotic a system is.
While there are more conditions for a system to be considered chaotic, one of the primary indicators of a chaotic system is \emph{extreme sensitivity to initial conditions}.
Strictly speaking, this is saying that a chaotic system is poorly conditioned.
Usually, in dynamical systems, the sensitivity to changes in initial conditions depends exponentially on the time the system is allowed to evolve.
If $\delta(t)$ represents the difference between two solution curves, when $\delta(t)$ is small, the following approximation holds.
\[\|\delta(t)\| \sim \|\delta(0)\| e^{\lambda t}\]
where $\lambda$ is a constant called the Lyapunov exponent.
For the Lorenz system, experimentally it can be verified that $\lambda \approx .9$.

\begin{figure}
\includegraphics[width=\textwidth]{figures/semilog.pdf}
\caption{A semilog plot of the separation between two solutions to the Lorenz equations together with a fitted line that gives a rough estimate of the Lyapunov exponent of the system.}
\label{fig:lyapunov_exponent}
\end{figure}

Suppose we want tol find the Lyapunov exponent for the Lorenz System.
Start by importing \li{linalg} and \li{linregress} modules from \li{scipy}.
\begin{lstlisting}
from scipy import linalg as la
from scipy.stats import linregress
\end{lstlisting}
To get a good estimate of the Lyapunov exponent, we want to make sure our initial solution already lies on the attractor.
To do this, choose some random initial conditions, run your \li{solve_lorenz} function, then pick out the final coordinates.
Let these coordinates be the starting point for our next system.

Next perturb the conditions slightly.
Find the solution curve using these two sets of initial coordinates and then calculate the \li{norm} between the solution at each point.
Plot the norm using \li{plt.semilogy(time, norm)}.

Finally we want to calculate the exponential line fitted to the data.
Because we are using the semilog plot, we can find this line as follows:
\begin{itemize}
\item take the \li{log} of the norms.
\item use \li{linregress()} to compute the linear regression of the log of the norms.
\item take the \li{exp} of the linear regression to turn back into an exponential regression.
\item plot the exponential regression using \li{plt.semilogy()} (your solution will be a line)
\end{itemize}

\begin{problem}
Write a Python function that finds an initial point on the strange attractor, runs the simulation to a given time $t$, and produces a semilog plot of the norm of the difference between the two solution curves.
Also have it plot an exponential line fitted to match the curve (this will be linear on the semilog plot).
Have it return a rough estimate of the Lyapunov exponent.
The output should be something like Figure \ref{fig:lyapunov_exponent}.

Note: In order to get a good estimate of the Lyapunov exponent, your initial guess should already lie on the strange attractor.
You can get a value on the attractor by running the system for a while to find a good initial guess.

Hint: To find the fitting line, take the logarithm of the norms of the differences, compute a linear fit, then take the exponential function of the resulting line.
The Lyapunov exponent will be approximately equal to the slope found by the linear regression.
\end{problem}


\lab{Image Compression (SVD)}{SVD}

\objective{Explore the SVD as a method of image compression}
\label{lab:SVD}

In this lab, we are going to explore how the SVD can be used to compress image data.
Recall that the SVD is a decomposition of an $m \times n$ matrix $A$ of rank $r$ into the product $A = U \Sigma V^H$, where $U$ and $V$ are unitary matrices having dimensions $m \times m$ and $n \times n$, respectively, and $\Sigma$ is an $m \times n$ diagonal matrix.
\begin{equation*}
\Sigma = \mbox{diag}(\sigma_1,\sigma_2,\ldots,\sigma_r,0,\ldots,0)
\end{equation*}
where $\sigma_1 \geq \sigma_2 \geq \ldots \geq \sigma_r > 0$ are the singular values of $A$.
Upon closer inspection, we see that we can write
\begin{equation*}
U = \begin{pmatrix}U_1 & U_2\end{pmatrix}, \quad \Sigma =
\begin{pmatrix}\Sigma_r & 0\\0 & 0\end{pmatrix}, \quad V =
\begin{pmatrix}V_1 & V_2\end{pmatrix},
\end{equation*}
where $U_1$ and $V_1$ have dimensions $m\times r$ and $n\times r$ respectively and $\Sigma_r$ is the $r\times r$ diagonal matrix of (nonzero) singular values.
Multiplying this out yields the reduced form of the SVD
\begin{equation*}
A =
\begin{pmatrix}U_1 & U_2\end{pmatrix}
\begin{pmatrix}\Sigma_r & 0\\0 & 0\end{pmatrix}
\begin{pmatrix}V^H_1 \\ V^H_2\end{pmatrix} =
U_1 \Sigma_r V_1^H
\end{equation*}

\subsection*{Low rank data storage}
If the rank of a given matrix is significantly smaller than its dimensions, the reduced form of the SVD offers a way to store $A$ with less memory.
Without the SVD, an $m\times n$ matrix requires storing $mn$ values.
By decomposing the original matrix into the SVD reduced form, $U_1$, $\Sigma_r$ and $V_1$ together require $mr+r+nr$ values.
Thus if $r$ is much smaller than both $m$ and $n$, we can obtain considerable efficiency.
For example, suppose $m=100$, $n=200$ and $r=20$.
Then the original matrix would require storing $20,000$ values whereas the reduces form of the SVD only requires storing $6020$ values.

\subsection*{Low rank approximation}
The reduced form of the SVD also provides a way to approximate a
matrix with another one of lower rank.
This idea is used in many areas of applied mathematics including signal processing, statistics, semantic indexing (search engines), and control theory.
If we are given a matrix $A$ of rank $r$, we can find an approximate matrix $\widehat A$ of rank $s<r$ by taking the SVD of $A$ and setting all of its singular values after $\sigma_s$ to zero, that is,
\begin{equation*}
\Sigma_{\widehat A} = \mbox{diag}(\sigma_1,\sigma_2,\ldots,\sigma_s,0,\ldots,0)
\end{equation*}
and then multiplying the matrix back together again.
The more singular values we keep, the closer our approximation is to $A$.
The number of singular values we decide to preserve depends on how close of an approximation we need and what our size requirements are for $U_1$, $\Sigma_{\widehat A}$, and $V_1$.
Try plotting the the singular values.
In Figure \ref{fig:hubble} we present an image and its singular values.
Matrix rank is on the x-axis and the singular values are the y-axis.
Note that SVD orders the singular values from greatest to least.
The greatest singular values contribute most to the image while the smallest singular values hardly contribute anything to the final approximation.
By looking at the graph in Figure \ref{fig:svals_plot} we can have a rough idea of how many singular values we need to preserve to have a good approximation of $A$.
The matrix rank of the image below is $670$.
However, as the plot shows, we could easily approximate the image using only the first half of the singular values.  In Figure \ref{fig:rankvalues}, we can see different rank approximations of the image in Figure \ref{fig:hubble}.
\begin{figure}
\centering
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{hubble_red}
\caption{NGC 3603 (Hubble Space Telescope).}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{hubble_svals}
\caption{Singular values from greatest to smallest.}
\label{fig:svals_plot}
\end{subfigure}
\caption{An image and its singular values.}
\label{fig:hubble}
\end{figure}

\begin{figure}
\centering
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{rank1.jpg}
\caption{Rank 1 approximation}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{rank14.jpg}
\caption{Rank 14 approximation}
\end{subfigure}

\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{rank27.jpg}
\caption{Rank 27 approximation}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{rank40.jpg}
\caption{Rank 40 approximation}
\end{subfigure}
\caption{Different rank approximations for SVD based compression.  Notice that higher rank is needed to resolve finer detail.}
\label{fig:rankvalues}
\end{figure}

The \li{scipy.linalg} module has a convenient method to calculate the SVD of a given matrix. We can use
this method to create a lower-rank approximation of a given matrix. Execute the following code.
\begin{lstlisting}
>>> import numpy as np
>>> from scipy.linalg import svd, norm
>>> A = np.array([[1,1,3,4], [5,4,3,7], [9,10,10,12], [13,14,15,16], [17,18,19,20]])
>>> U,s,Vt = svd(A, full_matrices=False)
\end{lstlisting}
In that last line of code, we included the keyword argument \li{full_matrices=False} to calculate the
reduced SVD rather than the full SVD. The arrays \li{U} and \li{Vt} correspond to the matrices
$U_1$ and $V_1^H$ discussed earlier in the lab. The array \li{s} simply gives the nonzero singular values
of the matrix \li{A}, and we can find the rank of \li{A} by inspecting the number of entries in \li{s} (in this
example, we have a rank 4 matrix).

Next, we calculate a rank 3 approximation. Instead of setting the smallest singular value to 0, we simply
omit it from the calculation. Note that we also omit the last column of \li{U} and last row of \li{Vt}, as
they correspond to the smallest eigenvalue.
\begin{lstlisting}
>>> S = np.diag(s[:-1])
>>> Ahat = U[:,:-1].dot(S).dot(Vt[:-1,:])
>>> norm(A-Ahat)
\end{lstlisting}
Note that $\widehat A$ is ``close'' to the original matrix $A$, but that its rank is 3 instead of 4. More
precisely, $\widehat A$ is the best rank 3 approximation of $A$ with respect to both the induced 2-norm and
the Frobenius norm.

\begin{problem}
Write a function \li{svd_approx} that takes as input a matrix $A$ and a positive integer $k$ and returns
the best rank $k$ approximation to $A$ with respect to the induced 2-norm.
\end{problem}

\subsection*{Application to Imaging}
Enter the following into IPython (note that any image you might have will work):
\begin{lstlisting}
>>> import matplotlib.pyplot as plt
>>> X = plt.imread('fingerprint.png')[:,:,0].astype(float)
>>> X.nbytes      #number of bytes needed to store X
>>> plt.imshow(X)
>>> plt.show()
\end{lstlisting}
Computing the SVD of your image is simple.
Remember to make the singular values a diagonal matrix before multiplying.
\begin{lstlisting}
>>> U,s,Vt = svd(X, full_matrices=False)
>>> S = sp.diag(s)
\end{lstlisting}
In the next code block, $n$ represents the desired rank of the output.
\begin{lstlisting}
>>> n = 50
>>> u1, s1, vt1 = U[:,0:n], S[0:n,0:n], Vt[0:n,:]
>>> Xhat = u1.dot(s1).dot(vt1)
>>> (u1.nbytes + np.diag(s1).nbytes + vt1.nbytes) - X.nbytes   #should be negative
>>> plt.imshow(Xhat)
>>> plt.show()
\end{lstlisting}
Recall that the error between the best rank $s$ approximation $\widehat{A_s}$ to $A$ with respect to the induced
2-norm is given by
$$
\|A - \widehat{A_s}\|_2 = \sigma_{s+1},
$$
where $\sigma_{s+1}$ is the $(s+1)$-th singular value of $A$.

\begin{problem}
Sometimes there is not enough available bandwidth to transmit a full resolution photograph.
You aim to reduce the amount of data that needs to be transmitted from a remote location such that loss of image detail is minimal, but the amount of data that needs to be sent has reduced as much as possible.
Write a function \li{lowest_rank_approx} that takes as input a matrix $A$ and a positive number $e$ and returns
the lowest rank approximation of $A$ with error less than $e$ (with respect to the induced 2-norm).
\end{problem}