\documentclass[]{simple}

\title{Sum of the Natural Numbers and the Riemann-Zeta Function}
\date{January, 2016}
\author{Matt McCarthy\\\href{mailto:matthew.mccarthy.12@cnu.edu}{matthew.mccarthy.12@cnu.edu}}

\addbibresource{naturals-and-zeta.bib}

\begin{document}

\maketitle

\section{Introduction}

We begin by introducing a rather interesting result, namely that the sum of the natural numbers is $-1/12$.
\begin{thm}\label{wrong}
	\[
		\sum_{n=1}^\infty n = -\frac{1}{12}
	\]
\end{thm}
We now provide the canonical proof as shown by \cite{wrong-pf}.
\begin{proof}[Proof (\autoref{wrong})]
	Define $s:=1+2+3+4+\ldots$, $s_1:=\sum_{n=0}^\infty(-1)^n$, and $s_2:=1-2+3-4+\ldots$.
	If we consider $s_1$, when we stop after an odd number of terms, the partial sum is $1$ but when we stop after an even number of terms, the partial sum is zero.
	Obviously, the series converges to the average of the two, thus $s_1=1/2$.
	Consider $2s_2$.
	\[
	\begin{array}{rrrrrrr}
		 &1 & -2 &+3 &-4&+\ldots\\
		+&  &1   &-2 &+3&-4&+\ldots\\\hline
		 &1 &-1  &+1 &-1&+\ldots
	\end{array}
	\]
	Since $2s_2=s_1=1/2$, $s_2=1/4$.
	Consider $s-s_2$.
	\[
		\begin{array}{rrrrrr}
			&1&+2&+3&+4&+\ldots\\
			-&1&-2&+3&-4&+\ldots\\\hline
			&0&+4&+0&+8&+\ldots
		\end{array}
	\]
	Moreover, we can factor out the four to get $s-s_2=4[1+2+3+\ldots]=4s$
	Solving for $s$ then yields $s=-1/12$ and completes the proof.
\end{proof}
\begin{analysis}
	The trick to this proof lies in exploiting the commutativity of addition.
	By doing so, we can manipulate the subtraction of two infinite series into a single alternating series.
	After that step, we then use the Taylor series we found in the lemma to complete the proof.
\end{analysis}

Now that we have ``shown'' that the natural numbers sum up to $-1/12$ the only things left to do are show that it is wrong and how this incorrect result came to be so prolific.

\section{Refutation}

We begin refuting this result by stating two of the limit laws and by providing the definition of an infinite series.
\begin{thm}[Limit Laws]
	Let $\set{x_n}_{n=1}^\infty$ and $\set{y_n}_{n=1}^\infty$ be real-valued sequences that converge to $x$ and $y$ respectively.
	Then all of the following hold.
	\begin{enumerate}
		\item The sequence $\set{x_n+y_n}_{n=1}^\infty$ converges to $x+y$.
		\item Let $k\in\RR$ then $\set{k\cdot x_n}_{n=1}^\infty$ converges to $k\cdot x$.
	\end{enumerate}
\end{thm}
These laws should seem familiar, considering that we used both in our ``proof'' of \autoref{wrong} however we may have invoked them without respecting their hypotheses.

\begin{definition}[Infinite Series]
	Let $\set{x_k}_{k=1}^\infty$ be a sequence of real numbers and let $x\in\RR$.
	An \textit{infinite series} is an expression of the form
	\[
		\sum_{k=1}^\infty x_k=x_1+x_2+\ldots.
	\]
	The corresponding \textit{sequence of partial sums} is defined by
	\[
		s_n:=\sum_{k=1}^n x_k.
	\]
	The series, $\sum_{k=1}^\infty x_k$ \textit{converges} to $x$ if and only if the sequence $\set{s_n}_{n=1}^\infty$ converges to $x$.
\end{definition}
This definition tells us that an infinite sum exists if and only if the its corresponding sequence of partial sums converges, which leads us to our biggest error: assuming the sum exists.

To determine whether or not the sum exists, we need to consider the sequence of partial sums,
\[
	s_n:=\sum_{k=1}^n k.
\]
If we inspect the sequence long enough, we see that the sum only increases and is what we call a \textit{monotone increasing sequence}.
\begin{definition}[Monotone Increasing Sequence]
	Let $\set{x_n}_{n=1}^\infty$ be a real-valued sequence.
	We say $\set{x_n}$ is \textit{monotone increasing} if and only if $x_n\leq x_{n+1}$ for each $n\in\NN$.
\end{definition}
If we consider the $(n+1)$ partial sum, we see
\[
	s_{n+1}=\sum_{k=1}^{n+1} k = \paren{\sum_{k=1}^{n} k} + (n+1) \geq \sum_{k=1}^{n} k=s_n.
\]
Thus, $\set{s_n}$ is a monotone increasing sequence.

Since $s_n$ is monotone increasing, we have a very nice theorem that turns our limit problem into a boundedness problem.

\begin{definition}[Upper Bound]
	Let $\set{x_n}_{n=1}^\infty$ be a real valued sequence and let $b\in\RR$.
	We say $b$ is an \textit{upper bound} of $\set{x_n}$ if and only if $x_n\leq b$ for each $n\in\NN$.
	If any such $b$ exists, we say $\set{x_n}$ is \textit{bounded above}.
\end{definition}

\begin{thm}[Monotone Convergence Theorem]
	Let $\set{x_n}_{n=1}^\infty$ be a real-valued monotone increasing sequence. Then $\set{x_n}$ converges if and only if $\set{x_n}$ is bounded above.
\end{thm}

With this theorem in our arsenal, we will now show that this infamous sum does not converge.

\begin{proposition}
	The sequence defined by
	\[
		s_n:=\sum_{k=1}^n k
	\]
	is unbounded.
\end{proposition}
\begin{proof}
Assume that $s_n$ is bounded above.
Ergo there exists a natural number $N > 0$ such that $s_n:=\sum_{k=1}^n k \leq N$ for every $n\in\NN$.
We know that
\[
	s_n:=\sum_{k=1}^n k = \frac{n(n+1)}{2}.
\]
Consider $s_{2N}$.
\[
	s_{2N} = \frac{2N(2N+1)}{2}=N(2N+1)=2N^2+N>N
\]
The previous statement contradicts our assumption and thus $s_n$ is an unbounded sequence.
\end{proof}
\begin{analysis}
	We perform a standard contradiction proof.
	The only tricks we use are the fact that
	\[
		1+2+\ldots+n=\frac{n(n+1)}{2}
	\]
	and a careful selection of $n$.
	Essentially, the proof boils down to picking a candidate for a bound and showing that we have a term in the sequence that is greater than our bound.
\end{analysis}

Since $s_n$ is unbounded, by the monotone convergence theorem, $s_n$ does not converge and neither does our sum.
Thus, we cannot invoke limit laws in order to compute $-4s$ and $s-4s$ in our proof of \autoref{wrong}.
In turn, this invalidates our derivation of $-3s$ which destroys the rest of the proof.

\section{The Riemann-Zeta Function}

To figure out why people believe this nonsense, we turn to the Riemann-Zeta function.
Historically, the Riemann-Zeta function arose as a way to extend the Euler-Zeta function, defined on real numbers greater than 1
\[
	\zeta(s)=\sum_{n=1}^\infty \frac{1}{n^s}
\]
to the rest of the complex plane \cite{edwards}.
When viewed in this light, one may think that
\[
	\zeta(-1)=\sum_{n=1}^\infty n.
\]
However, this is not the case since the Riemann-Zeta function is defined as follows.
\begin{definition}[Riemann-Zeta Function from \cite{edwards}]
	The \textit{Riemann-Zeta function} is the map $\zeta:\CC\rightarrow\CC$ defined by the contour integral
	\[
		\zeta(s):=\frac{\Pi(-s)}{2\pi i}\int_\gamma\frac{(-x)^s}{(e^x-1)x}dx
	\]
	where $\gamma$ is a curve that starts at $+\infty$, moves towards the origin along the positive real axis, circles the origin in a counterclockwise direction, and returns to $+\infty$ along the positive real axis.
	Furthermore, $\Pi(s)$ is defined as
	\[
		\Pi(s):=\int_0^\infty e^{-x}x^s dx.
	\]
\end{definition}

As we can see from the definition, this function does not resemble the geometric series in any way.
However, if $s$ is real and greater than $1$ (note: strictly greater than, not greater than or equal to) the zeta function can be written as
\begin{equation}\label{zeta-series}
	\zeta(s)=\sum_{n=1}^\infty\frac{1}{n^s}.
\end{equation}
Furthermore, since $-1$ is less than $1$,
\[
	\zeta(-1)\neq\sum_{n=1}^\infty n.
\]

Another interesting property of the Riemann-Zeta function pops up when we let $s$ be a negative integer.
\begin{proposition}[From \cite{edwards}]
	For any natural number $n$,
	\[
		\zeta(-n)=(-1)^n\frac{B_{n+1}}{n+1}
	\]
	where $B_{n+1}$ is the $n+1$th Bernoulli number.
\end{proposition}
By using this property we get
\begin{equation}\label{zeta-1}
	\zeta(-1)=(-1)\frac{B_2}{2}=(-1)\paren{\frac{1}{6}}\paren{\frac{1}{2}}=-\frac{1}{12}.
\end{equation}

Very likely, the incorrect conclusion that $1+2+3+\ldots=-1/12$ came out of some confusion arising from \autoref{zeta-series}, \autoref{zeta-1}, and the history of the zeta function.

\printbibliography

\end{document}


\section*{Exercises}

\begin{ex}
  True or false? ``Cramer's rule is useful for finding solutions
    to systems of linear equations in which there is an infinite set
    of solutions.''
  \begin{sol}
    False. Cramer's rule only works when the coefficient matrix is
    invertible. In these cases, the solution is always unique.
  \end{sol}
\end{ex}

\begin{ex}
  Use Cramer's rule to find the solution to
  \begin{equation*}
    \begin{array}{c}
      x+2y=1 \\
      2x-y=2
    \end{array}
  \end{equation*}
  \begin{sol}
    Solution is: $(x,y) = (1,0)$.
  \end{sol}
\end{ex}

\begin{ex}
  Use Cramer's rule to find the solution to
  \begin{equation*}
    \begin{array}{c}
      x+2y+z=3 \\
      2x-y-z=1 \\
      x+z=1
    \end{array}
  \end{equation*}
  \begin{sol}
    Solution is: $(x,y,z) = (1,1,0)$. For example,
    \begin{equation*}
      y=\frac{\begin{absmatrix}{rrr}
          1 & 3 & 1 \\
          2 & 1 & -1 \\
          1 & 1 & 1
        \end{absmatrix}}{\begin{absmatrix}{rrr}
          1 & 2 & 1 \\
          2 & -1 & -1 \\
          1 & 0 & 1
        \end{absmatrix}}=1
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Use Cramer's rule to solve the system of equations
  \begin{equation*}
    \begin{mymatrix}{rrr}
      2 & 1 &  1 \\
      2 & 2 & -1 \\
      2 & 1 &  2 \\
    \end{mymatrix} \begin{mymatrix}{c}
      x \\
      y \\
      z \\
    \end{mymatrix} =\begin{mymatrix}{r}
      3 \\
      -2 \\
      5 \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    The solution is $(x,y,z) = (1,-1,2)$.
  \end{sol}
\end{ex}

\begin{ex}
  Find the value of $y$ in the following system of equations:
  \begin{equation*}
    \begin{mymatrix}{ccc}
      1 & 1 &  1   \\
      1 & t &  t^2 \\
      1 & s &  s^2 \\
    \end{mymatrix} \begin{mymatrix}{c}
      x \\
      y \\
      z \\
    \end{mymatrix} =\begin{mymatrix}{c}
      t \\
      s \\
      1 \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    By Cramer's rule, we have
    \begin{equation*}
      y = \frac{
        \begin{absmatrix}{ccc}
          1 & t &  1   \\
          1 & s &  t^2 \\
          1 & 1 &  s^2 \\
        \end{absmatrix}
      }{
        \begin{absmatrix}{ccc}
          1 & 1 &  1   \\
          1 & t &  t^2 \\
          1 & s &  s^2 \\
        \end{absmatrix}
      }
      = \frac{s^3+t^3+1-2-ts^2-t^2}{ts^2+t^2+s-t-st^2-s^2}.
    \end{equation*}
  \end{sol}
\end{ex}


\section{Convergence}
We have described what it means for a Taylor series to converge to a function over some interval.
What if we're given an infinite series that's not a function, just an infinite sum of numbers?
Can we still check if the series converges or diverges?
We'll develop several tests that we can apply to check for convergence or divergence.

\subsection{nth Term Test for Divergence}
\begin{lemma}
	Let $a_n$ be the nth term of a series $s$.
	If
	\begin{equation*}
		\lim_{n\to\infty}{a_n} \neq 0,
	\end{equation*}
	then the series diverges.
\end{lemma}
\begin{proof}
	Assume not.
	Let $a_n$ be the nth term and $s_n$ the nth partial sum of a convergent series $s$ whose terms do not tend to 0.
	Since $s$ converges to some value $L$, there exists some positive integer $m$ such that for all  $n > m$ and $\epsilon > 0$,
	\begin{equation*}
		\abs{s_n - L} < \epsilon.
	\end{equation*}
	We can also say the same for $s_{n+1}$.
	\begin{equation*}
		\abs{s_{n+1}-L} < \epsilon.
	\end{equation*}
	So, subtracting one inequality from the other,
	\begin{equation*}
		\abs{s_{n+1}-s_n} < 2\epsilon.
	\end{equation*}
	The difference between the two partial sums is just $a_{n+1}$.
	So,
	\begin{equation*}
		\abs{a_{n+1}} < 2\epsilon.
	\end{equation*}
	Since the terms of $s$ don't tend to 0, there exists some positive integer $h$ and real value $\delta > 0$ such that for all $n > h$,
	\begin{equation*}
		\abs{a_n} > \delta.
	\end{equation*}
	We can also say the same for $a_{n+1}$.
	\begin{equation*}
		\abs{a_{n+1}} > \delta.
	\end{equation*}
	Combining the two inequalities involving $a_{n+1}$, for all $n > \max{(m,h)}$,
	\begin{equation*}
		\delta < \abs{a_{n+1}} < 2\epsilon.
	\end{equation*}
	However, for $\epsilon \leq \delta/2$, the inequality creates a contradiction.
\end{proof}

Although the proof has to be a bit specific to cover the case of alternating series, the idea behind the test makes sense.
If you're adding on terms that don't get smaller in absolute value, then you can't ``zero in'' on a particular value and converge.

\begin{example}
	Show that the following series diverges:
	\begin{equation*}
		\sum_{k=0}^{\infty}{(-1)^k} = 1 - 1 + 1 - 1 + \ldots.
	\end{equation*}
\end{example}
\begin{answer}
	We see that
	\begin{equation*}
		\lim_{n\to\infty}{a_n} = \text{DNE} \neq 0.
	\end{equation*}

	So, by the nth Term Test for Divergence, the series diverges.
\end{answer}

\subsection{Geometric Series with $\abs{r} < 1$}
\begin{lemma}
	If $s$ is a geometric series with common ratio $r$, then $s$ converges if and only if $\abs{r} < 1$.
\end{lemma}

We already proved this when talking about power series and gave a formula to find its value.

\begin{example}
	Show that the following series diverges:
	\begin{equation*}
		\sum_{k=0}^{\infty}{\frac{2^k}{3}}.
	\end{equation*}
\end{example}
\begin{answer}
	This is a geometric series with initial term $1/3$ and common ratio $r=2$.
	Since $\abs{r} > 1$, the series diverges.
\end{answer}

\subsection{P-Series Test}
\begin{lemma}
	The p-series
	\begin{equation*}
		\sum_{k=1}^{\infty}{\frac{1}{k^p}}
	\end{equation*}
	converges if and only if $p > 1$.
\end{lemma}

\begin{example}
	Show that the Harmonic Series diverges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{\frac{1}{k}}.
	\end{equation*}
\end{example}
\begin{answer}

	This is a p-series with $p=1$.
	So, by the P-Series Test, the sum diverges.
\end{answer}

\subsection{Direct Comparison Test}
\begin{lemma}
	Let $s = \sum_{k=0}^{\infty}{a_k}$ be a series where all $a_k \geq 0$.
	$s$ converges if there exists some convergent series $c = \sum_{k=0}^{\infty}{c_k}$ and positive integer $m$ such that for all $n > m$,
	\begin{equation*}
		c_n \geq a_n.
	\end{equation*}
	$s$ diverges if there exists some divergent series $d = \sum_{k=0}^{\infty}{d_k}$ and positive integer $m$ such that for all $n > m$,
	\begin{equation*}
		a_n \geq d_n.
	\end{equation*}
\end{lemma}

\begin{example}
	Show that the following series converges:
	\begin{equation*}
		\sum_{k=0}^{\infty}{\frac{1}{2 + 3^k}}.
	\end{equation*}
\end{example}
\begin{answer}
We see that this series looks really similar to a geometric series with common ratio 1/3, just with an extra 2 in the denominator.
For all $n > 0$,
\begin{equation*}
	\frac{1}{2+3^n} \leq \frac{1}{3^n}.
\end{equation*}

So, by the Direct Comparison Test, the series converges.
\end{answer}

\begin{example}
	Show that the following series diverges:
	\begin{equation*}
		\sum_{k=0}^{\infty}{\frac{1}{2+\sqrt{k}}}.
	\end{equation*}
\end{example}
\begin{answer}
	Although the sum looks like a p-series with $p=1/2$, we can use that series because our series has smaller terms.
	Instead, we can compare to a p-series where $p=1/2$.
	\begin{align*}
		0 \leq \frac{1}{n} &\geq \frac{1}{2+\sqrt{n}} \\
		0 \leq 2 + \sqrt{n} &\leq n \\
		n & \geq 4.
	\end{align*}

	So, for all $n \geq 4$, our series has larger values than the p-series with $p=1$.
	We know the p-series diverges by the P-Test, so by the Direct Comparison Test, our series also diverges.
\end{answer}

\subsection{Limit Comparison Test}
\begin{lemma}
	Let $a_n$ and $b_n$ be the nth terms of two series $a$ and $b$ that have all positive terms after some point.
	If
	\begin{equation*}
		\lim_{n\to\infty}{\frac{a_n}{b_n}}
	\end{equation*}
	converges to a finite value greater than 0, then $a$ and $b$ either both converge or both diverge.
	If the limit converges to 0 and $b$ converges, then $a$ also converges.
	If the limit goes to $\infty$ and $b$ diverges, then $a$ also diverges.
\end{lemma}

A common tactic is to select one of $a$ or $b$ to be a geometric or p-series.

\begin{example}
	Show that the following series diverges:
	\begin{equation*}
		\sum_{k=2}^{\infty}{\frac{2k}{k^2-k+a}}.
	\end{equation*}
\end{example}
\begin{answer}
	Looking at this rational function, we see a degree 1 term in the numerator and a degree 2 term in the denominator.
	So, we might expect that the series behaves similarly to $1/n$.
	\begin{equation*}
		\lim_{n\to\infty}{\frac{\frac{2n}{n^2-n+a}}{\frac{1}{n}}} = \lim_{n\to\infty}{\frac{2n^2}{n^2-n+a}} = 2.
	\end{equation*}

	Since the limit converges to a finite value greater than 0, and we know that $1/n$ diverges by the P-Test, then the series must also diverge by the Limit Comparison Test.
\end{answer}

\begin{example}
	Show that the following series converges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{\frac{1}{2^k - 1}}.
	\end{equation*}
\end{example}
\begin{answer}
	This series looks similar to the geometric series $1/2^n$.
	\begin{equation*}
		\lim_{n\to\infty}{\frac{\frac{1}{2^n}}{\frac{1}{2^n-1}}} = \lim_{n\to\infty}{\frac{2^n - 1}{2^n}} = 1.
	\end{equation*}

	Since the limit converges to a finite value greater than 0, and we know that $1/2^n$ converges because it's a geometric series with $r=1/2$, then the series must also converge the the Limit Comparison Test.
\end{answer}

\subsection{Integral Test}
\begin{lemma}
	Let $a$ be a sequence of positive terms where $a_n = f(n)$.
	If $f$ is continuous, positive after some $m$, and decreasing, then the series
	\begin{equation*}
		\sum_{k=m}^{\infty}{a_k}
	\end{equation*}
	converges if and only if the integral
	\begin{equation*}
		\int_{m}^{\infty}{f(x)\d{x}}
	\end{equation*}
	converges.
\end{lemma}

\begin{example}
	Show that the following series diverges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{\frac{2k}{k^2+1}}.
	\end{equation*}
\end{example}
\begin{answer}
	The function $f(x)=2x/(x^2+1)$ is continuous, positive for all $x \geq 1$, and decreasing.
	\begin{equation*}
		\int_{1}^{\infty}{\frac{2x}{x^2+1}\d{x}} = (u = x^2+1) \int_{2}^{\infty}{\frac{\d{u}}{u}} = \ln{u}\biggr\rvert_{2}^{\infty} = \text{diverges}.
	\end{equation*}

	So, by the Integral Test, the series also diverges.
\end{answer}

\begin{example}
	Show that the following series converges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{\frac{1}{k^2+1}}.
	\end{equation*}
\end{example}
\begin{answer}
	The function $f(x)=1/(x^2+1)$ is continuous, positive for all $x \geq 1$, and decreasing.
	\begin{equation*}
		\int_{1}^{\infty}{\frac{\d{x}}{x^2+1}} = \arctan{x}\biggr\rvert_{1}^{\infty} = \frac{\pi}{2} - \frac{\pi}{4} = \frac{\pi}{4}.
	\end{equation*}

	So, by the Integral Test\footnote{We omitted evaluating the improper integral with limits, but the value is what you would get from doing that.} the series also converges.
\end{answer}

\subsection{Ratio Test}
\begin{lemma}
	Let $a$ be a series with only positive terms after some index.
	If the limit
	\begin{equation*}
		\lim_{n\to\infty}{\frac{a_{n+1}}{a_n}}
	\end{equation*}
	is less than 1, then the series converges.
	If the limit is greater than 1, then the series diverges.
	If the limit is equal to 1, then the test is inconclusive.
\end{lemma}

\begin{example}
	State whether the following series converges or diverges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{\frac{k\ln{k}}{2^k}}.
	\end{equation*}
\end{example}
\begin{answer}
	Taking the limit of the ratio of subsequent terms,
	\begin{equation*}
		\lim_{n\to\infty}{\frac{\frac{(n+1)\ln{(n+1)}}{2^{n+1}}}{\frac{n\ln{n}}{2^n}}} = \lim_{n\to\infty}{\frac{(n+1)\ln{n+1}}{2n\ln{n}}} = \lim_{n\to\infty}{\frac{n+1}{2n}} = \frac{1}{2}.
	\end{equation*}

	Since the limit is less than 1, the series converges by the Ratio Test.
\end{answer}

\subsection{nth Root Test}
\begin{lemma}
	Let $a$ be a series with all positive terms after some index.
	If the limit
	\begin{equation*}
		\lim_{n\to\infty}{\sqrt[n]{a_n}}
	\end{equation*}
	is less than 1, then the series converges.
	If the limit is greater than 1, the series diverges.
	If the limit is equal to 1, the test is inconclusive.
\end{lemma}

This test is most useful for series that looks like geometric series, but the common ratio is not a constant.

\begin{example}
	State whether the following series converges or diverges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{\left(\frac{k}{2k-1}\right)^k}.
	\end{equation*}
\end{example}
\begin{answer}
	Taking the limit of the nth root,
	\begin{equation*}
		\lim_{n\to\infty}{\sqrt[n]{\left(\frac{n}{2n-1}\right)^n}} = \lim_{n\to\infty}{\frac{n}{2n-1}} = \frac{1}{2}.
	\end{equation*}

	Since the limit is less than 1, the series converges by the nth Root Test.
\end{answer}

\subsection{Alternating Series Test}
All of the previous 8 tests have tested whether a series converges absolutely.
That is, all terms in the series could be made positive and the series would still converge or diverge.
There are some series that do converge but don't converge absolutely.
We say that these series converge conditionally.
You've already seen  few, like the formula for $\pi/4$ using the Maclaurin series for $\arctan{x}$.
All series that converge absolutely also converge conditionally.
Series that converge conditionally do so when they pass the following test.

\begin{lemma}
	The series
	\begin{equation*}
		\sum_{k=0}^{\infty}{(-1)^ku_k}
	\end{equation*}
	converges if all of the following conditions are satisfied.
	\begin{enumerate}
		\item All $u_i$ are positive.
		\item There exists an integer $m$ such that for all $n > m$, $u_{n+1} \leq u_n$.
		\item The series of $u$'s pass the nth Term Test.
	\end{enumerate}
\end{lemma}

\begin{example}
	State whether the following series converges or diverges:
	\begin{equation*}
		\sum_{k=1}^{\infty}{(-1)^k\frac{1}{k}}.
	\end{equation*}
\end{example}
\begin{answer}
	\begin{enumerate}
		\item All the terms $1, 1/2, 1/3, \ldots$ are positive.
		\item All terms are less than the previous term.
		\item The terms tend to 0, passing the nth Term Test.
	\end{enumerate}

	So, by the Alternating Series Test, the series converges conditionally\footnote{You've already seen this series too. It converges to $\ln{2}$.}.
	Note that the series does not converge absolutely because it fails the P-Test.
\end{answer}

\subsubsection{Riemann Rearrangement Theorem}
\begin{theorem}[Riemann Rearrangement Theorem]
	If an infinite series is conditionally convergent, but not absolutely convergent, then its terms can be rearranged to form a divergent series or to converge to any constant.
\end{theorem}

This result may seem counter intuitive, given that we know addition is associative and commutative.
However, the result stems from the fact that we defined convergence of an infinite series at the limit of partial sums.
If we rearrange terms of the series, the partial sums and their limits can change, meaning the limit of partial sums can diverge or converge to some other value.


\begin{verbatim}
\subsection{Sine functions}
For given $x \in [0, 2\pi]$ with step size $\pi/12$, we can obtain the evaluations
of \eqref{eq:sine} at $x$ (see Table \ref{tab:sine}), and the corresponding plot
(see Figure \ref{fig:sine}).

\begin{equation}
  \label{eq:sine}
  \begin{cases}
    y_1 = \sin(x/2) \\
    y_2 = \sin(x)   \\
    y_3 = \sin(2x)
  \end{cases}
\end{equation}
\begin{table}[!hbtp]
\centering
\caption{Sine functions}
\label{tab:sine}
\begin{tabular}{ccrr}
\toprule
        $x$ & $\sin(x/2)$ &   $\sin(x)$ &  $\sin(2x)$ \\
\midrule
$0$      & $0$          & $0$  & $0$ \\
$\pi/2$  & $\sqrt{2}/2$ & $1$  & $0$ \\
$\pi$    & $1$          & $0$  & $0$ \\
$3\pi/2$ & $\sqrt{2}/2$ & $-1$ & $0$ \\
$2\pi$   & $0$          & $0$  & $0$ \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[!hbtp]
  \centering
  \includegraphics[width=0.3\textheight]{./fig/sine.pdf}
  \caption{Sine functions}
  \label{fig:sine}
\end{figure}

\subsection{Goldbach's Conjecture}
Pursuing this type of analysis more carefully, Hardy and Littlewood in 1923 conjectured
(as part of their famous \textsl{Hardy–Littlewood prime tuple conjecture}) that for any
fixed $c \geq 2$, the number of representations of a large integer $n$ as the sum of $c$
primes $n = p_1 + \cdots + p_{c}$ with $p_1 \leq \cdots \leq p_c$ should be asymptotically
equal to
\begin{equation}
    \label{eq:hardy}
    \left( \prod_{p} \frac{p \gamma_{c,p} (n)}{(p - 1)^c}\right)
    \int_{2 \leq x_1 \leq \cdots \leq x_c: x_1 + \cdots + x_c = n}
    \frac{d x_1 \cdots d x_{c - 1}}{\ln{x_1} \cdots \ln{x_c}},
\end{equation}
where the product is over all primes $p$, and $\gamma_{c, p}(n)$ is the number of
solutions to the equation $n = q_1 + \cdots + q_c \mod p$ in modular arithmetic,
subject to the constraints $q_1, \ldots, q_c \ne 0 \mod p$. This formula
\eqref{eq:hardy} has been rigorously proven to be asymptotically valid for
$c \geq 3$ from the work of Vinogradov, but is till only a conjecture when $c = 2$.
In the latter case, the above formula simplifies to $0$ when $n$ is odd, and to
$$
2 \Pi_2 \left( \prod_{p|n; p \geq 3} \frac{p - 1}{p - 2} \right)
\int_{2}^{n} \frac{dx}{(\ln{x})^2} \approx 2 \Pi_2
\left( \prod_{p|n; p \geq 3} \frac{p - 1}{p - 2} \right) \frac{n}{(\ln{n})^2},
$$
when $n$ is even, where $\Pi_2$ is Hardy-Littlewood's twin prime constant
$$
\Pi_2 := \prod_{p \geq 3} \left( 1 - \frac{1}{(p - 1)^2} \right) = 0.6601618158\ldots
$$
This sometimes known as the \textsf{extended Goldbach conjecture}.

\emph{Reference}: \href{https://en.wikipedia.org/wiki/Goldbach's_conjecture}{Goldbach's conjecture}.
\end{verbatim}


\begin{thm}[Abstract Bayes' Rule]
	Let $\Pf$ and $\Pm$ be probability measures on the probability space $(\Omega, \F)$ with $\Pm \ll \Pf$ and Radon-Nikod\'{y}m derivate
	\begin{align}
	L = \frac{ \dx \Pm }{ \dx \Pf } .
	\end{align}
	If $\G$ is a sub-$\sigma$ algebra of $\F$ and $X$ is integrable $\G$-measurable random variable, then
	\begin{align}
	\E_{\Pm} (X| \G) = \frac{\E_{\Pf} (LX| \G)}{\E_{\Pf} (L| \G)} .
	\end{align}
\end{thm}

\begin{proof}
	First we note that
	\begin{align}
	\Pm ( \E_{\Pf} (L| \G) = 0 ) &= \int \1_{ \{ \E_{\Pf} (L| \G) = 0 \} } \dx \Pm \\
	&= \int_{ \{ \E_{\Pf} (L| \G) = 0 \} } L \dx \Pf \\
	&= \int_{ \{ \E_{\Pf} (L| \G) = 0 \} } \E_{\Pf} (L| \G) \dx \Pf \\
	&= 0
	\end{align}
	which implies that $\E_{\Pf} (L| \G) > 0$. Now we have to prove only that
	\begin{align}
	\E_{\Pm} (X| \G) \E_{\Pf} (L| \G) = \E_{\Pf} (LX| \G) .
	\end{align}
	If $G \in \G$, then $\E_{\Pm} (X| \G)$ is $\G$-measurable and thus
	\begin{align}
	\int_G \E_{\Pm} (X| \G) \E_{\Pf} (L| \G) \dx \Pf &= \int_G \E_{\Pf} (L \E_{\Pm} (X| \G) | \G) \dx \Pf  \\
	&= \int_G L \E_{\Pm} (X| \G)  \dx \Pf  \\
	&= \int_G \E_{\Pm} (X| \G)  \dx \Pm \\
	&= \int_G X \dx \Pm \\
	&= \int_G L X \dx \Pf \\
	&= \int_G \E_{\Pf} (LX | \G ) \dx \Pf .
	\end{align}
	and concludes the proof.
\end{proof}

If filtration $(\F_t)$ consists of sub-$\sigma$-algebras of $\F$, then, by Radon-Nikod\'{y}m-theorem, we have Radon-Nikod\'{y}m derivates
\begin{align}
L_t = \frac{\dx \Pm}{\dx \Pf}
\end{align}
with respect to $\sigma$-algebra $\F_t$ for all $t \geq 0$. By equation \ref{Radon-Nikodymderivateconditional},
\begin{align}
L_t = \E_{\Pf} ( L_s | \F_t )
\end{align}
for $0 \leq t < s$ implying that $(L_t)$ is a martingale with respect to the filtration $(\F_t)$. The process $(L_t)$ is called likelihood-process.

Abstract Bayes' rule and the fact $\E_{\Pf} (L_t) = 1$ implies the following theorem.

\begin{thm}
	Let $\Pf$ and $\Pm$ be probability measures with $\Pm \ll \Pf$ and Radon-Nikod\'{y}m derivate
	\begin{align}
	L = \frac{ \dx \Pm }{ \dx \Pf } .
	\end{align}
	Now the process $(X_t)$ is $\Pm$-martingale if and only if $LX_t$ is $\Pf$-martingale, where $L$ is the likelihood-process.
\end{thm}

\subsection{A consequence of Radon-Nikod\'{y}m-theorem}

If $\G$ is a sub-$\sigma$-algebra of $\F$, then Radon-Nikod\'{y}m-derivates $L_{\G}$ and $L_{\F}$ of $\Pf$ with respect to $\Pm$ need not be equal as $L_{\F}$ may not be $\G$-measurable. However, we know that for every $G \in \G$
\begin{align}
\int_G \E_{\Pf} ( L_{\F} | \G ) \dx \Pf = \int_G L_{\F} \dx \Pf = \Pm (G)
\end{align}
meaning that
\begin{align}
L_{\G} = \E_{\Pf} ( L_{\F} | \G ) .
\end{align}

Thus we have proved the following.

\begin{lemma}
	If $\Pm \ll \Pf$ and $\G$ is a sub-$\sigma$-algebra of $\F$, then
	\begin{align}
	\label{Radon-Nikodymderivateconditional}
	L_{\G} = \E_{\Pf} ( L_{\F} | \G ) ,
	\end{align}
	where $L_{\G}$ is Radon-Nikod\'{y}m-derivate of $\Pf$ with respect to $\Pm$ and filtration $\G$ and $L_{\F}$ is Radon-Nikod\'{y}m-derivate of $\Pf$ with respect to $\Pm$ and filtration $\F$.
\end{lemma}

Let $\Pm$ and $\Pf$ be equivalent measures with a filtration $(\F_t)$. W define a density process by
\begin{align}
\Lambda_t = \E_{\Pf} \left( \frac{\dx \Pm}{\dx \Pf} \ | \ \F_t \right)
\end{align}
Now
\begin{align}
\E_{\Pf} \left( \Lambda_T \ | \ \F_t \right) &= \E_{\Pf} \left( \E_{\Pf} \left( \frac{\dx \Pm}{\dx \Pf} \ | \ \F_T \right) \ | \ \F_t \right) \\
&= \E_{\Pf} \left( \frac{\dx \Pm}{\dx \Pf} \ | \ \F_t \right) \\
&= \Lambda_t
\end{align}
for all $0 \leq t \leq T$ meaning that $\Lambda_t$ is a martingale. Also
\begin{align}
\Lambda_0 = \E_{\Pf} \left( \frac{\dx \Pm}{\dx \Pf} \ | \ \F_0 \right)
\end{align}


\section*{Problem 2 Solution}

\begin{enumerate}[a)]

\item

The transmitted uncollided intensity of a neutron beam through a material with macroscopic cross section $\Sigma$ is given by
$$ I = I(0) \, e^{-\Sigma x} .$$
Solving for $\Sigma$, we find
$$ \Sigma = \frac{1}{x}\ln\left(\frac{I(0)}{I}\right) $$
Using the values provided (and considering the intensity as per the target, rather than per cm$^2$), we have
$$ \Sigma = \frac{1}{10\text{ cm}}\ln\left(\frac{2\times10^{12}\text{ neutrons/(cm}^2{\cdot}\text{s)}\left( 5\text{ cm}^2 \right)}{3.0\times10^9\text{ neutrons/s}}\right) $$
$$ \Sigma = \frac{1}{10\text{ cm}}\ln\left(3.33\times10^{3}\right) $$
$$\boxed{ \Sigma = 0.811\text{ cm}^{-1} }$$

\item

The mean free path of a particle is defined as $\lambda \equiv \frac{1}{\Sigma}$.
$$\boxed{ \lambda = 1.233\text{ cm} }$$

\item

We are told that the beam pulse is 10 $\mu$s, so when considered in conjunction with the known intensity of the neutron beam, we can find the total number of particles produced by the beam per area. We define the neutron fluence, $\Phi$, as the number of neutrons per cm$^2$, calculated as
$$ \Phi = It = (2\times10^{12}\text{ neutrons/(cm}^2\cdot\text{s)})(10^{-5}\text{ s}) = 2\times10^7\text{ neutrons/cm}^2 $$
The number of interactions, $R$, is
$$ R = \Phi \Sigma V $$
where $\Phi$ is the incident neutron fluence in $\left[\frac{\text{neutrons}}{\text{cm}^2}\right]$, $\Sigma$ is the macroscopic cross section of the shielding material in $\left[\frac{1}{\text{cm}}\right]$, and $V$ is the volume of the shield in $\left[{\text{cm}^3}\right]$.

We have already found $\Phi$, we calculated $\Sigma$ in part (a), and we can determine $V$ by multiplying the area of the beam spot by the thickness of the target.
$$ V = 5\text{ cm}^2 \times 10\text{ cm} = 50\text{ cm}^3 $$
The total number of collisions is then
$$ R = (2\times10^7\text{ cm}^{-2})(0.811\text{ cm}^{-1})(50\text{ cm}^3) $$
$$\boxed{ R = 8.11\times10^8\text{ collisions} }$$
\end{enumerate}


In this section we will study the limitations of our previous encoding of natural numbers. We will then use a positional system (the simplest of which is binary numbers) to show a much more efficient conversion process.

\subsection{Performance considerations of unary encoding}
Let us begin with an analysis of how long it takes to perform addition of two Peano numbers. Let us assume that the cost of building or decomposing terms is $O(1)$, since it can be implemented with a constant number of low-level operations such as memory allocation, runtime type checks, and similar, but it does not require recursion or iteration. Consider addition between two numbers $n + m$ with the rules:

\begin{lstlisting}
----------- r0
z + a => a

a + b => c
----------------- r1
s(a) + b => s(c)
\end{lstlisting}

We have two alternatives that clearly only depend from $n$ ($m$ does not have influx on the number of operations, since it is just returned directly as the last step): either $n=0$, or $n=n'+1$. In case of $n=0$, then rule \texttt{r0} is used, producing a result directly. This means that the number of steps $T$ performed by the system is:

$$T(n) = 1 \text{ when } n=0$$

In the case of $n=n'+1$, then rule \texttt{r1} is used. Evaluating rule \texttt{r1} requires first evaluating the expression $n'+m$\footnote{Note that this again an application of \textit{induction}, albeit less formal}, and then performing a single operation which is the addition of the \texttt{s} keyword to intermediate result \texttt{c}:

$$T(n) = T(n') + 1 = T(n-1) + 1 \text{ when } n=n'+1$$

This means that the system will perform exactly $n+1$ computation steps, therefore having a complexity of $O(n)$ where $n$ is the value of the first number. This is highly inefficient.

\subsection{Binary encoding}
Let us consider positional encoding. Specifically, let us consider binary encoding (base 2 encoding). We will define a binary number as a series of binary digits, which are either zero or one:

\begin{lstlisting}
Data [] "d0" [] Priority 0 Type Digit
Data [] "d1" [] Priority 0 Type Digit
\end{lstlisting}

The binary number itself is defined as a digit, followed by the rest of the number. We will use the comma operator to specify this separation:

\begin{lstlisting}
Data [Num] "," [Digit] Priority 10 Type Num
\end{lstlisting}

Of course numbers need to end at some point\footnote{not in real life, where we could simply assume an infinite sequence of zero's, but in a computer with limited memory this simplification is ubiquitous}, so we define a special symbol that represents the ``end of list'' number:

\begin{lstlisting}
Data [] "nil" [] Priority 0 Type Num
\end{lstlisting}

A number such as $3 = 011$ would then be expressed as \texttt{(((nil,d0),d1),d1)}. Notice that we associate the digits to the left, so that the first digit we encounter is the one of the lowest order. This is not strictly necessary, but it makes it easier to extract the digits in order of significance, and this in turns simplifies the traditional definition of addition.

\subsection{Addition}
To add two numbers, we will encode the well-known process of adding numbers with a \textit{carry}. This means that we will add, repeatedly, three digits at a time: the first two digits of the numbers that we are adding, plus the carry that comes from the previous addition of digits. We now define this sum of digits:

\begin{lstlisting}
Func [] "addDigits" [Digit Digit Digit] Priority 5 Type Expr => Num
\end{lstlisting}

We have eight instances of \texttt{addDigits}, one for each possible combination of digits (each digit has two possible values, \texttt{d0} and \texttt{d1}, thus we have $2^3$ total combinations). For each such combination, we directly return the sum of the three digits in the format of a two-digit binary number. We need no more than two digits for the result, since the maximum number we will need to return is the sum of \texttt{d1} three times:

\begin{lstlisting}
----------------------------------
addDigits d0 d0 d0 => (nil,d0),d0

----------------------------------
addDigits d0 d0 d1 => (nil,d0),d1

...

----------------------------------
addDigits d1 d1 d1 => (nil,d1),d1
\end{lstlisting}

Addition with carry performs the digit addition of the current digits of the numbers being added, plus the current carry. The two resulting digits are the lower-order unit of the result, together with the carry that will be used to add together the remaining bits of the input numbers. The function that recursively adds the digits is \texttt{addCarry}:

\begin{lstlisting}
Func [] "addCarry" [Num Num Digit] Priority 5 Type Expr => Num
\end{lstlisting}

\begin{lstlisting}
addDigits da db dr => (nil,dr',d)
addCarry a b dr' => res
----------------------------------
addCarry a,da b,db dr => res,d
\end{lstlisting}

Of course addition ends when it encounters the end of the numbers, which needs to be accompanied with a null carry:

\begin{lstlisting}
--------------------------
addCarry nil nil d0 => nil
\end{lstlisting}

We might even add an \texttt{overflow} keyword and graciously handle the case of non-null last carry as follows:

\begin{lstlisting}
--------------------------------
addCarry nil nil d1 => overflow
\end{lstlisting}

The first step of addition simply instances \texttt{addCarry} with a null initial carry:

\begin{lstlisting}
addCarry a b d0 => c
--------------------
a + b => c
\end{lstlisting}

Multiplication follows a similar scheme, and as such we will not give it here. Also, notice that we only support numbers of the same length. Supporting addition of numbers of different lengths would be relatively trivial, and as such is also not shown here.

\subsection{Performance considerations of binary encoding}
Binary encoding allows us to perform operations much faster, that is with less rules used when compared with Peano addition. Remember that Peano addition took a significant $O(n)$ amount of operations to perform, with $n$ being the value of the first operand.

Consider only the rule which performs a single step of the binary addition:

\begin{lstlisting}
addDigits da db dr => (nil,dr',d)
addCarry a b dr' => res
----------------------------------
addCarry a,da b,db dr => res,d
\end{lstlisting}

\texttt{addDigits} is just one single step, in that it directly resolves the parameters and returns the appropriate result. The only thing that matters for the purpose of the complexity of \texttt{addCarry} is the recursive call. Let us observe that the recursive call \texttt{addCarry a b dr'} uses \texttt{a} and \texttt{b} as input numbers, whereas the original addition \texttt{addCarry a,da b,db dr} used \texttt{a,da} and \texttt{b,db}, which are precisely equal to \texttt{a} and \texttt{b}, but with the additional digits \texttt{da} and \texttt{db} respectively. What is the relationship between \texttt{a} and \texttt{a,da}? Simply enough, we know that \texttt{a} is \texttt{a,da} divided by two. In general, removing the least-significant digit in a number encoded in base $b$ is equivalent to dividing the number by $b$ itself. Consider the following examples of removal of a least-significant digit:

$\begin{array}{c c c}
  110 / 2 & = & 11 \\
  111 / 2 & = & 11 \\
  1000 / 2 & = & 100
\end{array}$

This happens because a number $n$ in an arbitrary base $b$ is decomposed into:

$$d_m \times b^m \dots + d_2 \times b^2 + d_1 \times b^1 + d_0 \times b^0$$

Removing one digit decreases all powers of $b$, resulting in:

$$d_m \times b^{m-1} \dots + d_2 \times b^{2-1} + d_1 \times b^{1-1}$$

which is precisely equal to the original number divided by $b$.

When there are no more digits, it means that the number has been divided by the base ($2$ for our binary numbers) enough times to render the highest coefficient null. If the original number was $n$, we will have stopped only after $k$ steps such that $\frac{n}{2^k} = 0$. The smallest solution to this is actually well known, and is $k = \log_2 n$. This leads us to the conclusion that the complexity of this method of addition is $O(\log n)$, which is significantly faster than $O(n)$. As a way of comparison, assume that we are computing the sum $1000000 + 1000000$: the Peano addition will take about $1000000$ steps, whereas binary addition will take about $6$.


\chapter{Data}

\begin{description}
  \item[Data] is collection of data object and their attributes.
  \item[Type of Attributes] Nominal, Ordinal, Interval, Ratio.
  \item[Attribute Values Property] Distinctness, Order, Addition, Multiplication
\end{description}
\par \noindent
\marginnote{I do not focus on Types of Data Sets and Characteristics of Structured Data}One idea to determine if an interval attribute is ratio is to check if the attribute has a true or natural absolute zero point. The ratio scales are very common in physical scenario.

\section{Discrete and Continuous}

\begin{description}
  \item[Discrete Attribute] Has only a finite or countably infinite set of values
  \item[Continuous Attribute] Has real numbers as attribute values
\end{description}

\section{Data Quality}

\begin{description}
  \item[Noise] modification of original values
  \item[Outlier] considerably different than most of the other data objects
\end{description}

\section{Discretization and Binarization}

\begin{description}
  \item[Binarization] transform either a continuous attribute or a categorical attribute into one or more binary attributes
  \item[Discretisation] transform a continuous attribute into a categorical attribute
\end{description}

\section{Normalization}

\underline{Min-Max Normalization}
$$(min_A, max_A) \rightarrow (new\_min_A, new\_max_A)$$
$${v^{'}={\frac{v-min_A}{max_A-min_A}} \times (new\_max_A-new\_min_A) + new\_min_A}$$
\\
\underline{Z-score Normalization}
$$v^{'}=\frac{v-\mu_A}{\sigma_A}$$
\\
\underline{Normalisation by Decimal Scaling}
$$v^{'}=\frac{v}{10^{j}}$$
where j = smallest integer such that $Max(\mid v^{'}\mid) < 1$

\section{Proximity}

\begin{description}
\item[Similarity] Numerical measure of how alike two data objects are
\item[Dissimilarity] Numerical measure of how different two data objects are
\end{description}
\underline{Euclidean Distance}
$$dist=\sqrt{\sum_{k=1}^{n}{(p_k-q_k)^2}}$$
\\
\underline{Minkowski Distance}
$$dist={\sum_{k=1}^{n}{(\mid p_k-q_k\mid^r)}}^{1/r}$$
\begin{align*}
r = 1&: Hamming\ distance / Manhattan\ distance\\
r = 2&: Euclidean\ distance \\
r = inf&: supremum\ /\ Chebyshev\ distance, max(x - y) \\
\end{align*}
\\
\underline{Simple Matching Coefficient}
$$SMC=\frac{M_{11} + M_{00}}{M_{00} + M_{01} + M_{10} + M_{11}}$$
\\
\underline{Jaccard Coefficient}
$$J=\frac{M_{11}}{M_{01} + M_{10} + M_{11}}$$
\\
\underline{Cosine Similarity}
$$cos(p,q)=\frac{p\bullet q}{\|p\|\|q\|}$$
\underline{Tanimoto Coefficient}
$$T(p,q)=\frac{p\bullet q}{\|p\|^{2}+\|q\|^{2}-p\bullet q}$$

\section{Correlation}

$$corr(p,q)=\frac{cov(p,q)}{std(p) \times std(q)}$$
$$cov(p,q)=\frac{1}{n-1}\sum_{k=1}^{n}{(p_k-\bar{p})(q_k-\bar{q})}$$
$$std(p)=\sqrt{\frac{1}{n-1}\sum_{k=1}^{n}{(p_k-\bar{p})^{2}}}$$


\subsection{Convex Functions}
Let us first give the definition of convex sets.
\begin{definition}[Convex set]
	A set $C$ is convex, if the line segment between any two points in $C$ lies in $C$, i.e., if any $x, y\in C$ and any $\alpha$ with $0\leq \alpha \leq 1$, there holds
	\begin{equation}
	\alpha x+ (1- \alpha)y \in C.
	\end{equation}
\end{definition}

%Based on the definition above, we have the next property of convex sets.
%\begin{lemma}
%[Homework] Let $C$ be a convex set, with $x_{1}, \dots, x_{k}\in C$, and let $\alpha_{1}, \dots, \alpha_{k} \in \mathbb{R}$ satisfy $\alpha_{i} \geq 0$ and  $\sum_{i =1}^{k} \alpha_{i} = 1$, then
%	\begin{equation}
%	\sum_{i = 1}^{k}\alpha_{i}x^{i} \in C,
%	\end{equation}
%\end{lemma}

Following the definition of convex set, we define convex function as following.
\begin{definition}[Convex function]
	Let $C \subset \mathbb{R}^n$ be a convex set and $f: C \rightarrow \mathbb{R}$:
	\begin{enumerate}
		\item $f$ is called {\bf convex} if for any $x, y \in C$ and $\alpha\in [0, 1]$
		\begin{equation}
		f(\alpha x + (1- \alpha)y) \leq \alpha f(x )+ (1- \alpha)f(y).
		\end{equation}
		\item $f$ is called {\bf strictly convex} if for any $x\neq y \in C$ and $\alpha\in (0, 1)$:
		\begin{equation}
		f(\alpha x + (1- \alpha)y) < \alpha f(x ) + (1 - \alpha)f(y).
		\end{equation}
		\item A function $f$ is said to be (strictly) {\bf concave} if $-f$ is (strictly) convex.
	\end{enumerate}
\end{definition}
%With mentioned previously property of convex sets, it is easy to expend the above definition of convex function. Here we write as one lemma without proof.
%\begin{lemma}
%[Homework]Let $C$ be a convex set, with $x_{1}, \dots, x_{k}\in C$, and let $\alpha_{1}, \dots, \alpha_{k} \in \mathbb{R}$ satisfy $\alpha_{i} \geq 0$ and  $\sum_{i =1}^{k} \alpha_{i} = 1$, then
%	\begin{equation}
%	f(\sum_{i = 1}^{k}\alpha_{i}x_{i}) \leq \sum_{i = 1}^{k}\alpha_{i}f(x_{i}) .
%	\end{equation}
%\end{lemma}
Fig. \ref{convexfunc} and \ref{nonconvexfunc} are diagrams  for convex function definition.
\begin{figure}[H]
\centering                    % 大图名称
\subfigure[Convex function]{                    %第二张子图
\begin{minipage}{4cm}\centering
\includegraphics[width=1\textwidth] {6DL/figures/convexfunction.png}\label{convexfunc}
\end{minipage}   }
 \quad
 \subfigure[Nonconvex function]{                    %第二张子图
\begin{minipage}{4cm}\centering
\includegraphics[width=1\textwidth,height=0.8\textwidth] {6DL/figures/nonconvexfunction.png}\label{nonconvexfunc}
\end{minipage}   }
\caption{Convex and nonconvex functions} %
\end{figure}

\begin{lemma}
	If $f(x)$ is differentiable on $\mathbb{R}^n$, then $f(x)$ is convex if and only if
	\begin{equation}
	f( x) \ge f( y) + \nabla f( y)\cdot ( x -  y), \forall x, y \in \mathbb{R}^n.
	\end{equation}
\end{lemma}
Based on the lemma, we can first have  Fig. \ref{fig:convextangent} for convex functions, namely, the curve of a convex function is always above the tangent lines.
\begin{figure}[H]
\centering
\includegraphics[width=.5\textwidth] {6DL/figures/convextangent.png}
\caption{Tangent line of a convex function}
\label{fig:convextangent}
\end{figure}

\begin{proof}
	Let $z=\alpha x+(1-\alpha )y, 0\leq \alpha \leq 1, \forall x,y \in \mathbb{R}^n$, we have these next two Taylor expansion:
	\begin{equation}\label{key}
	\begin{aligned}
	&f(x)\geq f(z)+ \nabla f(z)(x-z),\\
	&f(y)\geq f(z)+ \nabla f(z)(y-z).
	\end{aligned}
	\end{equation}
	Then we have
	\begin{equation}
	\begin{aligned}
	\alpha f(x) + (1-\alpha) f(y)
	\geq &f(z) + \nabla f(z)[\alpha(x-z)+(1-\alpha)(y-z)]\\
	=&f(z)
	=f(\alpha x +(1-\alpha)y).
	\end{aligned}
	\end{equation}
	Thus we have
	\begin{equation}\label{key}
	\alpha f(x) + (1-\alpha) f(y) \ge f(\alpha x +(1-\alpha)y).
	\end{equation}
	This finishes the proof.

	On the other hand if	$f(x)$ is differentiable on $\mathbb{R}^n$, then
	$$
	f( x) \ge f( y) + \nabla f( y)\cdot ( x -  y), \forall x, y \in \mathbb{R}^n
	$$
	if $f(x)$ is convex.
\end{proof}

\begin{definition}[$\lambda$-strongly convex]\label{def:strongconvex}
We say that $f(x)$ is $\lambda$-strongly convex if
	\begin{equation}\label{equ:lambdastronglyconvex}
	f(x)  \ge f(y) + \nabla f(y)\cdot(x-y) + \frac{\lambda}{2}\|x-y\|^2, \quad \forall x, y \in C,
	\end{equation}
	for some $\lambda >0$.
\end{definition}
\begin{lemma}
Let $\sigma(\nabla^2f)$ be the spectrum of the operator $\nabla^2 f$. If
$$
\sigma(\nabla^2f)\subset [\lambda, L],
$$
then the condition \eqref{equ:lambdastronglyconvex} holds.
\end{lemma}

\begin{lemma}
If $f(x)$ is $\lambda$-strongly convex,
	\begin{equation}\label{strongConvIneq}
		\nabla f(x)\cdot(x - x^*) \ge f(x) - f(x^*) +\frac{\lambda}{2}\|x - x^*\|^2\ge \lambda \|x - x^*\|^2.
		\end{equation}
\end{lemma}

\begin{example}
	Consider $f(x) = \|x\|^2$, then we have
	\begin{equation}\label{key}
	\frac{\partial f}{\partial x_{i}}=2x_{i}, \nabla f = 2x \in R^n,\quad \nabla^2f=2I_{n\times n}\in\mathbb{R}^{n\times n}.
	\end{equation}
	So, we have $\lambda_{\rm min}(\nabla^2f)=2>0$ and
	\begin{equation}
	\begin{aligned}
	f(x)-f(y)-\nabla f(y)(x-y)
	=&\left \| x \right \|^2-\left \| y \right \|^2-2y(x-y)\\
	=&\left \| x \right \|^2-\left \| y \right \|^2-2xy+2\left \| y \right \|^2\\
	=&\left \| x \right \|^2-2xy+\left \| y \right \|^2\\
	=&\left \| x-y \right \|^2\\
	=&\frac{\lambda }{2}\left \| x-y \right \|^2 , \quad \lambda=2.
	\end{aligned}
	\end{equation}
	Thus, $f(x) = \|x\|^2$ is 2-strongly convex
\end{example}
\begin{lemma}
If $f(x)$ is $\lambda$-strongly convex and $\lambda(\nabla^2 f)\le L$, then
$$
\lambda\|x-y\|\le \|\nabla f(x) - \nabla f(y)\|\le L\|x-y\|.
$$
\end{lemma}
\begin{assumption}\label{ass:GD}
We make the following assumptions:
\begin{enumerate}
	\item $f(x)$ is $\lambda$-strongly convex for some $\lambda >0$ with the definition in Definition \ref{def:strongconvex}.
	\item $\nabla f$ is $L$-Lipschitz, i.e.,
	\begin{equation}\label{key}
	\|\nabla f(x) - \nabla f(y)\| \le L\|x - y\|, \forall x,y.
	\end{equation}
\end{enumerate}
\end{assumption}
\begin{example}
Actually, the loss function of the logistic regression model
	\begin{equation}\label{key}
	L(\theta) = -\log P(\theta),
	\end{equation}
	is a convex  function of $\theta$.
	Furthermore, the loss function of the regularized logistic regression model
	\begin{equation}\label{key}
	L_\lambda(\theta) = -\log P(\theta) + \lambda \|\theta\|_F^2, \lambda > 0
	\end{equation}
	  is a $\lambda'$-strongly convex function of $\theta$($\lambda'$ is related to $\lambda$).
\end{example}

We also have these following interesting properties of convex function.

\begin{proposition}[basic properties of convex function]
	\begin{enumerate}
		\item If $f(x)$, $g(x)$ are both convex, then $\alpha f(x) + \beta g(x)$ is also convex, if $\alpha, \beta \geq 0$.
		\item  Linear function is both convex and concave. Here, $f(x)$ is concave if and only if $-f(x)$ is convex.
		\item If $f(x)$ is a convex function on $\mathbb{R}^n$, then $g(y) = f(Ay+b)$ is a convex function on $\mathbb{R}^m$. Here $A \in \mathbb{R}^{m \times n}$ and $b\in \mathbb{R}^m$.
		\item  If $g(x)$ is a convex function on $\mathbb{R}^n$, and the function $f(u)$ is convex function on $\mathbb{R}$ and non-decreasing, then the composite function $f \circ g(x) = f(g(x))$ is convex.
	\end{enumerate}
\end{proposition}
The proof is straightforward and left to the interested readers.

\begin{theorem}[Jensen's inequality]
If $p_1, \cdots, p_n$ are positive numbers which sum to 1 and $f$ is a real continuous function which is convex, then
$$
f(\sum_{i=1}^n\alpha_ix_i)\le \sum_{i=1}^n \alpha_if(x_i).
$$
If $f$ is concave, then
$$
f(\sum_{i=1}^n\alpha_ix_i)\ge \sum_{i=1}^n \alpha_if(x_i).
$$
\end{theorem}


\section{The matrix of a linear transformation}

\begin{outcome}
  \begin{enumerate}
  \item Find the matrix of a linear transformation and determine the
    action on a vector in $\R^n$.
  \end{enumerate}
\end{outcome}

In the above examples, the action of the linear transformations was to
multiply by a matrix.  It turns out that this is always the case for
linear transformations.  If $T$ is \textbf{any} linear transformation
which maps $\R^{n}$ to $\R^{m}$, there is \textbf{always} an
$m\times n$-matrix $A$ with the property that
\begin{equation}
T(\vect{x}) = A\vect{x} \label{matrix-of-transf}
\end{equation}
for all $\vect{x} \in \R^{n}$.

\begin{theorem}{Matrix of a linear transformation}{matrix-linear-transformation}
Let $T:\R^{n}\to \R^{m}$ be a linear transformation. Then we can find a matrix $A$ such that $T(\vect{x}) = A\vect{x}$.
 In this case, we say that $T$ is {\em determined} or {\em induced}
by the matrix $A$.
\end{theorem}

Here is why. Suppose $T:\R^{n}\to \R^{m}$ is a linear transformation and you want to find
the matrix defined by this linear transformation as described in {\eqref{matrix-of-transf}}.
 Note that
\begin{equation*}
\vect{x} =\begin{mymatrix}{c}
x_{1} \\
x_{2} \\
\vdots \\
x_{n}
\end{mymatrix} = x_{1}\begin{mymatrix}{c}
1 \\
0 \\
\vdots \\
0
\end{mymatrix} + x_{2}\begin{mymatrix}{c}
0 \\
1 \\
\vdots \\
0
\end{mymatrix} +\ldots + x_{n}\begin{mymatrix}{c}
0 \\
0 \\
\vdots \\
1
\end{mymatrix} = \sum_{i=1}^{n}x_{i}\vect{e}_{i}
\end{equation*}
where $\vect{e}_{i}$ is the $i\th$ column of $I_n$, that is the $n \times
1$ vector which has zeros in every slot but the $i\th$ and a 1 in
this slot.

Then since $T$ is linear,
\begin{eqnarray*}
T(\vect{x})&=&\sum_{i=1}^{n}x_{i}T(\vect{e}_{i}) \\
&=&\begin{mymatrix}{ccc}
| &  & | \\
T(\vect{e}_{1}) & \cdots & T(\vect{e}_{n}) \\
| &  & |
\end{mymatrix} \begin{mymatrix}{c}
x_{1} \\
\vdots \\
x_{n}
\end{mymatrix} \\
&=& A\begin{mymatrix}{c}
x_{1} \\
\vdots \\
x_{n}
\end{mymatrix}
\end{eqnarray*}
Therefore,  the desired matrix is obtained from constructing the $i\th$
column as $T(\vect{e}_{i})$. We state this formally as the
following theorem.

\begin{theorem}{Matrix of a linear transformation}{matrix-of-linear-transformation}
Let $T: \R^{n} \to \R^{m}$ be a linear transformation. Then the matrix $A$ satisfying $T(\vect{x})=A\vect{x}$\index{linear transformation!matrix} is given by
\begin{equation*}
A=
\begin{mymatrix}{ccc}
| &  & | \\
T(\vect{e}_{1}) & \cdots & T(\vect{e}_{n}) \\
| &  & |
\end{mymatrix}
\end{equation*}
where $\vect{e}_{i}$ is the $i\th$ column of $I_n$, and then $T(\vect{e}_{i}
)$ is the $i\th$ column of $A$.
\end{theorem}

The following Corollary is an essential result.

\begin{corollary}{Matrix and linear transformation}{matrix-linear-transformation-equivalence}
A transformation $T$ is a linear transformation if and only if it is a matrix transformation.
\end{corollary}

Consider the following example.

\begin{example}{The matrix of a linear transformation}{matrix-of-linear-transformation}
Suppose $T$ is a linear transformation, $T:\R^{3}\rightarrow \R^{2}$ where
\begin{equation*}
T\begin{mymatrix}{r}
1 \\
0 \\
0
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
2
\end{mymatrix} ,\ T\begin{mymatrix}{r}
0 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{r}
9 \\
-3
\end{mymatrix} ,\ T\begin{mymatrix}{r}
0 \\
0 \\
1
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
1
\end{mymatrix}
\end{equation*}
Find the matrix $A$ of $T$ such that $T (\vect{x})=A\vect{x}$  for all $\vect{x}$.
\end{example}

\begin{solution} By Theorem~\ref{thm:matrix-of-linear-transformation} we construct $A$ as follows:
\begin{equation*}
A =
\begin{mymatrix}{ccc}
| &  & | \\
T(\vect{e}_{1}) & \cdots & T(\vect{e}_{n}) \\
| &  & |
\end{mymatrix}
\end{equation*}

In this case, $A$ will be a $2 \times 3$-matrix, so we need to find $T
(\vect{e}_1), T (\vect{e}_2)$, and $T (\vect{e}_3
)$. Luckily, we have been given these values so we can fill in
$A$ as needed, using these vectors as the columns of $A$.  Hence,
\begin{equation*}
A=\begin{mymatrix}{rrr}
1 & 9 & 1 \\
2 & -3 & 1
\end{mymatrix}
\end{equation*}
\end{solution}

In this example, we were given the resulting vectors of $T (\vect{e}_1),
T (\vect{e}_2)$, and $T (\vect{e}_3)$. Constructing the matrix $A$ was simple, as we
could simply use these vectors as the columns of $A$. The next example shows how to find $A$ when we are not given the $T (\vect{e}_i)$ so clearly.

\begin{example}{The matrix of linear transformation: inconveniently \\ defined}{2x2-inconvenient-matrix-of-linear-transformation}
Suppose $T$ is a linear transformation, $T:\R^{2}\rightarrow \R^{2}$ and
\begin{equation*}
T\begin{mymatrix}{r}
1 \\
1
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
2
\end{mymatrix} ,\ T\begin{mymatrix}{r}
0 \\
-1
\end{mymatrix} =\begin{mymatrix}{r}
3 \\
2
\end{mymatrix}
\end{equation*}
Find the matrix $A$ of $T$ such that $T (\vect{x})=A\vect{x}$  for all $\vect{x}$.
\end{example}

\begin{solution} By Theorem~\ref{thm:matrix-of-linear-transformation} to find this matrix, we need to determine the action of $T$ on
$\vect{e}_{1}$ and $\vect{e}_{2}$. In Example~\ref{exa:matrix-of-linear-transformation}, we were given these resulting vectors.
However, in this example, we have been given $T$ of two different vectors. How can we find out the action
of $T$ on $\vect{e}_{1}$ and $\vect{e}_{2}$? In particular for $\vect{e}_{1}$, suppose there exist $x$ and $y$ such that
\begin{equation}
\begin{mymatrix}{r}
1 \\
0
\end{mymatrix} = x\begin{mymatrix}{r}
1\\
1
\end{mymatrix} +y\begin{mymatrix}{r}
0 \\
-1
\end{mymatrix}
\label{matrix-values}
\end{equation}

Then, since $T$ is linear,
\begin{equation*}
T\begin{mymatrix}{r}
1 \\
0
\end{mymatrix}  = x T\begin{mymatrix}{r}
1 \\
1
\end{mymatrix} +y T\begin{mymatrix}{r}
0 \\
-1
\end{mymatrix}
\end{equation*}

Substituting in values, this sum becomes
\begin{equation}
T\begin{mymatrix}{r}
1 \\
0
\end{mymatrix} =
 x\begin{mymatrix}{r}
1 \\
2
\end{mymatrix} +y\begin{mymatrix}{r}
3 \\
2
\end{mymatrix}
\label{matrix-values2}
\end{equation}

Therefore, if we know the values of $x$ and $y$ which satisfy {\eqref{matrix-values}}, we can substitute these into equation {\eqref{matrix-values2}}. By doing so,
we find $T(\vect{e}_1)$ which is the first column of the matrix $A$.

We proceed to find $x$ and $y$. We do so by solving {\eqref{matrix-values}}, which can be done by solving the system
\begin{equation*}
\begin{array}{c}
x = 1 \\
x - y = 0
\end{array}
\end{equation*}

We see that $x=1$ and $y=1$ is the solution to this system.
Substituting these values into equation {\eqref{matrix-values2}}, we have
\begin{equation*}
T\begin{mymatrix}{r}
1 \\
0
\end{mymatrix} =
 1 \begin{mymatrix}{r}
1 \\
2
\end{mymatrix} + 1 \begin{mymatrix}{r}
3 \\
2
\end{mymatrix}
=
 \begin{mymatrix}{r}
1 \\
2
\end{mymatrix} + \begin{mymatrix}{r}
3 \\
2
\end{mymatrix}
=
\begin{mymatrix}{r}
4 \\
4
\end{mymatrix}
\end{equation*}

Therefore $\begin{mymatrix}{r}
4 \\
4
\end{mymatrix}$
is the first column of $A$.

Computing the second column is done in the same way, and is left as an exercise.

The resulting matrix $A$ is given by
\begin{equation*}
A
=
\begin{mymatrix}{rr}
4 & -3 \\
4 & -2
\end{mymatrix}
\end{equation*}
\end{solution}

This example illustrates a very long procedure for finding the matrix of $A$. While this method is reliable and
will always result in the correct matrix $A$, the following procedure provides an alternative method.

\begin{procedure}{Finding the matrix of inconveniently defined linear transformation}{finding-matrix-of-linear-transformation}
Suppose $T:\R^{n}\rightarrow \R^{m}$ is a linear transformation. Suppose there exist vectors $\set{\vect{a}_{1},\ldots,\vect{a}_{n}} $ in $\R^{n}$ such that $\begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix} ^{-1}$ exists, and
\begin{equation*}
T (\vect{a}_{i})=\vect{b}_{i}
\end{equation*}
Then the matrix of $T$ must be of the form
\begin{equation*}
\begin{mymatrix}{ccc}
\vect{b}_{1} & \cdots & \vect{b}_{n}
\end{mymatrix} \begin{mymatrix}{ccc}
\vect{a}_{1} & \cdots & \vect{a}_{n}
\end{mymatrix} ^{-1}
\end{equation*}
\end{procedure}

We will illustrate this procedure in the following example. You may also find it useful
to work through Example~\ref{exa:2x2-inconvenient-matrix-of-linear-transformation} using this procedure.

\begin{example}{Matrix of a linear transformation \\ given inconveniently}{inconvenient-matrix-linear-transformation}
Suppose $T:\R^{3}\rightarrow \R^{3}$ is a linear
transformation and
\begin{equation*}
T\begin{mymatrix}{r}
1 \\
3 \\
1
\end{mymatrix} =\begin{mymatrix}{r}
0 \\
1 \\
1
\end{mymatrix} ,T\begin{mymatrix}{r}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{r}
2 \\
1 \\
3
\end{mymatrix} ,T\begin{mymatrix}{r}
1 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{r}
0 \\
0 \\
1
\end{mymatrix}
\end{equation*}
Find the matrix of this linear transformation.
\end{example}

\begin{solution}
By Procedure~\ref{proc:finding-matrix-of-linear-transformation},
$A=  \begin{mymatrix}{rrr}
1 & 0 & 1 \\
3 & 1 & 1 \\
1 & 1 & 0
\end{mymatrix} ^{-1}$ and
 $B=\begin{mymatrix}{rrr}
0 & 2 & 0 \\
1 & 1 & 0 \\
1 & 3 & 1
\end{mymatrix}$

Then, Procedure~\ref{proc:finding-matrix-of-linear-transformation} claims that the matrix of $T$ is
\begin{equation*}
C= BA^{-1}
=\begin{mymatrix}{rrr}
2 & -2 & 4 \\
0 & 0 & 1 \\
4 & -3 & 6
\end{mymatrix}
\end{equation*}

Indeed you can first verify that $T(\vect{x})=C\vect{x}$ for the 3 vectors above:

\begin{equation*}
 \begin{mymatrix}{ccc}
2 & -2 & 4 \\
0 & 0 & 1 \\
4 & -3 & 6
\end{mymatrix} \begin{mymatrix}{c}
1 \\
3 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} ,\ \begin{mymatrix}{ccc}
2 & -2 & 4 \\
0 & 0 & 1 \\
4 & -3 & 6
\end{mymatrix} \begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
2 \\
1 \\
3
\end{mymatrix}
\end{equation*}
\begin{equation*}
\begin{mymatrix}{ccc}
2 & -2 & 4 \\
0 & 0 & 1 \\
4 & -3 & 6
\end{mymatrix} \begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix}
\end{equation*}

But more generally $T(\vect{x})= C\vect{x}$ for any $\vect{x}$. To see this, let $\vect{y}=A^{-1}\vect{x}$ and then using linearity of $T$:
\[ T(\vect{x})= T(A\vect{y}) = T \paren{\sum_i \vect{y}_i\vect{a}_i} = \sum \vect{y}_i T(\vect{a}_i) \sum \vect{y}_i \vect{b}_i = B\vect{y} = BA^{-1}\vect{x} = C\vect{x}\]
\end{solution}

Recall the dot product discussed earlier. Consider the map $\vect{v}$\textbf{$\mapsto $}
$\proj_{\vect{u}}(\vect{v}) $ which takes a vector a transforms it to its projection onto a given vector $\vect{u}$. It turns out that
this map is linear, a result which follows from the properties of the
dot product. This is shown as follows.
\begin{eqnarray*}
\proj_{\vect{u}}(k \vect{v}+ p \vect{w})
&=&\paren{\vspace{0.05in}\frac{(k \vect{v}+ p \vect{w})\dotprod \vect{u}}{
\vect{u}\dotprod \vect{u}}} \vect{u} \\
&=& k  \paren{\vspace{0.05in}\frac{
\vect{v}\dotprod \vect{u}}{\vect{u}\dotprod \vect{u}}} \vect{u}+p \paren{\vspace{
0.05in}\frac{\vect{w}\dotprod \vect{u}}{\vect{u}\dotprod \vect{u}}} \vect{u} \\
&=& k \; \proj_{\vect{u}}(\vect{v}) +p \; \proj
_{\vect{u}}(\vect{w})
\end{eqnarray*}

Consider the following example.

\begin{example}{Matrix of a projection map}{projection-matrix}
Let $\vect{u} = \begin{mymatrix}{r}
1 \\
2 \\
3
\end{mymatrix}$ and let $T$ be the projection map $T: \R^3 \to \R^3$ defined by
\[
T(\vect{v}) = \proj_{\vect{u}}(\vect{v})
\]
for any $\vect{v} \in \R^3$.
\begin{enumerate}
\item Does this transformation come from
multiplication by a matrix?
\item If so, what is the matrix?
\end{enumerate}
\end{example}

\begin{solution}
\begin{enumerate}
\item
First, we have just seen that $T (\vect{v}) = \proj_{\vect{u}}(\vect{v})$ is linear. Therefore by Theorem~\ref{thm:matrix-linear-transformation}, we can find a matrix $A$ such that $T(\vect{x}) = A\vect{x}$.

\item
The columns of the matrix for $T$ are defined above as $T(\vect{e}_{i})$.
It follows that $T(\vect{e}_{i}) = \proj
_{\vect{u}}(\vect{e}_{i}) $ gives the $i\th$ column of the
desired matrix. Therefore, we need to find
\begin{equation*}
\proj_{\vect{u}}(\vect{e}_{i}) = \paren{\vspace{0.05in}
\frac{\vect{e}_{i}\dotprod \vect{u}}{\vect{u}\dotprod \vect{u}}}
\vect{u}
\end{equation*}
For the given vector $\vect{u}$ , this implies the columns of the desired
matrix are
\begin{equation*}
\vspace{0.05in}\frac{1}{14}\begin{mymatrix}{r}
1 \\
2 \\
3
\end{mymatrix} ,\vspace{0.05in}\frac{2}{14}\begin{mymatrix}{r}
1 \\
2 \\
3
\end{mymatrix} ,\vspace{0.05in}\frac{3}{14}\begin{mymatrix}{r}
1 \\
2 \\
3
\end{mymatrix}
\end{equation*}
which you can verify.
Hence the matrix of $T$ is
\begin{equation*}
\vspace{0.05in}\frac{1}{14}\begin{mymatrix}{rrr}
1 & 2 & 3 \\
2 & 4 & 6 \\
3 & 6 & 9
\end{mymatrix}
\end{equation*}
\end{enumerate}
\end{solution}


\documentclass[../searching.tex]{subfiles}
\begin{document}
To search in a sorted array or string using brute force with a for loop, it takes $O(n)$ time. Binary search is designed to reduce search time if the array or string is already sorted. It uses the divide and conquer method; each time we compare our target with the middle element of the array and with the comparison result to decide the next search region: either the left half or the right half. Therefore, each step we filter out half of the array which gives the time complexity function $T(n) = T(n/2) + O(1)$, which decrease the time complexity to $O(\log n)$.

Binary Search can be applied to different tasks:
\begin{enumerate}
    \item Find Exact target, find the first position that value>= target, find the last position that value<= target.(this is called lower\_bound, and upper\_bound.
\end{enumerate}

\subsection{Standard Binary Search and Python Module bisect}
Binary search is usually carried out on a Static sorted array or 2D matrix. There are three basic cases: (1) find the exact target that value = target; If there are duplicates, we are more likely to be asked to (2) find the first position that has value >= target; (3) find the first position that has value <= target. Here, we use two example array: one without duplicates and the other has duplicates.
\begin{lstlisting}[language=Python]
a = [2, 4, 5, 9]
b = [0, 1, 1, 1, 1, 1]
\end{lstlisting}

\paragraph{Find the Exact Target} This is the most basic application of binary search. We can set two pointers, l and r. Each time we compute the middle position, and check if it is equal to the target. If it is, return the position; if it is smaller than the target, move to the left half, otherwise, move to the right half. The Python code is given:
\begin{lstlisting}[language=Python]
def standard_binary_search(lst, target):
    l, r = 0, len(lst) - 1
    while l <= r:
        mid = l + (r - l) // 2
        if lst[mid] == target:
            return mid
        elif lst[mid] < target:
            l = mid + 1
        else:
            r = mid - 1
    return -1 # target is not found
\end{lstlisting}
Now, run the example:
\begin{lstlisting}[language=Python]
print("standard_binary_search: ", standard_binary_search(a,3), standard_binary_search(a,4), standard_binary_search(b, 1))
\end{lstlisting}
The print out is:
\begin{lstlisting}
standard_binary_search:  -1 1 2
\end{lstlisting}
From the example, we can see that multiple \textbf{duplicates} of the target exist, it can possibly return any one of them. And for the case when the target does not exist, it simply returns -1. In reality, we might need to find a position where we can potentially insert the target to keep the sorted array sorted. There are two cases: (1) the first position that we can insert, which is the first position that has value>= target (2) and the last position we can insert, which is the first position that has value > target. For example, if we try to insert 3 in a, and 1 in b, the first position should be 1 and 1 in each array, and the last position is 1 and 6 instead.  For these two cases, we have a Python built-in Module \textbf{bisect} which offers two methods: bisect\_left() and bisect\_right() for these two cases respectively.

\paragraph{Find the First Position that value >= target} This way the target position separates the array into two halves: value < target, target\_position, value>= target. In order to meet the purpose, we make sure that if value < target, we move to the right side, else, move to the left side.
\begin{lstlisting}[language=Python]
# bisect_left, no longer need to check the mid element,
# it separate the list in to two halfs: value < target, mid,  value >= target
def bisect_left_raw(lst, target):
    l, r = 0, len(lst)-1
    while l <= r:
        mid = l + (r-l)//2
        if lst[mid] < target: # move to the right half if the value < target, till
            l = mid + 1 #[mid+1, right]
        else:# move to the left half is value >= target
            r = mid - 1  #[left, mid-1]
    return l # the final position is where
\end{lstlisting}
% Now insert the value with:
% \begin{lstlisting}[language=Python]
% lst.insert(l+1, target)
% \end{lstlisting}

\paragraph{Find the First Position that value > target} This way the target position separates the array into two halves: value <= target, target\_position, value> target. Therefore, we simply change the condition of if value < target to if value <= target, then we move to the right side.
\begin{lstlisting}[language=Python]
#bisect_right: separate the list into two halfs: value<= target, mid, value > target
def bisect_right_raw(lst, target):
    l, r = 0, len(lst)-1
    while l <= r:
        mid = l + (r-l)//2
        if lst[mid] <= target:
            l = mid + 1
        else:
            r = mid -1
    return l
\end{lstlisting}
Now, run an example:
\begin{lstlisting}[language=Python]
print("bisect left raw: find 3 in a :", bisect_left_raw(a,3), 'find 1 in b: ', bisect_left_raw(b, 1))
print("bisect right raw: find 3 in a :", bisect_right_raw(a, 3), 'find 1 in b: ', bisect_right_raw(b, 1))
\end{lstlisting}
The print out is:
\begin{lstlisting}
bisect left raw: find 3 in a : 1 find 1 in b:  1
bisect right raw: find 3 in a : 1 find 1 in b:  6
\end{lstlisting}

\paragraph{Bonus} For the last two cases, if we return the position as l-1, then we get the last position that value < target, and the last position value <= target.

\paragraph{Python Built-in Module bisect} This module provides support for maintaining a list in sorted order without having to sort the list after each insertion. It offers six methods as shown in Table~\ref{tab:method_bisect}. However, only two are most commonly used: bisect\_left and bisect\_right.
\begin{table}[h]
\begin{small}
\centering
\noindent\captionof{table}{ Methods of \textbf{bisect}}
 \noindent \begin{tabular}{|p{0.25\columnwidth}|p{0.75\columnwidth}| }
  \hline
Method & Description   \\ \hline
bisect\_left(a, x, lo=0, hi=len(a)  &  The parameters lo and hi may be used to specify a subset of the list; the function is the same as bisect\_left\_raw  \\\hline
bisect\_right(a, x, lo=0, hi=len(a)  &  The parameters lo and hi may be used to specify a subset of the list; the function is the same as bisect\_right\_raw  \\\hline
bisect(a, x, lo=0, hi=len(a))  &Similar to bisect\_left(), but returns an insertion point which comes after (to the right of) any existing entries of x in a.\\ \hline
insort\_left(a, x, lo=0, hi=len(a))  &This is equivalent to a.insert(bisect.bisect\_left(a, x, lo, hi), x).\\ \hline
insort\_right(a, x, lo=0, hi=len(a)) & This is equivalent to a.insert(bisect.bisect\_right(a, x, lo, hi), x).\\ \hline
insort(a, x, lo=0, hi=len(a)) & Similar to insort\_left(), but inserting x in a after any existing entries of x.\\ \hline
\end{tabular}
  \label{tab:method_bisect}
  \end{small}
\end{table}
Let's see come examplary code:
\begin{lstlisting}[language=Python]
from bisect import bisect_left,bisect_right, bisect
print("bisect left: find 3 in a :", bisect_left(a,3), 'find 1 in b: ', bisect_left(b, 1)) # lower_bound, the first position that value>= target
print("bisect right: find 3 in a :", bisect_right(a, 3), 'find 1 in b: ', bisect_right(b, 1)) # upper_bound, the last position that value <= target
\end{lstlisting}
The print out is:
\begin{lstlisting}
bisect left: find 3 in a : 1 find 1 in b:  1
bisect right: find 3 in a : 1 find 1 in b:  6
\end{lstlisting}
\subsection{Binary Search in Rotated Sorted Array}
\label{concept_binary_search_in_array}
The extension of the standard binary search is on array that the array is ordered in its own way like rotated array.

\paragraph{Binary Search in Rotated Sorted Array } (See LeetCode problem, 33. Search in Rotated Sorted Array (medium). Suppose an array (without duplicates) sorted in ascending order is rotated at some pivot unknown to you beforehand. (i.e., 0 1 2 4 5 6 7 might become 4 5 6 7 0 1 2). You are given a target value to search. If found in the array return its index, otherwise return -1. You may assume no duplicate exists in the array.
\begin{lstlisting}[numbers=none]
Example 1:

Input: nums = [3, 4,5,6,7,0,1,2], target = 0
Output: 5

Example 2:

Input: nums = [4,5,6,7,0,1,2], target = 3
Output: -1
\end{lstlisting}

In the rotated sorted array, the array is not purely monotonic. Instead, there is one drop in the array because of the rotation, where it cuts the array into two parts. Suppose we are starting with a standard binary search with example 1, at first, we will check index 3, then we need to move to the right side?  Assuming we compare our middle item with the left item,
\begin{lstlisting}[numbers=none]
if nums[mid] > nums[l]: # the left half is sorted
elif nums[mid] < nums[l]: # the right half is sorted
else: # for case like [1,3], move to the right half
\end{lstlisting}
For a standard binary search, we simply need to compare the target with the middle item to decide which way to go. In this case, we can use objection. Check which side is sorted, because no matter where the left, right and the middle index is, there is always one side that is sorted. So if the left side is sorted, and the value is in the range of the [left, mid], then we move to the left part, else we object the left side, and move to the right side instead.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\columnwidth]{fig/rotated_array.png}
    \caption{Example of Rotated Sorted Array}
    \label{fig:rotated_sorted_array}
\end{figure}

The code is shown:
\begin{lstlisting}[language=Python]
'''implemente the rotated binary search'''
def RotatedBinarySearch(nums, target):
    if not nums:
        return -1

    l,r = 0,len(nums)-1
    while l<=r:
        mid = l+ (r-l)//2
        if nums[mid] == target:
            return mid
         if nums[l] < nums[mid]: # if the left part is sorted
                if nums[l] <= target <= nums[mid]:
                    r = mid-1
                else:
                    l = mid+1
            elif nums[l] > nums[mid]: # if the right side is sorted
                if nums[mid] <= target <= nums[r]:
                    l = mid+1
                else:
                    r = mid-1
            else:
                l = mid + 1
    return -1
\end{lstlisting}
\begin{bclogo}[couleur = blue!30, arrondi=0.1,logo=\bccrayon,ombre=true]{What happens if there is duplicates in the rotated sorted array? } In fact, similar comparing rule applies:
\begin{lstlisting}[numbers=none]
if nums[mid] > nums[l]: # the left half is sorted
elif nums[mid] < nums[l]: # the right half is sorted
else: # for case like [1,3], or [1, 3, 1, 1, 1] or [3, 1, 2, 3, 3, 3]
   only l++
\end{lstlisting}
\end{bclogo}

%%%%%%%%%%%%%%binary search on result space%%%%%%%
\subsection{Binary Search on Result Space}
If the question gives us the context: the target is in the range [left, right], we need to search the first or last position that satisfy a condition function. We can apply the concept of standard binary search and bisect\_left and bisect\_right and its mutant. Where we use the condition function to replace the value comparison between target and element at middle position. The steps we need:
\begin{enumerate}
    \item get the result search range [l, r] which is the initial value for l and r pointers.
    \item decide the valid function to replace such as if lst[mid] < target
    \item decide which binary search we use: standard, bisect\_left/ bisect\_right or its mutant.
\end{enumerate}

For example:
\begin{examples}[resume]
\item \textbf{441. Arranging Coins (easy)}. You have a total of n coins that you want to form in a staircase shape, where every k-th row must have exactly k coins. Given n, find the total number of full staircase rows that can be formed. n is a non-negative integer and fits within the range of a 32-bit signed integer.
\begin{lstlisting}[numbers=none]
Example 1:

n = 5

The coins can form the following rows:
*
* *
* *

Because the 3rd row is incomplete, we return 2.
\end{lstlisting}

\textbf{Analysis: } Given a number n>=1, the minimum row is 1, and the maximum is n. Therefore, our possible result range is [1, n]. These can be treated as indexes of the sorted array. For a given row, we write a function to check if it is possible. We need a function $r* (r+1) // 2 <= n$. For this problem, we need to search in the range of [1, n] to find the last position that is valid. This is bisect\_left or bisect\_right, where we use the function replace the condition check:
\begin{lstlisting}[language=Python]
def arrangeCoins(self, n):
    def isValid(row):
        return (row*(row+1))//2 <= n
    # we need to find the last position that is valid (<=)
    def bisect_right():
        l, r = 1, n
        while l <= r:
            mid = l + (r-l) // 2
            if isValid(mid): # replaced compared with the standard binary search
                l = mid + 1
            else:
                r = mid - 1
        return l-1
    return bisect_right()
\end{lstlisting}
\item \textbf{278. First Bad Version.} You are a product manager and currently leading a team to develop a new product. Unfortunately, the latest version of your product fails the quality check. Since each version is developed based on the previous version, all the versions after a bad version are also bad.

Suppose you have n versions [1, 2, ..., n] and you want to find out the first bad one, which causes all the following ones to be bad.

You are given an API bool isBadVersion(version) which will return whether version is bad. Implement a function to find the first bad version. You should minimize the number of calls to the API.

Solution: we keep doing binary search until we have searched all possible areas.
\begin{lstlisting}[language = Python]
class Solution(object):
    def firstBadVersion(self, n):
        """
        :type n: int
        :rtype: int
        """
        l,r=0,n-1
        last = -1
        while l<=r:
            mid = l+(r-l)//2
            if isBadVersion(mid+1): #move to the left, mid is index, s
                r=mid-1
                last = mid+1 #to track the last bad one
            else:
                l=mid-1
        return last
\end{lstlisting}
\end{examples}
% \subsection{Bisection Method} (second edition)
% The binary search principle can be used to find the root of a function that may be difficult to compute mathematically. We have not seen any problems that require this method on LeetCode yet. Thus we define the problem as:

% Find the monthly payment for a loan: You want to buy a car using loan and want to pay in monthly installment of d d
% \subsection{Python Library}
% Python has \textbf{bisect} module for binary search.
% \begin{lstlisting}[numbers=none]
% bisect.bisect_left(a,    x):  Return the leftmost index where we can  insert x into a to maintain sorted order! Leftmost rl that satisfy: x<=a[rl]

% bisect.bisect_right(a,    x):  Return the rightmost index where we can  insert x into a to maintain sorted order! Right most rr that satisfy: x>=a[rr]
% \end{lstlisting}
% For example:
% \begin{lstlisting}[language=Python]
% from bisect import bisect_left,bisect_right
% a = [1,    2,    3,    3,    3,    4,    5]
% p1, p2= bisect_left(a,3), bisect_right(a, 3)
% print(p1, p2)
% # output
% # 2, 5
% \end{lstlisting}

\subsection{LeetCode Problems}
\begin{examples}
\item \textbf{35. Search Insert Position (easy).} Given a sorted array and a target value, return the index if the target is found. If not, return the index where it would be if it were inserted in order.

You can assume that there are no duplicates in the array.
\begin{lstlisting}[numbers=none]
Example 1:

Input: [1,3,5,6], 5
Output: 2

Example 2:
Input: [1,3,5,6], 2
Output: 1

Example 3:
Input: [1,3,5,6], 7
Output: 4

Example 4:
Input: [1,3,5,6], 0
Output: 0
\end{lstlisting}

\textbf{Solution: Standard Binary Search Implementation.} For this problem, we just standardize the Python code of binary search, which takes $O(logn)$ time complexity and O(1) space complexity without using recursion function. In the following code, we use exclusive right index with len(nums), therefore it stops if l == r; it can be as small as 0 or as large as n of the array length for numbers that are either smaller or equal to the nums[0] or larger or equal to nums[-1]. We can also make the right index inclusive.
\begin{lstlisting}[language = Python]
# exclusive version
def searchInsert(self, nums, target):
    l, r = 0, len(nums) #start from 0, end to the len (exclusive)
    while l < r:
        mid = (l+r)//2
        if nums[mid] < target: #move to the right side
            l = mid+1
        elif nums[mid] > target: #move to the left side, not mid-1
             r= mid
        else: #found the traget
            return mid
    #where the position should go
    return l
\end{lstlisting}

\begin{lstlisting}[language = Python]
# inclusive version
def searchInsert(self, nums, target):
   l = 0
    r = len(nums)-1
    while l <= r:
        m = (l+r)//2
        if target > nums[m]: #search the right half
            l = m+1
        elif target < nums[m]: # search for the left half
            r = m-1
        else:
            return m
    return l
\end{lstlisting}
\end{examples}
Standard binary search
\begin{enumerate}
    \item 611. Valid Triangle Number (medium)
    \item 704. Binary Search (easy)

\item  74. Search a 2D Matrix) Write an efficient algorithm that searches for a value in an m x n matrix. This matrix has the following properties:
\begin{enumerate}
    \item Integers in each row are sorted from left to right.
    \item The first integer of each row is greater than the last integer of the previous row.
    \end{enumerate}
\begin{lstlisting}[numbers=none]
For example,
Consider the following matrix:

[
  [1,   3,  5,  7],
  [10, 11, 16, 20],
  [23, 30, 34, 50]
]

Given target = 3, return true.
\end{lstlisting}

Solution: 2D matrix search, time complexity from $O(n^2)$ to $O(lgm+lgn)$.
\begin{lstlisting}[language = Python]
def searchMatrix(self, matrix, target):
        """
        :type matrix: List[List[int]]
        :type target: int
        :rtype: bool
        """

        if not matrix:
            return False
        row, col = len(matrix), len(matrix[0])
        if row==0 or col==0: #for [[]]
            return False
        sr, er = 0, row-1
        #fisrst search the mid row
        while sr<=er:
            mid = sr+(er-sr)//2
            if target>matrix[mid][-1]: #go to the right side
                sr=mid+1
            elif target < matrix[mid][0]: #go the the left side
                er = mid-1
            else: #value might be in this row
                #search in this row
                lc, rc = 0, col-1
                while lc<=rc:
                    midc = lc+(rc-lc)//2
                    if matrix[mid][midc]==target:
                        return True
                    elif target<matrix[mid][midc]: #go to left
                        rc=midc-1
                    else:
                        lc=midc+1
                return False
        return False
\end{lstlisting}

Also, we can treat is as one dimensional, and the time complexity is $O(lg(m*n))$, which is the same as $O(log(m)+log(n))$.
\begin{lstlisting}[language = Python]
class Solution:
    def searchMatrix(self, matrix, target):
        if not matrix or target is None:
            return False

        rows, cols = len(matrix), len(matrix[0])
        low, high = 0, rows * cols - 1

        while low <= high:
            mid = (low + high) / 2
            num = matrix[mid / cols][mid % cols]

            if num == target:
                return True
            elif num < target:
                low = mid + 1
            else:
                high = mid - 1

        return False
\end{lstlisting}
\end{enumerate}

Check \url{http://www.cnblogs.com/grandyang/p/6854825.html} to get more examples.

Search on rotated and 2d matrix:
\begin{enumerate}
    \item 81. Search in Rotated Sorted Array II (medium)
    \item 153. Find Minimum in Rotated Sorted Array (medium) The key here is to compare the mid with left side, if mid-1 has a larger value, then that is the minimum
    \item 154. Find Minimum in Rotated Sorted Array II (hard)
\end{enumerate}
Search on Result Space:
\begin{enumerate}
    \item 367. Valid Perfect Square (easy) (standard search)
    \item 363. Max Sum of Rectangle No Larger Than K (hard)
    \item 354. Russian Doll Envelopes (hard)
    \item 69. Sqrt(x) (easy)
\end{enumerate}
\end{document}


\subsection{Analytic optimisation}

\subsubsection{Convex and concave functions}

Convex functions only have one minimum, and concave functions have only one maximum.

If a function is not concave or convex, it may have multiple minima

If a function is convex, then there is only one critical point – the local minimum. We can identify this this by looking for critical points using first-order conditions.

Similarly, if a function is concave, then there is only one critical point – the local maximum.

We can identify whether a function is concave or convex by evaluating the Hessian matrix.

\subsubsection{Evaluating multiple local optima}

We can evaluate each of the local minima or maxima, and compare the sizes.

We can identify these by taking partial derivatives of the function in question and identifying where this function is equal to zero.

\(u=f(x)\)

\(u_{x_i}=\dfrac{\delta f}{\delta x_i}=0\)

We can then solve this bundle of equations to find the stationary values of \(x\).

After identifying the vector \(x\) for these points we can then  determine whether or not the points are minima or maxima by  examining the second derivative at these points. If it is positive it is a local minima, and therefore not an optimal point. Points beyond these will be higher, and may be higher than any local maxima.


\subsection{Rank and independence}

If $\{\tb{v}_1,\tb{v}_2,\cdots ,\tb{v}_m\}$ is a collection if independent vectors then

\[
\left(\begin{array}{ccccc}
\mid & \mid & \mid & & \mid \\
\tb{v}_{1} & \bar{v}_{2} & \bar{v}_{3} & \ldots & \bar{v}_{m} \\
\mid & \mid & \mid & & \mid
\end{array}\right)
\]

must have a rank of $m$. This is because row reducing the matrix corresponds to the following relation:

\[c_1\tb{v}_1+c_2\tb{v}_2+c_3\tb{v}_3+\dots +c_m\tb{v}_m=\tb{0}\]

\noindent
Also, $m\leq n$ where $n$ is the number of rows in each column vector, in order to have linear independence for this set.

\subsection{Dimension}

Considering an $xy$-plane in $\R^3$:

\[V=\left\{\begin{bmatrix}s\\t\\0\\ \end{bmatrix}\Bigg| s,t \in \mathbb{R} \right\}\]

The basis of this set contains 2 vectors (e.g. dimension of 2), with example being:

\[\mathfrak{B}=\left\{\begin{bmatrix}1\\0\\0 \end{bmatrix} , \begin{bmatrix} 0\\1\\0 \end{bmatrix} \right\}\]

\begin{framed}
    If $V$ is a subspace of $\R^n$ and $\mathfrak{B}$ and $\mathfrak{C}$ are two bases of $V$, then
    $\mathfrak{B}$ and $\mathfrak{C}$ contain the same number of vectors.
\end{framed}

\textbf{Dimension} of a subspace is number of vectors in the basis.

\subsubsection{Example}

Considering the following matrix:

\[A=\begin{bmatrix}
    1&2&0&1&2\\
    1&2&0&2&3\\
    1&2&0&3&4\\
    1&2&0&4&5\\
    \end{bmatrix}\]

By discounting the redundant vectors, a possible basis for $\mathrm{Im}(A)$:

\[\mathfrak{B}_\mathrm{image}=\left \{\begin{bmatrix} 1\\1\\1\\1 \end{bmatrix},\begin{bmatrix}1\\2\\3\\4 \end{bmatrix} \right\}\]

So the dimension of $\mathrm{Im}(A)$ is 2.
Finding a basis for $\mathrm{ker}(A)$ is the same as solving $A\tb{x}=\tb{0}$:

\[\mbox{ker}(A)=\left\{\begin{bmatrix}-2s-w\\ s\\t\\-w\\w \end{bmatrix} \Bigg| s,t,w\in\mathbb{R} \right\}=
\left\{s\begin{bmatrix}-2\\1\\0\\0\\0\\\end{bmatrix}+t\begin{bmatrix}0\\0\\1\\0\\0\\\end{bmatrix}+w\begin{bmatrix}-1\\0\\0\\-1\\1\\\end{bmatrix}\Bigg|s,t,w\in\mathbb{R}\right\}\]

So the basis for $\mathrm{ker}(A)$:

\[\mathfrak{B}_\mathrm{kernel}=\left\{\begin{bmatrix}-2\\1\\0\\0\\0\\ \end{bmatrix}, \begin{bmatrix} 0\\0\\1\\0\\0\\ \end{bmatrix}, \begin{bmatrix}-1\\0\\0\\-1\\1\\ \end{bmatrix} \right\}\]

And dimension of $\mathrm{ker}(A)$ is 3.
However, it is shown that $\mathrm{rref}(A)$ gives dimension of \textbf{image and kernel}.

\subsection{Rank-Nullity Theorem}

\begin{framed}
    \begin{itemize}
        \item If $T$ is a linear transformation,
        then \\$\mathrm{dim}(\mathrm{Im}(T))+\mathrm{dim}(\mathrm{ker}(T))=\text{dimension of domain of }T$
        \item If $A$ is a matrix, then $\mathrm{rank}(A)+\mathrm{nullity}(A)=\text{number of columns of }A$
        \item In a linear system,
        \\$\text{number of leading variables}+\text{number of free variables}=\text{total number of variables}$
    \end{itemize}
\end{framed}

Considering non-invertible matrices $A$ and $B$, let $AB$ be invertible.
It must hold true that $\mathrm{ker}(B)=\{\tb{0}\}$. If the dimensions of $B$ are $p\times n$,
$\mathrm{Im}(B)$ is a subspace of $\R^p$ has dimension $n$. This means that it is a vertically rectangular
matrix with $n\leq p$. Thus, $A$ is $n\times p$ so it is horizontally rectangular.