\lab{Introduction to SymPy}{Introduction to SymPy}

\objective{Most implementations of numerical algorithms focus on crunching, relating, or visualizing numerical data.
However, it is sometimes convenient or necessary to represent parts of an algorithm symbolically.
The SymPy module provides a way to do symbolic mathematics in Python, including algebra, differentiation, integration, and more.
In this lab we introduce SymPy syntax and emphasize how to use symbolic algebra for numerical computing.}

\section*{Symbolic Variables and Expressions} % ===============================

Most variables in Python refer to a number, string, or data structure.
Doing computations on such variables results in more numbers, strings, or data structures.
A \emph{symbolic variable} is a variable that represents a mathematical symbol, such as $x$ or $\theta$, not a number or another kind of data.
Operating on symbolic variables results in an \emph{expression}, representative of an actual mathematical expression.
For example, if a symbolic variable \li{Y} refers to a mathematical variable $y$, the multiplication \li{3*Y} refers to the expression $3y$.
This is all done without assigning an actual numerical value to \li{Y}.

SymPy \cite{meurer2017sympy} is Python's library for doing symbolic algebra and calculus.
It is typically imported with \li{import sympy as sy}, and symbolic variables are usually defined using \li{sy.symbols()}.
% Expressions can also be derived from a string using \li{sy.sympify()} or the shortcut \li{sy.S()}.
% \li{sy.Symbol()} can also be used to define a single variable, but will not be used in this lab for consistency.

\begin{lstlisting}
>>> import sympy as sy
>>> x0 = sy.symbols('x0')                   # Define a single variable.

# Define multiple symbolic variables simultaneously.
>>> x2, x3 = sy.symbols('x2, x3')           # Separate symbols by commas,
>>> m, a = sy.symbols('mass acceleration')  # by spaces,
>>> x, y, z = sy.symbols('x:z')             # or by colons.
>>> x4, x5, x6 = sy.symbols('x4:7')

# Combine symbolic variables to form expressions.
>>> expr = x**2 + x*y + 3*x*y + 4*y**3
>>> force = m * a
>>> print(expr, force, sep='\n')
x**2 + 4*x*y + 4*y**3
acceleration*mass
\end{lstlisting}
%
% # Create an expression from a string.
% >>> sy.S("x0**2+x1**3+x2")
% x0**2 + x1**3 + x2

% Pre-defined variables, including greek letters, can be imported from \li{sympy.abc}.
% To see a list of these variables, run \li{dir(sy.abc)} after \li{sy.abc} has been imported.

SymPy has its own version for each of the standard mathematical functions like $\sin(x)$, $\log(x)$, and $\sqrt{x}$, and includes predefined variables for special numbers such as $\pi$.
The naming conventions for most functions match NumPy, but some of the built-in constants are named slightly differently.
\[
\begin{array}{cl}
\text{Functions}
&
\begin{tabular}{c|c|c|c|c|c}
    $\sin(x)$ & $\arcsin(x)$ & $\sinh(x)$ & $e^x$ & $\log(x)$ & $\sqrt{x}$
    \\ \hline
    \li{sy.sin()} & \li{sy.asin()} & \li{sy.sinh()} & \li{sy.exp()} & \li{sy.log()} & \li{sy.sqrt()}
\end{tabular}
\\ \\
\text{Constants}
&
\begin{tabular}{c|c|c|c}
    $\pi$ & $e$ & $i = \sqrt{-1}$ & $\infty$
    \\ \hline
    \li{sy.pi} & \li{sy.E} & \li{sy.I} & \li{sy.oo}
\end{tabular}
\end{array}
\]
Other trigonometric functions like $\cos(x)$ follow the same naming conventions.
For a complete list of SymPy functions, see \url{http://docs.sympy.org/latest/modules/functions/index.html}.

\begin{warn}
Always use SymPy functions and constants when creating expressions instead of using NumPy's functions and constants.
Later we will show how to make NumPy and SymPy cooperate.
\begin{lstlisting}
>>> import numpy as np

>>> x = sy.symbols('x')
>>> np.exp(x)                       # Try to use NumPy to represent e**x.
<<Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Symbol' object has no attribute 'exp'>>

>>> sy.exp(x)                       # Use SymPy's version instead.
<<exp(x)>>
\end{lstlisting}
\end{warn}

\begin{info}
SymPy defines its own numeric types for integers, floats, and rational numbers.
For example, the \li{sy.Rational} class is similar to the standard library's \li{fractions.Fraction} class, and should be used to represent fractions in SymPy expressions.

\begin{lstlisting}
>>> x = sy.symbols('x')
>>> (2/3) * sy.sin(x)               # 2/3 returns a float, not a rational.
0.666666666666667*sin(x)

>>> sy.Rational(2, 3) * sy.sin(x)   # Keep 2/3 symbolic.
2*sin(x)/3
\end{lstlisting}
Always be aware of which numeric types are being used in an expression.
Using rationals and integers where possible is important for simplifying expressions.
% To keep $\frac{2}{3}$ symbolic, use \li{sy.Rational(2,3)*sy.sin(x)} or \li{sy.Integer(2)/3*sy.sin(x)}.
\end{info}

\begin{problem} % Write a simple expression.
Write a function that returns the expression $\frac{2}{5} e^{x^2-y}\cosh(x+y) + \frac{3}{7}\log(xy+1)$ symbolically.
Make sure that the fractions remain symbolic.
\end{problem}

\subsection*{Sums and Products} % ---------------------------------------------

Expressions that can be written as a sum or a product can be constructed with \li{sy.summation()} or \li{sy.product()}, respectively.
Each of these functions accepts an expression that represents one term of the sum or product, then a tuple indicating the indexing variable and which values it should take on.
For example, the following code constructs the sum and product given below.
\[
\sum_{i=1}^{4} x + iy
\qquad\qquad
\prod_{i=0}^{5} x + iy
\]

\begin{lstlisting}
>>> x, y, i = sy.symbols('x y i')

>>> sy.summation(x + i*y, (i, 1, 4))    # Sum over i=1,2,3,4.
<<4*x + 10*y>>

>>> sy.product(x + i*y, (i, 0, 5))      # Multiply over i=0,1,2,3,4,5.
<<x*(x + y)*(x + 2*y)*(x + 3*y)*(x + 4*y)*(x + 5*y)>>
\end{lstlisting}

\subsection*{Simplifying Expressions} % ---------------------------------------

The expressions for the summation and product in the previous example are automatically simplified.
More complicated expressions can be simplified with one or more of the following functions.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
Function & Description \\ \hline
% \li{sy.apart()} & Partial fraction decomposition. \\
\li{sy.cancel()} & Cancel common factors in the numerator and denominator. \\
% \li{sy.collect()} & Factor out all rational powers of a specified variable. \\
\li{sy.expand()} & Expand a factored expression. \\
\li{sy.factor()} & Factor an expanded expression. \\
% \li{powsimp()} & Combine powers with similar bases and exponents. \\
\li{sy.radsimp()} & Rationalize the denominator of an expression. \\
% \li{sy.ratsimp()} & Return the simplest form of the addition of two rational expressions. \\
% \li{seperate()} & Rewrite a power of products to a product of powers. \\
\li{sy.simplify()} & Simplify an expression.\\
% \li{together()} & Combine rational functions.\\
\li{sy.trigsimp()} & Simplify only the trigonometric parts of the expression. \\
\end{tabular}
\end{table}

\begin{lstlisting}
>>> x = sy.symbols('x')
>>> expr = (x**2 + 2*x + 1) / ((x+1)*((sy.sin(x)/sy.cos(x))**2 + 1))
>>> print(expr)
(x**2 + 2*x + 1)/((x + 1)*(sin(x)**2/cos(x)**2 + 1))

>>> sy.simplify(expr)
(x + 1)*cos(x)**2
\end{lstlisting}

The generic \li{sy.simplify()} tries to simplify an expression in any possible way.
This is often computationally expensive; using more specific simplifiers when possible reduces the cost.

\begin{lstlisting}
>>> expr = sy.product(x + i*y, (i, 0, 3))
>>> print(expr)
x*(x + y)*(x + 2*y)*(x + 3*y)

>>> expr_long = sy.expand(expr)         # Expand the product terms.
>>> print(expr_long)
x**4 + 6*x**3*y + 11*x**2*y**2 + 6*x*y**3

>>> expr_long /= (x + 3*y)
>>> print(expr_long)
(x**4 + 6*x**3*y + 11*x**2*y**2 + 6*x*y**3)/(x + 3*y)

>>> expr_short = sy.cancel(expr_long)   # Cancel out the denominator.
x**3 + 3*x**2*y + 2*x*y**2

>>> sy.factor(expr_short)               # Factor the result.
x*(x + y)*(x + 2*y)

# Simplify the trigonometric parts of an expression.
>>> sy.trigsimp(2*sy.sin(x)*sy.cos(x))
<<sin(2*x)>>
\end{lstlisting}

See \url{http://docs.sympy.org/latest/tutorial/simplification.html} for more examples.

\begin{warn}
% Keep the following in mind when simplifying expressions.
\begin{enumerate}
\item Simplifications return new expressions; they do not modify existing expressions in place.

\item The \li{==} operator compares two expressions for exact structural equality, not algebraic equivalence.
Simplify or expand expressions before comparing them with \li{==}.

\item Expressions containing floats may not simplify as expected.
Always use integers and SymPy rationals in expressions when appropriate.
\end{enumerate}

\begin{lstlisting}
>>> expr = 2*sy.sin(x)*sy.cos(x)
>>> sy.trigsimp(expr)
sin(2*x)
>> print(expr)
2*sin(x)*cos(x)                 # The original expression is unchanged.

>>> 2*sy.sin(x)*sy.cos(x) == sy.sin(2*x)
<<False>>                           # The two expression structures differ.

>>> sy.factor(x**2.0 - 1)
x**2.0 - 1                      # Factorization fails due to the 2.0.
\end{lstlisting}
\end{warn}

\begin{problem}
Write a function that computes and simplifies the following expression.
\[\prod_{i=1}^{5} \sum_{j=i}^{5}j(\sin(x)+\cos(x))\]
\end{problem}

\section*{Evaluating Expressions} % ===========================================

Every SymPy expression has a \li{subs()} method that substitutes one variable for another.
The result is usually still a symbolic expression, even if a numerical value is used in the substitution.
The \li{evalf()} method actually evaluates the expression numerically after all symbolic variables have been assigned a value.
Both of these methods can accept a dictionary to reassign multiple symbols simultaneously.

\begin{lstlisting}
>>> x,y = sy.symbols('x y')
>>> expr = sy.expand((x + y)**3)
>>> print(expr)
x**3 + 3*x**2*y + 3*x*y**2 + y**3

# Replace the symbolic variable y with the expression 2x.
>>> expr.subs(y, 2*x)
27*x**3

# Replace x with pi and y with 1.
>>> new_expr = expr.subs({x:sy.pi, y:1})
>>> print(new_expr)
1 + 3*pi + 3*pi**2 + pi**3
>>> new_expr.evalf()                # Numerically evaluate the expression.
71.0398678443373

# Evaluate the expression by providing values for each variable.
>>> expr.evalf(subs={x:1, y:2})
27.0000000000000
\end{lstlisting}

These operations are good for evaluating an expression at a single point, but it is typically more useful to turn the expression into a reusable numerical function.
To this end, \li{sy.lambdify()} takes in a symbolic variable (or list of variables) and an expression, then returns a callable function that corresponds to the expression.

\begin{lstlisting}
# Turn the expression sin(x)^2 into a function with x as the variable.
>>> f = sy.lambdify(x, sy.sin(x)**2)
>>> print(f(0), f(np.pi/2), f(np.pi), sep='  ')
0.0  1.0  1.4997597826618576e-32

# Lambdify a function of several variables.
>>> f = sy.lambdify((x,y), sy.sin(x)**2 + sy.cos(y)**2)
>>> print(f(0,1), f(1,0), f(np.pi, np.pi), sep='  ')
0.2919265817264289  1.708073418273571  1.0
\end{lstlisting}

By default, \li{sy.lambdify()} uses the \li{math} module to convert an expression to a function.
For example, \li{sy.sin()} is converted to \li{math.sin()}.
By providing \li{"numpy"} as an additional argument, \li{sy.lambdify()} replaces symbolic functions with their NumPy equivalents instead, so \li{sy.sin()} is converted to \li{np.sin()}.
This allows the resulting function to act element-wise on NumPy arrays, not just on single data points.

\begin{lstlisting}
>>> f = sy.lambdify(x, 2*sy.sin(2*x), "numpy")
>>> f(np.linspace(0, 2*np.pi, 9))   # Evaluate f() at many points.
array([  0.00000000e+00,   2.00000000e+00,   2.44929360e-16,
        -2.00000000e+00,  -4.89858720e-16,   2.00000000e+00,
         7.34788079e-16,  -2.00000000e+00,  -9.79717439e-16])
\end{lstlisting}

\begin{info}
It is almost always computationally cheaper to lambdify a function than to use substitutions.
According to the SymPy documentation, using \li{sy.lambdify()} to do numerical evaluations ``takes on the order of hundreds of nanoseconds, roughly two orders of magnitude faster than the \li{subs()} method.''

\begin{lstlisting}
<g<In [1]:>g> import sympy as sy
<g<In [2]:>g> import numpy as np

# Define a symbol, an expression, and points to plug into the expression.
<g<In [3]:>g> x = sy.symbols('x')
<g<In [4]:>g> expr = sy.tanh(x)
<g<In [5]:>g> points = np.random.random(10000)

# Time using evalf() on each of the random points.
<g<In [6]:>g> <p<%time>p> _ = [expr.subs(x, pt).evalf() for pt in points]
<<CPU times: user 5.29 s, sys: 40.3 ms, total: 5.33 s
Wall time: 5.36 s>>

# Lambdify the expression and time using the resulting function.
<g<In [7]:>g> f = sy.lambdify(x, expr)
<g<In [8]:>g> <p<%time>p> _ = [f(pt) for pt in points]
<<CPU times: user 5.39 ms, sys: 648 micros, total: 6.04 ms
Wall time: 7.75 ms>>      # About 1000 times faster than evalf().

# Lambdify the expression with NumPy and repeat the experiment.
<g<In [9]:>g> f = sy.lambdify(x, expr, "numpy")
<g<In [10]:>g> <p<%time>p> _ = f(points)
<<CPU times: user 381 micros, sys: 63 micros, total: 444 micros
Wall time: 282 micros>>   # About 10 times faster than regular lambdify.
\end{lstlisting}
\end{info}

\begin{problem}
The Maclaurin series up to order $N$ for $e^x$ is defined as
\begin{equation}
e^x \approx \sum_{n=0}^N \frac{x^n}{n!}.
\label{eq:sympy-maclaurin-exponential}
\end{equation}
Write a function that accepts an integer $N$.
Define an expression for (\ref{eq:sympy-maclaurin-exponential}), then substitute in $-y^2$ for $x$ to get a truncated Maclaurin series of $e^{-y^2}$.
Lambdify the resulting expression and plot the series on the domain $y\in[-2,2]$.
Plot $e^{-y^2}$ over the same domain for comparison.
\\(Hint: use \li{sy.factorial()} to compute the factorial.)

Call your function with increasing values of $N$ to check that the series converges correctly.
\end{problem}

\section*{Solving Symbolic Equations} % =======================================

A SymPy expression by itself is not an equation.
However, \li{sy.solve()} equates an expression with zero and solves for a specified variable.
In this way, SymPy can be used to solve equations.

\begin{lstlisting}
>>> x,y = sy.symbols('x y')

# Solve x^2 - 2x + 1 = 0 for x.
>>> sy.solve(x**2 - 2*x + 1, x)
[1]                                 # The result is a list of solutions.

# Solve x^2 - 1 = 0 for x.
>>> sy.solve(x**2 - 1, x)
[-1, 1]                             # This equation has two solutions.

# Solutions can also be expressions involving other variables.
>>> sy.solve(x/(y-x) + (x-y)/y, x)
[y*(-sqrt(5) + 3)/2, y*(sqrt(5) + 3)/2]
\end{lstlisting}

\begin{problem}
The following equation represents a rose curve in cartesian coordinates:
\begin{equation}
0 = 1 - \frac{(x^2 + y^2)^{7/2} + 18x^5 y - 60x^3 y^3 + 18xy^5}{(x^2 + y^2)^3}.
\label{eq:prob-polar-parametric}
\end{equation}
The curve is not the image of a single function (such a function would fail the vertical line test), so the best way to plot it is to convert (\ref{eq:prob-polar-parametric}) to a pair of parametric equations that depend on the angle parameter $\theta$.

Construct an expression for the nonzero side of (\ref{eq:prob-polar-parametric}) and convert it to polar coordinates with the substitutions $x = r\cos(\theta)$ and $y = r\sin(\theta)$.
Simplify the result, then solve it for $r$.
There are two solutions due to the presence of an $r^2$ term; pick one and lambdify it to get a function $r(\theta)$.
Use this function to plot $x(\theta) = r(\theta)\cos(\theta)$ against $y(\theta) = r(\theta)\sin(\theta)$ for $\theta \in [0,2\pi]$.
\\(Hint: use \li{sy.Rational()} for the fractional exponent.)
\end{problem}

\subsection*{Linear Algebra} % ------------------------------------------------

Sympy can also solve systems of equations.
A system of linear equations $A\x = \b$ is solved in a slightly different way than in NumPy and SciPy: instead of defining the matrix $A$ and the vector $\b$ separately, define the augmented matrix $M = [A\mid\b]$ and call \li{sy.solve_linear_system()} on $M$.

SymPy matrices are defined with \li{sy.Matrix()}, with the same syntax as $2$-dimensional NumPy arrays.
For example, the following code solves the system given below.
\[
\begin{array}{rrrrrrr}
 x & + &   y & + &  z & = & 5 \\
2x & + &  4y & + & 3z & = & 2 \\
5x & + & 10y & + & 2z & = & 4
\end{array}
\]
\begin{lstlisting}
>>> x, y, z = sy.symbols('x y z')

# Define the augmented matrix M = [A|b].
>>> M = sy.Matrix([ [1,  1, 1, 5],
                    [2,  4, 3, 2],
                    [5, 10, 2, 4] ])

# Solve the system, providing symbolic variables to solve for.
>>> sy.solve_linear_system(M, x, y, z)
{x: 98/11, y: -45/11, z: 2/11}
\end{lstlisting}

SymPy matrices support the standard matrix operations of addition \li{+}, subtraction \li{-}, and multiplication \li{@}.
Additionally, SymPy matrices are equipped with many useful methods, some of which are listed below.
See \url{http://docs.sympy.org/latest/modules/matrices/matrices.html} for more methods and examples.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
Method & Returns \\ \hline
% \li{applyfunc()} & Apply a function to every element of the matrix. \\
% \li{condition_number()} & The condition number. \\
\li{det()} & The determinant. \\
% \li{diagonalize()} & Diagonalize a matrix. \\
\li{eigenvals()} & The eigenvalues and their multiplicities. \\
\li{eigenvects()} & The eigenvectors and their corresponding eigenvalues. \\
\li{inv()} & The matrix inverse. \\
\li{is_nilpotent()} & \li{True} if the matrix is nilpotent. \\
% \li{multiply_elementwise()} & Multiply matrices elementwise. \\
\li{norm()} & The Frobenius, $\infty$, $1$, or $2$ norm. \\
\li{nullspace()} & The nullspace as a list of vectors. \\
\li{rref()} & The reduced row-echelon form. \\
\li{singular_values()} & The singular values. \\
\end{tabular}
\end{table}

\begin{warn}
The \li{*} operator performs matrix multiplication on SymPy matrices.
To perform element-wise multiplication, use the \li{multiply_elementwise()} method instead.
\end{warn}

\begin{problem}
Find the eigenvalues of the following matrix by solving for $\lambda$ in the characteristic equation $\det(A - \lambda I) = 0$.
\begin{align*}
A =
\left[\begin{array}{cccc}
x-y & x   & 0   \\
x   & x-y & x   \\
0   & x   & x-y \\
\end{array}\right]
\end{align*}
Also compute the eigenvectors by solving the linear system $A - \lambda I = \0$ for each eigenvalue $\lambda$.
Return a dictionary mapping the eigenvalues to their eigenvectors.
\\(Hint: the \li{nullspace()} method may be useful.)

Check that $A\v = \lambda\v$ for each eigenvalue-eigenvector pair $(\lambda, \v)$.
Compare your results to the \li{eigenvals()} and \li{eigenvects()} methods for SymPy matrices.
\end{problem}

\section*{Calculus} % =========================================================

SymPy is also equipped to perform standard calculus operations, including derivatives, integrals, and taking limits.
Like other elements of SymPy, calculus operations can be temporally expensive, but they give exact solutions whenever solutions exist.

\subsection*{Differentiation} % -----------------------------------------------

The command \li{sy.Derivative()} creates a closed form, unevaluated derivative of an expression.
This is like putting $\frac{d}{dx}$ in front of an expression without actually calculating the derivative symbolically.
The resulting expression has a \li{doit()} method that can be used to evaluate the actual derivative.
Equivalently, \li{sy.diff()} immediately takes the derivative of an expression.

Both \li{sy.Derivative()} and \li{sy.diff()} accept a single expression, then the variable or variables that the derivative is being taken with respect to.

\begin{lstlisting}
>>> x, y = sy.symbols('x y')
>>> f = sy.sin(y)*sy.cos(x)**2

# Make an expression for the derivative of f with respect to x.
>>> df = sy.Derivative(f, x)
>>> print(df)
Derivative(sin(y)*cos(x)**2, x)

>>> df.doit()                       # Perform the actual differentiation.
-2*sin(x)*sin(y)*cos(x)

# Alternatively, calculate the derivative of f in a single step.
>>> sy.diff(f, x)
-2*sin(x)*sin(y)*cos(x)

# Calculate the derivative with respect to x, then y, then x again.
>>> sy.diff(f, x, y, x)
2*(sin(x)**2 - cos(x)**2)*cos(y)    # Note this expression could be simplified.
\end{lstlisting}

\begin{problem} % Single-variable optimization with differentiation.
Let $f:\mathbb{R}\rightarrow\mathbb{R}$ be a smooth function.
A \emph{critical point} of $f$ is a number $x_0\in\mathbb{R}$ satisfying $f'(x_0) = 0$.
The second derivative test states that a critical point $x_0$ is a local minimum of $f$ if $f''(x_0) > 0$, or a local maximum of $f$ if $f''(x_0) < 0$ (if $f''(x_0) = 0$, the test is inconclusive).

Now consider the polynomial
\[
p(x) = 2x^6 - 51x^4 + 48x^3 + 312x^2 - 576x - 100.
\]
Use SymPy to find all critical points of $p$ and classify each as a local minimum or a local maximum.
Plot $p(x)$ over $x\in[-5,5]$ and mark each of the minima in one color and the maxima in another color.
Return the collections of the x-values corresponding to the local minima and local maxima as sets.
\end{problem}

The \emph{Jacobian matrix} of a multivariable function $f:\mathbb{R}^n \rightarrow \mathbb{R}^m$ at a point $\x_0 \in \mathbb{R}^n$ is the $m \times n$ matrix $J$ whose entries are given by
\begin{equation*}
J_{ij} = \frac{\partial f_i}{\partial x_j}(\x_0).
\end{equation*}
For example, the Jacobian for a function $f:\mathbb{R}^3 \rightarrow \mathbb{R}^2$ is defined by
\[
J = \left[\begin{array}{c|c|c}
\arrayrulecolor{gray}
\frac{\partial f}{\partial x_1}&\frac{\partial f}{\partial x_2}&\frac{\partial f}{\partial x_3}
\end{array}\right]
=
\left[\begin{array}{ccc}
\frac{\partial f_1}{\partial x_1}&\frac{\partial f_1}{\partial x_2}&\frac{\partial f_1}{\partial x_3}
\\ \\
\frac{\partial f_2}{\partial x_1}&\frac{\partial f_2}{\partial x_2}&\frac{\partial f_2}{\partial x_3}
\end{array}\right],
\qquad
\text{where}
\qquad
f(\x) =
\left[\begin{array}{c}
f_1(\x) \\ f_2(\x)
\end{array}\right],
\quad
\x = \left[\begin{array}{c}
x_1 \\ x_2 \\ x_3
\end{array}\right].
\]

To calculate the Jacobian matrix of a multivariate function with SymPy, define that function as a symbolic matrix (\li{sy.Matrix()}) and use its \li{jacobian()} method.
The method requires a list of variables that prescribes the ordering of the differentiation.

\begin{lstlisting}
# Create a matrix of symbolic variables.
>>> r, t = sy.symbols('r theta')
>>> f = sy.Matrix([r*sy.cos(t), r*sy.sin(t)])

# Find the Jacobian matrix of f with respect to r and theta.
>>> J = f.jacobian([r,t])
>>> J
Matrix([
[cos(theta), -r*sin(theta)],
[sin(theta),  r*cos(theta)]])

# Evaluate the Jacobian matrix at the point (1, pi/2).
>>> J.subs({r:1, t:sy.pi/2})
Matrix([
[0, -1],
[1,  0]])

# Calculate the (symbolic) determinant of the Jacobian matrix.
>>> sy.simplify(J.det())
r
\end{lstlisting}

\subsection*{Integration} % ---------------------------------------------------

The function \li{sy.Integral()} creates an unevaluated integral expression.
This is like putting an integral sign in front of an expression without actually evaluating the integral symbolically or numerically.
The resulting expression has a \li{doit()} method that can be used to evaluate the actual integral.
Equivalently, \li{sy.integrate()} immediately integrates an expression.

Both \li{sy.Derivative()} and \li{sy.diff()} accept a single expression, then a tuple or tuples containing the variable of integration and, optionally, the bounds of integration.

\begin{lstlisting}
# Calculate the indefinite integral of sec(x).
>>> sy.integrate(sy.sec(x), x)
-log(sin(x) - 1)/2 + log(sin(x) + 1)/2

# Integrate cos(x)^2 from 0 to pi/2.
>>> sy.integrate(sy.cos(x)**2, (x,0,sy.pi/2))
pi/4

# Compute the integral of (y^2)(x^2) dx dy with x from 0 to 2, y from -1 to 1.
>>> sy.integrate(y**2 * x**2, (x,0,2), (y,-1,1))
16/9
\end{lstlisting}

\begin{problem}
% Many unpleasant integrals can be made tractable by a \emph{change of variables}, such as using polar, cylindrical, or spherical coordinates instead of the usual Cartesian system.
Let $f:\mathbb{R}^3\rightarrow\mathbb{R}$ be a smooth function.
The volume integral of $f$ over the sphere $S$ of radius $r$ can written in spherical coordinates as
\begin{align*}
\underset{S}{\iiint} f(x,y,z) dV = \int_0^\pi\int_0^{2\pi}\int_0^r f(h_1(\rho, \theta, \phi), h_2(\rho, \theta, \phi), h_3(\rho, \theta, \phi)) |\det(J)| \:d\rho\:d\theta\:d\phi,
\end{align*}
where $J$ is the Jacobian of the function $h:\mathbb{R}^3\rightarrow\mathbb{R}^3$ given by
\[
h(\rho, \theta, \phi) =
\left[\begin{array}{c}
h_1(\rho, \theta, \phi) \\
h_2(\rho, \theta, \phi) \\
h_3(\rho, \theta, \phi)
\end{array}\right]
=
\left[\begin{array}{c}
\rho\sin(\phi)\cos(\theta) \\
\rho\sin(\phi)\sin(\theta) \\
\rho\cos(\phi)
\end{array}\right].
\]
Calculate the volume integral of $f(x,y,z) = (x^2 + y^2 + z^2)^2$ over the sphere of radius $r$.
Lambdify the resulting expression (with $r$ as the independent variable) and plot the integral value for $r \in [0, 3]$.
In addition, return the value of the integral when $r=2$.
\\ (Hint: simplify the integrand before computing the integral.
In this case, $|\det(J)| = -\det(J)$.)

To check your answer, when $r=3$, the value of the integral is $\frac{8748}{7}\pi$.
\end{problem}

\begin{warn}
SymPy isn't perfect.
It solves some integrals incorrectly, simplifies some expressions poorly, and is significantly slower than numerical computations.
However, it is generally very useful for simplifying parts of an algorithm, getting exact answers, and handling tedious algebra quickly.
\end{warn}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Pretty Printing} % -----------------------------------------------

SymPy expressions, especially complicated ones, can be hard to read.
Calling \li{sy.init_printing()} changes the way that certain expressions are displayed to be more readable; in a Jupyter Notebook, the rendering is done with \LaTeX, as displayed below.
Furthermore, the function \li{sy.latex()} converts an expression into actual \LaTeX \ code for use in other settings.

\begin{center}
\includegraphics[width=\textwidth]{figures/pretty_printing.png}
\end{center}

% SymPy also has a simplified plotting wrapper around Matplotlib.
% See \url{http://docs.sympy.org/latest/modules/plotting.html} for further details.
% \begin{lstlisting}
% # Plot sin(x)e^x where x ranges from -3 to 3.
% >>> expr = sy.sin(x) * sy.exp(x)
% >>> sy.plot(expr, (x, -3, 3))
% \end{lstlisting}
%
% \begin{figure}[H]
% \includegraphics[width=.7\textwidth]{figures/plotting_wrapper.pdf}
% \end{figure}

\subsection*{Limits} % --------------------------------------------------------

Limits can be expressed, similar to derivatives or integrals, with \li{sy.Limit()}.
Alternatively, \li{sy.limit()} (lowercase) evaluates a limit directly.

\begin{lstlisting}
# Define the limit of a^(1/x) as x approaches infinity.
>>> a, x = sy.symbols('a x')
>>> sy.Limit(a**(1/x), x, sy.oo)
<<Limit(a**(1/x), x, oo, dir='-')>>

# Use the doit() method or sy.limit() to evaluate a limit.
>>> sy.limit((1+x)**(1/x), x, 0)
E

# Evaluate a limit as x approaches 0 from the negative direction.
>>> sy.limit(1/x, x, 0, '-')
-oo
\end{lstlisting}

Use limits instead of the \li{subs()} method when the value to be substituted is $\infty$ or is a singularity.

\begin{lstlisting}
>>> expr = x / 2**x
>>> expr.subs(x, sy.oo)
nan
>>> sy.limit(expr, x, sy.oo)
0
\end{lstlisting}

Refer to \url{http://docs.sympy.org/latest/tutorial/calculus.html} for SymPy's official documentation on calculus operations.

\begin{comment} % Too much additional material already.
\subsection*{Series Expansions} % ---------------------------------------------

Series can be depicted using the summation function, but SymPy also has built-in functions to depict them.
Expressions involving series can be created using \li{series()} as a function or as a method.
The function takes in the expression, variable, the point the series is centered at, and the order of the series.
The point the series is centered at defaults to zero and the order to six.

\begin{lstlisting}
# Find the terms in the series for cos(x) centered around x = 0 of order 6.
>>> sy.series(sy.cos(x),x)
1 - x**2/2 + x**4/24 + O(x**6)

# Center the series at x = pi of order 4.
>>> sy.series(sy.cos(x),x,sy.pi, 4)
-1 + (x - pi)**2/2 + O((x - pi)**4, (x,pi))

# Use removeO() to remove the order from the output.
>>> sy.series(sy.exp(sy.cos(x)),x,1,2).removeO()
-(x - 1)*exp(cos(1))*sin(1) + exp(cos(1))
\end{lstlisting}

Power series can be represented in SymPy with the ``Formal Power Series'' class.
When an \li{fps} object is created, it returns sequences of coefficients and powers, followed by the independent and then the common terms.
Use the \li{truncate()} or \li{polynomial()} methods to visualize individual terms of the series.
These methods input the highest order to be visualized and default to six.

\begin{lstlisting}
>>> f = sy.fps(sy.cos(x),x)
>>> f.truncate()
1 - x**2/2 + x**4/24 + O(x**6)
>>> f.polynomial()
x**4/24 - x**2/2 + 1
\end{lstlisting}

Further series features in SymPy include the computation of residues, laurent series and fourier series.
\end{comment}

\subsection*{Numerical Integration} % -----------------------------------------

Many integrals cannot be solved analytically.
% Additionally, the definite integral of a function may be undefined.
% If SymPy is unable to compute an integral, it returns an integral object.
As an alternative to the \li{doit()} method, the \li{as_sum()} method approximates the integral with a summation.
This method accepts the number of terms to use and a string indicating which approximation rule to use (\li{"left"}, \li{"right"}, \li{"midpoint"}, or \li{"trapezoid"}).

\begin{lstlisting}
>>> x = sy.symbols('x')

# Try integrating e^(x^2) from 0 to pi.
>>> I = sy.Integral(sy.exp(x**2), (x,0,sy.pi))
>>> I.doit()
sqrt(pi)*erfi(pi)/2                 # The result is not very helpful.

# Instead, approximate the integral with a sum.
>>> I.as_sum(10, 'left').evalf()
1162.85031639195
\end{lstlisting}

% In addition, SymPy includes several integral transforms, such as the Laplace, Fourier, sine, and cosine transforms.
% It can also do several types of integrals, such as line integrals.
See \url{http://docs.sympy.org/latest/modules/integrals/integrals.html} for more documentation on integration with SymPy.

\subsection*{Differential Equations} % ----------------------------------------

SymPy can be used to solve both ordinary and partial differential equations.
The documentation for working with PDE functions is at \url{http://docs.sympy.org/dev/modules/solvers/pde.html}

The general form of a first-order differential equation is $\frac{dx}{dt} = f(x(t),t)$.
To represent the unknown function $x(t)$, use \li{sy.Function()}.
Just as \li{sy.solve()} is used to solve an expression for a given \textbf{variable}, \li{sy.dsolve()} solves an ODE for a particular \textbf{function}.
When there are multiple solutions, \li{sy.dsolve()} returns a list; when arbitrary constants are involved they are given as \li{C1}, \li{C2}, and so on.
Use \li{sy.checkodesol()} to check that a function is a solution to a differential equation.

\begin{lstlisting}
>>> t = sy.symbols('t')
>>> x = sy.Function('x')

# Solve the equation x''(t) - 2x'(t) + x(t) = sin(t).
>>> ode = x(t).diff(t, t) - 2*x(t).diff(t) + x(t) - sy.sin(t)
>>> sy.dsolve(ode, x(t))
Eq(x(t), (C1 + C2*t)*exp(t) + cos(t)/2) # C1 and C2 are arbitrary constants.
\end{lstlisting}

Since there are many types of ODEs, \li{sy.dsolve()} may also take a hint indicating what solving strategy to use.
See \li{sy.ode.allhints} for a list of possible hints, or use \li{sy.classify_ode()} to see the list of hints that may apply to a particular equation.

% Old problem.
\begin{comment}
\begin{problem}
Return the solution to the following differential equation.
\begin{equation*}
 y_{xxx} + 3y_{xx} + 3y_{x} + y = x^{5}e^x + x^{4}\sin\left(x\right) + x^{3}\cos\left(3x\right)
\end{equation*}

Check the result $y(x)$ by plugging it back into an expression where one side of the equation is subtracted to another.
Use the \li{doit()} and \li{simplify()} methods to ensure that the new expression simplifies to $0$.
\end{problem}
\end{comment}


\newpage
%!TEX root = 497Notes-Temple.tex
\section{Batch Normalization in DNN and CNN}
Recall the classical (fully connected)  artificial deep neural network (DNN) $f^L$ in \eqref{compress-dnn}. We can define
%\begin{equation}\label{key}
%\mathcal{F}_\text{DNN}:= \{ f(x;\Theta) = \text{softmax} \circ f^J(x;\Theta) ~|~ x\in\mathbb{R}^n,\Theta \in \mathbb{R}^{\mathcal N} \}.
%\end{equation}
\begin{equation}\label{key}
\rm{DNN}_L:= \{  f^L(x;\Theta) ~|~ \Theta \in \mathbb{R}^{\mathcal N} \}.
\end{equation}
Or, we have a more comprehensive notation for classical DNN models.
\begin{equation}\label{eq:DNNdef_J}
\begin{aligned}
{\rm{DNN}_L} :=\{& f:f=
\theta^L \circ \sigma \circ \theta^{L-1} \cdots \sigma \circ \theta^1(x), \\
&\theta^\ell \in \mathbb{R}^{n^{\ell+1} \times (n^\ell+1)}, \quad n^0 = d, \quad n^{L} = 1, \quad n^\ell \in \mathbb{N}^+\}.
\end{aligned}
\end{equation}
Generally speaking, a deep learning problem consists of the next few components:
\begin{equation}\label{DL-process}
\text{Data}  \rightarrow \text{Model} \rightarrow \text{Training} \rightarrow \text{Testing}.
\end{equation}
We will try to give a different explanation for batch normalization based on the above outline.

\subsection{Ideas Behind the BN for DNN: Internal \mbox{Covariate} Shift in Training}
The { Internal Covariate Shift} is defined in \cite{ioffe2015batch} as
the change in the distribution of network activations
due to the change in network parameters during training.

For example, considering the training data with
$X=\{(x^i,y^i)~|~i=1,..,N\}\subset \mathcal{X} : =\{(x,y)~|~x,\in\mathbb{R}^n, y \in \mathbb{R}^c\}$,
then output of $\nu$-th layer (the input of activation function in $\nu+1$-the layer) $\{f^\nu ({x^i}) \}_{i=1}^N$
will satisfy different discrete distributions if you have different parameters $\Theta$ which
always happened during training process.

More intuitively, fixed distribution of inputs to a sub-network would have positive
consequences for the layers {\em outside} the sub-network, as
well. Consider a layer with a sigmoid activation function $f =
\sigma(W x+b)$ where $x$ is the layer input, the weight matrix $W$ and
bias vector $b$ are the layer parameters to be learned, and $\sigma(x) =
\frac{1}{1+e^{-x}}$.  It is easy to see
$$
\sigma'(x) \to 0 \quad \text{if} \quad |x| \to \infty.
$$
This means that for all dimensions of $y = Wx+b$ except those with
small absolute values, the gradient flowing down to $x$ will vanish
and the model will train slowly. However, since $y$ is affected by
$W, b$ and the parameters of all the layers below, changes to those
parameters during training will likely move many dimensions of $y$
into the saturated regime of the nonlinearity and slow down the
convergence. This effect is amplified as the network depth
increases. In practice, the saturation problem and the resulting
vanishing gradients are usually addressed by using ${\rm ReLU}(x)=\max(x,0)$, careful initialization and small learning rates.
However, if we could ensure that the distribution of nonlinearity inputs
remains more stable as the network trains, then the optimizer would be
less likely to get stuck in the saturated regime, and the training
would accelerate.

In summary, the gradient vanishes in the following two situations:
\begin{itemize}
	\item Saturated problem:
	$$
	\sigma'(x) \to 0 \quad \text{for $x$ located in some saturated domain}.
	$$
	\item Instability from multiplication:
	$$
	\nabla_{W^1}  \ell(f^L(x^i;\Theta), y^i) \approx \Pi_{\ell=1}^L W^{\ell} {\rm Diag}(\sigma'(f^{\ell-1})) \to 0~ , \quad \text{ if $L$ is big}.
	$$
\end{itemize}

Under this principle,
it has been long known \cite{lecun1998neural, wiesler2011convergence}
that the network training converges faster if its inputs are whitened, i.e.,
linearly transformed to have zero means and unit variances, and  decorrelated.

Thus to say, to improve the training,  one may seek to reduce the internal covariate shift.
By fixing the distribution of the layer inputs $f^\ell$ as the training progresses,
it is expected to improve the training speed. As each layer
observes the inputs produced by the layers below, it would be advantageous to
achieve the same whitening of the inputs of each layer.  By whitening the
inputs to each layer, BN would take a step towards achieving the fixed
distributions of inputs that would remove the ill effects of the internal covariate shift.

Within this framework, whitening the layer inputs is expensive, as it requires
computing the covariance matrix
\begin{equation}
{\rm Cov}[f^\ell] = \mathbb E_{ x \in X} [f^\ell(x) [f^\ell(x)]^T]-\mathbb E_{ x \in X} [f^{\ell}(x)]\mathbb E_{ x \in X} [f^\ell(x)]^T
\end{equation}
and its inverse square root, to produce the whitened activations
\begin{equation}
\tilde  f^\ell = \left({\rm Cov}[f^\ell \right)^{-1/2}\left(f^\ell-\mathbb E[f^\ell]\right),
\end{equation}
as well as the derivatives of these transforms for backpropagation.
However, this is impossible to take gradient for $\left({\rm Cov}[f^\ell] \right)^{-1/2}$
w.r.t $\Theta$ as $ f^\ell = f^\ell(x; \Theta)$.

The original version is to do whitening in Batch i.e. $\mathbb E_{ x \in X} [\cdot ]$, that. This
is why this is called Batch Normalization not mini-batch normalization which
is the practical version of batch normalization.

\subsection{Practical batch normalization: assume i.i.d and add scale and shift }
Since the full whitening of each layer's inputs is costly and not
everywhere differentiable, BN makes two necessary simplifications.

\paragraph{Take batch normalization for each scalar feature (neuron).}

The first is that instead of whitening the features in layer
inputs and outputs {\bf jointly}, batch normalization will normalize each {\bf scalar feature (neuron)}
independently, by making it have the mean of zero and the variance of
1.   Assume the distribution of
scalar features to be i.i.d.

For a layer with $n^\ell$-dimensional input $f^\ell = (f^\ell_1, \ldots,  f^\ell_{n^\ell})$, we
will normalize each dimension
$$ \hat{f}^\ell_k = \frac{f^\ell_k-\mathbb E [ f^\ell_k]}{
	\sqrt{{\rm Var}[f^\ell_k]}}$$
where the expectation and variance are
computed over the training data set. As shown in
\cite{lecun1998neural}, such normalization speeds up convergence,
even when the  features are not decorrelated.

\paragraph{Add scale and shift into batch normalization.}
Note that simply normalizing each input of a layer may change what the
layer can represent. For instance, normalizing the inputs of a
sigmoid would constrain them to the linear
regime of the nonlinearity. To address this, we make sure that {\em the transformation inserted in
	the network can represent the identity transform}.  To
accomplish this, we introduce, for each activation $\hat f^\ell_k$, a pair of
parameters $\gamma^\ell_k, \beta^\ell_k$, which scale and shift the
normalized value:
\begin{equation}
\tilde{f^\ell}_k  = \gamma^\ell_k \hat{f^\ell}_k +\beta^\ell_k.
\end{equation}
These parameters are learned along with the original model
parameters, and restore the representation power of the
network. Indeed, by setting $\gamma^\ell_k = \sqrt{{\rm Var}[f^\ell_k]}$ and
$\beta^\ell_k = \mathbb E [f^\ell_k]$, we could recover the original activations, if that were the optimal thing to do.

\subsection{Batch normalization for DNN}
\paragraph{Definition of batch normalization operation based on the batch}
Following the idea in \eqref{DL-process}, we consider that we have the all
training data as
\begin{equation}\label{eq:trainingdata}
(X,Y) := \{x^i, y^i\}_{i=1}^N.
\end{equation}
Since the normalization is applied to
each activation independently, let us focus on a particular activation $ f^\ell_k$ and omit $k$ as $f^\ell$ for clarity.
We have $N$ values of this activation
in the batch,
$$ X=\{x_1, \cdots, x_N\}.$$
Let the normalized values be
$\hat f^\ell$, and their linear transformations be $\tilde f^\ell$.
\begin{equation}\label{def:BNeq}
\begin{aligned}
\mu^\ell_{ X} & \leftarrow \mathbb{E}_{x \sim X} [f^\ell(x)] =  \frac{1}{N}\sum_{i=1}^N f^\ell(x^i) & \text{ batch mean}& \\
\sigma^\ell_{ X} & \leftarrow \mathbb{E}_{x \sim X}  \left[(f^\ell(x)-\mathbb{E}_{x \sim X}[ f^\ell(x)])^2 \right] = \frac{1}{N}\sum_{i=1}^N (f^\ell(x^i)-\mu_{ X}^\ell)^2 &  \text{ batch variance}&\\
\hat f^\ell (x) & \leftarrow \frac{f^\ell(x)-\mu^\ell_{ X}}{\sqrt{\sigma^\ell_{ X}+\epsilon}}   &\text{ normalize}&\\
\tilde f^\ell(x)  & \leftarrow \gamma^\ell \hat f^\ell (x) + \beta^\ell
&\text{ scale and shift}&
\end{aligned}
\end{equation}
Here we note that all these operations in the previous equation are defined  element-wise.
Then at last, we define the batch normalization operation based on the batch set as
\begin{equation}\label{eq:BNop}
{\rm BN}_{X}({f^\ell(x)}) = \tilde f^\ell(x) := \gamma^\ell  \frac{f^\ell(x)-\mu^\ell_{ X}}{\sqrt{\sigma^\ell_{ X}+\epsilon}}  + \beta^\ell  ,
\end{equation}
where $\tilde f^\ell(x)$, $\mu^\ell_{ X}$ and $\sigma^\ell_{ X}$  are given above.

\paragraph{Model with batch normalization}
Then we have the new DNN model with batch normalization as:
\begin{equation}\label{nn-BN0}
\begin{cases}
\tilde f^1(x^i)&= (\theta^1 (x^i) ),\\
\tilde f^\ell &= \theta^{\ell} \circ \sigma \circ {\rm BN}_{ X}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
%Here we would like to notice that all these input data
%$x_i \in X$, it is already normalized over all the original data set with
%the same fashion.
%Thai is to say
%$$
%x_i = {\rm BN}_{\bar X}(\bar x_i),
%$$
%for the original data $\bar x_i \in \bar X$.
For a more comprehensive notation,
we can use the next notation
\begin{equation}
\sigma_{\rm BN} := \sigma \circ {\rm BN}_{ X}.
\end{equation}
We can remove the basis $b^\ell$ in $\theta^\ell$, thus to say the real model we use should be
\begin{equation}\label{nn-BN}
\begin{cases}
\tilde f^1(x^i)&= W^1 x^i ,\\
\tilde f^\ell &= W^{\ell}  \sigma_{\rm BN}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
Combine the two definitions, we note
\begin{equation}
\tilde \Theta := \{W, \gamma, \beta\},
\end{equation}
where
$W = \{W^1, \cdots, W^l \}$, $\gamma := \{\gamma^2, \cdots, \gamma^L\}$ and $\beta := \{\beta^2, \cdots, \beta^L\}$.
Finally, we have the loss function
\begin{equation}\label{eq:loss-BN}
\mathcal L(\tilde \Theta) = \mathbb{E}_{(x,y)\sim (X,Y)} \approx \frac{1}{N}\sum_{i=1}^N \ell(\tilde f^L(x^i; \tilde \Theta), y^i).
\end{equation}
A key observation in \eqref{eq:loss-BN} and the new BN model \eqref{nn-BN} is that
\begin{equation}\label{eq:threeExpectation}
\begin{aligned}
\mu^\ell_{ X}
%& \leftarrow \frac{1}{N}\sum_{i=1}^N f^\ell(x_i)
&= \mathbb{E}_{x \sim X} [f^\ell(x)],\\
\sigma^\ell_{ X}
%& \leftarrow \frac{1}{N}\sum_{i=1}^N (f^\ell(x_i)-\mu_{ X})^2
&=  \mathbb{E}_{x \sim X}  \left[(f^\ell(x)-\mathbb{E}_{x \sim X}[ f^\ell(x)])^2 \right], \\
\mathcal L(\tilde \Theta)
%&= \sum_{i=1}^N L(\tilde f^J(x_i; \tilde \Theta), y_i)
&=  \mathbb{E}_{(x,y)\sim (X,Y)}  \left[\ell(\tilde f^L(x^i; \tilde \Theta), y^i) \right].
\end{aligned}
\end{equation}
Here we need to mention that
$$
x \sim X
$$
means $x$ subjects to the discrete distribution of all data $X$.
%For example:
%$$
%{p}_{X}(x = x_i) = \frac{1}{N}.
%$$
%or more mathematically we can use the Dirac distribution as:
%$$
%p_{X}(x) = \frac{1}{N} \sum_{i=1}^N\delta(x - x_i).
%$$

%Similar with the case from (batch) gradient descent to stochastic gradient descent.
%Therefore, BN makes the second simplification: since we use mini-batches
%in stochastic gradient training, {\em each mini-batch produces estimates of the mean and variance} of each
%activation for whole batch.

\subsection{Batch normalization: some ``modified" SGD training algorithm}
Following the key observation in \eqref{eq:threeExpectation}, and recall the
similar case in SGD, we do the the sampling trick in \eqref{eq:loss-BN} and
obtain the mini-batch SGD:
\begin{equation}\label{eq:mini-batch-sample}
x \sim X \approx x \sim \mathcal B,
\end{equation}
here $\mathcal B$ is a mini-batch of batch $X$ with $\mathcal B \subset X$.

However, for problem in \eqref{eq:loss-BN}, it is very difficult to find some
subtle sampling method because of the composition of $\mu^\ell_{\mathcal X}$
and $[\sigma^\ell_{\mathcal X}]$. However, one simple way for sampling
\eqref{eq:loss-BN} can be chosen as taking \eqref{eq:mini-batch-sample} for
all the expectation case in \eqref{eq:loss-BN} and \eqref{eq:threeExpectation}.
This is to say, in training process ($t$-th step for example), once we choose $B_t \subset X$
as the mini-batch, then the model becomes
\begin{equation}\label{nn-BN-training}
\begin{cases}
\tilde f^1(x^i)&= W^1 x^i ,\\
\tilde f^\ell &= W^{\ell}  \sigma_{\rm BN}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
where
\begin{equation}
\sigma_{\rm BN} := \sigma \circ {\rm BN}_{\mathcal B_t},
\end{equation}
or we can say that $X$ is replaced by $\mathcal B_t$ in this case.
Here ${\rm BN}_{\mathcal B_t}$ is defined by
\begin{equation}\label{def:BNeq-traning}
\begin{aligned}
\mu^\ell_{\mathcal B_t} & \leftarrow \frac{1}{m}\sum_{i=1}^m f^\ell(x^i) & \text{ mini-batch mean}& \\
\sigma^\ell_{\mathcal B_t} & \leftarrow \frac{1}{m}\sum_{i=1}^m (f^\ell(x^i)-\mu_{\mathcal B_t})^2 &  \text{ mini-batch variance}&\\
\hat f^\ell (x) & \leftarrow \frac{f^\ell(x)-\mu^\ell_{\mathcal B_t}}{\sqrt{\sigma^\ell_{\mathcal B_t}+\epsilon}}   &\text{ normalize}&\\
{\rm BN}_{\mathcal B_t}(\tilde f^{\ell}) := \tilde f^\ell(x) & \leftarrow \gamma^\ell \hat f^\ell (x) + \beta^\ell
&\text{ scale and shift}&
\end{aligned}
\end{equation}
Here batch normalization operation introduces some new parameters such as $\gamma$ and $\beta$.
Thus to say, for training phase, if we choose mini-batch as $\mathcal B_t$ in $t$-th training
step, we need to take gradient
\begin{equation}
\frac{1}{m}\nabla_{\tilde \Theta} \sum_{i \in \mathcal B_t} \ell(\tilde f^L(x^i; \tilde \Theta), y^i),
\end{equation}
which requires the gradient of $\mu_{B_t}^\ell$ or $[\sigma_{B_t}^\ell]$
w.r.t $W^i$ for $i \le \ell$.
To derive the new gradient formula for batch normalization step,
$$
\mu^\ell_{\mathcal B_t}, \quad \text{and} \quad \sigma^\ell_{\mathcal B_t},
$$
contain the output of $\tilde f^{\ell-1}$.
This is exact the batch normalization method described in \cite{ioffe2015batch}.

\subsection{Final model  with BN in DNN after training}
One key problem is that, in the batch normalization operator, we need to compute the mean and variance
in a data set (batch or mini-batch). However, in the inference step, we just input one data
into this DNN, how to compute the batch normalization operator in this situation.

Actually,  the $\gamma$ and $\beta$ parameter is fixed after training, the only problem is
to compute the mean $\mu$ and variance $\sigma$.
All the mean $ \mu_{\mathcal B_t}  $ and variance $\sigma_{\mathcal B_t} $ during the training phase
are just the approximation of the mean and variance of whole batch i.e.  $ \mu_{X}  $ and $\sigma_{X}$
as shown in \eqref{eq:threeExpectation}.

One natural idea might be just use the batch normalization operator w.r.t to the whole training data set, thus to say
just compute $\mu_X$ and $\sigma_X$ by definition in \eqref{def:BNeq}.

However, there are at least the next few problems:
\begin{itemize}
\item computation cost,
\item ignoring the statistical approximation (don't make use of the $ \mu_{\mathcal B_t}  $ and  $\sigma_{\mathcal B_t} $ in training phase).
\end{itemize}

Considering that we have the statistical approximation for $ \mu_{X}  $ and $\sigma_{X}$
during each SGD step, moving average might be a more straightforward way.
Thus to say, we define the $\mu^\ell$ and $[\sigma^\ell]$ for the inference (test) phase
as
\begin{equation}
\mu^\ell = \frac{1}{T}\sum_{t=1}^T \mu^\ell_{\mathcal B_t},
\quad \sigma^\ell =\frac{1}{T} \frac{m}{m-1}\sum_{t=1}^T\sigma^\ell_{\mathcal B_t}.
\end{equation}
Here we take Bessel's correction for unbiased variance.
The above moving average step is found in the original paper of batch normalization in \cite{ioffe2015batch}.

Another way to do this is to call the similar idea in momentum.
At each time step we update the running averages for mean and variance using an exponential decay based on the momentum parameter:
\begin{equation}
\begin{aligned}
\mu^\ell_{\mathcal B_t} &=\alpha \mu^\ell_{\mathcal B_{t-1}} + (1-\alpha)\mu^\ell_{\mathcal B_t} , \\
\sigma^\ell_{\mathcal B_t} &=\alpha \sigma^\ell_{\mathcal B_{t-1}} + (1-\alpha)\sigma^\ell_{\mathcal B_t} .
\end{aligned}
\end{equation}
$\alpha$ is close to $1$, we can take it as $0.9$ generally. Then we all take bath mean and variance as
$\mu^\ell_X  \approx \mu^\ell_{\mathcal B_T}$  and $\sigma^\ell_X  \approx \sigma^\ell_{\mathcal B_T}$.

Many people argue that the variance here should also use Bessel's correction.

\subsection{Batch Normalization for CNN}
One key idea in batch normalization is to do normalization with each scalar features (neurons)
separately along a mini-batch.
Thus to say, we need to identify what is neuron in CNN.
This is a historical problem, some people think neuron in CNN
should be the pixel in each channel, some other people think that each channel is just
one neuron. Batch normalization chooses the later one.
One important reason for this choice is the fact of computation cost.

For convolutional layers, batch normalization additionally requires the normalization
to  obey the convolutional property so that different elements
of the same feature map, at different locations, are normalized in the
same way.
To compute $\mu^\ell_{\mathcal B_t}$, we take mean of the set of all values in a feature map across both the
elements of a mini-batch and spatial locations, so for a mini-batch
of size $m$ and feature maps of size $m_\ell \times n_\ell$ (image geometrical size),
we use the effective mini-batch of size $ m m_\ell n_\ell$.
We learn a pair of parameters $\gamma_k$ and $\beta_k$ per feature map (k-th channel), rather than per activation.

For simplicity, then have the following batch normalization scheme for CNN
\begin{equation}\label{def:BNeq-traningCNN}
\begin{aligned}
\mu^{\ell, j}_{\mathcal B_t} & \leftarrow \frac{1}{m \times m_\ell \times n_\ell }\sum_{i=1}^m \sum_{1\le s\le m_\ell, 1\le t \le n_\ell} f^{\ell, j}_{s,t}(x^i)
 &&\text{ mean on channel }j \\
\sigma^{\ell, j}_{\mathcal B_t} & \leftarrow \frac{1}{m \times m_\ell \times n_\ell }\sum_{i=1}^m \sum_{1\le s\le m_\ell, 1\le t \le n_\ell}
 (f^{\ell, j}_{s,t}(x^i) - \mu^{\ell, j}_{\mathcal B_t})^2   &&\text{ variance on channel }j\\
\hat f^{\ell, j}_{s,t} (x) & \leftarrow \frac{f^{\ell, j}_{s, t}(x)] - \mu^{\ell, j}_{\mathcal B_t}}{\sqrt{\sigma^{\ell, j}_{\mathcal B_t} +\epsilon}}   &&\text{ normalize }\\
[{\rm BN}_{\mathcal B_t}(\tilde f^{\ell})]_{j;st} &:= \tilde f^{\ell, j}_{s,t}(x)  \leftarrow \gamma^{\ell,j} \hat f^{\ell, j}_{s,t} (x)  + \beta^{\ell,j}
&&\text{ scale and shift on channel}
\end{aligned}
\end{equation}

\subsection{Batch normalization for MgNet}
When we train MgNet, we also adopt the batch normalization mechanism.
This means that, when we do training for MgNet, we
apply the following basic block in MgNet
\begin{equation}\label{key}
u^{\ell,\nu} \leftarrow u^{\ell,\nu-1} + {\rm ReLU} \circ {\rm BN}_{\mathcal B_t} \circ B^{\ell,\nu}\ast {\rm ReLU} \circ {\rm BN}_{\mathcal B_t} ({f^\ell -  A^{\ell} \ast u^{\ell,\nu-1}}),
\end{equation}
or simply
\begin{equation}\label{key}
u^{\ell,\nu} \leftarrow u^{\ell,\nu-1} + {\rm ReLU}_{\rm BN} \circ B^{\ell,\nu}\ast {\rm ReLU}_{\rm BN} ({f^\ell -  A^{\ell} \ast u^{\ell,\nu-1}}),
\end{equation}
where
\begin{equation}\label{key}
{\rm ReLU}_{\rm BN} =  {\rm ReLU} \circ {\rm BN}_{\mathcal B_t}.
\end{equation}


\section*{Bayesian decision theory}
Given:\\
- Conditional distribution over labels $P(y|x)$\\
- Set of actions $\mathcal{A}$\\
- Cost function $C:Y\times \mathcal{A} \rightarrow \mathbb{R}$\\
Pick action that minimizes the expected cost:
$a^* = \underset{a \in \mathcal{A}}{\operatorname{argmin}} \mathbb{E}_y[C(y,a)|x] = \sum_y P(y|x) \cdot C(y,a)$
\subsection*{Optimal decision for logistic regression}
$a^* = \underset{y}{argmax} \hat{P}(y|x) = sign(w^T x)$

% \subsection*{Example: logistic regression}
% \begin{itemize}
% 	 Est. cond. dist: $P(y|x,w) = Ber(\sigma(w^Tx))$
% 	 Action set: $\mathcal{A} = \{ +1, -1\}$
% 	 Cost function: $C(y,a) = [y \not = a]$ \\
% 	$= \left \{
% 		\begin{array}{lr}
% 			1 \text{ , if } y \not = a\\
% 			0 \text{ , otherwise}
% 		\end{array}
% 		$
% \end{itemize}
% $a^* = \underset{a \in \mathcal{A}}{\operatorname{argmin}} \mathbb{E}_y[C(y,a)|x]$\\
% $= \underset{a \in \mathcal{A}}{\operatorname{argmin}} P(y=1|x,w)[a=1] + P(Y=-1|x,w)[a=+1]$\\
% $= \underset{a \in \mathcal{A}}{\operatorname{argmin}} P(y \not = a | x,w) = \underset{a \in \mathcal{A}}{\operatorname{argmin}} \frac{1}{1+exp(aw^Tx)}$\\
% $=\underset{a \in \mathcal{A}}{\operatorname{argmax}} (1 + exp(aw^Tx))$
% $=\underset{a \in \mathcal{A}}{\operatorname{argmax}} (aw^Tx)$\\
% $= sign (w^Tx)$

%\subsection*{Example: Asymmetric costs}
%	 Est. cond. dist: $\hat{P}(y|x,w) = Ber(\sigma(w^Tx))$
%	 Action set: $\mathcal{A} = \{ +1, -1\}$
%	 Cost function: $C(y,a) =
%	 \begin{cases}
%	 	c_{FP} \text{ , if $y=-1$ and $y=+1$}\\
%			c_{FN} \text{ , if $y=+1$ and $y=-1$}\\
%			0 \text{ , otherwise}
%	 \end{cases}
%		$
%The action that minimizes the expected cost is:\\
%$C_+ = \mathbb{E}_y[C(y,+1)|x] = P(y=+1|x) \cdot 0 + (P(y=-1)|x) \cdot c_{FP}$\\
%$C_- = \mathbb{E}_y[C(y,-1)|x] = P(y=+1|x) \cdot c_{FN} + P(y=-1|x) \cdot 0$\\
%Predict +1 if $C_+ \leq C_- \Leftrightarrow P(y=+1|x) \geq \frac{c_{FP}}{c_{FP} + c_{FN}}$

\subsection*{Doubtful logistic regression}
	 Est. cond. distr.: $\hat{P}(y|x) = Ber(y;\sigma(\hat{w}^Tx))$\\
	 Action set: $\mathcal{A} = \{ +1, -1, D\}$;  Cost function:\\
	 $C(y,a) = \begin{cases}
			[y \neq a] &\text{if } a \in \{+1,-1\}\\
			c &\text{if } a = D
       \end{cases}$\\
The action that minimizes the expected cost\\
$a^* = y \text{ if } \hat{P}(y|x) \geq 1-c\text{, D otherwise}$
%$a^* = \begin{cases}
%		y &\hat{P}(y|x) \geq 1-c\\
%		D &\text{otherwise}
%	   \end{cases}
%$

\subsection*{Linear regression}
	 Est. cond. distr.: $\hat{P}(y|x,w) = \mathcal{N}(y;w^Tx, \sigma^2)$\\
 $\mathcal{A} = \mathbb{R}$; $C(y,a) = (y-a)^2$\\
The action that minimizes the expected cost\\
$a^* = \mathbb{E}_y[y|x] = \int \hat{P}(y | x) \partial y = \hat{w}^Tx$

\subsection*{Asymmetric cost for regression}
	 Est. cond. distr.: $\hat{P}(y|x) = \mathcal{N}(\hat{y};\hat{w}^Tx, \sigma^2)$\\
	 $\mathcal{A} = \mathbb{R}$; $C(y,a) = c_1 \max(y-a,0) + c_2 \max(a-y,0)$
Action that minimizes the expected cost:\\
$a^* = \hat{w}^Tx + \sigma \Phi^{-1} (\frac{c_1}{c_1 + c_2})$, $\Phi$: Gaussian CDF\\
%$\frac{\partial}{\partial a}$\\
%$\frac{\partial}{\partial a} \mathbb{E}_y[C(y,a)|x] = \int_{-\infty}^{\infty} C(y,a) P(y | x) dy \overset{!}{=} 0$\\
%$= -c_1 \int_{a}^{\infty} P(y | x) dy + c_2 \int_{-\infty}^{a} P(y | x) dy$\\
%$= -c_1 [1-\phi(a; w^Tx, \sigma^2)] + c_2 \phi(a; w^Tx, \sigma^2)$\\
%$\phi(a; w^Tx, \sigma^2) = \frac{c_1}{c_1 + c_2}$\\
%using $\phi(u;v,w) = \phi((u-v)/\sqrt{w};0,1)$ and applying inverse CDF of std. ND $\phi^{-1}$ we get\\
%$a^* = w^Tx + \sigma\phi^{-1} (\frac{c_1}{c_1 + c_2})$


\subsection{Intervals of Increase/Decrease, and the First Derivative Test}\label{sec:FirstDer}
The method of Section~\ref{subsec:LocalExtremaSubsection} for deciding whether there is a
local maximum or minimum at a critical value is not always
convenient. We can instead use information about the derivative
$f'(x)$ to decide; since we have already had to compute the derivative
to find the critical values, there is often relatively little extra
work involved in this method.

How can the derivative tell us whether there is a maximum, minimum, or
neither at a point? Suppose that $f$ is differentiable at and around $x=a$, and suppose further that $a$ is a critical point of $f$. Then we have several possibilities:
\begin{enumerate}
\item	There is a local maximum at $x=a$. This happens if $f'(x)>0$ as we approach $x=a$ from the left (i.e. when $x$ is in the vicinity of $a$, and $x<a$) and $f'(x)<0$ as we move to the right of $x=a$ (i.e. when $x$ is in the vicinity of $a$, and $x>a$).
\item	There is a local minimum at $x=a$. This happens if $f'(x)<0$ as we approach $x=a$ from the left (i.e. when $x$ is in the vicinity of $a$, and $x<a$) and $f'(x)>0$ as we move to the right of $x=a$ (i.e. when $x$ is in the vicinity of $a$, and $x>a$).
\item	There is neither a local maximum or local minimum at $x=a$. If $f'(x)$ does not change from negative to positive, or from positive to negative, as we move from the left of $x=a$ to the right of $x=a$ (that is, $f'(x)$ is positive on both sides of $x=a$, or negative on both sides of $x=a$) then there is neither a maximum nor minimum when $x=a$.
\end{enumerate}
See the first graph in Figure~\ref{fig:max and min points}
and the graph in Figure~\ref{fig:non extremum}
for examples.

\begin{example}{Local Maximum and Minimum}{localmaxandmin}
Find all local maximum and minimum points for $f(x)=\sin x+\cos
x$ using the first derivative test.
\end{example}

\begin{solution}
The derivative is $f'(x)=\cos
x-\sin x$ and from Example~\ref{max and min} the critical
values we need to consider are $\pi/4$ and $5\pi/4$.

We analyze the graphs of $\sin x$ and $\cos x$.
Just to the left of $\pi/4$ the cosine is larger than the
  sine, so $f'(x)$ is positive; just to the right the cosine is
  smaller than the sine, so $f'(x)$ is negative. This means there is a
  local maximum at $\pi/4$. Just to the left of $5\pi/4$ the cosine is
  smaller than the sine, and to the right the cosine is larger than
  the sine. This means that the derivative $f'(x)$ is negative to the
  left and positive to the right, so $f$ has a local minimum at
  $5\pi/4$.
\end{solution}

The above observations have obvious intuitive appeal as
you examine the graphs in Figures~\ref{fig:max and min points} and \ref{fig:non extremum}. We can extend these ideas
further and then formulate and prove a theorem: If the graph of $f$ is
increasing before (i.e., to the left of) $x=a$ and decreasing after (i.e.,
to the right of) $x=a,$ then there is a local maximum at $x=a.$ If the graph
of $f$ is decreasing before $x=a$ and increasing after $x=a,$ then there is
a local minimum at $x=a.$ If the graph of $f$ is consistently increasing on
either side of $x=a$ or consistently decreasing on either side of $x=a,$
then there is neither a local maximum nor a local minimum at $x=a.$ We can
prove the following theorem using the Mean Value Theorem.

\begin{theorem}{Intervals of Increase and Decrease}{IntervalsIncDecTheorem}
If $f^{\prime }\left( x\right) >0$ for every $x$ in an interval, then $f$ is
increasing on that interval.

If $f^{\prime }\left( x\right) <0$ for every $x$ in an interval, then $f$ is
decreasing on that interval.
\end{theorem}
\begin{proof}
We will prove the increasing case. The proof of the decreasing case
is similar. Suppose that $f^{\prime }\left( x\right) >0$ on an interval $I.$
Then $f$ is differentiable, and hence also, continuous on $I.$ If $x_{1}$
and $x_{2}$ are any two numbers in $I$ and $x_{1}<x_{2},$ then $f$ is
continuous on $\left[ x_{1},x_{2}\right] $ and differentiable on $\left(
x_{1},x_{2}\right) .$ By the Mean Value Theorem, there is some $c$ in $%
\left( x_{1},x_{2}\right) $ such that
\begin{equation*}
f^{\prime }\left( c\right) =\frac{f\left( x_{2}\right) -f\left( x_{1}\right)
}{x_{2}-x_{1}}.
\end{equation*}%
But $c$ must be in $I,$ and thus, since $f^{\prime }\left( x\right) >0$ for
every $x$ in $I,$ $f^{\prime }\left( c\right) >0$. Also, since $x_{1}<x_{2},$
we have $x_{2}-x_{1}>0.$ Therefore, both the left hand side and the
denominator of the right hand side are positive. It follows that the
numerator of the right hand must be positive. That is, $f\left( x_{2}\right)
-f\left( x_{1}\right) >0,$ or in other words, $f\left( x_{1}\right) <f\left(
x_{2}\right) .$ This shows that between $x_{1}$ and $x_{2}$ in $I,$ the
larger one, $x_{2},$ necessarily has the larger function value, $f\left(
x_{2}\right) ,$ and the smaller one, $x_{1},$ necessarily have the smaller
function value, $f\left( x_{1}\right) .$ This means that $f$ is increasing
on $I$.
\end{proof}

\begin{example}{Local Minimum and Maximum}{LocalMaxMinExample}
Consider the function $f\left( x\right) =x^{4}-2x^{2}.$ Find
where $f$ is increasing and where $f$ is decreasing. Use this information to
find the local maximum and minimum points of $f$.
\end{example}
\begin{solution}
We compute $f^{\prime }\left( x\right) $ and analyze its sign.
\begin{equation*}
f^{\prime }\left( x\right) =4x^{3}-4x=4x\left( x^{2}-1\right) =4x\left(
x-1\right) \left( x+1\right) .
\end{equation*}%
The solution of the inequality $f^{\prime }\left( x\right) >0$ is $\left(
-1,0\right) \cup \left( 1,\infty \right) $. So, $f$ is increasing on the
interval $\left( -1,0\right) $ and on the interval $\left( 1,\infty \right)
. $ The solution of the inequality $f^{\prime }\left( x\right) <0$ is $%
\left( -\infty ,-1\right) \cup \left( 0,1\right) .$ So, $f$ is decreasing on
the interval $\left( -\infty ,-1\right) $ and on the interval $\left(
0,1\right) .$ Therefore, at the critical points $-1,$ $0$ and $1,$
respectively, $f$ has a local minimum, a local maximum and a local minimum.
\end{solution}

%\figure[!ht]
%\vbox{\beginpicture
%\normalgraphs
%%\ninepoint
%\setcoordinatesystem units <1.5truecm,1.5truecm>
%\setplotarea x from 0 to 6.28, y from -1 to 1
%\axis left shiftedto x=0 /
%\axis bottom shiftedto y=0 ticks withvalues {${\pi\over4}$}
%      {$5\pi\over4$} / at 0.7853981635 3.926990818 / /
%\setquadratic
%\plot
%0.000 0.000
%0.063 0.063 0.126 0.125 0.188 0.187 0.251 0.249
%0.314 0.309 0.377 0.368 0.440 0.426 0.503 0.482 0.565 0.536
%0.628 0.588 0.691 0.637 0.754 0.685 0.817 0.729 0.880 0.771
%0.942 0.809 1.005 0.844 1.068 0.876 1.131 0.905 1.194 0.930
%1.257 0.951 1.319 0.969 1.382 0.982 1.445 0.992 1.508 0.998
%1.571 1.000 1.634 0.998 1.696 0.992 1.759 0.982 1.822 0.969
%1.885 0.951 1.948 0.930 2.011 0.905 2.073 0.876 2.136 0.844
%2.199 0.809 2.262 0.771 2.325 0.729 2.388 0.685 2.450 0.637
%2.513 0.588 2.576 0.536 2.639 0.482 2.702 0.426 2.765 0.368
%2.827 0.309 2.890 0.249 2.953 0.187 3.016 0.125 3.079 0.063
%3.142 0.000 3.204 -0.063 3.267 -0.125 3.330 -0.187 3.393 -0.249
%3.456 -0.309 3.519 -0.368 3.581 -0.426 3.644 -0.482 3.707 -0.536
%3.770 -0.588 3.833 -0.637 3.896 -0.685 3.958 -0.729 4.021 -0.771
%4.084 -0.809 4.147 -0.844 4.210 -0.876 4.273 -0.905 4.335 -0.930
%4.398 -0.951 4.461 -0.969 4.524 -0.982 4.587 -0.992 4.650 -0.998
%4.712 -1.000 4.775 -0.998 4.838 -0.992 4.901 -0.982 4.964 -0.969
%5.027 -0.951 5.089 -0.930 5.152 -0.905 5.215 -0.876 5.278 -0.844
%5.341 -0.809 5.404 -0.771 5.466 -0.729 5.529 -0.685 5.592 -0.637
%5.655 -0.588 5.718 -0.536 5.781 -0.482 5.843 -0.426 5.906 -0.368
%5.969 -0.309 6.032 -0.249 6.095 -0.187 6.158 -0.125 6.220 -0.063
%6.283 0.000 /
%\plot
%0.000 1.000 0.063 0.998 0.126 0.992 0.188 0.982 0.251 0.969
%0.314 0.951 0.377 0.930 0.440 0.905 0.503 0.876 0.565 0.844
%0.628 0.809 0.691 0.771 0.754 0.729 0.817 0.685 0.880 0.637
%0.942 0.588 1.005 0.536 1.068 0.482 1.131 0.426 1.194 0.368
%1.257 0.309 1.319 0.249 1.382 0.187 1.445 0.125 1.508 0.063
%1.571 0.000 1.634 -0.063 1.696 -0.125 1.759 -0.187 1.822 -0.249
%1.885 -0.309 1.948 -0.368 2.011 -0.426 2.073 -0.482 2.136 -0.536
%2.199 -0.588 2.262 -0.637 2.325 -0.685 2.388 -0.729 2.450 -0.771
%2.513 -0.809 2.576 -0.844 2.639 -0.876 2.702 -0.905 2.765 -0.930
%2.827 -0.951 2.890 -0.969 2.953 -0.982 3.016 -0.992 3.079 -0.998
%3.142 -1.000 3.204 -0.998 3.267 -0.992 3.330 -0.982 3.393 -0.969
%3.456 -0.951 3.519 -0.930 3.581 -0.905 3.644 -0.876 3.707 -0.844
%3.770 -0.809 3.833 -0.771 3.896 -0.729 3.958 -0.685 4.021 -0.637
%4.084 -0.588 4.147 -0.536 4.210 -0.482 4.273 -0.426 4.335 -0.368
%4.398 -0.309 4.461 -0.249 4.524 -0.187 4.587 -0.125 4.650 -0.063
%4.712 0.000 4.775 0.063 4.838 0.125 4.901 0.187 4.964 0.249
%5.027 0.309 5.089 0.368 5.152 0.426 5.215 0.482 5.278 0.536
%5.341 0.588 5.404 0.637 5.466 0.685 5.529 0.729 5.592 0.771
%5.655 0.809 5.718 0.844 5.781 0.876 5.843 0.905 5.906 0.930
%5.969 0.951 6.032 0.969 6.095 0.982 6.158 0.992 6.220 0.998
%6.283 1.000 /
%\endpicture}
%\label{fig:sin and cos}
%\caption{The sine and cosine.}
%\endfigure

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:FirstDer}}

\begin{enumialphparenastyle}

Find all critical points and identify them as local maximum points, local minimum points, or neither.

%%%%%%%%%%
\begin{ex}
 $\ds y=x^2-x$
\begin{sol}
 min at $x=1/2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=2+3x-x^3$
\begin{sol}
 min at $x=-1$, max at $x=1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^3-9x^2+24x$
\begin{sol}
 max at $x=2$, min at $x=4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^4-2x^2+3$
\begin{sol}
 min at $x=\pm 1$, max at $x=0$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^4-4x^3$
\begin{sol}
 min at $x=1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=(x^2-1)/x$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^2-(1/x^2)$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $y=\cos(2x)-x$
\begin{sol}
 min at $x=7\pi/12+k\pi$, max at $x=-\pi/12+k\pi$, for integer $k$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
$\ds f(x) = (5-x)/(x+2)$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds f(x) = |x^2 - 121|$
\begin{sol}
 max at $x=0$, min at $x=\pm 11$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds f(x) = x^3/(x+1)$
\begin{sol}
 min at $x=-3/2$, neither at $x=0$
\end{sol}
\end{ex}

%%%%%%%%%%%
%\begin{ex}
% $\ds f(x)= \cases{x^2 \sin(1/x)  & $x\neq 0$ \cr
% 0  & $x=0$\cr}$

%%%%%%%%%%
\begin{ex}
 $\ds f(x) = \sin ^2 x$
\begin{sol}
 min at $n\pi$, max at $\pi/2+n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the maxima and minima of $f(x)=\sec x$.
\begin{sol}
 min at $2n\pi$, max at $(2n+1)\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
  Let $\ds f(\theta) = \cos^2(\theta) -
 2\sin(\theta)$.  Find the intervals where $f$ is increasing and the
 intervals where $f$ is decreasing in $[0,2\pi]$.  Use this
 information to classify the critical points of $f$ as either local
 maximums, local minimums, or neither.
\begin{sol}
 min at $\pi/2+2n\pi$, max at $3\pi/2+2n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Let $r>0$. Find the local
maxima and minima of the function $\ds f(x)
=\sqrt{r^2 -x^2 }$ on its domain $[-r,r]$.
\end{ex}

%%%%%%%%%%
\begin{ex}
 Let $\ds f(x) =a x^2 + bx + c$ with $a\neq 0$. Show that $f$
has exactly one critical point using the first derivative test. Give
conditions on $a$ and $b$ which guarantee that the critical point will
be a maximum. It is possible to see this without using calculus at
all; explain.
\end{ex}

\end{enumialphparenastyle}


\subsection{Fibonacci sequence}

\subsubsection{The generating function}

Let's use a generating function to create a function for the Fibonacci sequence's \(c\)th digit.
\(F(c)=\sum_{i=c} x^is_i\)

Let's look at it for other starts:

\(F(c+k)=\sum_{i=c} x^{i+k}s_{i+k}\)

\(F(c+k)=\sum_{i=c+k} x^is_i\)

\(F(c+1)=\sum_{i=c} x^{i+1}s_{i+1}\)

\(F(c+2)=\sum_{i=c} x^{i+2}s_{i+2}\)

This means

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^i s_i x^2 +\sum_{i=c} x^{i+1} s_{i+1} x\)

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^{i+2}s_i+\sum_{i=c} x^{i+2}s_{i+1}\)

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^{i+2}(s_i+s_{i+1})\)
\subsubsection{Using the definiton of the Fibonacci sequence}

From the definition of the fibonacci sequence, \(s_{i}+s_{i+1}=s_{i+2}\).

\(F(c)x^2+F(c+1)x=\sum_{i=c} x^{i+2}(s_{i+2})\)

\(F(c)x^2+F(c+1)x=F(c+2)\)

\subsubsection{Reducing the functions}

Next, we expand out \(F(c+1)\) and \(F(c+2)\).

\(F(c)-F(c+k)=\sum_{i=c} x^i s_i -\sum_{i=c+k} x^i s_i\)

\(F(c)-F(c+k)=\sum^{c+k}_{i=c} x^i s_i\)

\(F(c+k)=F(c)-\sum^{c+k}_{i=c} x^i s_i\)

So:

\(F(c+1)=F(c)-\sum^{c+1}_{i=c} x^i s_i\)

\(F(c+1)=F(c)-x^c s_c\)

\(F(c+2)=F(c)-\sum^{c+2}_{i=c} x^i s_i\)

\(F(c+2)=F(c)-x^{c+1}s_{c+1}-x^c s_c\)

Let's take our previous equation

\(F(c)x^2+F(c+1)x=F(c+2)\)

\(F(c)x^2+[F(c)-x^c s_c]x=F(c)-x^{c+1}s_{c+1}-x^c s_c\)

\(F(c)x^2+F(c)x-x^{c+1} s_c=F(c)-x^{c+1}s_{c+1}-x^c s_c\)

\(F(c)[x^2+x-1]=x^{c+1}s_c-x^{c+1}s_{c+1}-x^c s_c\)

\(F(c)=\dfrac{x^c s_c + x^{c+1}s_{c+1}-x^{c+1}s_c}{1-x-x^2}\)

\subsubsection{Using the first element in the sequence}

For the start of the sequence, \(c=0\), \(s_0=s_1=1\).

\(F(0)=\dfrac{x^0 1 + x - x}{1-x-x^2}\)

\(F(0)=\dfrac{1}{1-x-x^2}\)

Let's factorise this:

\(F(0)=\dfrac{-1}{(x+\dfrac{1}{2}+\dfrac{\sqrt 5}{2})(x+\dfrac{1}{2}-\dfrac{\sqrt 5}{2})}\)

We can then use partial fraction decomposition

\(\dfrac{1}{(M+\delta)(M-\delta)}=\dfrac{1}{2\delta}[\dfrac{1}{M-\delta}-\dfrac{1}{M+\delta}]\)

To show that

\(F(0)=\dfrac{-1}{\sqrt 5}[\dfrac{1}{x+\dfrac{1}{2}-\dfrac{\sqrt 5}{2}}-\dfrac{1}{x+\dfrac{1}{2}+\dfrac{\sqrt 5}{2}}]\)

\(F(0)=\dfrac{-1}{\sqrt 5}[\dfrac{\dfrac{1}{2}+\dfrac{\sqrt 5}{2}}{(x+\dfrac{1}{2}-\dfrac{\sqrt 5}{2})(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})}-\dfrac{\dfrac{1}{2}-\dfrac{\sqrt 5}{2}}{(x+\dfrac{1}{2}+\dfrac{\sqrt 5}{2})(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})}]\)

\(F(0)=\dfrac{-1}{\sqrt 5}[\dfrac{\dfrac{1}{2}+\dfrac{\sqrt 5}{2}}{x(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})-1}-\dfrac{\dfrac{1}{2}-\dfrac{\sqrt 5}{2}}{x(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})-1}]\)

\(F(0)=\dfrac{1}{\sqrt 5}[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})\dfrac{1}{1-x(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})}-(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})\dfrac{1}{1-x(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})}]\)

\subsubsection{Finishing off}

As we know

\(\dfrac{1}{1-x}=\sum_{i=0}x^i \)

So

\(F(0)=\dfrac{1}{\sqrt 5}[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})\sum_{i=0} x^i(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^i -(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})\sum_{i=0}x^i(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^i]\)

\(F(0)=\dfrac{1}{\sqrt 5}[\sum_{i=0} x^i(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^{i+1} -\sum_{i=0}x^i(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^{i+1}]\)

\(F(0)=\dfrac{1}{\sqrt 5}\sum_{i=0} x^i[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^{i+1} -(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^{i+1}]\)

So the \(n\)th number in the sequence (treating \(n=1\) as the first number) is:

\(\dfrac{1}{\sqrt 5}[(\dfrac{1}{2}+\dfrac{\sqrt 5}{2})^{n} -(\dfrac{1}{2}-\dfrac{\sqrt 5}{2})^{n}]\)


\subsection{Open Delta Neighborhoods}
\noindent
An open delta neighborhood of a point $x_0$ is defined as the set
\begin{equation*}
	N\left(x_0, \delta\right) = \left\{x \in \mathbb{R}^n \mid \norm{x-x_0} < \delta\right\}.
\end{equation*}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{./differentialMultivariableCalculus/open_delta.png}
	\caption{An open delta neighborhood centered at $(x_0, y_0)$}
\end{figure}

\noindent
This simply means all points less than a distance $\delta$ away from point $x_0$.\\
For example, $N( (1,2) , 7) = \left\{ (x,y) \mid \sqrt{(x-1)^2 + (y-2)^2}<7 \right\}$, which is a ball (filled-in circle) of radius 7 centered at $(1, 2)$.


\subsection{Spherical coordinate system}
A sphere with center $O=(x_0,y_0,z_0)$ and radius $r$ is the set of points at distance exactly $r$ from $O$.
We can describe it this by equation
\[(x-x_0)^2+(y-y_0)^2+(z-z_0)^2 = r^2\]

\centerFig{sph-0}

To describe a point on a sphere, we can either directly give its coordinates $x,y,z$ or use the spherical coordinate system: this is the system used to position locations on Earth. We use an angle $\varphi$, the latitude, which tells us how far North the point is (or South, if $\varphi<0$), and an angle $\lambda$, the longitude, which tells us how far East the point is (or West, if $\lambda<0)$. We usually take $\varphi \in [-\frac{\pi}{2},\frac{\pi}{2}]$ and $\lambda \in (-\pi,\pi]$.

\centerFig{sph-1}

If the sphere is centered at the origin, the position represented by coordinates $(\varphi,\lambda)$ is
\[(r\cos\varphi\cos\lambda, r\cos\varphi\sin\lambda, r\sin\varphi)\]
where the $z$-axis points North and the $x$-axis points towards meridian $\lambda=0$ (on Earth, the Greenwich meridian).
%Note that this also gives a parametric representation of the sphere.

The function below finds the position given angles in degrees:
\begin{lstlisting}
p3 sph(double r, double lat, double lon) {
    lat *= M_PI/180, lon *= M_PI/180;
    return {r*cos(lat)*cos(lon), r*cos(lat)*sin(lon), r*sin(lat)};
}
\end{lstlisting}

\subsection{Sphere-line intersection}
A sphere $(O,r)$ and a line $l$ have either 0, 1, or 2 intersection points.

\centerFig{sph-2}

Finding them is exactly like circle-line intersection: first compute the projection $P$ of the center onto the line, then find the intersections by moving the appropriate distance forward or backward along $l$.

\centerFig{sph-3}

This function returns the number of intersection points and places them in pair \lstinline|out| if they exist.
\begin{lstlisting}
int sphereLine(p3 o, double r, line3d l, pair<p3,p3> &out) {
    double h2 = r*r - l.sqDist(o);
    if (h2 < 0) return 0; // the line doesn't touch the sphere
    p3 p = l.proj(o); // point P
    p3 h = l.d*sqrt(h2)/abs(l.d); // vector parallel to l, of length h
    out = {p-h, p+h};
    return 1 + (h2 > 0);
}
\end{lstlisting}


\subsection{Maximum A-Priori (MAP) estimation}

Mode estimate

\(Arg max_\theta p(\theta | X)\)

Using Bayes theorem:

\(P(\theta | X)= \dfrac{P(X|\theta )P(\theta)}{P(X)}\)

So:

\(P(\theta | X)= \dfrac{P(X|\theta )P(\theta)}{P(X)}\)

\(Argmax_\theta p(\theta | X)=Argmax_\theta \dfrac{p(X|\theta )P(\theta)}{P(X)}\)

The denominator isn't affected so:

\(Arg max_\theta p(\theta | X)=Arg max_\theta p(X|\theta )P(\theta)\)

If \(P(\theta )\) is a constant then this is the same as the MLE estimator.

\subsubsection{Other}

\(Argmax_\theta p(\theta|X)\)

Mode estimate

\(p(\theta|X)= \dfrac{p(X| \theta)p(\theta )}{p(X)}\)

\(Argmax_\theta \dfrac{p(X| \theta)p(\theta )}{p(X)}\)

\(\theta \) doesn't change denominator so can instead use:

\(Argmax_\theta p(X| \theta)p(\theta )\)

It is the same as maximum likelihood estimator if \(p(\theta )\) is a constant.


\lab{Application}{Nearest Neighbor Search}{Nearest Neighbor Search}
\label{Ch:NNS}

\objective{Teach about branch and bound and the curse of dimensionality using the nearest neighbor search problem.}

\section*{The Nearest Neighbor Search Problem}

You move into a city that has several post offices.
You want to know which one is the closest.
This problem is  known as the nearest neighbor search problem or the post-office problem.
The general problem is to find the closest of a set of points to any new point.

This has many applications which include computer vision, pattern recognition, internet marketing and data compression.

The naive way to solve this problem is to check the distance of all the data against the point.

\begin{problem}
Write a function that solves the nearest neighbor search problem by exhaustively checking all the distances.
The function should take in the set of points that is the data and a single point.
The output should be the distance to the closest data point and the index of that point.
Your function should be able to take in data in an arbitrary dimension.
\end{problem}

The complexity of this algorithm is $O(kn)$.
Where  is the $k$ number of dimensions and $n$ is the number of data points.

\section*{K-D Trees}

A faster way to solve this problem is to build a k-d tree and search the k-d tree for the nearest neighbor.

A k-d tree is a binary tree where the nodes to the left of parent node have a lower value in the i-th dimension and the nodes to the right of the parent node have a greater value in the i-th dimension.
Which dimension you split the nodes alternates at different levels.
In the $3$ dimensional case the root node is divided in the $x$ dimension, children in the $y$ dimension, grandchildren in the $z$ dimension, and the great-grandchildren in the $x$ dimension and so on.
Each node stores its location, left child and right child.
This requires sorting at each level so the complexity is $O(n log^2(n))$, but we only need to build the k-d tree once and after that we can query it as many times as we want.

Included is a function that takes in a set of data and builds k-d tree.
The leaf nodes' children are  python's ``None" object.

TODO: revise algorithm description

The search of the tree is done recursively.
We will call the node that we are on the parent node.
We first see if the distance between the parent node at the less is current best.
If so, we update it.
We then compare the values in the i-th dimension of the point and the parent.
If it is point's value is less than the parent we recursively go down the left child, greater, the right child.
Then we have to check if the hypersphere around the point with radius being the current best distance crosses the dividing hyperplane created by the parent.
If the point's value is less than the parent, we do this by checking to see if the value in the i-th dimension plus the current best distance is greater than the parent's value in the i-th dimension.
If the point's value is greater than the parent, we do this by checking to see if the value n the i-th dimension minus the current best distance is less than the parent's value in the i-th dimension.
If this be the case, we recursively go down the other child as well.

\begin{problem}
Write a function that solves the nearest neighbor search problem by searching through a k-d tree.
The function should take in k-d tree and a single point.
The output should be the distance to the closest data point and the coordinates of that point.
\end{problem}

\begin{problem}
Time both the functions you have created with the number of data points being $10,000-100,000$ every multiple of $10,000$ with $4$ dimensions.
Time only the searching of the k-d tree not the building of it.
Plot both times on the same plot.
How do the two algorithms compare?
\end{problem}

\begin{figure}[H]
\includegraphics[scale = .5]{4dTime.pdf}
\caption{Your graph should look like this.
The green line is the naive version and blue line is using a kd-tree.
As you can see the kd-tree is much faster.}
\end{figure}

The complexity of this algorithm is $O(klog(n))$ in optimal time.
Its worst case is $O(k*n^{1-\frac{1}{k}})$ where $k$ is the number of dimensions and $n$ is the number of points in the tree.
The reasons for this are discussed in the next section.

\section*{Curse of Dimensionality}

As you increase the number of dimensions the number of times that you have to go down both branches increases.
You get to the point where you eliminate very few points by using a k-d tree.

\begin{problem}
Time both algorithms for the number of data points being $10,000-100,000$ every multiple of $10,000$ with $20$ dimensions.
Plot both times on the same plot. Now how do the two algorithms compare?
\end{problem}

\begin{figure}[H]
\includegraphics[scale = .5]{20dTime.pdf}
\caption{
Your graph should look like this.
The green line is the naive version and blue line is using a kd-tree.
As you can see the kd-tree is slightly slower than the naive version.}
\end{figure}

\begin{problem}
Time the SciPy built in function for searching a k-d tree (do not time the building of the kd-tree) for the number of dimensions points being $2-50$ with $20,000$ data points.
Plot the time.
What do you notice?
\li{from scipy.spatial import KDTree} will import the built in k-d tree.
Create the tree by \li{tree = KDTree(data)} and search it by \li{tree.query(point)}.

\li{scipy.spatial} includes a \li{cKDTree} object which is implemented in C.
Is it any better?

\end{problem}
\begin{figure}[h!]
\includegraphics[scale = .5]{curseD.pdf}
\caption{
Your graph should look similar to this.
Around 15 dimensions the time jumps.
At that point using a kd-tree is not more effective than checking all the combinations.}
\end{figure}


\subsection{k-medoids}

\subsubsection{Introduction}

k-mediods is similar to k-means clustering, with two key differences:

\begin{itemize}
\item Centroids are now always located on data points, rather than floating freely.
\item We mimimise \(l_1\) distance, rathern than \(l_2\).
\end{itemize}

\subsubsection{Partitioning Around Medoids (PAM) algorithm}

This is the most common approach for k-medoids.

We initialise randomly, as we do for k-means.

We then iterate the following:

\begin{itemize}
\item Calculate the loss for the current allocation
\item For each medoid, see if swapping allocation with another (non-medoid) data point decreases the cost.
\item If it does, make the swap.
\end{itemize}