\lab{Newton's Method, Julia Sets and Basins of Attraction}{Newton's Method}
\label{lab:NewtonsMethod}
\objective{Understand Newton's Method. Understand definition of basin of attraction.  Basic understanding of Julia Sets.}

Newton's method finds the zeros of functions; i.e., Newton's method finds $\overline{x}$ such that $f\left(\overline{x}\right) = 0$.
This method is used to optimize functions (find their maxima and minima), for example to find the zeros of the first derivative.

\section*{Newton's method}
Newton's method begins with an initial guess $x_0$.
Then recursively define a sequence by
\[
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
\]
In other words, Newton's method approximates a function by its tangent line, and then uses the zero of the tangent line as the next guess for $x_n$ (see Figure \ref{fig:newton}).

The sequence $\{x_n\}$ will converge to the zero $\overline{x}$ if
\begin{enumerate}
\item $f$, $f'$, and $f''$ exist and are continuous,
\item $f'(\overline{x})\neq0$, and
\item $\overline{x}$ is ``sufficiently close'' to $x_0$.
\end{enumerate}
In applications, the first two conditions usually hold.
However, if $\overline{x}$ and $x_0$ are not ``sufficiently close'', Newton's method may converge very slowly, or it may not converge at all.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{newton_iters}
\label{fig:newton}
\caption{An illustration of how one iteration of Newton's method works}
\end{figure}

Newton's method is powerful because given the three conditions above, it converges quickly.
In these cases, the $\{x_n\}$ converge to the actual root quadratically, meaning that the maximum error is squared at every iteration.

Let us do an example with $f(x) = x^2-1$.
We define $f(x)$ and $f'(x)$ in Python as follows.
\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt
>>> f = lambda x : x**2 - 1
>>> Df = lambda x : 2*x
\end{lstlisting}
Now we set $x_0 = 1.5$ and iterate.
\begin{lstlisting}
>>> xold = 1.5
>>> xnew = xold - f(xold)/Df(xold)
>>> xnew
1.0833333333333333
\end{lstlisting}
We can repeat this as many times as we desire.
\begin{lstlisting}
>>> xold = xnew
>>> xnew = xold - f(xold)/Df(xold)
>>> xnew
1.0032051282051282
\end{lstlisting}
We have already computed the root 1 to two digits of accuracy.

\begin{problem}
\label{prob:newton_arr}
\leavevmode
\begin{enumerate}
\item Implement Newton's method with the following function.
\begin{lstlisting}
def Newtons_method(f, x0, Df, iters=15, tol=.002):
    '''Use Newton's method to approximate a zero of a function.

    INPUTS:
    f     - A function handle. Should represent a function from
            R to R.
    x0    - Initial guess. Should be a float.
    Df    - A function handle. Should represent the derivative
            of `f`.
    iters - Maximum number of iterations before the function
            returns. Defaults to 15.
    tol   - The function returns when the difference between
            successive approximations is less than `tol`.

    RETURN:
    A tuple (x, converged, numiters) with
    x           - the approximation to a zero of `f`
    converged   - a Boolean telling whether Newton's method
                converged
    numiters    - the number of iterations the method computed
    '''
\end{lstlisting}

\item Run \li{Newtons_method()} on $f=\cos(x)$ with $x_0=1$ and $x_0=2$.
How many iterations are required to get 5 digits of accuracy?
\item Newton's method can be used to find zeros of functions that are hard to solve for analytically.
Plot $f(x) = \frac{sin(x)}{x}-x$ on $[-4, 4]$.
Note that this function can be made continuous on this domain by defining $f(0)=1$.
Use your function \li{Newtons_method()} to compute the zero of this function to 7 digits of accuracy.
\item Run \li{Newtons_method()} on $f(x)=x^9$ with $x_0=1$.
How many iterations are required to get 1 digit of accuracy?
What is the reason for this slow convergence?
\item Run \li{Newtons_method()} on $f(x)=x^{1/3}$ with $x_0=.01$.
What happens and why?
Hint: The command \li{x**(1/3)} will not work when \li{x} is negative.
Here is one way to define the function $f(x)=x^{1/3}$ in NumPy.
\begin{lstlisting}
f = lambda x: np.sign(x)*np.power(np.abs(x), 1./3)
\end{lstlisting}
\end{enumerate}
\end{problem}

\begin{problem}(Optional)
Modify the function \li{Newtons_method()} in Problem \ref{prob:newton_arr} so that the argument \li{Df} defaults to \li{None}.
If no derivative is passed to the function, use the centered coefficients method of Lab \ref{lab:NumericalDerivatives} to compute the derivative.
\end{problem}

\begin{comment}
\begin{problem}
Extend your Newton's method even further so that it will work on systems of equations.
Suppose that $F: \mathbb{R}^n \rightarrow \mathbb{R}^n $.
The relevant equation is
\[
x_{i+1} = x_i - J^{-1}F(x_i)
\]
Note that you should not calculate the inverse Jacobian.
\li{scipy.linalg.solve(A,b)} gives you the solution $x$ to the equation $Ax=b$.
Use this fact to calculate $J^{-1}F$ from $J$ and $F$.
You should be able to make this function work whether or not the user inputs a Jacobian.
This also means that you will have to use your own \li{jacobian} function.
\end{problem}
\end{comment}

\section*{Basins of attraction: Newton fractals}
When $f(x)$ has many roots, the root that Newton's method converges to depends on the initial guess $x_0$.
For example, the function $f(x)=x^2-1$ has roots at $-1$ and $1$.
If $x_0<0$, then Newton's method coverges to -1; if $x_0>0$ then it converges to 1 (see Figure \ref{fig:basins1}).
We call the regions $(-\infty, 0)$ and $(0, \infty)$ \emph{basins of attraction}.

\begin{figure}
\begin{center}
\includegraphics[scale=0.5]{basins1}
\caption{The plot of $f(x) = x^2 -1$ along with some values for $x_0$.
When Newton's method is initialized with a blue value for $x_0$ it converges to -1; when it is initialized with a red value it converges to 1.}
\label{fig:basins1}
\end{center}
\end{figure}

When $f$ is a polynomial of degree greater than 2, the basins of attraction are much more interesting.
For example, if $f(x) = x^3-x$, the basins are depicted in Figure \ref{fig:fractal_1d}.

\begin{figure}
\begin{center}
\includegraphics[scale=0.5]{fractal1d}
\caption{The plot of $f(x) = x^3 -x$ along with some values for $x_0$.
Blue values converge to $-1$, red converge to 0, and green converge to 1.}
\label{fig:fractal_1d}
\end{center}
\end{figure}

We can extend these examples to the complex plane.
Newton's method works in arbitrary Banach spaces with slightly stronger hypotheses (see theorem in textbook), and in particular it holds over $\mathbb{C}$.

Let us plot the basins of attraction for $f(x) = x^3-x$ on the domain $\{a+bi \mid (a, b) \in [-1.5, 1.5] \times [-1.5, 1.5] \}$ in the complex plane.
We begin by creating a $700 \times 700$ grid of points in this domain.
We create the real and imaginary parts of the points separately, and then use \li{np.meshgrid()} to turn them into a single grid of complex numbers.
\begin{lstlisting}
>>> xreal = np.linspace(-1.5, 1.5, 700)
>>> ximag = np.linspace(-1.5, 1.5, 700)
>>> Xreal, Ximag = np.meshgrid(xreal, ximag)
>>> Xold = Xreal+1j*Ximag
\end{lstlisting}
Recall that \li{1j} is the complex number $i$ in NumPy.
The array \li{Xold} contains $700^2$ complex points evenly spaced in the domain.

We may now perform Newton's method on the points in \li{Xold}.
\begin{lstlisting}
>>> f = lambda x : x**3-x
>>> Df = lambda x : 3*x**2 - 1
>>> Xnew = Xold - f(Xold)/Df(Xold)
\end{lstlisting}
After iterating the desired number of times, we have an array \li{Xnew} whose entries are various roots of $x^3-x$.

Finally, we plot the array \li{Xnew}. The result is similar to Figure \ref{fig:fractal_ex}.
\begin{lstlisting}
>>> plt.pcolormesh(Xreal, Ximag, Xnew)
\end{lstlisting}

Notice that in Figure \ref{fig:fractal_ex}, whenever red and blue try to come together, a patch of green appears inbetween.
This behavior repeats on an infinitely small scale, producing a fractal.
Because it arises from Newton's method, this fractal is called a \emph{Newton fractal}.

\begin{figure}
\begin{center}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{fractal_ex}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
\centering
\includegraphics[width=\textwidth]{fractal_zoom}
\end{subfigure}
\caption{ Basins of attraction for $x^3-x$ in the complex plane.
The picture on the right is a close-up of the figure on the left.}
\label{fig:fractal_ex}
\end{center}
\end{figure}

Newton fractals tell us that the long-term behavior of the Newton method is extremely sensitive to the initial guess $x_0$.
Changing $x_0$ by a small amount can change the output of Newton's method in a seemingly random way.
This is an example of \emph{chaos}.

\begin{problem}
\leavevmode
\begin{enumerate}
\item Complete the following function to plot the basins of attraction of a function.
\begin{lstlisting}
def plot_basins(f, Df, roots, xmin, xmax, ymin, ymax, numpoints=100, iters=15, colormap='brg'):
    '''Plot the basins of attraction of f.

    INPUTS:
    f       - A function handle. Should represent a function
            from C to C.
    Df      - A function handle. Should be the derivative of f.
    roots   - An array of the zeros of f.
    xmin, xmax, ymin, ymax - Scalars that define the domain
            for the plot.
    numpoints - A scalar that determines the resolution of
            the plot. Defaults to 100.
    iters   - Number of times to iterate Newton's method.
            Defaults to 15.
    colormap - A colormap to use in the plot. Defaults to 'brg'.
    '''
\end{lstlisting}
You can test your function on the example $f(x) = x^3-x$ above.

When the function \li{plt.pcolormesh()} is called on a complex array, it evaluates only on the real part of the complex numbers.
This means that if two roots of \li{f} have the same real part, their basins will be the same color if you plot directly using \li{plt.pcolormesh()}.

One way to fix this problem is to compute \li{Xnew} as usual.
Then iterate through the entries of \li{Xnew} and identify which root each entry is using the input \li{roots}.
Finally, create a new array whose entries are integers corresponding to these roots.
Plot the array of integers to view the basins of attraction.

\item Run \li{plot_basins()} on the function $x^3-1$ on the domain $\{a+bi \mid (a, b) \in [-1.5, 1.5] \times [-1.5, 1.5] \}$.
The resulting plot should look like Figure \ref{fig:fractal_hw}.

\end{enumerate}
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.66]{fractal_hw}
\caption{Basins of attraction for $x^3-1$.}
\label{fig:fractal_hw}
\end{center}
\end{figure}
\end{problem}

% The remainder of the material has nothing to do with this lab, except that it mentions fractals.
\begin{comment}
Another well-studied fractal in the complex plane is the Mandelbrot set.
It is defined as the points $c \in \mathbb{C}$ for which the sequence
\[z_n = z_{n-1}^2 + c\]
is bounded.

\begin{problem}
Generate another grid of complex numbers on $[-1.5,.5]\times[-i,i]$.
Consider the recurrence relation $x_n = x_{n-1}^2 + c$.
Run this iteration 30 times with a resolution of $200 \times 200$ and display the plot.
\end{problem}
\end{comment}


\section{Method}

\subsection{The Hopfield Network}
The Hopfield Network is a simple fully-connected neural network, i.e. all neurons are connected to each other. The network is a recurrent neural network popularized by John Hopfield in \cite{hopfield}. Each neuron is either on (firing) or off (not firing) encoded as positive or negative values, as showed in mathematical terms in \cref{eq:states}.
\begin{tcolorbox}[ams equation, title={Network States}] \label{eq:states}
    \mathbf{S} = \begin{bmatrix}
        s_1 \\ s_2 \\ \vdots \\ s_n
    \end{bmatrix}, \quad \mathbf{S} \in \{1, -1\}^{N\times1}
\end{tcolorbox}
The connection between two seperate neurons is weighted, and there are no self-connections. We used the Hebbian learning rule where the weight matrix $\bf W$ for storing $M$ different patterns $\mathbf{V}^{(m)} \in \{1, -1\}^{N\times 1}$ can be implemented using \cref{eq:weights}. Hebbian learning models how cells firing together gain stronger synaptic connections \cite{hebb}.
\begin{tcolorbox}[ams align, title={Network Weights}] \label{eq:weights}
    \mathbf{W} &= {\bar{\mathbf{W}}} - diag^{-1}(\bar{\mathbf{W}}) & \bar{\mathbf{W}} &= \frac{1}{M}\sum_{m=1}^M \mathbf{V}^{(m)} (\mathbf{V}^{(m)})^T \succeq 0
\end{tcolorbox}
Each element in \cref{eq:energy} in the weight matrix then corresponds to \cite{hopfield}:
\begin{equation*}
    w_{ij} = \begin{cases}
        \frac{1}{M}\sum_{m=1}^M v_{i}^{(m)} v_{j}^{(m)} & \forall i \neq j \\
        0 & \forall i = j
    \end{cases}
\end{equation*}
The weigths can be interpreted as whether neuron $i$ and neuron $j$ should be in the same or opposite states according to weight $w_{ij}$. A positive value indicates the neurons should be pulled toward the same state, while a negative value indicates they should be pushed away from each other. Each weight can be thought of as a spring connected between two neurons, either pushing or pulling depending on the state of the spring.

The network states can be updated by using the update rule in \cref{eq:update}. Intuitively we can think of the update step as the updating neuron being pushed or pulled towards the state of all the other neurons, and the strongest state "wins".
\begin{tcolorbox}[ams align, title={Update Step}]\label{eq:update}
    s_i &= sign(h_i) & h_i = \sum_{j=1}^N w_{ij} s_j \iff \mathbf{H} = \mathbf{W} \mathbf{S}
\end{tcolorbox}
The neurons can either be updated using an asynchronous or synchronous approach. With asynchronous updating, the neurons are updated in random order at different times, while with syncrhonous updating they are updated all at once. We utilized the asynchronous approach, and selected one neruon each timestep. The neurons were shuffled using a random permuation.

\subsection{Energy of Hopfield Network}
We can associate a number to each state of the network, which we refer to as the energy $E$ of the network.
\begin{tcolorbox}[title={Energy contained in network}]
    \begin{subequations}\label{eq:energy}
        \begin{align}
        E &= -\frac{1}{2} \sum_{i,j} w_{ij}s_i s_j \label{eq:energy-sum} \\
        &= - \frac{1}{2} \mathbf{S}^T \mathbf{W} \mathbf{S} \\
        &= -\frac{1}{2}( \mathbf{S}^T \bar{\mathbf{W}} \mathbf{S} - trace(\bar{\mathbf{W}})) \label{eq:energy-trace} \\
        &= -\frac{1}{2}(\mathbf{S}^T \bar{\mathbf{W}} \mathbf{S} - N) \label{eq:energy-compact}
        \end{align}
    \end{subequations}
\end{tcolorbox}
A more in-depth derivation of \cref{eq:energy} is available in \cref{sec:proofenergy}.

It is worth noticing that $\bf \bar{W}$ is positive semi-definite, which can be shown using the definition of positive semi-definite matrices $\mathbf{z}^T \mathbf{A} \mathbf{z} \geq 0 \iff \mathbf{A} \succeq 0$.
\begin{equation*}
\mathbf{z}^T\bar{\mathbf{W}} \mathbf{z} = \frac{1}{M}\sum_{m=1}^M \mathbf{z}^T \mathbf{V}^{(m)}(\mathbf{V}^{(m)})^T \mathbf{z} = \frac{1}{M}\sum_{m=1}^M (\mathbf{z}^T \mathbf{V}^{(m)})^2 \geq 0 \implies \mathbf{\bar{W}} \succeq 0
\end{equation*}
Using \cref{eq:energy-compact} we can form upper and lower bounds for the energy. We can think of these limits as how deep valleys we possibly can create using $N$ neurons. Using the fact that $\mathbf{\bar{W}} \succeq 0 \iff \mathbf{S}^T \mathbf{\bar{W}}\mathbf{S} \geq 0$, we get the upper limit.
\begin{equation*}
    E = -\frac{1}{2}(\mathbf{S}^T \bar{\mathbf{W}} \mathbf{S} - N) \leq \frac{N}{2}
\end{equation*}
For the lower limit we assume all the addends in \cref{eq:energy-sum} are $1$. With $N^2$ elements in the weight matrix and after removing the $N$ diagonal elements we get
\begin{equation*}
    E = -\frac{1}{2}(\mathbf{S}^T \bar{\mathbf{W}} \mathbf{S} - N) \geq -\frac{1}{2} (N^2 - N) = -\frac{N}{2} (N-1)
\end{equation*}

\begin{tcolorbox}[title={Energy Limits}]
    \begin{equation}\label{eq:energy-limits}
        -\frac{N}{2}(N-1) \leq E \leq \frac{N}{2}
    \end{equation}
\end{tcolorbox}

\begin{tcolorbox}[title={Intution behind the energy}]
    \textit{We can get a more intuitive sense of how the Hopfield network is able to recall memories by comparing the energy in \cref{eq:energy} to a ball resting on a peak between two valleys. The position of the ball represents the current state of the neurons, and the equilibrium points of the valleys represents the stored memories. If we give the ball even the slightest nudge to either side, the ball will eventually roll down until it reaches the lowest point of its closest valley, and the potential energy of the ball will decrease. To then move the ball further requires energy (i.e. work). As long as the ball is initialized within the correct valley, it will roll down to the correct equilibrium (memory). Using this intuition, the lower and upper energy limits in \cref{eq:energy-limits} correspond to the deepest possible valley and the highest possible peak respectively. While the large number of dimentions in the Hopfield network makes it difficult to visualize the valleys, the energy can still be interpereted in a physical sense.}
\end{tcolorbox}

\subsection{Simulating the Hopfield network}
The Hopfield network can be simulated using MATLAB and \crefrange{eq:states}{eq:energy}. The code is available in \cref{sec:matlab_code}.
\subsubsection*{Stability of a single pattern}
A single pattern can be simulated using $N=50$ neurons, and by storing a single (random generated) pattern $\mathbf{V}$. Two simulations were used, one where the states $\bf S$ were initialized to the pattern $\bf V$ and one with $20\%$ noise. The noise was generated by initializing $80\%$ using the stored pattern, and initializing the remaining $20\%$ randomly with equal probability of $1$ and $-1$. We ran the simulations using $T=50$ iterations to allow the network to properly converge.
\subsubsection*{Stability of multiple patterns}
Multiple patterns can be simulated by storing multiple patterns in $\bf W$ using \cref{eq:weights}. Simulations were then run for each pattern, initializing the states with and without noise. Each simulation was then compared to the same reference pattern.

The capacity of the network can be tested by simulating the network with different number of memories, and record the proportion of memories the network is able to recover from noisy inital conditions. We used $N=50$ neurons and simulated with up to $10$ memories using different noise levels. The average over $100$ simulations were used to reduce random effects.

\subsubsection*{Creating patterns from QR codes}
QR codes were generated using \href{https://miniwebtool.com/qr-code-generator/}{an online QR-code generator}, and resulted in $50 \times 50$ PNG images. The QR-codes were loaded in MATLAB, stripped for any white borders, converted from grayscale values to binary and encoded as a $\{1, -1\}^{N \times 1}$ vector. Two QR codes were generated and stored in the Hopfield network using \cref{eq:weights}. The first simulation was initialized to the first QR-code with large amounts of noise. The second simulation was initialized to the second QR-code with half of the bits forced to 1 (i.e. losing half of the pattern). The number of iterations was increased due to the large amount of neurons.


\chapter{Integral Transforms }
\section{Integral Transforms}
Frequently in mathematical physics we encounter pairs of functions related by an expression of the form
\begin{equation}
g(x)=\int_{a}^{b} f(t) K(x, t) d t\label{FT-01}
\end{equation}
where it is understood that $a, b$, and $K(x, t)$ (called the kernel) will be the same for all function pairs $f$ and $g$. We can write the relationship expressed in Eq. (\ref{FT-01}) in the more symbolic form
\begin{equation}
g(x)=\mathcal{L}{ f(t)}
\end{equation}
thereby emphasizing the fact that Eq. $(\ref{FT-01})$ can be interpreted as an operator equation. The function $g(x)$ is called the integral transform of $f(t)$ by the operator $\mathcal{L}$, with the specific transform determined by the choice of $a, b$, and $K(x, t)$. The operator defined by Eq. (\ref{FT-01}) will be linear:
\begin{align}
\int_{a}^{b}\left[f_{1}(t)+f_{2}(t)\right] K(x, t) d t&=\int_{a}^{b} f_{1}(t) K(x, t) d t+\int_{a}^{b} f_{2}(t) K(t) d t \\
\int_{a}^{b} c f(t) K(\alpha, t) d t&=c \int_{a}^{b} f(t) K(\alpha, t) d t
\end{align}
In order for transforms to be useful, we will shortly see that we need to be able to "undo" their effect. From a practical viewpoint, this means that not only must there exist an operator $\mathcal{L}^{-1}$, but also that we have a reasonably convenient and powerful method of evaluating
\begin{equation}
\mathcal{L}^{-1} g(x)=f(t)
\end{equation}
for an acceptably broad range of $g(x)$. The procedure for inverting a transform takes a wide variety of forms that depend on the specific properties of $K(x, t)$, so we cannot write a formula that is as general as that for $\mathcal{L}$ in Eq. (\ref{FT-01}).\\
Not all superficially reasonable choices for the kernel $K(x, t)$ will lead to operators $\mathcal{L}$ that have inverses, and even for strategically chosen kernels it may be the case that $\mathcal{L}$ and $\mathcal{L}^{-1}$ will only exist for substantially restricted classes of functions. Thus, the entire development of the present chapter is restricted (for any given integral transform) to functions for which the indicated operations can be carried out.\\
One of the frequent uses of integral transforms is to use one, together with its inverse, to form an \textbf{integral representation} of a function that we originally had in an explicit form. This move (which appears to be in the direction of generating greater complexity) has value that arises from the relatively simple behavior of the transforms of differentiation and integration operators. Procedures involving integral representations are also presented in later sections of this chapter.
\begin{figure}[H]
	\centering
	\includegraphics[height=5cm,width=9cm]{FT-01}
\end{figure}
\newpage
\begin{abox}
	Fourier Transform
	\end{abox}

The integral transform that has seen the widest use is the Fourier transform, defined as
\begin{equation}
g(\omega)=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(t) e^{i \omega t} d t
\end{equation}
The notation for this transform is not entirely universal; some writers omit the prefactor $1 / \sqrt{2 \pi}$; we keep it because it causes the transform and its inverse to have formulas that are more symmetrical. In applications involving periodic systems, one occasionally encounters a definition with kernel $\exp \left(2 \pi i \omega t / a_{0}\right)$, where $a_{0}$ is a lattice constant. These differences in notation do not change the mathematics, but cause formulas to differ by powers of $2 \pi$ or $a_{0}$. Caution is therefore advised when combining material from different sources.\\
\section{Infinite Fourier Transform}
\begin{align*}
\text{The fourier transform of }f(x)[-\infty<x<\infty] &:\\
f(s)=F\{f(x)\}&=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(x) e^{i s x} d x\\
\text{Inverse fourier transform of }f(s)[-\infty<x<\infty] &:\\
f(x)=F^{-1}\{f(s)\}&=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(s) e^{-i s x} d s .
\end{align*}
\section{Fourier Sine Transform}
\begin{align*}
f_{s}(\mathrm{~s})&=F_{s}\{f(x)\}=\sqrt{\frac{2}{\pi}} \int_{0}^{\infty} f(x) \sin s x d x \\
f(x)&=F_{s}^{-1}\left\{f_{s}(s)\right\}=\sqrt{\frac{2}{\pi}} \int_{0}^{\infty} f_{s}(s) \sin s x d s
\end{align*}
\section{Fourier Cosine Transform}
\begin{align*}
f_{c}(s)&=F_{c}\{f(x)\}=\sqrt{\frac{2}{\pi}} \int_{0}^{\infty} f(x) \cos s x d x \\
f(x)&=F_{c}^{-1}\left\{f_{c}(s)\right\}=\sqrt{\frac{2}{\pi}} \int_{0}^{\infty} f_{c}(s) \cos s x d s
\end{align*}
\subsubsection{Important Formulas:}
\begin{enumerate}
	\item $\int_{0}^{\infty} \frac{\sin a x}{x} d x=\frac{\pi}{2}$
	\item $\int_{0}^{\infty} e^{-x^{2}} d x=\sqrt{\frac{\pi}{2}}$
	\item $\int_{0}^{\infty} e^{-a x} \sin b x d x=\frac{b}{a^{2}+b^{2}}$
	\item $\int_{0}^{\infty} e^{-a x} \cos b x d x=\frac{a}{a^{2}+b^{2}}$
	\item $\int_{0}^{\infty} e^{-a x^{2}} d x=\frac{1}{2} \sqrt{\frac{\pi}{a}}$
	\item $\int_{0}^{\infty} e^{-a x^{2}} \cos b x d x=\frac{1}{2} \sqrt{\frac{\pi}{a}} e^{-b^{2} / 4 a}$
	\item $\int_{-\infty}^{\infty} e^{-a x^{2}+b x} d x=\sqrt{\frac{\pi}{a}} e^{b^{2} / 4 a}$
\end{enumerate}

\begin{exercise}
	Find the fourier transform of Gaussian distribution function $f(x)=N e^{-a x^{2}}$ where $N$ and 'a' are constant.
\end{exercise}
\begin{answer}
	\begin{align*}
	F\{f(x)\}=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(x) e^{i k x} d x=\frac{1}{\sqrt{2 \pi}} \int_{-\infty} N e^{-a x^{2}} e^{i k x} d x=\frac{N}{\sqrt{2 \pi}} e^{-k^{2} / 4 a}
	\end{align*}
\end{answer}
\begin{exercise}
	Find $F\{\delta(x)\}$
\end{exercise}
\begin{answer}
	\begin{align*}
	F\{f(x)\}=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(x) e^{i k x} d x=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} \delta(x) e^{i k x} d x=\frac{1}{\sqrt{2 \pi}}\left[\mathrm{e}^{i k x}\right]_{x=0}=\frac{1}{\sqrt{2 \pi}}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the fourier transform of $f(x)=e^{-a|x|} \quad(-\infty<x<\infty) \quad(a>0)$
\end{exercise}
\begin{answer}
	\begin{align*}
	\begin{aligned}
	F\left[e^{-a|x|}\right] &=\frac{1}{\sqrt{2 \pi}}\left[\int_{-\infty}^{0} e^{a x} e^{i s x} d x+\int_{0}^{\infty} e^{-a x} e^{i s x} d x\right]=\frac{1}{\sqrt{2 \pi}}\left[\left(\frac{e^{(a+i s) x}}{(a+i s)}\right)_{-\infty}^{0}+\left(\frac{e^{-(a-i s) x}}{(i s-a)}\right)_{0}^{\infty}\right] \\
	&=\frac{1}{\sqrt{2 \pi}}\left[\frac{1}{a+i s}+\frac{1}{a-i s}\right]=\frac{2 a}{\sqrt{2 \pi}\left(s^{2}+a^{2}\right)} \because
	\end{aligned}
	\end{align*}
\end{answer}

\section{Properties of Fourier Transform}
(i) \textbf{Linearity theorem:}
\begin{align*}
\text{If }F(x)&=a_{1} f_{1}(x)+a_{2} f_{2}(x)+\ldots\text{ then, }\\f(s)&=F[f(x)]=a_{1} f_{1}(s)+a_{2} f_{2}(s)+\ldots
\end{align*}
(ii) \textbf{Change of scale property: }
\begin{align*}
\text{If }F[f(x)]&=f(s),\text{ then }F\{f(a x)\}\\&=\frac{1}{a} f(\mathrm{~s} / a)
\end{align*}
(iii) $\text{If }F\{f(x)\}=f(s),\text{ then }F\left\{f^{*}(x)\right\}=f^{*}(-s)$\\\\
(iv) \textbf{Shifting property:}
 \begin{align*}
\text{ If $f(s)$ is the Fourier transform of }\mathrm{f}(\mathrm{x}), \text{ then }F\{f(x \pm a)\}=e^{\mp i s a} f(s)
 \end{align*}
(v) \textbf{Modulation theorem:} \\\\
If $F\{f(x)\}=f(s)$, then $F\{f(x) \cos a x\}=\frac{1}{2} f(s-a)+\frac{1}{2} f(s+a)$
\section{Parseval Identity for Fourier Transform}
If the Fourier transform of $f(x)$ and $g(x)$ be $f(s) \& g(s)$ respectively, then\\\\
(i) $\int_{-\infty}^{\infty} f(s) g^{*}(s) d s=\int_{-\infty}^{\infty} f(x) g^{*}(x) d x$\\\\
where $g^{*}(s)$ is complex conjugate of $g(s)$ and $g^{*}(x)$ is the complex conjugate of $g(x)$.\\\\
(ii) $\left.\int_{-\infty}^{\infty}|f(s)|\right.^{2} d s=\int_{-\infty}^{\infty}|f(x)|^{2} d x$
\begin{exercise}Using parseval's identity show that $\int_{b}^{\infty}\left(\frac{\sin x}{x}\right)^{2} d x=\pi / 2$
\end{exercise}
\begin{answer}
	\begin{align*}
	F\{f(x)\}&=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} f(x) e^{i x x} d x=\sqrt{\frac{2}{\pi}} \frac{\sin a s}{s}\\
	\text{Using parseval identity, }\int_{-\infty}^{\infty}|f(x)|^{2} d x&=\int_{-\infty}^{\infty}|f(s)|^{2} d s\\
	\int_{-a}^{\infty} l^{2} d x&=\int_{-\infty}^{\infty} \frac{2}{\pi} \frac{\sin ^{2} a s}{s^{2}} d s \\ 2 a&=\frac{2}{\pi} \int_{-\infty}^{\infty}\left(\frac{\sin a s}{s}\right)^{2} d s\\
	\text{Putting }a s&=x \Rightarrow a d s=d x\\
	\Rightarrow \pi&=\int_{-\infty}^{\infty}\left(\frac{\sin x}{x}\right)^{2} d x \\ \int_{0}^{\infty}\left(\frac{\sin x}{x}\right)^{2} d x&=\frac{\pi}{2}
	\end{align*}
\end{answer}
\begin{exercise}
	If the fourier transform of $f(x)$ is $f(s)$, then find the fourier transform of $\frac{d f}{d x}$
\end{exercise}
\begin{answer}
	\begin{align*}
	F\left[\frac{d f}{d x}\right]&=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} \frac{d f}{d x} e^{i s x} d x\\&=\frac{1}{\sqrt{2 \pi}}\left[e^{i s x} f(x)\right]_{-\infty}^{\infty}-\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} i s e^{i s x} f(x) d x\\
	\intertext{Since $f(x)$ should be a well behave function for the fourier tranform to exist, therefore, the first term of the above equation should be equal to zero.}
	\text{Then }F\left[\frac{d f}{d x}\right]&=-i s \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} e^{i s x} f(x) d x=-is f( {s})
	\end{align*}
\end{answer}
\begin{exercise}
	Find Fourier transform of $f(x)= \begin{cases}x^{2}, & |x|<a \\ 0, & |x|>a\end{cases}$
\end{exercise}
\begin{answer}
	\begin{align*}
	F\{f(x)\} &=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} e^{i s x} f(x) d x=\frac{1}{\sqrt{2 \pi}} \int_{-a}^{a} e^{i s x} x^{2} d x \\
	&=\frac{1}{\sqrt{2 \pi}}\left[\left(\frac{e^{i s x}}{i s} \cdot x^{2}\right)_{x=-a}^{a}-\frac{2}{i s} \int_{-a}^{a} x e^{i s x} d x\right] \\
	&=\frac{1}{\sqrt{2 \pi}}\left[\frac{a^{2}}{i s}\left(e^{i s a}-e^{-i s a}\right)-\frac{2}{i s}\left[\left(\frac{x e^{i s x}}{i s}\right)_{x=-a}^{a}-\frac{1}{i s} \int_{-a}^{a} e^{i s x} d x\right]\right]\\
	&=\frac{1}{\sqrt{2 \pi}}\left[\frac{2 a^{2}}{s} \sin (s a)+\frac{2 a}{s^{2}} \cos (s a)-\frac{4}{s^{3}} \sin (s a)\right]
	\end{align*}
\end{answer}
\begin{exercise}
	Find the complex fourier transform $e^{-|x|}$
\end{exercise}
\begin{answer}
	\begin{align*}
	F\left\{e^{-|x|}\right\}&=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{\infty} e^{-|x|} e^{i s x} d x\\&=\frac{1}{\sqrt{2 \pi}}\left[\int_{-\infty}^{0} e^{-|x|} e^{i s x} d x+\int_{0}^{\infty} e^{-|x|} \cdot e^{i s x} d x\right]\\
	&=\frac{1}{\sqrt{2 \pi}}\left[\int_{-\infty}^{0} e^{x} \cdot e^{i s x} d x+\int_{0}^{\infty} e^{-x(1-i s)} d x\right]\\&=\frac{1}{\sqrt{2 \pi}}\left[\left[\frac{e^{x(1+i s)}}{1+i s}\right]_{x=-\infty}^{0}+\left[\frac{e^{-x(1-i s)}}{-(1-i s)}\right]_{x=0}^{\infty}\right] \\
	&=\frac{1}{\sqrt{2 \pi}}\left[\frac{1}{1-i s}+\frac{1}{1+i s}\right]\\&=\frac{1}{\sqrt{2 \pi}}\left(\frac{2}{1+s^{2}}\right)
	\end{align*}
\end{answer}

\begin{exercise}
	The graph of a real periodic function $f(x)$ for the range $[-\infty, \infty]$ is shown below
	\begin{figure}[H]
		\centering
		\includegraphics[height=3cm,width=5cm]{FT01}
	\end{figure}
	Which of the following graphs represents the real part of its Fourier transform?
	 \begin{tasks}(2)
		\task[\textbf{a.}]
		\begin{figure}[H]
			\centering
			\includegraphics[height=2cm,width=2.5cm]{FT-06}
		\end{figure}
		\task[\textbf{b.}]	\begin{figure}[H]
			\centering
			\includegraphics[height=2cm,width=2.5cm]{FT-05}
		\end{figure}
		\task[\textbf{c.}]	\begin{figure}[H]
			\centering
			\includegraphics[height=2cm,width=2.5cm]{FT-03}
		\end{figure}
		\task[\textbf{d.}] 	\begin{figure}[H]
			\centering
			\includegraphics[height=2cm,width=2.5cm]{FT02}
		\end{figure}
	\end{tasks}
\end{exercise}
\begin{answer}
	\begin{align*}
	\text { The given graph in the question is of } f(x)&=\cos \omega x\\
	F(\cos \omega x)=\frac{1}{\sqrt{2 \pi}} \int_{-\infty}^{+\infty} \cos \omega x e^{i k x} d x&=\frac{1}{\sqrt{2 \pi}}\left(\int_{-\infty}^{+\infty} \frac{e^{i(k+\omega) x}}{2} d x+\int_{-\infty}^{+\infty} \frac{e^{i(k-\omega) x}}{2} d x\right)\\
	&\Rightarrow F(\cos \omega x) \frac{1}{\sqrt{2 \pi}}\left(\frac{1}{2} \delta(k+\omega)+\frac{1}{2} \delta(k-\omega)\right)
	\intertext{Therefore, the fourier transform of $f(x)=\cos x \omega x$ consist of two delta functions of same height at $k=\omega$ and $k=-\omega$.}
	\end{align*}
		So the correct answer is \textbf{option (b)}
\end{answer}
\newpage
\begin{abox}
	Laplace Transforms
\end{abox}
The Laplace transform $f(s)$ of a function $F(t)$ is defined by
\begin{equation}
f(s)=\mathcal{L}\{F(t)\}=\int_{0}^{\infty} e^{-s t} F(t) d t .
\end{equation}
A few comments on the existence of the integral are in order. The infinite integral of $F(t)$,
\begin{equation*}
\int_{0}^{\infty} F(t) d t,
\end{equation*}
need not exist. For instance, $F(t)$ may diverge exponentially for large $t$. However, if there are some constants $s_{0}, M$, and $t_{0} \geq 0$ such that for all $t>t_{0}$
\begin{equation}
\left|e^{-s_{0} t} F(t)\right| \leq M,\label{LT-01}
\end{equation}
the Laplace transform will exist for $s>s_{0} $. $F(t)$ is then said to be of exponential order. As a counter example, $F(t)=e^{t^{2}}$ does not satisfy the condition given by Eq. (\ref{LT-01}) and is not of exponential order. Thus, $\mathcal{L}\left\{e^{t^{2}}\right\}$ does not exist.
The Laplace transform may also fail to exist because of a sufficiently strong singularity in the function $F(t)$ as $t \rightarrow 0$. For example,
\begin{equation*}
\int_{0}^{\infty} e^{-s t} t^{n} d t
\end{equation*}
diverges at the origin for $n \leq-1$. The Laplace transform $\mathcal{L}\left\{t^{n}\right\}$ does not exist for $n \leq-1$.

\section{Important Laplace Transforms:}
\begin{enumerate}
	\item $\mathcal{L}(1)=\frac{1}{s}$
	\begin{exercise}
		$$
		\mathcal{L}(1)=\frac{1}{s}
		$$
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}(1)&=\int_{0}^{\infty} 1 \cdot e^{-s t} d t=\left[\frac{e^{-s t}}{-s}\right]_{0}^{\infty}\\&=-\frac{1}{s}\left[\frac{1}{e^{s t}}\right]_{0}^{\infty}=-\frac{1}{s}[0-1]\\&=\frac{1}{s}\\
		\text{ Hence }\mathcal{L}(1)&=\frac{1}{s}
		\end{align*}
	\end{answer}
	\item $\mathcal{L}\left(t^{n}\right)=\frac{n !}{s^{n+1}}(n=0,1,2, \ldots \ldots . .)$
	\begin{exercise}
		$\mathcal{L}\left(t^{n}\right)=\frac{n !}{s^{n+1}}$ where $n$ and $s$ are positive.
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}\left(t^{n}\right)&=\int_{0}^{\infty} e^{-s t} t^{n} d t\\
		\text{Putting }s t=&x \quad\text{ or }\quad t=\frac{x}{s} \quad\text{ or } \quad d t=\frac{d x}{s}\\
		\text{Thus we have}\mathcal{L}\left(t^{n}\right)&=\int_{0}^{\infty} e^{-x}\left(\frac{x}{s}\right)^{n} \frac{d x}{s_{r}} \quad \Rightarrow \quad \mathcal{L}\left(t^{n}\right)=\frac{1}{s^{n+1}} \int_{0}^{\infty} e^{-x} \cdot x^{n} d x\\
		\Rightarrow \mathcal{L}\left(t^{n}\right)&=\frac{\sqrt{n+1}}{s^{n+1}} \Rightarrow \mathcal{L}\left(t^{n}\right)=\frac{n !}{s^{n+1}}\hspace{2cm}\left[\begin{array}{c}
		{\left[n+1=\int_{0}^{\infty} e^{-x} \cdot x^{n} d x\right] } \\
		\text { and } \quad \mid \overline{n+1}=n !
		\end{array}\right]
		\end{align*}
	\end{answer}
	\item $\mathcal{L}\left(e^{a t}\right)=\frac{1}{s-a}(s>a)$
	\begin{exercise}
		$\mathcal{L}\left(e^{a t}\right)=\frac{1}{s-a}, \quad$ where $s>a$
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}\left(e^{a t}\right) &=\int_{0}^{\infty} e^{-s t} \cdot e^{a t} d t=\int_{0}^{\infty} e^{-s t+a t} \cdot d t \\
		&=\int_{0}^{\infty} e^{(-s+a) t} \cdot d t=\int_{0}^{\infty} e^{-(s-a) t} \cdot d t=\left[\frac{e^{-(s-a) t}}{-(s-a)}\right]_{0}^{\infty}=-\frac{1}{s-a}\left[\frac{1}{e^{(s-a) t}}\right]_{0}^{\infty}\\
		&=\frac{-1}{(s-a)}(0-1)=\frac{1}{s-a}
		\end{align*}
	\end{answer}
	\item $\mathcal{L}(\cos a t)=\frac{s}{s^{2}+a^{2}}$
	$(s>0)$
	\begin{exercise}
		$L(\cos a t)=\frac{s}{s^{2}+a^{2}}$
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}(\cos a t)&=\mathcal{L}\left(\frac{e^{t a t}+e^{-i a t}}{2}\right) \quad\left[\because \cos a t=\frac{e^{i a t}+e^{-i a t}}{2}\right]\\
		&=\frac{1}{2}\left[\mathcal{L}\left(e^{i a t}+e^{-i a t}\right)\right]=\frac{1}{2}\left[\mathcal{L}\left(e^{i a t}\right)+\mathcal{L}\left(e^{i a t}\right)\right]=\frac{1}{2}\left[\frac{1}{s-i a}+\frac{1}{s+i a}\right]=\frac{1}{2} \frac{s+i a+s-i a}{s^{2}+a^{2}}\\
		&=\frac{s}{s^{2}+a^{2}}
		\end{align*}
	\end{answer}
	\item $\mathcal{L}(\sin a t)=\frac{a}{s^{2}+a^{2}} \quad(s>0)$
		\begin{exercise}
		$\mathcal{L}(\sin a t)=\frac{a}{s^{2}+a^{2}}$
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}(\sin a t)&=\mathcal{L}\left[\frac{e^{i a t}-e^{-i a t}}{2 i}\right]\hspace{3cm} \left[\because \sin a t=\frac{e^{i a t}-e^{-i a t}}{2 i}\right]\\
		&=\frac{1}{2 i}\left[L\left(e^{i a t}-e^{-i a t}\right)\right]=\frac{1}{2 i}\left[L\left(e^{i a t}\right)-\mathcal{L}\left(e^{-i a t}\right)\right] \\
		&=\frac{1}{2 i}\left[\frac{1}{s-i a}-\frac{1}{s+i a}\right]=\frac{1}{2 i} \frac{s+i a-s+i a}{s^{2}+a^{2}}=\frac{1}{2 i} \frac{2 i a}{s^{2}+a^{2}}=\frac{a}{s^{2}+2^{2}}
		\end{align*}
	\end{answer}

	\item $\mathcal{L}(\cosh a t)=\frac{s}{s^{2}-a^{2}}\left(s^{2}>a^{2}\right)$
		\begin{exercise}
		$\mathcal{L}(\cosh a t)=\frac{s}{s^{2}-a^{2}}$
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}(\cosh a t) &=L\left[\frac{e^{a t}+e^{-a t}}{2}\right] \hspace{4cm}\left(\therefore \cosh a t=\frac{e^{a t}+e^{-a t}}{2}\right)\\
		&=\frac{1}{2} \mathcal{L}\left(e^{a t}\right)+\frac{1}{2} L\left(e^{-a t}\right)=\frac{1}{2}\left[\frac{1}{s-a}+\frac{1}{s+a}\right] \hspace{2cm}\left[\mathcal{L}\left(e^{a t}\right)=\frac{1}{s-a}\right]\\
		&=\frac{1}{2}\left[\frac{s+a+s-a}{s^{2}-a^{2}}\right]=\frac{s}{s^{2}-a^{2}}
		\end{align*}
	\end{answer}
	\item $\mathcal{L}(\sinh a t)=\frac{a}{s^{2}-a^{2}}\left(s^{2}>a^{2}\right)$
	\begin{exercise}
		$L(\sinh a t)=\frac{a}{s^{2}-a^{2}}$
	\end{exercise}
	\begin{answer}
		\begin{align*}
		\mathcal{L}(\sinh a t) &=\mathcal{L}\left[\frac{1}{2}\left(e^{a t}-e^{-a t}\right)\right] \\
		&=\frac{1}{2}\left[\mathcal{L}\left(e^{a t}\right)-L\left(e^{-a t}\right)\right]=\frac{1}{2}\left[\frac{1}{s-a}-\frac{1}{s+a}\right]=\frac{1}{2}\left[\frac{s+a-s+a}{s^{2}-a^{2}}\right] \\
		&=\frac{a}{s^{2}-a^{2}}
		\end{align*}
	\end{answer}
\end{enumerate}

\subsubsection{Important properties:}
\begin{enumerate}
	\item Linear Property: $\mathcal{L}\left[a_{1} f_{1}(t)+a_{2} f_{2}(t)\right]=a_{1} \mathcal{L}\left[f_{1}(t)\right]+a_{2} \mathcal{L}\left[f_{2}(t)\right]$
	\item Shifting Property: $\mathcal{L}\left[e^{a t} f(t)\right]=f(s-a)$
	\item Scaling Property: $\mathcal{L}[f(a t)]=\frac{1}{a} f\left(\frac{s}{a}\right)$
	\item $\mathcal{L}\left[t^{n} f(t)\right]=(-1)^{n} \frac{d^{n}}{d s^{n}}(f(s))$
	\item $\mathcal{L}\left[\frac{f(t)}{t}\right]=\int_{-\infty}^{\infty} f(s) d s$
	\item $\mathcal{L}\left[\int_{0}^{t} f(t) d t\right]=\frac{1}{s} f(s)$
	\item $\mathcal{L}\left[f^{\prime}(t)\right]=s L[f(t)]-f(0)$
	\item  $\mathcal{L}\left[f^{\prime \prime}(t)\right]=s^{2} L[f(t)]-f^{\prime}(0)-s f(0)$
	\item Laplace transform of a periodic function $f(t)$ having period ' $T$ ' is $$\mathcal{L}[f({t})]=\frac{1}{1-e^{-s T}} \int_{0}^{T} e^{-s t} f({t}) d {t}$$
	\item Laplace transform of the unit step function $u(t-a)=u_{a}(t)=\left\{\begin{array}{lll}0 & \text { for } & t<a \\ 1 & \text { for } & t \geq a\end{array}\right.$ is $\frac{e^{-a s}}{s}$.
	\item Laplace transform of the dirac delta function $\delta(t-a)$ is $e^{-a s}$.
\end{enumerate}
\begin{exercise}
	Find the laplace transform of $f(t)=(1+\cos 2 t)$
\end{exercise}
\begin{answer}
	\begin{align*}
	\begin{aligned}
	\mathcal{L}[f(t)] &=\int_{0}^{\infty} e^{-s t}\left(1+\cos 2 t \cdot d t=\left(\frac{e^{-s t}}{-s}\right)_{0}^{\infty}+\int_{0}^{\infty} e^{-s t} \cos 2 t d t\right.\\
	&=\frac{1}{s}+\frac{s}{4+s}=\frac{2 s^{2}+4}{s\left(s^{2}+4\right)}
	\end{aligned}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of $f(t)=2 \sin 2 t \cos 4 t$
\end{exercise}
\begin{answer}
	\begin{align*}
	\mathcal{L}[f(t)]=\mathcal{L}[2 \sin 2 t \cdot \cos 4 t]=\mathcal{L}[\sin 6 t-\sin 2 t]=\left(\frac{6}{s^{2}+36}-\frac{2}{s^{2}+4}\right)
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of $f(t)=e^{-t}(3 \sinh 2 t-5 \cosh 2 t)$
\end{exercise}
\begin{answer}
	\begin{align*}
	\begin{aligned}
	\mathcal{L}[f(t)] &=\mathcal{L}\left(3 e^{-t} \sinh 2 t\right)-\mathcal{L}\left(5 e^{-t} \cosh 2 t\right) \\
	&=3 \frac{2}{(s+1)^{2}-4}-5 \cdot \frac{(s+1)}{(s+1)^{2}-4}=\frac{(1-5 \mathrm{~s})}{\left(\mathrm{s}^{2}+2 \mathrm{~s}-3\right)} \quad \text { (Using 2nd property) }
	\end{aligned}
	\end{align*}
\end{answer}
\begin{exercise}
	Find the laplace transform of the fuction\\
	$f(t)=1$ for $2 n \leq t \leq 2 n+1$\\
	$=0 \quad$ for $2 n+1 \leq t \leq 2 n+2 \quad(n=0,1,2, \ldots . .)$
\end{exercise}
\begin{answer}
	\begin{align*}
	\mathcal{L}[f(t)] &=\frac{1}{1-e^{-s T}} \int_{0}^{T} e^{-s t} f(t) d t=\frac{1}{1-e^{-2 s}} \int_{0}^{2} e^{-s t} d t \\
	&=\frac{1}{1-e^{-2 s}}\left[\int_{0}^{1} e^{-s t} d t\right]=\frac{1}{1-e^{-2 s}}\left(\frac{1-e^{-s}}{s}\right)\\
	&=\frac{\left(1-e^{-s}\right)}{s\left(1+e^{-s}\right)\left(1-e^{-s}\right)}=\frac{1}{s\left(1+e^{-s}\right)}
	\end{align*}
\end{answer}

\begin{exercise}
	Find the laplace transform of $f(t)=\left(1+t e^{-t}\right)^{3}$
\end{exercise}
\begin{answer}
	\begin{align*}
	f(t)&=\left(1+t e^{-t}\right)^{3}=1+t^{3} e^{-3 t}+3 t e^{-t}+3 t^{2} e^{-2 t} \\
	\mathcal{L}[f(t)]&=\mathcal{L}(1)+\mathcal{L}\left(t^{3} e^{-3 t}\right)+\mathcal{L}\left(3 t e^{-t}\right)+\mathcal{L}\left(3 t^{2} e^{-2 t}\right)\\
	&=\frac{1}{s}+(-1)^{3} \frac{d^{3}}{d s^{3}}\left(\frac{1}{s+3}\right)+3(-1)^{1} \frac{d}{d s}\left(\frac{1}{s+1}\right)+3(-1)^{2} \frac{d^{2}}{d s^{2}}\left(\frac{1}{s+2}\right)\text{ (Using 4th property)}\\
	&=\frac{1}{s}+\frac{6}{(s+3)^{4}}+\frac{3}{(s+1)^{2}}+\frac{6}{(s+2)^{3}} \text {. }
	\end{align*}
\end{answer}

CSIR- Mathematical Physics/chapter/crash- Fourier.tex


\subsection{Set Difference and Set Complement}

So far, our notation for describing objects that live in sets has been very \textit{inclusive}. For example, set intersection describes objects that are contained in many sets all at once, and set union describes objects that are contained in at least one of many sets. Now we develop notation for descriptions which are \textit{exclusive}. More specifically, we will develop notation to describe objects that are \textit{not} contained in some specified set. To begin with we will define the relative complement of two sets.

\begin{definition}
Let $A$ and $B$ be sets. The \textbf{relative complement} of $A$ with respect to $B$ is denoted $B \setminus A$ and is defined as the set
\[B \setminus A = \{x \in B \colon x \not\in A\}.\]

The \textbf{divided difference} of $A$ and $B$ is denoted $A \Delta B$, and is defined by
\[A \Delta B = (A \setminus B) \cup (B \setminus A).\]
\end{definition}

Note first that $A \setminus B$ is not generally equal to $B \setminus A$. For example, consider the sets $\{1, 2, 3\}$ and $\{2, 3, 4\}$. Then we have
\begin{gather*}
	\{1,2,3\} \setminus \{2, 3, 4\} = \{1\} \\
	\{2, 3, 4\} \setminus\{1,2,3\} = \{4\}.
\end{gather*}

The relative complement is good notation to indicate whether or not some element does not lie in a certain set. Now suppose that we are working within a global set $U$. For example, $U$ can be the set of natural numbers, integers, or some other ``global'' set. We will define the global complement now.

\begin{definition}
Let $U$ be some set fixed in advance and let $A \subset U$. The global complement of $A$ (with respect to $U$) is the set $U \setminus A$.
\end{definition}

\begin{example}
Suppose I want to describe a set of elements for which each element $x$ is in $B$ but not in $A_1$ or $A_2$. We can describe this as follows. Since $x$ is not in $A_1$ or $A_2$, we know that $x \not\in A_1 \cup A_2$. So the set we are describing is just the set $B \setminus (A_1 \cup A_2)$.

Alternatively, we could employ the following derivation using set builder notation:
\begin{align*}
	\{x \colon (x \in B) \land (x \not\in A_1) \land (x \not\in A_2)\} &= \{x \in B \colon \lnot((x \in A_1) \lor (x \in A_2))\} \\
	&= \{x \in B \colon \lnot(x \in A_1 \cup A_2)\} \\
	&= \{x \in B \colon (x \not\in A_1 \cup A_2)\} \\
	&= B \setminus (A_1 \cup A_2).
\end{align*}

We will make this method of manipulating clauses in set builder notation more clear and general in the next section.
\end{example}

\subsection{Set Identities and Manipulation}

In this section we will indicate several set identities (relationships involving sets and various operations on sets) and in general we will develop the general technique for deriving and demonstrating that these identities are true. As in the previous section, we demonstrated various manipulations of set builder notation to abstract sets to other sets using logical identities and notation. We outline the general method of set builder notation manipulation below:

\begin{enumerate}
	\item Recall that set builder notation has the following form:
	\[S = \{\text{$x$} \colon \text{$x$ has some property}\}.\]
	In general, the property that $x$ has can be described in the form of a logical statement. In this sense this means that we can manipulate this statement using the rules of propositional logic.
	\item Suppose now we have some notation of the following form:
	\[S = \{x \in B \colon \text{$x$ satisfies some properties}\}.\]
	This set is the same as the set
	\[\{x \colon (x \in B) \land \text{$x$ satisfies some properties}\}.\]
	In this way, the set declaration on the left hand side can be seen as another property.
\end{enumerate}

We illustrate this method by showing several important set theoretic identities.

\begin{proposition}
For any three sets $A$, $B$, and $C$,
\begin{gather*}
	A \cap (B \cup C) = (A \cap B) \cup (A \cap C) \\
	A \cup (B \cap C) = (A \cup B) \cap (A \cup C)
\end{gather*}
\end{proposition}
To show, for example, the first claim is true, we provide the following chain of equalities.
\begin{align*}
	A \cap (B \cup C) &= \{x \colon (x \in A) \land (x \in B \cup C)\} \\
	&= \{x \colon (x \in A) \land ((x \in B) \lor (x \in C))\} \\
	&= \{x \colon ((x \in A) \land (x \in B)) \lor ((x \in A) \lor (x \in C))\} \\
	&= \{x \colon (x \in A \cap B) \lor (x \in A \cap C)\} \\
	&= (A \cap B) \cup (A \cap C)
\end{align*}
The set of equalities for the second claim is almost the same. In that case we can apply another law of distributivity to reach the conclusion.


%!TEX root =  ../main.tex
\subsection{Reflection}

\objective{Produce reflections of functions and describe their symmetry.}

As you know, multiplying numbers by $-1$ produces the additive opposite: negative values
become positive and positives become negative.  Zero is unaffected, being neither positive
nor negative.  The same principle applies to functions.  What would it look like to ``flip'' all
$x$ values?  $y$ values?

\paragraph{``Outside''}
$-f(x)$ is a reflection over the $x$-axis, leaving $x$'s untouched, and making $y$'s opposite.

We can obtain both the original and reflection across the $x$-axis by writing $\pm f(x)$.  Why
is that not a function?

\paragraph{``Inside''}
$f(-x)$ is a reflection over the $y$-axis, leaving $y$'s untouched, and making $x$'s opposite.

Does the ``opposite'' rule of inside transformations apply or not to the negative on the inside?
\index{transformation!reflection}

\subsection{Symmetry}
\paragraph{Even}\index{even functions}
\marginfig[-0in]{\chapdir/pics/Playing_card_spade_A.png}{The ace of spades (or of clubs or hearts) displays even symmetry about its center.}
Sometimes, multiplying by a negative makes no difference.  In mathematics, this can be very
helpful to know.  Functions that are the same left-to-right and right-to-left are called \textbf{even}.
Can you guess why?  Aren't only numbers even (or odd)?

\begin{derivation}{Even Functions}
An even function has the property $f(x)=f(-x)$ for every $x$ in its domain.
\end{derivation}

Among the more basic functions are power functions.  We will study them in great depth in
chapter 5.  The names ``even'' and ``odd'' come from the similar behavior of $x^n$ when
$n$ is even or odd.

Notice that evenness has the visual appearance of putting a mirror on the $y-$axis.  Did you
know human beings are made to find such left-right symmetry appealing?  Study the faces of
attractive people, and you will find evenness to be a rule of thumb.

\begin{derivation}{Odd Functions}\index{odd functions}
An odd function has the propety $-f(x)=f(-x)$ for every $x$ in its domain.
\end{derivation}

\marginfig[-0in]{\chapdir/pics/English_pattern_queen_of_hearts.png}{The queen of hearts (and many other playing cards) display an odd symmetry about the center.}
You might have supposed oddness to be top-to-bottom and bottom-to-top symmetry.  Why isn't
that possible for functions?  Instead, we find that these functions are the same whether we
proceed left from the origin, or flip the right half upside-down.

\paragraph{Rotational}\index{rotation!symmetry}
Looking at the odd power functions, it becomes clear that there are two ways to regard their
symmetry.  Either they are reflections left-right and up-down (in either order), or they are
$180^\circ$ of themselves.  Is it possible for a function to have any other angle of symmetry
with itself?

Moving on to symmetry \emph{between} functions (or relations), we see a lot of rotational
symmetry exists.  All lines of the form $y=ax$ are rotations of each other.  $f(x)=x^2+k$ and
the relation $y=\pm\sqrt{x-k}$ are $90^\circ$ rotations of each other.

Relations can be their own rotation.  Equilateral triangles with their center at the origin are
all $120^\circ$ rotations of themselves.  In fact, every regular $n-$gon (polygon) is its
own rotation every $\frac{360}{n}^\circ$.  As we take the limit and let $n$ increase,
we approach circular symmetry, continuous rotational symmetry.

Rotation is hard to discover under normal algebra, but it very easy with a matrix.
You can read in §18.1 how the rotational matrix is
$\begin{bmatrix}
	\cos { \theta  }  & -\sin { \theta  }  \\
	\sin { \theta  }  & \cos { \theta  }
\end{bmatrix}$.

\inlinefig{\chapdir/pics/Purple_Star}{Rotational symmetry becomes much more complicated in higher dimensions.}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Give an $\mathcal{O}(n \log k)$-time algorithm to merge $k$ sorted lists into one sorted list, where $n$ is the total number of elements in all the input lists.

Hint 1: Use a min-heap for $k$-way merging.

Hint 2: Think of the merging part of \texttt{MergeSort} and extend it to multiple lists.
Remember that the lists are not necessarily the same size.

\subsection*{Solution}

To merge $k$ sorted lists, we slightly modify the merging function of the original \textsc{MergeSort} algorithm so that it compares first elements of $k$ arrays instead of 2.

With no assumption on size of $k$ lists, we start with first elements of each array and build a min-heap structure.

For any heap structure with $n$ nodes where $n \leq k$ we pop the root element once after a call to \textsc{Heapify} and insert it into the final sorted array.
Once all nodes are popped, we take a new element (smallest element) from each $k$ list and repeat the procedure until all elements are inserted into the final sorted array.

A more formal explanation of the algorithm is given in the following Algorithm in which \textsc{Construct}($A$,$i$) would initialize array $A$ with elements in index $i$ of lists $L_1$ through $L_k$.

\begin{algorithm}[H]
\caption{\textsc{Merge-Sort-k-List}}
\begin{algorithmic}[1]
\For {$i \leftarrow 1$ : max($L_1$.length, $L_2$.length, ...,  $L_k$.length)}
\State \textsc{Construct}($A$, $i$)
\State \textsc{Build-Heap}($A$)
\State $A$.heapsize $\leftarrow$ $A$.length
\For {$j$ $\leftarrow$ $1$ to $A$.heapsize}
\State $F[k] \leftarrow A[1]$
\State $k \leftarrow k + 1$
\State \textsc{Heap-Delete}($A$, $1$)
\EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

The outer loop of proposed algorithm will be executed $\lceil \frac{n}{k} \rceil$ times while the inner loop is executed (at most) $k$ times.
Knowing from previous question that runtime of \textsc{Heap-Delete}($A$, $1$) would be $\log{k}$, the inner loop would give a runtime of $k\log{k}$.
Since runtime of \textsc{Build-Heap}($A$) is also known to be $\mathcal{O}(k)$, runtime of the outer loop would be $k \log k + k$.

Thus the runtime of the algorithm would be the runtime of the outer loop times the number of its execution, as given in Equation \ref{eq2}.

\begin{equation}\label{eq2}
\begin{aligned}
\mathcal{O}(f) &=
\mathcal{O}(\frac{n}{k}[(k\log k)+ k])\\ &= \mathcal{O}(n \log k + n)\\ &= \mathcal{O}(n\log k)
\end{aligned}
\end{equation}


\section{Multi-Index Notation}
Multi-index notation makes high-dimensional things faster and easier. A collection is indices is represented by a tuple $\alpha = \left(\alpha_1,\alpha_2,\alpha_3,... \right)$. The absolute value $|\alpha| = \sum_i \alpha$, partial derivatives $\partial^\alpha = \prod \partial^{\alpha_i}$, powers $\mathbf{x}^\alpha = \prod_i x_i^{\alpha_i}$.

\subsection{Example: Multinomial Coefficients}

Instead of:
\begin{equation}
\sum_{0\leq i_1,i_2,i_3,...,i_k \leq n} {n \choose i_1,i_2,i_3,...,i_k}
\end{equation}

Write:

\begin{equation}
\sum_{0\leq |\alpha|\leq n}{n \choose \alpha}
\end{equation}

\subsection{Example: Taylor Expansion}

For a vector valued function $\mathbf{f}: \mathbb{R}^n \rightarrow \mathbb{R}^m$ that is analytical in a neighborhood of the point $\mathbf{a}$:

\begin{equation}
f(\mathbf{x}) = \sum_{|\alpha|\geq0} \frac{(\mathbf{x} - \mathbf{a})^\alpha}{\alpha!}(\partial^\alpha)f
\end{equation}


\textbf{In this problem you will derive the barycentric weights for Chebyshev points of the second kind. Let}
\begin{align*}
x_j = \cos(j\pi /n),~j = 0,\dots,n,~~\text{and}~~w_j = 1/ \left( \prod_{k=0,k \neq j}^n (x_j - x_k) \right)
\end{align*}
\textbf{In Matlab, use the formula above to evaluate $w_j$, $j = 0, \dots, n$ with $n = 10$. Verify that the weights returned in Matlab are}
\begin{align}
w_0 = 2^{n-1}/(2n),~w_j = 2^{n-1}(-1)^j/n,~j = 1,\dots,n-1,~w_n = 2^{n-1}(-1)^n/(2n)~.
\end{align}
\textbf{Explain why the factor $2^{n-1}/n$ can be ignored when evaluating the barycentric formula. That is, one can use, instead, the weights}
\begin{align*}
\tilde{w}_0 = 1/2,~\tilde{w}_j = (-1)^j, ~j = 1,\dots, n-1,~\tilde{w}_n = (-1)^n/2~.
\end{align*}
By coding matlab both ways of calculating the weights we found the claim to be true since we obtained an infinity norm of the difference of order $10^{-14}$. We can prove the previous by simply substituting the definition of the weights in the baricentric formula that when using the barycentric formula,
\begin{align*}
p(x) &= \frac{\sum_{j=0}^{n}\frac{w_j}{(x - x_j)}f_j}{\sum_{j=0}^{n}\frac{w_j}{(x - x_j)}}\\
&= \frac{\sum_{j=0}^{n}\frac{\frac{2^{n-1}(-1)^j}{n}}{(x - x_j)}}{\sum_{j=0}^{n}\frac{\frac{2^{n-1}(-1)^j}{n}}{(x - x_j)}}f_j\\
& = \frac{\frac{2^{n-1}}{n}}{\frac{2^{n-1}}{n}}\frac{\sum_{j=0}^{n}\frac{(-1)^j}{(x - x_j)}}{\sum_{j=0}^{n}\frac{(-1)^j}{(x - x_j)}}f_j \\
& = \frac{\sum_{j=0}^{n}\frac{(-1)^j}{(x - x_j)}}{\sum_{j=0}^{n}\frac{(-1)^j}{(x - x_j)}}f_j.
\end{align*}
Hence, since the factor $2^{n-1}/n$ gets cancelled, it can be ignored from the weights.
\newpage
\textbf{Show that, for $x_j = cos(j\pi/n)$,}
\begin{align*}
\prod_{k=0}^n(x - x_k) = \frac{1}{2^{n-1}}(x^2-1)U_{n-1}(x)
\end{align*}
\textbf{and}
\begin{align*}
w_j = 1/ \left( \prod_{k=0,k \neq j}^n (x_j - x_k) \right) = \lim_{x \rightarrow x_j} \frac{2^{n-1}(x-x_j)}{(x^2 - 1)U_{n-1}(x)}~.
\end{align*}
\textbf{Here $U_{n-1}(x) = \frac{sin(n\arccos x)}{sin(\arccos x)}$ is the Chebyshev polynomial of the second kind of degree $n-1$.}

We start with the relation between the Chebishev's polynomials of first and second kind
\begin{align*}
T'_n(x) = nU_{n-1}(x).
\end{align*}
Using that
\begin{align*}
T_n(x) = 2^{n-1}\prod_{k=0}^{n-1}(x - x_k),
\end{align*}
and the previous relation we get
\begin{align*}
nU_{n-1}(x) = T'_n(x) = n2^{n-1}\prod_{k=1}^{n-1}(x - \tilde{x}_k)~.
\end{align*}
Note that now the roots have changed, now they are the roots of the Chebishev's polynomial of the second kind and the roots of the boundaries are missing. We can include them by multiplying
\begin{align*}
\prod_{k=0}^{n}(x - \tilde{x}_k)=(x-x_0)(x-x_n)\prod_{k=1}^{n-1}(x - \tilde{x}_k) = (x-x_0)(x-x_n)\frac{1}{2^{n-1}}U_{n-1}(x),
\end{align*}
where we have made $\tilde{x}_0=x_0$ and same for $x_n$. Lastly, recall that
\begin{align*}
(x-x_0)(x-x_n)=(x+1)(x-1)=(x^2-1),
\end{align*}
and we obtain the desired expression
\begin{align*}
\prod_{k=0}^{n}(x - \tilde{x}_k)= \frac{1}{2^{n-1}}(x^2-1)U_{n-1}(x).
\end{align*}

Further, observe that
\begin{align*}
\frac{d}{dx}\prod_{k=0}^{n}(x - x_k) = \sum_{k=0}^{n} \left( \prod_{j=0,j \neq k}^{n}(x - x_j) \right) = \prod_{k=0, k \neq j}^{n}(x_j - x_k).
\end{align*}
Hence,
\begin{align*}
 \frac{1}{\frac{d}{dx} \prod_{k=0}^{n}(x - x_k)} = \lim_{x \rightarrow x_j} \frac{(x - x_j)}{\prod_{k=0}^{n}(x - x_k)} = \lim_{x \rightarrow x_j} \frac{2^{n-1}(x-x_j)}{(x^2 - 1)U_{n-1}(x)}.
\end{align*}

\textbf{Use L'Hopital's Rule to evaluate the limit in (2) and derive the expressions in equation (1).}

Given the expression for $U_{n-1}(x)$ above we obtain the derivative
\begin{align*}
U'_{n-1}(x) = \frac{-n\cos(n\arccos(x))\sin(\arccos(x)) + \sin(n\arccos(x))\cos(\arccos(x))}{\left(\sqrt{1 - x^2}\right)\sin^2(\arccos(x))}.
\end{align*}
Recall the definition of $x_j=\cos(j\pi/n)$. Then,
\begin{align*}
U'_{n-1}(x_j) & = \frac{-n\cos(j \pi)}{\left(\sqrt{1 - x^2}\right) \sin(\frac{j \pi}{n})} + \frac{\sin(j \pi) \cos(\frac{j \pi}{n})}{\left(\sqrt{1 - x^2}\right)\sin^2(\frac{j \pi}{n})}  \\
&= \frac{1}{\sqrt{1 - x^2}} \left( \frac{n\cos(j\pi)}{\sin(\frac{j \pi}{n})} \right) \\
& = \frac{n(-1)^j}{\sin^2(\frac{j \pi}{n})}~.
\end{align*}
To finish, use L'Hopital's rule and the result from the previous part,
\begin{align*}
w_j & = \lim_{x \rightarrow x_j} \frac{2^{n-1}(x-x_j)}{(x^2 - 1)U_{n-1}(x)} \\ &= \lim_{x \rightarrow x_j} \frac{2^{n-1}}{2xU_{n-1}(x)+(x^2 -1)U'_{n-1}(x)} \\
& = \frac{2^{n-1}}{2x_jU_{n-1}(x_j)+(x_j^2 -1)U'_{n-1}(x_j)} \\
& = \frac {2^{n-1}}{\sin^2(\frac{j\pi}{n})\left( \frac{n(-1)^j}{\sin^2(\frac{j\pi}{n})}\right)} \\
& = \frac{2^{n-1}(-1)^j}{n}.
\end{align*}

\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 5
n=10;
j=0:n;
x=cos(j*pi/n);
w_a=zeros(n+1,1);
w_b=zeros(n+1,1);
for j=0:n
    xj=x(j+1);
    xk=x;
    xk(j+1)=[];
    w_a(j+1)=1/prod(xj-xk);
end
w_b(1)=2^(n-1)/(2*n);
for j=2:n
    w_b(j)=2^(n-1)*(-1)^(j-1)/n;
end
w_b(n+1)=2^(n-1)*(-1)^n/(2*n);
err=norm(w_a-w_b,inf)
\end{verbatim}


\section*{Exercises}

\begin{ex}
  Consider the set $\R^2$ with the following non-standard addition
  operation $\oplus$:
  \begin{equation*}
    (a,b) \oplus (c,d) = (a+d,b+c).
  \end{equation*}
  Scalar multiplication is defined in the usual way. Is this a vector space?
  Explain why or why not.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Consider $\R^2$ with the following non-standard addition operation
  $\oplus$:
  \begin{equation*}
    (a,b) \oplus (c,d) = (0,b+d).
  \end{equation*}
  Scalar multiplication is defined in the usual way. Is this a vector space?
  Explain why or why not.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Consider $\R^2$ with the following non-standard scalar multiplication:
  \begin{equation*}
    c\odot(a,b) = (a,cb).
  \end{equation*}
  Vector addition is defined as usual. Is this a vector space? Explain
  why or why not.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Consider $\R^2$ with the following non-standard addition operation
  $\oplus$:
  \begin{equation*}
    (a,b) \oplus (c,d) = (a-c,b-d).
  \end{equation*}
  Scalar multiplication is defined as usual. Is this a vector space?
  Explain why or why not.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Prove that the set $\Seq_K$ from
  Example~\ref{exa:vector-space-sequences} is a vector space.
  Hint: this is a special case of
  Example~\ref{exa:vector-space-function}, if you realize that a
  sequence $(a_i)_{i\in\N}$ is the same thing as a function $a:\N\to K$.
\end{ex}

\begin{ex}
  Prove that the set $\Poly$ from
  Example~\ref{exa:vector-space-polynomials} is a vector space.
\end{ex}

\begin{ex}
  Let $V$ be the set of functions defined on a set $X$ that have
  values in a vector space $W$. Is this a vector space? Explain.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Consider the set $\R^2$ with the following non-standard operations
  of addition and scalar multiplication:
  \begin{equation*}
    \begin{array}{rcl}
      (a,b) \oplus (c,d) &=& (a+c-1,\, b+d-1), \\
      k \odot (c,d) &=& (kc + (1-k),\, kd + (1-k)). \\
    \end{array}
  \end{equation*}
  Show that $\R^2$ is a vector space with these operations. Hint: the
  zero vector is not $(0,0)$, but $(1,1)$.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Consider the set $\R$ of real numbers. Addition of real numbers is
  defined in the usual way, and scalar multiplication is just
  multiplication of one real number by another. In other words, $x+y$
  means to add the two numbers and $xy$ means to multiply them. Show
  that $\R$, with these operations, is a real vector space.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $K=\Q$ be the field of rational numbers, and let $V$ be the set
  of real numbers of the form $a+b\sqrt{2}$, where $a$ and $b$ are
  rational numbers. Show that with the usual operations, $V$ is a
  $\Q$-vector space.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $K=\Q$ be the field of rational numbers, and let $V=\R$ be the
  set of real numbers. Show that with the usual operations, $V$ is a
  $\Q$-vector space.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $\Poly_3$ be the set of all polynomials of degree 3 or
  less. That is, these are of the form $ax^3+bx^2+cx+d$. Addition and
  scalar multiplication of polynomials are defined as usual.  Show
  that $\Poly_3$ is a vector space.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $X=\set{1,2,\ldots,n}$, and consider the space $\Func_{X,\R}$ of
  real-valued functions defined on $X$. Explain how $\Func_{X,\R}$ can be
  considered as $\R^n$.
  \begin{sol}
    Let $f(i)$ be the $i\th$ component of a vector
    $\vect{x}\in\R^n$. Thus a typical element in $\R^n$ is
    $(f(1),\ldots,f(n))$.
  \end{sol}
\end{ex}

\begin{ex}
  Prove the cancellation law from
  Proposition~\ref{prop:vector-space-elementary}.
\end{ex}


\subsection{Multiplication of natural numbers}

\subsection{Definition}

Multiplication can be defined by:

\(\forall a \in \mathbb{N} (a.0=0)\)

\(\forall a b \in \mathbb{N} (a.s(b)=a.b+a)\)

\subsection{Example}

Let’s calculate \(2.2\).

\(2.2=2.s(1)\)

\(2.s(1)=2.1+2\)

\(2.1+2=2.s(0)+2\)

\(2.s(0)+2=2.0+2+2\)

\(2.0+2+2=2+2\)

\(2+2=4\)

\subsection{Commutative property of multiplication}

Multiplication is commutative:

\(xy=yx\)

\subsection{Associative property of multiplication}

Multiplication is associative:

\(x(yz)=(xy)z\)

\subsection{Distributive property of multiplication}

Multiplication is distributive over addition:

\(a(b+c)=ab+ac\)

\((a+b)c=ac+bc\)


\section*{Exercises}

\begin{ex}
  Find the characteristic polynomial of the matrix
  \begin{equation*}
    \begin{mymatrix}{rr}
      3 &  -2 \\
      1 &   0 \\
    \end{mymatrix}.
  \end{equation*}
  Use the quadratic formula to find the eigenvalues.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Find the characteristic polynomial, eigenvalues, and basic
  eigenvectors of the matrix
  \begin{equation*}
    \begin{mymatrix}{rr}
      9 &  10 \\
      -5 &  -6 \\
    \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  Find the characteristic polynomial, eigenvalues, and basic
  eigenvectors of the matrix
  \begin{equation*}
    \begin{mymatrix}{rrr}
      0 &   3 &  -1 \\
      -2 &   4 &  -2 \\
      2 &  -3 &   3 \\
    \end{mymatrix}.
  \end{equation*}
  One eigenvalue is $1$.
\end{ex}

\begin{ex}
  Find the characteristic polynomial, eigenvalues, and basic
  eigenvectors of the matrix
  \begin{equation*}
    \begin{mymatrix}{rrr}
      3 &   0 &  -2 \\
      -2 &   1 &   2 \\
      0 &   0 &   1 \\
    \end{mymatrix}.
  \end{equation*}
  One eigenvalue is $3$.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Find the characteristic polynomial, eigenvalues, and basic
  eigenvectors of the matrix
  \begin{equation*}
    \begin{mymatrix}{rrr}
      9 & 2 & 8 \\
      2 & -6 & -2 \\
      -8 & 2 & -5
    \end{mymatrix}.
  \end{equation*}
  One eigenvalue is $-3$.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Which of the following matrices have no real eigenvalue?
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      1 & 1 \\
      1 & -1 \\
    \end{mymatrix},
    \quad
    B = \begin{mymatrix}{rr}
      1 & -1 \\
      1 & 1 \\
    \end{mymatrix},
    \quad
    C = \begin{mymatrix}{rr}
      0 & 1 \\
      1 & 0 \\
    \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  Find the eigenvalues and eigenvectors of the following triangular
  matrix:
  \begin{equation*}
    \begin{mymatrix}{rrr}
      3 & 2 & 2 \\
      0 & 1 & -2 \\
      0 & 0 & -1 \\
    \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  Is it possible for a non-zero matrix to have only $0$ as an eigenvalue?
  \vspace{1mm}
  \begin{sol}
    Yes. $\begin{mymatrix}{cc}
      0 & 1 \\
      0 & 0%
    \end{mymatrix}$ works.
  \end{sol}
\end{ex}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS671: Machine Learning
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 4}

Let $f_1: \mathbb{R}^n \longrightarrow \mathbb{R}$ and $f_2: \mathbb{R}^n \longrightarrow \mathbb{R}$ be two convex functions.
Prove that the function $f$ defined by $f(x) = \text{max}\{f_1(x), f_2(x)\}$ is convex.

\subsection*{Solution}

Since $f_1$ is a convex function, Eq. \ref{eq41} will hold by definition of convexity for two arbitrary points $x, y \in \mathbb{R}^n$.
\begin{equation}
f_1(tx+(1-t)y)\leq tf_1(x) + (1-t)f_1(y)
\label{eq41}
\end{equation}
Similarly,
\begin{equation}
f_2(tx+(1-t)y)\leq tf_2(x) + (1-t)f_2(y)
\label{eq42}
\end{equation}
On the other hand, since $f = \text{max}\{f_1, f_2\}$, for any point $z \in \mathbb{R}^2$, $f_1(z) \leq f(z)$ and $f_2(z) \leq f(z)$.
Therefore, $f_1(x), f_2(x) \leq f(x)$ and $f_1(y), f_2(y) \leq f(y)$.
Thus, Eq. \ref{eq41} and Eq. \ref{eq42} can be rewritten as follows.
\begin{equation}
f_1(tx+(1-t)y)\leq tf(x) + (1-t)f(y)
\label{eq43}
\end{equation}
Similarly,
\begin{equation}
f_2(tx+(1-t)y)\leq tf(x) + (1-t)f(y)
\label{eq44}
\end{equation}
But, since $f = max\{f_1,f_2\}$, $f(tx+(1-t)y)$ is either $f_1(tx+(1-t)y)$ or $f_2(tx+(1-t)y)$.
Based on Eq. \ref{eq43} and Eq. \ref{eq44}, however, this immediately results into Eq. \ref{eq45}.
\begin{equation}
f(tx+(1-t)y) = tf(x) + (1-t)f(y)
\label{eq45}
\end{equation}
And therefore, by definition, $f = max\{f_1, f_2\}$ is convex.


\section{The Cauchy-Schwarz inequality}

The Cauchy$-$Schwarz inequality states that for all vectors $x$ and $y$ of an inner product space it is true that
$$\left | \left \langle x, y \right \rangle \right | \leq \left \| x \right \| \cdot \left \| y \right \|$$


\section{DNN and CNN}
\subsection{Tensor notation}
\begin{definition}
A tensor $T$ of order $d$  is a multi-index array,
\begin{equation}
T \in \mathbb R^{n_1 \times n_2 \times \cdots \times n_d},
\end{equation}
with $i$-th dimension being $n_i$.
\end{definition}
\paragraph{Example 1: 2D grey image}
\begin{equation}\label{2DGreyImage}
T \in \mathbb{R}^{n_1 \times n_2}.
\end{equation}
\paragraph{Example 2: 2D color image}
\begin{equation}\label{2DColorImage}
T \in \mathbb{R}^{3 \times n_1 \times n_2}.
\end{equation}
\paragraph{Example 3: 3D grey image - MRI}
\begin{equation}\label{3DColorImage}
T \in \mathbb{R}^{ n_1 \times n_2 \times n_3}.
\end{equation}

\begin{definition}[Tensor Product]
If $X \in \mathbb R^{n_1 \times n_2 \times \cdots \times n_d}$ and $Y \in \mathbb R^{m_1 \times m_2 \times \cdots \times m_e}$, then the tensor product is noted as $\otimes$ with the next definition,
\begin{equation}
X \otimes Y \in \mathbb R^{n_1 \times n_2 \times \cdots \times n_d \times m_1 \times m_2 \times \cdots \times m_e},
\end{equation}
with
\begin{equation}
(X\otimes y)_{i_1, \cdots, i_d, j_1, \cdots,j_e} = X_{i_1, \cdots, i_d} Y_{j_1, \cdots, j_e}.
\end{equation}
\end{definition}
\paragraph{Example 5: Rank one matrix} If $x \in \mathbb{R}^n$ and $y \in \mathbb{R}^m$, then
\begin{equation}\label{3DColorImage}
x \otimes y \in \mathbb{R}^{n \times m},
\end{equation}
with
\begin{equation}
x \otimes y = x y^\top.
\end{equation}

\paragraph{Example 6: } If $X \in \mathbb{R}^{n_1\times n_2}$ and $Y \in \mathbb{R}^{m_1 \times m_2}$, then
\begin{equation}\label{3DColorImage}
X \otimes Y \in \mathbb{R}^{n_1 \times n_1 \times m_1 \times m_1},
\end{equation}
with
\begin{equation}
(X \otimes Y)_{i_1, i_2, j_1 ,j_2}  =  X_{i_1, i_2} Y_{j_1, j_2}.
\end{equation}

\begin{definition}[Tensor ``inner product"]
If
$$
X \in \mathbb R^{(n_1 \times n_2 \times \cdots \times n_d )\times
  (t_1\times t_2\times\cdots\times t_k)}
$$ and
$$
Y\in \mathbb R^{ (t_k\times t_{k-1}\times\cdots\times t_1)\times
  (m_1 \times m_2 \times \cdots \times m_e)},
$$
then the tensor ``inner product" with order $k$ is given by
\begin{equation}
X\odot_k Y \in \mathbb R^{n_1 \times n_2 \times \cdots \times n_d \times m_1 \times m_2 \times \cdots \times m_e},
\end{equation}
with
\begin{equation}
(X\odot_k Y)_{i_1, \cdots, i_d, j_1, \cdots,j_e}
=\sum_{s_1=1}^{t_1} \cdots\sum_{s_k=1}^{t_k}X_{i_1, \cdots, i_d, s_1,\cdots,s_k} Y_{s_k,\cdots,s_1,,j_1, \cdots, j_e}.
\end{equation}
\end{definition}
We note that
$$
X\otimes Y =X\odot_0Y
$$
For simplicity, we denote
\begin{equation}
X\cdot Y=X\odot_1Y \mbox{ and } X:Y=X\odot_2 Y  .
\end{equation}
\paragraph{Example 7: } If $x \in \mathbb{R}^{1\times n}$ and $y \in \mathbb{R}^{n}$, then
\begin{equation}\label{3DColorImage}
x \odot_1 y =xy= \sum_{i=1}^n x_{1,i} y_{i,1}.\in \mathbb{R}^{1}.
\end{equation}

\paragraph{Example 8: } If $X \in \mathbb{R}^{n_1 \times m}$ and $Y \in \mathbb{R}^{m \times n_2}$, then
\begin{equation}\label{3DColorImage}
X \odot_1 Y =XY\in \mathbb{R}^{n_1 \times n_2},
\end{equation}
with
\begin{equation}
(X \cdot Y)_{i,j} = \sum_{k=1}^m X_{i,k} Y_{k,j}.
\end{equation}
which is again the product of two matrices.

\begin{remark}
Without loss of generality, if the dimension and operation is clear, we note $\odot$ without $\odot_k$
\end{remark}

\newpage
\subsection{Single layer}
A singular layer linear neural network can be written as
$$
f(x, \theta)
=
\begin{pmatrix}
  f_1(x,\theta_1)\\
\vdots
\\
  f_m(x,\theta_m)
\end{pmatrix}
=
\begin{pmatrix}
 w_1 x+b_1, \\
\vdots
\\
 w_m x+b_m, \\
\end{pmatrix}
=
\begin{pmatrix}
 w_1\\
\vdots
\\
 w_m
\end{pmatrix}x
+
\begin{pmatrix}
b_1\\
\vdots
\\
b_m
\end{pmatrix}
=Wx+b=\theta \tilde x
$$
where
\begin{equation}
W\in \mathbb R^{m\times n},
\theta=(W,b)=
\begin{pmatrix}
  \theta_1\\
\vdots\\
\theta_m
\end{pmatrix}, \quad
\tilde x=
\begin{pmatrix}
  x\\
1
\end{pmatrix}
\end{equation}

We have
\begin{equation}
f(x; \theta )= W x+b, ~ \Theta = \{ \theta = (W,b) ~|~ W \in \mathbb{R}^{m \times n}, b \in \mathbb{R}^m \}.
\end{equation}

with $f: \mathbb{R}^n \to \mathbb{R}^m$.

\subsection{Deep neural networks (DNN)}
\begin{figure}[!ht]
\center{\includegraphics[width=12cm,height=6cm] {figures/ANN.png}}
\caption{A typical deep neural network}
\end{figure}
We consider a sequence of affine operations:
\begin{equation}
\theta^j: \mathbb R^{\hat{n}_{j}}\mapsto \mathbb R^{ n_{j}}.
\end{equation}
with
\begin{equation}\label{DNN_affinemap}
\theta^j(x) =  W^j x + b^j
=
(W^j, b^j)
\begin{pmatrix}
  x \\
1
\end{pmatrix}
=
\theta^j
\begin{pmatrix}
  x \\
1
\end{pmatrix}
\end{equation}
and
\begin{equation}
W^j\in \mathbb R^{n_j \times  \hat n_{j}}, b^j\in \mathbb R^{n_j}.
\end{equation}
With a slight abuse of notation, we also denote
\begin{equation}
\theta^j=(W^j, b^j)
=
\begin{pmatrix}
\theta^j_{1}\\
\vdots \\
\theta^j_{n_j}\\
\end{pmatrix}
\in \mathbb R^{n_j\times (\hat n_{j}+1)}.
\end{equation}

For $j=0$, we have the input data
$$
x\in \mathbb R^{\hat n_{0}}
$$
For MNIST data base, we have
$$
\hat n_0 = 784.
$$

We consider nonlinear activation function that is applied componentwise:
\begin{equation}\label{DNN_iteration_vector}
g:\mathbb R^{ n_{j-1}}\mapsto \mathbb R^{n_{j-1}},
\end{equation}
and polling functions
\begin{equation}\label{DNN_iteration_vector}
r^j:\mathbb R^{ n_{j-1}}\mapsto \mathbb R^{\hat n_{j}}.
\end{equation}
Poolling is not always applied after each application of activation.
When it is not applied, $r^j$ is identity and $\hat n_j=n_{j-1}$;   when
it is applied, $r_j$ is usually nonlinear and
$$
\hat n_j<n_{j-1}.
$$

We consider the pooled-activation functions:
\begin{equation}\label{DNN_iteration_vector}
g^j=r^j\circ g: \mathbb R^{{n}_{j-1}}\mapsto
\mathbb R^{\hat n_{j}}, \quad j =  1 :J.
\end{equation}
We note that
\begin{equation}
\theta^j\circ g^j:    \mathbb R^{{n}_{j-1}}\mapsto
\mathbb R^{n_{j}}
\end{equation}

We can then define a multi-layer neural network
	\begin{equation}\label{DNN_finallayer}
	f(x; \Theta) = f^J,
	\end{equation}
        with:
	\begin{equation}\label{DNN_iteration_vector}
	f^j(x,\Theta^j) = (\theta^j\circ g^j\circ
        f^{j-1})(x,\Theta^{j-1}), \quad j=1:J
	\end{equation}
where
\begin{equation}
\Theta^j=(\Theta^{j-1},\theta^j), \quad \Theta^0=\theta^0
\end{equation}
and
	\begin{equation}
	f^0(x)=\theta^0(x).
	\end{equation}

Here

\begin{align}
\Theta =\Theta_J =(\theta^0,\theta^1, \cdots, \theta^J)=( ({W^0}, {b^0}),({W^1}, {b^1}),
\cdots, ({W^J}, {b^J})) \\
\displaystyle
	\in  (\mathbb{R}^{n_0 \times (\hat n_0 + 1)}) \oplus \cdots
        \oplus  	(\mathbb{R}^{{n}_{J} \times (\hat
          n_{J}+1)})=
\oplus_{j=0}^J \mathbb{R}^{n_{j} \times (\hat n_{j} + 1)}
\end{align}

\subsection{Activation function}
        \begin{itemize}
	\item An general activation function(must be nonlinear) is
	$$g: \mathbb{R} \to  \mathbb{R}.$$
        \item The sigmoid function is
        $$\sigma(x) = \frac{1}{1 + e^{-x}}.$$
	\item The ReLU(rectified linear unit) is
	$$
	ReLU(x) = r(x) = \max(0, x).
	$$
	\item The Heaviside function is
	$$
	H(x ) = \begin{cases}
	0 \quad &\text{if} ~ x \le 0, \\
	1 \quad &\text{if} ~ x > 0.
	\end{cases}
	$$
	\item The new activation function as ReLU1 is:
	$$
	\tau(x) = r(x) - r(x-1) = \begin{cases}
	0 \quad &\text{if} ~ x \le 0, \\
	x \quad &\text{if} ~  0 < x \le 1, \\
	1  \quad &\text{if} ~ x > 1.
	\end{cases}
	$$

	\end{itemize}

\subsection{Classification properties under this notation}
\begin{theorem}
  If $\{A_k\}_{k=1}^c$ are separable with positive distance, then it
  can be separated by $f(x; \Theta) = f^2$, with $W^2= id, b^2=0$ and
  all activation function is $\tau$.
\end{theorem}

\subsection{CNN for images}
\subsection{General setup}
\begin{enumerate}
\item Data:
	\begin{itemize}
	\item Input set: $\bm{X} = \{x_1, \cdots, x_N\}$
	\item Output set: $\bm{y} =  \{y_1, \cdots, y_N\}$
	\item Generally,  input use $x \in \mathbb{R}^{\hat c_0 \times \hat n_1\times \cdots \times \hat n_d }$, with a label $y \in \mathbb{R}^c$ (output for classification problem)
	\item Here $\hat c_0$ means the channel dimension and $\hat n_1\times \cdots \times \hat n_d$ means the essential dimension.
	\end{itemize}
      \item Here we use $x \in \mathbb{R}^{\hat c_0 \times n \times n} $,
        $y \in \mathbb{R}^c$ as example.
\item Kernels: in CNN they generally use $K^i$ and $b^i$ to stand for the kernels and bias.
\end{enumerate}

\paragraph{Example 1: 2D grey image}
\begin{equation}\label{2DGreyImage}
c_0=1, d=2
\end{equation}
\paragraph{Example 2: 2D color image}
\begin{equation}\label{2DColorImage}
c_0=3, d=2
\end{equation}
\paragraph{Example 3: 3D grey image - MRI}
\begin{equation}\label{2DColorImage}
c_0=1, d=3
\end{equation}

\subsection{DNN for images under tensor notation}

 Multilayer structures in convolutional layers is totally the same for \eqref{DNN_affinemap} - \eqref{DNN_finallayer}: the only difference  is the structure of affine map:

\begin{equation}\label{CNN_affinemap}
\theta^j_{p}(x) =  W^j_p \odot_3 x + b^j_p \in \mathbb R^{n_j \times n_j}, \quad p=1:c_j
\end{equation}

\begin{equation}\label{CNN_affinemap}
\theta^j(x) =  W^j\odot_3 x + b^j
\end{equation}
with
\begin{equation}
\theta^j: \mathbb R^{\hat c_j  \times \hat{n}_{j} \times \hat n_{j} } \mapsto \mathbb{R}^{c_j \times n_{j} \times n_j }.
\end{equation}

Here
\begin{equation}
W^j \in \mathbb{R}^{(c_j \times n_{j} \times n_j) \times
  (\hat{n}_{j} \times \hat n_{j}\times \hat c_{j} )}, \quad
b^j\in  \mathbb R^{c_j \times n_{j} \times n_j},
\end{equation}
and
\begin{align}
\theta^j  &= \{W^j , b^j\}  \in \mathbb{R}^{(c_j \times n_{j} \times n_j) \times
  (\hat{n}_{j} \times \hat n_{j}\times \hat c_{j} )} \oplus \mathbb R^{c_j \times n_{j} \times n_j} \\
  &\cong  \mathbb{R}^{(c_j \times n_{j} \times n_j) \times
  (\hat{n}_{j} \times \hat n_{j}\times c_{j} + 1 )}
\end{align}

We note that
\begin{equation}
\hat n_0 =n  .
\end{equation}

We consider nonlinear activation function applied element wise:
\begin{equation}\label{CNN_iteration_vector}
g: \mathbb R^{c_{j-1} \times n_{j-1} \times n_{j-1} }\mapsto  \mathbb R^{c_{j-1} \times n_{j-1} \times n_{j-1}},
\end{equation}
and polling functions
\begin{equation}\label{CNN_iteration_vector}
r^j: \mathbb R^{c_{j-1} \times n_{j-1} \times n_{j-1} }\mapsto  \mathbb R^{\hat c_j \times \hat n_{j} \times\hat n_j }.
\end{equation}

We also consider ``pooled-activation'' functions:
\begin{equation}\label{CNN_iteration_vector}
g^j=r^j\circ g:  \mathbb R^{c_{j-1} \times n_{j-1} \times n_{j-1} }\mapsto
\mathbb  \mathbb R^{\hat c_j \times \hat n_{j} \times \hat n_j }, \quad j  = 1:J.
\end{equation}

We have the multi-layer DNN structure like:
	\begin{equation}\label{CNN_iteration_vector}
	f^j(x,\Theta^j) = (\theta^j\circ g^{j}\circ f^{j-1})(x,\Theta^{j-1}) \in \mathbb{R}^{c_j \times n_j \times n_j }
	\end{equation}
where
\begin{equation}
\Theta^j=(\Theta^{j-1},\theta^j), \quad \Theta^0=\theta^0
\end{equation}
	with
	\begin{equation}
	f^0(x)=\theta^0(x).
	\end{equation}

At last, we get
	\begin{equation}\label{CNN_finallayer}
	f(x; \Theta) = f^J.
	\end{equation}

        Roughly speaking, \eqref{CNN_affinemap} -
        \eqref{CNN_finallayer} is totally the same for \eqref{DNN_affinemap}
        - \eqref{DNN_finallayer} in tensor notation.

\subsection{CNN for image}
A convolution operation is a special linear mapping of the following form:
\begin{equation}
\theta^j(x) = K^j \circledast x + {\rm{diag}(b^j)} \cdot \bm{1}_{c_j \times n_j \times n_j} \quad
\in\mathbb R^{c_j\times n_j\times n_j}.
\end{equation}
with
	\begin{equation}
	K^j \in \mathbb{R}^{ c_{j} \times \hat c_{j} \times (2k+1) \times (2k+1)},  \quad \text{and} \quad b^j \in \mathbb{R}^{c_j},
	\end{equation}
	and
	\begin{equation}
	\bm{1}_{c_j \times n_j \times n_j} = 1_{c_{j}} \otimes 1_{n_j} \otimes 1_{n_j}
	\end{equation}
for $k = 0, 1, 2, 3...$.

Give $p\in \{1,\ldots c_j\}$
\begin{equation}
(\theta^j(x))_p= (K^j \circledast x)_p+b^j_p{\bm 1}_{n_j\times n_j}
\end{equation}
and
\begin{equation}
 (K^j \circledast x)_p
= \sum_{q=1}^{\hat c_{j}} K^{j}_{p,q} \ast x_q \in \mathbb{R}^{n_j \times n_j}, \quad p= 1:c_j.
\end{equation}
where
$$
K^{j}_{p,q}\in \mathbb R^{(2k+1)\times (2k+1)}, x_q\in \mathbb
R^{\hat n_{j}\times \hat n_{j}}.
$$
Here $\ast$ is the traditional convolution defined as:
\begin{equation}\label{ConvPadding}
(K \ast x)_{i,j} :=\sum_{s, t = -k}^k  K_{s+k+1,t+k+1} x_{i + s, j + t},
\end{equation}
Here we assume implicitly that a special ``padding'' is used such that
\begin{equation}
  \label{zero-padding}
x_{i,j}=0 \mbox{ if } i, j\le 0 \mbox{ or } i, j > n_j.
\end{equation}

\subsection{CNN with stride for image}
\subsubsection{Polling and stride}
If we use stride $s$, we have have the stander convolution operator with stride $s$ as:
\begin{equation}\label{equ:convstride}
(K\ast_{s} x)_{i,j} = \sum_{p, q = -k}^k K_{p+k+1,q+k+1}x_{is + p, js + q} .
\end{equation}
This stride properties in some application are often used as pooling(subsampling, coarsening).

Here we can define a special pooling operator as $S(\cdot,s)$ which likes the $C/F$ split for choosing coarse point:
\begin{equation}
S(X,s)_{i,j} = X_{is, js },
\end{equation}
then we have:
\begin{equation}\label{equ:stride}
X \ast_s K = S(X\ast K, s),
\end{equation}
with the $\ast$ and $\ast_s$ defined by \eqref{ConvPadding} and \eqref{equ:convstride}
\begin{proof}
\begin{align}
S(X \ast K,s)_{i,j} &= (X \ast K)_{is, js}  \\
&= \sum_{p, q = -k}^k K_{p+k+1,q+k+1}x_{is + p, js + q}  \\
&= (X \ast_s K)_{i,j}.
\end{align}
\end{proof}

We note convolution with stride as:
\begin{equation}
\theta^j: \mathbb R^{\hat c_j \times \hat{n}_{j} \times \hat n_{j} } \mapsto \mathbb{R}^{c_j \times n_{j} \times n_j }.
\end{equation}
with
\begin{equation}
\theta^j(x) = K^j \circledast_{s^j} x + {\rm{diag}(b^j)} \cdot \bm{1}_{c_j \times n_j \times n_j} \quad
\in\mathbb R^{c_j\times n_j\times n_j}.
\end{equation}
with
	\begin{equation}
	K^j \in \mathbb{R}^{ c_{j} \times \hat c_{j} \times (2k+1) \times (2k+1)},  \quad \text{and} \quad b^j \in \mathbb{R}^{c_j},
	\end{equation}
	and
	\begin{equation}
	\bm{1}_{c_j \times n_j \times n_j} = 1_{c_{j}} \otimes 1_{n_j} \otimes 1_{n_j}
	\end{equation}
for $k = 0, 1, 2, 3...$.

Give $p\in \{1,\ldots c_j\}$
\begin{equation}
(\theta^j(x))_p= (K^j \circledast_{s^j} x)_p+b^j_p{\bm 1}_{n_j\times n_j}
\end{equation}
and
\begin{equation}
 (K^j \circledast_{s^j} x)_p
= \sum_{q=1}^{\hat c_{j}} K^{j}_{p,q} \ast_{s^j} x_q \in \mathbb{R}^{n_j \times n_j}, \quad p= 1:c_j.
\end{equation}
where
$$
K^{j}_{p,q}\in \mathbb R^{(2k+1)\times (2k+1)}, x_q\in \mathbb
R^{\hat n_{j}\times \hat n_{j}}.
$$
Here $\ast_{s^j}$ is the stander convolution with stride as definition before.


\section{Linear equations}
\subsection{Linear and affine functions}
\textbullet Superposition condition:$f(\alpha x + \beta y) = \alpha f(x) + \beta f(y)$\\
\textbullet Such an f is called Linear\\
\textbf{Matrix vector product function}:\\
\textbullet A is $m \times n$ matrix such that $f(x)=Ax$\\
\textbullet f is linear:  $f(\alpha x + \beta y)=A(\alpha x + \beta y)=\alpha f(x) + \beta f(y)$\\
\textbullet Converse is true: If $f:R^{n} \mapsto R^{m}$ is linear, then \\
$f(x)=f(x_{1}e_{1}+x_{2}e_{2}+...x_{n}e_{n})$\\
$=x_{1}f(e_{1})+x_{2}f(e_{2})+...x_{n}f(e_{n})$
$=Ax$ with $A = [f(e_{1})+f(e_{2})+...f(e_{n})]$
\newline \textbf{Affine Functions}: $f:R^{n} \mapsto R^{m}$ is affine if it is a linear function plus a constant i.e $f(x)=Ax+b$ same as $f(\alpha x + \beta y)=\alpha f(x)+\beta f(y)$ holds for all x, y and $\alpha,\beta$ such that $\alpha+\beta=1$
\newline A and b can be calculated as \\
$A=[f(e_{1})-f(0)\thickspace f(e_{2})-f(0) ... f(e_{n})-f(0)]] ; $\\
$b= f(0)$\\
\textbullet Affine functions sometimes incorrectly called linear functions
\subsection{Linear function models}
Price elasticity of demand
$\delta_i^{price}=(p_i^{new}-p_{i})/p_{i}$: fractional changes in prices
$\delta_i^{dem}=(d_i^{new}-d_{i})/d_{i}$: fractional change in demand
Price demand elasticity model: $\delta^{dem}=E\delta^{price}$\\
\textbf{Taylor series approximation}\\
\textbullet The (first-order) Taylor approximation of f near (or at) the point z:\\
\begin{scriptsize}
$\hat{f}(x)= f(z) + \frac{\partial f}{\partial x_1}(z)(x_1 - z_1) + ... +  \frac{\partial f}{\partial x_n}(z)(x_n - z_n)$
\end{scriptsize}\\
\textbullet in compact notation:  \\$\hat{f}(x) = f (z) + Df (z)(x - z)$
\subsection{Systems of linear equations}
\textbullet set (or system) of m linear equations in n variables $x_1,.., x_n$: \\
$A_{11}x_1 + A_{12}x_2 + ... + A_{1n}x_n = b_1$\\
$A_{21}x_1 + A_{22}x_2 + ... + A_{2n}x_n = b_2$\\
.\\
.\\
$A_{m1}x_1 + A_{m2}x_2 + ... + A_{mn}x_n = b_m$\\

\textbullet \textbf{systems of linear equations classified as}\\
\thickspace -- under-determined if m < n (A wide)\\
\thickspace -- square if m = n (A square)\\
\thickspace -- over-determined if m > n (A tall)\\

\textbf{Balancing equation example}\\
\textbullet consider reaction with m types of atoms, p reactants, q products\\
\textbullet $m \times p$ reactant matrix R is defined by\\
$R_{ij}$ = number of atoms of type i in reactant $R_j$\\
for i = 1,..,m and j = 1,..,p\\
\textbullet with $a = (a_1,..,a_p)$ (vector of reactant coefficients)\\
$Ra$ = (vector of) total numbers of atoms of each type in reactants\\
\textbullet define product $m \times q$ matrix P in similar way\\
\textbullet m-vector $Pb$ is total numbers of atoms of each type in products\\
\textbullet conservation of mass is $Ra = Pb$
\textbullet  conservation of mass is\\
$[R - P][a\thickspace b]^T = 0$\\
\textbullet simple solution is $a = b = 0$\\
\textbullet to find a nonzero solution, set any coefficient (say, $a_1$) to be 1\\
\textbullet balancing chemical equations can be expressed as solving a set of m + 1 linear equations in p + q variables \\
$
\begin{bmatrix}
    R & -P \\
    e_1^T & 0

\end{bmatrix}
\begin{bmatrix}
    a \\
    b
\end{bmatrix}=e_{m+1}
$\\
(we ignore here that $a_i$ and $b_i$ should be nonnegative integers)


\section*{Ex.2.1}
\subsection*{Is the truly independent hash function universal}

Yes - the truly independent hash function is universal.
To prove it we want to show that $P(h(x) = h(y))\leq 1/m$.

The truly random hash function $h$ is an independent and identical discrete uniformly distributed random variable.
Hence $h(x)\sim U(0,m-1)$ which means $P(h(x) = j) = 1/m$ for all $j\in[m]$. So for two distinct keys $x,y \in U$
\begin{align*}
P(h(x) = h(y)) &= \sum_{j=0}^{m-1} P([h(x)=j] \cap [h(y)=j])&\\
               &= \sum_{j=0}^{m-1}   P(h(x)=j) P(h(y)=j)    &\text{equality holds as $h$ is independent}\\
							 &= m P^2(h(x)=j)                             &\text{equality holds as $h$ is identical distributed}\\
							 &= m \left(\frac{1}{m}\right)^2              &\text{equality holds as $h$ is uniform distributed}\\
							 &= \frac{1}{m}                               &
\end{align*}
Which shows that the truly independent hash function universial.


\subsection{Integral}
$integral(f,x)$ returns the integral of $f$ with respect to $x$.
The $x$ can be omitted for expressions in $x$.
The argument list can be extended for multiple integrals.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
integral(x^2)
\end{Verbatim}

$\displaystyle \frac{1}{3}x^3$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
integral(x*y,x,y)
\end{Verbatim}

$\displaystyle \frac{1}{4}x^2y^2$

$defint(f,x,a,b,\ldots)$
computes the definite integral of $f$ with respect to $x$ evaluated from
$a$ to $b$.
The argument list can be extended for multiple integrals.
The following example computes the integral of $f=x^2$
over the domain of a semicircle.
For each $x$ along the abscissa, $y$ ranges from 0 to $\sqrt{1-x^2}$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
defint(x^2,y,0,sqrt(1-x^2),x,-1,1)
\end{Verbatim}

$\displaystyle \frac{1}{8}\pi$

As an alternative, the $eval$ function can be used to compute a definite integral step by step.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
I = integral(x^2,y)
I = eval(I,y,sqrt(1-x^2))-eval(I,y,0)
I = integral(I,x)
eval(I,x,1)-eval(I,x,-1)
\end{Verbatim}

$\displaystyle \frac{1}{8}\pi$


%
% Chapter 2.2
%

\section*{2.2 Derivatives as Functions}

$$f'(x)=\lim_{h \to 0}\frac{f(x+h)-f(x)}{h}$$
\(f'(x)\) can be interpreted geometrically as the slope of the tangent line to the graph of \(f\) at the point \((x, f(x))\)
\\\\
A function \(f\) is \(\textbf{differentiable}\) at \(a\) if \(f'(a)\) exists. It is differentiable on an open interval \((a,b)\) if it is differentiable at every number in the interval.
\\\\
If \(f\) is differentiable at \(a\), the \(f\) is continuous at \(a\).

\subsection*{Ways a Function can be Discontinuous}

\begin{enumerate}
    \item corner point
    \item discontinuity
    \item vertical tangent
\end{enumerate}