\subsection{Damage Evolution and Stiffness Degredation}

The evolution of the equivalent plastic strains are formulated by assuming the stress-strain curves can be converted  into stress vs plastic strain curves where the tensile and compressive stresses are streated seperately:
\begin{equation}
\label{eqn:dam1}
\begin{aligned}
\boldsymbol{\sigma}_t = \boldsymbol{\sigma}_t(\boldsymbol{\bar{\epsilon}^{pl}_t},
	\boldsymbol{\dot{\bar{\epsilon}}^{pl}_t}) \\
\boldsymbol{\sigma}_c = \boldsymbol{\sigma}_c(\boldsymbol{\bar{\epsilon}^{pl}_c},
	\boldsymbol{\dot{\bar{\epsilon}}^{pl}_c})
\end{aligned}
\end{equation}

Loading a quasi-brittle in compression or tension causes damage in the material, which reduces the effective stiffness, weakening the unloading response. This damage is characterized by two damage variables, one of which represents the damage due to tensile loading, the other represents damage due to compressive loading.
\begin{equation}
\label{eqn:dam2}
\begin{aligned}
D_t = D_t(\boldsymbol{\bar{\epsilon}^{pl}_t}),\qquad 0 \leq D_t \leq 1 \\
D_c = D_c(\boldsymbol{\bar{\epsilon}^{pl}_c}),\qquad 0 \leq D_t \leq 1
\end{aligned}
\end{equation}

The damage in both compression and tension is a neccesarily increasing function of the equivalent plastic strains. This formulation will adopt the convention where $\boldsymbol{sigma_c}$ is positive in compression, as with the respectiove strains.
\begin{equation}
\label{eqn:dam3}
\begin{aligned}
\boldsymbol{\sigma}_t = (1-D_t)\mathbf{E}:(\boldsymbol{\epsilon_t} - \boldsymbol{\bar{\epsilon}^{pl}_t}) \\
\boldsymbol{\sigma}_c = (1-D_c)\mathbf{E}:(\boldsymbol{\epsilon_c} - \boldsymbol{\bar{\epsilon}^{pl}_c})
\end{aligned}
\end{equation}

For cyclic loading, both the compressive and tensile damage need to be considered. Two stiffness recovery factors are introduced, $s_t$ and $s_c$, which represent the stiffness recovery effects associated with stress reversals. The damage can be said to take the form of:
\begin{equation}
\label{eqn:dam4}
(1-D) = (1-s_t D_c)(1-s_c D_t),\qquad 0 \leq s_t, s_c, \leq 1
\end{equation}

In the case of tensile loading followed by compressive loading, the stiffness is assumed to completely recover


\subsection{Neural Network Design and Training}
\label{subsec:nntraining}

%\begin{wrapfigure}{r}{0.5\textwidth}
%	\centering
%    \includegraphics[width=0.35\textwidth]{img/nnlayout}
%	\caption{Network Layers}
%	\label{fig:network-layers}
%\end{wrapfigure}

The network was implemented in PyTorch \cite{Paszke:2019aa} as well as Tensorflow \cite{MartinAbadi:2015aa}. The backend was later exclusively switched to PyTorch (which is also the most common deep learning framework in Science) due to its better support of quantization. The layers of the neural network are depicted in Figure~\ref{fig:eggnet}.
The network was trained on the \emph{MNIST} dataset which consists of 60.000 images of handwritten digits. Those where split up in 50.000 images used for training and 10.000 used for evaluation. The network maps mathematically an $28 \times 28$ input image $x \in \mathbb{R}^{28,28}$ to an output of vector of probabilities $y \in \mathbb{R}^{10}$ where each value corresponds how likely the input image belongs to that class. The layers of the network are listed in Table~\ref{tab:eggnet-layers}.

\begin{table}[h]
	\centering
	\begin{tabular}{clcl}
		\toprule
		Layer & Type	& 	Data Size (output)	& Purpose \\
		\midrule
		 1 & Conv01		&	$[28,28,16]$	& Extract features ($[3,3]$ kernels)		  \\
		 2 & ReLU		&	$[28,28,16]$	& Introduce nonlinearity  \\
		 3 & Pool01		&	$[14,14,16]$	& Introduce nonlinearity, reduce dimensions  \\
		 4 & Conv02 	&	$[14,14,32]$	& Extract features 	($[3,3]$ kernels)	  \\
		 5 & ReLU		&	$[14,14,32]$	& Introduce nonlinearity  \\
		 6 & Pool02		&	$[7,7,32]$	& Introduce nonlinearity, reduce dimensions  \\
		 7 & Dense01	&	$[32]$		& Combine features  \\
		 8 & ReLU		&	$[32]$		& Introduce nonlinearity  \\
		 9 & Dense02	&	$[10]$		& Combine features  \\
		10 & Softmax 	&	$[10]$		& Normalize output  \\
		\bottomrule
	\end{tabular}
	\caption{Eggnet Layers}
	\label{tab:eggnet-layers}
\end{table}

Further for training of the network the \emph{ADAM} optimization algorithm \cite{Kingma:2014aa} was used to minimize the cross-entropy-loss function which is defined as
\begin{equation}
    J = - y  \log(h) + (1-y)  \log(1-h)
\end{equation}
This loss function is standard for classification problems where each sample belongs to exactly one class.
The ADAM algorithm can be adjusted by parameters, which control the speed of convergence. The recommended values, listed in Table~\ref{tab:train-params}, by \cite{Kingma:2014aa} were used.
\begin{table}[ht]
	\centering
    \begin{tabular}{cc}
        \toprule
            Parameter & Value \\
        \midrule
            $\alpha$   & $0.001$ \\
            $\beta_1$  & $0.9$   \\
            $\beta_2$  & $0.999$  \\
        \bottomrule
    \end{tabular}
    \caption{Network Training Parameters}
    \label{tab:train-params}
\end{table}

A useful guide for implementing convolutions can be found in \cite{dumoulin2016guide}. The training of the network yielded very high accuracy rates that are typical for the MNIST dataset, which is an easy challenge for machine learning. Even though the network performance could be improved, e.g. by hyperparameter tuning the results were acceptable for our case. The progress of the training in terms of accuracy and loss can be seen in Figure~\ref{fig:network-train-acc} respectively in Figure~\ref{fig:network-train-loss}. The final output of the network over the training is evaluated in Figure~\ref{fig:network-test-cm} for real values and in Figure~\ref{fig:network-test-qcm} for fake quantized values.

Both, PyTorch and Tensorflow/Keras are Python libraries but most \gls{acr:NN} operations are not implemented in Python directly because of performance reasons. This makes debugging more difficult which is why we reimplemented the operations we needed in Python. This was done using NumPy and SWIG, which will be discussed more extensively in Section~\ref{sec:sw-python}, and enabled us to verify the correct ordering of parameters, weights and activations.

\begin{figure}[hbtp]
\centering
\begin{subfigure}[t]{0.5\textwidth}
	\includegraphics[width=0.8\textwidth]{../../net/images/training_loss}
	\caption{Training Loss}
	\label{fig:network-train-loss}
\end{subfigure}%
~
\begin{subfigure}[t]{0.5\textwidth}
	\includegraphics[width=0.8\textwidth]{../../net/images/training_accuracy}
	\caption{Training Accuracy}
	\label{fig:network-train-acc}
\end{subfigure}
\caption[Network loss and accuracy over the training iterations]{Network loss and accuracy over the training iterations. The blue lines show spikes which occur because of the randomly selected mini batches. The orange line shows the smoothed version over 50 periods}
\label{fig:network-training-graphs}
\end{figure}

\begin{figure}[hbtp]
\centering
\begin{subfigure}[t]{0.5\textwidth}
	\includegraphics[width=0.9\textwidth]{../../net/images/cm}
	\caption{Floating Point}
	\label{fig:network-test-cm}
\end{subfigure}%
~
\begin{subfigure}[t]{0.5\textwidth}
	\includegraphics[width=0.9\textwidth]{../../net/images/qcm}
	\caption{Quantized Values}
	\label{fig:network-test-qcm}
\end{subfigure}
\caption{Confusion matrix for the floating point and quantized version of the network.}
\label{fig:network-confusion-matrix}
\end{figure}

\subsection{Quantization}
\label{sec:nn-quant}

The network is trained and created using \SI{32}{\bit} floating point values in Python. Directly porting this all the weights and biases to the FPGA is due to the limited amount of available resources not feasible. The goal is therefore to reduce the amount of required hardware cells by switching from floating point arithmetic to the less expensive integer arithmetic. Then a floating point value $v$ can be approximately represented as
\begin{equation}
	v \approx Q \cdot 2 ^{-m}
\end{equation}
where $Q$ and $m$ are integers. In our case all input values of the first layer are guaranteed to lie in the interval $[0,1]$ and all layer weights are known from training. It is therefore possible to precompute the expected range where the output values will be. Depending on this range it is then possible to select a suitable bit width for both $Q$ and $m$.

This is a cost-accuracy trade-off where higher bit widths would improve accuracy as well as increase the amount of hardware resources needed.
In \cite{Wu:2018aa} different strategies of choosing bit widths for $Q$ and $m$ are compared and they observed three main configurations, which are (from simple to advanced):
\begin{enumerate}
	\item Use a $(Q,m)$ configuration for the whole network
	\item Use a $(Q,m)$ configuration for each layer
	\item Use a $(Q,m)$ configuration for each output channel
\end{enumerate}
In the third configuration the authors could reduce the bit widths the most without sacrificing accuracy this increases the complexity in transferring the weights from layer to layer because the additional shift operations are necessary in order to adjust for the different values of $m$.
In \cite{Wu:2018aa} the authors also deduced from their experiments that the accuracy of the weights can be reduced the most, followed by the activations. By analysing the weights of our network (see Figure~\ref{fig:network-weight-distributions}) a per channel quantization is not necessary, because all weights in a Convolutional Layer are equally distributed among the output channels. Another important property that can be noted is the that the weights do have zero mean and most of the values lie very close to zero. Because of the usage of ReLU layer the situation is different for the activations where unsigned integers can be used, the distributions are shown in Figure~\ref{fig:network-activations-distributions}.

Using the distribution histograms we then derived the necessary bitwidths for $Q$ and $m$. In our experiments we were able to reduce them to \SI{8}{\bit}, if we used a single configuration for the whole network and also reducing them down to \SI{4}{\bit} if the bitwidth configuration is selected for each layer independently with an accuracy drop from around \SI{98.35}{\percent} to \SI{97.37}{\percent}. The strategy to the select the values for $(Q,m)$ was
\begin{enumerate}
	\item Find the value range of the weights and output activations of each layer
	\item Select suitable $(Q,m)$ values that most activations fall in that range
	\item Calculate the bit widths and exponents of the multiplication operation
	\item Add $\lceil \log_2(n) \rceil$ extra bits to account for the accumulation of $n$ values
	\item Compare the accumulated exponents and with the exponents of the successive layers input exponents. The difference is the amount of shift required
\end{enumerate}
It is noteworthy that the values for $m$ do not need to be stored in the final network, because those are only used to determine the amount of shifts between the layers. Also the values need to be clipped to their maximum and minimum values. The complete configuration of the network is summarized in Table~\ref{tab:quantization-linear-params}.

Ad 4 and 5: The transition from a layer to the next often changes the exponent $m$ and the available bitwidth. To account for this the values need to accordingly shifted. Also the decreased bitwidth needs clipping to maximum available values for the target bitwidth. This directly alters the behaviour of the network which should be accounted for during training, which is done via a saturated version of ReLU, defined as:
\begin{equation}
    \text{ReLU}_{\text{sat}}: ~ f(x;p) = \begin{cases}
		0 	\quad \text{if} \quad x < 0 \\
		p 	\quad \text{if} \quad x > p \\
		x	\quad \text{else}
	\end{cases}
\end{equation}

For our network only linear quantization has been used but also non-linear quantization, e.g. in a $\log_2$ way which is proposed in \cite{Lee:2017aa}. Experiments showed that using this technique even further down to \SI{3}{\bit} weights in our case.
Another optimization technique that could be explored is the systematically removing of weights (connections) of the network and reduce the amount of operations needed to be performed, a process referred to as ''pruning'' \cite{Zhu:2017aa}. This was not explicitly performed but is implicitly done by low bit quantization.

\begin{table}[hbt]
  \centering
  \begin{tabular}{lcccc}
	\toprule
    Network Part 	  & $|Q|$ & $m$ & $\pm$ & $v$ (real value range) \\
	\midrule
    Input 		 	  &  8    & 8  & $+$   & $[0,1]  $ \\
    L1: Weights 	  &  4    & 2  & $\pm$ & $[-2,2] $ \\
    L1: Intermediates & 12    & 10 & $\pm$ & $[-2,2] $ \\
    L1: Accumulated   & 16    & 10 & $\pm$ & 		   \\
    \midrule
    L1 $\to$ L2 	  & \multicolumn{4}{c}{Rshift by $10-2$ and clip values in range $[0,15]$} \\
    \midrule
    L2: Input 		  &  4    & 2  & $+$   & $[-2,2] $ 	   \\
    L2: Weights 	  &  4    & 5  & $\pm$ & $[-0.5,0.5] $ \\
    L2: Intermediates &  8    & 7  & $\pm$ & $[-1,1] $ \\
    L2: Accumulated   & 16    & 7  & $\pm$ &  			   \\
    \midrule
    L2 $\to$ L3 	  & \multicolumn{4}{c}{Rshift by $7-0$ and clip values in range $[0,15]$} \\
    \midrule
    L3: Input 		  &  4    & 0  & $+$   & $[0,15] $     \\
    L3: Weights 	  &  4    & 5  & $\pm$ & $[-0.5,0.5] $ \\
    L3: Intermediates &  8    & 5  & $\pm$ & $[-7.5,7.5] $ \\
    L3: Accumulated   & 19    & 5  & $\pm$ &               \\
    \midrule
    L3 $\to$ L4 	  & \multicolumn{4}{c}{Rshift by $5-0$ and clip values in range $[0,15]$} \\
    \midrule
    L4: Input 		  &  4    & 0  & $+$   & $[0,15] $ \\
    L4: Weights 	  &  4    & 5  & $\pm$ & $[-0.5,0.5] $ \\
    L4: Intermediates &  8    & 5  & $\pm$ & $[-7.5,7.5] $ \\
    L4: Accumulated   & 14    & 5  & $\pm$ & $[0,1] $ \\
    \bottomrule
  \end{tabular}
  \caption[Quantization parameters for the \SI{4}{\bit} network]{Quantization parameters for the \SI{4}{\bit} network. The intermediate terms are the values after the multiplication operation and the accumulated term denotes values after summing up of weighted inputs including bias in a channel.}
  \label{tab:quantization-linear-params}
\end{table}

%% WEIGHT DISTTRIBUTIONS
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \includegraphics[height=1.6in]{../../net/images/hist_cn1_k}
        \caption{Convolutional Layer 1}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
         \includegraphics[height=1.6in]{../../net/images/hist_cn2_k}
        \caption{Convolutional Layer 2}
    \end{subfigure}%
    \\
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \includegraphics[height=1.6in]{../../net/images/hist_fc1_w}
        \caption{Fully Connected Layer 1}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
         \includegraphics[height=1.6in]{../../net/images/hist_fc2_w}
        \caption{Fully Connected Layer 2}
    \end{subfigure}
    \caption[Distribution of the network weights for the different layers]{Distribution of the network weights for the different layers. It can be seen that the weights are distributed close to zero.}
    \label{fig:network-weight-distributions}
\end{figure}

%% ACTIVATIONS DISTTRIBUTIONS
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \includegraphics[height=1.6in]{../../net/images/hist_ao1}
        \caption{Convolutional Layer 1}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
         \includegraphics[height=1.6in]{../../net/images/hist_ao2}
        \caption{Convolutional Layer 2}
    \end{subfigure}%
    \\
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
        \includegraphics[height=1.6in]{../../net/images/hist_ao3}
        \caption{Fully Connected Layer 1}
    \end{subfigure}%
    ~
    \begin{subfigure}[t]{0.5\textwidth}
        \centering
         \includegraphics[height=1.6in]{../../net/images/hist_ao4}
        \caption{Fully Connected Layer 2}
    \end{subfigure}
    \caption[Distribution of the activations for a randomly selected batch of the input data]{Distribution of the activations for a randomly selected batch of the input data. The occurring minimum and maximum values were used to define the quantization bit width.}
    \label{fig:network-activations-distributions}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 3}

Prove that a simple loop contains no edge more than one.

\subsection*{Solution}

By definition, the loop $v_0 \rightarrow v_1 \rightarrow \cdots \rightarrow v_k$ is simple if $k \geq 3$ and it contains no vertex more than once except for the first and last vertex which are the same and occur only twice.

Proof is thus given by contradiction.
We assume that a simple loop might contain the edge $v_i \rightarrow v_{i+1}$ more than once, where $0 \leq i < k$.
If $i \neq 0$, $v_i$ (and $v_{i+1}$) would occur twice while neither of them are first or last vertex.
Therefore, the assumption would violate definition of the simple loop.
If $i = 0$, the path would be of the form $v_i \rightarrow \cdots \rightarrow v_i \rightarrow \cdots \rightarrow v_k$.
In this case, since $v_0$ and $v_k$ are the same, $v_i$ will occur more than twice which again violates the definition.
Therefore the assumption is false and proof is complete.


\section{Functions}

\begin{definition}
  We define the \textbf{composite} $g \circ f$ of $f : A \to B$ and $g : B \to C$ as the function $g \circ f : A \to C$ defined by the equation
  \begin{equation}
    (g \circ f)(a) = g(f(a))
  \end{equation}

  and

  \begin{equation}
    \pbrac{(a, c) \mid \text{for some } b \in B, f(a) = b \text{ and } g(b) = c}
  \end{equation}
\end{definition}

\begin{definition}
  A function $f$ is \textbf{injective} if
  \begin{equation}
    \pbra{
      f(a) = f(a')
    } \implies \pbra{
      a = a'
    }
  \end{equation}
\end{definition}

\begin{definition}
  A function $f$ is \textbf{surjective} if
  \begin{equation}
    \pbra{b \in B} \implies \pbra{
      \exists a \in A, b = f(a)
    }
  \end{equation}
\end{definition}

\begin{lemma}
  Let $f : A \to B$. If there are functions $g : B \to A$ and $h : B \to A$ such
  that $g(f(a) = a, \forall a \in A$ and $f(h(b)) = b, \forall b \in B$, then
  $f$ is bijective and $g = h = f^{-1}$.
\end{lemma}

\begin{definition}
  $f(A_0)$ is the \textbf{image} of $A_0$, and
  \begin{equation}
    f(A_0) = \pbrac{
      b \mid \exists a \in A_0, b = f(a)
    }
  \end{equation}
\end{definition}

\begin{definition}
  $f^{-1}(B_0)$ is the preimage of $B_0$ under $f$, formally
  \begin{equation}
    f^{-1}(B_0) = \pbrac{
      a \mid f(a) \in B_0
    }
  \end{equation}
\end{definition}

\section*{Exercises}

\bx{
  \ea{
    \item Suppose $x \in A_0$. Then consider

    \begin{equation*}
      f^{-1}(f(A_0)) = \pbrac{x' \mid f(x') \in f(A_0)}.
    \end{equation*}

    Since $x \in A_0$, we know $f(x) \in f(A_0)$, and so we can conclude $x \in f^{-1}(f(A_0))$.

    If $f$ is injective, then we know $f(x) = f(x'), x' \in A_0$ implies that $x = x' \implies x \in A_0$, proving equality.

    \item Suppose $y \in f(f^{-1}(B_0)) = \pbrac{y' \mid \exists x \in f^{-1}(B_0), y' = f(x)}$.

    We have $x \in \pbrac{x' \mid f(x') \in B_0}$, which means $y' = f(x) \in B_0$, so therefore $y \in B_0$.

    If $f$ is surjective, then we know $y \in B_0 \implies \exists x \in A$ such that $f(x) = y$.

    In particular, $f(x) = y \in B_0$, this set of $x \in f^{-1}(B_0)$, so we can conclude that $y \in f(f^{-1}(B_0))$.
  }
}

\bx{
  \ea{
    \item We can write some definitions first
    \begin{itemize}
      \item $f^{-1}(B_0) = \pbrac{x \mid f(x) \in B_0}$
      \item $f^{-1}(B_1) = \pbrac{x \mid f(x) \in B_1}$
    \end{itemize}

    If we know $B_0 \subset B_1$, then for some $x \in B_0$, we know $x \in B_1$.

    This means for some $f(x) \in B_0, f(x) \in B_1$ as well, so therefore $f^{-1}(B_0) \subset f^{-1}(B_1)$.

    \item
    \begin{align*}
      f^{-1}(B_0 \cup B_1) &= \pbrac{x \mid f(x) \in B_0 \cup B_1}\\
      &= \pbrac{x \mid f(x) \in B_0 \text{ or } f(x) \in B_1}\\
      &= \pbrac{x \mid f(x) \in B_0} \cup \pbrac{x \mid f(x) \in B_1}\\
      &= f^{-1}(B_0) \cup f^{-1}(B_1)
    \end{align*}

    \label{item:inverse_or}

    \item Basically the same proof as \ref{item:inverse_or}.
    \item Basically the same proof as \ref{item:inverse_or}.
    \item Suppose $x \in A_0$ means $x \in A_1$ as well.
    Consider $y \in f(A_0) = \pbrac{y' \mid y' = f(x) \text{ for some } x \in A_0}$. Because of our assumptions, it is also the case that
    $y \in \pbrac{y' \mid y' \text{ for some } x \in A_1} = f(A_1)$.

    Notice that $f(A_0) \subset f(A_1)$ does not imply that $A_0 \subset A_1$. E.g. think parabola.

    \item
    \begin{align*}
      f(A_0 \cup A_1)
      &= \pbrac{y \mid y = f(x) \text{ for some } x \in A_0 \cup A_1} \\
      &= \pbrac{y \mid y = f(x) \text{ for some } x \in A_0 \text{ or } x \in A_1} \\
      &= \pbrac{y \mid y = f(x) \text{ for some } x \in A_0} \cup \pbrac{y \mid y \text{ for some } x \in A_1} \\
      &= f(A_0) \cup f(A_1)
    \end{align*}

    \item
    \begin{equation*}
      f(A_0 \cap A_1) = \pbrac{y \mid y = f(x) \text{ for some } x \in A_0 \cap A_1}
    \end{equation*}
    which implies that $f(A_0 \cap A_1) \in f(A_0)$, since $x \in A_0$ and $f(A_0 \cap A_1) \in f(A_1)$, since $x \in A_1$.

    This means that $f(A_0 \cap A_1) \subset f(A_0) \cap f(A_1)$.

    Now, if $f$ is injective, then if we start with $y \in f(A_0) \cap f(A_1)$,
    we know $y \in \pbrac{y' \mid y' = f(x), x \in A_0}$ and
    $y \in \pbrac{y' \mid y' = f(x), x \in A_1}$. Since $f$ is injective, the
    common $y'$ values in $f(A_0)$ and $f(A_1)$ will map to the same $x$ values
    in $A_0$ and $A_1$, which means $y \in \pbrac{y' \mid y' = f(x), x \in A_0 \cap A_1} = f(A_0 \cap A_1)$.
  }
}

\bx{
  too lazy
}

\bx{
  \ea{
    \item Consider
    \begin{align*}
      (g \circ f)^{-1}(C_0)
      &= \pbrac{a \mid (g \circ f)(x) \in C_0} \tag{by definition}\\
      &= \pbrac{a \mid \text{for some } b \in B, f(a) = b, g(b) = c \in C_0} \\
      &= \pbrac{a \mid \text{for some } b \in B, f(a) = b, b \in g^{-1}(C_0)} \tag{we know $b \in g^{-1}(C_0)$ since $C_0 \subset C$}\\
      &= \pbrac{a \mid f(a) \in g^{-1}(C_0)}\\
      &= f^{-1}\pa{g^{-1}\pa{C_0}}
    \end{align*}

    Just a note for this problem, it's easy to get caught up with definitions
    and forget why we need assumptions. It might seem easy to do this problem
    without the fact that $C_0 \subset C$, but if you look at the step where we
    use that property, if $C_0 \not\subset C$, we cannot assume that $b \in
    g^{-1}(C_0)$, e.g. if $C_0$ contains elements that are not in $C$.

    \item Suppose we have
    \begin{align*}
      (g \circ f)(a) &= (g \circ f)(a')\\
      g(f(a)) &= g(f(a'))\tag{by def.}\\
      f(a) &= f(a')\tag{because $g$ is injective}\\
      a &= a'\tag{because $f$ is injective}
    \end{align*}
    therefore we conclude that $g \circ f$ is also injective.

    \item If we know that $g \circ f$ is injective,
    \begin{itemize}
      \item \AFSOC $f$ is not injective. Then $\exists a_1, a_2$ such that
      $f(a_1) = f(a_2)$ but $a_1 \neq a_2$. If this is the case, then $(g \circ
      f)(a_1) = g(f(a_1)) = g(f(a_2)) = (g \circ f)(a_2)$ which shows that $g
      \circ f$ is not injective. Which is a contradiction. Therefore, $f$ must
      be injective.

      \item It is possible for $g$ to not be injective. We can have some $b \in
      B$ that $\not\exists a \in A$ such that $f(a) = b$. In this case, we will
      not be able to find some input $a \neq a' \in $ where we break injectivity
      for $g \circ f$.
    \end{itemize}

    \item Suppose $f$ and $g$ are surjective. Now consider some $c \in C$.
    Since $g$ is surjective, we know $\exists b \in B$ such that $g(b) = c$.
    For this $b \in B$, since $f$ is surjective, we know that $\exists a \in A$
    such that $f(a) = b$.
    This means for any $c \in C$, we know $\exists a$ such that $g(f(a)) = (g
    \circ f)(a) = c$, which means $g \circ f$ is surjective.

    \item If we know that $g \circ f$ is surjective,
    \begin{itemize}
      \item It is possible for $f$ to not be surjective. Intuitively, a
      counterexample would show that there is some $b \in B$ such that
      $\not\exists a \in A$ such that $f(a) = b$. But all we have to make sure
      in our example is that whatever $g(b) = c$ maps to, $\exists a' \neq a \in
      A$ such that $(g \circ f)(a') = c$.

      \item \AFSOC $g$ is not surjective. Then $\exists c \in C$ such that
      $\not\exists b \in B$ such that $g(b) = c$. If this is the case, then
      $\not\exists a \in A$ such that $(g \circ f)(a) = c$, which means $g \circ
      f$ is not surjective. This is a contradiction, therefore $g$ must be
      surjective.
    \end{itemize}

    \item Summary should be pretty clear :) from the results above.
  }
}

\bx{
  \item Alright let's consider the two cases in this problem
  \begin{itemize}
    \item $f$ has a left inverse, i.e. $\exists g, g \circ f = i_A$.

    \AFSOC $f$ is not injective. This means that $f(a) = f(a') = b \in B$ but $a \neq a' \in A$.

    If this is the case, then $g(b) = a$ or $g(b) = a'$, but cannot be both, by the definition of a function, which means that $g \circ f$ is not $i_A$ by counterexample of either $a$ or $a'$.

    \item $f$ has a right inverse, i.e. $\exists h, f \circ h = i_B$.

    \AFSOC $f$ is not surjective. Then $\exists b \in B$ such that $\not\exists a \in A$ such that $f(a) = b$. If this is the case, then for this $b$, $(f \circ h)(b) \neq b$, in which case $f \circ h \neq i_B$.
  \end{itemize}

  \label{chap1:p5:part:a}

  \item The proofs above in \ref{chap1:p5:part:a} are by contradiction, and illustrate how to construct such a counterexample.
  \item The proofs above in \ref{chap1:p5:part:a} are by contradiction, and illustrate how to construct such a counterexample.
  \item No, these left/right inverses are unique. Not rigorous, but if you have an identity mapping and you change any of the mappings, it will no longer be an identity mapping.
  \item So we have $f$ that has both a left and right inverse, $g, h$
  respectively.In that case, we know that $f$ is injective and surjective, by
  the results of \ref{chap1:p5:part:a}, so we can conclude that $f$ is
  bijective. We have $g = h = f^{-1}$ by Lemma 2.1 in the text.

  (We could've just used Lemma 2.1 directly, but I think it's important to
  remind ourselves that showing a function is injective and surjective is a
  problem solving technique for showing a function is bijective.)
}

\bx{
  Let us draw out our function first,

  \begin{figure}[H]
    \centering
    \def\domainSize{2}
    \begin{tikzpicture}
      \begin{axis}[
        axis y line = middle,
        axis x line = middle,
      ]

      \addplot[
        domain=-\domainSize:\domainSize,
        samples=100
      ]{
        x^3 - x
      };

      \end{axis}
    \end{tikzpicture}
    \caption{Plotting $f(x) = x^3 - x$}
    \label{chap1:sec2:p6:fig:1}
  \end{figure}

  A less formal way to imagine injective functions is to use the horizontal line
  test. If you sweep a horizontal line and it intersects the plot in more than 1
  spot, then you know the function is not injective. We see our function $f(x)$
  here has issues between $[-1, 1]$.

  For surjectivity, we just need to map all of $\mathbb{R}$ on the $y$-axis. Our
  function does this nicely already, so we just have to make our function
  injective, and then make sure to keep the surjectivity.

  We have two choices here, we can either restrict our domain to be
  \begin{itemize}
    \item $D = (-\infty, -1) \cup [-1, \infty)$
    \item $D = (-\infty, -1] \cup (-1, \infty)$
  \end{itemize}

  If we use the first choice, we will get the following plot as $g$:
  \begin{figure}[H]
    \centering
    \begin{tikzpicture}
      \begin{axis}[
        axis y line = middle,
        axis x line = middle,
      ]

      \addplot[
        domain=-2:-1,
        samples=100
      ]{
        x^3 - x
      };

      \addplot[
        domain=1:2,
        samples=100
      ]{
        x^3 - x
      };

      \addplot[only marks, mark=o] table[row sep=crcr] {
        x y\\
        -1 0\\
      };

      \addplot[only marks] table[row sep=crcr] {
        x 0\\
        1 0\\
      };
      \end{axis}
    \end{tikzpicture}
    \caption{Plotting $g(x)$, a bijective function. Open circles are exclusive, closed circles are inclusive.}
    \label{chap1:sec2:p6:fig:2}
  \end{figure}

  To find $g^{-1}$, a classic algebra way to do this is to solve the function in
  terms of $x$. Because of the horizontal line issues from before, we will
  encounter, some issues, but because of our domain restriction, things should
  be ok.

  Instead of solving for $x$, since we already have a plot, we can just do a
  reflection across $y=x$, and we will get the inverse function $g^{-1}$. The
  intuitive way to think about this is that we are essentially swapping all $(x,
  y)$ coordinates to become $(y, x)$.

  \begin{figure}[H]
    \centering
    \begin{tikzpicture}
      \begin{axis}[
        axis y line = middle,
        axis x line = middle,
      ]

      \addplot[domain=-2:-1] (x^3 - x, x);

      \addplot[domain=1:2] (x^3 - x, x);

      \addplot[only marks, mark=o] table[row sep=crcr] {
        x y\\
        0 -1\\
      };

      \addplot[only marks] table[row sep=crcr] {
        x y\\
        0 1\\
      };
      \end{axis}
    \end{tikzpicture}
    \caption{Plotting $g(x)$, a bijective function. Open circles are exclusive, closed circles are inclusive.}
    \label{chap1:sec2:p6:fig:3}
  \end{figure}
}


\documentclass{memoir}
\usepackage{notestemplate}

\begin{document}
\section{Sums of Subspaces}
Let $(V,F,+,\cdot )$ be a vector space, and let $U_1,\ldots,U_m$ be subspaces. Define
\begin{align*}
	U_1 +\ldots+U_m := \left\{ u_1+\ldots+u_m \mid u_i \in U_i \text{ for } i=1,\ldots,m \right\}
\end{align*}
to be the sum of two subspaces.

\begin{exmp}
Let $V = \R^{3}$, $F=\R$ and consider
\begin{align*}
	U_1 = \left\{ (x,0,0) \mid x\in \R \right\} \in \R^{3}\\
	U_2 = \left\{ (0,y,0) \mid y \in \R \right\} \in \R^{3}
\end{align*}. Then
\begin{align*}
	U_1 + U_2 = \left\{ (x,y,0) \mid x,y\in \R \right\} .
\end{align*}
\end{exmp}

\begin{thm}[Sum of subspaces]
	If $U_1,\ldots,U_m \subseteq V$ are subspaces of $(V,F,+,\cdot )$, then $U_1+\ldots+U_m$ is the "smallest" subspace of $V$ that contains $U_1,\ldots,U_m$. That is, if there exists \(U'\subset V\) such that \(U_1,\ldots,U_m\subset U'\), then \(U_1+\ldots+U_m \subset U'\).
\end{thm}
\begin{proof}
	First we show that $U_1 + \ldots + U_m$ is a subspace of $V$. It is easy to see that $\overline{0} = \overline{0}+\ldots+\overline{0}$ and so the zero vector is in our space. We also know that
	\begin{align*}
		(u_1+\ldots+u_m) + (v_1+\ldots+v_m) = (u_1+v_1)+(u_2+v_2)+\ldots+(u_m+v_m) \in U_1+\ldots+U_m
	\end{align*}
	Finally,

	\begin{align*}
		\lambda \cdot (u_1+\ldots+u_m) = \lambda \cdot u_1 + \ldots + \lambda \cdot u_m \in U_1 + \ldots+U_m.
	\end{align*}
	which gives us that \(U_1+\ldots+U_m\) is a subspace of \(V\) as desired.\\

	Now we show that $U_i \subseteq U_1 + \ldots + U_m$. Let $u_i \in U_i.$ Then
	\begin{align*}
		u_2 = \overline{0} + \ldots + \overline{0} + u_i + \overline{0} + \ldots + \overline{0} \implies u_i \in U_1 + \ldots + U_m
	\end{align*}
	as desired.

Finally, we want to show the statement. Let $U' \subseteq V$ be a subspace with $U_1,\ldots,U_m \subseteq U'$, and let $u_1+\ldots+u_m \in U_1+\ldots+U_m$. Our work above gives us that $u_i \in U_i$. Because we have \(U_i \subset U'\), that implies that each \(u_i \in U'\), and so
\begin{align*}
	u_1 + \ldots+ u_m \in U'
\end{align*}
as desired.
\end{proof}

	\section{Direct sums}
	\begin{defn}[Direct sum]
		Let $U_1,\ldots,U_m$ be subspace of $(V,F,+,\cdot )$. If each $v \in U_1 + \ldots +U_m$ can be written in \textit{exactly} one way as
\begin{align*}
	v = u_1 + \ldots +u_m \quad u_i \in  U_i
\end{align*}
		 then we say $U_1 + \ldots + U_m$ is a \textbf{direct sum}, and denote it by
		\begin{align*}
		U_1 \oplus U_2 \oplus \ldots \oplus U_m .
		\end{align*}
	\end{defn}
	\begin{exmp}
	Let $V = \R^{3}$, $F = \R$, and consider $U_1 = \left\{ (x,y,0) \mid x,y \in R \right\} $ and $U_2 = \left\{ (0,0,z) | z \in \R \right\} $. Then $U_1 \bigoplus U_2$ is direct.\\

	But consider $U_3 = \left\{ (0,y,y) \mid y \in \R \right\} $. Then $U_1 + U_2 + U_3$ is NOT direct.
\end{exmp}
\begin{lemma}[Criterion for direct sums]
	$U_1 \oplus \ldots \oplus U_m \iff \left[ u_1 + \ldots + u_m = 0 \implies u_1,\ldots,u_m=0\right]$ where $u_i \in U_i$.
\end{lemma}
	\begin{proof}
	First we will prove the forward direction. Observe that $\overline{0} = \overline{0} + \ldots + \overline{0}$. Because it is a direct sum, it is the only way to write the zero vector, and so each $u_i \in U_i$ must be $\overline{0}$. \\

	For the reverse direction, suppose that the only way to write $\overline{0} = u_1 + \ldots + u_m$ is to take $u_i = \overline{0}$ . \\ Let $v \in U_1 + \ldots + U_m$. Suppose that
	\begin{align*}
	v = u_1+\ldots+u_m \quad u_i \in U_i \\
	v = u'_1+\ldots+u'_m \quad u'_i \in U_i
	\end{align*}
	Subtraction yields $\overline{0}=(u_1 - u'_1) + \ldots + (u_m - u'_m)$. By hypothesis, each parentheses must be zero, and so \(u_i = u'_i\) and hence the representation of \(v\) is unique.
\end{proof}

\begin{lemma}[Direct sum of two subspaces]
Let $U,W \subseteq V$ be two subspaces. Then
\begin{align*}
	U\oplus W \iff U\cap W = \left\{ \vec{0} \right\} .
\end{align*}
\end{lemma}

\end{document}


\subsection{The type of interaction that the OSLR algorithm can and cannot model, and how to improve it for the latter situation.}
    The algorithm can model linear data interactions. However, it would not perform well with data that display non-linearity, for example, a quadratic distribution of data. In this case, the model could work better by transforming the variables to square-root to get a linear relationship. Yo

    \subsection{Assumptions and Derivation of Analytical Solution}
    \textbf{Assumptions:}
    \begin{enumerate}
    \item expected residual error value is zero. $\forall : E(\varepsilon_i) = 0$
    \item zero correlation and equal variance of residuals. $\forall : Var(\varepsilon_i) = \sigma^2$
   \item the residuals are normally distributed. $\varepsilon_i \sim N(0,\sigma^2)$
   \end{enumerate}
   \textbf {Derivation:}\\
    Consider the augmented vector notation:
    \begin{equation*}
        y = Xw + \epsilon
    \end{equation*}
    To minimize the sum of squared residual errors,
    \newcommand{\argmin}{\mathop{\mathrm{argmin}}}
    \begin{equation*}
        \argmin_w ||\varepsilon||^2 = \argmin_w ||y - Xw||^2
    \end{equation*}
    Here,
    \begin{align*}
        \argmin_w ||y - Xw||^2 &= (y-Xw)^T (y-Xw)\\
        &= y^Ty - (Xw)^Ty - y^TXw + (Xw)^TXw\\
        &= y^Ty - w^TX^Ty - y^TXw + w^TX^TXw\\
        \intertext{Since $w^TX^Ty$ is scalar, so $w^TX^Ty = (w^TX^Ty)^T = y^TXw$. Therefore,}
        &= y^Ty - 2y^TXw + w^TX^TXw\\
        \intertext{applying partial derivatives with respect to $w$ and equating it to 0}
        \frac{\partial}{\partial w} (y^Ty - 2y^TXw + w^TX^TXw)   &= 0\\
        - 2y^TX + w^T(X^TX + (X^TX)^T) &= 0\\
        - 2y^TX + 2w^TX^TX &= 0\\
         \intertext{dividing by 2,}
        - y^TX + w^TX^TX &= 0\\
        \intertext{adding $y^TX$ on both sides,}
        w^TX^TX &= y^TX\\
        X^TXw &= X^Ty\\
        \intertext{multiplying by inverse of $X^TX$,}
        w &= (X^TX)^{-1}X^Ty
    \end{align*}
    \subsection{Situation when squared error loss function would not perform well and alternative loss functions.}
    This method is highly sensitive to outliers, since squares of the residual errors of outliers would be large, which would significantly influence the decision. Thus, it would not perform well on noisy data. Some alternative loss functions that could be used include the absolute loss (L1) function and Huber loss function.  BLAH BLAH


%------------------------------------------------
% FILENAME: assignment_0_key.tex
%  PROJECT: mathbootcamp
%   AUTHOR: Brett R. Devine
%    EMAIL: brett.devine@wsu.edu
%  WEBSITE: http://brettdevine.github.io
%------------------------------------------------
\documentclass[a4paper, 11pt]{article}
\usepackage{assignment_0_style}

\title{ Assignment 0 Key }
\author{ Brett Devine }
\date{ Jun 19, 2016 }

\begin{document}
\maketitle

\section{Limits}
\label{sec:limits}

\paragraph{Problem 1} Compute
\begin{align}
    \lim_{x\rightarrow 3} \frac{5x^2 - 8x -13}{x^2-5}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 1 Answer
%------------------------------------------------
\paragraph{Problem 1 Answer}
\begin{align}
    \lim_{x\rightarrow 3} \frac{5x^2-8x-13}{x^2-5} &= \frac{5(3)^2-8(3)-13}{(3)^2-5} \nonumber \\
    &= \frac{8}{2} = 2 \nonumber
\end{align}

%------------------------------------------------
%   END Problem 1 Answer
%------------------------------------------------

\paragraph{Problem 2} Compute
\begin{align}
    \lim_{x\rightarrow 3} \frac{x^4 - 81}{2x^2-5x-3}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 2 Answer
%------------------------------------------------

\paragraph{Problem 2 Answer}
Substitution of 3 into the expression will result in the indeterminate form $0/0$ so factor out the expressions causing the indeterminate form
\begin{align}
    \lim_{x\rightarrow 3} \frac{x^4 - 81}{2x^2-5x-3} &= \lim_{x\rightarrow 3} \frac{(x^2-9)(x^2+9)}{(x-3)(2x+1)} \nonumber \\
    &= \lim_{x\rightarrow 3} \frac{(x-3)(x+3)(x^2+9)}{(x-3)(2x+1)} \nonumber \\
    &= \lim_{x\rightarrow 3} \frac{(x+3)(x^2+9)}{(2x+1)} \nonumber \\
    &= \frac{((3) + 3)((3)^2 + 9)}{2(3) + 1} \nonumber \\
    &= \frac{108}{7} \nonumber
\end{align}

%------------------------------------------------
%   END Problem 2 Answer
%------------------------------------------------

\paragraph{Problem 3}
Compute
\begin{align}
    \lim_{x\rightarrow 4} \frac{3-\sqrt{x+5}}{x-4}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 3 Answer
%------------------------------------------------

\paragraph{Problem 3 Answer}
\begin{align}
    &= \lim_{x\rightarrow 4} \frac{3-\sqrt{x+5}}{x-4} \cdot \frac{3+\sqrt{x+5}}{3 + \sqrt{x+5}}  \nonumber \\
    &= \lim_{x\rightarrow 4} \frac{9-(x+5)}{(x-4)(3+\sqrt{x+5})} \nonumber \\
    &= \lim_{x\rightarrow 4} \frac{4-x}{(x-4)(3 + \sqrt{x + 5}} \nonumber \\
    &= \lim_{x\rightarrow 4} \frac{-(x-4)}{(x-4)(3 + \sqrt{x + 5})} \nonumber \\
    &= \lim_{x\rightarrow 4} \frac{-1}{3 + \sqrt{x+5}} \nonumber \\
    &= \frac{-1}{3 + \sqrt{4 + 5}} \nonumber \\
    &= -\frac{1}{6} \nonumber
\end{align}

%------------------------------------------------
%   END Problem 3 Answer
%------------------------------------------------

\paragraph{Problem 4}
Consider the values of constants $a$ and $b$ so that $\lim_{x \rightarrow 2} f(x)$ exists and is equal to $f(2)$ where $f(x)$ is defined as below.
\begin{align}
    f(x) = \begin{cases}
    			a + bx & \text{ if } x > 2 \\
    			3 & \text{ if } x=2 \\
    			b-ax^2 & \text{ if } x < 2
    	   \end{cases}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 4 Answer
%------------------------------------------------

\paragraph{Problem 4 Answer}
\begin{align}
    \lim_{x\rightarrow 2+} f(x) = \lim_{x\rightarrow 2+} (a+bx) &= a + 2b = 3  \nonumber \\
    \lim_{x\rightarrow 2-} f(x) = \lim_{x\rightarrow 2-} (b-ax^2) &= b - 4a = 3 \nonumber
\end{align}
The above expressions give us a system of equations
\begin{align}
    a + 2b &= 3  \nonumber \\
    b-4a &= 3 \nonumber
\end{align}
Solving the system of 2 equations in 2 unknowns gives us $a = -\frac{1}{3}$ and $b = \frac{5}{3}$.

%------------------------------------------------
%   END Problem 4 Answer
%------------------------------------------------

\paragraph{Problem 5}
Compute the following limit.
\begin{align}
    \lim_{x\rightarrow \infty} \frac{100}{x^2 + 5}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 5 Answer
%------------------------------------------------

\paragraph{Problem 5 Answer}
\begin{align}
    \lim_{x\rightarrow \infty} \frac{100}{x^2 + 5} = \frac{100}{\infty} = 0  \nonumber
\end{align}

%------------------------------------------------
%   END Problem 5 Answer
%------------------------------------------------

\paragraph{Problem 6}
Compute
\begin{align}
    \lim_{x\rightarrow \infty} (3x^3 - 1000x^2)  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 6 Answer
%------------------------------------------------

\paragraph{Problem 6 Answer}
Substitution yields an indeterminate form ``$\infty - \infty$'' which we circumvent by factoring.
\begin{align}
    &= \lim_{x\rightarrow 3} x^2(3x - 1000)  \nonumber \\
    &= \infty \cdot \infty \nonumber
\end{align}
which means that the limit does not exist.

%------------------------------------------------
%   END Problem 6 Answer
%------------------------------------------------

\paragraph{Problem 7}
Compute
\begin{align}
    \lim_{x\rightarrow \infty} \frac{7x^2 + x - 100}{2x^2 - 5x}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 7 Answer
%------------------------------------------------

\paragraph{Problem 7 Answer}
\begin{align}
    &= \lim_{x\rightarrow \infty} \frac{7x^2 + x - 100}{2x^2 - 5x} \cdot \frac{\frac{1}{x^2}}{\frac{1}{x^2}} \nonumber \\
    &= \lim_{x\rightarrow \infty} \frac{\frac{7x^2}{x^2} + \frac{x}{x^2} - \frac{100}{x^2}}{ \frac{2x^2}{x^2} - \frac{5x}{x^2} } \nonumber \\
    &= \lim_{x\rightarrow \infty} \frac{ 7 + \frac{1}{x} - \frac{100}{x^2} }{ 2 - \frac{5}{x} } \nonumber \\
    &= \frac{7 + 0 - 0}{2 - 0} \nonumber \\
    &= \frac{7}{2} \nonumber
\end{align}

%------------------------------------------------
%   END Problem 7 Answer
%------------------------------------------------

\paragraph{Problem 8}
Compute
\begin{align}
    \lim_{x\rightarrow \infty} \left( 3^x + 3^{2x} \right)^{\frac{1}{x}}  \nonumber
\end{align}

%------------------------------------------------
%   BEGIN Problem 8 Answer
%------------------------------------------------

\paragraph{Problem 8 Answer}
\begin{align}
    &= \lim_{x\rightarrow \infty} \left( 3^x + \left[3^2\right]^x \right)^{\frac{1}{x}}  \nonumber \\
    &= \lim_{x\rightarrow \infty} \left( 3^x + 9^x \right)^{\frac{1}{x}} \nonumber \\
    &= \lim_{x\rightarrow \infty} \left( 9^x \left[ \frac{3^x}{9^x} + \frac{9^x}{9^x} \right] \right)^{\frac{1}{x}} \nonumber \\
    &= \lim_{x\rightarrow \infty} \left( 9^x \left[ \left(\frac{3}{9}\right)^x + 1 \right] \right)^{\frac{1}{x}} \nonumber \\
    &= \lim_{x\rightarrow \infty} \left( 9^x \left[ \left(\frac{1}{3}\right) + 1 \right] \right)^{\frac{1}{x}} \nonumber \\
    &= \lim_{x\rightarrow \infty} \left(9^x \right)^{\frac{1}{x}} \cdot \left[ \left(\frac{1}{3}\right)^x + 1 \right]^{\frac{1}{x}} \nonumber \\
    &= \lim_{x\rightarrow \infty} 9 \left[ \left(\frac{1}{3}\right)^x + 1 \right]^{\frac{1}{x}} \nonumber \\
    &= (9)[0 + 1]^0 \nonumber \\
    &= 9 \nonumber
\end{align}

%------------------------------------------------
%   END Problem 8 Answer
%------------------------------------------------

\paragraph{Problem 9}
Compute $\lim_{x\rightarrow 0^+} \; x \cdot \ln x$.

%------------------------------------------------
%   BEGIN Problem 9 Answer
%------------------------------------------------

\paragraph{Problem 9 Answer}
\begin{align}
    &= \lim_{x\rightarrow 0^+} \frac{\ln x}{1/x} \nonumber \\
    &= \frac{\ln 0}{1/0^+} \nonumber
\end{align}
which leads to the indeterminate form $-\infty / \infty$ so use l'Hopital's Rule.
\begin{align}
    &= \lim_{x\rightarrow 0^+} \frac{1/x}{-1/x^2}  \nonumber \\
    &= \lim_{x\rightarrow 0^+} \frac{1}{x} \cdot \frac{x^2}{-1} \nonumber \\
    &= \lim_{x\rightarrow 0^+} (-x) \nonumber \\
    &= 0 \nonumber
\end{align}
%------------------------------------------------
%   END Problem 9 Answer
%------------------------------------------------

\paragraph{Problem 10}
Compute $\lim_{x \rightarrow 0^+} x \cdot \left( \ln x \right)^2$.

%------------------------------------------------
%   BEGIN Problem 10 Answer
%------------------------------------------------

\paragraph{Problem 10 Answer}
\begin{align}
    &= \lim_{x\rightarrow 0^+} \frac{\left(\ln x\right)^2}{1/x} = \frac{\infty}{\infty}  \nonumber
\end{align}
Use l'Hopital's Rule
\begin{align}
    &= \lim_{x\rightarrow 0^+} \frac{2\ln x \cdot (1/x)}{\frac{-1}{x^2}} = \frac{\infty}{\infty}  \nonumber
\end{align}
So use l'Hopital's Rule again
\begin{align}
    &= \lim_{x\rightarrow 0^+} \frac{2 \cdot 1/x}{\frac{1}{x^2}}  \nonumber \\
    &= \lim_{x\rightarrow 0^+} 2x \nonumber \\
    &= 2 \cdot 0 \nonumber \\
    &= 0 \nonumber
\end{align}

%------------------------------------------------
%   END Problem 10 Answer
%------------------------------------------------

\paragraph{Problem 11}
Compute $\lim_{x \rightarrow 0 } (1-x)^{1/x}$.

%------------------------------------------------
%   BEGIN Problem 11 Answer
%------------------------------------------------

\paragraph{Problem 11 Answer}
Rewrite problem
\begin{align}
    \lim_{x\rightarrow 0} (1-x)^{1/x} &= \lim_{x\rightarrow 0} e^{\ln(1-x)^{1/x}}  \nonumber \\
    &= \lim_{x\rightarrow 0} e^{(1/x)\cdot \ln(1-x)} \nonumber \\
    &= e^{\lim_{x\rightarrow 0} \frac{\ln(1-x)}{x} } \nonumber \\
    &= e^{(\ln 1)/0} = e^{0/0} \nonumber
\end{align}
Use l'Hopital's rule for the indeterminate form
\begin{align}
    &= e^{\lim_{x\rightarrow 0} \frac{\frac{1}{1-x} \cdot (-1)}{1} }  \nonumber \\
    &= e^{-1/(1-0)} \nonumber \\
    &= e^{-1} \nonumber \\
    &= \frac{1}{e} \nonumber
\end{align}
%------------------------------------------------
%   END Problem 11 Answer
%------------------------------------------------

\paragraph{Problem 12}
Consider the following functions for $w \geq 0$ and $0 < \sigma < 1$.
\begin{align}
    u(w) &= \frac{w^{1-\sigma}-1}{1-\sigma}  \nonumber \\
    u(w) &= \ln(w) \nonumber \nonumber \\
    u(w) &= \sqrt{w} \nonumber
\end{align}
For each function above, find the following limits (if they exist) where $u'(\cdot)$ and $u''(\cdot)$ represent the first and second derivatives respectively.
\begin{enumerate}[(i)]
	\item $\lim_{w\rightarrow 0} \; -u''(w)/u'(w)$.
	\item $\lim_{w\rightarrow \infty} \; -u''(w)/u'(w)$.
	\item $\lim_{w\rightarrow 0} \; (-u''(w)\cdot w)/u'(w)$.
	\item $\lim_{w\rightarrow \infty} \; (-u''(w)\cdot w)/u'(w)$.
\end{enumerate}

\paragraph{Problem 13} Differentiate $y = x^x$ (with respect to $x$).

%------------------------------------------------
%   BEGIN Problem 13 Answer
%------------------------------------------------

\paragraph{Problem 13 Answer}
$x^{x} (1 + \ln x)$.

%------------------------------------------------
%   END Problem 13 Answer
%------------------------------------------------

\paragraph{Problem 14} Differentiate $y = x^{e^x}$ (with respect to $x$).

%------------------------------------------------
%   BEGIN Problem 14 Answer
%------------------------------------------------

\paragraph{Problem 14 Answer} $x^{e^x-1} e^{x} (1 + x \ln x)$

%------------------------------------------------
%   END Problem 14 Answer
%------------------------------------------------

\paragraph{Problem 15} Compute the following limits associated with the functions $f(x) = |x|$.
\begin{align}
    &\lim_{h\rightarrow 0} \frac{f(-2 + h) - f(-2)}{h}  \nonumber \\
    &\lim_{h\rightarrow 0} \frac{f(0 + h) - f(0)}{h} \nonumber \\
    &\lim_{h\rightarrow 0} \frac{f(3 + h) + f(3)}{h} \nonumber
\end{align}

\paragraph{Problem 16}
Define the intervals (if any) over which the following function is continuous.
\[
	f(x) = \frac{7x^5 + x - 2}{x^2-4}
\]

%------------------------------------------------
%   BEGIN Problem 16 Answer
%------------------------------------------------

\paragraph{Problem 16 Answer} \; $(-\infty, -2) \cup (-2,2) \cup (2, \infty)$.

%------------------------------------------------
%   END Problem 16 Answer
%------------------------------------------------

\paragraph{Problem 17}
Show that there is a root of the equation $3x^7 - 2x^5 + x -1 = 0$ between $0$ and $1$.

%------------------------------------------------
%   BEGIN Problem 17 Answer
%------------------------------------------------

\paragraph{Problem 17 Answer}
We see that $f(0) = -1 < 0$ and $f(1) = 3 - 2 + 1 - 1 = 1 > 0$.
By the Intermediate Value Theorem it follows that there exists a number $c$ in $(0,1)$ such that $f(c)=0$ since $f(x)$ is continuous and 0 is between $f(0)$ and $f(1)$.

%------------------------------------------------
%   END Problem 17 Answer
%------------------------------------------------

\paragraph{Problem 18}
Compute the following limit associated with $f(x) = x^2 - 8x + 9$.
\[
	\lim_{x\rightarrow a} \; \frac{(x^2-8x+9)-(a^2-8a+9)}{x-a}
\]

%------------------------------------------------
%   BEGIN Problem 18 Answer
%------------------------------------------------

\paragraph{Problem 18 Answer}
\; $2a - 8$.

%------------------------------------------------
%   END Problem 18 Answer
%------------------------------------------------

\paragraph{Problem 19}
If the function $f(x)$ is differentiable in the interval $(a,b)$ and $|f'(x)| \leq B < \infty$ for all $x$ in the interval $(a,b)$ then is the maximum change in the function over any sub interval $(c,d) \subseteq (a,b)$ finite or infinite? Prove it.

\paragraph{Problem 20}
If the function $f(x)$ is differentiable on $(a,b)$, but not continuously differentiable, then is $f$ continuous everywhere on $(a,b)$? Prove it.

\paragraph{Problem 21}
Can you differentiate the expression $2x = 1$? If so what is the derivative?  If not, why not?

\paragraph{Problem 22} Find the derivative of the function
\[
	f(x) = \frac{\sqrt[4]{x}}{x^{-1} \sqrt{x^5}}
\]

%------------------------------------------------
%   BEGIN Problem 22 Answer
%------------------------------------------------

\paragraph{Problem 22 Answer}
\[
	f'(x) = - \frac{5}{4x^2 \sqrt[4]{x} }
\]

%------------------------------------------------
%   END Problem 22 Answer
%------------------------------------------------

\paragraph{Problem 23} Differentiate the function
\[
	f(x) = \frac{3x^2 - 5\sqrt{x} }{ 6x^4 }
\]

%------------------------------------------------
%   BEGIN Problem 23 Answer
%------------------------------------------------

\paragraph{Problem 23 Answer}
\[
	f'(x) = -x^{-3} + \frac{35}{12} x^{-9/2}
\]

%------------------------------------------------
%   END Problem 23 Answer
%------------------------------------------------

\paragraph{Problem 24}
Find the linearization of the function $f(x) = \sqrt[3]{1+x}$ at $a=0$ and use it to approximate the numbers $f(-0.05)$ and $f(0.1)$.
Are these approximations overestimates or underestimates?

%------------------------------------------------
%   BEGIN Problem 24 Answer
%------------------------------------------------

\paragraph{Problem 24 Answer}
\begin{align}
    L(x) &= f(0) + f'(0)(x-0) = 1 + \frac{x}{3}  \nonumber \\
    L(-0.05) &= 0.9833 \approx 0.98305 \quad \text{over estimate} \nonumber \\
    L(0.1) &= 1.0333 \approx 1.0323 \quad \text{over estimate} \nonumber
\end{align}

%------------------------------------------------
%   END Problem 24 Answer
%------------------------------------------------

\paragraph{Problem 25}
Let $f(x) = (3x-5)/(4-2x)$ and find $f^{-1}(x)$.  Then compare $f'(x)$ and $\frac{d}{dx} f^{-1}(x)$ and describe the relationship (if any).

%------------------------------------------------
%   BEGIN Problem 25 Answer
%------------------------------------------------

\paragraph{Problem 25 Answer}
The inverse function is
\[
	f^{-1}(x) = \frac{4x+5}{3+2x}
\]

%------------------------------------------------
%   END Problem 25 Answer
%------------------------------------------------

\paragraph{Problem 26}
Compute the derivative of the following function, $H(p)$, with respect to $p$ and use it to show the answer to the following questions.
\begin{align}
    H(p) &= p \log_2 \left(\frac{1}{p} \right) + (1-p)\log_2 \left(\frac{1}{1-p} \right)  \nonumber
\end{align}
\begin{enumerate}[(i)]
	\item Is there a global maximum and minimum over the interval $[0,1]$, if so, what is it?
	\item If so, over what subset of $[0,1]$, if any, is the function increasing?
	\item If so, over what subset of $[0,1]$, if any, is the function decreasing?
\end{enumerate}

%------------------------------------------------
%   BEGIN Problem 26 Answer
%------------------------------------------------

\paragraph{Problem 26 Answer}
\begin{enumerate}[(i)]
	\item Yes, there is a global max because it is a continuous function on a compact set.  The maximum is $p^* = 0.5$.
	\item The function increases over $[0,0.5]$.
	\item The function decreases over $[0.5,1]$.
\end{enumerate}

%------------------------------------------------
%   END Problem 26 Answer
%------------------------------------------------

\paragraph{Problem 27}
Consider the function $f(x) = x^4 e^x$ with domain all real numbers.
\begin{enumerate}[(i)]
	\item Find the $x$-value(s) of all roots ($x$-intercepts) of $f$.
	\item Find the $x$- and $y$-value(s) of all critical points and identify each as a local max, local min, or neither.
	\item Find the $x$- and $y$-value(s) of all global extrema and identify each as a global max or global min.
	\item Find the $x$-value(s) of all inflection points.
\end{enumerate}

%------------------------------------------------
%   BEGIN Problem 27 Answer
%------------------------------------------------

\paragraph{Problem 27 Answer}
\begin{enumerate}[(i)]
	\item The equation $x^4e^x=0$ means $x^4=0 \implies x=0$ so the only root is $x=0$.
	\item $f'(x) = 4x^3 e^x + x^4 e^x = 0 \implies x = 0,-4$.  Plugging in values shows us that $f$ has a local max at $(-4, 256e^{-4})$ and a local min at $(0,0)$.
	\item There is a global minimum at $(0,0)$, but there is no global maximum since $f(x)\rightarrow \infty$ as $x\rightarrow \infty$.
	\item Look for solutions to $f''(x) = 0$ which yields $x=\{0,-2,-6\}$.  Checking the value of $f$ and $f''$ at those points tells us that the inflection points are $x=\{-2,-6\}$ but not $x=0$.
\end{enumerate}

%------------------------------------------------
%   END Problem 27 Answer
%------------------------------------------------

\paragraph{Problem 28}
Use the Intermediate Value Theorem to show that $f(x) = x^3 - 2x - 1$ has a root on $[1,2]$.

%------------------------------------------------
%   BEGIN Problem 28 Answer
%------------------------------------------------

IVT: if $f$ is continuous on $[a,b]$ and $y$ is a number between $f(a)$ and $f(b)$, then there is a number $c$ between $a$ and $b$ such that $f(c)=y$.
For the function given above, $f(1)=-2$ and $f(2)=3$.
Since $0$ is a number between $-2$ and $3$, the IVT says there is a number $c$ between $1$ and $2$ such that $f(c)=0$; this $c$ is the desired root.

%------------------------------------------------
%   END Problem 28 Answer
%------------------------------------------------

\paragraph{Problem 29}
Does the Extreme Value Theorem say anything about the function $f(x) = x^2$ on each of the following intervals? If so what does it say?  In either case, explain why.
\begin{enumerate}[(i)]
	\item $[1,4]$
	\item $(1,4)$
\end{enumerate}

%------------------------------------------------
%   BEGIN Problem 29 Answer
%------------------------------------------------

\paragraph{Problem 29 Answer}
\begin{enumerate}[(i)]
	\item $f$ has a maximum and a mininum on $[1,4]$ by EVT.
	\item Because $(1,4)$ is not compact, the EVT does not apply and so although there may be, the theorem doesn't say anything to us in this case.
\end{enumerate}

%------------------------------------------------
%   END Problem 29 Answer
%------------------------------------------------

\paragraph{Problem 30}
Find the value of the constant $c$ that the Mean Value Theorem specifies for $f(x)=x^3 + x$ on the interval $[0,3]$.

%------------------------------------------------
%   BEGIN Problem 30 Answer
%------------------------------------------------

\paragraph{Problem 30 Answer}
MVT: If $f$ is continuous on $[a,b]$, then there is a number $c$ between $a$ and $b$ such that $f'(c) = (f(b)-f(a))/(b-a)$.

For the provided function we have $\frac{f(3)-f(0)}{3-0} = \frac{30-0}{3} = 10$.
And $f'(x) = 3x^2 + 1$ so we know $f'(c) = 3c^2 + 1$.
Solving $f'(c)=3c^2 + 1 = 10$ for $c$ yields $c = \sqrt{3}$.

%------------------------------------------------
%   END Problem 30 Answer
%------------------------------------------------

\paragraph{Problem 31}
For the equation $x^3 + y^3 = \ln(xy) - 1$ use implicit differentiation to find $\dd y/ \dd x$.

%------------------------------------------------
%   BEGIN Problem 31 Answer
%------------------------------------------------

\paragraph{Problem 31 Answer}
\begin{align}
    \frac{\dd y}{\dd x} &= \frac{y - 3x^3 y}{3xy^3-x} = \frac{\frac{1}{x}-3x^2}{3y^2-\frac{1}{y}}  \nonumber
\end{align}

%------------------------------------------------
%   END Problem 31 Answer
%------------------------------------------------

\paragraph{Problem 32}
Consider the function
\[
	f(x) = \begin{cases}
				b-x^2 & \text{ if } x < 3 \\
				ax & \text{ if } x \geq 3
	 	   \end{cases}
\]
\begin{enumerate}[(i)]
	\item What condition(s) must be placed on the constants $a$ and $b$ in order for $f$ to be continuous on $(-\infty, \infty)$?
	\item For what values of the constants $a$ and $b$ will $f$ be differentiable on $(-\infty, \infty)$?
\end{enumerate}

%------------------------------------------------
%   BEGIN Problem 32 Answer
%------------------------------------------------

\paragraph{Problem 32 Answer}
\begin{enumerate}[(i)]
	\item The only problem area is $x=3$ so we want $\lim_{x \rightarrow 3^-} f(x) = \lim_{x\rightarrow 3^+} f(x)$.
	This gives us limits $b-9$ and $3a$ so we need $3a=b-9 \iff a = (1/3)b - 3$.
	\item Diffentiate piecewise to find $f'(x)$ for $x < 3$ and $x > 3$ giving us $-2x$ and $a$ respectively.
	Existence will require that $\lim_{x\rightarrow 3^-} f'(x) = \lim_{x\rightarrow 3^+} f'(x) = f'(3)$.
	The piecewise limits of the derivatives are $-6$ and $a$ so $a=-6$ and using the equation found before we know $3(-6) = b-9$ which implies $-18 + 9 = -9 = b$ so we have $\{a=-6, b=-9\}$.
\end{enumerate}

%------------------------------------------------
%   END Problem 32 Answer
%------------------------------------------------

\paragraph{Problem 33}
Consider the function $f(x) = \ln(x^2)$.
Find the fourth-order Taylor polynomial for $f(x)$ centered at $x_0=1$.

%------------------------------------------------
%   BEGIN Problem 33 Answer
%------------------------------------------------

\paragraph{Problem 33 Answer}
\[
	P_3(x) = 2(x-1) - (x-1)^2 + \frac{2}{3}(x-1)^3 - \frac{1}{2}(x-1)^4
\]

%------------------------------------------------
%   END Problem 33 Answer
%------------------------------------------------

\paragraph{Problem 34}
Let $X$ be a random variable with a probability density function (PDF) $f(x)$
\[
	f(x) = \begin{cases}
 				ce^{-x/3} & \text{ for } x > 0 \\
 				0 & \text{ otherwise }
 		   \end{cases}
\]
Remember that a PDF has the property that $\int_{\infty}^{\infty} f(x)\dd x = 1$.
\begin{enumerate}[(i)]
	\item Find the value of the constant $c$ that makes $f(x)$ a valid PDF.
	\item Find the probability that $X \leq 1/4$.
\end{enumerate}

%------------------------------------------------
%   BEGIN Problem 34 Answer
%------------------------------------------------

\paragraph{Problem 34 Answer}
\begin{enumerate}[(i)]
	\item $c = - \frac{1}{3}$.
	\item $1 - e^{-1/12}$.
\end{enumerate}

%------------------------------------------------
%   END Problem 34 Answer
%------------------------------------------------

\paragraph{Problem 35}
Find the derivative of the following function
\[
	G(x) = \int_{1}^{\sin{x}} t \dd t
\]

%------------------------------------------------
%   BEGIN Problem 35 Answer
%------------------------------------------------

\paragraph{Problem 35 Answer}
\[
	G'(x) = \sin{x} \cdot \cos{x}
\]

%------------------------------------------------
%   END Problem 35 Answer
%------------------------------------------------

\paragraph{Problem 36}
Find the derivative of the following function
\[
	G(x) = \int_{72}^{\ln(x)} t \dd t
\]

%------------------------------------------------
%   BEGIN Problem 36 Answer
%------------------------------------------------

\paragraph{Problem 36 Answer}
\[
	G'(x) = \frac{\ln x}{x}
\]

%------------------------------------------------
%   END Problem 36 Answer
%------------------------------------------------

\paragraph{Problem 37}
Suppose that $\sum_{n=1}^{\infty} a_n$ represents a convergent series and that no term of the series equals zero, i.e., $a_n \neq 0$ for all $n=1,2, \dots$.
Prove that $\sum_{n=1}^{\infty} 1/a_n$ is a divergent series.

%------------------------------------------------
%   BEGIN Problem 37 Answer
%------------------------------------------------

\paragraph{Problem 37 Answer}
If $\sum_{n=1}^{\infty} a_n$ is a convergent series, then $a_n \rightarrow 0$ (i.e., the terms go to zero).
If $a_n \rightarrow 0$, then $|1/a_n| \rightarrow \infty $.
As the terms of $\sum_{n=1}^{\infty} 1/a_n$ do not go to zero, by the $n$th term test for divergence, the series diverges.

%------------------------------------------------
%   END Problem 37 Answer
%------------------------------------------------

\paragraph{Problem 38}
Determine whether the give sequence is increasing, decreasing, or not monotonic.
Is the sequence bounded?  On the basis of what you find, does the series converge, diverge or can't be determined?
\[
	a_n = \frac{1}{5^n}
\]

%------------------------------------------------
%   BEGIN Problem 38 Answer
%------------------------------------------------

\paragraph{Problem 38 Answer}
The sequence is \textbf{decreasing} and the sequence is \textbf{bounded} between 0 and 1 and by the monotonic sequence theorem the sequence is \textbf{convergent}.

%------------------------------------------------
%   END Problem 38 Answer
%------------------------------------------------

\paragraph{Problem 39}
Differentiate the function $y = 3(x^2-1)^3 (x^2 + 1)^5$.

%------------------------------------------------
%   BEGIN Problem 39 Answer
%------------------------------------------------

\paragraph{Problem 39 Answer}
\[
	y' = 18x(x^2-1)^2 (x^2+1)^5 + 30x(x^2-1)^3 (x^2 + 1)^4
\]

%------------------------------------------------
%   END Problem 39 Answer
%------------------------------------------------

\paragraph{Problem 40}
Consider a continuously differentiable utility function $u(\cdot)$ such that $u'>0$ and $u'' < 0$.
Utility comes from income $I$ which takes on two different values $I_H(x)$ and $I_L(x)$ and that the probability of that $I=I_H$ is given by $p(x)$ where $x\geq 0$ and $p'(x)>0$.
Differentiate the expected utility with respect to $x$.
\[
	E[u] = p(x)u[I_H(x)] + (1-p(x))u[I_L(x)]
\]

\paragraph{Problem 41}
Consider a twice continuously differentiable utility function $U(\cdot)$ such that $U'>0$ and $U''< 0$.
Utility comes from wealth $W$ and a person takes a gamble $h$ that represents a gain or loss to the a person's wealth and $h=0$ on average ($E(h)=0$).
Let $p$ represent the size of an insurance premium paid to avoid taking the gamble and makes the person exactly indifferent between the gamble $h$ and paying $p$ with certainty.
Let $E$ represent the expectation operator.
Use the identity below to answer the questions.
\[
	E[U(W + h)] \equiv U(W-p)
\]
\begin{enumerate}[(i)]
	\item Derive a linear (first order) Taylor approximation of the right-hand side of the identity.
	\item Derive a quadratic (second order) Taylor approximation of the left-hand side of the identity.
	\item Let $r(W) = -U''(W)/U'(W)$ (this is the Arrow-Pratt measure of risk-aversion) use the previous approximations and the identity relationship to sign the derivative $\dd P / \dd r(W)$.  What is the sign and what relationship does this imply about the level of risk aversion and the amount of insurance premium an individual is willing to pay?
\end{enumerate}

\paragraph{Problem 42}
A new machine as a rental rate $v(s)$ at any time $s$ and depreciates at rate $d$.
With interest rate $r$ the present discounted value of this machine is
\[
	PDV(t) = \int_{t}^{\infty} e^{(r+d)t} v(s) e^{-(r+d)s} \dd s
\]
Let $p(t)$ be the purchase price of the machine at time $t$.
In equilibrium the purchase price at time $t$, $p(t)$ will equal the present discounted value and hence we have
\[
	p(t) = \int_{t}^{\infty} e^{(r+d)t} v(s)e^{-(r+d)s} \dd s
\]
\begin{enumerate}[(i)]
	\item Using the last expression for $p(t)$, find $\frac{dp(t)}{dt}$.
	\item Using your expression for $\frac{dp(t)}{dt}$ solve for $v(t)$ in terms of $p(t)$ to show the relationship between a new machines rental rate at time $t$ and the machine's purchase price at time $t$.
\end{enumerate}

\paragraph{Problem 43}
A forester must decide when to cut down a growing tree.
The value at any time $t$ is given by $f(t)$ where $f' > 0$ and $f'' < 0$ and there was an initial investment of $L$.
The continuous interest rate is $r$.
The forester must choose $t$ (the time of harvest) to maximize the present discounted value of her profits.
\[
	PDV(t) = e^{-rt}f(t) - L
\]
\begin{enumerate}[(i)]
	\item Differentiate $PDV(t)$ with respect to $t$.
	\item Use the derivative as a first order condition to characterize the relationship between the $t$ that maximizes the forester's present discounted value of profit and the interest rate $r$.
\end{enumerate}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 5}

Give asymptotic upper and lower bounds for $T(n)$ in each of the following recurrences.
Assume that $T(n)$ is constant for $n \leq 2$.
Make your bounds as tight as possible, and justify your answers.

\begin{enumerate}[label=(\alph*)]
\item $T(n) = 2T(n/2) + n^4$.
\item $T(n) = T(7n/10) + n$.
\item $T(n) = 16T(n/4) + n^2$.
\item $T(n) = 2T(n/4) + \sqrt{n}$.
\item $T(n) = T(n-2) + n^2$.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]
\item $T(n) = 2T(n/2) + n^4$

Recurrence is of the form $T(n) = aT(n/b)+f(n)$ where $a=2 \geq 1$, $b=2 > 1$ and $f(n) = n^4$ is ultimately positive.
Therefore the \emph{Master Theorem} can be applied, according to which $T(n)$ can be asymptotically estimated as follows.

Since $f(n) = n^4$ is significantly large compared to $n^{\log_b a}=n$, $f(n)$ can be written as $f(n) = \Omega(n^{1+\epsilon})$ for $0 < \epsilon \leq 1$ provided that for some $0<c<1$ and $n_0$, if $n > n_0$, $af(n/b) \leq c f(n)$.
This is true because by choosing $0.125 \leq c < 1$, $2f(n/2)=2\frac{n^4}{16}\leq cf(n)=cn^4$ for all $n$.

Therefore,
\begin{equation}
T(n) = \Theta(f(n)) = \Theta(n^4)
\end{equation}

\item $T(n) = T(7n/10) + n$

Recurrence is of the form $T(n) = aT(n/b) + f(n)$ where $a = 1 \geq 1$, $b = \frac{10}{7} > 1$ and $f(n) = n$ is ultimately positive.
With conditions of the \emph{Master Theorem} satisfied, $T(n)$ can be asymptotically estimated as follows.

As $\log_b a = \log_{10/7} 1 = 0$, $f(n) = n$ is larger than $n^{\log_b a} = 1$.
Therefore, $f(n) = \Omega(n^\epsilon)$ for $0 < \epsilon \leq 1$ provided that for some $0<c<1$ and $n_0$, if $n > n_0$, $af(n/b) \leq c f(n)$.
This is true because by choosing $0.7 \leq c < 1$, $f(7n/10) = 0.7n \leq cf(n) = cn$ for all $n \geq n_0 = 0$.

Therefore,
\begin{equation}
T(n) = \Theta(f(n)) = \Theta(n)
\end{equation}

\item $T(n) = 16T(n/4) + n^2$

Recurrence is of the form $T(n) = aT(n/b)+f(n)$ where $a=16 \geq 1$, $b=4 > 1$ and $f(n) = n^2$ is ultimately positive.
Therefore the \emph{Master Theorem} can be applied, according to which $T(n)$ can be asymptotically estimated as follows.

Since $f(n) = n^2$ is comparable to $n^{\log_b a}=n^{\log_4 16}=n^2$, we have $ f(n) = \Theta(n^2)$ and thus

\begin{equation}
T(n) = \Theta(n^{\log_b a}\log n) = \Theta(n^2\log n)
\end{equation}

\item $T(n) = 2T(n/4) + \sqrt{n}$

Recurrence is of the form $T(n) = aT(n/b)+f(n)$ where $a=2 \geq 1$, $b=4 > 1$ and $f(n) = \sqrt{n}$ is ultimately positive.
Therefore the \emph{Master Theorem} can be applied, according to which $T(n)$ can be asymptotically estimated as follows.

Since $f(n) = \sqrt{n}$ is comparable to $n^{\log_b a}=n^{\log_4 2}=\sqrt{n}$, we have $ f(n) = \Theta(\sqrt{n})$ and thus

\begin{equation}
T(n) = \Theta(n^{\log_b a}\log n) = \Theta(\sqrt{n} \log n)
\end{equation}

\item $T(n) = T(n-2) + n^2$

It is claimed that the given recurrence is bound by $T(n) = \mathcal{O}(n^2)$.
Proof is given by induction on $n$.

\emph{Base step}: As stated by the question, $T(n)$ is constant for $n \leq 2$.
Let $T(n) = c$ for any $n \leq 2$.
We can take any $n_0 \leq 2$ as base step of the induction and show that $T(n) = c + n^2 = \mathcal{O}(n^2)$.
This is directly obtained by choosing $n_0 = 2$ and $d = c$ to satisfy $c + n^2 \leq d n^2$.

\emph{Induction Step}: Inductive hypothesis is formed as $T(n) = \mathcal{O}(n^2)$ for any $n \leq n_1$.
It suffices to show that $T(n_1+1) = \mathcal{O}(n^2)$.
Using the inductive hypothesis, $T(n_1+1) = T(n_1-1) + n^2$.
But it is explicitly stated by inductive hypothesis that $T(n_1-1) = \mathcal{O}(n^2)$.
We also know that $n^2 = \mathcal{O}(n^2)$.
Therefore, $T(n_1+1) = \mathcal{O}(n^2) + \mathcal{O}(n^2) = \mathcal{O}(n^2)$.

Therefore,
\begin{equation}
T(n) = \mathcal{O}(n^2)
\end{equation}

\end{enumerate}


\documentclass{notes}

  \title{Pumping Lemma}
  \author{ian.mcloughlin@gmit.ie}
  \date{\today}

\begin{document}

  \section*{Theorem}
    Let \(L\) be an infinite regular language.
    Let \(p\) be the number of states in a deterministic finite automaton that recognises \(L\).
    Then any string \(s\) in \(L\) of length at least \(p = |s|\) can be broken into three substrings \(s=xyz\) such that:
    \begin{itemize}
      \item \(|y| > 0\) ,
      \item \(|xy| \leq p\), and
      \item \(xy^iz \in L\) for all \(i \in \mathbb{N}_0\).
    \end{itemize}

    \section*{Rationale}
      Once we read \(p\) characters from \(s\), we must have visited some state twice\footnote{At least one state at least twice.}.
      Suppose \(q\) is a state we visit twice, and call \(y\) the substring of \(s\) that we read between the two visits.

      When we can delete \(y\) from \(s\) and the automaton must accept this new string also.
      Likewise we can repeat \(y\) any number of times to create a new string that must also be accepted.

      Note the automaton essentially forgets the path it took to a given state -- once it arrives at a given state it can't remember how it got there.

    \section*{Example}

    \section*{Non-regular example}
      \[L = \{0^i1^i | i \in \mathbb{N}_0\}\]
      \[s = 0^p1^p = xyz \]
      \[|xy| \leq p \Rightarrow y = 0^n, n \in \mathbb{N}\]
      \[\Rightarrow 0^{p-n}0^{2n}1^n \in L \]
      That's a contradiction.

  %\bibliography{bibliography}
\end{document}