\chapter{The Combined Filtering Method}
% \section{The Combined Filtering Method}
\section{Introduction}
\label{sect_comb}

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_resi_lena_f24.ps,bbllx=2cm,bblly=13.5cm,bburx=13cm,bbury=24.5cm,width=7.5cm,height=7.5cm,clip=}
\psfig{figure=fig_resi_lena_cur.ps,bbllx=2cm,bblly=13.5cm,bburx=13cm,bbury=24.5cm,width=7.5cm,height=7.5cm,clip=}
}}
\caption{Residual for thresholding of the undecimated
wavelet transform and thresholding of
the curvelet transform.}
\label{fig_lenna_resi}
\end{figure}

Although the results obtained by simply thresholding the curvelet expansion
are encouraging, there is of course ample room for further
improvement. A quick inspection of the residual images for both the
wavelet and curvelet transforms shown in Figure~\ref{fig_lenna_resi}
reveals the existence of very different features.  For instance,
wavelets do not restore long edges with high fidelity while curvelets
are seriously challenged by small features such as {\tt Lena}'s eyes. Loosely
speaking, each transform has its own area of expertise and this
complementarity may be of great potential. This section will develop a
denoising strategy based on the idea of combining both transforms.

In general, suppose that we are given $K$ linear transforms $T_1,
\ldots, T_K$ and let $\alpha_k$ be the coefficient sequence of an
object $x$ after applying the transform $T_k$, i.e. $\alpha_k = T_k
x$. We will suppose that for each transform $T_k$ we have available a
reconstruction rule that we will denote by $T^{-1}_k$ although this is
clearly an abuse of notation.  Finally, $T$ will denote the block
diagonal matrix with the $T_k$'s as building blocks and $\alpha$ the
amalgamation of the $\alpha_k$'s.

A hard thresholding rule associated with the transform $T_k$ synthesizes
an estimate $\tilde{s}_k$ via the formula
\begin{equation}
\label{eq:ht}
\tilde{s}_k = T_k^{-1} \delta(\alpha_k)
\end{equation}
where $\delta$ is a rule that sets to zero all the coordinates of
$\alpha_k$ whose absolute value falls below a given sequence of
thresholds (such coordinates are said to be non-significant).

In practice, a widely used approach is to compute the average of the
$\tilde{s}_k$'s giving a reconstruction of the form
 \begin{equation}
\tilde{s} = \sum_k \tilde{s}_k/K.
\end{equation}
For instance, in the literature of image processing it is common to
average reconstructions obtained after thresholding the wavelet
coefficients of translated versions of the original dataset
(cycle-spinning), i.e. the $T_k$'s are obtained by composing
translations and the wavelet transform.  In our setup, we do not find
this solution very appealing since this creates the opportunity to
average high-quality and low-quality reconstructions.

\section{The Combined Filtering Principle}

Given data $y$ of the form $y = s + \sigma z$, where $s$ is the image
we wish to recover and $z$ is standard white noise, we propose solving
the following optimization problem \cite{starck:spie01a}:
\begin{equation}
  \label{eq:l1-min}
  \min \|T\tilde{s}\|_{\ell_1}, \quad \mbox{subject to} \quad s \in C,
\end{equation}
where $C$ is the set of vectors $\tilde{s}$
which obey the linear constraints
\begin{equation}
\label{eq:constraints}
\left\{  \begin{array}{ll}
  \tilde{s} \ge 0, \\
  |T\tilde{s} - Ty| \le e;
  \end{array}
  \right.
\end{equation}
here, the second inequality constraint
only concerns the set of significant coefficients,
i.e. those indices $\mu$ such that $\alpha_\mu =
(Ty)_\mu$ exceeds (in absolute value) a threshold $t_\mu$. Given a
vector of tolerance $(e_\mu)$, we seek a solution whose coefficients
  $(T\tilde{s})_\mu$ are within $e_\mu$ of the noisy
empirical $\alpha_\mu$'s.  Think of $\alpha_\mu$ as being given by
\[
y = \langle y, \varphi_\mu \rangle,
\]
so that $\alpha_\mu$ is normally distributed with mean $\langle f,
\varphi_\mu \rangle$ and variance $\sigma^2_\mu = \sigma^2
\|\varphi_\mu\|^2_2$. In practice, the threshold values range
typically between three and four times the noise level $\sigma_\mu$
and in our experiments we will put $e_\mu = \sigma_\mu/2$. In short,
our constraints guarantee that the reconstruction will take into
account any pattern which is detected as significant by  any of the
$K$ transforms.

We use an $\ell_1$ penalty on the coefficient sequence because we are
interested in {\em low complexity} reconstructions. There are other
possible choices of complexity penalties; for instance, an alternative
to (\ref{eq:l1-min}) would be
\[
\label{eq:tv-min}
  \min \|\tilde{s}\|_{TV}, \quad \mbox{subject to} \quad s \in C.
\]
where $\|\cdot\|_{TV}$ is the Total Variation norm, i.e.\ the discrete
equivalent of the integral of the Euclidean norm of the gradient.

\section{The Minimization Method}

We propose solving (\ref{eq:l1-min}) using the method of hybrid
steepest descent (HSD) \cite{wave:yamada01}. HSD consists of building
the sequence
\begin{eqnarray}
 s^{n+1} = P(s^{n}) - \lambda_{n+1} \nabla_J(P(s^{n}));
\end{eqnarray}
Here, $P$ is the $\ell_2$ projection operator onto the feasible set
$C$, $\nabla_J$ is the gradient of equation~\ref{eq:l1-min}, and
$(\lambda_{n})_{n \ge 1}$ is a sequence obeying $(\lambda_{n})_{n\ge
  1} \in [0,1] $ and $\lim_{ n \rightarrow + \infty } \lambda_{n} = 0$.

Unfortunately, the projection operator $P$ is not easily determined
and in practice we will use the following proxy; compute $T \tilde{s}$
and replace those coefficients which do not obey the constraints $|T
\tilde{s} - Ty| \le e$ (those which fall outside of the prescribed
interval) by those of $y$; apply the inverse transform.

% {\tt Needs work.} Then apply thresholding. This approximation gives
The combined filtering algorithm is:
\begin{enumerate}
\baselineskip=0.4truecm
\itemsep=0.1truecm
\item Initialize $L_{\max} = 1$, the number of iterations $N_i$, and
  $\delta_{\lambda} = \frac{L_{\max}}{N_i}$.
\item Estimate the noise standard deviation $\sigma$, and set $e_k =
  {\sigma \over 2}$.
\item For k = 1, .., $K$ calculate the transform: $\alpha^{(s)}_k
  = T_k s$.
\item Set $\lambda = L_{\max}$, $n = 0$, and $\tilde s^{n}$ to 0.
\item While $\lambda >= 0$ do
\begin{itemize}
\item $u = \tilde s^{n}$.
\item For k = 1, .., $K$ do
  \begin{itemize}
  \item Calculate the transform $\alpha_{k} = T_k u$.
  \item For all coefficients $\alpha_{k,l}$ do
     \begin{itemize}
     \item Calculate the residual $r_{k,l} = \alpha^{(s)}_{k,l} -
       \alpha_{k,l}$

     \item if $\alpha^{(s)}_{k,l}$ is significant and $ \mid r_{k,l}
       \mid > e_{k,l}$ then $\alpha_{k,l} = \alpha^{(s)}_{k,l}$
     \item $\alpha_{k,l} = sgn(\alpha_{k,l}) ( \mid \alpha_{k,l} \mid - \lambda)_{+}$.
     \end{itemize}
   \item $u = T_k^{-1} \alpha_{k}$
  \end{itemize}
\item Threshold negative values in $u$ and $\tilde s^{n+1} = u$.
\item $n = n + 1$, $\lambda = \lambda - \delta_{\lambda} $, and goto 5.
\end{itemize}
\end{enumerate}

\section{Experiments}

\voffset -1truecm
\begin{table}[htb]
\begin{center}
\begin{tabular}{lccccc} \hline \hline
Method                          & PSNR   &  Comments   \\ \hline \hline
Noisy image                     & 22.13  &     \\
OWT7-9 + k-sigma  Hard thresh.   & 28.35  &   many artifacts \\
UWT7-9 + k-sigma  Hard thresh.   & 31.94  &   very few artifacts \\
Curvelet (B=16)                 & 31.95  &   no artifact  \\
Combined filtering              & 32.72  &   no artifact  \\ \hline \hline
\end{tabular}
\caption{PSNR after filtering the simulated image (Lena + Gaussian
noise, sigma=20).
In the combined filtering, a curvelet and an undecimated wavelet
transform were used.}
\vspace{0.5cm}
\label{comptab2}
\end{center}
\end{table}

The noisy {\tt Lena} image (the noise standard deviation being equal 20)
was filtered by the undecimated
wavelet transform, the curvelet transform, and by our combined
transform approach (curvelet and undecimated wavelet transforms). The
results are reported in Table~\ref{comptab2}.
Figure~\ref{fig_cb2_lenna} displays the noisy image (top left), and
the restored image after denoising by the combined transforms (bottom right).
Details are
displayed in Figure~\ref{fig_cb2_lenna} bottom left.
Figure~\ref{fig_cb2_lenna}  bottom  right shows the full residual image, and
can be compared  to the residual images shown in Figure~\ref{fig_lenna_resi}.
The residual is much better when the combined filtering is applied, and no
feature can be detected any more by eye. This was not the case for either the
wavelet and the curvelet filtering.

\begin{figure}[htb]
\centerline{
\vbox{
\hbox{
 \psfig{figure=fig_lena_g20.ps,bbllx=2cm,bblly=13.5cm,bburx=13cm,bbury=24.5cm,width=7.5cm,height=7.5cm,clip=}
 \psfig{figure=fig_lenna_cfil_wt_cur.ps,bbllx=2cm,bblly=13.5cm,bburx=13cm,bbury=24.5cm,width=7.5cm,height=7.5cm,clip=}
 }
\hbox{
\psfig{figure=fig_lena_cfil_rid_cur_x185_y342_sub.ps,bbllx=2cm,bblly=13.5cm,bburx=13cm,bbury=24.5cm,width=7.5cm,height=7.5cm,clip=}
\psfig{figure=fig_resi_cfil_wt_cur.ps,bbllx=2cm,bblly=13.5cm,bburx=13cm,bbury=24.5cm,width=7.5cm,height=7.5cm,clip=}
}
}}
\caption{Noisy image (top left), and filtered image based on the
  combined transform (top right). Bottom left panel shows a detail
  of the filtered image. The full residual image is displayed on the bottom right.}
\label{fig_cb2_lenna}
\end{figure}

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_ctm_iter.ps,bbllx=2.5cm,bblly=13.cm,bburx=19.5cm,bbury=25cm,width=10cm,height=7cm,clip=}
}}
\caption{PSNR versus the number of iterations.}
\label{fig_iter_cvg}
\end{figure}

Figure~\ref{fig_iter_cvg} displays the PSNR of the solution versus the
number of iterations. In this example, the algorithm is shown to
converge rapidly. From a practical viewpoint only four or five
iterations are truly needed.  Note that the result obtained after a
single iteration is already superior to those available using
methods based on the thresholding of wavelet or curvelet coefficients
alone.

\begin{table*}[htb]
\begin{center}
\begin{tabular}{lcc} \hline \hline
Method                         & PSNR   & PSNR  \\
                               & (coeff. $l_1$ norm minim.) & (TV minim.) \\ \hline \hline
 Undecimated wavelet only      & 32.00   &  32.43 \\
 Curvelet only                 & 32.03   &  32.40 \\
 Wavelet + curvelet            & 32.72   &  32.77 \\
 (Combined filtering)          &         &   \\ \hline \hline
\end{tabular}
\caption{PSNR after filtering the simulated image
(Lena + Gaussian noise, sigma=20).
In the combined filtering, a curvelet and an undecimated wavelet
transform have been used.}
\vspace{0.5cm}
\label{cur_comptab3}
\end{center}
\end{table*}

Several papers have been recently published, based on the concept
of minimizing the total variation under constraints in the
wavelet domain \cite{rest:froment01,rest:froment02a,rest:malgouyres02} or in
the curvelet domain \cite{rest:candes02}. Our combined approach
can be seen as a generalization of these methods.
We carried out a set of experiments in order to estimate (i) if the
total variation is better than the $l_1$ norm of the multiscale
coefficients, and (ii) if the combined approach improves the
results compared to a single transform based method.

In our example, Gaussian noise with a standard deviation equal
to 20 was added to the classical {\tt Lena} image (512 by 512).
Several methods were used to filter the noisy image:
\begin{enumerate}
\item TV + constraint in the wavelet domain.
\item TV + constraint in the curvelet domain.
\item Wavelet $l_1$ norm minimization + wavelet constraints.
\item Curvelet $l_1$ norm minimization + curvelet constraints.
\item Combined filtering method using multiscale coefficient $l_1$
norm minimization.
\item Combined filtering method using TV minimization.
\end{enumerate}
We use the PSNR as an ``objective'' measure of performance.
The noisy image PSNR is $22.13$.
PSNR results from the different tested methods
are reported in Table~\ref{cur_comptab3}.

We observe that combined filtering leads to a significant improvement
when compared to a single transform based method.
The TV penalization gives better results when a single transform is used,
while it
seems not to have too much importance for the combined filtering approach.
We will see in the following that the latter
is not true for the deconvolution problem.

% \clearpage
\section{Discussion}
We believe that the denoising experiments presented in this paper are
of very high quality:
\begin{enumerate}
\item Combined filtering leads to a real
improvement both in terms of PSNR and visual appearance.
\item The combined approach arguably challenges the eye to distinguish
  structure/features from residual images of real image data (at least
  for the range of noise levels that was considered here). Single
  transforms cannot manage such a feat.
\end{enumerate}
We also note that the combined reconstruction may tend to be free of
major artifacts which is very much unlike typical thresholding rules.
Although the ease of implementation is clear we did not address the
computational issues associated with our method. In a nutshell, the
algorithms we described require calculating each transform and its
inverse only a limited  number of times.

In our examples, we constructed a combined transform from linear
transforms (wavelets, ridgelets and curvelets) but our paradigm
extends to any kind of nonlinear transform such as the Pyramidal
Median Transform \cite{starck:book98} or morphological multiscale
transforms \cite{wave:goutsias99b}.

\clearpage
\newpage


%Suggested order of slides
% slides-regu-intro
% slides-regu-l1l2
% slides-regu-l1vsl2
% slides-regu-enetlogreg
% slides-regu-underdetermined
% slides-regu-l0
% slides-regu-nonlin-bayes
% slides-regu-geom-l2-wdecay
% slides-regu-geom-l1
% slides-regu-early-stopping

\subsection{Introduction to Regularization}
\includepdf[pages=-]{../slides-pdf/slides-regu-intro.pdf}

\subsection{Lasso and Ridge Regression}
\includepdf[pages=-]{../slides-pdf/slides-regu-l1l2.pdf}

\subsection{Lasso vs. Ridge Regression}
\includepdf[pages=-]{../slides-pdf/slides-regu-l1vsl2.pdf}

\subsection{Elastic Net and Regularization for GLMs}
\includepdf[pages=-]{../slides-pdf/slides-regu-enetlogreg.pdf}

\subsection{Regularization for Underdetermined Problem}
\includepdf[pages=-]{../slides-pdf/slides-regu-underdetermined.pdf}

\subsection{L0 Regularization}
\includepdf[pages=-]{../slides-pdf/slides-regu-l0.pdf}

\subsection{Nonlinear and Bayes}
\includepdf[pages=-]{../slides-pdf/slides-regu-nonlin-bayes.pdf}

\subsection{Geometric Analysis of L2-Regularization and Weight Decay}
\includepdf[pages=-]{../slides-pdf/slides-regu-geom-l2-wdecay.pdf}

\subsection{Geometric Analysis of L1-regularization}
\includepdf[pages=-]{../slides-pdf/slides-regu-geom-l1.pdf}

\subsection{Early Stopping}
\includepdf[pages=-]{../slides-pdf/slides-regu-early-stopping.pdf}


\subsection{Benchmarking instances}

This section explains the two sets of benchmarks used.The medium size instance set consists of 38 small or medium size instances, and the large size instance set consists of 200 problem instances. The medium size set is used to compare the ILP model versus the Metaheuristics models. The large size set is used to test parameter values in order to find the best performing setup for each BRKGA and GRASP model, and finally to compare them solving one large problem instance.

\subsubsection{The instance generation process}

The algorithm used to generate problem instances is based on generating random but valid schedules of nurses and then adding the number of nurses working at each hour to compute the demand. This approach assures that the problem instance will always have a solution. Besides that, the algorithm places the schedules close to few selected hours of the schedule (adding some variability). The input values of the algorithm are all the variables of the problem instance, the selected hours of the schedule to place nurse schedules around and some proportion of extra nurses to add to the problem without using them to create the demand.

\subsubsection{The medium and large instance sets}

To decide how the size of a problem is determined, we choose to reference the time it takes for the ILP model to solve the problem instance using the Cplex solver. We generate and solve a series of instances with the ILP until we have around 20 instances that take 60 minutes or less to solve by ILP. Then, as we cannot perform the same procedure to generate the large instance set, we execute a series of experiments to determine the influence of the problem variables to the steps the ILP model has to do to decrease the gap between the best integer and the best bound.

The results are shown in figure~\ref{fig_ilp_size}, figure~\ref{fig_ilp_size2} and figure~\ref{fig_ilp_size3} page~\pageref{fig_ilp_size}. The table~\ref{tab:ilp_size} page~\pageref{tab:ilp_size} summarizes the findings and shows how we applied them to create the large benchmark instance set.

\begin{figure}[h!]

\begin{subfigure}[b]{.49\linewidth}
\centering
\includegraphics[width=\linewidth]{./img/instances_nurses_ilp_evol.png}
\caption{ Evolution of Gap of the ILP model with different number of nurses}\label{fig1a}
\end{subfigure}%
\begin{subfigure}[b]{.49\linewidth}
\centering
\includegraphics[width=\linewidth]{./img/instances_hours_ilp_evol.png}
\caption{ Evolution of Gap of the ILP model with different number of hours }\label{fig1b}
\end{subfigure}\vfill
\caption{Evolution of Gap of the ILP model with different values of number of nurses (\subref{fig1a}) and number of hours (\subref{fig1b}). }
\label{fig_ilp_size}
\end{figure}

\begin{figure}[h!]
\begin{subfigure}[b]{.49\linewidth}
\centering
\includegraphics[width=\linewidth]{./img/instances_maxpresence_ilp_evol.png}
\caption{Evolution of Gap of the ILP model with different values of maxPresence parameter }\label{fig1c}
\end{subfigure}
\begin{subfigure}[b]{.49\linewidth}
\centering
\includegraphics[width=\linewidth]{./img/instances_maxconsec_ilp_evol.png}
\caption{Evolution of Gap of the ILP model with different values of maxConsec parameter }\label{fig1d}
\end{subfigure}
\caption{Evolution of Gap of the ILP model with different values of maxPresence (\subref{fig1c}) and maxConsec (\subref{fig1d}).  }
\label{fig_ilp_size2}
\end{figure}

\begin{figure}[h!]
\begin{subfigure}[b]{.49\linewidth}
\centering
\includegraphics[width=\linewidth]{./img/instances_maxhours_ilp_evol.png}
\caption{Evolution of Gap of the ILP model with different values of maxHours parameter }\label{fig1c}
\end{subfigure}
\begin{subfigure}[b]{.49\linewidth}
\centering
\includegraphics[width=\linewidth]{./img/instances_minhours_ilp_evol.png}
\caption{Evolution of Gap of the ILP model with different values of minHours parameter }\label{fig1d}
\end{subfigure}
\caption{Evolution of Gap of the ILP model with different values of maxHours (\subref{fig1c}) and minHours (\subref{fig1d}).  }
\label{fig_ilp_size3}
\end{figure}

\begin{table}[ht]
\centering
\begin{tabularx}{0.75\textwidth}{|l|c|c|c|}
\hline
\textbf{Problem} & \textbf{Effect on problem size}  & \textbf{Medium size set} & \textbf{Large size set} \\

\textbf{Variable}  		& \textbf{when increasing}  & 38 instances &  199 instances \\
\hline
\textbf{$nNurses$} & increases      &  64	&  64 to 4096 \\
\textbf{$hour$}   & increases		& 24	&  24, 48, 72 \\
\textbf{$maxPrensce$} & decreases	& 16	& 8 to 27 \\
\textbf{$maxConsec$} & decreases		& 5	&  4 to 13		\\
\textbf{$maxHours$} & decreases		& 4 to 10	& 2 to 12 \\
\textbf{$minHours$} & increases		& 1 to 5	& 1 to 9\\
\hline
\end{tabularx}
\caption{Benchmark datasets and variables influence on problem size}
\label{tab:ilp_size}
\end{table}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Example 2: One Dimensional Heat Diffusion in an Iron Rod}
\sslist{example02.py}
\label{Sec:1DHDv0}

Our second example is of a cold iron bar at a constant temperature of
$T_{ref}=20^{\circ} C$, see \reffig{fig:onedhdmodel}. The bar is
perfectly insulated on all sides with a heating element at one end keeping the
temperature at a constant level $T_0=100^{\circ} C$.  As heat is
applied energy will disperse along the bar via conduction. With time the bar
will reach a constant temperature equivalent to that of the heat source.

\begin{figure}[ht]
\centerline{\includegraphics[width=4.in]{figures/onedheatdiff002}}
\caption{Example 2: One dimensional model of an Iron bar}
\label{fig:onedhdmodel}
\end{figure}

This problem is very similar to the example of temperature diffusion in granite
blocks presented in the previous Section~\ref{Sec:1DHDv00}. Thus, it is possible
to modify the script we have already developed for the granite blocks to suit
the iron bar problem.
The obvious differences between the two problems are the dimensions of the
domain and different materials involved. This will change the time scale of the
model from years to hours. The new settings are
\begin{python}
#Domain related.
mx = 1*m #meters - model length
my = .1*m #meters - model width
ndx = 100 # mesh steps in x direction
ndy = 1 # mesh steps in y direction - one dimension means one element
#PDE related
rho = 7874. *kg/m**3 #kg/m^{3} density of iron
cp = 449.*J/(kg*K) # J/Kg.K thermal capacity
rhocp = rho*cp
kappa = 80.*W/m/K   # watts/m.Kthermal conductivity
qH = 0 * J/(sec*m**3) # J/(sec.m^{3}) no heat source
Tref = 20 * Celsius  # base temperature of the rod
T0 = 100 * Celsius # temperature at heating element
tend= 0.5 * day # - time to end simulation
\end{python}
We also need to alter the initial value for the temperature. Now we need to set
the temperature to $T_{0}$ at the left end of the rod where we have
$x_{0}=0$ and
$T_{ref}$ elsewhere. Instead of \verb|whereNegative| function we now
use \verb|whereZero| which returns the value one for those sample points where
the argument (almost) equals zero and the value zero elsewhere. The initial
temperature is set to
\begin{python}
# ... set initial temperature ....
T= T0*whereZero(x[0])+Tref*(1-whereZero(x[0]))
\end{python}

\subsection{Dirichlet Boundary Conditions}
In the iron rod model we want to keep the initial temperature $T_0$ on
the left side of the domain constant with time.
This implies that when we solve the PDE~\refEq{eqn:hddisc}, the solution must
have the value $T_0$ on the left hand side of the domain. As mentioned
already in Section~\ref{SEC BOUNDARY COND} where we discussed boundary
conditions, this kind of scenario can be expressed using a
\textbf{Dirichlet boundary condition}. Some people also use the term
\textbf{constraint} for the PDE.

To define a Dirichlet boundary condition we need to specify where to apply the
condition and determine what value the
solution should have at these locations. In \esc we use $q$ and $r$ to define
the Dirichlet boundary conditions for a PDE. The solution $u$ of the PDE is set
to $r$ for all sample points where $q$ has a positive value.
Mathematically this is expressed in the form;
\begin{equation}
  u(x) = r(x) \mbox{ for any } x \mbox{ with } q(x) > 0
\end{equation}
In the case of the iron rod we can set
\begin{python}
q=whereZero(x[0])
r=T0
\end{python}
to prescribe the value $T_{0}$ for the temperature at the left end of
the rod where $x_{0}=0$.
Here we use the \verb|whereZero| function again which we have already used to
set the initial value.
Notice that $r$ is set to the constant value $T_{0}$ for all sample
points. In fact, values of $r$ are used only where $q$ is positive. Where $q$
is non-positive, $r$ may have any value as these values are not used by the PDE
solver.

To set the Dirichlet boundary conditions for the PDE to be solved in each time
step we need to add some statements;
\begin{python}
mypde=LinearPDE(rod)
A=zeros((2,2)))
A[0,0]=kappa
q=whereZero(x[0])
mypde.setValue(A=A, D=rhocp/h, q=q, r=T0)
\end{python}
It is important to remark here that if a Dirichlet boundary condition is
prescribed on the same location as any Neumann boundary condition, the Neumann
boundary condition will be \textbf{overwritten}. This applies to Neumann
boundary conditions that \esc sets by default and those defined by the user.

Besides some cosmetic modification this is all we need to change. The total
energy over time is shown in \reffig{fig:onedheatout1 002}. As heat
is transferred into the rod by the heater the total energy is growing over time
but reaches a plateau when the temperature is constant in the rod, see
\reffig{fig:onedheatout 002}.
You will notice that the time scale of this model is several order of
magnitudes faster than for the granite rock problem due to the different length
scale and material parameters.
In practice it can take a few model runs before the right time scale has been
chosen\footnote{An estimate of the
time scale for a diffusion problem is given by the formula $\frac{\rho
c_{p} L_{0}^2}{4 \kappa}$, see
\url{http://en.wikipedia.org/wiki/Fick\%27s_laws_of_diffusion}}.

\begin{figure}[ht]
\begin{center}
\includegraphics[width=4in]{figures/ttrodpyplot150}
\caption{Example 2: Total Energy in the Iron Rod over Time (in seconds)}
\label{fig:onedheatout1 002}
\end{center}
\end{figure}

\begin{figure}[ht]
\begin{center}
\includegraphics[width=4in]{figures/rodpyplot001}
\includegraphics[width=4in]{figures/rodpyplot050}
\includegraphics[width=4in]{figures/rodpyplot200}
\caption{Example 2: Temperature ($T$) distribution in the iron rod at time steps
$1$, $50$ and $200$}
\label{fig:onedheatout 002}
\end{center}
\end{figure}

\section{For the Reader}
\begin{enumerate}
 \item Move the boundary line between the two granite blocks to another part of
the domain.
 \item Split the domain into multiple granite blocks with varying temperatures.
 \item Vary the mesh step size. Do you see a difference in the answers? What
does happen with the compute time?
 \item Insert an internal heat source (Hint: The internal heat source is given
by $q_{H}$.)
 \item Change the boundary condition for the iron rod example such that the
temperature
 at the right end is kept at a constant level $T_{ref}$, which
corresponds to the installation of a cooling element (Hint: Modify $q$ and
$r$).
\end{enumerate}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\subsection{Worksheet - Intro to Coupled Oscillators}
\begin{p}
Give some examples of coupled oscillators.
\end{p}
\begin{s}
\phantom{i}
\begin{itemize}
    \item Masses coupled together by springs
    \item A crystal lattice, which can be approximated as atoms being connected by springs in a lattice structure
    \item Molecules (e.g. CO2) which we can treat as a carbon atom connected linearly to two oxygen atoms.
\end{itemize}
\end{s}

\begin{center}
    \includegraphics[scale=0.5]{Lecture-10/w10-img1.png}
\end{center}
\begin{p}
Find the Lagrangian for the two coupled oscillators(two masses connected by springs). Find the equations of motion.
\end{p}
\begin{s}
The Lagrangian is given by:
\[\LL = T - U = \frac{m}{2}\left(\dot{x}^2_1 + \dot{x}^2_2\right) - \left[\frac{k}{2}x_1^2 + \frac{k_{12}}{2}(x_1 - x_2)^2 + \frac{k}{2}x_2^2\right]\]
To obtain the equations of motion, we use the EL equations:
\[\dod{}{t}\dpd{\LL}{\dot{x}_1} = m\ddot{x}_1 = \dpd{\LL}{x_1} = -kx_1 - k_{12}x_1 + k_{12}x_2\]
\[\dod{}{t}\dpd{\LL}{\dot{x}_2} = m\ddot{x}_2 = \dpd{\LL}{x_2} = -kx_2 - k_{12}x_2 + k_{12}x_1\]
Which agrees with our analysis using the Newtonian formulation. This is clearly a coupled system of ODEs. To start solving this, we start by writing this in a nicer form, defining a column vector $\v{x} = \m{x_1 \\ x_2}$. This transforms things into a matrix equation:
$$\left(\begin{array}{cc}m & 0 \\ 0 & m\end{array}\right)\left(\begin{array}{c}\ddot{x}_{1} \\ \ddot{x}_{2}\end{array}\right)=-\left(\begin{array}{cc}k+k_{12} & -k_{12} \\ -k_{12} & k+k_{12}\end{array}\right)\left(\begin{array}{c}x_{1} \\ x_{2}\end{array}\right)$$
Which we can write as:
\[\MM\ddot{\v{x}} = -\KK\v{x}\]
\end{s}

\begin{p}
Using the complex quantity $\v{z} = \v{a}\exp(i\omega t)$, show that the equation for the normal modes can be written as $(\KK - \omega\MM)\v{a} = 0$.
\end{p}
\begin{s}
We define the complex quantity $\v{z}$ as above (such that $\v{x} = \Re\v{z}$, and then we have:
\[\MM\ddot{\v{z}} = \MM\v{a}(-\omega^2)\exp(i\omega t) = -\KK\v{a}\exp(i\omega t)\]
Then cancelling out the exponentials, we have:
\[\KK\v{a} = \MM\omega^2\v{a}\]
Rearranging, we have:
\[(\KK - \omega^2\MM)\v{a} = 0\]
This is an \textbf{eigenvalue problem}, which we are familiar with from linear algebra. We note that here, $\MM$ is just an identity matrix multiplied by $m$.
\end{s}

\begin{p}
Write out the characteristic equation. What are the roots? (i.e. the eigenfrequencies).
\end{p}
\begin{s}
To solve this eigenvalue problem (i.e. for the system to have nontrivial solutions) we require that $\det(\KK - \omega^2\MM) = 0$. This is a polynomial/characteristic equation, for which the solution are the eigenvalues. For $\MM$, $\KK$ as we have defined them, this looks like:
\[\det(\KK - \omega^2\MM) = (k + k_{12}- m\omega^2)^2 - k_{12}^2 = 0\]
Where the first term is the product of the diagonals and the second term is the product of the off diagonals. Factoring, we have:
\[(k - m\omega^2)(k + 2_{k12} - m\omega^2) = 0\]
So the characterstic equation hence has the two roots of:
\[\omega_1 = \sqrt{\frac{k + 2k_{12}}{m}}, \omega_2 = \sqrt{\frac{k}{m}}\]
The types of motion described by these two eigenfrequencies are as follows. For $\omega_1$, the blocks move with the same frequency and exactly out of phase. For $\omega_2$, the blocks move with the same frequency, and exactly in phase. We will see why this is in the last two question, by solving for the amplitudes $a_1, a_2$.
\end{s}

\begin{p}
Find the normal mode corresponding to the eigenfrequency $\sqrt{k/m}$.
\end{p}
\begin{s}
To find the normal mode, we solve for the eigenvector corresponding to the above eigenfrequency/eigenvalue:
\[(\KK - \omega_2\MM)\m{a_1 \\ a_2} = \m{k_{12} & - k_{12} \\ -k_{12} & k_{12}}\m{a_1 \\ a_2} = 0\]
Or equivalently:
\[k_{12}\m{1 & -1 \\ -1 & 1}\m{a_1 \\ a_2} = 0\]
From which we can see that the restriction on $a_1, a_2$ is that:
\[a_1 = a_2 = A\exp(-i\delta)\]
Hence solving for $\v{z}$:
\[\v{z} = \m{a_1 \\ a_2}\exp(i\omega_2 t) = \m{A \\ A}\exp(i(\omega_2 t - \delta))\]
Therefore, finding $\v{x}$ we have:
\[\v{x}_{II} = \Re\v{z} = \m{A \\ A}\cos(\omega_2 t - \delta)\]
This is an eigenmode, at the lower frequency. We can see from this that the two masses oscillate in phase with each other.
\end{s}

\begin{p}
Find the normal mode corresponding to the eigenfrequency $\sqrt{(k+2k_{12})/m}$.
\end{p}
\begin{s}
Similarly solving for the eigenvector, we have:
\[(\KK - \omega^2\MM)\v{a} = \m{-k_{12} & -k_{12} \\ -k_{12} & -k_{12}}\v{a} = -k_{12}\m{1 & 1 \\ 1 & 1}\m{a_1 \\ a_2} = 0\]
Therefore we obtain the requirement that $a_1 = -a_2$, and hence:
\[\v{x}_I = \m{A \\ -A}\cos(\omega_1 t - \delta)\]
\end{s}

\begin{p}
What is the general solution?
\end{p}
\begin{s}
The general solution is the sum of the eigenmodes:
\[\v{x}(t) = \v{x}_I(t) + \v{x}_{II}(t)\]
\end{s}

\begin{p}
If block 1 oscillates while block 2 is held fixed, what is the frequency of oscillations?
\end{p}
\begin{s}
The frequency of oscillations of the first block would be given by $\omega_0 = \sqrt{\frac{k + k_{12}}{m}}$; the reasoning for this is the block feels a restoring force $-kx$ from the left, restoring force $k_{12}x$ from the right, which leads to an effective spring constant $k + k_{12}$ and hence leads to the solution as stated.
\end{s}

\begin{p}
How does the uncoupled frequency above compare to the two eigenfrequencies?
\end{p}
\begin{s}
\[\omega_2 < \omega_0 < \omega_1\]
\end{s}
\noindent$\omega_2$ is the frequency of the lowest mode, the blocks are in phase. The middle frequency is where we fix one block and just let the other oscillate. $\omega_1$ corresponds to the higher eigenmode, with the blocks oscillating out of phase, at the highest frequency. Next day, we will look at coupled pendulums and normal coordinates (which is changing bases to diagonalize our matrices), where we obtain the useful result that the normal coordinates are independent of one another.
\end{document}


\documentclass{article}
\usepackage[margin=1in]{geometry}
\usepackage{blindtext}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{minted}
\begin{document}
\tableofcontents
\section{Priority Queue}
\begin{minted}{c++}
    priority_queue<int, vector<int>, greater<int>> p; // min heap
    priority_queue<int, vector<int>, less<int>> p; // max heap
\end{minted}
\section{Connected Components}
\inputminted[linenos,breaklines]{c++}{connected_component.cpp}

\section{BFS}
\inputminted[linenos,breaklines]{c++}{bfs.cpp}

\section{Dijkstra}
\inputminted[linenos,breaklines]{c++}{dijkstra.cpp}

\section{Line Equations}
\inputminted[linenos,breaklines]{c++}{lines_equation.cpp}

\section{Non Prime}
\inputminted[linenos,breaklines]{c++}{Nonprime.cpp}

\section{Prim}
\inputminted[linenos,breaklines]{c++}{prim.cpp}

\section{Prime Sieve}
\inputminted[linenos,breaklines]{c++}{prime_sieve.cpp}

\section{TopSort}
\inputminted[linenos,breaklines]{c++}{topsort.cpp}

\end{document}


\subsection{Outer Valence Green's Function}\label{gf}
This section is based on materials supplied by
\begin{center}Dr David Danovich\\The Fritz Haber Research Center for
Molecular Dynamics\\ The Hebrew University of Jerusalem\\ 91904 Jerusalem\\
Israel \end{center}

The OVGF technique was used with the self-energy part extended to include
third order perturbation corrections,~\cite{gf1}.  The higher order contributions
were estimated by the renormalization procedure.  The actual expression used to
calculate the self-energy part, $\sum_{pp}(w)$, chosen in the diagonal form,
is given in equation~(\ref{gfeq1}), where $\sum_{pp}^{(2)}(w)$ and $\sum_{pp}^{(3)}(w)$ are
the second- and third-order corrections, and $A$ is the screening factor accounting
for all the contributions of higher orders.
\begin{equation}\label{gfeq1}
\sum_{pp}(w) = \sum_{pp}^{(2)}(w)+(1-A)^{-1}\sum_{pp}^{(3)}(w).
\end{equation}
The particular expression which was used for the second-order corrections is given in
equation~(\ref{gfeq2}).
\begin{equation}\label{gfeq2}
\sum_{pp}^{(2)}(w) = \sum_a\sum_{i,j}\frac{(2V_{paij}-V_{paji})V_{paij}}{w+e_a-e_i-e_j}
+\sum_{a,b}\sum_i\frac{(2V_{piab}-V_{piba})V_{piab}}{w+e_i-e_a-e_b},
\end{equation}
where
$$
V_{pqrs} = \int\int\psi_p^*(1)\psi_q^*(2)(1/r_{12})\psi_r^*(1)\psi_s^*(2){\rm d}\tau_1{\rm d}\tau_2.
$$

In equation~(\ref{gfeq2}), $i$ and $j$ denote occupied orbitals, $a$ and $b$
denote virtual orbitals, $p$ denotes orbitals of unspecified occupancy, and
$e$  denotes an orbital energy. The equations are solved by an iterative
procedure, represented in  equation~(\ref{gfeq3}).
\begin{equation}\label{gfeq3}
w_p^{i+1}=e_p+\sum_{pp}(w^i).
\end{equation}

The SCF energies and the corresponding integrals, which were calculated by one
of the semiempirical methods (MNDO, AM1, or PM3), were taken as the zero'th
approximation and all M.O.s may be included in the active space for the OVGF
calculations.

The expressions used for $\sum_{pp}^{(3)}$ and $A$ are given in \cite{gf2}.

The OVGF method itself, is described in detail in \cite{gf1}.

%\subsubsection{Example of OVGF calculation}
%  The data-set {\bf test\_greenf.dat} will calculate the first 8 I.P.s
%for dimethoxy-$s$-tetrazine.  This calculation is discussed in detail in \cite{gf6}.
%The experimental and calculated I.P.s are shown in Table~\ref{gftab}.
%\begin{table}
%\caption{\label{gftab}OVGF Calculation, Comparison with Experiment}
%\begin{center}
%\begin{tabular}{lccccc}\\
%M.O.    &  Expt*  &   PM3   & Error   &  OVGF(PM3)  &  Error \\
%$n_1$    &  9.05   &   10.15 &  1.10   &   9.46      &   0.41 \\
%$\pi_1$  &  9.6    &   10.01 &  0.41   &   9.65      &   0.05 \\
%$n_2$    &  11.2   &   11.96 &  0.76   &  11.13      &  -0.07 \\
%$\pi_2$  &  11.8   &   12.27 &  0.47   &  11.43      &  -0.37 \\
%\end{tabular}
%
%*: R. Gleiter,  V. Schehlmann, J. Spanget-Larsen, H. Fischer and F. A. Neugebauer,
%{\em J. Org. Chem.}, {\bf 53}, 5756 (1988).
%\end{center}
%\end{table}
%
%
%From this, we see that for PM3 the average error is 0.69eV, but after OVGF
%correction, the error drops to 0.22eV.  This is typical of nitrogen heterocycle
%%calculations.


\documentclass{article}

\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{siunitx}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage[citestyle=ieee,sorting=none,bibencoding=utf8,backend=biber]{biblatex}
\usepackage{caption}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}

\graphicspath{{figures/}}
\bibliography{bibliography}
\makeatletter
\def\BState{\State\hskip-\ALG@thistlm}
\makeatother

\author{J.R. Powers-Luhn}
\title{CS528 Project 2: PCA and K Means}
\date{October 22nd, 2018}

\begin{document}
	\maketitle

	\section{Objective}
	The goal of this project was to explore unsupervised clustering and means of evaluating the
	clusters generated by the K-Means algorithm. The ability of Principal Component Analysis to
	improve the performance of this algorithm was also explored.

	\section{Preprocessing the data}
	A dataset was obtained with several summary statistics for fifty seven U.S. universities,
	mostly obtained from the Department of Education's Integrated Postsecondary Education Data System (IPEDS).

	The dataset was processed in several ways before numerical analysis. Two missing values for
	Clemson University were obtained: the IPEDS ID number (from the IPEDS website) and the 2017
	endowment balance\footnote{Source: \url{https://www.clemson.edu/giving/cufoundations/documents/allocations.pdf}}.
	Some information (medical and agricultural school research funding) was absent for many schools
	and was therefore removed from the data. None of the schools included in the dataset were
	historically black colleges, so the ``HBC'' column had zero variance and was therefore removed.
	Only some of the schools had Wall Street Journal rankings; this column was removed. Some columns
	were determined to be categorical--these were ``one-hot'' encoded.

	For all processing in this project, the numerical attributes were mean-centered and scaled to
	unit variance. This was accomplished using the scikit-learn \texttt{StandardScaler} class
	\cite{scikit-learn}. Scaling is necessary due to the different units of the various measurements.
	Without this correction the variance would be skewed by the units and mean value of the individual
	measurements.

	\section{PCA}
	Principal component analysis (PCA) was applied to the data. PCA is a linear transformation of a
	dataset into a new basis set. The principal components have the following properties:

	\begin{itemize}
		\item the components are orthogonal to each other, and
		\item the components are ordered by the amount of variance they exhibit.
	\end{itemize}

	This means that the dimensionality of the data can be reduced by transforming the data into the
	new space (where the transformed values are referred to as ``scores'') and throwing away all but
	\texttt{k} columns of the transformed data.

	The transformation into the new space is performed by taking the singular value decomposition of
	the original data as indicated in equation \ref{eq:svd}.

	\begin{equation}
		\mathbf{U}, \mathbf{S}, \mathbf{V}^T = svd(\mathbf{X})
		\label{eq:svd}
	\end{equation}

	In equation \ref{eq:svd}, $\mathbf{US}$ is the transformed data, $\mathbf{S}$ are the singular
	values, and $\mathbf{V}$ are the principal component loadings used to transform $\mathbf{X}$ into
	the PC space. $\mathbf{S}$ is a diagonal matrix with the property that the diagonal values are
	proportional to the amount of variance captured by the associated principal component. The number
	of principal components to capture some fraction $f$ of the variance is shown in equation
	\ref{eq:var_frac}.

	\begin{equation}
		\sum_{i=0}^k \vec{s^2} = f
		\label{eq:var_frac}
	\end{equation}

	For the university dataset, the data in its pre-processed form had sixty two numeric dimensions.
	However, the vast majority of the variance was contained in the first few principal components, as
	shown in figure \ref{fig:explained_variance}.

	\begin{figure}[h]
		\centering
		\includegraphics[width=0.8\textwidth]{explained_variance}
		\caption{Variance explained by each principal component (unscaled)}
		\label{fig:explained_variance}
	\end{figure}

	In order to simplify the dataset while retaining 95\% of the variance (and, therefore, the original
	information), the cumulative sum of the singular values squared (normalized to the sum of the square
	of all singular values) was calculated, and a threshold set at the first value that exceeded \num{0.95}.

	\begin{figure}[h]
		\centering
		\includegraphics[width=0.8\textwidth]{cumulative_explained_variance}
		\caption{Cumulative variance explained by the first $k$ principal components}
		\label{fig:cumulative_explained_variance}
	\end{figure}

	As shown in figure \ref{fig:cumulative_explained_variance}, this corresponded to the first \num{17}
	components. A scatter graph of the first two components plotted against each other is shown in
	figure \ref{fig:pc1_vs_pc2}

	\begin{figure}[h]
		\centering
		\includegraphics[width=0.8\textwidth]{pc1_v_pc2}
		\caption{The first two principal components plotted against each other. Since the PC's are
				 orthogonal to each other, no linear relationship is apparent. Note the scales of the
				 two different axes. Since PC1 captures more variance than PC2, this axis convers a
				 larger range.}
		\label{fig:pc1_vs_pc2}
	\end{figure}

	\section{K-means}
	In order to determine whether the schools could be grouped into categories, the \texttt{k-means}
	algorithm was employed.

	\begin{enumerate}
		\item Select $k$ initial vectors from the data as seeds, $s_i \in \{s_0, s_1, \mathellipsis, s_k\}$
		\item Calculate the distance from each $s_i$ to each vector
		\item Assign to each vector the label associated with the seed with the minimum distance to that vector
		\item Recalculate each $s_i = \frac{\sum_j x_j}{j}$ for all $x_j$ with label $i$
		\item Repeat 2-4 until the values of $s$ converge
	\end{enumerate}

	This algorithm was found to converge in less than ten iterations.

	This algorithm is not guaranteed to converge to the optimal solution since it is sensitive to the
	selection of the initial seeds. Because of this, the algorithm was repeated several times in order to
	improve the likelihood of optimal selection.

	\subsection{K-means++}
	While the \texttt{k-means} algorithm is guaranteed to converge, it is not guaranteed to converge to an
	optimum value. Specifically, it is subject to the selection of the initial seed vectors. In order to
	improve upon this, the \texttt{k-means++} algorithm was employed\cite{Arthur2007}. In this the first
	seed $s_1$ is selected at random from the data. Subsequent seeds $s_2, \mathellipsis, s_k$ are selected
	using a weighted probability proportional to the distance of each vector from the closest seed. In this
	way the initial selection of seeds avoids selecting vectors that are too close to each other. The
	\texttt{k-means++} algorithm therefore converges much more quickly than \texttt{k-means}, usually in
	a single iteration.

	\subsection{Cluster selection}
	Ideally, data clusters should exhibit both tight grouping (minimal intra-cluster distance) and wide
	separation (maximal inter-cluster distance). The ratio of these two values (the minimum inter-cluster
	distance to the maximum intra-cluster distance) is referred to as the ``Dunn Index'' \cite{dunn1974}.
	Because the \texttt{k-means} algorithm is not guaranteed to converge to the optimal solution, the
	Dunn Index is subject to the random selection of the initial vector. Still, it provides a tool for
	evaluating the ``true'' number of clusters in the data.

	The \texttt{k-means} algorithm was applied to the data with $k$ ranging from \numrange{2}{56}. In each
	case the clusters were allowed to converge and the Dunn index was calculated. The results are shown
	in figure \ref{fig:dunn_index_raw_data}.

	\begin{figure}[h]
		\centering
		\includegraphics[width=0.8\textwidth]{dunn_index_raw_data}
		\caption{Dunn index for $k$ clusters. The value oscillates but has a clear maximum.}
		\label{fig:dunn_index_raw_data}
	\end{figure}

	Unfortunately, the Dunn index did not converge to a consistent maximum. Therefore an appropriate value
	was chosen for the number of clusters that ``looked right''.

	\section{PCA plus K-Means}
	Lorem ipsum dolor simet \cite{Yeung2000}

	\section{Conclusions}
	Lorem ipsum dolor simet

	\printbibliography
\end{document}


\documentclass[10pt,a4paper]{article}
\usepackage[latin1]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{float}
\usepackage{bm}
\title{Week 6 exercises}
\begin{document}
	\maketitle
	\begin{enumerate}
		\item The linear probability model for binary response is simply $ P(y = 1) =   \pi_i = \bm{x_i'\beta} = \beta_{0i} +
		\beta_{1i} x_{1i}  + \dots + \beta_{ni} x_{ni}   $ and require the restriction that $ 0  \leq  \bm{X'\beta}  \leq 1 $
		\item In the GLM we combine the probability output to the linear prediction through a function \textit{h} called \textit{response function} that it is a cumulative distribution function with co domain in [0,1]. In formula we can express the GLM as $$ P(y = 1) =   \pi_i = h(\eta_i) = h(\mathbf{x_i'\beta}) = h(\beta_{0i} +
		\beta_{1i} x_{1i}  + \dots + \beta_{ni} x_{ni})  $$\\$ g = h^{-1} $ is the \textit{link function} and it is used to calculate the linear predictor in function of probability: $ \eta_i = g(\pi_i) $
		The logit model use as response function the logistic function: $$ \pi = h(\eta) = \dfrac{e^\eta}{1 + e^\eta} $$. The linear predictor returns the log odds $$ \mathbf{x_i'\beta} = \beta_{0i} +
		\beta_{1i} x_{1i}  + \dots + \beta_{ni} x_{ni} =  \pi_i = \log\left(\dfrac{\pi}{1 - \pi}\right)   $$.
		The probit model use instead a normal distribution cumulative function.
		The c-log-log model use as response function the extreme minimum-value cumulative
		distribution function
		$$ h(\eta) = 1 - e^{-e^\eta}$$ with the following link function $$ g(\pi) = \log(- \log( 1 - \pi )) $$
		\item
		\begin{figure}[H]
			\centering
			\includegraphics[width=0.7\linewidth]{plot_comparison_response_functions}
			\label{fig:plotcomparisonresponsefunctions}
		\end{figure}
		From the latter plot we can note that the \textit{probit} and \textit{logit} are symmetric around the 0 while the c-log-log is not symmetric. The c-log-log is similar to the logit but tend more speedly to one. If we do a comparison with the same variance between the logit and probit we have that the coefficients differ for a values of $ \frac{\pi}{3} = 1.814 $
		\item With a latent continuous variable we use the standard normal distribution for the errors because we don't know the variance of the latent variable so the new coefficients are $ \tilde{\beta} = \frac{\beta}{\sigma} $. We can't calculate the original $ \beta $ since doesn't know $ \sigma $ but the ratio between coefficients are constants $ \dfrac{\tilde{\beta}_i}{\tilde{\beta}_j} = \dfrac{\beta_i}{\beta_j}$
		\item In the logit model if increase $ x_k $ by 1 we have an increments on the odds by $ e^\beta $ while the increment of probability is not the same in every point. \\ In the probit model we have an increment equal to $ \phi^{-1}(\beta) $
		\end{enumerate}
		\subsection*{Solution to applied exercise}
		\begin{enumerate}
			\item 		$$ \eta = 0.42 + 0.06 \cdot kidsge6 - 1.44 \cdot kidslt6 - 0.09 \cdot age + 0.21 \cdot exper - 0.0031 \cdot exper^2 + 0.22 \cdot educ - 0.021 \cdot nwifeinc$$
			$$
			P(y = 1) =
			\hat{\pi} = h(\hat{\eta}) =   \dfrac{e^{\hat{\eta}}}{1 + e^{\hat{\eta}}} $$
			\item $$ 0.42 + 0.06 \cdot 0 - 1.44 \cdot 0 - 0.09 \cdot 40 + 0.21 \cdot 0 - 0.0031 \cdot 0 + 0.22 \cdot 10 - 0.021 \cdot 0 = -0.98$$
			\item one year of more eduction when all the other variables are the same is a multiplicative effect of $ e^{0.22} $
			\item The probability is 0.27
			\item -0.0174
			\item 0.046
			\item $ 0.22 * 0.5^2 =  0.055 $
			\item The probit coefficients are obtained by dividing by 1.84
		\end{enumerate}

\end{document}


% !TeX spellcheck = en_GB
\documentclass[12pt]{article}

\usepackage[a4paper, margin=0.7in]{geometry}
\usepackage{algorithm,algpseudocode}
\usepackage{caption}
\usepackage{algpseudocode}
\usepackage{graphicx}
\usepackage{verbatim}
\usepackage{comment}
\usepackage{amsmath}
\usepackage{amssymb}

\newtheorem{theorem}{Theorem}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proof}[theorem]{Proof}

\begin{document}
\title{An Efficient Recursive Approach for Workload Distribution on Heterogeneous Systems}
\author{Hamidreza Khaleghzadeh}
\date{26 Jan 2017}

\maketitle

\section{Introduction}
In this report, we propose a new method called \textit{WorkPart} which benefits from Branch-and-Bound programming technique to find the optimal workload distribution on $p$ heterogeneous processors so that the parallel execution time is minimized. We will explain the proposed approach in this report.

\section{Formulation of Performance Optimization Problem}
Consider a workload of size $n$ executed using $p$ heterogeneous processors. Let time functions of processors be represented by set $T$ where $T_i=\{(x_i^0,t_i(x_i^0)),(x_i^1,t_i(x_i^1)),\cdots,\allowbreak (x_i^{m-1},t_i(x_i^{m-1}))\}$, $0 \le i \le p-1$, $m \in \mathbb{Z}_{>0}$. There is one time function per each processor where $T_i$ represents the time function of the processor $P_i$ which is represented by a discrete set of experimental data points separated by minimum granularity, $\Delta x$. A given data point $(x,t(x))$ determines a workload with size of $x$ along with its execution time ($t(x)$). It should be noted that the workload with size of $n$ should be a multiple of $\Delta x$. The problem, $WorkPart(n,p,T,t_{opt},D_{opt})$, is to find a partitioning, $D_{opt} = <x_{opt}^0,\cdots,x_{opt}^{p-1}>$, of the workload of size $n$ among $p$ processors that minimizes the parallel computation time of the workload ($t_{opt}$). The parameters of $(n,p,T)$ and $(t_{opt},D_{opt})$ are the inputs and outputs of the problem, respectively. The problem can be formulated as following:
\begin{equation} \label{eq1}
\begin{split}
t_{opt} = minimize \quad \max_{i=0}^{p-1} \quad t_i(x^i) \\
\text{Subject to } x^0+x^1+\cdots+x^{p-1} = n \\
0 \le x^i \le n \qquad i = 0,\cdots,p-1 \\
\text{Where } p,n,x^i \in \mathbb{Z}_{>0} \text{ and } t_i(x^i) \in \mathbb{R}_{>0}.
\end{split}
\end{equation}

It is noteworthy that if a given $x^i$ is equal with 0, no problem is assigned to $P_i$. Therefore, the number of selected processors in the optimal workload distribution may be less than $p$.

\section{An Example}
In this section, using a simple example, we will explain how the problem $WorkPart$ is solved by the proposed algorithm called $HetWD$ which is responsible to find optimal distribution. Let a workload size of $n = 16$ runs on $p = 4$ processors with time functions $T=\{T_0,\cdots,T_3\}$. Assume, for the sake of simplicity, the minimum granularity is 1 ($\Delta x = 1$).

Figure \ref{ex_fig1} illustrates time functions sorted in non-decreasing order of time. Each cell in time functions shows execution time and its label is the problem size. To restrict the search area, the algorithm applies a time threshold represented by $t_{opt}$. All data points with greater or equal execution time than the time threshold are ignored. During the initialization step of the algorithm, the time threshold is set to load-equal execution time. According to the load-equal distribution, the assigned problem size to each processor should be $\frac{16}{4}=4$ where its parallel execution time is $\max_{i=0}^{3} t_i(4) = \max \{12,6,4,4\} = 12$. All data points with smaller execution time of 12 consist our search area highlighted in time functions (Figure \ref{ex_fig1}).

In the figure, there is a matrix called $Mem$ so that rows $Mem[1]$ and $Mem[2]$ store intermediate results for processors $P_1$ and $P_2$ (one row per processor). Each row in $Mem$ consists of 17 compound cells ($=16+1$) where $Mem[i][w]$, $i \in \mathbb{Z}_{[1,2]}, w \in \mathbb{Z}_{[0,n]}$ holds some information about distribution of work-size of $w$ on $P_i$ such as:

\begin{itemize}
	\item \textbf{$Mem[i][w].eTime$:} Parallel execution time of $w$ on $P_i,\cdots,P_{p-1}$,
	\item \textbf{$Mem[i][w].lastIndex$:} The index of last evaluated data point in the time function $T_i$. In case a memory cell stores the optimal result, it is labelled as $Finalized$ and $Mem[i][w].lastIndex = \_FI$.
	\item \textbf{$Mem[i][w].size$:} The problem size assigned to $P_i$ along with its execution time
	\item \textbf{$Mem[i][w].time$:} The execution time of the assigned partition to $P_i$ ($t_i(Mem[i][w].size)$).

\end{itemize}

The matrix is empty at the beginning of the algorithm.

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig1.png}}
	\caption{Data structures and applying time threshold to restrict search area}
	\label{ex_fig1}
\end{figure}

To find optimal solution, all possible distributions should be evaluated. To this end, the method uses Branch-and-Bound technique which finds the optimal solution using in-depth tree traverse. Figure \ref{ex_fig2} shows the search tree built and traversed by $HetWD$ for finding the optimal workload distribution. There are two values associated with each node. Suppose node $X/Y$ on the level $L_{i \in \mathbb{Z}_{[0,\cdots,p - 1]}}$ where $X$ is remaining workload should be distributed, and $Y$ is a size threshold which is maximum possible work-size can be distributed on processors $P_i$ to $P_{p-1}$. According to the time functions, the maximum workload size with execution times less than 12 are 16, 9, 6 and 8 for $T_0,\cdots,T_3$, respectively. As shown in the figure, size thresholds are as following: for the level $L_0$ is $16+9+6+8=40$, for $L_1$ is $9+6+8=24$, for $L_2$ is $9+6=15$ and for $L_3$ is 8. In addition, the pair assigned to the edge on level $L_{i}$ determines assigned work-size to the processor $P_i$ and its corresponding execution time, respectively. If the edge label of a given level $L_{i \in \mathbb{Z}_[0,p-1]}$ is $0,0$, it means that no workload is assigned to $P_i$.

Each level in the tree is responsible for one processor to examine all data points existing in its time function from the first to the last elements which have less execution time than time threshold. Let's take a look at figure \ref{ex_fig2}. Before examination of data points existing in time functions, zero problem sizes are assigned to processors $P_0$ and $P_1$. Thus the remaining work-size should be distributed on $P_2$ and $P_3$ is still 16. But, since the remaining size (16) is greater than the size threshold of $L_2$, the node 16/15 will not be extended more and cut.

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig2.png}}
	\caption{Using size thresholds for cutting branches}
	\label{ex_fig2}
\end{figure}

Figure \ref{ex_fig3} depicts a solution found with parallel execution time of 9. The partitions assigned to processors $P_0$ to $P_3$ are $D_{opt}=<0,8,0,8>$, respectively.

Having found a solution, $HetWD$ are required to perform some post-solution operations as follows:
\begin{enumerate}
	\item updating time threshold ($t_{opt}$)
	\item updating size thresholds
	\item memorizing the current solution into $Mem$
	\item finding the uppermost node in the tree with maximum execution time and backtracking to its ancestor.
\end{enumerate}

In this example, $t_{opt}$ is updated to 9. As shown in the figure, the search area highlighted in each time function is also reduced due to the new $t_{opt}$. Once the green search area is shrunk, then size thresholds are recalculated. New size thresholds are replaced with $\{38,23,14,7\}$ (figure \ref{ex_fig4}). In addition, we are required to store the intermediate solutions in $Mem$. To this end, the solution found for the workload sizes on $P_1$ and $P_2$ should be stored. If the solution path from the root to the leaf is traced in the figure \ref{ex_fig3}, the problem size for $P_1$ and $P_2$ are respectively 8 and 16. Therefore, $Mem[2][8]$ and $Mem[1][16]$ has been updated to store these intermediate solutions.

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig3.png}}
	\caption{Finding a solution an its post-solution operations}
	\label{ex_fig3}
\end{figure}

The algorithm keeps on examination of more data points. Figure \ref{ex_fig4} illustrates new solution with better computation time of 7. All post-solution operations should again be performed. As time threshold decreases to 7, new size thresholds will be $\{37,22,13,6\}$ (figure \ref{ex_fig5}). In addition, $Mem[2][8]$ and $Mem[1][16]$ are updated (figure \ref{ex_fig4}).

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig4.png}}
	\caption{Converging to the optimal solution}
	\label{ex_fig4}
\end{figure}

In figure \ref{ex_fig5}, the parallel execution time is 4, and the maximum time occurs at level $L_2$. Since we are looking for a new solution with the execution time less than 4, more expansion of node 8/13 makes no sense. Because of non-decreasing order of execution time in time functions, following data points in the time function $T_2$ do not have less execution time than $t_{opt}$. It means that more expanding of this node would not bring a better distribution with an execution time less than 4. So, we backtrack to its ancestor in level $L_1$. Since there is no better solution than $t_{opt}=4$ for workload size of 8 on $P_2$ and $P_3$, $Mem[2][8]$ is finalized in (Labelled as $\_FI$).

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig5.png}}
	\caption{Backtracking}
	\label{ex_fig5}
\end{figure}

Having backtracked, the process continues from level $L_1$. Figure \ref{ex_fig6} shows this step. The algorithm selects (9,3) from $T_1$. Its expansion results in a solution with better execution time. Thus, all post-solution operations are done after finding a solution are applied where $t_{opt}$ decreases to 3, and $Mem[2][7]$ and $Mem[1][16]$ are updated. It is noteworthy that the last index examined in $T_2$ is stored in $Mem[2][7].lastIndex$. For instance, suppose that we again want to find a distribution for workload 7 on $P_2$ and $P_3$. In this case, we will not be required to start from the beginning of the time function $T_2$. Having restored the last stored solution for work-size 7, the process is resumed from the node with index 1.

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig6.png}}
	\caption{Storing last evaluated index in Mem for resuming the process from where was interrupted}
	\label{ex_fig6}
\end{figure}

After backtracking to the root, the algorithm again needs to find a solution for workload with size of 8 on $P_2$ (Node 8/13 in figure \ref{ex_fig7}). Since the problem for this workload on $P_2$ has already been solved, we are not required to resolve it from scratch. Instead, the solution can be retrieved from $Mem$. The stored solution is finalized and it can be used to make final decision. It should be mentioned that if it is not finalized, we are required to extract the previous stored solution and then resume the process from where it was interrupted using the stored index in the memory.

In this example, using the memory, we extract the distribution of work-size 8 on $P_2$ and $P_3$. Since its parallel execution time is greater than current one, this solution is ignored.

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig7.png}}
	\caption{Extracting a sub-solution from Mem}
	\label{ex_fig7}
\end{figure}

Finally, $HetWD$ examines the data point $(8,1)$ from $T_1$ (Figure \ref{ex_fig8}). It finds the solution $D_{opt}=<8,8,0,0>$ with the parallel execution time of 1 ($t_{opt}=1$). Since time threshold is updated with 1, there is no data points remaining to examine and the last distribution in $D_{opt}$ with the parallel execution time $t_{opt}$ is returned as the optimal distribution.

It should be noted that the algorithm returns load-equal workload distribution if it cannot find any solution with less parallel execution time. It happens when there is no distribution with sum to $n$ and less parallel execution time than that of load-equal workload distribution.

\begin{figure}[!t]
	\centering
	\fbox{\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{Images/example/fig8.png}}
	\caption{Finding the optimal solution}
	\label{ex_fig8}
\end{figure}

\section{The Proposed Efficient Algorithm}
In this section, we present an efficient algorithm to solve $WorkPart$ problem (Algorithm \ref{alg1_code}). The inputs to the algorithm are: the workload size, $n$ ($n$ should be a multiple of $\Delta x$), the number of heterogeneous processors, $p$ and a list of $p$ time functions, $T=\{T_0,T_1,\cdots,T_{p-1}\}$. The outputs of the algorithm are the optimal execution time, $t_{opt}$ and the optimal workload distribution, $D_{opt}$. It is noteworthy that the number of processors selected by WorkPart in the optimal workload distribution may be less than $p$.

The function obtains load-equal distribution an initializes $t_{opt}$ and $D_{opt}$. It returns the workload distribution so that $x_{opt}^i=\frac{n}{p}$, $t_{opt}=\max_{j=0}^{p-1} t_i(\frac{n}{p})$, $\forall i \in [0,p-1]$ (Algorithm \ref{alg1_code}, line \ref{alg1_leq}). It is important to note that $\sum_{i=0}^{p-1} x_{opt}^i = n$.

Array S contains the size thresholds, which is illustrated in the example. IT is a list of sizes where $S_{i \in \mathbb{Z}_{[0..p - 1]}}$ represents the maximum workload size can be distributed on processors $P_i,...,P_{p - 1}$ with maximum parallel execution time of $t_{opt}$. It is the function size\_thresholds responsible for determining these values (Algorithm \ref{alg1_code}, line \ref{alg1_s_th}). There is a two-dimensional matrix with size of $(p-2)*(n+1)$ called $Mem$ to memorize the points that have already been visited during the recursive invocations. The array value $Mem[i][n]$, $\forall i \in [1,p-2]$ contains some information for workload with size of $n$ distributed on processors $P_i,\cdots,p-1$. the information includes: the parallel execution time, the partition assigned to $P_i$ along with its execution time,the index of last examined data point in $T_i$ and the status of memory (Finalized or Not\_Finalized). This memorization ensures that there are only $O(n*m*p)$ recursive invocations of the core function (Algorithm \ref{alg2_code}). Having initialized $Mem$, $WorkPart$ invokes the $HetWD$ to find the optimal workload distribution.

\begin{algorithm}
	\scriptsize
	\caption{Algorithm Finding Optimal workload Distribution of Size $n$ for Maximizing Performance} \label{alg1_code}
	\begin{algorithmic}[1]
		\Function{WorkPart}{$n, p, T, t_{opt}, D_{opt}$}
		\Statex \textbf{INPUT:}
		\Statex Workload size, $n \in \mathbb Z_{> 0}$
		\Statex Number of processors, $p \in \mathbb Z_{> 0}$
		\Statex Time functions, $T = \{T_0,...,T_{p - 1}\}$. $T_i={(x_i^0,t_i(x_i^0)),\cdots,(x_i^{m-1},t_i(x_i^{m-1}))}, $ $t_i(x_i^0) \le t_i(x_i^1) \le \cdots \le t_i(x_i^{m-1}), $ $x^j \in \mathbb Z_{> 0},$ $\forall j \in [0,m-1],$ $\forall i \in [0,\cdots,p-1]$
		\Statex \textbf{OUTPUT:}
		\Statex Optimal execution time, $t_{opt} \in \mathbb R_{> 0}$
		\Statex Optimal workload distribution, $D_{opt} = {x_{opt}^0,...,x_{opt}^{p-1}}, x_{opt}^i \in T_i $ $or 0,$ $\forall i \in [0,\cdots,p-1]$.
		\Statex
		\State $(t_{opt}, D_{opt})$ $\gets$ \Call{load\_equal\_dist}{$n,p$}	\label{alg1_leq}
		\State $S$ $\gets$ \Call {size\_thresholds}{$T,t_{opt}$}	\label{alg1_s_th}
		\State $\forall i \in [1,\cdots,p-2],$ $j \in [0,\cdots,n]$
		\State \hskip\algorithmicindent $Mem[i][j]$ $\gets$ $(0,0)$
		\State \Call {HetWD}{$n,p,0,T,S,NULL,D_{cur},Mem,t_{opt},D_{opt}$} \label{alg1_hetwdCall}
		\State \Return $(t_{opt},D_{opt})$
		\EndFunction
	\end{algorithmic}
\end{algorithm}

\section{Recursive Algorithm $HetWD$} \label{sec_code}
In order workload with size of $n$ to be optimally distributed on $p$ processors, WorkPart invokes a core routine called $HetWD$ (Algorithm \ref{alg1_code} line \ref{alg1_hetwdCall}). This proposed function is responsible to examine possible workload distributions using the Branch-and-Bound technique and find optimal solution. The search area is a tree data structure which is constructed implicitly using Depth First Search (DFS) technique and consists of maximum $p$ levels. Each level belongs to one processor where level $L_0$ to $P_0$ and so on, and all possible partitions can be assigned to a given processor $P_i$ are examined in $L_i$.

To cut sub-optimal branches, we benefits from some optimizations that will be explained in section \ref{sec_opt}. In addition, we will explain how memorization of intermediate solutions prevent duplicate recalculations (Section \ref{sec_mem}).

$HetWD$ is illustrated in Algorithm \ref{alg2_code}. The while loop (Algorithm\ref{alg2_code}, lines \ref{Alg1_mainLoop1}-\ref{Alg1_mainLoop2}) builds the main section of the algorithm which is responsible for scanning time functions from left to right to examine all data points with execution time less than $t_{opt}$. The current processor which is under processing is represented by $c \in \{P_0,\cdots,P_{p-1}\}$. Selected work-sizes for each processor is stored in $D_{cur}={x^0,\cdots,x^{p-1}}$ where $d^i \in T_i$ determines current selected partition for $P_i$. It is important to note that $D_{opt}={x_{opt}^0,\cdots,x_{opt}^{p-1}}$ holds the best distribution found so far.

The algorithm goes ahead in depth from $P_0$ (the root of the implicit tree) up to $P_{p-1}$ (a leaf) while $n$ is larger than zero. In each level, it stores current partition in $D_{cur}$. Once a solution is found, Post\_Solution\_Operations is called to perform post-solution activities such as: updating $t_{opt}$, $D_{opt}$, array $S$ and determining to which level the process should backtrack (determined with $bk$).

\begin{algorithm}
\scriptsize
\caption{Pseudocode of recursive workload partitioning on heterogeneous platforms} \label{alg2_code}
\begin{algorithmic}[1]
\Function{HetWD}{$n, p, c, T, S, bk, D_{cur}, Mem,t_{opt}, D_{opt}$}
\Statex
	\If{$c = p - 1$}	\label{Alg1_leaf1}
		\If{$t_c(n) < t_{opt}$}
			\State $D_{cur}[c]$ $\gets$ $(n,t_c(n))$
			\State \Call{Post\_Solution\_Operations}{$D_{cur},t_{opt},D_{opt},S,bk,Mem$}
		\EndIf
		\State \textbf{return}
	\EndIf				\label{Alg1_leaf2}
	\State $<size_c,time_c>$ $\gets$ $(0,0)$
	\State $curIndex=-1$
	\If{$c > 0 \wedge c \leq p - 2$}	\label{retrieveMem1}
		\State $<isReturn,curIndex>$ $\gets$ \Call{ReadFromMemory}{$n,p,c,t_{opt},T,bk, D_{cur}$}
		\If{$isReturn=true$}
			\State \textbf{return}
		\Else
			\State $<size_c,time_c>$ $\gets$ $(x_c^{curIndex},t_c(x_c^{curIndex}))$
		\EndIf
	\EndIf							\label{retrieveMem2}
	\While{$time_c < t_{opt}$} \label{Alg1_mainLoop1}
		\If{$n = size_c \vee n > size_c \wedge S[c + 1] \geq (n - size_c)$}	\label{Alg1_non_leaf1}
			\State $D_{cur}[c]$ $\gets$ $(size_c, time_c)$
			\If{$n = size_c$}
				\State \Call{Post\_Solution\_Operations}{$D_{cur},t_{opt},D_{opt},S,bk,Mem$}
			\Else
				\State \Call{HetWD}{$n-size_c,p,c+1,t_{opt},T,S,bk,D_{cur}$}	\label{Alg1_recall}
			\EndIf
			\If{$bk < c$}
				\If{$t_c(x_c^{curIndex})=t_{opt}$}
					\State $Mem[c][n].$\Call{makeFinal}{ }
				\Else
					\State $Mem[c][n].last_{index} = curIndex$ \label{alg1_lastIndex}
				\EndIf
				\State \textbf{return}
			\ElsIf{$bk = c$}
				\State $bk = NULL$
				\State $Mem[c][n].$\Call{makeFinal}{ }
				\State \textbf{return}
			\Else
				\State $bk = NULL$
			\EndIf
		\EndIf	\label{Alg1_non_leaf2}
		\If{$T_c.$\Call{isEnd}{ }}
			\State \textbf{break}
		\EndIf
		\State $curIndex++$
		\State $<size_c,time_c>$ $\gets$ $(x_c^{curIndex},t_c(x_c^{curIndex}))$
	\EndWhile	\label{Alg1_mainLoop2}
	\State $Mem[c][n].$\Call{makeFinal}{ }
\EndFunction
\end{algorithmic}
\end{algorithm}

\subsection{Optimizations} \label{sec_opt}
$HetWD$ uses a set of optimizations to avoid branches that lead to sub-optimal solutions and therefore reduce the number of recursive calls.

\subsubsection{Time threshold} \label{sec_time_threshold}
To find the optimal solution, all possible work-size combinations should be examined. It means that $n$ data points from each time function are required to be examined. Using a time threshold can eliminate some sub-optimal points from our search area (Algorithm\ref{alg2_code}, Line \ref{Alg1_mainLoop1}). In $HetWD$, it is $t_{opt}$ which applies a time threshold initialized to load-equal execution time. At runtime, it is updated by post-solution operations.
For instance, figure \ref{ex_fig1} shows how the time threshold removes some data points from the search area. The threshold is updated by the function Post\_Solution\_Operations() at run-time when a solution with less execution than $t_{opt}$ is found (Figures \ref{ex_fig4}-\ref{ex_fig7}).

\subsubsection{Size threshold} \label{sec_size_Threshold}
An acceptable distribution $<x^0,\cdots,x^{p-1}>$ for workload with size of $n$ running on $p$ processors should satisfies Eq. \ref{eq1} where $\sum_{i=0}^{p-1}x^i=n$. We introduce size thresholds $S=\{S_0,S_1,\cdots,S_{p-1}\}$ where $S_i$ determines the maximum workload size can be distributed on processors $P_i,\cdots,P_{p-1}$ with parallel execution time less than $t_{opt}$. In other words, if $s_{p-1}=x_g^{p-1}$, $x_g^{p-1}$ represents the greatest work-size in $T_{p-1}$ where $t_{p-1}(x_g^{p-1})<t_{opt}$, then $S_i=x_g^i+s_{i+1}$, $i \in [0,p-2]$. Size threshold array $S$ is updated when $t_{opt}$ is replaced with new execution time. Figure \ref{ex_fig2} shows how size thresholds is able to cut branches will not result any solution.

\subsubsection{Backtracking}	\label{sec_backtracking}
According to the Eq. \ref{eq1}, the parallel execution time for a given solution $<x^0,\cdots,x^{p-1}>$ will be $t_{opt}=\max_{i=0}^{p-1}t_i(x^i)$. Suppose $x^i$ is the uppermost node in the search tree where $t_i(x^i)=t_{opt}$. It should be noted that the execution times of all $x^k$s, $\forall k \in [0,i-1]$ are less than $t_{opt}$. Since time function are sorted in increasing order of execution time, more expansion of the given node $x^i$ will not result a distribution with execution time less than current $t_{opt}$. Therefore, when a distribution is found, the recursive process should find the uppermost node in the search tree with the execution time equals to $t_{opt}$ and backtrack to its ancestor. This operation is performed by Post\_Solution\_Operations(). Take figure \ref{ex_fig5} as an example. Having found the distribution $<0,8,3,5>$ with parallel execution time $t_{opt}=4$, the process should return to node $16/21$ at level $L_1$ which is the ancestor of node $8/13$.

\subsection{Memorization}	\label{sec_mem}
Storing intermediate solutions is an effective way to prevent resolving the nodes which have been already visited. To this end a two-dimensional array called $Mem$, is defined to store intermediate solutions for processors $P_1,\cdots,P_{p - 2}$. The matrix size depends on workload size ($n$) and the number of processors ($p$) and consists of $(p-2) * (n+1)$ elements. A solution for workload size $w$ on processor $P_i$ is stored in $Mem[i][w]$. The stored information is the work-size should be assigned to processor $P_i$, $Mem[i][w].size$, its corresponding execution time on $P_i$ extracted from $T_i$, $Mem[i][w].time$, the parallel execution time of $w$ distributed on processors $P_i,\cdots,P_{p-1}$, $Mem[i][w].eTime$, and the index of last data point examined from time function $T_i$, $Mem[i][w].lastIndex$. $lastIndex$ helps to resume the operation from where it has been interrupted When all possible data points existing in $T_i$ is examined or further expanding of $P_i$ for work-size $w$ does not improve final execution time (refer to section \ref{sec_backtracking}), it means that $Mem[i][w]$ contains the optimal solution for $w$, and the memory cell should be finalized. To this end, $Mem[i][w].lastIndex$ is equal to $\_FI$.

\subsubsection{Retrieval from Memory}
In every recursion, in case under processing processor is $c \in [1,\cdots,p - 2]$, function ReadFromMemory is invoked to read (Algorithm \ref{alg2_code} lines \ref{retrieveMem1}-\ref{retrieveMem2}). It either retrieves final solution if the accessed memory cell is $\_FI$ or determines from where the process should be resumes by reading $lastIndex$. Algorithm \ref{alg3_code} illustrates the function ReadFromMemory. Let $w$ is the size of workload. Firstly, $Mem[c][w]$ is accessed to read the last stored solution (Algorithm \ref{alg3_code}, Line \ref{alg3_memAcc}). $Mem[c][w]$ contains the size of partition assigned to $P_c$, the execution time of the stored solution on processors $P_c,\cdots,P_{p-1}$, the index of last data point of $T_c$ which has been examined and assigned workload to $P_c$ stored in $MTime$, $MTime$ and $MLast$, respectively According to the values of $MPoint$ and $MLast$, the following scenarios occurs:

\begin{itemize}
\item \textbf{No Solution}: This case occurs when there is no stored execution time in memory ($\_NE$), and the result is $\_FI$. It means that there is no solution for $n$ on processor $P_c$ (Algorithm \ref{alg3_code}, Lines \ref{alg3_noSol_1}-\ref{alg3_noSol_2}).

\item \textbf{Solution}:  This occurs when there is a finalized solution for $n$ on processor $c$. While the first part of solution has been stored in $D_{cur}[i], i \in [0 \cdots c - 1]$, the second part, $D_{cur}[i], i \in [c \cdots p - 1]$ will be read using the previously stored solution from $Mem$. the final solution is extracted from memory and processed by Post\_Solution\_Operations() (Algorithm \ref{alg3_code}, Lines \ref{alg3_fsol_1}-\ref{alg3_fsol_2}).

\item \textbf{Solution and Resume}: This is similar to the second items, but the solution is not finalized. The solution which has been already stored at $Mem$ is firstly extracted. Having examined the solution (Algorithm \ref{alg3_code}, Lines \ref{alg3_sol_1}-\ref{alg3_sol_2}), the process is again resumed from where is determined by $MLast$ (Algorithm \ref{alg3_code}, Lines \ref{alg3_resume_1}-\ref{alg3_resume_2}).

\item \textbf{Resume}: This condition happens when there has yet to be no solution for $n$ on processor $c$, but the memory cell is not $\_FI$ and $MLast$ points to the index where the processing work should be resumed (Algorithm \ref{alg3_code}, Lines \ref{alg3_resume_1}-\ref{alg3_resume_2}).
\end{itemize}

If the function returns $true$, it means that the caller of ReadFromMemory should return, too. Otherwise, the process is resumed from where $MLast$ determines.

\begin{algorithm}
\scriptsize
\caption{Pseudocode of solution retrieval from memory} \label{alg3_code}
\begin{algorithmic}[1]
\Function{<bool,MLast> ReadFromMemory}{$w, p, c, t_{opt},T,bk, D_{cur}$}
	\State $(MTime, MLast, MPoint)$ $\gets$ $Mem[c][w]$ \label{alg3_memAcc}
	\If{$MLast = \_FI$}
		\If{$MTime = \_NE$}	\label{alg3_noSol_1}
			\State \textbf{return $<true,MLast>$}	\label{alg3_noSol_2}
		\Else						\label{alg3_fsol_1}
			\If{$MTime < t_{opt}$}
				\State $x^c = MPoint$
				\State $x^{c+1,\cdots,p-1}$ $\gets$ \Call{RetrieveFromMemo}{ }
				\State \Call{Post\_Solution\_Operations}{ }
				\If{$bk >= c$}
					\State $bk = NULL$
				\EndIf
			\EndIf
			\State \textbf{return $<true,MLast>$}
		\EndIf						\label{alg3_fsol_2}
	\ElsIf{$MLast \neq \_FI$}
		\If{$MTime \neq \_NE \wedge MTime<t{opt} \wedge t_c(x_{MLAST}) \neq MPoint.size$}	\label{alg3_sol_1}
			\State $x^c = MPoint$
			\State $x^{c+1,\cdots,p-1}$ $\gets$ \Call{RetrieveFromMemo}{ }
			\State \Call{Post\_Solution\_Operations}{ }
			\If{$bk > c$}
				\State $bk = NULL$
			\ElsIf{$bk = c$}
				\State $bk = NULL$
				\State $Mem[c][w].$\Call{makeFinal}{ }
				\State \textbf{return $<true,MLast>$}
			\Else
				\State \textbf{return $<true,MLast>$}
			\EndIf
		\EndIf								\label{alg3_sol_2}
		\If{$MLast \neq \_NE$}	\label{alg3_resume_1}
			\State \textbf{return $<false,MLast>$}
		\EndIf						        \label{alg3_resume_2}
	\EndIf
	\State \textbf{return $<false,MLast>$}
\EndFunction
\end{algorithmic}
\end{algorithm}

\subsubsection{Store to Memory}
When a solution is found, the function Post\_Solution\_Operations() is responsible for storing it into the memory. Figures \ref{ex_fig3}-\ref{ex_fig8} depicts storing intermediate results in $Mem$.

\subsubsection{Solution Finalization}
When all possible data points for a workload size on a processor is examined, the stored solution for the processor is set $\_FI$ using the function $makeFinal()$. When a solution is found, the status of all processors having the same execution time with the maximum one are set $\_FI$. It is because that finding a solution with smaller execution time on these processors does not improve optimal solution (Section \ref{sec_backtracking}). Take figures \ref{ex_fig5}, \ref{ex_fig6} and \ref{ex_fig8} as examples.

\subsubsection{Store Last Index}
Storing last index helps process to be resumed form where it has been interrupted Suppose we want to backtrack to $P_i$. Indexes of last examined data points for processors $P_{i+1},\cdots,P_{p-2}$ are recursively stored.(Algorithm \ref{alg2_code}, Line \ref{alg1_lastIndex}). Figure \ref{ex_fig6} shows storing last examined index of $T_2$ for work-size of 7.

\section{Correctness Proof of $HetWD$}
Depth-First Search (DFS) is an option for searching a tree data structures to examine the search area and find the optimal solution. Suppose a tree with maximum height $p$ where the level $L_0$ is responsible for examination of all data points in the time function $T_0$, $L_1$ for $T_1$ and so on. In addition to data points existing in the function, a data point with zero size and zero execution time (no workload is assigned to a processor) is examined on levels $L_0$ to $L_{p-2}$. Depth-first spanning of the tree enables us to examine all combinations, extracts all possible solutions and then find the optimal one which is minimum in parallel execution time.

Since full spanning a search tree is exponential, we proposed algorithm called $HetWD$ applies Branch-and-Bound technique to find the optimal solution much quicker. It uses a series of optimizations (section \ref{sec_opt}) to reduce the exponential search space to polynomial. In this section, we are going to prove that these optimizations only cut the branches which do not involve any optimal solution.

\begin{lemma}	\label{lem_time_threshold}
	Branches cut by time threshold optimization do not involve the optimal solution.
\end{lemma}

\textit{Proof.} We are looking for a solution with parallel execution time less than the time threshold $t_{opt}$. Suppose a solution with the partition $x^i$ on level $L_i$ where $t_i(x^i) > t_{opt}$. According to the Eq. \ref{eq1}, the parallel computation time of this solution is:

$$Time = \max_{j=0}^{p-1} \quad x^j$$

Since the solution involves the partition $x^i$ with the execution time greater than $t_{opt}$, the parallel execution time of the solution will be greater than $t_{opt}$. Thus, the solution cannot be the optimal one we are looking for. \textit{End of Proof}.

\begin{lemma}	\label{lem_size_threshold}
	Branches cut by size threshold optimization do not involve the optimal solution.
\end{lemma}

\textit{Proof.} Suppose the node $r_i/S_i$ in level $L_i$ where $r_i$ represents remaining workload size should be distributed on processors $P_i,\cdots,P_{p-1}$ and $S_i$ determines size threshold for the level $L_i$ and $x_g^i$ represents the maximum workload size in $T_i$ that its execution time is less than $t_{opt}$ (Section \ref{sec_size_Threshold}).

Suppose a distribution for $r_i = {x^i,\cdots,x^{p-1}}$ on processors $P_i,\cdots,P_{p-1}$ which satisfies the equation \ref{lem2_r_i}.

\begin{equation}	\label{lem2_r_i}
\sum_{j=i}^{p-1} x_j = r_i
\end{equation}

We know that $S_i$ is the summation of $x_g^{i \in [i,p-1]}$ and $x^j \le x_g^j, j \in [i,p-1] , x^j \in r_i$. Thus any distribution on processors $P_i,\cdots,P_{p-1}$ should satisfy the equation \ref{lem2_con}.
\begin{equation}	\label{lem2_con}
\sum_{j=i}^{p-1} x_j \le \sum_{j=i}^{p-1} x_g^j \Longrightarrow \sum_{j=i}^{p-1} x_j \le S_i \Longrightarrow r_i \le S_i
\end{equation}

It means that $\forall i \in [0,\cdots, p-1]$ there is no distribution for $r_i$ in case $r_i > S_i$.

\textit{End of Proof}.

\begin{lemma}	\label{lem_backtracking}
	Subtrees ignored by backtracking do not involve an distribution better than current solution.
\end{lemma}

\textit{Proof.} Suppose $D_{cur} = <x^0, x^1,\cdots,x^{p-1}>$ be the best distribution found till now for work-size $n$ on $p$ processors, and $Time(D_{cur}) = t_{opt}$ represents its parallel execution time. Let consider a given $i \in \mathbb{Z}_{[1..p - 1]}$ where $x^i \in D_{cur}$ and it is the closet node to the tree's root with the execution time equals with $t_{opt}$ $t_i(x^i)=t_{opt}$. Therefore:
$$\max_{j=0}^{i-1} Time(x^j) < t_{opt}$$
and
$$\max_{j=i}^{p-1} Time(x^j) = t_{opt}$$

We know time functions are sorted in non-decreasing order of execution times. So the execution times of all data points located after $x^i$ in $T_i$ will be greater than or equal with $t_{opt}$. Therefore, since we are looking for a solution with parallel execution time less than $t_{opt}$, the examination of following data points from $T_i$ will not bring us a distribution with the execution time less than current $t_{opt}$. \textit{End of Proof}.

\begin{proposition}
	Suppose $\Delta x$ be the minimum granularity of workload so that each processor is allocated a multiple of $\Delta x$ only. Let the time function of a processor, $T_i$, be represented by a discrete set of experimental points separated by $\Delta x$ and sorted in non-descending order of time. Then the proposed algorithm distribute the workload with size of $n$ on $p$ processors so that the parallel execution time stays minimum. Then, the algorithm $HetWD$ gives the optimal solution.
\end{proposition}

\textit{Proof.} $HetWD$ algorithm is based on the DFS technique which examines all possible distributions to find the optimal one. However, the proposed algorithm tries to make search area smaller with Branch-and-Bound method. To this end, we have applied some optimizations including time threshold, size threshold and backtracking. Thus, the correctness of $HetWD$ can be proved just if we prove that these three optimizations do cut sub-trees of the main DFS which do not involve better solutions than the distribution already found. It can be proved using the lemmas \ref{lem_time_threshold}, \ref{lem_size_threshold}, \ref{lem_backtracking}. They prove the applied optimizations only cut branches involving either sub-optimal distributions or not-better distribution than the solution currently has been found. \textit{End of Proof}.

\begin{proposition}
	$HetWD$ is terminable.
\end{proposition}

\textit{Proof.} There is a while loop in the $HetWD$ that its iteration is bound to the time threshold $t_{opt}$. We know that $t_{opt}$ is updated when a distribution with less execution time is found and it means that the values for the time threshold have a non-increasing order. Since time functions are sorted in non-decreasing order of time and the algorithm go ahead from smaller execution times to greater ones, after some iterations the time of data points will be greater than the time threshold, and then the while loop finishes. \textit{End of Proof}.

\section{Complexity of The Proposed Algorithm}	\label{sec_timeComp}
Let a workload size of $n$ which is a multiple of $\Delta x$. Let the workload should be distributed on $p$ processors. The core function $HetWD$ implicitly builds a tree data structure consisting of $p$ levels. There are at most $n$ work sizes should be resolved on every level except the leaf one. Let $m$ be the maximum number of points with execution time less than the time threshold in a time function. Thus, the maximum number of points should be examined for each time function is at most $m + 1$ ($m$ data points from time function plus 1 point for zero-size problem). At the mercy of the memorization, each data point for a given work size is required to be examined just only one time. So, the number of recursive calls for each non-leaf level is $n * (m + 1)$. Despite the last level, recursion happens in $p - 1$ non-leaf ones. Thus, the maximum number of recursion calls without considering optimizations (section \ref{sec_opt}) is bounded by $O(n * m * p)$.

\section{Experimental Results}
To evaluate the proposed algorithm, some random time functions with lots of fluctuations have been built. Figure \ref{fig:randomTimeFunction} depicts a randomly built time function. The vertical axis shows execution times and the other one is workload sizes. There are 1000 data points in each function with the granularity of 64 ($\Delta x = 64$).

\begin{figure}[!t]
\centering
\includegraphics[width=6in]{Images/randonTimeFunction.jpg}
\caption{A random time function}
\label{fig:randomTimeFunction}
\end{figure}

\subsection{Parallel Execution Time Improvement}
Figure \ref{fig:etimeComp} compared the execution time of load-equal distribution with that of heterogeneous one for 16 processors. In this figure, horizontal and vertical axes show workload sizes and parallel execution times, respectively. Workload sizes rage from 1024 ($=16*64$) to 1024000 ($=16*64000$) with granularity of 64.

\begin{figure}[!t]
\centering
\includegraphics[width=6in]{Images/eTimeCompare_set16.jpg}
\caption{Load-equal vs. heterogeneous distribution}
\label{fig:etimeComp}
\end{figure}

\subsection{Processing Time}
In this section we are going to examine the processing speed of the proposed algorithm. Figures \ref{fig:pTime16} and \ref{fig:pTime128} show the time taken to find optimal solutions for 16 and 128 processors. Each time function consists of 1000 data points with granularity of 64. The experiments have been run on HCLServer.

\begin{figure}[!t]
\centering
\includegraphics[width=6in]{Images/procTime_set16.jpg}
\caption{The time consumed to find optimal heterogeneous distributions for 16 processors.}
\label{fig:pTime16}
\end{figure}

\begin{figure}[!t]
\centering
\includegraphics[width=6in]{Images/procTime_set128.jpg}
\caption{The time consumed to find optimal heterogeneous distributions for 128 processors.}
\label{fig:pTime128}
\end{figure}

\subsection{Time Complexity}
According to section \ref{sec_timeComp}, the proposed algorithm's processing time is bounded by $O(n * d * p)$. Since the number of maximum points in every time function ($d$) is not too large, it can be considered as a constant, and it means that the processing time will be a parameter of both work-size ($n$) and the number of processors ($p$). To evaluate the correctness, figures \ref{fig:n_p_time} and \ref{fig:n_p_recCall} depict how $n$ and $p$ affect processing time and the number of recursive calls, respectively. Work sizes range from 8192 to 256000, and the number of processors varies between 4 and 128. It is apparent that there is a linear relationship between both $n$ and $p$ with processing and recursion calls.

\begin{figure}[!t]
\centering
\includegraphics[width=6in]{Images/n_p_time.jpg}
\caption{The relationship of work-size and number of processors with processing time.}
\label{fig:n_p_time}
\end{figure}

\begin{figure}[!t]
\centering
\includegraphics[width=6in]{Images/n_p_recCall.jpg}
\caption{The relationship of work-size and number of processors with recursion calls}
\label{fig:n_p_recCall}
\end{figure}

\end{document}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\section{Lecture 31}
\subsection{Lecture Notes - The Strain Tensor and Hooke's Law for Solids}
\subsubsection{The Stress Tensor}
Recall we can write the vector area element as $d\v{A} = \hat{\v{n}}dA$. What is then the stress tensor? We consider the force on a surface element $\v{F}(d\v{A})$, which we can express using the stress tensor:
\[\v{F}(d\v{A}) = \sigma d\v{A}\]
Where $\sigma$ is the stress tensor, which is a 3x3 matrix. We could alternatively write this as:
\[F_i(d\v{A}) = \sum_{j=1}^3 \sigma_{ij}dA_j\]

\subsubsection{Stress Tensor Elements}
Let
\[F_1(\text{On area dA normal to} \hat{\v{e}}_1) = \sigma_{11}dA\]
This is of course true for $\sigma_{22}$ and $\sigma_{33}$. $\sigma_{ii}$ is therefore ith component of the force $\perp$ to the i-axis. These are volumetric forces, and this tells us that the tensile and compressive forces correspond to the diagonal entries of the stress tensor. Next, consider \[F_2(\text{on area dA normal to } \hat{\v{e}}_1) = \sigma_{21}dA\]
And similarly for $\sigma_{31}$. These are clearly shearing forces, forces that act perpendicular to the plane.

\subsubsection{Symmetry of the Stress Tensor}
We said above that the stress tensor has 9 entries (3x3) but there is good news; it turns out that the tensor is actually symmetric (i.e. just 6 components to worry about!). To see this, consider the following geometry:
\begin{center}
    \includegraphics[scale=0.8]{Lecture-31/l31-img1.png}
\end{center}
We now apply a force in the $1$ direction, perpendicular to the $2$ direction, i.e. $F_{a} = \sigma_{12}dA$. We can also apply a force in the $2$ direction, perpendicular to the $1$ direction, which gives $F_b = \sigma_{21}dA$. It is clear to see that these forces induce a torque. WE can also apply the same forces at the opposite corner, on the opposite direction (of equal magnitude):
\begin{center}
    \includegraphics[scale=0.8]{Lecture-31/l31-img2.png}
\end{center}
The torque is then given by:
\[\tau = F_b l - F_q l = (\sigma_{21} - \sigma_{12})ldA = \Gamma_3\]
(this torque is in the 3-direction). Furthermore, we have that:
\[\Gamma_3 = \dod{L_3}{t}\]
We now argue that $\Gamma_3$ is zero (and hence that $\sigma_{21} = \sigma_{12}$. To see that this is the case, consider shrinking all sides of the square by a factor $\lambda$. If I change $l$ by $\lambda l$, I pick up a factor of $\lambda$, and the area picks up a factor of $\lambda^2$, for a total scaling of:
\[\Gamma_3 \mapsto \lambda_3 \Gamma_3\]
Then, what happens to the angular momentum? We pick up $\lambda^2$ from the $\v{r} \times \v{p}$, and then integrating over the plane, we have that we pick up a $\lambda^2$, so it follows that:
\[\lambda^3\Gamma_3 = \lambda^4\dod{L_3}{t}\]
But this is ture for all $\lambda$, so it follows that $\Gamma_3$ must be zero, and hence $\sigma_{21} = \sigma_{12}$. An identical argument shows that $\sigma_{ij} = \sigma{ji}$ in general.

\subsubsection{Displacements}
We now have a measure of force on the system, but we also need a measure of deformation. In general, we can write down a vector $\v{r}$ from the origin to any position in the original configuration, and we change this $\v{r}$ to a new vector $\v{r} + \v{u}(\v{r})$ where $\v{u}$ is the displacement from the reference to the current position.
\begin{center}
    \includegraphics[scale=0.8]{Lecture-31/l31-img3.png}
\end{center}
This displacement vector in general depends on the position, not all points in the object will move the same amount. One might ask why do we want $\v{r} + \v{u}(\v{r})$ and not just $\v{u}$ by itself; consider that $\v{u}$ itself would change during a constant translation ($\v{u}(\v{r}) = \v{u}_0$) of the entire object, and hence is not a good measure of the strain. We need to look at \textbf{distortions}. A general way to write down/pick up these distortions:
\[du_i = \sum_{j}\dpd{u_i}{r_j}dr_j\]
Or we can write this vectorially:
\[d\v{u} = \DD d\v{r}\]
Where:
\[\DD = \m{\dpd{u_1}{r_1} & \dpd{u_1}{r_2} & \dpd{u_1}{r_3} \\ \dpd{u_2}{r_1} & \dpd{u_2}{r_2} & \dpd{u_2}{r_3} \\
\dpd{u_3}{r_1} & \dpd{u_3}{r_2} & \dpd{u_3}{r_3}}\]
And this matrix contains the rate of change of the displacement. This is nice, as evidently this is now insensitive to any constant translations. The gradient of the constant translation will be zero, which is what we want as we should not have to pay any energy just by moving our rigid body back and forth.

\subsubsection{The Strain Tensor}
But there is another wrinkle to consider; rotating the body should also not change the energy of the body/the energy should not depend on the orientation. What do we then do about rotations?
COnsider that for a small rotation:
\[\bm{\theta} = \theta\v{u}\]
About an axis $\v{u}$. To see this,
\[\v{v} = \bm{\omega} \times \v{r}\]
So we can use this to write:
\[\v{u}(\v{r}) = \v{v}dt = \bm{\omega} dt \times \v{r} = \bm{\theta} \times \v{r}\]
Then we have that the displacement gradient has the form:
\[\DD = \m{0 & \theta_3 & -\theta_3 \\ -\theta_3 & 0 & \theta_1 \\ \theta_2 & -\theta_1 & 0}\]
Which we can see is an antisymmetric matrix. The antisymmetry means that:
\[\DD^T = -\DD\]
We need to get rid of this; we dont want a measure that picks up these rotations. We can construct this by remembering that any matrix can be decomposed into a symmetric and antisymmetric part. So, we write:
\[\DD = \frac{1}{2}(\DD - \DD^T) + \frac{1}{2}(\DD + \DD^T)\]
Where the first term is by construction anti-symmetric, and the second term is by construction symmetric. Hence, we will just keep the second term, and use this as the measure of strain. The first term corresponds to the vorticity/curl part, but here we only want to keep the symmetric part. Hence, we can define the small-strain tensor as:
\[\e = \frac{1}{2}\left(\DD + \DD^T\right)\]
Which we can see is symmetric by construction;
\[\e_{ij} = \frac{1}{2}\left(\dpd{u_i}{r_j} + \dpd{u_j}{r_i}\right)\]

\subsubsection{Example: Thin/thick plate in xy plane}
Consider a thin plate subject to in plane (xy) tensile, compressive, or shear forces. Are $\sigma_{zz}$ and $\e_{zz}$ zero? nonzero?
\begin{center}
    \includegraphics[scale=0.8]{Lecture-31/l31-img4.png}
\end{center}
\begin{s}
$\sigma_{zz} = 0$ and $\e_{zz} \neq 0$. For the first point, we can recognize that pulling on the plate in the xy plane induces no stress on the plane in the z direction. For the second point, we consider a sheet of rubber which changes in thickness as we pull it.
\end{s}
What if we ask the same question, but this time the plate is thick?
\begin{s}
If we compare the thick to the thing plate, any length change from the contraction effect would be very very small as the rod is tall. Hence, if we elongate it a little bit, then to first order, there is no strain. But, there can be a stress, as the system would like to contract.
\end{s}

The first case (thin plate) corresponds to a plane stress condition. There is no stress in the z-axis, but if we pull, we get an appreciable change in the thickness of the plate and hence a nonzreo strain. On the other hand, we have the plane strain condition, where there is no strain in the z axis (rod is so tall such that the strain in the z-direction is negligeble/the rod does not change thickness when we pull) but we could still have a stress in the z-axis.

\subsubsection{Hooke's Law for Isotropic and Homogenous Solids}
Consider a decomposition of strain tensor. Consider the quantity of average dilation:
\[e = \frac{1}{3}(\e_{11} + \e_{22} + \e_{33}) = \frac{1}{3}\Tr(\e)\]
Which is a measure of how much the system is compressed/pulled. The last equality we just write the expression as the trace of the strain tensor. Then, decomposing we have:
\[\e = e\II + \e^{dev} = \text{Vol}(\e) + \text{Dev}(\e)\]
Where the first term is the spherical term (the term that couples to the volume changes) and the second term is the deviatoric part (everything else, e.g. shear). Then, Hooke's Law says that:
\[\sigma = f(\e)\]
Where $f$ is a linear function. Then, we write (Without proof) that in the linear case, we can use this decomposition to obtain:
\[\sigma = 3B\text{Vol}(\e) + 2G\text{Dev}(\e)\]
Where $B$, $G$ are the bulk and shear moduli. We can alternatively write this as:
\[\sigma = 2\mu\e + \lambda \Tr(\e) \II\]
Where $\mu = G$ and $\lambda = B - \frac{2}{3}\mu$. What is important to realize is that this is true for an isotropic and homoegnous solid, in which case only two elastic moduli are sufficient to characterize this response (we only need to know the bulk and shear moduli). Of course in something like an anisotropic metal, this would be more complicated.
\end{document}