\documentclass{article}
\usepackage[utf8]{inputenc}

\title{MAT257 Notes}
\author{Jad Elkhaleq Ghalayini}
\date{March 18 2019}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{mathtools}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{cancel}

\usepackage[margin=1in]{geometry}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem*{corollary}{Corollary}
\newtheorem{exercise}{Exercise}
\newtheorem{claim}{Claim}
\newtheorem{proposition}{Proposition}

\DeclareMathOperator{\Int}{Int}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Div}{div}
\DeclareMathOperator{\curl}{curl}
\DeclareMathOperator{\Ker}{Ker}
\DeclareMathOperator{\Ima}{Im}
\DeclareMathOperator{\Vol}{vol}
\DeclareMathOperator{\D}{D}

\newcommand{\reals}[0]{\mathbb{R}}
\newcommand{\nats}[0]{\mathbb{N}}
\newcommand{\ints}[0]{\mathbb{Z}}
\newcommand{\rationals}[0]{\mathbb{Q}}
\newcommand{\brac}[1]{\left(#1\right)}
\newcommand{\sbrac}[1]{\left[#1\right]}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\eval}[3]{\left.#3\right|_{#1}^{#2}}
\newcommand{\ip}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\prt}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\mb}[1]{\mathbf{#1}}
\newcommand{\hlfspc}[0]{\mathbb{H}}
\newcommand{\loint}[0]{\operatorname{L}\int}
\newcommand{\hiint}[0]{\operatorname{U}\int}
\newcommand{\indic}[1]{\chi_{#1}}

\begin{document}

\maketitle

\section{The Volume Element}

We already talked about the volume element in the context of alternating tensors. Of course, we're going to apply that to the tangent space at every point of an oriented \(k\)-dimensional manifold \(M \subseteq \reals^n\) (with or without bound). Consider a particular orientation \(\mu\). If we look at the tangent space \(M_x\) at a point \(x \in M\), it has an orientation \(\mu_x\) and also has an inner product \(T_x\) given by the standard inner product in \(\reals^n\).

These two things, as we saw before, are what we need to talk about a volume element (at the point \(x\)), which is an alternating \(k\)-tensor on the tangent space
\begin{equation}
  \omega(x) \in \Omega^k(M_x)
\end{equation}
which is the unique element in the space of alternating \(k\)-tensors such that if \(v_1,...,v_k\) is a positively oriented orthonormal basis of \(M_x\),
\begin{equation}
  \omega(x)(v_1,...,v_k) = 1
\end{equation}
In particular, for any \(w_1,...,w_k \in M_x\),
\begin{equation}
  \omega(x)(w_1,...,w_k)
\end{equation}
is the \textit{oriented} \(k\)-dimensional volume of the parallelopiped determined by \(w_1,...,w_k\), implying it's absolute value is the \(k\)-dimensional volume of the parallelopiped determined by \(w_1,...,w_k\). I didn't say these \(w_1,...,w_k\) were linearly independent, but if they weren't the \(k\)-dimensional volume would be zero. Also notice that if \(w_1,...,w_k\) are linearly independent, then
\begin{equation}
  [w_1,...,w_k] = [v_1,...,v_k] \implies \omega(x)(w_1,...,w_k) > 0
\end{equation}
For an example, note that \(\det\) is the volume element of \(\reals^n\) (at every point) with the standard inner product and orientation.
Note that, taken over the whole manifold, \(\omega\) is by definition a (nonzero, and \(\mc{C}^r\) if \(M\) is \(\mc{C}^r\)) \(k\)-form. We often call this form \(dV\), but be careful: it's not necessarily the differential of some \((k - 1)-\) form ``\(V\)''. In low dimensions,
\begin{itemize}
  \item Where \(k = 1\), \(dV\) is often written \(ds\) for \underline{length}
  \item Where \(k = 2\), \(dV\) is often written \(dA\) or \(dS\) for \underline{surface area}
\end{itemize}
We're going to use these for specific versions of Stokes' Theorem.

\section{The Volume of \(M\)}
\begin{definition}
  Provided it exists (e.g., if \(M\) is compact), we define the \underline{volume} of \(M\) to be
  \begin{equation}
    \int_MdV
  \end{equation}
\end{definition}
Let's look at some examples:
\begin{enumerate}

  \item Let \(M\) be some \(n\)-dimensional submanifold of \(\reals^n\) with the standard orientation. Then
  \begin{equation}
    dV = dx_1 \wedge ... \wedge dx^n
  \end{equation}
  Hence we have that
  \begin{equation}
    \int_MdV = \int_M1
  \end{equation}
  giving us that the volume of \(M\) agrees with our old definition.

  \item Another situation that we've looked at before is the case where \(M\) is a \(1\)-dimensional oriented submanifold of \(\reals^n\) (a curve). Note that all \(1\)-dimensional submanifolds are orientable.

  It doesn't make sense in general to say ``let's compute the length of this'', because the length may not even be finite. But let's compute the length of a finite piece. This means in general that we'll be looking at an oriented 1-cube in \(M\)
  \begin{equation}
    c: [0, 1] \to M
  \end{equation}
  Let's try to compute the length of \(c([0, 1])\), i.e. just that piece. That means we're going to integrate
  \begin{equation}
    \int_{c([0, 1])}ds = \int_{c}ds = \int_{[0, 1]}c^*(ds) = \int_{[0, 1]}\sqrt{\dot{c_1}^2 + ... + \dot{c_n}^2}
  \end{equation}
  Why? Say \(c^*(ds) = f(t)dt\). That means that
  \begin{equation}
    f(t) = c^*(ds)(t)(e_{1, t}) = ds(c(t))(c_{*,t}(e_{1, t})) = ds(c(t))(c'(t)_{c(t)}) = |c'(t)|
  \end{equation}

  \item Let's see how to compute \(k\)-dimensional volume (in fact, the integral of a \(k\)-form in general) on a \(k\)-dimensional manifold: let \(M\) be a \(k\)-dimensional oriented submanifold of \(\reals^n\) with orientation \(\mu\), and let \(\omega\) be a \(k\)-form on \(M\). We can write
  \begin{equation}
    \omega = \lambda dV
  \end{equation}
  where \(\lambda\) is a function on \(M\). Let \(\varphi: W \to M\) be a \(\mc{C}^{r + 1}\) one-to-one function of constant rank \(k\) (e.g. a coordinate chart). Provided the integral exists, then, we have
  \begin{equation}
    \int_{\varphi(W)}\omega = \int\omega\varphi^*(\omega) = \int_W\lambda \circ \varphi\varphi^*dV
  \end{equation}
  Say, for \(x \in W\),
  \begin{equation}
    \varphi^*dV = h(x)dx_1 \wedge ... \wedge dx_k
  \end{equation}
  How do we compute \(h(x)\). It's:
  \begin{equation}
    (\varphi^*dV)(x)(e_{1,x},...,e_{k,x})
  \end{equation}

\end{enumerate}

\end{document}


\documentclass{article}
\usepackage[fleqn]{amsmath}
\usepackage{graphicx}
\usepackage
    [tight,
%     dvips, % <- replace with any supported driver
     designi,%forpaper,
     navibar
    ]{web}
\usepackage{exerquiz}
\usepackage[ImplMulti]{dljslib}
\usepackage{pifont}

\title{Extending the Exerquiz Package Math Fillin Questions}
\author{D. P. Story}
\subject{Sample file}
\keywords{LaTeX, PDF, derivative, calculus, JavaScript}
\university{NORTHWEST FLORIDA STATE COLLEGE\\
Department of Mathematics}
\email{dpstory@acrotex.net}
\version{3.0}
\copyrightyears{1999-\the\year}
\norevisionLabel

% Uncomment the lines below to create buttons
% for "Begin Quiz", "End Quiz" instead of links
%\useBeginQuizButton[\textColor{blue}]
\useEndQuizButton[\textColor{blue}]

% Uncomment this line to require the student
% to answer all questions in the quiz before
% the quiz is evaluated.
%\renewcommand\minQuizResp{highThreshold}

\font\hv=cmtt10 \def\hvperk{\char`^}
%\font\hv=hv at 9pt \def\hvperk{\char142 }
{\catcode`\^=\active
\gdef\js{\bgroup\hv\catcode`\^=\active \let^=\hvperk \jsi}
}\def\jsi#1{#1\egroup}

\def\hr#1{\textcolor{red}{#1}}
\def\hb#1{\textcolor{blue}{#1}}

\def\D{\dfrac {\text{d}}{\text{d}x}}
\def\dPose#1{$\D#1=$ }

\begin{document}

\maketitle
%\tableofcontents

\section{Math Fill-In Question}

The current version of \textsf{exerquiz} included only multiple
choice questions.  That's all well and good, but instructors would
occasionally like to ask questions that would require the student
to fill in the answer, whether it be numerical or symbolic.

This new question type, the Math Fill-in, can be used in either
the \texttt{shortquiz} or \texttt{quiz} environment.   In the
former case, there is immediate feedback as to right or wrong; in
the latter case, correction is delayed until the quiz is completed
and the user requests to be corrected.

In either case, you can choose to include the correct answer,
which can be obtained by the click of a button.

The next section outlines the syntax for the user to enter the
answer.

\subsection{Instructions}

When responding to a Math Fill-in question, you answer by typing
in your answer.  Use the following notation to enter your answers.

\begin{itemize}
\item Use \js{*} to indicate multiplication: Type \js{4*x} for $4x$;

\item Use \js{^} to indicate powers: Type \js{4*x^3} for
$4x^3$; \js{12*x^-6} for $12x^{-6}$.

\item Use parentheses to delimit the argument of a function; i.e., type
\js{sin(x)} rather than \js{sin x}.

\item Use parentheses to define the {\it scope\/} of an operation: For
example, type \js{4*x*(x^2+1)^3} for $4x(x^2+1)^3$; \js{ 4^(2*x+1)} for
$4^{2x+1}$; \js{(sin(x))^2} for $(\sin(x))^2$. {\it Do not\/} type
\js{sin^2(x)} for $\sin^2(x)$, type \js{(sin(x))^2} instead.

\item You can also use brackets \js{[ ]} or braces \js{\{ \}},
to delimit a mathematics expression.

\item Functions you may use:
\begin{itemize}
    \item  Trig: \js{sin}, \js{cos}, \js{tan}, \js{cot}, \js{sec}, \js{csc};
    \item  Inverse Trig: \js{asin}, \js{acos}, \js{atan};
    \item  Log: \js{ln} (natural log), or use \js{\tt log}; e.g. \js{ln(x)};
    \item  Exponential: The natural exponential function, $e^x$,
           can be entered as \js{exp(x)} or as \js{e^x}.
    \item  The absolute function, \js{abs($\cdot$)} can also be written
           in the usual way \js{|$\cdot$|}; thus, you can type either
           \js{abs(x)} or \js{|x|}.
    \item  Misc.: \js{sqrt}, usage \js{sqrt(x)} for $\sqrt{x}$ (or, use
           exponential notation: \js{x^(1/2)}).
\end{itemize}
\end{itemize}

When you enter your response, some attempt will be made to
determine whether the response is a valid mathematical expression.
For example, if you say \js{san(x)}, the function `\js{san}' will
not be recognized as a valid mathematical function; an error
message is generated, and the user is not penalized for a possible
typing error.  The JavaScript routines will also check for
unbalanced parentheses; thus, \js{((x^4+1) + sin(x)^2} will be
flagged as a syntax error.

\medskip\noindent\hr{\textbf{Important:}} When you enter a function
of a single variable---the only type currently supported---use as
the independent variable implied by the statement of the problem.
If the problem statement involves the variable $x$, use \js{x} as
the independent variable; if the problem statement uses $t$, use
\js{t} in your answer.  To enter a function of $t$ when a function
of $x$ is expected will, no doubt, result in missed problem.

\subsection{Answers and Solutions}

For fill-in questions, if the document author so wishes, answers
and (optionally) solutions can be provided.  The author provides
an ``Ans'' button. This button is visible for a \texttt{shortquiz}
and hidden for a \texttt{quiz}.

For a \texttt{shortquiz}, the ``Ans'' button can be clicked at
anytime. In the case of a \texttt{quiz}, after a \texttt{quiz} has
been completed, the hidden ``Ans'' buttons appear. Click on the
button to get an answer to the problem.

Concerning solutions. If the ``Ans'' button has a green boundary,
that means that question has a solution. Performing a
\texttt{Shift-Click} on the ``Ans'' button causes the viewer to
jump to the solution.  For multiple choice questions, the boundary
for the correct answer is colored green as well. Click on the
answer field to jump to the solution.

Solutions to a \texttt{quiz} can be protected from prying eyes
with the \texttt{\string\NoPeeking} command. See the
\hyperref[s:quiz] {Section~\ref*{s:quiz}} for an example and a
brief discussion.

\subsection{Practice Example}

\noindent\hb{\textbf{Example:}}  Consider the following sample question.
Practice by typing in the answer:
\js{2*x^3*(x^4+1)^(-1/2)}

\medskip
\begin{oQuestion}{example}

\def\RBW{120pt}\def\DefaultHeightOfWidget{16pt}\def\TBW{18pt}
\edef\STRUT{\noexpand\rule[\ifxetex-1bp\else-5bp\fi]{0pt}{0pt}}

\noindent$\dfrac{\text{d}}{\text{d}x} {(x^4 + 1)^{1/2}} =
\underset{\text{\rotatebox{-45}{\rlap{The Response Box}}}}
    {\underbrace{\STRUT\efKern{1bp}{0bp}\RespBoxMath{2*x^3*(x^4+1)^(-1/2)}{4}{.0001}{[1,2]}\relax\efKern{1bp}{0bp}}}\kern-1bp
\underset{\text{\rotatebox{-45}{\rlap{Answer Button}}}}
    {\underbrace{\STRUT\efKern{1bp}{0bp}\CorrAnsButton{2*x^3*(x^4+1)^(-1/2)}\efKern{1bp}{0bp}}}\kern-1bp
\underset{\text{\rotatebox{-45}{\rlap{Num.\ of incorrect tries}}}}
    {\underbrace{\STRUT\efKern{1bp}{0bp}\sqTallyBox\relax\efKern{1bp}{0bp}}}\olBdry\efKern{-1bp}{0bp}\sqClearButton$
\end{oQuestion}

\vspace{5\bigskipamount}

When a correct answer is input into the response box, the color of
the border surrounding the response box changes to \textcolor{webgreen}{green}.
The answer can also be expressed correctly using the \js{sqrt}
operator, try modifying the answer using \js{sqrt}.

\newpage
\subsection{Short Quiz Environment}\label{s:shortquiz}

%In the following problems, answer by typing in your
%response using the correct syntax.

Problem \#2 illustrates how to define a problem with a variable
other than the default $x$. Click on the ``Ans'' button to get
answers, shift-click on ``Ans'' buttons that have a green boundary
to get a solution.

\begin{shortquiz*}[answer] Answer each of the following. Passing is 100\%.

\answersEndHook{\hfill\makebox[0pt][r]{\sqTallyBox}}

\begin{questions}

\item If $f$ is differentiable, then $f$ is continuous.
\begin{answers}[diff]{4}
\Ans1 True & \Ans0 False
\end{answers}
\begin{solution}
Yes, differentiability at a point implies continuity at that point.
\end{solution}

%If you want another variable other than $x$, then specify the variable immediately
%following the correct answer in the \RespBoxMath command. Following the optional
%variable specification is the optional named destination for the solution the
%the problem.

\item $\dfrac{\text{d}}{\text{d}t} {4 t^{-1/2}}
    = \RespBoxMath{-2*t^(-3/2)}(t)[diffprob]{4}{.0001}{[1,2]}$\hfill
\CorrAnsButton{-2*t^(-3/2)}\cgBdry\sqTallyBox
\begin{solution}
We use the power rule:
\begin{equation*}
        4 t^{-1/2}
            = 4(-1/2) t^{-3/2}
            = \boxed{-2 t^{-3/2}}
\end{equation*}
Or, in the syntax of this quiz: \js{-2*t^(-3/2)}.
\end{solution}

\item \dPose{e^{x^2}}\RespBoxMath{2*x*e^(x^2)}{4}{.0001}{[0,1]}\hfill
\CorrAnsButton{2*x*e^(x^2)}\olBdry\sqTallyBox

\item $\displaystyle\int_0^\pi \sin(x)\,dx
    = \RespBoxMath{2}{1}{.0001}{[0,1]}$\hfill\CorrAnsButton{2}\olBdry\sqTallyBox
\end{questions}
\end{shortquiz*}

\begin{flushright}
\sqClearButton\olBdry\sqTallyTotal
\end{flushright}

\subsection{Quiz Environment}\label{s:quiz}

The example below illustrates multiple choice questions
and objective math questions in the \texttt{quiz} environment
and is protected by the \texttt{\string\NoPeeking}
command. First, browse the solutions at the end
of the file, when you encounter a solution protected by
\texttt{\string\NoPeeking}, see what happens.
Click on the ``Ans'' button to get answers, shift-click on ``Ans''
buttons that have a green boundary to get a solution.

\NoPeeking % or put in preamble

\begin{quiz}{mathquiz} Answer each of the following. Passing
is 100\%.

\begin{questions}

\item If $\lim_{x\to a} f(x) = f(a)$, then we say that $f$ is\dots
\begin{answers}[cont]{3}
\Ans0 differentiable &\Ans1 continuous &\Ans0 integrable
\end{answers}
\begin{solution}
A function $f$ is said to be continuous at $x=a$ if $x\in\operatorname{Dom}(f)$,
$\lim_{x\to a} f(x) $ exists and $\lim_{x\to a} f(x) = f(a)$.
\end{solution}

\item $\cos(\pi) = \RespBoxMath{-1}[cospi]{1}{.0001}{[2,4]}\cgBdry\kern1bp\CorrAnsButton{-1}$
\begin{solution}
Of course, everyone knows that $\cos(\pi) = -1 $.
\end{solution}

\item \dPose{e^{x^2}}\RespBoxMath{2*x*e^(x^2)}[diffexp]{4}{.0001}{[0,1]}\cgBdry\kern1bp
\CorrAnsButton{2*x*e^(x^2)}
\begin{solution}
First apply the rule for differentiating an the natural exponential, then apply
the power rule:
\begin{align*}
        \dfrac d{dx}\,e^{x^2} &
           = e^{x^2}\,\dfrac d{dx}\,x^2 \\&
           = e^{x^2}\,(2x) \\&
           = 2x e^{x^2}
\end{align*}
In the syntax of this document, \js{2*x*e^(x^2)}.
\end{solution}
\end{questions}
\end{quiz}\quad\ScoreField\currQuiz\olBdry\eqButton\currQuiz

\noindent
Answers: \AnswerField\currQuiz

\end{document}


\chapter{Conclusion}

In this paper, we proposed a novel Android malware detection system using static features extracted from the application files. This system uses a model implemented with several machine learning algorithms combined with our proposed Stacking method to get the best performance of all algorithms. We evaluated our proposed system with 20,000 samples of applications downloaded from Androzoo \cite{androzoo} database.
Among these, static features have proved to be a powerful instrument to represent the behavior of each sample, and they have been used to train different ensemble classifiers that have reached high accuracy values.

Using only one model with static features allow reaching an appealing almost 87.9\verb+%+ of accuracy.
But with our proposed Stacking method, we could improve the accuracy of classification to about 93.6\verb+%+ at the 2nd training stage.
In the 3rd stage, the accuracy down about 1\verb+%+ because of the small dataset.
We could demonstrate that the Stacking method is a powerful way to improve classification models but using it repeatedly may cause the model poor performance.

Another remarkable result is that the description of a single component of the Android platform describing the use of API calls enables to build the most representative feature space, that is to say, the space
where instances of malware and benignware can be better differentiated.

For future work, we want to try to find the best combination of static features to create the best classification model. For example, trying to build the model with only the best 3 or 2 static features (API calls, Opcodes, Receivers) in Figure \ref{fig:1ststage}.


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{graphics}
\usepackage{graphicx}
\usepackage{wrapfig}

\title{Regularization for Deep Learning}
\author{Chen Yuyan}
\date{\today}

\begin{document}
	\maketitle
	A central problem in machine learning is how to make an algorithm that will
	perform well not just on the training data, but also on new inputs. To make an algorithm perform well on training data is called regularization, we will discuss the regularization for deep learning in this passage.
	\section{Regularization Strategies}
	 Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. Developing more effective regularization strategies has been one of the major research efforts in the field.

	Suppose we have already known these basic concepts of generalization:(The relationship between them are showed in Figure \ref{1}.)
	\begin{itemize}
		\item underfitting, overfitting, capacity
		\item bias, variance
		\item regularization
	\end{itemize}

	\begin{figure}\label{1}
	\caption{The relationship between capacity, underfitting, and overfitting.}
	\includegraphics[width=\textwidth]{figures/MachineLearning3}
	\end{figure}

	There are three typical strategies for regularization, and we list them below:
		\begin{itemize}
			\item Put extra	constraints on a machine learning model: such as adding restrictions on the	parameter values.
			\item Add extra terms in the objective function that can be	thought of as corresponding to a soft constraint on the parameter values.
			\item Ensemble methods.
	\end{itemize}
	The	constraints and penalties for the first or second strategies usually have the properties list below:
			\begin{itemize}
				\item They are designed to encode specific kinds of prior knowledge.
				\item They are designed to express a generic preference for a simpler model class in order to promote generalization.
				\item They are necessary to make an underdetermined problem determined
			\end{itemize}

		\section{Parameter Norm Penalties}
		\subsection{Deep learning problem with penalty}
		The deep learning problem with penalty which we discussed in this section is
		\begin{itemize}
			\item $\mathbb X = \{(\bm x^{(1)},y^{(1)}),...,(\bm x^{(m)},y^{(m)})\} \subset \mathbb R^n \times \mathbb R$ is a dataset. Let $\bm X =(\bm x^{(1)},...,\bm x^{(m)})$, $\bm Y=(y^{(1)},...,y^{(m)})$.
			\item Let $f:\mathbb R^n\rightarrow \mathbb R$ is the neural network function of $\bm x\in \mathbb R^n$, and has parameter $\bm w \in \mathbb R^p$, write as $f(\bm x;\bm w)$.
			\item $L(u,v):\mathbb R \times \mathbb R \rightarrow \mathbb R$ is the loss function. And
			$$
			\mathcal L(\bm u,\bm v)=\dfrac 1m\sum_{i=1}^m L(u_i,v_i)
			$$
			\item $J:\mathbb R^p\rightarrow \mathbb R$ is a function of $\bm w$ which describe the complexity of $f$. $\alpha$ is a constant.
			\item Let $\bm F(\bm X;\bm w)=(f(\bm x^{(1)};\bm w),...,f(\bm x^{(m)};\bm w))$, $\mathcal{L}(\bm w) = \mathcal L(\bm F(\bm X;\bm w),\bm Y) $ and $\tilde{\mathcal L}(\bm w) = \mathcal L(\bm w) + \alpha J(\bm w) $\\ Solve:
			$$
			\bm \theta_{ML}=\arg \min_{\bm w}(\tilde{\mathcal L}(\bm w) )
			$$
	\end{itemize}

		\subsection{$L^2$ Parameter Regularization}

		One of the simplest and most common kinds of parameter norm penalty: the $L^2$ parameter norm penalty commonly known as \textbf{weight decay}:
		$$
		J(\bm w) = \dfrac{1}{2}||\bm w||_2^2
		$$

		In other academic communities, $L^2$ regularization is also known as \textbf{ridge regression} or \textbf{Tikhonov regularization}.

		$$
		\tilde{\mathcal{L}} (\bm w) = \dfrac{\alpha}{2} {\bm w}^T\bm w + \mathcal L(\bm F(\bm X;\bm w),\bm Y)
		$$
		$$
		\nabla_{\bm w}\tilde{\mathcal{L}} (\bm w) = \alpha\bm w + \nabla_{\bm w} \mathcal L(\bm F(\bm X;\bm w),\bm Y)
		$$
			To explain what happens over the entire course of training, we can consider the approximation of $\mathcal L$. Let
			$$
				\bm w^*= \text{arg} \min_{\bm w} \mathcal{L}(w)
			$$
			and $\hat{\mathcal{L}}$ is the approximation(2-order Taylor series) of $\mathcal L$ near the $\bm w*$, i.e.
			$$
				\hat{\mathcal{L}} = \mathcal{L}(\bm w^*) + \dfrac{1}{2}(\bm w - \bm w^*)^T\bm H(\bm w - \bm w^*)
			$$
			where $\bm H$ is the Hessian matrix of $\mathcal{L}$. (Notice $\nabla_{\bm w}\mathcal{L}(\bm w^*)=0$.)
			Then the gradient of $\hat{\mathcal{L}}$ is
			\begin{equation}\label{1}
				\nabla_{\bm w} \hat{\mathcal{L}} = \bm H(\bm w - \bm w^*)
			\end{equation}

			To study the effect of weight decay, we modify equation \eqref{1} by adding the
			weight decay gradient. We can now solve for the minimum of the regularized
			version of $\hat{\mathcal{L}}$. We use the variable $\tilde{\bm w}$ to represent the location of the minimum.
			$$
			\alpha \tilde{\bm w} + \bm H(\tilde{\bm w} - \bm w^*)=0
			$$
			Then we can get
			\begin{equation}\label{2}
				\tilde{\bm w} = (\bm H + \alpha \bm I)^{-1}\bm H \bm w^*
			\end{equation}

			where $\bm I$ is the identity matrix.

			As $\alpha \rightarrow 0$, $\tilde{\bm w}\rightarrow \bm w^*$, but what happens as $\alpha$ grows? Because	$\bm H$	is real and symmetric, then we have eigenvalue decomposition:
			$$ \bm H = \bm Q^T \bm \Lambda \bm Q$$
			Applying the decomposition to equation \eqref{2}, we obtain:
			$$
			\tilde{\bm w} = \bm Q(\bm \Lambda +\alpha \bm I)^{-1}\bm \Lambda \bm Q^T \bm w^*
			$$
			We see that the effect of weight decay is to rescale $\bm w^*$ along the axes defined by the eigenvectors of $\bm H$. Specifically, the component of
			$\bm w^*$ that is aligned with the $i$-th eigenvector of $\bm H$ is rescaled by a factor of	$\dfrac{\lambda_i}{\lambda_i+\alpha}$. It's the same as Tikhonov regularization in inverse problem theory.
		\subsection{$L^1$ Regularization}
		Formally, $L^1$	regularization on the model parameter w is defined as
		$$
		J(\bm w) = ||\bm w||_1 = \sum_i|w_i|
		$$
		and
		$$
		\tilde{\mathcal{L}}(\bm w) =\alpha ||\bm w||_1 + \mathcal L(\bm F(\bm X;\bm w),\bm Y)
		$$
		with gradient:
		$$
		\nabla_{\bm w}\tilde{\mathcal{L}}(\bm w) =\alpha \text{sgn}(\bm w) + \nabla_{\bm w} \mathcal L(\bm F(\bm X;\bm w),\bm Y)
		$$
			Because the $L^1$ penalty does not admit clean algebraic expressions in the case
			of a fully general Hessian, we will also make the further simplifying assumption
			that the Hessian is diagonal, $\bm H =\text{diag}(H_{1,1},\cdots,H_{n,n})$, where $H_{i,i}>0$.

			Then we have
			$$
			\hat{\mathcal{L}} = \mathcal{L}(\bm w^*) +\sum_i \dfrac{1}{2}H_{i,i}(w_i - w_i^*)^2
			$$
			when can get the
			$$
			\tilde w_i =\text{sign}(w_i^*)\max \{|w_i^*|-\dfrac{\alpha}{H_{i,i}},0 \}
			$$
		Consider the situation where $w^*_i>0$ for all $i$. There are two possible outcomes:
			\begin{itemize}
				\item The case where $ w^*_i\leq\dfrac{\alpha}{H_{i,i}}$, then $\tilde{w}_i=0$.  This occurs because the contribution of $\mathcal{L}$ to the regularized objective $\tilde{\mathcal L}$ is overwhelmed--in direction $i$--by the $L^1$ regularization which pushes the value of $w_i$ to zero.

				\item The case where $ w^*_i >\dfrac{\alpha}{H_{i,i}}$.  In this case, the regularization does not move the optimal value of $w_i$ to zero but instead it just shifts it in that direction by a distance equal to $\dfrac{\alpha}{H_{i,i}}$
			\end{itemize}
			A similar process happens when $w^*_i<0$.
			In comparison to $L^2$ regularization, $L^1$ regularization results in a solution that is more sparse.

	\section{Ensemble Methods}
	\subsection{Dataset Augmentation}
		The best way to make a machine learning model generalize better is to train it on
	more data. Of course, in practice, the amount of data we have is limited. One way
	to get around this problem is to create fake data and add it to the training set.
	For some machine learning tasks, it is reasonably straightforward to create new
	fake data.

	This approach is easiest for classification. A classifier needs to take a complicated, high dimensional input $\bm x$ and summarize it with a single category identity $y$. This means that the main task facing a classifier is to be invariant to a wide variety of transformations. We can generate new $(\bm x, y)$ pairs easily just by transforming
	the $\bm x$ inputs in our training set. But this approach is not as readily applicable to many other tasks.

	\subsection{Bagging and Other Ensemble Methods}
		\begin{itemize}
			\item \textbf{Bootstrap} comes from a common saying "pull up by your own bootstraps", which means use your won resource to solve problem.
			\item \textbf{Bagging} (short for \textbf{bootstrap aggregating}) is a technique for reducing generalization error by combining several models. The idea is to train several different models separately, then have all of the models vote on the	output for test examples.This is an example of a general strategy in machine learning called	model averaging. Techniques employing this strategy are known as ensemble methods.
			\item The reason that model averaging works is that different models will usually
			not make all the same errors on the test set.
		\end{itemize}
		Neural networks reach a wide enough variety of solution points that they can often benefit from model averaging even if all of the models are trained on the same dataset. Differences in random initialization, random selection of minibatches,
		differences in hyperparameters, or different outcomes of non-deterministic implementations of neural networks are often enough to cause different members of the ensemble to make partially independent errors.
	\subsection{Dropout}
			 \textbf{Dropout} provides a computationally inexpensive but	powerful method of regularizing a broad family of models. To a first approximation, dropout can be thought of as a method of making bagging practical for ensembles of very many large neural networks. Specifically, dropout trains the ensemble consisting of all sub-networks that can be formed by removing non-output units from an underlying base network.

		\newpage
		\begin{figure}
			\includegraphics[width=\linewidth]{figures/dropout}
			\caption{Dropout trains an ensemble consisting of all sub-networks that can be
				constructed by removing non-output units from an underlying base network.}
			\label{fig:dropout}
		\end{figure}

		\begin{wrapfigure}{r}{0.4\textwidth}
			\vspace{-20pt}
			\begin{center}
				\includegraphics[width=0.38\textwidth]{figures/dropoutexample}\label{dropoutexample}
			\end{center}
			\vspace{-20pt}
			\vspace{-10pt}
		\end{wrapfigure}

		To perform forward propagation with dropout, we randomly sample a vector $\bm \mu$ with one entry for each input
		or hidden unit in the network. The entries of $\bm \mu$ are binary and are sampled independently from each other. The probability of each entry being 1 is a hyperparameter, usually $0.5$ for the hidden layers and $0.8$ for the input. Each unit in the network is multiplied by the corresponding mask, and then forward propagation continues through the rest of the	network as usual. This is equivalent to randomly selecting one of the sub-networks from figure and running forward propagation through it.

		 Recall that to learn with bagging, we define $k$ different models, construct $k$ different datasets by sampling from the training set with replacement, and then train model $i$ on dataset $i$. Dropout aims to approximate this process, but with an exponentially large number of neural networks. Specifically, to train with dropout, we use a minibatch-based learning algorithm that makes small steps, such as stochastic gradient descent. Each time we load an example into a minibatch, we randomly sample a different binary mask to apply to all of the input and hidden units in the network.

		Dropout training is not quite the same as bagging training. In the case of
		bagging, the models are all independent. In the case of dropout, the models share
		parameters, with each model inheriting a different subset of parameters from the
		parent neural network. This parameter sharing makes it possible to represent an
		exponential number of models with a tractable amount of memory. In the case of
		bagging, each model is trained to convergence on its respective training set. In the
		case of dropout, typically most models are not explicitly trained at all—usually,
		the model is large enough that it would be infeasible to sample all possible sub-
		networks within the lifetime of the universe. Instead, a tiny fraction of the possible
		sub-networks are each trained for a single step, and the parameter sharing causes
		the remaining sub-networks to arrive at good settings of the parameters. These
		are the only differences. Beyond these, dropout follows the bagging algorithm.

\end{document}

6DL/Regularization for Deep Learning Notes.tex


\chapter{Reference Implementation}

\section{Overview}

The program in this project plays the role of the code breaker. The goal is to find out the secret using as few guesses as possible. The program supports both the Mastermind rules and the GuessNumber rules, subject to the following limits:

Maximum number of colors (defined by MM\_MAX\_COLORS): 10.

Maximum number of pegs (defined by MM\_MAX\_PEGS): 6.

Program Optimization Techniques

While the code-breaking algorithms are quite straightforward, much effort of this project has been put to optimize the performance of a real-time code breaker. Some of the hot-spots are identified by the profiler. The major points of optimization are described below. Most of the optimized routines are implemented in standalone source files for clarity.

Search space pruning. While all codewords are candidates for making a guess, some of them are equivalent in terms of bringing new information. For example, in the first round of a Number Guessing game, any guess works the same. Aware of this, we implement pruning by classifying digits into three classes: impossible, unguessed, and the rest. After each round of feedback, we update the list of distinct guesses (in terms of bringing information) and search within this list only. This reduces the search space significantly.

Codeword comparison. This is the most intensive operation in the program, accounting for 40\% of all CPU time. The program uses SSE2 instructions (implemented via compiler intrinsics) to compare each pair of codewords in four instructions.

Feedback frequency counting. The heuristic code breaker relies heavily on these routines to count statistics on partitions. This is an intensive operation which accounts for about 20\% of all CPU time. The program uses an ASM implementation to maximize performance. See Frequency.cpp.

\section{Data Structure}

\subsection{Codeword}

\subsection{Feedback}

\subsection{Strategy Tree}

\section{Basic operations}

\subsection{Codeword comparison}

\subsection{Frequency table generation}

\subsection{Codeword set partitioning}

\section{Implementing Heuristic Strategies}

 Obvious guesses

\section{Implementing Optimal Strategies}

\section{Implementing randomized strategies}


%%
%% Author: novitoll
%% 3/9/18
%%

% Preamble
\documentclass[11pt]{article}

% Packages
\usepackage{amsmath}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{listings}

\graphicspath{ {pics/} }
\lstset{language=Python}
\lstset{%
  escapeinside={(*}{*)},%
}

\title{CVT: Lecture 7}
\date{2018-03-09}
\author{Novitoll}

\begin{document}
    \maketitle
    \pagenumbering{arabic}

    \section{Machine learning} \label{sec:mlintro}

    \begin{itemize}
        \item Supervised learning
        \item Unsupervised learning
        \item Reinforcement learning
    \end{itemize}

    \section{Linear regression} \label{sec:linearreg}

    Simple linear function $y = mx + b$ in Supervised Learning has the hypothesis $h_\theta(x) = \theta_0 + \theta_1 x$
    to predict with given X N-dimensional matrix of features, $\theta$ is coefficient (something to be found during model training)

    \[
        h_\theta(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + .. + \theta_i x_i = \sum_{i=0}^{m} \theta_i x_i = \theta^T x
    \]

    , where $m$ - is number of features.

    Now we want to minimize the difference between hypothesis $h_\theta(x)$ and ground truth $y$ for high accuracy,
    the difference is called Cost function - $J(\theta)$.

    \[
        J(\theta) = \frac{1}{2}\sum_{i=0}^{m} (h_\theta(x^i) - y^i)^2
    \]

    This is the generalized cost function, in practice you will find its implementations as:
    \begin{itemize}
        \item mean squared error (MSE) = $\frac{1}{2m}\sum_{i=0}^{m}..$
        \item hinge loss
        \item log loss
        \item ..
    \end{itemize}

    In order to $\min_{\theta} J(\theta)$ let's differentiate it (keeping in mind that X - is a matrix of independent variables):

    \begin{gather*}
        \frac{\partial{}}{\partial{\theta_i}}J(\theta) = \frac{\partial{}}{\partial{\theta_i}} \
            \frac{1}{2}(h_\theta(x) - y)^2 = \\
            2 \ \frac{1}{2} \frac{\partial{}}{\partial{\theta_i}} \ (h_\theta(x) - y) \
            \frac{\partial{}}{\partial{\theta_i}} \ (h_\theta(x) - y) = \\
            (h_\theta(x) - y) \ \frac{\partial{}}{\partial{\theta_i}} (\theta_0 + \theta_1 x_1 + .. + \theta_n - y) = \\
            (h_\theta(x) - y) \ \frac{\partial{}}{\partial{\theta_i}} (h_\theta(x) - y) = \\
            (h_\theta(x) - y) \ x_i
    \end{gather*}

    Update of our cost function (Batch gradient descent) is:
    \begin{align}
        \theta_i := \theta_i - \alpha \sum_{i=0}^{m} (h_\theta(x^i) - y^i) \ x^i
    \end{align}

    , where $\alpha$ - is learning rate.
    BGD (1) is good for small dataset and always finds the local minima, however with the real world dataset it's slow.
    For that, usually Stochastic gradient descent (SGD) is used:

    \begin{lstlisting}
        Repeat 1..10 {
            for j=1 .. to m {(*\begin{align}
                    \theta_i := \theta_i - \alpha (h_\theta(x^j) - y^j) \ x_i^j
                \end{align}*)
            }
        }
    \end{lstlisting}

    SGD works well with big dataset, however wont converge to the 0 (minima), but though will be close as much as possible.
    The idea is to find global minima in order not to \textbf{overfit} with finding local minima.

    \includegraphics[scale=0.5]{sgd.png}

\end{document}


\section{Redesigning the WordCount Database}
In section \ref{InitialDesign}, we established that the database design would lead to duplicate or redundant data in the WordCount database.
Namely, the terms occurring in the Term entity set (see section \ref{newdatabaseER}) will not be unique.
Thus, the database can contain duplicate values.

To address this issue, we created relations using the relational model (see section \ref{relational_databases}) and ensured they are well-designed.
After ensuring that a normal form is reached, we created the relations using SQL as a DDL.
We also needed to adjust the data access models (see section \ref{models}) to ensure access using EF core.
When implementing the new database structures, we needed to ensure that the pipeline is operational and has minimum downtime.
Therefore, we had to ensure we could always roll back changes to the code and database structures.
As described in section \ref{CI/CD}, we can easily deploy versions of our code.
By reverting the changes to the old implementation of the models, we were able to ensure a quick response in case the new implementation failed.
We needed to ensure that this was also the case for the database structures.
This was done by taking a dump (backup) of the database, and verifying that no database relations are deleted during the data migration.

\input{sections/Chapter 8 Sprint 6/CreatingNewModels.tex}
\input{sections/Chapter 8 Sprint 6/BCNF.tex}
\input{sections/Chapter 8 Sprint 6/ImplementingTheNewModels.tex}
\subsection{Result}
After having implemented the new database structure, we assessed the changes to the database.
The number of words in the newly implemented \texttt{Word} relation is 260,000.
The number of unique tuples in the newly implemented \texttt{WordOccurance} table is 6,900,000, the same as the number of tuples representing words in the old \texttt{Term} relation.
Thus, the number of tuples in the new relation is the same, but the number of duplicate words have been reduced greatly.

sections/Chapter 8 Sprint 6/database_redesign.tex


% !TEX root = BusSim.tex
\section{Numerical experiment}
\label{s:experiments}
\subsection{Experiment set up}

To generate the synthetic `historical' and `real-time' GPS data used in BusSim-deterministic and BusSim-stochastic, we predetermine a set of model parameters to generate realistic GPS data. Table \ref{tab:param} lists the fixed parameters being used in this experiment.

\begin{table}[htbp]
  \centering
  \caption{Fixed parameters in BusSim-truth}
    \begin{tabular}{rll}
    %\toprule
    \multicolumn{1}{l}{\textbf{Class}} & \textbf{Parameter} & \textbf{Value} \\
    \midrule
    \multicolumn{1}{l}{Bus} & FleetSize  & Unique ID of the bus agent \\
          & Acceleration & 3 $m/s^2$ \\
          & [$\theta_1,\theta_2,\theta_3$] & [3,1,0.85] $s$ \\
    \midrule
    \multicolumn{1}{l}{BusStop} & Number of Stops & 20 \\
          & Length between stops & 2000m \\
          & GeoFence & 50m \\
    \bottomrule
    \end{tabular}%
  \label{tab:param}%
\end{table}%

Second, the dynamic parameter set $S_t = [Arr_m^t \ Dep_m^t \ V^t]$ (Equation \ref{eq:S}) are time-varying and therefore, randomly generated using fixed rules. We first generate an initial arrival rate $Arr_m^0$ at stop $m$ at time 0 by a random generation from an uniform distribution between the minimum and maximum passenger arrival rate [$minDemand, maxDemand$].

  \begin{equation}
   Arr_m = \mathcal{U} (minDemand, maxDemand) \quad m = 1,...,M
   \label{eq:arrival_rate}
\end{equation}

The departure rate is also generated from an uniform distribution, but also ordered non-decreasingly to represent the fact that more passengers alight at the end of the route than at the beginning. The departure rate at the last stop (stop M) is set as 1 to let every remaining passengers to alight the bus at the last stop.
\begin{equation}
   Dep_m =  ordered \ (\mathcal{U} (0.05, 0.5)), \quad Dep_M = 1 \quad \& \quad m = 1,...,M
     \label{eq:departure_rate}
\end{equation}

\subsection{The stochastic and dynamic nature of BusSim-truth}

Figures  \ref{fig:2stochastic} and \ref{fig:2dynamic} provide a simple verification that demonstrates BusSim-truth generates realistic synthetic GPS data under different sets of parameters. Other variables have also been verified and will be used in the sensitivity analysis. This section outlines how this validation was achieved.

\begin{figure}[h]
	\centering
	\includegraphics[width=1\textwidth]{Figures/Fig_spacetime_2stochastic.pdf}
	\caption{Synthetic bus GPS trajectory at low and high passenger demand. Red, dashed lines are bus trajectories when $maxDemand$ equals 0.5, while black, solid lines are bus trajectories when $maxDemand$ equals 2.}
	\label{fig:2stochastic}
\end{figure}

\begin{figure}[h!]
	\centering
	\includegraphics[width=1\textwidth]{Figures/Fig_spacetime_2dynamic.pdf}
	\caption{Synthetic bus GPS trajectory with two different value of $\xi$.}
	\label{fig:2dynamic}
\end{figure}

We aim to control the stochastic and dynamic level in BusSim-truth using only a single parameter for each. Equation \ref{eq:arrival_rate} controls the level of stochasticity in BusSim-truth. For instance, a pair of values [$minDemand, maxDemand$]=[0.5,1] means 0.5 to 1 passenger arriving at the bus stop each minute. By fixing the $minDemand$ to be a small number (e.g. equals 0.5), we can control the stochasticity of BusSim-truth by a single parameter $maxDemand$, with a larger $maxDemand$ meaning more stochasticity and vice versa. We control the level of dynamicity in BusSim-truth by a dynamic change rate parameter $\xi$, which gradually changes the arrival rate and surrounding traffic speed over the simulation period.

To implement an inner verification of the BusSim-Truth model and to investigate the impacts of the \textit{stochastic} and \textit{dynamic} natures of the system under study, we evaluate the outputs from BusSim-truth under different values of stochasticity and dynamicity. Figure \ref{fig:2stochastic} gives an insight into the differences in bus trajectories when $maxDemand$ equals 0.5 and 2. Note that when $maxDemand$ equals 0.5, BusSim-truth reduces to a deterministic model (similar to BusSim-deterministic) because $maxDemand$ would then be equal to $minDemand$.

Each line in Figure \ref{fig:2stochastic} shows the GPS trajectory of bus location, as generated by BusSim-truth. The solid lines show the trajectory of buses at high and stochastic demand ($maxDemand$ equals 3), whereas the dashed lines are for low and deterministic demand ($maxDemand$ equals $minDemand$). The trajectories in Figure \ref{fig:2stochastic} show that as the $maxDemand$ increases, there are more delays for each individual buses and less likely that buses are able to keep stable headway from each other.

The dynamic nature of BusSim-truth is illustrated in
Figure \ref{fig:2dynamic} when the dynamic change rate parameter $\xi$ is equal to 1\% and 10\%. Because the arrival rate and traffic speed gradually change, there is little change in the bus trajectories of BusSim-truth with $\xi$ equals 1\% and 10\%. As time passes, there are more delays for BusSim-truth with $\xi$ equals 10\%. This is because there are more passengers (higher arrival rate) and the buses are also travelling slower (lower traffic speed).

\subsection{Scenario 1: no calibration (benchmark)}

This scenario aims to evaluate the prediction results from BusSim-deterministic and BusSim-stochastic \textit{without} calibrating their parameters or performing data assimilation. This is necessary so that later we can evidence the additional predictive performance of the model after calibration \textit{and} data assimilation. The two models are implemented using random parameters generated from Equation \ref{eq:arrival_rate} and \ref{eq:departure_rate}. The outputs from these models are bus locations at each time step $t$, which can be compiled to space-time trajectories and compared to the synthetic `real-time' bus trajectories, as illustrated in Figure \ref{fig:do_nothing}.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\textwidth]{Figures/Fig_do_nothing_IncreaseRate_7.pdf}
    \caption{Prediction results from Scenario 1: no calibration}
    \label{fig:do_nothing}
\end{figure}

Figure \ref{fig:do_nothing} shows one particular case where $maxDemand$ equals 2, and $\xi$ equals 7\%, as an example of the prediction results. Both models poorly predict the trajectories of the `real' buses. This is expected because the models do not have the optimal parameters to capture the bus route operations. These models are therefore not useful for real-time prediction without parameter calibration or data assimilation.

\subsection{Scenario 2: Parameter calibration}

In this scenario, BusSim-deterministic and BusSim-stochastic are calibrated using the Cross-Entropy Method, as described in Section \ref{s:calibration}. The two calibrated models are used to predict the bus locations at each time step $t$, which can be compiled to trajectories. Figure \ref{fig:calibration} shows an example of the comparison between the predictions from BusSim-deterministic and BusSim-stochastic versus the synthetic `real-time' GPS data, where the $maxDemand$ equals 2 and $\xi$ equals 7\%.

\begin{figure}[htb]
    \centering
    \includegraphics[width=1\textwidth]{Figures/Fig_calibration_IncreaseRate_7.pdf}
    \caption{Prediction results from Scenario 2: Parameter calibration}
    \label{fig:calibration}
\end{figure}

Although they are improvements compared the un-calibrated versions -- Figure \ref{fig:calibration} shows that both models outperform the models in the Scenario 1 (no calibration [see Figure \ref{fig:do_nothing}]) -- the models can only predict well early in the simulation when there is little deviation in passenger arrival rate and surrounding traffic speed. There are large observable gaps between the predictions from BusSim-deterministic and BusSim-stochastic as buses reach the end of their routes. The two models were trained with synthetic `historical' data, but evaluated with `real-time' data. Recall that there are differences between the `historical' and `real-time' data due to the stochastic nature of the system under study (see Figure \ref{fig:historical_realtime}). Therefore a data assimilation procedure is required to prevent the errors gradually increasing throughout the simulation.

\subsection{Scenario 3: Applying a Particle Filter}

This section applies a PF to the calibrated BusSim-deterministic and BusSim-stochastic, as described in section \ref{s:PF}. At each time step $t$, the two models are only provided with the observation vector $O_t$, and then attempts to use $O_t$ to correct their prediction of future state vectors $X_t$ to $X_T$, where $T$ is the last time step. Figure~\ref{fig:calibration_plus_PF} illustrates the results after the models have been calibrated \textit{and} have `real-time' data incorporated (assimilated) into them during runtime.

\begin{figure*}[htb]
    \centering
    \includegraphics[width=1\textwidth]{Figures/Fig_PF_IncreaseRate_5.pdf}
    \caption{Prediction results from Scenario 3: Parameter calibration and Particle Filtering}
    \label{fig:calibration_plus_PF}
\end{figure*}

The predicted bus trajectories in Figure~\ref{fig:calibration_plus_PF} fit much closer to the synthetic `real-time' data than the previous scenarios (Figure \ref{fig:do_nothing} and \ref{fig:calibration}). There are still observable gaps between the prediction and the synthetic `real-time' GPS data, because the underlying models do not know the underlying stochasticiy and dynamicity in the synthetic data, but the improvements (which will be quantified shortly) certainly \textit{appear} to be substantial.

\subsection{Sensitivity analysis}

In this section, we perform a sensitivity analysis to compare the prediction error in each scenario. The same experiments, as described in Scenario 1 to 3, are repeated at different values of $maxDemand$ and dynamic change rate $\xi$. To increase the robustness of the comparison, 10 replications have been made for each experiment, and the average Root Mean Squared Error (RMSE) values are reported. RMSE is calculated as the difference in prediction bus location and synthetic `real-time' bus location:

\begin{equation}
RMSE = \sqrt{\frac{1}{T}\sum_{k=1}^{T}{\Big(\hat{y}_k -y_k\Big)^2}}
\end{equation}

Where $\hat{y}_k$ and $y_k$ is the bus location at time $k$ from the model prediction and synthetic `real-time' data, respectively. Table \ref{tab:sensitivity} compares the RMSE from each scenario. It is clear that the Scenario 3 (combination of parameter calibration and data assimilation) outperforms the other two Scenarios.

\begin{table}[ht]
  \centering
  \caption{Sensitivity analysis of $maxDemand$ and dynamic change rate $\xi$}
    \begin{tabular}{ccccc}
    \toprule
          & \textbf{Values} & \textbf{Scenario 1} & \textbf{Scenario 2} & \textbf{Scenario 3} \\
    \midrule
    \multicolumn{1}{c}{\multirow{9}[2]{*}{maxDemand }} & 0.5   & 302   & 102   & 24 \\
          & 1     & 313   & 107   & 25 \\
          & 1.5   & 319   & 112   & 35 \\
          & 2     & 335   & 125   & 49 \\
          & 2.5   & 340   & 119   & 52 \\
          & 3     & 337   & 127   & 62 \\
          & 3.5   & 346   & 133   & 66 \\
          & 4     & 338   & 148   & 59 \\
          & 4.5   & 341   & 145   & 55 \\
    \midrule
    \multicolumn{1}{c}{\multirow{8}[2]{*}{Dynamic change rate}} & 0     & 197   & 75    & 41 \\
          & 2.5   & 203   & 77    & 44 \\
          & 5     & 208   & 82    & 40 \\
          & 7.5   & 211   & 89    & 39 \\
          & 10    & 218   & 90    & 49 \\
          & 12.5  & 220   & 93    & 47 \\
          & 15    & 232   & 97    & 45 \\
          & 17.5  & 235   & 102   & 49 \\
    \bottomrule
    \end{tabular}%
  \label{tab:sensitivity}%
\end{table}%


%!TEX program = xelatex
\documentclass[50pt]{article}
\usepackage[a4paper, left=1in, right=1in, top=1in, bottom=1in]{geometry}
\usepackage{xeCJK} %调用 xeCJK 宏包
\usepackage{xcolor}
\usepackage{listings}
\usepackage{amsmath}
\usepackage{color}
\usepackage{geometry}
\usepackage{graphicx}
\usepackage{tikz}
\usepackage{verbatimbox}
\allowdisplaybreaks
\usetikzlibrary{trees}

\title{Team Project Part 1}
\date{\today}
\begin{document}

\maketitle

\clearpage
\tableofcontents
\section{background knowledge}
\subsection{Decision Tree}
Decision tree has advantages:
\begin{itemize}
    \item Simple Idea: IF\dots THEN\dots
    \item It can deal with high dimension data and winnow importan variables.
    \item The results are easy to understand.
    \item Quick calculation
    \item Ideal correctness
\end{itemize}
CART decision tree is called Classification and Regression tree. When the dataset is of
continuous type, the tree can be a Regression Tree. We can predict the value
by the expected value of leaf nodes. When dataset is of discrete type, we can regard it
as a Classification Tree. The tree \textbf{is a binary tree}. One feature can
be used many times. Every non-leaf node can only extend to two children.

\subsection{Entropy}
    Definition: the degree of disorder or randomness in the system.\\
    Suppose X is a discrete random variable, the pmf:
    $$P(X = X_i) = p_i, i = 1, 2, \dots, n$$
    then the entropy of RV X is:
    $$H(X) = -\sum_{i=1}^{n}p_{i}log_{2}{p_i}$$
    The more the entropy is, the unceritainty the RV is.
\subsection{Conditional Entropy}
    In the given condition of X, the conditional entropy of RV Y $H(Y|X)$ is defined as:
    $$H(Y|X) = \sum_{i=1}^{n}p_{i}H(Y|X=X_{i})$$
    In the equation, $p_{i} = P(X=X_{i})$
\subsection{Information Gain}
    Definition: Information gain is the reduction in entropy or surprise by transforming
    a dataset and is often used in training decision trees. Information gain is calculated
    by comparing the entropy of the dataset before and after a transformation.\\
    The information gain that the feature A contribuites to dataset D is called
    $$g(D, A) = H(D) - H(D|A)$$
    For the dataset D, we need to calculate the information regarding to each feature and
    each feature value, and choose the largest one, which is the best.\\
    Suppose a training dataset D, the capacity is |D|, has $k$ categories $C_k$, $|C_k|$
    is the sample number of $C_k$. Suppose one feature $A$ has n values $a_{1},a_{2},\dots, a{n}$
     We can divide D into n subsets $D_{1}, D_{2}, \dots, D_{n}$, $|D_{i}|$ is
     the sample number of $D_{i}$. We denote $D_{ik}$ as a subset of $D_{i}$ which belong to
     $C_{k}$, $|D_{ik}|$ is the sample number of $D_{ik}$. We then calculate the information
     gain as follows:\\
     \begin{enumerate}
         \item calculate $$H(D) = -\sum_{k=1}^{K}\frac{|C_{k}|}{|D|}log_{2}\frac{|C_{k}|}{|D|}$$.
         \item calculate the conditional entropy of feature A contributing to D
         $$H(D|A) = \sum_{i=1}^{n}\frac{|D_{i}}{|D|}\sum_{k=1}^{K}\frac{|D_{ik}|}{|D{i}|}log_{2}\frac{|D_{ik}|}{|D_{i}|}$$
         \item calculate information gain $$g(D,A) = H(D) - H(D|A)$$
     \end{enumerate}
\subsection{Information Gain Ratio}
     Sometimes we may choose improperly a feature that has too much values. Such situation makes no sense.
     We must correct it using information gain ratio.
     $$g_{R}(D,A) = \frac{g(D,A)}{H_{A}(D)}$$
     $$H_{A}(D) = -\sum_{i=1}^{n}\frac{|D_{i}|}{|D|} log_{2}\frac{|D_{i}|}{|D|}$$
\subsection{Gini Index}
    $$Gini(D) = 1 - \sum_{i=1}^{m}p^{2}_{i}$$
    In the equation, $p_i$ is the probability of class $C_i$ in $D$\\
    For a discrete variable, we need to calculate the weight sum of each zone's impurity, As the following:
    $$Gini_{A}(D) = \frac{|D_1|}{|D|}Gini(D_1) + \frac{|D_2|}{|D|}Gini(D_2)$$
    For a continuous variable, we can set a dividing point to get the same goal.\\
    Our goal is to make the weight sum as small as possible by choose the best feature and
    the best feature value.
\section{Get Access to Data}
     Use open() function to extract data from csv files.
     \begin{lstlisting}[language={python}]
        def get_data(file_name):
            f = open(file_name,'r').readlines()
            data_name = f[0].split(',')[:-1]
            data = []
            for i in f[1:]:
                d = [float(j) for j in i.split(',')]
                d[-1] = d[-1] > 6 # Transform the quality figures into True or False
                data.append(d)
            return data_name, data
     \end{lstlisting}
\section{Build a Decition Tree}
\begin{itemize}
    \item Starting from root node, calculate the possible \textbf{information gain/information gain
    ratio/gini index} regarding each feature
    and value. Choose the best information gain/ratio/gini index. Construct different child nodes according to
    the feature and value.
    \item Use recursion to the child node and build the tree.
    \item Until all the labels are the same after selection.
\end{itemize}
\subsection{Feature Choice}
The most popular methods are:
\begin{itemize}
    \item ID3: Depend on information gain
    \item CD4.5: Depend on information gain ratio
    \item CART: Depend on Gini Index when it is Classification Tree, on MSE when it is Regression Tree.
\end{itemize}
    Here we use CART to construct the classification tree.
\section{Prune the Tree}
We can do pruning to optimize the tree size and reduce overfitting. Generally, there are two kinds of pruning: Pre-pruning and Post-pruning.
\subsection{Pre-pruning}
\begin{itemize}
    \item Pre-pruning is also called early stoping.
    \item Pre-pruning stops the tree before it grows perfectly.
    \item People can set maximum depth of the tree to restrict the tree.
\end{itemize}
\subsection{Post-pruning}
\subsubsection{REP(Reduced-Error Pruning)}
REP is one of the simplest forms of Post-pruning.\\
First, calculate the error of a subtree $E_{r}(t)$.
Then, calculate the error of each leaf node of this subtree $E_{t}(Tt)$.
If $$\ E_{r}(t) < \sum E_{r}(T_{t})$$
Then, replace the subtree with a leaf node, whose label is determined by the majority label of the subtree. Repeat this process from the bottom of the tree to the top.
\subsubsection{PEP(Pessimistic-Error Pruning)}
For a leaf node with N samples and E errors, its error rate $e$ is $\frac{E+0.5}{N}$. "0.5" is called penalty factor. For a subtree with L leaf nodes, its error rate is
$$ \ p = \frac{\sum_{i=1}^{L}\ E_{i}+0.5L}{\sum_{i=1}^{L}\ N_{i}}$$
Suppose that all the samples in a subtree is of binomial distribution $B(N,P)$, then the expectation and standard deviation of error before pruning are:
$$\ E_{T} = N*p = N*\frac{\sum_{i=1}^{L}\ E_{i}+0.5L}{\sum_{i=1}^{L}\ N_{i}} = \sum_{i=1}^{L}\ E_{i}+0.5L $$
$$\sigma = \sqrt{N*p*(1-p)}$$
Expectation of error after pruning is:
$$\ E_{t} = N*e = N*\frac{E+0.5}{N} = E+0.5$$
If $$E_{t} - E_{T} < \sigma$$
then prune the subtree. Repeat this process from the top of the tree to the bottom.
\subsubsection{CCP(Cost-Complexity Pruning)}
$\alpha$ is a real number called the complexity parameter, it's defined as follows:
$$\alpha = \frac{R(t)- R(T_{t})}{|T_{t}|-1},$$ where $R(t)$ is the error after pruning, $R(T_{t})$ is the error of the original subtree and $|T_{t}|$ is the number of samples in the subtree.\\
Step1: Calculate the value of $\alpha$ for all the subtrees from the bottom to the top, each time prune the subtree with minimal $\alpha$. Get a set $\{T_{0}, T_{1},..., T_{M}\}$, where $T_{0}$ is a complete tree and $T_{M}$ is a root node.\\
Step2: Pick the best tree from $\{T_{0}, T_{1},..., T_{M}\}$, according to its performance on the testing sets.\\
*Note that if we use CCP to prune the tree, we should separate a testing set from the given training set before training begins.
\section{Code structure}
\tikzstyle{every node}=[draw=black,thick,anchor=west]
	\tikzstyle{selected}=[draw=red,fill=red!30]
	\tikzstyle{optional}=[dashed,fill=gray!50]
	\begin{tikzpicture}[
	grow via three points={one child at (0.5,-0.7) and two children at (0.5,-0.7) and (0.5,-1.4)},
	edge from parent path={(\tikzparentnode.south) |- (\tikzchildnode.west)}]
    \node {main.py}
    child { node {data.py}
        child {node {def get-data()}}
    }
    child [missing] {}
	child { node [selected] {ccp-prune.py}
		child { node {class Tree}
			child { node {class Node}}
		}
		child [missing] {}
		child { node {def process()}
		}
    }
    child [missing] {}
	child [missing] {}
	child [missing] {}
	child { node [selected] {pep-prune.py}};
    \end{tikzpicture}
\section{Result}
We can attain 89.7916666667 percent accuracy
\makeatletter
\setlength{@fptop}{5pt}
\subsection{The Confusion Matrix}

\begin{table}[]
    \begin{tabular}{lll}
     & True Bad Wine & True Good Wine \\
     Predicted Bad Wine & 395 & 28 \\
     Predicted Good Wine & 21 & 36 \\
     &  &
    \end{tabular}
\end{table}

\end{document}


\chapter{Initial implementation}

\lstinputlisting{code/MLDU_Simple.m}

\noindent The block MLDU algorithm was implemented using MATLAB. It represents a typical first attempt without many optimizations. It uses pre-allocation for the output matrices and a nested function for the splitting of the matrix.

\section{Verification and performance analysis}

A simple test was required to verify the correctness of the implementation of the algorithm and analyze its performance. The following code was used:

\lstinputlisting{code/Test_Function_1.m}
\lstinputlisting{code/Matrix_A.m}

\vspace{5mm}

\newpage
\noindent As an example, matrix A and s are provided below ($n = 4$):

$$
A =
\left\lbrack\begin{array}{rrrr|rrrr|rrrr|rrrr}
           4&          -1&           0&           0&          -1&           0&           0&           0&           0&           0&           0&           0&           0&           0&           0&           0\\
          -1&           4&          -1&           0&           0&          -1&           0&           0&           0&           0&           0&           0&           0&           0&           0&           0\\
           0&          -1&           4&          -1&           0&           0&          -1&           0&           0&           0&           0&           0&           0&           0&           0&           0\\
           0&           0&          -1&           4&           0&           0&           0&          -1&           0&           0&           0&           0&           0&           0&           0&           0\\ \hline
          -1&           0&           0&           0&           4&          -1&           0&           0&          -1&           0&           0&           0&           0&           0&           0&           0\\
           0&          -1&           0&           0&          -1&           4&          -1&           0&           0&          -1&           0&           0&           0&           0&           0&           0\\
           0&           0&          -1&           0&           0&          -1&           4&          -1&           0&           0&          -1&           0&           0&           0&           0&           0\\
           0&           0&           0&          -1&           0&           0&          -1&           4&           0&           0&           0&          -1&           0&           0&           0&           0\\ \hline
           0&           0&           0&           0&          -1&           0&           0&           0&           4&          -1&           0&           0&          -1&           0&           0&           0\\
           0&           0&           0&           0&           0&          -1&           0&           0&          -1&           4&          -1&           0&           0&          -1&           0&           0\\
           0&           0&           0&           0&           0&           0&          -1&           0&           0&          -1&           4&          -1&           0&           0&          -1&           0\\
           0&           0&           0&           0&           0&           0&           0&          -1&           0&           0&          -1&           4&           0&           0&           0&          -1\\  \hline
           0&           0&           0&           0&           0&           0&           0&           0&          -1&           0&           0&           0&           4&          -1&           0&           0\\
           0&           0&           0&           0&           0&           0&           0&           0&           0&          -1&           0&           0&          -1&           4&          -1&           0\\
           0&           0&           0&           0&           0&           0&           0&           0&           0&           0&          -1&           0&           0&          -1&           4&          -1\\
           0&           0&           0&           0&           0&           0&           0&           0&           0&           0&           0&          -1&           0&           0&          -1&           4\\
\end{array}\right\rbrack
$$

$$
s =
\left\lbrack\begin{array}{rrrrrrrr}
           2&           2&           2&           2&           2&           2&           2&           2\\
\end{array}\right\rbrack
$$

\noindent Running the test with $n = 100$ creates a $10.000 \times 10.000$ matrix. Calculating the Frobenius norm of the error results in:

\vspace{5mm}

\noindent \texttt{>> [ E ] = Test\_Function\_1( 100, @MLDU\_Simple )}\\
\\
\texttt{E =}\\
\\
\texttt{   1.1409e-13}

\vspace{5mm}

\noindent The resulting error is small enough to be confident in the correct operation of the implementation, at least for this test case.

\vspace{5mm}

\noindent Running the profiler with the same input results in the following table:

\begin{figure}[h!]
    \includegraphics[width=\linewidth]{figures/Profile_MLDU_Simple_1.eps}
    \centering
\end{figure}

\noindent The first thing to note is that the runtime is about $30$ seconds for this test case, of which roughly $20$ seconds are spend by the \texttt{MLDU\_Simple} function. A $10.000 \times 10.000$ matrix is not really small, but it's not large either. As such, $20$ seconds to provide the factorization is a rather poor result.

\newpage

\noindent To get a better understanding of what is going on, the following profiler information was pulled up:

\begin{figure}[h!]
    \includegraphics[width=0.95\linewidth]{figures/Profile_MLDU_Simple_2.eps}
    \centering
\end{figure}

\noindent About 12 seconds of the runtime are spent calculating the Frobenius norm of the error, which can clearly be omitted from the performance analysis. Zooming in on the profiler information of the \texttt{MLDU\_Simple.m} function yields:

\begin{figure}[b!]
    \includegraphics[width=0.95\linewidth]{figures/Profile_MLDU_Simple_3.eps}
    \centering
    \label{Piramid}
\end{figure}

\newpage

\section{Discussion of the results}

\noindent The results mostly speak for themselves. A huge part of the runtime of the implementation is spent on line $27$, where the results of the Schur complement update gets subtracted from the matrix. Even though there are some calculations involved in this line, they could not account for this excessive runtime.\\

\noindent Calculating the Schur complement happens at lines $23$, $24$ and $25$, which takes about $1.5$ seconds. Most of that time is not actually spent doing any calculations, but rather accessing the elements of matrix \texttt{U}. This is to be expected, as the sparse storage format of MATLAB is catered towards accessing columns rather than rows. Further proof of this can be found at line $21$, where separating the elements of \texttt{U} from \texttt{A} actually takes more time than calculating the Schur complement update.\\

\noindent The point is that most of the runtime of line $27$ will most likely be spent on the memory operations associated with inserting columns and rows of data into an existing sparse matrix. There are a couple of obvious possibilities that might be tried to circumvent this problem:

\begin{itemize}
    \item Switch from sparse to full matrix storage.
    \item Store rows in a transposed fashion.
    \item Use a matrix storage format that doesn't require as many memory operations to insert rows and columns.
\end{itemize}

\noindent The first possibility may seem silly, but is not as stupid as may at first appear. Fill-in is a real nemesis of algorithms like Block MLDU, and can make a sparse matrix very full in some instances. Switching to full matrix storage in these situations makes perfect sense and will provide a very healthy performance improvement.\\

\noindent Storing the rows as transposed columns can have tangible benefits, but it has certain disadvantages as well. The most obvious downside is the extra memory consumption, as the matrix will probably need to be stored twice. The second problem is that the block nature of the algorithm makes a lot of the work required to adapt the code that much harder.\\
\noindent It should be noted that this possible improvement should definitely be investigated further, as it could be a huge advantage in situations where the input matrix is symmetric.\\

\noindent Switching to a different matrix storage format will probably provide the most benefits, especially if the storage format is designed from scratch and tailored to the needs of the Block MLDU algorithm. The second implementation of the algorithm will focus on this possibility.


\section{GEOP}
\label{GEOP}
\begin{ipifield}{}%
{A Geometry Optimization class implementing most of the standard methods}%
{}%
{\ipiitem{mode}%
{The geometry optimization algorithm to be used}%
{default: `lbfgs'; data type: string; options: `sd', `cg', `bfgs', `bfgstrm', `lbfgs'; }%
}
\begin{ipifield}{biggest\_step}%
{The maximum step size for (L)-BFGS line minimizations.}%
{default:  100.0 ; data type: float; }%
{}
\end{ipifield}
\begin{ipifield}{scale\_lbfgs}%
{Scale choice for the initial hessian.
                                            0 identity.
                                            1 Use first member of position/gradient list.
                                            2 Use last  member of position/gradient list.}%
{default:  2 ; data type: integer; }%
{}
\end{ipifield}
\begin{ipifield}{corrections\_lbfgs}%
{The number of past vectors to store for L-BFGS.}%
{default:  6 ; data type: integer; }%
{}
\end{ipifield}
\begin{ipifield}{tolerances}%
{Convergence criteria for optimization. Default values are extremely conservative. Set them to appropriate values for production runs.}%
{}%
{}
\end{ipifield}
\begin{ipifield}{glist\_lbfgs}%
{List of previous gradient differences for L-BFGS, if known.}%
{default:  [ ] ; data type: float; }%
{\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{old\_direction}%
{The previous direction in a CG or SD optimization.}%
{default:  [ ] ; data type: float; }%
{\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{old\_force}%
{The previous force in an optimization step.}%
{dimension: force; default:  [ ] ; data type: float; }%
{\ipiitem{units}%
{The units the input data is given in.}%
{default: `automatic'; data type: string; }%
\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{invhessian\_bfgs}%
{Approximate inverse Hessian for BFGS, if known.}%
{default:  [ ] ; data type: float; }%
{\ipiitem{shape}%
{The shape of the array.}%
{default:  (0, 0) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{tr\_trm}%
{The trust radius in trm.}%
{dimension: length; default:  [ ] ; data type: float; }%
{\ipiitem{units}%
{The units the input data is given in.}%
{default: `automatic'; data type: string; }%
\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{hessian\_trm}%
{Approximate Hessian for trm, if known.}%
{default:  [ ] ; data type: float; }%
{\ipiitem{shape}%
{The shape of the array.}%
{default:  (0, 0) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{ls\_options}%
{"Options for line search methods. Includes:
                              tolerance: stopping tolerance for the search (as a fraction of the overall energy tolerance),
                              iter: the maximum number of iterations,
                              step: initial step for bracketing,
                              adaptive: whether to update initial step.
                              }%
{}%
{}
\end{ipifield}
\begin{ipifield}{old\_pos}%
{The previous positions in an optimization step.}%
{dimension: length; default:  [ ] ; data type: float; }%
{\ipiitem{units}%
{The units the input data is given in.}%
{default: `automatic'; data type: string; }%
\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{old\_pot}%
{The previous potential energy in an optimization step.}%
{dimension: energy; default:  [ ] ; data type: float; }%
{\ipiitem{units}%
{The units the input data is given in.}%
{default: `automatic'; data type: string; }%
\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\begin{ipifield}{qlist\_lbfgs}%
{List of previous position differences for L-BFGS, if known.}%
{default:  [ ] ; data type: float; }%
{\ipiitem{shape}%
{The shape of the array.}%
{default:  (0,) ; data type: tuple; }%
\ipiitem{mode}%
{If 'mode' is 'manual', then the array is read from the content of 'cell' takes a 9-elements vector containing the cell matrix (row-major). If 'mode' is 'abcABC', then 'cell' takes an array of 6 floats, the first three being the length of the sides of the system parallelopiped, and the last three being the angles (in degrees) between those sides. Angle A corresponds to the angle between sides b and c, and so on for B and C. If mode is 'abc', then this is the same as for 'abcABC', but the cell is assumed to be orthorhombic. 'pdb' and 'chk' read the cell from a PDB or a checkpoint file, respectively.}%
{default: `manual'; data type: string; options: `manual', `file'; }%
}
\end{ipifield}
\end{ipifield}