\documentclass[paper=a4, fontsize=11pt]{scrartcl} % A4 paper and 11pt font size
\usepackage{physics}
\usepackage[T1]{fontenc} % Use 8-bit encoding that has 256 glyphs
\usepackage{fourier} % Use the Adobe Utopia font for the document - comment this line to return to the LaTeX default
\usepackage[english]{babel} % English language/hyphenation
\usepackage{amsmath,amsfonts,amsthm} % Math packages
\usepackage{braket}
\usepackage{lipsum} % Used for inserting dummy 'Lorem ipsum' text into the template
\usepackage{tikz}
\usepackage{amsmath}
\usepackage{sectsty} % Allows customizing section commands
\allsectionsfont{\centering \normalfont\scshape} % Make all sections centered, the default font and small caps

\usepackage{bm}
\newcommand{\uvec}[1]{\boldsymbol{\hat{\textbf{#1}}}}
\usepackage[thinlines]{easytable}
\usepackage{fancyhdr} % Custom headers and footers
\pagestyle{fancyplain} % Makes all pages in the document conform to the custom headers and footers
\fancyhead{} % No page header - if you want one, create it in the same way as the footers below
\fancyfoot[L]{} % Empty left footer
\fancyfoot[C]{} % Empty center footer
\fancyfoot[R]{\thepage} % Page numbering for right footer
\renewcommand{\headrulewidth}{0pt} % Remove header underlines
\renewcommand{\footrulewidth}{0pt} % Remove footer underlines
\setlength{\headheight}{13.6pt} % Customize the height of the header
\usepackage{float}
\numberwithin{equation}{section} % Number equations within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{figure}{section} % Number figures within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{table}{section} % Number tables within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)

\setlength\parindent{0pt} % Removes all indentation from paragraphs - comment this line for an assignment with lots of text

%----------------------------------------------------------------------------------------
%	TITLE SECTION
%----------------------------------------------------------------------------------------

\newcommand{\horrule}[1]{\rule{\linewidth}{#1}} % Create horizontal rule command with 1 argument of height

\title{
\normalfont \normalsize
\textsc{California State University San Marcos \\ Dr. De Leone, Physics 323} \\ [25pt] % Your university, school and/or department name(s)
\horrule{0.5pt} \\[0.4cm] % Thin top horizontal rule
\huge H.W. II \\ % The assignment title
\horrule{2pt} \\[0.5cm] % Thick bottom horizontal rule
}

\author{Josh Lucas} % Your name

\date{\normalsize\today} % Today's date or a custom date

\begin{document}

\maketitle % Print the title

%----------------------------------------------------------------------------------------
%	PROBLEM 1
%----------------------------------------------------------------------------------------

\section{SPINS Lab 1}
This lab entailed preparing particles in a particular state, $\ket{\pm}_n$, and passing them through a second analyzer in the orthogonal directions noting the output on the table.
\renewcommand{\arraystretch}{2}
\begin{center}
  \begin{tabular}{ | c | c | c | c | c | c | c | }
    \hline
    $|\bra{out}\ket{in}|^2$ & $\ket{+}$& $\ket{-}$ & $\ket{+}_x$ & $\ket{-}_x$ & $\ket{+}_y$ & $\ket{-}_y$ \\ \hline
  $  \bra{+}$ & 1 &0   & $\tfrac{1}{2}$ & $\tfrac{1}{2}$  & $\tfrac{1}{2}$ & $\tfrac{1}{2}$   \\ \hline
    $\bra{-}$ & 0 & 1 & $\tfrac{1}{2}$ &$\tfrac{1}{2}$  & $\tfrac{1}{2}$ &  $\tfrac{1}{2}$ \\
    \hline
       $_x\bra{+}$ & $\tfrac{1}{2}$ & $\tfrac{1}{2}$ & 1 & 0 & $\tfrac{1}{2}$ & $\tfrac{1}{2}$  \\
    \hline
       $_x\bra{-}$ & $\tfrac{1}{2}$ & $\tfrac{1}{2}$ & 0 & 1 &$\tfrac{1}{2}$  & $\tfrac{1}{2}$  \\
    \hline
       $_y\bra{+}$ & $\tfrac{1}{2}$ &$\tfrac{1}{2}$  &$\tfrac{1}{2}$  &$\tfrac{1}{2}$  & 1 &  0 \\
    \hline
       $_y\bra{-}$ &$\tfrac{1}{2}$  &$\tfrac{1}{2}$  & $\tfrac{1}{2}$ & $\tfrac{1}{2}$ & 0 & 1  \\
    \hline
  \end{tabular}
\end{center}
We can see the symmetry in the results of the table, if the in's and out's were reversed the results would be the same.
%------------------------------------------------
\section{Ch.1 Definitions}
\textbf{Orbital Angular Momentum L} $\bullet$ The electron, from a classical viewpoint has angular momentum around a radius. $L = mvr$ This momentum of a charged particle creates current loops which induce a magnetic moment.\\
\\
 \textbf{Intrinsic Angular Momentum, S, Spin} $\bullet$ Similar to a satellite in orbit rotating as it revolves an electron classically has a rotation called spin. The spin also produces current loops.\\
\\
 \textbf{Gyroscopic Ratio} $\bullet$ A dimensionless scalar constant that relates spin of a particle with the magnetic moment $\mu$ as in $\vec{\mu} = g\frac{q}{2m}\vec{S}$.\\
\\
\textbf{Quantization} $\bullet$ The act of organizing information into discrete sets.\\
\\
\textbf{Spin $\frac{1}{2}$ system} $\bullet$ The Stern-Gerlach observations showed that spins of the particles were orientated in either a up or down configuration, $\vec{S_z}= \pm \frac{\hbar}{2} $\\
\\
 \textbf{Observable} $\bullet$ The physical quantity that is measured.\\
 \\
 \textbf{Analyzer} $\bullet$ A device which can distinguish the spin configuration of particles.\\
 \\
 \textbf{Ket} $\bullet$ Symbolic representation of the quantum state.\\
 \\
 \textbf{Bra} $\bullet$ Symbolic representation of the complex conjugate of quantum state.\\
 \\
 \textbf{State Preparation Device} $\bullet$ When a device such as Stern-Gerlach device separates the up and down spins of particles they are said to be prepared in that state. \\
 \\
 \textbf{Local Hidden Variable Theory} $\bullet$ The theory that there could be another yet undiscovered fact or trait about the system that would account for the observations.\\
 \\
 \textbf{Incompatible Observables} $\bullet$  Observables that cannot be know simultaneously as in knowing orthogonal spin components of a particle.\\
 \\
 \textbf{Quantum State Vectors} $\bullet$ The mathematical representation of the quantum state of a system containing both the real and complex information. \\
 \\
\textbf{Hilbert Space} $\bullet$ Vector space defined for a system that limits space to the necessary dimensions.\\
 \\
 \textbf{Dot Product, inner product, scalar product} $\bullet$ The magnitude of two vectors multiplied by the cosine of the angle between them. It is the magnitude of the projection of one vector on to another. \\
 \\
 \textbf{$S_z$ basis} $\bullet$ The possible contents of the set.\\
 \\
 \textbf{Orthonormality} $\bullet$ The combination of normalizing and making orthogonal a vector or state.\\
 \textbf{Basis Vectors :}\\
 \textbf{Normalized} $\bullet$ The spatial vector is normalized if it is a unit vector with magnitude of one.
 \begin{equation*}
  \hat{i}\cdot\hat{i}=\hat{j}\cdot\hat{j}=\hat{k}\cdot\hat{k} = 1
 \end{equation*}
 \textbf{Orthogonal} $\bullet$ The spatial vectors are orthogonal if their directions are perpendicular to each other.
 \begin{equation*}
 \hat{i}\cdot\hat{j}=\hat{i}\cdot\hat{k}=\hat{j}\cdot\hat{k} = 0
 \end{equation*}
 \textbf{Completeness} $\bullet$ The spatial vectors are complete if any vector to some point in the space can be written as linear superposition of the unit vectors, scaling them accordingly and summing directions.
 \begin{equation*}
 \vec{A} = a_x\hat{i} = a_y\hat{j} = a_z\hat{k}
 \end{equation*}
\textbf{Normalization Constant} $\bullet$  The complex constant C\\
 \\
 \textbf{Probability} $\bullet$ Mathematical possibility of an event occurring, the chance that the quantum state of the ket is measured to be in the corresponding basis state.\\
 \\
 \textbf{Probability Amplitude} $\bullet$ The inner product of input and output states. \\
 \\
 \textbf{Superposition State, Coherent Superposition} $\bullet$ The combination of general spin 1/2 vector containing both up and down kets\\
 \\
 \textbf{Mixed State} $\bullet$ A state created from a mixture of up and down spin atoms \\
\\
 \textbf{Column-Row Vectors} $\bullet$ Matrices for indexing information, framework for linear algebra or computational calculations. \\
 \\
 \textbf{Representation} $\bullet$ Collection of coefficients that multiply the basis set\\
 \\
 \textbf{Kronecker Delta} $\bullet$ Binary logical operator, True if equal, false if not.  \\
 \\

%----------------------------------------------------------------------------------------

\end{document}

H.W. 2 Due Wednesday 9-5-2018/H.W. 2.tex


\section{Background}

%\subsection{Supervised learning and gradient tree boosting}

\begin{frame}{Question 1: Linear regression}

	\begin{columns}[T]
		\begin{column}{0.35\textwidth}
			%\begin{figure}
				\includegraphics<1->[height=2.9cm,width=3.5cm]{figures/growth_girls_lm.pdf}
				\includegraphics<1->[height=2.9cm,width=3.5cm]{figures/growth_boys_lm.pdf}
			%\end{figure}
		\end{column}
		\begin{column}{0.6\textwidth}
			\begin{myblock}{Researcher asks...}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
				How can I model the \textcolor{orange}{\texttt{height}} of children given their \textcolor{orange}{\texttt{age}} and \textcolor{orange}{\texttt{sex}}?
				And I need a model fast! [Berkeley growth curve dataset]
			\end{myblock}
		\visible<2>{
			\begin{myblock}{The statistician responds...}{bg=yellow!05,fg=black}{bg=yellow!20, fg=black}
				Easy! Just try a linear regression:
				$\textcolor{orange}{\texttt{height}}\approx \beta_0 + \beta_1
				\textcolor{orange}{\texttt{age}} + \beta_2 \textcolor{orange}{\texttt{sex}}$.
				Estimate parameters $\mathbf{\beta} = \{\beta_0, \beta_1, \beta_2\}$ by minimizing the mean squared error (MSE):\\ $\hat{\mathbf{\beta}} = \arg\min_\mathbf{\beta} \sum_i \left(y_i - f(\textcolor{orange}{\texttt{age}}_i,\textcolor{orange}{\texttt{sex}}_i;\mathbf{\beta})\right)^2$.
			\end{myblock}}
		\end{column}
	\end{columns}

\end{frame}

\begin{frame}{Question 2: Generalized linear models}

	\begin{columns}[T]
	\begin{column}{0.35\textwidth}
		%\begin{figure}
		\includegraphics<1->[height=5.5cm,width=4cm]{figures/claims_glm.pdf}
		%\includegraphics<1->[height=2.9cm,width=3.5cm]{figures/growth_boys_lm.pdf}
		%\end{figure}
	\end{column}
	\begin{column}{0.6\textwidth}
		\begin{myblock}{Researcher asks...}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
			Is there an efficient way to model the \textcolor{orange}{\texttt{risk}} of customers of insurance given some history of \textcolor{orange}{\texttt{claims}} and information about the customers?
			The model needs to be production friendly!
		\end{myblock}
		\visible<2>{
			\begin{myblock}{The actuary responds...}{bg=yellow!05,fg=black}{bg=yellow!20, fg=black}
				Easy! Divide and conquer: split the \textcolor{orange}{\texttt{claims}} into \textcolor{orange}{\texttt{size}} and \textcolor{orange}{\texttt{frequency}} and model them using a gamma and a Poisson generalized linear model, respectively. The \texttt{glm()}-function in R is your friend.
		\end{myblock}}
	\end{column}
\end{columns}

\end{frame}

\begin{frame}{Supervised learning}

	\begin{itemize}
		\item The above problems may be framed as \textcolor{orange}{supervised learning}:
	\end{itemize}

	\begin{block}{Supervised learning}
		Find the best (in expectation, relative to loss $l$) predictive function:
		\begin{align*}
		\hat{f} = \arg\min_f E_{\features\response}\left[l(\response,f(\features))\right]
		\end{align*}
		The loss often correspond to a nll.
\end{block}

\visible<2->{
\begin{myblock}{User restricted $f$, is it...}{bg=yellow!05,fg=black}{bg=yellow!20, fg=black}
	\begin{itemize}
		\item Non-linear?
		\item Continuous?
		\item Which features should it use?
		\item Do we have enough data to parametrize $f$?
	\end{itemize}}
\end{myblock}

%	\begin{columns}[T]
%		\begin{column}{0.52\textwidth}
%			\begin{block}{Maximum likelihood}
%				LR and GLM both learn their parameters from data:%, by maximizing the likelihood of observing the data, given that the data was generated by the model.
%				\begin{align*}
%					\hat{\beta} = \arg\max_\beta p(\texttt{data}|\beta)
%				\end{align*}
%				\visible<2->{
%				Equivalent to minimizing the nll
%				\begin{align*}
%					\hat{\beta} = \arg\min_\beta \left\{- \sum_i \log (p(\texttt{obs}_i|\beta))\right\}
%				\end{align*}
%				assuming independent observations
%				\begin{itemize}
%					\item Minimizing MSE correspond to minimizing a Gaussian nll.
%				\end{itemize}}
%			\end{block}
%		\end{column}
%		\begin{column}{0.5\textwidth}
%			\visible<3->{
%			\begin{block}{Supervised learning}
%				Find the best (in expectation, relative to loss $l$) predictive function:
%				\begin{align*}
%					\hat{f} = \arg\min E_{\features\response}\left[l(\response,f(\features))\right]
%				\end{align*}
%
%				The loss often correspond to a nll.
%				\visible<4->{
%				User restricted $f$:
%				Is it
%				\begin{itemize}
%					\item Non-linear?
%					\item Continuous?
%					\item Which features should it use?
%					\item Do we have enough data to parametrize $f$?
%				\end{itemize}}
%			\end{block}}
%		\end{column}
%	\end{columns}

\end{frame}

\begin{frame}{Question 3: Gradient boosting}

	\begin{myblock}{Researcher asks...}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
		I want to do well in this ML-competition, but...
		\begin{itemize}
			\item<2-> My data has missing values
			\item<2-> Both $n$ and $p$ are very large (design matrix with some billion elements)
			\item<2-> Relationships are non-linear and possibly discontinuous
			\item<2-> I don't care about explainability, just give me predictive power!
		\end{itemize}
	\end{myblock}
	\visible<3->{\begin{myblock}{The data scientist/Kaggle master responds...}{bg=yellow!05,fg=black}{bg=yellow!20, fg=black}
		Try gradient boosting?
		\begin{itemize}\small
			\item<3-> State-of-the-art gradient boosting libraries: XGBoost, LightGBM and CatBoost.
		\end{itemize}
	\end{myblock}}

\end{frame}

\begin{frame}{Gradient boosting, what is going on behind the scenes?}

	Gradient boosting attacks the supervised learning problem directly

	%For many problems coined as supervised learning problems, gradient boosting

	\begin{myblock}{}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
		\begin{itemize}
			\item Start with a constant value: $f^{(0)} = \arg\min_\eta \sum_i l(y_i,\eta)$
			\item Iteratively, add $\delta f_k$ to $f^{(k-1)}$, where $f_k$ is trained on the "error" (MSE case) of $f^{(k-1)}$, and $\delta$ is some small number scaling $f_k$.
		\end{itemize}
	\end{myblock}

	\begin{columns}[T]
		\begin{column}{0.35\textwidth}
			% p0
			\includegraphics<1>[height=4.5cm,width=4.5cm]{figures/lm_boost_p.pdf}
			\includegraphics<2-4>[height=4.5cm,width=4.5cm]{figures/lm_boost_p_0.pdf}
					%p1
			\includegraphics<5-7>[height=4.5cm,width=4.5cm]{figures/lm_boost_p1.pdf}
			%p2
			\includegraphics<8-10>[height=4.5cm,width=4.5cm]{figures/lm_boost_p2.pdf}
			%p3
			\includegraphics<11-13>[height=4.5cm,width=4.5cm]{figures/lm_boost_p3.pdf}
			%p4
			\includegraphics<14>[height=4.5cm,width=4.5cm]{figures/lm_boost_p4.pdf}
		\end{column}
		\begin{column}{0.35\textwidth}
			%res1
			\includegraphics<3>[height=4.5cm,width=4.5cm]{figures/lm_boost_res1.pdf}
			\includegraphics<4-5>[height=4.5cm,width=4.5cm]{figures/lm_boost_resf1.pdf}
			%res2
			\includegraphics<6>[height=4.5cm,width=4.5cm]{figures/lm_boost_res2.pdf}
			\includegraphics<7-8>[height=4.5cm,width=4.5cm]{figures/lm_boost_resf2.pdf}
						%res3
			\includegraphics<9>[height=4.5cm,width=4.5cm]{figures/lm_boost_res3.pdf}
			\includegraphics<10-11>[height=4.5cm,width=4.5cm]{figures/lm_boost_resf3.pdf}
						%res4
			\includegraphics<12>[height=4.5cm,width=4.5cm]{figures/lm_boost_res4.pdf}
			\includegraphics<13-14>[height=4.5cm,width=4.5cm]{figures/lm_boost_resf4.pdf}
		\end{column}
	\end{columns}

\end{frame}

\begin{frame}{Why this iterative procedure is a good idea}

	\begin{myblock}{The procedure...}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
		\visible<2->{\begin{itemize}
			\item Adapts the complexity of the model, $f$, to the data,
			\item Only add as much complexity in a certain direction as it deserves
			\item Builds sparse models: Connection to the LARS algorithm for computing LASSO solution paths.
		\end{itemize}}
	\end{myblock}

	\visible<3->{
	\begin{myblock}{It can be generalized beyond MSE:}{bg=yellow!30, fg=black}{bg=yellow!50, fg=black}
	\begin{itemize}
		\item Given a differentiable loss function $l$
		\item Instead of building a model on the "errors" in the MSE case,
		\item Compute derivatives from $l(y_i,\hat{y}_i)$ over the data given predictions $\hat{y}_i$ from the current model.
		\item Build a model on the derivatives.
	\end{itemize}
	\end{myblock}}

\end{frame}

\begin{frame}{Trees: where boosting gets interesting}

	We used a linear model for "base learners" $f_k$:
	\begin{itemize}
		\item The linear combination of linear functions is still a linear model...
	\end{itemize}
	\visible<2->{
	\begin{myblock}{}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
	\begin{itemize}
			\item<2-> More interesting with non-linear learning procedures for $f_k$ %(adapting to error / derivatives)...
			\item<2-> But needs to retain the possibility of a simple (sparse) model.
			\item<2-> We need something that can be non-linear but adapts this to data!
			\item<3-> Trees: complexity from the simple mean or "tree-stumps" to potentially a complete fit to training data.
	\end{itemize}
	\end{myblock}}

	%important with splitting algorithm
\end{frame}

\begin{frame}{The tree-learning prcedure: recursive binary splitting}

	Trees are constant predictions in $T$ regions, $R_t$, of feature space:
	\begin{columns}[T]
		\begin{column}{0.6\textwidth}
			\begin{align*}
			\pred = \sum_{t=1}^{T} w_t I(\features \in R_t)
			\end{align*}
			But how do we choose the regions $R_t$?
		\end{column}
		\begin{column}{0.39\textwidth}
			\begin{tikzpicture}
			\draw (0,0)coordinate(A) -- node[below] {$X_1$}  ++ (3,0)coordinate(B) -- (3,3)coordinate(C) -- (0,3)coordinate(D) -- node[left]{$X_2$} cycle;

			% initial figure
			\draw[name path=ef] (0,1)coordinate(E) -- (3,1)coordinate(F);
			\draw[name path=gh] (1,1)coordinate(G) -- (1,3)coordinate(H);
			\draw (1,2)coordinate(I) -- (3,2)coordinate(J);

			\fill[blue!50](A)--(B)--(F)--(E)--cycle;
			\fill[orange](E)--(G)--(H)--(D)--cycle;
			\fill[red](G)--(F)--(J)--(I)--cycle;
			\fill[green] (I)--(J)--(C)--(H)--cycle;

			\node[fill=blue!50,circle,text=black] at (barycentric cs:A=1,B=1,F=1 ,E=1) {$R_1$};
			\node[text=black] at (barycentric cs:E=1,G=1,H=1 ,D=1) {$R_2$};
			\node[text=black] at (barycentric cs:G=1,F=1,J=1 ,I=1) {$R_3$};
			\node[text=black] at (barycentric cs:I=1,J=1,C=1 ,H=1) {$R_4$};

			%split points
			\node[right=0.2cm of F,visible on=<3->]{$s_2^{(1)}$};
			\node[above=0.2cm of H,visible on=<4->]{$s_1^{(2)}$};
			\node[right=0.2cm of J,visible on=<5->]{$s_2^{(3)}$};

			%\draw (0, 0) -- node[below]{$X_1$} -- (0, 3)coordinate(B)node[above left]{B} -- (3, 3)coordinate(C)node[above right]{C} -- (3, 0)coordinate(D)node[below right]{D} --cycle;
			%\draw[name path=ae] (B) -- (2, 1)coordinate(E)node[right]{E};
			%\draw[name path=ac] (A) -- (C);
			%\path[name intersections ={of= ae and ac,by=I}];
			%\fill[green](A)--(B)--(I)--cycle;
			%\fill[blue](B)--(C)--(I)--cycle;
			%\fill[red](C)--(E)--(I)--cycle;
			%\fill[violet](A)--(D)--(E)--(I)--cycle;
			\end{tikzpicture}
		\end{column}
	\end{columns}

			\visible<2->{\begin{myblock}{Recursive binary splitting}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
				\begin{enumerate}
					\item Start with a constant prediction for all of feature space
					\item Split a leaf-node into two regions (on feature $j$ and split-point $s_j$ chosen by some criteria).
					\item Continue step 2 recursively on all leaves.
				\end{enumerate}
			\end{myblock}
		}

\end{frame}

\begin{frame}{Second order gradient tree boosting}

	\begin{myblock}{}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
	Iteratively add $\delta f_k$ where $f_k$ are trees trained on derivatives
	\begin{align*}
	g_{ik} =  \left.\frac{\partial}{\partial \pred_i} \loss(\response_i, \pred_i)\right|_{\substack{\pred_i = f^{(k-1)}(\features_i)}}
	\text{ and }
	h_{ik} = \left.\frac{\partial^2}{\partial \pred_i^2} \loss(\response_i, \pred_i)\right|_{\substack{\pred_i = f^{(k-1)}(\features_i)}}.
	\end{align*}
	\end{myblock}

		\begin{columns}[T]
		\begin{column}{0.35\textwidth}
			% p0
			\includegraphics<1>[height=4.5cm,width=4.5cm]{figures/tree_boost_p.pdf}
			\includegraphics<2-4>[height=4.5cm,width=4.5cm]{figures/tree_boost_p_0.pdf}
			%p1
			\includegraphics<5-7>[height=4.5cm,width=4.5cm]{figures/tree_boost_p1.pdf}
			%p2
			\includegraphics<8-10>[height=4.5cm,width=4.5cm]{figures/tree_boost_p2.pdf}
			%p3
			\includegraphics<11-13>[height=4.5cm,width=4.5cm]{figures/tree_boost_p3.pdf}
			%p4
			\includegraphics<14>[height=4.5cm,width=4.5cm]{figures/tree_boost_p4.pdf}
		\end{column}
		\begin{column}{0.35\textwidth}
			%res1
			\includegraphics<3>[height=4.5cm,width=4.5cm]{figures/tree_boost_res1.pdf}
			\includegraphics<4-5>[height=4.5cm,width=4.5cm]{figures/tree_boost_resf1.pdf}
			%res2
			\includegraphics<6>[height=4.5cm,width=4.5cm]{figures/tree_boost_res2.pdf}
			\includegraphics<7-8>[height=4.5cm,width=4.5cm]{figures/tree_boost_resf2.pdf}
			%res3
			\includegraphics<9>[height=4.5cm,width=4.5cm]{figures/tree_boost_res3.pdf}
			\includegraphics<10-11>[height=4.5cm,width=4.5cm]{figures/tree_boost_resf3.pdf}
			%res4
			\includegraphics<12>[height=4.5cm,width=4.5cm]{figures/tree_boost_res4.pdf}
			\includegraphics<13-14>[height=4.5cm,width=4.5cm]{figures/tree_boost_resf4.pdf}
		\end{column}
	\end{columns}

\end{frame}

\begin{frame}{Second order gradient tree boosting: Complexity}
	\begin{myblock}{This was too easy!}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
		\begin{itemize}
			\item How do we choose the complexity of the trees?
			\item And how many boosting iterations?
		\end{itemize}
\end{myblock}
	\begin{columns}[T]
		\begin{column}{0.35\textwidth}
		\includegraphics<2->[height=4.5cm,width=4.5cm]{figures/xgb_boost_p.pdf}
		\end{column}
		\begin{column}{0.6\textwidth}
			\visible<3->{
			\begin{myblock}{Regularization}{bg=yellow!05,fg=black}{bg=yellow!20, fg=black}
			\begin{itemize}
			\item Choose a maximum depth?
			\item A maximum number of leaf-nodes?
			\item A minimum observations in node?
			\item A minimum reduction in loss when splitting?
			\item A set number of boosting iterations?
			\end{itemize}
			\end{myblock}
			}
		\end{column}
	\end{columns}

\end{frame}

\begin{frame}{The researcher contemplates}

	\begin{myblock}{The researcher goes home...}{bg=blue!5,fg=black}{bg=blue!10, fg=black}
		He is determined to win that ML-competition!...
		\begin{itemize}
			\item<2-> But he only has the 2-gb ram laptop running Windows XP that his job graciously gave him as his every-day workhorse.
		\end{itemize}
	\end{myblock}

	\visible<3->{

	\begin{columns}[T]

		\begin{column}{0.6\linewidth}

			\begin{myblock}{So what does he do?}{bg=yellow!05,fg=black}{bg=yellow!20, fg=black}
				\begin{itemize}
					\item<4-> Opt 1: Buy a new computer?
					\item<5-> Opt 2: Expert knowledge on data and tuning?
					\item<6-> Opt 3: Get lucky?
					\item<7-> Opt 4: Rebuild the algorithm to not require tuning?
				\end{itemize}
			\end{myblock}

		\end{column}

		\begin{column}{0.4\linewidth}
			\visible<8->{
			\begin{myblock}{\small{Plot twist: the researcher is me!}}{bg=green!05,fg=black}{bg=green!20, fg=black}
				\begin{itemize}
					\item<9-> Opt 1: I am a PhD student...
					\item<10-> Opt 2: I am too lazy to be an expert!
					\item<11-> Opt 3: I like good expectations.
					\item<12-> Opt 4: Hmm...
				\end{itemize}
			\end{myblock}
		}

		\end{column}

	\end{columns}

	}

\end{frame}

%\subsection{Model complexity selection}


\chapter{Dataset}\label{chap:dataset}

For our experiments, we make use of a dataset composed of Reber sequences. A Reber sequence is a grammar string made of finite states, in simple words, formed by using a confined set of characters. In the research paper that proposed the LSTM \cite{lstm}, the authors use embedded Reber grammar due to its short time lags. For our purpose, we use a simple version of this embedded Reber grammar, called Reber grammar (\cite{reber}).

The following figure shows the flow diagram to generate Reber grammar sequences:

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\linewidth]{images/dataset/reber.png}
    \caption[Flow diagram to generate Reber grammar sequences]{Flow diagram to generate Reber grammar sequences}
    \label{fig:reber}
\end{figure}

As shown in the above graph, a sequence is a true Reber sequence if
\begin{enumerate}
    \item it contains only these characters: \textit{B, E, P, S, T, V, X}
    \item it starts with a $B$ and ends with an $E$, \textit{always!}
    \item it strictly follows the transition diagram shown in figure \ref{fig:reber}.
\end{enumerate}

After starting at $B$, we move from one node to the next until we reach $E$. If we have more than one path to choose from, we can randomly select one of them. Two of the allowed characters, $S$ and $T$, have a self-loop at nodes $2$ and $4$, respectively, meaning we can have more than two consecutive $S$s and $T$s. For example, the following table shows some of the true and false Reber sequences from our dataset:

\begin{table}[h]
	\centering
	\begin{tabular}{|c|c|}
	    \hline
		\textbf{True Reber sequences} & \textbf{False Reber sequences} \\
		\hline
		BPTVPXTTVVE & BTTVPXTVPSE \\
		BTSXXTTTTVVE & BPSXXTTTVPSE \\
		BTSSXXTTVVE & BPSSSXXTVVE \\
		BTXXVPXTVVE & BTTTVPXTVVE \\
		BPTTTTTVPSE & BPSSXXTTVVE \\
		\hline
	\end{tabular}
	\caption[A few examples of true and false Reber sequences]{A few examples of true and false Reber sequences.}
	\label{tab:reber}
\end{table}

Since, to our knowledge, there is no public dataset of Reber sequences available, we created our own dataset of $25000$ true and false Reber sequences combined. We follow the same flow, as shown in figure \ref{fig:reber}, to generate true Reber sequences. Then, by slightly altering the same procedure, we generate false Reber sequences.

We then split this dataset into a train-test set, details of which are shown in the following bar chart:

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\linewidth]{images/dataset/string_counts.png}
    \caption[Validity and number of strings in train-test dataset]{Number of valid (true) and invalid (false) Reber sequences in our train-test set}
    \label{fig:string_count}
\end{figure}

Our dataset has $12500$ true Reber sequences and $12500$ false, thus totaling $25000$. Although the true Reber sequences only contain allowed characters (i.e., \textit{B, E, P, S, T, V, X}), the false sequences may contain any character from A to Z.

Out of $25000$ total sequences, our train-test set contains $18750$ and $6250$ sequences, respectively. Furthermore, this train-test individually include the following number of true and false Reber sequences:

\begin{table}[h]
	\centering
	\begin{tabular}{|c|c|c|}
	    \hline
		 & \textbf{Training set} & \textbf{Test set} \\
		\hline
		\textbf{Valid (true)} & 9339 & 3161 \\
		\textbf{Invalid (false)} & 9411 & 3089 \\
		\hline
	\end{tabular}
	\caption[The number of true and false Reber sequences in our train-test set]{The number of true and false Reber sequences in our train-test set.}
	\label{tab:train-test_strings}
\end{table}
Next, we visualize the string length distribution that shows not only the minimum and maximum string length but also the number of strings for each length value.

The following plot displays the string length distribution of our entire dataset:

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\linewidth]{images/dataset/entire_dataset_string_len.png}
    \caption[String length distribution of entire dataset]{String length distribution of entire dataset}
    \label{fig:string_len}
\end{figure}

As we can see in the above plot, in our entire dataset, the minimum string length is $11$ with $216$ sequences, and the maximum string length is $53$ with $1$ sequence. The highest number of strings, $2112$ strings, in our entire dataset is of length $22$.

The following plot displays the string length distribution of our train dataset:

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\linewidth]{images/dataset/train_dataset_string_len.png}
    \caption[String length distribution of train dataset]{String length distribution of train dataset}
    \label{fig:train_string_len}
\end{figure}

As we can see in the above plot, in our train dataset, the minimum string length is $11$ with $160$ sequences, and the maximum string length is $49$ with $2$ sequence. The highest number of strings, $1566$ strings, in our train dataset is of length $22$.

The following plot displays the string length distribution of our test dataset:

\begin{figure}[h]
    \centering
    \includegraphics[width=0.9\linewidth]{images/dataset/test_dataset_string_len.png}
    \caption[String length distribution of test dataset]{String length distribution of test dataset}
    \label{fig:test_string_len}
\end{figure}

As we can see in the above plot, in our test dataset, the minimum string length is $11$ with $56$ sequences, and the maximum string length is $53$ with $1$ sequence. The highest number of strings, $546$ strings, in our test dataset is of length $22$.


\documentclass{article}
\usepackage[a4paper, total={6in, 9in}]{geometry}
\usepackage{fancyhdr}
\usepackage{amsmath}
\usepackage{algorithm}
\usepackage{algpseudocode}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{proof}{Proof}[section]

\pagestyle{fancy}
\fancyhf{}
\rhead{DanDoge}
\lhead{Notes on k-NN}
\rfoot{Page \thepage}
\cfoot{latest version: 2018/01/20}

\title{Notes on k-NN}
\date{2018-01-20}
\author{DanDoge}

\begin{document}
\section{defination}
  \paragraph{}
    k-nearest neighbors algorithm is a non-parametric method for classification and regerssion.  In k-NN classification, the output is a class membership classified by a majority vote of its neighbors, especially, if k == 1 then the object is simply assigned to the classs of its nearest neighbor. In k-NN regerssion, the out put value is the average of the values of its k nearest neighbors.

\section{algorithm}
  \begin{algorithm}
    \caption{k-NN algorithm}
    \hspace*{0.02in}{\bf Input:} $\vec x$, training set \textit{T} = ${(x_1, y_1), ..., (x_n, y_n)}$ , where $\vec x_i \in \textit{X}$ being eigenvectors and $y_i \in \textit{Y}$ = $\{c_1, c_2, ..., c_k\}$ being classes
    \newline{}
    \hspace*{0.02in}{\bf Output:} class y that $\vec x$ belongs to.
    \begin{algorithmic}[1]
      \State find k nearest neighbors using metric given, using $N_k(x)$ to denote these dots
      \State use dicision rule, e.g. majority rule, to dicide the class of x
    \end{algorithmic}
  \end{algorithm}

\section{metric}
  \paragraph{}
    usually we use Euclidean distance, $L_2(x_i, x_j) = (\sum_{l = 1}^{n}(x_i^{(l)} - x_j^{(l)})^{2})^{1/2}$, more generally, $\textit{L}_p$ distance is $L_p(x_i, x_j) = (\sum_{l = 1}^{n}(x_i^{(l)} - x_j^{(l)})^{p})^{1/p}$, If k == 1, it is called Manhattan distance, and when k == $\inf$, $L_{\inf}(x_i, x_j) = max_{l} \|x_i^{(l)} - x_j^{(l)}\|$

\section{weighted nearest neighbor classifier}
  \paragraph{}
    the k-nearest neighbor classifier could be viewed as assigning the k nearest neighbors a weight of 1/k and all others 0 weight, we can generlize it to weighted nearest neighbor classifiers with the \textit{i}th nearest neighbor a weight $\textit{w}_{ni}$, and $\sum_{i = 1}^{n} \textit{w}_{ni}$ = 1

\section{CNN for data reduction}
  \paragraph{}
    condensed nearest neighbor is an algorithm designed to reduce the data set for k-NN classification.
    \begin{algorithm}
      \caption{CNN for data reduction}
      \hspace*{0.02in}{\bf Input:} training set \textit{X}
      \newline{}
      \hspace*{0.02in}{\bf Output:} the set of prototypes \textit{U}
      \begin{algorithmic}[1]
        \State scan all elements of \textit{X}, looking for an element whose nearest element in \textit{U} has a different label than x.
        \State remove x from \textit{X}, and add it to \textit{U}.
        \State repeat till no more prototypes are added to \textit{U}.
      \end{algorithmic}
    \end{algorithm}
    \newline{}
    and use \textit{U} instead of \textit{X} for classification.
\end{document}


\chapter{Descretization of the dynamics}
\label{chap:descretization dynamics}
{\bf \Large
\begin{tabular}{ccc}
\hline
  Corresponding author & : & Hirofumi Tomita\\
\hline
\end{tabular}
}

\section{Time integration method}

For the time integration of Eqs.(\ref{eq:rhotot_d2})-(\ref{eq:etot_d2}),
we adopt the full explicit scheme with
the $p$-th order Runge-Kutta scheme.
\begin{eqnarray}
&& \phi^{*}_{0} = \phi^{t}\\
&&  \phi^{*}_{1}
= \phi^{t} + \left(\frac{\partial \phi}{\partial t}\right)^{*}_{0}\frac{\Delta t}{p}\\
&&  \phi^{*}_{2}
= \phi^{t} + \left(\frac{\partial \phi}{\partial t}\right)^{*}_{1}\frac{\Delta t}{p-1}\\
&&  \cdot \cdot \cdot\nonumber\\
&&  \cdot \cdot \cdot\nonumber\\
&&  \cdot \cdot \cdot\nonumber\\
&&  \phi^{*}_{p} = \phi^{t} + \left(\frac{\partial \phi}{\partial t}\right)^{*}_{p-1} \Delta t\\
&& \phi^{t+\Delta t} = \phi^{*}_{p}
\end{eqnarray}
where $\left(\frac{\partial \phi}{\partial t}\right)^{*}_{k}$ is
the estimated time tendency by using $\phi^{*}_{k}$.
Usually, we use $p=2,3$ and $4$.

\section{Spatial descretization}
We employ the Arakawa-C staggered grid with the 3-dimensional momentum
($\rho u, \rho v, \rho w$), density ($\rho$) and mass-weighted potentail temperature($\rho \theta$)
as the prognostic variables.
Figure \ref{fig:cntrl-volume}(a) shows the structure of the control volume for the mass,
indicating the location of each of prognostic variables.
Conceptually, we use the 4th order central difference scheme
for the advection or convection terms and
the 2nd order central difference scheme for the other terms.
Before the descretization of differential equations,
we should diagnose several quantities from the prognostic variables.
\begin{description}
\item[Full-level pressure and potential temperature]
\begin{eqnarray}
&&p_{i,j,k}=p_{00}\left[\frac{(\rho \theta)_{i,j,k} R^*}{p_{00}} \right]^{\frac{c_{p}^*}{c_{p}^*- R^*}}\\
&&\theta_{i,j,k} = \frac{(\rho \theta)_{i,j,k}}{\rho_{i,j,k}} \label{eq:theta full} \\
\end{eqnarray}
\item[Half-level density]
\begin{eqnarray}
&&  \overline{\rho}_{i+\frac{1}{2},j,k} = \frac{\rho_{i+1,j,k}+\rho_{i,j,k}}{2} \label{eq:rho half i} \\
&&  \overline{\rho}_{i,j+\frac{1}{2},k} = \frac{\rho_{i,j+1,k}+\rho_{i,j,k}}{2} \label{eq:rho half j} \\
&&  \overline{\rho}_{i,j,k+\frac{1}{2}} = \frac{\rho_{i,j,k+1}+\rho_{i,j,k}}{2} \label{eq:rho half k}
\end{eqnarray}
\item[Half-level velocity]
\begin{eqnarray}
&&  \overline{u}_{i+\frac{1}{2},j,k} = \frac{(\rho u)_{i+\frac{1}{2},j,k}}{\overline{\rho}_{i+\frac{1}{2},j,k}}\\
&&  \overline{v}_{i,j+\frac{1}{2},k} = \frac{(\rho v)_{i,j+\frac{1}{2},k}}{\overline{\rho}_{i,j+\frac{1}{2},k}}\\
&&  \overline{w}_{i,j,k+\frac{1}{2}} = \frac{(\rho w)_{i,j,k+\frac{1}{2}}}{\overline{\rho}_{i,j,k+\frac{1}{2}}}
\end{eqnarray}
\item[Full-level velocity]
\begin{eqnarray}
&&  \overline{u}_{i,j,k} = \frac{(\rho u)_{i+\frac{1}{2},j,k}+(\rho u)_{i-\frac{1}{2},j,k}}{2\rho_{i,j,k}} \label{eq:u full} \\
&&  \overline{v}_{i,j,k} = \frac{(\rho v)_{i,j+\frac{1}{2},k}+(\rho v)_{i,j-\frac{1}{2},k}}{2\rho_{i,j,k}} \label{eq:v full} \\
&&  \overline{w}_{i,j,k} = \frac{(\rho w)_{i,j,k+\frac{1}{2}}+(\rho w)_{i,j,k-\frac{1}{2}}}{2\rho_{i,j,k}} \label{eq:w full}
\end{eqnarray}
\end{description}

\subsection{Continuity equation}
\begin{eqnarray}
%%
\left(\frac{\partial \rho}{\partial t}\right)_{i,j,k}
&=& - \frac{(\rho u)_{i+\frac{1}{2},j,k} -(\rho u)_{i-\frac{1}{2},j,k}}{\Delta x}\nonumber\\
& & - \frac{(\rho v)_{i,j+\frac{1}{2},k} -(\rho v)_{i,j-\frac{1}{2},k}}{\Delta y}\nonumber\\
& & - \frac{(\rho w)_{i,j,k+\frac{1}{2}} -(\rho w)_{i,j,k-\frac{1}{2}}}{\Delta z}
\end{eqnarray}

\subsection{Momentum equations}
Figure \ref{fig:cntrl-volume}(a) shows the structure of the control volume for the momentum
in the x direction.
The momentum equation is descretized as
\begin{eqnarray}
%%
\left(\frac{\partial \rho u}{\partial t}\right)_{i+\frac{1}{2},j,k}
&=& - \frac{\overline{(\rho u)}_{i+1,j,k} \overline{u}_{i+1,j,k}
           -\overline{(\rho u)}_{i,j,k} \overline{u}_{i,j,k}}
     {\Delta x}\nonumber\\
& & - \frac{\overline{(\rho u)}_{i+\frac{1}{2},j+\frac{1}{2},k}  \overline{v}_{i+\frac{1}{2},j+\frac{1}{2},k}
           -\overline{(\rho u)}_{i+\frac{1}{2},j-\frac{1}{2},k}  \overline{v}_{i+\frac{1}{2},j-\frac{1}{2},k}}
     {\Delta y}\nonumber\\
& & - \frac{\overline{(\rho u)}_{i+\frac{1}{2},j,k+\frac{1}{2}}  \overline{w}_{i+\frac{1}{2},j,k+\frac{1}{2}}
           -\overline{(\rho u)}_{i+\frac{1}{2},j,k-\frac{1}{2}}  \overline{w}_{i+\frac{1}{2},j,k-\frac{1}{2}}}
     {\Delta z}\nonumber\\
& & -\frac{p_{i+1,j,k}-p_{i,j,k}}{\Delta x},
\end{eqnarray}
where
\begin{eqnarray}
&& \overline{(\rho u)}_{i,j,k} \nonumber\\
= && \frac{-(\rho u)_{i+\frac{3}{2},j,k}+7(\rho u)_{i+\frac{1}{2},j,k}+7(\rho u)_{i-\frac{1}{2},j,k}-(\rho u)_{i-\frac{3}{2},j,k}}{12}\\
&& \overline{(\rho u)}_{i+\frac{1}{2},j+\frac{1}{2},k} \nonumber\\
= && \frac{-(\rho u)_{i+\frac{1}{2},j+2,k}+7(\rho u)_{i+\frac{1}{2},j+1,k}+7(\rho u)_{i+\frac{1}{2},j,k}-(\rho u)_{i+\frac{1}{2},j-1,k}}{12}\\
&& \overline{(\rho u)}_{i+\frac{1}{2},j,k+\frac{1}{2}} \nonumber\\
= &&\frac{-(\rho u)_{i+\frac{1}{2},j,k+2}+7(\rho u)_{i+\frac{1}{2},j,k+1}+7(\rho u)_{i+\frac{1}{2},j,k}-(\rho u)_{i+\frac{1}{2},j,k-1}}{12}
\end{eqnarray}
and the velocities at the cell wall for the staggered control volume to $x$ direction are
defined as
\begin{eqnarray}
\overline{u}_{i,j,k} &=& \frac{\overline{u}_{i+\frac{1}{2},j,k}+\overline{u}_{i-\frac{1}{2},j,k}}{2}\\
\overline{v}_{i+\frac{1}{2},j+\frac{1}{2},k} &=&
\frac{\overline{v}_{i,j+\frac{1}{2},k}+\overline{v}_{i+1,j+\frac{1}{2},k}}{2}\\
\overline{w}_{i+\frac{1}{2},j,k+\frac{1}{2}} &=&
\frac{\overline{w}_{i,j,k+\frac{1}{2}}+\overline{w}_{i+1,j,k+\frac{1}{2}}}{2}
\end{eqnarray}
In this form, the 4th order accuracy is guaranteed
on the condition of the constant velocity.

The momentum equations in the $y$ and $z$ directions are descretized
in the same way:
\begin{eqnarray}
%%
\left(\frac{\partial \rho v}{\partial t}\right)_{i,j+\frac{1}{2},k}
&=& - \frac{\overline{(\rho v)}_{i+\frac{1}{2},j+\frac{1}{2},k}  \overline{u}_{i+\frac{1}{2},j+\frac{1}{2},k}
           -\overline{(\rho v)}_{i-\frac{1}{2},j+\frac{1}{2},k}  \overline{u}_{i-\frac{1}{2},j+\frac{1}{2},k}}
     {\Delta x}\nonumber\\
& & - \frac{\overline{(\rho v)}_{i,j+1,k} \overline{v}_{i,j+1,k}
           -\overline{(\rho v)}_{i,j,k} \overline{v}_{i,j,k}}
     {\Delta y}\nonumber\\
& & - \frac{\overline{(\rho v)}_{i,j+\frac{1}{2},k+\frac{1}{2}}  \overline{v}_{i,j+\frac{1}{2},k+\frac{1}{2}}
           -\overline{(\rho v)}_{i,j+\frac{1}{2},k-\frac{1}{2}}  \overline{v}_{i,j+\frac{1}{2},k-\frac{1}{2}}}
     {\Delta z}\nonumber\\
& & -\frac{p_{i,j+1,k}-p_{i,j,k}}{\Delta y},\\
%%
\left(\frac{\partial \rho w}{\partial t}\right)_{i,j,k+\frac{1}{2}}
&=& - \frac{\overline{(\rho w)}_{i+\frac{1}{2},j,k+\frac{1}{2}}  \overline{u}_{i+\frac{1}{2},j,k+\frac{1}{2}}
           -\overline{(\rho w)}_{i-\frac{1}{2},j,k+\frac{1}{2}}  \overline{u}_{i-\frac{1}{2},j,k+\frac{1}{2}}}
     {\Delta x}\nonumber\\
& & - \frac{\overline{(\rho w)}_{i,j+\frac{1}{2},k+\frac{1}{2}}  \overline{w}_{i,j+\frac{1}{2},k+\frac{1}{2}}
           -\overline{(\rho w)}_{i,j-\frac{1}{2},k+\frac{1}{2}}  \overline{w}_{i,j-\frac{1}{2},k+\frac{1}{2}}}
     {\Delta y}\nonumber\\
& & - \frac{\overline{(\rho w)}_{i,j,k+1} \overline{w}_{i,j,k+1}
           -\overline{(\rho w)}_{i,j,k} \overline{w}_{i,j,k}}
     {\Delta z}\nonumber\\
& & -\frac{p_{i,j,k+1}-p_{i,j,k}}{\Delta z}-\overline{\rho}_{i,j,k+\frac{1}{2}} g
\end{eqnarray}

\subsection{Energy equation}

\begin{eqnarray}
%%
\left(\frac{\partial \rho \theta}{\partial t}\right)_{i,j,k}
&=& - \frac{(\rho u)_{i+\frac{1}{2},j,k} \overline{\theta}_{i+\frac{1}{2},j,k}
           -(\rho u)_{i-\frac{1}{2},j,k} \overline{\theta}_{i-\frac{1}{2},j,k}}
     {\Delta x}\nonumber\\
& &  - \frac{(\rho v)_{i,j+\frac{1}{2},k} \overline{\theta}_{i,j+\frac{1}{2},k}
           -(\rho v)_{i,j-\frac{1}{2},k} \overline{\theta}_{i,j-\frac{1}{2},k}}
     {\Delta y}\nonumber\\
& &  - \frac{(\rho w)_{i,j,k+\frac{1}{2}} \overline{\theta}_{i,j,k+\frac{1}{2}}
           -(\rho w)_{i,j,k-\frac{1}{2}} \overline{\theta}_{i,j,k-\frac{1}{2}}}
     {\Delta z}\nonumber\\
\end{eqnarray}
where
\begin{eqnarray}
&& \overline{\theta}_{i+\frac{1}{2},j,k} =
\frac{-\theta_{i+2,j,k}+7\theta_{i+1,j,k}+7\theta_{i,j,k}-\theta_{i-1,j,k}}{12}\\
&& \overline{\theta}_{i,j+\frac{1}{2},k} =
\frac{-\theta_{i,j+2,k}+7\theta_{i,j+1,k}+7\theta_{i,j,k}-\theta_{i,j-1,k}}{12}\\
&& \overline{\theta}_{i,j,k+\frac{1}{2}} =
\frac{-\theta_{i,j,k+2}+7\theta_{i,j,k+1}+7\theta_{i,j,k}-\theta_{i,j,k-1}}{12}
\end{eqnarray}

\subsection{Tracer advection}
The tracer advection process is done after the time integration of
the dynamical variables ($\rho$, $\rho u,\rho v,\rho w$, and $\rho \theta$).
We impose two constraints to tracer advection:
\begin{description}
\item[Consistency With Continuity ( CWC )]
On the condition without any source/sink,
the mass concentration in the advection process
should be conserved along the trajectory.
It is, at least,  necessary that
the spatially constant mass concentration should be kept
in any motion of fluid.
In order to satisfy this condition, we use the same mass flux at the last Runge-Kutta
process of Eqs.() and () for integration of tracers:
\begin{eqnarray}
%%
\frac{\left(\rho q\right)^{n+1}_{i,j,k} - \left(\rho q\right)^{n}_{i,j,k}}{\Delta t}
&=& - \frac{(\rho u)_{i+\frac{1}{2},j,k} \overline{q}_{i+\frac{1}{2},j,k}
           -(\rho u)_{i-\frac{1}{2},j,k} \overline{q}_{i-\frac{1}{2},j,k}}
     {\Delta x}\nonumber\\
& &  - \frac{(\rho v)_{i,j+\frac{1}{2},k} \overline{q}_{i,j+\frac{1}{2},k}
           -(\rho v)_{i,j-\frac{1}{2},k} \overline{q}_{i,j-\frac{1}{2},k}}
     {\Delta y}\nonumber\\
& &  - \frac{(\rho w)_{i,j,k+\frac{1}{2}} \overline{q}_{i,j,k+\frac{1}{2}}
           -(\rho w)_{i,j,k-\frac{1}{2}} \overline{q}_{i,j,k-\frac{1}{2}}}
     {\Delta z}\nonumber\\
\label{eq:tracer_int}
\end{eqnarray}
\item[Monotonicity]
In order to satisfy the monotonicity of tracer advection,
we employ the Flux Corrected Trasport scheme, which is a hybrid scheme
with the 4th order central difference scheme and 1st order upwind scheme.
If The 4th order central difference is applied,
$\overline{q}$ is descritized as
\begin{eqnarray}
&& \overline{q}_{i+\frac{1}{2},j,k}^{high} =
\frac{-q_{i+2,j,k}+7q_{i+1,j,k}+7q_{i,j,k}-q_{i-1,j,k}}{12}\\
&& \overline{q}_{i,j+\frac{1}{2},k}^{high} =
\frac{-q_{i,j+2,k}+7q_{i,j+1,k}+7q_{i,j,k}-q_{i,j-1,k}}{12}\\
&& \overline{q}_{i,j,k+\frac{1}{2}}^{high} =
\frac{-q_{i,j,k+2}+7q_{i,j,k+1}+7q_{i,j,k}-q_{i,j,k-1}}{12}.
\end{eqnarray}
On the other hand, in the 1st order upwind scheme
$\overline{q}$ is described as
\begin{eqnarray}
\overline{q}_{i+\frac{1}{2},j,k}^{low} =
\begin{cases}
  q_{i,j,k} & ( (\rho u)_{i+\frac{1}{2},j,k}>0 )\\
  q_{i+1,j,k} & ({\rm otherwise})\\
\end{cases}
\\
\overline{q}_{i,j+\frac{1}{2},k}^{low} =
\begin{cases}
  q_{i,j,k} & ( (\rho u)_{i,j+\frac{1}{2},k}>0 )\\
  q_{i,j+1,k} & ({\rm otherwise})\\
\end{cases}
\\
\overline{q}_{i,j,k+\frac{1}{2}}^{low} =
\begin{cases}
  q_{i,j,k} & ( (\rho u)_{i,j,k+\frac{1}{2}}>0 )\\
  q_{i,j,k+1} & ({\rm otherwise})\\
\end{cases}
\end{eqnarray}
The actual $\overline{q}$ is described as
\begin{eqnarray}
  \overline{q}_{i+\frac{1}{2},j,k}
&=& C_{i+\frac{1}{2},j,k} \overline{q}_{i+\frac{1}{2},j,k}^{high}
+ \left( 1 - C_{i+\frac{1}{2},j,k}\right) \overline{q}_{i+\frac{1}{2},j,k}^{low}\\
  \overline{q}_{i,j+\frac{1}{2},k}
&=& C_{i,j+\frac{1}{2},k} \overline{q}_{i,j+\frac{1}{2},k}^{high}
+ \left( 1 - C_{i,j+\frac{1}{2},k}\right) \overline{q}_{i,j+\frac{1}{2},k}^{low}\\
  \overline{q}_{i,j,k+\frac{1}{2}}
&=& C_{i,j,k+\frac{1}{2}} \overline{q}_{i,j,k+\frac{1}{2}}^{high}
+ \left( 1 - C_{i,j,k+\frac{1}{2}}\right) \overline{q}_{i,j,k+\frac{1}{2}}^{low}
\end{eqnarray}
See the appedix for the method to determine the flux limter.
\end{description}

\section{boundary condition}
The boundary condition only for the vertical velocity at the top and bottom
boundaries is needed:
\begin{eqnarray}
&&  w_{i,j,k_{max}+\frac{1}{2}} = 0\\
&&  w_{i,j,k_{min}-\frac{1}{2}} = 0
\end{eqnarray}
This leads to the boundary condition of the prognostic variable as
\begin{eqnarray}
&&  (\rho w)_{i,j,k_{max}+\frac{1}{2}} = 0\\
&&  (\rho w)_{i,j,k_{min}-\frac{1}{2}} = 0
\end{eqnarray}

\section{Numerical filters}

In the dynamics exept for the tracer advection,
we can consider two ways to stabilize the calculation conceptually.
\begin{enumerate}
\item We impose no explicit numerical filter.
The numerical stability can be achieved
by aid of tiny portion of turbulence scheme.
\item We impose tiny explicit numerical filter by the numerical viscousity or diffusion.
In order to damp the higher wavenumber component selectively,
we adapt the super-viscousty or diffusion as the traditional way.
\end{enumerate}
In this section, we formulate the latter case.
The Laplacian of an arbitray prognostic valuable $f (\in \rho, u, v, w, \theta)$
is descritized as
{\footnotesize
\begin{eqnarray}
(\Delta f)_i &=&
\frac{1}{\Delta x_i}\left[
\frac{1}{\Delta x_{i+\frac{1}{2}}}f_{i+1}
-\left(\frac{1}{\Delta x_{i+\frac{1}{2}}}+\frac{1}{\Delta x_{i-\frac{1}{2}}}\right)f_i
+\frac{1}{\Delta x_{i-\frac{1}{2}}}f_{i-1}\right]
%
\end{eqnarray}
}
The super-diffusion can be descritezed as
{\footnotesize
\begin{eqnarray}
&&\frac{\partial }{\partial x}\left[\nu \rho \frac{\partial \Delta f}{\partial x}\right]  =\nonumber\\
&&
\frac{1}{\Delta x_{i}}
\Bigg[\nonumber\\
&&\frac{\nu_{i+\frac{1}{2}}\rho_{i+\frac{1}{2}}}{\Delta x_{i+\frac{1}{2}}}
\Bigg\{\frac{1}{\Delta x_{i+1}}\left[
\frac{1}{\Delta x_{i+\frac{3}{2}}}f_{i+2}
-\left(\frac{1}{\Delta x_{i+\frac{3}{2}}}+\frac{1}{\Delta x_{i+\frac{1}{2}}}\right)f_{i+1}
+\frac{1}{\Delta x_{i+\frac{1}{2}}}f_{i}\right]\nonumber\\
&&~~~~~~~~~~~~ -\frac{1}{\Delta x_i}\left[
\frac{1}{\Delta x_{i+\frac{1}{2}}}f_{i+1}
-\left(\frac{1}{\Delta x_{i+\frac{1}{2}}}+\frac{1}{\Delta x_{i-\frac{1}{2}}}\right)f_i
+\frac{1}{\Delta x_{i-\frac{1}{2}}}f_{i-1}\right]\Bigg\}
\nonumber\\
%
&-&
\frac{\nu_{i-\frac{1}{2}}\rho_{i-\frac{1}{2}}}{\Delta x_{i-\frac{1}{2}}}
\Bigg\{\frac{1}{\Delta x_{i}}\left[
\frac{1}{\Delta x_{i+\frac{1}{2}}}f_{i+1}
-\left(\frac{1}{\Delta x_{i+\frac{1}{2}}}+\frac{1}{\Delta x_{i-\frac{1}{2}}}\right)f_{i}
+\frac{1}{\Delta x_{i-\frac{1}{2}}}f_{i-1}\right]\nonumber\\
&&~~~~~~~~~~~~ -\frac{1}{\Delta x_{i-1}}\left[
\frac{1}{\Delta x_{i-\frac{1}{2}}}f_{i}
-\left(\frac{1}{\Delta x_{i-\frac{1}{2}}}+\frac{1}{\Delta x_{i-\frac{3}{2}}}\right)f_{i-1}
+\frac{1}{\Delta x_{i-\frac{3}{2}}}f_{i-2}\right]\Bigg\}\nonumber\\
&&\Bigg]
%
%
\end{eqnarray}
}
In Eq.(), the numerical diffiusion flux can be written as
{\footnotesize
\begin{eqnarray}
F_{i+\frac{1}{2}}&=&\frac{\nu_{i+\frac{1}{2}}\rho_{i+\frac{1}{2}}}{\Delta x_{i+\frac{1}{2}}}
\Bigg\{\frac{1}{\Delta x_{i+1}}\left[
\frac{1}{\Delta x_{i+\frac{3}{2}}}f_{i+2}
-\left(\frac{1}{\Delta x_{i+\frac{3}{2}}}+\frac{1}{\Delta x_{i+\frac{1}{2}}}\right)f_{i+1}
+\frac{1}{\Delta x_{i+\frac{1}{2}}}f_{i}\right]\nonumber\\
&&~~~~~~~~~~~~ -\frac{1}{\Delta x_i}\left[
\frac{1}{\Delta x_{i+\frac{1}{2}}}f_{i+1}
-\left(\frac{1}{\Delta x_{i+\frac{1}{2}}}+\frac{1}{\Delta x_{i-\frac{1}{2}}}\right)f_i
+\frac{1}{\Delta x_{i-\frac{1}{2}}}f_{i-1}\right]\Bigg\}\nonumber\\
\end{eqnarray}
}
The coefficient $\nu$ is determined by
\begin{eqnarray}
  \nu_{i+\frac{1}{2}} &=& \gamma \frac{\Delta x_{i+\frac{1}{2}}^4}{\Delta t}
\end{eqnarray}
where  the $\gamma$ is non-dimensional coefficient and it should satisfy that
\begin{eqnarray}
\gamma < \frac{1}{16}
\end{eqnarray}
for the numerical stability of numerical filter itself.
If Eq.() is rewritten by using Eq.(),
\begin{eqnarray}
\frac{\partial }{\partial x}\left[\nu \rho \frac{\partial \Delta f}{\partial x}\right]
=
\frac{F_{i+\frac{1}{2}}-F_{i-\frac{1}{2}}}{\Delta x_{i}}
%
%
\end{eqnarray}
In this numerical diffusion process,
we can consider that the advective flux is modified as
\begin{eqnarray}
  (\rho u f)_{i+\frac{1}{2}}^{\dagger} = (\rho u f)_{i+\frac{1}{2}}+F_{i+\frac{1}{2}}
\end{eqnarray}
where the first term of R.H.S. is the flux by the advection scheme.
In this dynamics, it is the 4th order central difference scheme.
This concept is very important for the CWC condition in the continuity equation.
The mass flux in the tracer advection should be used as this formulation,
otherwize the CWC condition is violated.

The numerical diffusion in the $y$ and $z$ directions
are formulated in the same way.

\begin{figure}[t]
\begin{center}
  \begin{tabular}{l}
    (a) Control volume for the mass\\
  \includegraphics[scale=0.6]{./figure/cntrl-volume.eps}\\
    (b) Control volume for the momentum\\
  \includegraphics[scale=0.6]{./figure/cntrl-volume2.eps}\\
  \end{tabular}
\end{center}
  \caption{Control volume}
  \label{fig:cntrl-volume}
\end{figure}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{algorithm}
\usepackage{graphicx}
\usepackage[noend]{algpseudocode}
\usepackage{url}
\usepackage{listings}

\newlength\tindent
\setlength{\tindent}{\parindent}
\setlength{\parindent}{0pt}
\renewcommand{\indent}{\hspace*{\tindent}}

\title{COMP 312 Assignment 6}
\author{Daniel Braithwaite}

\begin{document}
	\pagenumbering{gobble}
	\maketitle
	\newpage
  	\pagenumbering{arabic}

  	\section{Python}
  		\subsection{Program A}
			\subsubsection{Code}
				\lstinputlisting[language=Python]{Programs/coxian.py}
			\subsubsection{Output}
				$E[t]$: (0.75741101633238683, 0.78099765992129067)\\
$E[t^2]$: (0.92233300526267215, 0.98510362267500762)

			The theoretical values are within the 95\% confidence interval which means we can conclude that $E[t]$ and $E[t^2]$ are equal to the theoretical values

  		\subsection{Program B}
  			\subsubsection{Code}
				\lstinputlisting[language=Python]{Programs/mg1-coxian.py}
			\subsubsection{Output}
				Mean Y: -0.00233069473309\\
Conf Y: (-0.011739640704165668, 0.0070782512379870006)\\

				We want Y to be 0 as this would show $w_q = \frac{\lambda E[t^2]}{2(1-\rho)}$. We see that 0 is within the confidence interval of the average Y. Not only this but our mean Y is small enough for us to conclude that $w_q = \frac{\lambda E[t^2]}{2(1-\rho)}$

\end{document}


\section{Performance Evaluation}

We evaluated the performance of our code, measuring runtime of our serial and
graph parallel code for a variety of $N$, $M$, and $T$. We generated synthetic
data with random $(A, B, \pi)$ as the true parameters, and initialized our
Baum-Welch training parameters randomly. For both the serial and parallel
versions, we ran 5 update iterations of the Baum-Welch algorithm.  Additionally,
we also varied the number of cores available for GraphLab. R~\cite{r} was used
for data analysis and plotting.

Our parallel code's runtime, compared to that of the serial version, is shown in
Figure~\ref{fig:runtime-256-500}. We see that, while our GraphLab implementation
runs slower than our serial implementation, the running time still decreases as
we increase the number of processing cores.

\begin{figure*}[htb]
    \centering
    \includegraphics[width=0.6\textwidth]{../figure/runtime-N_256-T_500.pdf}
    \caption{Runtime as a function of number of cores used.}
    \label{fig:runtime-256-500}
\end{figure*}

Figures for other testing conditions are shown in the appendix, in
Figures~\ref{fig:runtime1}--\ref{fig:runtime4}. In general, runtime for the
serial code remains constant, and runtime for the parallel code decreases with
more processors. In many cases, runtime increases when going from 2 to 4
cores---this may represent increased communication cost between more processors.
Noise in our plots may be caused by changes in processing load on the Odyssey
server.

We examined how our implementation scales as we increase the
problem size, especially $N$. A log-log plot of running time vs. $N$ is shown in
Figure~\ref{fig:scaling-16-200}. Judging by the slopes on the graph, our
GraphLab code scales better than the serial code for smaller $N$, but unfortunately not for other values of $N$.

\begin{figure*}[htb]
    \centering
    \includegraphics[width=0.6\textwidth]{../figure/scaling-cores_16-T_200.pdf}
    \caption{Log-Log plot of runtime as a function of number of states $N$.}
    \label{fig:scaling-16-200}
\end{figure*}

Additional figures for other testing conditions display similar trends and are
shown in the appendix, Figures~\ref{fig:scaling1}--\ref{fig:scaling4}.


\documentclass{article}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{listings}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{mwe}
\usepackage{float}
\usepackage{epstopdf}
\usepackage[top=1.5in, bottom=1in, left=1in, right=1in]{geometry}

\title{Project 2 Advanced Land Finding Writeup}

\begin{document}
\maketitle
\section{Project Goals}

The goals / steps of this project are the following:\\\\
* Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.\\
* Apply a distortion correction to raw images.\\
* Use color transforms, gradients, etc., to create a thresholded binary image.\\
* Apply a perspective transform to rectify binary image ("birds-eye view").\\
* Detect lane pixels and fit to find the lane boundary.\\
* Determine the curvature of the lane and vehicle position with respect to center.\\
* Warp the detected lane boundaries back onto the original image.\\
* Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.\\

\section{Write up}
In this section I will go over the pipeline code in project2.py
\subsection{Camera Calibration and Distortion Correction}
The first step is to calibrate the camera using the provided figures using the defined function $cameraCalibration$ and $undistort$. \\
The first function defines object and find image points for camera calibration using $cv2.findChessboardCorners()$. The second function creates undistort images witg $cv2.calibrateCamera()$ and $cv2.undistort()$. An example of the figure before and after distortion correction is shown in Figure \ref{fig1a} , Figure \ref{fig1b}, Figure \ref{fig2a} and Figure \ref{fig2b}.

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{../camera_cal/calibration2.jpg}
         \caption{Before Distortion Correction}
         \label{fig1a}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{../output_images/undistort/alibration2.jpg}
         \caption{After Distortion Correction}
         \label{fig1b}
     \end{subfigure}
\end{figure}
\subsection{Create Color Thresholded Image}

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{../test_images/test1.jpg}
         \caption{Before Distortion Correction}
         \label{fig2a}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{../output_images/undistort/test1.jpg}
         \caption{After Distortion Correction}
         \label{fig2b}
     \end{subfigure}
\end{figure}
The function $thresholdImage$ is used to create the binary threshold images. It use threshold in HLS S-Channel and Sobel X direction to create the thresholded binary image for line detection.  Additionally a HLS L-Channel threshold is added to remove the dark tree shadow that also cross S-Channel threshold.\\\\
The final threshold values are $170<HLS_S<255$, $HLS_L>=100$ and $40<Sobel_x<100$, an example of the output is shown in Figure \ref{fig3}.\\

\begin{figure}[h!t]
\begin{center}
\includegraphics[width=0.75\textwidth]{../output_images/threshold/v2test5.jpg}
\caption{Binary Thresholded Output of Test Image 5}
\label{fig3}
\end{center}
\end{figure}

\subsection{Perspective Transform "Bird-Eye View"}
The function $birdsEye$ is used to warp the image to bird-eye view. The key parameters source and destination points are defined in the following Table \ref{table1}. The values are verified by checking on a straight line as shown in Figure \ref{fig4}.\\

\begin{table}[h!t]
  \begin{center}
    \caption{Perspective Transform Parameters}
    \label{table1}
    \begin{tabular}{l|l} % <-- Alignments: 1st column left, 2nd middle and 3rd right, with vertical lines in between
      \textbf{Source Point} & \textbf{Destination Point}\\
      \hline
      (592,450) & (240,0)\\
      (688,450) & (1040,0)\\
      (1120,720) & (1040,720)\\
      (200,720) & (240,720)\\
    \end{tabular}
  \end{center}
\end{table}

\begin{figure}[h!t]
\begin{center}
\includegraphics[width=0.75\textwidth]{../output_images/birdeye/straight_lines2.jpg}
\caption{Perspective Transform on Straight Road}
\label{fig4}
\end{center}
\end{figure}

\subsection{Polynomial Fit}
The function $search_around_poly$ is the main one to warp the image to bird-eye view. Before that, the function $fit_firstPoly$ uses sliding window method (as shown in Figure \ref{fig5}) to find the line to calculate the first polynomial and update the variable $self.left\_fit$ and $self.right\_fit$.  After that, the line in the new frame is searched through with a smaller margin in an area based on the previous polynomial as  Figure \ref{fig6} shows. Then the calculated polynomial is shown in Figure \ref{fig7}.

\begin{figure}[h!t]
\begin{center}
\includegraphics[width=0.75\textwidth]{../output_images/polynomialfit/test5.jpg}
\caption{Sliding Windows}
\label{fig5}
\end{center}
\end{figure}

\begin{figure}[h!t]
\begin{center}
\includegraphics[width=0.75\textwidth]{../output_images/polynomialfit/v2test5.jpg}
\caption{Polynomial Area}
\label{fig6}
\end{center}
\end{figure}

\begin{figure}[h!t]
\begin{center}
\includegraphics[width=0.75\textwidth]{../output_images/polynomialfit/v3test5.jpg}
\caption{Polynomial Visualization}
\label{fig7}
\end{center}
\end{figure}

\subsection{Curvature and Vehicle Position Calculation}
Curvature is calculated in the function $measure\_curvature\_real$ through the following formula, where $A$ and $B$ are the coefficients in the polynomial and $y$ is the location at the bottom of the image.\\
\begin{equation}
R_{curve}=\frac{(1+(2Ay+B)^2)^{3/2}}{|2A|}
\end{equation}
Then the vehicle position is calculate in function $measure\_vehicle\_position$, subtracted the average of the two polynomial points at the bottom of the image from the center point at image bottem.

\subsection{Inverse Perspective and Adding Comment}
To finish the pipeline, A inverse perspective transformation is applied to the bird-eye view picture in function $inverse\_perspective$. Then the calculated curvature and vehicle position is added the inverse perspective transformed image through function $add\_word$. An example is shown in Figure \ref{fig8}. The output video is attached in submission

\begin{figure}[h!t]
\begin{center}
\includegraphics[width=0.75\textwidth]{../output_images/test/test5final.jpg}
\caption{Example Output Frame}
\label{fig8}
\end{center}
\end{figure}

\section{Discussion}
The part that need most calibration effort is the to create the thresholded image. When the light condition changes or more noise on the road, the resulting thresholded picture can easily include lots of unwanted frame.  To process the project video, the tree shadows create some difficulty so another restraint from the L channel in HLS space is added. But the same trick won't work on challenge video, a different set of calibration parameters in $thresholdImage$ function or update the parameters dynamically based on the overall image parameter such as lightness might work. But I could not spare more time on it. Also put all code in a overall object is another challenge for myself. I hope I have time to polish and update my code in future.

\end{document}