% ===> this file was generated automatically by noweave --- better not edit it
\section{Introduction}

As a convenience to users, we provide a mechanism for declaring
nested coordinate systems.  This is particularly useful
for creating \emph{substructures} or \emph{subnets}.  Different
instances of a subnet may have the same relative placements
of nodes, but at different translations and orientations.

\section{Interface}

We support nested coordinate systems via a \emph{transformation stack}.
The entries in this stack data structures are all affine transformations that
take the current coordinate system to the global system.  New transformations
are \emph{composed} with the previous top-of-stack, so that if
$T_0$ is the current transformation to global coordinates and a
transformation $S$ is pushed onto the stack, then $T_0 S$ will
be the new transformation from current to global coordinates.

The transformation $T = Ax + b$ is given by a table of twelve entries
which represents the components of the matrix $[A; b]$ in column major
order.  A vector $x$ is given by a table with coordinate entries at
indices 1 to 3.  Note that the table can contain additional information
(e.g. a node name).  Part of the reason the transformation functions
overwrite the input vector table is so that entries besides those
at indices 1 to 3 can be retained unchanged.

The functions provided by this module are
\begin{itemize}
  \item {\Tt{}xform{\_}push(T)\nwendquote}: compose a transformation onto the stack
  \item {\Tt{}xform{\_}pop\nwendquote}: pop a level off the transformation stack
  \item {\Tt{}top{\_}xform(x)\nwendquote}: overwrite $x$ with $T_\mathit{top} x$,
        where $T_\mathit{top}$ is the top transform on the stack
  \item {\Tt{}xform{\_}apply(T,\ x)\nwendquote}: overwrite $x$ with $Tx$.
  \item {\Tt{}xform{\_}applyA(t,\ x)\nwendquote}: overwrite $x$ with $A_T x$,
        where $A_T$ is the linear part of the affine transform $T$.
  \item {\Tt{}xform{\_}compose(T,\ S)\nwendquote}: return $TS$
  \item {\Tt{}xform{\_}identity\nwendquote}: return the identity transform
  \item {\Tt{}xform{\_}ox(r),\ xform{\_}oy(r),\ xform{\_}oz(r)\nwendquote}:
        return right-handed rotations about the coordinate axes
  \item {\Tt{}xform{\_}translate(z)\nwendquote}: return a translation by the
        vector $z$
  \item {\Tt{}subnet(f)\nwendquote}: return a function which creates a nested
        coordinate system according to any {\Tt{}ox\nwendquote}, {\Tt{}oy\nwendquote},
        and {\Tt{}oz\nwendquote} parameters, calls {\Tt{}f\nwendquote}, and then pops the
        nested coordinate system.
\end{itemize}

\section{Implementation}

\nwfilename{xformstack.nw}\nwbegincode{1}\sublabel{NW4ZsHyK-4TVv9z-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-4TVv9z-1}}}\moddef{xformstack.lua~{\nwtagstyle{}\subpageref{NW4ZsHyK-4TVv9z-1}}}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
\LA{}data~{\nwtagstyle{}\subpageref{NW4ZsHyK-3zsvuH-1}}\RA{}
\LA{}functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}\RA{}
\nwnotused{xformstack.lua}\nwendcode{}\nwbegindocs{2}\nwdocspar

\subsection{The transform stack}

The size of the stack is maintained in the field $n$.

\nwenddocs{}\nwbegincode{3}\sublabel{NW4ZsHyK-3zsvuH-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-3zsvuH-1}}}\moddef{data~{\nwtagstyle{}\subpageref{NW4ZsHyK-3zsvuH-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwenddeflinemarkup
xform_stack = \{n = 0\};

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{4}\nwdocspar

Note that we compose new transformations onto the stack, interpreting
the transformation argument as a transformation into the current coordinates
instead of a transformation into the global coordinates.

\nwenddocs{}\nwbegincode{5}\sublabel{NW4ZsHyK-nRuDO-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{\relax}{NW4ZsHyK-nRuDO-2}\nwenddeflinemarkup
function xform_push(T)
  local n = xform_stack.n + 1
  if n == 1 then
    xform_stack[n] = T
  else
    xform_stack[n] = xform_compose(xform_stack[n-1], T);
  end
  xform_stack.n = n;
end

\nwalsodefined{\\{NW4ZsHyK-nRuDO-2}\\{NW4ZsHyK-nRuDO-3}\\{NW4ZsHyK-nRuDO-4}\\{NW4ZsHyK-nRuDO-5}\\{NW4ZsHyK-nRuDO-6}\\{NW4ZsHyK-nRuDO-7}\\{NW4ZsHyK-nRuDO-8}\\{NW4ZsHyK-nRuDO-9}\\{NW4ZsHyK-nRuDO-A}\\{NW4ZsHyK-nRuDO-B}\\{NW4ZsHyK-nRuDO-C}}\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{6}\nwdocspar

\nwenddocs{}\nwbegincode{7}\sublabel{NW4ZsHyK-nRuDO-2}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-2}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-1}{NW4ZsHyK-nRuDO-3}\nwenddeflinemarkup
function xform_pop()
  xform_stack.n = xform_stack.n - 1
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{8}\nwdocspar

\nwenddocs{}\nwbegincode{9}\sublabel{NW4ZsHyK-nRuDO-3}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-3}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-2}{NW4ZsHyK-nRuDO-4}\nwenddeflinemarkup
function top_xform(x)
  local n = xform_stack.n
  if n > 0 then
    return xform_apply(xform_stack[n], x)
  else
    return x
  end
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{10}\nwdocspar

\subsection{Transformation functions}

\nwenddocs{}\nwbegincode{11}\sublabel{NW4ZsHyK-nRuDO-4}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-4}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-3}{NW4ZsHyK-nRuDO-5}\nwenddeflinemarkup
-- Overwrite x with Tx
function xform_apply(T, x)
  local y1 = T[1]*x[1] + T[4]*x[2] + T[7]*x[3] + T[10];
  local y2 = T[2]*x[1] + T[5]*x[2] + T[8]*x[3] + T[11];
  local y3 = T[3]*x[1] + T[6]*x[2] + T[9]*x[3] + T[12];
  x[1] = y1;
  x[2] = y2;
  x[3] = y3;
  return x;
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{12}\nwdocspar

\nwenddocs{}\nwbegincode{13}\sublabel{NW4ZsHyK-nRuDO-5}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-5}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-4}{NW4ZsHyK-nRuDO-6}\nwenddeflinemarkup
-- Overwrite x with A_T * x
function xform_applyA(T, x)
  local y1 = T[1]*x[1] + T[4]*x[2] + T[7]*x[3];
  local y2 = T[2]*x[1] + T[5]*x[2] + T[8]*x[3];
  local y3 = T[3]*x[1] + T[6]*x[2] + T[9]*x[3];
  x[1] = y1;
  x[2] = y2;
  x[3] = y3;
  return x;
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{14}\nwdocspar

Recall that
\[
  T(S(x)) = A_T (A_S x + b_S) + b_T = (A_T A_S) x + T(b_T)
\]

\nwenddocs{}\nwbegincode{15}\sublabel{NW4ZsHyK-nRuDO-6}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-6}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-5}{NW4ZsHyK-nRuDO-7}\nwenddeflinemarkup
-- Return TS
function xform_compose(T, S)
  local TS = \{\};
  for k = 1,4 do
    local base = 3*k-3;
    TS[base+1] = T[1]*S[base+1] + T[4]*S[base+2] + T[7]*S[base+3];
    TS[base+2] = T[2]*S[base+1] + T[5]*S[base+2] + T[8]*S[base+3];
    TS[base+3] = T[3]*S[base+1] + T[6]*S[base+2] + T[9]*S[base+3];
  end
  TS[10] = TS[10] + T[10];
  TS[11] = TS[11] + T[11];
  TS[12] = TS[12] + T[12];
  return TS;
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{16}\nwdocspar

\nwenddocs{}\nwbegincode{17}\sublabel{NW4ZsHyK-nRuDO-7}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-7}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-6}{NW4ZsHyK-nRuDO-8}\nwenddeflinemarkup
-- Return the identity
function xform_identity()
  return \{ 1, 0, 0,
           0, 1, 0,
           0, 0, 1,
           0, 0, 0 \};
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{18}\nwdocspar

Getting right hand rotations correct is always a trick.
Important note -- remember that the transformation matrix is
interpreted as column-major, but is ``typographically'' row
major.  So put on your transposing hat before reading the
following three functions.

\nwenddocs{}\nwbegincode{19}\sublabel{NW4ZsHyK-nRuDO-8}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-8}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-7}{NW4ZsHyK-nRuDO-9}\nwenddeflinemarkup
-- Return a rotation about the x axis
function xform_ox(r)
  local c = cos(r)
  local s = sin(r)
  return \{ 1,   0,  0,
           0,   c,  s,
           0,  -s,  c,
           0,   0,  0 \}
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{20}\nwdocspar

\nwenddocs{}\nwbegincode{21}\sublabel{NW4ZsHyK-nRuDO-9}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-9}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-8}{NW4ZsHyK-nRuDO-A}\nwenddeflinemarkup
-- Return a rotation about the y axis
function xform_oy(r)
  local c = cos(r)
  local s = sin(r)
  return \{ c,   0, -s,
           0,   1,  0,
           s,   0,  c,
           0,   0,  0 \}
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{22}\nwdocspar

\nwenddocs{}\nwbegincode{23}\sublabel{NW4ZsHyK-nRuDO-A}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-A}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-9}{NW4ZsHyK-nRuDO-B}\nwenddeflinemarkup
-- Return a rotation about the z axis
function xform_oz(r)
  local c = cos(r)
  local s = sin(r)
  return \{ c,   s,  0,
          -s,   c,  0,
           0,   0,  1,
           0,   0,  0 \}
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{24}\nwdocspar

\nwenddocs{}\nwbegincode{25}\sublabel{NW4ZsHyK-nRuDO-B}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-B}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-A}{NW4ZsHyK-nRuDO-C}\nwenddeflinemarkup
-- Return a translation
function xform_translate(z)
  return \{ 1,    0,    0,
           0,    1,    0,
           0,    0,    1,
           z[1], z[2], z[3] \}
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{26}\nwdocspar

\subsection{{\Tt{}node\nwendquote} and {\Tt{}subnetize\nwendquote} functions}

We assume that node positions are expressed in the current coordinate
system (at least, that's what we assume for most purposes).
So the {\Tt{}nodex\nwendquote} (``node transformed'') function transforms
the input coordinates from local to global, and them makes a node.

We still want to leave the option of expressing node coordinates
directly in the global coordinate system, though.  For instance,
we may want to put a node halfway between two other nodes which
have already been transformed into global coordinates.  For this
reason, we leave the {\Tt{}node\nwendquote} function alone.

\nwenddocs{}\nwbegincode{27}\sublabel{NW4ZsHyK-nRuDO-C}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-C}}}\moddef{functions~{\nwtagstyle{}\subpageref{NW4ZsHyK-nRuDO-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-4TVv9z-1}}\nwprevnextdefs{NW4ZsHyK-nRuDO-B}{\relax}\nwenddeflinemarkup
function nodex(p)
  if p[1] then
    top_xform(p)
  end
  return node(p)
end

\nwused{\\{NW4ZsHyK-4TVv9z-1}}\nwendcode{}\nwbegindocs{28}\nwdocspar

\subsection{Test code}

\nwenddocs{}\nwbegincode{29}\sublabel{NW4ZsHyK-pcrvx-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-pcrvx-1}}}\moddef{xformtst.lua~{\nwtagstyle{}\subpageref{NW4ZsHyK-pcrvx-1}}}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
use("xformstack.lua")

\LA{}test transform constructors~{\nwtagstyle{}\subpageref{NW4ZsHyK-1zasbh-1}}\RA{}
\LA{}test composition~{\nwtagstyle{}\subpageref{NW4ZsHyK-2X3VkV-1}}\RA{}
\nwnotused{xformtst.lua}\nwendcode{}\nwbegindocs{30}\nwdocspar

\nwenddocs{}\nwbegincode{31}\sublabel{NW4ZsHyK-1zasbh-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-1zasbh-1}}}\moddef{test transform constructors~{\nwtagstyle{}\subpageref{NW4ZsHyK-1zasbh-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-pcrvx-1}}\nwenddeflinemarkup
-- First check out the basics

rx45 = xform_ox(45)
ry45 = xform_oy(45)
rz45 = xform_oz(45)
trans = xform_translate \{1, 2, 3\}

x1 = xform_apply(rx45,  \{1, 0, 0\})
x2 = xform_apply(ry45,  \{1, 0, 0\})
x3 = xform_apply(rz45,  \{1, 0, 0\})
x4 = xform_apply(trans, \{1, 0, 0\})
x5 = xform_applyA(trans, \{1, 0, 0\})

print("Rotate e1 by rx45: ",     x1[1], x1[2], x1[3])
print("Rotate e1 by ry45: ",     x2[1], x2[2], x2[3])
print("Rotate e1 by rz45: ",     x3[1], x3[2], x3[3])
print("Translate e1 by 1,2,3: ", x4[1], x4[2], x4[3])
print("Apply A e1 by 1,2,3: ",   x5[1], x5[2], x5[3])

\nwused{\\{NW4ZsHyK-pcrvx-1}}\nwendcode{}\nwbegindocs{32}\nwdocspar

\nwenddocs{}\nwbegincode{33}\sublabel{NW4ZsHyK-2X3VkV-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW4ZsHyK-2X3VkV-1}}}\moddef{test composition~{\nwtagstyle{}\subpageref{NW4ZsHyK-2X3VkV-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW4ZsHyK-pcrvx-1}}\nwenddeflinemarkup
-- Now check out composition

T = xform_compose(trans, rz45)
x = xform_apply(T, \{1, 0, 0\})

print("Rotate then translate: ", x[1], x[2], x[3])

undoT = xform_compose(xform_oz(-45), xform_translate\{-1,-2,-3\})
xform_apply(undoT, x)

print("After undo operation: ", x[1], x[2], x[3])

\nwused{\\{NW4ZsHyK-pcrvx-1}}\nwendcode{}

\nwixlogsorted{c}{{data}{NW4ZsHyK-3zsvuH-1}{\nwixu{NW4ZsHyK-4TVv9z-1}\nwixd{NW4ZsHyK-3zsvuH-1}}}%
\nwixlogsorted{c}{{functions}{NW4ZsHyK-nRuDO-1}{\nwixu{NW4ZsHyK-4TVv9z-1}\nwixd{NW4ZsHyK-nRuDO-1}\nwixd{NW4ZsHyK-nRuDO-2}\nwixd{NW4ZsHyK-nRuDO-3}\nwixd{NW4ZsHyK-nRuDO-4}\nwixd{NW4ZsHyK-nRuDO-5}\nwixd{NW4ZsHyK-nRuDO-6}\nwixd{NW4ZsHyK-nRuDO-7}\nwixd{NW4ZsHyK-nRuDO-8}\nwixd{NW4ZsHyK-nRuDO-9}\nwixd{NW4ZsHyK-nRuDO-A}\nwixd{NW4ZsHyK-nRuDO-B}\nwixd{NW4ZsHyK-nRuDO-C}}}%
\nwixlogsorted{c}{{test composition}{NW4ZsHyK-2X3VkV-1}{\nwixu{NW4ZsHyK-pcrvx-1}\nwixd{NW4ZsHyK-2X3VkV-1}}}%
\nwixlogsorted{c}{{test transform constructors}{NW4ZsHyK-1zasbh-1}{\nwixu{NW4ZsHyK-pcrvx-1}\nwixd{NW4ZsHyK-1zasbh-1}}}%
\nwixlogsorted{c}{{xformstack.lua}{NW4ZsHyK-4TVv9z-1}{\nwixd{NW4ZsHyK-4TVv9z-1}}}%
\nwixlogsorted{c}{{xformtst.lua}{NW4ZsHyK-pcrvx-1}{\nwixd{NW4ZsHyK-pcrvx-1}}}%
\nwbegindocs{34}\nwdocspar

\nwenddocs{}


\lab{Pandas 1: Introduction}{Pandas 1: Introduction}
\objective{Though NumPy and SciPy are powerful tools for numerical computing, they lack some of the high-level functionality necessary for many data science applications.
Python's \emph{pandas} library, built on NumPy, is designed specifically for data management and analysis.
In this lab, we introduce pandas data structures, syntax, and explore its capabilities for quickly analyzing and presenting data.
}
\label{lab:pandas1}

% Just as NumPy is built on the \li{ndarray} data structure suited for efficient
% scientific and numerical computation, pandas is centered around a handful of
% core data structures custom built for data analysis: the \li{Series}, \li{DataFrame}, and \li{Panel}, which roughly correspond to one, two, and three-dimensional arrays.
% The \li{Panel} data structure is not used nearly

\begin{info}
This lab will be done using Colab Notebooks.
These notebooks are similar to Jupyter Notebooks but run remotely on Google's servers.
Open a Google Colab notebook by going to your Google Drive account and creating a new Colaboratory file.
If making a Colaboratory file is not an option, download the application Colaboratory onto your Google Drive.
Once opening a new Colab Notebook, upload the file \li{pandas1.ipynb}.
To make the data files accessible, run the following at the top of the lab:
\begin{lstlisting}
>>> from google.colab import files

>>> uploaded = files.upload()
\end{lstlisting}
This will prompt you upload files for this notebook.
For this lab, upload \li{budget.csv} and \li{crime_data.csv}.

Once the lab is complete, delete BOTH lines of code used for uploading files (the import statement and the upload statement) and download as a \li{.py} file to your git repository.
Push the newly made \li{pandas1.py} file.
\end{info}

\section*{Pandas Basics}

Pandas is a python library used primarily to analyze data.
It combines functionality of NumPy, MatPlotLib, and SQL to create a easy to understand library that allows for the manipulation of data in various ways.
In this lab, we focus on the use of Pandas to analyze and manipulate data in ways similar to NumPy and SQL.

\subsection*{Pandas Data Structures}

\subsubsection*{Series}

The first pandas data structure is a \li{Series}.
A \li{Series} is a one-dimensional array that can hold any datatype, similar to a \li{ndarray}.
However, a \li{Series} has an \li{index} that gives a label to each entry.
An \li{index} generally is used to label the data.

Typically a \li{Series} contains information about one feature of the data.
For example, the data in a \li{Series} might show a class's grades on a test and the \li{Index} would indicate each student in the class.
To initialize a \li{Series}, the first parameter is the data and the second is the index.

\begin{lstlisting}
>>> import pandas as pd
>>>
>>> # Initialize Series of student grades
>>> math = pd.Series(np.random.randint(0,100,4), ['Mark', 'Barbara', 'Eleanor', 'David'])
>>> english = pd.Series(np.random.randint(0,100,5), ['Mark', 'Barbara', 'David', 'Greg', 'Lauren'])
\end{lstlisting}

\subsubsection*{DataFrame}

The second key pandas data structure is a \li{DataFrame}.
A \li{DataFrame} is a collection of multiple \li{Series}.
It can be thought of as a 2-dimensional array, where each row is a separate datapoint and each column is a feature of the data.
The rows are label with an \li{index} (as in a \li{Series}) and the columns are labelled in the attribute \li{columns}.

There are many different ways to initialize a \li{DataFrame}.
One way to initialize a \li{DataFrame} is passing in a dictionary as the data of the \li{DataFrame}.
The keys of the dictionary will become the labels in \li{columns} and the values are the \li{Series} associated with the label.

\begin{lstlisting}
>>> # Create a DataFrame of student grades
>>> grades = pd.DataFrame({"Math": math, "English": english}
>>> grades
	      Math  English
Barbara   52.0     73.0
David     10.0     39.0
Eleanor   35.0      NaN
Greg       NaN     26.0
Lauren     NaN     99.0
Mark      81.0     68.0
\end{lstlisting}

Notice that \li{pd.DataFrame} automatically lines up data from both \li{Series} that have the same index.
If the data only appears in one of the \li{Series}, the entry for the second \li{Series} is \li{NaN}.

We can also initialize a \li{DataFrame} with a NumPy array.
In this way, the data is passed in as a 2-dimensional NumPy array, while the column labels and index are passed in as parameters.
The first column label goes with the first column of the array, the second with the second, etc.
The same holds for the index.

\begin{lstlisting}
>>> import numpy as np
>>> # Initialize DataFrame with NumPy array
>>> data = np.array([[52.0, 73.0], [10.0, 39.0], [35.0, np.nan], [np.nan, 26.0], [np.nan, 99.0], [81.0, 68.0]])
>>> grades = pd.DataFrame(data, columns = ['Math', 'English'], index = ['Barbara', 'David', 'Eleanor', 'Greg', 'Lauren', 'Mark'])
>>> grades
	      Math  English
Barbara   52.0     73.0
David     10.0     39.0
Eleanor   35.0     NaN
Greg       NaN     26.0
Lauren     NaN     99.0
Mark      81.0     68.0
\end{lstlisting}

A \li{DataFrame} can also be viewed as a NumPy array using the attribute \li{values}.

\begin{lstlisting}
>>> # View the DataFrame as a NumPy array
>>> grades.values
array([[ 52.,  73.],
       [ 10.,  39.],
       [ 35.,  nan],
       [ nan,  26.],
       [ nan,  99.],
       [ 81.,  68.]])
\end{lstlisting}

\begin{problem}
Write a function \li{random_dataframe()} that accepts a dictionary \li{d} which defaults to \li{None}.
If a dictionary is passed in, initialize a Pandas \li{DataFrame}.
Return a tuple of the attributes \li{index}, \li{columns}, and \li{values} of the \li{DataFrame}.

If a dictionary is not passed in, generate random data as a \li{ndarray} and initialize a \li{DataFrame}.
The columns of the \li{DataFrame} should be the letters \li{'A'} through \li{'E'}.
The index of the \li{DataFrame} should be the roman numerals 1-6.
Return a tuple of the attributes \li{index}, \li{columns}, and \li{values} of the \li{DataFrame}.

(Hint: What should the dimension of the data be if no dictionary is passed in?)
\end{problem}

\begin{comment}
The Pandas python library is used to manipulate and analyze data.
The two main pandas structures are \li{Series} and \li{DataFrames}.
They are built off of \li{ndarrays} and thereby carry many functionalities of arrays.
A \li{Series} is a one-dimensional array that can hold any datatype.
Unlike a NumPy array, every \li{Series} has an \emph{index} that labels each entry, and a \li{Series} object can also be given a name to label the entire data set.
A group of \li{Series} makes a 2-d structure called a \li{DataFrame}.
A \li{DataFrame} is like a table while the \li{Series} are the columns.
The row labels are collectively called the \emph{index}, and the column labels are collectively called the \emph{columns}.
There are many ways to initialize a \li{DataFrame}.
In the following code, we build a \li{DataFrame} out of a dictionary of \li{Series}.

\begin{lstlisting}
>>> x = pd.Series(np.random.randn(4), ['a', 'b', 'c', 'd'])
>>> y = pd.Series(np.random.randn(5), ['a', 'b', 'd', 'e', 'f'])
>>> d = {"series 1": x, "series 2": y}
>>> df1 = pd.DataFrame(d)
>>> df1
<<   series 1  series 2
a -0.365542  1.227960
b  0.080133  0.683523
c  0.737970       NaN
d  0.097878 -1.102835
e       NaN  1.345004
f       NaN  0.217523>>
\end{lstlisting}

The index of this \li{DataFrame} is the union of the indices of \li{Series} \li{x} and \li{Series} \li{y}.
The columns are given by the keys of the dictionary \li{d}.
Since \li{x} doesn't have a label \li{e}, the value in row \li{e}, column \li{1} is \li{NaN}.
This same reasoning explains the other missing values as well.
We can recover the \li{Series} \li{x} by slicing using the axis label for the first column on the \li{DataFrame} and \li{dropna()}, which removes the \li{NaN}:
 \begin{lstlisting}
>>> df1["series1"].dropna()
<<a   -0.365542
b    0.080133
c    0.737970
d    0.097878
Name: series 1, dtype: float64>>
\end{lstlisting}

\begin{warn}
A pandas \li{DataFrame} cannot be sliced in exactly the same way as a NumPy array.
Notice how we just used \li{df1["series 1"]} to access a \emph{column} of the
the \li{DataFrame} \li{df1}.
We will discuss this in more detail later on.
\end{warn}

We can also initialize a \li{DataFrame} using a NumPy array, creating custom
row and column labels.

\begin{lstlisting}
>>> data = np.random.random((3, 4))
>>> pd.DataFrame(data, index=['A', 'B', 'C'], columns=np.arange(1, 5))
             1         2         3         4
A  0.065646  0.968593  0.593394  0.750110
B  0.803829  0.662237  0.200592  0.137713
C  0.288801  0.956662  0.817915  0.951016
3 rows     4 columns
\end{lstlisting}

If we don't specify the index or columns, the default is
\li{np.arange(n)}, where \li{n} is either the number of rows or columns.
\end{comment}

\subsection*{Data I/O}

The pandas library has functions that make importing and exporting data simple.
The functions allow for a variety of file formats to be imported and exported, including CSV, Excel, HDF5, SQL, JSON, HTML, and pickle files.

\begin{table}[H]
\begin{tabular}{r|l}
Method & Description \\ \hline
\begin{comment}
\li{describe()}  & Return a \li{Series} describing the data structure \\
\li{head()}      & Return the first $n$ rows, defaulting to 5 \\
\li{tail()}      & Return the last $n$ rows, defaulting to 5 \\
\end{comment}
\li{to_csv()}    & Write the index and entries to a CSV file \\
\li{to_json()}   & Convert the object to a JSON string \\
\li{to_pickle()} & Serialize the object and store it in an external file \\
\li{to_sql()}    & Write the object data to an open SQL database \\
\end{tabular}
\caption{Methods for exporting data in a pandas \li{Series} or \li{DataFrame}.}
\label{table:pandas-view-or-export}
\end{table}

The CSV (comma separated values) format is a simple way of storing tabular data
in plain text.
Because CSV files are one of the most popular file formats for
exchanging data, we will explore the \li{read_csv()} function in more detail.
To learn to read other types of file formats, see the online pandas
documentation.
To read a CSV data file into a \li{DataFrame}, call the
\li{read_csv()} function with the path to the CSV file, along with the
appropriate keyword arguments.
Below we list some of the most important keyword
arguments:

\begin{itemize}
\item \li{delimiter}:
The character that separates data fields. It is often a
comma or a whitespace character.
\item \li{header}:
The row number (0 indexed) in the CSV file that contains the column names.
 \item \li{index_col}:
The column (0 indexed) in the CSV file that is the index for the
\li{DataFrame}.
 \item \li{skiprows}:
If an integer $n$, skip the first $n$ rows of the file, and then start reading
in the data.
If a list of integers, skip the specified rows.
 \item \li{names}:
If the CSV file does not contain the column names, or you wish to use other
column names, specify them in a list.
\end{itemize}

\begin{comment}
\begin{problem}
Write a function \li{prob2()} that accepts a filename.
Check whether the file is in CSV format and read the data in appropriately.
If the file is not in CSV format, raise a \li{TypeError}.

\label{prob:read-csv}
\end{problem}

To save data, pandas has functions that write to several different
file formats.
A typical example is the \li{to_csv()} function method attached to
\li{Series} and \li{DataFrame} objects, which writes the data to a CSV file.
Keyword arguments allow you to specify the separator character, omit writing the
columns names or index, and specify many other options.
The code below
demonstrates its typical usage:
\begin{lstlisting}
>>> df.to_csv("my_df.csv")
\end{lstlisting}
\end{comment}

\section*{Data Manipulation}

\subsection*{Accessing Data}

While array slicing can be used to access data in a \li{DataFrame}, it is always preferable to use the \li{loc} and \li{iloc} indexers.
Accessing \li{Series} and \li{DataFrame} objects
using these indexing operations is more efficient than slicing because the bracket indexing has to check many cases before it can determine how to slice the data structure.
Using \li{loc}/\li{iloc} explicitly, bypasses the extra checks.
The \li{loc} index selects rows and columns based on
their labels, while \li{iloc} selects them based on their
integer position.
When using these indexers, the first and second arguments refer to the rows and columns, respectively, just as array slicing.

\begin{lstlisting}
>>> grades
	      Math  English
Barbara   52.0     73.0
David     10.0     39.0
Eleanor   35.0     NaN
Greg       NaN     26.0
Lauren     NaN     99.0
Mark      81.0     68.0

>>> # Use loc to select the Math scores of David and Greg
>>> grades.loc[['David', 'Greg'],'Math']
David    10.0
Greg      NaN
Name: Math, dtype: float64

>>> # Use iloc to select the Math scores of David and Greg
>>> grades.iloc[[1,3], 0]
David    10.0
Greg      NaN
\end{lstlisting}

An entire column of a \li{DataFrame} can be accessed using simple square
brackets and the name of the column.
In addition, to create a new column or reset the values of an entire column, simply call this column in the same fashion and set the value.

\begin{lstlisting}
>>> # Set new History column with array of random values
>>> grades['History'] = np.random.randint(0,100,6)
>>> grades['History']
Barbara     4
David      92
Eleanor    25
Greg       79
Lauren     82
Mark       27
Name: History, dtype: int64

>>> # Reset the column such that everyone has a 100
>>> grades['History'] = 100.0
>>> grades
         Math  English  History
Barbara  52.0     73.0    100.0
David    10.0     39.0    100.0
Eleanor  35.0      NaN    100.0
Greg      NaN     26.0    100.0
Lauren    NaN     99.0    100.0
Mark     81.0     68.0    100.0
\end{lstlisting}

Often datasets can be very large and difficult to visualize.
Pandas offers various methods to make the data easier to visualize.
The methods \li{head} and \li{tail} will show the first or last $n$ data points, respectively, where $n$ defaults to 5.
The method \li{sample} will draw $n$ random entry of the dataset, where $n$ defaults to 1.

\begin{lstlisting}
>>> # Use head to see the first n rows
>>> grades.head(n=2)
         Math  English  History
Barbara  52.0     73.0    100.0
David    10.0     39.0    100.0

>>> # Use sample to sample a random entry
>>> grades.sample()
        Math  English  History
Lauren   NaN     99.0    100.0
\end{lstlisting}

It may also be useful to re-order the columns or rows or sort according to a
given column.

\begin{lstlisting}
>>> # Re-order columns
>>> grades.reindex(columns['English','Math','History'])
         English  Math  History
Barbara     73.0  52.0    100.0
David       39.0  10.0    100.0
Eleanor      NaN  35.0    100.0
Greg        26.0   NaN    100.0
Lauren      99.0   NaN    100.0
Mark        68.0  81.0    100.0

>>> # Sort descending according to Math grades
>>> grades.sort_values('Math', ascending=False)
         Math  English  History
Mark     81.0     68.0    100.0
Barbara  52.0     73.0    100.0
Eleanor  35.0      NaN    100.0
David    10.0     39.0    100.0
Greg      NaN     26.0    100.0
Lauren    NaN     99.0    100.0
\end{lstlisting}

Other methods used for manipulating \li{DataFrame} and \li{Series} panda structures can be found in Table \ref{table:pandas-manage-data}.

\begin{table}[H]
\begin{tabular}{r|l}
Method & Description \\ \hline
\li{append()} & Concatenate two or more \li{Series}. \\
\li{drop()} & Remove the entries with the specified label or labels \\
\li{drop_duplicates()} & Remove duplicate values \\
\li{dropna()} & Drop null entries \\
\li{fillna()} & Replace null entries with a specified value or strategy \\
\li{reindex()} & Replace the index \\
\li{sample()} & Draw a random entry \\
\li{shift()} & Shift the index \\
% \li{truncate()} & \\
\li{unique()} & Return unique values \\
% \li{where()} & Search for entires that satisfy some criteria \\
\end{tabular}
\caption{Methods for managing or modifying data in a pandas \li{Series} or \li{DataFrame}.}
\label{table:pandas-manage-data}
\end{table}

\begin{problem}
The file \li{budget.csv} contains the budget of a college student over the course of 4 years.
Write a function \li{prob2()} reads in \li{budget.csv} as a \li{DataFrame}.
Perform the following operations:
\begin{enumerate}
\item Reindex the columns such that amount spent on food is the first column and all other columns maintain the same ordering.
\item Sort the \li{DataFrame} in descending order based on how much money was spent on \li{Groceries}
\item Reset all values in the \li{'Rent'} column to \li{800.0}
\item Reset all values in the first 5 data points to \li{0.0}
\end{enumerate}
Return the values of the updated \li{DataFrame} as a NumPy array.
\label{prob:budget}
\end{problem}

\subsection*{Basic Data Manipulation}
Because the primary pandas data structures are subclasses of \li{ndarray}, most NumPy functions work with pandas structure.
For example, basic
vector operations work as expected:

\begin{lstlisting}
>>> # Sum history and english grades of all students
>>> grades['English'] + grades['History']
Barbara    173.0
David      139.0
Eleanor      NaN
Greg       126.0
Lauren     199.0
Mark       168.0
dtype: float64

>>> # Double all Math grades
>>>  grades['Math']*2
Barbara    104.0
David       20.0
Eleanor     70.0
Greg         NaN
Lauren       NaN
Mark       162.0
Name: Math, dtype: float64
\end{lstlisting}
In addition to arithmetic, \li{Series} have a variety of other methods similar to NumPy arrays.
A collection of these methods is found in Table \ref{table:pandas-numerical-methods}.
\begin{table}[H]
\begin{tabular}{r|l}
Method & Returns \\ \hline
\li{<<abs>>()}     & Object with absolute values taken (of numerical data) \\
\li{idxmax()}  & The index label of the maximum value \\
\li{idxmin()}  & The index label of the minimum value \\
\li{count()}   & The number of non-null entries \\
\li{cumprod()} & The cumulative product over an axis \\
\li{cumsum()}  & The cumulative sum over an axis \\
\li{<<max>>()}     & The maximum of the entries \\
\li{mean()}    & The average of the entries \\
\li{median()}  & The median of the entries \\
\li{<<min>>()}     & The minimum of the entries \\
\li{mode()}    & The most common element(s) \\
\li{prod()}    & The product of the elements \\
\li{<<sum>>()}     & The sum of the elements \\
\li{var()}     & The variance of the elements \\
\end{tabular}
\caption{Numerical methods of the \li{Series} and \li{DataFrame} pandas classes.
% Methods marked with a * are not methods of NumPy's \li{ndarray} class.
}
\label{table:pandas-numerical-methods}
\end{table}

%The default
% missing value \li{NaN} is given for labels that are not shared by both inputs.

\subsection*{Basic Statistical Functions}

The pandas library allows us to easily calculate basic summary statistics of our data,
useful when we want a quick description of the data.
The \li{describe()} function
outputs several such summary statistics for each column in a \li{DataFrame}:
\begin{lstlisting}
>>> # Use describe to better understand the data
>>> grades.describe()
            Math   English  History
count   4.000000   5.00000      6.0
mean   44.500000  61.00000    100.0
std    29.827281  28.92231      0.0
min    10.000000  26.00000    100.0
25%    28.750000  39.00000    100.0
50%    43.500000  68.00000    100.0
75%    59.250000  73.00000    100.0
max    81.000000  99.00000    100.0
\end{lstlisting}

Functions for calculating means and variances, the covariance and correlation matrices, and other
basic statistics are also available.

\begin{lstlisting}
>>> # Find the average grade for each student
>>> grades.mean(axis=1)
Barbara    75.000000
David      49.666667
Eleanor    67.500000
Greg       63.000000
Lauren     99.500000
Mark       83.000000
dtype: float64

>>> # Solve for the unbiased variance between subjects
>>> grades.cov()
               Math  English  History
Math     889.666667    557.0      0.0
English  557.000000    836.5      0.0
History    0.000000      0.0      0.0

>>> # Give correlation matrix between subjects
>>> grades.corr()
            Math  English  History
Math     1.00000  0.84996      NaN
English  0.84996  1.00000      NaN
History      NaN      NaN      NaN
\end{lstlisting}

The method \li{rank} gives a ranking based on methods such as average, minimum, and maximum.
This method defaults ranking in ascending order (the least will be ranked 1 and the greatest will be ranked the highest number).

\begin{lstlisting}
>>> # Rank each student's performance based on their highest grade in any class in descending order
>>> grades.rank(axis=1,method='max',ascending=False)
         Math  English  History
Barbara   3.0      2.0      1.0
David     3.0      2.0      1.0
Eleanor   2.0      NaN      1.0
Greg      NaN      2.0      1.0
Lauren    NaN      2.0      1.0
Mark      2.0      3.0      1.0
\end{lstlisting}

These methods can be very effective in interpreting data.
For example, the \li{rank} example above shows use that Barbara does best in History, then English and then Math.

\subsection*{Dealing with Missing Data}

Missing data is a ubiquitous problem in data science.
Fortunately, pandas is particularly well-suited to
handling missing and anomalous data.
As we have already seen, the pandas default for a missing value is \li{NaN}.
In basic arithmetic operations, if one of the operands is \li{NaN}, then the output is also \li{NaN}.
If we are not interested in the missing values, we can simply drop them from the data altogether:

\begin{lstlisting}
>>> # Grades with all NaN values dropped
>>> grades.dropna()
         Math  English  History
Barbara  52.0     73.0    100.0
David    10.0     39.0    100.0
Mark     81.0     68.0    100.0
\end{lstlisting}

This is not always the desired behavior, however.
Missing data could actually correspond to some default value, such as zero.
For example, in the budget dataset, filling \li{NaN} value with 0 indicates that no money was spent on that item.
In the grade dataset, we can replace all instances of \li{NaN} with a specified value:
\begin{lstlisting}
>>> # fill missing data with 50.0
>>> grades.fillna(50.0)
         Math  English  History
Barbara  52.0     73.0    100.0
David    10.0     39.0    100.0
Eleanor  35.0     50.0    100.0
Greg     50.0     26.0    100.0
Lauren   50.0     99.0    100.0
Mark     81.0     68.0    100.0
\end{lstlisting}

When dealing with missing data, make sure you are aware of the behavior of the pandas functions you are using.
For example, \li{<<sum()>>} and \li{mean()} ignore NaN values in the computation.

\begin{warn}
Always consider missing data carefully when analyzing a dataset.
It may not always be helpful to drop the data or fill it in with a random number.
Consider filling the data with the mean of surrounding data or the mean of the feature in question.
Overall, the choice for how to fill missing data should make sense with the dataset.
\end{warn}

\begin{problem}
Write a function \li{prob3()} that uses \li{budget.csv} to answer the questions "Which category affects living expenses the most? Which affects other expenses the most? How much is generally spent in these two categories?".
Use the functions above to manipulate the data to perform the following manipulations:
\begin{enumerate}
\item Fill all \li{NaN} values with \li{0.0}.
\item Create two new columns, \li{'Living Expenses'} and \li{'Other'}.
Sum the columns \li{'Rent', 'Groceries', 'Gas'} and \li{'Utilities'} and set as the value of \li{'Living Expenses'}.
Sum the columns \li{'Dining Out', 'Out With Friends'} and \li{'Netflix'} and set as the value of \li{'Other'}.
\item  Identify which column correlates most with \li{'Living Expenses'} and which correlates most with \li{'Other'}.
This can indicate which columns in the budget affects the overarching categories the most.
\end{enumerate}
Return the mean of each the two columns found in 3 as a tuple.
The first mean should be of the column corresponding to \li{'Living Expenses'} and the second to \li{'Other'}.
\end{problem}

\begin{comment}
\begin{problem}
Read in the file \li{crime_data.csv} as a pandas object.
The file contains data on types of crimes in the U.S. from 1960 to 2016.
Set the index as the column 'Year'.

Create a new column \li{Rate} which contains the crime rate for each year.
Use this table to answer the following questions:
\begin{enumerate}
    \item What is the mean crime rate? In what years was the crime rate above the mean?
    \item On average, using the mean, what is the least common crime?
    \item Identify two crimes which have close means. Do these crimes have any kind of correlation from year to year? If so, what is the correlation?

\end{enumerate}
\end{problem}
\end{comment}

\begin{comment}
\subsection*{Series}

A pandas \li{Series} is generalization of a one-dimensional NumPy array, or the column of a DataFrame.
Like a NumPy array, every \li{Series} has a data type (\li{dtype}), and the entries of the \li{Series} are all of that type.
Unlike a NumPy array, every \li{Series} has an \emph{index} that labels each entry, and a \li{Series} object can also be given a name to label the entire data set.

\begin{lstlisting}
>>> import numpy as np
>>> import pandas as pd

# Initialize a Series of random entries with an index of letters.
>>> pd.Series(np.random.random(4), index=['a', 'b', 'c', 'd'])
<<a    0.474170
b    0.106878
c    0.420631
d    0.279713
dtype: float64>>

# The default index is integers from 0 to the length of the data.
>>> pd.Series(np.random.random(4), name="uniform draws")
<<0    0.767501
1    0.614208
2    0.470877
3    0.335885
Name: uniform draws, dtype: float64>>
\end{lstlisting}

The index in a \li{Series} is a pandas object of type \li{Index} and is stored as the \li{index} attribute of the \li{Series}.
The plain entries in the \li{Series} are stored as a NumPy array and can be accessed as such via the \li{values} attribute.

\begin{lstlisting}
>>> s1 = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd'], name="some ints")

>>> s1.values                           # Get the entries as a NumPy array.
<<array([1, 2, 3, 4])>>

>>> print(s1.name, s1.dtype, sep=", ")  # Get the name and dtype.
<<some ints, int64>>

>>> s1.index                            # Get the pd.Index object.
<<Index(['a', 'b', 'c', 'd'], dtype='object')>>
\end{lstlisting}

The elements of a \li{Series} can be accessed by either the regular position-based integer index, or by the corresponding label in the index.
New entries can be added dynamically as long as a valid index label is provided, similar to adding a new key-value pair to a dictionary.
A \li{Series} can also be initialize from a dictionary: the keys become the index labels, and the values become the entries.

\begin{lstlisting}
>>> s2 = pd.Series([10, 20, 30], index=["apple", "banana", "carrot"])
>>> s2
<<apple     10
banana    20
carrot    30
dtype: int64>>

# s2[0] and s2["apple"] refer to the same entry.
>>> print(s2[0], s2["apple"], s2["carrot"])
<<10 10 30>>

>>> s2[0] += 5              # Change the value of the first entry.
>>> s2["dewberry"] = 0      # Add a new value with label 'dewberry'.
>>> s2
<<apple       15
banana      20
carrot      30
dewberry     0
dtype: int64>>

# Initialize a Series from a dictionary.
>>> pd.Series({"eggplant":3, "fig":5, "grape":7}, name="more foods")
<<eggplant    3
fig         5
grape       7
Name: more foods, dtype: int64>>
\end{lstlisting}

Slicing and fancy indexing also work the same way in \li{Series} as in NumPy arrays.
In addition, multiple entries of a \li{Series} can be selected by indexing a list of labels in the index.

\begin{lstlisting}
>>> s3 = pd.Series({"lions":2, "tigers":1, "bears":3}, name="oh my")
>>> s3
<<bears     3
lions     2
tigers    1
Name: oh my, dtype: int64>>

# Get a subset of the data by regular slicing.
>>> s3[1:]
<<lions     2
tigers    1
Name: oh my, dtype: int64>>

# Get a subset of the data with fancy indexing.
>>> s3[np.array([len(i) == 5 for i in s3.index])]
<<bears    3
lions    2
Name: oh my, dtype: int64>>

# Get a subset of the data by providing several index labels.
>>> s3[ ["tigers", "bears"] ]
tigers    1                     # Note that the entries are reordered,
bears     3                     # and the name stays the same.
<<Name: oh my, dtype: int64>>
\end{lstlisting}

\begin{comment} % Unnecessary, already getting too long.
We can create a \li{Series} having constant values in the following manner:
\begin{lstlisting}
>>> val = 4     #desired constant value of Series
>>> n = 6       #desired length of Series
>>> s3 = pd.Series(val, index=np.arange(n))
>>> s3
0    4
1    4
2    4
3    4
4    4
5    4
dtype: int64
\end{lstlisting}

\begin{problem}
Create a pandas \li{Series} where the index labels are the even integers $0,2,\ldots,50$, and the entries are $n^2 - 1$, where $n$ is the entry's label.
Set all of the entries equal to zero whose labels are divisible by $3$.
\end{problem}

\subsection*{Data Manipulation with Series} % ----------------------------------------

A \li{Series} object, or column of a \li{DataFrame}, has all of the advantages of a NumPy array, including entry-wise arithmetic, plus a few additional features (see Table \ref{table:pandas-numerical-methods}).

\begin{lstlisting}
>>> s4 = pd.Series([1, 2, 4], index=['a', 'c', 'd'])
>>> s5 = pd.Series([10, 20, 40], index=['a', 'b', 'd'])
>>> 2*s4 + s5
<<a    12.0>>
<<b     NaN>>                           # s4 doesn't have an entry for b, and
<<c     NaN>>                           # s5 doesn't have an entry for c, so
<<d    48.0>>                           # the combination is Nan (np.nan / None).
<<dtype: float64>>
\end{lstlisting}

\begin{table}[H]
\begin{tabular}{r|l}
Method & Returns \\ \hline
\li{<<abs>>()}     & Object with absolute values taken (of numerical data) \\
\li{idxmax()}  & The index label of the maximum value \\
\li{idxmin()}  & The index label of the minimum value \\
\li{count()}   & The number of non-null entries \\
\li{cumprod()} & The cumulative product over an axis \\
\li{cumsum()}  & The cumulative sum over an axis \\
\li{<<max>>()}     & The maximum of the entries \\
\li{mean()}    & The average of the entries \\
\li{median()}  & The median of the entries \\
\li{<<min>>()}     & The minimum of the entries \\
\li{mode()}    & The most common element(s) \\
\li{prod()}    & The product of the elements \\
\li{<<sum>>()}     & The sum of the elements \\
\li{var()}     & The variance of the elements \\
\end{tabular}
\caption{Numerical methods of the \li{Series} and \li{DataFrame} pandas classes.
% Methods marked with a * are not methods of NumPy's \li{ndarray} class.
}
\label{table:pandas-numerical-methods}
\end{table}

\li{Series} are more useful than NumPy arrays when dealing with data primarily because of their index.
For example, a \li{Series} can be indexed by time with a pandas \li{DatetimeIndex}, an index with date and/or time values.
The usual way to create this kind of index is with \li{pd.date_range()}.

\begin{lstlisting}
# Make an index of the first t11	`hree days in July 2000.
>>> pd.date_range("7/1/2000", "7/3/2000", freq='D')
<<DatetimeIndex(['2000-07-01', '2000-07-02', '2000-07-03'],
                dtype='datetime64[ns]', freq='D')>>
\end{lstlisting}

\end{comment}

\begin{comment}
\begin{info}
The \li{Series} in Problem \ref{prob:pandas-random-walk} is an example of a \emph{time series}, since it is indexed by time.
Time series show up often in data science; we will explore them in more depth in another lab.
\end{info}
\end{comment}

\begin{comment}
A \li{DataFrame} is a collection of \li{Series} that share the same index, and is therefore a two-dimensional generalization of a NumPy array.
The row labels are collectively called the \emph{index}, and the column labels are collectively called the \emph{columns}.
An individual column in a \li{DataFrame} object is one \li{Series}.

There are many ways to initialize a \li{DataFrame}.
In the following code, we build a \li{DataFrame} out of a dictionary of \li{Series}.

\begin{lstlisting}
>>> x = pd.Series(np.random.randn(4), ['a', 'b', 'c', 'd'])
>>> y = pd.Series(np.random.randn(5), ['a', 'b', 'd', 'e', 'f'])
>>> df1 = pd.DataFrame({"series 1": x, "series 2": y})
>>> df1
<<   series 1  series 2
a -0.365542  1.227960
b  0.080133  0.683523
c  0.737970       NaN
d  0.097878 -1.102835
e       NaN  1.345004
f       NaN  0.217523>>
\end{lstlisting}

Note that the index of this \li{DataFrame} is the union of the index of \li{Series} \li{x} and that of \li{Series} \li{y}.
The columns are given by the keys of the dictionary \li{d}.
Since \li{x} doesn't have a label \li{e}, the
value in row \li{e}, column \li{1} is \li{NaN}.
This same reasoning explains the other missing values as well.
Note that if we take the first column of the \li{DataFrame} and drop the missing values, we recover the \li{Series} \li{x}:

\begin{lstlisting}
>>> df1["series1"].dropna()
<<a   -0.365542
b    0.080133
c    0.737970
d    0.097878
Name: series 1, dtype: float64>>
\end{lstlisting}

We can also initialize a \li{DataFrame} using a NumPy array, creating custom
row and column labels.
\begin{lstlisting}
>>> data = np.random.random((3, 4))
>>> pd.DataFrame(data, index=['A', 'B', 'C'], columns=np.arange(1, 5))

            1	        2	        3	        4
A	 0.065646	 0.968593	 0.593394	 0.750110
B	 0.803829	 0.662237	 0.200592	 0.137713
C	 0.288801	 0.956662	 0.817915	 0.951016
3 rows     4 columns
\end{lstlisting}

As with Series, if we don't specify the index or columns, the default is
\li{np.arange(n)}, where \li{n} is either the number of rows or columns.

\begin{comment} % Not necessary here. Too much info.
It is also possible to create multi-indexed arrays, for example:
\begin{lstlisting}
>>> grade=['eighth', 'ninth', 'tenth']
>>> subject=['math', 'science', 'english']
>>> myindex = pd.MultiIndex.from_product([grade, subject], names=['grade', 'subject'])
>>> myseries = pd.Series(np.random.randn(9), index=myindex)
>>> myseries
grade   subject
eighth  math       1.706644
        science   -0.899587
        english   -1.009832
ninth   math       2.096838
        science    1.884932
        english    0.413266
tenth   math      -0.924962
        science   -0.851689
        english    1.053329
dtype: float64
\end{lstlisting}

Multi-indexing is visually convenient, and will be explored further in Lab
\ref{lab:pandas3}, where we will discuss pandas pivot tables.
\end{comment}

\begin{comment} % Moved random walk to Series section.
\subsection*{Plotting}

Plotting is often a much more effective way to view and gain understanding of a dataset than simply viewing the raw numbers.
Fortunately, pandas interfaces well with matplotlib, allowing relatively
painless data visualization.

Most of the functionality of plotting using pandas will be discussed in Lab \ref{lab:pandas2}.
 In this lab, we will simply show how to plot a \li{Series}.
Doing so is easy, as the \li{Series} object is equipped with its own plot
function.
% Change this example to use the titanic data, which will be cleaned up.

Let's start with visualizing a simple random walk.
By way of background, a \emph{random walk} is a
stochastic process used to model a non-deterministic path through some space.
It is a useful construct in
many fields and can be used to explain things like the motion of a molecule as it travels through a liquid to modeling the fluctuations
of stock prices.
Here we will simulate a one-dimensional symmetric random walk on the integers, which can be
described as follows.
\begin{enumerate}
  \item Start at 0.
  \item Flip a fair coin.
  \item If heads, move one unit to the right.
Otherwise, move one unit to the left.
  \item Go to Step 2.
\end{enumerate}
How can we simulate this random walk efficiently? Note that the walk is really characterized by the outcomes
of the coin flip.
If we represent heads by the number $1$ and tails by $-1$, then our position at a given moment
is just the cumulative sum of all previous outcomes.
Below, we simulate a sequence of coin flips, build the
resulting random walk, and plot the outcome.
\begin{lstlisting}
>>> import matplotlib.pyplot as plt
>>> N = 1000        # length of random walk
>>> s = np.zeros(N)
>>> s[1:] = np.random.binomial(1, .5, size=(N-1,))*2-1 #coin flips
>>> s = pd.Series(s)
>>> s = s.cumsum()  # random walk
>>> s.plot()
>>> plt.ylim([-50, 50])
>>> plt.show()
>>> plt.close()
\end{lstlisting}

The random walk is shown in Figure \ref{fig:PandasRandomWalk}.

\begin{figure}
\centering
\includegraphics[width=.7 \textwidth]{randomWalk.pdf}
\caption{Random walk of length 1000.}
\label{fig:PandasRandomWalk}
\end{figure}

\begin{problem}
Create five random walks of length 100, and plot them together in the same plot.

Next, create a ``biased'' random walk by changing the coin flip probability of head from 0.5 to 0.51.
Plot this biased walk with lengths 100, 10000, and then 100000.
Notice the definite trend that emerges.
Your results should be comparable to those in Figure \ref{pandas:biasedRandomWalk}.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=.7 \textwidth]{biasedRandomWalk.pdf}
\caption{Biased random walk of length 100 (above) and 10000 (below).}
\label{pandas:biasedRandomWalk}
\end{figure}
\end{comment}

\section*{SQL Operations in pandas} % =========================================

\li{DataFrames} are tabular data structures bearing an obvious resemblance to a typical relational
database table.
SQL is the standard for working with relational databases; however, pandas can accomplish many of the same tasks as SQL.
The SQL-like functionality of pandas is
one of its biggest advantages, eliminating the need to switch between programming languages
for different tasks.
Within pandas, we can handle both the querying \emph{and} data analysis.

For the examples below, we will use the following data:
\begin{lstlisting}
>>> name = ['Mylan', 'Regan', 'Justin', 'Jess', 'Jason', 'Remi', 'Matt', 'Alexander', 'JeanMarie']
>>> sex = ['M', 'F', 'M', 'F', 'M', 'F', 'M', 'M', 'F']
>>> age = [20, 21, 18, 22, 19, 20, 20, 19, 20]
>>> rank = ['Sp', 'Se', 'Fr', 'Se', 'Sp', 'J', 'J', 'J', 'Se']
>>> ID = range(9)
>>> aid = ['y', 'n', 'n', 'y', 'n', 'n', 'n', 'y', 'n']
>>> GPA = [3.8, 3.5, 3.0, 3.9, 2.8, 2.9, 3.8, 3.4, 3.7]
>>> mathID = [0, 1, 5, 6, 3]
>>> mathGd = [4.0, 3.0, 3.5, 3.0, 4.0]
>>> major = ['y', 'n', 'y', 'n', 'n']
>>> studentInfo = pd.DataFrame({'ID': ID, 'Name': name, 'Sex': sex, 'Age': age, 'Class': rank})
>>> otherInfo = pd.DataFrame({'ID': ID, 'GPA': GPA, 'Financial_Aid': aid})
>>> mathInfo = pd.DataFrame({'ID': mathID, 'Grade': mathGd, 'Math_Major': major})
\end{lstlisting}

Before querying our data, it is important to know some of its basic properties,
such as number of columns, number of rows, and the datatypes of the columns.
This can be done by simply calling the \li{info()} method on the desired
\li{DataFrame}:

\begin{lstlisting}
>>> mathInfo.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 3 columns):
Grade         5 non-null float64
ID            5 non-null int64
Math_Major    5 non-null object
dtypes: float64(1), int64(1), object(1)
\end{lstlisting}

\begin{comment}
We can also get some basic information about the structure of the \li{DataFrame}
using the \li{head()} or \li{tail()} methods.

\begin{lstlisting}
>>> mathInfo.head()
   Grade  ID Math_Major  ID  Age  GPA
0    4.0   0          y   0   20  3.8
1    3.0   1          n   2   18  3.0
2    3.5   5          y   4   19  2.8
3    3.0   6          n   6   20  3.8
4    4.0   3          n   7   19  3.4
\end{lstlisting}
\end{comment}

SQL SELECT statements can be done by column indexing.
WHERE statements can be included by adding masks (just like in a NumPy array).
The method \li{isin()} can also provide a useful WHERE statement.
This method accepts a list, dictionary, or \li{Series} containing possible values of the \li{DataFrame} or \li{Series}.
When called upon, it returns a \li{Series} of booleans, indicating whether an entry contained a value in the parameter pass into \li{isin()}.

\begin{lstlisting}
>>> # SELECT ID, Age FROM studentInfo
>>> studentInfo[['ID', 'Age']]
   ID  Age
0   0   20
1   1   21
2   2   18
3   3   22
4   4   19
5   5   20
6   6   20
7   7   19
8   8   29

>>> # SELECT ID, GPA FROM otherInfo WHERE Financial_Aid = 'y'
>>> mask = otherInfo['Financial_Aid'] == 'y'
>>> otherInfomask][['ID', 'GPA']]
   ID  GPA
0   0  3.8
3   3  3.9
7   7  3.4

>>> # SELECT Name FROM studentInfo WHERE Class = 'J' OR Class = 'Sp'
>>>  studentInfo[studentInfo['Class'].isin(['J','Sp'])]['Name']
0        Mylan
4        Jason
5         Remi
6         Matt
7    Alexander
Name: Name, dtype: object
\end{lstlisting}

\begin{comment}
\begin{problem}
Using panda commands, find the crime rates of the years where Property crimes were less than 90$\%$ of the total crime and burglary crimes were more than 20$\%$ of the total crime or property crimes were less than 90$\%$ of the total crime while violent crimes were more than 12$\%$ of the total crime.
\end{problem}
\end{comment}

Next, let's look at JOIN statements.
In pandas, this is done with the \li{merge} function.
\li{merge} takes the two \li{DataFrame} objects to join as parameters, as well as keyword arguments specifying
the column on which to join, along with the type (left, right, inner, outer).

\begin{lstlisting}
>>> # SELECT * FROM studentInfo INNER JOIN mathInfo ON studentInfo.ID = mathInfo.ID
>>> pd.merge(studentInfo, mathInfo, on='ID') # INNER JOIN is the default
   Age Class  ID    Name Sex  Grade Math_Major
0   20    Sp   0   Mylan   M    4.0          y
1   21    Se   1   Regan   F    3.0          n
2   22    Se   3    Jess   F    4.0          n
3   20     J   5    Remi   F    3.5          y
4   20     J   6    Matt   M    3.0          n
[5 rows x 7 columns]

>>> # SELECT GPA, Grade FROM otherInfo FULL OUTER JOIN mathInfo ON otherInfo.ID = mathInfo.ID
>>> pd.merge(otherInfo, mathInfo, on='ID', how='outer')[['GPA', 'Grade']]
   GPA  Grade
0  3.8    4.0
1  3.5    3.0
2  3.0    NaN
3  3.9    4.0
4  2.8    NaN
5  2.9    3.5
6  3.8    3.0
7  3.4    NaN
8  3.7    NaN
[9 rows x 2 columns]
\end{lstlisting}

\begin{problem}
Read in the file \li{crime_data.csv} as a pandas object.
The file contains data on types of crimes in the U.S. from 1960 to 2016.
Set the index as the column 'Year'.

Create a new column \li{Rate} which contains the crime rate for each year.
Using panda commands, find the number of murders in the years where the crime rate was greater than 5\% and the number of \li{Violent} was more than on average.
Return an array containing the number of murders in these years.

(Hint: To do \li{AND} statements, use two masks. Use \li{values} attribute to create array.)
\label{prob:rate}
\end{problem}

\begin{problem}
Answer the following questions using the file \li{crime_data.csv} and the pandas methods learned in this lab.
The answer of each question should be saved as indicated.
Return the answers to each question as a tuple (i.e. \li{(answer_1,answer_2,answer_3)}).

\begin{enumerate}
	\item Identify the three crimes that have a mean over 1,500,000.
	Of these three crimes, which two are very correlated?
	Which of these two crimes has a greater maximum value?
	Save the title of this column as a variable to return as the answer.
	\item Examine the data since 2000.
	Sort this data (in ascending order) according to number of murders.
	SELECT Aggravated Assault WHERE Aggravated Assault is greather than 850,000.
	Save the reordered and SQL queried \li{DataFrame} as a NumPy array to return as the answer.
	\item What decade had the most crime?
	In this decade, which crime was committed the most?
	What percentage of the total crime that year was it?
	Save this value as a float.
\end{enumerate}
\end{problem}

\begin{comment}
\begin{problem}
Read in the files \li{final_accidents2.pickle} and \li{final_drivers.pickle} using the function \li{read_pickle()}.
The accidents file contains data on various accidents that occurred in the U.S. and their causes.
The drivers data has data on various tickets given to drivers, how many tickets given, and what the ticket is for.
The following table shows the meanings of the codes in the data.

\begin{table}[H]
\begin{tabular}{r|l}
Code & Description \\ \hline
\li{describe()}  & Return a \li{Series} describing the data structure \\
\li{head()}      & Return the first $n$ rows, defaulting to 5 \\
\li{tail()}      & Return the last $n$ rows, defaulting to 5 \\

\li{DRUNK_DR} & 0-99 - Number of drunk drivers \\ \hline
\li{WEATHER} & 0 - Unknown \\
             & 1 - Clear \\
             & 2 - Rain \\
             & 3 - Sleet, Hail \\
             & 4 - Snow \\
             & 5 - Fog, Smog, Smoke \\
             & 6 - Severe Crosswinds \\
             & 7 - Blowing Sand, Soil, and Dirt \\
             & 8 - Other \\
             & 10 - Cloudy \\
             & 11 - Blowing Snow \\
             & 12 - Freezing Rain or Drizzle \\
             & 98 - Not Reported \\
             & 99 - Unknown \\ \hline
\li{SP} & 0 - Not Speeding \\
        & 1 - Speeding \\
        & 8 - Not Reported \\
        & 9 - Unknown \\ \hline
\li{FATALS} & 0-99 - Number of fatalities \\ \hline
\li{VEH_NO} & Assigned number of motor vehicle \\ \hline
\li{DRINKING} & 0 - Alcohol Not Involved \\
              & 1 - Alcohol Involved \\
              & 8 - Not Reported \\
              & 9 - Unknown \\ \hline
\li{SPEEDREL} & 0 - No \\
              & 1 - Yes \\
              & 2 - Yes, Racing \\
              & 3 - Yes, Exceeded Speed Limit \\
              & 4 - Yes, Too Fast for Conditions \\
              & 5 - Yes, Specifics Unknown \\
              & 8 - No Driver Present/Unknown if Driver Present \\
              & 9 - Unknown

\end{tabular}
\caption{Codes for \li{FARS} data.}
\label{table:fars-codes}
\end{table}

Create a \li{DataFrame} containing the Case Number, Age, whether the accident was caused by a drunk driver, whether the driver had DUIs, and if the accident was fatal, using an inner join operation.
Create a second \li{DataFrame} containing Case Number, Age, and fatalities using an inner join operation. In this \li{DataFrame}, also include the speed column from each dataframe.

Use these dataframes to answer the following questions.
Make sure to support your claims with appropriate and presentable data.

\begin{enumerate}
\item What is the most common kind of weather involved with accidents?

\item How many accidents involved speeding?

\item Of accidents where the type of speeding is known, what is the most common type of speeding in accidents?

\item Rounding to the nearest integer, what is the mean number of drunk drivers involved in each accident?

\item What is the mean number of fatalities per accident each year?

\item What month has the most fatalities each year?
\end{enumerate}
\end{problem}
\end{comment}

\begin{comment}
This may be going into too much detail for an introductory lab...
It is sometimes desirable to join to DataFrames on their indexes rather than on a particular column.
The \li{join} method makes this operation convenient.
\begin{lstlisting}
>>> #create new DataFrame
>>> sibs = [0, 1, 0, 5, 2, 9]
>>> sibInfo = pd.DataFrame({'Siblings':sibs})
>>> sibInfo
   Siblings
0         0
1         1
2         0
3         5
4         2
5         9
[6 rows x 1 columns]

>>> #now join studentInfo with sibInfo
>>> studentInfo.join(sibInfo)
   Age Class  ID    Name Sex  Siblings
0   20    Sp   0    Mylan   M         0
1   21    Se   1   Regan   F         1
2   18    Fr   2     Justin   M         0
3   22    Se   3   Jess   F         5
4   19    Sp   4     Jason   M         2
5   20     J   5  Remi   F         9
6   20     J   6   Matt   M       NaN
7   19     J   7  Alexander   M       NaN
8   20    Se   8     JeanMarie   F       NaN
[9 rows x 6 columns]
\end{lstlisting}
The default is a left join, but this can be altered.
When attempting to join to DataFrames that share common columns on their indexes,
we must rename the columns to avoid contradictions.
We do this by simply appending
characters to the existing column names, specified by the \li{lsuffix} and \li{rsuffix}
parameters (for the left and right DataFrames, respectively).
\begin{lstlisting}
>>> #join studentInfo and mathInfo on indexes
>>> studentInfo.join(mathInfo, lsuffix='_left', rsuffix='_right', how='inner')
   Age Class  ID_left   Name Sex  Grade  ID_right Math_Major
0   20    Sp        0   Mylan   M    4.0         0          y
1   21    Se        1  Regan   F    3.0         1          n
2   18    Fr        2    Justin   M    3.5         5          y
3   22    Se        3  Jess   F    3.0         6          n
4   19    Sp        4    Jason   M    4.0         3          n
[5 rows x 8 columns]
\end{lstlisting}

The final SQL-like operation we will discuss is the UNION, or concatenation of DataFrames.
In the simplest setting, this operation is useful when we have two DataFrames that have the
same columns but possibly different rows.
\begin{lstlisting}
>>> #create another df holding more math info
>>> mathID2 = [0, 5, 2, 4]
>>> mathGd2 = [4.0, 3.5, 2.0, 3.0]
>>> major2 = ['y', 'y', 'n', 'y']
>>> mathInfo2 = pd.DataFrame({'Grade':mathGd2, 'ID':mathID2, 'Math_Major':major2})
>>> pd.concat([mathInfo, mathInfo2], ignore_index=True).drop_duplicates()
   Grade  ID Math_Major
0    4.0   0          y
1    3.0   1          n
2    3.5   5          y
3    3.0   6          n
4    4.0   3          n
7    2.0   2          n
8    3.0   4          y
[7 rows x 3 columns]
\end{lstlisting}
The \li{ignore_index=True} argument means we discard the indexes of the two input DataFrames and create
a new one for the concatenated DataFrame.
The \li{drop_duplicates} method simply drops duplicate rows.
\end{comment}

\begin{comment}
\begin{problem}
In 1912 the RMS \emph{Titanic} sank after colliding with an iceberg.
The file \texttt{titanic.csv} contains data on the incident.
Each row represents a different passenger, and the columns describe various features of the passengers (age, sex, whether or not they survived, etc.)

Start by cleaning the data.
\begin{itemize}
    \item Read the data into a \li{DataFrame}.
    Use the first row of the file as the column labels, but do not use any of the columns as the index.
    \item Drop the columns \li{"Sibsp"}, \li{"Parch"}, \li{"Cabin"}, \li{"Boat"}, \li{"Body"}, and \li{"home.dest"}.
    \item Drop any entries without data in the \li{"Survived"} column, then change the remaining entries to \li{True} or \li{False} (they start as 1 or 0).
    \item Replace null entries in the \li{"Age"} column with the average age.
    \item Save the new \li{DataFrame} as \texttt{titanic\_clean.csv}.
\end{itemize}
Next, answer the following questions.
\begin{itemize}
    \item How many people survived? What percentage of passengers survived?
    \item What was the average price of a ticket? How much did the most expensive ticket cost?
    \item How old was the oldest survivor? How young was the youngest survivor? What about non-survivors?
\end{itemize}

\end{problem}
\end{comment}


\XtoCBlock{Int2Real}
\label{block:Int2Real}
\begin{figure}[H]\includegraphics{Int2Real}\end{figure}

\begin{XtoCtabular}{Inports}
In & Integer input\tabularnewline
\hline
\end{XtoCtabular}

\begin{XtoCtabular}{Outports}
Out & Real output\tabularnewline
\hline
\end{XtoCtabular}

\begin{XtoCtabular}{Mask Parameters}
Scale & Scaling factor from integer to real\tabularnewline
\hline
\end{XtoCtabular}

\subsubsection*{Description:}
Conversion block from integer (fixed point) datatypes to real (floating point) datatypes.

  Out = In * Scale

% include optional documentation file
\InputIfFileExists{\XcHomePath/Library/General/Doc/Int2Real_Info.tex}{\vspace{1ex}}{}

\subsubsection*{Implementations:}
\begin{tabular}{l l}
\textbf{FiP8\_Float32} & 8 Bit Fixed Point to 32 Bit Floating Point Implementation\tabularnewline
\textbf{FiP16\_Float32} & 16 Bit Fixed Point to 32 Bit Floating Point Implementation\tabularnewline
\textbf{FiP32\_Float32} & 32 Bit Fixed Point to 32 Bit Floating Point Implementation\tabularnewline
\textbf{FiP8\_Float64} & 8 Bit Fixed Point to 64 Bit Floating Point Implementation\tabularnewline
\textbf{FiP16\_Float64} & 16 Bit Fixed Point to 64 Bit Floating Point Implementation\tabularnewline
\textbf{FiP32\_Float64} & 32 Bit Fixed Point to 64 Bit Floating Point Implementation\tabularnewline
\end{tabular}

\XtoCImplementation{FiP8\_Float32}
\index{Block ID!192}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP8\_Float32 \tabularnewline
\textbf{ID} & 192 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Int2Real\_FiP8\_Float32.c \tabularnewline
\textbf{H filename} & Int2Real\_FiP8\_Float32.h \tabularnewline
\end{tabular}
\vspace{1ex}

8 Bit Fixed Point to 32 Bit Floating Point Implementation

\begin{XtoCtabular}{Controller Parameters}
scale & Scaling factor\tabularnewline
\hline
\end{XtoCtabular}

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int8          *In;
     float32       Out;
     float32       scale;
} INT2REAL_FIP8_FLOAT32;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/General/Doc/Test_Int2Real_FiP8_Float32.tex}{}{}
\fi
\XtoCImplementation{FiP16\_Float32}
\index{Block ID!193}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP16\_Float32 \tabularnewline
\textbf{ID} & 193 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Int2Real\_FiP16\_Float32.c \tabularnewline
\textbf{H filename} & Int2Real\_FiP16\_Float32.h \tabularnewline
\end{tabular}
\vspace{1ex}

16 Bit Fixed Point to 32 Bit Floating Point Implementation

\begin{XtoCtabular}{Controller Parameters}
scale & Scaling factor\tabularnewline
\hline
\end{XtoCtabular}

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int16         *In;
     float32       Out;
     float32       scale;
} INT2REAL_FIP16_FLOAT32;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/General/Doc/Test_Int2Real_FiP16_Float32.tex}{}{}
\fi
\XtoCImplementation{FiP32\_Float32}
\index{Block ID!194}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP32\_Float32 \tabularnewline
\textbf{ID} & 194 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Int2Real\_FiP32\_Float32.c \tabularnewline
\textbf{H filename} & Int2Real\_FiP32\_Float32.h \tabularnewline
\end{tabular}
\vspace{1ex}

32 Bit Fixed Point to 32 Bit Floating Point Implementation

\begin{XtoCtabular}{Controller Parameters}
scale & Scaling factor\tabularnewline
\hline
\end{XtoCtabular}

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int32         *In;
     float32       Out;
     float32       scale;
} INT2REAL_FIP32_FLOAT32;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/General/Doc/Test_Int2Real_FiP32_Float32.tex}{}{}
\fi
\XtoCImplementation{FiP8\_Float64}
\index{Block ID!195}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP8\_Float64 \tabularnewline
\textbf{ID} & 195 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Int2Real\_FiP8\_Float64.c \tabularnewline
\textbf{H filename} & Int2Real\_FiP8\_Float64.h \tabularnewline
\end{tabular}
\vspace{1ex}

8 Bit Fixed Point to 64 Bit Floating Point Implementation

\begin{XtoCtabular}{Controller Parameters}
scale & Scaling factor\tabularnewline
\hline
\end{XtoCtabular}

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int8          *In;
     float64       Out;
     float64       scale;
} INT2REAL_FIP8_FLOAT64;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/General/Doc/Test_Int2Real_FiP8_Float64.tex}{}{}
\fi
\XtoCImplementation{FiP16\_Float64}
\index{Block ID!196}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP16\_Float64 \tabularnewline
\textbf{ID} & 196 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Int2Real\_FiP16\_Float64.c \tabularnewline
\textbf{H filename} & Int2Real\_FiP16\_Float64.h \tabularnewline
\end{tabular}
\vspace{1ex}

16 Bit Fixed Point to 64 Bit Floating Point Implementation

\begin{XtoCtabular}{Controller Parameters}
scale & Scaling factor\tabularnewline
\hline
\end{XtoCtabular}

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int16         *In;
     float64       Out;
     float64       scale;
} INT2REAL_FIP16_FLOAT64;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/General/Doc/Test_Int2Real_FiP16_Float64.tex}{}{}
\fi
\XtoCImplementation{FiP32\_Float64}
\index{Block ID!197}
\nopagebreak[0]
% Implementation details
\begin{tabular}{l l}
\textbf{Name} & FiP32\_Float64 \tabularnewline
\textbf{ID} & 197 \tabularnewline
\textbf{Revision} & 0.1 \tabularnewline
\textbf{C filename} & Int2Real\_FiP32\_Float64.c \tabularnewline
\textbf{H filename} & Int2Real\_FiP32\_Float64.h \tabularnewline
\end{tabular}
\vspace{1ex}

32 Bit Fixed Point to 64 Bit Floating Point Implementation

\begin{XtoCtabular}{Controller Parameters}
scale & Scaling factor\tabularnewline
\hline
\end{XtoCtabular}

% Implementation data structure
\XtoCDataStruct{Data Structure:}
\begin{lstlisting}
typedef struct {
     uint16        ID;
     int32         *In;
     float64       Out;
     float64       scale;
} INT2REAL_FIP32_FLOAT64;
\end{lstlisting}

\ifdefined \AddTestReports
\InputIfFileExists{\XcHomePath/Library/General/Doc/Test_Int2Real_FiP32_Float64.tex}{}{}
\fi


\section*{Welcome to SnuggleTeX!}

\begin{itemize}
  \item This is an itemized list.

  \item This is some inline maths: $x^2+ y^2 = 1$

  \item Some block maths:
        \[ \sum_{n=1}^\infty \frac{1}{n^2} = \frac{\pi^2}{6} \]
        (\textbf{Note}: This currently doesn't display nicely in Firefox 3. Boo\ldots)

  \item A $2\times 2$ matrix created via a user-defined command:
        \newcommand{\mat}[4]{\left( \begin{array}{cc} #1 & #2 \\ #3 & #4 \end{array} \right)}
        \[ A = \mat{\alpha}{\beta}{\gamma}{\delta} \]

\end{itemize}

Here is an example of a SnuggleTeX error message. You can set it up to display the messages inline (as we have here) or report them in various other ways:

\doh


% declare document class and geometry
\documentclass[12pt]{article} % use larger type; default would be 10pt
\usepackage[margin=1in]{geometry} % handle page geometry

% import packages and commands
\input{../header2.tex}

\newcommand{\Gr}{\opname{Gr}}

\title{Math 217 -- Geometry and Physics -- Lec10}
\author{UCLA, Fall 2014}
\date{\formatdate{24}{10}{2014}} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed

\begin{document}
\maketitle

\section{Universal vector bundles}

Let's start with complex vector bundles. Recall the grassmannian
\begin{eqn}
\opname{Gr}(n,N) = \set{V \subseteq \complexes^N | \dim V = n},
\end{eqn}
which has the property
\begin{eqn}
\begin{matrix}
\opname{Gr}(n,N) & \subseteq & \opname{Gr} (n,N+!) & \subseteq \dots \\
(z_1, \dots, z_N) & \mapsto & (z_1, \dots, z_N, 0) &
\end{matrix}
\end{eqn}
The ``infinite'' Grassmannian is denoted
\begin{eqn}
BU(n) = \opname{Gr}(n, \infty)
\end{eqn}
and has a universal subbundle $S \rightarrow \Gr(n,\infty) = BU(n)$. Furthermore, we find that
\begin{align}
\opname{Vect}_n(M) &= \set{ \text{complex vector bundles on $M$} } \\
	& \cong [M, BU(n)] = \set{ \text{homotopy classes of $f: M \rightarrow BU(n)$} }.
\end{align}
Then we have
\begin{eqn}
H^*(BU(n), \ints) \cong \ints[c_1, \dots, c_n], \quad \text{where} \quad c_i = c_i(S).
\end{eqn}
We also have
\begin{eqn}
c_i(E) = f^* c_i (S) = f^* c_i.
\end{eqn}

For grassmannians on real vector bundles, we have the infinite grassmannian
\begin{eqn}
BO(n) = \Gr_\reals (n, \infty).
\end{eqn}
Then
\begin{align}
\opname{Vect}_n (M) &\cong [M, BO(n)] \\
H^* (BO(n), \irrats) &= \irrats[p_1, \dots, p_{\lceil n/2 \rceil}] \\
H^*(BO(n), \ints_2) &= \ints[w_1, \dots, w_n]
\end{align}
where $P_j (E) = f^* p_j(S)$, $w_j(E) = f^* w_j(S)$, and $w_j = w_j(S)$ (Universal Stiefle-Whitney [?])

[$O(n) \rightarrow U(n)$, $BO(n) \rightarrow BU(n)$, $H^*(BU(n), \ints) \rightarrow H^*(BO(n),\ints)$, $p_j = (-1)^j c_{2j}$]

\subsection{Axioms of SW [Stiefle-Whitney?] classes}
Given real vector bundle $E$ over closed $M$, there exist $\set{w_i(E)}$ in $H^*(M, \ints_2)$ ($w_i \in H^i(M, \ints_2)$) such that
\begin{enumerate}
\item $w_0 = 1$,
\item $w_i (f^*E') = f^* w_i (E)$,
\item $w(E \oplus E') = w(E) w(E')$,
\item $\gamma_1$ (Universal [subdim?] bundle) $\rightarrow \RP^1$, where $w_1(\gamma_1)$ the generator of $H^1(\RP^1, \ints_2)$.
\end{enumerate}

\subsubsection{stuff}

[missed some stuff here]

so we have
\begin{eqn}
c_j = \sigma_j(x_1, \dots, x_n)
\end{eqn}

\subsubsection{stuff}

[missed bit more here]

\subsection{Axioms for Chern classes}

Given complex vector bundle $E^n \rightarrow M$,
\begin{enumerate}
\item $c_0(E) = 1$, $c_j(E) = 0$ for $j > n$.
\item $f^* c_j(E') = c_j (f^* E')$,
\item $c(E \oplus E') = c(E) c(E')$,
\item $\bigO(-1) = L \rightarrow \CP^\infty$ and $c_1(L) = x$ is generator of $H^*(\CP^\infty, \ints)$.
\end{enumerate}

We can write
\begin{eqn}
i \partial \bar\partial \log(1 + \abs{z}^2) = c_1 (\bigO(-1)),
\end{eqn}
where $\bigO(-1) \rightarrow \CP^1$ (exercise). We find that
\begin{eqn}
\int_{\CP^1} c_1 \bigO(-1) = \int_{z \in \complexes} i \partial \bar\partial \log(1 + \abs{z}^2) = \pm 1.
\end{eqn}
We have $\pi^* c_j(E) = \sigma_j (x_1, \dots, x_n)$, where $x_j = c_1(L_j)$, where
\begin{eqn}
\pi^* : H^*(M) \hookrightarrow H^*(F(E)).
\end{eqn}
So we have
\begin{eqn}
\CP^{n-1} \hookrightarrow P(E) \overset{\pi}{\rightarrow} M
\end{eqn}
Let $t = -c_1(L)$, and note that
\begin{eqn}
\pi^* E \cong \underbrace{L}_{\bigO(-1)} \oplus \underbrace{Q}_\text{quotient bundle}.
\end{eqn}
Writing out our expansion
\begin{eqn}
c_t = \det (tI + \frac{i}{2\pi} \Omega) = t^n + c_1 t^{n-1} + \dots + c_n,
\end{eqn}
since we have
\begin{eqn}
c_t(\pi^* E) = \underbrace{c_t(L)}_{=0} c_t(Q),
\end{eqn}
so that
\begin{eqn}
0 = t^n + \pi^* c_1(E) t^{n-1} + \dots + \pi^* c_n(E).
\end{eqn}
So we've found \textit{Grothendieck's characterization of Chern classes},
\begin{eqn}
H^*(P(E), \irrats) = H^*(M) [x_E] \Big/ \set{x_E^n + \pi^* c_1(E) x_E^{n-1} + \dots + \pi^* c_n(E) = 0}.
\end{eqn}

The most useful characteristic classes is probably Hirzebruch's:

1. $L$-classes, $TM \rightarrow M^{2n}$. We have
\begin{eqn}
L(TM) = \prod_{j=1}^n \frac{x_j}{\tanh x_j} = \text{a symmetric function in $\set{x_j^2}$}
\end{eqn}
As an exercise, for $n=2$, can show
\begin{eqn}
L(TM)^\text{top} = \frac{x_1}{\tanh x_1} \frac{x_2}{\tanh x_2} = \frac{1}{3} P_1 (TM).
\end{eqn}
Then $\hat{A} (TM^4) = -\frac{1}{24} P_1 (TM)$

2. The $\hat{A}$-classes are given by
\begin{eqn}
\hat{A} (TM) = \prod_{j=1}^n \frac{x_j / 2}{\sinh (x_j/2} = f(p_1, \dots, p_n)
\end{eqn}

\end{document}


%&LaTeX

\section{The Matlab Lab, or How to Not Teach a Programming Language}

Labs in this class will make liberal use of the Matlab numerical
programming environment. Because this class assumes that you are an
experienced computer science student, you are expected to be able to
learn how to use computer tools, and how to program in new programming
languages, pretty much on your own. So, the first thing you should do
is check out the Matlab documentation built into the Matlab help
system, or online at
\url{http://www.mathworks.com/help/matlab/index.html}. Of course, you
can always search online for Matlab tutorials and the like. We'll
include a very brief overview of Matlab below, and then more detailed
information about the code developed specifically for this class that
you will be using.

\subsection{Matlab in a (Very Small) Nutshell}
\label{sc:basic-matlab}

The Matlab GUI environment is very similar to IDEs that you are
already familiar with. As you might expect, there are some
idiosyncrasies here and there, but nothing terribly unexpected. The
major areas of difference are tools and panes that have to do with
viewing variables (something in other IDEs that you'd only see when
debugging a program), the command pane, and figure windows that open
to display graphs. The first two of these differences have to do with
the fact that Matlab is an interpreted language/programming
environment. You primary area of interaction with the Matlab
interpreter will be through the command window, with variable
information panes providing views of variables and their contents
related to your interaction. It's interesting to note that, if you
want, you can run Matlab without the GUI, providing just a command
line interface.

Interpreted languages have their advantages and disadvantages.  One
advantage is that anything you can use as a line of code in a program
you can use immediately as a command on the command line. This lets
you test code interactively and then copy it into the script or
function you're writing. Like any IDE, Matlab includes an editor with
syntax highlighting and debugger integration.

The disadvantage is the interpreted programs are slower. In Matlab, we
get around this by using built-in functions that operate on entire
vectors or arrays as single data objects. The core loops of the
built-in functions are compiled for speed. If you make good use of
those functions, Matlab code can often be as fast as completely
compiled code.

Here are some things to try:
\begin{enumerate}
\item Immediate calculations and variables:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
radius = 5    % Comments start with "%"
circumference = 2 * pi * radius
area = pi * radius^2
\end{lstlisting}
\item Complex numbers:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
sqrt(-1)
x = 7 + 14j
conj(x)       % complex conjugate
abs(x)        % magnitude (e.g., for polar representation)
angle(x)      % and angle for polar rep
real(x)
imag(x)
\end{lstlisting}

\item Complex exponentials:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
exp(j * pi)
exp(j * pi/2)
exp(j * pi/4)
\end{lstlisting}

\item Vectors:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
v1 = [0 1 2 3]    % Four elements
v2 = [0 : 2 : 10] % like a loop (start value : increment : end value)
v3 = pi * [-0.5 -0.25 0 0.25 0.5] % All operations are vectorized
exp(j * v3)
mistake = v1 * v1     % a mistake; vector mult doesn't work this way
dotproduct = v1 * v1' % transposing will work, if you want to do this
arrayprod = v1 .* v1  % element-by-element ops include: .+, .-, .*, ./
\end{lstlisting}

\item Simple plots (note that ``;'' suppresses outputting results to
  the command window --- useful if that would generate massive amounts
  of text, or just if you want things neat):
\label{it:simple-plots}
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
t = [0 : 0.01 : 2*pi];
x = sin(t);
plot(t, x);       % default plots points connected by lines in blue
plot(t, x, 'r');  % change the line color
plot(t, x, 'r.'); % change the plot style; zoom in to see individual points
xlabel('t, ms');  % X axis label (all of your plots should have this)
ylabel('Mag');    % Y axis label (all of your plots should have this)
title('Triangle');% Graph title (all of your plots should have this)
\end{lstlisting}

\end{enumerate}

If you take a sequence of commands and save them in a file with an
extension of \texttt{.m}, the result is a \emph{script}. Assuming that the
script is saved in a directory in the MATLAB search path, you can then
execute the script by just typing its name (without the \texttt{.m}),
just as if it were a command. If you want your code to take
parameters, return a return value, and have local variables, start
your code with a line like:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
function retval = funcname(parm1, parm2)
\end{lstlisting}

Your code is now a function. MATLAB functions can take variable
numbers of arguments and even return variable numbers of return
values, but that's getting beyond what we need right now.

\paragraph{Step 1.1} It's easy to create, concatenate, extract, and modify
vectors or parts of vectors. Execute the following lines of Matlab
code and explain what each echoes out:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
a = ones(1,3)
b = zeros(1,5)
x = [b, a, [1:2:12]]
x(7:end)
length(x)
x(1:2:12)
\end{lstlisting}

Also, explain the difference between the square bracket notation
\verb|[1:2:12]| and the parenthetical notation \verb|(1:2:12)|.

\paragraph{Step 1.2} Consider the result of the following assignment:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
x(7:11) = pi*(1:5)
\end{lstlisting}

Write a \emph{single} statement that will replace the odd-indexed
elements of \texttt{x} with the constant -10 (i.e., \texttt{x(1)},
\texttt{x(3)}, etc).

\paragraph{Step 1.3} One of the side benefits of learning Matlab is
that it trains you to think in terms of parallel operations --- an
increasingly important skill in a profession becoming dominated by
multi-core, GPU, and distributed computing. That doesn't mean you
can't write loops in Matlab; it's just that your code will be more
concise and efficient if you can avoid that. The efficiency arises
from the fact that the vectorized Matlab commands are mostly compiled;
while the loops you write are interpreted. Consider the following loop:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
for k=0:7,
   x(k+1) = cos(k*pi/4);
end
x
\end{lstlisting}

Why is \verb|x| indexed by \verb|k+1| rather than \verb|k|? What
happens to the length of \verb|x| for each iteration of the loop?
Rewrite this computation without using the loop (as in list
item~\ref{it:simple-plots}). Besides the increase in efficiency from
avoiding an interpreted loop, what other major efficiency results from
this change?

\paragraph{Step 1.4} Consider the following code that plots a
sinusoid:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
t = [0 : 0.01 : 1]; % time in seconds
f = 5;              % freq in Hertz
x = sin(2*pi*f*t);
plot(t, x);
xlabel('Time (sec)');
\end{lstlisting}

Use the MATLAB editor to create a script file called
\texttt{firstsin.m}, verify that you've saved it in a directory in the
MATLAB path (or add that directory to the path), and test its
execution by typing \texttt{firstsin} at the MATLAB command
prompt. Note that you can also do:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
type firstsin   % prints out contents of the script
which firstsin  % shows directory (useful when your code shadows built-ins)
\end{lstlisting}

If you included documentation for this script (comments at the
beginning), the command \verb|help firstsin| would also produce useful
output.

Add three lines of code to your script, so that it will plot a cosine
using the same axes as the sine (i.e., ``on top of the sine''). Use
the \texttt{hold} function to add a plot of
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
0.75*cos(2*pi*f*t)
\end{lstlisting}
to the plot. So, your final graph will have two functions
plotted. Save the plot using the MATLAB \texttt{print} command as a
PNG file named \texttt{step14.png} by typing:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
print -dpng step14
\end{lstlisting}

You should include all plots and code snippets in your lab report,
following the instructions in the report rubric.

\paragraph{Step 1.5} You can also use Matlab to generate sounds. A
pure tone is merely a sinusoid, which you already know how to
generate. Let's generate one with a frequency of 3 kHz and a duration
of 1 second:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
T = 1.0;
f = 3000;
fs = 8000;
t = [0 : (1/fs) : T];
x = sin(2*pi*f*t);
soundsc(x, fs)
\end{lstlisting}

The vector of numbers \texttt{x} are converted into a sound waveform
at a certain rate, \verb|fs|, called the \emph{sampling rate} (we will
learn a lot more about this in this class). In this case, the sampling
rate was set to 8000 samples/second. What is the length of
the vector \texttt{x}?

\paragraph{Step 1.6} Write a new function that performs the same task
as the following function without using any loops. Use the idea in
step~1.3 and also consult the section on the \verb|find| function,
relational operators, and vector logicals in the MATLAB documentation.
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
function B = denegify(A)
% DENEGIFY Replace negative elements of matrix with zeros
% Usage:
%    B = denegify(A)
%
[W,H] = size(A);
for i=1:W
   for j=1:H
      if A(i,j) < 0
         B(i,j) = 0;
      else
         B(i,j) = A(i,j);
      end
   end
end
\end{lstlisting}

\subsection{Trigonometric Functions and Complex Mathematics in Matlab}

\paragraph{Step 2.1} In this step, you are asked to complete a Matlab
function to synthesize a waveform in the form of:
\begin{equation*}
x(t) = \sum_{k=1}^N a_k\cos(2\pi f t + \phi_k)
\end{equation*}
This is a sum of cosines, all at the same frequency but with different
phases and amplitudes.  Use the following function prototype to start you off:
\begin{lstlisting}[style=Matlab-editor,basicstyle=\mlttfamily\small]
   function x = sumcos(f, phi, a, fs, dur)
   % SUMCOS Synthesize a sum of cosine waves
   % Usage:
   %    x = sumcos(f, phi, a, fs, dur)
   %        Returns sum of cosines at a single frequency f, sampling
   %        rate fs, and duration dur, each with a phase phi and
   %        amplitude a.
   %    f = frequency (scalar)
   %    phi = vector of phases
   %    a = vector of amplitudes
   %    fs = the sampling rate in Hz (scalar)
   %    dur = total time duration of signal (scalar)
\end{lstlisting}

Include your code in your writeup. Additionally, include a plot of
\texttt{x = sumcos(20, [0 pi/4 pi/2 3*pi/2], [1 2 3 4], 200, 0.25);} versus
time.

Hint: the MATLAB \verb|length| function is useful in determining the
number of elements in a vector; the \verb|size| function returns both
dimensions of a vector or an array.

\paragraph{Step 2.2} Now, let's see how complex exponentials can
simplify things. Re-implement your \texttt{sumcos} function using
complex exponentials. Take advantage of the fact that multiplying a
complex sinusoid $e^{j2\pi f t}$ by the complex amplitude
$a_ie^{j\phi}$ will shift its phase and change its amplitude. Thus,
you should be able to create a \emph{single} complex sinusoid at the
given frequency \texttt{f} and then multiply it by different \texttt{a
  * exp(j * phi)} to get multiple phase shifted cosines. Remember that
we want a real value to plot; the cosine is the real part of a complex
sinusoid. Include your code in your writeup and provide a plot that
demonstrates that this function produces the same result as the
original implementation.

\paragraph{Step 2.3} Generate four sinusoids with the following
amplitudes and phases:
\begin{align}
x_1(t) &= 6 \cos(2\pi(10)t - 0.5\pi) \\
x_2(t) &= 3 \cos(2\pi(10)t + 0.25\pi) \\
x_3(t) &= 2 \cos(2\pi(10)t - 0.3\pi) \\
x_4(t) &= 8 \cos(2\pi(10)t + 0.9\pi)
\end{align}

\begin{enumerate}\renewcommand{\theenumi}{\alph{enumi}}
\item Make a single plot of all four signals together over a range of
  $t$ that will generate approximately 3 cycles. Make sure the plot
  includes negative time so that the phase at $t = 0$ can be
  measured. In order to get a smooth plot make sure that your have at
  least 20 samples per period of the wave. Include your plot in your
  writeup.

\item Verify that the phase of all four signals is correct at $t = 0$,
  and also verify that each one has the correct maximum amplitude. Use
  \verb|subplot(3,2,i)| to make a six-panel subplot that puts all of
  these plots in the same figure, with space for two additional plots
  at the bottom. Use the \verb|xlabel|, \verb|ylabel|, and
  \verb|title| functions so that the reader can figure out what the
  plots mean; reinforce this with your report's figure caption. (You
  should include the final figure, with all subplots, that results
  from finishing all of the parts of this step.)

\item Create the sum sinusoid,
  $x_5(t)=x_1(t)+x_2(t)+x_3(t)+x_4(t)$. Plot $x_5(t)$ over the same
  range of time as used in the last plot. Include this as the lower
  left panel in the plot by using \verb|subplot(3,2,5)|.

\item Now do some complex arithmetic; create the complex amplitudes
  corresponding to the sinusoids $x_i(t)$: $z_i = A_ie^{j\phi_i}$,
  $i=1,2,3,4,5$. Include a table in your report of the $z_i$ in polar
  and rectangular form, showing $A_i$, $\phi_i$, $\Real\{z_i\}$, and
  $\Imag\{z_i\}$.
\end{enumerate}

\subsection{Representing Analog, Discrete, and Digital Signals}

In our class, we will need to manipulate analog signals (real-valued
signals that are functions of continuous time), discrete signals
(real-valued signals that are functions of discrete time), and digital
signals (discrete-valued signals that are functions of discrete
time). No need to worry about the details or these; that will become
clear later. The trickiest part of this is representing anything other
than digital signals on a digital computer, because you can't. So,
we'll need to employ two key elements of software design: information
hiding and make-believe.

\begin{figure}
\begin{center}
\includegraphics[width=0.75\textwidth]{lab1/AnalogSignal-help}
\end{center}
\caption{Example help screen for the \texttt{AnalogSignal}
  class. This example may be out-of-date; use the Matlab
  \texttt{doc AnalogSignal} command to get current
  documentation.\label{fg:analogsignal-help}}
\end{figure}

Information hiding is used in our implementation of analog signals. We
make use of the object-oriented programming aspects of Matlab to
create an \texttt{AnalogSignal} class. If you look up the
documentation for \texttt{AnalogSignal} (using
\verb|doc AnalogSignal|), you'll see something like
figure~\ref{fg:analogsignal-help}. The key operations on these analog
signals are:
\begin{itemize}
\item Creating an analog signal (ex: \texttt{a =
    AnalogSignal('sawtooth', 2.0, 1.0, 10.0)}).
\item Scaling an analog signal (ex: \texttt{b = a * 5})
\item Adding two analog signals (ex: \texttt{c = a + b})
\item Subtracting two analog signals (ex: \texttt{d = c - a})
\item Plotting an analog signal (ex: \texttt{plot(d)})
\item Sampling an analog signal (ex: \texttt{x = d.samplehold(0.1)})
\end{itemize}

You will get a lot more experience working with analog signals shortly
in this class, so for the time being just play with this a bit.


Role of BNN. List the parameters predicted $\xi_{lens}$ and which of the $\xi_{light}$ (source position, lens effective radius). Input, output. Build up to hierarchical inference. Focus on consistency between \cite{hezaveh2017fast} and \cite{wagnercarena2019double}.

\subsection{Training set generation}
Cite baobab paper in prep and link the GH. Say that we experimented with two different conditional PDFs, from which we sampled the macromodel parameters. One was where each parameter was sampled independently. Have a table like the TDLMC table? See \ref{table:conditional_pdf}. Another was where the parameters followed empirical distributions.

Training set is samples from the interim prior...

\begin{table}
\label{table:conditional_pdf}
\centering
\caption{Parameter distribution}\label{para_config}
\begin{tabular}{ l l}
\hline
Simulating ingredient & model and parameter values \\
\hline\hline
A): redshift \\\hline
deflector redshift & $z_d \in {\rm Norm} (0.5\pm0.2)$ \\
source redshift & $z_s \in {\rm Norm} (2.0\pm0.4)$ \\
 \hline\hline
% Rung1 and Rung2\\ \hline
\\B): deflector (image plane) \\
\hline
lensing galaxy mass & elliptical power-law \\
\hline
SIS velocity dispersion & $v_d\in {\rm Norm} (250\pm25)$ km/s \\
Einstein radius$^a$ &$R_{\rm Ein} = 4\pi v_d^2 \frac{D_{ds}}{D_{s}}$ \\
mass slope & $s \in {\rm Norm} (2.0\pm 0.1)$ \\
ellipticity & $q \in {\rm Uni} (0.7 \sim 1.0)$ \\
elliptical axis angle & $\phi \in {\rm Uni} (0 \sim \pi) $ \\
\hline
lensing galaxy SB & \sersic\ profile \\
\hline
total magnitude$^{b}$ & $ mag \in {\rm Uni} (17.0 \sim 19.0) $ magnitude \\
effective radius & $R_{\rm eff} = R_{\rm Ein} * {\rm Uni} (0.5 \sim 1.0) $ \\
\sersic\ index & $n \in {\rm Uni} (2.0 \sim 4.0) $\\
ellipticity & $q \in {\rm Uni} (0.7 \sim 1.0)$ \\
elliptical axis angle & $\phi \in {\rm Uni} (0 \sim \pi) $ \\
\hline \hline

\\C): AGN (source plane) \\
\hline
host galaxy SB & realistic galaxy\\
\hline
total magnitude & $ mag \in {\rm Uni} (22.5 \sim 20.0) $ magnitude \\
effective radius$^c$ & $R_{\rm eff} \in {\rm Uni} (0\farcs{}37, 0\farcs{}45), $ {\scriptsize $1.0<z_s<1.5$ \par} \\
&$R_{\rm eff} \in {\rm Uni} (0\farcs{}34, 0\farcs{}42), $  {\scriptsize $1.5<z_s<2.0$  \par} \\
&$R_{\rm eff} \in {\rm Uni} (0\farcs{}31, 0\farcs{}35), $  {\scriptsize $2.0<z_s<2.5$  \par} \\
&$R_{\rm eff} \in {\rm Uni} (0\farcs{}23, 0\farcs{}33), $  {\scriptsize $2.5<z_s<3.0$  \par} \\
\hline
active nuclear light & scaled point source \\
\hline
source plane total flux & $f_{\rm AGN} = f_{\rm host} * {\rm Uni} (0.8 \sim 1.25) $ \\
 \\\hline \hline
external shear \\
\hline
amplitudes & $ \gamma\ \in {\rm Uni} (0 \sim 0.05) $ \\
shear axis angle & $\phi \in {\rm Uni} (0 \sim \pi) $ \\
\hline
external convergency \\
\hline
external kappa$^d$ & $ \kappa_{\rm ext} \in {\rm Norm} (0 \pm 0.025) $ \\
\hline
\hline
\end{tabular}
  \begin{tablenotes}
 	 \footnotesize
 	 \item Notes:- In Rung3, the non-parameterized deflector (i.e., lensing galaxy mass and surface brightness) are adopted. Thus, the B part in the table is not adoptable for this rung. The distribution of ``Norm" means normal distribution and the ``Uni" means uniform distribution.
	 \item ~~~~a: The scale of Einstein Radius by is in range [1$\farcs$00, 1$\farcs$20].
	 \item ~~~~b: The flux value and the magnitude value are related by the equation: \\ $mag= -2.5 * log10(flux) + zp$, where $zp$ is the filter zeropoint in AB system. For filter WFC3/F160W, $zp = 25.9463$.
	 \item ~~~~c: The effective radius of the realistic galaxy is measured by the Galfit, by considering them as \sersic\ profile.
	 \item ~~~d: The value of $\kappa_{\rm ext}$ is randomly generated to calculate the time delay data. The information of this distribution is provided to the ``Good" teams as a measurement data.
  \end{tablenotes}
\end{table}

\subsection{BNN posterior}
What's the term to use for this? Cite \cite{wagnercarena2019double} and say the form was double Gaussian.

\subsection{Posterior on $H_0$}
Time delays, velocity dispersion are external data. Conditional PDF
Introduce training set are samples from the interim prior.

With importance sampling,
\begin{align}
    &p(\Omega | \Delta t_{AB}, \sigma_V, d_{HST}) \\
    &\propto p(\Omega) \prod_{k} \int p(\Delta t_{AB}^{(k)} | \Omega, \xi_{lens}^{(k)}, \kappa_{ext}^{(k)})  \nonumber \\
    & \quad \times p(\sigma_V^{(k)} | \Omega, \xi_{lens}^{(k)}, \xi_{light}^{(k)}, \kappa_{ext}^{(k)}, \beta_{ani}^{(k)})  \nonumber \\
    & \quad \times p(\xi_{lens}^{(k)}, \xi_{light}^{(k)} | d_{HST}^{(k)}, \Omega_i) \times \frac{p(\xi_{lens}^{(k)}, \xi_{light}^{(k)} | \Omega)}{ p(\xi_{lens}^{(k)}, \xi_{light}^{(k)} | \Omega_i)} \nonumber \\
    & \quad \times p(\kappa_{ext}^{(k)}) p(\beta_{ani}^{(k)})
    \quad d \left( \xi_{lens}^{(k)}, \xi_{light}^{(k)} \right) d \kappa_{ext}^{(k)} d \beta_{ani}^{(k)} \nonumber
\end{align}