#!/usr/bin/env python
# coding: utf-8

# # Self-Driving Car Engineer Nanodegree
#
# ## Deep Learning
#
# ## Project: Build a Traffic Sign Recognition Classifier
#
# In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.
#
# > **Note**: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to  \n",
#     "**File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission.
#
# In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a [write up template](https://github.com/udacity/CarND-Traffic-Sign-Classifier-Project/blob/master/writeup_template.md) that can be used to guide the writing process. Completing the code template and writeup template will cover all of the [rubric points](https://review.udacity.com/#!/rubrics/481/view) for this project.
#
# The [rubric](https://review.udacity.com/#!/rubrics/481/view) contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.
#
#
# >**Note:** Code and Markdown cells can be executed using the **Shift + Enter** keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.

# ---
# ## Step 0: Load The Data

# In[26]:

# Load pickled data
import pickle
import numpy as np
# Load the training, validation and testing data

training_file = '../data/train.p'
validation_file= '../data/valid.p'
testing_file = '../data/test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)

X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

# Lets check if data is not missing. Length of features and labels should match
assert(len(X_train) == len(y_train))
assert(len(X_valid) == len(y_valid))
assert(len(X_test) == len(y_test))

# Print some stats for data that we gonna use
print()
print("Image Shape: {}".format(X_train[0].shape))
print()
print("Training Set: {} Samples".format(len(X_train)))
print("Validation Set: {} Samples".format(len(X_valid)))
print("Tesging Set: {} Samples".format(len(X_test)))
print()
print("Feature Shape: {}".format(np.shape(X_train)))
print("Labels Shape: {}".format(np.shape(y_train)))
# import matplotlib.pyplot as plt
# img = X_train[0].squeeze()
# plt.figure(figsize=(1,1))
# plt.imshow(img)
# print(X_train[0])

# ---
#
# ## Step 1: Dataset Summary & Exploration
#
# The pickled data is a dictionary with 4 key/value pairs:
#
# - `'features'` is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
# - `'labels'` is a 1D array containing the label/class id of the traffic sign. The file `signnames.csv` contains id -> name mappings for each id.
# - `'sizes'` is a list containing tuples, (width, height) representing the original width and height the image.
# - `'coords'` is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. **THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES**
#
# Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the [pandas shape method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shape.html) might be useful for calculating some of the summary results.

# ### Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

# In[27]:

### Replace each question mark with the appropriate value.
### Use python, pandas or numpy methods rather than hard coding the results

# Number of training examples
n_train = len(X_train)

# Number of validation examples
n_validation = len(X_valid)

# Number of testing examples.
n_test = len(X_test)

# What's the shape of an traffic sign image?
image_shape = X_train[0].shape

# How many unique classes/labels there are in the dataset.
n_classes = len(np.unique(y_train))

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Number of validation examples =", n_validation)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)

# ### Include an exploratory visualization of the dataset

# Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
#
# The [Matplotlib](http://matplotlib.org/) [examples](http://matplotlib.org/examples/index.html) and [gallery](http://matplotlib.org/gallery.html) pages are a great resource for doing visualizations in Python.
#
# **NOTE:** It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?

# In[32]:

### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
f, axarr = plt.subplots(2,2)

# count of each traffic sign in training data set
unique, counts = np.unique(y_train, return_counts=True)
axarr[0,0].bar(unique, counts)

top_5_idx = np.argsort(counts)[-5:]
top_5_signs = [unique[i] for i in top_5_idx]
print("5 Sign with highest Samples: ", top_5_signs)

bottom_5_idx = np.argsort(counts)[0:4]
bottom_5_signs = [unique[i] for i in bottom_5_idx]
print("5 Sign with lowest Samples: ", bottom_5_signs)

# plt.ylabel('Counts')
# plt.xlabel('Sign')

# show image of one of the least occuring sign and one most occuring sign
idx_least_occur_image = np.where(y_train==0)
idx_most_occur_image = np.where(y_train==1)
img_sign_least_count = X_train[idx_least_occur_image[0][0]]
axarr[0,1].imshow(img_sign_least_count)
#print(idx_most_occur_image[0])
img_sign_max_count = X_train[idx_most_occur_image[0][6]]
axarr[1,0].imshow(img_sign_max_count)

# ----
#
# ## Step 2: Design and Test a Model Architecture
#
# Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the [German Traffic Sign Dataset](http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset).
#
# The LeNet-5 implementation shown in the [classroom](https://classroom.udacity.com/nanodegrees/nd013/parts/fbf77062-5703-404e-b60c-95b78b2f3f9e/modules/6df7ae49-c61c-4bb2-a23e-6527e69209ec/lessons/601ae704-1035-4287-8b11-e2c2716217ad/concepts/d4aca031-508f-4e0b-b493-e7b706120f81) at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!
#
# With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.
#
# There are various aspects to consider when thinking about this problem:
#
# - Neural network architecture (is the network over or underfitting?)
# - Play around preprocessing techniques (normalization, rgb to grayscale, etc)
# - Number of examples per label (some have more than others).
# - Generate fake data.
#
# Here is an example of a [published baseline model on this problem](http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf). It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

# ### Pre-process the Data Set (normalization, grayscale, etc.)

# Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, `(pixel - 128)/ 128` is a quick way to approximately normalize the data and can be used in this project.
#
# Other pre-processing steps are optional. You can try different techniques to see if it improves performance.
#
# Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.

# In[5]:

### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.
from skimage import color
import matplotlib.pyplot as plt
from sklearn.utils import shuffle

X_train, y_train = shuffle(X_train, y_train)
X_test, y_test = shuffle(X_test, y_test)

X_train = X_train.astype("float32")
X_valid = X_valid.astype("float32")
X_test = X_test.astype("float32")

# Normalize Data
def normalize(data):
    return (data - 128)/128

# grayscale data
X_train = color.rgb2gray(X_train)
X_valid = color.rgb2gray(X_valid)
X_test = color.rgb2gray(X_test)

# some debug information
# img = X_train[0].squeeze()
# plt.figure(figsize=(1,1))
# plt.imshow(img, cmap="gray")
# print(X_train.shape)
# print(X_valid.shape)

# now reshape them
X_train_processed = np.reshape(X_train,(len(X_train),32,32,1))
X_valid_processed = np.reshape(X_valid,(len(X_valid),32,32,1))
X_test_processed = np.reshape(X_test,(len(X_test),32,32,1))
# some debug information to confirm matrix reshape
# print(X_train_processed.shape)
# print(X_valid_processed.shape)
# img = X_train_processed[0].squeeze()
# plt.figure(figsize=(1,1))
# plt.imshow(img, cmap="gray")

X_train_processed = normalize(X_train_processed)
X_valid_processed = normalize(X_valid_processed)
X_test_processed = normalize(X_test_processed)

# print(X_train_processed.shape)
# print(X_valid_processed.shape)
print("PreProcessing Done")

# ### Model Architecture

# In[6]:

### Define your architecture here.
### Feel free to use as many code cells as needed.
# Tunebale parameters
import tensorflow as tf

EPOCHS = 11
BATCH_SIZE = 128
input_channel = 1
print("Hyper Parameter Set")

# In[7]:

from tensorflow.contrib.layers import flatten

def LeNet(x):
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1

    # Layer 1: Convolutional. Input = 32x32x input_channel. Output = 28x28x6.
    # for this filter size = 5, Stride = 1, padding = 0 and number of filter is 6
    F_W = tf.Variable(tf.truncated_normal(shape=(5, 5, input_channel, 6), mean = mu, stddev = sigma))
    F_b = tf.Variable(tf.zeros(6))
    strides = [1, 1, 1, 1]
    padding = 'VALID'
    conv_1_layer = tf.nn.conv2d(x, F_W, strides, padding)
    conv_1_layer = tf.nn.bias_add(conv_1_layer, F_b)

    # Activation. RELU
    conv_1_layer = tf.nn.relu(conv_1_layer)

    # Pooling. Input = 28x28x6. Output = 14x14x6.
    # for this filter size = 2, Stride = 2, padding = 0 and number of filter is 6
    conv_1_pool = tf.nn.max_pool(conv_1_layer, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # Layer 2: Convolutional. Output = 10x10x16.
    # for this filter size = 5, Stride = 1, padding = 0 and number of filter is 16
    F_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
    F_b = tf.Variable(tf.zeros(16))
    strides = [1, 1, 1, 1]
    padding = 'VALID'
    conv_2_layer = tf.nn.conv2d(conv_1_pool, F_W, strides, padding)
    conv_2_layer = tf.nn.bias_add(conv_2_layer, F_b)

    # Activation. RELU
    conv_2_layer = tf.nn.relu(conv_2_layer)
    # Pooling. Input = 10x10x16. Output = 5x5x16.
    # for this filter size = 2, Stride = 2, padding = 0 and number of filter is 6
    conv_2_pool = tf.nn.max_pool(conv_2_layer, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # Flatten. Input = 5x5x16. Output = 400.
    flat_conv = flatten(conv_2_pool)

    # Layer 3: Fully Connected. Input = 400. Output = 120.
    weigth_1 = tf.Variable(tf.truncated_normal(shape=(400,120), mean=mu, stddev=sigma))
    bias_1 = tf.Variable(tf.zeros(120))
    fc1 = tf.add(tf.matmul(flat_conv, weigth_1), bias_1)

    # Activation. RELU
    fc1 = tf.nn.relu(fc1)
    fc1 = tf.nn.dropout(fc1, keep_prob)

    # Layer 4: Fully Connected. Input = 120. Output = 84.
    weigth_2 = tf.Variable(tf.truncated_normal(shape=(120,84), mean=mu, stddev=sigma))
    bias_2 = tf.Variable(tf.zeros(84))
    fc2 = tf.add(tf.matmul(fc1, weigth_2), bias_2)

    # Activation. RELU
    fc2 = tf.nn.relu(fc2)
    fc2 = tf.nn.dropout(fc2, keep_prob)

    # Layer 5: Fully Connected. Input = 84. Output = 43.
    weigth_3 = tf.Variable(tf.truncated_normal(shape=(84,43), mean=mu, stddev=sigma))
    bias_3 = tf.Variable(tf.zeros(43))
    logits = tf.add(tf.matmul(fc2, weigth_3), bias_3)

    return logits

# ### Train, Validate and Test the Model

# A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation
# sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.

# In[8]:

### Train your model here.
### Calculate and report the accuracy on the training and validation set.
### Once a final model architecture is selected,
### the accuracy on the test set should be calculated and reported as well.
### Feel free to use as many code cells as needed.
# `x` is a placeholder for a batch of input images.
# `y` is a placeholder for a batch of output labels.
x = tf.placeholder(tf.float32, (None, 32, 32, input_channel))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)
keep_prob = tf.placeholder(tf.float32)

# In[9]:

# Training Pipeline
rate = 0.001

logits = LeNet(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)

# In[10]:

# ## Model Evaluation
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

# In[12]:

# ## Train the Model
from sklearn.utils import shuffle
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)

    print("Training...")
    print()
    for i in range(EPOCHS):
        X_train_processed, y_train = shuffle(X_train_processed, y_train)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train_processed[offset:end], y_train[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 0.7})

        validation_accuracy = evaluate(X_valid_processed, y_valid)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print()

    saver.save(sess, './lenet_traffic_sign')
    print("Model saved")

# ---
#
# ## Step 3: Test a Model on New Images
#
# To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.
#
# You may find `signnames.csv` useful as it contains mappings from the class id (integer) to the actual sign name.

# In[13]:

# Test the model on test data from above
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test_processed, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))

# ### Load and Output the Images

# In[14]:

### Load the images and plot them here.
### Feel free to use as many code cells as needed.
# open web downloaded images and also do pre-processing as done above
# Convert image to Gray and then normalize images
import glob
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
from skimage import color
import numpy as np
from scipy import misc

# Normalize Data
def normalize_web_images(data):
    return (data - 128)/128

images = glob.glob('GermanDownloadImage/*.png')
X_web_test = np.ndarray(shape=(5,32,32,1), dtype=float)
web_image_idx = 0

y_web_labels = np.array([-1, -1, -1, -1, -1])
for image in images:
#     print(image)
    img = mpimg.imread(image)
    image_resized = misc.imresize(img, (32, 32))
    gray = color.rgb2gray(image_resized).astype("float32")
    gray_reshaped = np.reshape(gray,(32,32,1))
#     plt.figure(figsize=(1,1))
#     plt.imshow(gray)
#     print(gray.shape)
    X_web_test[web_image_idx] = gray_reshaped
    if image == "GermanDownloadImage/ice_road.png":
        y_web_labels[web_image_idx] = 30
    elif image == "GermanDownloadImage/dang_curve_right.png":
        y_web_labels[web_image_idx] = 20
    elif image == "GermanDownloadImage/road_narrow_right.png":
        y_web_labels[web_image_idx] = 24
    elif image == "GermanDownloadImage/dang_curve_left.png":
        y_web_labels[web_image_idx] = 19
    elif image == "GermanDownloadImage/priority_road.png":
        y_web_labels[web_image_idx] = 12
    web_image_idx += 1

X_web_test = normalize_web_images(X_web_test)

print("Web Image Labels:")
print(y_web_labels)

img = X_web_test[2].squeeze()
plt.figure(figsize=(1,1))
plt.imshow(img, cmap="gray")
# print(X_web_test.shape)

# ### Predict the Sign Type for Each Image

# In[15]:

### Run the predictions here and use the model to output the prediction for each image.
### Make sure to pre-process the images with the same pre-processing pipeline used earlier.
### Feel free to use as many code cells as needed.
# as pre-processing is already done so we can go ahead and start evaluation
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    web_test_accuracy = evaluate(X_web_test, y_web_labels)
    print("Test Accuracy = {:.3f}".format(web_test_accuracy))

# ### Analyze Performance

# In[ ]:

### Calculate the accuracy for these 5 new images.
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.
# Accuracy for the web image is shown in previous cell

# ### Output Top 5 Softmax Probabilities For Each Image Found on the Web

# For each of the new images, print out the model's softmax probabilities to show the **certainty** of the model's predictions (limit the output to the top 5 probabilities for each image). [`tf.nn.top_k`](https://www.tensorflow.org/versions/r0.12/api_docs/python/nn.html#top_k) could prove helpful here.
#
# The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.
#
# `tf.nn.top_k` will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.
#
# Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. `tf.nn.top_k` is used to choose the three classes with the highest probability:
#
# ```
# # (5, 6) array
# a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
#          0.12789202],
#        [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
#          0.15899337],
#        [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
#          0.23892179],
#        [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
#          0.16505091],
#        [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
#          0.09155967]])
# ```
#
# Running it through `sess.run(tf.nn.top_k(tf.constant(a), k=3))` produces:
#
# ```
# TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
#        [ 0.28086119,  0.27569815,  0.18063401],
#        [ 0.26076848,  0.23892179,  0.23664738],
#        [ 0.29198961,  0.26234032,  0.16505091],
#        [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
#        [0, 1, 4],
#        [0, 5, 1],
#        [1, 3, 5],
#        [1, 4, 3]], dtype=int32))
# ```
#
# Looking just at the first row we get `[ 0.34763842,  0.24879643,  0.12789202]`, you can confirm these are the 3 largest probabilities in `a`. You'll also notice `[3, 0, 5]` are the corresponding indices.

# In[16]:

### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web.
### Feel free to use as many code cells as needed.
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    softmax_probability = tf.nn.softmax(logits)
    top_5 = tf.nn.top_k(softmax_probability, 5)
    top_5_result = sess.run(top_5, feed_dict={x: X_web_test, keep_prob: 1.0})
    print("Values:")
    print(top_5_result.values)
    print("Indices:")
    print(top_5_result.indices)

# ### Project Writeup
#
# Once you have completed the code implementation, document your results in a project writeup using this [template](https://github.com/udacity/CarND-Traffic-Sign-Classifier-Project/blob/master/writeup_template.md) as a guide. The writeup can be in a markdown or pdf file.

# > **Note**: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to  \n",
#     "**File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission.

# ---
#
# ## Step 4 (Optional): Visualize the Neural Network's State with Test Images
#
#  This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.
#
#  Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the [LeNet lab's](https://classroom.udacity.com/nanodegrees/nd013/parts/fbf77062-5703-404e-b60c-95b78b2f3f9e/modules/6df7ae49-c61c-4bb2-a23e-6527e69209ec/lessons/601ae704-1035-4287-8b11-e2c2716217ad/concepts/d4aca031-508f-4e0b-b493-e7b706120f81) feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.
#
# For an example of what feature map outputs look like, check out NVIDIA's results in their paper [End-to-End Deep Learning for Self-Driving Cars](https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/) in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.
#
# <figure>
#  <img src="visualize_cnn.png" width="380" alt="Combined Image" />
#  <figcaption>
#  <p></p>
#  <p style="text-align: center;"> Your output should look something like this (above)</p>
#  </figcaption>
# </figure>
#  <p></p>
#

# In[ ]:

### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")


#!/usr/bin/env python
# coding: utf-8

#
#

# # PHY321: Classical Mechanics 1
# **Homework 1, due January 21 (midnight)**
#
# Date: **Jan 11, 2022**

# ### Practicalities about  homeworks and projects
#
# 1. You can work in groups (optimal groups are often 2-3 people) or by yourself. If you work as a group you can hand in one answer only if you wish. **Remember to write your name(s)**!
#
# 2. Homeworks (final version) are available approximately ten days before the  deadline.
#
# 3. How do I(we)  hand in?  You can hand in the paper and pencil exercises as a scanned document. For this homework this applies to exercises 1-5. You should upload the scan to D2L. Alternatively, you can hand in everyhting (if you are ok with typing mathematical formulae using say Latex) as a jupyter notebook at D2L. The numerical exercise (exercise 6 here) should always be handed in as a jupyter notebook by the deadline at D2L.

# ### Exercise 1 (12 pt), math reminder, properties of exponential function
#
# The first exercise is meant to remind ourselves about properties of
# the exponential function and imaginary numbers. This is highly
# relevant later in this course when we start analyzing oscillatory
# motion and some wave mechanics. As physicists we should thus feel comfortable with expressions that
# include $\exp{(\imath\omega t)}$. Here $t$ could be interpreted as time and $\omega$ as a frequency and $\imath$ is the imaginary unit number.
#
# * 1a (3pt): Perform Taylor expansions in powers of $\omega t$ of the functions $\cos{(\omega t)}$ and $\sin{(\omega t)}$.
#
# * 1b (3pt): Perform a Taylor expansion of $\exp{(i\omega t)}$.
#
# * 1c (3pt): Using parts (a) and (b) here, show that $\exp{(\imath\omega t)}=\cos{(\omega t)}+\imath\sin{(\omega t)}$.
#
# * 1d (3pt): Show that $\ln{(−1)} = \imath\pi$.

# ### Exercise 2 (12 pt), Vector algebra
#
# * 2a (6pt) One of the many uses of the scalar product is to find the angle between two given vectors. Find the angle between the vectors $\boldsymbol{a}=(1,2,4)$ and $\boldsymbol{b}=(4,2,1)$ by evaluating their scalar product.
#
# * 2b (6pt) For a cube with sides of length 1, one vertex at the origin, and sides along the $x$, $y$, and $z$ axes, the vector of the body diagonal from the origin can be written $\boldsymbol{a}=(1, 1, 1)$ and the vector of the face diagonal in the $xy$ plane from the origin is $\boldsymbol{b}=(1,1,0)$. Find first the lengths of the body diagonal and the face diagonal. Use then part (2a) to find the angle between the body diagonal and the face diagonal.

# ### Exercise 3 (10 pt), More vector mathematics
#
# * 3a (5pt) Show (using the fact that multiplication of reals is distributive) that $\boldsymbol{a}(\boldsymbol{b}+\boldsymbol{c})=\boldsymbol{a}\boldsymbol{b}+\boldsymbol{a}\boldsymbol{c}$.
#
# * 3b (5pt) Show that (using product rule for differentiating reals)  $\frac{d}{dt}(\boldsymbol{a}\boldsymbol{b})=\boldsymbol{a}\frac{d\boldsymbol{b}}{dt}+\boldsymbol{b}\frac{d\boldsymbol{a}}{dt}$

# ### Exercise 4 (10 pt), Algebra of cross products
#
# * 4a (5pt) Show that the cross products are distribuitive $\boldsymbol{a}\times(\boldsymbol{b}+\boldsymbol{c})=\boldsymbol{a}\times\boldsymbol{b}+\boldsymbol{a}\times\boldsymbol{c}$.
#
# * 4b (5pt) Show that $\frac{d}{dt}(\boldsymbol{a}\times\boldsymbol{b})=\boldsymbol{a}\times\frac{d\boldsymbol{b}}{dt}+\frac{d\boldsymbol{a}}{dt}\times \boldsymbol{b}$. Be careful with the order of factors

# ### Exercise 5 (10 pt), Area of triangle and law of sines
#
# Exercise 1.18 in the textbook of Taylor, Classical Mechanics. Part (1.18a) gives 5pt and part (1.18b) gives also 5pt.

# ### Exercise 6 (40pt), Numerical elements, getting started with some simple data
#
# **This exercise should be handed in as a jupyter-notebook** at D2L. Remember to write your name(s).
#
# Our first numerical attempt will involve reading data from file or
# just setting up two vectors, one for position and one for time. Our data are from
# [Usain Bolt's world record 100m during the olympic games in Beijing in
# 2008](https://www.youtube.com/watch?v=93dC0o2aHto). The data show the time used in units of 10m (see below). Before we however
# venture into this, we need to repeat some basic Python syntax with an
# emphasis on
#
# * basic Python syntax for arrays
#
# * define and operate on vectors and matrices in Python
#
# * create plots for motion in 1D space
#
# For more information, see the [introductory slides](https://mhjensen.github.io/Physics321/doc/pub/week2/html/week2.html).
# Here are some of the basic packages we will be using this week

# In[1]:

get_ipython().run_line_magic('matplotlib', 'inline')

import numpy as np
import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')

# The first exercise here deals with simply getting familiar with vectors and matrices.
#
# We will be working with vectors and matrices to get you familiar with them
#
# 1. Initalize two three-dimensional $xyz$ vectors in the below cell using np.array([x,y,z]). Vectors are represented through arrays in python
#
# 2. V1 should have x1=1, y1 =2, and z1=3.
#
# 3. Vector 2 should have x2=4, y2=5,  and z2=6.
#
# 4. Print both vectors to make sure your code is working properly.

# In[2]:

V1 = np.array([1,2,3])
V2 = np.array([4,5,6])
print("V1: ", V1)
print("V2: ", V2)

# If this is not too familiar, here's a useful link for creating vectors in python
# <https://docs.scipy.org/doc/numpy-1.13.0/user/basics.creation.html>. Alternatively, look up the [introductory slides](https://mhjensen.github.io/Physics321/doc/pub/week2/html/week2.html).
#
# Now lets do some basic mathematics with vectors.
#
# Compute and print the following, and double check with hand calculations:
#
# * 6a (2pt)  Calculate $\boldsymbol{V}_1-\boldsymbol{V}_2$.
#
# * 6b (2pt)  Calculate $\boldsymbol{V}_2-\boldsymbol{V}_1$.
#
# * 6c (2pt) Calculate the dot product $\boldsymbol{V}_1\boldsymbol{V}_2$.
#
# * 6d (2pt) Calculate the cross product $\boldsymbol{V}_1\times\boldsymbol{V}_2$.
#
# Here is some useful explanation on numpy array operations if you feel a bit confused by what is happening,
# see <https://www.pluralsight.com/guides/overview-basic-numpy-operations>.
#
# The following code prints the first two exercises

# In[3]:

print(V1-V2)
print(V2-V1)

# For the dot product of V1 and V2 below we can use the **dot** function of **numpy** as follows

# In[4]:

print(V1.dot(V2))

# As a small challenge try to write your own function for the **dot** product of two vectors.
#
# Matrices can be created in a similar fashion in python.  In this
# language we can work with them through the package numpy (which we
# have already imported)

# In[5]:

M1 = np.matrix([[1,2,3],
             [4,5,6],
             [7,8,9]])
M2 = np.matrix([[1,2],
             [3,4],
             [5,6]])
M3 = np.matrix([[9,8,7],
             [4,5,6],
             [7,6,9]])

# Matrices can be added in the same way vectors are added in python as shown here

# In[6]:

print("M1+M3: ", M1+M3)

# What happens if we try to do $M1+M2$?
#
# That's enough vectors and matrices for now. Let's move on to some physics problems! Yes, the actual subject we are studying for.
#
# We can opt for two different ways of handling the data. The data is listed in the table here and represents the total time Usain Bolt used in steps of  10 meters of distance. The label $i$ is just a counter and we start from zero since Python arrays are by default set from zero. The variable $t$ is time in seconds and $x$ is the position in meters.
#
#

#

#
 i  	  0  	  1  	  2  	  3  	  4  	  5  	  6  	  7  	  8  	  9

#

#

#
   x[m]    	    10      	    20      	    30      	    40      	    50      	    60      	    70      	    80      	    90      	    100

#
   t[s]    	    1.85    	    2.87    	    3.78    	    4.65    	    5.50    	    6.32    	    7.14    	    7.96    	    8.79    	    9.69

#

#

#
# * 6e (6pt) You can here make a file with the above data and read them in and set up two vectors, one for time and one for position. Alternatively, you can just set up these two vectors directly and define two vectors in your Python code.
#
# The following example code may help here

# In[7]:

# we just initialize time and position
x = np.array([10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0])
t = np.array([1.85, 2.87, 3.78, 4.65, 5.50, 6.32, 7.14, 7.96, 8.79, 9.69])
plt.plot(t,x, color='black')
plt.xlabel("Time t[s]")
plt.ylabel("Position x[m]")
plt.title("Usain Bolt's world record run")
plt.show()

# * 6f (6pt) Plot the position as function of time
#
# * 6g (10pt) Compute thereafter the mean velocity for every interval $i$ and the total velocity (from $i=0$ to the given interval $i$) for each interval and plot these two quantities as function of time. Comment your results.
#
# * 6h (10pt) Finally, compute and plot the mean acceleration for each interval and the total acceleration. Again, comment your results. Can you see whether he slowed down during the last meters?


# Copyright 2017 Battelle Energy Alliance, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
  This Module performs Unit Tests for the DataSet class.
  It can not be considered part of the active code but of the regression test system
"""

#For future compatibility with Python 3
from __future__ import division, print_function, unicode_literals, absolute_import
import warnings
warnings.simplefilter('default',DeprecationWarning)

import xml.etree.ElementTree as ET
import sys, os
import pickle as pk
import numpy as np
import xarray as xr
import copy

# find location of crow, message handler
frameworkDir = os.path.abspath(os.path.join(*([os.path.dirname(__file__)]+[os.pardir]*4+['framework'])))
sys.path.append(frameworkDir)

from utils.utils import find_crow
find_crow(frameworkDir)
import MessageHandler

import DataObjects

mh = MessageHandler.MessageHandler()
mh.initialize({'verbosity':'debug', 'callerLength':10, 'tagLength':10})

print('Module undergoing testing:')
print(DataObjects.DataSet )
print('')

def createElement(tag,attrib=None,text=None):
  """
    Method to create a dummy xml element readable by the distribution classes
    @ In, tag, string, the node tag
    @ In, attrib, dict, optional, the attribute of the xml node
    @ In, text, str, optional, the dict containig what should be in the xml text
  """
  if attrib is None:
    attrib = {}
  if text is None:
    text = ''
  element = ET.Element(tag,attrib)
  element.text = text
  return element

results = {"pass":0,"fail":0}

def checkFloat(comment,value,expected,tol=1e-10,update=True):
  """
    This method is aimed to compare two floats given a certain tolerance
    @ In, comment, string, a comment printed out if it fails
    @ In, value, float, the value to compare
    @ In, expected, float, the expected value
    @ In, tol, float, optional, the tolerance
    @ In, update, bool, optional, if False then don't update results counter
    @ Out, res, bool, True if same
  """
  if np.isnan(value) and np.isnan(expected):
    res = True
  elif np.isnan(value) or np.isnan(expected):
    res = False
  else:
    res = abs(value - expected) <= tol
  if update:
    if not res:
      print("checking float",comment,'|',value,"!=",expected)
      results["fail"] += 1
    else:
      results["pass"] += 1
  return res

def checkTrue(comment,res,update=True):
  """
    This method is a pass-through for consistency and updating
    @ In, comment, string, a comment printed out if it fails
    @ In, res, bool, the tested value
    @ In, update, bool, optional, if False then don't update results counter
    @ Out, res, bool, True if test
  """
  if update:
    if res:
      results["pass"] += 1
    else:
      print("checking bool",comment,'|',res,'is not True!')
      results["fail"] += 1
  return res

def checkSame(comment,value,expected,update=True):
  """
    This method is aimed to compare two identical things
    @ In, comment, string, a comment printed out if it fails
    @ In, value, float, the value to compare
    @ In, expected, float, the expected value
    @ In, update, bool, optional, if False then don't update results counter
    @ Out, res, bool, True if same
  """
  res = value == expected
  if update:
    if res:
      results["pass"] += 1
    else:
      print("checking string",comment,'|',value,"!=",expected)
      results["fail"] += 1
  return res

def checkArray(comment,first,second,dtype,tol=1e-10,update=True):
  """
    This method is aimed to compare two arrays
    @ In, comment, string, a comment printed out if it fails
    @ In, value, float, the value to compare
    @ In, expected, float, the expected value
    @ In, tol, float, optional, the tolerance
    @ In, update, bool, optional, if False then don't update results counter
    @ Out, res, bool, True if same
  """
  res = True
  if len(first) != len(second):
    res = False
    print("checking answer",comment,'|','lengths do not match:',len(first),len(second))
  else:
    for i in range(len(first)):
      if dtype == float:
        pres = checkFloat('',first[i],second[i],tol,update=False)
      elif dtype.__name__ in ('str','unicode'):
        pres = checkSame('',first[i],second[i],update=False)
      if not pres:
        print('checking array',comment,'|','entry "{}" does not match: {} != {}'.format(i,first[i],second[i]))
        res = False
  if update:
    if res:
      results["pass"] += 1
    else:
      results["fail"] += 1
  return res

def checkRlz(comment,first,second,tol=1e-10,update=True,skip=None):
  """
    This method is aimed to compare two realization
    @ In, comment, string, a comment printed out if it fails
    @ In, first, dict, the first dict, the "calculated" value -> should be as obtained from the data object
    @ In, second, dict, the second dict, the "expected" value -> should be as a realization submitted
    @ In, tol, float, optional, the tolerance
    @ In, update, bool, optional, if False then don't update results counter
    @ In, skip, list, optional, keywords not to check
    @ Out, res, bool, True if same
  """
  if skip is None:
    skip = []
  res = True
  if abs(len(first) - len(second)) > len(skip):
    res = False
    print("checking answer",comment,'|','lengths do not match:',len(first),len(second))
  else:
    for key,val in first.items():
      if key in skip:
        continue
      if isinstance(val,(float,int,np.int64,np.int32)):
        pres = checkFloat('',val,second[key][0],tol,update=False)
      elif type(val).__name__ in ('str','unicode','str_','unicode_'):
        pres = checkSame('',val,second[key][0],update=False)
      elif isinstance(val,np.ndarray):
        if isinstance(val[0],(float,int)):
          pres = (val - second[key]).sum()<1e-20 #necessary due to roundoff
        else:
          pres = val == second[key]
      elif isinstance(val,xr.DataArray):
        if isinstance(val.item(0),(float,int)):
          pres = (val - second[key]).sum()<1e-20 #necessary due to roundoff
        else:
          pres = val.equals(second[key])
      else:
        raise TypeError(type(val))
      if not pres:
        print('checking dict',comment,'|','entry "{}" does not match: {} != {}'.format(key,first[key],second[key]))
        res = False
  if update:
    if res:
      results["pass"] += 1
    else:
      results["fail"] += 1
  return res

def checkNone(comment,entry,update=True):
  """
    Checks if entry is None.
    @ In, comment, string, a comment printed out if it fails
    @ In, entry, object, to test if against None
    @ In, update, bool, optional, if False then don't update results counter
    @ Out, res, bool, True if None
  """
  res = entry is None
  if update:
    if res:
      results["pass"] += 1
    else:
      print("checking answer",comment,'|','"{}" is not None!'.format(entry))
      results["fail"] += 1

def checkFails(comment,errstr,function,update=True,args=None,kwargs=None):
  """
    Checks if expected error occurs
    @ In, comment, string, a comment printed out if it fails
    @ In, errstr, str, expected fail message
    @ In, function, method, method to run to test for failure
    @ In, update, bool, optional, if False then don't update results counter
    @ In, args, list, arguments to pass to function
    @ In, kwargs, dict, keyword arguments to pass to function
    @ Out, res, bool, True if failed as expected
  """
  print('Error testing ...')
  if args is None:
    args = []
  if kwargs is None:
    kwargs = {}
  try:
    function(*args,**kwargs)
    res = False
    msg = 'Function call did not error!'
  except Exception as e:
    res = checkSame('',e.args[0],errstr,update=False)
    if not res:
      msg = 'Unexpected error message.  \n    Received: "{}"\n    Expected: "{}"'.format(e.args[0],errstr)
  if update:
    if res:
      results["pass"] += 1
      print(' ... end Error testing (PASSED)')
    else:
      print("checking error",comment,'|',msg)
      results["fail"] += 1
      print(' ... end Error testing (FAILED)')
  print('')
  return res

def formatRealization(rlz):
  """
    Converts types of each input.
    @ In, rlz, dict, var:val
    @ Out, rlz, dict, formatted
  """
  for k,v in rlz.items():
    rlz[k] = np.atleast_1d(v)

######################################
#            CONSTRUCTION            #
######################################
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input',text='a,b,c'))
xml.append(createElement('Output',text='x,y,z'))
xml.append(createElement('Index',attrib={'var':'time'},text='c,y'))

# check construction
data = DataObjects.DataSet()
# inputs, outputs
checkSame('DataSet __init__ name',data.name,'DataSet')
checkSame('DataSet __init__ print tag',data.printTag,'DataSet')
checkNone('DataSet __init__ _data',data._data)
checkNone('DataSet __init__ _collector',data._collector)

# check initialization
data.messageHandler = mh
data._readMoreXML(xml)
# NOTE histories are currently disabled pending future work (c,y are history vars)
checkArray('DataSet __init__ inp',data._inputs,['a','b','c'],str)
checkArray('DataSet __init__ out',data._outputs,['x','y','z'],str)
checkArray('DataSet __init__ all',data.vars,['a','b','c','x','y','z'],str)
checkNone('DataSet __init__ _data',data._data)
checkNone('DataSet __init__ _collector',data._collector)

######################################
#    SAMPLING AND APPENDING DATA     #
######################################
# test ND construction
vals = np.array([[1.0,1.1,1.2],[2.0,2.1,2.2]])
right = xr.DataArray(vals,dims=['x','time'],coords={'time':[1e-6,2e-6,3e06],'x':[1e-3,2e-3]})
dims = ['x','time']
coords = {'time':[1e-6,2e-6,3e06],'x':[1e-3,2e-3]}
test = data.constructNDSample(vals,dims,coords)
checkTrue('ND instance construction',test.equals(right))

# append some data to get started
# TODO expand this to ND not just History
data.addExpectedMeta(['prefix'])
rlz0 = {'a': 1.0,
        'b': 2.0,
        'c': np.array([3.0, 3.1, 3.2]),
        'x': 4.0,
        'y': np.array([5.0, 5.1, 5.2]),
        'prefix': 'first',
        'time':np.array([3.1e-6,3.2e-6,3.3e-6]),
       }
rlz1 = {'a' :11.0,
        'b': 12.0,
        'c': [13.0, 13.1, 13.2],
        'x': 14.0,
        'y': [15.0, 15.1, 15.2],
        'z': 16.0,
        'prefix': 'second',
        'time':[13.1e-6,13.2e-6,13.3e-6],
       }
rlz2 = {'a' :21.0,
        'b': 22.0,
        'c': [23.0, 23.1, 23.2],
        'x': 24.0,
        'y': [25.0, 25.1, 25.2],
        'z': 26.0,
        'prefix': 'third',
        'time':[23.1e-6,23.2e-6,23.3e-6],
       }
formatRealization(rlz0)
formatRealization(rlz1)
formatRealization(rlz2)
# test missing data
rlzMissing = dict(rlz0)
rlz0['z'] = 6.0
formatRealization(rlz0)
checkFails('DataSet addRealization err missing','Provided realization does not have all requisite values for object \"DataSet\": \"z\"',data.addRealization,args=[rlzMissing])
# bad formatting
rlzFormat = dict(rlz0)
rlzFormat['c'] = list(rlzFormat['c'])
checkFails('DataSet addRealization err format','Realization was not formatted correctly for "DataSet"! See warnings above.',data.addRealization,args=[rlzFormat])
# test appending
data.addRealization(dict(rlz0))

# get realization by index, from collector
checkRlz('Dataset append 0',data.realization(index=0),rlz0,skip=['time'])
# try to access the inaccessible
checkFails('DataSet get nonexistant realization by index','DataSet: Requested index "1" but only have 1 entries (zero-indexed)!',data.realization,kwargs={'index':1})
# add more data
data.addRealization(dict(rlz1))
data.addRealization(dict(rlz2))
# get realization by index
checkRlz('Dataset append 1 idx 0',data.realization(index=0),rlz0,skip=['time'])
checkRlz('Dataset append 1 idx 1',data.realization(index=1),rlz1,skip=['time'])
checkRlz('Dataset append 1 idx 2',data.realization(index=2),rlz2,skip=['time'])
######################################
#      GET MATCHING REALIZATION      #
######################################
m,match = data.realization(matchDict={'a':11.0})
checkSame('Dataset append 1 match index',m,1)
checkRlz('Dataset append 1 match',match,rlz1,skip=['time'])
idx,rlz = data.realization(matchDict={'x':1.0})
checkSame('Dataset find bogus match index',idx,3)
checkNone('Dataset find bogus match',rlz)
# TODO more checks on reading collector, writing to file, etc

######################################
#        COLLAPSING DATA SET         #
######################################
# collapse dataset
data.asDataset()
# check sample tag IDs
checkArray('Dataset first collapse sample IDs',data._data['RAVEN_sample_ID'].values,[0,1,2],float)
# check time coordinate
times = [ 3.1e-6, 3.2e-6, 3.3e-6,
         13.1e-6,13.2e-6,13.3e-6,
         23.1e-6,23.2e-6,23.3e-6]
checkArray('Dataset first collapse "time"',data._data['time'].values,times,float)
# check values for scalars "a"
checkArray('Dataset first collapse "a"',data._data['a'].values,[1.0,11.0,21.0],float)
# check values for timeset "c"
c = np.array(
    [           [ 3.0, 3.1, 3.2]+[np.nan]*6,
     [np.nan]*3+[13.0,13.1,13.2]+[np.nan]*3,
     [np.nan]*6+[23.0,23.1,23.2]           ])
checkArray('Dataset first collapse "c" 0',data._data['c'].values[0],c[0],float)
checkArray('Dataset first collapse "c" 1',data._data['c'].values[1],c[1],float)
checkArray('Dataset first collapse "c" 2',data._data['c'].values[2],c[2],float)
# check values for timeset "y"
y = np.array(
    [           [ 5.0, 5.1, 5.2]+[np.nan]*6,
     [np.nan]*3+[15.0,15.1,15.2]+[np.nan]*3,
     [np.nan]*6+[25.0,25.1,25.2]           ])
checkArray('Dataset first collapse "y" 0',data._data['y'].values[0],y[0],float)
checkArray('Dataset first collapse "y" 1',data._data['y'].values[1],y[1],float)
checkArray('Dataset first collapse "y" 2',data._data['y'].values[2],y[2],float)
# check values for metadata prefix (unicode, not float)
checkArray('Dataset first collapse "prefix"',data._data['prefix'].values,['first','second','third'],str)

# get dimensions
checkSame('Dataset getDimensions "None" num entries',len(data.getDimensions()),7)
checkArray('Dataset getDimensions "None" entry "c"',data.getDimensions()['c'],['time'],str)
checkArray('Dataset getDimensions "c"',data.getDimensions('c')['c'],['time'],str)
checkSame('Dataset getDimensions "inp" num entries',len(data.getDimensions('input')),3)
checkArray('Dataset getDimensions "inp" entry "c"',data.getDimensions('input')['c'],['time'],str)
checkSame('Dataset getDimensions "out" num entries',len(data.getDimensions('output')),3)
checkArray('Dataset getDimensions "out" entry "y"',data.getDimensions('output')['y'],['time'],str)
checkSame('Dataset getDimensions "dummy" num entries',len(data.getDimensions('dummy')),1)
######################################
#     SAMPLING AFTER COLLAPSING      #
######################################
# take a couple new samples to test simultaneous collector and data
# use the same time stamps as rlz0 to test same coords
rlz3 = {'a' :31.0,
        'b': 32.0,
        'c': [33.0, 33.1, 33.2],
        'x': 34.0,
        'y': [35.0, 35.1, 35.2],
        'z': 36.0,
        'prefix': 'fourth',
        'time':[ 3.1e-6, 3.2e-6, 3.3e-6],
       }
formatRealization(rlz3)
data.addRealization(dict(rlz3))
# get new entry (should be in the collector, but we shouldn't care)
checkRlz('Dataset append 2 idx 3',data.realization(index=3),rlz3,skip=['time'])
# make sure old entry is still there
checkRlz('Dataset append 2 idx 1',data.realization(index=1),rlz1,skip=['time'])
# test grabbing negative indices
checkRlz('Dataset append 2 idx -1',data.realization(index=-1),rlz3,skip=['time'])
checkRlz('Dataset append 2 idx -3',data.realization(index=-3),rlz1,skip=['time'])

data.asDataset()
# check new sample IDs
checkArray('Dataset first collapse sample IDs',data._data['RAVEN_sample_ID'].values,[0,1,2,3],float)
# "times" should not have changed
times = [ 3.1e-6, 3.2e-6, 3.3e-6,
         13.1e-6,13.2e-6,13.3e-6,
         23.1e-6,23.2e-6,23.3e-6]
#checkArray('Dataset first collapse "time"',data._data['time'].values,times,float)
# check new "a"
checkArray('Dataset first collapse "a"',data._data['a'].values,[1.0,11.0,21.0,31.0],float)
# check new "c"
c = np.array(
    [           [ 3.0, 3.1, 3.2]+[np.nan]*6,
     [np.nan]*3+[13.0,13.1,13.2]+[np.nan]*3,
     [np.nan]*6+[23.0,23.1,23.2]           ,
                [33.0,33.1,33.2]+[np.nan]*6])
checkArray('Dataset post collapse "c" 0',data._data['c'].values[0],c[0],float)
checkArray('Dataset post collapse "c" 1',data._data['c'].values[1],c[1],float)
checkArray('Dataset post collapse "c" 2',data._data['c'].values[2],c[2],float)
checkArray('Dataset post collapse "c" 3',data._data['c'].values[3],c[3],float)
# check string prefix
checkArray('Dataset post collapse "prefix"',data._data['prefix'].values,['first','second','third','fourth'],str)

# check removing variable from collector and data
rlz4 = {'a' :41.0,
        'b': 42.0,
        'c': [43.0, 43.1, 43.2],
        'x': 44.0,
        'y': [45.0, 45.1, 45.2],
        'z': 46.0,
        'prefix': 'five',
        'time':[ 4.1e-6, 4.2e-6, 4.3e-6],
       }
######################################
#         GENERAL META DATA          #
######################################
# add scalar metadata
data.addMeta('TestPP',{'firstVar':{'scalarMetric1':10.0,
                                   'scalarMetric2':'20',
                                   'vectorMetric':{'a':1,'b':'2','c':u'3','d':4.0}
                                   },
                       'secondVar':{'scalarMetric1':100.}
                      })
# directly test contents, without using API
checkSame('Metadata top level entries',len(data._meta),2)
treePP = data._meta['TestPP'].getRoot()
checkSame('Metadata TestPP',treePP.tag,'TestPP')
first,second = (c for c in treePP) # TODO always same order?

checkSame('Metadata TestPP/firstVar tag',first.tag,'firstVar')
sm1,sm2,vm = (c for c in first) # TODO always same order?
checkSame('Metadata TestPP/firstVar/scalarMetric1 tag',sm1.tag,'scalarMetric1')
checkSame('Metadata TestPP/firstVar/scalarMetric1 value',sm1.text,'10.0')
checkSame('Metadata TestPP/firstVar/scalarMetric2 tag',sm2.tag,'scalarMetric2')
checkSame('Metadata TestPP/firstVar/scalarMetric2 value',sm2.text,'20')
checkSame('Metadata TestPP/firstVar/vectorMetric tag',vm.tag,'vectorMetric')
for child in vm:
  if child.tag == 'a':
    checkSame('Metadata TestPP/firstVar/vectorMetric/a value',child.text,'1')
  elif child.tag == 'b':
    checkSame('Metadata TestPP/firstVar/vectorMetric/b value',child.text,'2')
  elif child.tag == 'c':
    checkSame('Metadata TestPP/firstVar/vectorMetric/c value',child.text,'3')
  elif child.tag == 'd':
    checkSame('Metadata TestPP/firstVar/vectorMetric/d value',child.text,'4.0')
  else:
    checkTrue('Unexpected node in TestPP/firstVar/vectorMetric nodes: '+child.text,False)

checkSame('Metadata TestPP/secondVar tag',second.tag,'secondVar')
checkSame('Metadata TestPP/secondVar entries',len(second),1)
child = second[0]
checkSame('Metadata TestPP/secondVar/scalarMetric1 tag',child.tag,'scalarMetric1')
checkSame('Metadata TestPP/secondVar/scalarMetric1 value',child.text,'100.0')

treeDS = data._meta['DataSet'].getRoot()
checkSame('Metadata DataSet',treeDS.tag,'DataSet')
checkSame('Metadata DataSet entries',len(treeDS),2)
dims,general = treeDS[:]
checkSame('Metadata DataSet/dims tag',dims.tag,'dims')
checkSame('Metadata DataSet/dims entries',len(dims),2)
c,y = dims[:]
checkSame('Metadata DataSet/dims/y tag',y.tag,'y')
checkSame('Metadata DataSet/dims/y value',y.text,'time')
checkSame('Metadata DataSet/dims/c tag',c.tag,'c')
checkSame('Metadata DataSet/dims/c value',c.text,'time')
checkSame('Metadata DataSet/general tag',general.tag,'general')
checkSame('Metadata DataSet/general entries',len(general),4)
inputs, outputs, pointwise_meta, sampleTag = general[:]
checkSame('Metadata DataSet/general/inputs tag',inputs.tag,'inputs')
checkSame('Metadata DataSet/general/inputs value',inputs.text,'a,b,c')
checkSame('Metadata DataSet/general/outputs tag',outputs.tag,'outputs')
checkSame('Metadata DataSet/general/outputs value',outputs.text,'x,y,z')
checkSame('Metadata DataSet/general/pointwise_meta tag',pointwise_meta.tag,'pointwise_meta')
checkSame('Metadata DataSet/general/pointwise_meta value',pointwise_meta.text,'prefix')
checkSame('Metadata DataSet/general/sampleTag tag',sampleTag.tag,'sampleTag')
checkSame('Metadata DataSet/general/sampleTag value',sampleTag.text,'RAVEN_sample_ID')

# use getters to access contents (using API)
meta = data.getMeta(pointwise=True,general=True)
checkArray('Metadata get keys',sorted(meta.keys()),['DataSet','TestPP','prefix'],str)
# fail to find pointwise in general
checkFails('Metadata get missing general','Some requested keys could not be found in the requested metadata: (prefix)',data.getMeta,kwargs=dict(keys=['prefix'],general=True))
# fail to find general in pointwise
checkFails('Metadata get missing general','Some requested keys could not be found in the requested metadata: (DataSet)',data.getMeta,kwargs=dict(keys=['DataSet'],pointwise=True))
# check that poorly-aligned set checks out as such
checkTrue('Check misaligned data is not aligned',not data.checkIndexAlignment())
# check aligned data too
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input',text='a'))
xml.append(createElement('Output',text='b'))
xml.append(createElement('Index',attrib={'var':'t'},text='b'))
dataAlign = DataObjects.DataSet()
dataAlign.messageHandler = mh
dataAlign._readMoreXML(xml)
rlz = {'a':np.array([1.9]),
       'b':np.array([3.4, 2.4, 6.5]),
       't':np.array([0.4, 0.9, 10])}
dataAlign.addRealization(rlz)
rlz = {'a':np.array([7.9]),
       'b':np.array([0.3, -0.8, 9.7]),
       't':np.array([0.4, 0.9, 10])}
dataAlign.addRealization(rlz)
checkTrue('Check aligned data is aligned', dataAlign.checkIndexAlignment('t'))

######################################
#        READ/WRITE FROM FILE        #
######################################
# to netCDF
# NOTE: due to a cool little seg fault error in netCDF4 versions less than 1.3.1, we cannot test it currently.
# Leaving implementation for the future.
#netname = 'DataSetUnitTest.nc'
#data.write(netname,style='netcdf',format='NETCDF4') # WARNING this will fail if netCDF4 not installed
#checkTrue('Wrote to netcdf',os.path.isfile(netname))
## read fresh from netCDF
#dataNET = DataSet.DataSet()
#dataNET.messageHandler = mh
#dataNET.load(netname,style='netcdf')
# validity of load is checked below, in ACCESS USING GETTERS section

# to CSV
## test writing to file
csvname = 'DataSetUnitTest'
data.write(csvname,style='CSV',**{'what':'a,b,c,x,y,z,RAVEN_sample_ID,prefix'})
## test metadata written
correct = ['<DataObjectMetadata name="DataSet">',
           '  <DataSet type="Static">',
           '    <dims>',
           '      <c>time</c>',
           '      <y>time</y>',
           '    </dims>',
           '    <general>',
           '      <inputs>a,b,c</inputs>',
           '      <outputs>x,y,z</outputs>',
           '      <pointwise_meta>prefix</pointwise_meta>',
           '      <sampleTag>RAVEN_sample_ID</sampleTag>',
           '    </general>',
           '  </DataSet>',
           '  ',
           '  <TestPP type="Static">',
           '    <firstVar>',
           '      <scalarMetric1>10.0</scalarMetric1>',
           '      <scalarMetric2>20</scalarMetric2>',
           '      <vectorMetric>',
           '        <a>1</a>',
           '        <b>2</b>',
           '        <c>3</c>',
           '        <d>4.0</d>',
           '      </vectorMetric>',
           '    </firstVar>',
           '    <secondVar>',
           '      <scalarMetric1>100.0</scalarMetric1>',
           '    </secondVar>',
           '  </TestPP>',
           '  ',
           '</DataObjectMetadata>']
# read in XML
lines = open(csvname+'.xml','r').readlines()
# remove line endings
for l,line in enumerate(lines):
  lines[l] = line.rstrip(os.linesep).rstrip('\n')
# check
checkArray('CSV XML',lines,correct,str)
## read from CSV/XML
xml = createElement('DataSet',attrib={'name':'csv'})
# scramble the IO space, also skip 'z' for testing
xml.append(createElement('Input',text='a,x,y'))
xml.append(createElement('Output',text='c,b'))
xml.append(createElement('Index',attrib={'var':'t'},text='y,c'))
dataCSV = DataObjects.DataSet()
dataCSV.messageHandler = mh
dataCSV._readMoreXML(xml)
dataCSV.load(csvname,style='CSV')

for var in data.getVars():
  if var == 'z':
    # not included in XML input specs, so should be left out
    checkFails('CSV var z','z',dataCSV.getVarValues,args=var)
  elif isinstance(data.getVarValues(var).item(0),(float,int)):
    checkTrue('CSV var {}'.format(var),(dataCSV._data[var] - data._data[var]).sum()<1e-20) #necessary due to roundoff
  else:
    checkTrue('CSV var {}'.format(var),bool((dataCSV._data[var] == data._data[var]).prod()))

# clean up temp files
os.remove(csvname+'.csv')
os.remove(csvname+'.xml')

######################################
#        ACCESS USING GETTERS        #
######################################
# test contents of data in parallel
# by index
checkRlz('Dataset full origin idx 1',data.realization(index=1),rlz1,skip=['time'])
#checkRlz('Dataset full netcdf idx 1',dataNET.realization(index=1),rlz1,skip=['time'])
checkRlz('Dataset full csvxml idx 1',dataCSV.realization(index=1),rlz1,skip=['time','z'])
# by match
idx,rlz = data.realization(matchDict={'prefix':'third'})
checkSame('Dataset full origin match idx',idx,2)
checkRlz('Dataset full origin match',rlz,rlz2,skip=['time'])
#idx,rlz = dataNET.realization(matchDict={'prefix':'third'})
#checkSame('Dataset full netcdf match idx',idx,2)
#checkRlz('Dataset full netCDF match',rlz,rlz2,skip=['time'])
idx,rlz = dataCSV.realization(matchDict={'prefix':'third'})
checkSame('Dataset full csvxml match idx',idx,2)
checkRlz('Dataset full csvxml match',rlz,rlz2,skip=['time','z'])
# TODO metadata checks?

## remove files, for cleanliness (comment out to debug)
## first have to close file (for Windows' sake)
#dataNET._data.close()
#os.remove(netname)

######################################
#        ADDING NEW VARIABLE         #
######################################
f = np.array([9., 19., 29., 39.])
data.addVariable('f',f)
rlzAdd = dict(rlz2)
rlzAdd['f'] = np.atleast_1d(29.)
checkArray('Dataset add variable column',data.asDataset()['f'].values,f,float)
checkRlz('Dataset add variable rlz 2',data.realization(index=2),rlzAdd,skip='time')

######################################
#           SLICE BY INDEX           #
######################################
slices = data.sliceByIndex('time')
checkFloat('Index slicing "time" [2] "time"',slices[2]['time'].item(0),3.3e-6)
checkArray('Index slicing "time" [2] "a"',slices[2]['a'].values,[1.0, 11.0, 21.0, 31.0],float)
checkArray('Index slicing "time" [2] "c"',slices[2]['c'].values,[3.2, np.nan, np.nan, 33.2],float)

slices = data.sliceByIndex('RAVEN_sample_ID')
checkFloat('Index slicing sampleTag [3] sampleTag',slices[3]['RAVEN_sample_ID'].item(0),3)
checkFloat('Index slicing sampleTag [3] "a"',slices[3]['a'].values,31.0)
checkArray('Index slicing sampleTag [3] "c"',slices[3]['c'].values,[33.0,33.1,33.2,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan],float)

######################################
#        CONSTRUCT FROM DICT         #
######################################
seed = {}
# vector variable, 10 entries with arbitrary lengths
seed['b'] = np.array([ np.array([1.00]),
                       np.array([1.10, 1.11]),
                       np.array([1.20, 1.21, 1.22]),
                       np.array([1.30, 1.31, 1.32, 1.33]),
                       np.array([1.40, 1.41, 1.42, 1.43, 1.44]),
                       np.array([1.50, 1.51, 1.52, 1.53, 1.54, 1.55]),
                       np.array([1.60, 1.61, 1.62, 1.63, 1.64, 1.65, 1.66]),
                       np.array([1.70, 1.71, 1.72, 1.73, 1.74, 1.75, 1.76, 1.77]),
                       np.array([1.80, 1.81, 1.82, 1.83, 1.84, 1.85, 1.86, 1.87, 1.88]),
                       np.array([1.90, 1.91, 1.92, 1.93, 1.94, 1.95, 1.96, 1.97, 1.98, 1.99])
                       ])
# coordinate, as vector
seed['t'] = np.array([ np.linspace(0,1,1),
                       np.linspace(0,1,2),
                       np.linspace(0,1,3),
                       np.linspace(0,1,4),
                       np.linspace(0,1,5),
                       np.linspace(0,1,6),
                       np.linspace(0,1,7),
                       np.linspace(0,1,8),
                       np.linspace(0,1,9),
                       np.linspace(0,1,10) ])
# set up data object
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input',text='a'))
xml.append(createElement('Output',text='b'))
xml.append(createElement('Index',attrib={'var':'t'},text='b'))
data = DataObjects.DataSet()
data.messageHandler = mh
data._readMoreXML(xml)
# load with insufficient values
checkFails('Load from dict missing variable','Variables are missing from "source" that are required for data object " DataSet ": a',data.load,args=[seed],kwargs=dict(style='dict',dims=data.getDimensions()))
# add a scalar variable, 10 entries
seed['a'] = np.array([1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9])
# load properly
data.load(seed,style='dict',dims=data.getDimensions())
# test contents
checkArray('load from dict "a"',data.asDataset()['a'].values,seed['a'],float)
checkArray('load from dict "b"[3]',data.asDataset().isel({'RAVEN_sample_ID':3},True)['b'].dropna('t').values,seed['b'][3],float)
rlz = data.realization(index=2)
checkFloat('load from dict rlz 2 "a"',rlz['a'],1.2)
checkArray('load from dict rlz 2 "b"',rlz['b'].values,[1.2,1.21,1.22],float)
# Here I am testing the functionality that converts the dataObject into a dict
convertedDict = data.asDataset(outType='dict')
# check that the dictionary entries are the same
checkArray('asDict "a"',seed['a'],convertedDict['data']['a'],float)
checkArray('asDict "b[0]"',convertedDict['data']['b'][0],seed['b'][0],float)
checkArray('asDict "b[4]"',convertedDict['data']['b'][4],seed['b'][4],float)
checkArray('asDict "b[9]"',convertedDict['data']['b'][9],seed['b'][9],float)
checkArray('asDict "t[0]"',convertedDict['data']['t'][0],seed['t'][0],float)
checkArray('asDict "t[4]"',convertedDict['data']['t'][4],seed['t'][4],float)
checkArray('asDict "t[9]"',convertedDict['data']['t'][9],seed['t'][9],float)
checkSame('asDict dims "a"',convertedDict['dims']['a'],[])
checkSame('asDict dims "b"',convertedDict['dims']['b'],['t'])
# TODO check metadata?
# double-check there's no errors using this to construct a new dataset (full loop)
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input',text='a'))
xml.append(createElement('Output',text='b'))
xml.append(createElement('Index',attrib={'var':'t'},text='b'))
dataRe = DataObjects.DataSet()
dataRe.messageHandler = mh
dataRe._readMoreXML(xml)
dataRe.load(convertedDict['data'],style='dict',dims=convertedDict['dims'])
# use exact same tests as originally loading from dict, but for dataRe
checkArray('load from dict "a"',dataRe.asDataset()['a'].values,seed['a'],float)
checkArray('load from dict "b"[3]',dataRe.asDataset().isel({'RAVEN_sample_ID':3},True)['b'].dropna('t').values,seed['b'][3],float)
rlz = dataRe.realization(index=2)
checkFloat('load from dict rlz 2 "a"',rlz['a'],1.2)
checkArray('load from dict rlz 2 "b"',rlz['b'].values,[1.2,1.21,1.22],float)

# construct from dict, but data is provided in numpy ND array instead of np array of np array objects
seed = {}
seed['a'] = np.array([0., 1., 2., 3.])
seed['b'] = np.array([ [0.1, 0.2, 0.3],
                       [1.1, 1.2, 1.3],
                       [2.1, 2.2, 2.3],
                       [3.1, 3.2, 3.3]])
seed['t'] = np.array([ [1e-6,2e-6,3e-6],
                       [1e-6,2e-6,3e-6],
                       [1e-6,2e-6,3e-6],
                       [1e-6,2e-6,3e-6]])
# set up data object
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input',text='a'))
xml.append(createElement('Output',text='b'))
xml.append(createElement('Index',attrib={'var':'t'},text='b'))
data = DataObjects.DataSet()
data.messageHandler = mh
data._readMoreXML(xml)
# load
data.load(seed,style='dict',dims=data.getDimensions())
# check data
checkArray('load from dict of ND, "a"',data.asDataset()['a'].values,seed['a'],float)
checkArray('load from dict of ND, "b[0]"',data.asDataset()['b'][0].values,seed['b'][0],float)
checkArray('load from dict of ND, "b[1]"',data.asDataset()['b'][1].values,seed['b'][1],float)
checkArray('load from dict of ND, "b[2]"',data.asDataset()['b'][2].values,seed['b'][2],float)
checkArray('load from dict of ND, "b[3]"',data.asDataset()['b'][3].values,seed['b'][3],float)

######################################
#        REMOVING VARIABLES          #
######################################
# first, add a sample so the variable is also in the collector
rlz = {'a':np.atleast_1d(2.0), 'b':np.array([2.0,2.1,2.2]), 't':np.linspace(0,1,3)}
data.addRealization(rlz)
del rlz['b']
del rlz['t']
checkArray('Remove variable starting vars',data.getVars(),['a','b'],str)
data.remove(variable='b')
checkArray('Remove variable remaining vars',data.getVars(),['a'],str)
checkRlz('Remove variable rlz -1',data.realization(index=-1),rlz)
# collapse and re-check
data.asDataset()
checkArray('Remove variable remaining vars',data.getVars(),['a'],str)
checkRlz('Remove variable rlz -1, ds',data.realization(index=-1),rlz)
# check we can add a new realization
data.addRealization({'a':np.array([2.1]), 't':np.array([0])})

######################################
#          CLUSTER LABELING          #
######################################
# as used by the Optimizer, for example.  We store as a flat point set, then
#   divide up by cluster label for printing.
# create data object
xml = createElement('PointSet',attrib={'name':'test'})
xml.append(createElement('Input',text='a,b'))
xml.append(createElement('Output',text='x,y'))
data = DataObjects.DataSet()
data.messageHandler = mh
data._readMoreXML(xml)
# register "trajID" (cluster label) and "varsUpdate" (iteration number/monotonically increasing var) as meta
data.addExpectedMeta(['trajID','varsUpdate'])
# add two trajectories to get started, like starting two trajectories
rlz0_0 = {'trajID': np.array([1]),
          'a': np.array([  1.0]),
          'b': np.array([  5.0]),
          'x': np.array([ 10.0]),
          'y': np.array([100.0]),
          'varsUpdate': np.array([0])}
rlz1_0 = {'trajID': np.array([2]),
          'a': np.array([  2.0]),
          'b': np.array([  6.0]),
          'x': np.array([ 20.0]),
          'y': np.array([200.0]),
          'varsUpdate': np.array([0])}
data.addRealization(rlz0_0)
data.addRealization(rlz1_0)
checkRlz('Cluster initial traj 1',data.realization(index=0),rlz0_0,skip='varsUpdate')
checkRlz('Cluster initial traj 2',data.realization(index=1),rlz1_0,skip='varsUpdate')
# now sample a new trajectory point, going into the collector
rlz0_1 = {'trajID': np.array([1]),
          'a': np.array([  1.1]),
          'b': np.array([  5.1]),
          'x': np.array([ 10.1]),
          'y': np.array([100.1]),
          'varsUpdate': np.array([1])}
data.addRealization(rlz0_1)
checkRlz('Cluster extend traj 1[0]',data.realization(matchDict={'trajID':1,'varsUpdate':0})[1],rlz0_0,skip='varsUpdate')
checkRlz('Cluster extend traj 1[1]',data.realization(matchDict={'trajID':1,'varsUpdate':1})[1],rlz0_1,skip='varsUpdate')
checkRlz('Cluster extend traj 2[0]',data.realization(matchDict={'trajID':2,'varsUpdate':0})[1],rlz1_0,skip='varsUpdate')
# now collapse and then append to the data
data.asDataset()
rlz1_1 = {'trajID': np.array([    2]),
               'a': np.array([  2.1]),
               'b': np.array([  6.1]),
               'x': np.array([ 20.1]),
               'y': np.array([200.1]),
          'varsUpdate': np.array([1])}
data.addRealization(rlz1_1)
tid = data._collector[-1,data._orderedVars.index('trajID')]
checkRlz('Cluster extend traj 2[1]',data.realization(matchDict={'trajID':2,'varsUpdate':1})[1],rlz1_1,skip='varsUpdate')
# print it
fname = 'DataUnitTestClusterLabels'
data.write(fname,style='csv',clusterLabel='trajID')
# manually check contents
for l,line in enumerate(open(fname+'.csv','r')):
  if l == 0:
    checkSame('Cluster CSV main [0]',line.strip(),'trajID,filename')
  elif l == 1:
    checkSame('Cluster CSV main [1]',line.strip(),'1,{}_1.csv'.format(fname))
  elif l == 2:
    checkSame('Cluster CSV main [2]',line.strip(),'2,{}_2.csv'.format(fname))
for l,line in enumerate(open(fname+'_1.csv','r')):
  if l == 0:
    checkSame('Cluster CSV id1 [0]',line.strip(),'RAVEN_sample_ID,a,b,x,y,varsUpdate')
  elif l == 1:
    line = list(float(x) for x in line.split(','))
    checkArray('Cluster CSV id1 [1]',line,[0,1.0,5.0,10.0,100.0,0],float)
  elif l == 2:
    line = list(float(x) for x in line.split(','))
    checkArray('Cluster CSV id1 [1]',line,[2,1.1,5.1,10.1,100.1,1],float)
for l,line in enumerate(open(fname+'_2.csv','r')):
  if l == 0:
    checkSame('Cluster CSV id1 [0]',line.strip(),'RAVEN_sample_ID,a,b,x,y,varsUpdate')
  elif l == 1:
    line = list(float(x) for x in line.split(','))
    checkArray('Cluster CSV id2 [1]',line,[1,2.0,6.0,20.0,200.0,0],float)
  elif l == 2:
    line = list(float(x) for x in line.split(','))
    checkArray('Cluster CSV id2 [1]',line,[3,2.1,6.1,20.1,200.1,1],float)
# load it as a history # TODO first, loading needs to be fixed to use DataObject params instead of XML params
xml = createElement('HistorySet',attrib={'name':'test'})
xml.append(createElement('Input',text='trajID'))
xml.append(createElement('Output',text='a,b,x,y'))
options = createElement('options')
options.append(createElement('pivotParameter',text='varsUpdate'))
xml.append(options)
data2 = DataObjects.HistorySet()
data2.messageHandler = mh
data2._readMoreXML(xml)
data2.load(fname,style='csv')
# check data is correct by realization
correct = {'a':np.array([  1.0,  1.1]),
           'b':np.array([  5.0,  5.1]),
           'x':np.array([ 10.0, 10.1]),
           'y':np.array([100.0,100.1]),
           'trajID':np.array([1])}
checkRlz('Cluster read [0]',data2.realization(index=0),correct)
correct = {'a':np.array([  2.0,  2.1]),
           'b':np.array([  6.0,  6.1]),
           'x':np.array([ 20.0, 20.1]),
           'y':np.array([200.0,200.1]),
           'trajID':np.array([2])}
checkRlz('Cluster read [1]',data2.realization(index=1),correct)

######################################
#             DATA TYPING            #
######################################
## check that types are set correctly, both for histories and scalars
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input', text=' fl, in, st, un, bo'))
xml.append(createElement('Output',text='dfl,din,dst,dun,dbo'))
xml.append(createElement('Index',attrib={'var':'t'},text='dfl,din,dst,dun,dbo'))
data = DataObjects.DataSet()
data.messageHandler = mh
data._readMoreXML(xml)

rlz = {'fl' :np.array([   1.0]),
       'in' :np.array([     2]),
       'st' :np.array([ 'msg']),
       'un' :np.array([u'utf']),
       'bo' :np.array([  True]),
       'dfl':np.array([ 1.0,   2.0,  3.0]),
       'din':np.array([   4,     5,    6]),
       'dst':np.array([ 'a',   'b',  'c']),
       'dun':np.array([ u'x', u'y', u'z']),
       'dbo':np.array([ True,False, True]),
         't':np.array(['one','two','three'])}
rlz2= {'fl' :np.array([   10.0]),
       'in' :np.array([     20]),
       'st' :np.array([ 'msg2']),
       'un' :np.array([u'utf2']),
       'bo' :np.array([  False]),
       'dfl':np.array([ 10.0,   20.0,  30.0]),
       'din':np.array([   40,     50,    60]),
       'dst':np.array([ 'a2',   'b2',  'c2']),
       'dun':np.array([ u'x2', u'y2', u'z2']),
       'dbo':np.array([ False,  True, False]),
         't':np.array(['one','two','manystringchars'])}
data.addRealization(rlz)
data.asDataset()
# check types
for var in rlz.keys():
  correct = rlz[var].dtype
  if correct.type in [np.unicode_,np.string_,str]:
    correct = object
  checkSame('dtype checking "{}"'.format(var),data.asDataset()[var].dtype,correct)

data.addRealization(rlz2)

######################################
#          DATA RENAMING             #
######################################
# use the renaming, needed for alias operations
xml = createElement('DataSet',attrib={'name':'test'})
xml.append(createElement('Input', text='a,b'))
xml.append(createElement('Output',text='c,d'))
xml.append(createElement('Index',attrib={'var':'t'},text='b,d'))
data = DataObjects.DataSet()
data.messageHandler = mh
data._readMoreXML(xml)
data.addExpectedMeta(['prefix'])
rlz0 = {'a':np.array([0.0]),
        'b':np.array([0.0, 0.1, 0.2]),
        'c':np.array([0.5]),
        'd':np.array([0.5, 0.6, 0.7]),
        'prefix':np.array(['0']),
        't':np.array([0.01, 0.02, 0.03])}
data.addRealization(rlz0)
# make a copy, to test renaming with just collector
data2 = copy.deepcopy(data)
data2.renameVariable('a','alpha')
data2.renameVariable('b','beta')
data2.renameVariable('c','gamma')
data2.renameVariable('d','delta')
data2.renameVariable('prefix','jobID')
data2.renameVariable('t','timelike')
# check everything was changed
correct0 = {'alpha':np.array([0.0]),
            'beta':np.array([0.0, 0.1, 0.2]),
            'gamma':np.array([0.5]),
            'delta':np.array([0.5, 0.6, 0.7]),
            'jobID':np.array(['0']),
            'timelike':np.array([0.01, 0.02, 0.03])}
checkRlz('Rename in collector: variables',data2.realization(index=0),correct0,skip='timelike')
checkArray('Rename in collector: index',data2.indexes,['timelike'],str)

# now asDataset(), then rename
data3 = copy.deepcopy(data)
data3.asDataset()
data3.renameVariable('a','alpha')
data3.renameVariable('b','beta')
data3.renameVariable('c','gamma')
data3.renameVariable('d','delta')
data3.renameVariable('prefix','jobID')
data3.renameVariable('t','timelike')
checkRlz('Rename in dataset: variables',data3.realization(index=0),correct0,skip='timelike')
checkArray('Rename in dataset: index',data3.indexes,['timelike'],str)

# now asDatset, then append, then rename
data.asDataset()
rlz1 = {'a':np.array([1.0]),
        'b':np.array([1.0, 1.1, 1.2]),
        'c':np.array([1.5]),
        'd':np.array([1.5, 1.6, 1.7]),
        'prefix':np.array(['1']),
        't':np.array([0.01, 0.02, 0.03])}
data.addRealization(rlz1)
data.renameVariable('a','alpha')
data.renameVariable('b','beta')
data.renameVariable('c','gamma')
data.renameVariable('d','delta')
data.renameVariable('prefix','jobID')
data.renameVariable('t','timelike')
correct1 = {'alpha':np.array([1.0]),
            'beta':np.array([1.0, 1.1, 1.2]),
            'gamma':np.array([1.5]),
            'delta':np.array([1.5, 1.6, 1.7]),
            'jobID':np.array(['1']),
            'timelike':np.array([0.01, 0.02, 0.03])}
checkRlz('Rename in both: variables[0]',data.realization(index=0),correct0,skip='timelike')
checkRlz('Rename in both: variables[1]',data.realization(index=1),correct1,skip='timelike')
checkArray('Rename in both: index',data.indexes,['timelike'],str)
# make sure adding a new realization without the renaming fails
checkFails('Add old-named data after renaming variables','Provided realization does not have all requisite values for object \"DataSet\": \"alpha\"',data.addRealization,args=[rlz1])

print(results)

sys.exit(results["fail"])
"""
  <TestInfo>
    <name>framework.test_datasets</name>
    <author>talbpaul</author>
    <created>2017-10-20</created>
    <classesTested>DataSet</classesTested>
    <description>
       This test is a Unit Test for the DataSet classes.
    </description>
  </TestInfo>
"""