#!/usr/bin/env python
# coding: utf-8

# # Self-Driving Car Engineer Nanodegree
#
# ## Deep Learning
#
# ## Project: Build a Traffic Sign Recognition Classifier
#
# In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.
#
# > **Note**: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to  \n",
#     "**File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission.
#
# In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a [write up template](https://github.com/udacity/CarND-Traffic-Sign-Classifier-Project/blob/master/writeup_template.md) that can be used to guide the writing process. Completing the code template and writeup template will cover all of the [rubric points](https://review.udacity.com/#!/rubrics/481/view) for this project.
#
# The [rubric](https://review.udacity.com/#!/rubrics/481/view) contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.
#
#
# >**Note:** Code and Markdown cells can be executed using the **Shift + Enter** keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.

# ---
# ## Step 0: Load The Data

# In[1]:

# Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data

training_file = "Data/train.p"
validation_file= "Data/valid.p"
testing_file = "Data/test.p"

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)

X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

# ---
#
# ## Step 1: Dataset Summary & Exploration
#
# The pickled data is a dictionary with 4 key/value pairs:
#
# - `'features'` is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
# - `'labels'` is a 1D array containing the label/class id of the traffic sign. The file `signnames.csv` contains id -> name mappings for each id.
# - `'sizes'` is a list containing tuples, (width, height) representing the original width and height the image.
# - `'coords'` is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. **THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES**
#
# Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the [pandas shape method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shape.html) might be useful for calculating some of the summary results.

# ### Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

# In[2]:

### Replace each question mark with the appropriate value.
### Use python, pandas or numpy methods rather than hard coding the results

# TODO: Number of training examples
n_train = format(len(X_train))

# TODO: Number of validation examples
n_validation = format(len(X_valid))

# TODO: Number of testing examples.
n_test = format(len(X_test))

# TODO: What's the shape of an traffic sign image?
image_shape = format(X_train[0].shape)

# TODO: How many unique classes/labels there are in the dataset.
n_classes = 43

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)

# ### Include an exploratory visualization of the dataset

# Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
#
# The [Matplotlib](http://matplotlib.org/) [examples](http://matplotlib.org/examples/index.html) and [gallery](http://matplotlib.org/gallery.html) pages are a great resource for doing visualizations in Python.
#
# **NOTE:** It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?

# In[3]:

### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import random
import numpy as np
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
get_ipython().run_line_magic('matplotlib', 'inline')

index1 = random.randint(0, len(X_train))
image1 = X_train[index1].squeeze()

index2 = random.randint(0, len(X_train))
image2 = X_train[index2].squeeze()

plt.figure(figsize=(1,1))
plt.imshow(image1,cmap='gray')

plt.figure(figsize=(1,1))
plt.imshow(image2,cmap='gray')
print(y_train[index1], y_train[index2])

# In[4]:

hist, bins = np.histogram(y_train, bins=n_classes)
width = 0.7 * (bins[1] - bins[0])
center = (bins[:-1] + bins[1:]) / 2
plt.bar(center, hist, align='center', width=width)
plt.show()

# ----
#
# ## Step 2: Design and Test a Model Architecture
#
# Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the [German Traffic Sign Dataset](http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset).
#
# The LeNet-5 implementation shown in the [classroom](https://classroom.udacity.com/nanodegrees/nd013/parts/fbf77062-5703-404e-b60c-95b78b2f3f9e/modules/6df7ae49-c61c-4bb2-a23e-6527e69209ec/lessons/601ae704-1035-4287-8b11-e2c2716217ad/concepts/d4aca031-508f-4e0b-b493-e7b706120f81) at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!
#
# With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.
#
# There are various aspects to consider when thinking about this problem:
#
# - Neural network architecture (is the network over or underfitting?)
# - Play around preprocessing techniques (normalization, rgb to grayscale, etc)
# - Number of examples per label (some have more than others).
# - Generate fake data.
#
# Here is an example of a [published baseline model on this problem](http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf). It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

# ### Pre-process the Data Set (normalization, grayscale, etc.)

# Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, `(pixel - 128)/ 128` is a quick way to approximately normalize the data and can be used in this project.
#
# Other pre-processing steps are optional. You can try different techniques to see if it improves performance.
#
# Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.

# In[5]:

### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.
from sklearn.utils import shuffle
import cv2
import tensorflow as tf

def convert_gray(image):
    new_image = np.asarray(image)
    new_image_gray = np.sum(image/3,axis=3,keepdims=True)
    return new_image_gray

def normalize(image):
    return np.divide(np.subtract(image,128),128)

def scale_image(image):
    #print(image.shape)
    row,col,_ = image.shape
    x = np.random.randint(-2,2)
    pts1 = np.float32([[0,0],[row,0],[0,col],[row,col]])
    pts2 = np.float32([[0+x,0+x],[row-x,0+x],[0+x,col-x],[row-x,col-x]])
    M = cv2.getPerspectiveTransform(pts2,pts1)
    dst = cv2.warpPerspective(image,M,(row,col))
    dst = dst[:,:,np.newaxis]

    #print(dst.shape)
    return dst

def rotate_image(image):
    image_center = tuple(np.array(image.shape[1::-1]) / 2)
    rot_mat = cv2.getRotationMatrix2D(image_center, 90, 1.0)
    result = cv2.warpAffine(image, rot_mat, image.shape[1::-1], flags=cv2.INTER_LINEAR)
    result = result[:,:,np.newaxis]
    return result

def random_brightness(image):
    shifted = image + 1.0   # shift to (0,2) range
    img_max_value = max(shifted.flatten())
    max_coef = 2.0/img_max_value
    min_coef = max_coef - 0.1
    coef = np.random.uniform(min_coef, max_coef)
    dst = shifted * coef - 1.0
    return dst

X_train_gray = convert_gray(X_train)
X_test_gray = convert_gray(X_test)
X_valid_gray = convert_gray(X_valid)

X_train_norm = normalize(X_train_gray)
X_test_norm = normalize(X_test_gray)
X_valid_norm = normalize(X_valid_gray)

test_img = X_train_norm[2234]

X_train_rot = rotate_image(test_img)
X_train_rand = random_brightness(test_img)
X_train_scale = scale_image(test_img)
#X_train_gray = convert_gray(test_img)

#plt.figure(figsize=(1,1))
#plt.imshow(X_train_rot)

#plt.figure(figsize=(1,1))
#plt.imshow(X_train_rand)

#plt.figure(figsize=(1,1))
#plt.imshow(X_train_scale)

#plt.figure(figsize=(1,1))
#plt.imshow(X_train_gray)

print("X_train_rot shape:",X_train_rot.shape)

input_indices = []
output_indices = []

for class_n in range(n_classes):
    class_indices = np.where(y_train == class_n)
    n_samples = len(class_indices[0])
    if n_samples < 800:
        for i in range(800 - n_samples):
            input_indices.append(class_indices[0][i%n_samples])
            output_indices.append(X_train_norm.shape[0])
            new_img = X_train_norm[class_indices[0][i % n_samples]]
            new_img = rotate_image(scale_image(random_brightness(new_img)))
        X_train_norm = np.concatenate((X_train_norm, [new_img]), axis=0)
        y_train = np.concatenate((y_train, [class_n]), axis=0)

print('X, y shapes:', X_train_norm.shape, y_train.shape)

# In[6]:

from sklearn.model_selection import train_test_split

X_train, X_valid, y_train, y_valid = train_test_split(X_train_norm, y_train, test_size=0.20, random_state=42)

print("Old X_train size:",len(X_train_norm))
print("New X_train size:",len(X_train))
print("X_validation size:",len(X_valid))

# ### Model Architecture

# In[7]:

### Define your architecture here.
### Feel free to use as many code cells as needed.

from tensorflow.contrib.layers import flatten

def conv2d(x, W, b, strides=1):
    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='VALID')
    x = tf.nn.bias_add(x, b)
    return tf.nn.relu(x)

def maxpool2d(x, k=2):
    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1], padding='SAME')

def LeNet(x):
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1
    n_classes = 43

    weights = {
    'wc1': tf.Variable(tf.truncated_normal(shape = (5, 5, 1, 6), mean = mu, stddev = sigma)),
    'wc2': tf.Variable(tf.truncated_normal(shape = (5, 5, 6, 16), mean = mu, stddev = sigma)),
    'wd1': tf.Variable(tf.truncated_normal(shape = (400, 400), mean = mu, stddev = sigma)),
    'wd2': tf.Variable(tf.truncated_normal(shape = (400,120), mean = mu, stddev = sigma)),
    'wd3': tf.Variable(tf.truncated_normal(shape = (120,84), mean = mu, stddev = sigma)),
    'out': tf.Variable(tf.truncated_normal(shape = (84, n_classes), mean = mu, stddev = sigma))}

    biases = {
    'bc1': tf.Variable(tf.zeros([6])),
    'bc2': tf.Variable(tf.zeros([16])),
    'bd1': tf.Variable(tf.zeros([400])),
    'bd2': tf.Variable(tf.zeros([120])),
    'bd3': tf.Variable(tf.zeros([84])),
    'out': tf.Variable(tf.zeros([n_classes]))}

    lay1 = conv2d(x, weights['wc1'], biases['bc1'])
    lay1 = maxpool2d(lay1, k=2)

    lay2 = conv2d(lay1, weights['wc2'], biases['bc2'])
    lay2 = maxpool2d(lay2, k=2)

    fc0 = flatten(lay2)
    fc1 = tf.add(tf.matmul(fc0, weights['wd1']), biases['bd1'])
    fc1 = tf.nn.relu(fc1)

    fc2 = tf.add(tf.matmul(fc1, weights['wd2']), biases['bd2'])
    fc2 = tf.nn.relu(fc2)

    fc3 = tf.add(tf.matmul(fc2, weights['wd3']), biases['bd3'])
    fc3 = tf.nn.relu(fc3)

    logits = tf.add(tf.matmul(fc3, weights['out']), biases['out'])

    return logits

# ### Train, Validate and Test the Model

# A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation
# sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.

# In[8]:

### Train your model here.
### Calculate and report the accuracy on the training and validation set.
### Once a final model architecture is selected,
### the accuracy on the test set should be calculated and reported as well.
### Feel free to use as many code cells as needed.

EPOCHS = 10
BATCH_SIZE = 128

x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.int32, (None))
keep_prob = tf.placeholder(tf.float32)
one_hot_y = tf.one_hot(y, 43)

# In[9]:

rate = 0.001

logits = LeNet(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)

# In[12]:

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_prob:1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples

# In[13]:

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)

    print("Training...")
    print()
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y})

        validation_accuracy = evaluate(X_valid, y_valid)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print()

    saver.save(sess, './lenet')
    print("Model saved")

# ---
#
# ## Step 3: Test a Model on New Images
#
# To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.
#
# You may find `signnames.csv` useful as it contains mappings from the class id (integer) to the actual sign name.

# In[14]:

#with tf.Session() as sess:
    #sess.run(tf.global_variables_initializer())
    #saver2 = tf.train.import_meta_graph('./lenet.meta')
    #saver2.restore(sess, "./lenet")
    #test_accuracy = evaluate(X_test_norm, y_test)
    #print("Test Set Accuracy = {:.3f}".format(test_accuracy))

# ### Load and Output the Images

# In[15]:

### Load the images and plot them here.
### Feel free to use as many code cells as needed.

import glob
import cv2
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

fig, axs = plt.subplots(2,3, figsize=(4,2))
fig.subplots_adjust(hspace = .2, wspace=.001)
axs = axs.ravel()

gts_images = []

for i, img in enumerate(glob.glob('GTS example images/*.png')):
    new_image = cv2.imread(img)
    axs[i].axis('off')
    axs[i].imshow(cv2.cvtColor(new_image, cv2.COLOR_BGR2RGB))
    gts_images.append(new_image)

gts_images = np.asarray(gts_images)

gts_images_gray = np.sum(gts_images/3, axis = 3, keepdims=True)

gts_images_norm = (gts_images_gray-128)/128

print(gts_images_norm.shape)

labels = [0,28,40,17,39,32]

label_id = np.asarray(labels)

print(label_id.shape)

# In[16]:

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver2 = tf.train.import_meta_graph('./lenet.meta')
    saver2.restore(sess, "./lenet")
    test_accuracy = evaluate(gts_images_norm, label_id)
    print("Test Set Accuracy = {:.3f}".format(test_accuracy))

# ### Predict the Sign Type for Each Image

# In[17]:

### Run the predictions here and use the model to output the prediction for each image.
### Make sure to pre-process the images with the same pre-processing pipeline used earlier.
### Feel free to use as many code cells as needed.

# ### Analyze Performance

# In[18]:

### Calculate the accuracy for these 5 new images.
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.

softmax_logits = tf.nn.softmax(logits)
top_k = tf.nn.top_k(softmax_logits, k=3)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver = tf.train.import_meta_graph('./lenet.meta')
    saver.restore(sess, "./lenet")
    my_softmax_logits = sess.run(softmax_logits, feed_dict={x: gts_images_norm, keep_prob: 1.0})
    my_top_k = sess.run(top_k, feed_dict={x: gts_images_norm, keep_prob: 1.0})

    fig, axs = plt.subplots(len(gts_images),4, figsize=(12, 14))
    fig.subplots_adjust(hspace = .4, wspace=.2)
    axs = axs.ravel()

    for i, image in enumerate(gts_images):
        axs[4*i].axis('off')
        axs[4*i].imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
        axs[4*i].set_title('input')
        guess1 = my_top_k[1][i][0]
        index1 = np.argwhere(y_train == guess1)[0]
        axs[4*i+1].axis('off')
        axs[4*i+1].imshow(X_train[index1].squeeze(), cmap='gray')
        axs[4*i+1].set_title('top guess: {} ({:.0f}%)'.format(guess1, 100*my_top_k[0][i][0]))
        guess2 = my_top_k[1][i][1]
        index2 = np.argwhere(y_train == guess2)[0]
        axs[4*i+2].axis('off')
        axs[4*i+2].imshow(X_train[index2].squeeze(), cmap='gray')
        axs[4*i+2].set_title('2nd guess: {} ({:.0f}%)'.format(guess1, 100*my_top_k[0][i][1]))
        guess3 = my_top_k[1][i][2]
        index3 = np.argwhere(y_train == guess3)[0]
        axs[4*i+3].axis('off')
        axs[4*i+3].imshow(X_train[index3].squeeze(), cmap='gray')
        axs[4*i+3].set_title('3rd guess: {} ({:.0f}%)'.format(guess1, 100*my_top_k[0][i][2]))

# ### Output Top 5 Softmax Probabilities For Each Image Found on the Web

# For each of the new images, print out the model's softmax probabilities to show the **certainty** of the model's predictions (limit the output to the top 5 probabilities for each image). [`tf.nn.top_k`](https://www.tensorflow.org/versions/r0.12/api_docs/python/nn.html#top_k) could prove helpful here.
#
# The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.
#
# `tf.nn.top_k` will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.
#
# Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. `tf.nn.top_k` is used to choose the three classes with the highest probability:
#
# ```
# # (5, 6) array
# a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
#          0.12789202],
#        [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
#          0.15899337],
#        [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
#          0.23892179],
#        [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
#          0.16505091],
#        [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
#          0.09155967]])
# ```
#
# Running it through `sess.run(tf.nn.top_k(tf.constant(a), k=3))` produces:
#
# ```
# TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
#        [ 0.28086119,  0.27569815,  0.18063401],
#        [ 0.26076848,  0.23892179,  0.23664738],
#        [ 0.29198961,  0.26234032,  0.16505091],
#        [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
#        [0, 1, 4],
#        [0, 5, 1],
#        [1, 3, 5],
#        [1, 4, 3]], dtype=int32))
# ```
#
# Looking just at the first row we get `[ 0.34763842,  0.24879643,  0.12789202]`, you can confirm these are the 3 largest probabilities in `a`. You'll also notice `[3, 0, 5]` are the corresponding indices.

# In[ ]:

### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web.
### Feel free to use as many code cells as needed.

# ### Project Writeup
#
# Once you have completed the code implementation, document your results in a project writeup using this [template](https://github.com/udacity/CarND-Traffic-Sign-Classifier-Project/blob/master/writeup_template.md) as a guide. The writeup can be in a markdown or pdf file.

# > **Note**: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to  \n",
#     "**File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission.

# ---
#
# ## Step 4 (Optional): Visualize the Neural Network's State with Test Images
#
#  This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.
#
#  Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the [LeNet lab's](https://classroom.udacity.com/nanodegrees/nd013/parts/fbf77062-5703-404e-b60c-95b78b2f3f9e/modules/6df7ae49-c61c-4bb2-a23e-6527e69209ec/lessons/601ae704-1035-4287-8b11-e2c2716217ad/concepts/d4aca031-508f-4e0b-b493-e7b706120f81) feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.
#
# For an example of what feature map outputs look like, check out NVIDIA's results in their paper [End-to-End Deep Learning for Self-Driving Cars](https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/) in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.
#
# <figure>
#  <img src="visualize_cnn.png" width="380" alt="Combined Image" />
#  <figcaption>
#  <p></p>
#  <p style="text-align: center;"> Your output should look something like this (above)</p>
#  </figcaption>
# </figure>
#  <p></p>
#

# In[19]:

### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")

# In[ ]:

# In[ ]:


#!/usr/bin/env python
# license removed for brevity
import rospy
from std_msgs.msg import String
from control_msgs.msg import JointTrajectoryControllerState
from control_msgs.msg import FollowJointTrajectoryActionGoal
from trajectory_msgs.msg import JointTrajectoryPoint
import math
import numpy as np

def talker():
    rospy.loginfo('starting the talker node')
    global counter
    rospy.init_node('talker', anonymous=True)
    rate = rospy.Rate(10) # 10hz

    pub= rospy.Publisher('/arm_controller/follow_joint_trajectory/goal', FollowJointTrajectoryActionGoal, queue_size=10)

    rospy.sleep(2)

    command = FollowJointTrajectoryActionGoal()
    command.header.stamp = rospy.Time.now()
    command.goal.trajectory.joint_names = ['elbow']
    point = JointTrajectoryPoint()
    point.positions=[math.pi/2]
    point.velocities=[1.5] #0.5
    #point.positions = [math.sin(counter/4)*math.pi]
    # in which range is sinus defined?
    point.time_from_start = rospy.Duration(10)
    command.goal.trajectory.points.append(point)

    pub.publish(command)
    rospy.loginfo('=====send command %r', command.goal.trajectory.points[0])

    rospy.sleep(2)
    counter+=1

    rate.sleep()

if __name__ == '__main__':
    try:
        global counter
        counter=0
        talker()

    except rospy.ROSInterruptException:
        pass


"""
Soft Actor-Critic Algorithms and Applications:
    https://arxiv.org/abs/1812.05905
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor:
   https://arxiv.org/abs/1801.01290
"""
from math import floor
from cv2 import exp
from mani_skill_learn.env.builder import build_replay
from mani_skill_learn.env.replay_buffer import ReplayMemory
import numpy as np
from numpy.core.numeric import ones
import torch
from torch._C import device
import torch.nn as nn
import torch.nn.functional as F

from mani_skill_learn.networks import build_model, hard_update, soft_update
from mani_skill_learn.optimizers import build_optimizer

from mani_skill_learn.utils.data import to_torch,to_np,merge_dict
from ..builder import MFRL
from mani_skill_learn.utils.torch import BaseAgent

@MFRL.register_module()
class GAIL(BaseAgent):
    def __init__(self, policy_cfg, value_cfg, discriminator_cfg, obs_shape, action_shape, action_space, batch_size=128, discrim_batch=1024, gamma=0.99,
                 update_coeff=0.005, alpha=0.2, target_update_interval=1, automatic_alpha_tuning=True,use_expert=0,
                 alpha_optim_cfg=None):
        super(GAIL, self).__init__()
        policy_optim_cfg = policy_cfg.pop("optim_cfg")
        value_optim_cfg = value_cfg.pop("optim_cfg")
        discriminator_optim_cfg = discriminator_cfg.pop("optim_cfg")

        self.gamma = gamma
        self.update_coeff = update_coeff
        self.alpha = alpha
        self.batch_size = batch_size
        self.discrim_batch = discrim_batch
        self.target_update_interval = target_update_interval
        self.automatic_alpha_tuning = automatic_alpha_tuning
        self.use_expert = use_expert

        policy_cfg['obs_shape'] = obs_shape
        policy_cfg['action_shape'] = action_shape
        policy_cfg['action_space'] = action_space
        value_cfg['obs_shape'] = obs_shape
        value_cfg['action_shape'] = action_shape
        discriminator_cfg['obs_shape'] = obs_shape
        discriminator_cfg['action_shape'] = action_shape

        self.policy = build_model(policy_cfg)
        self.critic = build_model(value_cfg)
        self.discriminator = build_model(discriminator_cfg)
        self.discrim_criterion = nn.BCELoss()

        self.target_critic = build_model(value_cfg)
        hard_update(self.target_critic, self.critic)

        self.log_alpha = nn.Parameter(torch.zeros(1, requires_grad=True))
        self.target_entropy = -np.prod(action_shape)
        if self.automatic_alpha_tuning:
            self.alpha = self.log_alpha.exp().item()

        self.alpha_optim = build_optimizer(self.log_alpha, alpha_optim_cfg)
        self.policy_optim = build_optimizer(self.policy, policy_optim_cfg)
        self.critic_optim = build_optimizer(self.critic, value_optim_cfg)
        self.discriminator_optim = build_optimizer(self.discriminator, discriminator_optim_cfg)

    def update_discriminator(self, expert_replay, tmp_replay, expert_split = False,
                             progressive_PN = False, progressive_TN = False, PN_alpha = 0, enable_TN_progressive = False, TN_inc_iter = 0):
        if expert_split:
            assert type(expert_replay) is dict, f'expert_replay should be the dict type, {type(expert_replay)} found instead'
            num_expert_replay_is_not_null=0
            if expert_replay is not None:
                for env_id in expert_replay.keys():
                    if(len(expert_replay[env_id])>0):
                        num_expert_replay_is_not_null+=1
            if num_expert_replay_is_not_null==0:
                return 0,0 #! 还没有成功过
            # else:
            #     print("we are training discriminator")
            tmp_split_replay_cfg = dict(
                type='ReplayMemory',
                capacity=self.discrim_batch,
            )
            tmp_split_replay = build_replay(tmp_split_replay_cfg)
            tmp_split_replay.reset()
            assert num_expert_replay_is_not_null>0, f' you shouldn\'t get here '
            single_batchsize = floor(self.discrim_batch/num_expert_replay_is_not_null)
            for env_id in expert_replay.keys():
                assert type(expert_replay[env_id]) is ReplayMemory, f'element of expert_replay should be the ReplayMemory type, {type(expert_rep)} found instead'
                if(len(expert_replay[env_id])>0):
                    tmp_batch = expert_replay[env_id].sample(single_batchsize)
                    tmp_split_replay.push_batch(**tmp_batch)
            expert_sampled_batch = tmp_split_replay.get_all()
            tmp_sampled_batch = tmp_replay.sample(single_batchsize * num_expert_replay_is_not_null)
        else:
            expert_sampled_batch = expert_replay.sample(self.discrim_batch)
            tmp_sampled_batch = tmp_replay.sample(self.discrim_batch)
        expert_sampled_batch = to_torch(expert_sampled_batch, dtype='float32', device=self.device, non_blocking=True)
        tmp_sampled_batch = to_torch(tmp_sampled_batch, dtype='float32', device=self.device, non_blocking=True)

        expert_out = self.discriminator(expert_sampled_batch['obs'], expert_sampled_batch['actions'])
        tmp_out = self.discriminator(tmp_sampled_batch['obs'], tmp_sampled_batch['actions'])

        self.discriminator_optim.zero_grad()
        expert_loss = self.discrim_criterion(expert_out, torch.zeros((expert_out.shape[0],1), device = self.device))
        tmp_loss = self.discrim_criterion(tmp_out, torch.ones((tmp_out.shape[0],1), device=self.device))
        # print(torch.mean(expert_loss),torch.mean(tmp_loss))
        discrim_loss = expert_loss + tmp_loss

        discrim_loss = discrim_loss.mean()
        discrim_loss.backward()
        self.discriminator_optim.step()

        return expert_loss.mean().item(),  tmp_loss.mean().item()

    def expert_reward(self, obs, action,
                      progressive_PN = False, progressive_TN = False, PN_alpha = 0, enable_TN_progressive = False, TN_inc_iter = 0):
        obs = to_torch(obs, dtype='float32', device=self.device, non_blocking=True)
        action = to_torch(action, dtype='float32', device=self.device, non_blocking=True)
        # print(torch.mean(self.discriminator(obs, action)))
        exr = -torch.log(self.discriminator(obs, action))
        return exr.cpu().detach().numpy()

    def return_feature(self, sampled_batch):
        sampled_batch = to_torch(sampled_batch, dtype='float32', device=self.device, non_blocking=True)
        features = self.discriminator(sampled_batch['obs'],sampled_batch['actions'], feature_only = True)
        return features.cpu().detach().numpy()

    def update_parameters(self, memory, updates,expert_replay=None,
                          progressive_PN = False, progressive_TN = False, PN_alpha = 0, enable_TN_progressive = False, TN_inc_iter = 0, env_r = 1):
        num_expert_replay_is_not_null=0
        if expert_replay is not None:
            for env_id in expert_replay.keys():
                if(len(expert_replay[env_id])>0):
                    num_expert_replay_is_not_null+=1
        beta_each = 0.01 if num_expert_replay_is_not_null>0 else 0
        if self.use_expert:
            sampled_batch3=[]
            expert_data = 0
            for env_id in expert_replay.keys():
                if(len(expert_replay[env_id])>0):
                    data_to_sample=min(len(expert_replay[env_id]),int(self.batch_size*beta_each))
                    expert_data += data_to_sample
                    sampled_batch3.append(expert_replay[env_id].sample(data_to_sample))
            sampled_batch = memory.sample(self.batch_size-expert_data)

            for key in sampled_batch:
                if not isinstance(sampled_batch[key], dict) and sampled_batch[key].ndim == 1:
                    sampled_batch[key] = sampled_batch[key][..., None]

            for i in range(len(sampled_batch3)):
                for key in sampled_batch3[i]:
                    if not isinstance(sampled_batch3[i][key], dict) and sampled_batch3[i][key].ndim == 1:
                        sampled_batch3[i][key] = sampled_batch3[i][key][..., None]
                sampled_batch=merge_dict(sampled_batch,sampled_batch3[i])

        else:
            sampled_batch = memory.sample(self.batch_size)
            sampled_batch = to_torch(sampled_batch, dtype='float32', device=self.device, non_blocking=True)

        sampled_batch = to_torch(sampled_batch, dtype='float32', device=self.device, non_blocking=True)
        for key in sampled_batch:
            if not isinstance(sampled_batch[key], dict) and sampled_batch[key].ndim == 1:
                sampled_batch[key] = sampled_batch[key][..., None]

        with torch.no_grad():
            if num_expert_replay_is_not_null>0:
                expert_rewards = -torch.log(self.discriminator(sampled_batch['obs'], sampled_batch['actions']))
                assert not torch.isnan(expert_rewards.any()) ,f'something strange happens'
                # print(torch.isnan(expert_rewards).int().sum())
                # print(torch.mean(expert_rewards))
                # print(sampled_batch['rewards'])
                sampled_batch['rewards'] =sampled_batch['rewards'] + expert_rewards
            # print(sampled_batch['next_obs'])
            next_action, next_log_prob = self.policy(sampled_batch['next_obs'], mode='all')[:2]
            q_next_target = self.target_critic(sampled_batch['next_obs'], next_action)
            min_q_next_target = torch.min(q_next_target, dim=-1, keepdim=True).values - self.alpha * next_log_prob
            q_target = sampled_batch['rewards'] + (1 - sampled_batch['dones']) * self.gamma * min_q_next_target
        q = self.critic(sampled_batch['obs'], sampled_batch['actions'])
        critic_loss = F.mse_loss(q, q_target.repeat(1, q.shape[-1])) * q.shape[-1]
        abs_critic_error = torch.abs(q - q_target.repeat(1, q.shape[-1]))

        self.critic_optim.zero_grad()
        critic_loss.backward()
        self.critic_optim.step()

        pi, log_pi = self.policy(sampled_batch['obs'], mode='all')[:2]
        q_pi = self.critic(sampled_batch['obs'], pi)
        q_pi_min = torch.min(q_pi, dim=-1, keepdim=True).values
        policy_loss = -(q_pi_min - self.alpha * log_pi).mean()

        self.policy_optim.zero_grad()
        policy_loss.backward()
        self.policy_optim.step()

        if self.automatic_alpha_tuning:
            alpha_loss = self.log_alpha.exp() * (-(log_pi + self.target_entropy).detach()).mean()
            self.alpha_optim.zero_grad()
            alpha_loss.backward()
            self.alpha_optim.step()
            self.alpha = self.log_alpha.exp().item()
        else:
            alpha_loss = torch.tensor(0.).to(self.device)
        if updates % self.target_update_interval == 0:
            soft_update(self.target_critic, self.critic, self.update_coeff)

        return {
            'critic_loss': critic_loss.item(),
            'max_critic_abs_err': abs_critic_error.max().item(),
            'policy_loss': policy_loss.item(),
            'alpha': self.alpha,
            'alpha_loss': alpha_loss.item(),
            'q': torch.min(q, dim=-1).values.mean().item(),
            'q_target': torch.mean(q_target).item(),
            'log_pi': torch.mean(log_pi).item(),
            'num_expert': num_expert_replay_is_not_null,
        }


import math
from typing import Tuple

import numpy as np

from ..data import train_test_split

class DataLoader:
    """
    Loads individual items of a dataset.
    You must implement the `__len__` and `__getitem__` functions for the loader to work correctly.

    Usage:
    ```python
    class MyDataLoader(DataLoader):
        def __init__(self, paths):
            self.paths = paths

        def __len__(self):
            return len(self.paths)

        def __getitem__(self, index: int):
            return load_item(self.paths[index])
    ```
    """

    def __len__(self):
        raise NotImplementedError

    def __getitem__(self, index: int):
        raise NotImplementedError

    def __bool__(self):
        return len(self) > 0

    def reset(self):
        pass

    def __iter__(self):
        """Create a generator that iterate over the loader."""
        for item in (self[i] for i in range(len(self))):
            yield item

    def split(self, test_ratio: float) -> Tuple["DataLoader", "DataLoader"]:
        """
        Splits the loader into two loaders, one with train data, and another one with test data.
        """
        raise NotImplementedError("This loader cannot be split")

class ListDataLoader(DataLoader):
    """
    Loader that simply returns all elements of the passed list.

    Usage:
    ```python
    items = [Image("a.jpg"), Image("b.jpg"), Image("c.jpg")]
    loader = ListDataloader(items)
    ```
    """

    def __init__(self, items):
        self.items = items

    def __len__(self):
        return len(self.items)

    def __getitem__(self, index: int):
        return self.items[index]

    def split(self, test_ratio: float) -> Tuple["ListDataLoader", "ListDataLoader"]:
        train, test = train_test_split(list(self.items), test_ratio)
        return (ListDataLoader(train), ListDataLoader(test))

class LoaderWrapper(DataLoader):
    """
    Wraps a loader and delegates everything to it.
    Inherit from this class if you need to wrap another data loader.
    """

    def __init__(self, loader: DataLoader):
        self.loader = loader

    def __len__(self):
        return len(self.loader)

    def __getitem__(self, index: int):
        return self.loader[index]

class EagerLoader(ListDataLoader):
    """
    Preloads all samples from the given loader and keeps them in memory.

    Usage:
    ```python
    class MyDataLoader(DataLoader):
        ...

        def __getitem__(self, index: int):
            return expensive_load(index)

    loader = MyDataLoader(...)
    loader = EagerLoader(loader)  # loads all items into memory
    ```
    """

    def __init__(self, loader: DataLoader):
        super().__init__(list(loader))

class MappingLoader(LoaderWrapper):
    """
    Loader that wraps another loader and applies a function to each item.

    Usage:
    ```python
    loader = BGRImageDataLoader(...)
    loader = MappingLoader(loader, lambda image: bgr_to_rgb(image))
    ```
    """

    def __init__(self, loader: DataLoader, map_fn):
        super().__init__(loader)
        self.map_fn = map_fn

    def __getitem__(self, index: int):
        return self.map_fn(self.loader[index])

class ZipLoader(DataLoader):
    def __init__(self, *loaders):
        assert len(loaders) > 0
        self.loaders = loaders

    def __getitem__(self, item):
        return tuple(loader[item] for loader in self.loaders)

    def __len__(self):
        return min(len(loader) for loader in self.loaders)

    def reset(self):
        for loader in self.loaders:
            loader.reset()

class BatchIndexer:
    def __init__(self, length: int, batch_size: int, seed=None, shuffle=True):
        assert batch_size > 0
        assert length > 0

        if not shuffle:
            assert seed is None

        self.batch_size = batch_size
        self.batch_count = int(math.ceil(length / self.batch_size))
        self.indices = np.arange(0, length)
        self.shuffle = shuffle

        if shuffle:
            self.random = np.random.RandomState(seed)
            self.random.shuffle(self.indices)

    def get_indices(self, start, end):
        return self.indices[start:end]

    def reset(self):
        if self.shuffle:
            self.random.shuffle(self.indices)

    def __len__(self):
        return self.batch_count

class BatchLoader(DataLoader):
    """
    Loader that takes another loader and aggregates its items into batches.

    Usage:
    ```python
    loader = ListDataLoader([1, 2, 3, 4, 5])
    loader = BatchLoader(loader, batch_size=2)

    batches = list(loader)  # [[1, 3], [4, 5], [2]]
    ```
    """

    def __init__(self, loader: DataLoader, batch_size: int, seed=None, shuffle=True):
        self.indexer = BatchIndexer(len(loader), batch_size, seed, shuffle)
        self.loader = loader

    @property
    def batch_size(self):
        return self.indexer.batch_size

    def __len__(self):
        return len(self.indexer)

    def reset(self):
        self.indexer.reset()

    def __getitem__(self, index: int):
        batch = []

        start = index * self.batch_size
        end = start + self.batch_size
        for index in self.indexer.get_indices(start, end):
            item = self.loader[index]
            batch.append(item)

        return batch


#! /usr/bin/env python3

"""
Name:   app.py
Author: Joshua Tice
Date:   June 30, 2019

Description
-----------
This app allows a user to predict which breed of dog a human or dog
looks like. The workflow starts with the user uploading the image, then
the user clicks a link to initiate the app's algorithm. Finally, the
output is displayed alongside the image.

Acknowledgements
----------------
The main portions of the flask backend were adopted from Dustin
D'Avignon's Medium post entitled "Upload multiple images with Python,
Flask, and Flask Dropzone." Many thanks to Dustin for the head start.

https://medium.com/@dustindavignon/upload-multiple-images-with-python-
flask-and-flask-dropzone-d5b821829b1d
"""

import os
from pathlib import Path
import pickle

import cv2
from flask import Flask, redirect, render_template, request, session, url_for
from flask_dropzone import Dropzone
from flask_uploads import UploadSet, configure_uploads, IMAGES, patch_request_class
from keras.applications import resnet50, xception
from keras.backend import clear_session
from keras.layers import Conv2D, Dense, MaxPooling2D, GlobalAveragePooling2D
from keras.models import Sequential
from keras.preprocessing import image
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

app = Flask(__name__)

# App configuration
app.config["SECRET_KEY"] = "supersecretkeygoeshere"
os.environ["KMP_DUPLICATE_LIB_OK"] = "True"
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

# Dropzone configuration
dropzone = Dropzone(app)
app.config["DROPZONE_UPLOAD_MULTIPLE"] = True
app.config["DROPZONE_ALLOWED_FILE_CUSTOM"] = True
app.config["DROPZONE_ALLOWED_FILE_TYPE"] = "image/*"
app.config["DROPZONE_REDIRECT_VIEW"] = "uploaded_images"

# Uploads configuration
app.config["UPLOADED_PHOTOS_DEST"] = Path.cwd() / "uploads"
photos = UploadSet("photos", IMAGES)
configure_uploads(app, photos)
patch_request_class(app)

def load_resnet50_model():
    """Instantiate the keras Resnet50 convolutional neural network

    Returns
    -------
    keras.applications.resnet50.Resnet50
        Resnet50 CNN with weights trained on imagenet
    """

    clear_session()
    resnet50_model = resnet50.ResNet50(weights="imagenet")
    global resnet50_graph
    resnet50_graph = tf.get_default_graph()
    return resnet50_model

def load_xception_model():
    """Instantiate the keras Xception convolutional neural network

    Returns
    -------
    keras.models.Sequential
        The final fully connected layers of an Xception-based CNN where
        the weights of the fully connected layers are trained to
        differentiate between different dog breeds
    """

    clear_session()
    xception_model = Sequential()
    xception_model.add(GlobalAveragePooling2D(input_shape=(7, 7, 2048)))
    xception_model.add(Dense(133, activation="softmax"))
    xception_model.compile(
        loss="categorical_crossentropy", optimizer="rmsprop", metrics=["accuracy"]
    )
    xception_model.load_weights("utility_files/weights.best.Xception.hdf5")
    global xception_graph
    xception_graph = tf.get_default_graph()
    return xception_model

def face_detector(img_path):
    """Detect human faces in a given image

    Parameters
    ----------
    img_path : str
        Path to the image

    Returns
    -------
    bool
        True if human face(s) detected in image
    """

    face_cascade = cv2.CascadeClassifier(
        "utility_files/haarcascade_frontalface_alt.xml"
    )
    img = cv2.imread(img_path)
    grayscale_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(grayscale_img)
    return len(faces) > 0

def path_to_tensor(img_path):
    """Converts image to tensor for input to deep learning model

    Parameters
    ----------
    img_path : str
        Path to image file to be converted to tensor

    Returns
    np.ndarray
        Tensor to serve as input for deep learning model
    """

    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    return np.expand_dims(x, axis=0)

def dog_detector(img_path):
    """Returns "True" if a dog is detected in the image at img_path

    Parameters
    ----------
    img_path : str
        Path to the image of interest

    Returns
    -------
    bool
        "True if a dog is detected in the image
    """

    resnet50_model = load_resnet50_model()
    img = resnet50.preprocess_input(path_to_tensor(img_path))
    with resnet50_graph.as_default():
        prediction = np.argmax(resnet50_model.predict(img))

    return (prediction <= 268) & (prediction >= 151)

def predict_breed(img_path):
    """Predict the breed of a dog in a given image using a trained CNN

    Parameters
    ----------
    img_path : str
        The path to the image to be classified

    Returns
    -------
    str
        The predicted breed of the dog in the image
    """

    with open("utility_files/dog_names.pickle", "rb") as f:
        dog_names = pickle.load(f)

    xception_model = load_xception_model()
    img = xception.preprocess_input(path_to_tensor(img_path))
    bottleneck_feature = xception.Xception(
        weights="imagenet", include_top=False
    ).predict(img)
    with xception_graph.as_default():
        predicted_vector = xception_model.predict(bottleneck_feature)
    prediction = dog_names[np.argmax(predicted_vector)]

    return prediction

def main_algorithm(img_path):
    """Detect whether an image contains a dog or human, then guess breed

    Parameters
    ----------
    img_path : str
        Path to an image for classification
    """

    if face_detector(img_path):
        breed = predict_breed(img_path)
        return (
            "Looks like a human! If this were a dog, though, I would"
            " guess a {}".format(breed)
        )
    elif dog_detector(img_path):
        breed = predict_breed(img_path)
        return "Looks like a dog! Perhaps a {}".format(breed)
    else:
        return (
            "Well...I have now idea what this is. Apologies. "
            "Please try another image."
        )

@app.route("/", methods=["GET", "POST"])
def index():

    [file.unlink() for file in Path("./uploads").iterdir()]
    if "file_urls" in session:
        session["file_urls"] = []

    # list to hold uploaded image urls
    file_urls = []
    predictions = []

    # handle image upload from Dropszone
    if request.method == "POST":

        if "file_urls" not in session:
            session["file_urls"] = []
        file_urls = session["file_urls"]
        for f in request.files:
            file = request.files.get(f)
            filename = photos.save(file, name=file.filename)
            file_urls.append(photos.url(filename))
        session["file_urls"] = file_urls

        return "uploading..."

    return render_template("index.html")

@app.route("/results")
def uploaded_images():

    # redirect to home if no images to display
    if "file_urls" not in session or session["file_urls"] == []:
        return redirect(url_for("index"))
    file_urls = session["file_urls"]

    return render_template("uploaded_images.html", file_urls=file_urls)

@app.route("/predictions")
def predictions():

    file_urls = session["file_urls"]
    session.pop("file_urls", None)
    img_paths = [str(file) for file in Path.cwd().joinpath("uploads").iterdir()]
    predictions = []
    for img_path in img_paths:
        prediction = main_algorithm(img_path)
        predictions.append(prediction)
    return render_template(
        "predictions.html", file_urls=file_urls, predictions=predictions
    )


import numpy as np
import copy
import tempfile
import os
import abc
from typing import Dict, Sequence, Union
from collections import OrderedDict

from .base import ObjectiveBase
from .constants import MODE_FUN, MODE_RES, FVAL, RDATAS
from .amici_calculator import AmiciCalculator
from .amici_util import (
    map_par_opt_to_par_sim, create_identity_parameter_mapping)

try:
    import amici
    import amici.petab_objective
    import amici.parameter_mapping
    from amici.parameter_mapping import ParameterMapping
except ImportError:
    pass

AmiciModel = Union['amici.Model', 'amici.ModelPtr']
AmiciSolver = Union['amici.Solver', 'amici.SolverPtr']

class AmiciObjectBuilder(abc.ABC):
    """Allows to build AMICI model, solver, and edatas.

    This class is useful for pickling an :class:`pypesto.AmiciObjective`,
    which is required in some parallelization schemes. Therefore, this
    class itself must be picklable.
    """

    @abc.abstractmethod
    def create_model(self) -> AmiciModel:
        """Create an AMICI model."""

    @abc.abstractmethod
    def create_solver(self, model: AmiciModel) -> AmiciSolver:
        """Create an AMICI solver."""

    @abc.abstractmethod
    def create_edatas(self, model: AmiciModel) -> Sequence['amici.ExpData']:
        """Create AMICI experimental data."""

class AmiciObjective(ObjectiveBase):
    """
    This class allows to create an objective directly from an amici model.
    """

    def __init__(self,
                 amici_model: AmiciModel,
                 amici_solver: AmiciSolver,
                 edatas: Union[Sequence['amici.ExpData'], 'amici.ExpData'],
                 max_sensi_order: int = None,
                 x_ids: Sequence[str] = None,
                 x_names: Sequence[str] = None,
                 parameter_mapping: 'ParameterMapping' = None,
                 guess_steadystate: bool = True,
                 n_threads: int = 1,
                 fim_for_hess: bool = True,
                 amici_object_builder: AmiciObjectBuilder = None,
                 calculator: AmiciCalculator = None):
        """
        Constructor.

        Parameters
        ----------
        amici_model:
            The amici model.
        amici_solver:
            The solver to use for the numeric integration of the model.
        edatas:
            The experimental data. If a list is passed, its entries correspond
            to multiple experimental conditions.
        max_sensi_order:
            Maximum sensitivity order supported by the model. Defaults to 2 if
            the model was compiled with o2mode, otherwise 1.
        x_ids:
            Ids of optimization parameters. In the simplest case, this will be
            the AMICI model parameters (default).
        x_names:
            Names of optimization parameters.
        parameter_mapping:
            Mapping of optimization parameters to model parameters. Format
            as created by `amici.petab_objective.create_parameter_mapping`.
            The default is just to assume that optimization and simulation
            parameters coincide.
        guess_steadystate:
            Whether to guess steadystates based on previous steadystates and
            respective derivatives. This option may lead to unexpected
            results for models with conservation laws and should accordingly
            be deactivated for those models.
        n_threads:
            Number of threads that are used for parallelization over
            experimental conditions. If amici was not installed with openMP
            support this option will have no effect.
        fim_for_hess:
            Whether to use the FIM whenever the Hessian is requested. This only
            applies with forward sensitivities.
            With adjoint sensitivities, the true Hessian will be used,
            if available.
            FIM or Hessian will only be exposed if `max_sensi_order>1`.
        amici_object_builder:
            AMICI object builder. Allows recreating the objective for
            pickling, required in some parallelization schemes.
        calculator:
            Performs the actual calculation of the function values and
            derivatives.
        """
        if amici is None:
            raise ImportError(
                "This objective requires an installation of amici "
                "(https://github.com/icb-dcm/amici). "
                "Install via `pip3 install amici`.")

        self.amici_model = amici.ModelPtr(amici_model.clone())
        self.amici_solver = amici.SolverPtr(amici_solver.clone())

        # make sure the edatas are a list of edata objects
        if isinstance(edatas, amici.amici.ExpData):
            edatas = [edatas]

        # set the experimental data container
        self.edatas = edatas

        # set the maximum sensitivity order
        self.max_sensi_order = max_sensi_order

        self.guess_steadystate = guess_steadystate

        # optimization parameter ids
        if x_ids is None:
            # use model parameter ids as ids
            x_ids = list(self.amici_model.getParameterIds())
        self.x_ids = x_ids

        # mapping of parameters
        if parameter_mapping is None:
            # use identity mapping for each condition
            parameter_mapping = create_identity_parameter_mapping(
                amici_model, len(edatas))
        self.parameter_mapping = parameter_mapping

        # preallocate guesses, construct a dict for every edata for which we
        # need to do preequilibration
        if self.guess_steadystate:
            if self.amici_model.ncl() > 0:
                raise ValueError('Steadystate prediction is not supported for '
                                 'models with conservation laws!')

            if self.amici_model.getSteadyStateSensitivityMode() == \
                    amici.SteadyStateSensitivityMode_simulationFSA:
                raise ValueError('Steadystate guesses cannot be enabled when'
                                 ' `simulationFSA` as '
                                 'SteadyStateSensitivityMode!')
            self.steadystate_guesses = {
                'fval': np.inf,
                'data': {
                    iexp: {}
                    for iexp, edata in enumerate(self.edatas)
                    if len(edata.fixedParametersPreequilibration) or
                    self.amici_solver.getPreequilibration()
                }
            }
        # optimization parameter names
        if x_names is None:
            # use ids as names
            x_names = x_ids

        self.n_threads = n_threads
        self.fim_for_hess = fim_for_hess
        self.amici_object_builder = amici_object_builder

        if calculator is None:
            calculator = AmiciCalculator()
        self.calculator = calculator
        super().__init__(x_names=x_names)

    def initialize(self):
        super().initialize()
        self.reset_steadystate_guesses()
        self.calculator.initialize()

    def __deepcopy__(self, memodict: Dict = None) -> 'AmiciObjective':
        other = self.__class__.__new__(self.__class__)

        for key in set(self.__dict__.keys()) - \
                {'amici_model', 'amici_solver', 'edatas'}:
            other.__dict__[key] = copy.deepcopy(self.__dict__[key])

        # copy objects that do not have __deepcopy__
        other.amici_model = amici.ModelPtr(self.amici_model.clone())
        other.amici_solver = amici.SolverPtr(self.amici_solver.clone())
        other.edatas = [amici.ExpData(data) for data in self.edatas]

        return other

    def __getstate__(self) -> Dict:
        if self.amici_object_builder is None:
            raise NotImplementedError(
                "AmiciObjective does not support __getstate__ without "
                "an `amici_object_builder`.")

        state = {}
        for key in set(self.__dict__.keys()) - \
                {'amici_model', 'amici_solver', 'edatas'}:
            state[key] = self.__dict__[key]

        # write amici solver settings to file
        amici_solver_file = tempfile.mkstemp()[1]
        try:
            amici.writeSolverSettingsToHDF5(
                self.amici_solver, amici_solver_file)
        except AttributeError as e:
            e.args += ("Pickling the AmiciObjective requires an AMICI "
                       "installation with HDF5 support.",)
            raise

        # read in byte stream
        amici_solver_settings = open(amici_solver_file, 'rb').read()
        state['amici_solver_settings'] = amici_solver_settings
        # remove temporary file
        os.remove(amici_solver_file)

        return state

    def __setstate__(self, state: Dict):
        if state['amici_object_builder'] is None:
            raise NotImplementedError(
                "AmiciObjective does not support __setstate__ without "
                "an `amici_object_builder`.")
        self.__dict__.update(state)

        # note: attributes not defined in the builder are lost
        model = self.amici_object_builder.create_model()
        solver = self.amici_object_builder.create_solver(model)
        edatas = self.amici_object_builder.create_edatas(model)

        try:
            # write solver settings to temporary file
            _file = tempfile.mkstemp()[1]
            with open(_file, 'wb') as f:
                f.write(state['amici_solver_settings'])
            # read in solver settings
            amici.readSolverSettingsFromHDF5(_file, solver)
            # remove temporary file
            os.remove(_file)
        except AttributeError as err:
            if not err.args:
                err.args = ('',)
            err.args += ("Amici must have been compiled with hdf5 support",)
            raise
        self.amici_model = model
        self.amici_solver = solver
        self.edatas = edatas

    def check_sensi_orders(self, sensi_orders, mode) -> bool:
        sensi_order = max(sensi_orders)

        # dynamically obtain maximum allowed sensitivity order
        max_sensi_order = self.max_sensi_order
        if max_sensi_order is None:
            max_sensi_order = 1
            # check whether it is ok to request 2nd order
            sensi_mthd = self.amici_solver.getSensitivityMethod()
            mthd_fwd = amici.SensitivityMethod_forward
            if mode == MODE_FUN and (
                    self.amici_model.o2mode or (
                    sensi_mthd == mthd_fwd and self.fim_for_hess)):
                max_sensi_order = 2

        # evaluate sensitivity order
        return sensi_order <= max_sensi_order

    def check_mode(self, mode):
        return mode in [MODE_FUN, MODE_RES]

    def call_unprocessed(self, x, sensi_orders, mode):
        sensi_order = max(sensi_orders)

        x_dct = self.par_arr_to_dct(x)

        # update steady state
        if self.guess_steadystate and \
                self.steadystate_guesses['fval'] < np.inf:
            for data_ix in range(len(self.edatas)):
                self.apply_steadystate_guess(data_ix, x_dct)

        ret = self.calculator(
            x_dct=x_dct, sensi_order=sensi_order, mode=mode,
            amici_model=self.amici_model, amici_solver=self.amici_solver,
            edatas=self.edatas, n_threads=self.n_threads,
            x_ids=self.x_ids, parameter_mapping=self.parameter_mapping,
            fim_for_hess=self.fim_for_hess,
        )

        nllh = ret[FVAL]
        rdatas = ret[RDATAS]

        # check whether we should update data for preequilibration guesses
        if self.guess_steadystate and \
                nllh <= self.steadystate_guesses['fval'] and \
                nllh < np.inf:
            self.steadystate_guesses['fval'] = nllh
            for data_ix, rdata in enumerate(rdatas):
                self.store_steadystate_guess(data_ix, x_dct, rdata)

        return ret

    def par_arr_to_dct(self, x: Sequence[float]) -> Dict[str, float]:
        """Create dict from parameter vector."""
        return OrderedDict(zip(self.x_ids, x))

    def apply_steadystate_guess(self, condition_ix: int, x_dct: Dict):
        """
        Use the stored steadystate as well as the respective  sensitivity (
        if available) and parameter value to approximate the steadystate at
        the current parameters using a zeroth or first order taylor
        approximation:
        x_ss(x') = x_ss(x) [+ dx_ss/dx(x)*(x'-x)]
        """
        mapping = self.parameter_mapping[condition_ix].map_sim_var
        x_sim = map_par_opt_to_par_sim(mapping, x_dct, self.amici_model)
        x_ss_guess = []  # resets initial state by default
        if condition_ix in self.steadystate_guesses['data']:
            guess_data = self.steadystate_guesses['data'][condition_ix]
            if guess_data['x_ss'] is not None:
                x_ss_guess = guess_data['x_ss']
            if guess_data['sx_ss'] is not None:
                linear_update = guess_data['sx_ss'].transpose().dot(
                    (x_sim - guess_data['x'])
                )
                # limit linear updates to max 20 % elementwise change
                if (x_ss_guess/linear_update).max() < 0.2:
                    x_ss_guess += linear_update

        self.edatas[condition_ix].x0 = tuple(x_ss_guess)

    def store_steadystate_guess(
            self, condition_ix: int, x_dct: Dict, rdata: 'amici.ReturnData'):
        """
        Store condition parameter, steadystate and steadystate sensitivity in
        steadystate_guesses if steadystate guesses are enabled for this
        condition
        """
        if condition_ix not in self.steadystate_guesses['data']:
            return
        preeq_guesses = self.steadystate_guesses['data'][condition_ix]

        # update parameter
        condition_map_sim_var = \
            self.parameter_mapping[condition_ix].map_sim_var
        x_sim = map_par_opt_to_par_sim(
            condition_map_sim_var, x_dct, self.amici_model)
        preeq_guesses['x'] = x_sim

        # update steadystates
        preeq_guesses['x_ss'] = rdata['x_ss']
        preeq_guesses['sx_ss'] = rdata['sx_ss']

    def reset_steadystate_guesses(self):
        """Resets all steadystate guess data"""
        if not self.guess_steadystate:
            return

        self.steadystate_guesses['fval'] = np.inf
        for condition in self.steadystate_guesses['data']:
            self.steadystate_guesses['data'][condition] = {}


import os
import logging
import argparse
import warnings
import traceback
from timeit import time

import cv2
import mxnet as mx
import numpy as np
import gluoncv as gcv
from gluoncv import model_zoo

from deep_sort import preprocessing
from deep_sort import nn_matching
from deep_sort.detection import Detection
from deep_sort.tracker import Tracker
from tools import generate_detections as gdet

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('deepsort')
warnings.filterwarnings('ignore')

BASE_FPS = 30

def parse_args():
    parser = argparse.ArgumentParser(description='DeepSORT using mxnet YOLO3.')
    parser.add_argument('--network', type=str, default='yolo3_darknet53_voc',
                        help='Base network name which serves as feature extraction base.')
    parser.add_argument('--short', type=int, default=416,
                        help='Input data shape for short-side, use 320, 416, 608...')
    parser.add_argument('--gpu', type=int, default=0,
                        help='Is GPU accelration available')
    parser.add_argument('--fps', type=int, default=0,
                        help=f'Frame per sec to process based {BASE_FPS} fps,' +
                        ' 0 to consume with no skipped frame')
    parser.add_argument('--src', type=str, default='video.mp4',
                        help='Video filename to tracking person')
    parser.add_argument('--out-dir', type=str, default='.',
                        help='Location for missed frame image will be stored.')

    args = parser.parse_args()
    return args

def main(args):
    logger.info('Start Tracking...')

    ctx = mx.gpu(0) if args.gpu else mx.cpu()
    fps = max(0, min(BASE_FPS, args.fps))
    net = model_zoo.get_model(args.network, pretrained=True, ctx=ctx)
    net.reset_class(classes=['person'], reuse_weights=['person'])

    # Definition of the parameters
    max_cosine_distance = 0.3
    nn_budget = None
    nms_max_overlap = 1.0

    # feature extractor for deepsort re-id
    encoder = gdet.BoxEncoder()

    metric = nn_matching.NearestNeighborDistanceMetric(
        'cosine', max_cosine_distance, nn_budget
    )
    tracker = Tracker(metric)

    capture = cv2.VideoCapture(args.src)
    frame_interval = BASE_FPS // fps if fps > 0 else 0
    frame_index = 0
    while True:
        ret, frame = capture.read()
        if ret != True:
            break

        if 0 < fps and frame_index % frame_interval != 0:
            frame_index += 1
            continue

        x, img = gcv.data.transforms.presets.yolo.transform_test(
            mx.nd.array(frame).astype('uint8'),
            short=args.short,
        )

        class_IDs, det_scores, det_boxes = net(x.as_in_context(ctx))

        boxs = []
        person = mx.nd.array([0])
        score_threshold = mx.nd.array([0.5])
        for i, class_ID in enumerate(class_IDs[0]):
            if class_ID == person and det_scores[0][i] >= score_threshold:
                boxs.append(det_boxes[0][i].asnumpy())

        if boxs:
            features = encoder(img, boxs)
        else:
            features = np.array([])

        # score to 1.0 here).
        detections = [Detection(bbox, 1.0, feature)
                      for bbox, feature in zip(boxs, features)]

        # Run non-maxima suppression.
        boxes = np.array([d.tlwh for d in detections])
        scores = np.array([d.confidence for d in detections])
        indices = preprocessing.non_max_suppression(
            boxes, nms_max_overlap, scores
        )
        detections = [detections[i] for i in indices]

        # Call the tracker
        tracker.predict()
        tracker.update(detections)

        frame_index += 1

        # store original scene
        cv2.imwrite(os.path.join(args.out_dir, f'{frame_index}.jpg'), img)

        show_img = img.copy()
        # check missed
        for track in tracker.tracks:
            bbox = [max(0, int(x)) for x in track.to_tlbr()]
            if not track.is_confirmed() or track.time_since_update > 1:
                if 2 <= track.time_since_update < 10:
                    try:
                        cv2.imwrite(
                            os.path.join(
                                args.out_dir,
                                f'missed-{frame_index}-{track.track_id}.jpg'
                            ),
                            img,
                        )
                    except:
                        traceback.print_exc()
                logger.info('Skipped by time_since_update')
                continue

            logger.info(f'Frame #{frame_index} - Id: {track.track_id}')
            cv2.rectangle(show_img,
                          (bbox[0], bbox[1]), (bbox[2], bbox[3]),
                          (255,255,255), 2)
            cv2.putText(show_img,
                        str(track.track_id),
                        (bbox[0], bbox[1]+30),
                        0, 5e-3 * 200,
                        (0,255,0), 2)

        # show image
        cv2.imwrite(os.path.join(args.out_dir, f'anno-{frame_index}.jpg'), show_img)
        cv2.imshow('', show_img)

        # Press Q to stop!
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

    logger.info(f'Missed obj: {tracker.missed_obj}, Missed frame: {tracker.missed_frame}')

    capture.release()
    cv2.destroyAllWindows()

if __name__ == '__main__':
    args = parse_args()
    if not os.path.exists(args.out_dir):
        os.mkdir(args.out_dir)
    main(args)


from scipy.ndimage import gaussian_filter
import cv2

class Deidentifier:
    def __init__(self):
        pass

    def deidentify(self, image_file, rectangles=None):
        img = cv2.imread(image_file)
        image_blurred = img
        if rectangles:
            for (x, y, w, h) in rectangles:
                image_blurred[y:y+h, x:x+w] = gaussian_filter(image_blurred[y:y+h, x:x+w], 5)
        else:
            image_blurred = gaussian_filter(img, 5)

        return image_blurred

if __name__ == '__main__':
    di = Deidentifier()
    img = di.deidentify('Images/people-02.jpg')
    cv2.imshow('output', img)
    cv2.waitKey(0)
    cv2.destroyAllWindows()


import numpy as np
from quartical.calibration.solver import solver_wrapper
from quartical.utils.dask import Blocker
from collections import namedtuple

term_spec_tup = namedtuple("term_spec_tup", "name type shape pshape")

def construct_solver(data_xds_list,
                     gain_xds_lod,
                     t_bin_list,
                     t_map_list,
                     f_map_list,
                     d_map_list,
                     solver_opts,
                     chain_opts):
    """Constructs the dask graph for the solver layer.

    This constructs a custom dask graph for the solver layer given the slew
    of solver inputs. This is arguably the most important function in V2 and
    should not be tampered with without a certain level of expertise with dask.

    Args:
        data_xds_list: A list of xarray.Dataset objects containing MS data.
        gain_xds_lod: A list of dicts containing xarray.Dataset objects
            describing the gain terms.
        t_map_list: List of dask.Array objects containing time mappings.
        f_map_list: List of dask.Array objects containing frequency mappings.
        d_map_list: List of dask.Array objects containing direction mappings.
        solver_opts: A Solver config object.
        chain_opts: A Chain config object.

    Returns:
        solved_gain_xds_lod: A list of dicts containing xarray.Datasets
            describing the solved gains.
    """

    solved_gain_xds_lod = []
    output_data_xds_list = []

    for xds_ind, data_xds in enumerate(data_xds_list):

        model_col = data_xds.MODEL_DATA.data
        data_col = data_xds.DATA.data
        ant1_col = data_xds.ANTENNA1.data
        ant2_col = data_xds.ANTENNA2.data
        weight_col = data_xds.WEIGHT.data
        flag_col = data_xds.FLAG.data
        chan_freqs = data_xds.CHAN_FREQ.data
        t_bin_arr = t_bin_list[xds_ind]
        t_map_arr = t_map_list[xds_ind]
        f_map_arr = f_map_list[xds_ind]
        d_map_arr = d_map_list[xds_ind]
        gain_terms = gain_xds_lod[xds_ind]
        corr_mode = data_xds.dims["corr"]

        # Grab the number of input chunks - doing this on the data should be
        # safe.
        n_t_chunks, n_f_chunks, _ = data_col.numblocks

        # Take the compact chunking info on the gain xdss and expand it.
        spec_list = expand_specs(gain_terms)

        # Create a blocker object.
        blocker = Blocker(solver_wrapper, "rf")

        # Add relevant inputs to the blocker object. TODO: Only pass in values
        # required by the specific terms in use.
        blocker.add_input("model", model_col, "rfdc")
        blocker.add_input("data", data_col, "rfc")
        blocker.add_input("a1", ant1_col, "r")
        blocker.add_input("a2", ant2_col, "r")
        blocker.add_input("weights", weight_col, "rfc")
        blocker.add_input("flags", flag_col, "rf")
        blocker.add_input("t_bin_arr", t_bin_arr, "prj")  # Not always needed.
        blocker.add_input("t_map_arr", t_map_arr, "prj")
        blocker.add_input("f_map_arr", f_map_arr, "pfj")
        blocker.add_input("d_map_arr", d_map_arr)
        blocker.add_input("corr_mode", corr_mode)
        blocker.add_input("term_spec_list", spec_list, "rf")
        blocker.add_input("chan_freqs", chan_freqs, "f")  # Not always needed.
        blocker.add_input("solver_opts", solver_opts)
        blocker.add_input("chain_opts", chain_opts)

        # If the gain dataset already has a gain variable, we want to pass
        # it in to initialize the solver.
        for term_name, term_xds in gain_terms.items():
            if "gains" in term_xds.data_vars:
                blocker.add_input(f"{term_name}_initial_gain",
                                  term_xds.gains.data,
                                  "rfadc")

        if hasattr(data_xds, "ROW_MAP"):  # We are dealing with BDA.
            blocker.add_input("row_map", data_xds.ROW_MAP.data, "r")
            blocker.add_input("row_weights", data_xds.ROW_WEIGHTS.data, "r")
        else:
            blocker.add_input("row_map", None)
            blocker.add_input("row_weights", None)

        # Add relevant outputs to blocker object.
        blocker.add_output("weights",
                           "rfc",
                           weight_col.chunks,
                           weight_col.dtype)

        blocker.add_output("flags",
                           "rf",
                           flag_col.chunks,
                           flag_col.dtype)

        for term_name, term_xds in gain_terms.items():

            blocker.add_output(f"{term_name}-gain",
                               "rfadc",
                               term_xds.GAIN_SPEC,
                               np.complex128)

            blocker.add_output(f"{term_name}-gain_flags",
                               "rfad",
                               term_xds.GAIN_SPEC[:-1],
                               np.int8)

            # If there is a PARAM_SPEC on the gain xds, it is also an output.
            if hasattr(term_xds, "PARAM_SPEC"):
                blocker.add_output(f"{term_name}-param",
                                   "rfadp",
                                   term_xds.PARAM_SPEC,
                                   np.float64)

                blocker.add_output(f"{term_name}-param_flags",
                                   "rfad",
                                   term_xds.PARAM_SPEC[:-1],
                                   np.int8)

            else:  # Only non-parameterised gains return a jhj (for now).
                blocker.add_output(f"{term_name}-jhj",
                                   "rfadc",
                                   term_xds.GAIN_SPEC,
                                   np.complex128)

            chunks = ((1,)*n_t_chunks, (1,)*n_f_chunks)
            blocker.add_output(f"{term_name}-conviter",
                               "rf",
                               chunks,
                               np.int64)
            blocker.add_output(f"{term_name}-convperc",
                               "rf",
                               chunks,
                               np.float64)

        # Apply function to inputs to produce dask array outputs (as dict).
        output_dict = blocker.get_dask_outputs()

        # Assign column results to the relevant data xarray.Dataset object.
        # NOTE: Only update FLAG if we are honouring solver flags.
        flag_field = "FLAG" if solver_opts.propagate_flags else "_FLAG"

        output_data_xds = data_xds.assign(
            {"_WEIGHT": (data_xds.WEIGHT.dims, output_dict["weights"]),
             flag_field: (data_xds.FLAG.dims, output_dict["flags"])}
        )
        output_data_xds_list.append(output_data_xds)

        # Assign results to the relevant gain xarray.Dataset object.
        solved_gain_dict = {}

        for term_name, term_xds in gain_terms.items():

            result_vars = {}

            gain = output_dict[f"{term_name}-gain"]
            result_vars["gains"] = (term_xds.GAIN_AXES, gain)

            flags = output_dict[f"{term_name}-gain_flags"]
            result_vars["gain_flags"] = (term_xds.GAIN_AXES[:-1], flags)

            convperc = output_dict[f"{term_name}-convperc"]
            result_vars["conv_perc"] = (("t_chunk", "f_chunk"), convperc)

            conviter = output_dict[f"{term_name}-conviter"]
            result_vars["conv_iter"] = (("t_chunk", "f_chunk"), conviter)

            if hasattr(term_xds, "PARAM_SPEC"):
                params = output_dict[f"{term_name}-param"]
                result_vars["params"] = (term_xds.PARAM_AXES, params)

                param_flags = output_dict[f"{term_name}-param_flags"]
                result_vars["param_flags"] = \
                    (term_xds.PARAM_AXES[:-1], param_flags)
            else:
                jhj = output_dict[f"{term_name}-jhj"]
                result_vars["jhj"] = (term_xds.GAIN_AXES, jhj)

            solved_xds = term_xds.assign(result_vars)

            solved_gain_dict[term_name] = solved_xds

        solved_gain_xds_lod.append(solved_gain_dict)

    return solved_gain_xds_lod, output_data_xds_list

def expand_specs(gain_terms):
    """Convert compact spec to a per-term list per-chunk."""

    # TODO: This was rejiggered to work with the updated Blocker. Could stand
    # to be made a little neater/smarter, but works for now. Assembles nested
    # list where the outer list represnts time chunks, the middle list
    # represents frequency chunks and the inner-most list contains the
    # specs per term. Might be possible to do this with arrays instead.

    n_t_chunks = set(xds.dims["t_chunk"] for xds in gain_terms.values())
    n_f_chunks = set(xds.dims["f_chunk"] for xds in gain_terms.values())

    assert len(n_t_chunks) == 1, "Chunking in time is inconsistent."
    assert len(n_f_chunks) == 1, "Chunking in freq is inconsistent."

    n_t_chunks = n_t_chunks.pop()
    n_f_chunks = n_f_chunks.pop()

    tc_list = []
    for tc_ind in range(n_t_chunks):
        fc_list = []
        for fc_ind in range(n_f_chunks):
            term_list = []
            for xds in gain_terms.values():

                term_name = xds.NAME
                term_type = xds.TYPE
                gain_chunk_spec = xds.GAIN_SPEC

                tc = gain_chunk_spec.tchunk[tc_ind]
                fc = gain_chunk_spec.fchunk[fc_ind]

                ac = gain_chunk_spec.achunk[0]  # No chunking.
                dc = gain_chunk_spec.dchunk[0]  # No chunking.
                cc = gain_chunk_spec.cchunk[0]  # No chunking.

                term_shape = (tc, fc, ac, dc, cc)

                # Check if we have a spec for the parameters.
                parm_chunk_spec = getattr(xds, "PARAM_SPEC", ())
                if parm_chunk_spec:
                    tc_p = parm_chunk_spec.tchunk[tc_ind]
                    fc_p = parm_chunk_spec.fchunk[fc_ind]
                    pc = parm_chunk_spec.pchunk[0]

                    parm_shape = (tc_p, fc_p, ac, dc, pc)
                else:
                    parm_shape = (0,) * 5  # Used for creating a dummy array.

                term_list.append(term_spec_tup(term_name,
                                               term_type,
                                               term_shape,
                                               parm_shape))

            fc_list.append(term_list)
        tc_list.append(fc_list)

    return tc_list


# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Run a trained model on frames from the camera and play random sounds when
class 0 is detected.
"""
from __future__ import print_function

from sys import argv, stderr, exit
from os import getenv

import numpy as np
from tensorflow import keras
import picamera
import time

from camera import Camera
from pinet import PiNet
from randomsound import RandomSound

# Smooth out spikes in predictions but increase apparent latency. Decrease on a Pi Zero.
SMOOTH_FACTOR = 0.8

SHOW_UI = getenv("DISPLAY")

if SHOW_UI:
    import pygame

def main():
    if len(argv) != 3 or argv[1] == '--help':
        print("""Usage: run.py MODEL token
Use MODEL to classify camera frames and play sounds when class 0 is recognised.""")
        exit(1)

    model_file = argv[1]

    print(argv[2])

    # We use the same MobileNet as during recording to turn images into features
    print('Loading feature extractor')
    extractor = PiNet()

    # Here we load our trained classifier that turns features into categories
    print('Loading classifier')
    classifier = keras.models.load_model(model_file)

    # Initialize the camera and sound systems
    camera = Camera(training_mode=False)
    #random_sound = RandomSound()

    # Smooth the predictions to avoid interruptions to audio
    smoothed = np.ones(classifier.output_shape[1:])
    smoothed /= len(smoothed)

    print('Now running!')
    while True:
        while True:
            raw_frame = camera.next_frame()

            # Use MobileNet to get the features for this frame
            z = extractor.features(raw_frame)

            # With these features we can predict a 'normal' / 'yeah' class (0 or 1)
            # Keras expects an array of inputs and produces an array of outputs
            classes = classifier.predict(np.array([z]))[0]

            # smooth the outputs - this adds latency but reduces interruptions
            smoothed = smoothed * SMOOTH_FACTOR + classes * (1.0 - SMOOTH_FACTOR)
            selected = np.argmax(smoothed) # The selected class is the one with highest probability

            # Show the class probabilities and selected class
            summary = 'Class %d [%s]' % (selected, ' '.join('%02.0f%%' % (99 * p) for p in smoothed))
            stderr.write('\r' + summary)

            # Perform actions for selected class. In this case, play a sound from the sounds/ dir
            if selected == 0:
                print("Detected")
                break
        # hand detected, send notifcation
        dateAndTime = time.asctime( time.localtime(time.time()) )
        camera.camera.resolution = (1280,720)
        camera.camera.capture(dateAndTime + ".jpg")
        camera.camera.resolution = (128, 128)
        time.sleep(2)

if __name__ == '__main__':
    main()


import copy
from random import choice
from typing import Dict

import numpy as np
from tqdm import tqdm

from domain.const import SIMULATION_ENV
from domain.data.cluster import Cluster
from domain.data.human import Human
from domain.data.simulation_result import SimulationResult
from lib.log import logger

def lockdown(human_dict: Dict[str, Human], cluster_dict: Dict[str, Cluster]) -> SimulationResult:
    """
    simulation of lockdown scenario
    ロックダウンシナリオのシミュレーション
    """
    logger.info("start evaluation lockdown scenario")
    result = SimulationResult(
        economic_losses_transition=[], serious_cases_transition=[],
        cumulative_economic_losses_transition=[], cumulative_serious_cases_transition=[]
    )
    lockdown_flg = False
    lockdown_days = 0

    for _ in tqdm(range(SIMULATION_ENV.simulation_days)):
        if lockdown_flg is True:
            logger.info("lockdown")
            lockdown_days += 1
            if lockdown_days == 10:
                lockdown_flg = False
        onset_ids = []
        economic_loss = 0
        freeze_rate = 0.0
        freeze_count = 0
        # humanの状態更新
        for key in human_dict.keys():
            if human_dict[key].infected is True:
                human_dict[key].passed_day_after_infected += 1
                if human_dict[key].passed_day_after_infected == human_dict[key].onset_delay_days:
                    human_dict[key].onset = True
                    onset_ids.append(human_dict[key].id)

            # 凍結期間の更新
            if human_dict[key].freeze is True:
                human_dict[key].freeze_remaining -= 1
                if human_dict[key].freeze_remaining == 0:
                    human_dict[key].freeze = False
                freeze_count += 1
                # 凍結時に1日分の経済損失の計算(20 ~ 70歳の生産人口の場合に経済損失を計上する)
                if (human_dict[key].age <= 70) and (human_dict[key].age >= 20):
                    economic_loss += int(SIMULATION_ENV.GDP_per_person / 365)

        if freeze_count != 0:
            freeze_rate = freeze_count / SIMULATION_ENV.population

        # クラスタの凍結日数が0なら凍結解除
        for key in cluster_dict.keys():
            if cluster_dict[key].freeze_remaining == 0:
                cluster_dict[key].freeze = False

        # 凍結可能クラスタかつ、感染を知っている or 発症をしている人がいればクラスタを凍結する
        for key in cluster_dict.keys():
            for human in cluster_dict[key].people:
                if (human_dict[human.id].onset is True) or (human_dict[human.id].know_infection is True):
                    cluster_dict[key].freeze = True
                    cluster_dict[key].freeze_remaining = 10

        # 同日内に再帰的に評価元の配列が変わってしまわないために評価用配列と別オブジェクトにしておく
        evaluation_human_dict = copy.deepcopy(human_dict)
        evaluation_cluster_dict = copy.deepcopy(cluster_dict)

        human_list = []
        for key in human_dict.keys():
            human_list.append(human_dict[key])

        for human_key in human_dict.keys():
            # 感染中かつ発症3日前よりあとなら接触人数に基づいてランダム感染する
            if (human_dict[human_key].infected is True) and (
                    human_dict[human_key].onset_delay_days - human_dict[human_key].passed_day_after_infected >= 3) and (
                    human_dict[human_key].freeze is False):
                # 平均で人に合う人数に凍結レートをかけたものが接触人数となる
                if freeze_rate == 0.0:
                    meet_per_day = human_dict[human_key].random_meet_per_day
                else:
                    meet_per_day = int(human_dict[human_key].random_meet_per_day * freeze_rate)
                for _ in range(meet_per_day):
                    is_freeze_human = True
                    while is_freeze_human is True:
                        selected_human = choice(human_list)
                        if selected_human.freeze is False:
                            is_freeze_human = False
                    # 抗体なしなら確率で感染
                    if selected_human.has_antibodies is False:
                        # マスク着用有無により感染確率の判断
                        if human_dict[human_key].masked:
                            factor_1 = SIMULATION_ENV.utility_mask_to_someone
                        else:
                            factor_1 = 1

                        if selected_human.masked:
                            factor_2 = SIMULATION_ENV.utility_mask_from_someone
                        else:
                            factor_2 = 1

                        # 感染判断
                        probability_of_infect = SIMULATION_ENV.infection_rate * factor_1 * factor_2
                        is_infect = bool(np.random.choice([True, False],
                                                     p=[probability_of_infect, 1 - probability_of_infect]))
                        evaluation_human_dict[human_key].infected = is_infect
                        if is_infect is True:
                            evaluation_human_dict[human_key].passed_day_after_infected = 0
                            evaluation_human_dict[human_key].has_antibodies = True

                for cluster_id in human_dict[human_key].cluster_ids:
                    # クラスターのhumanを呼び出してクラスター内の人間の評価を行う
                    cluster = cluster_dict[cluster_id]
                    # クラスターが凍結状態であれば、そのクラスターの評価は行わない。クラスターの凍結日を1日下げる。
                    if cluster.freeze is True:
                        evaluation_cluster_dict[cluster_id].freeze_remaining -= 1
                        continue

                    cluster_human_list = cluster.people
                    for human in cluster_human_list:
                        if (human.infected is True) and (human.onset_delay_days - human.passed_day_after_infected >= 2):
                            for eval_human in evaluation_cluster_dict[cluster_id].people:
                                if eval_human.has_antibodies is False:
                                    if human.masked:
                                        factor_1 = SIMULATION_ENV.utility_mask_to_someone
                                    else:
                                        factor_1 = 1
                                    if eval_human.masked:
                                        factor_2 = SIMULATION_ENV.utility_mask_from_someone
                                    else:
                                        factor_2 = 1
                                    probability_of_infect = SIMULATION_ENV.infection_rate * factor_1 * factor_2
                                    is_infect = bool(np.random.choice(a=[True, False],
                                                                 p=[probability_of_infect, 1 - probability_of_infect]))
                                    evaluation_human_dict[eval_human.id].infected = is_infect
                                    if is_infect is True:
                                        evaluation_human_dict[eval_human.id].passed_day_after_infected = 0
                                        evaluation_human_dict[eval_human.id].has_antibodies = True

        # 評価後の配列を次の日の評価元配列として保存する
        human_dict = evaluation_human_dict
        cluster_dict = evaluation_cluster_dict

        # 発症している人間を凍結する
        for key in human_dict.keys():
            if human_dict[key].onset is True and human_dict[key].freeze is False:
                human_dict[key].freeze = True
                human_dict[key].freeze_remaining = 10

        # 感染中の人間の数をカウントし、しきい値を超えていたらロックダウン手続きを行う
        counts = 0
        for human_key in human_dict.keys():
            if human_dict[human_key].onset is True:
                counts += 1
        threshold = int(SIMULATION_ENV.population * SIMULATION_ENV.lockdown_threshold)
        if (threshold < counts) and (lockdown_flg is False):
            lockdown_flg = True
            logger.info("lockdown triggered!")
            for human_key in human_dict.keys():
                if human_dict[human_key].does_lock_down is True:
                    human_dict[human_key].freeze = True
                    human_dict[human_key].freeze_remaining = 10

        for key in human_dict.keys():
            if human_dict[key].onset is True:
                human_dict[key].onset_days += 1
                # 10日経過時点で発症をFalse
                if human_dict[key].onset_days == 10:
                    human_dict[key].onset = False

        # 重症評価
        aggravated_count = 0
        for _id in onset_ids:
            human = human_dict[_id]
            if counts > SIMULATION_ENV.population * SIMULATION_ENV.capacity_of_medical_care:
                table = SIMULATION_ENV.aggravation_rate_when_medical_care_capacity_exceeded
            else:
                table = SIMULATION_ENV.aggravation_rate
            if (human.age >= 0) and (human.age < 10):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["0~9"], 1-table["0~9"]]))
            elif (human.age >= 10) and (human.age < 20):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["10~19"], 1 - table["10~19"]]))
            elif (human.age >= 20) and (human.age < 30):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["20~29"], 1 - table["20~29"]]))
            elif (human.age >= 30) and (human.age < 40):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["30~39"], 1 - table["30~39"]]))
            elif (human.age >= 40) and (human.age < 50):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["40~49"], 1 - table["40~49"]]))
            elif (human.age >= 50) and (human.age < 60):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["50~59"], 1 - table["50~59"]]))
            elif (human.age >= 60) and (human.age < 70):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["60~69"], 1 - table["60~69"]]))
            elif (human.age >= 70) and (human.age < 80):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["70~79"], 1 - table["70~79"]]))
            elif (human.age >= 80) and (human.age < 90):
                aggravated = bool(np.random.choice(a=[True, False], p=[table["80~89"], 1 - table["80~89"]]))
            if aggravated is True:
                aggravated_count += 1
        result.serious_cases_transition.append(aggravated_count)
        if len(result.cumulative_serious_cases_transition) == 0:
            result.cumulative_serious_cases_transition.append(aggravated_count)
        else:
            cumulative = result.cumulative_serious_cases_transition[-1] + aggravated_count
            result.cumulative_serious_cases_transition.append(cumulative)

        # 経済損失の計上
        result.economic_losses_transition.append(economic_loss)
        if len(result.cumulative_economic_losses_transition) == 0:
            result.cumulative_economic_losses_transition.append(economic_loss)
        else:
            cumulative = result.cumulative_economic_losses_transition[-1] + economic_loss
            result.cumulative_economic_losses_transition.append(cumulative)

    return result