#!/usr/bin/env python
# coding: utf-8

# # Face Recognition
#
# Welcome! In this assignment, you're going to build a face recognition system. Many of the ideas presented here are from [FaceNet](https://arxiv.org/pdf/1503.03832.pdf). In the lecture, you also encountered [DeepFace](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf).
#
# Face recognition problems commonly fall into one of two categories:
#
# **Face Verification** "Is this the claimed person?" For example, at some airports, you can pass through customs by letting a system scan your passport and then verifying that you (the person carrying the passport) are the correct person. A mobile phone that unlocks using your face is also using face verification. This is a 1:1 matching problem.
#
# **Face Recognition** "Who is this person?" For example, the video lecture showed a [face recognition video](https://www.youtube.com/watch?v=wr4rx0Spihs) of Baidu employees entering the office without needing to otherwise identify themselves. This is a 1:K matching problem.
#
# FaceNet learns a neural network that encodes a face image into a vector of 128 numbers. By comparing two such vectors, you can then determine if two pictures are of the same person.
#
# By the end of this assignment, you'll be able to:
#
# * Differentiate between face recognition and face verification
# * Implement one-shot learning to solve a face recognition problem
# * Apply the triplet loss function to learn a network's parameters in the context of face recognition
# * Explain how to pose face recognition as a binary classification problem
# * Map face images into 128-dimensional encodings using a pretrained model
# * Perform face verification and face recognition with these encodings
#
# **Channels-last notation**
#
# For this assignment, you'll be using a pre-trained model which represents ConvNet activations using a "channels last" convention, as used during the lecture and in previous programming assignments.
#
# In other words, a batch of images will be of shape $(m, n_H, n_W, n_C)$.

# ## Table of Contents
#
# - [1 - Packages](#1)
# - [2 - Naive Face Verification](#2)
# - [3 - Encoding Face Images into a 128-Dimensional Vector](#3)
#     - [3.1 - Using a ConvNet to Compute Encodings](#3-1)
#     - [3.2 - The Triplet Loss](#3-2)
#         - [Exercise 1 - triplet_loss](#ex-1)
# - [4 - Loading the Pre-trained Model](#4)
# - [5 - Applying the Model](#5)
#     - [5.1 - Face Verification](#5-1)
#         - [Exercise 2 - verify](#ex-2)
#     - [5.2 - Face Recognition](#5-2)
#         - [Exercise 3 - who_is_it](#ex-3)
# - [6 - References](#6)

#
# ## 1 - Packages
#
# Go ahead and run the cell below to import the packages you'll need.

# In[1]:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, ZeroPadding2D, Activation, Input, concatenate
from tensorflow.keras.models import Model
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import MaxPooling2D, AveragePooling2D
from tensorflow.keras.layers import Concatenate
from tensorflow.keras.layers import Lambda, Flatten, Dense
from tensorflow.keras.initializers import glorot_uniform
from tensorflow.keras.layers import Layer
from tensorflow.keras import backend as K
K.set_image_data_format('channels_last')
import os
import numpy as np
from numpy import genfromtxt
import pandas as pd
import tensorflow as tf
import PIL

get_ipython().run_line_magic('matplotlib', 'inline')
get_ipython().run_line_magic('load_ext', 'autoreload')
get_ipython().run_line_magic('autoreload', '2')

#
# ## 2 - Naive Face Verification
#
# In Face Verification, you're given two images and you have to determine if they are of the same person. The simplest way to do this is to compare the two images pixel-by-pixel. If the distance between the raw images is below a chosen threshold, it may be the same person!
#
#
#
   Figure 1

#
# Of course, this algorithm performs poorly, since the pixel values change dramatically due to variations in lighting, orientation of the person's face, minor changes in head position, and so on.
#
# You'll see that rather than using the raw image, you can learn an encoding, $f(img)$.
#
# By using an encoding for each image, an element-wise comparison produces a more accurate judgement as to whether two pictures are of the same person.

#
# ## 3 - Encoding Face Images into a 128-Dimensional Vector
#
#
# ### 3.1 - Using a ConvNet to Compute Encodings
#
# The FaceNet model takes a lot of data and a long time to train. So following the common practice in applied deep learning, you'll load weights that someone else has already trained. The network architecture follows the Inception model from [Szegedy *et al*..](https://arxiv.org/abs/1409.4842) An Inception network implementation has been provided for you, and you can find it in the file `inception_blocks_v2.py` to get a closer look at how it is implemented.
#
# *Hot tip:* Go to "File->Open..." at the top of this notebook. This opens the file directory that contains the `.py` file).
#
# The key things to be aware of are:
#
# - This network uses 160x160 dimensional RGB images as its input. Specifically, a face image (or batch of $m$ face images) as a tensor of shape $(m, n_H, n_W, n_C) = (m, 160, 160, 3)$
# - The input images are originally of shape 96x96, thus, you need to scale them to 160x160. This is done in the `img_to_encoding()` function.
# - The output is a matrix of shape $(m, 128)$ that encodes each input face image into a 128-dimensional vector
#
# Run the cell below to create the model for face images!

# In[2]:

from tensorflow.keras.models import model_from_json

json_file = open('keras-facenet-h5/model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
model = model_from_json(loaded_model_json)
model.load_weights('keras-facenet-h5/model.h5')

# Now summarize the input and output shapes:

# In[3]:

print(model.inputs)
print(model.outputs)

# By using a 128-neuron fully connected layer as its last layer, the model ensures that the output is an encoding vector of size 128. You then use the encodings to compare two face images as follows:
#
#
#
   Figure 2:
  By computing the distance between two encodings and thresholding, you can determine if the two pictures represent the same person

#
# So, an encoding is a good one if:
#
# - The encodings of two images of the same person are quite similar to each other.
# - The encodings of two images of different persons are very different.
#
# The triplet loss function formalizes this, and tries to "push" the encodings of two images of the same person (Anchor and Positive) closer together, while "pulling" the encodings of two images of different persons (Anchor, Negative) further apart.
#
#

#
   Figure 3:
   In the next section,  you'll call the pictures from left to right: Anchor (A), Positive (P), Negative (N)

#
# ### 3.2 - The Triplet Loss
#
# **Important Note**: Since you're using a pretrained model, you won't actually need to implement the triplet loss function in this assignment. *However*, the triplet loss is the main ingredient of the face recognition algorithm, and you'll need to know how to use it for training your own FaceNet model, as well as other types of image similarity problems. Therefore, you'll implement it below, for fun and edification. :)
#
# For an image $x$, its encoding is denoted as $f(x)$, where $f$ is the function computed by the neural network.
#
#
#
# Training will use triplets of images $(A, P, N)$:
#
# - A is an "Anchor" image--a picture of a person.
# - P is a "Positive" image--a picture of the same person as the Anchor image.
# - N is a "Negative" image--a picture of a different person than the Anchor image.
#
# These triplets are picked from the training dataset. $(A^{(i)}, P^{(i)}, N^{(i)})$ is used here to denote the $i$-th training example.
#
# You'd like to make sure that an image $A^{(i)}$ of an individual is closer to the Positive $P^{(i)}$ than to the Negative image $N^{(i)}$) by at least a margin $\alpha$:
#
# $$
# || f\left(A^{(i)}\right)-f\left(P^{(i)}\right)||_{2}^{2}+\alpha<|| f\left(A^{(i)}\right)-f\left(N^{(i)}\right)||_{2}^{2}
# $$
#
#
# You would thus like to minimize the following "triplet cost":
#
# $$\mathcal{J} = \sum^{m}_{i=1} \large[ \small \underbrace{\mid \mid f(A^{(i)}) - f(P^{(i)}) \mid \mid_2^2}_\text{(1)} - \underbrace{\mid \mid f(A^{(i)}) - f(N^{(i)}) \mid \mid_2^2}_\text{(2)} + \alpha \large ] \small_+ \tag{3}$$
# Here, the notation "$[z]_+$" is used to denote $max(z,0)$.
#
# **Notes**:
#
# - The term (1) is the squared distance between the anchor "A" and the positive "P" for a given triplet; you want this to be small.
# - The term (2) is the squared distance between the anchor "A" and the negative "N" for a given triplet, you want this to be relatively large. It has a minus sign preceding it because minimizing the negative of the term is the same as maximizing that term.
# - $\alpha$ is called the margin. It's a hyperparameter that you pick manually. You'll use $\alpha = 0.2$.
#
# Most implementations also rescale the encoding vectors to haven L2 norm equal to one (i.e., $\mid \mid f(img)\mid \mid_2$=1); you won't have to worry about that in this assignment.
#
#
# ### Exercise 1 - triplet_loss
#
# Implement the triplet loss as defined by formula (3). These are the 4 steps:
#
# 1. Compute the distance between the encodings of "anchor" and "positive": $\mid \mid f(A^{(i)}) - f(P^{(i)}) \mid \mid_2^2$
# 2. Compute the distance between the encodings of "anchor" and "negative": $\mid \mid f(A^{(i)}) - f(N^{(i)}) \mid \mid_2^2$
# 3. Compute the formula per training example: $ \mid \mid f(A^{(i)}) - f(P^{(i)}) \mid \mid_2^2 - \mid \mid f(A^{(i)}) - f(N^{(i)}) \mid \mid_2^2 + \alpha$
# 4. Compute the full formula by taking the max with zero and summing over the training examples:$$\mathcal{J} = \sum^{m}_{i=1} \large[ \small \mid \mid f(A^{(i)}) - f(P^{(i)}) \mid \mid_2^2 - \mid \mid f(A^{(i)}) - f(N^{(i)}) \mid \mid_2^2+ \alpha \large ] \small_+ \tag{3}$$
#
# *Hints*:
#
# - Useful functions: `tf.reduce_sum()`, `tf.square()`, `tf.subtract()`, `tf.add()`, `tf.maximum()`.
#
# - For steps 1 and 2, sum over the entries of $\mid \mid f(A^{(i)}) - f(P^{(i)}) \mid \mid_2^2$ and $\mid \mid     f(A^{(i)}) - f(N^{(i)}) \mid \mid_2^2$.
#
# - For step 4, you will sum over the training examples.
#
# *Additional Hints*:
#
# - Recall that the square of the L2 norm is the sum of the squared differences: $||x - y||_{2}^{2} = \sum_{i=1}^{N}(x_{i} - y_{i})^{2}$
#
# - Note that the anchor, positive and negative encodings are of shape (*m*,128), where *m* is the number of training examples and 128 is the number of elements used to encode a single example.
#
# - For steps 1 and 2, maintain the number of *m* training examples and sum along the 128 values of each encoding. `tf.reduce_sum` has an axis parameter. This chooses along which axis the sums are applied.
#
# - Note that one way to choose the last axis in a tensor is to use negative indexing (axis=-1).
#
# - In step 4, when summing over training examples, the result will be a single scalar value.
#
# - For `tf.reduce_sum` to sum across all axes, keep the default value axis=None.

# In[18]:

# UNQ_C1(UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: triplet_loss

def triplet_loss(y_true, y_pred, alpha = 0.2):
    """
    Implementation of the triplet loss as defined by formula (3)

    Arguments:
    y_true -- true labels, required when you define a loss in Keras, you don't need it in this function.
    y_pred -- python list containing three objects:
            anchor -- the encodings for the anchor images, of shape (None, 128)
            positive -- the encodings for the positive images, of shape (None, 128)
            negative -- the encodings for the negative images, of shape (None, 128)

    Returns:
    loss -- real number, value of the loss
    """

    anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]

    ### START CODE HERE
    #(≈ 4 lines)
    # Step 1: Compute the (encoding) distance between the anchor and the positive
    pos_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, positive)), axis=-1)
    # Step 2: Compute the (encoding) distance between the anchor and the negative
    neg_dist = tf.reduce_sum(tf.square(tf.subtract(anchor, negative)), axis=-1)
    # Step 3: subtract the two previous distances and add alpha.
    basic_loss = pos_dist - neg_dist + alpha
    # Step 4: Take the maximum of basic_loss and 0.0. Sum over the training examples.
    loss = tf.reduce_sum(tf.maximum(basic_loss, 0))
    ### END CODE HERE

    return loss

# In[19]:

# BEGIN UNIT TEST
tf.random.set_seed(1)
y_true = (None, None, None) # It is not used
y_pred = (tf.keras.backend.random_normal([3, 128], mean=6, stddev=0.1, seed = 1),
          tf.keras.backend.random_normal([3, 128], mean=1, stddev=1, seed = 1),
          tf.keras.backend.random_normal([3, 128], mean=3, stddev=4, seed = 1))
loss = triplet_loss(y_true, y_pred)

assert type(loss) == tf.python.framework.ops.EagerTensor, "Use tensorflow functions"
print("loss = " + str(loss))

y_pred_perfect = ([1., 1.], [1., 1.], [1., 1.,])
loss = triplet_loss(y_true, y_pred_perfect, 5)
assert loss == 5, "Wrong value. Did you add the alpha to basic_loss?"
y_pred_perfect = ([1., 1.],[1., 1.], [0., 0.,])
loss = triplet_loss(y_true, y_pred_perfect, 3)
assert loss == 1., "Wrong value. Check that pos_dist = 0 and neg_dist = 2 in this example"
y_pred_perfect = ([1., 1.],[0., 0.], [1., 1.,])
loss = triplet_loss(y_true, y_pred_perfect, 0)
assert loss == 2., "Wrong value. Check that pos_dist = 2 and neg_dist = 0 in this example"
y_pred_perfect = ([0., 0.],[0., 0.], [0., 0.,])
loss = triplet_loss(y_true, y_pred_perfect, -2)
assert loss == 0, "Wrong value. Are you taking the maximum between basic_loss and 0?"
y_pred_perfect = ([[1., 0.], [1., 0.]],[[1., 0.], [1., 0.]], [[0., 1.], [0., 1.]])
loss = triplet_loss(y_true, y_pred_perfect, 3)
assert loss == 2., "Wrong value. Are you applying tf.reduce_sum to get the loss?"
y_pred_perfect = ([[1., 1.], [2., 0.]], [[0., 3.], [1., 1.]], [[1., 0.], [0., 1.,]])
loss = triplet_loss(y_true, y_pred_perfect, 1)
if (loss == 4.):
    raise Exception('Perhaps you are not using axis=-1 in reduce_sum?')
assert loss == 5, "Wrong value. Check your implementation"
# END UNIT TEST

# **Expected Output**:
#
#

#

#
#             loss
#
#
#            527.2598
#
#

#

#
# ## 4 - Loading the Pre-trained Model
#
# FaceNet is trained by minimizing the triplet loss. But since training requires a lot of data and a lot of computation, you won't train it from scratch here. Instead, you'll load a previously trained model in the following cell; which might take a couple of minutes to run.

# In[20]:

FRmodel = model

# Here are some examples of distances between the encodings between three individuals:
#
#

#
   Figure 4:
   Example of distance outputs between three individuals' encodings

#
# Now use this model to perform face verification and face recognition!

#
# ## 5 - Applying the Model
#
# You're building a system for an office building where the building manager would like to offer facial recognition to allow the employees to enter the building.
#
# You'd like to build a face verification system that gives access to a list of people. To be admitted, each person has to swipe an identification card at the entrance. The face recognition system then verifies that they are who they claim to be.
#
#
# ### 5.1 - Face Verification
#
# Now you'll build a database containing one encoding vector for each person who is allowed to enter the office. To generate the encoding, you'll use `img_to_encoding(image_path, model)`, which runs the forward propagation of the model on the specified image.
#
# Run the following code to build the database (represented as a Python dictionary). This database maps each person's name to a 128-dimensional encoding of their face.

# In[21]:

#tf.keras.backend.set_image_data_format('channels_last')
def img_to_encoding(image_path, model):
    img = tf.keras.preprocessing.image.load_img(image_path, target_size=(160, 160))
    img = np.around(np.array(img) / 255.0, decimals=12)
    x_train = np.expand_dims(img, axis=0)
    embedding = model.predict_on_batch(x_train)
    return embedding / np.linalg.norm(embedding, ord=2)

# In[22]:

database = {}
database["danielle"] = img_to_encoding("images/danielle.png", FRmodel)
database["younes"] = img_to_encoding("images/younes.jpg", FRmodel)
database["tian"] = img_to_encoding("images/tian.jpg", FRmodel)
database["andrew"] = img_to_encoding("images/andrew.jpg", FRmodel)
database["kian"] = img_to_encoding("images/kian.jpg", FRmodel)
database["dan"] = img_to_encoding("images/dan.jpg", FRmodel)
database["sebastiano"] = img_to_encoding("images/sebastiano.jpg", FRmodel)
database["bertrand"] = img_to_encoding("images/bertrand.jpg", FRmodel)
database["kevin"] = img_to_encoding("images/kevin.jpg", FRmodel)
database["felix"] = img_to_encoding("images/felix.jpg", FRmodel)
database["benoit"] = img_to_encoding("images/benoit.jpg", FRmodel)
database["arnaud"] = img_to_encoding("images/arnaud.jpg", FRmodel)

# Load the images of Danielle and Kian:

# In[23]:

danielle = tf.keras.preprocessing.image.load_img("images/danielle.png", target_size=(160, 160))
kian = tf.keras.preprocessing.image.load_img("images/kian.jpg", target_size=(160, 160))

# In[24]:

np.around(np.array(kian) / 255.0, decimals=12).shape

# In[25]:

kian

# In[26]:

np.around(np.array(danielle) / 255.0, decimals=12).shape

# In[27]:

danielle

# Now, when someone shows up at your front door and swipes their ID card (thus giving you their name), you can look up their encoding in the database, and use it to check if the person standing at the front door matches the name on the ID.
#
#
# ### Exercise 2 - verify
#
# Implement the `verify()` function, which checks if the front-door camera picture (`image_path`) is actually the person called "identity". You will have to go through the following steps:
#
# - Compute the encoding of the image from `image_path`.
# - Compute the distance between this encoding and the encoding of the identity image stored in the database.
# - Open the door if the distance is less than 0.7, else do not open it.
#
# As presented above, you should use the L2 distance `np.linalg.norm`.
#
# **Note**: In this implementation, compare the L2 distance, not the square of the L2 distance, to the threshold 0.7.
#
# *Hints*:
#
# - `identity` is a string that is also a key in the database dictionary.
# - `img_to_encoding` has two parameters: the image_path and model.

# In[30]:

# UNQ_C2(UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: verify

def verify(image_path, identity, database, model):
    """
    Function that verifies if the person on the "image_path" image is "identity".

    Arguments:
        image_path -- path to an image
        identity -- string, name of the person you'd like to verify the identity. Has to be an employee who works in the office.
        database -- python dictionary mapping names of allowed people's names (strings) to their encodings (vectors).
        model -- your Inception model instance in Keras

    Returns:
        dist -- distance between the image_path and the image of "identity" in the database.
        door_open -- True, if the door should open. False otherwise.
    """
    ### START CODE HERE
    # Step 1: Compute the encoding for the image. Use img_to_encoding() see example above. (≈ 1 line)
    encoding = img_to_encoding(image_path, model)
    # Step 2: Compute distance with identity's image (≈ 1 line)
    dist = np.linalg.norm(database[identity] - encoding)
    # Step 3: Open the door if dist < 0.7, else don't open (≈ 3 lines)
    if dist < 0.7:
        print("It's " + str(identity) + ", welcome in!")
        door_open = True
    else:
        print("It's not " + str(identity) + ", please go away")
        door_open = False
    ### END CODE HERE
    return dist, door_open

# Younes is trying to enter the office and the camera takes a picture of him ("images/camera_0.jpg"). Let's run your verification algorithm on this picture:
#
#

# In[31]:

# BEGIN UNIT TEST
assert(np.allclose(verify("images/camera_1.jpg", "bertrand", database, FRmodel), (0.54364836, True)))
assert(np.allclose(verify("images/camera_3.jpg", "bertrand", database, FRmodel), (0.38616243, True)))
assert(np.allclose(verify("images/camera_1.jpg", "younes", database, FRmodel), (1.3963861, False)))
assert(np.allclose(verify("images/camera_3.jpg", "younes", database, FRmodel), (1.3872949, False)))

verify("images/camera_0.jpg", "younes", database, FRmodel)
# END UNIT TEST

# **Expected Output**:
#
#

#

#
#             It's Younes, welcome in!
#
#
#            (0.5992946, True)
#
#

#

# Benoit, who does not work in the office, stole Kian's ID card and tried to enter the office. Naughty Benoit! The camera took a picture of Benoit ("images/camera_2.jpg).
#
#
#
# Run the verification algorithm to check if Benoit can enter.

# In[32]:

verify("images/camera_2.jpg", "kian", database, FRmodel)

# **Expected Output**:
#
#

#

#
#             It's not Kian, please go away
#
#
#            (1.0259346, False)
#
#

#

#
# ### 5.2 - Face Recognition
#
# Your face verification system is mostly working. But since Kian got his ID card stolen, when he came back to the office the next day he couldn't get in!
#
# To solve this, you'd like to change your face verification system to a face recognition system. This way, no one has to carry an ID card anymore. An authorized person can just walk up to the building, and the door will unlock for them!
#
# You'll implement a face recognition system that takes as input an image, and figures out if it is one of the authorized persons (and if so, who). Unlike the previous face verification system, you will no longer get a person's name as one of the inputs.
#
#
# ### Exercise 3 - who_is_it
#
# Implement `who_is_it()` with the following steps:
#
# - Compute the target encoding of the image from `image_path`
# - Find the encoding from the database that has smallest distance with the target encoding.
# - Initialize the `min_dist` variable to a large enough number (100). This helps you keep track of the closest encoding to the input's encoding.
# - Loop over the database dictionary's names and encodings. To loop use for (name, db_enc) in `database.items()`.
# - Compute the L2 distance between the target "encoding" and the current "encoding" from the database. If this distance is less than the min_dist, then set min_dist to dist, and identity to name.

# In[33]:

# UNQ_C3(UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: who_is_it

def who_is_it(image_path, database, model):
    """
    Implements face recognition for the office by finding who is the person on the image_path image.

    Arguments:
        image_path -- path to an image
        database -- database containing image encodings along with the name of the person on the image
        model -- your Inception model instance in Keras

    Returns:
        min_dist -- the minimum distance between image_path encoding and the encodings from the database
        identity -- string, the name prediction for the person on image_path
    """

    ### START CODE HERE

    ## Step 1: Compute the target "encoding" for the image. Use img_to_encoding() see example above. ## (≈ 1 line)
    encoding =  img_to_encoding(image_path, model)

    ## Step 2: Find the closest encoding ##

    # Initialize "min_dist" to a large value, say 100 (≈1 line)
    min_dist = 100

    # Loop over the database dictionary's names and encodings.
    for (name, db_enc) in database.items():

        # Compute L2 distance between the target "encoding" and the current db_enc from the database. (≈ 1 line)
        dist = np.linalg.norm(db_enc - encoding)

        # If this distance is less than the min_dist, then set min_dist to dist, and identity to name. (≈ 3 lines)
        if dist < min_dist:
            min_dist = dist
            identity = name
    ### END CODE HERE

    if min_dist > 0.7:
        print("Not in the database.")
    else:
        print ("it's " + str(identity) + ", the distance is " + str(min_dist))

    return min_dist, identity

# Younes is at the front door and the camera takes a picture of him ("images/camera_0.jpg"). Let's see if your `who_it_is()` algorithm identifies Younes.

# In[34]:

# BEGIN UNIT TEST
# Test 1 with Younes pictures
who_is_it("images/camera_0.jpg", database, FRmodel)

# Test 2 with Younes pictures
test1 = who_is_it("images/camera_0.jpg", database, FRmodel)
assert np.isclose(test1[0], 0.5992946)
assert test1[1] == 'younes'

# Test 3 with Younes pictures
test2 = who_is_it("images/younes.jpg", database, FRmodel)
assert np.isclose(test2[0], 0.0)
assert test2[1] == 'younes'
# END UNIT TEST

# **Expected Output**:
#
#

#

#
#             it's Younes, the distance is 0.5992946
#
#
#            (0.5992946, 'younes')
#
#

#

#
# You can change "camera_0.jpg" (picture of Younes) to "camera_1.jpg" (picture of Bertrand) and see the result.

# **Congratulations**!
# You've completed this assignment, and your face recognition system is working well! It not only lets in authorized persons, but now people don't need to carry an ID card around anymore!
#
# You've now seen how a state-of-the-art face recognition system works, and can describe the difference between face recognition and face verification. Here's a quick recap of what you've accomplished:
#
# - Posed face recognition as a binary classification problem
# - Implemented one-shot learning for a face recognition problem
# - Applied the triplet loss function to learn a network's parameters in the context of face recognition
# - Mapped face images into 128-dimensional encodings using a pretrained model
# - Performed face verification and face recognition with these encodings
#
# Great work!

#
#
# **What you should remember**:
#
# - Face verification solves an easier 1:1 matching problem; face recognition addresses a harder 1:K matching problem.
#
# - Triplet loss is an effective loss function for training a neural network to learn an encoding of a face image.
#
# - The same encoding can be used for verification and recognition. Measuring distances between two images' encodings allows you to determine whether they are pictures of the same person.

# **Ways to improve your facial recognition model**:
#
# Although you won't implement these here, here are some ways to further improve the algorithm:
#
# - Put more images of each person (under different lighting conditions, taken on different days, etc.) into the database. Then, given a new image, compare the new face to multiple pictures of the person. This would increase accuracy.
#
# - Crop the images to contain just the face, and less of the "border" region around the face. This preprocessing removes some of the irrelevant pixels around the face, and also makes the algorithm more robust.

#
# ## 6 - References
# 1. Florian Schroff, Dmitry Kalenichenko, James Philbin (2015). [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/pdf/1503.03832.pdf)
#
# 2. Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, Lior Wolf (2014). [DeepFace: Closing the gap to human-level performance in face verification](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf)
#
# 3. This implementation also took a lot of inspiration from the official FaceNet github repository: https://github.com/davidsandberg/facenet
#
# 4. Further inspiration was found here: https://machinelearningmastery.com/how-to-develop-a-face-recognition-system-using-facenet-in-keras-and-an-svm-classifier/
#
# 5. And here: https://github.com/nyoki-mtl/keras-facenet/blob/master/notebook/tf_to_keras.ipynb

4 - Convolutional Neural Networks/Face_Recognition.py


import face_recognition
import cv2
import numpy as np
import os
import sys
import subprocess
import argparse
from utils import col, removeExtension

def parse_args():
    parser = argparse.ArgumentParser(
        description="Highlight recognized faces in videos")
    parser.add_argument("files", metavar="filepath", type=str,
                        nargs="+", help="file path of videos to process")
    parser.add_argument("-a", "--accuracy", dest="accuracy", type=float,
                        help="the factor the video will get scaled down by before processing (faster; noninversive)")
    arguments = vars(parser.parse_args())
    return arguments

def main(options):
    if options.get("accuracy", None) != None:
        ACCURACY = options["accuracy"]
    else:
        ACCURACY = 0.25

    print("using accuracy of " + str(ACCURACY))

    for video in options.get("files", []):
        FILE_PATH = options.get("files", [])[options.get("files").index(video)]
        FILE_NAME = removeExtension(FILE_PATH).replace("videos/", "")
        NEW_FILE_PATH = "videos/" + FILE_NAME + "-rec.mp4"

        # extracting audio as mp3
        os.system("ffmpeg -i " + FILE_PATH + " -loglevel warning temp/" +
                  FILE_NAME + ".mp3")

        # getting framerate of video
        cmd = ['ffprobe', '-v', '0', '-of', 'csv=p=0', '-select_streams',
               'v:0', '-show_entries', 'stream=r_frame_rate', FILE_PATH]
        FRAME_RATE = subprocess.Popen(
            cmd, stdout=subprocess.PIPE).communicate()[0]
        FRAME_RATE = str(FRAME_RATE).replace("b", "").replace(
            "\n", "").replace("\\n", "").replace("'", "")
        frame_rate_divisors = FRAME_RATE.split("/")
        FRAME_RATE = float(
            int(frame_rate_divisors[0]) / int(frame_rate_divisors[1]))

        video_capture = cv2.VideoCapture(FILE_PATH)

        known_face_encodings = []
        known_face_names = []

        for i in os.listdir("faces"):
            if i == ".DS_Store":
                continue
            f = face_recognition.load_image_file("faces/" + str(i))
            try:
                encoding = face_recognition.face_encodings(f)[0]
            except IndexError:
                print("no encoding available for " + str(i))
                continue
            known_face_encodings.append(encoding)
            known_face_names.append(str(i))

        # Initialize some variables
        face_locations = []
        face_encodings = []
        face_names = []
        process_this_frame = True

        cc = cv2.VideoWriter_fourcc(*"MP4V")
        writer = cv2.VideoWriter(NEW_FILE_PATH, cc, FRAME_RATE, (1920, 1080))

        try:
            while True:
                # Grab a single frame of video
                ret, frame = video_capture.read()

                # Resize frame of video to 1/4 size for faster face recognition processing
                small_frame = cv2.resize(
                    frame, (0, 0), fx=ACCURACY, fy=ACCURACY)

                # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses)
                rgb_small_frame = small_frame[:, :, ::-1]

                # Only process every other frame of video to save time
                if process_this_frame:
                    # Find all the faces and face encodings in the current frame of video
                    face_locations = face_recognition.face_locations(
                        rgb_small_frame)
                    face_encodings = face_recognition.face_encodings(
                        rgb_small_frame, face_locations)

                    face_names = []
                    for face_encoding in face_encodings:
                        # See if the face is a match for the known face(s)
                        matches = face_recognition.compare_faces(
                            known_face_encodings, face_encoding)
                        name = "Unknown"

                        # # If a match was found in known_face_encodings, just use the first one.
                        # if True in matches:
                        #     first_match_index = matches.index(True)
                        #     name = known_face_names[first_match_index]

                        # Or instead, use the known face with the smallest distance to the new face
                        face_distances = face_recognition.face_distance(
                            known_face_encodings, face_encoding)
                        best_match_index = np.argmin(face_distances)
                        if matches[best_match_index]:
                            name = known_face_names[best_match_index]

                        face_names.append(name)

                process_this_frame = not process_this_frame

                # Display the results
                for (top, right, bottom, left), name in zip(face_locations, face_names):
                    # Scale back up face locations since the frame we detected in was scaled to 1/4 size
                    top *= int(1 / ACCURACY)
                    right *= int(1 / ACCURACY)
                    bottom *= int(1 / ACCURACY)
                    left *= int(1 / ACCURACY)

                    # Draw a box around the face
                    cv2.rectangle(frame, (left, top),
                                  (right, bottom), (0, 0, 255), 2)

                    # Draw a label with a name below the face
                    cv2.rectangle(frame, (left, bottom - 35),
                                  (right, bottom), (0, 0, 255), cv2.FILLED)
                    font = cv2.FONT_HERSHEY_DUPLEX
                    cv2.putText(frame, name, (left + 6, bottom - 6),
                                font, 1.0, (255, 255, 255), 1)

                # Display the resulting image
                cv2.imshow('Video', frame)
                writer.write(frame)

                # Hit 'q' on the keyboard to quit!
                if cv2.waitKey(1) & 0xFF == ord('q'):
                    break
        except Exception as e:
            print(col.FAIL + "Error: " + col.ENDC + str(e))
        finally:
            writer.release()
            video_capture.release()
            cv2.destroyAllWindows()

        # write extracted mp3 from temp/ to FILE_PATH
        os.system(
            f"ffmpeg -i {NEW_FILE_PATH} -i temp/{FILE_NAME}.mp3 -loglevel warning -y {removeExtension(NEW_FILE_PATH)}-with_audio.mp4")
        os.remove("temp/" + str(FILE_NAME) + ".mp3")
        os.remove(NEW_FILE_PATH)

if __name__ == "__main__":
    main(parse_args())


"""
NCL_xy_18.py
============
This script illustrates the following concepts:
    - Filling the area between two curves in an XY plot
    - Labeling the bottom X axis with years
    - Drawing a main title on three separate lines
    - Calculating a weighted average
    - Changing the size/shape of an XY plot using viewport resources
    - Manually creating a legend
    - Overlaying XY plots on each other
    - Maximizing plots after they've been created

See following URLs to see the reproduced NCL plot & script:
    - Original NCL script: https://www.ncl.ucar.edu/Applications/Scripts/xy_18.ncl
    - Original NCL plot: https://www.ncl.ucar.edu/Applications/Images/xy_18_lg.png
"""

###############################################################################
# Import packages:

import numpy as np
import xarray as xr
from matplotlib import pyplot as plt

import geocat.datafiles as gdf
from geocat.viz import util as gvutil

###############################################################################
# Read in data:
# -------------
#
# Open files and read in monthly data
#
# Xarray's ``open_mfdataset`` (open multi-file dataset) method will attempt to
# merge all of the individual datasets (i.e., NetCDF files) into one single
# Xarray ``Dataset``.  The ``concat_dim`` and ``combine`` keyword arguments to
# this method give you control over how this merging takes place (see the
# Xarray documentation for more information).
#
# In the below example, each NetCDF file represents the same variables and
# coordinates, but from a different ensemble member.  There is no ``case`` (or
# ensemble) dimension explicitly declared in the files, so we use the
# ``concat_dim`` argument to state that we will create a new dimension called
# ``case`` that spans the ensemble members.  Here, each file contains a
# ``TREFHT`` variable that depends upon dimensions ``(time, lat, lon)`` and
# coordinate variables ``time``, ``lat`` and ``lon``.  After opening these
# files with ``open_mfdataset``, the resulting Xarray ``Dataset`` will consist
# of a ``TREFHT`` variable that depends upon dimensions ``(case, time, lat, lon)``
# and coordinate variables ``case``, ``time``, ``lat`` and ``lon``.
#
# **NOTE:** One of the files (``TREFHT.B06.69.atm.1890-1999ANN.nc``) contains
# a ``time`` coordinate variable with a ``calendar`` attribute having the
# value ``noleap`` (i.e., the "No leap year" non-standard calendar).  The
# ``time`` coordinate variable in all of the other files do not have a
# ``calendar`` attribute *at all*.  By default, when Xarray's ``open_mfdataset``
# reads each individual dataset, it will attempt to decode the ``time`` coordinate
# into an appropriate ``datetime`` object, so that you can then take advantage of
# Xarray's (and Pandas's) excellent time-series manipulation capabilities.
# However, due to the lacking ``calendar`` attribute in most of the files
# (which, according to CF conventions, defaults to the ``standard`` Gregorian
# calendar) and the ``noleap`` calendar attribute in one of the files, the
# ``time`` coordinate variable will be interpreted as "non-uniform" across all
# of the datasets.  To fix this problem, we tell Xarray's ``open_mfdataset``
# function to *not* decode the ``time`` coordinate into ``datetime`` objects
# by passing the ``decode_times=False`` argument.  Second, we pass a pre-processing
# function via the ``preprocess`` argument to ``open_mfdataset``, telling
# Xarray to read each individual dataset from file (with the ``decode_times=False``
# option) and then modify the resulting dataset according to the pre-processing
# function.  In this case, the pre-processing function (``assume_noleap_calendar``)
# takes the single-file dataset, sets the ``calendar`` attribute of the ``time``
# coordinate variable to ``noleap``, and returns the *decoded* dataset (using
# the Xarray function ``decode_cf``).  Work-arounds like this are needed
# whenever you have "errors" or "inconsistancies" in your data.

# Define the xarray.open_mfdataset pre-processing function
# (Must take an xarray.Dataset as input and return an xarray.Dataset)
def assume_noleap_calendar(ds):
    ds.time.attrs['calendar'] = 'noleap'
    return xr.decode_cf(ds)

# Create a dataset for the "natural" (i.e., no anthropogenic effects) data
nfiles = [
    gdf.get("netcdf_files/TREFHT.B06.66.atm.1890-1999ANN.nc"),
    gdf.get("netcdf_files/TREFHT.B06.67.atm.1890-1999ANN.nc"),
    gdf.get("netcdf_files/TREFHT.B06.68.atm.1890-1999ANN.nc"),
    gdf.get("netcdf_files/TREFHT.B06.69.atm.1890-1999ANN.nc")
]
nds = xr.open_mfdataset(nfiles,
                        concat_dim='case',
                        combine='nested',
                        preprocess=assume_noleap_calendar,
                        decode_times=False)

# Create a dataset for the "natural + anthropogenic" data
vfiles = [
    gdf.get("netcdf_files/TREFHT.B06.61.atm.1890-1999ANN.nc"),
    gdf.get("netcdf_files/TREFHT.B06.59.atm.1890-1999ANN.nc"),
    gdf.get("netcdf_files/TREFHT.B06.60.atm.1890-1999ANN.nc"),
    gdf.get("netcdf_files/TREFHT.B06.57.atm.1890-1999ANN.nc")
]
vds = xr.open_mfdataset(vfiles,
                        concat_dim='case',
                        combine='nested',
                        preprocess=assume_noleap_calendar,
                        decode_times=False)

# Read the "weights" file
# (The xarray.Dataset.expand_dims call adds the longitude dimension to the
# dataset, which originally depends only upon the latitude dimension. This
# arguably makes computing the weighted means below more straight-forward.)
gds = xr.open_dataset(gdf.get("netcdf_files/gw.nc"))
gds = gds.expand_dims(dim={'lon': nds.lon})

###############################################################################
# Observations:
# -------------
#
# Read in the observational data from an ASCII (text) file.  Here, we use
# Numpy's nice ``loadtxt`` method to read the data from the text file and
# return a Numpy array with ``float`` type.  Then, we construct an Xarray
# ``DataArray`` explicitly, since the time values are not stored in the
# ASCII data file (we have to know them!).

obs_data = np.loadtxt(gdf.get("ascii_files/jones_glob_ann_2002.asc"),
                      dtype=float)
obs_time = xr.cftime_range('1856-07-16T22:00:00',
                           freq='365D',
                           periods=len(obs_data),
                           calendar='noleap')
obs = xr.DataArray(name='TREFHT', data=obs_data, coords=[('time', obs_time)])

###############################################################################
# NCL-based Weighted Mean Function:
# ---------------------------------
#
# We define this function just for convenience.  This is equivalent to how
# NCL computes the weighted mean.

def horizontal_weighted_mean(var, wgts):
    return (var * wgts).sum(dim=['lat', 'lon']) / wgts.sum(dim=['lat', 'lon'])

###############################################################################
# Natural data:
# -------------
#
# We compute the weighted mean across the latitude and longitude dimensions
# (leaving only the ``case`` and ``time`` dimensions), and then we compute the
# anomaly measured from the average of the first 30 years.

gavn = horizontal_weighted_mean(nds["TREFHT"], gds["gw"])
gavan = gavn - gavn.sel(time=slice('1890', '1920')).mean(dim='time')

###############################################################################
# Natural + Anthropogenic data:
# -----------------------------
#
# We do the same thing for the "natural + anthropogenic" data.

gavv = horizontal_weighted_mean(vds["TREFHT"], gds["gw"])
gavav = gavv - gavv.sel(time=slice('1890', '1920')).mean(dim='time')

###############################################################################
# Observation data:
# -----------------
#
# We do the same thing for the observation data.

obs_avg = obs.sel(time=slice('1890', '1999')) - obs.sel(
    time=slice('1890', '1920')).mean(dim='time')

###############################################################################
# Calculate the ensemble Min. & Max. & Mean:
# ------------------------------------------
#
# Here we find the ``min``, ``max``, and ``mean`` along the ``case`` (i.e.,
# ensemble) dimension (leaving only the ``time`` dimension) for both of our
# datasets.  We compute the equivalent anomaly for the observations data.

gavan_min = gavan.min(dim='case')
gavan_max = gavan.max(dim='case')
gavan_avg = gavan.mean(dim='case')

gavav_min = gavav.min(dim='case')
gavav_max = gavav.max(dim='case')
gavav_avg = gavav.mean(dim='case')

###############################################################################
# Plot:
# -----

# Generate figure (set its size (width, height) in inches) and axes
fig, ax = plt.subplots(figsize=(10.5, 6))

# We create the time axis data, not as datetime objects, but as just years
# The following line of code is equivalent to this:
#     time = [t.year for t in gavan.time.values]
# but it uses Xarray's convenient DatetimeAccessor functionality.
time = gavan.time.dt.year

# Plot data and add a legend
ax.plot(time, obs_avg, color='black', label='Observations', zorder=4)
ax.plot(time, gavan_avg, color='blue', label='Natural', zorder=3)
ax.plot(time, gavav_avg, color='red', label='Anthropogenic + Natural', zorder=2)
ax.legend(loc='upper left', frameon=False, fontsize=18)

# Use geocat.viz.util convenience function to add minor and major tick lines
gvutil.add_major_minor_ticks(ax,
                             x_minor_per_major=4,
                             y_minor_per_major=3,
                             labelsize=20)

# Use geocat.viz.util convenience function to set axes limits & tick values without calling several matplotlib functions
gvutil.set_axes_limits_and_ticks(ax,
                                 xlim=(1890, 2000),
                                 ylim=(-0.4, 1),
                                 xticks=np.arange(1900, 2001, step=20),
                                 yticks=np.arange(-0.3, 1, step=0.3))

# Set three titles on top of each other using axes title and texts
ax.set_title('Parallel Climate Model Ensembles', fontsize=24, pad=60.0)
ax.text(0.5,
        1.125,
        'Global Temperature Anomalies',
        fontsize=18,
        ha='center',
        va='center',
        transform=ax.transAxes)
ax.text(0.5,
        1.06,
        'from 1890-1919 average',
        fontsize=14,
        ha='center',
        va='center',
        transform=ax.transAxes)
ax.set_ylabel('$^\circ$C', fontsize=24)
ax.fill_between(time, gavan_min, gavan_max, color='lightblue', zorder=0)
ax.fill_between(time, gavav_min, gavav_max, color='lightpink', zorder=1)

# Show the plot
plt.tight_layout()
plt.show()


import numpy as np
import cv2
import os
from PIL import Image
#from tensorflow import keras
from tensorflow.keras.models import model_from_json
from colorama import Fore, Back
#import tensorflow as tf

RESULT_LIST = ["Positive, Bacterial", "Negative", "Positive, Viral"]

def create_model(model_file):
    with open(model_file, "r") as json_file:
        loaded_model_json = json_file.read()
        loaded_model_json = model_from_json(loaded_model_json)
    return loaded_model_json

def run_predictions(path_to_img):
    model = create_model("model/model.json")
    model.load_weights("model/weights.h5")
    #print(model.summary())
    img = cv2.imread(path_to_img)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    #width, height = img.shape[:2]
    img = cv2.resize(img , (200, 200))
    res = np.argmax(model.predict(img[np.newaxis, :, :]))
    if (res == 0):
        print(Fore.MAGENTA + Back.BLACK + "Positive, Bacterial")
    elif (res == 1):
        print(Fore.GREEN + Back.BLACK + "Negative")
    else:
        print(Fore.CYAN + Back.BLACK + "Positive, Viral")

input_img = input("Enter image name: ")

path_to_img = os.path.join("images", input_img)
run_predictions(path_to_img)


# -*- coding: utf-8 -*-
# <nbformat>3.0</nbformat>

# <markdowncell>

# ># IOOS System Test: [Extreme Events Theme:](https://github.com/ioos/system-test/wiki/Development-of-Test-Themes#theme-2-extreme-events) Inundation

# <markdowncell>

# ### This is a single "spin-off notebook" for the basic oceanography variables (wind, waves, currents, and water level) to test all CSW end points for multiple geographies.
#
# ### Questions
# * Is data available for the basic oceanography variables in the CSW endpoints for multiple locations?
# * Let's take it one step further. Is the data recent (< 1 month)?
#
# ####Methodology:
#
# * Define temporal and spatial bounds of interest
# * Show bounding boxes being tested on a map
# * Define standard names of variables of interest to search for in data sets
# * Search for available service endpoints in the CSW catalogs meeting the search criteria for each variable
# * Plot the results in a horizontal bar graph

# <markdowncell>

# ### import required libraries

# <codecell>

import matplotlib.pyplot as plt
import numpy as np

from owslib.csw import CatalogueServiceWeb
from owslib import fes
from owslib.ows import ExceptionReport

import folium
import pandas as pd
import itertools
import datetime as dt
from utilities import (fes_date_filter, service_urls, get_coordinates, inline_map, css_styles,
                       insert_progress_bar, update_progress_bar)
css_styles()

# <markdowncell>

# ### Define spatial bounds of interest

# <codecell>

bounding_box_type = "box"

# Bounding Box [lon_min, lat_min, lon_max, lat_max]
locations = {'Hawaii': [-160.0, 18.0, -154., 23.0],
             'Caribbean': [-75, 12, -55, 26],
             'East Coast': [-77, 30, -70, 40],
             'North West': [-130, 38, -121, 50],
             'Gulf of Mexico': [-94, 26, -84, 32],
             'Arctic': [-179, 63, 179, 80],
             'North East': [-74, 40, -67, 46]}

# <markdowncell>

# ### Plot the bounding boxes

# <codecell>

lat_center = 45
lon_center = -90
m = folium.Map(location=[lat_center, lon_center], zoom_start=2)

# Loop through bounding boxes
for location, bounding_box in locations.iteritems():
    # Create popup string for the bounding box
    popup_string = location
    m.line(get_coordinates(bounding_box, bounding_box_type), line_color='#FF0000', line_weight=5)

inline_map(m)

# <markdowncell>

# ### Define standard names of variable of interest to search for in data sets

# <markdowncell>

#
 - We need to specify all the names we know for each variable, names that will get used in the CSW search, and also to find data in the datasets that are returned. This is ugly and fragile. There hopefully will be a better way in the future...

# <codecell>

# put the names in a dict for ease of access
names_dict = {}
names_dict["waves"] = {"names": ['sea_surface_wave_significant_height',
                                 'significant_wave_height',
                                 'significant_height_of_wave',
                                 'sea_surface_wave_significant_height(m)',
                                 'sea_surface_wave_significant_height (m)',
                                 'water_surface_height'],
                      "sos_name": ["waves"]}

names_dict['winds'] = {"names": ['eastward_wind', 'u-component_of_wind',
                                 'u-component_of_wind_height_above_ground',
                                 'ugrd10m',
                                 'wind'],
                       "v_names": ['northward_wind',
                                   'v-component_of_wind',
                                   'v-component_of_wind_height_above_ground',
                                   'vgrd10m',
                                   'wind'],
                       "sos_name": ['winds']}

names_dict['currents'] = {"names": ['eastward_sea_water_velocity_assuming_no_tide',
                                    'surface_eastward_sea_water_velocity',
                                    '*surface_eastward_sea_water_velocity*',
                                    'eastward_sea_water_velocity'],
                          "v_names": ['northward_sea_water_velocity_assuming_no_tide',
                                      'surface_northward_sea_water_velocity',
                                     '*surface_northward_sea_water_velocity*',
                                     'northward_sea_water_velocity'],
                          "sos_name": ['currents']}

names_dict['water_level'] = {"names": ['water_surface_height_above_reference_datum',
                                       'sea_surface_height_above_geoid',
                                       'sea_surface_elevation',
                                       'sea_surface_height_above_reference_ellipsoid',
                                       'sea_surface_height_above_sea_level',
                                       'sea_surface_height','water level']}

# <markdowncell>

# ### Define the csw endpoints we know about

# <markdowncell>

#
This cell lists catalog endpoints. The list is updated by the IOOS Program Office here: https://github.com/ioos/system-test/wiki/Service-Registries-and-Data-Catalogs

# <codecell>

endpoints = ['http://www.nodc.noaa.gov/geoportal/csw',
             'http://data.nodc.noaa.gov/geoportal/csw',
             'http://www.ngdc.noaa.gov/geoportal/csw',
             'http://catalog.data.gov/csw-all',
             'https://data.noaa.gov/csw',
             'http://geoport.whoi.edu/geoportal/csw',
             'https://edg.epa.gov/metadata/csw',
             'http://cmgds.marine.usgs.gov/geonetwork/srv/en/csw',
             'http://cida.usgs.gov/gdp/geonetwork/srv/en/csw',
             'http://geodiscover.cgdi.ca/wes/serviceManagerCSW/csw',
             'http://cwic.csiss.gmu.edu/cwicv1/discovery',
             'https://www.sciencebase.gov/catalog/item/519bee13e4b0e4e151f0232c/csw'
             ]
# 'http://pacioos.org/search/'
# 'http://geoport.whoi.edu/gi-cat/services/cswiso',

# Set the maximum number of records the CSW will return
max_records = 2000

# <markdowncell>

# ### Is data available for the basic oceanography variables in the CSW endpoints for multiple locations?
#
# #### Check the CSW endpoints for each variable and location

# <markdowncell>

#
This next cell takes a long time to process!
Go grab a coffee

# <codecell>

# Add a waitbar to monitor status
divid = insert_progress_bar(title='Searching catalogs. Please wait...', color='red')

# Save all of the results in a list of Dataframes
results = {}
all_data = []

count = 0
# Loop through the csw endpoints
for endpoint in endpoints:
    print '\n' + endpoint

    csw = CatalogueServiceWeb(endpoint, timeout=60)
    # loop through the variables
    for var_name in names_dict:
#         print '\n' + var_name.upper()
        num_recs = []
        for location, bounding_box in locations.iteritems():
#             print location

            bbox = fes.BBox(bounding_box)
            #use the search name to create search filter
            or_filt = fes.Or([fes.PropertyIsLike(propertyname='apiso:AnyText',
                                                 literal='*%s*' % val,
                                                 escapeChar='\\',
                                                 wildCard='*',
                                                 singleChar='?') for val in names_dict[var_name]["names"]])
            filter_list = [fes.And([ bbox, or_filt])]
            # try request using multiple filters "and" syntax: [[filter1,filter2]]
            try:
                csw.getrecords2(constraints=filter_list, maxrecords=max_records, resulttype='hits')
            except Exception as e:
                print '\t' + 'ERROR - ' + str(e)
                num_recs.append(np.NaN)
            else:
#                 print csw.results['matches']
                num_recs.append(csw.results['matches'])

        results[var_name] = np.array(num_recs)

    # Save the results
    prod = list(itertools.product([endpoint], locations.keys()))
    mi = pd.MultiIndex.from_tuples(prod, names=['endpoint', 'location'])
    all_data.append(pd.DataFrame(results, index=mi))

    # Update progress bar
    count += 1
    percent_complete = (float(count)/float(len(endpoints)))*100
    update_progress_bar(divid, percent_complete)

# all_data_concat = pd.concat(all_data)

# <markdowncell>

#
 Some servers have a maximum amount of records you can retrieve at once. See: https://github.com/ioos/system-test/issues/126

# <markdowncell>

# #### Let's plot the results in a bar graph

# <codecell>

alldata_concat = pd.concat(all_data)
endpoint_group = alldata_concat.groupby(level=0)
# can uncomment this for a terser, but less well annotated plot
# endpoint_group.plot(kind='barh')
for grp_name, grp in endpoint_group:
    fig, ax = plt.subplots()
    # eliminate endpoint from index since it will be the graph title
    grp.reset_index(0, drop=True).plot(ax=ax, kind="barh", figsize=(10, 8,),
                                       title=grp_name)
    ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    ax.set_xlabel('Number of records')

# <codecell>

# By location across all endpoints
location_group = alldata_concat.fillna(0).groupby(level='location').sum()
ax = location_group.plot(kind='barh', title='All records by location', figsize=(10, 8))
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.set_xlabel('Number of records')
print(location_group)

# <markdowncell>

# ### Is the data recent (< 1 month)?
#
# #### Let's add a temporal extent to the search

# <codecell>

#temporal range - last 28 days and next 3 days (forecast data)
jd_now = dt.datetime.utcnow()
jd_start,  jd_stop = jd_now - dt.timedelta(days=28), jd_now + dt.timedelta(days=3)

start_date = jd_start.strftime('%Y-%m-%d %H:00')
stop_date = jd_stop.strftime('%Y-%m-%d %H:00')

print start_date + ' to ' + stop_date

# <codecell>

# Add a waitbar to monitor status
divid = insert_progress_bar(title='Searching catalogs. Please wait...', color='red')

# Save all of the results in a list of Dataframes
results = {}
recent_data = []

count = 0
# Loop through the csw endpoints
for endpoint in endpoints:
    print '\n' + endpoint

    try:
        csw = CatalogueServiceWeb(endpoint, timeout=60)
    # continue processing if an endpoint is down or otherwise nonfunctional
    # but report the exception returned from OWSLib
    except ExceptionReport as e:
        print('Error accessing CSW endpoint "{0}". Error report: {1}'.format(endpoint, e))
        continue
    # loop through the variables
    for var_name in names_dict:
#         print '\n' + var_name.upper()
        num_recs = []
        for location, bounding_box in locations.iteritems():
#             print location
            # convert User Input into FES filters
            start, stop = fes_date_filter(start_date, stop_date)
            bbox = fes.BBox(bounding_box)

            #use the search name to create search filter
            or_filt = fes.Or([fes.PropertyIsLike(propertyname='apiso:AnyText',
                                                 literal='*%s*' % val,
                                                 escapeChar='\\',
                                                 wildCard='*',
                                                 singleChar='?') for val in names_dict[var_name]["names"]])
            filter_list = [fes.And([ bbox, start, stop, or_filt])]
            # try request using multiple filters "and" syntax: [[filter1,filter2]]
            try:
                csw.getrecords2(constraints=filter_list, #maxrecords=max_records,
                                resulttype='hits')

            except Exception as e:
                print '\t' + 'ERROR - ' + str(e)
                num_recs.append(np.NaN)
            else:
#                 print '\t' + str(len(csw.records)) + " csw records found"
#                 print csw.results['matches']
                num_recs.append(csw.results['matches'])

        results[var_name] = np.array(num_recs)

    # Save the results
    prod = list(itertools.product([endpoint], locations.keys()))
    mi = pd.MultiIndex.from_tuples(prod, names=['endpoint', 'location'])
    df = pd.DataFrame(results, index=mi)
    # if all the entries in the entire endpoint have zero counts, do not include this
    # endpoint provider
    if (df.stack().fillna(0) != 0).any():
        recent_data.append(df)
    else:
        continue

    # Update progress bar
    count += 1
    percent_complete = (float(count)/float(len(endpoints)))*100
    update_progress_bar(divid, percent_complete)

recent_data_concat = pd.concat(recent_data)

# <markdowncell>

# #### Once again, let's plot the results in a bar graph

# <codecell>

endpoint_group_recent = recent_data_concat.groupby(level='endpoint')
# can uncomment this for a plot with tuple groups as y axis marks
# endpoint_group.plot(kind='barh', figsize=(10, 8,))
for grp_name, grp in endpoint_group_recent:
#     print grp_name, grp
    fig, ax = plt.subplots()
    # eliminate endpoint from index since it will be the graph title
    grp.reset_index(0, drop=True).plot(ax=ax, kind="barh", figsize=(10, 8,),
                                       title=grp_name)
    ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    ax.set_xlabel('Number of records')

# <markdowncell>

# ### Reorganize the data by variable name and plot

# <codecell>

#return counts of each variables as a series and place the variable type as the first index
stacked = recent_data_concat.stack().reorder_levels([2,1,0])
stacked.index.names = ['variable_type', 'location', 'endpoint']
stacked.name = 'record_counts'
#stacked.index.names = stacked.index.names[['variable_type']
by_var = stacked.unstack()
for grpname, grps in by_var.groupby(level='variable_type'):
   # get rid of variable index since we are already grouping by it and have its name
   cur_grp = grps.reset_index(level='variable_type', drop=True)
   fig, ax = plt.subplots()
   cur_grp.plot(ax=ax, kind='barh', stacked=True, figsize=(10, 8,), legend=False, title=grpname)
   ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
   ax.set_xlabel('Number of records')

# <codecell>

# show
ax = by_var.groupby(level='variable_type').sum().plot(kind='bar',
                                                      figsize=(10, 8,),
                                                      title='Recent variable counts by endpoint')

# <markdowncell>

# ### Conclusions
# * The core oceanographic variables are available from numerous CSW endpoints
# * But if you are looking for recent (< 1 month) data, the NGDC and NODC CSW is the best bet
# * Each of the locations tested seemed to have good data coverage except currents in the Arctic


#!/usr/bin/env python
# coding: utf-8

#
#
# # Earth Analytics Education - EA  Python Course Spring 2021

# ## Important  - Assignment Guidelines
#
# 1. Before you submit your assignment to GitHub, make sure to run the entire notebook with a fresh kernel. To do this first, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart & Run All)
# 2. Always replace the `raise NotImplementedError()` code with your code that addresses the activity challenge. If you don't replace that code, your notebook will not run.
#
# ```
# # YOUR CODE HERE
# raise NotImplementedError()
# ```
#
# 3. Any open ended questions will have a "YOUR ANSWER HERE" within a markdown cell. Replace that text with your answer also formatted using Markdown.
# 4. **DO NOT RENAME THIS NOTEBOOK File!** If the file name changes, the autograder will not grade your assignment properly.
# 6. When you create a figure, comment out `plt.show()` to ensure the autograder can grade your plots. For figure cells, DO NOT DELETE the code that says `DO NOT REMOVE LINE BELOW`.
#
# ```
# ### DO NOT REMOVE LINE BELOW ###
# student_plot1_ax = nb.convert_axes(plt)
# ```
#
# * Only include the package imports, code, and outputs that are required to run your homework assignment.
# * Be sure that your code can be run on any operating system. This means that:
#    1. the data should be downloaded in the notebook to ensure it's reproducible
#    2. all paths should be created dynamically using the `os.path.join`
#
# ## Follow to PEP 8 Syntax Guidelines & Documentation
#
# * Run the `autopep8` tool on all cells prior to submitting (HINT: hit shift + the tool to run it on all cells at once!
# * Use clear and expressive names for variables.
# * Organize your code to support readability.
# * Check for code line length
# * Use comments and white space sparingly where it is needed
# * Make sure all python imports are at the top of your notebook and follow PEP 8 order conventions
# * Spell check your Notebook before submitting it.
#
# For all of the plots below, be sure to do the following:
#
# * Make sure each plot has a clear TITLE and, where appropriate, label the x and y axes. Be sure to include UNITS in your labels.
#

# ### Add Your Name Below
# **Your Name: Christian Haselwimmer**

#

# ---

# # Week 04 and 05 Homework - Automate NDVI Workflow
#
# For this assignment, you will write code to generate a plot of the mean normalized difference vegetation index (NDVI) for two different sites in the United States across one year of data:
#
# * San Joaquin Experimental Range (SJER) in Southern California, United States
# * Harvard Forest (HARV) in the Northeastern United States
#
# The data that you will use for this week is available from **earthpy** using the following download:
#
# `et.data.get_data('ndvi-automation')`
#
# ## Assignment Goals
#
# Your goal in this assignment is to create the most efficient and concise workflow that you can that allows for:
#
# 1. The code to scale if you added new sites or more time periods to the analysis.
# 2. Someone else to understand your workflow.
# 3. The LEAST and most efficient (i.e. runs fast, minimize repetition) amount of code that completes the task.
#
# ### HINTS
#
# * Remove values outside of the landsat valid range of values as specified in the metadata, as needed.
# * Keep any output files SEPARATE FROM input files. Outputs should be created in an outputs directory that is created in the code (if needed) and/or tested for.
# * Use the functions that we demonstrated during class to make your workflow more efficient.
# * BONUS - if you  chose - you can export your data as a csv file. You will get bonus points for doing this.
#
#
# ## Assignment Requirements
#
# Your submission to the GitHub repository should include:
# * This Jupyter Notebook file (.ipynb) with:
#     * The code to create a plot of mean NDVI across a year for  2 NEON Field Sites:
#         * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object
#     * The **data should be cleaned to remove the influence of clouds**. See the [earthdatascience website for an example of what your plot might look like with and without removal of clouds](https://www.earthdatascience.org/courses/earth-analytics-python/create-efficient-data-workflows/).
# * BONUS: Create one output `.csv` file that has 3 columns - NDVI, Date and Site Name - with values for SJER and HARV.
#
# Your notebook should:
# * Have *at least* 2 well documented and well named functions with docstrings.
# * Include a Markdown cell at the top of the notebook that outlines the overall workflow using pseudocode (i.e. plain language, not code)
# * Include additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# # Replace this cell with your pseudocode  for this workflow
#
# If you happen to be a diagram person a diagram is ok too
#
#

# ## Pseudocode
# 1. Query input directory and build list of sites (in this case SJER and HARV) from top level directories
# 2. Create placeholder list to store data for each site
# 3. Loop through each site/directory and:
#    - Create empty site specific dataframe with required columns
#    - Open the sites bounding vector as a geodataframe
#    - Build list of available Landsat scenes within the site directory
#    - Loop through the list of Landsat scenes and for each Landsat scene:
#      - Get paths for bands 4 & 5 and for the QA data
#      - Open bands 4 & 5 using the bounding extent and calculate NDVI
#      - Open the QA data and use to mask the NDVI result
#      - Calculate average NDVI value from masked result and populate dataframe with the value
#    - Append the site specific dataframe to the list
# 4. Concatenate the list items into a single dataframe
# 5. Create plots

# In[1]:

# Autograding imports - do not modify this cell
import matplotcheck.autograde as ag
import matplotcheck.notebook as nb
import matplotcheck.timeseries as ts
from datetime import datetime

# In[2]:

# Import needed packages in PEP 8 order
# and no unused imports listed (10 points total)

import os
from glob import glob

import matplotlib.pyplot as plt
import matplotlib as mpl
import pandas as pd
import geopandas as gpd
import numpy as np
import xarray as xr
import rioxarray as rxr
import earthpy as et
import seaborn as sns

# Get data
et.data.get_data('ndvi-automation')

# Set working directory
os.chdir(os.path.join(et.io.HOME,
                      'earth-analytics',
                      'data'))

# Setting consistent plotting style throughout notebook
sns.set_style("white")
sns.set(font_scale=1.5)

# In[3]:

# DO NOT MODIFY THIS CELL
# Tests that the working directory is set to earth-analytics/data

path = os.path.normpath(os.getcwd())
student_wd_parts = path.split(os.sep)

if student_wd_parts[-2:] == ['earth-analytics', 'data']:
    print("\u2705 Great - it looks like your working directory is set correctly to ~/earth-analytics/data")
else:
    print("\u274C Oops, the autograder will not run unless your working directory is set to earth-analytics/data")

# # Figure 1: Plot 1 - Mean NDVI For Each Site Across the Year (50 points)
#
# Create a plot of the mean normalized difference vegetation index (NDVI) for the two different sites in the United States across the year:
#
# * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object.
# * Each site should be identified with a different color in the plot and legend.
# * The final plot **data should be cleaned to remove the influence of clouds**.
# * Be sure to include appropriate title and axes labels.
#
# Add additional cells as needed for processing data (e.g. defining functions, etc), but be sure to:
# * follow the instructions in the code cells that have been provided to ensure that you are able to use the sanity check tests that are provided.
# * include only the plot code in the cell identified for the final plot code below

# ## Task 1:
#
# In the cell below, create a single dataframe containing MEAN NDVI, the site name,
# and the date of the data for the HARV site
# scene `HARV/landsat-crop/LC080130302017031701T1-SC20181023151837`.  The column names for the  final
# DataFrame should be`mean_ndvi`, and `site`, and the data should be **indexed on the date**.
#
# Use the functions that we reviewed in class (or create your own versions of them) to implement your code
#
# ### In the Cell below Place  All Functions Needed to Run this Notebook (20 points)

# In[4]:

### DO NOT REMOVE THIS LINE OR EDIT / MOVE THIS CELL ###
start_time = datetime.now()

# In[5]:

# In this cell place all of the functions needed to run your notebook
# You will be graded here on function application, docstrings, efficiency so ensure
# All functions are placed here!

#  Function to open a Landsat band for a specific geographic bounds
def open_clean_bands(band_path,
                     crop_extent,
                     valid_range=None):
    """Opens a Landsat band and returns data for a specific geographic bounds with the
    option to enforce a valid data range.

    Parameters
    -----------
    band_path : string
        A path to the array to be opened
    crop_extent: geopandas GeoDataFrame
        A geopandas dataframe to be used to crop the raster data using rasterio mask().
    valid_range : tuple (optional)
        A tuple of min and max range of values for the data. Default = None

    Returns
    -----------
    band : xarray DataArray
        An xarray DataArray with the band values
    """
    band = rxr.open_rasterio(band_path, masked=True).rio.clip(crop_extent.geometry,
                                                              from_disk=True).squeeze()
    # Only run this step if a valid range tuple is provided
    if valid_range:
        mask = ((band < valid_range[0]) | (band > valid_range[1]))
        band = band.where(~xr.where(mask, True, False))

    return band

# Function to calculate average NDVI value
def mask_crop_ndvi(all_bands,
                   crop_extent,
                   pixel_qa_path,
                   vals):
    """Calculates an average NDVI value from Landsat bands 4 and 5 for a specific geographic
    extent using the Landsat cloud mask to remove anomalous values.

    Parameters
    -----------
    all_bands : list
        A list containing two xarray objects for landsat bands 4 and  5
    crop_extent: geopandas GeoDataFrame
        A geopandas dataframe to be used to crop the raster data using rasterio mask().
    pixel_qa_path: string
        A path to a pixel qa tif file.
    vals: list
        A list of values needed to create the cloud mask

    Returns
    -----------
    ndvi_mean : float
        Mean NDVI value
    """

    # crop_json = crop_bound.geometry

    # Open and clip qa layer
    # pixel_qa = rxr.open_rasterio(pixel_qa_path[0], masked=True).rio.clip(crop_json,
    #                                                            from_disk = True).squeeze()

    # Open and clip qa layer
    pixel_qa = rxr.open_rasterio(pixel_qa_path[0], masked=True).rio.clip(crop_extent.geometry,
                                                                         from_disk=True).squeeze()

    # Calculate NDVI
    ndvi_xr = (all_bands[1]-all_bands[0]) / (all_bands[1]+all_bands[0])

    # Apply cloud mask to NDVI
    ndvi_mask = ndvi_xr.where(~pixel_qa.isin(vals))

    # Calculate mean NDVI value
    ndvi_mean = np.nanmean(ndvi_mask)

    return ndvi_mean

# ## Script to calculate mean NDVI for a single Landsat 8 scene
# The script below will calculate the average NDVI value from Bands 4 and 5 of Landsat 8 data for a single scene.

# In[6]:

# Create dataframe of mean NDVI in this cell using the functions created above
# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Call the dataframe at the end of the cell so the tests run on it!
# Be sure that the date column is an index of type date
# HINT: the time series lessons may help you remember how to do this!

# Define site name
site_name = "HARV"

# Define site path
site = os.path.join("ndvi-automation", "sites", site_name)

# Geographic bounds
# Open up the shapefile for clipping your landsat data to the study area
vector_dir = os.path.join(site, "vector")

# Open crop boundary
site_boundary_path = os.path.join(vector_dir,  site_name + "-crop.shp")
crop_extent = gpd.read_file(site_boundary_path)

# Landsat bands
# Select just a single directory and grab bands 4-5 from the directory
adir = os.path.join(site, "landsat-crop",
                    "LC080130302017031701T1-SC20181023151837")

# Define bands paths
band_paths = sorted(glob(os.path.join(adir, "*band*[4-5].tif")))

# Open the bands and append into Xarray
all_bands = []
for aband in band_paths:
    print("Opening up", aband)
    cleaned_band = open_clean_bands(band_path=aband,
                                    crop_extent=crop_extent,
                                    valid_range=(0, 10000))
    all_bands.append(cleaned_band)

# QA dataset
# Cloud no data vals for Landsat 8 -
vals = [328, 392, 840, 904, 1350, 352, 368, 416,
        432, 480, 864, 880, 928, 944, 992, 480, 992]

# Open cloud mask layer
pixel_qa_path = glob(os.path.join(adir, "*qa*"))

# Calculate mean NDVI
site_NDVI = mask_crop_ndvi(all_bands=all_bands,
                           crop_extent=crop_extent,
                           pixel_qa_path=pixel_qa_path,
                           vals=vals)

# Extract the date from the Landsat directory name
path_components = adir.split(os.sep)
dir_name = path_components[4]
date = dir_name[10:18]
date

# Create the summary dataframe
ndvi_list = []
ndvi_list.append([site_name, date, site_NDVI])
ndvi_df = pd.DataFrame(ndvi_list,
                       columns=["site", "date", "mean_ndvi"])

# convert the 'Date' column to datetime format
ndvi_df['date']= pd.to_datetime(ndvi_df['date'])
ndvi_df.set_index("date", inplace=True)
ndvi_df

# In[7]:

# This cell  is testing your data output above

student_ndvi_ts_single_site = _

single_scene_points = 0

# Ensure the data is stored in a dataframe.
if isinstance(student_ndvi_ts_single_site, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    single_scene_points += 1
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Ensure that the date column is the index
if isinstance(student_ndvi_ts_single_site.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    single_scene_points += 2
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_ts_single_site.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    single_scene_points += 2
else:
    print('\u274C The data in your date column is not datetime.')

# Ensure the site name is correct
if student_ndvi_ts_single_site.site.values[0] == 'HARV':
    print('\u2705 You have the correct site name!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct site name.')

if np.allclose(0.281131628228094, student_ndvi_ts_single_site.mean_ndvi.values[0]):
    print('\u2705 You have the correct mean NDVI value!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct mean ndvi value.')

print("\n \u27A1 You received {} out of 15 points for creating a dataframe.".format(
    single_scene_points))
single_scene_points

# ## Task 2:
#
# In the cell below, process all of the landsat scenes. Create a DataFrame that contains the following
# information for each scene
#
#
# |   | index  | site  | mean_ndvi  |
# |---|---|---|---|
# | Date  |   |   |   |
# |  2017-01-07  | 0  | SJER  | .4  |
#
# Be sure to call your dataframe at the end of the cell to ensure autograding works.
# HINT: FOR THIS STEP, leave any rows containing missing values (`NAN`).

# ## Script to calculate average NDVI values from multiple NEON sites and Landsat 8 scenes
# This script searches through a specified directory for NEON sites and then calculates the average NDVI values for Landsat 8 datasets associated with these sites. The Landsat 8 data includes acquisitions for the year 2017.

# In[8]:

# Create dataframe of NDVI including the cleaning data to deal with clouds

# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Don't forget to set date as the index and make the values of type datetime

# Get list of sites
all_sites = glob(os.path.join("ndvi-automation", "sites", "*/"))

# Define the directory name where the Landsat directories are stored
landsat_dir = "landsat-crop"

# Define cloud no data vals for Landsat 8 -
vals = [328, 392, 840, 904, 1350, 352, 368, 416,
        432, 480, 864, 880, 928, 944, 992, 480, 992]

# Placeholder list to store the NDVI data
ndvi_list = []

# Loop through the sites
for site in all_sites:

    # Get the site name
    path_components = site.split(os.sep)
    site_name = path_components[2]

    # Open up the site extents shapefile to clip Landsat data
    vector_dir = os.path.join(site, "vector")

    # Open crop boundary
    site_boundary_path = os.path.join(vector_dir,  site_name + "-crop.shp")
    crop_extent = gpd.read_file(site_boundary_path)

    # Get a list of the Landsat sub-directories
    new_path = os.path.join(site, landsat_dir)
    all_dirs = glob(new_path + "/*/")

    #  Loop through  each Landsat subdirectory
    for adir in all_dirs:

        # Define Landsat bands paths
        band_paths = sorted(glob(os.path.join(adir, "*band*[4-5].tif")))

        # Open the bands and append into Xarray
        all_bands = []
        for aband in band_paths:
            cleaned_band = open_clean_bands(band_path=aband,
                                            crop_extent=crop_extent,
                                            valid_range=(0, 10000))
            all_bands.append(cleaned_band)

        # Open cloud mask layer
        pixel_qa_path = glob(os.path.join(adir, "*qa*"))

        # Calculate mean NDVI
        site_NDVI = mask_crop_ndvi(all_bands=all_bands,
                                   crop_extent=crop_extent,
                                   pixel_qa_path=pixel_qa_path,
                                   vals=vals)

        # Extract the date from the Landsat directory name
        path_components = adir.split(os.sep)
        dir_name = path_components[4]
        date = dir_name[10:18]

        # Append values to the list
        ndvi_list.append([site_name, date, site_NDVI])

# Format the NDVI data
ndvi_df = pd.DataFrame(ndvi_list,
                       columns=["site", "date", "mean_ndvi"])

# Convert the 'Date' column to datetime format
ndvi_df['date']= pd.to_datetime(ndvi_df['date'])
ndvi_df.set_index("date", inplace=True)

# Export the df as a CSV file
output_path = os.path.join("ndvi-automation", "landsat_ndvi.csv")
ndvi_df.to_csv(output_path, index=True, header=True)

ndvi_df

# In[9]:

# Last sanity check before creating your plot (10 points)

# Ensure that you call your dataframe at the bottom of the cell above
# and that it has columns called: mean_ndvi and site

# Ensure the data is stored in a dataframe.
student_ndvi_df = _

df_points = 0

if isinstance(student_ndvi_df, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    df_points +=2
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Check that dataframe contains the appropriate number of NAN values
if student_ndvi_df.mean_ndvi.isna().sum() == 15:
    print('\u2705 Correct number of masked data values!')
    df_points +=2
else:
    print('\u274C The amount of null data in your dataframe is incorrect.')

# Ensure that the date column is the index
if isinstance(student_ndvi_df.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    df_points +=3
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_df.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    df_points +=3
else:
    print('\u274C The data in your date column is not datetime.')

# Output for timer, # DO NOT MODIFY
end_time = datetime.now()
total_time = end_time - start_time
print(
    "Your total run time for processing the data was {0}.".format(total_time))

print("\n \u27A1 You received {} out of 10 points for creating a dataframe.".format(
    df_points))

df_points

# In[10]:

# Add only the plot code to this cell

# This is the final figure of mean NDVI
# for both sites across the year
# with data cleaned to deal with clouds

# Remove NaN rows that represent cloud covered scenes
ndvi_df = ndvi_df.dropna()

fig, ax = plt.subplots(figsize=(15, 15))

sites = ndvi_df.site.unique()
ndvi_df.reset_index().groupby('site').plot(x='date', y='mean_ndvi', ax=ax)
plt.legend(sites)

ax.set(title="Mean NDVI January-December 2017 calculated from Landsat 8 data with clouds masked",
        xlabel="Date",
        ylabel="NDVI")

#plt.show()

### DO NOT REMOVE LINES BELOW ###
final_masked_solution = nb.convert_axes(plt, which_axes="current")

# In[11]:

# Ignore this cell for the autograding tests

# In[12]:

# Ignore this cell for the autograding tests

# # Question 1 (10 points)
#
# Imagine that you are planning NEON’s upcoming flight season to capture remote sensing data in these locations and want to ensure that you fly the area when the vegetation is the most green.
#
# When would you recommend the flights take place for each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# I would recommend for the HARV site that the collection take place towards the end of June and for the SJER site that this take places towards the end of April.

# # Question 2 (10 points)
#
# How could you modify your workflow to look at vegetation changes over time in each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# You could look at vegetation change over multiple years for each site by downloading further datasets. For each site you could overlay each years NDVI trends (perhaps by grouping by year) to inspect the variability in NDVI temporal signatures from year to year. This would allow you to more robustly identify the periods of peak greeness.

# # Do not edit this cell! (10 points)
#
# The notebook includes:
# * additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# # Do not edit this cell! (20 points)
#
# The notebook will also be checked for overall clean code requirements as specified at the **top** of this notebook. Some of these requirements include (review the top cells for more specifics):
#
# * Notebook begins at cell [1] and runs on any machine in its entirety.
# * PEP 8 format is applied throughout (including lengths of comment and code lines).
# * No additional code or imports in the notebook that is not needed for the workflow.
# * Notebook is fully reproducible. This means:
#    * reproducible paths using the os module.
#    * data downloaded using code in the notebook.
#    * all imports at top of notebook.

# ## BONUS - Export a  .CSV File to Share (10 points possible)
#
# This is optional - if you export a **.csv** file with the columns specified above: Site, Date and NDVI Value you can get an additional 10 points.
#
# * FULL CREDIT: File exists in csv format and contains the columns specified.
# We will check your github repo for this file!
#


import numpy as np
a=np.arange(10)
print(a)

#save numpy array
np.save("saved",a)
#new file is craeted with name saved.npy

new_a=np.load("saved.npy")  #load the saved file
print(new_a)

#saving multiple arrays as zip or archive file
a1=np.arange(25)
a2=np.arange(30)
np.savez("saved_archive.npz",x=a1,y=a2) #savez is used for saving multiple arrays
load_npz=np.load("saved_archive.npz")
print(load_npz['x'])
print(load_npz['y'])

#save the arrays to textfile

np.savetxt('text.txt',a1,delimiter=',')
#loading of txt files

load_txt=np.loadtxt('text.txt',delimiter=',')
print("Text File=",load_txt)
#convets an integrer into a float number


#!/usr/bin/env python3

# we will be using nltk.lm and numpy
#%% Imports
import random
from typing import Dict, List, Tuple

from nltk import everygrams, text
from nltk.probability import FreqDist
from numpy import lookfor
import pandas as pd

from nltk.lm import MLE
from nltk.lm.preprocessing import padded_everygram_pipeline

random_state = 42

# 0. Before you get started, make sure to download the `theses.txt` data set.
#%%
def get_data() -> List :
    f = open('../res/theses.txt', 'r')
    lines = f.readlines()
    f.close()

    for idx, line in enumerate(lines):
        lines[idx] = line.replace("\n", "")
    return lines

# 1. Spend some time on pre-processing. How would you handle hyphenated words
#    and abbreviations/acronyms?
#%% Preprocessing
def preprocess(data: List[str]):

    # remove empty entries
    sentences = data.remove("")

    # Remove duplicates
    sentences = list(dict.fromkeys(data))

    # Make every character lowercase
    for idx, sentence in enumerate(sentences):
        sentences[idx] = sentence.lower()

    return sentences

def tokenize(data: str) -> List[str]:
    return data.split()

# 2. Train n-gram models with n = [1, ..., 5]. What about <s> and </s>?
#%%
def get_trained_model(data: List[List[str]], n : int):
    train, vocab = padded_everygram_pipeline(n, data)
    model = MLE(n)
    model.fit(train, vocab)
    return model

def get_everygrams(data: List[List[str]], n : int):
    grams = []
    for idx, sentence in enumerate(data):

        gram = everygrams(
            sentence,
            min_len=2,
            max_len=n,
            pad_left=True,
            pad_right=True,
            left_pad_symbol="<s>",
            right_pad_symbol="</s>")

        grams.append(gram)
    return grams

# 3. Write a generator that provides thesis titles of desired length. Please
#    do not use the available `lm.generate` method but write your own.
#    nb: If you fix the seed in numpy.random.choice, you get reproducible
#        results.
# 3.1 How can you incorporate seed words?
# 3.2 How do you handle </s> tokens (w.r.t. the desired length?)

def generate_thesis(ngrams: List[List[str]], text_seed : Tuple = ("<s>",), length: int = 10):
    lookup = generate_lookup(ngrams)
    current_token = text_seed
    n = 3

    def get_next(token : Tuple):
        values = lookup[token]

        choices = []
        counts =  []

        for choice in values.keys():
            choices.append(choice)
            counts.append(values[choice])

        word = random.choices(choices, weights=counts, k=1)
        return word[0]

    title : List = []
    counter = 0
    while counter <= length:
        w = get_next(current_token)
        if w == "<s>":
            continue

        if w == "</s>":
            break

        title.append(w)
        current_token = (w,) # TODO: this currently is always a bigram
    return title

def generate_lookup(ngrams: List[List[str]]):
    fdist = FreqDist()

    for entry in ngrams:
        fdist.update(list(entry))

    lookup = {}
    for ngram in fdist:
        key =  ngram[:-1]
        word = ngram[-1]
        count = fdist[ngram]

        if key not in lookup:
            lookup[key] = {}

        lookup[key][word] = count
    return lookup

# 3.3 If you didn't just copy what nltk's lm.generate does: compare the
#     outputs

# %%

data = get_data()
preprocessed = preprocess(data)

tokenized = []
for idx in range(0, len(preprocessed)):
    tokenized.append(tokenize(preprocessed[idx]))

for i in range(0, 6):
    model = get_trained_model(tokenized, 3)
    ngrams = get_everygrams(tokenized, 3)

    print(f"########## {i} ############ ")
    print(model.generate(20, text_seed=["<s>"]))
    print(generate_thesis(ngrams, text_seed=("<s>",), length=20))

# %%

# %%


#from math import
import numpy as np

x0 = np.array(zeros)
n = input('Input length of solution vector: ')

for i in range(1:n):
    x0[i] = float(input('Insert xi coord. of x0 vector: '))
print(x0)

f1 = lambda x: (6 - (-2)*[])/5
f2 =
f3 =
f4 =

x1 =
x2 =
x3 =
x4 =