#!/usr/bin/env python
# coding: utf-8

#
#
# # Earth Analytics Education - EA  Python Course Spring 2021

# ## Important  - Assignment Guidelines
#
# 1. Before you submit your assignment to GitHub, make sure to run the entire notebook with a fresh kernel. To do this first, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart & Run All)
# 2. Always replace the `raise NotImplementedError()` code with your code that addresses the activity challenge. If you don't replace that code, your notebook will not run.
#
# ```
# # YOUR CODE HERE
# raise NotImplementedError()
# ```
#
# 3. Any open ended questions will have a "YOUR ANSWER HERE" within a markdown cell. Replace that text with your answer also formatted using Markdown.
# 4. **DO NOT RENAME THIS NOTEBOOK File!** If the file name changes, the autograder will not grade your assignment properly.
# 6. When you create a figure, comment out `plt.show()` to ensure the autograder can grade your plots. For figure cells, DO NOT DELETE the code that says `DO NOT REMOVE LINE BELOW`.
#
# ```
# ### DO NOT REMOVE LINE BELOW ###
# student_plot1_ax = nb.convert_axes(plt)
# ```
#
# * Only include the package imports, code, and outputs that are required to run your homework assignment.
# * Be sure that your code can be run on any operating system. This means that:
#    1. the data should be downloaded in the notebook to ensure it's reproducible
#    2. all paths should be created dynamically using the `os.path.join`
#
# ## Follow to PEP 8 Syntax Guidelines & Documentation
#
# * Run the `autopep8` tool on all cells prior to submitting (HINT: hit shift + the tool to run it on all cells at once!
# * Use clear and expressive names for variables.
# * Organize your code to support readability.
# * Check for code line length
# * Use comments and white space sparingly where it is needed
# * Make sure all python imports are at the top of your notebook and follow PEP 8 order conventions
# * Spell check your Notebook before submitting it.
#
# For all of the plots below, be sure to do the following:
#
# * Make sure each plot has a clear TITLE and, where appropriate, label the x and y axes. Be sure to include UNITS in your labels.
#

# ### Add Your Name Below
# **Your Name: Svetlana Kurakina**

#

# ---

# # Week 04 and 05 Homework - Automate NDVI Workflow
#
# For this assignment, you will write code to generate a plot of the mean normalized difference vegetation index (NDVI) for two different sites in the United States across one year of data:
#
# * San Joaquin Experimental Range (SJER) in Southern California, United States
# * Harvard Forest (HARV) in the Northeastern United States
#
# The data that you will use for this week is available from **earthpy** using the following download:
#
# `et.data.get_data('ndvi-automation')`
#
# ## Assignment Goals
#
# Your goal in this assignment is to create the most efficient and concise workflow that you can that allows for:
#
# 1. The code to scale if you added new sites or more time periods to the analysis.
# 2. Someone else to understand your workflow.
# 3. The LEAST and most efficient (i.e. runs fast, minimize repetition) amount of code that completes the task.
#
# ### HINTS
#
# * Remove values outside of the landsat valid range of values as specified in the metadata, as needed.
# * Keep any output files SEPARATE FROM input files. Outputs should be created in an outputs directory that is created in the code (if needed) and/or tested for.
# * Use the functions that we demonstrated during class to make your workflow more efficient.
# * BONUS - if you  chose - you can export your data as a csv file. You will get bonus points for doing this.
#
#
# ## Assignment Requirements
#
# Your submission to the GitHub repository should include:
# * This Jupyter Notebook file (.ipynb) with:
#     * The code to create a plot of mean NDVI across a year for  2 NEON Field Sites:
#         * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object
#     * The **data should be cleaned to remove the influence of clouds**. See the [earthdatascience website for an example of what your plot might look like with and without removal of clouds](https://www.earthdatascience.org/courses/earth-analytics-python/create-efficient-data-workflows/).
# * BONUS: Create one output `.csv` file that has 3 columns - NDVI, Date and Site Name - with values for SJER and HARV.
#
# Your notebook should:
# * Have *at least* 2 well documented and well named functions with docstrings.
# * Include a Markdown cell at the top of the notebook that outlines the overall workflow using pseudocode (i.e. plain language, not code)
# * Include additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# # Replace this cell with your pseudocode  for this workflow
#
# If you happen to be a diagram person a diagram is ok too
#
#

# # Pseudocode for this workflow
# * Import libraries necessary to complete the workflow
# * Set home directory
# * Set paths to data
# * Specify cloud values using earthpy (need to do once)
# * Read study sites boundaries (need to do once for each site)
# * Read landsat bands and qa layers
# * Restrict landsat data to "valid range" of 0 to 10000
# * Clip extent of bands to match study sites boundaries
# * Clip extent of qa layers to match study sites boundaries
# * Calculate NDVI for each pair of red/infrared bands for each landsar scene
# * Apply cloud mask to resulting NDVI
# * Calculate Mean NDVI and append those to a list
# * Get site names from paths and append those to a list
# * Get scene dates from from paths and append those to a list
# * Create dataframe and populate  with lists of site names, dates, mean NDVIs
# * Set dataframe index to datetime
# * Drop NA data
# * Plot data
# * Export data to CSV

# In[1]:

# Autograding imports - do not modify this cell
import matplotcheck.autograde as ag
import matplotcheck.notebook as nb
import matplotcheck.timeseries as ts
from datetime import datetime

# In[2]:

# Import needed packages in PEP 8 order
# and no unused imports listed (10 points total)

# YOUR CODE HERE
import os
from glob import glob

import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter
import matplotlib.dates as mdates
import numpy as np
import pandas as  pd
import geopandas as gpd
import xarray as xr
import rioxarray as rxr
from rasterio.plot import plotting_extent
import earthpy as et
import earthpy.mask as em
import earthpy.spatial as es
import earthpy.plot as ep

# Download data and set working directory
data = et.data.get_data('ndvi-automation')
os.chdir(os.path.join(et.io.HOME,
                      'earth-analytics',
                      'data'))

# In[3]:

# DO NOT MODIFY THIS CELL
# Tests that the working directory is set to earth-analytics/data

path = os.path.normpath(os.getcwd())
student_wd_parts = path.split(os.sep)

if student_wd_parts[-2:] == ['earth-analytics', 'data']:
    print("\u2705 Great - it looks like your working directory is set correctly to ~/earth-analytics/data")
else:
    print("\u274C Oops, the autograder will not run unless your working directory is set to earth-analytics/data")

# # Figure 1: Plot 1 - Mean NDVI For Each Site Across the Year (50 points)
#
# Create a plot of the mean normalized difference vegetation index (NDVI) for the two different sites in the United States across the year:
#
# * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object.
# * Each site should be identified with a different color in the plot and legend.
# * The final plot **data should be cleaned to remove the influence of clouds**.
# * Be sure to include appropriate title and axes labels.
#
# Add additional cells as needed for processing data (e.g. defining functions, etc), but be sure to:
# * follow the instructions in the code cells that have been provided to ensure that you are able to use the sanity check tests that are provided.
# * include only the plot code in the cell identified for the final plot code below

# # MEAN NDVI for one HARV landsat scene without functions (below)
# * I want to be sure I understand all the transformations and calculations involved in the process
# * To achieve that, I start from processing all data for one HARV landsat scene without using any functions
# * I check my resulting dataframe using autograder below to make sure I am getting correct result
# * After that, I adjust my pseudocode and start working on functions
# * Once I get my functions I will use them to calculate df for single HARV landsat scene again
# * Once I am sure my functions work as intended, I will comment out code cell below, but I leave it for future reference

# In[4]:

# # Calculation Mean NDVI for one HARV landsat scene without functions

# # Set paths
# path_harv = os.path.join('ndvi-automation', 'sites', 'HARV')
# path_harv_scene = os.path.join(path_harv,'landsat-crop',
#                                'LC080130302017031701T1-SC20181023151837')
# path_harv_bands = sorted(glob(os.path.join(path_harv_scene,
#                                            '*band*[4-5].tif')))
# path_harv_crop_bound = os.path.join(path_harv, 'vector', 'HARV-crop.shp')
# path_harv_qa = os.path.join(
#     path_harv_scene,
#     'LC08_L1TP_013030_20170317_20170328_01_T1_pixel_qa.tif'
# )

# # Open crop boundary
# harv_crop_bound = gpd.read_file(path_harv_crop_bound)

# # Open band 4 using rioxarray
# harv_band_4 = rxr.open_rasterio(path_harv_bands[0], masked=True).squeeze()
# #harv_band_4.plot()

# # Clip band 4
# harv_band_4_crop = harv_band_4.rio.clip(harv_crop_bound.geometry)

# # Specify the valid range
# valid_range = (0, 10000)

# if valid_range:
#     mask = ((
#         harv_band_4_crop < valid_range[0]) | (
#         harv_band_4_crop > valid_range[1]))
#     harv_band_4_crop = harv_band_4_crop.where(~xr.where(mask, True, False))
#harv_band_4_crop.plot()

# # Open band 5 using rioxarray
# harv_band_5 = rxr.open_rasterio(path_harv_bands[1], masked=True).squeeze()

# # Clip band 5
# harv_band_5_crop = harv_band_5.rio.clip(harv_crop_bound.geometry)

# # Specify the valid range for band 5

# if valid_range:
#     mask = ((
#         harv_band_5_crop < valid_range[0]) | (
#         harv_band_5_crop > valid_range[1]))
#     harv_band_5_crop = harv_band_5_crop.where(~xr.where(mask, True, False))

# #harv_band_5_crop.plot()

# # Calculate NDVI (since I used cropped bands it is already cropped)
# harv_ndvi = (harv_band_5_crop-harv_band_4_crop)/(
#     harv_band_5_crop+harv_band_4_crop)

# # Grab cloud values using earthpy
# high_cloud_confidence = (
#     em.pixel_flags["pixel_qa"]["L8"]["High Cloud Confidence"])
# cloud = em.pixel_flags["pixel_qa"]["L8"]["Cloud"]
# cloud_shadow = em.pixel_flags["pixel_qa"]["L8"]["Cloud Shadow"]
# all_masked_values = cloud_shadow + cloud + high_cloud_confidence

# # Open and crop harv qa layer
# harv_qa = rxr.open_rasterio(path_harv_qa).squeeze()
# harv_qa_cropped = harv_qa.rio.clip(harv_crop_bound.geometry)

# Create and apply cloud mask to NDVI
# harv_ndvi_cl_free = harv_ndvi.where(
#     ~harv_qa_cropped.isin(all_masked_values))
#harv_ndvi_cl_free.plot()

# # Calculate mean NDVI
# print('My mean NDVI for HARV site is:', harv_ndvi_cl_free.mean().values)

# #Get site name from path
# harv_sitename = [os.path.basename(os.path.normpath(path_harv))]

# # Get date
# harv_date = [os.path.basename(os.path.normpath(path_harv_scene))[10:18]]

# # Get NDVI value
# harv_mean = [harv_ndvi_cl_free.mean().values]

# # Construct dataframe
# harv_df = pd.DataFrame(columns=["site","date","mean_ndvi"])
# harv_df['site'] = harv_sitename
# harv_df['date'] = harv_date
# harv_df['mean_ndvi'] = harv_mean
# harv_df['date'] = pd.to_datetime(harv_df['date'])
# harv_df.set_index('date', inplace=True)
# harv_df

# ## Task 1:
#
# In the cell below, create a single dataframe containing MEAN NDVI, the site name,
# and the date of the data for the HARV site
# scene `HARV/landsat-crop/LC080130302017031701T1-SC20181023151837`.  The column names for the  final
# DataFrame should be`mean_ndvi`, and `site`, and the data should be **indexed on the date**.
#
# Use the functions that we reviewed in class (or create your own versions of them) to implement your code
#
# ### In the Cell below Place  All Functions Needed to Run this Notebook (20 points)

# In[5]:

### DO NOT REMOVE THIS LINE OR EDIT / MOVE THIS CELL ###
start_time = datetime.now()

# In[6]:

# In this cell place all of the functions needed to run your notebook
# You will be graded here on function application, docstrings, efficiency so ensure
# All functions are placed here!

# YOUR CODE HERE

# Define function which opens, crops, and cleans landsat bands
def open_clean_bands(band_path,
                     crop_extent,
                     valid_range=None,):
    """Open, crop and define valid range of values for single ladsat band.

    Parameters
    -----------
    band_path : string
        A path to the array to be opened
    valid_range : tuple (optional)
        A tuple of min and max range of values for the data. Default = None

    Returns
    -----------
    arr : xarray DataArray
        An xarray DataArray with values in a valid range.
    """

    # Open and crop bands
    band = rxr.open_rasterio(
        band_path, masked=True).rio.clip(
        crop_extent.geometry, from_disk=True).squeeze()

    # Only run this step if a valid range tuple is provided
    if valid_range:
        mask = ((band < valid_range[0]) | (band > valid_range[1]))
        band = band.where(~xr.where(mask, True, False))

    return band

# Define function that calculates NDVI and masks clouds using cropped qa layer
def mask_crop_ndvi(all_bands,
                   crop_bound,
                   pixel_qa_path,
                   vals):
    """Takes 2 landsat bands, calc. NDVI, masks clouds, gets NDVI mean value.

    Parameters
    -----------
    all_bands : list
        A list containing two xarray objects for landsat bands 4 and 5
    crop_bound: geopandas GeoDataFrame
        A geopandas dataframe to be used to crop
        a pixel qa tif using rasterio mask().
    pixel_qa_path: string
        A path to a pixel qa tif file.
    vals: list
        A list of values needed to create the cloud mask.

    Returns
    -----------
    ndvi_mean : array
        an array object containing mean NDVI value
    """

    crop_json = crop_bound.geometry

    # Open and clip qa layer
    pixel_qa = rxr.open_rasterio(
        pixel_qa_path[0], masked=True).rio.clip(
        crop_json, from_disk=True).squeeze()

    # Calculate NDVI
    ndvi_xr = (all_bands[1]-all_bands[0]) / (all_bands[1]+all_bands[0])

    # Apply cloud mask to NDVI
    ndvi_mask = ndvi_xr.where(~pixel_qa.isin(vals))

    # Calculate mean NDVI
    ndvi_mean = ndvi_mask.mean().values

    return ndvi_mean

# Define function that constructs df with Mean NDVI value for a landsat scene
# Might not be that useful if needs to be scaled up
# Served as an exploratory exercise for an author of this notebook
def construct_df(site_name,
                 scene_path,
                 ndvi_mean):
    """Contructs df indexed on date with NDVI value for a landsat scene.

    Parameters
    -----------
    site_name : string
        A string with the name of the study site
    scene_path: string
        A string with the path to landsat scene directory.
    ndvi_mean: array
        An array with NDVI mean value.

    Returns
    -----------
    ndvi_mean_df : dataframe
        a pandas Data Frame with date, site name and mean NDVI value
    """

    #Prep site name
    sitename = [site_name]

    # Get date
    date = [os.path.basename(scene_path)[10:18]]

    # Get NDVI value
    mean = [ndvi_mean]

    # Construct dataframe
    ndvi_mean_df = pd.DataFrame(columns = ["site","date","mean_ndvi"])
    ndvi_mean_df['site'] = sitename
    ndvi_mean_df['date'] = date
    ndvi_mean_df['mean_ndvi'] = mean
    ndvi_mean_df['date'] = pd.to_datetime(ndvi_mean_df['date'])
    ndvi_mean_df.set_index('date', inplace=True)

    return ndvi_mean_df

# Define function that uses open_clean_bands and mask_crop_ndvi
# Function is suitable to use with multiple landsat scenes
def mean_ndvi(scene_path,
              crop_bound):
    """Opens landsat bands, crops and cleans data, calculates mean NDVI.

    Parameters
    -----------
    scene_path : string
        A string with the path of the landsat scene directory.
    crop_extent : crop_bound: geopandas GeoDataFrame
        A geopandas dataframe to be used to crop
        the raster data using rasterio mask().

    Returns
    -----------
    ndvi : array
        An array with NDVI mean value.
    """

    # Construct paths
    band_paths = glob(os.path.join(scene_path, '*band[4-5].tif'))
    qa_paths = glob(os.path.join(scene_path, '*qa.tif'))

    # Run open_clean_bands function
    bands = []
    for aband in band_paths:
        clean_band = open_clean_bands(aband, crop_bound, (0, 10000))
        bands.append(clean_band)

    # Grab cloud values using earthpy
    high_cloud_confidence = (
    em.pixel_flags["pixel_qa"]["L8"]["High Cloud Confidence"])
    cloud = em.pixel_flags["pixel_qa"]["L8"]["Cloud"]
    cloud_shadow = em.pixel_flags["pixel_qa"]["L8"]["Cloud Shadow"]
    all_masked_values = cloud_shadow + cloud + high_cloud_confidence

    # Run mask_crop_ndvi function
    ndvi = mask_crop_ndvi(bands, crop_bound, qa_paths, all_masked_values)

    return ndvi

# In[7]:

# Create dataframe of mean NDVI in this cell using the functions created above
# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Call the dataframe at the end of the cell so the tests run on it!
# Be sure that the date column is an index of type date
# HINT: the time series lessons may help you remember how to do this!

# YOUR CODE HERE

# Get a list of each directory
path = os.path.join("ndvi-automation", "sites")
# print (path)

# Get a list of both site directories
# Added sorted to make sure sites come in the same order on all machines
sites = sorted(glob(path + "/*/"))
# print(sites)

# Get the HARV site name
site_name = os.path.basename(os.path.normpath(sites[0]))
# site_name

# Set path to directory with HARV site boundary
vector_dir = os.path.join(sites[0],
                          "vector")
# print(vector_dir)

# Set path to HARV site boundary
site_boundary_path = os.path.join(vector_dir,  site_name + "-crop.shp")
# print(site_boundary_path)

# Open crop boundary
crop_bound = gpd.read_file(site_boundary_path)
# crop_bound.plot()

# Set path to umbrella directory with HARV landsat data
landsat_dir = os.path.join(sites[0],
                           "landsat-crop")
# print(landsat_dir)

# Set path to HARV landsat scenes
landsat_dirs = sorted(glob(os.path.join(landsat_dir, "LC08*")))
# print(landsat_dirs)

# Set path to scene LC080130302017031701T1-SC20181023151837
adir = landsat_dirs[4]
#print(adir)

# Set paths to bands 4-5 tifs
band_paths = sorted(glob(os.path.join(adir, "*band*[4-5].tif")))
# print(band_paths)

# Use function to open, crop and clean bands 4 and 5
all_bands = []
for aband in band_paths:
#     print("Opening up", aband)
    cleaned_band = open_clean_bands(band_path=aband,
                                    crop_extent=crop_bound,
                                    valid_range=(0, 10000))
    all_bands.append(cleaned_band)

# Grab cloud values using earthpy
high_cloud_confidence = (
    em.pixel_flags["pixel_qa"]["L8"]["High Cloud Confidence"])
cloud = em.pixel_flags["pixel_qa"]["L8"]["Cloud"]
cloud_shadow = em.pixel_flags["pixel_qa"]["L8"]["Cloud Shadow"]
all_masked_values = cloud_shadow + cloud + high_cloud_confidence

# Set path to qa tif
pixel_qa_path = glob(os.path.join(adir, "*qa*"))
# print(pixel_qa_path)

# Use function to calculate NDVI, and mask clouds with cropped qa layer
ndvi_harv_mean = mask_crop_ndvi(all_bands=all_bands,
                            crop_bound=crop_bound,
                            pixel_qa_path=pixel_qa_path,
                            vals=all_masked_values)

# Use function to get dataframe
single_scene_df = construct_df(site_name=site_name,
                         scene_path=adir,
                         ndvi_mean=ndvi_harv_mean)

single_scene_df

# In[8]:

# This cell  is testing your data output above

student_ndvi_ts_single_site = _

single_scene_points = 0

# Ensure the data is stored in a dataframe.
if isinstance(student_ndvi_ts_single_site, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    single_scene_points += 1
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Ensure that the date column is the index
if isinstance(student_ndvi_ts_single_site.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    single_scene_points += 2
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_ts_single_site.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    single_scene_points += 2
else:
    print('\u274C The data in your date column is not datetime.')

# Ensure the site name is correct
if student_ndvi_ts_single_site.site.values[0] == 'HARV':
    print('\u2705 You have the correct site name!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct site name.')

if np.allclose(0.281131628228094, student_ndvi_ts_single_site.mean_ndvi.values[0]):
    print('\u2705 You have the correct mean NDVI value!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct mean ndvi value.')

print("\n \u27A1 You received {} out of 15 points for creating a dataframe.".format(
    single_scene_points))
single_scene_points

# ## Task 2:
#
# In the cell below, process all of the landsat scenes. Create a DataFrame that contains the following
# information for each scene
#
#
# |   | index  | site  | mean_ndvi  |
# |---|---|---|---|
# | Date  |   |   |   |
# |  2017-01-07  | 0  | SJER  | .4  |
#
# Be sure to call your dataframe at the end of the cell to ensure autograding works.
# HINT: FOR THIS STEP, leave any rows containing missing values (`NAN`).

# In[9]:

# Create dataframe of NDVI including the cleaning data to deal with clouds

# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Don't forget to set date as the index and make the values of type datetime

# YOUR CODE HERE

# Get list of study site directories
harv_sjer_sites = sorted(glob(path + "/*/"))

# Construct NDVI list
all_ndvi = []

# Loop through sites and read two crop_bounds
for asite in harv_sjer_sites:
    sitename = os.path.basename(os.path.normpath(asite))
    site_boundary_path = os.path.join(
        asite,
        'vector',
        sitename +
        '-crop.shp'
    )
    crop_bound = gpd.read_file(site_boundary_path)
    # Get a list of all landsat scenes
    scene_dirs = sorted(glob(os.path.join(
        path,
        sitename,
        'landsat-crop',
        '*'
    )))

    # Loop though all landsat scenes
    for ascene in scene_dirs:
        # get date for each scene
        date = ascene[-29:-21]
        # run function which reads and cleans all bands and
        # calculates NDVI for each pait of red/infrared bands
        # belonging to the same scene
        ndvi = mean_ndvi(ascene, crop_bound)
        # append site names, dates and mean ndvi to a list
        all_ndvi.append([sitename, date, ndvi])

# Construct data frame using list above
ndvi = pd.DataFrame(all_ndvi,
                    columns = ['site', 'date', 'mean_ndvi'])

# Convert string with date to a datetime object
ndvi['date'] = pd.to_datetime(ndvi['date'])

# Set dataframe index to a datetime
ndvi.set_index('date', inplace=True)

# Remove invalid ndvi values (i.e. remove rows with no values)
ndvi_clean = ndvi[ndvi['mean_ndvi'] > 0]
ndvi_clean

# In[10]:

# Last sanity check before creating your plot (10 points)

# Ensure that you call your dataframe at the bottom of the cell above
# and that it has columns called: mean_ndvi and site

# Ensure the data is stored in a dataframe.
student_ndvi_df = _

df_points = 0

if isinstance(student_ndvi_df, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    df_points +=2
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Check that dataframe contains the appropriate number of NAN values
if student_ndvi_df.mean_ndvi.isna().sum() == 15:
    print('\u2705 Correct number of masked data values!')
    df_points +=2
else:
    print('\u274C The amount of null data in your dataframe is incorrect.')

# Ensure that the date column is the index
if isinstance(student_ndvi_df.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    df_points +=3
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_df.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    df_points +=3
else:
    print('\u274C The data in your date column is not datetime.')

# Output for timer, # DO NOT MODIFY
end_time = datetime.now()
total_time = end_time - start_time
print(
    "Your total run time for processing the data was {0}.".format(total_time))

print("\n \u27A1 You received {} out of 10 points for creating a dataframe.".format(
    df_points))

df_points

# In[11]:

# Add only the plot code to this cell

# This is the final figure of mean NDVI
# for both sites across the year
# with data cleaned to deal with clouds

# YOUR CODE HERE

# Set a list of sites to loop through
plot_sites = ['HARV', 'SJER']
# Set a list of unique colors
plot_colors = ['purple', 'green']

# Set figure
fig, ax = plt.subplots(figsize=(12, 6))

# Loop through dataframe and plot NDVI for each site using assigned color
for idx, asite in enumerate(plot_sites):
    temp = ndvi_clean[ndvi_clean['site']==asite]
    ax.plot(temp.index,
            temp.mean_ndvi,
            label=plot_sites[idx],
            color = plot_colors[idx])

#Set plot title and axes labels
ax.set(title = "Mean NDVI For HARV and SJER Study Sites\n (February 2017 - December 2017)\nInfluence of Clouds Removed",
       xlabel = "Month",
       ylabel = "Mean NDVI")

# Add legend
plt.legend()

# Add grid
plt.grid(color='grey', linestyle='-', linewidth=0.5)

# Format dates on x axis
# Add month names
date_form = DateFormatter("%b")
ax.xaxis.set_major_formatter(date_form)
# Add ticks for each months
ax.xaxis.set_major_locator(mdates.MonthLocator(interval=1))

### DO NOT REMOVE LINES BELOW ###
final_masked_solution = nb.convert_axes(plt, which_axes="current")

# In[12]:

# Ignore this cell for the autograding tests

# In[13]:

# Ignore this cell for the autograding tests

# # Question 1 (10 points)
#
# Imagine that you are planning NEON’s upcoming flight season to capture remote sensing data in these locations and want to ensure that you fly the area when the vegetation is the most green.
#
# When would you recommend the flights take place for each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# YOUR ANSWER HERE
# 1. I would plan flights for HARV site starting from mid May and until the first week of October.
# 2. Flights for SJER would have to be planned for a short window between mid February and end of March.

# # Question 2 (10 points)
#
# How could you modify your workflow to look at vegetation changes over time in each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# YOUR ANSWER HERE:
#
# 1. I would probably ended up making separate dataframes for each site.
# 2. I think, it may be good idea to process more landsat scenes (for few years). I can see seasonal changes on the data from one year, but long-term trend would be interesting to explore.
# 3. Would be interesting to compare NDVI pattern with tempreture and precipitation variations across years.

# # Do not edit this cell! (10 points)
#
# The notebook includes:
# * additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# # Do not edit this cell! (20 points)
#
# The notebook will also be checked for overall clean code requirements as specified at the **top** of this notebook. Some of these requirements include (review the top cells for more specifics):
#
# * Notebook begins at cell [1] and runs on any machine in its entirety.
# * PEP 8 format is applied throughout (including lengths of comment and code lines).
# * No additional code or imports in the notebook that is not needed for the workflow.
# * Notebook is fully reproducible. This means:
#    * reproducible paths using the os module.
#    * data downloaded using code in the notebook.
#    * all imports at top of notebook.

# ## BONUS - Export a  .CSV File to Share (10 points possible)
#
# This is optional - if you export a **.csv** file with the columns specified above: Site, Date and NDVI Value you can get an additional 10 points.
#
# * FULL CREDIT: File exists in csv format and contains the columns specified.
# We will check your github repo for this file!
#

# In[14]:

# Export final datafarame to csv stored in the outputs directory

# Check if output folder exists
# Create output folder if necessary

ndvi_output = os.path.join(
    'ndvi-automation',
    'outputs'
)

if os.path.exists(ndvi_output):
    print('Output directory exists')
else:
    print('Output directory does not exist but it is being created.')
    os.makedirs(ndvi_output)

# Create path and filename
ndvi_output_csv = os.path.join(
    ndvi_output,
    'ndvi_output.csv'
)

# Export final df to csv
ndvi_clean.to_csv(ndvi_output_csv)

# Check that output csv exists
if os.path.exists(ndvi_output_csv):
    print('Output CSV exists')
else:
    print('Oh no, final CSV does not exist in the outputs directory.')


#!/usr/bin/env python
# coding: utf-8

# # Detecting Payment Card Fraud
#
# In this section, we'll look at a credit card fraud detection dataset, and build a binary classification model that can identify transactions as either fraudulent or valid, based on provided, *historical* data. In a [2016 study](https://nilsonreport.com/upload/content_promo/The_Nilson_Report_10-17-2016.pdf), it was estimated that credit card fraud was responsible for over 20 billion dollars in loss, worldwide. Accurately detecting cases of fraud is an ongoing area of research.
#
#
#
# ### Labeled Data
#
# The payment fraud data set (Dal Pozzolo et al. 2015) was downloaded from [Kaggle](https://www.kaggle.com/mlg-ulb/creditcardfraud/data). This has features and labels for thousands of credit card transactions, each of which is labeled as fraudulent or valid. In this notebook, we'd like to train a model based on the features of these transactions so that we can predict risky or fraudulent transactions in the future.
#
# ### Binary Classification
#
# Since we have true labels to aim for, we'll take a **supervised learning** approach and train a binary classifier to sort data into one of our two transaction classes: fraudulent or valid.  We'll train a model on training data and see how well it generalizes on some test data.
#
# The notebook will be broken down into a few steps:
# * Loading and exploring the data
# * Splitting the data into train/test sets
# * Defining and training a LinearLearner, binary classifier
# * Making improvements on the model
# * Evaluating and comparing model test performance
#
# ### Making Improvements
#
# A lot of this notebook will focus on making improvements, as discussed in [this SageMaker blog post](https://aws.amazon.com/blogs/machine-learning/train-faster-more-flexible-models-with-amazon-sagemaker-linear-learner/). Specifically, we'll address techniques for:
#
# 1. **Tuning a model's hyperparameters** and aiming for a specific metric, such as high recall or precision.
# 2. **Managing class imbalance**, which is when we have many more training examples in one class than another (in this case, many more valid transactions than fraudulent).
#
# ---

# First, import the usual resources.

# In[1]:

import io
import os
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import boto3
import sagemaker
from sagemaker import get_execution_role

get_ipython().run_line_magic('matplotlib', 'inline')

# I'm storing my **SageMaker variables** in the next cell:
# * sagemaker_session: The SageMaker session we'll use for training models.
# * bucket: The name of the default S3 bucket that we'll use for data storage.
# * role: The IAM role that defines our data and model permissions.

# In[2]:

# sagemaker session, role
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# S3 bucket name
bucket = sagemaker_session.default_bucket()

# ## Loading and Exploring the Data
#
# Next, I am loading the data and unzipping the data in the file `creditcardfraud.zip`. This directory will hold one csv file of all the transaction data, `creditcard.csv`.
#
# As in previous notebooks, it's important to look at the distribution of data since this will inform how we develop a fraud detection model. We'll want to know: How many data points we have to work with, the number and type of features, and finally, the distribution of data over the classes (valid or fraudulent).

# In[3]:

# only have to run once
get_ipython().system('wget https://s3.amazonaws.com/video.udacity-data.com/topher/2019/January/5c534768_creditcardfraud/creditcardfraud.zip')
get_ipython().system('unzip creditcardfraud')

# In[4]:

# read in the csv file
local_data = 'creditcard.csv'

# print out some data
transaction_df = pd.read_csv(local_data)
print('Data shape (rows, cols): ', transaction_df.shape)
print()
transaction_df.head()

# ### Calculate the percentage of fraudulent data
#
# Take a look at the distribution of this transaction data over the classes, valid and fraudulent.
#
# Complete the function `fraudulent_percentage`, below. Count up the number of data points in each class and calculate the *percentage* of the data points that are fraudulent.

# In[5]:

# Calculate the fraction of data points that are fraudulent
def fraudulent_percentage(transaction_df):
    '''Calculate the fraction of all data points that have a 'Class' label of 1; fraudulent.
       :param transaction_df: Dataframe of all transaction data points; has a column 'Class'
       :return: A fractional percentage of fraudulent data points/all points
    '''

    # counts for all classes
    counts = transaction_df['Class'].value_counts()

    # get fraudulent and valid cnts
    fraud_cnts = counts[1]
    valid_cnts = counts[0]

    # calculate percentage of fraudulent data
    fraud_percentage = fraud_cnts/(fraud_cnts+valid_cnts)

    return fraud_percentage

# Test out your code by calling your function and printing the result.

# In[6]:

# call the function to calculate the fraud percentage
fraud_percentage = fraudulent_percentage(transaction_df)

print('Fraudulent percentage = ', fraud_percentage)
print('Total # of fraudulent pts: ', fraud_percentage*transaction_df.shape[0])
print('Out of (total) pts: ', transaction_df.shape[0])

# ### Split into train/test datasets
#
# In this example, we'll want to evaluate the performance of a fraud classifier; training it on some training data and testing it on *test data* that it did not see during the training process. So, we'll need to split the data into separate training and test sets.
#
# Complete the `train_test_split` function, below. This function should:
# * Shuffle the transaction data, randomly
# * Split it into two sets according to the parameter `train_frac`
# * Get train/test features and labels
# * Return the tuples: (train_features, train_labels), (test_features, test_labels)

# In[7]:

# split into train/test
def train_test_split(transaction_df, train_frac= 0.7, seed=1):
    '''Shuffle the data and randomly split into train and test sets;
       separate the class labels (the column in transaction_df) from the features.
       :param df: Dataframe of all credit card transaction data
       :param train_frac: The decimal fraction of data that should be training data
       :param seed: Random seed for shuffling and reproducibility, default = 1
       :return: Two tuples (in order): (train_features, train_labels), (test_features, test_labels)
       '''

    # convert the df into a matrix for ease of splitting
    df_matrix = transaction_df.as_matrix()

    # shuffle the data
    np.random.seed(seed)
    np.random.shuffle(df_matrix)

    # split the data
    train_size = int(df_matrix.shape[0] * train_frac)

    # features are all but last column
    train_features = df_matrix[:train_size, :-1]

    # class labels *are* last column
    train_labels = df_matrix[:train_size, -1]

    # test data
    test_features = df_matrix[train_size:, :-1]
    test_labels = df_matrix[train_size:, -1]

    return (train_features, train_labels), (test_features, test_labels)

    # shuffle and split the data
    train_features = None
    train_labels = None
    test_features = None
    test_labels = None

    return (train_features, train_labels), (test_features, test_labels)

# ### Test Cell
#
# In the cells below, I'm creating the train/test data and checking to see that result makes sense. The tests below test that the above function splits the data into the expected number of points and that the labels are indeed, class labels (0, 1).

# In[8]:

# get train/test data
(train_features, train_labels), (test_features, test_labels) = train_test_split(transaction_df, train_frac=0.7)

# In[9]:

# manual test

# for a split of 0.7:0.3 there should be ~2.33x as many training as test pts
print('Training data pts: ', len(train_features))
print('Test data pts: ', len(test_features))
print()

# take a look at first item and see that it aligns with first row of data
print('First item: \n', train_features[0])
print('Label: ', train_labels[0])
print()

# test split
assert len(train_features) > 2.333*len(test_features),         'Unexpected number of train/test points for a train_frac=0.7'
# test labels
assert np.all(train_labels)== 0 or np.all(train_labels)== 1,         'Train labels should be 0s or 1s.'
assert np.all(test_labels)== 0 or np.all(test_labels)== 1,         'Test labels should be 0s or 1s.'
print('Tests passed!')

# ---
# # Modeling
#
# Now that you've uploaded your training data, it's time to define and train a model!
#
# In this notebook, you'll define and train the SageMaker, built-in algorithm, [LinearLearner](https://sagemaker.readthedocs.io/en/stable/linear_learner.html).
#
# A LinearLearner has two main applications:
# 1. For regression tasks in which a linear line is fit to some data points, and you want to produce a predicted output value given some data point (example: predicting house prices given square area).
# 2. For binary classification, in which a line is separating two classes of data and effectively outputs labels; either 1 for data that falls above the line or 0 for points that fall on or below the line.
#
#
#
# In this case, we'll be using it for case 2, and we'll train it to separate data into our two classes: valid or fraudulent.

# ### Create a LinearLearner Estimator
#
# You've had some practice instantiating built-in models in SageMaker. All estimators require some constructor arguments to be passed in. See if you can complete this task, instantiating a LinearLearner estimator, using only the [LinearLearner documentation](https://sagemaker.readthedocs.io/en/stable/linear_learner.html) as a resource. This takes in a lot of arguments, but not all are required. My suggestion is to start with a simple model, utilizing default values where applicable. Later, we will discuss some specific hyperparameters and their use cases.
#
# #### Instance Types
#
# It is suggested that you use instances that are available in the free tier of usage: `'ml.c4.xlarge'` for training and `'ml.t2.medium'` for deployment.

# In[10]:

# import LinearLearner
from sagemaker import LinearLearner

# specify an output path
prefix = 'creditcard'
output_path = 's3://{}/{}'.format(bucket, prefix)

# instantiate LinearLearner
linear = LinearLearner(role = role,
                       train_instance_count = 1,
                       train_instance_type = 'ml.c4.xlarge',
                       predictor_type = 'binary_classifier',
                       output_path = output_path,
                       sagemaker_session = sagemaker_session,
                       epochs = 15)

# ### Convert data into a RecordSet format
#
# Next, prepare the data for a built-in model by converting the train features and labels into numpy array's of float values. Then you can use the [record_set function](https://sagemaker.readthedocs.io/en/stable/linear_learner.html#sagemaker.LinearLearner.record_set) to format the data as a RecordSet and prepare it for training!

# In[11]:

# convert features/labels to numpy
train_x_np = train_features.astype('float32')
train_y_np = train_labels.astype('float32')

# create RecordSet of training data
formatted_train_data = linear.record_set(train_x_np, labels = train_y_np)

# ### Train the Estimator
#
# After instantiating your estimator, train it with a call to `.fit()`, passing in the formatted training data.

# In[12]:

get_ipython().run_cell_magic('time', '', '# train the estimator on formatted training data\nlinear.fit(formatted_train_data)')

# ### Deploy the trained model
#
# Deploy your model to create a predictor. We'll use this to make predictions on our test data and evaluate the model.

# In[13]:

get_ipython().run_cell_magic('time', '', "# deploy and create a predictor\nlinear_predictor = linear.deploy(initial_instance_count = 1,\n                                 instance_type = 'ml.t2.medium')")

# ---
# # Evaluating Your Model
#
# Once your model is deployed, you can see how it performs when applied to the test data.
#
# According to the deployed [predictor documentation](https://sagemaker.readthedocs.io/en/stable/linear_learner.html#sagemaker.LinearLearnerPredictor), this predictor expects an `ndarray` of input features and returns a list of Records.
# > "The prediction is stored in the "predicted_label" key of the `Record.label` field."
#
# Let's first test our model on just one test point, to see the resulting list.

# In[14]:

# test one prediction
test_x_np = test_features.astype('float32')
result = linear_predictor.predict(test_x_np[0])

print(result)

# ### Helper function for evaluation
#
#
# The provided function below, takes in a deployed predictor, some test features and labels, and returns a dictionary of metrics; calculating false negatives and positives as well as recall, precision, and accuracy.

# In[15]:

# code to evaluate the endpoint on test data
# returns a variety of model metrics
def evaluate(predictor, test_features, test_labels, verbose=True):
    """
    Evaluate a model on a test set given the prediction endpoint.
    Return binary classification metrics.
    :param predictor: A prediction endpoint
    :param test_features: Test features
    :param test_labels: Class labels for test data
    :param verbose: If True, prints a table of all performance metrics
    :return: A dictionary of performance metrics.
    """

    # We have a lot of test data, so we'll split it into batches of 100
    # split the test data set into batches and evaluate using prediction endpoint
    prediction_batches = [predictor.predict(batch) for batch in np.array_split(test_features, 100)]

    # LinearLearner produces a `predicted_label` for each data point in a batch
    # get the 'predicted_label' for every point in a batch
    test_preds = np.concatenate([np.array([x.label['predicted_label'].float32_tensor.values[0] for x in batch])
                                 for batch in prediction_batches])

    # calculate true positives, false positives, true negatives, false negatives
    tp = np.logical_and(test_labels, test_preds).sum()
    fp = np.logical_and(1-test_labels, test_preds).sum()
    tn = np.logical_and(1-test_labels, 1-test_preds).sum()
    fn = np.logical_and(test_labels, 1-test_preds).sum()

    # calculate binary classification metrics
    recall = tp / (tp + fn)
    precision = tp / (tp + fp)
    accuracy = (tp + tn) / (tp + fp + tn + fn)

    # printing a table of metrics
    if verbose:
        print(pd.crosstab(test_labels, test_preds, rownames=['actual (row)'], colnames=['prediction (col)']))
        print("\n{:<11} {:.3f}".format('Recall:', recall))
        print("{:<11} {:.3f}".format('Precision:', precision))
        print("{:<11} {:.3f}".format('Accuracy:', accuracy))
        print()

    return {'TP': tp, 'FP': fp, 'FN': fn, 'TN': tn,
            'Precision': precision, 'Recall': recall, 'Accuracy': accuracy}

# ### Test Results
#
# The cell below runs the `evaluate` function.
#
# The code assumes that you have a defined `predictor` and `test_features` and `test_labels` from previously-run cells.

# In[16]:

print('Metrics for simple, LinearLearner.\n')

# get metrics for linear predictor
metrics = evaluate(linear_predictor,
                   test_features.astype('float32'),
                   test_labels,
                   verbose=True) # verbose means we'll print out the metrics

# ## Delete the Endpoint
#
# I've added a convenience function to delete prediction endpoints after we're done with them. And if you're done evaluating the model, you should delete your model endpoint!

# In[17]:

# Deletes a precictor.endpoint
def delete_endpoint(predictor):
        try:
            boto3.client('sagemaker').delete_endpoint(EndpointName=predictor.endpoint)
            print('Deleted {}'.format(predictor.endpoint))
        except:
            print('Already deleted: {}'.format(predictor.endpoint))

# In[18]:

# delete the predictor endpoint
delete_endpoint(linear_predictor)

# ---
#
# # Model Improvements
#
# The default LinearLearner got a high accuracy, but still classified fraudulent and valid data points incorrectly. Specifically classifying more than 30 points as false negatives (incorrectly labeled, fraudulent transactions), and a little over 30 points as false positives (incorrectly labeled, valid transactions). Let's think about what, during training, could cause this behavior and what we could improve.
#
# **1. Model optimization**
# * If we imagine that we are designing this model for use in a bank application, we know that users do *not* want any valid transactions to be categorized as fraudulent. That is, we want to have as few **false positives** (0s classified as 1s) as possible.
# * On the other hand, if our bank manager asks for an application that will catch almost *all* cases of fraud, even if it means a higher number of false positives, then we'd want as few **false negatives** as possible.
# * To train according to specific product demands and goals, we do not want to optimize for accuracy only. Instead, we want to optimize for a metric that can help us decrease the number of false positives or negatives.
#
#
#
# In this notebook, we'll look at different cases for tuning a model and make an optimization decision, accordingly.
#
# **2. Imbalanced training data**
# * At the start of this notebook, we saw that only about 0.17% of the training data was labeled as fraudulent. So, even if a model labels **all** of our data as valid, it will still have a high accuracy.
# * This may result in some overfitting towards valid data, which accounts for some **false negatives**; cases in which fraudulent data (1) is incorrectly characterized as valid (0).
#
# So, let's address these issues in order; first, tuning our model and optimizing for a specific metric during training, and second, accounting for class imbalance in the training set.
#

# ## Improvement: Model Tuning
#
# Optimizing according to a specific metric is called **model tuning**, and SageMaker provides a number of ways to automatically tune a model.
#
#
# ### Create a LinearLearner and tune for higher precision
#
# **Scenario:**
# * A bank has asked you to build a model that detects cases of fraud with an accuracy of about 85%.
#
# In this case, we want to build a model that has as many true positives and as few false negatives, as possible. This corresponds to a model with a high **recall**: true positives / (true positives + false negatives).
#
# To aim for a specific metric, LinearLearner offers the hyperparameter `binary_classifier_model_selection_criteria`, which is the model evaluation criteria for the training dataset. A reference to this parameter is in [LinearLearner's documentation](https://sagemaker.readthedocs.io/en/stable/linear_learner.html#sagemaker.LinearLearner). We'll also have to further specify the exact value we want to aim for; read more about the details of the parameters, [here](https://docs.aws.amazon.com/sagemaker/latest/dg/ll_hyperparameters.html).
#
# I will assume that performance on a training set will be within about 5% of the performance on a test set. So, for a recall of about 85%, I'll aim for a bit higher, 90%.

# In[19]:

# instantiate a LinearLearner
# tune the model for a higher recall
linear_recall = LinearLearner(role = role,
                              train_instance_count = 1,
                              train_instance_type = 'ml.c4.xlarge',
                              predictor_type = 'binary_classifier',
                              output_path = output_path,
                              sagemaker_session = sagemaker_session,
                              epochs = 15,
                              binary_classifier_model_selection_criteria = 'precision_at_target_recall', # target recall
                              target_recall = 0.9) # 90% recall

# ### Train the tuned estimator
#
# Fit the new, tuned estimator on the formatted training data.

# In[20]:

get_ipython().run_cell_magic('time', '', '# train the estimator on formatted training data\nlinear_recall.fit(formatted_train_data)')

# ### Deploy and evaluate the tuned estimator
#
# Deploy the tuned predictor and evaluate it.
#
# We hypothesized that a tuned model, optimized for a higher recall, would have fewer false negatives (fraudulent transactions incorrectly labeled as valid); did the number of false negatives get reduced after tuning the model?

# In[21]:

get_ipython().run_cell_magic('time', '', "# deploy and create a predictor\nrecall_predictor = linear_recall.deploy(initial_instance_count=1,\n                                        instance_type='ml.t2.medium')")

# In[22]:

print('Metrics for tuned (recall), LinearLearner.\n')

# get metrics for tuned predictor
metrics = evaluate(recall_predictor,
                   test_features.astype('float32'),
                   test_labels,
                   verbose=True)

# ## Delete the endpoint
#
# As always, when you're done evaluating a model, you should delete the endpoint. Below, I'm using the `delete_endpoint` helper function I defined earlier.

# In[23]:

# delete the predictor endpoint
delete_endpoint(recall_predictor)

# ---
# ## Improvement: Managing Class Imbalance
#
# We have a model that is tuned to get a higher recall, which aims to reduce the number of false negatives. Earlier, we discussed how class imbalance may actually bias our model towards predicting that all transactions are valid, resulting in higher false negatives and true negatives. It stands to reason that this model could be further improved if we account for this imbalance.
#
# To account for class imbalance during training of a binary classifier, LinearLearner offers the hyperparameter, `positive_example_weight_mult`, which is the weight assigned to positive (1, fraudulent) examples when training a binary classifier. The weight of negative examples (0, valid) is fixed at 1.
#
# ### Create a LinearLearner with a `positive_example_weight_mult` parameter
#
# In **addition** to tuning a model for higher recall (you may use `linear_recall` as a starting point), you should *add* a parameter that helps account for class imbalance. From the [hyperparameter documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/ll_hyperparameters.html) on `positive_example_weight_mult`, it reads:
# > "If you want the algorithm to choose a weight so that errors in classifying negative vs. positive examples have equal impact on training loss, specify `balanced`."
#
# You could also put in a specific float value, in which case you'd want to weight positive examples more heavily than negative examples, since there are fewer of them.

# In[24]:

# instantiate a LinearLearner

# include params for tuning for higher recall
# *and* account for class imbalance in training data
linear_balanced = LinearLearner(role = role,
                                train_instance_count = 1,
                                train_instance_type = 'ml.c4.xlarge',
                                predictor_type='binary_classifier',
                                output_path = output_path,
                                sagemaker_session = sagemaker_session,
                                epochs = 15,
                                binary_classifier_model_selection_criteria = 'precision_at_target_recall',
                                target_recall = 0.9,
                                positive_example_weight_mult = 'balanced')

# ### Train the balanced estimator
#
# Fit the new, balanced estimator on the formatted training data.

# In[25]:

get_ipython().run_cell_magic('time', '', '# train the estimator on formatted training data\nlinear_balanced.fit(formatted_train_data)')

# ### Deploy and evaluate the balanced estimator
#
# Deploy the balanced predictor and evaluate it. Do the results match with your expectations?

# In[26]:

get_ipython().run_cell_magic('time', '', "# deploy and create a predictor\nbalanced_predictor = linear_balanced.deploy(initial_instance_count = 1,\n                                            instance_type='ml.t2.medium')")

# In[27]:

print('Metrics for balanced, LinearLearner.\n')

# get metrics for balanced predictor
metrics = evaluate(balanced_predictor,
                   test_features.astype('float32'),
                   test_labels,
                   verbose=True)

# ## Delete the endpoint
#
# When you're done evaluating a model, you should delete the endpoint.

# In[28]:

# delete the predictor endpoint
delete_endpoint(balanced_predictor)

# A note on metric variability:
#
# The above model is tuned for the best possible precision with recall fixed at about 90%. The recall is fixed at 90% during training, but may vary when we apply our trained model to a test set of data.

# ---
# ## Model Design
#
# Now that you've seen how to tune and balance a LinearLearner. Create, train and deploy your own model. This exercise is meant to be more open-ended, so that you get practice with the steps involved in designing a model and deploying it.
#
# Train and deploy a LinearLearner with appropriate hyperparameters, according to the given scenario
#
# **Scenario:**
# * A bank has asked you to build a model that optimizes for a good user experience; users should only ever have up to about 15% of their valid transactions flagged as fraudulent.
#
# This requires that you make a design decision: Given the above scenario, what metric (and value) should you aim for during training?
#
# You may assume that performance on a training set will be within about 5-10% of the performance on a test set. For example, if you get 80% on a training set, you can assume that you'll get between about 70-90% accuracy on a test set.
#
# Your final model should account for class imbalance and be appropriately tuned.

# In[29]:

get_ipython().run_cell_magic('time', '', "# instantiate and train a LinearLearner\n\n# include params for tuning for higher precision\n# *and* account for class imbalance in training data\nlinear_precision = LinearLearner(role = role,\n                                train_instance_count = 1,\n                                train_instance_type = 'ml.c4.xlarge',\n                                predictor_type='binary_classifier',\n                                output_path = output_path,\n                                sagemaker_session = sagemaker_session,\n                                epochs = 15,\n                                binary_classifier_model_selection_criteria = 'recall_at_target_precision',\n                                target_precision = 0.9,\n                                positive_example_weight_mult = 'balanced')\n\n# train the estimator on formatted training data\nlinear_precision.fit(formatted_train_data)")

# In[30]:

get_ipython().run_cell_magic('time', '', "# deploy and evaluate a predictor\nprecision_predictor = linear_precision.deploy(initial_instance_count=1,\n                                              instance_type='ml.t2.medium')")

# In[31]:

print('Metrics for tuned (precision), LinearLearner.\n')

# get metrics for balanced predictor
metrics = evaluate(precision_predictor,
                   test_features.astype('float32'),
                   test_labels,
                   verbose=True)

# In[32]:

## IMPORTANT
# delete the predictor endpoint after evaluation
delete_endpoint(precision_predictor)

# ## Final Cleanup!
#
# * Double check that you have deleted all your endpoints.
# * I'd also suggest manually deleting your S3 bucket, models, and endpoint configurations directly from your AWS console.
#
# You can find thorough cleanup instructions, [in the documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html).

# ---
# # Conclusion
#
# In this notebook, you saw how to train and deploy a LinearLearner in SageMaker. This model is well-suited for a binary classification task that involves specific design decisions and managing class imbalance in the training set.
#
# Following the steps of a machine learning workflow, you loaded in some credit card transaction data, explored that data and prepared it for model training. Then trained, deployed, and evaluated several models, according to different design considerations!


#!/usr/bin/env python
# coding: utf-8

# ## Pandas

# In[1]:

import pandas as pd
import os

# In[2]:

os.getcwd()

# In[7]:

titanic_df=pd.read_csv('/Users/kangjunseo/python programming/파이썬 머신러닝 완벽 가이드/titanic_train.csv')
titanic_df.head(3)

# In[8]:

print(type(titanic_df))
print(titanic_df.shape)

# In[9]:

titanic_df.info()

# In[10]:

titanic_df.describe()

# In[11]:

print(titanic_df['Pclass'].value_counts())

# In[12]:

titanic_df['Pclass'].head()

# ### DataFrame 변환

# In[13]:

import numpy as np

# In[17]:

#1d array to DataFrame

col_name1=['col1']
df_list=pd.DataFrame([1,2,3],columns=col_name1)
print(df_list)

arr=np.array([1,2,3])
df_arr=pd.DataFrame(arr,columns=col_name1)
print(df_arr)

# In[18]:

#2d array to DataFrame

col_name2=['col1','col2','col3']

df_list=pd.DataFrame([[1,2,3],
                      [11,12,13]],columns=col_name2)
print(df_list)

arr=np.array([[1,2,3],
              [11,12,13]])
df_arr=pd.DataFrame(arr,columns=col_name2)
print(df_arr)

# In[20]:

#Dictionary to DataFrame
dict={'col1':[1,11],'col2':[2,22],'col3':[3,33]}
df_dict = pd.DataFrame(dict)
print(df_dict)

# #### Dataframe to Others

# In[85]:

#ndarray
arr=df_dict.values
print(arr)

#list
_list=df_dict.values.tolist()
print(_list)

#dictionary
dict=df_dict.to_dict()
print(dict)

# ### DataFrame의 Column Data Set 생성 및 수정

# #### 새로운 칼럼 추가

# In[25]:

titanic_df['Age_0']=0
titanic_df.head(3)

# In[27]:

titanic_df['Age_by_10']=titanic_df['Age']*10
titanic_df['Family_No']=titanic_df['SibSp']+titanic_df['Parch']+1
titanic_df.head(3)

# In[28]:

titanic_df['Age_by_10']=titanic_df['Age_by_10']+100
titanic_df.head(3)

# ### DataFrame 데이터 삭제

# #### inplace = False 인 경우

# In[32]:

#axis0=row, axis1=col
titanic_drop_df=titanic_df.drop('Age_0',axis=1)
titanic_drop_df.head()

# #### inplace = True 인 경우

# In[33]:

drop_result=titanic_df.drop(['Age_0','Age_by_10','Family_No'],axis=1,inplace=True)
print(drop_result) #inplace=True 이어서 반환값이 None
titanic_df.head(3)

# In[34]:

pd.set_option('display.width',1000)
pd.set_option('display.max_colwidth',15)
print(titanic_df.head(3))
titanic_df.drop([0,1,2],axis=0,inplace=True)
print(titanic_df.head(3))

# ### Index 객체

# In[37]:

titanic_df=pd.read_csv('titanic_train.csv')
indexes = titanic_df.index
print(indexes)
print(indexes.values) #1d array

# In[39]:

print(type(indexes.values))
print(indexes[:5].values)
print(indexes.values[:5])

# In[40]:

#한 번 생성된 DataFrame/Series의 Index 객체는 immutable
indexes[0]=5

# series 객체에 연산 함수 --> `Index`는 연산에서 제외

# In[42]:

series_fair = titanic_df['Fare']
print(series_fair.max() , series_fair.sum() ,sum(series_fair))
(series_fair+3).head(3)

# In[44]:

#인덱스가 연속된 int 숫자형 데이터가 아닐 경우 사용
titanic_reset_df=titanic_df.reset_index(inplace=False)
titanic_reset_df.head(3)

# In[45]:

value_counts = titanic_df['Pclass'].value_counts()
print(value_counts)
print(type(value_counts))
new_value_counts = value_counts.reset_index(inplace=False) #drop=True 시 index가 column으로 추가되지 않고 삭제됨
print(new_value_counts)
print(type(new_value_counts))

# ### Data Selection & Filtering

# #### DataFrame의 [] 연산자

# In[46]:

print(titanic_df['Pclass'].head(3)) #가능
print(titanic_df[0]) #불가능

# In[47]:

titanic_df[0:2] #pandas의 Index 형태로 변환가능, 하지만 사용하지 않는 것이 좋음

# In[48]:

titanic_df[titanic_df['Pclass']==3].head(3) #boolean indexing

# #### iloc[] - 위치 기반 Indexing

# In[50]:

data={'Name':['Chulmin','Eunkyung','Jinwoong','Soobeom'],
     'Year':[2011,2016,2015,2015],
     'Gender':['Male','Female','Male','Male']
     }
data_df=pd.DataFrame(data,index=['one','two','three','four'])
data_df

# In[51]:

data_df.iloc[0,0]

# In[52]:

data_df.iloc[0,'Name'] #명칭 입력시 오류
data_df.iloc['one,0']

# In[57]:

data_df_reset=data_df.reset_index()
data_df_reset=data_df_reset.rename(columns={'index':'old_index'})
data_df_reset.index = data_df_reset.index+1
print(data_df_reset)
data_df_reset.iloc[0,1] #index 값과 헷갈리지 않음

# #### loc[] - 명칭 기반 Indexing

# In[53]:

data_df.loc['one','Name']

# In[58]:

data_df_reset.loc[1,'Name'] #이 경우에는 인덱스 값으로 1이 존재하므로 가능

# In[59]:

print('위치 기반 iloc slicing\n', data_df.iloc[0:1,0],'\n') # 0부터 1-1까지
print('명칭 기반 loc slicing\n', data_df.loc['one':'two','Name']) #'one'부터 'two'까지 ('two'-1이 불가능하므로)

# In[60]:

print(data_df_reset.loc[1:2,'Name'])

# #### Boolean Indexing

# In[61]:

#Using []
titanic_df = pd.read_csv('titanic_train.csv')
titanic_boolean = titanic_df[titanic_df['Age']>60]
print(type(titanic_boolean))
titanic_boolean

# In[62]:

titanic_df[titanic_df['Age']>60][['Name','Age']].head(3)

# In[63]:

#Using loc[]
titanic_df.loc[titanic_df['Age']>60,['Name','Age']].head(3)

# In[65]:

# and = &, or = |, not = ~
# 60세 이상, 1등급, 여성
titanic_df[(titanic_df['Age']>60) & (titanic_df['Pclass']==1) & (titanic_df['Sex']=='female')]
# 각 조건연산자 사이 괄호 필수

# In[66]:

IsOld = titanic_df['Age']>60
IsFirst = titanic_df['Pclass']==1
IsWoman = titanic_df['Sex']=='female'
titanic_df[IsOld & IsFirst & IsWoman]

# ### Sort, Aggregation, GroupBy

# #### sort_values()

# In[67]:

# main args - by, ascending, inplace
titanic_sorted = titanic_df.sort_values(by=['Name'])
titanic_sorted.head(3)

# In[68]:

titanic_sorted = titanic_df.sort_values(by=['Pclass','Name'],ascending=False)
titanic_sorted.head(3)

# #### Aggregation

# In[69]:

#min(), max(), sum(), count()
titanic_df.count()

# In[70]:

titanic_df[['Age','Fare']].mean()

# #### groupby()

# In[71]:

titanic_groupby = titanic_df.groupby(by='Pclass')
print(type(titanic_groupby))

# In[72]:

titanic_groupby = titanic_groupby.count()
titanic_groupby

# In[73]:

titanic_groupby = titanic_df.groupby('Pclass')[['PassengerId','Survived']].count()
titanic_groupby

# In[74]:

#aggregation 적용
titanic_df.groupby('Pclass')['Age'].agg([max, min])

# In[75]:

#각각 다른 agg 적용
agg_format = {'Age':'max','SibSp':'sum','Fare':'mean'}
titanic_df.groupby('Pclass').agg(agg_format)

# ### 결손 데이터 처리

# NULL 데이터는 NaN으로 나타남
# 머신러닝 알고리즘은 이 값을 처리하지 않으므로 대체해야함

# #### isna() - 결손 데이터 여부 확인

# In[76]:

titanic_df.isna().head(3)

# In[77]:

titanic_df.isna().sum()

# #### fillna() - 결손 데이터 대체

# In[78]:

titanic_df['Cabin'] = titanic_df['Cabin'].fillna('C000')
titanic_df.head()

# In[79]:

titanic_df['Age'] = titanic_df['Age'].fillna(titanic_df['Age'].mean())
titanic_df['Embarked']=titanic_df['Embarked'].fillna('S')
titanic_df.isna().sum()

# ### apply lambda

# In[89]:

arr=np.arange(10)
sq_arr = map(lambda x : x**2,arr)
print(list(sq_arr))

# In[90]:

titanic_df['Name_len']=titanic_df['Name'].apply(lambda x :len(x))
titanic_df[['Name','Name_len']].head(3)

# In[93]:

titanic_df['Child_Adult']=titanic_df['Age'].apply(lambda x : 'Child' if x<=15 else 'Adult')
titanic_df[['Age','Child_Adult']].head(8)

# In[95]:

titanic_df['Age_cat']=titanic_df['Age'].apply(lambda x : 'Child' if x<=15 else('Adult' if x<=60 else 'Elder'))
titanic_df['Age_cat'].value_counts()

# In[96]:

def get_cat(age):
    cat =''
    if age<=5: cat='Baby'
    elif age<=12: cat='Child'
    elif age<=18: cat='Teen'
    elif age<=25: cat='학식'
    elif age<=35: cat='Adult'
    else : cat='Grand Grand Adult'

    return cat

titanic_df['Age_cat']= titanic_df['Age'].apply(lambda x : get_cat(x))
titanic_df[['Age','Age_cat']].head()


# Copyright 2018-2022 Xanadu Quantum Technologies Inc.

# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
r"""
This module contains the :class:`~.LightningGPU` class, a PennyLane simulator device that
interfaces with the NVIDIA cuQuantum cuStateVec simulator library for GPU-enabled calculations.
"""
from warnings import warn

import numpy as np
from pennylane import (
    math,
    BasisState,
    QubitStateVector,
    DeviceError,
    Projector,
    Hermitian,
    Rot,
    QuantumFunctionError,
    QubitStateVector,
)
from pennylane_lightning import LightningQubit
from pennylane.operation import Tensor, Operation
from pennylane.measurements import Expectation
from pennylane.wires import Wires

# Remove after the next release of PL
# Add from pennylane import matrix
import pennylane as qml

from ._version import __version__

try:
    from .lightning_gpu_qubit_ops import (
        LightningGPU_C128,
        LightningGPU_C64,
        AdjointJacobianGPU_C128,
        AdjointJacobianGPU_C64,
        device_reset,
        is_gpu_supported,
        get_gpu_arch,
        DevPool,
        ObsStructGPU_C128,
        ObsStructGPU_C64,
        OpsStructGPU_C128,
        OpsStructGPU_C64,
    )

    from ._serialize import _serialize_obs, _serialize_ops
    from ctypes.util import find_library
    from importlib import util as imp_util

    if find_library("custatevec") == None and not imp_util.find_spec("cuquantum"):
        raise ImportError(
            'cuQuantum libraries not found. Please check your "LD_LIBRARY_PATH" environment variable,'
            'or ensure you have installed the appropriate distributable "cuQuantum" package.'
        )
    if not is_gpu_supported():
        raise ValueError(f"CUDA device is an unsupported version: {get_gpu_arch()}")

    CPP_BINARY_AVAILABLE = True
except (ModuleNotFoundError, ImportError, ValueError) as e:
    warn(str(e), UserWarning)
    CPP_BINARY_AVAILABLE = False

def _gpu_dtype(dtype):
    if dtype not in [np.complex128, np.complex64]:
        raise ValueError(f"Data type is not supported for state-vector computation: {dtype}")
    return LightningGPU_C128 if dtype == np.complex128 else LightningGPU_C64

class LightningGPU(LightningQubit):
    """PennyLane-Lightning-GPU device.

    Args:
        wires (int): the number of wires to initialize the device with
        sync (bool): immediately sync with host-sv after applying operations
        c_dtype: Datatypes for statevector representation. Must be one of ``np.complex64`` or ``np.complex128``.
    """

    name = "PennyLane plugin for GPU-backed Lightning device using NVIDIA cuQuantum SDK"
    short_name = "lightning.gpu"
    pennylane_requires = ">=0.22"
    version = __version__
    author = "Xanadu Inc."
    _CPP_BINARY_AVAILABLE = True

    observables = {
        "PauliX",
        "PauliY",
        "PauliZ",
        "Hadamard",
        "Identity",
    }

    def __init__(self, wires, *, sync=True, c_dtype=np.complex128, shots=None, batch_obs=False):
        super().__init__(wires, c_dtype=c_dtype, shots=shots)
        self._gpu_state = _gpu_dtype(self._state.dtype)(self._state)
        self._sync = sync

    def reset(self):
        super().reset()
        self._gpu_state.resetGPU(False)  # Sync reset

    def syncH2D(self, use_async=False):
        """Explicitly synchronize CPU data to GPU"""
        self._gpu_state.HostToDevice(self._state.ravel(order="C"), use_async)

    def syncD2H(self, use_async=False):
        """Explicitly synchronize GPU data to CPU"""
        self._gpu_state.DeviceToHost(self._state.ravel(order="C"), use_async)
        self._pre_rotated_state = self._state

    @classmethod
    def capabilities(cls):
        capabilities = super().capabilities().copy()
        capabilities.update(
            model="qubit",
            supports_reversible_diff=False,
            supports_inverse_operations=True,
            supports_analytic_computation=True,
            supports_finite_shots=False,
            returns_state=True,
        )
        capabilities.pop("passthru_devices", None)
        return capabilities

    def statistics(self, observables, shot_range=None, bin_size=None):
        ## Ensure D2H sync before calculating non-GPU supported operations
        if self._sync:
            self.syncD2H()
        return super().statistics(observables, shot_range, bin_size)

    def apply_cq(self, operations, **kwargs):
        # Skip over identity operations instead of performing
        # matrix multiplication with the identity.
        skipped_ops = ["Identity"]

        for o in operations:
            if o.base_name in skipped_ops:
                continue
            name = o.name.split(".")[0]  # The split is because inverse gates have .inv appended
            method = getattr(self._gpu_state, name, None)

            wires = self.wires.indices(o.wires)

            if method is None:
                # Inverse can be set to False since qml.matrix(o) is already in inverted form
                try:
                    mat = qml.matrix(o)
                except AttributeError:  # pragma: no cover
                    # To support older versions of PL
                    mat = o.matrix

                if len(mat) == 0:
                    raise Exception("Unsupported operation")
                self._gpu_state.apply(
                    name,
                    wires,
                    False,
                    [],
                    mat.ravel(order="C"),  # inv = False: Matrix already in correct form;
                )  # Parameters can be ignored for explicit matrices; F-order for cuQuantum

            else:
                inv = o.inverse
                param = o.parameters
                method(wires, inv, param)

    def apply(self, operations, **kwargs):
        # State preparation is currently done in Python
        if operations:  # make sure operations[0] exists
            if isinstance(operations[0], QubitStateVector):
                self._apply_state_vector(operations[0].parameters[0].copy(), operations[0].wires)
                del operations[0]
                self.syncH2D()
            elif isinstance(operations[0], BasisState):
                self._apply_basis_state(operations[0].parameters[0], operations[0].wires)
                del operations[0]
                self.syncH2D()

        for operation in operations:
            if isinstance(operation, (QubitStateVector, BasisState)):
                raise DeviceError(
                    "Operation {} cannot be used after other Operations have already been "
                    "applied on a {} device.".format(operation.name, self.short_name)
                )

        self.apply_cq(operations)
        if self._sync:
            self.syncD2H()

    def adjoint_diff_support_check(self, tape):
        """Check Lightning adjoint differentiation method support for a tape.

        Raise ``QuantumFunctionError`` if ``tape`` contains not supported measurements,
        observables, or operations by the Lightning adjoint differentiation method.

        Args:
            tape (.QuantumTape): quantum tape to differentiate
        """
        for m in tape.measurements:
            if m.return_type is not Expectation:
                raise QuantumFunctionError(
                    "Adjoint differentiation method does not support"
                    f" measurement {m.return_type.value}"
                )
            if not isinstance(m.obs, Tensor):
                if isinstance(m.obs, Projector):
                    raise QuantumFunctionError(
                        "Adjoint differentiation method does not support the Projector observable"
                    )
                if isinstance(m.obs, Hermitian):
                    raise QuantumFunctionError(
                        "Lightning adjoint differentiation method does not currently support the Hermitian observable"
                    )
            else:
                if any([isinstance(o, Projector) for o in m.obs.non_identity_obs]):
                    raise QuantumFunctionError(
                        "Adjoint differentiation method does not support the Projector observable"
                    )
                if any([isinstance(o, Hermitian) for o in m.obs.non_identity_obs]):
                    raise QuantumFunctionError(
                        "Lightning adjoint differentiation method does not currently support the Hermitian observable"
                    )

        for op in tape.operations:
            if op.num_params > 1 and not isinstance(op, Rot):
                raise QuantumFunctionError(
                    f"The {op.name} operation is not supported using "
                    'the "adjoint" differentiation method'
                )

    def adjoint_jacobian(self, tape, starting_state=None, use_device_state=False):
        if self.shots is not None:
            warn(
                "Requested adjoint differentiation to be computed with finite shots."
                " The derivative is always exact when using the adjoint differentiation method.",
                UserWarning,
            )

        if len(tape.trainable_params) == 0:
            return np.array(0)

        # Check adjoint diff support
        self.adjoint_diff_support_check(tape)

        # Initialization of state
        if starting_state is not None:
            ket = np.ravel(starting_state, order="C")
        else:
            if not use_device_state:
                self.reset()
                self.execute(tape)
            ket = np.ravel(self._pre_rotated_state, order="C")

        if self.use_csingle:
            adj = AdjointJacobianGPU_C64()
            ket = ket.astype(np.complex64)
        else:
            adj = AdjointJacobianGPU_C128()

        obs_serialized = _serialize_obs(tape, self.wire_map, use_csingle=self.use_csingle)
        ops_serialized, use_sp = _serialize_ops(tape, self.wire_map, use_csingle=self.use_csingle)

        ops_serialized = adj.create_ops_list(*ops_serialized)

        trainable_params = sorted(tape.trainable_params)

        tp_shift = []
        record_tp_rows = []
        all_params = 0

        for op_idx, tp in enumerate(trainable_params):
            op, _ = tape.get_operation(
                op_idx
            )  # get op_idx-th operator among differentiable operators

            if isinstance(op, Operation) and not isinstance(op, (BasisState, QubitStateVector)):
                # We now just ignore non-op or state preps
                tp_shift.append(tp)
                record_tp_rows.append(all_params)
            all_params += 1

        if use_sp:
            # When the first element of the tape is state preparation. Still, I am not sure
            # whether there must be only one state preparation...
            tp_shift = [i - 1 for i in tp_shift]

        jac = adj.adjoint_jacobian(self._gpu_state, obs_serialized, ops_serialized, tp_shift)
        jac = np.array(jac)  # only for parameters differentiable with the adjoint method
        jac = jac.reshape(-1, len(tp_shift))
        jac_r = np.zeros((jac.shape[0], all_params))
        jac_r[:, record_tp_rows] = jac
        return jac_r

    def expval(self, observable, shot_range=None, bin_size=None):
        if observable.name in [
            "Projector",
            "Hamiltonian",
            "SparseHamiltonian",
        ]:
            self.syncD2H()
            return super().expval(observable, shot_range=shot_range, bin_size=bin_size)

        if self.shots is not None:
            # estimate the expectation value
            samples = self.sample(observable, shot_range=shot_range, bin_size=bin_size)
            return np.squeeze(np.mean(samples, axis=0))

        par = (
            observable.parameters
            if (
                len(observable.parameters) > 0 and isinstance(observable.parameters[0], np.floating)
            )
            else []
        )
        return self._gpu_state.ExpectationValue(
            observable.name,
            self.wires.indices(observable.wires),
            par,  # observables should not pass parameters, use matrix instead
            qml.matrix(observable).ravel(order="C"),
        )

    def probability(self, wires=None, shot_range=None, bin_size=None):
        if self.shots is not None:
            return self.estimate_probability(wires=wires, shot_range=shot_range, bin_size=bin_size)

        wires = wires or self.wires
        wires = Wires(wires)

        # translate to wire labels used by device
        device_wires = self.map_wires(wires)
        # Device returns as col-major orderings, so perform transpose on data for bit-index shuffle for now.
        return (
            self._gpu_state.Probability(device_wires)
            .reshape([2] * len(wires))
            .transpose()
            .reshape(-1)
        )

    def generate_samples(self):
        """Generate samples

        Returns:
            array[int]: array of samples in binary representation with shape ``(dev.shots, dev.num_wires)``
        """
        return self._gpu_state.GenerateSamples(len(self.wires), self.shots).astype(int)

    def var(self, observable, shot_range=None, bin_size=None):
        if self.shots is not None:
            # estimate the var
            # Lightning doesn't support sampling yet
            samples = self.sample(observable, shot_range=shot_range, bin_size=bin_size)
            return np.squeeze(np.var(samples, axis=0))

        adjoint_matrix = math.T(math.conj(qml.matrix(observable)))
        sqr_matrix = np.matmul(adjoint_matrix, qml.matrix(observable))

        mean = self._gpu_state.ExpectationValue(
            [i + "_var" for i in observable.name],
            self.wires.indices(observable.wires),
            observable.parameters,
            qml.matrix(observable).ravel(order="C"),
        )

        squared_mean = self._gpu_state.ExpectationValue(
            [i + "_sqr" for i in observable.name],
            self.wires.indices(observable.wires),
            observable.parameters,
            sqr_matrix.ravel(order="C"),
        )

        return squared_mean - (mean**2)

if not CPP_BINARY_AVAILABLE:

    class LightningGPU(LightningQubit):
        name = "PennyLane plugin for GPU-backed Lightning device using NVIDIA cuQuantum SDK: Lightning CPU fall-back"
        short_name = "lightning.gpu"
        pennylane_requires = ">=0.22"
        version = __version__
        author = "Xanadu Inc."
        _CPP_BINARY_AVAILABLE = False

        def __init__(self, wires, *, c_dtype=np.complex128, **kwargs):
            w_msg = """
            !!!#####################################################################################
            !!!
            !!! WARNING: INSUFFICIENT SUPPORT DETECTED FOR GPU DEVICE WITH `lightning.gpu`
            !!!          DEFAULTING TO CPU DEVICE `lightning.qubit`
            !!!
            !!!#####################################################################################
            """
            warn(
                w_msg,
                RuntimeWarning,
            )
            super().__init__(wires, c_dtype=c_dtype, **kwargs)


import numpy as np
import pdb
import math
from . import data_generators
import copy
from log import logger

def calc_iou(rois, augmented_annotation, C, class_name_idx_mapping):
    """
    分别计算每一个 roi 和所有 gt_bboxes 的 best iou, 并根据 best_iou 判断 roi 的类型
        如果 best_iou < C.rcnn_min_overlap, 忽略此 roi
        如果 C.rcnn_min_overlap <= best_iou < C.rcnn_max_overlap, 设置该 roi 的 class='bg',regr=[0,0,0,0]
        如果 C.rcnn_max_overlap <= best_iou, 设置 best_iou 对应的 gt_bbox 的 class 为该 roi 的 class, 并计算 regr
    :param rois:
    :param augmented_annotation:
    :param C:
    :param class_name_idx_mapping:
    :return: X,Y21,Y22 X: 表示合格的 rois 组成的数组作为 rcnn 的输入, (1, num_rois,4)
        Y21: 表示对所有 rois 分类的结果, (1, num_rois, num_classes)
        Y22: shape 为 (1,num_rois, (num_classes -1) * 4 * 2)
            前 (num_classes-1)*4 表示 rois 是否是 bg, 不参与回归的 loss 计算
            后 (num_classes-1)*4 表示对所有非 bg 的 rois 的回归梯度
        IoUs: 表示每一个 roi 和所有 gt_bboxes 的 best iou
    """
    bboxes = augmented_annotation['bboxes']
    (width, height) = (augmented_annotation['width'], augmented_annotation['height'])
    # get image dimensions for resizing
    (resized_width, resized_height) = data_generators.get_new_image_size(width, height, C.image_min_size)
    # ground truth bbox 在 feature map 上的坐标
    gt_bboxes = np.zeros((len(bboxes), 4))
    width_ratio = resized_width / float(width)
    height_ratio = resized_height / float(height)
    for bbox_idx, bbox in enumerate(bboxes):
        # get the GT box coordinates, and resize to account for image resizing
        gt_bboxes[bbox_idx, 0] = int(round(bbox['x1'] * width_ratio / C.rpn_stride))
        gt_bboxes[bbox_idx, 1] = int(round(bbox['x2'] * width_ratio / C.rpn_stride))
        gt_bboxes[bbox_idx, 2] = int(round(bbox['y1'] * height_ratio / C.rpn_stride))
        gt_bboxes[bbox_idx, 3] = int(round(bbox['y2'] * height_ratio / C.rpn_stride))

    x_roi = []
    # 分类模型的 y,是 one-hot 形式,(None,len(class_name_idx_mapping))
    y_class = []
    # 回归模型的 y,(None,len(class_name_idx_mapping - 1) * 4)
    y_regr = []
    # 标记 regr 是否参与 loss 计算
    y_regr_valid = []
    IoUs = []  # for debugging only

    for ix in range(rois.shape[0]):
        (x1, y1, x2, y2) = rois[ix, :]
        x1 = int(round(x1))
        y1 = int(round(y1))
        x2 = int(round(x2))
        y2 = int(round(y2))

        best_iou = 0.0
        best_bbox_idx = -1
        # roi 和每一个 bbox 计算 iou, 取 best_iou
        for bbox_idx in range(len(bboxes)):
            current_iou = data_generators.iou([gt_bboxes[bbox_idx, 0], gt_bboxes[bbox_idx, 2], gt_bboxes[bbox_idx, 1], gt_bboxes[bbox_idx, 3]],
                                              [x1, y1, x2, y2])
            if current_iou > best_iou:
                best_iou = current_iou
                best_bbox_idx = bbox_idx

        if best_iou < C.rcnn_min_overlap:
            continue
        else:
            w = x2 - x1
            h = y2 - y1
            x_roi.append([x1, y1, w, h])
            IoUs.append(best_iou)

            if C.rcnn_min_overlap <= best_iou < C.rcnn_max_overlap:
                # hard negative example
                class_name = 'bg'
            elif C.rcnn_max_overlap <= best_iou:
                class_name = bboxes[best_bbox_idx]['class']
                # ground truth bbox 的中心点坐标
                gcx = (gt_bboxes[best_bbox_idx, 0] + gt_bboxes[best_bbox_idx, 1]) / 2.0
                gcy = (gt_bboxes[best_bbox_idx, 2] + gt_bboxes[best_bbox_idx, 3]) / 2.0
                gw = gt_bboxes[best_bbox_idx, 1] - gt_bboxes[best_bbox_idx, 0]
                gh = (gt_bboxes[best_bbox_idx, 3] - gt_bboxes[best_bbox_idx, 2])
                # roi 的中心点坐标
                cx = x1 + w / 2.0
                cy = y1 + h / 2.0
                # 计算梯度
                tx = (gcx - cx) / float(w)
                ty = (gcy - cy) / float(h)
                tw = np.log(gw / float(w))
                th = np.log(gh / float(h))
            else:
                logger.error('roi={}'.format(best_iou))
                raise RuntimeError

        class_idx = class_name_idx_mapping[class_name]
        # 构建分类模型的 y
        class_label = len(class_name_idx_mapping) * [0]
        class_label[class_idx] = 1
        y_class.append(copy.deepcopy(class_label))
        regr_label = [0] * 4 * (len(class_name_idx_mapping) - 1)
        regr_label_valid = [0] * 4 * (len(class_name_idx_mapping) - 1)
        if class_name != 'bg':
            regr_pos = 4 * class_idx
            # UNCLEAR: C.classifier_regr_std 的作用是？
            sx, sy, sw, sh = C.classifier_regr_std
            regr_label[regr_pos:4 + regr_pos] = [sx * tx, sy * ty, sw * tw, sh * th]
            regr_label_valid[regr_pos:4 + regr_pos] = [1, 1, 1, 1]
            y_regr.append(copy.deepcopy(regr_label))
            y_regr_valid.append(copy.deepcopy(regr_label_valid))
        else:
            y_regr.append(copy.deepcopy(regr_label))
            y_regr_valid.append(copy.deepcopy(regr_label_valid))

    if len(x_roi) == 0:
        return None, None, None, None

    X = np.array(x_roi)
    Y_class = np.array(y_class)
    Y_regr = np.concatenate([np.array(y_regr_valid), np.array(y_regr)], axis=-1)
    # expand_dims 是为了 batch 那一维度
    return np.expand_dims(X, axis=0), np.expand_dims(Y_class, axis=0), np.expand_dims(Y_regr, axis=0), IoUs

def apply_regr(x, y, w, h, tx, ty, tw, th):
    """
    对单个矩形框进行修正
    :param x:
    :param y:
    :param w:
    :param h:
    :param tx:
    :param ty:
    :param tw:
    :param th:
    :return:
    """
    try:
        cx = x + w / 2.
        cy = y + h / 2.
        # rcx,rcy 表示修正后的矩形框的中心点 x,y
        # (rcx - cx) / w = tx
        rcx = tx * w + cx
        rcy = ty * h + cy
        # rw,rh 表示修正后的矩形框的宽和高
        # log(rw / w) = tw
        rw = math.exp(tw) * w
        rh = math.exp(th) * h
        # 左上角的点的 x,y
        rx = rcx - rw / 2.
        ry = rcy - rh / 2.
        rx = int(round(rx))
        ry = int(round(ry))
        rw = int(round(rw))
        rh = int(round(rh))
        return rx, ry, rw, rh
    except ValueError:
        return x, y, w, h
    except OverflowError:
        return x, y, w, h
    except Exception as e:
        logger.error(e)
        return x, y, w, h

def apply_regr_np(X, T):
    """
    根据回归梯度对所有的 anchor 进行校正
    :param X: anchor 的坐标, shape 为 (m, n, 4)
    :param T: regr 梯度, shape 为 (m, n, 4)
    :return: 修正后的 anchor 的坐标 array([x,y,w,h]...), shape 为 (m, n, 4)
    """
    try:
        height, width = X.shape[:2]
        # anchor 的坐标
        xa = X[:, :, 0]
        ya = X[:, :, 1]
        wa = X[:, :, 2]
        ha = X[:, :, 3]
        # regr 梯度
        tx = T[:, :, 0]
        ty = T[:, :, 1]
        tw = T[:, :, 2]
        th = T[:, :, 3]
        # anchor 的中心点坐标 x,y
        cxa = xa + wa / 2.
        cya = ya + ha / 2.
        # tx = (cxg - cxa) / wa
        # rectified_anchor 的中心点坐标 x,y
        cxg = tx * wa + cxa
        cyg = ty * ha + cya
        # tw = log(wg / wa)
        wg = np.exp(tw.astype(np.float64)) * wa
        hg = np.exp(th.astype(np.float64)) * ha
        xg = cxg - wg / 2.
        yg = cyg - hg / 2.

        xg = np.round(xg)
        yg = np.round(yg)
        wg = np.round(wg)
        hg = np.round(hg)

        xg = xg.reshape(height, width, 1)
        yg = yg.reshape(height, width, 1)
        wg = wg.reshape(height, width, 1)
        hg = hg.reshape(height, width, 1)
        rectified_anchor_coordinates = np.concatenate((xg, yg, wg, hg), axis=-1)
        return rectified_anchor_coordinates
    except Exception as e:
        logger.exception(e)
        return X

def non_max_suppression_fast(boxes, probs, overlap_thresh=0.9, max_boxes=300):
    """

    :param boxes: proposal 的 box 的坐标 np.array[[x1,y1,x2,y2]...]
    :param probs: box 包含物体的概率 np.array([p1,p2...])
    :param overlap_thresh: 覆盖阈值
    :param max_boxes: 最大返回的 box 个数
    :return: filtered_boxes and related probs
    """
    # code used from here: http://www.pyimagesearch.com/2015/02/16/faster-non-maximum-suppression-python/
    # https://www.pyimagesearch.com/2014/11/17/non-maximum-suppression-object-detection-python
    # if there are no boxes, return an empty list
    if len(boxes) == 0:
        return np.array([]), np.array([])

    # grab the coordinates of the bounding boxes
    x1 = boxes[:, 0]
    y1 = boxes[:, 1]
    x2 = boxes[:, 2]
    y2 = boxes[:, 3]

    # https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.testing.assert_array_less.html
    # 如果两数组的 shape 不同,会抛出异常
    # 如果 shape 相同但是有前一个数组的元素大于后一个数组的元素,也会抛出异常
    np.testing.assert_array_less(x1, x2)
    np.testing.assert_array_less(y1, y2)

    # if the bounding boxes integers, convert them to floats --
    # this is important since we'll be doing a bunch of divisions
    if boxes.dtype.kind == "i":
        boxes = boxes.astype("float")

    # initialize the list of picked indexes
    pick = []

    # calculate the areas
    # 数组的每个元素分别相乘 [area1,area2...]
    area = (x2 - x1) * (y2 - y1)

    # sort the bounding boxes, 返回的结果是下标数组,每一个元素的值为原数组的元素的下标
    # 如最后一个元素的值表示的是原数组最大值的下标,第一个元素的值表示原数组最小值的下标
    # https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html
    idxs = np.argsort(probs)
    # keep looping while some indexes still remain in the indexes list
    while len(idxs) > 0:
        # grab the last index in the indexes list and add the
        # index value to the list of picked indexes
        last = len(idxs) - 1
        # 最大 prob 的 box 在原 boxes 中的下标
        i = idxs[last]
        pick.append(i)

        # 计算最大 prob 的 box 和其他所有 box 的交集
        # x1 的 shape (num_boxes,), x1[i] 的 shape 为 (1,)
        # x1[idxs[:last]] 的 shape 为 (last, )
        # x1_intersection 的 shape 为 (last, )
        #  _____________________
        # |  (max_x1,max_y1)    |
        # |                 ____|________________
        # |                |    |                |
        # |                |    |                |
        # |                |    |                |
        # |________________|____|(min_x2,min_y2) |
        #                  |                     |
        #                  |_____________________|
        # x1[idxs[:last]] 其实这 last 个 x1 就已经按照 probs 排好序了
        x1_intersection = np.maximum(x1[i], x1[idxs[:last]])
        y1_intersection = np.maximum(y1[i], y1[idxs[:last]])
        x2_intersection = np.minimum(x2[i], x2[idxs[:last]])
        y2_intersection = np.minimum(y2[i], y2[idxs[:last]])
        w_intersection = np.maximum(0, x2_intersection - x1_intersection)
        h_intersection = np.maximum(0, y2_intersection - y1_intersection)
        # shape 为 (last,)
        area_intersection = w_intersection * h_intersection
        # 计算最大 prob 的 box 和其他所有 box 的并集
        area_union = area[i] + area[idxs[:last]] - area_intersection
        # compute the ratio of overlap, 就是 IOU
        # shape 为 (last,)
        overlap = area_intersection / (area_union + 1e-6)

        # delete all indexes from the index list that have
        # 删除当前最大 prob 的 box 和当前最大 prob 的 box 重叠超过阈值的其他 box
        # 这里因为 overlap 是按照 idxs 获得的, 所以 np.where 其实返回的 idxs 的下标
        idxs = np.delete(idxs, np.concatenate(([last],
                                               np.where(overlap > overlap_thresh)[0])))

        if len(pick) >= max_boxes:
            break

    # return only the bounding boxes that were picked using the integer data type
    boxes = boxes[pick].astype("int")
    probs = probs[pick]
    return boxes, probs

def rpn_to_roi(rpn_class, rpn_regr, C, max_rois=300, overlap_thresh=0.9):
    """
    对所有的 anchor, 根据 rpn 的分类和回归结果进行筛选和修正, 生成 roi
    :param rpn_class: rpn 分类的结果 (1,m,n,9) m,n 表示 feature map 的高和宽
    :param rpn_regr: rpn 回归的结果 (1,m,n,36)
    :param C: config 对象
    :param max_rois: 最大 roi 个数
    :param overlap_thresh:
    :return: 返回 rois 在 feature map 上的坐标 (x1,y1,x2,y2)
    """
    # 在生成 rpn target 时乘过 C.std_scaling
    rpn_regr = rpn_regr / C.std_scaling
    anchor_scales = C.anchor_scales
    anchor_ratios = C.anchor_ratios

    assert rpn_class.shape[0] == 1
    assert rpn_regr.shape[0] == 1
    # feature map 的高,宽和深, d=9
    (m, n, d) = rpn_class.shape[1:]

    anchor_idx = 0
    # (m,n,9,4)
    A = np.zeros((m, n, d, 4))

    for anchor_scale in anchor_scales:
        for anchor_ratio in anchor_ratios:
            # anchor 在特征图上的宽
            anchor_width = (anchor_scale * anchor_ratio[0]) / C.rpn_stride
            # anchor 在特征图上的高
            anchor_height = (anchor_scale * anchor_ratio[1]) / C.rpn_stride
            # 该 anchor 的回归梯度
            regr = rpn_regr[0, :, :, 4 * anchor_idx:4 * anchor_idx + 4]
            # X: (m, n) [[0,1...,n-1],[0,1...,n-1]...]
            # Y: (m, n) [[0,0...,0],[1,1...,1],...[m-1,m-1...,m-1]]
            # 其实生成的是一个坐标,特征图上的每个点都有了相应的坐标
            # X 表示的是所有 anchor 的中心点的 x 坐标
            # Y 表示的是所有 anchor 的中心点的 y 坐标
            X, Y = np.meshgrid(np.arange(n), np.arange(m))
            # A 的最后一维的四个值表示 anchor 的 (x,y,w,h), 其中 x,y 表示左上方点的坐标
            A[:, :, anchor_idx, 0] = X - anchor_width / 2
            A[:, :, anchor_idx, 1] = Y - anchor_height / 2
            A[:, :, anchor_idx, 2] = anchor_width
            A[:, :, anchor_idx, 3] = anchor_height
            # 根据回归梯度修正
            A[:, :, anchor_idx, :] = apply_regr_np(A[:, :, anchor_idx, :], regr)
            # 宽度最小为 1
            A[:, :, anchor_idx, 2] = np.maximum(1, A[:, :, anchor_idx, 2])
            # 高度最小为 1
            A[:, :, anchor_idx, 3] = np.maximum(1, A[:, :, anchor_idx, 3])
            # A[:,:,anchor_idx, 2] 变成了 x2, 即右下角 x 坐标
            A[:, :, anchor_idx, 2] += A[:, :, anchor_idx, 0]
            # A[:,:,anchor_idx, 3] 变成了 y2, 即右下角 y 坐标
            A[:, :, anchor_idx, 3] += A[:, :, anchor_idx, 1]
            # x1 最小为 0
            A[:, :, anchor_idx, 0] = np.maximum(0, A[:, :, anchor_idx, 0])
            # y1 最小为 0
            A[:, :, anchor_idx, 1] = np.maximum(0, A[:, :, anchor_idx, 1])
            # x2 最大为 n - 1
            A[:, :, anchor_idx, 2] = np.minimum(n - 1, A[:, :, anchor_idx, 2])
            # y2 最大为 m - 1
            A[:, :, anchor_idx, 3] = np.minimum(m - 1, A[:, :, anchor_idx, 3])

            anchor_idx += 1

    # (m*n*9,4)
    all_rois = np.reshape(A, (-1, 4))
    # (m*n*9,)
    all_probs = rpn_class.reshape((-1))

    x1 = all_rois[:, 0]
    y1 = all_rois[:, 1]
    x2 = all_rois[:, 2]
    y2 = all_rois[:, 3]

    # 删除不合理的矩形存在, 左上角的坐标不能大于右下角的坐标
    idxs = np.where((x1 - x2 >= 0) | (y1 - y2 >= 0))
    all_rois = np.delete(all_rois, idxs, axis=0)
    all_probs = np.delete(all_probs, idxs, axis=0)

    rois = non_max_suppression_fast(all_rois, all_probs, overlap_thresh=overlap_thresh, max_boxes=max_rois)[0]

    return rois