#!/usr/bin/env python
# coding: utf-8

#

# # Capstone Project: Starbucks
#
# 1. [Background](#background)
# 2. [Exploratory Data Analysis](#exploratory-data-analysis)
# 3. [Data Pre-processing](#data-pre-processing)
# 4. [Analysis: Rationale, Transaction Labelling, PCA (dimension rediction), K-means clustering](#analysis)
# 5. [Interpretation](#interpretation)

#

# # 1.0 Background

# [back to top menu](#toplevel)

# The Starbucks Capstone details a test which was conducted on a sample of customers. The data captures customers' responses to different offers and the accompanying analyses aims to use the information to determine the effectiveness of these offers across the different customer segments.
#
# My proposed solution leverages an understanding of customer purchase behaviour in framing an approach to segmenting customers within the sample and assessing the performance of each offer within the resulting customer segments.
#
# Specifically, the approach involves:
#
# 1. Reviewing customer transactions to identify those influenced by specific offers and those that were not. The resulting insights will be used to define individual customer behaviour/preferences (unstimulated spend, BOGO spend, Discount spend)
#
#
# 2. Segmenting the sample customer base using behaviour/preferences. Segments should contain customers of similiar purchase behaviour.
#
#
# 3. Reviewing value/count of offer-influenced transactions vs "uninfluenced" transactions at customer and segment levels. value/count of offer-influenced transactions represent incremental change due to offers while magnitude of change is established by comparing against relevant "uninfluenced" transaction measures.

#

# [back to top menu](#toplevel)
# # 2.0 Exploratory Data Analysis
#
# Within this section, the relevant datasets will be imported and characteristics identified. The insights gained from this activity will guide the extent of data cleaning and transformation required. In addition, these insights will guide the choice of machine learning models most suitable for the project.Datasets to import and review include: Portfolio; Profile; Transcript

# ## 2.1 Data Import

# In[468]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import math
import json
import seaborn as sns
sns.set()
#%matplotlib inline

# read in the json files
portfolio = pd.read_json('portfolio.json', orient='records', lines=True)
profile = pd.read_json('profile.json', orient='records', lines=True)
transcript = pd.read_json('transcript.json', orient='records', lines=True)

# ## 2.2 Portfolio Data
# To Review: dataframe structure; data types; unique values; distribution

# In[469]:

# View the dataframe and values
portfolio.head(3)

# In[470]:

# Review the dimension of the dataframe
portfolio.shape

# In[471]:

# View data structure and types
portfolio.info()

# In[472]:

# View Offer Types
portfolio['offer_type'].unique().tolist()

# In[473]:

# Distribution of values within numeric fields
portfolio.hist(alpha=0.5, figsize=(5, 5));

# #### Portfolio EDA: Observation(s)
#
# 1. The Portfolio data contains 10 unique records and 6 columns.
# 2. There are no null values and data types include: int64 and objects.
# 3. There are 3 types of offers - bogo, Informational and discount.
# 4. Mode for reward, difficulty and duration values: 5, 10, 7.
#
# #### Portfolio EDA: Action Point(s)
#
# 1. Create a new id column with more recognizable and easier to process values for offer id.
# 2. Duration (days) needs to be converted to hours which is the unit of measure for time in transcript dataset.

# ## 2.3 Profile Dataset
# To Review: dataframe structure; data types; unique values; distribution

# In[474]:

# View the dataframe and values
profile.head(3)

# In[475]:

# Review the dimension of the dataframe
profile.shape

# In[476]:

# View data structure and types
profile.info()

# In[477]:

# Null check: Count and % proportion
print('Null Check: Count')
print('-----------------')
print(profile.isnull().sum())
print('')
print('Null Check: % occurrance')
print('------------------------')
print(profile.isnull().mean())

# In[478]:

# Distribution of values within numeric fields
profile[['age','income']].hist(alpha=0.5, figsize=(8, 3));

# In[479]:

# Understand categories within gender
profile['gender'].unique()

# In[480]:

# Understand distribution of gender
profile['gender'].hist(alpha=0.5, figsize=(5, 3));

# #### Profile EDA: Observation(s)
#
# 1. The Portfolio data contains 17,000 unique records and 5 columns.
# 2. There are 2,175 null values in gender and income columns. The proportion of null values is 12.5% of dataset
# 3. Data types include: int64, object and null
# 4. Age column has an outlier value: 118.
# 5. Income distribution is right skewed
# 6. "became_member_on" column should be date and not integer
#
# #### Profile EDA: Action Point(s)
# 1. Create a short version of id (user_id).
# 2. Address null records: Keep (Impute) or Remove (Delete)
# 3. Attempt a correction for error in age (118). Most likely a data entry error for 18.
# 4. Convert "became_member_on" to datetype

# ## 2.4 Transcript Dataset
# To Review: table structure; data types; unique values; distribution

# In[481]:

# View the table structure and values
transcript.head(3)

# In[482]:

# Review the dimension of the dataframe
transcript.shape

# In[483]:

# View data structure and types
transcript.info()

# In[484]:

# Understand categories within event
transcript['event'].value_counts()

# In[485]:

# Highlight data structure within events column
transcript['value'].tail()

# In[486]:

# Highlight data structure within events column
transcript['value'].head()

# In[487]:

# Uniqueness check of person column
transcript['person'].nunique()

# #### Transcript EDA: Observation(s)
# 1. The Transcript data contains 306534 unique records and 4 columns.
# 2. There are no null values in any columns.
# 3. Data types include: int64, object (incl. dictionary in value column)
#
# #### Transcript EDA: Action Point(s)
# 1. Rename person column to user_id and map to profile user id (short version).
# 2. Unpack value column to extract key and values

#

# [back to top menu](#toplevel)
#
# # 3.0 Data Pre-processing
# Within this section, data transformation requirements identified as action points within the Exploratory Data Analysis (EDA) section would be completed. Further data tranformation activities will be performed on relevant data in preparation for various analysis and machine learning models

# ## 3.1 Portfolio
# - Create a new id column with more recognizable and easier to process values for offer id.
# - Duration (days) needs to be converted to hours which is the unit of measure for time in transcript dataset.
# - Create mapping files: short version id vs id; short id vs duration in hours

# Create a new id column with more recognizable and easier to process values for offer id

# In[488]:

# define rules
rule01 = lambda x:x[:4]
rule02 = lambda x:'0'+ str(x) if x<10 else str(x)
rule03 = lambda x:'0'+ str(x) if x<10 else str(x)

# apply to relevant columns within dataframe
portfolio['short_id'] = portfolio['offer_type'].apply(rule01) + portfolio['duration'].apply(rule02) + portfolio['difficulty'].apply(rule03)

# In[489]:

# view impact on dataframe
portfolio.head(3)

# Create a column for duration in hours

# In[490]:

# set conversion of day to hours (24hrs = 1 day)
day_hrs = 24.0

# apply to relevant columns within dataframe
portfolio['duration_hrs'] = portfolio['duration'] * day_hrs

# In[491]:

# view impact on dataframe
portfolio

# Create Mapping Files: short version id vs id; short id vs duration in hours

# In[492]:

# Map long offer id to new short id
map_id_shortid = dict(portfolio[['id','short_id']].values.tolist()) # map required to simplify data analysis

# Map new short id to duration (in hours)
map_shortid_hrs = dict(portfolio[['short_id','duration_hrs']].values.tolist()) # map is required for transcript data

# Additional: Create list of offers to automate analysis of impact of offers on different customer segment categories

# In[493]:

# generate list for portfolio.short_id column
offer_list_shortid = portfolio['short_id'].values.tolist()

# remove informational
offer_list_shortid.remove('info0300')
offer_list_shortid.remove('info0400')

# sort to group similiar offers
offer_list_shortid.sort()

# ## 3.2 Profile
# - Create short version of id (user_id)
# - Address null records: Keep (Impute) or Remove (Delete)
# - Attempt a correction for error in age (118). Most likely a data entry error for 18
# - Convert "became_member_on" to date

# Create short version of id (user_id)

# In[494]:

# Create short version of id (user_id)
def id_mapper():
    coded_dict = dict()
    cter = 1
    id_encoded = []

    for val in profile['id']:
        if val not in coded_dict:
            coded_dict[val] = cter
            cter+=1

        id_encoded.append(coded_dict[val])
    return id_encoded

id_encoded = id_mapper()
profile['user_id'] = id_encoded

# show header
profile.head(3)

# Address null records: Keep (Impute) or Remove (Delete)

# In[495]:

# Impute Numeric Fields: Fill nans with mean, median, mode
profile['income_meanfill'] = np.round(profile['income'].fillna(profile['income'].mean()),1)
profile['income_medianfill'] = np.round(profile['income'].fillna(profile['income'].median()),1)
profile['income_modefill'] = np.round(profile['income'].fillna(profile['income'].mode(dropna=True)[0]),1)

# Review impact on distribution (kde) and select most suitable option
fig = plt.figure()
ax = fig.add_subplot(111)
profile['income'].plot(kind='kde', ax=ax)
profile['income_meanfill'].plot(kind='kde', ax=ax, color='red')
profile['income_medianfill'].plot(kind='kde', ax=ax, color='green')
profile['income_modefill'].plot(kind='kde', ax=ax, color='purple')
lines, labels = ax.get_legend_handles_labels()
ax.legend(lines, labels, loc='best')
plt.show()

# In[496]:

# Impute Categorical Fields: Fill nans mode
profile['gender-fill'] = profile['gender'].fillna(profile['gender'].mode()[0])

# Review impact on distribution (kde) and select most suitable option
profile['gender'].hist(alpha=0.5, figsize=(5, 3));
profile['gender-fill'].hist(alpha=0.5, figsize=(5, 3));

# Attempt a correction for error in age (118). Most likely a data entry error for 18

# In[497]:

# Create new age column (age-fill) and impute 118 with mean age (after excluding 118 entries by converting to Nan)

profile['age_fill'] = profile['age'] # create new age column to use and preserve original data field
profile['age_fill'].replace(to_replace=118,value=np.nan, inplace=True) # replace 118 with Nan
profile['age_fill'].fillna(profile['age_fill'].median(), inplace=True) # impute Nan with median
profile[['age','age_fill']].hist(alpha=0.5, figsize=(8, 3)); # plot histogram before and after change

# Convert "became_member_on" to date

# In[498]:

profile['became_member_on'] = pd.to_datetime(profile['became_member_on'], format='%Y%m%d')

# Additional Task: Create mapping for id, user_id

# In[499]:

id_userid_map = profile[['id','user_id']]
map_userid_person = dict(id_userid_map.values.tolist()) # map required to simplify data analysis

# ### Profile Comments: Filling nulls vs Deleting nulls
#
#
# **Background:**
#
# New columns were created for columns with null values inorder to determine the impact of filling nulls. This approach also creates a fall-back in the event the option to delete nulls is considered.
#
# **Observations:**
#
# 1. Median fill appears most suitable for income column as the underlying distribution is skewed (see density plots)
# 2. Replacing 118 with Nan and imputing with the median of age has changed the kurtosis of the distribution.
#
# **Conclusion:**
#
# Overall, the choice of either filling nulls or deleting will depend on the model selected for segmenting customers. The basis for segmentation ( customer behaviour, demographics, psycograhics etc.) will determine which features are relevant. For example, the choice of customer behaviour as basis for segmentation will not require the use income, age and sex columns which are are relevant in the demographics approach.

# ## 3.3 Transcript
# - Unpack value column to extract key and values
# - Apply unpacked value column to transcript dataframe
# - Map person to profile user id (short version).

# Unpack value column to extract keys and values

# In[500]:

# Extract value column into dataframe
value_unpacked = pd.DataFrame(transcript['value'].values.tolist(), index=transcript.index)
value_unpacked['offerid'] = value_unpacked['offer id'].combine_first(value_unpacked['offer_id'])
value_unpacked.drop(['offer id','offer_id'],axis=1, inplace=True)
value_unpacked.head(3)

# Apply unpacked value column to transcript dataframe

# In[501]:

# Applying unpacked value column to transcript dataframe
transcript_unpacked = pd.merge(pd.merge(transcript, value_unpacked, left_index=True, right_index=True),id_userid_map,how='left',left_on='person',right_on='id')
transcript_unpacked.drop(['value'],axis=1, inplace=True)
transcript_unpacked.drop(['id'],axis=1, inplace=True)
transcript_unpacked.head(3)

# Map person to profile user id (short version).

# In[502]:

# Map person to profile user id (short version).
transcript_unpacked['short_id'] = transcript_unpacked['offerid'].map(map_id_shortid)
transcript_unpacked['duration_hrs'] = transcript_unpacked['short_id'].map(map_shortid_hrs)
transcript_unpacked.fillna(0, inplace=True)
transcript_unpacked.head(5)

#

# [back to top menu](#toplevel)

# # 4.0 Analysis
# - Background information for context
# - Transaction Labelling
# - PCA (dimension reduction)
# - K-means clustering

# ## 4.1 Background information for context

# Key Ideas:
#
# 1. There are 2 states to customer behaviour: offer-induced ("excited") state and steady ("uninfluenced") state.
#
#
# 2. Categorizing customer transactions by influence type - bogo, discount, self - and establishing the relative contribution of each category provides a quantitative definition of individual customer preferences/behaviour (Transaction Labelling).
#
#
# 3. Transactions are said to be influenced by an offer when these transactions occur after a valid offer is received and viewed. In the event a transaction is completed before an offer is viewed, it is deemed to be uninfluenced (self). In addition, transactions completed outside the influence of any offer are deemed to be "uninfluenced" and labelled "self" (Transaction labelling)
#
#
# 4. Informationals do not influence transactions because these do not have a motivator/reward (assumption)
#
#
# 5. Customer behaviour can be used as a basis to segment customer base in addition to the use of demographic, psychograhic and other factors. There is a need to identify the most suitable factors (PCA and Clustering).
#
#
# 6. Count and value of transactions completed under the influence of an offer provide a measure of the impact of the offer. These measures can be reviewed at individual and segment levels (Impact of offer - incremental transactions and revenue)
#

# ## 4.2 Transaction Labelling

# Data Preparation

# In[503]:

# Creating new columns in preparation for transaction labelling
transcript_unpacked = transcript_unpacked.drop(columns=['person','offerid']) # dropping redundant columns to streamline
transcript_unpacked['influence'] = ''    # required to capture name of influencing offer
transcript_unpacked['completiontype'] = '' # required to identify offer completion which occur after offer view
transcript_unpacked['time_btw_offers'] = 0.0 # required to comment on frequency/volume of offers in test
transcript_unpacked['active_offers'] = '' # required to evaluate occurence and impact of multiple active offers

# Sort dataframe by userid and time to create blocks of user-specific, sequenced transactions
transcript_unpacked.sort_values(by=['user_id','time'], ascending=True, inplace=True)
transcript_unpacked.reset_index(inplace=True, drop=True)

# Labelling Transactions

# In[504]:

# Iterating through txn history (transcript) and labeling relevant fields
# ***********************************************************************

userid = "" # initializing user id which is necessary to track user transactions

# Looping through modified transcript dataframe to label transactions based on influencing offer

for index, row in transcript_unpacked.iterrows():

    if row['user_id'] == userid:

            if row['event'] == 'offer received':
                transcript_unpacked.at[index, 'time_btw_offers'] =  row['time'] - txntime # time between offers

                if ('info' in offerid) or ('info' in row['short_id']):
                    transcript_unpacked.at[index, 'active_offers'] = ''
                else:
                    if (row['time'] - txntime) - offer_duration < 0: # remember to exclude info
                        if offercomplete == 0:
                            transcript_unpacked.at[index, 'active_offers'] = '2+'

                offerid = row['short_id']
                txntime = row['time']
                offer_duration = row['duration_hrs']
                offercomplete == 0

            if row['event'] == 'offer viewed':
                if row['short_id'] == offerid:
                    if ('info' in offerid):
                        influencedby = ""
                    else:
                        if row['time'] - txntime < offer_duration:
                            influencedby = offerid

            if row['event'] == 'transaction':
                if row['time'] - txntime < offer_duration: #transaction is within the period of an offer
                    if len(influencedby) != 0: #test to determine if offer has been viewed
                        transcript_unpacked.at[index, 'influence'] = influencedby
                    else:
                        transcript_unpacked.at[index, 'influence'] = 'self'
                else:
                    transcript_unpacked.at[index, 'influence'] = 'self'

            if row['event'] == 'offer completed':
                if row['short_id'] == offerid:
                    if len(influencedby) == 0: #test to determine if offer has been viewed
                        transcript_unpacked.at[index, 'completiontype'] = 'offer: not viewed'
                    else:
                        transcript_unpacked.at[index, 'completiontype'] = 'offer:viewed'
                    influencedby = ""
                    offercomplete = 1
    else:

        userid = row['user_id']
        offerid = ""
        txntime = 0
        offer_duration = 0
        offercomplete = 0
        influencedby = ""

        if row['event'] == 'offer received':
            offerid = row['short_id']
            txntime = row['time']
            offer_duration = row['duration_hrs']
            offercomplete == 0

        if row['event'] == 'transaction':
            transcript_unpacked.at[index, 'influence'] = 'self'

# In[505]:

# Displaying updates made (1st 10 records)
transcript_unpacked.head(10)

# In[506]:

# Displaying updates made (last 10 records)
transcript_unpacked.tail(10)

# customer-offer interaction matrix

# In[507]:

# Create customer-offer interaction matrix (view into customer preferences/behaviour)

df_extract_txns = transcript_unpacked[transcript_unpacked.event=='transaction'] # extract transactions
cust_offer_matrix = df_extract_txns.groupby(by=['user_id','influence'])['amount'].sum().unstack() # matrix by spend
df_cust_offer_nonull = cust_offer_matrix.fillna(0) # create a copy of customer-offer interaction matrix and fill null
df_cust_offer_nonull.head(10)

# Interpretation:
#
# - user 1 is a bargain hunter who only purchases with discount offers. Preference is disc0710, disc1010 and disc1020
# - user 2 is not influenced by bogo and discount offers. Self motivated
# - user 3 is both self motivated and interested in bogo offers. Preference is bogo0705

# In[508]:

# Identify customers with no transactions across self, bogo and discount

cust_with_txns = df_cust_offer_nonull.shape[0] # customers with at least 1 txn across self and offers
total_cust_base = transcript_unpacked.user_id.nunique() # total number of customers in sample

print("number of customers with at least 1 transaction: {:,}".format(cust_with_txns))
print("number of customers in sample: {:,}".format(total_cust_base))
print("number of customers without transactions: {:,}".format((total_cust_base-cust_with_txns)))

# Comment(s):
#
# There are 422 customers who neither responded to the offers nor performed any transaction within the test period. Their user ids are not captured in the user-offer matrix and their preferences are yet unknown. However, with the addition of relevant demographics-related columns from the profile dataframe to create a comprehensive reference table/dataframe, the customer user ids will reflect in the matrix (no nulls version) with 0 across offer/self columns
#

# Additional tasks: Create a comprehensive reference dataframe by adding relevant columns to user-interaction dataframe

# In[509]:

# Transaction count per customer
df_cust_txnCount = df_extract_txns[df_extract_txns.influence=='self'].groupby(['user_id']).agg(txn_count=('event',pd.Series.count)).reset_index()

# In[510]:

# Profile dataframe: Create column for number of months as member
cutoff_date = pd.to_datetime('20210618',format='%Y%m%d')
profile['membership_mnths'] = ((cutoff_date - profile['became_member_on'])/np.timedelta64(1,'M')).astype(int)

# In[511]:

# Profile dataframe: Extract relevant fields - userid, age, income (median fill), months as member
cust_profile_trim = profile[['user_id','age_fill','income_medianfill','membership_mnths','gender-fill']]

# In[512]:

# join profile trim with count of transactions
cust_profile_txn = cust_profile_trim.merge(df_cust_txnCount, how='left', on='user_id')

# In[513]:

# create master dataframe with demographics and customer-offer interaction details
cust_profile_txn_matrix = cust_profile_txn.merge(df_cust_offer_nonull, how='left',on='user_id')

# In[514]:

# cleaning master dataframe
cust_profile_txn_matrix.set_index('user_id',drop=True,inplace=True) # set user_id as index
cust_profile_txn_matrix.fillna(0, inplace=True) # fill nulls

# In[515]:

# Creating subtotals for bogo and discount offers
cust_profile_txn_matrix['bogo'] = (cust_profile_txn_matrix['bogo0505']
                                        +cust_profile_txn_matrix['bogo0510']
                                        +cust_profile_txn_matrix['bogo0705']
                                        +cust_profile_txn_matrix['bogo0710'])

cust_profile_txn_matrix['discount'] = (cust_profile_txn_matrix['disc0707']
                                            +cust_profile_txn_matrix['disc0710']
                                            +cust_profile_txn_matrix['disc1010']
                                            +cust_profile_txn_matrix['disc1020'])

# In[516]:

# create categories for membership. Business rule (Assumption) - Members are new for 3 months after joining

membership_filter = lambda x: "new" if x <= 37 else "old"
cust_profile_txn_matrix['custtype'] = cust_profile_txn_matrix['membership_mnths'].apply(membership_filter)

#comments:
#********
#The latest membership date is 34 months from cutoff date. 34mnths + 3mnths (37) will filter out new members
# Choice of 3 months was based on optimization. 3 months returned highest silhouette score (0.7456)
# compared to 6 (0.6881), 12 (0.6473) etc

# In[517]:

# view master reference dataframe
cust_profile_txn_matrix.head(3)

# In[518]:

# understand master reference dataframe
cust_profile_txn_matrix.describe()

# In[519]:

cust_profile_txn_matrix.corr()

# # 4.2 PCA (Dimension Reduction)

# In[520]:

cust_profile_txn_matrix.columns

# The columns of the cust_profile_txn_matrix dataframe represent the different factors (features) which could be used to segment the sample customer base. The preferred factors would depend on:
# - choice of basis for segmentation: demograhics, customer behaviour etc.
# - relative strengths of individual factors as determined by PCA analysis
#
# To facilitate decision making, 2 segmentation scenarios would be reviewed. Scenario 1 will review the use of demographic factors while Scenario 2 will explore the use of customer behavior related factors. Both will be reviewed and the most suitable scenario adopted for the rest of the analysis

# Definitions

# In[521]:

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# In[522]:

def showkeycomponents(df):
    '''
    INPUT:
    df - (pandas dataframe) with features for principal component analysis

    OUTPUT:
    PCA_components - (pandas dataframe) transformed data of principal components
    '''
    X_std = StandardScaler().fit_transform(df.values)
    pca = PCA(n_components=0.90) # return components that explain 90% of variance in data
    principalComponents = pca.fit_transform(X_std)
    PCA_components = pd.DataFrame(principalComponents)
    features = range(pca.n_components_)
    plt.bar(features, pca.explained_variance_ratio_, color='black')
    plt.plot(features, pca.explained_variance_ratio_.cumsum(), color='blue', linestyle='--', marker ='o')
    plt.grid(True)
    return PCA_components

# In[523]:

def threeDplot(df):
    '''
    INPUT:
    df - (pandas dataframe) with features for scatter plot

    OUTPUT:

    '''
    fig = plt.figure(figsize = (5, 5))
    ax = plt.axes(projection ="3d")

    # Creating plot
    ax.scatter3D(df[0],df[1],df[2], color = "black")
    plt.title("3D scatter plot")
    ax.set_xlabel('X Label')
    ax.set_ylabel('Y Label')
    ax.set_zlabel('Z Label')
    ax.view_init(30, 60)

    # show plot
    plt.show()
    return None

# One Hot Encoding for categorical Columns/Features

# In[524]:

# transformation for gender and membership class
df_temp = pd.get_dummies((pd.get_dummies(cust_profile_txn_matrix, columns=['gender-fill'])), columns=['custtype'])

# **SCENARIO 1: Segmentation Using Demograhics Data + Membership Class (Income, Age, Sex, Membership Type)**

# In[525]:

# Features list and resulting dataframe
features_list = ['income_medianfill','age_fill','custtype_new','custtype_old','gender-fill_F','gender-fill_M','gender-fill_O']
df_cust_test = df_temp[features_list]
df_cust_test.shape

# In[526]:

# Call PCA showkeycomponents
analysis_demographics = showkeycomponents(df_cust_test)

# In[527]:

# Data dimension before and after PCA for Experiment 1 (Segmenting Using Demographics Data)
print("Dataframe dimension before PCA: {}".format(df_cust_test.shape))
print("Dataframe dimension after PCA: {}".format(analysis_demographics.shape))

# In[528]:

# 3D plot of top 3 components
threeDplot(analysis_demographics)

# **SCENARIO 2: Segmentation Using Customer Bahaviour (Self, Bogo, Discount, Membership Class)**

# In[529]:

# Features list and resulting dataframe
features_list =['self','bogo','discount','custtype_new','custtype_old']
df_cust_test = df_temp[features_list]
df_cust_test.shape

# In[530]:

# Call PCA showkeycomponents
analysis_cust_behave = showkeycomponents(df_cust_test)

# In[531]:

# Data dimension before and after PCA for Experiment 1 (Segmenting Using Demographics Data)
print("Dataframe dimension before PCA: {}".format(df_cust_test.shape))
print("Dataframe dimension after PCA: {}".format(analysis_cust_behave.shape))

# In[532]:

# 3D plot of top 3 components
threeDplot(analysis_cust_behave)

# Comment(s):
#
# Scenario 1
# - requires 5 components to describe 90% of the variance in data
# - 3D plot of top 3 components suggests the presence of distinct clusters
#
# Scenario 2
# - requires 4 components to describe 90% of the variance in data
# - 3D plot of top 3 components suggests the presence of distinct clusters

# # 4.3 K-means Clustering

# Definitions

# In[533]:

from sklearn.preprocessing import MaxAbsScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

# In[534]:

def normalize_data(df_to_normalize, no_of_components):
    '''
    INPUT:
    df_to_normalize - (pandas dataframe) with features requiring standardization
    no_of_components - (int)

    OUTPUT:
    df_normalized - (pandas dataframe) with standardized data using MaxAbsScaler
    '''
    scaler = MaxAbsScaler()
    scaler.fit(df_to_normalize)
    df_normalized = pd.DataFrame(scaler.transform(df_to_normalize), columns=df_to_normalize.columns)
    return df_normalized.iloc[:,:no_of_components]

# In[535]:

def kmeans_cluster_no(df_normalized, components, max_no_cluster=20):
    '''
    INPUT:
    df_to_normalize - (pandas dataframe) with features requiring standardization
    no_of_components - (int) number of features to use in function
    max_no_cluster - (int) number of clusters to iterate to

    OUTPUT:
    None
    '''
    wcss = []
    X = df_normalized.iloc[:,:components].values
    for i in range(1, max_no_cluster):
        kmeans = KMeans(n_clusters=i, init='k-means++', random_state=0)
        kmeans.fit(X)
        wcss.append(kmeans.inertia_)
    plt.plot(range(1,max_no_cluster),wcss)
    plt.xlabel('number of clusters')
    plt.ylabel('wcss values')
    plt.title('The Elbow Method')
    plt.grid(True)
    plt.show()
    return None

# In[536]:

def silhouette_cluster_no(df_normalized, max_no_cluster=10):
    '''
    INPUT:
    df_to_normalize - (pandas dataframe) with features requiring standardization
    max_no_cluster - (int) number of clusters required

    OUTPUT:
    num_cluster - (int) number of segments based on highest silhouette score
    '''

    range_n_clusters = list (range(2,max_no_cluster))
    cluster_scores = []

    for n_clusters in range_n_clusters:
        clusterer = KMeans(n_clusters=n_clusters,init='k-means++', random_state=0)
        preds = clusterer.fit_predict(df_normalized)
        centers = clusterer.cluster_centers_
        score = silhouette_score(df_normalized, preds)

        cluster_scores.append(score)

        print("For n_clusters = {}, silhouette score is {})".format(n_clusters, score))

    num_cluster = np.argmax(np.array(cluster_scores)) + 2
    print("Recommended number of clusters: {}".format(num_cluster))

    return num_cluster

# In[537]:

def cluster_data(df_normalized, cluster_num):
    '''
    INPUT:
    df_to_normalize - (pandas dataframe) with features requiring standardization
    cluster_num - (int) number of clusters required

    OUTPUT:
    kcategories - (series) index values with cluster number assigned
    '''
    kmeansmodel = KMeans(n_clusters=cluster_num, init='k-means++', random_state=0)
    y_kmeans = kmeansmodel.fit_predict(df_normalized)
    kcategories = y_kmeans
    return kcategories

# **SCENARIO 1: Segmentation Using Demograhics Data + Membership Class (Income, Age, Sex, Membership Type)**

# In[538]:

# Identify Number of Clusters: Elbow Method
kmeans_cluster_no(df_normalized=analysis_demographics,components=analysis_demographics.shape[1],max_no_cluster=15)

# In[539]:

# Identify Number of Clusters: Silhouette Method
cluster_no_s1 = silhouette_cluster_no(df_normalized=analysis_demographics)

# **SCENARIO 2: Segmentation Using Customer Bahaviour (Self, Bogo, Discount, Membership Class)**

# In[540]:

# Identify Number of Clusters: Elbow Method
kmeans_cluster_no(df_normalized=analysis_cust_behave,components=analysis_cust_behave.shape[1],max_no_cluster=15)

# In[541]:

# Identify Number of Clusters: Silhouette Method
cluster_no_s2 = silhouette_cluster_no(df_normalized=analysis_cust_behave)

# **Segmentation: Demograhics vs Customer Behaviour**
#
# - Both methods result in four(4)/ five(5) clusters though customer behaviour has a higher silhouette score (0.7456) compared to demograhics (0.6234). By definition, the closer the score is to 1, the better.
#
#
# - The use of demograhics data (age, income, sex etc.) required extensive data cleaning/manipulation to address for nulls. These impacted the underlying distribution of original data as well as accuracy of the clustering/segmentation task. In addition, the dependence on customers to provide these data is a weakness of this approach as changes in certain factors (income, sex etc.) is usually not communicated in real time or at all. Overall, the use of demographic data will create a static and most likely obsolete segmentation which might work for this test but would be unable to adapt quickly to changes in factors.
#
#
# - The use of customer behaviour data addresses the concerns of using demographics data to segment. Willingness to spend and preferences are better factors for segmenting customers especially for profit-seeking businesses like Starbucks. In addition, data required is sourced from business systems and returns with no nulls. Most importantly, this approach is more responsive as changes in customer behaviour are inferred from transactions and eliminates need to depend on customers for key information.
#
#
# - Finally, in the event of new customers who refuse to supply income/age/sex information, the demographics based model will fail while the customer behaviour based model will not as customer transactions will suffice.
#

# **Conclusion**
#
# Scenario 2 is the preferred solution and customer behaviour will be the basis for segmenting sample customer base

#

# # 5.0 Insights

# [back to top menu](#toplevel)

# # 5.1 Mapping users/customers to categories

# In[542]:

# mapping users to categories

# cluster categories
cluster_category = cluster_data(df_normalized=analysis_cust_behave, cluster_num = cluster_no_s2)

# Assign to matrix
cust_profile_txn_matrix = df_temp
cust_profile_txn_matrix['category']=cluster_category
cust_profile_txn_matrix.head(3)

# ## 5.2 Basic Cluster Characteristics

# In[543]:

for cluster in range(cluster_no_s2):
    df_sample = cust_profile_txn_matrix[cust_profile_txn_matrix.category==cluster][features_list]
    key_par = ['count','min','mean','max']
    print("Summary: Cluster {}".format(cluster))
    print(round(df_sample.describe().loc[key_par],1))
    print("\n")

# **Explanation:**
#
# - Category 0: Old customers with low average spend (sub 100) across self, bogo and discount.
# - Category 1: New customers with low average spend (sub 100) across self, bogo and discount.
# - Category 2: Customers (new and old) with high average spend (600 range) on Discount offers (clear preference for Discount).
# - Category 3: Customers (new and old) with high average spend (600 range) on Self (not influenced by offers).
# - Category 4: Customers (new and old) with high average spend (600 range) on BOGO (clear preference for BOGO).
# - High Value Customers: Category 2, 3, 4
# - Mass Market Customers: Category 0, 1

# # 5.3 Offer Performance Within Categories

# Data Processing for Analysis

# In[544]:

#create mapping for userid and cluster category for use on transcript dataframe
userid = cust_profile_txn_matrix.index.values
category_no = cust_profile_txn_matrix.category.values
map_userid_catno = dict(zip(userid,category_no))

#labelling transactions performed by users in transcript with category values
transcript_unpacked['category'] = transcript_unpacked['user_id'].map(map_userid_catno)
transcript_unpacked.head(3)

# Defining Relevant Functions

# Cluster Level Measures

# In[545]:

def category_size(df=cust_profile_txn_matrix):
    '''
    INPUT:
    df - (pandas dataframe) with customer profile and transaction data

    OUTPUT:
    df_temp_grp - (series) number of customers per cluster/category
    '''
    df_temp_grp = df.groupby(['category'])['age_fill'].count()

    return df_temp_grp

# In[546]:

def received_offer(offerid, txnhist=transcript_unpacked, event='offer received'):
    '''
    INPUT:
    offerid - (string) short_id of promotional offer
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (series) number of customers that received offer within each cluster
    '''
    df_temp = txnhist[(txnhist['short_id']==offerid) & (txnhist['event']==event)]

    df_temp_grp = df_temp.groupby(['category'])['user_id'].count()

    return df_temp_grp

# In[547]:

def viewd_offer(offerid, txnhist=transcript_unpacked, event='offer viewed'):
    '''
    INPUT:
    offerid - (string) short_id of promotional offer
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (series) number of customers by clusters that viewed an offer
    '''
    df_temp = txnhist[(txnhist['short_id']==offerid) & (txnhist['event']==event)]
    df_temp_grp = df_temp.groupby(['category'])['user_id'].count()
    return df_temp_grp

# In[548]:

def viewd_complete(offerid, txnhist=transcript_unpacked, event='offer:viewed'):
    '''
    INPUT:
    offerid - (string) short_id of promotional offer
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (series) number of customers by clusters that completed an offer after viewing
    '''
    df_temp = txnhist[(txnhist['short_id']==offerid) & (txnhist['completiontype']==event)]
    df_temp_grp = df_temp.groupby(['category'])['user_id'].count()
    return df_temp_grp

# In[549]:

def no_txns(offerid, txnhist=transcript_unpacked, event='transaction'):
    '''
    INPUT:
    offerid - (string) short_id of promotional offer
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (series) number of transactions by clusters influenced by offer
    '''
    df_temp = txnhist[(txnhist['influence']==offerid) & (txnhist['event']==event)]
    df_temp_grp = df_temp.groupby(['category'])['user_id'].count()
    return df_temp_grp

# In[550]:

def txns_value(offerid, txnhist=transcript_unpacked, event='transaction'):
    '''
    INPUT:
    offerid - (string) short_id of promotional offer
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (series) value of transactions by clusters influenced by offer
    '''
    df_temp = txnhist[(txnhist['influence']==offerid) & (txnhist['event']==event)]
    df_temp_grp = df_temp.groupby(['category'])['amount'].sum()
    return df_temp_grp

# In[551]:

def cat_results(offer_short_id):
    '''
    INPUT:
    offer_short_id - (string) short_id of promotional offer

    OUTPUT:
    df_all - (pandas dataframe) with descriptive statistics for offer
    '''
    df_cat_size = pd.DataFrame(category_size())
    df_cat_size.rename(columns = {'age_fill':'cat_pop'},inplace=True)

    df_rec_offer = pd.DataFrame(received_offer(offer_short_id))
    df_rec_offer.rename(columns = {'user_id':'recv'},inplace=True)

    df_viewd_offer = pd.DataFrame(viewd_offer(offer_short_id))
    df_viewd_offer.rename(columns = {'user_id':'view'},inplace=True)

    df_viewd_complete = pd.DataFrame(viewd_complete(offer_short_id))
    df_viewd_complete.rename(columns = {'user_id':'compltn'},inplace=True)

    df_no_txns = pd.DataFrame(no_txns(offer_short_id))
    df_no_txns.rename(columns = {'user_id':'notxns'},inplace=True)

    df_txns_value = pd.DataFrame(txns_value(offer_short_id))
    df_txns_value.rename(columns = {'amount':'txnval'},inplace=True)

    df_all = pd.concat([df_cat_size, df_rec_offer,df_viewd_offer,df_viewd_complete,df_no_txns,df_txns_value], axis=1)
    df_all['avgtxnval'] = round((df_all['txnval']/df_all['notxns']),2)
    df_all['%compltn'] = round(100*(df_all['compltn']/df_all['recv']),2)

    return df_all

# Total Measures

# In[552]:

def offer_txns(txnhist=transcript_unpacked, event='transaction'):
    '''
    INPUT:
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (pandas dataframe) total value of transactions influenced by offer
    '''

    df_temp = txnhist[txnhist['event']==event]
    df_temp_grp = pd.DataFrame(df_temp.groupby(['influence'])['event'].count())
    df_temp_grp.rename(columns={'event':'offer_txn_count'},inplace=True)
    return df_temp_grp

# In[553]:

def offer_revenue(txnhist=transcript_unpacked, event='transaction'):
    '''
    INPUT:
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (pandas dataframe) total value of transactions influenced by offer
    '''
    df_temp = txnhist[txnhist['event']==event]
    df_temp_grp = pd.DataFrame(df_temp.groupby(['influence'])['amount'].sum())
    df_temp_grp.rename(columns = {'amount':'txn_value'},inplace=True)
    return df_temp_grp

# In[554]:

def offer_payout(txnhist=transcript_unpacked, event='offer completed'):
    '''
    INPUT:
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (pandas dataframe) total value of payouts on offer completion
    '''
    df_temp = txnhist[txnhist['event']==event]
    df_temp_grp = pd.DataFrame(df_temp.groupby(['short_id'])['reward'].sum())
    df_temp_grp.rename(columns = {'reward':'reward_payout'},inplace=True)
    return df_temp_grp

# In[555]:

def received_offer_total(txnhist=transcript_unpacked, event='offer received'):
    '''
    INPUT:
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (pandas dataframe) total number of offers sent/received
    '''
    df_temp = txnhist[txnhist['event']==event]
    df_temp_grp = pd.DataFrame(df_temp.groupby(['short_id'])['user_id'].count())
    df_temp_grp.rename(columns = {'user_id':'num_offer_sent'},inplace=True)
    return df_temp_grp

# In[556]:

def viewd_complete_total(txnhist=transcript_unpacked, event='offer:viewed'):
    '''
    INPUT:
    txnhist - (pandas dataframe) customer transaction history
    event - (string) event type

    OUTPUT:
    df_temp_grp - (pandas dataframe) total number of offers completed after viewing
    '''
    df_temp = txnhist[txnhist['completiontype']==event]
    df_temp_grp = pd.DataFrame(df_temp.groupby(['short_id'])['user_id'].count())
    df_temp_grp.rename(columns = {'user_id':'num_offers_completed'},inplace=True)
    return df_temp_grp

# In[557]:

def offer_results():
    '''
    INPUT:

    OUTPUT:
    df_all - (pandas dataframe) with descriptive statistics for offer
    '''

    df_offer_txn = pd.DataFrame(offer_txns())
    df_offer_revenue = pd.DataFrame(offer_revenue())
    df_offer_payout = pd.DataFrame(offer_payout())

    df_offer_view = pd.DataFrame(received_offer_total())
    df_offer_compltd = pd.DataFrame(viewd_complete_total())

    df_all = pd.concat([df_offer_view, df_offer_compltd, df_offer_txn,df_offer_revenue,df_offer_payout], axis=1)
    df_all.drop(index=['info0300','info0400'], inplace=True)

    df_all['net_txn_value'] = df_all['txn_value'] - df_all['reward_payout']
    df_all['payout_ratio'] = round((df_all['reward_payout']/df_all['txn_value']),2)
    df_all['percent_completn'] = round(100*(df_all['num_offers_completed']/df_all['num_offer_sent']),2)

    df_all.fillna(0, inplace=True)

    return df_all

# In[558]:

total_offer_perform = offer_results()
total_offer_perform

# In[559]:

#sorting
total_offer_perform.sort_values(by=['txn_value'], ascending=False)

# Comments:
# - Top 3 offers based on transaction value: disc1010; bogo0710; disc0707
#
#
# - Offers have been ranked by transaction value because the real aim of promotional offers is to incentivise customers to transact more. "Number of offers completed" as a measure of the success of a promotional offer is inadequate as it does not capture the real value to Starbucks e.g. disc0707 has a higher completion rate but lower transaction value (revenue) compared to disc1010. When you factor reward payouts, the higher completion rates reduces the net transaction value (net revenue) a lot more for disc0707 compared to disc1010
#
#
# - Self represents expected transaction numbers and value in the absence of any offers
#

# In[560]:

print("Description for Performance Measures")
print("--------------------------")
print("cat_pop: number of customers in category")
print("recv: number of customers that received offer")
print("view: number of customers that viewed offer")
print("compltn: number of customers that completed offer after viewing")
print("notxns: number of transactions influenced by offer")
print("txnval: monetary value of transactions")
print("%compltn: percentage of customers who completed offer after viewing")

# In[561]:

# Generate performance measures for offer across categories
for offer in offer_list_shortid:
    df_temp = cat_results(offer)
    print("category results for {}:".format(offer))
    print("------------------------------")
    print(df_temp)
    print("\n")

# **General Trends**

# - Offers which have difficulty value = duration are completed at higher rates than other offers. for example, bogo0505, disc0707, disc1010
#
# - Low difficulty/short duration offers generate more incremental transaction value as well as completion rates compared to high difficulty/long duration offers
#
# - Value of transactions influenced by an offer is a better measure of performance than completion rates (see impact of mass market categories 0, 1 with relatively low completion rates but high number of small transactions and total transaction value).

# **Category-Offer Ranking Table (Descending Order of Txv Value)**
#
# - Category 0: disc1010; bogo0710; disc0707; bogo0510; bogo0505
#
# - Category 1: disc1010; bogo0710; disc0707; bogo0505; bogo0510
#
# - Category 2: disc1010; disc0707; disc1020; disc0710; bogo0710
#
# - Category 3: disc1010; bogo0710; bogo0510; bogo0505; disc0707
#
# - Category 4: bogo0505; bogo0510; bogo0710; bogo0705; disc1010

# ## Review of Starbucks Experiment: Concern Areas

# Categorization of offer completion into offer-viewed and offer-not-viewed

# In[562]:

# Categorization of offer completion into offer-viewed and offer-not-viewed
transcript_unpacked[transcript_unpacked['event']=='offer completed'].tail(5)

# In[563]:

# Total count of offer completions which happened without offer viewing
transcript_unpacked[transcript_unpacked['completiontype']=='offer: not viewed'].shape[0]

# In[564]:

# Total value of reward paid on offer completions which happened without offer viewing
transcript_unpacked[transcript_unpacked['completiontype']=='offer: not viewed']['reward'].sum()

# Multiple Active Offers

# In[565]:

# Total number of cases with 2+ active offers
transcript_unpacked[transcript_unpacked['active_offers']=='2+'].shape[0]

# In[566]:

# Total number of offer completions from multiple active offers which were double counted
transcript_unpacked[(transcript_unpacked['event']=='offer completed') & (transcript_unpacked['completiontype']=="")].shape[0]

# In[567]:

# Total value  of reward paid on double counted completions
transcript_unpacked[(transcript_unpacked['event']=='offer completed') & (transcript_unpacked['completiontype']=="")]['reward'].sum()


# coding: utf-8

# # AI4M Course 1 week 3 lecture notebook

#
# # Explore the data
#
#
#
# In this week's assignment, you'll be working with 3D MRI brain scans from the public [Medical Segmentation Decathlon](https://decathlon-10.grand-challenge.org/) challenge project. This is an incredibly rich dataset that provides you with labels associated with each point (voxel) inside a 3D representation of a patient's brain. Ultimately, in this week's assignment, you will train a neural network to make three-dimensional spatial segmentation predictions for common brain disorders.
#
# In this notebook, you're all set up to explore this exciting dataset. Run the code below and tweak it to explore further!

# ### Import packages
# For this lab, you'll import some of the packages you've seen before (`numpy`, `matplotlib` and `seaborn`) as well as some new ones for reading (`nibabel`) and visualizing (`itk`, `itkwidgets`, `ipywidgets`) the data. Run the next cell to import these packages.

# In[13]:

# Import all the necessary packages
import numpy as np
import nibabel as nib
import itk
import itkwidgets
from ipywidgets import interact, interactive, IntSlider, ToggleButtons
import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')
import seaborn as sns
sns.set_style('darkgrid')

# ### Loading Images of the brain
# Run the next cell to grab a single 3D MRI brain scan

# In[14]:

# Define the image path and load the data
image_path = "BraTS-Data/imagesTr/BRATS_001.nii.gz"
image_obj = nib.load(image_path)
print(f'Type of the image {type(image_obj)}')

# ### Extract the data as a numpy array
# Run the next cell to extract the data using the `get_fdata()` method of the image object

# In[15]:

# Extract data as numpy ndarray
image_data = image_obj.get_fdata()
type(image_data)

# In[16]:

# Get the image shape and print it out
height, width, depth, channels = image_data.shape
print(f"The image object has the following dimensions: height: {height}, width:{width}, depth:{depth}, channels:{channels}")

# As you can see these "image objects" are actually 4 dimensional! With the exploratory steps below you'll get a better sense of exactly what each of these dimensions represents.
#
# ### Visualize the data
# The "depth" listed above indicates that there are 155 layers (slices through the brain) in every image object. To visualize a single layer, run the cell below. Note that if the layer is one of the first or the last (`i` near 0 or 154), you won't find much information and the screen will be dark. Run this cell multiple times to look at different layers.
#
# The code is set up to grab a random layer but you can select a specific layer by choosing a value for `i` from 0 to 154. You can also change which channel you're looking at by changing the `channel` variable.
#
# Keep in mind that you could just as easily look at slices of this image object along the height or width dimensions. If you wish to do so, just shift `i` to a different dimension in the `plt.imshow()` command below. Which slice direction looks the most interesting to you?

# In[17]:

# Select random layer number
maxval = 154
i = np.random.randint(0, maxval)
# Define a channel to look at
channel = 3
print(f"Plotting Layer {i} Channel {channel} of Image")
plt.imshow(image_data[: , :, i, channel], cmap='gray')
plt.axis('off');

# ### Interactive exploration
# Another way to visualize this dataset is by using IPython Widgets to allow for an interactive exploration of the data.
#
# Run the next cell to explore across different layers of the data. Move the slider to explore different layers. Change the `channel` value to explore different channels. See if you can tell which layer corresponds to the top of the brain and which corresponds to the bottom!
#
# If you're feeling ambitious, try modifying the code below to slice along a different axis through the image object and look at other channels to see what you can discover!

# In[18]:

# Define a function to visualize the data
def explore_3dimage(layer):
    plt.figure(figsize=(10, 5))
    channel = 3
    plt.imshow(image_data[:, :, layer, channel], cmap='gray');
    plt.title('Explore Layers of Brain MRI', fontsize=20)
    plt.axis('off')
    return layer

# Run the ipywidgets interact() function to explore the data
interact(explore_3dimage, layer=(0, image_data.shape[2] - 1));

# ### Explore the data labels
# In this section, you'll read in a new dataset containing the labels for the MRI scan you loaded above.
#
# Run the cell below to load the labels dataset for the image object you inspected above.

# In[19]:

# Define the data path and load the data
label_path = "./BraTS-Data/labelsTr/BRATS_001.nii.gz"
label_obj = nib.load(label_path)
type(label_obj)

# ### Extract the data as a numpy array
# Run the next cell to extract the data labels using the `get_fdata()` method of the image object

# In[20]:

# Extract data labels
label_array = label_obj.get_fdata()
type(label_array)

# In[21]:

# Extract and print out the shape of the labels data
height, width, depth = label_array.shape
print(f"Dimensions of labels data array height: {height}, width: {width}, depth: {depth}")
print(f'With the unique values: {np.unique(label_array)}')
print("""Corresponding to the following label categories:
0: for normal
1: for edema
2: for non-enhancing tumor
3: for enhancing tumor""")

# ### Visualize the labels for a specific layer
# Run the next cell to visualize a single layer of the labeled data. The code below is set up to show a single layer and you can set `i` to any value from 0 to 154 to look at a different layer.
#
# Note that if you choose a layer near 0 or 154 there might not be much to look at in the images.

# In[22]:

# Define a single layer for plotting
layer = 50
# Define a dictionary of class labels
classes_dict = {
    'Normal': 0.,
    'Edema': 1.,
    'Non-enhancing tumor': 2.,
    'Enhancing tumor': 3.
}
# Set up for plotting
fig, ax = plt.subplots(nrows=1, ncols=4, figsize=(50, 30))
for i in range(4):
    img_label_str = list(classes_dict.keys())[i]
    img = label_array[:,:,layer]
    mask = np.where(img == classes_dict[img_label_str], 255, 0)
    ax[i].imshow(mask)
    ax[i].set_title(f"Layer {layer} for {img_label_str}", fontsize=45)
    ax[i].axis('off')
plt.tight_layout()

# ### Interactive visualization across layers
# As another way of looking at the data, run the code below to create a visualization where you can choose the class you want to look at by clicking a button to choose a particular label and scrolling across layers using the slider!

# In[23]:

# Create button values
select_class = ToggleButtons(
    options=['Normal','Edema', 'Non-enhancing tumor', 'Enhancing tumor'],
    description='Select Class:',
    disabled=False,
    button_style='info',

)
# Create layer slider
select_layer = IntSlider(min=0, max=154, description='Select Layer', continuous_update=False)

# Define a function for plotting images
def plot_image(seg_class, layer):
    print(f"Plotting {layer} Layer Label: {seg_class}")
    img_label = classes_dict[seg_class]
    mask = np.where(label_array[:,:,layer] == img_label, 255, 0)
    plt.figure(figsize=(10,5))
    plt.imshow(mask, cmap='gray')
    plt.axis('off');

# Use the interactive() tool to create the visualization
interactive(plot_image, seg_class=select_class, layer=select_layer)

# #### And there you have it! We hope this lab has helped you get a better sense of the data you'll be working with in this week's assignment.


# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     cell_metadata_json: true
#     comment_magics: false
#     formats: py:light,notebooks//ipynb
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.6.0
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# # Modules
#
# If your Python program gets longer, you may want to split it into several files for easier maintenance. To support this, Python has a way to put definitions in a file and use them in a script or in an interactive instance of the interpreter. Such a file is called a module.

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# Run the cell below to create a file named fibo.py with several functions inside:

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:31.963243Z", "iopub.execute_input": "2020-09-12T13:29:31.964404Z", "iopub.status.idle": "2020-09-12T13:29:31.967533Z", "shell.execute_reply": "2020-09-12T13:29:31.968160Z"}}
%%file fibo.py
""" Simple module with
    two functions to compute Fibonacci series """

def fib1(n):
   """ write Fibonacci series up to n """
   a, b = 0, 1
   while b < n:
      print(b, end=', ')
      a, b = b, a+b

def fib2(n):
    """ return Fibonacci series up to n """
    result = []
    a, b = 0, 1
    while b < n:
        result.append(b)
        a, b = b, a+b
    return result

if __name__ == "__main__":
    import sys
    fib1(int(sys.argv[1]))

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# You can use the function fib by importing fibo which is the name of the file without .py extension.

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:31.972627Z", "iopub.execute_input": "2020-09-12T13:29:31.973839Z", "iopub.status.idle": "2020-09-12T13:29:31.978991Z", "shell.execute_reply": "2020-09-12T13:29:31.978327Z"}}
import fibo
print(fibo.__name__)
print(fibo.__file__)
fibo.fib1(1000)

# + {"execution": {"iopub.status.busy": "2020-09-12T13:29:31.983810Z", "iopub.execute_input": "2020-09-12T13:29:31.985343Z", "iopub.status.idle": "2020-09-12T13:29:31.990318Z", "shell.execute_reply": "2020-09-12T13:29:31.989653Z"}}
%run fibo.py 1000

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:31.995175Z", "iopub.execute_input": "2020-09-12T13:29:31.996128Z", "iopub.status.idle": "2020-09-12T13:29:31.998886Z", "shell.execute_reply": "2020-09-12T13:29:31.999551Z"}}
help(fibo)

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# ## Executing modules as scripts
#
# When you run a Python module with
# ```bash
# $ python fibo.py <arguments>
# ```
# the code in the module will be executed, just as if you imported it, but with the __name__ set to "__main__". The following code will be executed only in this case and not when it is imported.
# ```python
# if __name__ == "__main__":
#     import sys
#     fib(int(sys.argv[1]))
# ```
# In Jupyter notebook, you can run the fibo.py python script using magic command.

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.005047Z", "iopub.execute_input": "2020-09-12T13:29:32.006164Z", "iopub.status.idle": "2020-09-12T13:29:32.009821Z", "shell.execute_reply": "2020-09-12T13:29:32.010715Z"}}
%run fibo.py 1000

# + [markdown] {"slideshow": {"slide_type": "fragment"}}
# The module is also imported.

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.016819Z", "iopub.execute_input": "2020-09-12T13:29:32.017788Z", "iopub.status.idle": "2020-09-12T13:29:32.020332Z", "shell.execute_reply": "2020-09-12T13:29:32.020947Z"}}
fib1(1000)

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# ## Different ways to import a module
# ```python
# import fibo
# import fibo as f
# from fibo import fib1, fib2
# from fibo import *
# ```

# + [markdown] {"slideshow": {"slide_type": "fragment"}}
# - Last command with '*' imports all names except those beginning with an underscore (_). In most cases, do not use this facility since it introduces an unknown set of names into the interpreter, possibly hiding some things you have already defined.

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# - If a function with same name is present in different modules imported. Last module function imported replace the previous one.

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.025313Z", "iopub.execute_input": "2020-09-12T13:29:32.026261Z", "iopub.status.idle": "2020-09-12T13:29:32.179026Z", "shell.execute_reply": "2020-09-12T13:29:32.179636Z"}}
from numpy import sqrt
from scipy import sqrt
sqrt(-1)

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.184726Z", "iopub.execute_input": "2020-09-12T13:29:32.185794Z", "iopub.status.idle": "2020-09-12T13:29:32.189313Z", "shell.execute_reply": "2020-09-12T13:29:32.189893Z"}}
from scipy import sqrt
from numpy import sqrt
sqrt(-1)

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.194542Z", "iopub.execute_input": "2020-09-12T13:29:32.195587Z", "iopub.status.idle": "2020-09-12T13:29:32.199684Z", "shell.execute_reply": "2020-09-12T13:29:32.199040Z"}}
import numpy as np
import scipy as sp

print(np.sqrt(-1+0j), sp.sqrt(-1))

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# - For efficiency reasons, each module is only imported once per interpreter session. Therefore, if you change your modules, you must restart the interpreter
# – If you really want to test interactively after a long run, use :
# ```python
# import importlib
# importlib.reload(modulename)
# ```

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# ## The Module Search Path
#
# When a module is imported, the interpreter searches for a file named module.py in a list of directories given by the variable sys.path.
# - Python programs can modify sys.path
# - export the PYTHONPATH environment variable to change it on your system.

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.204098Z", "iopub.execute_input": "2020-09-12T13:29:32.205177Z", "iopub.status.idle": "2020-09-12T13:29:32.207852Z", "shell.execute_reply": "2020-09-12T13:29:32.208467Z"}}
import sys
sys.path

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.212786Z", "iopub.execute_input": "2020-09-12T13:29:32.213743Z", "iopub.status.idle": "2020-09-12T13:29:32.216521Z", "shell.execute_reply": "2020-09-12T13:29:32.217111Z"}}
import collections
collections.__path__

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# `sys.path` is a list and you can append some directories:

# + {"execution": {"iopub.status.busy": "2020-09-12T13:29:32.221298Z", "iopub.execute_input": "2020-09-12T13:29:32.222227Z", "iopub.status.idle": "2020-09-12T13:29:32.224691Z", "shell.execute_reply": "2020-09-12T13:29:32.225273Z"}}
sys.path.append("/Users/navaro/python-notebooks/")
print(sys.path)

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# When you import a module `foo`, following files are searched in this order:
#
# - **foo.dll**, **foo.dylib** or **foo.so**
# - **foo.py**
# - **foo.pyc**
# - **foo/\_\_init__.py**
#

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# ## Packages
#
# - A package is a directory containing Python module files.
# - This directory always contains a file name \_\_init\_\_.py

# + [markdown] {"slideshow": {"slide_type": "slide"}}
#

# sklearn
# ├── base.py
# ├── calibration.py
# ├── cluster
# │   ├── __init__.py
# │   ├── _kmeans.py
# │   ├── _mean_shift.py
# ├── ensemble
# │   ├── __init__.py
# │   ├── _bagging.py
# │   ├── _forest.py
#

#
# cluster `__init__.py`
#
#

# from ._mean_shift import mean_shift, MeanShift
# from ._kmeans import k_means, KMeans, MiniBatchKMeans
#

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# ## Relative imports
#
# These imports use leading dots to indicate the current and parent packages involved in the relative import. In the sugiton module, you can use:
# ```python
# from . import cluster # import module in the same directory
# from .. import base   # import module in parent directory
# from ..ensemble import _forest # import module in another subdirectory of the parent directory
# ```

# + [markdown] {"slideshow": {"slide_type": "slide"}}
# ## Reminder
#
# Don't forget that importing * is not recommended

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.229868Z", "iopub.execute_input": "2020-09-12T13:29:32.230907Z", "iopub.status.idle": "2020-09-12T13:29:32.233262Z", "shell.execute_reply": "2020-09-12T13:29:32.233875Z"}}
sum(range(5),-1)

# + {"slideshow": {"slide_type": "fragment"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.240382Z", "iopub.execute_input": "2020-09-12T13:29:32.241342Z", "iopub.status.idle": "2020-09-12T13:29:32.243778Z", "shell.execute_reply": "2020-09-12T13:29:32.244478Z"}}
from numpy import *
sum(range(5),-1)

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.249481Z", "iopub.execute_input": "2020-09-12T13:29:32.250399Z", "iopub.status.idle": "2020-09-12T13:29:32.253014Z", "shell.execute_reply": "2020-09-12T13:29:32.253616Z"}}
del sum # delete imported sum function from numpy
help(sum)

# + {"slideshow": {"slide_type": "slide"}, "execution": {"iopub.status.busy": "2020-09-12T13:29:32.258118Z", "iopub.execute_input": "2020-09-12T13:29:32.259095Z", "iopub.status.idle": "2020-09-12T13:29:32.261731Z", "shell.execute_reply": "2020-09-12T13:29:32.262316Z"}}
import numpy as np
help(np.sum)

import numpy,from numpy,import scipy,from scipy


#!/usr/bin/env python
# coding: utf-8

# # 16 - AgriPV - 3-up and 4-up collector optimization
#
#
# This journal helps the exploration of varying collector widths and xgaps in the ground underneath as well as on the rear irradiance for bifacial AgriPV. The optimization varies the numpanels combinations with xgaps for having 3-up and 4-up collectors with varying space along the row (xgap). The actual raytracing is not performed in the jupyter journal but rather on the HPC, but the geometry is the same as presented here.
#
# The steps on this journal:
#

#
  Making Collectors for each number panel and xgap case

#
  Builds the Scene so it can be viewed with rvu

#
#
# An area of 40m x 20 m area is sampled on the HPC, and is highlighted in the visualizations below with an appended terrain of 'litesoil'. The image below shows the two extremes of the variables optimized and the raytrace results, including the worst-case shading experienced under the array ( 100 - min_irradiance *100 / GHI).
#
#
#
# ![AgriPV Collector Width and Xgap Optimization](../images_wiki/AdvancedJournals/AgriPV_CWandXgap_Optimization.PNG)
#

# In[1]:

import os
from pathlib import Path

testfolder = Path().resolve().parent.parent / 'bifacial_radiance' / 'TEMP' /  'Tutorial_16'
if not os.path.exists(testfolder):
    os.makedirs(testfolder)

print ("Your simulation will be stored in %s" % testfolder)

# In[2]:

import bifacial_radiance
import numpy as np

rad_obj = bifacial_radiance.RadianceObj('tutorial_16', str(testfolder))

#

# ## 1. Making Collectors for each number panel and xgap case

# In[3]:

x = 2
y = 1
ygap = 0.1524 # m = 6 in
zgap = 0.002 # m, veyr little gap to torquetube.

tubeParams = {'diameter':0.15,
              'tubetype':'square',
              'material':'Metal_Grey',
              'axisofrotation':True,
               'visible': True}

ft2m = 0.3048
xgaps = [3, 4, 6, 9, 12, 15, 18, 21]
numpanelss = [3, 4]

# Loops
for ii in range(0, len(numpanelss)):
    numpanels = numpanelss[ii]
    for jj in range(0, len(xgaps)):
        xgap = xgaps[jj]*ft2m

        moduletype = 'test-module_'+str(numpanels)+'up_'+str(round(xgap,1))+'xgap'
        rad_obj.makeModule(moduletype,
                    x=x, y=y,
                    xgap=xgap, zgap=zgap, ygap = ygap, numpanels=numpanels,
                    tubeParams=tubeParams)

#

# ## 2. Build the Scene so it can be viewed with rvu

# In[4]:

xgaps = np.round(np.array([3, 4, 6, 9, 12, 15, 18, 21]) * ft2m,1)
numpanelss = [3, 4]
sensorsxs = np.array(list(range(0, 201)))

# Select CASE:
xgap = np.round(xgaps[-1],1)
numpanels = 4

# All the rest

ft2m = 0.3048
hub_height = 8.0 * ft2m
y = 1
pitch = 0.001 # If I recall, it doesn't like when pitch is 0 even if it's a single row, but any value works here.
ygap = 0.15
tilt = 18

sim_name = ('Coffee_'+str(numpanels)+'up_'+
            str(round(xgap,1))+'_xgap')

albedo = 0.35 # Grass value from Torres Molina, "Measuring UHI in Puerto Rico" 18th LACCEI
            # International Multi-Conference for Engineering, Education, and Technology

azimuth = 180
if numpanels == 3:
    nMods = 9
if numpanels == 4:
    nMods = 7
nRows = 1

moduletype = 'test-module_'+str(numpanels)+'up_'+str(round(xgap,1))+'xgap'

rad_obj.setGround(albedo)
lat = 18.202142
lon = -66.759187
metfile = rad_obj.getEPW(lat,lon)
rad_obj.readWeatherFile(metfile)

sceneDict = {'tilt':tilt,'pitch':pitch,'hub_height':hub_height,'azimuth':azimuth, 'nMods': nMods, 'nRows': nRows}
scene = rad_obj.makeScene(module=moduletype,sceneDict=sceneDict,  radname = sim_name)

rad_obj.gendaylit(4020)

octfile = rad_obj.makeOct(filelist = rad_obj.getfilelist(), octname = rad_obj.basename)

name='SampleArea'
text='! genbox litesoil cuteBox 40 20 0.01 | xform -t -20 -10 0.01'
customObject =rad_obj.makeCustomObject(name,text)
rad_obj.appendtoScene(scene.radfiles, customObject, '!xform -rz 0')

octfile = rad_obj.makeOct(rad_obj.getfilelist())

#
# ### To View the generated Scene, you can navigate to the testfolder on a terminal and use:
#
# front view:
# > rvu -vf views\front.vp -e .0265652 -vp 2 -21 2.5 -vd 0 1 0 makemod.oct
#
#  top view:
# > rvu -vf views\front.vp -e .0265652 -vp 5 0 70 -vd 0 0.0001 -1 makemod.oct
#
# ### Or run it directly from Jupyter by removing the comment from the following cell:
#

# In[5]:

## Comment the ! line below to run rvu from the Jupyter notebook instead of your terminal.
## Simulation will stop until you close the rvu window

#!rvu -vf views\front.vp -e .0265652 -vp 2 -21 2.5 -vd 0 1 0 makemod.oct
#!rvu -vf views\front.vp -e .0265652 -vp 5 0 70 -vd 0 0.0001 -1 makemod.oct

docs/tutorials/16 - AgriPV - 3-up and 4-up collector optimization.py


# solutions.py
"""Volume II Lab 18: Conjugate Gradient. Test Driver."""

import numpy as np
from real_solutions import prob2, prob3

def test(student_module):
    """Grade a student's entire solutions file.

    20 points for problem 2.
    20 points for problem 3.

    Inputs:
        student_module: the imported module for the student's file.

    Returns:
        score (int): the student's score, out of TOTAL.
        feedback (str): a printout of test results for the student.
    """
    tester = _testDriver()
    tester.test_all(student_module)
    return tester.score, tester.feedback

class _testDriver(object):
    """Class for testing a student's work.

    Attributes:
        Score (int)
        Feedback (str)
    """
    # Constructor -------------------------------------------------------------
    def __init__(self):
        """Initialize the feedback attribute."""
        self.feedback = ""

    # Main routine -----------------------------------------------------------
    def test_all(self, student_module, total=40):
        """Grade the provided module on each problem and compile feedback."""
        # Reset feedback and score.
        self.feedback = ""
        self.score = 0

        def test_one(problem, number, value):
            """Test a single problem, checking for errors."""
            try:
                self.feedback += "\n\nProblem {} ({} points):".format(
                                                                number, value)
                points = problem(student_module)
                self.score += points
                self.feedback += "\nScore += {}".format(points)
            except BaseException as e:
                self.feedback += "\n{}: {}".format(self._errType(e), e)

        # Grade each problem.
        test_one(self.problem2, 2, 20)   # Problem 1: 20 points.
        test_one(self.problem3, 3, 20)   # Problem 2: 20 points.

        # Report final score.
        percentage = (100. * self.score) / total
        self.feedback += "\n\nTotal score: {}/{} = {}%".format(
                                    self.score, total, round(percentage, 2))
        if   percentage >=  98: self.feedback += "\n\nExcellent!"
        elif percentage >=  90: self.feedback += "\n\nGreat job!"

        # Add comments (optionally).
        print(self.feedback)
        comments = str(raw_input("Comments: "))
        if len(comments) > 0:
            self.feedback += '\n\n\nComments:\n\t{}'.format(comments)

    # Helper Functions --------------------------------------------------------
    @staticmethod
    def _errType(error):
        """Get just the name of the exception 'error' in string format."""
        return str(type(error).__name__)

    def _eqTest(self, correct, student, message):
        """Test to see if 'correct' and 'student' are equal.
        Report the given 'message' if they are not.
        """
        if np.allclose(correct, student, atol=1e-4, rtol=1e-4):
            return 1
        else:
            self.feedback += "\n{}".format(message)
            self.feedback += "\n\tCorrect response: {}".format(correct)
            self.feedback += "\n\tStudent response: {}".format(student)
            return 0

    # Problems ----------------------------------------------------------------
    def problem2(self, s):
        """Test prob2() (linregression problem). 20 points."""

        correct, student = prob2(), s.prob2()
        if type(student) != np.ndarray:
            raise TypeError("Failed to return a NumPy array.")

        return 20 * self._eqTest(correct, student, "Incorrect Answer.")

    def problem3(self, s):
        """Test prob3() (logregression problem). 20 points."""

        correct, student = prob3(), s.prob3()
        if type(student) != np.ndarray:
            raise TypeError("Failed to return a NumPy array.")

        return 20 * self._eqTest(correct, student, "Incorrect Answer.")

# Validation ==================================================================
if __name__ == '__main__':
    """Validate the test driver by testing the solutions file."""
    import solutions
    test(solutions)


# coding: utf-8

# # Self-Driving Car Engineer Nanodegree
#
#
# ## Project: **Finding Lane Lines on the Road**
# ***
# In this project, you will use the tools you learned about in the lesson to identify lane lines on the road.  You can develop your pipeline on a series of individual images, and later apply the result to a video stream (really just a series of images). Check out the video clip "raw-lines-example.mp4" (also contained in this repository) to see what the output should look like after using the helper functions below.
#
# Once you have a result that looks roughly like "raw-lines-example.mp4", you'll need to get creative and try to average and/or extrapolate the line segments you've detected to map out the full extent of the lane lines.  You can see an example of the result you're going for in the video "P1_example.mp4".  Ultimately, you would like to draw just one line for the left side of the lane, and one for the right.
#
# In addition to implementing code, there is a brief writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a [write up template](https://github.com/udacity/CarND-LaneLines-P1/blob/master/writeup_template.md) that can be used to guide the writing process. Completing both the code in the Ipython notebook and the writeup template will cover all of the [rubric points](https://review.udacity.com/#!/rubrics/322/view) for this project.
#
# ---
# Let's have a look at our first image called 'test_images/solidWhiteRight.jpg'.  Run the 2 cells below (hit Shift-Enter or the "play" button above) to display the image.
#
# **Note: If, at any point, you encounter frozen display windows or other confounding issues, you can always start again with a clean slate by going to the "Kernel" menu above and selecting "Restart & Clear Output".**
#
# ---

# **The tools you have are color selection, region of interest selection, grayscaling, Gaussian smoothing, Canny Edge Detection and Hough Tranform line detection.  You  are also free to explore and try other techniques that were not presented in the lesson.  Your goal is piece together a pipeline to detect the line segments in the image, then average/extrapolate them and draw them onto the image for display (as below).  Once you have a working pipeline, try it out on the video stream below.**
#
# ---
#
#

#
#

#

#
 Your output should look something like this (above) after detecting line segments using the helper functions below

#

#

#

#

#
#

#

#
 Your goal is to connect/average/extrapolate line segments to get output like this

#

#

# **Run the cell below to import some packages.  If you get an `import error` for a package you've already installed, try changing your kernel (select the Kernel menu above --> Change Kernel).  Still have problems?  Try relaunching Jupyter Notebook from the terminal prompt.  Also, consult the forums for more troubleshooting tips.**

# ## Import Packages

# In[1]:

#importing some useful packages
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import cv2
get_ipython().run_line_magic('matplotlib', 'inline')

# ## Read in an Image

# In[2]:

#reading in an image
image = mpimg.imread('test_images/solidWhiteRight.jpg')

#printing out some stats and plotting
print('This image is:', type(image), 'with dimensions:', image.shape)

plt.figure(figsize=(20,20))
plt.subplot(1,2,1)
plt.imshow(image)  # if you wanted to show a single color channel image called 'gray', for example, call as plt.imshow(gray, cmap='gray')
gray = cv2.cvtColor(image,cv2.COLOR_RGB2GRAY)

plt.subplot(1,2,2)
plt.imshow(gray, cmap='gray')

print("origin image shape",image.shape)
print("gray image shape",gray.shape)

# ## Ideas for Lane Detection Pipeline

# **Some OpenCV functions (beyond those introduced in the lesson) that might be useful for this project are:**
#
# `cv2.inRange()` for color selection
# `cv2.fillPoly()` for regions selection
# `cv2.line()` to draw lines on an image given endpoints
# `cv2.addWeighted()` to coadd / overlay two images
# `cv2.cvtColor()` to grayscale or change color
# `cv2.imwrite()` to output images to file
# `cv2.bitwise_and()` to apply a mask to an image
#
# **Check out the OpenCV documentation to learn about these and discover even more awesome functionality!**

# ## Helper Functions

# Below are some helper functions to help get you started. They should look familiar from the lesson!

# In[3]:

import math
import os
from collections import deque

images = os.listdir("test_images/")

fig_size = 30
COL = 2

# 转换为灰阶图像
def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    (assuming your grayscaled image is called 'gray')
    you should call plt.imshow(gray, cmap='gray')"""

    return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    # Or use BGR2GRAY if you read an image with cv2.imread()
    # return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# canny 边缘检测
def canny(img, low_threshold, high_threshold):
    """Applies the Canny transform"""
    return cv2.Canny(img, low_threshold, high_threshold)

# 图像平滑：高斯模糊
def gaussian_blur(img, kernel_size):
    """Applies a Gaussian Noise kernel"""
    return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)

# 目标区域选取
def region_of_interest(img, vertices):
    """
    Applies an image mask.

    Only keeps the region of the image defined by the polygon
    formed from `vertices`. The rest of the image is set to black.
    `vertices` should be a numpy array of integer points.
    """
    #defining a blank mask to start with
    mask = np.zeros_like(img)

    #defining a 3 channel or 1 channel color to fill the mask with depending on the input image
    if len(img.shape) > 2:
        channel_count = img.shape[2]  # i.e. 3 or 4 depending on your image
        ignore_mask_color = (255,) * channel_count
    else:
        ignore_mask_color = 255

    #filling pixels inside the polygon defined by "vertices" with the fill color
    cv2.fillPoly(mask, vertices, ignore_mask_color)
    #returning the image only where mask pixels are nonzero
    masked_image = cv2.bitwise_and(img, mask)

    return masked_image,mask

def draw_lines(img, lines, color=[0, 255, 0], thickness=4):
    """
    NOTE: this is the function you might want to use as a starting point once you want to
    average/extrapolate the line segments you detect to map out the full
    extent of the lane (going from the result shown in raw-lines-example.mp4
    to that shown in P1_example.mp4).

    Think about things like separating line segments by their
    slope ((y2-y1)/(x2-x1)) to decide which segments are part of the left
    line vs. the right line.  Then, you can average the position of each of
    the lines and extrapolate to the top and bottom of the lane.

    This function draws `lines` with `color` and `thickness`.
    Lines are drawn on the image inplace (mutates the image).
    If you want to make the lines semi-transparent, think about combining
    this function with the weighted_img() function below
    """
    #fit_left = np.polyfit((left_bottom[0], apex[0]), (left_bottom[1], apex[1]), 1)

    for line in lines:
        for x1,y1,x2,y2 in line:
            slope = (y2-y1)/(x2-x1)
            if abs(slope) < 0.4 or abs(slope)>1:
                continue
            cv2.line(img, (x1, y1), (x2, y2), color, thickness)

def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap):
    """
    `img` should be the output of a Canny transform.

    Returns an image with hough lines drawn.
    """
    lines = cv2.HoughLinesP(img, rho, theta, threshold, np.array([]), minLineLength=min_line_len, maxLineGap=max_line_gap)
    line_img = np.zeros((*img.shape, 3), dtype=np.uint8)
    #draw_lines(line_img, lines)
    draw_lane_lines(line_img, lines)
    return line_img,lines

def weighted_img(img, initial_img, α=0.8, β=1., γ=0.):
    """
    `img` is the output of the hough_lines(), An image with lines drawn on it.
    Should be a blank image (all black) with lines drawn on it.

    `initial_img` should be the image before any processing.

    The result image is computed as follows:

    initial_img * α + img * β + γ
    NOTE: initial_img and img must be the same shape!
    """
    return cv2.addWeighted(initial_img, α, img, β, γ)

def draw_imgs(imgs,vertices=None):
    """
    show images as subplots
    """
    #print("==",imgs.keys())
    fig = plt.figure(figsize=(fig_size,fig_size),dpi=100)
    for index,img in enumerate(imgs):
        plt.subplot(len(imgs)/COL+1, COL, index+1)
        img_name = list(img.keys())[0]
        image = list(img.values())[0]
        plt.title(img_name)
        plt.imshow(image)
        print(img_name)
    #fig.savefig("./examples/test_image_after.jpg",bbox_inches='tight')

def save_image(filename,img,size=(288,162)):
    """
    resize and save image to local
    """
    img2save = cv2.resize(img,size,cv2.INTER_LINEAR)
    if len(size)>2:
        plt.imsave("./examples/{0}.jpg".format(filename),img2save)
    else:
        plt.imsave("./examples/{0}.jpg".format(filename),img2save,cmap="gray")

def draw_poly(filename,img,vertices):
    """
    draw the poly lines of roi and save it to local
    """
    fig = plt.figure(figsize=(3,2),dpi=100)
    x = []
    y = []
    for v in vertices:
        for xx,yy in v:
            x.append(xx)
            y.append(yy)
            plt.plot(xx,yy,'.', color='red',markersize=10)
            #note_point = "({0},{1})".format(xx,yy)
            #plt.annotate(note_point,xy=(xx,yy),color='red')

    x.append(x[0])
    y.append(y[0])
    #plt.xticks([])
    #plt.yticks([])
    #plt.axis('off')
    plt.plot(x, y, 'b--', lw=4)
    plt.imshow(img)

    fig.savefig("./examples/{0}_with_dash.jpg".format(filename),bbox_inches='tight')

def draw_lines_in_xyspace(filename,lines,size=(28,16)):
    """
    draw a plot to show all the line segments
    red line : lines with illegal slope
    blue line : lines with negative slope which is considering as the left line
    green line : lines with postive slope which is considering as the right line
    """
    fig = plt.figure(figsize=size,dpi=100)
    ax = plt.gca()

    for line in lines:
        for x1,y1,x2,y2 in line:
            x = [x1,x2]
            y = [y1,y2]
            slope = (y2-y1)/(x2-x1)
            if abs(slope) < 0.4 or abs(slope)>1:
                color = "r"
            elif slope < 0:
                color = "b"
            elif slope > 0:
                color = "g"
            plt.plot(x,y,color, lw=2)
            plt.plot(x1,y1,'x', markersize=12)
            plt.plot(x2,y2,'x', markersize=12)
            note_start = "({0},{1})".format(x1,y1)
            note_end = "({0},{1})".format(x2,y2)
            note_slope = "{0}".format(slope)
            plt.annotate(note_start,xy=(x1,y1))
            plt.annotate(note_end,xy=(x2,y2))
            plt.annotate(note_slope,xy=(x1,y1+10))
    ax.invert_yaxis()
    plt.show()
    fig.savefig("./examples/{0}.jpg".format(filename),bbox_inches='tight')

def draw_polyfit_line(group,l,r,size=(28,16)):
    """
    draw all the end points and the line fit them
    """
    left_x,left_y,right_x,right_y = group
    fig = plt.figure(figsize=size,dpi=100)
    ax = plt.gca()

    min_left_y = min(left_y) if len(left_y) > 0 else 99999
    max_left_y = max(left_y) if len(left_y) > 0 else -99999
    min_right_y = min(right_y) if len(right_y) > 0 else 99999
    max_right_y = max(right_y) if len(right_y) > 0 else -99999

    maxy = max(max_right_y,max_left_y)
    miny = min(min_left_y,min_right_y)
    y1 = maxy
    y2 = miny

    if len(left_x) > 0 and len(left_y) > 0:
        plt.plot(left_x,left_y,'xb',markersize="15")
        lx1 = l(maxy)
        lx2 = l(miny)
        plt.plot([lx1,lx2],[y1,y2],'*r',markersize="25")
        plt.plot((lx1,lx2),(y1,y2),'-g')

    if len(right_x) > 0 and len(right_y) > 0:
        plt.plot(right_x,right_y,'xg',markersize="15")
        rx1 = r(maxy)
        rx2 = r(miny)
        plt.plot([rx1,rx2],[y1,y2],'*r',markersize="25")
        plt.plot((rx1,rx2),(y1,y2),'-y')

    ax.invert_yaxis()
    fig.savefig("./examples/polyfit.jpg",bbox_inches='tight')

def draw_lane_lines(img, lines, color=[255,30, 0], thickness=12,size=(28,16)):
    """
    calculate the slope for each line segments and divide its end points into two groups
    points on lines with postive slope which is considering as the right line
    points on lines with negative slope which is considering as the left line
    also rule out the lines whose absulute value of slope is not betweent 0.4~1

    using np.polyfit to get a line most fit the points in left group or right group,and this function
    will return [m,b] for slope and intercept

    finally,using poly1d to get the polynomial and calculate the x by y. Then draw two lines on the image

    """
    i = 0
    left_x = []
    left_y = []
    right_x = []
    right_y = []
    l = None
    r = None
    topY = int(330) # top of rio
    bottomY = int(img.shape[1]) #bottom of rio

    for line in lines:
        for x1,y1,x2,y2 in line:
            slope = (y2-y1)/(x2-x1)
            if abs(slope) > 0.4 and abs(slope)<1:
                if slope < 0: #left
                    left_x += [x1,x2]
                    left_y += [y1,y2]
                elif slope > 0: #right
                    right_x += [x1,x2]
                    right_y += [y1,y2]

    # polyfit for valid point

    # if we get a valid line , do polyfit and append it to the cache
    if len(left_y) > 0 and len(left_y) > 0:
        z1 = np.polyfit(left_y,left_x,1)
        left_line_queue.append(z1)

    # if we get a valid line , do polyfit and append it to the cache
    if len(right_y) > 0 and len(right_y) > 0:
        z2 = np.polyfit(right_y,right_x,1)
        right_line_queue.append(z2)

    left_ave_z = get_ave_z(left_line_queue)
    right_ave_z = get_ave_z(right_line_queue)

    l = np.poly1d(left_ave_z)
    lx1 = int(l(bottomY))
    lx2 = int(l(topY))

    r = np.poly1d(right_ave_z)
    rx1 = int(r(bottomY))
    rx2 = int(r(topY))

    cv2.line(img, (lx1,bottomY),(lx2,topY),[0,0,255],15)
    cv2.line(img, (rx1,bottomY),(rx2,topY),[0,0,255],15)

    #draw_polyfit_line((left_x,left_y,right_x,right_y),l,r)

def get_ave_z(queue):
    m = 0
    b = 0
    for p in queue:
        m+=p[0]
        b+=p[1]
    ave_m = m/len(queue)
    ave_b = b/len(queue)
    return ave_m,ave_b

# ## Test Images
#
# Build your pipeline to work on the images in the directory "test_images"
# **You should make sure your pipeline works well on these images before you try the videos.**

# ## Build a Lane Finding Pipeline
#
#

# Build the pipeline and run your solution on all test_images. Make copies into the `test_images_output` directory, and you can use the images in your writeup report.
#
# Try tuning the various parameters, especially the low and high Canny thresholds as well as the Hough lines parameters.

# In[4]:

# TODO: Build your pipeline that will draw lane lines on the test_images
# then save them to the test_images_output directory.

MAX_FRAME_NUM = 6 # max priors line cache numbers

left_line_queue = deque(maxlen=MAX_FRAME_NUM) # priors left line cache
right_line_queue = deque(maxlen=MAX_FRAME_NUM) # priors right line cache

def _pipeline(image):
    """
    build a pipeline to draw lane lines on the image

    1. tansfrom color image to gray scale image then using gaussian blur to smooth the gray image
    2. using canny to detect edges
    3. set 4 vertices of our interest region and using this region as a mask to rule out the non-related lines.
    4. apply hough transform to detect the line segment
    5. Draw hough lines on the original image.

    """
    # must make a copy otherwise when pipeline
    # failed we cannot return the original image
    # beacuse the original image has been changed
    # by part of the pipeline

    image_copy = np.copy(image)

    imshape = image_copy.shape

    kernel_size = 3       # kernel size for gaussian blur
    low_threshold = 100   # low threshold for canny
    high_threshold = 300  # high threshold for canny

    # vertices of the region of interest(roi)
    v1 = (0,imshape[0])
    v2 = (450, 290)
    v3 = (490, 290)
    v4 = (imshape[1],imshape[0])
    vertices = np.array([[v1,v2, v3, v4]], dtype=np.int32)

    # step1
    gray_image = grayscale(image_copy)

    blur_image = gaussian_blur(gray_image, kernel_size)
    #plt.imshow(blur_image)
    # step2
    edges = canny(blur_image, low_threshold, high_threshold)
    #plt.imshow(edges)
    # step3
    masked_edges,rio= region_of_interest(edges,vertices)
    #plt.imshow(masked_edges)

    # step4: hough transform
    # Define the Hough transform parameters
    # Make a blank the same size as our image to draw on
    rho = 1 # distance resolution in pixels of the Hough grid
    theta = np.pi/180 # angular resolution in radians of the Hough grid
    threshold = 60    # minimum number of votes (intersections in Hough grid cell)
    min_line_len = 80 #minimum number of pixels making up a line
    max_line_gap = 100    # maximum gap in pixels between connectable line segments
    line_img,lines = hough_lines(masked_edges, rho, theta, threshold, min_line_len, max_line_gap)
    #plt.imshow(line_img)

    #draw_lines_in_xyspace("hough_line_slope",lines)

    # step5
    result = weighted_img(line_img, image, α=0.8, β=1., γ=0.)

    #save_image("gray_image",gray_image)
    #save_image("blur_image",blur_image)
    #save_image("edges",edges)
    #save_image("masked_edges",masked_edges)

    #save_image("hough_lines",line_img)
    #save_image("hough_line_on_origin",result)
    #save_image("lane_lines",result,size=(960,540))
    #save_image("rio",rio)
    #draw_poly("rio",edges,vertices)

    return result

#draw_lines(line_img, lines)

#cv2.line(test_image,v1,v2,color=[255, 0, 0], thickness=2)
#color_edges = np.dstack((edges, edges, edges))
#plt.imshow(result_exp)
# Draw the lines on the edge image
#lines_edges = cv2.addWeighted(color_edges, 0.8, line_image, 1, 0)

ximages = os.listdir("test_images/")[1:]
def draw_test_images():
    check_images = []
    for img in ximages :
        filename = "test_images/"+img
        _img = mpimg.imread(filename)
        check_images.append({filename : _pipeline(_img)})
    draw_imgs(check_images)

import os
image_output_dir = 'test_image_output/'

def draw_bad_images():
    bad_images = []
    for img in os.listdir(image_output_dir)[1:]:
        _img = mpimg.imread(image_output_dir+img)
        bad_images.append({img : _pipeline(_img)})

    draw_imgs(bad_images)

draw_test_images()
#draw_bad_images()

# ## Test on Videos
#
# You know what's cooler than drawing lanes over images? Drawing lanes over video!
#
# We can test our solution on two provided videos:
#
# `solidWhiteRight.mp4`
#
# `solidYellowLeft.mp4`
#
# **Note: if you get an import error when you run the next cell, try changing your kernel (select the Kernel menu above --> Change Kernel). Still have problems? Try relaunching Jupyter Notebook from the terminal prompt. Also, consult the forums for more troubleshooting tips.**
#
# **If you get an error that looks like this:**
# ```
# NeedDownloadError: Need ffmpeg exe.
# You can download it by calling:
# imageio.plugins.ffmpeg.download()
# ```
# **Follow the instructions in the error message and check out [this forum post](https://discussions.udacity.com/t/project-error-of-test-on-videos/274082) for more troubleshooting tips across operating systems.**

# In[5]:

# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML

# In[6]:

def process_image(image):
    global i
    i+=1
    # NOTE: The output you return should be a color image (3 channel) for processing video below
    # TODO: put your pipeline here,
    # you should return the final output (image where lines are drawn on lanes)
    try :
        result = _pipeline(image)
    except Exception as e: #need to handle the error that pipeline failed to find any valid line
        result = image
        plt.imsave(image_output_dir + "frame_{0}.jpg".format(i),result)
        #error_img.append({"test" : image})

    return result

# Let's try the one with the solid white lane on the right first ...

# In[7]:

i = 0
white_output = 'test_videos_output/solidWhiteRight.mp4'

## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4").subclip(0,5)
#clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
get_ipython().run_line_magic('time', 'white_clip.write_videofile(white_output, audio=False)')
white_clip = white_clip.resize(0.5)
white_clip.write_gif("examples/solidWhiteRight.gif",fps=10,fuzz=1)
#draw_imgs(error_img)

# Play the video inline, or if you prefer find the video in your filesystem (should be in the same directory) and play it in your video player of choice.

# In[8]:

HTML("""
<video width="960" height="540" controls>

  <source src="{0}">
</video>
""".format(white_output))

# ## Improve the draw_lines() function
#
# **At this point, if you were successful with making the pipeline and tuning parameters, you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform. As mentioned previously, try to average and/or extrapolate the line segments you've detected to map out the full extent of the lane lines. You can see an example of the result you're going for in the video "P1_example.mp4".**
#
# **Go back and modify your draw_lines function accordingly and try re-running your pipeline. The new output should draw a single, solid line over the left lane line and a single, solid line over the right lane line. The lines should start from the bottom of the image and extend out to the top of the region of interest.**

# Now for the one with the solid yellow lane on the left. This one's more tricky!

# In[9]:

yellow_output = 'test_videos_output/solidYellowLeft.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4').subclip(0,5)
#clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')

yellow_clip = clip2.fl_image(process_image)
get_ipython().run_line_magic('time', 'yellow_clip.write_videofile(yellow_output, audio=False)')
yellow_clip = yellow_clip.resize(0.5)
yellow_clip.write_gif("examples/solidYellowLeft.gif",fps=10,fuzz=1)

# In[10]:

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))

# ## Writeup and Submission
#
# If you're satisfied with your video outputs, it's time to make the report writeup in a pdf or markdown file. Once you have this Ipython notebook ready along with the writeup, it's time to submit for review! Here is a [link](https://github.com/udacity/CarND-LaneLines-P1/blob/master/writeup_template.md) to the writeup template file.
#

# ## Optional Challenge
#
# Try your lane finding pipeline on the video below.  Does it still work?  Can you figure out a way to make it more robust?  If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!

# In[11]:

challenge_output = 'test_videos_output/challenge.mp4'
## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line below
## Where start_second and end_second are integer values representing the start and end of the subclip
## You may also uncomment the following line for a subclip of the first 5 seconds
clip3 = VideoFileClip('test_videos/challenge.mp4').subclip(0,5)
#clip3 = VideoFileClip('test_videos/challenge.mp4')
challenge_clip = clip3.fl_image(process_image)
get_ipython().run_line_magic('time', 'challenge_clip.write_videofile(challenge_output, audio=False)')
challenge_clip = challenge_clip.resize(0.5)
challenge_clip.write_gif("examples/challenge.gif",fps=10,fuzz=1)

# In[12]:

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))