#!/usr/bin/env python
# coding: utf-8

# # Analyzing borrowers’ risk of defaulting
#
# Your project is to prepare a report for a bank’s loan division. You’ll need to find out if a customer’s marital status and number of children has an impact on whether they will default on a loan. The bank already has some data on customers’ credit worthiness.
#
# Your report will be considered when building a **credit scoring** of a potential customer. A ** credit scoring ** is used to evaluate the ability of a potential borrower to repay their loan.

# In[1]:

# import libraries
import pandas as pd
import numpy as np

# In[2]:

# load the data
try:
    data = pd.read_csv('/Users/rraven/Desktop/a_final_yandex/datasets/credit_scoring_eng.csv')
except:
    data = pd.read_csv('/datasets/credit_scoring_eng.csv')

# In[3]:

# print general info & first 10 rows
#data = pd.read_csv('../data/credit_scoring_eng.csv')
print('\nGeneral Info for credit_scoring_eng\n')
data.info()
print('\nFirst 10 rows of credit_scoring_eng.csv')
data.head(10)

# In[4]:

# find the number of missing values per column
print('Total Rows:', len(data))
print('\nColumn\t\tMissing Rows')
data.isnull().sum()

# In[5]:

ratio = 100*data.isnull().sum() / len(data)
print('Column\t\tPercent Missing')
ratio

# In[6]:

# check data for duplicates in entire dataFrame
print('Number of duplicate rows:')
data.duplicated().sum()

# **Initial Observations for credit_scoring_eng.csv**
#
# * 21525 rows
# * 12 columns
# * A mix of data types (float64(2), int64(5), object(5))
# * 2174 missing values in 2 columns (days_employed, total_income)
# * days_employed and total_income are missing over 10% of total values
# * 54 duplicate rows

# In[7]:

# investigate children column
data['children'].value_counts()

# In[8]:

# investigate family_status column
data['family_status'].value_counts()

# In[9]:

# investigate family_status_id column
data['family_status_id'].value_counts()

# In[10]:

# investigate purpose column
data['purpose'].value_counts()

# In[11]:

# how many unique entries in purpose column
data['purpose'].nunique()

# In[12]:

# investigate debt column
data['debt'].value_counts()

# In[13]:

# investigate dob_years column
data['dob_years'].value_counts().sort_index()

# In[14]:

# categorize dob_years in 10 year increments
less_than_20 = 0
btwn20and30 = 0
btwn30and40 = 0
btwn40and50 = 0
btwn50and60 = 0
btwn60and70 = 0
over_equal70 = 0
nan_count = 0
unknown = 0
for x in data['dob_years']:
    if x < 20:
        less_than_20 +=1
    elif x < 30:
        btwn20and30 +=1
    elif x < 40:
        btwn30and40 +=1
    elif x < 50:
        btwn40and50 +=1
    elif x < 60:
        btwn50and60 +=1
    elif x < 70:
        btwn60and70 +=1
    elif x >= 70:
        over_equal70 +=1
    elif str(x) == 'nan':
        nan_count +=1
    else:
        # to check there are no other values
        unknown +=1
print("Age < 20:\t", less_than_20,
      "\n20 <= Age < 30:\t", btwn20and30,
      "\n30 <= Age < 40:\t", btwn30and40,
      "\n40 <= Age < 50:\t", btwn40and50,
      "\n50 <= Age < 60:\t", btwn50and60,
      "\n60 <= Age < 70:\t", btwn60and70,
      "\nAge >= 70:\t", over_equal70,
      "\nnan:\t\t", nan_count,
      "\nUnknown:\t", unknown)

# In[15]:

# investigate education column
data['education'].value_counts()

# In[16]:

# investigate education_id column
data['education_id'].value_counts()

# In[17]:

# investigate gender column
data['gender'].value_counts()

# In[18]:

# investigate income_type column
data['income_type'].value_counts()

# In[19]:

# how many unique entries in income_type column
data['income_type'].nunique()

# In[20]:

# investigate total_income
data['total_income'].value_counts()

# In[21]:

# categorize income_level into 10K levels to see distribution

# create a new function, income_level_fx
def income_level_fx(row):
    # the income_level is returned according to total_income
    income = row['total_income']

    if income < 10000:
        return 'less_than_10K'
    elif income < 20000:
        return 'btwn10Kand20K'
    elif income < 30000:
        return 'btwn20Kand30K'
    elif income < 40000:
        return 'btwn30Kand40K'
    elif income < 50:
        return 'btwn40Kand50K'
    elif income < 60000:
        return 'btwn50Kand60K'
    elif income < 70000:
        return 'btwn60Kand70K'
    elif income < 80000:
        return 'btwn70Kand80K'
    elif income < 90000:
        return 'btwn80Kand90K'
    elif income < 100000:
        return 'btwn90Kand100K'
    elif income >= 100000:
        return 'greater_than_100K'

# create a new column, income_level, based on total_income
data['income_level'] = data.apply(income_level_fx, axis=1)

# verfiy new column: income_level
data['income_level'].value_counts()

# ### Conclusion

# **The datafile credit_scoring_eng.csv contains**
# * 21525 rows
# * 12 columns
# * A mix of data types (float64(2), int64(5), object(5))
# * 2174 missing values in 2 columns (days_employed, total_income)
# * days_employed and total_income are missing over 10% of total values
# * 54 duplicate rows
#
# **Targeted areas - having kids, marital status, income level and loan purpose r/t repayment**
# * children
#     - int64
#     - contains 47 entries with -1 and 76 with 20
#     - maybe -1 was meant to be 1? keying error? correct or remove?
#     - maybe 20 was meant to be 2? keying error? correct or remove?
#     - ultimately can try different groups (none or some), (none, 1, or > 1), etc
# * family_status and family_status_id
#     - string(object) and int64
#     - no problems noted, appear to correlate well
#         - married              12380
#         - civil partnership     4177
#         - unmarried             2813
#         - divorced              1195
#         - widow / widower        960
#         - 0    12380
#         - 1     4177
#         - 4     2813
#         - 3     1195
#         - 2      960
# * total_income
#     - float64
#     - will need to address missing values
# * purpose
#     - string(object)
#     - 38 unique entries
#     - appears to be 4 main categories
#         - wedding
#         - property
#         - car
#         - education
#     - will need to work with stem and/or lemmas
#
# **How to judge risk of default / likelihood of repayment of loan?**
# Discussion: There are 12 total columns. 5 of these are specifically involved in the task (children, family_status/family_status_id, total_income, and purpose). This leaves days_employed, dob_years, education,
# education_id, gender, income_type, and debt.
#
# Which of these are likely to correlate with risk of default?
# - days_employed might, because it could demonstrate stability, but the values in that column have too many problems to be useful. Therefore, that column will likely be ignored.
# - dob_years: could younger people (< 30 or 40) have a higher risk of default? maybe
# - education / education_id: could less education correlate with higher risk of default? maybe
# - gender: could gender correlate with risk of default? maybe
# - income_type: could source of income correlate with risk of default? maybe
# - debt: does "whether the customer has ever defaulted on a loan" correlate with risk of default? very likely
#
# **Of these choices, debt is the most likely to produce useful correlations with risk of default. **
#
# * debt
#     - int64
#     - 2 values, no nan or unknowns
#     - assume 0 hasn't ever defaulted
#     - assume 1 for has defaulted in the past
#         - 0    19784
#         - 1     1741
#
# **General observations of other columns**
# * days_employed (will investigate in missing info section)
#     - float64 (why?)
#     - will need to address missing values
# * dob_years
#     - int64
#     - age range from 19 to 75, but 101 values of 0 for age
#     - will need to categorize, maybe by decade
# * education has a mixture of upper and lower case strings, numerous variations
#     - string(object)
#     - change all to lowercase to unify and compare to education_id
#     - could use it to help fill in missing income info
# * eduction_id column
#     - int64
#     - could be used instead of education if codes correlate?
#         - 0 = bachelor's degree
#         - 1 = secondary education
#         - 2 = some college
#         - 3 = primary education
#         - 4 = graduate degree
#     - maybe could reorganize because this order doesn't make sense?
# * gender
#     - string(object)
#     - could change it to number code
# * income_type
#     - string(object)
#     - 8 unique entries
#     - could change it to number code
#     - could combine it with other info to fill in missing incomes
#         - employee                       11119
#         - business                        5085
#         - retiree                         3856
#         - civil servant                   1459
#

# ## Data preprocessing

# ### Processing missing values

# 1. Address days_employed
# 2. Address total_income

# In[22]:

# investigate days_employed column
data['days_employed'].value_counts().sort_index()

# In[23]:

# there are missing values, but how many?
# find percent of total rows where days_employed is missing a value
missing_days = data['days_employed'].isnull().sum()
pct_missing = missing_days/len(data)
print('Percentage of rows missing total_income: {:.2%}'.format(pct_missing))

# In[24]:

# add column & calculate years worked using days_employed
data['years_employed'] = data['days_employed'] / 365

# In[25]:

# investigate range of years_employed column
data['years_employed'].value_counts().sort_index()

# There are some unrealistic values (working for negative 50 years and over 1100 years).

# In[26]:

# find counts of positive and negative values for days employed
pos_count = 0
neg_count = 0
nan_count = 0
unknown = 0
for x in data['days_employed']:
    if x >= 0:
        pos_count +=1
    elif x < 0:
        neg_count +=1
    elif str(x) == 'nan':
        nan_count +=1
    else:
        # to check there are no other values
        unknown +=1
print("Positive days employed:\t", pos_count,
      "\nNegative days employed:\t", neg_count,
      "\nnan in days employed:\t", nan_count,
      "\nUnknown in days employed:", unknown)

# Analysis of days_employed reveals:
# - float64 (why?) and has many negative values (why?)
# - is missing 2174 values
# - has a range from -18388.949901 to 401755.400475 days employed
# - has a range from -50 years to over 1100 years employed
# - positive days employed: 3445
# - negative days employed: 15906
#
# Possible reasons for problem entries:
# 1. Human error when entering data (adding - by accident).
# 2. Error in units (perhaps the person entering the data got confused between hours worked and days worked?). Confusing hours for days may explain the very high numbers.
# 3. There may have been errors when merging different data sets.
#
# Plan to fill missing values:
# 1. Since negative values are likely due to human error when entering data, use absolute value on days_employed.
# 2. Evaluate min/max values to determine range again, verify no more negative values.
# 3. Refresh years_employed and verify no more negative values.
# 4. Compare mean and median to gain an overall sense of data.
# 5. If people mistakenly entered hours worked instead of days, we can assume the maximum work period could be 70 years (70 yrs * 365 days/yrs) = 25550 days.
# 6. More realistically, we could assume the maximum work period could be 50 years (50 yrs * 365 days/yrs) = 18250.
# 7. With values greater than work period (either 25550 days or 18250 days), divide by 24 to get a days worked: data_50_yrs and data_70_years.
# 8. Compare statistics data_50_yrs and data_70_years.
# 9. Replace values in data df with appropriate cutoff value (#/24).
# 10. Compare mean and median of days_employed column.
# 11. Fill missing values with appropriate value (mean or median).

# In[27]:

# apply abs() to change negative values to positive
data['days_employed'] = data['days_employed'].abs()
# verify no more negative values after abs()
negative_count = data.loc[data['days_employed']  < 0, 'days_employed'].count()
print('After applying abs(), verify the total number of negative values in days_employed = 0.')
negative_count

# In[28]:

positive_count = data.loc[data['days_employed']  > 0, 'days_employed'].count()
print('After applying abs(), verify the total number of positive values in days_employed = 19351')
positive_count

# In[29]:

# add column & calculate years worked using days_employed
data['years_employed'] = data['days_employed'] / 365

days_neg = data.loc[data['years_employed']  < 0, 'years_employed'].count()
print('After applying abs(), verify the total number of negative values in years_employed = 0.')
days_neg

# In[30]:

# find info on mean, min, max for days_employed
print('Statistical info for days_employed')
data['days_employed'].describe()

# In[31]:

# find info on mean, min, max for years_employed
print('Statistical info for years_employed')
data['years_employed'].describe()

# In[32]:

# find the median of years_employed
data['years_employed'].median()

# This data is skewed with some very high values. The median for years is just over 6, while the mean is over 183 years worked. This suggests there are many very high values distorting the data.

# In[33]:

# make a copy of data, name it data_50_yrs
data_50_yrs = data.copy()
# change all the values greater than 18250 days (50 years) by dividing by 24 and saving results
data_50_yrs.loc[data_50_yrs['days_employed'] > 18250, 'days_employed'] = data_50_yrs['days_employed']/24
# find info on mean, min, max for days employed where values changed
print('Statistical info for days_employed')
print('where values > 18250 (50 years) and divided by 24')
data_50_yrs['days_employed'].describe()

# In[34]:

# make a copy of data, name it data_70_yrs
data_70_yrs = data.copy()
# change all the values greater than 25550 days (70 years) by dividing by 24 and saving results
data_70_yrs.loc[data_70_yrs['days_employed'] > 25550, 'days_employed'] = data_70_yrs['days_employed']/24
# find info on mean, min, max for days employed where values changed
print('Statistical info for days_employed')
print('where values > 25550 (70 years) and divided by 24')
data_70_yrs['days_employed'].describe()

# The means and medians for the two options are very close:
#
# - 50 yrs  70 yrs
# - 4640 vs 4641
# - 2194 vs 2194
#
# The max values are a bit different
# 17615 (48 years) vs 18388 (50 years).
#
# The 70 year cutoff will be used in the data df just in case that entry with a 50 year work history is accurate. There are 3800+ retirees in the sample after all.

# In[35]:

# change all the values greater than 25550 days (70 years) by dividing by 24 and saving results
data.loc[data['days_employed'] > 25550, 'days_employed'] = data['days_employed']/24

# find info on mean, min, max for days_employed
print('Statistical info for days_employed')
data['days_employed'].describe()

# In[36]:

# look for nan, negative, and other values in days_employed
# print first 20 rows of duplicates to look for patterns
pos_count = 0
neg_count = 0
nan_count = 0
unknown = 0
counter= -1
for x in data['days_employed']:
    counter +=1
    if x >= 0:
        pos_count +=1
    elif x < 0:
        neg_count +=1
    elif str(x) == 'nan':
        nan_count +=1
        if nan_count < 20:
            print(data.iloc[counter])
    else:
        # to check there are no other values
        unknown +=1
print("Positive income:", pos_count,
      "\nNegative income:", neg_count,
      "\nnan:\t\t", nan_count,
      "\nUnknown:\t", unknown)

# In[37]:

# find median of days_employed
median_days = data['days_employed'].median()
print('The median of days_employed')
median_days

# In[38]:

# fill in missing values with median income
data['days_employed'].fillna(value=median_days, inplace = True)

# In[39]:

# check for any missing values
# find info on mean, min, max for days_employed
print('Statistical info for days_employed')
data['days_employed'].describe()

# There are no more missing values and the median remains the same, 2194. The mean actually went down, which makes sense because over 2000 missing values were replaced with 2194, bringing the overall mean downward.

# In[40]:

# find percent of total rows where total_income is missing
missing_income = data['total_income'].isnull().sum()
pct_missing = missing_income/len(data)
print('Percentage of rows missing total_income: {:.2%}'.format(pct_missing))

# In[41]:

# look for nan, negative, and other values in total_income
# print first 20 rows of duplicates to look for patterns
pos_count = 0
neg_count = 0
nan_count = 0
unknown = 0
counter= -1
for x in data['total_income']:
    counter +=1
    if x >= 0:
        pos_count +=1
    elif x < 0:
        neg_count +=1
    elif str(x) == 'nan':
        nan_count +=1
        if nan_count < 20:
            print(data.iloc[counter])

    else:
        # to check there are no other values
        unknown +=1
print("Positive income:", pos_count,
      "\nNegative income:", neg_count,
      "\nnan:\t\t", nan_count,
      "\nUnknown:\t", unknown)

# In[42]:

# categorize income_level into 10K levels

# create a new function, income_level_fx
def income_level_fx(row):
    # the income_level is returned according to total_income
    income = row['total_income']

    if income < 10000:
        return 'less_than_10K'
    elif income < 20000:
        return 'btwn10Kand20K'
    elif income < 30000:
        return 'btwn20Kand30K'
    elif income < 40000:
        return 'btwn30Kand40K'
    elif income < 50:
        return 'btwn40Kand50K'
    elif income < 60000:
        return 'btwn50Kand60K'
    elif income < 70000:
        return 'btwn60Kand70K'
    elif income < 80000:
        return 'btwn70Kand80K'
    elif income < 90000:
        return 'btwn80Kand90K'
    elif income < 100000:
        return 'btwn90Kand100K'
    elif income >= 100000:
        return 'greater_than_100K'

# create a new column, income_level, based on total_income
data['income_level'] = data.apply(income_level_fx, axis=1)

# verfiy new column: income_level
data['income_level'].value_counts()

# In[43]:

# find info on mean, min, max for total_income
print('Statistical info for total_income')
data['total_income'].describe()

# In[44]:

# find median of total_income
median_income = data['total_income'].median()
print('The median of total_income')
median_income

# total_income is vital to the analysis. Over 10% of the values are missing.
# Those values could be replaced by the mean or median.
#
# Since the mean (26787) > median (23202), high outling values are pulling the mean up. The median should be used to replace the values. A copy of the dataframe, titled data_median_income, will be created to store data where the median total_income replaces the missing values.
#
# Another option will be to look at correlations between other factors and total_income and replace missing values based on those factors.
#
# The two options will be compared in the final analysis.

# In[45]:

# make a copy of data and replace the missing values with median
data_median_income = data.copy()

# In[46]:

# fill in missing values with median income
data_median_income['total_income'].fillna(value=median_income, inplace = True)

# In[47]:

# print the number of missing values in data_median_income after filling nan
print('Verify that there are no missing values in data_median_income')
data_median_income['total_income'].isnull().sum()

# In[48]:

# categorize data_median_income income_level into 10K levels

# create a new function, income_level_fx
def income_level_fx(row):
    # the income_level is returned according to total_income
    income = row['total_income']

    if income < 10000:
        return 'less_than_10K'
    elif income < 20000:
        return 'btwn10Kand20K'
    elif income < 30000:
        return 'btwn20Kand30K'
    elif income < 40000:
        return 'btwn30Kand40K'
    elif income < 50:
        return 'btwn40Kand50K'
    elif income < 60000:
        return 'btwn50Kand60K'
    elif income < 70000:
        return 'btwn60Kand70K'
    elif income < 80000:
        return 'btwn70Kand80K'
    elif income < 90000:
        return 'btwn80Kand90K'
    elif income < 100000:
        return 'btwn90Kand100K'
    elif income >= 100000:
        return 'greater_than_100K'

# create a new column, income_level, based on total_income
data_median_income['income_level'] = data_median_income.apply(income_level_fx, axis=1)

# verfiy new column: income_level
data_median_income['income_level'].value_counts()

# In[49]:

# find info on mean, min, max for total_income
print('Statistical info for data_median_income')
data_median_income['total_income'].describe()

# Since debt, children, family_status, purpose will be examined later, those will not be used when considering filling missing values in total_income.
#
# Before considering relationships between total_income and (gender and/or education_id and/or dob_years and/or income_type), dob_years and education need to be addressed.
#
# 1. Process dob_years into categories (called age_group)
# 2. Change the items in education to lowercase so it can be used for analysis.
# 3. Investigate gender, education, dob_years, income_type with total_income

# In[50]:

# process dob_years into age_group

# create a new function, age_group_fx
def age_group_fx(row):
    # the age group is returned according to dob_years
    age = row['dob_years']

    if age < 20:
        return 'less_than_20'
    elif age < 30:
        return 'btwn20and30'
    elif age < 40:
        return 'btwn30and40'
    elif age < 50:
        return 'btwn40and50'
    elif age < 60:
        return 'btwn50and60'
    elif age < 70:
        return 'btwn60and70'
    elif age >= 70:
        return 'over_equal70'

# create a new column, age_group, based on dob_years
data['age_group'] = data.apply(age_group_fx, axis=1)

# calculate number of items per age level
print('Number of items per age group')
data['age_group'].value_counts()

# In[51]:

# calculate number of items per education column
data['education'].value_counts()

# In[52]:

# change items to lowercase
data['education'] = data['education'].str.lower()

# calculate number of items per education column
print('Number of items per education category')
data['education'].value_counts()

# In[53]:

# compare with education #s with education_id #s
print('Number of items per education_id category')
data['education_id'].value_counts()

# The education column and the education_id column values match.

# In[54]:

# investigate gender and total_income
data.groupby('gender').agg({'total_income': ['count', 'mean', 'median']})

# There is close to 5K difference, quite significant, between values for F or M.
#
# Gender will be used to help fill in missing values for total income.

# In[55]:

# investigate education and total_income
data.groupby('education').agg({'total_income': ['count', 'mean', 'median']})

# There is a significant difference between incomes for different education levels.
#
# Education will be used to help fill in missing values for total income.

# In[56]:

# investigate count of missing total_income by education
print('Count of missing total income by education')
print(data[data['total_income'].isnull()]['education'].value_counts())

# In[57]:

# investigate age_group and total_income
data.groupby('age_group').agg({'total_income': ['count', 'mean', 'median']})

# There is not as significant difference between incomes for different age groups.
#
# Age_groups will not be used to help fill in missing values for total income.

# In[58]:

# investigate income_type and total_income
data.groupby('income_type').agg({'total_income': ['count', 'mean', 'median']})

# There is a sizeable difference between business, employee, and retiree and all of those have over 3K applicants.
#
# Income_type will be used to help fill in missing values for total income.

# In[59]:

# investigate count of missing total_income by income_type
print('Count of missing total income by income_type')
print(data[data['total_income'].isnull()]['income_type'].value_counts())

# In[60]:

# investigate if there are any correlations with total_income and
# gender and education and income_type and age group
data.pivot_table(values=['total_income'], columns=['gender', 'education', 'income_type'])

# This suggests some wide variations in value when categorized by gender, education, and income type.

# In[61]:

# fill in values using education_id and income_type
print('Fill in missing values of total_income based on education, gender, and income_type')
data['total_income'] = data['total_income'].fillna(data.groupby(['education_id', 'income_type', 'gender'])['total_income'].transform('median'))

# print the number of missing values in data after filling nan
print('Verify that there are no missing values in data')
data['total_income'].isnull().sum()

# In[62]:

miss_value = data[data['total_income'].isna()]
miss_value

# 1 row still has a missing value for total_income.
# The applicant is a male entrepreneur with a bachelor's degree.

# In[63]:

# investigate if there are any correlations with total_income and
# gender and income_type
data.pivot_table(values=['total_income'], columns=['gender', 'income_type'])

# Since there is not a listing for a M who is an entrepreneur, the program couldn't fill in a value. The value for a F who is an entrepreneur is very high, so it may be better to base the missing value on a M having a bachelor's degree.

# In[64]:

# investigate if there are any correlations with total_income and
# gender and education_id
data.pivot_table(values=['total_income'], columns=['gender', 'education'])

# In[65]:

# replace the missing value with value for a M with a bachelor's degree
print('Fill in missing value of total_income based on education and gender')
data['total_income'] = data['total_income'].fillna(data.groupby(['education_id', 'gender'])['total_income'].transform('median'))

# print the number of missing values in data after filling nan
print('Verify that there are no missing values in data')
data['total_income'].isnull().sum()

# In[66]:

# categorize income_level into 10K levels
# important to do after all the missing values added to update

# create a new function, income_level_fx
def income_level_fx(row):
    # the income_level is returned according to total_income
    income = row['total_income']

    if income < 10000:
        return 'less_than_10K'
    elif income < 20000:
        return 'btwn10Kand20K'
    elif income < 30000:
        return 'btwn20Kand30K'
    elif income < 40000:
        return 'btwn30Kand40K'
    elif income < 50:
        return 'btwn40Kand50K'
    elif income < 60000:
        return 'btwn50Kand60K'
    elif income < 70000:
        return 'btwn60Kand70K'
    elif income < 80000:
        return 'btwn70Kand80K'
    elif income < 90000:
        return 'btwn80Kand90K'
    elif income < 100000:
        return 'btwn90Kand100K'
    elif income >= 100000:
        return 'greater_than_100K'

# create a new column, income_level, based on total_income
data['income_level'] = data.apply(income_level_fx, axis=1)

# verfiy new column: income_level
data['income_level'].value_counts()

# In[67]:

data.info()

# In[68]:

# find info on mean, min, max for total_income
print('Statistical info for total_income')
data['total_income'].describe()

# ### Conclusion

# 1. days_employed may or may not be useful for analysis. Negative values may have occured through human data entry error. Very large values likely occured because of confusion over units (hours versus days), but they might be due to problems merging datasets. To handle the negative values, the absolute value was applied and the new values stored in place. Large values (equivalent to working > 70 years) were divided by 24 (for 24 hours) and replaced. The first 20 rows with NAN values revealed no particular pattern, so the missing values are likely MAR. Then the missing values were filled using the median because there were still large values skewing the results (mean 4641 vs median 2194).
#
#
# 2. total_income is vital to analysis (in fact it is one of the key categories to report on). Over 10% of the values are missing. The first 20 duplicates were printed and there is no obvious pattern, therefore these missing values are MAR. Reasons for missing values could include human error, information not provided by applicant, or a mix up when datasets were merged. total_income was categorized into 10K increments and general statistics were displayed. The mean is greater than the median, therefore the median values will be used for replacing missing values.
#
# 3. Replacing missing values:
# - approach one is to replace those missing values with the median. data_median_income
# - approach two involves filling missing values based on a composite value drawn from the influence of gender, income_type, and education (except for one stray value where only gender and education were used).
# - these approaches will be compared in the final analysis.

# ### Data type replacement

# 1. total_income should be changed from float 64 to an int for visual appeal / ease of understanding.
# 2. Data types could be changed to conserve memory. This is especially useful for very large files.

# In[69]:

# investigate data types
data.info()

# In[70]:

# check memory usage
print('Memory useage before')
data.memory_usage()

# In[71]:

# calculate total memory usage before
print('Memory useage before in MB')
memory_before = data.memory_usage().sum() / (1024**2) #converting to megabytes
memory_before

# In[72]:

# change data types using astype and apply w/numpy
data['children'] = data['children'].astype('int16')
# converting days_employed took 2 steps, 1st numpy to int, then astype
data['days_employed'] = data['days_employed'].apply(np.int)
data['days_employed'] = data['days_employed'].astype('int16')
data['dob_years'] = data['dob_years'].astype('int16')
data['education'] = data['education'].astype('category')
data['education_id'] = data['education_id'].astype('int16')
data['family_status'] = data['family_status'].astype('category')
data['family_status_id'] = data['family_status_id'].astype('int16')
data['gender'] = data['gender'].astype('category')
data['income_type'] = data['income_type'].astype('category')
data['debt'] = data['debt'].astype('int16')
# converting total_income took 2 steps, 1st numpy to int, then astype
data['total_income'] = data['total_income'].apply(np.int)
data['total_income'] = data['total_income'].astype('int16')
data['purpose'] = data['purpose'].astype('category')
data['years_employed'] = data['years_employed'].astype('float32')
data['age_group'] = data['age_group'].astype('category')

# In[73]:

# check memory usage after
print('Memory useage after')
data.memory_usage()

# In[74]:

# calculate total memory usage after
print('Memory useage after in MB')
memory_after = data.memory_usage().sum() / (1024**2) #converting to
memory_change = memory_before - memory_after
memory_after

# ### Conclusion

# In[75]:

print('Changing data types saved', memory_change, 'MB of memory and changing total_income and days_employed to int type allows for easier reading. Data types were mostly changed using astype, but when changing from float to int apply.np needed to be used. It is good to know that apply.np can only be used on columns with no missing values.')

# ### Processing duplicates

# 1. Manage duplicate rows
# - Calculate duplicate rows.
# - There is no reason to check for duplicates within columns, as values can repeat.
#
# 2. Manage purpose column with stemming or lemmatization
#

# In[76]:

dup_rows = data.duplicated().sum()
pct_duplicated = dup_rows/len(data)
print('There are', dup_rows, 'duplicate rows in the file')
print('Percentage of duplicate rows = {:.2%}'.format(pct_duplicated))

# In[77]:

# remove the duplicate rows
data = data.drop_duplicates()

# In[78]:

# verify that there are no more duplicate rows
print('Number of duplicates after dropping:')
data.duplicated().sum()

# In[79]:

data.info()

# In[80]:

# use stemming to categorize purpose column
from nltk.stem import SnowballStemmer

english_stemmer = SnowballStemmer('english')

data['purpose_words'] = data['purpose'].str.split().apply(lambda x: [english_stemmer.stem(y) for y in x])

def purpose_group(purpose_words):

    if 'wed' in purpose_words:
        return 'wedding'
    elif 'estat' in purpose_words or 'hous' in purpose_words or 'properti' in purpose_words:
        return 'real_estate'
    elif 'car' in purpose_words:
        return 'car'
    else:# 'educ' or 'uni' in purpose_words:
        return 'education'

data['purpose_cat'] = data['purpose_words'].apply(purpose_group)
data.head(10)

# In[81]:

print('Purpose        Count')
data['purpose_cat'].value_counts()

# In[82]:

print('Total number of purpose values')
len(data['purpose_cat'])

# ### Conclusion

# 1. Duplicate rows increased from 54 to 71 after filling in missing values for total_income, but that is still only 0.33%. Duplicate rows can happen when datasets are merged or through human error.
# 2. Duplicate rows were deleted using drop_duplicates() since it is simple to use. Duplicate removal verified.
# 3. 4 categories (real_estate, car, education, wedding) used to filter purpose column.
# 4. Total number of values verified (each purpose_cat assigned a category)

# ### Categorizing Data

# Categorizing data stratifies a large collection of values into groups or levels. It is ideal to use when working with age, income, time or anything that could be continuous but needs to be examined in groups.
#
# 1. Verify total_income properly categorized income_level for both data df and data_median_income df. See section 1.1 for initial categorizing total_income into income_level.
# 2. Categorize children into child_groups.
# 3. See section 1.1 for categorizing dob_years into age_group

# In[83]:

# verfiy new column: income_level
data['income_level'].value_counts()

# In[84]:

# verfiy new column: income_level
data_median_income['income_level'].value_counts()

# In[85]:

# create new column based on no children, 1 child, 2 or more children
# first investigate problem values
data['children'].value_counts()

# In[86]:

# calculate percentage of problem values (-1)
print('Percent of erroneous values (-1)')
print(47/21454)

# In[87]:

# change the -1 values to 1
data.loc[data['children'] < 0, ['children']] = 1

# In[88]:

# calculate percentage of problem values (20)
print('Percent of likely erroneous values (20)')
print(76/21454)

# In[89]:

# change the -1 values to 1
data.loc[data['children'] == 20, ['children']] = 2

# In[90]:

data['children'].value_counts()

# Replaced likely erroneous values (1 for -1) and (2 for 20).

# In[91]:

# create a new function, children_fx
def children_fx(row):
    # the income_level is returned according to total_income
    child = row['children']

    if child == 0:
        return 'no children'
    if child == 1:
        return '1 child'
    else:
        return '2 or more children'

# create a new column, income_level, based on total_income
data['child_groups'] = data.apply(children_fx, axis=1)
print('New categories for child groups')
data['child_groups'].value_counts()

# In[92]:

# verify child_groups added
data.head()

# In[93]:

# verify no missing values
print('Total number of entries')
data['child_groups'].count()

# ### Conclusion

# 1. income_level verfied for both data df and data_median_income df.
# 2. Replaced -1 and 20 values in children, grouped into 3 categories in child_groups. Verified no missing values.

# At this point, clean up df and get rid of cols not used

# In[94]:

# remove days_employed and years_employed columns
del data['days_employed']
del data['years_employed']
del data['dob_years']
del data['purpose_words']
del data['education_id']
del data['family_status_id']
data.head()

# ## Answer these questions

# - Is there a relation between having kids and repaying a loan on time?

# In[95]:

# create a formatting rule for ease of visualization
def format_float(value):
    value = value*100
    return f'{value:,.2f}%'
pd.options.display.float_format = format_float

# In[96]:

# create pivot table with percents per child group category
print('Pecentage of applicants with a history of default')
data.pivot_table(values=['debt'], columns=['child_groups'])

# ### Conclusion

# Yes, there is a clear relationship between debt and no children versus debt and any children. Only 7.54% of applicants without a child defaulted, while appliants with 1 or more children defaulted at a rate of 9.17% to 9.29%. Therefore, a applicant with a child may pose a greater default risk.

# - Is there a relation between marital status and repaying a loan on time?

# In[97]:

# create pivot table with percents per marital status category
print('Pecentage of applicants with a history of default')
data.pivot_table(values=['debt'], columns=['family_status'])

# ### Conclusion

# Yes, there are differences in default history amoung applicants with different family status. Widowers pose the best risk, as only 6.57% of them defaulted on a loan in the past. Both married and divorced applicants pose a moderate risk (between 7.11% and 7.55% defaulted in the past). The biggest risk is for unmarried applicants (9.75%) and those in civil partnerships (9.35%). Therefore, appliants who are unmarried or in a civil partnership may pose the greatest default risk.

# - Is there a relation between income level and repaying a loan on time?

# In[98]:

# create pivot table with percents per income level category
print('Pecentage of applicants with a history of default')
print('for per income level where missing values replaced')
print('based on gender, education level, and income type.')
data.pivot_table(values=['debt'], columns=['income_level'])

# In[99]:

# create pivot table with percents per income level category
print('Pecentage of applicants with a history of default')
print('for per income level (where missing values replaced')
print('by median value')
data_median_income.pivot_table(values=['debt'], columns=['income_level'])

# ### Conclusion

# Interestingly, the percentages for both df (the one where 1 median value replaced all missing values and the one where gender, education level, and income type guided replacement values) are very similar.
#
# Yes, there does seem to be a general trend where the higher the income, the less likely an applicant has defaulted in the past. Those making below60K demonstrate a clear trend towards higher risk as the income drops (7.29%, 7.79%, 8.46%, and then 8.54% for those making less than 20K. The upper half (greater than 60K) defaulted at a rate between 5.13% and 7.23%). Overall, those applicants earning less than 60K may need greater scrutiny as there is a higher likelyhood that they defaulted in the past.

# - How do different loan purposes affect on-time repayment of the loan?

# In[100]:

# create pivot table with percents per purpose category
print('Pecentage of applicants with a history of default')
data.pivot_table(values=['debt'], columns=['purpose_cat'])

# ### Conclusion

# Yes, once again there is a relationship between the purpose of the loan and a history of default. Applicants wishing to purchase a car have the greatest historical defalut rate (9.36%), followed by those who want money for education (9.22%). Applicants looking to buy real estate pose the least risk, as only 7.23% defaulted in the past.
#

# Investigate other columns for potential relationships to guide future analysis.

# In[101]:

# create pivot table with percents per age group category
print('Pecentage of applicants with a history of default')
data.pivot_table(values=['debt'], columns=['age_group'])

# Age group: Looking into age may be worthwhile as there appears to be a sharp increase in historical default for those under 40.

# In[102]:

# create pivot table with percents per gender category
print('Pecentage of applicants with a history of default')
data.pivot_table(values=['debt'], columns=['gender'])

# Gender: It appears male applicants have a higher rate of default in this sample. Further analysis may be useful.

# In[103]:

# create pivot table with percents per income_type category
print('Pecentage of applicants with a history of default')
data.pivot_table(values=['debt'], columns=['income_type'])

# In[104]:

data['income_type'].value_counts()

# Income type: Only the 4 groups (employee, business, retiree, civil servant) with the greatest number of applicants could be used for analysis since a sample of 1 or 2 isn't useful. Still, there does seem to be a significant difference between the lowest risk (retiree 5.64%) and highest risk (employee 9.57%). It would be worth more analysis.

# ## General conclusion

# Number of children, family status, income, and the stated purpose of the loan can be used to increase the validity of the credit scoring system.
#
# Better loan risk (lower percentage of historical defaults on loans):
# - no children
# - widowers
# - married
# - divorced
# - income > 60K
# - purpose of loan: real estate
# - purpose of loan: wedding
#
# Higher loan risk (higher percentage of historical defaults on loans):
# - have children
# - unmarried
# - in a civil union
# - income < 60K
# - purpose of loan: education
# - purpose of loan: car
#
# Additionally, it may be advantageous to pursue analysis of age group, gender, and income type as there do appear to be relationships beween subsections and percentage of defaults.


# my_lambdata/oop_ refactored_script.py
# Refactored script code with OOP approach

import pandas as pd
import numpy as np

class NullFinder():
    def __init__(self, num_list):
        self.num_list = num_list

    def findnulls(self):
        is_null = []
        for x in self.num_list:
            if x is np.NaN:
                is_null.append('Yes')
            else:
                is_null.append('No')
        return is_null

if __name__ == '__main__':
    test_list = [4, 2, 0, np.NaN, 2, 4]
    nullf = NullFinder(test_list)
    print(nullf.findnulls())
    nulls = nullf.findnulls()
    print(nulls[0])


from bootcamp_example import myfunctions

import pytest
import numpy as np

def test_constant_function_is_one():
    # Use pytest.approx, or numpy.allclose when testing for floating point equality
    assert myfunctions.constant_function(123) == pytest.approx(1)

def test_constant_function_typed_is_one():
    assert myfunctions.constant_function_typed(np.random.randn(10)).sum() == pytest.approx(10)

def test_mean_function_is_correct():
    # Intentionally failing test for demonstration purposes
    assert myfunctions.incorrect_mean([0, 2]) == pytest.approx(1)


import pandas as pd
import numpy as np

'''
Standard python indexing methods work fine with pandas data structures but its recommended to use pandas indexing methods which are
heavily optimized
'''
dict = {
    'A' : pd.Series(np.random.randn(5)),
     'g' : pd.Series(np.random.randn(5)),
    'r' : pd.Series(np.random.randn(5)),

}
df = pd.DataFrame(dict)
df.index = [ chr(i) for i in range(65,70)]
print(df)

#selecting using standard selection methods, ny using label or index
#print(df[0],'\n',df[0]) #access using index
print('\n',df['A'],'\n',df['A':'C']) #Access using labels 1st method selects columns 2nd slices through rows
print('\n',df[0:2])
print('\nList selection:\n',df[['A','r']])

#Let's use pandas optimized method accessing
#1)accessing using labels:
print('\n D of A' ,df.loc['D',['A']])
print('\n\n List selection ',df.loc['B':'D',['A','r']])

#2)Selection using index:
print('3\n\n',df.iloc[1]) #print particular column
print('\n\n',df.iloc[0:3,0:2])

#accessing a scalar:
print('\n scalar at 1,1',df.iloc[1,1])
####!!!NOTE: it is recommended to use iat for accessing a scalar as it is optimized
print('\n scalar at 1,1',df.iat[1,1])

'''
More about indexing and advanced indexing:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing
http://pandas.pydata.org/pandas-docs/stable/advanced.html#advanced
'''


# reading file with pandas
# Pandas is a python module to read files and manipulate its contents (and also plot it)
# the read data is stored in what is called as a dataframe (like excel worksheet, which can be viewed in Spyder)
# This example shows how to read file which we wrote in the previous assignment (09-file-write)
# Pandas can also be used to write files.

import numpy as np
import matplotlib.pyplot as plt

# You must also import pandas and if it is not installed use the anaconda navigator -> environments and search for pandas and install it
# You can also see detailed instructions to install in https://github.com/deepaksamuel/python-tutorials and see the README.md
import pandas as pd

# We have no headers in the file, we name the columns as i and i-squared as shown below. The names will be used while plotting
df = pd.read_csv("out.txt",header=None, names=["i","i-squared"]) # the dataframe will be shown in Spyder IDE

print(df.head()) # this prints out the first 5 elements in the file

# you can also plot from a dataframe
df.plot(x="i",y="i-squared")
plt.show()

# if the file has NO header rows, you must state that explicitly
# more info at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

# if you want to have the full view of the dataframe, go to variable explorer in the Spyder IDE and click on the parameter df.
# A separate window will open show the file in an Excel Sheet like view.


import numpy as np
from example import algs

def test_pointless_sort():
    # generate random vector of length 10
    x = np.random.rand(10)

    # check that pointless_sort always returns [1,2,3]
    assert np.array_equal(algs.pointless_sort(x), np.array([1,2,3]))

    # generate a new random vector of length 10
    x = np.random.rand(10)

    # check that pointless_sort still returns [1,2,3]
    assert np.array_equal(algs.pointless_sort(x), np.array([1,2,3]))

def test_bubblesort():
    # Actually test bubblesort here. It might be useful to think about
    # some edge cases for your code, where it might fail. Some things to
    # think about: (1) does your code handle 0-element arrays without
    # failing, (2) does your code handle characters?

    x = np.array([1,2,4,0,1])

    # for now, just attempt to call the bubblesort function, should
    # actually check output
    algs.bubblesort(x)

def test_quicksort():

    x = np.array([1,2,4,0,1])
    # for now, just attempt to call the quicksort function, should
    # actually check output
    algs.quicksort(x)


import pandas as pd

## READ & WRITE
    # Pickle
df = pd.read_pickle('psi.pickle')
df.to_pickle('normal.pkl')
    # CSV
df = pd.read_csv('shenzhen_processed.csv', low_memory=False)
df = pd.read_csv('olympics.csv', index_col=0, skiprows=1)   #take 1st col as index, and remove 1st row
df.to_csv('shenzhen_processed.csv', index=False)
df = pd.read_csv(file, usecols=['col1','col2']) #use only specific columns; can save a lot of memory
    # APPENDING DF TO EXISTING CSV
df = df.to_csv('my_csv.csv', mode='a', header=False)
    # EXCEL
    # reading excel has various differences compared to csv
    # dtype of a col to str will convert NaN into 'nan', while csv preserves the NaN
    # dtype of a col to str will not preserve numeric 0 padding, while csv preserves
df = pd.read_excel('shenzhen_processed.xlsx', sheet_name=0) #sheetname starts from 0
df = pd.read_csv("P00000001-ALL.csv", nrows=20) # limit to only 20 rows
df.to_excel('output.xlsx', index=False)
    # output multiple df in different Excel sheets
from pandas import ExcelWriter
writer = ExcelWriter(xls_path)
for n, df in enumerate(list_dfs):
    df1.to_excel(writer,'sheet%s' % n)
writer.save()
    # TXT
utown=pd.read_table('university_towns.txt', sep=',', header=None)
df1 = pd.read_table('training_text', sep='\|\|', engine='python', skiprows=1, names=["ID","Text"]) #note that any delimiter more than 1 is a reg, have to use \ to override
    # convert a clip board into dataframe!!!
pd.read_clipboard()
    # JSON
df=pd.read_json(path)
df.to_json('/Users/xxx/Desktop/d.json') # display by index
out = df.to_json(orient="records") # display by row, key=colname, value=cell value
dict_ = df.to_dict(orient="list") # display by cols, key=colname, value=list of values
    # DBF
from simpledbf import Dbf5
dbf = Dbf5('test.dbf')
df = dbf.to_dataframe()
    # Sample the data to speed up computation
df = df.sample(frac=0.1, random_state=10)

# set column as string
df = pd.read_csv('sample.csv', dtype={'ID': object}) #no diff if you '' the data type

# encoding error, eg: 'utf-8' codec can't decode byte 0x92 in position 763: invalid start byte
# use below to decode
df = pd.read_csv(email, encoding = "ISO-8859-1")

#--------------------------------------------------------
## SETTINGS
pd.set_option('display.max_columns',1) # expand column height
pd.set_option('display.max_rows', None) # show all rows
pd.set_option('display.max_colwidth', -1) # no limit to column width
pd.reset_option('all') # reset set options

#--------------------------------------------------------
## PERFORMANCE
first_five = pd.read_csv('loans_2007.csv', nrows=5) #call only first 5 rows

df._data #see how BlockManager classify dataframe by dtypes
df.info(memory_usage="deep") # display memory usage, dtype, null/non-null
df.memory_usage(deep=True) # display only memory usage for each column

# step1: filter dataframe to select certain dtype
df_obj = df.select_dtypes(include=['integer'])
# step2: auto-determinine optimal dtype so that memory usage is minimised; eg change form int64 to int16
df['columnNm'] = pd.to_numeric(df['columnNm'], downcast='integer')
df['columnNm'].dtype

# convert objects into category to minimise memory usage as its converted to int backend
# only use this when unique values <50% of rows & that there is no need for numeric calculations
df['columnNm'] = df['columnNm'].astype('category')

#--------------------------------------------------------
## BUILDING A NEW DATAFRAME
    #build dataframe from a for loop
x = 0
list = []
for i in df.columns: # how many nan in each column? Value
    list.append({'column':x, 'nan_count':df[i].isnull().values.sum(), 'variable':i})
    x+=1
df_nan = pd.DataFrame(list)

    #create random dataframe
df1 = pd.DataFrame(np.random.randint(1, 5, (10,2)), columns=['a','b']) #10 rows, 2 columns, with numbers 1 to 5

    #from a dictionary
df = pd.DataFrame()
newdf['Date'] = x.keys()    #where x is a dictionary
df['DateValue'] = x.values()

    # build a df with just one row of data. Note the nested list to change it to row
prediction = pd.DataFrame([[4, 21, 1, 5, 91,1984]], \
                          columns=['flat_type_code','town_code','flat_model_code', \
                                   'storey_range_code','floor_area_sqm', 'lease_commence_date'])

    # from dictionary
d = {'Desert':2345,'Mountain':8764,'Water':6689,'Land':7332,'Forest':1050,'Snow':3741, \
              'Is_Raining_ec':0,'Had_A_Good_Sleep_ec':0,'Average_Temperature':40}
df = pd.DataFrame([list(d.values())], columns=list(d.keys()))

    # from nested list (with headers)
df = pd.DataFrame(data[1:],columns=data[0])

    #duplicate a dataframe
df2 = df.copy()

#--------------------------------------------------------
## EXPLORATORY
df.info() #total non-null rows, dtypes, columns
df.shape #total number of rows by columns
df.size #total number of rows
len(df) #total number of rows too
len(df.columns) #total number of columns
df.head(2) #top 2 rows
df.dtypes #format
df.describe() #mean, std, count, etc. only numeric formated columns

#--------------------------------------------------------
## FORMAT
df.dtypes
df['hour'] = df['hour'].astype('int64')
df['text'] = df['text'].astype('str') # string will not preserve NaN, unlike object
df['col3'] = df['col2'].astype('category') # category type has int code in the backend
    #coerce, any errors will be converted to NaN
df['price'] = pd.to_numeric(df['price'], errors='coerce')
df['Time'] = pd.to_datetime(df['Time'], errors='coerce')

#--------------------------------------------------------
## CHUNK-SIZE
dtypes = {"ConstituentBeginDate": "float", "ConstituentEndDate": "float"}
chunk_iter = pd.read_csv("moma.csv", chunksize=250, dtype=dtypes) #each chunk is 250 rows
lifespans = []
for chunk in chunk_iter:
    diff = chunk['ConstituentEndDate'] - chunk['ConstituentBeginDate']
    lifespans.append(diff)
lifespans_dist = pd.concat(lifespans)
print(lifespans_dist)

#--------------------------------------------------------
## Using SQL

# Connection to database
    # sqlite connection
import sqlite3
conn = sqlite3.connect(sqlitePath)
df= pd.read_sql_query("SELECT * FROM table", conn)

from sqlalchemy import create_engine
engine = sqlalchemy.create_engine('sqlite:///my_db.sqlite')

    # postgres connection
import psycopg2
conn = psycopg2.connect(database="postgres", user="postgres", password="***", host="127.0.0.1", port="5432")
    # OR use sqlalchemy, which supports most databases
    # database engine + database connector package://username:password @ host ip / database? client encoding
    # latin1 or utf8 depending on client encoding
from sqlalchemy import create_engine
import psycopg2
conn = create_engine('postgresql+psycopg2://postgres:password@localhost:5432/postgres?client_encoding=latin1') # postgres
conn = create_engine('mysql+pymysal://{}:{}@{}:{}/{}'.format(username, password, address, port, db_name)) # mysql
query = ''' SELECT * FROM customer  '''

# reading from sql
df = pd.read_sql(query, conn)
df = pd.read_sql_query(query, conn)

# upload dataframe to database as new table by default (use if_exist for appending), only available using sqlalchemy as engine
# if_exists can 'replace' entire table, default is fail
df.to_sql(name='wsg_ap_list3', con=conn, index=False, if_exists='append') #OR
df.to_sql('pa', conn, if_exists='append', index=False, dtype='text')
    # upload using chunk, per 1000
df.to_sql('table', engine, chunksize=20000)

#--------------------------------------------------------
## INDEX NAMES
df.index
df['country'] = df.index #transfer index to a column
df = df.set_index('Gold') #set index from a column

df = df.set_index(['STNAME', 'CTYNAME']) #can set hierachical index
df.loc['Michigan', 'Washtenaw County'] #querying from index

df = df.reset_index(drop=True) #reset index; drop=True to remove original index as a column

df.loc['Animal'] #index name
df.iloc[5:10] #index location; row number

    #iloc can also detect by both index & column
df.iloc[1,0] #index 1, column 0
df.iloc[:,0] #all index, column 0

# ix indexing works just the same as .loc when passed strings
df.ix[['Andrade']] == df.loc[['Andrade']]
# ix indexing works the same as .iloc when passed integers.
df.ix[[33]] == df.iloc[[33]]

#--------------------------------------------------------
## COLUMNS NAMES
    ## identify column names
df.columns
df.columns[:2] ## first 3 columns
    ## show column names and position
x = 0
for i in df.columns:
    print(x, i)
    x += 1
    #renaming columns
df.columns = ['newcolumn1', 'newcolumn2', 'newcolumn3'] #easiest way to change, but error if total columns does not match
df.name = 'Original' #changing pd.Series name
df2 = df.rename(columns={'diam_circle_image':'diameter','depth_rimfloor_topog':'depth', 'number_layers':'layers'})
hdata2.rename(columns=dict(zip(hdata2.columns,date_change.tolist())), inplace=True) #change two lists into dictionary
df.columns = map(str.lower, df.columns) # change to lower case
    #drop columns
df.drop(df.columns[[0, 1, 3]], axis=1)
df.drop('column_name', axis=1, inplace=True) #note that if inplace value is not set to true, need to reassign a new df
del df['column_name']
    #choose column names by condition
col = [i for i in df.columns if i[-4:]=='2012']

    #concat two lists of columns together
df[df.columns[1:11] | df.columns[12:14]]

    #ordering columns in a df
df[sorted(df.columns.tolist())].head(3)
    #specific ordering of columns
df = df[['a', 'b', 'd', 'c']]

#--------------------------------------------------------
## CREATE DATAFRAMES BASED ON UNIQUE COLUMN CATEGORY VALUE, STORED IN A DICT

# https://datascience.stackexchange.com/questions/29825/create-new-data-frames-from-existing-data-frame-based-on-unique-column-values

# store dataframes in dict, based on unique column value 'company_id'
dict_of_companies = {key: value for key, value in df.groupby('company_id')}
# get all keys
keys = [i for i in dict_of_companies]
# print each dataframe out
for i in keys:
    print(dict_of_companies[i])

#--------------------------------------------------------
## SET VALUES PER CELL, GOOD FOR ITERATION
df.set_value(i, 'Y_svy', svy[1]) # index, column name, value

# new alternative
df.at[4, 'B'] = 10 # index, column name = value
df.at[4, 'B'] #querying a cell
# >>> 10

#--------------------------------------------------------
## COUNTING
df['EVENT_TYPE'].value_counts()
    # using groupby
df.groupby(['Fruit','Name'])['Number'].sum() #sum of Number grouping by fruit and name
df.groupby('name')['activity'].value_counts() #multi-dimension counts
df['number_layers'].value_counts(normalize=True)*100 # by percentage
df.describe()

#--------------------------------------------------------
## DELETE ROWS
df.drop(df.index[[2,3,10,20]])

#--------------------------------------------------------
## NAN NULL VALUES
    # note that NAN is only for numerical null values
df.isnull().any().any() # is there any nan in entire dataframe? Boolean
df.isnull().values.sum() #total number of nan in dataframe? Value
df.isnull().any() # which column is the nan in? Boolean
df[df['Timestamp'].isnull()] #filter rows with nan

# how many nan in each column? Value
df.isnull().sum() # by counts
df.isnull().sum() / len(df) # by percent
#filter dataframe to only rows NaN of a specific column
df[df['colnm'].isnull()]

    #drop NaN
df3 = df2.dropna() #drop all rows with nan in any columns
df3 = df2.dropna(how='all') #drop only rows with all nan values
df3 = df2.dropna(threa=2) #drop only rows with 2 or more nan values
df.dropna(subset=['x277_2012'],inplace=True) #drop all rows for specific columns
df[df['Col2'].notnull()] # same as above

    #fill NaN
df = df.fillna(value=99) #change NaN a value
df = df.fillna(method='ffill') #forward filling, note need to sort index
df = df.fillna(method='bfill') #back filling, note to sort index
df['colname'].interpolate(method='linear', limit=2) #interpolation, very useful for timeseries

    #set value as NaN
import numpy as np
df2=df2.replace('nan',np.nan)
df.replace({'99':np.nan}, inplace=True) #multiple rows

    #select null within lambda
df['colnm'] = df['colnm'].apply(lambda x: '' if pd.isnull(x)) else x)

#--------------------------------------------------------
# CHECK NEGATIVE VALUES
#entire df
any(df<0)
# each columns
for i in cfoul.columns:
    if any(cfoul[i]<0) == True:
        print(i)

#--------------------------------------------------------
## SORTING
    # sort by index
df.sort_index(ascending=False) # reverse order
df.sort_values #note no brackets

    # sort by value (column)
df.sort_values(ascending=False)

    # sort 1 column out of many
df2=df[['country','x277_2012']]
df2.sort_values('x277_2012',ascending=False)
    # sort multiple columns
df1.sort_values(['a', 'b'], ascending=[True, False])
df3[['a','b']].sort_values(['a','b'], ascending=[True, True]) # sort a first then b

#--------------------------------------------------------
## ENCODING
encode = {'Y':0, 'N':1}
df['Hired'] = df['Hired'].map(encode)

# Convert dummies back to single column
wild_dummies = df[['Wilderness_Area1','Wilderness_Area2','Wilderness_Area3','Wilderness_Area4']]
wild = wild_dummies.idxmax(axis=1)
wild.name = 'Wilderness' # set pd.Series name
wild = pd.concat([df2['Cover_Type'],wild], axis=1)

#--------------------------------------------------------
## STRING MANIPULATIONS

df['column1'].str.len() # length of each cell
df['column1'].str.strip() # remove spacing front & back. such spaces is not visible in dataframe

# regular expressions is also enabled here
df['text'].str.count(r'\d') # find how many times a digit occurs in each string
df['text'].str.findall(r'(\d?\d):(\d\d)') # group and find the hours and minutes
df['text'].str.replace(r'\w+day\b', '???') # replace weekdays with '???'
df['text'].str.replace(r'(\w+day\b)', lambda x: x.groups()[0][:3]) # replace weekdays with 3 letter abbrevations
    # Extract match to new columns
df['text'].str.extract(r'(\d?\d):(\d\d)') # create new columns from first match of extracted groups
df['text'].str.extractall(r'((\d?\d):(\d\d) ?([ap]m))') # extract the entire time, the hours, the minutes, and the period
df['text'].str.extractall(r'(?P<time>(?P<hour>\d?\d):(?P<minute>\d\d) ?(?P<period>[ap]m))') # extract the entire time, the hours, the minutes, and the period with group names

#--------------------------------------------------------
## FILTERING, SQL WHERE CLAUSE
df[:100] # first 100 rows
df[df['EVENT_TYPE'] == 'Thunderstorm Wind'] # one value
df[df['A'].str.contains("hello")] # SQL like; cannot use if there are NaN values
df[df['EVENT_TYPE'].isin(['Thunderstorm Wind', 'Hail', 'Winter Weather'])] # multiple values, like an SQL where~in clause

# Multiple Conditions, add parenthensis ()!
df3 = df2[(df2['layers']>0) & (df2['depth']>0)] # multiple columns using 'AND'
df3 = df2[(df2['layers']>0) | (df2['depth']>0)] # multiple columns using 'OR'

# REMOVE ROWS
df = df[df['Rating'] != 3]

# SQL NOT IN, note the curly '~' which act as a boolean
list = ['WORLD', 'INCOME', 'DEVELOPING']
df = df[~df['country'].isin(list)]
# SQL NOT LIKE OR
list = ['WORLD', 'ALL', 'DEVELOPING', 'ASIA', 'OTHER', 'MEMBERS', 'INCOME', 'DEVELOPED',  \
        'COUNTRIES', 'SITUATIONS', 'EUROPE', 'STATES']
for i in list:
    df = df[~df['country'].str.contains(i)]

# FILTER ROWS WITH NAN
df[df['mgmtsalary'].isnull()]

# FILTER ROWS WITHOUT NAN
df[df['mgmtsalary'].notnull()]

# CHECK FOR ALPHA OR NUMERIC
df[colnm].str.isnumeric()
df[colnm].str.isalpha()

# FILTER BY INDEX
filter_df  = df[df.index.isin(index_list)]

#--------------------------------------------------------
# set values within a for loop
for i in range(len(df)):
    x = df[longtidue][i]
    y = df[latitude][i]
    svy = pyproj.transform(wgs84, svy21, x, y)
    df.set_value(i, 'X_svy', svy[0]) # row_no, column_nm, value
    df.set_value(i, 'Y_svy', svy[1])

# set value for single cell
df = df.set_value(100, 'colnm', 'value') # row_no, column_nm, value

#--------------------------------------------------------
## UNIQUE VALUES, DUPLICATES
df.nunique() # unique values for each column
df['EVENT_TYPE'].unique() # single column, array
    # multiple columns, dataframe
df[['EVENT_TYPE', 'EVENT_ID']].drop_duplicates()
    # set for entire dataframe but target specific columns
df.drop_duplicates(subset=['col1', 'col2']) # only 1 the pair of a duplicate
df.drop_duplicates(subset=['A', 'C'], keep=False) # drop all duplicates
df.drop_duplicates(subset=['A', 'C'], keep='first') # keep only first of the pair of duplicates (default)
    # display the duplicates only
df[df.duplicated(keep=False)] # all columns
df[df[['colA','colB']].duplicated(keep=False)] # specific columns
    # count number of duplicates
len(df) - len(df.drop_duplicates())

# comparing duplicates between two columns
df[df['States'].ne(df['Region'])]

#group by, add a new count field for duplicate counts
df1 = pd.DataFrame(np.random.randint(1, 15, (100,4)), columns=['a','b','c','d'])
df1.groupby(df1.columns.tolist()).size().reset_index().rename(columns={0:'count'})

    # sometimes, might need to change all to str first as nulls are interferring w groupby
df2 = (df.astype('str').groupby(df.columns.tolist()).size()
         .sort_values(ascending=False)
         .reset_index()
         .rename(columns={0:'duplicates'})
         .replace('nan',np.nan)) #set nulls back to NaN

#--------------------------------------------------------
## DROP ROW
    # by index
df = df.drop(df.index[2938])

#--------------------------------------------------------
## NEW COLUMN CALCAULATIONS
    ## SINGLE COLUMN CONDITION
    ## by summing
df['TOTAL'] = df[df.columns[:]].sum(axis=1)

    ## by function, single column condition
def timerange(x):
    if x >= 0 and x < 3:
        return '00:00-03:00'
    elif x >= 3 and x < 6:
        return '03:00-06:00'
    else:
        return '21:00-24:00'
df['time_range'] = df['hour'].apply(timerange, axis=1)

    ## MULTIPLE COLUMN CONDITION
def peak(x):
    if x['Public Holiday'] == 'National Day' or x['Public Holiday'] == 'New Year''s Day':
        return 'Super Peak'
    if x['dow'] == 'Sunday' or x['dow'] == 'Saturday' or x['dow'] == 'Friday':
        return 'Peak'
    else:
        return 'Non-Peak'
df3['day_period'] = df3.apply(peak, axis=1)

    ## lambda function iterates a simple function. x below refers to each row of Top15{'PopEst']
    ## Note that lambda must contain if-else when using condition
Top15['PopEst']=Top15['PopEst'].apply(lambda x: "{:,}".format(x))
df['data']=df['data'].apply(lambda x: 'true' if x <= 2.5 else 'false')
df['date'] = df['raw'].str.extract('(....-..-..)', expand=False) #note that expand will split it into different columns
    #for this case x refers to the entire dataframe, you have to specify the column within the function.
    #this gives if else conditions from multiple columns
    #have to include axis=1 or will prompt error
ticketcat['funpass_days'] = ticketcat.apply(lambda x: '2' if x['ItemDescription'].find('2Day')>=0 else x['funpass_days'],axis=1)

    ## if else using np.where
df['logic'] = np.where(df['AAA'] > 5,'high','low')

    # calculating current to one row below
df['new'] = df['Reviewed']-df['Reviewed'].shift()
df['new'] = df['Reviewed'].diff() #this is more straightforward

    ## quartile cut
df3['diameter2'] = pd.qcut(df3.diameter, 2, labels=['<50%','>50%'])
    ## equal interval cut
df3['diameter2'] = pd.cut(df3.diameter, 2, labels=['<50%','>50%'])

df['hour'] = df['time'].str[:2] #select 1st 2 left characters in a string
    ## multiple columns
df3['DIFF_DAY'] = df3['END_DAY']-df3['BEGIN_DAY']

    ##Boolean, contains
df['StatesT']=df['states].str.contains('edit')

    ## calculate new column row by row
for i in range(len(df)):
    x = df[longtidue][i]
    y = df[latitude][i]
    svy = pyproj.transform(wgs84, svy21, x, y)
    df.set_value(i, 'X_svy', svy[0]) #set_value(index, column, value)
    df.set_value(i, 'Y_svy', svy[1])

    ## regex
df['date'] = df['raw'].str.extract('(....-..-..)', expand=True) #extract
# 0    2014-12-23
# 1    2010-02-23
# 2    2014-06-20
# 3    2014-03-14
df['node'] = df['node'].str.replace(r'[1-9]+\s','') #replace

    ## split by delimiter
# example of value 'Online,Sales,Adult'
ticketcat['sales'] = ticketcat['TicketDescription'].apply(lambda x: x.split(',')[1])
ticketcat['medium'] = ticketcat['TicketDescription'].apply(lambda x: x.split(',')[0])

    ## using numpy, if > 3, x=1 else x=0
df['Positively Rated'] = np.where(df['Rating'] > 3, 1, 0)

    ## remove words duplicate, assuming there is a delimiter of '|' btw words
def removedup(x):
    x = x.split('|')
    x = set(x)
    x = '|'.join(x)
    return x

df['colnm'] = df['colnm'].apply(removedup)
df['colnm'] = df['colnm'].apply(lambda x: '|'.join(set(x.split('|')))) #alternatively

#--------------------------------------------------------
# SHIFT COLUMN VALUES UP OR DOWN; LAG OR LEAD
df['temp_observed'] = df['temp_observed'].shift(periods=1) #push column down by a row
df['temp_observed'] = df['temp_observed'].shift(periods=-1) #push column up by a row
# compare row with previous
df['match'] = df.col1.eq(df.col1.shift())

#--------------------------------------------------------
# ITERATION
for index, row in df.iterrows():
    print(index, row['Desert'])

#--------------------------------------------------------
# REPLACE VALUES
    #option 1: single value
df = df.replace('value1', 'value2')
    #option 2: multiple values
dict={4:88, 1:11}
df1['a'].replace(dict,inplace=True)
    #option 3
df['Country'].apply(lambda x: dict.get(x,x))
    #option 4: replace part of string in value
ticketcat['price']=ticketcat['price'].str.replace('$', '')

#--------------------------------------------------------
# CONFUSION MATRIX
pd.crosstab(df['target'], df['predicted'])

#--------------------------------------------------------
## GROUP BY AND CALCULATING
df[['STNAME', 'COUNTY']].groupby(['STNAME']).sum() #SELECT sum(county), stname FROM tablenm GROUP BY stname
df3.groupby(['longitude', 'latitude']).count() #shows all column counts
df3.groupby(['longitude', 'latitude']).sum()
df.groupby(['LocationDescription','LocationCode']).size() #size include NAN counts, counts() does not, shows row size instead of columns

    #group by to show just top 3 records for each STNAME
df.groupby(['STNAME']).head(3)

## GROUPBY AGGREGATIONS
# https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.core.groupby.DataFrameGroupBy.agg.html

    #multiple aggregations
Top15.groupby('Continent')['PopEst'].agg({'size': np.count_nonzero, 'mean': np.mean, 'sum': np.sum, 'std': np.std})

    #merge nearly duplicate rows based on column value
    # https://stackoverflow.com/questions/36271413/pandas-merge-nearly-duplicate-rows-based-on-column-value
df.groupby(['Name','Sid','Revenue'])['Use_Case'].apply(''.join).reset_index()
df.groupby('Name').agg({'Sid':'first',
                        'Use_Case': ', '.join,
                        'Revenue':'first' }).reset_index()

# GROUPBY OUTPUT AS A DATAFRAME
    # note the as_index
df.groupby('StationID', as_index=False)['BiasTemp'].mean()
#   StationID  BiasTemp
# 0        BB       5.0
# 1     KEOPS       2.5
# 2    SS0279      15.0

# GROUPBY & STORE AS A LIST OF DF
listdf = [df for date, df in df.groupby('date')]

#--------------------------------------------------------
## SIMPLE MATHS
max(df['Gold']) #get the max value in the column
new3['CENSUS2010POP'].nlargest(3) #get top 3 by number; note all columns still show
new3['CENSUS2010POP'].nsmallest(3) #get bottom 3 by number

Top15['Avg']=Top15[df.columns[10:]].mean(axis=1) #mean for columns

Top15[['est','cit']].corr() #default is pearson's; give a correlation matrix
# can add method='pearson'

df2=df.pct_change() #difference of each row and the next with percentage. good for calculating stocks daily retuns
sma10 = CMT['Close'].rolling(10).mean() #calcluating moving averages
df.std() #standard deviation

CMT['Adj Close'].quantile(0.75) #get value by quantile, in this case 75%

#--------------------------------------------------------
## TRANSPOSING
df.T

#--------------------------------------------------------
## JOINS
# http://pandas.pydata.org/pandas-docs/stable/comparison_with_sql.html#compare-with-sql-join
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html
    #default is inner join
    #can also use left_on=, right_on=
df = pd.merge(df1, df2, on='Country')
df = pd.merge(df1, df2, how='left', left_on=['id_key'], right_on=['fk_key']) #if join fields are different
df = pd.merge(df1, df2, how='left', left_on=['id_key','field2','field3'], right_on=['fk_key','field2','field3']) #multiple join fields
hudf=pd.merge(hdf, ul, how ='left', on=['State','RegionName']) #join on multiple columns
    #indicator give an additional field '_merge'.
    # Can use groupby size to count number of 'left_only', 'right_only', or both
df=pd.merge(df1, df2, how='outer', on='Country', indicator=True)
df.groupby('_merge').size()

    #join by index
df=pd.concat([df1,df2], axis=1, join_axes=[df1.index])
df=pd.concat([df1,df2], axis=1) #or if they are already sorted properly

## UNION
df1=pd.read_csv('Redemption_Part1.csv')
df2=pd.read_csv('Redemption_Part2.csv')
df3=pd.read_csv('Redemption_Part3.csv')
frames=[df1,df2,df3]
df_new = pd.concat(frames)

## APPEND
df = df.append(df2, ignore_index=True)

#--------------------------------------------------------
## APPLY TO ALL CELLS IN DF

df.applymap(function_to_change)

#--------------------------------------------------------
## COMBINE TWO OR MORE DATAFRAMES

# all must be numerical
# sum
df1 + df2
# multiply
df1 * df2

#--------------------------------------------------------
## ENCODE CATEGORICAL TO INTEGERS

df2['Gene'] = df2['Gene'].astype('category') # first change an object to category
df2['code'] = df2['Gene'].cat.codes     # then extract their code out
# when training a model, can just use ".cat.codes" to fit the model

# or use the map function
replace = {'Logistics': 1, 'Commercial':2, 'Hospitality': 3, 'Retail': 4,
           'Medical': 5, 'Industrial': 6, 'Retail/Commercial': 7}
df['Type_Cat'] = df['Type'].map(replace)

#--------------------------------------------------------
## DATES
    #change string to date format
col = pd.to_datetime(df.columns[6:])
col = pd.to_datetime(df['date'],dayfirst=True)  #sometimes the auto-format is wrong, and you need specify dayfirst or yearfirst
df['timestamp'] = pd.to_datetime(df['timestamp'], format ='%d/%m/%Y') #sometimes the day, mth or yr are mixed up with each other, so we have to specify directly
df["timestamp"] = pd.to_datetime(df["timestamp"], format='%Y-%m-%d %H:%M:%S')

    #set datetime col to index
df = df.set_index('datetime')

    #aggregation by date intervals
    #list of rules can be found in url: http://pandas.pydata.org/pandas-docs/stable/timeseries.html#offset-aliases
    #date must be in index, while the dataframe contains only the col you want to aggregate
date=date.resample('Q').mean()  #by quarter, also (Day: D, Week: W, Month: M, Quarter: Q, Year: Y)
df.resample('1Min').mean()  #by seconds (1S), minutes (1Min)

    #change date to string
date_yr=a['column_nm'].dt.strftime('%Y') #change to year

    #change to hour / mth
df['hour'] = df['Time'].dt.hour
df['day'] = df['Time'].dt.day
df['mth'] = df['Time'].dt.month
df['dayofweek'] = df['timestamp'].dt.dayofweek
    #change to day of week
df['day_of_week'] = df['my_dates'].dt.weekday_name
    # change to date or time
df['date'] = df['datetime'].dt.date
df['time'] = df['datetime'].dt.time
    #set constant for date, minute, second
df2['hour'] = pd.to_datetime('1900-01-01') + pd.to_timedelta(df3['Time'].dt.hour, unit='H')

    # from epoch, i.e., seconds since 1970
    # check datetime format: https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
dfr['event_ts']=  dfr['Event-Timestamp'].apply(lambda x: time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(x)))
dfr['event_ts']= pd.to_datetime(dfr['Event-Timestamp'], unit='s', errors='coerce') #note that this STILL needs to convert to local time

    # filter by list of dates
df[df['Timestamp'].dt.date.isin(dates)]

#--------------------------------------------------------
## Split commas in cells and stack them in new rows
    # https://stackoverflow.com/questions/17116814/pandas-how-do-i-split-text-in-a-column-into-multiple-rows

# 1) isolate column with comma, split them, and stack them as single rows
p = df['producer'].str.split(', ').apply(pd.Series, 1).stack()
# 2) remove multi dimensions index
p.index = p.index.droplevel(-1)
# 3) add series must have a name
p.name = 'producer'
# 4) use inner join to add based on index value
df.join(p)

#--------------------------------------------------------
## STYLING
# https://pbpython.com/styling-pandas.html
import seaborn as sns
df.style.background_gradient(cmap='Greens')


def test_uppercase():
    assert "loud noises".upper() == "LOUD NOISES"

def test_reversed():
    assert list(reversed([1, 2, 3, 4])) == [4, 3, 2, 1]

import numpy as np

def test_squaring():
	'Test calculating the element-wise square of an array'
	a = np.array([1., 2., 3.])
	a_squared = np.array([1., 4., 9.])
	np.testing.assert_array_almost_equal(a**2, a_squared)

def test_sum():
	a = np.array([1.2, -1.0])
	assert a.sum() == 0.2
	# np.testing.assert_approx_equal(a.sum(), 0.2) # Try this instead!


import numpy as np
import pandas as pd

data_url = 'https://github.com/ine-rmotr-projects/project-files/files/4086772/playstore.xlsx'


import cv2
import numpy as np

def find_road_number(image: np.ndarray) -> int:
    """
    Найти номер дороги, на которой нет препятсвия в конце пути.

    :param image: исходное изображение
    :return: номер дороги, на котором нет препятсвия на дороге
    """
    road_number = None
    # Ваш код тут
    pass
    # Ваш код тут

    return road_number


import numpy as np     #the Numpy library
import matplotlib.pyplot as pyplot   #Matplotlibs pyplot

import sys      #give access to a c-like sys library
import os       #give access to operating sys

print(sys.argv)    #prints any command line arguments, incl program name
print(os.getcwd()) #print the current working directory


# %% [markdown]
# [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jun-hyeok/SUP5001-41_Deep-Neural-Networks_2022Spring/blob/main/DNN_HW4/pt_1.ipynb)

# %% [markdown]
# # DNN HW4 : Part 1
#
# 2022.03.16
# 박준혁

# %%
from util import *

# %% [markdown]
# ## Part 1: Python

# %% [markdown]
# ### 1-1. Python: List

# %%
print("1-1", color=GRAY)

# define
list1 = [1, 2, 3]

# index
print(f"{type(list1) = }")
print(f"{list1[0], list1[-1] = }")

# element
list1[1] = "str1"
print(f"{list1[1] = }")

# range
list2 = range(10)
print(f"{type(list2) = }")

# type cast
list2 = list(list2)
print(f"{type(list2) = }")

# %% [markdown]
# ### 1-2. Python: List

# %%
print("1-2", color=GRAY)

# slice range
print(f"{list2[2:4] = }")

# slice from # to end
print(f"{list2[2:] = }")

# slice from start to #
print(f"{list2[:2] = }")

# slice all (from start to end)
print(f"{list2[:] = }")

# slice step
print(f"{list2[::2] = }")

# slice neg index
print(f"{list2[:-1] = }")

# assign new list to sliced list
print()
print(f"{list2 = } : before")
list2[2:4] = [8, 9]
print("list2[2:4] = [8, 9]", color=GRAY, input=True)
print(f"{list2 = } : after")

print()
print(f"{list2 = } : before")
list2[2:5] = [8, 9]
print("list2[2:4] = [8, 9]", color=GRAY, input=True)
print(f"{list2 = } : after")

# %% [markdown]
# ### 1-3. Python: For Loop with List

# %%
print("1-3", color=GRAY)

# define animals list
animals = ["cat", "dog", "monkey"]

# print for loop
for animal in animals:
    print(f"{animal = }")
print()

# define nums list
nums = [0, 1, 2, 3, 4]
# empty list
squares = []

# calculate square of nums
for num in nums:
    squares.append(num**2)
print(f"{squares = }")

# %% [markdown]
# ### 1-4. Python: Function

# %%
print("1-4", color=GRAY)

# define function sign(x)
def sign(x):
    if x > 0:
        return "positive"
    elif x < 0:
        return "negative"
    else:
        return "zero"

# print sign of [-1, 0, 1]
for x in [-1, 0, 1]:
    print(f"{x = :2}, {sign(x) = }")

# %% [markdown]
# ### 1-5. Python: Function

# %%
print("1-5", color=GRAY)

# define function hello(name, loud=False)
def hello(name, loud=False):
    if loud:
        print(f"HELLO, {name.upper()}!")
    else:
        print(f"Hello, {name}!")

# hello loud default
print("hello('Bob')", color=GRAY, input=True)
hello("Bob")

# hello loud
print("hello('Fred', loud=True)", color=GRAY, input=True)
hello("Fred", loud=True)

# %% [markdown]
# ### 1-6. Python: zip

# %%
print("1-6", color=GRAY)

# print 1 4 7  / 2 5 8 / 3 6 9
for x, y, z in zip(range(1, 4), range(4, 7), range(7, 10)):
    print(f"{x, y, z = }")
print()

# error
try:
    for x, y in zip(range(1, 4), range(4, 7), range(7, 10)):
        print(f"{x, y, z = }")
except Exception as e:
    print(f"x, y, z = {type(e).__name__}: {e}", color=RED)

# %% [markdown]
# ### 1-7. Python: Magic method \_\_init\_\_()

# %%
print("1-7", color=GRAY)

# define class HelloWorld
class HelloWorld:
    def __init__(self):
        print("init")

print("helloworld = HelloWorld()", color=GRAY, input=True)
helloworld = HelloWorld()

# %% [markdown]
# ### 1-8. Python: Magic method \_\_call\_\_()

# %%
print("1-8", color=GRAY)

class HelloWorld:
    def __init__(self):
        print("init", color=GRAY)

print("helloworld = HelloWorld()", color=GRAY, input=True)
helloworld = HelloWorld()

# helloworld is callable?
print(f"{callable(helloworld) = }")
try:
    print(f"{helloworld() = }")
except Exception as e:
    print(f"helloworld() = {type(e).__name__}: {e}", color=RED)
print()

class HelloWorld:
    def __init__(self):
        print("init", color=GRAY)

    def __call__(self):
        print("Hello world")

print("helloworld = HelloWorld()", color=GRAY, input=True)
helloworld = HelloWorld()

print("helloworld()", color=GRAY, input=True)
helloworld()

# %% [markdown]
# ## Part 1: NumPy

# %% [markdown]
# ### NumPy: Install

# %%
try:
    import numpy as np
except ImportError:
    print("numpy is not installed", color=RED)
    import pip

    pip.main(["install", "numpy"])
else:
    print("numpy is installed", color=GREEN)

# %% [markdown]
# ### 1-9. Numpy: np.array

# %%
print("1-9", color=GRAY)

import numpy as np

# rank 1 array
a = np.array([1, 2, 3])

print(f"{type(a) = }")
print(f"{a.shape = }")
print(f"{a.ndim = }")
print(f"{a[0], a[1], a[2] = }")
print()

print(f"{a = } : before")
a[0] = 5
print("a[0] = 5", color=GRAY, input=True)
print(f"{a = } : after")
print()

# rank 2 array
b = np.array([[1, 2, 3], [4, 5, 6]])

print(f"{b.shape = }")
print(f"{b.ndim = }")
print(f"{b[0, 0], b[0, 1], b[0, 2] = }")

# %% [markdown]
# ### 1-10. Numpy: np.zeros, np.ones, np.full, np.eye, np.random

# %%
print("1-10", color=GRAY)

import numpy as np

# array with all zeros
a = np.zeros((2, 2))
print(f"{a = }")

# array with all ones
b = np.ones((1, 2))
print(f"{b = }")

# array with the specific value
c = np.full((2, 2), 7)
print(f"{c = }")

# 2x2 identity matrix
d = np.eye(2)
print(f"{d = }")

# array with random values
e = np.random.random((2, 2))
print(f"{e = }")

# %% [markdown]
# ### 1-11. Numpy: np.where

# %%
print("1-11", color=GRAY)

import numpy as np

# array with 0 to 9
a = np.arange(10)

# np.where(condition)
print(f"{np.where(a < 5) = }")

# np.where(condition, x, y)
print(f"{np.where(a < 5, a, 10 * a) = }")

# np.where(condition, x, y)
print(f"{np.where(a < 4, a, -1) = }")

# %% [markdown]
# ### 1-12. Numpy: Array slice

# %%
print("1-12", color=GRAY)

import numpy as np

# 3x4 array
a = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])

# slice the array a, the first 2 rows and column 1, 2
# shape of sub-array is (2, 2)
b = a[:2, 1:3]

# sub-array refer to the same array as original array, so
# change the sub-array will change the original array as well
# b[0, 0] == a[0, 1]
print(f"{b[0,0] == a[0,1] = }", color=GRAY)
print(f"{a[0, 1] = } : before")
b[0, 0] = 77
print("b[0, 0] = 77", color=GRAY, input=True)
print(f"{a[0, 1] = } : after")

# %% [markdown]
# ### 1-13. Numpy: Array reshape

# %%
print("1-13", color=GRAY)

import numpy as np

# 3x4 array
a = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])

# reshape the array a to be a 2x6 array
b = np.reshape(a, (2, 6))
print(f"{b = }")

# reshape also refer to the same array as original array, so
# change the reshaped-array will change the original array as well
# b[0, 0] == a[0, 0]
print(f"{b[0,0] == a[0,0] = }", color=GRAY)
print(f"{a[0, 0] = } : before")
b[0, 0] = 77
print("b[0, 0] = 77", color=GRAY, input=True)
print(f"{a[0, 0] = } : after")

# %% [markdown]
# ### 1-14. Numpy: Array reshape

# %%
print("1-14", color=GRAY)

import numpy as np

# 2 arrays with shape (2, 2)
a = np.array([[1, 0], [0, 1]])
print(f"{a = }", color=GRAY, input=True)
b = np.array([[4, 1], [2, 2]])
print(f"{b = }", color=GRAY, input=True)

# dot product of a and b
print(f"{np.dot(a, b) = }")

# %% [markdown]
# ## Part 1: Pytorch

# %% [markdown]
# ### Pytorch: Install

# %%
try:
    import torch
except ImportError:
    print("torch is not installed", color=RED)
    import pip

    pip.main(["install", "torch"])
else:
    print("torch is installed", color=GREEN)

# %% [markdown]
# ### 1-15. Pytorch: Tensor

# %%
print("1-15", color=GRAY)

import numpy as np
import torch

# float type tensor
t1 = torch.FloatTensor([0, 1, 2, 3, 4, 5, 6])
t2 = torch.tensor(np.arange(7))

print(f"{t1.shape = }")
print(f"{t2.shape = }")
print(f"{t1.dim() = }")
print(f"{t1.size() = }")
print(f"{t1[:2] = }")
print(f"{t1[3:] = }")

# %% [markdown]
# ### 1-16. Pytorch: NumPy array vs PyTorch tensor

# %%
print("1-16", color=GRAY)

import numpy as np
import torch

b = np.arange(7)
t1 = torch.FloatTensor([0, 1, 2, 3, 4, 5, 6])
print(f"{b = }")
print(f"{t1 = }")
print(f"{type(b) = }")
print(f"{type(t1) = }")
print()

# transform numpy array to torch tensor
tt = torch.tensor(b)
t_from = torch.from_numpy(b)
print(f"{tt = }")
print(f"{t_from = }")
print(f"{type(tt) = }")
print(f"{type(t_from) = }")

# %% [markdown]
# ### 1-17. Pytorch: NumPy array vs PyTorch tensor

# %%
print("1-17", color=GRAY)

print(f"{tt = } : before")
print(f"{t_from = } : before")
b[0] = -10
print("b[0]= -10", color=GRAY, input=True)
print(f"{tt = } : after")
print(f"{t_from = }: after")
print()

# transform torch tensor to numpy array
t_to_np = t_from.numpy()
print(f"{t_to_np = }")
print(f"{type(t_to_np) = }")

# %% [markdown]
# ### 1-18. Pytorch: Broadcasting

# %%
print("1-18", color=GRAY)

import numpy as np
import torch

# addition of two tensors
m1 = torch.FloatTensor([[3, 3]])
print(f"{m1 = }", color=GRAY, input=True)
m2 = torch.FloatTensor([[2, 2]])
print(f"{m2 = }", color=GRAY, input=True)
print(f"{m1 + m2 = }")

# vector + scalar
m1 = torch.FloatTensor([[1, 2]])
print(f"{m1 = }", color=GRAY, input=True)
m2 = torch.FloatTensor([3])
print(f"{m2 = }", color=GRAY, input=True)
print(f"{m1 + m2 = }")

# 2x1 vector + 1x2 vector
m1 = torch.FloatTensor([[1, 2]])
print(f"{m1 = }", color=GRAY, input=True)
m2 = torch.FloatTensor([[3], [4]])
print(f"{m2 = }", color=GRAY, input=True)
print(f"{m1 + m2 = }")

# %% [markdown]
# ### 1-19. Pytorch: torch.mul vs torch.matmul

# %%
print("1-19", color=GRAY)

m1 = torch.FloatTensor([[1, 2], [3, 4]])
print(f"{m1 = }", color=GRAY, input=True)
m2 = torch.FloatTensor([[1], [2]])
print(f"{m2 = }", color=GRAY, input=True)
print(f"{m1 * m2 = }")
print(f"{m1.mul(m2) = }")
print(f"{m1.matmul(m2) = }")

# %% [markdown]
# ### 1-20. Pytorch: torch.view (np.reshape in PyTorch)

# %%
print("1-20", color=GRAY)

import numpy as np
import torch

t = np.arange(12).reshape(-1, 2, 3)
floatT = torch.FloatTensor(t)

print(f"{floatT.shape = }")
print(f"{floatT.view([-1, 3]) = }")


import numpy as np
import torch
import os
import random

def set_seed(seed = int):
    '''Sets the seed of the entire notebook so results are the same every time we run.
    This is for REPRODUCIBILITY.'''
    np.random.seed(seed)
    random_state = np.random.RandomState(seed)
    random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    os.environ['PYTHONHASHSEED'] = str(seed)
    return random_state


import numpy as np
import pandas as pd
from pandas import Series,DataFrame

s1=Series([10,20,30,40],index=['A','B','C','D'])
print(s1)

index1=s1.index
print(index1)

#index operations
print(index1[2])
print(index1[2:])

#negative indexes
print(index1[-2:]) #ignores the first 2 elemnts and prints the rest
print(index1[:-2]) #ignores the last 2 elemnts and prints the rest

#range of indexes
print(index1[1:4])

#interesting

#index1[0]='a'  #index cannot be changed using =  operator
#print(index1)


# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.13.8
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# # Raster-vector interactions {#raster-vector}
#
# ## Prerequisites

#| echo: false
import pandas as pd
import matplotlib.pyplot as plt
pd.options.display.max_rows = 6
pd.options.display.max_columns = 6
pd.options.display.max_colwidth = 35
plt.rcParams["figure.figsize"] = (5, 5)

# Let's import the required packages:

import numpy as np
import matplotlib.pyplot as plt
import geopandas as gpd
import rasterio
import rasterio.mask
from rasterio.plot import show

# and load the sample data:

src_srtm = rasterio.open("data/srtm.tif")
zion = gpd.read_file("data/zion.gpkg")
zion_points = gpd.read_file("data/zion_points.gpkg")

# ## Introduction
#
# ## Raster cropping
#
# Many geographic data projects involve integrating data from many different sources, such as remote sensing images (rasters) and administrative boundaries (vectors). Often the extent of input raster datasets is larger than the area of interest. In this case raster **cropping** and **masking** are useful for unifying the spatial extent of input data. Both operations reduce object memory use and associated computational resources for subsequent analysis steps, and may be a necessary preprocessing step before creating attractive maps involving raster data.
#
# We will use two objects to illustrate raster cropping:
#
# * The `srtm.tif` raster representing elevation (meters above sea level) in south-western Utah
# * The `zion.gpkg` vector layer representing the Zion National Park
#
# Both target and cropping objects must have the same projection. The following reprojects the vector layer `zion` into the CRS of the raster `src_srtm`:

zion = zion.to_crs(src_srtm.crs)

# To mask the image, i.e., convert all pixels which do not intersect with the `zion` polygon to "No Data", we use the `rasterio.mask.mask` function as follows:

out_image_mask, out_transform_mask = rasterio.mask.mask(
    src_srtm,
    zion["geometry"],
    crop=False,
    nodata=9999
)

# Note that we need to specify a "No Data" value in agreement with the raster data type. Since `srtm.tif` is of type `uint16`, we choose `9999` (a positive integer that is guaranteed not to occur in the raster).
#
# The result is the `out_image` array with the masked values:

out_image_mask

# and the new `out_transform`:

out_transform_mask

# Note that masking (without cropping!) does not modify the raster spatial configuration. Therefore, the new transform is identical to the original:

src_srtm.transform

# Unfortunately, the `out_image` and `out_transform` object do not contain any information indicating that `9999` represents "No Data". To associate the information with the raster, we must write it to file along with the corresponding metadata. For example, to write the cropped raster to file, we need to modify the "No Data" setting in the metadata:

out_meta = src_srtm.meta
out_meta.update(nodata=9999)
out_meta

# Then we can write the cropped raster to file:

new_dataset = rasterio.open("output/srtm_masked.tif", "w", **out_meta)
new_dataset.write(out_image_mask)
new_dataset.close()

# Now we can re-import the raster:

src_srtm_mask = rasterio.open("output/srtm_masked.tif")

# The `.meta` property contains the `nodata` entry. Now, any relevant operation (such as plotting) will take "No Data" into account:

src_srtm_mask.meta

# Cropping means reducing the raster extent to the extent of the vector layer:
#
# * To crop *and* mask, we can use the same in `rasterio.mask.mask` expression shown above for masking, just setting `crop=True` instead of `crop=False`.
# * To just crop, *without* masking, we can derive the extent polygon and then crop using it.
#
# For example, here is how we can obtain the extent polygon of `zion`, as a `shapely` geometry object:

bb = zion.unary_union.envelope
bb

# The extent can now be used for masking. Here, we are also using the `all_touched=True` option so that pixels partially overlapping with the extent are included:

out_image_crop, out_transform_crop = rasterio.mask.mask(
    src_srtm,
    [bb],
    crop=True,
    all_touched=True,
    nodata=9999
)

# Figure ... shows the original raster, and the cropped and masked results.

fig, axes = plt.subplots(ncols=3, figsize=(9,5))
show(src_srtm, ax=axes[0])
zion.plot(ax=axes[0], color="none", edgecolor="black")
show(src_srtm_mask, ax=axes[1])
zion.plot(ax=axes[1], color="none", edgecolor="black")
show(out_image_crop, transform=out_transform_crop, ax=axes[2])
zion.plot(ax=axes[2], color="none", edgecolor="black")
axes[0].set_title("Original")
axes[1].set_title("Mask")
axes[2].set_title("Crop");

# ## Raster extraction
#
# From points...
#
# From line...
#
# From polygon (srtm)...
#
# From polygon (nlcd)...
#
# ## Rasterization
#
# ...
#
# ## Spatial vectorization
#
# Spatial vectorization is the counterpart of rasterization (Section ...), but in the opposite direction. It involves converting spatially continuous raster data into spatially discrete vector data such as points, lines or polygons.
#
# There are three standard methods to convert a raster to a vector layer:
#
# * Raster to polygons
# * Raster to points
# * Raster to contours
#
# The most straightforward form of vectorization is the first one, converting raster cells to polygons, where each pixel is represented by a rectangular polygon. The second method, raster to points, has the additional step of calculating polygon centroids. The third method, raster to contours, is somewhat unrelated. Let us demonstrate the three in the given order.

src = rasterio.open("data/grain.tif")

# To polygons... ...
#
#
# FIGURE 6.9: Illustration of vectorization of raster (left) into polygons (dissolve = FALSE; center) and aggregated polygons (dissolve = TRUE; right).

src = rasterio.open("data/elev.tif")

# To points...

src = rasterio.open("data/elev.tif")

# To contours...
#
# ...
#
# Another common type of spatial vectorization is the creation of contour lines representing lines of continuous height or temperatures (isotherms) for example. We will use a real-world digital elevation model (DEM) because the artificial raster elev produces parallel lines (task for the reader: verify this and explain why this happens). Contour lines can be created with the terra function as.contour(), which is itself a wrapper around filled.contour(), as demonstrated below (not shown):
#
# Contours can also be added to existing plots with functions such as contour(), rasterVis::contourplot() or tmap::tm_iso(). As illustrated in Figure 6.8, isolines can be labelled.
#
# The final type of vectorization involves conversion of rasters to polygons. This can be done with terra::as.polygons(), which converts each raster cell into a polygon consisting of five coordinates, all of which are stored in memory (explaining why rasters are often fast compared with vectors!).
#
# This is illustrated below by converting the grain object into polygons and subsequently dissolving borders between polygons with the same attribute values (also see the dissolve argument in as.polygons()).
#
#
#
# ## Exercises
#