# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.4'
#       jupytext_version: 1.2.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
#

#

#

#
Scientific Visualization with Python

#
A super quick crash course

#

#

#

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# Seismo-Live: http://seismo-live.org
#
# ##### Authors:
# * Stephanie Wollherr ([@swollherr](https://github.com/swollherr))
#
# ---

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# This notebook introduces some basic plotting examples using matplotlib.

# + {"deletable": true, "editable": true}
#we need the following packages, always execute this cell at the beginning

#plots inside the notebook
# %matplotlib inline

#you can use these libraries by refering to their appreviation plt., np. or pd.
#basic plotting library
import matplotlib.pyplot as plt

#scientifc computing library
import numpy as np

#data analysis tool
import pandas as pd

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# # Reading in your data
#
# To handle data in python we first have to read it in.
# ### Simple ascii/txt/csv files.
#
# #### loadtxt by numpy

# + {"deletable": true, "editable": true}
#read in a seismogram
#4 columns with
#time North-South East-West Up-Down
time, ns, ew, ud = np.loadtxt('data/station_1.dat').T
print(time)
print(ns)

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# usefull parameters for loadtxt:
# - comments : The characters or list of characters used to indicate the start of a comment; default: ‘#’.
# - skiprows : Skip the first skiprows lines; default: 0.
# - usecols : Which columns to read, with 0 being the first. For example, usecols = (1,4,5) will extract the 2nd, 5th and 6th columns. The default, None, results in all columns being read.

# + {"deletable": true, "editable": true}
#in action: we only need the time series and the North-South component
time, ns = np.loadtxt('data/station_1.dat', usecols=(0,1)).T
print(time)
print(ns)

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# #### read_csv by pandas
# much faster than loadtxt, in particular for large files

# + {"deletable": true, "editable": true}
#read it in without any specifications
data_pd = pd.read_csv('data/station_1.dat')
print (data_pd)

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# **It reads in everything!**
#
# So we have to set a couple of parameters. Possible parameters that we can set:
# - sep : string; delimiter to use; default ‘,’
# - header : integer or list of integers; row number(s) to use as the column names and the start of the data
# - names :  List of column names to use. If file contains no header row, then you should explicitly pass header=None; array-like; default None.
# - usecols : Return a subset of the columns; array-like or callable; default None.
# - skiprows : Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file; list-like or integer or callable; default None.
#
# ... and many more: full list available at https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html
#
# Let's try this again:

# + {"deletable": true, "editable": true}
#files uses tab-seperation, so we'll let read_csv know
#names can be used for header names, header can also be read in but now we're skipping them with comment='#'
data_pd = pd.read_csv('data/station_1.dat', sep = '\t', comment='#', names=['time','NS','EW', 'UD'])
print (data_pd)

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# Now we can nicely refer to our data with:

# + {"deletable": true, "editable": true}
print('time')
print (data_pd['time'])

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ### Other data formats
#
# There are other possibilties to read in your data when it's not in ascii format.
#
# For example, we will later use netcdf-based data formats in the second tutorial.
#
# More packages for handling data input:
# - netcdf4 package (Network Common Data Form)
# - read in hdf with panda (Hierarchical Data Format)
#
# **What kind of data format are you using?**
#
#
#

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## Plotting your data - Simple Plots
#
# Matplotlib is a widely used plotting library that we will use here for our first simple plots.
#
# 1. Single Figures
# 2. Axis labels, titles
# 3. Subplots
# 4. Styles
# 5. Scatter plots
# 6. Other types of plots
# 7. How to save your plots
#

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 1. Single figures
# Call the plotting function together with its package that we called 'plt'.
#
# The plot gets visible by inserting plt.show() at the end.

# + {"deletable": true, "editable": true}
plt.plot(data_pd['time'], data_pd['NS'])
plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 2. Labels and titles
# Let's insert more information!
# - label the axis
# - insert a legend
# - give a title

# + {"deletable": true, "editable": true}
#default label is the name of the array, but you can also label it with a own name

plt.plot(data_pd['time'], data_pd['NS'])
#plt.plot(data_pd['time'],data_pd['NS'], label='North-South')

plt.legend()

#axis labels
plt.xlabel('time [s]')
plt.ylabel('acceleration cm/s²')

#title
plt.title('Station LUC')

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 3. Subplots
#
# We now want to plot all three components of the seismogram in one single plot.
#
# Subplots are structured as follows:
#
#
# We will now introduce three axis in one figure.

# + {"deletable": true, "editable": true}
#create a figure f and three subplots
f, (ax1, ax2, ax3) = plt.subplots(3, 1)

ax1.plot(data_pd['time'], data_pd['NS'])
#for axis properties we use set_*
ax1.set_xlabel('time [s]')
ax1.set_ylabel('acceleration cm/s²')
ax1.legend()

ax2.plot(data_pd['time'], data_pd['EW'])
ax2.set_xlabel('time [s]')
ax2.set_ylabel('acceleration cm/s²')
ax2.legend()

ax3.plot(data_pd['time'], data_pd['UD'])
ax3.set_xlabel('time [s]')
ax3.set_ylabel('acceleration cm/s²')
ax3.legend()

#plot title over all subplots
f.suptitle('station LUC', size=20)

#needs to be shifted when tight_layout is used
f.subplots_adjust(top=0.5)
#makes all axis labels visible
f.tight_layout()

f.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# Beautify the plot!
# - plots can share axis, plots can even share boxes
# - focus on the area of interest

# + {"deletable": true, "editable": true}
#share x axis
f, (ax1, ax2, ax3) = plt.subplots(3, 1, sharex=True, sharey=True)

ax1.plot(data_pd['time'], data_pd['NS'])
ax1.set_ylabel('acc. cm/s²')
ax1.legend()

ax2.plot(data_pd['time'], data_pd['EW'])
ax2.set_ylabel('acc. cm/s²')
ax2.legend()

ax3.plot(data_pd['time'], data_pd['UD'])
ax3.set_ylabel('acc. cm/s²')
ax3.set_xlabel('time')
ax3.legend()

#share the box
plt.subplots_adjust(hspace=0)

#reduce number of ticks
plt.locator_params(axis='both', numtick=8)

#focus on the time where the signal is
plt.xlim((10.0,40.0))

#plot title over all subplots
f.suptitle('station LUC', size=20)

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 4. Styles
#
# You can use different style option to render your plot.
#
# Full documentation available here: https://matplotlib.org/devdocs/gallery/style_sheets/style_sheets_reference.html

# + {"deletable": true, "editable": true}
print(plt.style.available)

# + {"deletable": true, "editable": true}
# choose a style from above
plt.style.use('default')

#you can also combine different styles
#plt.style.use(('fivethirtyeight', 'seaborn-white', 'seaborn-pastel'))

f, (ax1, ax2, ax3) = plt.subplots(3, 1, sharex=True, sharey=True)

ax1.plot(data_pd['time'],data_pd['NS'])
ax1.set_ylabel('acc. cm/s²')

ax2.plot(data_pd['time'],data_pd['EW'])
ax2.set_ylabel('acc. cm/s²')

ax3.plot(data_pd['time'],data_pd['UD'], label='data1')
ax3.set_ylabel('acc. cm/s²')
ax3.set_xlabel('time')
ax3.legend()

#reduce number of ticks
plt.locator_params(axis='both',numtick=8)

#focus on the time where the signal is
plt.xlim((10.0,40.0))

#plot title over all subplots
f.suptitle('station LUC', size=20)

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# or you can change the **line style and colors**.

# + {"deletable": true, "editable": true}
#read in another dataset
data2_pd = pd.read_csv('data/station_2.dat', sep = '\t', comment='#', names=['time','NS','EW', 'UD'])

# + {"deletable": true, "editable": true}
f, (ax1, ax2, ax3) = plt.subplots(3,1, sharex=True, sharey=True)

ax1.plot(data_pd['time'], data_pd['NS'])
ax1.plot(data2_pd['time'], data2_pd['NS'], ls='-.')
ax1.set_ylabel('acc. cm/s²')

ax2.plot(data_pd['time'], data_pd['EW'])
ax2.plot(data2_pd['time'], data2_pd['EW'], ls='--')
ax2.set_ylabel('acc. cm/s²')

ax3.plot(data_pd['time'], data_pd['UD'], label='station 1')
ax3.plot(data2_pd['time'], data2_pd['UD'], ls=':', label='station 2')
ax3.set_ylabel('acc. cm/s²')
ax3.set_xlabel('time')
ax3.legend()

#reduce number of ticks
plt.locator_params(axis='both',numtick=8)

#focus on the time where the signal is
plt.xlim((10.0,40.0))

#plot title over all subplots
f.suptitle('Landers earthquake', size=20)

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 5. Scatter Plots

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# Using scatter plots we can show discrete data, for example measurements in dependence of two (or even three) dimensions.
#
# Full documentation: https://matplotlib.org/devdocs/api/_as_gen/matplotlib.pyplot.scatter.html
#

# + {"deletable": true, "editable": true}
# For this example we first create some random input.
# You can use your own dataset by reading in your data first
x = np.random.randn(100)
y = x + np.random.randn(100) + 10

#-------
# Insert linear fit

#from scipy import stats
#slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
#line = slope*x+intercept
#we can add some oppacity with alpha (0 to 1)
#plt.plot(x, line, alpha=0.5)

#-------
plt.xlabel('x')
plt.ylabel('y')
plt.title('Example Scatter Plot')

plt.scatter(x,y)

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# Additionally, we can use color and size as a third dimension of information in our scatter plot.
#
# Available colorbars: https://matplotlib.org/users/colormaps.html

# + {"deletable": true, "editable": true}
#create a third dimension, for example temperature
z = np.random.randn(100)*70

#plt.scatter(x,y, cmap='viridis', c=z)

#even better to spot the difference:
#change the size according to the third data dimension
#colorbar can be restricted by vmin and vmax
plt.scatter(x, y, cmap='viridis', c=z, s=z, vmin=-40, vmax=140)

plt.xlabel('x')
plt.ylabel('y')
plt.title('Example Scatter Plot')

plt.colorbar()

#set colorbar axis to customized range
#plt.set_cmap([0,150])

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ### Additional options on markers and labels
#
# You can customize your scatter plot by adding different markers for different datasets and labels.
#
# Markerstyles can be found here: https://matplotlib.org/api/markers_api.html

# + {"deletable": true, "editable": true}
#create a second data set
x_2 = np.random.randn(100)
y_2 = x_2 + np.random.randn(100) + 10
z_2 = np.random.randn(100)*70

#shades of red with z=color itensity
plt.scatter(x, y, cmap='Reds', c=z, label='dataset 1')

#shades of blue with z=color intensity
plt.scatter(x_2, y_2, cmap='Blues', c=z_2, marker ='v', label='dataset 2')

plt.xlabel('x')
plt.ylabel('y')
plt.legend()

plt.title('Difference day and night')
plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ### 3D scatter plots
#
# You can also use a  3D figure to plot your three dimensional data set.

# + {"deletable": true, "editable": true}
# we need the following package
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

#colored by the z values
ax.scatter(x, y, z, c=z)

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.title('3D scatter plot')
plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 6. Other types of plots
#
# Some additional graphics and inspiration.
#
# ### Bar plots
#
#

# + {"deletable": true, "editable": true}
#Generates
n = 12
#creates a array ranging from 0 to 11
X = np.arange(n)

#n random number, uniform distribution
Y1 = np.random.uniform(0.5, 1.0, n)
Y2 = np.random.uniform(0.5, 1.0, n)

#set a face and edge color
plt.bar(X, +Y1, facecolor='#9999ff')
plt.bar(X, -Y2, facecolor='#ff9999')

for x,y in zip(X,Y1):
    #annotations
    #shift the values slightly above the bars
    plt.text(x+0.1, y+0.05, '%.2f' % y, ha='center', va= 'bottom')

for x,y in zip(X,Y2):
    plt.text(x+0.1, -y-0.05, '%.2f' % y, ha='center', va= 'top')

plt.xlim(-.5, n)
#remove ticks on x axis
plt.xticks([])

plt.ylim(-1.25, +1.25)
#or fully remove the ticks/labels
plt.yticks([])

plt.title('Histogram example')

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ### Seaborn
#
# Seaborn is a package dedicated to statistical graphics.
#
# More general information: http://seaborn.pydata.org/examples/
#
# Jointplots combine scatter plots with distribution plots along the two axis.
#
# More information about jointplots: https://seaborn.pydata.org/generated/seaborn.jointplot.html

# + {"deletable": true, "editable": true}
#same random data set as before for the scatter plot
#this time with seaborn
import seaborn as sns

x_2 = np.random.randn(100)
y_2 = x_2 + np.random.randn(100) + 10

#data needs to put into this format
data = pd.DataFrame({"x": x_2, "y": y_2})

#scatter plot with distribution along x and y coordinate
#first plot: default
sns.jointplot(x=x_2, y=y_2, data=data)

#but you can also play around with the "kind" option
#second plot: hex + changed color
sns.jointplot(x=x_2, y=y_2, data=data, kind="reg", space=0, color="r")

#third plot: kde + changed color
sns.jointplot(x=x_2, y=y_2, data=data, kind="kde", space=0, color="g")

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# Some more inspiration

# + {"deletable": true, "editable": true}
#Error bar plots

sns.set(style="ticks")

#Load the example tips dataset
tips = sns.load_dataset("tips")

#Draw a nested boxplot to show bills by day and sex
sns.boxplot(x="day", y="total_bill", hue="sex", data=tips, palette="PRGn")

#This setting removes the borders to minimalize the figure
sns.despine(offset=10, trim=True)

plt.show()

# + {"deletable": true, "editable": true}
# Heatmaps
sns.set()

# Load the example flights dataset and conver to long-form
flights_long = sns.load_dataset("flights")
flights = flights_long.pivot("month", "year", "passengers")

# Draw a heatmap with the numeric values in each cell
f, ax = plt.subplots(figsize=(9, 6))

#main command
sns.heatmap(flights, annot=True, fmt="d", linewidths=.5, ax=ax)
plt.title('Passengers')

plt.show()

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ## 7. How to save your plots

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# You can easily save your nice figures by using
#
# ```python
# plt.savefig('my_figure.png')
# ```
# before calling
#
# ```python
# plt.show()
# ```
#
# The resolution can be increase by increasing dpi values or saving in pdf or svg format.
#
# ```python
# plt.savefig('my_figure.png', dpi=800)
# plt.savefig('my_figure.svg')
# ```


#!/usr/bin/env python
# coding: utf-8

# # Week 3 - Ungraded Lab: Data Labeling
#
#
# Welcome to the ungraded lab for week 3 of Machine Learning Engineering for Production. In this lab, you will see how the data labeling process affects the performance of a classification model. Labeling data is usually a very labor intensive and costly task but it is of great importance.
#
# As you saw in the lectures there are many ways to label data, this is dependant on the strategy used. Recall the example with the iguanas, all of the following are valid labeling alternatives but they clearly follow different criteria.
#
#

#
# **You can think of every labeling strategy as a result of different labelers following different labeling rules**. If your data is labeled by people using different criteria this will have a negative impact on your learning algorithm. It is desired to have consistent labeling across your dataset.
#
# This lab will touch on the effect of labeling strategies from a slighlty different angle. You will explore how different strategies affect the performance of a machine learning model by simulating the process of having different labelers label the data. This, by defining a set of rules and performing automatic labeling based on those rules.
#
# **The main objective of this ungraded lab is to compare performance across labeling options to understand the role that good labeling plays on the performance of Machine Learning models**, these options are:
# 1. Randomly generated labels (performance lower bound)
# 2. Automatic generated labels based on three different label strategies
# 3. True labels (performance upper bound)
#
# Although the example with the iguanas is a computer vision task, the same concepts regarding labeling can be applied to other types of data. In this lab you will be working with text data, concretely you will be using a dataset containing comments from the 2015 top 5 most popular Youtube videos. Each comment has been labeled as `spam` or `not_spam` depending on its contents.

# In[1]:

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# ## Loading the dataset
#
# The dataset consists of 5 CSV files, one for each video. Pandas `DataFrame` are very powerful to handle data in CSV format. The following helper function will load the data using pandas:

# In[ ]:

def load_labeled_spam_dataset():
    """Load labeled spam dataset."""

    # Path where csv files are located
    base_path = "./data/"

    # List of csv files with full path
    csv_files = [os.path.join(base_path, csv) for csv in os.listdir(base_path)]

    # List of dataframes for each file
    dfs = [pd.read_csv(filename) for filename in csv_files]

    # Concatenate dataframes into a single one
    df = pd.concat(dfs)

    # Rename columns
    df = df.rename(columns={"CONTENT": "text", "CLASS": "label"})

    # Set a seed for the order of rows
    df = df.sample(frac=1, random_state=824)

    return df.reset_index()

# Save the dataframe into the df_labeled variable
df_labeled = load_labeled_spam_dataset()

# To have a feeling of how the data is organized, let's inspect the top 5 rows of the data:

# In[ ]:

# Take a look at the first 5 rows
df_labeled.head()

# ## Further inspection and preprocessing
#
#
# ### Checking for data imbalance
#
# It is fairly common to assume that the data you are working on is balanced. This means that the dataset contains a similar proportion of examples for all classes. Before moving forward let's actually test this assumption:

# In[ ]:

# Print actual value count
print(f"Value counts for each class:\n\n{df_labeled.label.value_counts()}\n")

# Display pie chart to visually check the proportion
df_labeled.label.value_counts().plot.pie(y='label', title='Proportion of each class')
plt.show()

# There is roughly the same number of data points for each class so class imbalance is not an issue for this particular dataset.
#
#
# ### Cleaning the dataset
#
# If you scroll back to the cell where  you inspected the data, you will realize that the dataframe includes information that is not relevant for the task at hand. At the moment, you are only interested in the comments and the corresponding labels (the video that each comment belongs to will be used later). Let's drop the remaining columns.

# In[ ]:

# Drop unused columns
df_labeled = df_labeled.drop(['index', 'COMMENT_ID', 'AUTHOR', 'DATE'], axis=1)

# Look at the cleaned dataset
df_labeled.head()

# Now the dataset only includes the information you are going to use moving forward.
#
# ### Splitting the dataset
#
# Before jumping to the data labeling section let's split the data into training and test sets so you can use the latter to measure the performance of models that were trained using data labeled through different methods. As a safety measure when doing this split, remember to use stratification so the proportion of classes is maintained within each split.

# In[ ]:

from sklearn.model_selection import train_test_split

# Save the text into the X variable
X = df_labeled.drop("label", axis=1)

# Save the true labels into the y variable
y = df_labeled["label"]

# Use 1/5 of the data for testing later
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# Print number of comments for each set
print(f"There are {X_train.shape[0]} comments for training.")
print(f"There are {X_test.shape[0]} comments for testing")

# Let's do a visual to check that the stratification actually worked:

# In[ ]:

plt.subplot(1, 3, 1)
y_train.value_counts().plot.pie(y='label', title='Proportion of each class for train set', figsize=(10, 6))

plt.subplot(1, 3, 3)
y_test.value_counts().plot.pie(y='label', title='Proportion of each class for test set', figsize=(10, 6))

plt.tight_layout()
plt.show()

# Both, the training and test sets a balanced proportion of examples per class. So, the code successfully implemented stratification.
#
# Let's get going!

# ## Data Labeling
#
# ### Establishing performance lower and upper bounds for reference
#
# To properly compare different labeling strategies you need to establish a baseline for model accuracy, in this case you will establish both a lower and an upper bound to compare against.
#
#

# ### Calculate accuracy of a labeling strategy
#
# [CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer) is a handy tool included in the sklearn ecosystem to encode text based data.
#
# For more information on how to work with text data using sklearn check out this [resource](https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html).

# In[ ]:

from sklearn.feature_extraction.text import CountVectorizer

# Allow unigrams and bigrams
vectorizer = CountVectorizer(ngram_range=(1, 5))

# Now that the text encoding is defined, you need to select a model to make predictions. For simplicity you will use a [Multinomial Naive Bayes](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html) classifier. This model is well suited for text classification and is fairly quick to train.
#
# Let's define a function which will handle the model fitting and print out the accuracy on the test data:

# In[ ]:

from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import MultinomialNB

def calculate_accuracy(X_tr, y_tr, X_te=X_test, y_te=y_test,
                       clf=MultinomialNB(), vectorizer=vectorizer):

    # Encode train text
    X_train_vect = vectorizer.fit_transform(X_tr.text.tolist())

    # Fit model
    clf.fit(X=X_train_vect, y=y_tr)

    # Vectorize test text
    X_test_vect = vectorizer.transform(X_te.text.tolist())

    # Make predictions for the test set
    preds = clf.predict(X_test_vect)

    # Return accuracy score
    return accuracy_score(preds, y_te)

# Now let's create a dictionary  to store the accuracy of each labeling method:

# In[ ]:

# Empty dictionary
accs = dict()

# ### Random Labeling
#
# Generating random labels is a natural way to establish a lower bound. You will expect that any successful alternative labeling model to outperform randomly generated labels.
#
# Now let's calculate the accuracy for the random labeling method

# In[ ]:

# Calculate random labels
rnd_labels = np.random.randint(0, 2, X_train.shape[0])

# Feed them alongside X_train to calculate_accuracy function
rnd_acc = calculate_accuracy(X_train, rnd_labels)

rnd_acc

# You will see a different accuracy everytime you run the previous cell. This is due to the fact that the labeling is done randomly. Remember, this is a binary classification problem and both classes are balanced, so you can expect to see accuracies that revolve around 50%.
#
# To further gain intuition let's look at the average accuracy over 10 runs:

# In[ ]:

# Empty list to save accuracies
rnd_accs = []

for _ in range(10):
    # Add every accuracy to the list
    rnd_accs.append(calculate_accuracy(X_train, np.random.randint(0, 2, X_train.shape[0])))

# Save result in accs dictionary
accs['random-labels'] = sum(rnd_accs)/len(rnd_accs)

# Print result
print(f"The random labelling method achieved and accuracy of {accs['random-labels']*100:.2f}%")

# Random labelling is completely disregarding the information from the solution space you are working on, and is just guessing the correct label. You can't probably do worse than this (or maybe you can). For this reason, this method serves as reference for comparing other labeling methods
#
#
# ### Labeling with true values
#
# Now let's look at the other end of the spectrum, this is using the correct labels for your data points. Let's retrain the Multinomial Naive Bayes classifier with the actual labels

# In[ ]:

# Calculate accuracy when using the true labels
true_acc = calculate_accuracy(X_train, y_train)

# Save the result
accs['true-labels'] = true_acc

print(f"The true labelling method achieved and accuracy of {accs['true-labels']*100:.2f}%")

# Training with the true labels produced a noticeable boost in accuracy. This is expected as the classifier is now able to properly identify patterns in the training data which were lacking with randomly generated labels.
#
# Achieving higher accuracy is possible by either fine-tunning the model or even selecting a different one. For the time being you will keep the model as it is and use this accuracy as what we should strive for with the automatic labeling algorithms you will see next.

# ## Automatic labeling - Trying out different labeling strategies

# Let's suppose that for some reason you don't have access to the true labels associated with each data point in this dataset. It is a natural idea to think that there are patterns in the data that will provide clues of which are the correct labels. This is of course very dependant on the kind of data you are working with and to even hypothesize which patterns exist requires great domain knowledge.
#
# The dataset used in this lab was used for this reason. It is reasonable for many people to come up with rules that might help identify a spam comment from a non-spam one for a Youtube video. In the following section you will be performing automatic labeling using such rules. **You can think of each iteration of this process as a labeler with different criteria for labeling** and your job is to hire the most promising one.
#
# Notice the word **rules**. In order to perform automatic labeling you will define some rules such as "if the comment contains the word 'free' classify it as spam".
#
# First things first. Let's define how we are going to encode the labeling:
# - `SPAM` is represented by 1
#
#
# - `NOT_SPAM` by 0
#
#
# - `NO_LABEL` as -1
#
#
# You might be wondering about the `NO_LABEL` keyword. Depending on the rules you come up with, these might not be applicable to some data points. For such cases it is better to refuse from giving a label rather than guessing, which you already saw yields poor results.

# ### First iteration - Define some rules
#
# For this first iteration you will create three  rules based on the intuition of common patterns that appear on spam comments. The rules are simple, classify as SPAM if any of the following patterns is present within the comment or NO_LABEL otherwise:
# - `free` - spam comments usually lure users by promoting free stuff
# - `subs` - spam comments tend to ask users to subscribe to some website or channel
# - `http` - spam comments include links very frequently

# In[ ]:

def labeling_rules_1(x):

    # Convert text to lowercase
    x = x.lower()

    # Define list of rules
    rules = [
        "free" in x,
        "subs" in x,
        "http" in x
    ]

    # If the comment falls under any of the rules classify as SPAM
    if any(rules):
        return 1

    # Otherwise, NO_LABEL
    return -1

# In[ ]:

# Apply the rules the comments in the train set
labels = [labeling_rules_1(label) for label in X_train.text]

# Convert to a numpy array
labels = np.asarray(labels)

# Take a look at the automatic labels
labels

# For lots of points the automatic labeling algorithm decided to not settle for a label, this is expected given the nature of the rules that were defined. These points should be deleted since they don't provide information about the classification process and tend to hurt performance.

# In[ ]:

# Create the automatic labeled version of X_train by removing points with NO_LABEL label
X_train_al = X_train[labels != -1]

# Remove predictions with NO_LABEL label
labels_al = labels[labels != -1]

print(f"Predictions with concrete label have shape: {labels_al.shape}")

print(f"Proportion of data points kept: {labels_al.shape[0]/labels.shape[0]*100:.2f}%")

# Notice that only 379 data points remained out of the original 1564. The rules defined didn't provide enough context for the labeling algorithm to settle on a label, so around 75% of the data has been trimmed.
#
# Let's test the accuracy of the model when using these automatic generated labels:

# In[ ]:

# Compute accuracy when using these labels
iter_1_acc = calculate_accuracy(X_train_al, labels_al)

# Display accuracy
print(f"First iteration of automatic labeling has an accuracy of {iter_1_acc*100:.2f}%")

# Save the result
accs['first-iteration'] = iter_1_acc

# Let's compare this accuracy to the baselines by plotting:

# In[ ]:

def plot_accuracies(accs=accs):
    colors = list("rgbcmy")
    items_num = len(accs)
    cont = 1

    for x, y in accs.items():
        if x in ['true-labels', 'random-labels', 'true-labels-best-clf']:
            plt.hlines(y, 0, (items_num-2)*2, colors=colors.pop())
        else:
            plt.scatter(cont, y, s=100)
            cont+=2
    plt.legend(accs.keys(), loc="center left",bbox_to_anchor=(1, 0.5))
    plt.show()

plot_accuracies()

# This first iteration had an accuracy very close to the random labeling, we should strive to do better than this.

# Before moving forward let's define the `label_given_rules` function that performs all of the steps you just saw, these are:
# - Apply the rules to a dataframe of comments
# - Cast the resulting labels to a numpy array
# - Delete all data points with NO_LABEL as label
# - Calculate the accuracy of the model using the automatic labels
# - Save the accuracy for plotting
# - Print some useful metrics of the process

# In[ ]:

def label_given_rules(df, rules_function, name,
                      accs_dict=accs, verbose=True):

    # Apply labeling rules to the comments
    labels = [rules_function(label) for label in df.text]

    # Convert to a numpy array
    labels = np.asarray(labels)

    # Save initial number of data points
    initial_size = labels.shape[0]

    # Trim points with NO_LABEL label
    X_train_al = df[labels != -1]
    labels = labels[labels != -1]

    # Save number of data points after trimming
    final_size = labels.shape[0]

    # Compute accuracy
    acc = calculate_accuracy(X_train_al, labels)

    # Print useful information
    if verbose:
        print(f"Proportion of data points kept: {final_size/initial_size*100:.2f}%\n")
        print(f"{name} labeling has an accuracy of {acc*100:.2f}%\n")

    # Save accuracy to accuracies dictionary
    accs_dict[name] = acc

    return X_train_al, labels, acc

# Going forward we should come up with rules that have a better coverage of the training data, thus making pattern discovery an easier task. Also notice how the rules were only able to label as either SPAM or NO_LABEL, we should also create some rules that help the identification of NOT_SPAM comments.

# ### Second iteration - Coming up with better rules
#
# If you inspect the comments in the dataset you might be able to distinguish certain patterns at a glimpse. For example, not spam comments often make references to either the number of views since these were the most watched videos of 2015 or the song in the video and its contents . As for spam comments other common patterns are to promote gifts or ask to follow some channel or website.
#
# Let's create some new rules that include these patterns:

# In[ ]:

def labeling_rules_2(x):

    # Convert text to lowercase
    x = x.lower()

    # Define list of rules to classify as NOT_SPAM
    not_spam_rules = [
        "view" in x,
        "song" in x
    ]

    # Define list of rules to classify as SPAM
    spam_rules = [
        "free" in x,
        "subs" in x,
        "gift" in x,
        "follow" in x,
        "http" in x
    ]

    # Classify depending on the rules
    if any(not_spam_rules):
        return 0

    if any(spam_rules):
        return 1

    return -1

# This new set of rules looks more promising as it includes more patterns to classify as SPAM as well as some patterns to classify as NOT_SPAM. This should result in more data points with a label different to NO_LABEL.
#
# Let's check if this is the case.

# In[ ]:

label_given_rules(X_train, labeling_rules_2, "second-iteration")

plot_accuracies()

# This time 44% of the original dataset was given a decisive label and there were data points for both labels, this helped the model reach a higher accuracy when compared to the first iteration. Now the accuracy is considerably higher than the random labeling but it is still very far away from the upper bound.
#
# Let's see if we can make it even better!

# ### Third Iteration - Even more rules
#
# The rules we have defined so far are doing a fair job. Let's add two additional rules, one for classifying SPAM comments and the other for the opposite task.
#
# At a glimpse it looks like NOT_SPAM comments are usually shorter. This may be due to them not including hyperlinks but also in general they tend to be more concrete such as "I love this song!".
#
# Let's take a look at the average number of characters for SPAM comments vs NOT_SPAM oned:

# In[ ]:

from statistics import mean

print(f"NOT_SPAM comments have an average of {mean([len(t) for t in df_labeled[df_labeled.label==0].text]):.2f} characters.")
print(f"SPAM comments have an average of {mean([len(t) for t in df_labeled[df_labeled.label==1].text]):.2f} characters.")

# It sure looks like there is a big difference in the number of characters for both types of comments.
#
# To decide on a threshold to classify as NOT_SPAM let's plot a histogram of the number of characters for NOT_SPAM comments:

# In[ ]:

plt.hist([len(t) for t in df_labeled[df_labeled.label==0].text], range=(0,100))
plt.show()

# The majority of NOT_SPAM comments have 30 or less characters so we'll use that as a threshold.
#
# Another prevalent pattern in spam comments is to ask users to "check out" a channel, website or link.
#
# Let's add these two new rules:

# In[ ]:

def labeling_rules_3(x):

    # Convert text to lowercase
    x = x.lower()

    # Define list of rules to classify as NOT_SPAM
    not_spam_rules = [
        "view" in x,
        "song" in x,
        len(x) < 30
    ]

    # Define list of rules to classify as SPAM
    spam_rules = [
        "free" in x,
        "subs" in x,
        "gift" in x,
        "follow" in x,
        "http" in x,
        "check out" in x
    ]

    # Classify depending on the rules
    if any(not_spam_rules):
        return 0

    if any(spam_rules):
        return 1

    return -1

# In[ ]:

label_given_rules(X_train, labeling_rules_3, "third-iteration")

plot_accuracies()

# These new rules do a pretty good job at both, covering the dataset and having a good model accuracy. To be more concrete this labeling strategy reached an accuracy of ~86%! We are getting closer and closer to the upper bound defined by using the true labels.
#
# We could keep going on adding more rules to improve accuracy and we do encourage you to try it out yourself!
#
#
# ### Come up with your own rules
#
# The following cells contain some code to help you inspect the dataset for patterns and to test out these patterns. The ones used before are commented out in case you want start from scratch or re-use them.

# In[ ]:

# Configure pandas to print out all rows to check the complete dataset
pd.set_option('display.max_rows', None)

# Check NOT_SPAM comments
df_labeled[df_labeled.label==0]

# In[ ]:

# Check SPAM comments
df_labeled[df_labeled.label==1]

# In[ ]:

def your_labeling_rules(x):

    # Convert text to lowercase
    x = x.lower()

    # Define your rules for classifying as NOT_SPAM
    not_spam_rules = [
#         "view" in x,
#         "song" in x,
#         len(x) < 30
    ]

    # Define your rules for classifying as SPAM
    spam_rules = [
#         "free" in x,
#         "subs" in x,
#         "gift" in x,
#         "follow" in x,
#         "http" in x,
#         "check out" in x
    ]

    # Classify depending on your rules
    if any(not_spam_rules):
        return 0

    if any(spam_rules):
        return 1

    return -1

try:
    label_given_rules(X_train, your_labeling_rules, "your-iteration")
    plot_accuracies()

except ValueError:
    print("You have not defined any rules.")

# **Congratulations on finishing this ungraded lab!**
#
# By now you should have a better understanding of having good labelled data. In general, **the better your labels are, the better your models will be**. Also it is important to realize that the process of correctly labeling data is a very complex one. **Remember, you can think of each one of the iterations of the automatic labeling process to be a different labeler with different criteria for labeling**. If you assume you are hiring labelers you will want to hire the latter for sure!
#
# Another important point to keep in mind is that establishing baselines to compare against is really important as they provide perspective on how well your data and models are performing.
#
# **Keep it up!**


# -*- coding: utf-8 -*-

# This code is part of Qiskit.
#
# (C) Copyright IBM 2017, 2021.
#
# This code is licensed under the Apache License, Version 2.0. You may
# obtain a copy of this license in the LICENSE.txt file in the root directory
# of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
#
# Any modifications or derivative works of this code must retain this
# copyright notice, and modified files need to carry a notice indicating
# that they have been altered from the originals.
"""TEST COMPONENT NOT FOR ACTUAL USE."""

from qiskit_metal import draw, Dict
from qiskit_metal.qlibrary.core import QComponent
import numpy as np

class SmileyFace(QComponent):
    """TEST COMPONENT It is for fun only.  Can view a smiley face. Can make it
    wink or frown.

    .. image::
        SmileyFace.png

    .. meta::
        Smiley Face :)

    Default Options:
        * happy: True
        * wink: False
        * orientation: 0
    """

    component_metadata = Dict(short_name='Smile')
    """Component metadata"""

    default_options = Dict(happy=True, wink=False, orientation=0)
    """Default connector options"""

    TOOLTIP = """TEST COMPONENT It is for fun only"""

    def make(self):
        """Build the component."""
        face = draw.shapely.geometry.Point(0, 0).buffer(1)
        eye = draw.shapely.geometry.Point(0, 0).buffer(0.2)
        eye_l = draw.translate(eye, -0.4, 0.4)
        eye_r = draw.translate(eye, 0.4, 0.4)

        smile = draw.shapely.geometry.Point(0, 0).buffer(0.8)
        cut_sq = draw.shapely.geometry.box(-1, -0.3, 1, 1)
        smile = draw.subtract(smile, cut_sq)

        frown = draw.rotate(smile, 180)
        frown = draw.translate(frown, 0, 0.3)
        frown = draw.subtract(frown,
                              draw.shapely.geometry.Point(0, -0.8).buffer(0.7))

        face = draw.subtract(face, eye_l)

        if self.p.happy:
            face = draw.subtract(face, smile)
        else:
            face = draw.subtract(face, frown)

        if self.p.wink:
            face = draw.subtract(
                face,
                draw.shapely.geometry.LineString([(0.2, 0.4),
                                                  (0.6, 0.4)]).buffer(0.02))
        else:
            face = draw.subtract(face, eye_r)

        face = draw.rotate(face, self.p.orientation, origin=(0, 0))

        self.add_qgeometry('poly', {'Smiley': face})


"""
This is a unit test for monte_carlo.py.
"""

# Impoart package, test suit, and other packages as needed
import numpy as np
import mm_2019_sss_2 as MM2
import pytest
from argparse import Namespace

default_system = MM2.system.SystemSetup(num_particles=20, reduced_density=0.9)
MC_energy = MM2.energy.Energy()
default_namespace = Namespace(build_method='random', energy_function='UnitlessLJ', \
filename=None, freq=1000, max_displacement=0.1, n_steps=1000000, num_particles=20, \
reduced_density=0.9, reduced_temperature=0.9, simulation_cutoff=3.0, tune_displacement=True, output_traj=False, traj_file=None, traj_freq=1000, plot=False)

default_MC = MM2.monte_carlo.MonteCarlo(new_system=default_system, energy=MC_energy, arguments=default_namespace)
# Here we use default_system, MC_energy and default_namespace to generate an instance of the class MonteCarlo

@pytest.mark.parametrize("delta_e, beta, expected", [
    (-5.0, 1.0, True), (0.1, 1.0, True)
])

def test_accept_or_reject(delta_e, beta, expected):
    """
    For delta_e < 0, the expected result should always be true (proposed movement accepted).
    For delta_e > 0, the random seed 2019 makes the first random number always be 0.9034822144192743.
    In the test case, since we have delta_e = 0.1 and beta = 1.0, p_acc = 0.9048374180359595, which
    is larger than the random number, so the expected result should be "True".
    """

    np.random.seed(2019)
    calculated = default_MC.accept_or_reject(delta_e, beta)
    try:
        assert expected == calculated
    finally:
        np.random.seed()

@pytest.mark.parametrize("n_trials, n_accept, max_displacement, expected", [
    (100, 36, 10, (8, 0, 0)), (100, 40, 10, (10, 0, 0)), (100, 44, 10, (12, 0, 0))
])

def test_adjust_displacement(n_trials, n_accept, max_displacement, expected):
    """
    First test case (100, 36, 10, (8, 0, 0)): acc_rate = 0.36 < 0.38, so max_displacement = 8
    Second test case (100, 40, 10, (10, 0, 0)): 0.38 < acc_rate = 0.40 < 0.42, so max_displacement = 10
    Third test case (100, 44, 10, (12, 0, 0)): acc_rate = 0.44 > 0.42, so max_displacement = 12
    """
    calculated = default_MC.adjust_displacement(max_displacement, n_accept, n_trials)
    assert expected == calculated

def test_parse_arguments_defaults():
    """
    This testing function tests if the defaults are overwritten correctly if the arugments are specified
    differently from the defaults. Codewisely, note that default_MC.argument is a namespace, which can be
    converted to a dictionary by using vars(default_MC.arguments). Then, by using list(vars(default_MC.arguments).values()),
    we can get the list of defaults, which is to be compared with the list of specified values.
    """

    # create an instance of MonteCarlo for the test case (all arugments are specified to be different from the defaults
    # except for the build method)
    test_system = MM2.system.SystemSetup(num_particles=50, reduced_density=0.2)
    test_namespace = Namespace(build_method='file', energy_function='LJ', \
    filename=None, freq=500, max_displacement=0.5, n_steps=500000, num_particles=50, \
    reduced_density=0.2, reduced_temperature=0.5, simulation_cutoff=2.0, tune_displacement=False)
    test_energy = MM2.energy.Energy()
    test_case = MM2.monte_carlo.MonteCarlo(new_system=test_system, energy=test_energy, arguments=test_namespace)

    # things to be compared
    test_case_values = list(vars(test_case.arguments).values())
    default_values = list(vars(default_MC.arguments).values())

    # start to compare
    count = 0
    for i in range(len(test_case_values)):
        print("test_case:", test_case_values[i])
        print("default_values:", default_values[i])
        if test_case_values[i] != default_values[i]:
            count += 1

    expected_count = len(test_case_values) - 1
    assert  expected_count == count

def test_run_simulation():

    """
    This is a testing code for the method run_simulation.
    """
    import numpy as np

    runtest_namespace = default_namespace
    runtest_namespace.n_steps = 5000
    test_MC = MM2.monte_carlo.MonteCarlo(new_system=default_system, energy=MC_energy, arguments=runtest_namespace)
    calculated = test_MC.run_simulation()
    assert np.abs(test_MC.energy_array[-1]) < 10e+06


#!/usr/bin/env python
# coding: utf-8

# # Business Case: Netflix - Data Exploration and Visualisation

# ![alt text](https://akm-img-a-in.tosshub.com/indiatoday/images/story/202012/Netflix-New-Feature-Audio-Only_1200x768.jpeg?9TmAZq3wvsTH1jXQNlPkiSKJprCtGBAx& "Logo Title Text 1")

# ### Business Problem
#
#   - Analyze the data and generate insights that could help Netflix decide which type of shows/movies to produce and how to grow the business.
#
# ### Dataset  -  Netflix Dataset Link
#
# The dataset provided to you consists of a list of all the TV shows/movies available on Netflix:
#
#   - **Show_id:** Unique ID for every Movie / Tv Show
#   - **Type:** Identifier - A Movie or TV Show
#   - **Title:** Title of the Movie / Tv Show
#   - **Director:** Director of the Movie
#   - **Cast:** Actors involved in the movie/show
#   - **Country:** Country where the movie/show was produced
#   - **Date_added:** Date it was added on Netflix
#   - **Release_year:** Actual Release year of the movie/show
#   - **Rating:** TV Rating of the movie/show
#   - **Duration:** Total Duration - in minutes or number of seasons
#   - **Listed_in:** Genre
#   - **Description:** The summary description

# ## A high level overview of the Neflix Dataset Exploration and Visualization
#
#   - **Loading and inspecting the Dataset**
#     - Checking Shape of the Dateset
#     - Meaningful Column names
#     - Validating Duplicate Records
#     - Checking Missing values
#     - Unique values (counts) for each Feature
#     - Unique values (names) are checked for Features with a unique value count below 100
#     - Data validation - like for rating feature value cannot be duration of the movie.
#   - **Dataset Preparation**
#     - DataType Validation
#     - Dervied Columns
#   - **Univariante Analysis**
#     - Movies & TV shows - Distribution
#     - A pattern for adding Movies & TV shows content annually, monthly, etc.
#     - Release year of a movie or TV show
#     - Identify how content is distributed based on maturity level - kids, teens, and adults
#     - Netflix's most popular genre
#     - Top 20 cast contributed to Netflix content
#     - Distribution of Movie Duration
#     - No. of seasons per TV Shows
#     - Distribution of Movies and TV Shows based on Country
#   - **Biivariante Analysis**
#     - day content added and Type (Movie or Tv Show)
#     - rating and type
#   - **Summary of final recommendations**

# ### Importing the required libraries or packages for EDA

# In[1118]:

#Importing packages
import numpy as np
import pandas as pd

# Importing matplotlib and seaborn for graphs
import matplotlib.pyplot as plt
import seaborn as sns

import warnings
warnings.filterwarnings('ignore')

# Importing Date & Time util modules
from dateutil.parser import parse

from collections import Counter

# ## Loading and inspecting the Dataset

# ### Loading the csv file

# In[1119]:

netflix_data = pd.read_csv("./netflix.csv")

# In[1120]:

netflix_data.head()

# ### Checking Shape and Column names

# In[1121]:

netflix_data.shape

# In[1122]:

netflix_data.columns

# ### To make the column names more meaningful, "listed_in" has been changed to "genres".

# In[1123]:

netflix_data.rename(columns = {"listed_in":"genres"},inplace= True)

# ### Validating Duplicate Records.

# In[1124]:

# Dropping Duplicates if any
netflix_data=netflix_data.drop_duplicates()
netflix_data.shape

# ### Inference
#   - No duplicates records found.

# ### Missing Data Anaysis

# In[1125]:

#Identifying Missing data. Already verified above. To be sure again checking.
total_null = netflix_data.isnull().sum().sort_values(ascending = False)
percent = ((netflix_data.isnull().sum()/netflix_data.isnull().count())*100).sort_values(ascending = False)
print("Total records = ", netflix_data.shape[0])

missing_data = pd.concat([total_null,percent.round(2)],axis=1,keys=['Total Missing','In Percent'])
missing_data.head(7)

# ### Inference
#   -  **0.11%** of total records have missing data for "date_added". **These records can be removed while analyzing the "date_added" feature.**
#   - Only **4 records** are missing for **rating feature** which can be fixed using **imputation technique.**
#   - Only **3 records** are missing for **duration** which can be fixed using **imputation technique.**
#   - Rest missing data will be addressed during the analysis of each column.

# ### Handling Missing  - ratings
#   - Using Imputation technique. Imputing the rating missing value with NR

# In[1126]:

netflix_data['rating'].replace(to_replace = np.nan, value = "NR",inplace=True)

# ### Handling missing value for Country.

# In[1127]:

netflix_data['country'] = netflix_data['country'].fillna(netflix_data['country'].mode()[0])

# ### Unique values (counts) for each Feature

# In[1128]:

netflix_data.nunique()

# ### Inference
#   - dropping show ID as it is just for reference and no use.
#   - Description - 32 movies have the same description. This may be due to movies being released in different languages.

# In[1129]:

# dropping show ID as it is just for reference and no use.
netflix_data = netflix_data.drop('show_id',axis=1)

# ### Unique values (names) are checked for Features with a unique value count below 100

# In[1130]:

netflix_data['type'].unique()

# In[1131]:

netflix_data['rating'].unique()

# ### Inference
#   - For **rating** feature value cannot be duration of the movie. Hence need to be fixed.
#   - There were **3 missing records** for duration which are actually part of rating feature

# ### Using imputation technique to replace the duration feature value to with 74 min, 84 min and 66 min

# In[1132]:

null_columns=netflix_data.columns[netflix_data.isnull().any()]
print(netflix_data[netflix_data["duration"].isnull()][null_columns])

# In[1133]:

netflix_data.loc[netflix_data['rating'] == '74 min']

# In[1134]:

netflix_data.loc[netflix_data['rating'] == '84 min']

# In[1135]:

netflix_data.loc[netflix_data['rating'] == '66 min']

# In[1136]:

# Updated the correct duration value.
netflix_data.loc[5541,'duration'] = '74 min'
netflix_data.loc[5794,'duration'] = '84 min'
netflix_data.loc[5813,'duration'] = '66 min'

# In[1137]:

#Identifying Missing data. Already verified above. To be sure again checking.
total_null = netflix_data.isnull().sum().sort_values(ascending = False)
percent = ((netflix_data.isnull().sum()/netflix_data.isnull().count())*100).sort_values(ascending = False)
print("Total records = ", netflix_data.shape[0])

missing_data = pd.concat([total_null,percent.round(2)],axis=1,keys=['Total Missing','In Percent'])
missing_data.head(5)

# ### Using imputation technique to replace the duration value to 'NR'

# In[1138]:

netflix_data['rating'].replace({'74 min':'NR'},inplace=True)
netflix_data['rating'].replace({'84 min':'NR'},inplace=True)
netflix_data['rating'].replace({'66 min':'NR'},inplace=True)

# In[1139]:

netflix_data['release_year'].unique()

# ## Data Preparation

# ### DataType Validation

# In[1140]:

netflix_data.info()

# ### Inference
#   - date_added and release_year is a datetime data type , hence need to update the Dtype

# In[1141]:

netflix_data["date_added"] = pd.to_datetime(netflix_data['date_added'])

# In[1142]:

netflix_data.info()

# ### Dervied Columns
#
#   - Added new feature - **"Year Added"** from the **date_added** feature
#   - Added new feature - **"Month Added"** from the **date_added** feature
#   - Added new feature - **"Day Added"** from the **date_added** feature
#   - Added new feature - **"Weekday Added"** from the **date_added** feature
#   - Added new feature - **rating Category** based on whether the content is suitable for **Kids, Teenagers and Adults**
#   - Added new feature - **season Count** for each TV shows.
#

# In[1143]:

# Creating the copy of the day before manipulating the date time information.
netflix_date = netflix_data.copy()

# In[1144]:

netflix_date.shape

# ### Removed the missing values before Analysising

# In[1145]:

netflix_date.dropna(subset = ['date_added'],inplace= True)

# In[1146]:

netflix_date.shape

# ### New feature Added - year_added,month_added,day_added & Weekday_added

# In[1147]:

netflix_date["year_added"] = netflix_date['date_added'].dt.year
netflix_date["year_added"] = netflix_date["year_added"].astype("Int64")
netflix_date["month_added"] = netflix_date['date_added'].dt.month
netflix_date["month_added"] = netflix_date["month_added"].astype("Int64")
netflix_date["day_added"] = netflix_date['date_added'].dt.day
netflix_date["day_added"] = netflix_date["day_added"].astype("Int64")
netflix_date['Weekday_added'] = netflix_date['date_added'].apply(lambda x: parse(str(x)).strftime("%A"))

# In[1148]:

netflix_date.info()

# ### New Feature added "ratings_cat"

#   - Classifying the 'rating' feature into three categories. (Kids, Teenagers, Adults)
#
# |Rating|Category|
# |-----|-------|
# |TV-Y|Kids|
# |TV-Y7|Kids|
# |TV-Y7-FV|Kids|
# |G|Kids|
# |TV-G|Kids|
# |PG|Kids|
# |TV-PG|Kids|
# |PG-13|Teenagers|
# |TV-14|Teenagers|
# |TV-MA|Adults|
# |R|Adults|
# |NC-17|Adults|
# |NR|Adults|
# |UR|Adults|

# In[1149]:

netflix_data['ratings_cat'] = netflix_data['rating']

# In[1150]:

custom_rating = {  'TV-Y':'Kids','TV-Y7':'Kids' ,'TV-G':'Kids','PG':'Kids','TV-PG':'Kids','TV-Y7-FV':'Kids','G': 'Kids',
                 'PG-13':'Teenagers','TV-14' : 'Teenagers',
                 'R'     : 'Adults', 'TV-MA' : 'Adults','NC-17' : 'Adults','NR': 'Adults','UR': 'Adults'}

netflix_data['ratings_cat'] = netflix_data['rating'].replace(to_replace = custom_rating)

# In[1151]:

netflix_data.head()

# ### New feature - Season Count for each TV shows.

# In[1152]:

netflix_data['season_cnt'] = netflix_data.apply(lambda x : x['duration'].split(" ")[0] if "Season" in x['duration'] else "", axis = 1)
netflix_data['duration'] = netflix_data.apply(lambda x : x['duration'].split(" ")[0] if "Season" not in x['duration'] else "", axis = 1)

netflix_data.head()

# ### Analyzing basic statistics about each feature, such as count, min, max, and mean.

# In[1153]:

netflix_date.describe()

# ### Inference
#   - Netflix has **25%** of movies and TV shows that were released within the **last two years**
#   - About **75%** of Netflix's content consists of movies and TV shows **released after 2013**
#   - Data from Netflix shows that **new trend movies or TV shows are more prevalent**.
#   - For more subscribers, Netflix should invest in **classic Movies and TV shows.**

# ## Univariante Analysis
# #### Feature Name
#   - **Type** - Movies & TV shows - Distribution
#   - **date_added** - Checking number of new Contents added yearly, monthly, which date and Weekend-Weekday
#   - **release_year** - Movies and TV shows release_year trend.
#   - **ratings & ratings_cat** - Identify how content is distributed based on maturity level - kids, teens, and adults
#   - **genres** - Netflix's most popular genre
#   - **cast** - Top 20 cast contributed to Netflix content

# ### Movies & TV shows - Distribution

# In[1154]:

data = netflix_data.groupby("type")['type'].count()

explode=(0.08,0)
labels = ['Movie', 'TV Show']
colors = sns.color_palette("Reds")
plt.pie(data, labels=labels,colors = colors, autopct = '%0.0f%%', explode = explode)
plt.show()

# ### Inference
#   - Netflix has **70%** of its content as movies
#   - **Movies** are clearly more **popular on Netflix than TV shows**.
#

# ### Checking number of new Contents added yearly

# In[1155]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
count = (netflix_date['year_added'].value_counts(normalize=True)*100)
count.plot.bar(color=sns.color_palette('Reds'))
plt.title('Count Plot (%) - Movies added to Netflix by year ', fontsize=14)
plt.ylabel('"Frequency -> (Movies Added in %)', fontsize=12)
plt.xlabel('Year -> (Movies added to Netflix) ', fontsize=12)

# In[1156]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
sns.countplot(data=netflix_date,x = 'year_added',palette ="Reds")
plt.title('Count Plot - Movies added to Netflix by year ', fontsize=14)
plt.ylabel('"No. of movies added to Netflix', fontsize=12)
plt.xlabel('Year -> (Movies added to Netflix) ', fontsize=12)

# ### Inference
#   - According to the above graph, Netflix has started adding content since 2014.
#   - The popularity of OTT has boomed in the last 5 years, so we're seeing a dramatic increase in content being added.
#   - There were **2000 (23%) Movies and TV shows** added in the year 2019 (Highest until date).
#

# ### Checking number of new Contents added montly

# In[1157]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
sns.countplot(data=netflix_date,x = 'month_added',palette ="Reds")
plt.title('Count Plot - Movies added to Netflix by month ', fontsize=14)
plt.ylabel('"No. of movies added to Netflix', fontsize=12)
plt.xlabel('Month -> (Movies added to Netflix) ', fontsize=12)

# ### Inference
#   - Each month, we see consistent content additions.

# ### Checking number of new Contents on Weekends

# In[1158]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
count = (netflix_date['Weekday_added'].value_counts(normalize=True)*100)
count.plot.bar(color=sns.color_palette('Reds'))
plt.title('Frequency Plot (%) - Movies added to Netflix by day (Mon-Sun) ', fontsize=14)
plt.ylabel('"Frequency of Movies added in %', fontsize=12)
plt.xlabel('Year on which Movies added on Netflix', fontsize=12)

# In[1159]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
sns.countplot(data=netflix_date,x = 'Weekday_added',palette ="Reds")
plt.title('Count Plot - Movies added to Netflix by day (Mon-Sun) ', fontsize=14)

# ### Inference
#   - Netflix adds **45%** of its content on Thursdays and Fridays.
#   - **On Friday, new content should be added.**
#   - Over the weekend, less than 20% of content is added.

# ### Checking when should new content be added to the site.

# In[1160]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
count = (netflix_date['day_added'].value_counts(normalize=True)*100)
count.plot.bar(color=sns.color_palette('Reds'))
plt.title('Frequency plot (%) - Movies added to Netflix by date of the month ', fontsize=14)
plt.ylabel('"Frequency of Movies added in %', fontsize=12)
plt.xlabel('Date on which Movies added on Netflix', fontsize=12)

# ### Inference
#   - It was evident that **1st of every month** was when the most content was added.

# ### Distribution of Release year

# In[1161]:

bins = [1941,2000,2011,2014,2016,2018,2020,2021]
netflix_date_v1 = netflix_date.groupby(pd.cut(netflix_date['release_year'], bins=bins)).release_year.count()
bins = [1941,2000,2011,2013,2021]
netflix_date_v2 = netflix_date.groupby(pd.cut(netflix_date['release_year'], bins=bins)).release_year.count()

# In[1162]:

netflix_date_bin = netflix_date_v1.to_frame()
netflix_date_bin.rename(columns = {"release_year":"count"},inplace= True)
netflix_date_bin1 = netflix_date_v2.to_frame()
netflix_date_bin1.rename(columns = {"release_year":"count"},inplace= True)

# In[1163]:

fig = plt.figure(figsize=(12,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Bar plot - based on release_year of Movies & TV shows', fontsize=12)
plt.ylabel('"Count of Movies & TV shows release by year', fontsize=12)
plt.xlabel('Bin of release year ', fontsize=12)
plt.xticks(rotation = 80,fontsize=12)
sns.barplot(x=netflix_date_bin.index,y='count',data=netflix_date_bin,palette="Reds")

# In[1164]:

fig = plt.figure(figsize=(12,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Bar plot - based on release_year of Movies & TV shows', fontsize=12)
plt.ylabel('"Count of Movies & TV shows release by year', fontsize=12)
plt.xlabel('Bin of release year ', fontsize=12)
plt.xticks(rotation = 80,fontsize=12)
sns.barplot(x=netflix_date_bin1.index,y='count',data=netflix_date_bin1,palette="Reds")

# ### Inference
#   - Netflix began adding content in 2014 as indicated by the "added_year" analysis mentioned above.
#   - Added content has a release date ranging from 2014 till now for 75% of movies and TV shows. This shows Netflix encourages content creators to add new content in the platform.
#   - **New content creators have a lot of opportunities to share their content on Netflix.**

# ### Identify how content is distributed based on maturity level - kids, teens, and adults

# In[1165]:

fig = plt.figure(figsize=(12,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
count = netflix_data['ratings_cat'].value_counts(normalize=True)*100
count.plot.bar(color=sns.color_palette('Reds'))
plt.title('Bar plot - Rating based on the Category (Kids,Teenagers & Adults)', fontsize=12)
plt.ylabel('Frequency of Age rating (Adult,Kids,Teens) added in %', fontsize=12)
plt.xlabel('Category - Kids,Teenager & Adults', fontsize=12)
plt.xticks(rotation = 10,fontsize=12)

# In[1188]:

fig = plt.figure(figsize=(15,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
count = netflix_data['rating'].value_counts(normalize=True)*100
count.plot.bar(color=sns.color_palette('Reds'))
plt.xticks(rotation = 0,fontsize=12)
plt.title('Bar plot - Rating based on the Category ', fontsize=12)
plt.ylabel('"Frequency of age rating category added in %', fontsize=12)
plt.xlabel('Category ', fontsize=12)

# ### Inference
#   - On Netflix, **48%** of the content (both Movies and TV shows) is for adults.
#   - According to the graph above, more than **60%** of "TV & Shows" content is not suitable for kids.
#   - More **kid-friendly** content could increase subscriber numbers.

# ### Genres

# In[1167]:

genres = ", ".join(netflix_data['genres']).split(", ")
genres

genre_cnt = Counter()
for genre in genres:
    genre_cnt[genre] += 1

top_20_genre = genre_cnt.most_common(20)
netflix_genres = pd.DataFrame (top_20_genre, columns = ['Genres','Genres Count'])
netflix_genres.head()

# In[1189]:

fig = plt.figure(figsize=(15,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Top 20 Genres count', fontsize=12)
plt.ylabel('"Count of Movies & TV shows by genres', fontsize=12)
plt.xlabel('Genres', fontsize=12)
plt.xticks(rotation = 60,fontsize=12)
sns.barplot(x='Genres',y='Genres Count',data=netflix_genres,palette="Reds_r")

# ### Inference
#   - As can be seen from the graph above, **"International Movies & Dramas"** are the top genre contributor to Netflix.

# ### Top 20 cast contributed to Netflix content
#   - Top 20 actors who have contributed the most movies to Netflix content.
#   - Top 20 actors who have contributed the most TV shows to Netflix content.

# ### Top 20 actors who have contributed the most movies to Netflix content

# In[1169]:

netflix_cast = netflix_data.groupby(["type","cast"])["cast"].count().unstack('type')
netflix_cast.columns =['Movie', 'TV Show']
netflix_movie_cast = netflix_cast[netflix_cast["Movie"].notnull()].iloc[:,:1]
netflix_movie_cast.reset_index(level='cast', inplace=True)
netflix_movie_cast.head()

# In[1170]:

movieCast = ", ".join(netflix_movie_cast['cast']).split(", ")
movieCast
cnt_movieCast = Counter()

for cast in movieCast:
    cnt_movieCast[cast] += 1

movieCast = cnt_movieCast.most_common(20)
netflix_movieCast = pd.DataFrame (movieCast, columns = ['Actor','Actor Count'])
netflix_movieCast.head()

# In[1190]:

fig = plt.figure(figsize=(15,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Top Actor count for Movies', fontsize=12)
plt.ylabel('"Count of Movies by Actors', fontsize=12)
plt.xlabel('Actor Names', fontsize=12)
plt.xticks(rotation = 70,fontsize=12)
sns.barplot(x='Actor',y='Actor Count',data=netflix_movieCast,palette="Reds_r")

# ### Inference
#   - Based on the above data, the majority of Netflix movies are **starring Indian actors.**

# ### Top 20 actors who have contributed the most TV shows to Netflix content.

# In[1172]:

netflix_tv_show_cast = netflix_cast[netflix_cast["TV Show"].notnull()].iloc[:,1:2]
netflix_tv_show_cast.reset_index(level='cast', inplace=True)
tv_show_cast = ", ".join(netflix_tv_show_cast['cast']).split(", ")
tv_show_cast
cnt_tv_show_cast = Counter()

for cast in tv_show_cast:
    cnt_tv_show_cast[cast] += 1

tv_show_cast = cnt_tv_show_cast.most_common(20)
netflix_tvshow_cast = pd.DataFrame (tv_show_cast, columns = ['Actor','Actor Count'])
netflix_tvshow_cast.head()

# In[1191]:

fig = plt.figure(figsize=(15,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Top Actor count for TV Shows', fontsize=12)
plt.ylabel('"Count of TV Shows by Actors', fontsize=12)
plt.xlabel('Actor Names', fontsize=12)
plt.xticks(rotation = 70,fontsize=12)
sns.barplot(x='Actor',y='Actor Count',data=netflix_tvshow_cast,palette="Reds_r")

# ### Inference
#   - In TV and shows, there was a mix of stars from around the world.

# ### Distribution of Movie Duration

# In[1174]:

netflix_data_mv = netflix_data[netflix_data["type"] == "Movie"]
netflix_data_mv['duration'] = netflix_data_mv['duration'].fillna(0.0).astype(float)

# In[1175]:

sns.displot(data=netflix_data_mv,x='duration',bins=100,color="r")

# ### No. of seasons per TV Shows.

# In[1176]:

netflix_season_cnt = netflix_data['season_cnt'].value_counts().reset_index()
netflix_season_cnt = netflix_season_cnt.rename(columns = {'season_cnt' : "count", "index" : 'season'})
netflix_season_cnt.loc[0,'season'] = 0
netflix_season_cnt.head()

# In[1177]:

fig = plt.figure(figsize=(15,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Season count for TV shows', fontsize=12)
plt.xticks(rotation = 0,fontsize=12)
sns.barplot(x='season',y='count',data=netflix_season_cnt,palette="Reds_r")
plt.ylabel('Count of no. of season per TV show', fontsize=12)
plt.xlabel('No. of Seasons', fontsize=12)

# ### Distribution of Movies and TV Shows based on Country

# In[1178]:

# splitting the countries in different rows
netflix_country_data = netflix_data[['title','type', 'country' ]]
netflix_country_data = (netflix_country_data.drop('country', axis=1)
              .join(netflix_country_data.country.str.split(', ',expand=True).stack().reset_index(drop=True, level=1).rename('country')))
netflix_country_data.head()

# In[1179]:

netflix_country_data.country.nunique()

# In[1180]:

#as we can see we have records from 127 different countries, we'll only work with top 10 highest contributing countries
nf_country_top_15 = netflix_country_data.country.value_counts().sort_values(ascending=False)[:15]

# In[1181]:

fig = plt.figure(figsize=(15,5))
sns.set(style = "darkgrid")
fig.set_facecolor("lightgrey")
plt.title('Top 15 Countries contibution towards Netflix content', fontsize=12)
plt.xticks(rotation =60,fontsize=12)
sns.barplot(x=nf_country_top_15.index,y=nf_country_top_15.values,palette="Reds_r")
plt.ylabel('Number of Content released', fontsize=12)
plt.xlabel('Name of the country', fontsize=12)

# ### Inference
#   - US is the top contributor, followed by India and UK.

# ## Bi-variant Analysis

# ## Bi-variante analysis for below mentioned variables
#   - day content added and Type (Movie or Tv Show)
#   - rating and type

# ### Day content added and Type (Movie or Tv Show)
#   - As we have seen earlier that It was evident that 1st of every month was when the most content was added. Now it make sense to undersand how is the distribution based on Type.

# In[1182]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
sns.countplot(data=netflix_date,x = 'day_added',hue = 'type',palette ="Reds_r",
              order = netflix_date['day_added'].value_counts().index[0:15])
plt.title('Movies and TV Shows added added to Netflix by date ', fontsize=14)

# ### Infernce
#   - It was evident that 1st of every month was when the most content was added. Among these, **71% are movies, while 21% are TV shows**.It highly recommend that **Movie are added at the beginning of every month**.
#
#   - In addition, **Netflix will know when the majority of content is being added. If Netflix team needs to increase the servers, etc., they can work ahead of time.**

#
#

# In[1183]:

sns.catplot(data=netflix_data, kind="count", x="ratings_cat", hue="type", palette="Reds")

# In[1184]:

fig = plt.figure(figsize=(15,5))
sns.set(style='whitegrid')
fig.set_facecolor("lightgrey")
sns.countplot(data=netflix_data,x = 'rating',hue = 'type',palette ="Reds_r",
              order = netflix_data['rating'].value_counts().index[0:15])
plt.title('Content available at Netflix based on the Maurity level ', fontsize=14)

# ## Summary of Final Recommendations
#
#   - Only 25% of Netflix's content consists of movies and TV shows released before 2013.
# ###### Recommendations
#     - **For more subscribers, Netflix should invest in classic Movies and TV shows.**
#
#
#   - Netflix adds 45% of its content on Thursdays and Fridays. This may be due to the fact that people are likely to watch more content during weekends.
# ###### Recommendations -
#     - For **content creators or Netflix, Thursday and Friday are recommended dates to release content.**
#
#
#   - It was evident that 1st of every month was when the most content was added. Among these, 71% are Movies, while 21% are TV shows.
# ###### Recommendations -
#     - It highly **recommend that Movies be added at the beginning of every month**.
#     - In addition, **Netflix will know when the majority of content is being added. If Netflix team needs to increase the servers, etc., they can work ahead of time and it will have one time cost per month or quarter.**
#
#
#   - Netflix began adding content in 2014 as indicated by the "added_year" analysis mentioned above. Added content has a release date ranging from 2014 till now for 75% of movies and TV shows. This shows Netflix encourages content creators to add new content in the platform.
# ###### Recommendations -
#     - **New content creators have a lot of opportunities to share their content on Netflix.**
#
#
#   - On Netflix, **48%** of the content (both Movies and TV shows) is for adults.Based on the analysis, more than **60%** of "TV & Shows" content is not suitable for kids.
# ###### Recommendations -
#     - More **kid-friendly** content could increase subscriber numbers.

# In[ ]:

Netflix Movies & TV Shows/Netflix Data Analysis.py