# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     formats: ipynb,py
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.6.0
#   kernelspec:
#     display_name: deep_ml_curriculum
#     language: python
#     name: deep_ml_curriculum
# ---

# # Introduction
#
# This notebook provides a basic data visualisation tutorial using Matplotlib (MPL) and Seaborn python library. Please refer to the table below to navigate through the notebook.
#
# ## Table of Contents
#
# 1. [Matplotlib Basics](#matplotlib-basics)
# 2. [Importing Libraries](#libraries)
# 3. [Dataset](#dataset)
# 3. [Scatterplot](#scatter)
#     * [MPL Styling: Adding different markers](#scatter-styling-markers)
#     * [MPL Styling: Customise color](#scatter-styling-color)
# 4. [Bar Chart](#bar)
#     * [Useful Functions (getattr and operator)](#bar-useful)
# 5. [Histogram](#histogram)
# 6. [Line](#line)
#     * [MPL Styling: Line Styling](#line-styling)
#     * [MPL Styling: Colourmap](#line-styling-colourmap)
# 7. [Seaborn](#seaborn)
#    * [Basic Plots](#seaborn-basics)
#    * [Other Plots](#seaborn-others)

# # Plotting libraries
#
# We mention a few plotting libraries:
#
# - [Matplotlib/pyplot](https://matplotlib.org/3.1.1/gallery/index.html) is for basic plotting, it is the classic library that can do 80% of your plots
#     - [Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/visualization.html) wraps this to make it easier with dataframes
#     - [Geopandas](https://geopandas.org/gallery/index.html) adds maps
#     - [Seaborn](https://seaborn.pydata.org/examples/index.html) is for statistical visualization
# - [Holoviews](http://holoviews.org/gallery/index.html) - very high-level tool for interactive dashboards and large data (when using datashader backend). Not as well-suited to making figures for publication.
#     - [Bokeh](https://docs.bokeh.org/en/latest/docs/gallery.html) is usefull for interactive plotting but because it's browser based, it can struggle with large amounts of data, unless you use server-based plotting. We will use it as a Holoviews backend but it's worth looking into by itself.
#     - Datashader is useul for plotting large quantity of data points. We will use it as a holoview backend
#
# But the important thing is to get good at one so you can easily produce plots. Matplotlib is the default choice unless you expect to need interactivity or large datasets.
#
# Most plotting workflows start by finding an example and modifying it. For that we browse the galleries.

#

#
Exercise

#
#   A lot of the work of programming is knowing what to look for and how to look. This is a bit like riding a bike - it's a habit you need to build. We sometimes have blindspots where we don't know the right jargon, or stop because we get tired or overwhelmed with unfamiliar information.
#
#   For this excercise you've been asked to remake this plot in python but you need somewhere to start coding. Find the matplotlib and seaborn galleries, then look through for similar plot. Copy the code into a new cell and you are done.
#
#   If you're unsure, please expand the hints for keywords and tips of where and how to look.
#
#
#
#
#
#   → Hints
#
#   * Keywords are important, you are looking for a `heatmap` that is `categorical`
#   * You want to go to google `matplotlib gallery` or `seaborn gallery`
#   * With information overload don't get bogged down, try a quick scan of the whole page first or a Ctrl-F
#
#
#
#

#

#
#
#     → Solution
#
#
# - [matplotlib annotated heatmap](https://matplotlib.org/3.1.1/gallery/images_contours_and_fields/image_annotated_heatmap.html#sphx-glr-gallery-images-contours-and-fields-image-annotated-heatmap-py)
# - [seaborn pairwise correlations](https://seaborn.pydata.org/examples/many_pairwise_correlations.html)
# - many more
#
# If you found an example and copied the code:  congratulations, you are now a software developer.
#
#
#
#
#
#
#

# ## 1. Matplotlib Basics

# This notebook includes basic instructions about how to create different charts using the libraries Matplotlib and Seaborn. In python, we can use pandas to manipulate the data and import the libraries matplotlib and seaborn to create the charts.
#
#
# **Note:** %matplotlib inline This line of code sets the matplotlib backend to inline. More information about this [here](https://ipython.readthedocs.io/en/stable/interactive/plotting.html).
#
# From the official documentation:
#
# > With this backend, the output of plotting commands is displayed inline within frontends like the Jupyter notebook, directly below the code cell that produced it. The resulting plots will then also be stored in the notebook document.
#

# ## 2. Importing libraries

# Good practice to use short but clear aliases for the imported libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
# Magic Function
# %matplotlib inline
# Hide all warnings
import warnings
warnings.filterwarnings('ignore') # warnings.filterwarnings(action='once')

# ## 3. Dataset

# For this section, we will work with the Iris dataset. A classic in Machine Learning and data visualisation!
# This is a small but popular dataset from 1936. Each row represents an iris flower, including its species and dimensions of its sepal and petals in centimetres.
#
# More information about the dataset can be found [here](https://archive.ics.uci.edu/ml/datasets/iris). Source image from [Wikipedia](https://en.wikipedia.org/wiki/Iris_(plant)). Some of the examples were inspired from this [tutorial](https://github.com/TannerGilbert/Tutorials/blob/master/Introduction%20to%20Data%20Visualization%20in%C2%A0Python/Introduction%20to%20Data%20Visualization%20in%C2%A0Python.ipynb) which is licensed under [MIT](https://github.com/TannerGilbert/Tutorials/blob/master/LICENSE).
#
# This is an Iris (plant):
#
#

# The library seaborn provides an easy way to test some datasets
# Check the list of datasets available
print(sns.get_dataset_names())

# For this notebook we will use the popular iris dataset. We can load it using the seaborn library
iris = sns.load_dataset("iris")
# By default the function head() shows only the first 5 rows
iris

# +
# We might want to select only the numeric columns
iris_numeric = iris._get_numeric_data()

# and an unbalanced one
iris_unbalanced = iris[iris.petal_width>0.25]
# -

# ## 4. Scatterplots

# A scatter plot uses dots to represent values for two different dimensions. The position of each dot on the horizontal and vertical axis indicates values for an individual data point. Scatter plots are used to observe relationships between variables.

iris[["sepal_length", "sepal_width"]]

# Note: there are a few inferfaces to matplotlib, we will be using [pyplot](https://matplotlib.org/3.1.1/tutorials/introductory/pyplot.html) which is a high level interface

help(plt)

help(plt.scatter)

# scatter the sepal.length against the sepal.width
plt.scatter(x=iris["sepal_length"], y=iris["sepal_width"])
# set a title and labels
plt.title("Iris Dataset")
plt.xlabel("sepal length")
plt.ylabel("sepal width")

# At the moment, it is hard to know which properties belong to different varieties or classes. By adding some colour or some styling to each data point we can add more meaning the last chart.

# ### MPL Styling: Adding different markers

# +
group1 = iris[iris["species"] == "setosa"]
group2 = iris[iris["species"] == "versicolor"]
group3 = iris[iris["species"] == "virginica"]

# scatter the sepal_length against the sepal_width
# By adding a marker MPL will assign some default colours
plt.scatter(x=group1["sepal_length"], y=group1["sepal_width"], marker="^")
plt.scatter(group2["sepal_length"], group2["sepal_width"], marker="x")
plt.scatter(group3["sepal_length"], group3["sepal_width"], marker="+")

# set a title and labels
plt.title("Iris Dataset")
plt.xlabel("sepal length")
plt.ylabel("sepal width")
plt.show()
# -

# ### MPL Styling: Customise Colour

# The basic colours available are:
#
# - b: blue
# - g: green
# - r: red
# - c: cyan
# - m: magenta
# - y: yellow
# - k: black
# - w: white
#
# We can either use a single letter or write the color name. Let's use subplots and some add some styling.

# +
group1 = iris[iris["species"] == "setosa"]
group2 = iris[iris["species"] == "versicolor"]
group3 = iris[iris["species"] == "virginica"]

# scatter the sepal_length against the sepal_width
# By adding a marker MPL will assign some default colours
plt.scatter(x=group1["sepal_length"], y=group1["sepal_width"],color='purple')
plt.scatter(group2["sepal_length"], group2["sepal_width"], c="blue")
plt.scatter(group3["sepal_length"], group3["sepal_width"], c="green")

# set a title and labels
plt.title("Iris Dataset")
plt.xlabel("sepal length")
plt.ylabel("sepal width")
# -

basic_colors = {"setosa": "red", "versicolor": "green", "virginica": "blue"}

# Shape returns the number of rows and columns in the dataset
rows, columns = iris.shape
# Let's create a function that receives a color dictionary as a parameter
# Tip: It is good practice to reuse code as much as possible. Functions in Python are perfect for that !
def display_scatter(colors):
    # create a figure and axis
    fig, ax = plt.subplots()

    for species in colors.keys():
        df_species = iris[iris.species == species]
        ax.scatter(
            df_species["sepal_length"],
            df_species["sepal_width"],
            color=colors[species],
            label=species
        )

    # set a title and labels
    ax.set_title("Iris Dataset")
    ax.set_xlabel("sepal length")
    ax.set_ylabel("sepal width")
    plt.legend()

# **Note:** Gray shades can be given as a string encoding a float in the 0-1 range. Ranging from black to white.

gray_colors = {"setosa": "0", "versicolor": "0.5", "virginica": "0.85"}
display_scatter(gray_colors)

# We can also specify custom colours in hexadecimal format. More information about the colour API [here](https://matplotlib.org/2.0.2/api/colors_api.html).

# Create custom color dictionary mapping the specie with a hex colour
custom_colors = {"setosa": "#7F58AF", "versicolor": "#64C5EB", "virginica": "#E84D8A"}
display_scatter(custom_colors)

# #### Scatterplot Matrices
#
# Sometimes you might want to plot a scatter matrix which allows you to plot a grid of pairwise relationships in a dataset.

# +
from pandas.plotting import scatter_matrix

fig, ax = plt.subplots(figsize=(10, 10))
scatter_matrix(iris, alpha=1, ax=ax)
plt.show()
# -

# ## 5. Bar Chart
#
# You may have gathered that charts in pyplot are plt.`name`, but we still need to know the arguments. What are the arguments for a bar chart?

help(plt.bar)

# count the occurrence of each species
data = iris_unbalanced["species"].value_counts()
data

# create a figure and axis
fig, ax = plt.subplots()
# get x and y data
species = data.index
frequency = data.values
# create bar chart
ax.bar(species, frequency)
# set title and labels
ax.set_title("Frequency of Iris flowers by species")
ax.set_xlabel("Species")
ax.set_ylabel("Frequency")

# Or you can use the pandas wrapper
iris_unbalanced["species"].value_counts().plot.bar()

# Let's create another function to display our Bar chart
def display_bar(dataset, colors):
    # create a figure and axis
    fig, ax = plt.subplots()
    # count the occurrence of each specie
    data = dataset["species"].value_counts()
    print(data)

    # For loop to set a color per data point
    for i in range(len(data)):
        # get x and y data
        species = data.index[i]
        frequency = data.loc[species]
        ax.bar(species, frequency, color=colors[species])

    # set title and labels
    ax.set_title("Frequency of Iris flowers by specie")
    ax.set_xlabel("Species")
    ax.set_ylabel("Frequency")

display_bar(iris_unbalanced, basic_colors)
display_bar(iris_unbalanced, gray_colors)
display_bar(iris_unbalanced, custom_colors)

# Now let's try to create a more useful bar chart. Imagine that you are interested to see the frequency of iris flowers which have a minimum length (given in centimeters)

# +
def iris_minimum_sepal_length(length_cm):
    # use reset index to map correctly the new indices for the new dataset we are creating
    return iris[iris["sepal_length"] >= length_cm].reset_index(drop=True)

iris_minimum_sepal_length(5)
# -

display_bar(iris_minimum_sepal_length(5), custom_colors)

display_bar(iris_minimum_sepal_length(7), custom_colors)

# # 6. Histograms

# According to [Wikipedia](https://en.wikipedia.org/wiki/Histogram):
# > A histogram is an approximate representation of the distribution... To construct a histogram, the first step is to "bin" (or "bucket") the range of values—that is, divide the entire range of values into a series of intervals—and then count how many values fall into each interval. The bins are usually specified as consecutive, non-overlapping intervals of a variable. The bins (intervals) must be adjacent, and are often (but not required to be) of equal size.

# +
# Fix the seed for reproducibility
np.random.seed(2020)

# Let's generate a random list of numbers with a normal distribution
x = np.random.normal(size=1000)

# Let's use the pyplot function for a histogram
plt.hist(x, bins=10)
plt.show()
# -

#
#

#
Exercise

#
# Let's practice some of the key concepts we have learned so far. Using the same data in x for the histogram. Please complete the code below
#
# ```python
# # 1. Create a function that receives the number of bins and allow to plot a histogram
#
#
# def plot_hist(x, num_bins):
#     # COMPLETE CODE HERE
#     print("Complete code")
#
#
# # 2. Modify the last function to plot the histogram for values greater than the standard deviation of the data (`x.std()`).
# def plot_hist_gt_std(x, num_bins):
#     # COMPLETE CODE HERE
#     print("Complete code")
#
# ```
#
#
# → Hints
#
# * You already have x from above
# * To get rows greater than 1 std dev, look at the `iris_minimum_sepal_length` code above. `df[df<df.std()]`
# * Call the functions above with x, and the number of bins
#
#
#
#
#
#
# → Solution
#
#
# ```python
# # 1. Create a function that receives the number of bins and allows to plot a histogram
# def plot_hist(x, num_bins):
#     print(f"Histogram with {num_bins} bins")
#     plt.hist(x, bins=num_bins)
#     plt.show()
#
#
# # 2. Modify the last function to plot the histogram for values greater than the standard deviation of the data.
# def plot_hist_gt_std(x, num_bins):
#     print(f"Histogram for values greater than {x.std()}")
#     # You can use conditional selection in numpy like this:
#     above_std = x[x > x.std()]
#     plt.hist(above_std, bins=num_bins)
#     plt.show()
#
#
#
# # Let's test this new functions
# plot_hist(x, 10)
# plot_hist_gt_std(x, 10)
#
# ```
#
#
#
#

# ## Line Chart
#
# Suprisingly it's not plt.line, but plt.plot:

plt.plot(iris.index, iris['sepal_length'], label="sepal_length")
plt.plot(iris.index, iris['sepal_width'], color="green", label="sepal_width")
plt.legend()

# Or using the pandas wrapper:

iris_numeric.plot.line()

# # MPL Line Styling

# Now let's do some styling:
#
# More information about linestyling [here](https://matplotlib.org/3.1.0/gallery/lines_bars_and_markers/linestyles.html)

plt.plot(iris['sepal_length'], linestyle="dotted")
plt.plot(iris['sepal_width'], linestyle="--", color="green")
plt.plot(iris['petal_length'], linestyle="-", color="red")
plt.plot(iris['petal_width'], linestyle=":", color="c", linewidth=2, alpha=0.5, marker='x')
plt.legend()

# # Pandas plotting
#
# Pandas will handle a lot of the work for you if your dataframe is already clean. This is great if it works, so don't be afraid to try it first.

iris.plot()

iris.plot.hist()

iris.plot.scatter('sepal_length', 'sepal_width')

# ## Subplots
#
# This lets us combine multiple plots into one

# +
xi = range(10)
yi = range(10)

# Start a figure, with a certain size
plt.figure(figsize=(8, 8))

# For 2 rows and 2 columns of subplots, use subplot 1
plt.subplot(2, 2, 1)
plt.plot(xi, yi)

# activate the next plot
plt.subplot(2, 2, 2)
plt.plot(xi, yi)

plt.subplot(2, 2, 3)
plt.plot(xi, yi)
plt.title('Plot 3')
plt.xlabel('x')
plt.ylabel('y')

plt.subplot(2, 2, 4)
plt.plot(xi, yi)

plt.suptitle('The super title')
plt.show()
# -

# ### MPL Styling: Adding Colormaps  and subplots
#
# Sometimes you want to change to colors for a predefined color palette. MPL supports many color maps that can be specified in any chart using the parameter cmap
#
# We are also using subplots here
#
# Find the complete list of supported colormaps in the [official documentation](https://matplotlib.org/3.1.0/tutorials/colors/colormaps.html).

# +
# Examples using some colormaps available
colormap_list = ["viridis", "magma", "plasma", "rainbow", "Dark2", "hsv"]

def plot_lines(ax, cmap, df):
    df.plot(kind="line", cmap=cmap, ax=ax)

def plot_multiple_lines(df, colormap_list):
    fig, axes = plt.subplots(nrows=3, ncols=2, figsize=(12, 12))
    # Axes.flat is one the methods that allows iterating over the axes in a subplot
    for idx, ax in enumerate(axes.flat):
        cmap = colormap_list[idx]
        plot_lines(ax, cmap, df)
        plt.title(cmap)
    plt.tight_layout()
    plt.show()

# -

plot_multiple_lines(iris, colormap_list)

#
#

#
Exercise

#
# 1. Try different values of colormaps for the plot_multiple_lines
# 2. Modify the function plot_multiple_lines so it shows the plots in 3 columns and 2 rows.
#
#
#

#
#

# # Seaborn
#
# From the [Seaborn documentation](https://seaborn.pydata.org/)
#
# > Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.
#
# Given that this library is based on matplotlib, it is possible to style the plots using some of the styling tricks from matplotlib. However, it provides more advanced plots with less code.
#

# ## Seaborn Basic Plots
#
# We will start by plotting some of the charts that we have already used in this notebook. We will use some of the example code from the official documentation.

# %matplotlib inline

# ### Scatter plot

sns.scatterplot(x='sepal_width', y='sepal_length', data=iris)

# ### Line plot

# +
sns.lineplot(data=iris_numeric)
plt.show()

# Notice how seaborn adds by default color and line styles
# -

# ### Scatter / Pairplot

# +
import seaborn as sns

sns.set(style="ticks")
# Pairplot is the equivalent to scatter plots in MPL
sns.pairplot(iris, hue="species")
plt.show()
# -

# # Other plots

# ### Boxplot

# This plot shows the boxplots w.r.t species and sepal length
df = iris[(iris["sepal_length"] >= 2)]
sns.boxplot("sepal_length", "species", data=df)
plt.show()

# ### Multiple Linear Regression

# +
# Plot sepal width as a function of sepal length
g = sns.lmplot(x="sepal_length", y="sepal_width", hue="species", height=5, data=iris)

# Use more informative axis labels than are provided by default
g.set_axis_labels("Sepal length", "Sepal width")
plt.show()
# -

# #### Heatmaps
#
# Seaborn have interesting functions to create useful plots such as heatmap

# This heatmap shows the correlation between variables for the iris dataset in just one line of code!
sns.heatmap(iris.corr(), annot=True)
plt.show()

# Seaborn provides many more useful plots.
#
# Check some of the examples [here](https://seaborn.pydata.org/tutorial)

# ## References and further reading
# The following sources have been used in the creation of this notebook:
# - [Matplotlib documentation](https://matplotlib.org/3.2.2/contents.html)
# - [Seaborn documentation](https://seaborn.pydata.org/)
# - [Introduction to Data Visualization in Python](https://github.com/TannerGilbert/Tutorials/blob/master/Introduction%20to%20Data%20Visualization%20in%C2%A0Python/Introduction%20to%20Data%20Visualization%20in%C2%A0Python.ipynb)
#
# Many more examples:
#
# - [Seaborn examples](https://seaborn.pydata.org/examples/index.html)
# - [Matplotlib examples](https://matplotlib.org/3.2.2/tutorials/introductory/sample_plots.html)
# - [3D Plotting](https://towardsdatascience.com/an-easy-introduction-to-3d-plotting-with-matplotlib-801561999725)


# coding: utf-8

# In[31]:

import os
cwd = os.getcwd()
DB_FILE = "%s\Data\loans.db" % cwd

# In[32]:

import numpy as np
import pandas as pd
import seaborn as sns
from sklearn import preprocessing

# In[33]:

##############################################################################
##########                DATABASE FUNCTIONS                     #############
##############################################################################
#### Read function to import data from the SQL to a pandas dataframe.
def readSQL(query):
    import pandas as pd
    import sqlite3 as sql3
    db = sql3.connect(DB_FILE)
    df = pd.read_sql_query(query, db)
    db.close()
    return(df)

#### Write a pandas dataframe into an SQL table. Use overwrite=True if you want to delete
#### first a pre-existent table with the same name. Use append=True if you want to append
#### the data in the dataframe to a pre-existent table.
def writeSQL(df,tablename,overwrite=False, append=False):
    import pandas as pd
    import sqlite3 as sql
    db = sql.connect(DB_FILE)
    if (overwrite):
        action = "replace"
    elif (append):
        action = "append"
    else:
        action = "fail"
    df.to_sql(tablename, db, if_exists=action)
    db.close()

# In[34]:

loans = readSQL('''SELECT * FROM loans_dataset_missing''')

# In[35]:

loans.head()

# In[36]:

loans.dtypes

# In[37]:

df = loans.copy()

# In[38]:

df.drop(['issue_d'],axis=1,inplace=True)

# In[39]:

##Get Series of columns
##cat_vars = list(df.select_dtypes(include=['object']).columns)
##cat_vars
##Turn
#for column in cat_vars:
 #   print(df[column])
 #   df[column] = df[column].astype('category')

# In[40]:

## check unique - df.select_dtypes('category').apply(pd.Series.nunique, axis = 0)

#
 Encoding the Categorical Variables

#  For categorical variables with only 2 unique values we will use LabelEncoder

# In[41]:

count = 0
for col in df:
    if df[col].dtype == 'object':
        if len(list(df[col].unique())) <= 2:
            le = preprocessing.LabelEncoder()
            df[col] = le.fit_transform(df[col])
            count += 1
            print (col)

print('%d columns were label encoded.' % count)

#  For categorical variables with more than 2 unique values we will use dummies(onehotencoding)

# In[42]:

df = pd.get_dummies(df)
print(df.shape)
df.head()

# In[43]:

##check for NA
df.columns[df.isna().any()].tolist()

# In[44]:

##Turn Our Outcome To Category For TableOne

# In[45]:

df['default'] = df['default'].astype('category')

# # Feature Selection Strategy

#
After the dataset is ready we need to find what features are good for modeling.

#
We will do it with Univariable and Multivariable methods.

#
For Univariate we will check with tableOne what features have p-value<0.05

#
For Multivariate we will check with numerous modeling:

#

#
LASSO (L1 penalization)

#
Random Forest

#
Gradient Boosting classification

#
SVM classification

#

# In[46]:

##Get Thomas Library
import pyMechkar as mechkar

# prepare the data for multiple checks

# In[47]:

##Get All Columns except default
df_columns = df.select_dtypes(exclude=['category']).columns

# In[48]:

X = df.loc[:,df_columns]
X.head()
X.shape

# In[49]:

y = df[['default']]
print(y.shape)

# Prepare table with all of our features and fill it with the result of each analysis method

# In[53]:

varSelection = pd.DataFrame({'Variable': df_columns})
varSelection.size

# Univariable check

# In[55]:

tab1 = mechkar.pyMechkar().Table1(data=df, y='default')

# In[56]:

tab1[tab1['p_value']<0.05]

# In[57]:

vn1 = tab1.loc[tab1['p_value']<0.05,'Variables'].unique()
print(len(vn1))
vn1

# We will add these variables to our variable selection table

# In[58]:

varSelection['Univarable'] = 0
varSelection.loc[varSelection['Variable'].isin(vn1), 'Univarable'] = 1
varSelection

# Multivariable Analysis checks

# ### Variable Selection using LASSO (L1 penalization)

# In[59]:

from sklearn.linear_model import Lasso
from sklearn.feature_selection import SelectFromModel

lassomod = Lasso(alpha=0.1,max_iter=10000).fit(X, y)

# In[60]:

model = SelectFromModel(lassomod, prefit=True)
#model.get_support()

# In[61]:

varSelection['Lasso'] = model.get_support().astype('int64')
#varSelection

# ### Variable Selection using Random Forest

# In[62]:

from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel

rfmod = RandomForestClassifier().fit(X, y.values.ravel())
#rfmod.feature_importances_

# In[63]:

model = SelectFromModel(rfmod, prefit=True)
#model.get_support()

# In[64]:

varSelection['RandomForest'] = model.get_support().astype('int64')
#varSelection

# ### Variable Selection using Gradient Boosting classification

# In[65]:

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.feature_selection import SelectFromModel

gbmod = GradientBoostingClassifier().fit(X, y.values.ravel())

# In[66]:

model = SelectFromModel(gbmod, prefit=True)
#model.get_support()

# In[67]:

varSelection['GradientBoost'] = model.get_support().astype('int64')
#varSelection

# ### Variable Selection using SVM classification

# In[68]:

from sklearn.svm import LinearSVC
from sklearn.feature_selection import SelectFromModel

svmmod = LinearSVC(C=0.01, penalty="l1",dual=False).fit(X, y.values.ravel())

# In[69]:

model = SelectFromModel(svmmod, prefit=True)
#model.get_support()

# In[70]:

varSelection['SVM'] = model.get_support().astype('int64')
#varSelection

# ### Summarization and Selection of Variables

# In[71]:

varSelection['Sum'] =  np.sum(varSelection,axis=1)
varSelection

# In[73]:

varSelection.groupby('Sum')['Variable'].count()

# We can now decide a threshold for selecting our variables!

# In[74]:

v=varSelection[varSelection['Sum']>1]

# In[77]:

cols = v["Variable"]

# In[80]:

cols = cols.append(pd.Series('default'))

# In[81]:

df2 = df.loc[:,cols]

# In[82]:

print(df2.shape)

# In[83]:

writeSQL(df=df2,tablename="full_dataset")


import gym
import random
import torch
import numpy as np
from collections import deque
import matplotlib.pyplot as plt
import pdb

# ### 2. Instantiate the Environment and Agent
#
# Initialize the environment in the code cell below.

# In[2]:

env = gym.make('LunarLander-v2')
env.sparse_rewards=True
env.seed(0)
print('State shape: ', env.observation_space.shape)
print('Number of actions: ', env.action_space.n)

# Please refer to the instructions in `Deep_Q_Network.ipynb` if you would like to write your own DQN agent.  Otherwise, run the code cell below to load the solution files.

# In[3]:

from dqn_agent import Agent

agent = Agent(state_size=8, action_size=4, seed=0, enable_curiosity=False)

# watch an untrained agent

state = env.reset()

for j in range(200):
    action = agent.act(state)
    env.render()
    state, reward, done, _ = env.step(action)
    if done:
        break

env.close()

# ### 3. Train the Agent with DQN
#
# Run the code cell below to train the agent from scratch.  You are welcome to amend the supplied values of the parameters in the function, to try to see if you can get better performance!
#
# Alternatively, you can skip to the next step below (**4. Watch a Smart Agent!**), to load the saved model weights from a pre-trained agent.

# In[4]:

def dqn(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):
    """Deep Q-Learning.

    Params
    ======
        n_episodes (int): maximum number of training episodes
        max_t (int): maximum number of timesteps per episode
        eps_start (float): starting value of epsilon, for epsilon-greedy action selection
        eps_end (float): minimum value of epsilon
        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon
    """
    scores = []                        # list containing scores from each episode
    scores_window = deque(maxlen=100)  # last 100 scores
    eps = eps_start                    # initialize epsilon
    for i_episode in range(1, n_episodes+1):
        state = env.reset()
        score = 0
        for t in range(max_t):
            action = agent.act(state, eps)
            next_state, reward, done, _ = env.step(action)
            agent.step(state, action, reward, next_state, done)
            state = next_state
            score += reward
            if done:
                break
        scores_window.append(score)       # save most recent score
        scores.append(score)              # save most recent score
        eps = max(eps_end, eps_decay*eps) # decrease epsilon
        print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end="")
        if i_episode % 100 == 0:
            print('\rEpisode {}\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))
        if np.mean(scores_window)>=200.0:
            print('\nEnvironment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))
            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth')
            break
    return scores

scores = dqn()

fig=plt.figure()
ax = fig.add_subplot(111)
l1 = [tup[0] for tup in agent.loss_list]
l2 = [tup[1] for tup in agent.loss_list]
l3 = [tup[2] for tup in agent.loss_list]

plt.plot(np.arange(len(agent.loss_list)), l1)
plt.plot(np.arange(len(agent.loss_list)), l2)
plt.plot(np.arange(len(agent.loss_list)), l3)
plt.legend(["loss1","loss2","loss3"])
plt.show()

# plot the scores
fig = plt.figure()
ax = fig.add_subplot(111)
plt.plot(np.arange(len(scores)), scores)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.show()

# ### 4. Watch a Smart Agent!
#
# In the next code cell, you will load the trained weights from file to watch a smart agent!

# In[ ]:

"""
# load the weights from file
agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth'))

for i in range(3):
    state = env.reset()
    for j in range(200):
        action = agent.act(state)
        env.render()
        state, reward, done, _ = env.step(action)
        if done:
            break

env.close()
"""

# ### 5. Explore
#
# In this exercise, you have implemented a DQN agent and demonstrated how to use it to solve an OpenAI Gym environment.  To continue your learning, you are encouraged to complete any (or all!) of the following tasks:
# - Amend the various hyperparameters and network architecture to see if you can get your agent to solve the environment faster.  Once you build intuition for the hyperparameters that work well with this environment, try solving a different OpenAI Gym task with discrete actions!
# - You may like to implement some improvements such as prioritized experience replay, Double DQN, or Dueling DQN!
# - Write a blog post explaining the intuition behind the DQN algorithm and demonstrating how to use it to solve an RL environment of your choosing.


# Test helpers

import numpy as np

def assertions(user_vals, expected_vals, test_type, test_name):
    if test_type == 'type':
        try:
            assert type(user_vals) == type(expected_vals)
        except Exception as e:
            print('Type error, your type doesnt match the expected type.')
            print('Wrong type for %s' % test_name)
            print('Your type:   ', type(user_vals))
            print('Expected type:', type(expected_vals))
            return False
    elif test_type == 'shape':
        try:
            assert user_vals.shape == expected_vals.shape
        except Exception as e:
            print('Shape error, your shapes doesnt match the expected shape.')
            print('Wrong shape for %s' % test_name)
            print('Your shape:    ', user_vals.shape)
            print('Expected shape:', expected_vals.shape)
            return False
    elif test_type == 'closeness':
        try:
            assert np.allclose(user_vals, expected_vals)
        except Exception as e:
            print('Closeness error, your values dont match the expected values.')
            print('Wrong values for %s' % test_name)
            print('Your values:    ', user_vals)
            print('Expected values:', expected_vals)
            return False
    return True

def print_failure(cur_test):
    print('*'*77)
    print('The local autograder will not work if you do not pass %s.' % cur_test)
    print('*'*77)
    print(' ')

def print_name(cur_question):
    print('-'*20)
    print(cur_question)

def print_outcome(short, outcome):
    print(short + ': ', 'PASS' if outcome else '*** FAIL ***')
    print('-'*20)
    print()


# -*- coding: utf-8 -*-
#
# Author: Taylor Smith <taylor.smith@alkaline-ml.com>
#
# This is the lynx dataset found in R.

import numpy as np
import pandas as pd

from ..compat import DTYPE

__all__ = [
    'load_lynx'
]

def load_lynx(as_series=False, dtype=DTYPE):
    """Annual numbers of lynx trappings for 1821–1934 in Canada.

    This time-series records the number of skins of predators (lynx) that were
    collected over several years by the Hudson's Bay Company. The dataset was
    taken from Brockwell & Davis (1991) and appears to be the series
    considered by Campbell & Walker (1977).

    Parameters
    ----------
    as_series : bool, optional (default=False)
        Whether to return a Pandas series. If True, the index will be set to
        the observed years. If False, will return a 1d numpy array.

    dtype : type, optional (default=np.float64)
        The type to return for the array. Default is np.float64, which is used
        throughout the package as the default type.

    Examples
    --------
    >>> from pmdarima.datasets import load_lynx
    >>> load_lynx()
    array([ 269,  321,  585,  871, 1475, 2821, 3928, 5943, 4950, 2577,  523,
             98,  184,  279,  409, 2285, 2685, 3409, 1824,  409,  151,   45,
             68,  213,  546, 1033, 2129, 2536,  957,  361,  377,  225,  360,
            731, 1638, 2725, 2871, 2119,  684,  299,  236,  245,  552, 1623,
           3311, 6721, 4254,  687,  255,  473,  358,  784, 1594, 1676, 2251,
           1426,  756,  299,  201,  229,  469,  736, 2042, 2811, 4431, 2511,
            389,   73,   39,   49,   59,  188,  377, 1292, 4031, 3495,  587,
            105,  153,  387,  758, 1307, 3465, 6991, 6313, 3794, 1836,  345,
            382,  808, 1388, 2713, 3800, 3091, 2985, 3790,  674,   81,   80,
            108,  229,  399, 1132, 2432, 3574, 2935, 1537,  529,  485,  662,
           1000, 1590, 2657, 3396])

    >>> load_lynx(True).head()
    1821     269
    1822     321
    1823     585
    1824     871
    1825    1475
    dtype: int64

    Notes
    -----
    This is annual data and not seasonal in nature (i.e., :math:`m=1`)

    References
    ----------
    .. [1] Brockwell, P. J. and Davis, R. A. (1991)
           Time Series and Forecasting Methods. Second edition.
           Springer. Series G (page 557).

    .. [2] https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/lynx.html

    Returns
    -------
    lynx : array-like, shape=(n_samples,)
        The lynx dataset. There are 114 observations.
    """  # noqa: E501
    rslt = np.array([269, 321, 585, 871, 1475, 2821, 3928, 5943, 4950,
                     2577, 523, 98, 184, 279, 409, 2285, 2685, 3409,
                     1824, 409, 151, 45, 68, 213, 546, 1033, 2129,
                     2536, 957, 361, 377, 225, 360, 731, 1638, 2725,
                     2871, 2119, 684, 299, 236, 245, 552, 1623, 3311,
                     6721, 4254, 687, 255, 473, 358, 784, 1594, 1676,
                     2251, 1426, 756, 299, 201, 229, 469, 736, 2042,
                     2811, 4431, 2511, 389, 73, 39, 49, 59, 188,
                     377, 1292, 4031, 3495, 587, 105, 153, 387, 758,
                     1307, 3465, 6991, 6313, 3794, 1836, 345, 382, 808,
                     1388, 2713, 3800, 3091, 2985, 3790, 674, 81, 80,
                     108, 229, 399, 1132, 2432, 3574, 2935, 1537, 529,
                     485, 662, 1000, 1590, 2657, 3396]).astype(dtype)

    # Set the index if necessary
    if as_series:
        return pd.Series(rslt, index=range(1821, 1935))
    return rslt


# pylint: disable=invalid-name

"""
    utils.py
    --------

    The 'utils' module is created to define the helper functions, which would
    be used in the Jupyter Notebooks, during the exploration of the sample
    dataset. Following functions are included in this module.

    - :func:`plot_unclassified` plots the unclassified raw sample data.

    - :func:`plot_decision_boundary` uses the model object, data, and
    the ground truth labels in order to draw a decision boundary, which
    distinguishes between two possible classes of the sample dataset.

"""

import numpy as np
import matplotlib.pyplot as plt
from dl.nn import Sequential

def plot_unclassified(x: np.ndarray, y: np.ndarray) -> None:
    """Plots unclassified data.

    :param x: Training or validation data
    :type x: np.ndarray
    :param y: Training or validation labels
    :type y: np.ndarray
    """
    plt.figure(figsize=(6, 4))
    plt.clf()
    plt.scatter(x.T[:, 0], x.T[:, 1], c=y.T, s=40)
    plt.ylabel('x2')
    plt.xlabel('x1')
    plt.title('Classes in the unclassified data')
    plt.show()

def plot_decision_boundary(model: Sequential, x: np.ndarray,
                           y: np.ndarray, **kwargs) -> None:
    """Plots the decision boundary for 101exploratory training dataset.

    :param model: Model object representing a neural network architecture.
    :type model: Sequential
    :param x: Training or validation data.
    :type x: np.ndarray
    :param y: Training or validation labels.
    :type y: np.ndarray
    :param kwargs: Optional keyword arguments for customizing the decision
        boundary plot. Expected keys are, ep (Number of epochs),
        set (Data set), alg (optimization algorithm),
        bs (batch size for training), lr (learning rate),
        rp (regularization parameter).
    :type kwargs: str=Any
    """
    # Set min and max values and give it some padding
    x_min, x_max = x[0, :].min() - 1, x[0, :].max() + 1
    y_min, y_max = x[1, :].min() - 1, x[1, :].max() + 1
    interval: float = 0.01
    # Generate a grid of points with distance h between them
    gird_x, grid_y = np.meshgrid(np.arange(x_min, x_max, interval),
                                 np.arange(y_min, y_max, interval))

    # Predict the function value for the whole grid, for this we take the
    # gird_x, grid_y mesh matrices, flatten them,concatenate as columns and
    # finally make a transpose in order give them exact shape of training
    # dataset.
    mesh_dataset: np.ndarray = np.c_[gird_x.ravel().T, grid_y.ravel().T].T
    hyp = model.predict(data=mesh_dataset, tp=0.5)
    hyp = hyp.reshape(gird_x.shape)
    # Plot the contour and training examples
    plt.figure(figsize=(6, 4))
    plt.clf()
    plt.contourf(gird_x, grid_y, hyp)
    plt.ylabel('x2')
    plt.xlabel('x1')
    ep = kwargs.get('ep', None)
    st = kwargs.get('set', None)
    alg = kwargs.get('alg', None)
    bs = kwargs.get('bs', None)
    lr = kwargs.get('lr', None)
    rp = kwargs.get('rp', None)
    plt.title('Decision Boundary')
    plt.scatter(x[0, :], x[1, :], c=y, cmap=plt.cm.get_cmap('tab10'))
    plt.show()
    if st:
        print(f'For: {st} set')
    if alg:
        print(f'Optimization algorithm: {alg}')
    if ep:
        print(f'Epochs: {ep}')
    if bs:
        print(f'Batch size: {bs}')
    if lr:
        print(f'Learning rate: {lr}')
    if rp:
        print(f'Regularization parameter: {rp}')

if __name__ == '__main__':
    pass


#!/usr/bin/env python
# coding: utf-8

#
Table of Contents

#

1  Desafio 1

1.1  Set up da análise

1.2  Inicie sua análise a partir daqui

1.3  Questão 1

1.4  Questão 2

1.5  Questão 3

1.6  Questão 4

1.7  Questão 5

1.8  Questão 6

1.9  Questão 7

1.10  Questão 8

1.11  Questão 9

1.12  Questão 10

# # Desafio 1
#
# Para esse desafio, vamos trabalhar com o data set [Black Friday](https://www.kaggle.com/mehdidag/black-friday), que reúne dados sobre transações de compras em uma loja de varejo.
#
# Vamos utilizá-lo para praticar a exploração de data sets utilizando pandas. Você pode fazer toda análise neste mesmo notebook, mas as resposta devem estar nos locais indicados.
#
# > Obs.: Por favor, não modifique o nome das funções de resposta.

# ## _Set up_ da análise

# In[2]:

import pandas as pd
import numpy as np

# In[3]:

black_friday = pd.read_csv("black_friday.csv")

# ## Inicie sua análise a partir daqui

# In[4]:

black_friday.head(5)

# ## Questão 1
#
# Quantas observações e quantas colunas há no dataset? Responda no formato de uma tuple `(n_observacoes, n_colunas)`.

# In[5]:

def q1():
    return black_friday.shape

# In[6]:

black_friday.shape

# ## Questão 2
#
# Há quantas mulheres com idade entre 26 e 35 anos no dataset? Responda como um único escalar.

# In[43]:

def q2():
    return int(black_friday.query('Gender == "F" and Age=="26-35"')['Age'].value_counts()[0])

# In[8]:

black_friday.Age.unique()

# In[9]:

genders = black_friday.groupby('Gender')
genders

# In[10]:

black_friday.query('Gender == "F" and Age=="26-35"')['Age'].value_counts()

# In[44]:

int(black_friday.query('Gender == "F" and Age=="26-35"')['Age'].value_counts()[0])

# ## Questão 3
#
# Quantos usuários únicos há no dataset? Responda como um único escalar.

# In[12]:

def q3():
    return len(black_friday.User_ID.unique())

# In[13]:

len(black_friday.User_ID.unique())

# ## Questão 4
#
# Quantos tipos de dados diferentes existem no dataset? Responda como um único escalar.

# In[14]:

def q4():
    return len(black_friday.dtypes.unique())

# In[15]:

black_friday.dtypes

# In[16]:

len(black_friday.dtypes.unique())

# ## Questão 5
#
# Qual porcentagem dos registros possui ao menos um valor null (`None`, `ǸaN` etc)? Responda como um único escalar entre 0 e 1.

# In[17]:

def q5():
    return (black_friday.shape[0] - black_friday.dropna().shape[0])/black_friday.shape[0]

# In[18]:

black_friday.info()

# In[19]:

black_friday.isna()

# In[20]:

black_friday.shape[0]

# In[21]:

black_friday.dropna().shape[0]

# In[22]:

percentual = (black_friday.shape[0] - black_friday.dropna().shape[0])/black_friday.shape[0]
percentual

# ## Questão 6
#
# Quantos valores null existem na variável (coluna) com o maior número de null? Responda como um único escalar.

# In[46]:

def q6():
    return int(black_friday['Product_Category_3'].isna().sum())

# In[24]:

black_friday['Product_Category_3'].isna().sum()

# ## Questão 7
#
# Qual o valor mais frequente (sem contar nulls) em `Product_Category_3`? Responda como um único escalar.

# In[25]:

def q7():
    return black_friday['Product_Category_3'].value_counts().index[0]

# In[26]:

black_friday['Product_Category_3'].value_counts().index[0]

# ## Questão 8
#
# Qual a nova média da variável (coluna) `Purchase` após sua normalização? Responda como um único escalar.

# In[52]:

def q8():
    normalized = black_friday[['Purchase']]
    normalized_min = black_friday[['Purchase']].min()
    normalized_max = black_friday[['Purchase']].max()
    normalized_DF = (normalized - normalized_min) / (normalized_max - normalized_min)
    return float(normalized_DF.mean()[0])

# In[28]:

black_friday['Purchase'].isna().sum()

# In[29]:

normalized = black_friday[['Purchase']]
normalized

# In[30]:

normalized_min = black_friday[['Purchase']].min()
normalized_min

# In[31]:

normalized_max = black_friday[['Purchase']].max()
normalized_max

# In[32]:

normalized_DF = (normalized - normalized_min) / (normalized_max - normalized_min)
normalized_DF

# In[33]:

normalized_DF.mean()[0]

# ## Questão 9
#
# Quantas ocorrências entre -1 e 1 inclusive existem da variáel `Purchase` após sua padronização? Responda como um único escalar.

# In[34]:

def q9():
    stand = black_friday[['Purchase']]
    stand_mean = black_friday[['Purchase']].mean()
    stand_std = black_friday[['Purchase']].std()
    stand_DF = (stand - stand_mean) / (stand_std)
    return stand_DF.query('Purchase >= -1 and Purchase <= 1').Purchase.shape[0]

# In[35]:

stand = black_friday[['Purchase']]

# In[36]:

stand_mean = black_friday[['Purchase']].mean()
stand_mean

# In[37]:

stand_std = black_friday[['Purchase']].std()
stand_std

# In[38]:

stand_DF = (stand - stand_mean) / (stand_std)
stand_DF

# In[39]:

stand_DF.query('Purchase >= -1 and Purchase <= 1').Purchase.shape[0]

# ## Questão 10
#
# Podemos afirmar que se uma observação é null em `Product_Category_2` ela também o é em `Product_Category_3`? Responda com um bool (`True`, `False`).

# In[58]:

def q10():
    return bool(black_friday.isna().query('Product_Category_2 == True & Product_Category_3 == True').Product_Category_2.unique()[0])

# In[59]:

resultado = bool(black_friday.isna().query('Product_Category_2 == True & Product_Category_3 == True').Product_Category_2.unique()[0])
resultado


"""
=========================================
The :mod:`mpi_array.globale_ufunc` Module
=========================================

Defines :obj:`numpy.ufunc` functions for :obj:`mpi_array.globale.gndarray`.

Classes
=======

.. autosummary::
   :toctree: generated/

   GndarrayArrayUfuncExecutor - Creates :obj:`gndarray` outputs and forwards to `numpy.ufunc`.

Functions
=========

.. autosummary::
   :toctree: generated/

   get_dtype_and_ndim - Return :obj:`numpy.dtype` and :samp:`ndim` properties for an object.
   ufunc_result_type - Like :func:`numpy.result_type`.
   broadcast_shape - Calculates broadcast shape from sequence of shape arguments.
   shape_extend_dims - Prepend ones to 1D *shape* sequence to make it a specified dimension.
   gndarray_array_ufunc - A :obj:`numpy.ndarray` like distributed array.

"""

from __future__ import absolute_import

import sys as _sys
import numpy as _np
import mpi4py.MPI as _mpi

from .license import license as _license, copyright as _copyright, version as _version
from . import logging as _logging  # noqa: E402,F401
from . import globale_creation as _globale_creation
from . import comms as _comms
from .distribution import ScalarLocaleExtent, ScalarGlobaleExtent, LocaleExtent, GlobaleExtent

__author__ = "Shane J. Latham"
__license__ = _license()
__copyright__ = _copyright()
__version__ = _version()

def get_dtype_and_ndim(array_like):
    """
    Returns :samp:`(dtype, ndim)` pair for the given :samp:`{array_like}` argument.
    If the :samp:`{array_like}` has *both* :samp:`"dtype"` and :samp:`"ndim"`
    attributes, then the return tuple is :samp:`({array_like}.dtype, {array_like}.ndim)`.
    Otherwise,
    returns :samp:`(numpy.asanyarray({array_like}).dtype, numpy.asanyarray({array_like}).ndim)`.

    :type array_like: castable to :obj:`numpy.ndarray`
    :param array_like: Returns dtype and ndim for this object.
    :rtype: two element :obj:`tuple`
    :return: The :obj:`numpy.dtype` and integer :samp:`ndim` properties for :samp:`{array_like}`.

    Example::

       >>> get_dtype_and_ndim(1.0)
       (dtype('float64'), 0)
       >>> get_dtype_and_ndim((1.0, 2.0, 3.0, 4.0))
       (dtype('float64'), 1)
       >>> get_dtype_and_ndim([(1.0, 2.0, 3.0, 4.0), (5.0, 6.0, 7.0, 8.0)])
       (dtype('float64'), 2)
    """
    dt, nd = None, None
    if not ((hasattr(array_like, "dtype") and hasattr(array_like, "ndim"))):
        array_like = _np.asanyarray(array_like)

    dt, nd = array_like.dtype, array_like.ndim

    return dt, nd

def ufunc_result_type(
    ufunc_types,
    inputs,
    outputs=None,
    casting="safe",
    input_match_casting="safe"
):
    """
    Attempts to calculate the result type from given ufunc :samp:`{inputs}`
    and ufunc types (:attr:`numpy.ufunc.types`).
    Like :obj:`numpy.result_type`, but
    handles :obj:`mpi_array.globale.gndarray` in the :samp:`{inputs}`
    and handles multiple :samp:`{outputs}` cases.

    :type ufunc_types: sequence of `str`
    :param ufunc_types: The :attr:`numpy.ufunc.types` attribute,
       e.g. :samp:`['??->?', 'bb->b', 'BB->B', 'hh->h', 'HH->H', ..., 'mm->m', 'mM->M', 'OO->O']`.
    :type inputs: sequence of :obj:`object`
    :param inputs: The inputs (e.g. :obj:`numpy.ndarray`, scalars
       or :obj:`mpi_array.globale.gndarray`) to a :obj:`numpy.ufunc` call.
    :type outputs: :samp:`None` or sequence of :obj:`object`
    :param outputs: The output arrays these are explicitly checked casting correctness.
    :type casting: :obj:`str` :samp:`{'no', 'equiv', 'safe', 'same_kind', 'unsafe'}`
    :param casting: Casting mode applied to outputs. See :func:`numpy.can_cast`.
    :type input_match_casting: :obj:`str` :samp:`{'no', 'equiv', 'safe', 'same_kind', 'unsafe'}`
    :param input_match_casting: Casting mode applied to match :samp:`{ufunc_types}` inputs
       with the :samp:`{inputs}`. See :func:`numpy.can_cast`.
    :rtype: :obj:`tuple` of :obj:`numpy.dtype`
    :return: A tuple of :obj:`numpy.dtype` indicating the output types produced for
       the given inputs.
    :raises ValueError: If the the inputs (and outputs) cannot be cast to an
       appropriate element of :samp:`{ufunc_types}`.

    Example::

       >>> import numpy as np
       >>> import mpi_array as mpia
       >>> inp = (
       ... np.zeros((10,10,10), dtype='float16'),
       ... 16.0,
       ... mpia.zeros((10,10,10), dtype='float32'),
       ... )
       >>> ufunc_result_type(['eee->e?', 'fff->f?', 'ddd->d?'], inputs=inp)
       (dtype('float32'), dtype('bool'))
       >>> out = (mpia.zeros((10,10,10), dtype="float64"),)
       >>> ufunc_result_type(['eee->e?', 'fff->f?', 'ddd->d?'], inputs=inp, outputs=out)
       (dtype('float64'), dtype('bool'))
       >>> out += (mpia.zeros((10, 10, 10), dtype="uint16"),)
       >>> ufunc_result_type(['eee->e?', 'fff->f?', 'ddd->d?'], inputs=inp, outputs=out)
       (dtype('float64'), dtype('uint16'))
       >>> mpia.free_all(inp + out)
    """
    logger = _logging.get_rank_logger(__name__)
    result_dtypes = None
    ufunc_in_types = tuple(in2out_str.split("->")[0] for in2out_str in ufunc_types)
    ufunc_in_dtypes = \
        _np.asarray(
            tuple(
                tuple(_np.dtype(c) for c in ufunc_in_types[i])
                for i in range(len(ufunc_in_types))
            )
        )
    ufunc_out_types = tuple(in2out_str.split("->")[1] for in2out_str in ufunc_types)
    ufunc_out_dtypes = \
        _np.asarray(
            tuple(
                tuple(_np.dtype(c) for c in ufunc_out_types[i])
                for i in range(len(ufunc_out_types))
            )
        )

    in_dtypes_and_ndims = \
        _np.asarray(tuple(get_dtype_and_ndim(input) for input in inputs))

    in_dtypes = in_dtypes_and_ndims[:, 0]
    in_ndims = in_dtypes_and_ndims[:, 1]

    logger.debug("inputs=%s", inputs)
    logger.debug("in_dtypes=%s", in_dtypes)
    logger.debug("in_ndims=%s", in_ndims)
    logger.debug("ufunc_in_dtypes=%s", ufunc_in_dtypes)

    out_dtypes = None
    if (outputs is not None) and (len(outputs) > 0):
        out_dtypes = \
            _np.asarray(
                tuple(
                    output.dtype
                    if hasattr(output, "dtype") else _np.asarray(output).dtype
                    for output in outputs
                )
            )

    idx = None
    idxs = _np.where(_np.logical_and.reduce(ufunc_in_dtypes == in_dtypes, axis=1))
    if len(idxs) > 0 and len(idxs[0]) > 0:
        idx = idxs[0][0]

    if idx is None:
        in_scalars_and_dtypes = \
            tuple(
                inputs[i]
                if in_ndims[i] <= 0 else in_dtypes[i]
                for i in range(len(inputs))
            )
        idxs = \
            _np.where(
                _np.asarray(
                    tuple(
                        _np.all(
                            tuple(
                                _np.can_cast(
                                    in_scalars_and_dtypes[j],
                                    ufunc_in_dtypes[i, j],
                                    casting=input_match_casting
                                )
                                for j in range(ufunc_in_dtypes.shape[1])
                            )
                        )
                        for i in range(ufunc_in_dtypes.shape[0])
                    )
                )
            )
        if len(idxs) > 0 and len(idxs[0]) > 0:
            idx = idxs[0][0]

    if idx is not None:
        ufunc_out_dtypes_for_in = ufunc_out_dtypes[idx]
        if (
            (out_dtypes is not None)
            and
            _np.any(ufunc_out_dtypes_for_in[:len(out_dtypes)] != out_dtypes)
        ):
            if (
                _np.any(
                    tuple(
                        not _np.can_cast(ufunc_out_dtypes_for_in[i], out_dtypes[i], casting=casting)
                        for i in range(len(out_dtypes))
                    )
                )
            ):
                raise ValueError(
                    "Could not cast ufunc-output-types %s to desired output-types = %s."
                    %
                    (
                        tuple(ufunc_out_dtypes_for_in),
                        tuple(out_dtypes)
                    )
                )
        if out_dtypes is None:
            out_dtypes = _np.array((), dtype='O')
        result_dtypes = \
            tuple(
                out_dtypes.tolist()
                +
                ufunc_out_dtypes_for_in[len(out_dtypes):].tolist()
            )
    else:
        raise ValueError(
            "Could not cast (with input_match_casting='%s') inputs types = %s to ufunc types=\n%s"
            %
            (input_match_casting, in_dtypes, ufunc_in_dtypes, )
        )

    return result_dtypes

def broadcast_shape(*shape_args):
    """
    Returns
    the :mod:`numpy` `broadcast <https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html>`_
    shape for the give shape arguments.

    :type shape1, shape2, ...: sequence of `int`
    :param shape1, shape2, ...: Array shapes to be broadcast.
    :rtype: sequence of `int`
    :return: The broadcast shape.

    Examples::

        >>> broadcast_shape((4,), (4,))
        (4,)
        >>> broadcast_shape((4, 1), (1, 5))
        (4, 5)
        >>> broadcast_shape((4, 1, 3, 7), (1, 8, 1, 7))
        (4, 8, 3, 7)
        >>> broadcast_shape((3, 7), ())
        (3, 7)
    """
    ndim = _np.max(tuple(len(shape) for shape in shape_args))

    bcast_shape = ()
    if ndim > 0:
        ndim_shapes = \
            _np.asarray(tuple((1,) * (ndim - len(shape)) + tuple(shape) for shape in shape_args))
        bcast_shape = _np.amax(ndim_shapes, axis=0)

        if (_np.any(_np.logical_and(ndim_shapes != 1, ndim_shapes != bcast_shape))):
            raise ValueError(
                "shape mismatch - objects cannot be broadcast to a single shape:\n%s"
                %
                (shape_args,)
            )

        bcast_shape = tuple(bcast_shape)

    return bcast_shape

def shape_extend_dims(ndim, shape):
    """
    Returns :obj:`shape` pre-prepended with ones so returned 1D array has length :samp:`{ndim}`.

    :type ndim: :obj:`int`
    :param ndim: Length of returned 1D sequence.
    :type shape: sequence of :obj:`object`
    :param shape: Length of returned 1D sequence.
    :rtype: :obj:`tuple`
    :return: Sequence pre-pended with one elements so that sequence length equals :samp:`{ndim}`.

    Example::

       >>> shape_extend_dims(5, (3, 1, 5))
       (1, 1, 3, 1, 5)
       >>> shape_extend_dims(3, (3, 1, 5))
       (3, 1, 5)
       >>> shape_extend_dims(1, (3, 1, 5))
       (3, 1, 5)

    """
    return (1,) * (ndim - len(shape)) + tuple(shape)

def get_extents(input, locale_info):
    """
    Returns a :samp:`(locale_extent, globale_extent)` pair for
    the given :samp:`input`, where :samp:`locale_extent` is
    a :obj:`mpi_array.distribution.LocaleExtent` instance and :samp:`globale_extent` is
    a :obj:`mpi_array.distribution.GlobaleExtent` instance.

    :type input: scalar, array like or :obj:`mpi_array.globale.gndarray`
    :param input: Return extents for this input.
    :type locale_info: :obj:`mpi_array.comms.ThisLocaleInfo`
    :param locale_info: The rank info required for constructing
        a :obj:`mpi_array.distribution.LocaleExtent` instance
        for :samp:`input` types which are not :obj:`mpi_array.globale.gndarray`.
    :rtype: :obj:`tuple`
    :return: A :samp:`(locale_extent, globale_extent)` pair indicating the
       extents of the :samp:`{input}` array-like.
    """
    locale_extent = None
    globale_extent = None
    if not (hasattr(input, "shape") and hasattr(input, "ndim")):
        input = _np.asanyarray(input)
    if hasattr(input, "lndarray_proxy") and hasattr(input, "distribution"):
        locale_extent = input.lndarray_proxy.locale_extent
        globale_extent = input.distribution.globale_extent
    elif input.ndim > 0:
        start = (0,) * input.ndim
        globale_extent = GlobaleExtent(start=start, stop=input.shape)
        locale_extent = \
            LocaleExtent(
                peer_rank=locale_info.peer_rank,
                inter_locale_rank=locale_info.inter_locale_rank,
                globale_extent=globale_extent,
                start=start,
                stop=input.shape
            )
    else:
        locale_extent = \
            ScalarLocaleExtent(
                peer_rank=locale_info.peer_rank,
                inter_locale_rank=locale_info.inter_locale_rank
            )
        globale_extent = ScalarGlobaleExtent()

    return (locale_extent, globale_extent)

def calc_matching_locale_slices(out_locale_extent, out_globale_extent, inp_locale_extents):
    """
    Returns :obj:`tuple` of :obj:`slice` (one tuple for each pair-element
    in :samp:`{inp_locale_extents}`). The returned *slices* indicate the
    portion of the corresponding input extent which broadcasts
    to the output extent :samp:`{out_locale_extent}`.

    Assumes :samp:`{out_locale_extent}.ndim >= {inp_locale_extents}[i].ndim`
    for :samp:`i in range(0, len({inp_locale_extents})`.

    :type out_locale_extent: :obj:`mpi_array.distribution.LocaleExtent`
    :param out_locale_extent: A locale extent of the output array.
    :type out_globale_extent: :obj:`mpi_array.distribution.GlobaleExtent`
    :param out_globale_extent: The globale extent of the output :obj:`mpi_array.globale.gndarray`.
    :type inp_locale_extents: sequence of extent pairs
    :param inp_locale_extents: This is the sequence
       of :samp:`(inp_locale_extent, inp_globale_extent)` pairs, one pair for
       each ufunc input.
    :rtype: :obj:`tuple` of :obj:`tuple` elements
    :return: For each pair :samp:`(inp_locale_extent, inp_globale_extent)`
       in :samp:`{inp_locale_extents}` returns a :obj:`tuple`-of-:obj:`slice`
       indicating the portion of :samp:`inp_locale_extent` which is to be broadcast
       with :samp:`{out_locale_extent}`. Tuple indices are globale.
    """
    slice_list = []
    out_loc_start = out_locale_extent.start
    out_loc_shape = out_locale_extent.shape
    for inp_loc, inp_glb in inp_locale_extents:
        slc_tuple = None
        if inp_glb.ndim >= 1:
            inp_glb_shape = inp_glb.shape
            inp_loc_start = inp_loc.start
            inp_loc_shape = inp_loc.shape
            inp_slc_start = _np.zeros_like(inp_loc_start)
            inp_slc_shape = inp_loc_shape.copy()

            slc_tuple = []
            for a in range(-1, -(len(inp_loc_shape) + 1), -1):
                if inp_glb_shape[a] == 1:
                    inp_slc_start[a] = 0
                    inp_slc_shape[a] = 1
                else:
                    inp_slc_start[a] = out_loc_start[a]
                    inp_slc_shape[a] = out_loc_shape[a]
                slc = slice(inp_slc_start[a], inp_slc_start[a] + inp_slc_shape[a])
                slc_tuple.insert(0, slc)
            slc_tuple = tuple(slc_tuple)
        slice_list.append(slc_tuple)

    return tuple(slice_list)

def calc_matching_peer_rank_slices(out_slice, inp_arys):
    """
    For each input array in :samp:`{inp_arys}, calculates the portion
    which broadcasts to the :samp:`{out_slice}`.
    Returns :obj:`tuple` of :obj:`slice` (one tuple for each array/scalar element
    in :samp:`{inp_arys}`). The returned *slices* indicate the
    portion of the input which matches the specified :samp:`{out_slice}`
    for broadcasting.

    Assumes :samp:`len({out_slice}) >= {inp_arys}[i].ndim`
    for :samp:`i in range(0, len({inp_arys})`.

    :type out_slice: :obj:`tuple` of :obj:`slice`
    :param out_slice: Slice indicating a portion (sub-array) of an output array.
    :type inp_arys: Sequence of :obj:`numpy.ndarray`
    :param inp_arys: The ufunc input arrays.
    """
    slice_list = []
    for inp_ary in inp_arys:
        slc_tuple = None
        if hasattr(inp_ary, "ndim") and (inp_ary.ndim >= 1):
            inp_shape = _np.array(inp_ary.shape)
            inp_slc_start = _np.zeros_like(inp_shape)
            inp_slc_stop = inp_slc_start + inp_shape

            slc_tuple = []
            for a in range(-1, -(len(inp_shape) + 1), -1):
                if inp_shape[a] == 1:
                    inp_slc_start[a] = 0
                    inp_slc_stop[a] = 1
                else:
                    inp_slc_start[a] = out_slice[a].start
                    inp_slc_stop[a] = out_slice[a].stop
                slc = slice(inp_slc_start[a], inp_slc_stop[a])
                slc_tuple.insert(0, slc)
            slc_tuple = tuple(slc_tuple)
        slice_list.append(slc_tuple)

    return tuple(slice_list)

def convert_to_array_like(inputs):
    """
    Uses :obj:`numpy.asanyarray` to convert input ufunc arguments
    to array-like objects.

    :type inputs: sequence of :obj:`object`
    :param inputs: Elements of this sequence which to not have both :samp:`"shape"`
       and :samp:`"ndim"` attributes are converted to a new object
       using :obj:`numpy.asanyarray`.
    :rtype: sequence of :obj:`object`
    :return: Sequence where elements of :samp:`{inputs}` have been converted to array-like objects.

    Example::
       >>> import numpy as np
       >>> inputs = (np.array([1, 2, 3, 4], dtype="uint8"), 32.0, [[1, 2], [3, 4], [5, 6]])
       >>> convert_to_array_like(inputs)
       (array([1, 2, 3, 4], dtype=uint8), array(32.0), array([[1, 2],
              [3, 4],
              [5, 6]]))
       >>> converted = convert_to_array_like(inputs)
       >>> converted[0] is inputs[0]
       True
       >>> converted[1] is inputs[1]
       False
       >>> converted[2] is inputs[2]
       False
    """
    return \
        tuple(
            input
            if hasattr(input, "shape") and hasattr(input, "ndim") else _np.asanyarray(input)
            for input in inputs
        )

def check_equivalent_inter_locale_comms(
    gndarrays,
    equivalent_compare=(_mpi.IDENT, _mpi.CONGRUENT)
):
    """
    Checks that all the :obj:`mpi_array.globale.gndarray` elements
    of :samp:`{gndarrays}` have equivalent inter-locale communicators.

    :raises ValueError: if the arrays do not have equivalent inter-locale communicators.
    """
    if (gndarrays is not None) and (len(gndarrays) > 0):
        inter_locale_comm0 = gndarrays[0].locale_comms.inter_locale_comm
        for c in (gndary.locale_comms.inter_locale_comm for gndary in gndarrays[1:]):
            if (
                (
                    (c == _mpi.COMM_NULL)
                    and
                    (inter_locale_comm0 != _mpi.COMM_NULL)
                )
                or
                (
                    (c != _mpi.COMM_NULL)
                    and
                    (inter_locale_comm0 == _mpi.COMM_NULL)
                )
                or
                _mpi.Comm.Compare(inter_locale_comm0, c) not in equivalent_compare
            ):
                raise ValueError(
                    (
                        "Got inter_locale_comm=%s (name=%s) non-congruent with "
                        +
                        " inter_locale_comm=%s (name=%s)."
                    )
                    %
                    (
                        inter_locale_comm0,
                        inter_locale_comm0.name if inter_locale_comm0 != _mpi.COMM_NULL else "",
                        c,
                        c.name if c != _mpi.COMM_NULL else ""
                    )
                )

class GndarrayArrayUfuncExecutor(object):

    """
    Instances execute a ufunc for a :obj:`mpi_array.globale.gndarray`.
    Takes care of creating outputs, remote fetching of required parts of inputs
    and forwarding call to :obj:`numpy.ufunc` instance to perform
    the computation on the locale :obj:`numpy.ndarray` instances.
    """

    def __init__(self, array_like_obj, ufunc, method, *inputs, **kwargs):
        """
        Initialise.

        :type array_like_obj: :obj:`mpi_array.globale.gndarray`
        :param array_like_obj: The :obj:`mpi_array.globale.gndarray` which
           triggered the :samp:`__array_ufunc__` call.
        :type ufunc: :obj:`numpy.ufunc`
        :param ufunc: The ufunc to be executed.
        :type method: :obj:`str`
        :param method: The name of the method of :samp:`{ufunc}` which is
           to be executed.
        :type inputs: array like
        :param inputs: The ufunc inputs.
        :type kwargs: keyword args
        :param kwargs: The ufunc keyword arguments.
        """
        self._array_like_obj = array_like_obj
        self._ufunc = ufunc
        self._method = method
        self._inputs = convert_to_array_like(inputs)
        self._kwargs = kwargs
        self._outputs = None
        if "out" in self._kwargs.keys():
            self._outputs = self._kwargs["out"]
        self._casting = None
        if "casting" in self._kwargs.keys():
            self._casting = self._kwargs["casting"]
        else:
            self._casting = "same_kind"

    @property
    def array_like_obj(self):
        """
        The :obj:`mpi_array.globale.gndarray` object which triggered the
        construction of this :obj:`GndarrayArrayUfuncExecutor` object.
        """
        return self._array_like_obj

    @property
    def peer_comm(self):
        """
        The peer :obj:`mpi4py.MPI.Comm` communicator.
        """
        return self._array_like_obj.locale_comms.peer_comm

    @property
    def intra_locale_comm(self):
        """
        The intra-locale :obj:`mpi4py.MPI.Comm` communicator.
        """
        return self._array_like_obj.locale_comms.intra_locale_comm

    @property
    def inter_locale_comm(self):
        """
        The inter-locale :obj:`mpi4py.MPI.Comm` communicator.
        """
        return self._array_like_obj.locale_comms.inter_locale_comm

    @property
    def ufunc(self):
        """
        The :obj:`numpy.ufunc` to be executed.
        """
        return self._ufunc

    @property
    def outputs(self):
        """
        The ufunc :obj:`mpi_array.globale.gndarray` output arrays.
        """
        return self._outputs

    @property
    def inputs(self):
        """
        The sequence of ufunc inputs.
        """
        return self._inputs

    @property
    def casting(self):
        """
        A :obj:`str` indicating the casting mode.
        """
        return self._casting

    @property
    def method(self):
        """
        A :obj:`str` indicating the method of the :attr:`ufunc` to be executed.
        """
        return self._method

    def get_inputs_shapes(self):
        """
        Returns a *shape* :obj:`tuple` for each element of :attr:`inputs`.

        :rtype: :obj:`tuple`
        :return: Shape of each ufunc input.
        """
        return \
            tuple(
                input.shape
                if hasattr(input, "shape") else
                _np.asarray(input).shape
                for input in self._inputs
            )

    def get_best_match_input(self, result_shape):
        """
        Returns the element of :attr:`inputs` whose globale shape
        best matches :samp:`{result_shape}`.

        :rtype: :samp:`None` or :obj:`mpi_array.globale.gndarray`.
        :return: The input array whose shape matches :samp:`{result_shape}`,
           or :samp:`None` if none of the inputs are a good match.
        """
        best_input = None
        result_shape = _np.array(result_shape, dtype="int64")
        input_shapes = self.get_inputs_shapes()
        are_same_shape = \
            _np.array(
                tuple(
                    (len(result_shape) == len(in_shape)) and _np.all(result_shape == in_shape)
                    for in_shape in input_shapes
                )
            )
        if _np.any(are_same_shape):
            best_input = self._inputs[_np.where(are_same_shape)[0][0]]
        else:
            input_shapes = \
                _np.array(
                    tuple(
                        _np.array(shape_extend_dims(len(result_shape), in_shape))
                        for in_shape in input_shapes
                    ),
                    dtype="int64"
                )
            d = input_shapes - result_shape
            d *= d
            d = d.sum(axis=1)
            best_input = self._inputs[_np.argmin(d)]

        return best_input

    def create_outputs(self, outputs, result_shape, result_types):
        """
        Returns list of output :obj:`mpi_array.globale.gndarray` instances.

        :type outputs: :samp:`None` or :obj:`tuple` of :obj:`mpi_array.globale.gndarray`
        :param outputs: Output arrays passed in as the :samp:`out` argument
           of the :obj:`numpy.ufunc`.
        :type result_shape: sequence of :obj:`int`
        :param result_shape: The shape of all output arrays.
        :type result_types: sequence of :samp:`numpy.dtype`
        :param result_types: The :samp:`dtype` of each output array. Note
            that this is the list for all outputs including any
            in the :samp:`outputs` argument. This determines the
            number of output arrays.
        :rtype: :obj:`list` of :obj:`mpi_array.globale.gndarray`
        :return: A list of length :samp:`len(result_types)` elements,
           each element is a :obj:`mpi_array.globale.gndarray`.
        """

        template_output_gary = None
        if (outputs is not None) and (len(outputs) > 0):
            check_equivalent_inter_locale_comms(outputs)
            template_output_gary = outputs[-1]
        else:
            best_match_input = self.get_best_match_input(result_shape)
            comms_distrib = None
            if best_match_input is not None:
                comms_distrib = \
                    _comms.reshape_comms_distribution(
                        best_match_input.comms_and_distrib,
                        result_shape
                    )
            if comms_distrib is not None:
                template_output_gary = \
                    _globale_creation.empty(
                        result_shape,
                        comms_and_distrib=comms_distrib,
                        dtype=result_types[0]
                    )
            else:
                template_output_gary = \
                    _globale_creation.empty(
                        result_shape,
                        dtype=result_types[0],
                        peer_comm=self.peer_comm,
                        intra_locale_comm=self.intra_locale_comm,
                        inter_locale_comm=self.inter_locale_comm
                    )
            outputs = (template_output_gary,)
        outputs = \
            (
                outputs
                +
                tuple(
                    _globale_creation.empty_like(template_output_gary, dtype=result_types[i])
                    for i in range(len(outputs), len(result_types))
                )
            )

        return outputs

    def get_input_extents(self, locale_info):
        """
        Returns tuple of :samp:`(locale_extent, globale_extent)` pairs,
        one for each of the :attr:`inputs`.

        :type locale_info: :obj:`mpi_array.comms.ThisLocaleInfo`
        :param locale_info: The rank info required for constructing
            a :obj:`mpi_array.distribution.LocaleExtent` instance
            for :samp:`input` types which are not :obj:`mpi_array.globale.gndarray`.
        :rtype: :obj:`tuple`
        :return: Pairs which indicate the locale extent of the ufunc :attr:`inputs`.

        .. seealso:: :func:`get_extents`
        """
        return \
            tuple(
                get_extents(inp, locale_info) for inp in self.inputs
            )

    def get_numpy_ufunc_peer_rank_inputs_outputs(self, gndarray_outputs):
        """
        Returns two element tuple of :samp:`(input_arrays, output_arrays)` which
        are to be passed to the :obj:`numpy.ufunc` object :attr:`ufunc`.

        :type gndarray_outputs: sequence of :obj:`mpi_array.globale.gndarray`
        :param gndarray_outputs: The output arrays. All arrays should be the
           same shape and same distribution.
        :rtype: :samp:`None` or :obj:`tuple`
        :return: A tuple :samp:`(input_arrays, output_arrays)` of inputs and
           outputs which are to be passed to :obj:`numpy.ufunc` call.
           Returns :samp:`None` if the output locale extents are empty (i.e. no
           array elements to compute on this locale).
        """
        # First fetch/slice the parts of the input required for the locale extent
        out_gndarray = gndarray_outputs[0]
        out_globale_extent = out_gndarray.distribution.globale_extent
        out_locale_extent = out_gndarray.lndarray_proxy.locale_extent
        ret = None
        if _np.product(out_locale_extent.shape_n) > 0:
            inp_locale_extents = \
                self.get_input_extents(out_gndarray.comms_and_distrib.this_locale)
            inp_locale_slices = \
                calc_matching_locale_slices(
                    out_locale_extent,
                    out_globale_extent,
                    inp_locale_extents
                )

            inp_locale_arys = [None, ] * len(self.inputs)
            for i in range(len(self.inputs)):
                input = self.inputs[i]
                slice_tuple = inp_locale_slices[i]
                if slice_tuple is not None:
                    if hasattr(input, "locale_get"):
                        # is a gndarray
                        inp_locale_arys[i] = input.locale_get(slice_tuple)
                    else:
                        # is a numpy array (or similar)
                        inp_locale_arys[i] = input[slice_tuple]
                else:
                    # is a scalar
                    inp_locale_arys[i] = input

            # Now slice the locale input arrays to match the peer-rank portions of the output.
            out_peer_rank_slice = out_gndarray.lndarray_proxy.intra_partition.rank_view_slice_n
            out_peer_rank_slice = out_locale_extent.locale_to_globale_slice_h(out_peer_rank_slice)
            out_peer_rank_slice = out_locale_extent.globale_to_locale_slice_n(out_peer_rank_slice)

            inp_peer_rank_slices = calc_matching_peer_rank_slices(
                out_peer_rank_slice, inp_locale_arys)

            inp_peer_rank_arys = [None, ] * len(inp_locale_arys)
            for i in range(len(inp_locale_arys)):
                input = inp_locale_arys[i]
                slice_tuple = inp_peer_rank_slices[i]
                if slice_tuple is not None:
                    # is a numpy array (or similar)
                    inp_peer_rank_arys[i] = input[slice_tuple]
                else:
                    # is a scalar
                    inp_peer_rank_arys[i] = input

            ret = \
                (
                    tuple(inp_peer_rank_arys),
                    tuple(
                        out_gndarray.view_n[out_peer_rank_slice]
                        for out_gndarray in gndarray_outputs
                    )
                )
        return ret

    def need_remote_data(self, gndarray_outputs):
        """
        Returns :samp:`True` if any locale needs to fetch remote
        input data in order to compute the all elements of the
        outputs :samp:`{gndarray_outputs}`.

        :type gndarray_outputs: sequence of :obj:`mpi_array.globale.gndarray`
        :param gndarray_outputs: Check whether any of the locales require remote
           data in order to compute these outputs.
        :rtype: :obj:`bool`
        :return: :samp:`True` if remote fetch of input data is required
           in order to compute ufunc for the given outputs.
        """
        out_gndary = gndarray_outputs[0]
        need_remote = False
        if out_gndary.locale_comms.inter_locale_comm != _mpi.COMM_NULL:
            START_STR = LocaleExtent.START_N_STR
            STOP_STR = LocaleExtent.STOP_N_STR
            gndarray_inputs = \
                tuple(
                    input for input in self.inputs
                    if hasattr(input, "distribution") and hasattr(input, "locale_comms")
                )
            out_s_ext = out_gndary.distribution.struct_locale_extents
            for inp_gndary in gndarray_inputs:
                need_remote = \
                    (
                        _mpi.Comm.Compare(
                            out_gndary.locale_comms.inter_locale_comm,
                            inp_gndary.locale_comms.inter_locale_comm
                        )
                        ==
                        _mpi.UNEQUAL
                    )
                if not need_remote:
                    # first make sure that the inter_locale_comm is compatible
                    # between input and output
                    translated_ranks = \
                        _mpi.Group.Translate_ranks(
                            out_gndary.locale_comms.inter_locale_comm.group,
                            _np.arange(out_gndary.locale_comms.inter_locale_comm.group.size),
                            inp_gndary.locale_comms.inter_locale_comm.group
                        )
                    inp_s_ext = \
                        inp_gndary.distribution.struct_locale_extents[_np.asarray(translated_ranks)]

                    # Now check that the output locale extent is contained
                    # within the input locale extent.
                    # Dimension of input can be smaller than the output
                    # because of broadcasting rules.
                    need_remote = True
                    not_out_empty = \
                        _np.product(out_s_ext[STOP_STR] - out_s_ext[START_STR], axis=1) > 0

                    ndim = inp_gndary.ndim

                    beyond_out_extent = \
                        _np.logical_or.reduce(
                            (out_s_ext[START_STR][:, -ndim:] < inp_s_ext[START_STR])
                            |
                            (out_s_ext[STOP_STR][:, -ndim:] <= inp_s_ext[START_STR])
                            |
                            (out_s_ext[START_STR][:, -ndim:] >= inp_s_ext[STOP_STR])
                            |
                            (out_s_ext[STOP_STR][:, -ndim:] > inp_s_ext[STOP_STR]),
                            axis=1
                        )

                    need_remote = \
                        _np.any(
                            not_out_empty
                            &
                            beyond_out_extent
                        )

                if need_remote:
                    break
        # All ranks in the locale need to know the result, broadcast.
        need_remote = out_gndary.locale_comms.intra_locale_comm.bcast(need_remote, 0)

        return need_remote

    def execute___call__(self):
        """
        """
        from .globale import gndarray as _gndarray

        # Calculate the shape of the output arrays.
        result_shape = broadcast_shape(*(self.get_inputs_shapes()))
        self.array_like_obj.rank_logger.debug("result_shape=%s", result_shape)

        # Calculate the result dtype for each output array
        result_types = ufunc_result_type(self.ufunc.types, self.inputs, self.outputs, self.casting)
        self.array_like_obj.rank_logger.debug("result_types=%s", result_types)

        # Create the output gndarray instances
        gndarray_outputs = self.create_outputs(self.outputs, result_shape, result_types)
        self.array_like_obj.rank_logger.debug(
            "output shapes=%s", [o.shape for o in gndarray_outputs]
        )

        # Check whether remote fetch of data is needed
        # for any locale before calling this barrier. If all locales
        # have local data then this barrier isn't be necessary.
        # Otherwise, we have to sync to make sure that remote ranks have
        # finished writing data before starting to fetch it.
        if self.need_remote_data(gndarray_outputs):
            for i in self.inputs:
                if isinstance(i, _gndarray):
                    i.initialise_windows()
            gndarray_outputs[0].inter_locale_barrier()

        # Fetch the peer-rank sub-arrays of the input arrays needed
        # to calculate the corresponding sub-array of the outputs.
        np_ufunc_inputs_and_outputs = \
            self.get_numpy_ufunc_peer_rank_inputs_outputs(gndarray_outputs)

        if np_ufunc_inputs_and_outputs is not None:
            np_ufunc_inputs, np_ufunc_outputs = np_ufunc_inputs_and_outputs

            # Call the self.ufunc.__call__ method to perform the computation
            # in the sub-arrays
            kwargs = dict()
            kwargs.update(self._kwargs)
            kwargs["out"] = np_ufunc_outputs
            self.array_like_obj.rank_logger.debug(
                "Calling numpy.ufunc=%s:\ninputs=%s\noutputs=%s",
                self.ufunc, np_ufunc_inputs, kwargs["out"]
            )
            self.ufunc.__call__(*np_ufunc_inputs, **kwargs)
            self.array_like_obj.rank_logger.debug(
                "Finished numpy.ufunc=%s:\noutputs=%s",
                self.ufunc,
                kwargs["out"]
            )
        else:
            self.array_like_obj.rank_logger.debug(
                "Locale output extent is empty, skipping call to self.ufunc=%s:\nOutput extent=%s",
                self.ufunc,
                gndarray_outputs[0].lndarray_proxy.locale_extent
            )

        gndarray_outputs[0].intra_locale_barrier()

        # return the outputs
        if len(gndarray_outputs) == 1:
            gndarray_outputs = gndarray_outputs[0]
        return gndarray_outputs

    def execute_accumulate(self):
        """
        Not implemented.
        """
        return NotImplemented

    def execute_reduce(self):
        """
        Not implemented.
        """
        return NotImplemented

    def execute_reduceat(self):
        """
        Not implemented.
        """
        return NotImplemented

    def execute_at(self):
        """
        Not implemented.
        """
        return NotImplemented

    def execute_outer(self):
        """
        Not implemented.
        """
        return NotImplemented

    def execute(self):
        """
        Perform the ufunc operation. Call is forwarded to one
        of: :meth:`execute___call__`, :meth:`execute_accumulate`, :meth:`execute_at`
        , :meth:`execute_outer`, :meth:`execute_reduce` or :meth:`execute_reduceat`.
        """
        return getattr(self, "execute_" + self.method)()

#: Factory for generating instance of :obj:`GndarrayArrayUfuncExecutor`.
gndarray_ufunc_executor_factory = GndarrayArrayUfuncExecutor

def gndarray_array_ufunc(array_like_obj, ufunc, method, *inputs, **kwargs):
    """
    The implementation for  :meth:`mpi_array.globale.gndarray.__array_ufunc__`.
    """
    ufunc_executor = \
        gndarray_ufunc_executor_factory(
            array_like_obj,
            ufunc,
            method,
            *inputs,
            **kwargs
        )

    return ufunc_executor.execute()

def set_numpy_ufuncs_as_module_attr(set_attr_module, search_module):
    """
    Finds all :obj:`numpy.ufunc` attributes in the :samp:`{search_module}` :obj:`module`
    and sets corresponding attributes of :samp:`{set_attr_module}` :obj:`module`.

    :type set_attr_module: :obj:`module`
    :param set_attr_module: Set ufunc attributes of this module to those found
       in the :samp:`{search_module}` module
    :type search_module: :obj:`module`
    :param search_module: Find :obj:`numpy.ufunc` attributes in this module.

    """
    for attr in dir(search_module):
        numpy_attr_value = getattr(search_module, attr)
        if isinstance(numpy_attr_value, _np.ufunc):
            setattr(set_attr_module, attr, numpy_attr_value)

set_numpy_ufuncs_as_module_attr(_sys.modules[__name__], _np)

__all__ = [s for s in dir() if not s.startswith('_')]


import numpy as np
import pandas as pd
import sys # can use sys to take command line arguments

class Recommender():
    '''
    What is this class all about - write a really good doc string here
    '''
    def __init__(self, ):
        '''
        what do we need to start out our recommender system
        '''

    def fit(self, ):
        '''
        fit the recommender to your dataset and also have this save the results
        to pull from when you need to make predictions
        '''

    def predict_rating(self, ):
        '''
        makes predictions of a rating for a user on a movie-user combo
        '''

    def make_recs(self,):
        '''
        given a user id or a movie that an individual likes
        make recommendations
        '''

if __name__ == '__main__':
    # test different parts to make sure it works


import pandas as pd
import numpy as np
import unittest

class Mytest(unittest.TestCase):
    '''
    Checks scalar or array if there is anything missing in it
    '''
    def setUp(self):
        self.str='Hira'
        self.testNone=None
        self.testNan=' '
        self.testarray=[1,2,3]
        self.test1=[6,'h',None]

    def test_notnull(self):
        self.assertEqual(pd.notna(self.str),True)
        self.assertEqual(pd.notna(self.testNone),False)
        self.assertEqual(pd.notna(self.testNan),True)
        self.assertEqual(pd.notna(self.testarray[2]), True)
        self.assertEqual(pd.notna(self.testarray).all(),(True))
        self.assertEqual(pd.notna(self.test1).all(), (False))

if __name__  == '__main__':
    unittest.main()


# Lint as: python3
# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Tests lib for quant_utils test."""

# pylint: disable=bad-whitespace
# pylint: disable=bad-continuation

import lingvo.compat as tf
from lingvo.core import py_utils
from lingvo.core import quant_utils
from lingvo.core import test_utils
import numpy as np

class SampleQuantizedProjectionLayer(quant_utils.QuantizableLayer):
  """Simple projection layer to demonstrate quantization."""

  @classmethod
  def Params(cls):
    p = super().Params()
    p.Define('input_dim', 2, 'Depth of the input.')
    p.Define('output_dim', 3, 'Depth of the output.')
    return p

  def __init__(self, params):
    super().__init__(params)
    p = self.params
    self.CreateAqtWeight(
        'aqt_w', shape=[p.input_dim, p.output_dim], feature_axis=-1)

  def _CreateLayerVariables(self):
    super()._CreateLayerVariables()
    p = self.params

    w_pc = py_utils.WeightParams(
        shape=[p.input_dim, p.output_dim],
        init=p.params_init,
        dtype=p.dtype,
        collections=[self.__class__.__name__ + '_vars'])
    self.CreateVariable('w', w_pc)

    self.TrackQTensor('inputs', 'transformed')

  def FProp(self, theta, inputs, paddings):
    p = self.params
    fns = self.fns

    # It is the most important that weights and top-level activations
    # be tagged for quantization:
    #   - Weights use the self.QWeight() decorator
    #   - Inputs/activations are decorated with self.QTensor(). In general,
    #     the provided name should match a call to self.TrackQTensor in the
    #     constructor. This creates an tensor that is individually accounted
    #     for.
    w = fns.qweight(theta.w)

    # TODO(shivaniagrawal): change this to ToAqtWeight and FromAqtWeight.
    w = self.ToAqtWeight(
        'aqt_w', w, feature_axis=-1, expected_scale_shape=(1, p.output_dim))

    inputs = self.QTensor('inputs', inputs)

    # Note the use of the qmatmul from the function library. This will
    # automatically track the output against the qtensor 'transformed'.
    out = fns.qmatmul(
        tf.reshape(inputs, [-1, p.input_dim]), w, qt='transformed')
    out = self.FromAqtWeight('aqt_w', out, feature_axis=-1)

    out = tf.reshape(out, tf.concat([tf.shape(inputs)[:-1], [p.output_dim]], 0))

    # Decorate outputs of simple activation functions with their corresponding
    # range decorator. This will ensure that the result does not exceed the
    # precision of the underlying representation.
    out = fns.qtanh(out)

    # Perform padding manipulation via booleans instead of:
    #   out *= 1.0 - paddings
    # Because the paddings can exist in entirely different numeric ranges than
    # the tensor they are being applied to, it is best to not perform
    # arithmetic directly between them. Instead, broadcast them to the needed
    # size (if different) and perform an exact mask with tf.where.
    # For added numeric range protection, the QRPadding decorator ensures
    # the correct range. This is mostly needed for cases where padding is
    # dynamic at inference time.
    paddings = self.QRPadding(paddings)
    paddings *= tf.ones_like(out)  # Broadcast to 'out' size.
    out = tf.where(paddings > 0.0, tf.zeros_like(out), out)

    return out

class QuantUtilsBaseTest(test_utils.TestCase):
  """Base test class for testing quantizable layer."""

  # pyformat: disable
  NO_QDOMAIN_EXPECTED = [
   [[ 0.00071405, -0.03868543, -0.01999986, -0.00994987],
    [ 0.08905827,  0.13636404, -0.03180931,  0.06056439],
    [ 0.        ,  0.        ,  0.        ,  0.        ],
    [-0.0208858 , -0.17595209, -0.05192588,  0.02618068]],
   [[ 0.        ,  0.        ,  0.        ,  0.        ],
    [ 0.        ,  0.        ,  0.        ,  0.        ],
    [-0.02125708, -0.10454545, -0.01147466,  0.06903321],
    [ 0.0276652 , -0.14823943, -0.09726462,  0.01415125]]]
  # pyformat: enable

  def _testLayerHelper(self,
                       test_case,
                       p,
                       expected=None,
                       not_expected=None,
                       global_step=-1):
    tf.random.set_seed(398847392)
    np.random.seed(12345)
    p.name = 'proj'
    p.input_dim = 3
    p.output_dim = 4
    p.params_init = py_utils.WeightInit.Gaussian(0.1)
    l = p.Instantiate()
    in_padding = tf.zeros([2, 4, 1], dtype=tf.float32)
    in_padding = tf.constant(
        [[[0], [0], [1], [0]], [[1], [1], [0], [0]]], dtype=tf.float32)
    inputs = tf.constant(
        np.random.normal(0.1, 0.5, [2, 4, 3]), dtype=tf.float32)
    output = l.FPropDefaultTheta(inputs, in_padding)
    self.evaluate(tf.global_variables_initializer())

    if global_step >= 0:
      self.evaluate(tf.assign(py_utils.GetOrCreateGlobalStepVar(), global_step))

    output = output.eval()
    print('QuantizableLayerTest output', test_case, ':\n',
          np.array_repr(output))
    if expected is not None:
      self.assertAllClose(output, expected)
    if not_expected is not None:
      self.assertNotAllClose(output, not_expected)
    return l


"""
Used to train ResNet-50
Author: Kaihua Tang
"""

import math
import time
import tensorflow as tf
import ResNet as resnet
import numpy as np
import scipy.io as scio
from scipy import misc
from utils import *

# image size
WIDTH = 224
HEIGHT = 224
CHANNELS = 3
#"Mini batch size"
MINI_BATCH_SIZE = 32
#"Path of Label.npy"
label_path = "./label/label_1200.npy"
#"Path of image file names"
image_name_path = "./label/name_1200.npy"
# image path
parentPath = "F:\\CACD2000_Crop\\"
# data Path: n * 224 * 224 * 3 numpy matrix
data_path = 'F:\\Dataset\\1200_data.npy'

def Train():
    """
    HyperParameters of the Net
    model_path: path of pretrained model, set None if there is no such a model.
    LABELSNUM: Number of output labels
    learning_rate_orig : original learning rate
    NUM_EPOCHS: number of epochs
    save_frequency: frequency of saving model (number of epoches)
    """
    model_path ="./model/03.npy"
    LABELSNUM = 1200
    learning_rate_orig = 1e-05
    NUM_EPOCHS = 1000
    save_frequency = 2
    """
    Classification Layer
    final_layer_type: softmax or sigmoid
    is_sparse: when final layer is softmax, is it sparse
    """
    final_layer_type ="softmax"
    is_sparse = True
    """
    Tensorboard Setting
    tensorboard_on: Turn on Tensorboard or not
    TensorBoard_refresh: refresh rate (number of batches)
    monitoring_rate: Print output rate
    """
    tensorboard_on = False
    TensorBoard_refresh = 50
    monitoring_rate = 50

    #Lists that store name of image and its label
    trainNameList = np.load(image_name_path)
    trainLabelList = np.load(label_path)
    if(data_path is None):
        allImageData = load_all_image(trainNameList, HEIGHT, WIDTH, CHANNELS, parentPath)
    else:
        allImageData = np.load(data_path)

    #num of total training image
    num_train_image = trainLabelList.shape[0]

    with tf.Session() as sess:
        images = tf.placeholder(tf.float32, shape = [None, WIDTH, HEIGHT, CHANNELS])
        if(is_sparse):
            labels = tf.placeholder(tf.int64, shape = [None])
        else:
            labels = tf.placeholder(tf.float32, shape = [None, LABELSNUM])

        # build resnet model
        resnet_model = resnet.ResNet(ResNet_npy_path = model_path)
        resnet_model.build(images, LABELSNUM, final_layer_type)
        # number of batches per epoch
        num_minibatches = int(num_train_image / MINI_BATCH_SIZE)

        # cost function
        learning_rate = learning_rate_orig
        with tf.name_scope("cost"):
            if(final_layer_type == "sigmoid"):
                print("Using weighted sigmoid loss")
                loss = tf.nn.weighted_cross_entropy_with_logits(logits = resnet_model.fc1, targets = labels, pos_weight = 5.0)
            elif(final_layer_type == "softmax" and is_sparse):
                print("Using sparse softmax loss")
                loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits = resnet_model.fc1, labels = labels)
            elif(final_layer_type == "softmax" and (not is_sparse)):
                print("Using softmax loss")
                loss = tf.nn.softmax_cross_entropy_with_logits(logits = resnet_model.fc1, labels = labels)
            cost = tf.reduce_sum(loss)
        with tf.name_scope("train"):
            global_steps = tf.Variable(0)
            learning_rate = tf.train.exponential_decay(learning_rate_orig, global_steps, num_minibatches * 40, 0.1, staircase = True)
            #train = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
            #train = tf.train.AdamOptimizer(learning_rate).minimize(cost)
            train = tf.train.MomentumOptimizer(learning_rate, 0.9).minimize(cost)

        sess.run(tf.global_variables_initializer())
        print(resnet_model.get_var_count())

        if(tensorboard_on):
            merged_summary = tf.summary.merge_all()
            writer = tf.summary.FileWriter("./TensorBoard/Result")
            writer.add_graph(sess.graph)
            # used in tensorboard to count record times
            summary_times = 0

        for epoch in range(NUM_EPOCHS):
            print("Start Epoch %i" % (epoch + 1))

            minibatch_cost = 0.0
            # count the number of batch
            batch_index = 0
            # get index for all mini batches
            minibatches = random_mini_batches(num_train_image, MINI_BATCH_SIZE, random = True)

            for minibatch in minibatches:
                # get train examples from each mini batch
                (minibatch_X, minibatch_Y) = get_minibatch(minibatch, trainLabelList, HEIGHT, WIDTH, CHANNELS, LABELSNUM, allImageData, is_sparse)

                # change learning rate
                sess.run(global_steps.assign(epoch * num_minibatches + batch_index))

                # record examples to monitoring the training process
                if((batch_index % monitoring_rate == 0)):
                    resnet_model.set_is_training(False)
                    fc1, prob = sess.run([resnet_model.fc1, resnet_model.prob], feed_dict={images: minibatch_X})
                    countMax = np.sum(np.argmax(prob,1) == minibatch_Y)
                    print("Epoch %i Batch %i Before Optimization Count %i" %(epoch + 1,batch_index, countMax))

                # Training and calculating cost
                resnet_model.set_is_training(True)
                temp_cost, _ = sess.run([cost, train], feed_dict={images: minibatch_X, labels: minibatch_Y})
                minibatch_cost += np.sum(temp_cost)

                # tensorboard
                if(tensorboard_on) and (batch_index % tfBoard_refresh == 0):
                    s = sess.run(merged_summary, feed_dict={images: minibatch_X, labels: minibatch_Y})
                    writer.add_summary(s, summary_times)
                    summary_times = summary_times + 1
                    # record cost in tensorflow
                    tf.summary.scalar('cost', temp_cost)

                # record examples to monitoring the training process
                if((batch_index % monitoring_rate == 0)):
                    resnet_model.set_is_training(False)
                    fc1, prob = sess.run([resnet_model.fc1, resnet_model.prob], feed_dict={images: minibatch_X})
                    countMax = np.sum(np.argmax(prob,1) == minibatch_Y)
                    print("Epoch %i Batch %i After Optimization Count %i" %(epoch + 1,batch_index, countMax))
                    # Temp Cost & learning rate
                    print("Epoch %i Batch %i Batch Cost %f Learning_rate %f" %(epoch + 1,batch_index, np.sum(temp_cost), sess.run(learning_rate) * 1e10))

                batch_index += 1

            # print total cost of this epoch
            print("End Epoch %i" % (epoch + 1))
            print("Total cost of Epoch %f" % minibatch_cost)

            # save model
            if((epoch + 1) % save_frequency == 0):
                resnet_model.save_npy(sess, './model/temp-model%i.npy' % (epoch + 1))

if __name__ == '__main__':
    Train()