# Business Objectives/Statistical Questions and Feature Selection

*Hayley Boyce, May 12th, 2021*

**Attribution:**

- Tomas Beuzen - Previous BAIT 509 Lecture 6
- Varada Kolhatkar - Heavily guiding me with Feature Selection

# Importing our libraries
import pandas as pd
import altair as alt
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.dummy import DummyClassifier, DummyRegressor
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.model_selection import cross_validate, train_test_split
from sklearn import datasets
from sklearn.linear_model import Ridge

import sys
sys.path.append('code/')
from display_tree import display_tree
from plot_classifier import plot_classifier
import matplotlib.pyplot as plt

# Preprocessing and pipeline
from sklearn.impute import SimpleImputer
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.compose import make_column_transformer
from sklearn.linear_model import LogisticRegression

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, MinMaxScaler

import scipy
from sklearn.model_selection import RandomizedSearchCV

## House Keeping

- Project instructions are out!
- Time in class for groups!
- Assignment 3 was released yesterday!
- More reading in this lecture than usual (sorry!)
- Run code at your own risk (Think section can take up to 20 minutes so if you are running this locally, Start running everything NOW from top to bottom)

## Lecture Learning Objectives

- In the context of supervised learning, form statistical questions  from business questions/objectives.
- Understand the different forms your client may expect you to communicate results.
- Explain the general concept of feature selection.
- Discuss and compare different feature selection methods at a high level.
- Use sklearn's implementation of recursive feature elimination (RFE).
- Implement the forward search algorithm.

## Five Minute Recap/ Lightning Questions

- What is the name of the function used to bound our values between 0 and 1
- What is the name of the function that gives "hard" predictions?
- What is the name of the function that gives "soft" predictions?
- What is the hyperparameter we learned for Ridge and how does it affect the Fundamental Trade-off?
- What is the hyperparameter we learned for Logistic Regression and how does it affect the Fundamental Trade-off?

### Some lingering questions

- How can we start forming good business questions that can be addressed with Machine Learning?
- How to select features for our models?

## Forming statistical questions to answer business objectives

So far you've seen how to solve predictive problems using machine learning but today, we are going to look at the process involved in asking the questions and problems faced by organizations.

Generally, there are four parts of a machine learning analysis. In order from high to low level:

1. **The business question/objective**
2. **The statistical question/objective**
3. **The data and model**
4. **The data product**

Doing a machine learning analysis is about distilling from the highest level to the lowest level. As such, there are three distillations to keep in mind: 1-2, 2-3, and 3-4:

- **1-2 is about asking the right questions**
- **2-3 is about building a useful model**
- **3-4 is about communicating the results**

Note that an analysis isn’t a linear progression through these “steps”; rather, the process is iterative. This is because none of the components are independent. Making progress on any of the three distillations gives you more information as to what the problem is.

We’ll look at each of these distillations in turn.

## (1 - 2) Asking useful statistical questions

- Usually, a company is not served up a machine learning problem, complete with data and a description of the response and predictors.
    - Companies don't exactly know what question is the right question but they do know what they want  to be accomplished.
- Instead, they’re faced with some high-level objective/question that we’ll call the **business question/objective**.
- This question needs refining to a **statistical question/objective** – one that is directly addressable by machine learning.

Example:

- My Capstone Case study.

### Business objectives: examples
- This [altexsoft blog post](https://www.altexsoft.com/blog/business/supervised-learning-use-cases-low-hanging-fruit-in-data-science-for-businesses/) is a great introduction to business use cases of data science/ML
- Examples of business objectives (for which machine learning is a relevant approach)
    - Reduce the amount of spam email received
    - Early prediction of product failure
    - Find undervalued mines
    - Make a transit system more efficient
    - Hire efficient staff

### Refining business objectives to statistical objectives

- Statistical objectives need to be specific
- Remember that supervised learning is about predicting a response $Y$ from predictors $X_1,…,X_p$
- So we need to refine our business objectives to a statistical question(s) we can answer
- This typically involves:
    - Identifying the **response variable** ($Y$) that is most aligned with the business objective.
    - Identifying the **data** (observations + features) that will be used for model development/testing.
    - Note: part of this is the task of feature selection (a topic that we've covered briefly) – but, this is also largely, a human decision based on what we think is more informative, as well as a resource questions (what data is actually available?)

### Statistical objectives: examples
Statistical objectives corresponding to the above business objective examples might be:

| Business Objective | Statistical Question |
| :--- | :--- |
| Reduce the amount of spam email received |

$Y$ = classifying an email as spam/not spam
 $X$ = words present in name and the body of email and other metadata (sender email, time, etc.)
 Cases of spam will be gathered over time as employees identify emails as spam/not spam. The model can be improved as misclassifications are encountered.

| Early prediction of product failure (Kickstarter?) |

$Y$ = classifying a product as faulty/not faulty
 $X$ = Relevant features chosen by an expert
 Data obtained from the test facility

| Find undervalued mines |

$Y$ = total volume of gold and silver at a site
 $X$ =  concentrations of other minerals found in drill samples, geographic information, historical data, etc
 Data obtained from mines where total volumes are already known

| Make a transit system more efficient |

$Y$ = predict the time it takes a bus to travel between set stops
 $X$ = time of day/week/year, weather, etc.
 Use data from company server tracking bus movements

| Hire efficient staff |

$Y$ = predict monthly sales
 $X$ = a personality test, years of work experience, field of experience, etc.
 Use data based on current employees

### Statistical questions are not the full picture!
- Almost always, the business objective is more complex than the statistical question.
- By refining a business objective to a statistical one, we may lose part of the essence of the business objective.
- It’s important to have a sense of the ways in which your statistical objective falls short, and the ways in which it’s on the mark, so that you keep an idea of the big picture.
- For example, predicting whether a new staff hire will be efficient or not is a useful statistical question, but doesn't consider why a company might be attracting certain applicants, how long staff will remain, how staff work together, etc.

### Statistical objectives unrelated to supervised learning

- We are only focussing on statistical questions related to supervised learning and prediction in this course
- But there are other kinds of questions you can ask too
- Consider the following example

**Business objective**: To gain insight into the productivity of two branches of a company.

Examples of statistical questions:

- **Hypothesis testing**: Is the mean number of sick days per employee different between two branches of a company?
    - Supervised learning doesn’t cover testing for differences.
- **Unsupervised learning**: What is the mean sentiment of the internal discussion channels from both branches?
    - There is no data of feature + response here, as required by supervised learning (by definition).
- **Statistical inference**: Estimate the mean difference in monthly revenue generated by both branches, along with how certain you are with that estimate.
    - Supervised learning typically isn’t concerned about communicating how certain your estimate is. (BUT it should and progress is occuring to change this!)

## (2 - 3) Building a useful model
- This is really the main focus of this course, This is the meat/beyond meat patty in your burger!
- This involves using ML algorithms (kNN, loess, decision trees, etc) to build a predictive model from data
- You always should include a baseline model to assess how well the models you build are giving you some leg up.
- A simple model like logistic regression does as well as more complex approaches! At the very least, they can help guide you on what more complex approaches to take next.

## (3 - 4) Communicating results

- So you've distilled your business objectives to a statistical question.
- You've developed a model to answer the statistical question.
- Now your model needs to be delivered and used by others (or your future self)!
- The final delivery is often called "the data product" because it may consist of a variety of things:
    - a report
    - a presentation
    - an app
    - a dashboard
    - a software package/pipeline
- Sometimes the client requests a specific data product -> But note that their suggestion might not always be the best option.
    - Perhaps they request a report and presentation communicating your findings, when a more appropriate product also includes an interactive app that allows them to explore your findings for themselves.
- Either way, the key here is communication. Two import challenges (relevant to your final project):
    - Using appropriate language: there is a lot of jargon in ML, the key is to talk more about the output and the general idea of your model(s), but not machine learning or statistical jargon.
    - Communication with visual design: this is about choosing what visuals are the most effective for communicating. -> Plug: https://viz-learn.mds.ubc.ca/en/
- Usually, the first step is to set up a framework for your data product. For a report, this means outlining what you intend to write about, and where.
- Showing it to your client is useful as a sanity check to see that you’re about to produce something that the client currently sees as being potentially useful.

## Let's Practice

1\. What question is usually more complex?
2\. What model needs to be made for all problems?
3\. In supervised learning, once we have our business objective, part of our statistical question is identifying what?

**True or False:**

4\. When writing your reports, it's important to consider who is reading it.
5\. Sometimes you may need to dig a little to figure out exactly what the client wants.
6\. In supervised learning, we should take into consideration the uncertainty of our models.

```{admonition} Solutions!
:class: dropdown

1. Business question/objective
2. Baseline - Dummy
3. Our target variable
4. True
5. True
6. True

```

## Feature Selection

### Motivation

Remember the curse of dimensionality?

We spoke about this briefly when we discussed $k$-nn and how when we add many different dimensions (features) it can confuse the model with any irrelevant features and the model can disintegrate into predictions no better than random guessing.

Reasons like this are why we need to be careful about which features we include.

**Feature selection** can be described as *finding the features (columns) $X$ that are important for predicting $y$ and removing the features that aren’t.*

Feature selection can be aided using domain knowledge and manually however we can also use tools that help us either tell us:

- which features are most important to a model
- or that can select a number of features that will result with the optimal validation score.

cities_df = pd.read_csv("data/canada_usa_cities.csv")
train_df, test_df = train_test_split(cities_df, test_size=0.2, random_state=123)
X_train, y_train = train_df.drop(columns=["country"], axis=1), train_df["country"]
X_test, y_test = test_df.drop(columns=["country"], axis=1), test_df["country"]

train_df.head()

### Feature importance

Remember our Decision Tree models? Well, we can find out which features are most important in a model using an attribute called `feature_importances_`.

dt_modelku_model = DecisionTreeClassifier(max_depth=5)
dt_model.fit(X_train, y_train)
dt_model.feature_importances_

Here we can see that most of the importance is on the column `latitude`.

X_train.columns

If we graph this, the root of the decision tree will usually reflect the top feature. Just like we can see here:

sys.path.append('code/')
from display_tree import display_tree
display_tree(X_train.columns, dt_model, "imgs/decision_tree")

### New housing data

I know at this point you are probably annoyed and bored of housing data, but good, interesting open-source data is hard to come by. For this example, I really want to show you an example with LOTS of features.

Here is (yet another) housing dataset we acquired from [this GitHub repo](https://github.com/melindaleung/Ames-Iowa-Housing-Dataset), originally created by Dean De Cock.
*(We are using the raw data so we do not need to store it and import it simply from the url.)*

**Attribution:**

The Ames Housing dataset was compiled by Dean De Cock for use in data science education.

His publication can be found [here](http://jse.amstat.org/v19n3/decock.pdf).

ames_df = pd.read_csv('https://raw.githubusercontent.com/melindaleung/Ames-Iowa-Housing-Dataset/master/data/ames%20iowa%20housing.csv', index_col=0)
ames_df.loc[ames_df['SaleCondition'] != 'Normal', 'SaleCondition'] = 'Abnormal'
ames_df

ames_df.info()

Here, we use `.info()` to identify our categorical and numeric features. I split my features and identify my target.

The target variable for this question is `SaleCondition` and we are doing classification.

train_df, test_df = train_test_split(ames_df, test_size=0.2, random_state=77)

X_train = train_df.drop(columns=['SaleCondition', 'PoolQC', 'MiscFeature', 'Alley'])
X_test =  test_df.drop(columns=['SaleCondition', 'PoolQC', 'MiscFeature', 'Alley'])

y_train = train_df['SaleCondition']
y_test = test_df['SaleCondition']

 Note, you should be looking at these individually but I'm being a little lazy here.

numeric_features = X_train.select_dtypes('number').columns.to_list()
categorical_features = X_train.select_dtypes('object').columns.to_list()

Here are our numeric features:

numeric_features

And here are our categorical features:

categorical_features

Next, we need to make our pipelines and column transformer.

We can also cross-validate, but here my main goal is to show you how to get our feature importances from our pipeline!

numeric_pipe = make_pipeline(SimpleImputer(strategy='median'),
                            StandardScaler())
categoric_pipe = make_pipeline(SimpleImputer(strategy="constant", fill_value="missing"),
                            OneHotEncoder(dtype=int, handle_unknown="ignore"))

preprocessor = make_column_transformer((numeric_pipe, numeric_features),
                                       (categoric_pipe, categorical_features))

main_pipe = make_pipeline(preprocessor, LogisticRegression(max_iter=1000))

scores = cross_validate(main_pipe, X_train, y_train, return_train_score=True)

pd.DataFrame(scores).mean()

Once we fit our pipeline outside of `cross_validate()` we can use `coef_` to get our features that contribute to our predictions.

main_pipe.fit(X_train, y_train)
feats_coef = main_pipe.named_steps['logisticregression'].coef_
feats_coef

The problem here, is we don't know which value corresponds to which feature!

Let's first take a look at how many features we have now after preprocessing.

feats_coef.shape

The 282 is refering to the number of features after preprocessing!

Let's get the feature names after preprocessing.

We can obtain the categorical features and combine them with the  numeric features.

cat_feats = preprocessor.named_transformers_['pipeline-2'].named_steps[
    'onehotencoder'].get_feature_names(categorical_features).tolist()

all_feat_names = numeric_features + cat_feats

We can see now that we have the same number of feature names as we do coefficients.

len(all_feat_names)

Let's get them into a dataframe now and sort them:

features_df = pd.DataFrame(data = [all_feat_names,
                                feats_coef.flatten()]).T.rename(columns={0:'feature', 1:'feature_coefs'})
features_df.sort_values('feature_coefs',key= abs, ascending=False)

We can see that `SaleType_New` is the most important feature in our model.

From here we can decide to manually remove some of the columns that are not taken into consideration as much.... or we can instead use a tool to help us!

Enter - **Recursive feature elimination**

### Recursive feature elimination - RFE

We can use feature importances to eliminate unimportant features.

The basic idea with recursive feature elimination is we:
1. We decide $k$ - the number of features to select.
2. Assign importances to features, e.g. by fitting a model and looking at coef_ or feature_importances_.
3. Remove the least important feature.
4. Repeat steps 2-3 until only $k$ features are remaining.

**Note that this is not the same as just removing all the less important features in one shot!**

Let's take a look at how we can do this.

First we import `RFE` from `sklearn.feature_selection`:

from sklearn.feature_selection import RFE

Now instead of simply using `LogisticRegression`, we can wrap it around the `RFE` function and specify how many features we want with `n_features_to_select`.

Here I'm capping the number of features to 30 (an arbitrary number I picked).

This is going to take about 1-2 minutes to run because now, it's recursively removing 1 feature at a time and cross-validating on the final result.

main_pipe = make_pipeline(preprocessor, RFE(LogisticRegression(max_iter=1000),
                                            n_features_to_select=30))

scores = cross_validate(main_pipe, X_train, y_train, return_train_score=True)

pd.DataFrame(scores)

pd.DataFrame(scores).mean()

Looking at this mean validation score compared to when the model was using all the features, we can see it increased a tiny bit!

But now our next question is how do we set $k$? How do we know how many features is the optimal amount... Well, you guessed it! There is a tool for that too!

### RFECV

You can find the optimal number of features using cross-validation with `RFECV` where the optimal $k$ value is selected based on the highest validation score.

You would definitely not want to use the training score! - Why?
 > Because with training score the more features you add the higher the score, this isn't the case with validation score.

We can import `RFECV` from `sklearn.feature_selection` like we did for `RFE`.

from sklearn.feature_selection import RFECV

Instead of `RFE` now we simply use `RFECV` in our pipeline and we do not need to specify the argument `n_features_to_select` like we did with `RFE` since $k$ is selected based on the highest validation score.

(*This is also going to take a couple of minutes*)

main_pipe = make_pipeline(preprocessor, RFECV(LogisticRegression(max_iter=1000), cv=5))

scores = cross_validate(main_pipe, X_train, y_train, return_train_score=True)

Now we have ~91% for our validation score!

pd.DataFrame(scores).mean()

main_pipe.fit(X_train,y_train)
print(main_pipe.named_steps["rfecv"].support_)

print("The number of features selected by RFE: ",
      main_pipe.named_steps["rfecv"].n_features_)

feature_names = all_feat_names
support = main_pipe.named_steps["rfecv"].support_
RFE_selected_feats = np.array(feature_names)[support]
RFE_selected_feats

RFECV selects the features by references their `feature_importances`/`coefs_` as well as the validation score after each feature is removed and seeing if it is increasing.

When a feature is removed and the validation score is no longer increasing, then it stops removing features.

## Forward Selection

Unlike with RFE where we start with all our features and gradually remove the leat important ones, **Forward Selection** is a process where we start with no features and gradually add them!

With RFE we removed the least important feature, whereas with forward selection we add features untill our cross-validation score starts to decreases.

Forward Selection does not guaranty finding the best features set but reduces many problems.
Computationally cheaper (aka faster!)
Overfits less

Forward selection is recently implemented in `sklearn` so please make sure it is up to date! You need version 0.24!

from sklearn.feature_selection import SequentialFeatureSelector

We can import it as follows:

pipe_forward = make_pipeline(preprocessor,
                             SequentialFeatureSelector(LogisticRegression(max_iter=1000),
                                                       direction='forward',
                                                       n_features_to_select=20),
                            LogisticRegression(max_iter=1000))

Running this next cell is going to take a LONG LONG LONG time.

scores = cross_validate(pipe_forward, X_train, y_train,
                        return_train_score=True)
pd.DataFrame(scores).mean()

## Let's Practice

1\. As we increase features, which score will always increase?
2\. Between `RFE` and `RFECV` which one finds the optimal number of features for us?
3\. Which method starts with all our features and iteratively removes them from our model?
4\. Which method starts with no features and iteratively adds features?
5\. Which method does not take into consideration `feature_importances_`/`coefs_` when adding/removing features?

```{admonition} Solutions!
:class: dropdown

1. Training score
2. `RFECV`
3. Recursive Feature Elimination
4. Forward Selection
5. Forward Selection
```

## Extra time? Project time

First -> Project expectations.

Breakout rooms in your project groups!

Use this time to:

- Meet with your team mates;
- Think about a project - choose the data and business objective.
- Propose a statistical objective to address this.
- Also, elaborate on the statistical objective. What’s your plan for the analysis?

## What We've Learned Today

- How to construct a statistical question from a business objective.
- What steps are important in building your analysis.
- How to discover important features in your model.
- the 2 different methods (RFE, Forward selection) to conduct feature selection on your model.


"""
handling numpy data for json

author: anton feldmann <anton.feldmann@gmail.com>
"""
from json.encoder import JSONEncoder
from json.decoder import JSONDecoder

import numpy as np

# pylint: disable=C0103,R0201,R1705,R0911

class NumpyEncoder(JSONEncoder):
    """ Custom encoder for numpy data types
        The NumpyEncoder is a JSONEncoder

    Examples:
    ..  example_code::
        >>> import numpy as np
        >>> import json
        >>> from apu.encoding.json.np import NumpyEncoder
        >>> arr = array([   0,  239,  479,  717,  952, 1192, 1432, 1667],
        ...      dtype=int64)
        >>> json.dumps(arr,cls=NumpyEncoder)
    """
    def np_list(self, obj):
        """ numpy array object to json list """
        return obj.tolist()

    def np_float(self, obj):
        """ numpy float type to float """
        return float(obj)

    def np_int(self, obj):
        """ numpy int to int """
        return int(obj)

    def np_complex(self, obj):
        """ numpy complex to dict.
        because the decoder has to decode
        complex i use a dict """
        return {"real": obj.real, "imag": obj.imag}

    def np_bool(self, obj):
        """ numpy boolean to boolean """
        return bool(obj)

    def np_null(self, obj):
        """ numpy null no None or json null """
        return None

    def np(self, obj):
        """ np function to check for all numpy objects """
        if isinstance(obj, np.integer):
            return self.np_int(obj)

        elif isinstance(obj, np.floating):
            return self.np_float(obj)

        elif isinstance(obj, (np.complex_, np.complex64, np.complex128)):
            return self.np_complex(obj)

        elif isinstance(obj, (np.ndarray, )):
            return self.np_list(obj)

        elif isinstance(obj, (np.bool_)):
            return self.np_bool(obj)

        elif isinstance(obj, (np.void)):
            return self.np_null(obj)

        return JSONEncoder.default(self, obj)

    # pylint: disable=E0202,W0237
    def default(self, obj):
        """ default Encoder entrypoint to encode """
        return self.np(obj)

class NumpyDecoder(JSONDecoder):
    """ Custom decode for numpy data types

    Examples:
    ..  example_code::
        >>> import numpy as np
        >>> import json
        >>> from apu.encoding.json.np import NumpyDecoder
        >>> arr = '[[2.468031, 0.0, 0.0],
                    [-1.234015, 2.137377, 0.0],
                    [0.0, 0.0, 19.998293]]'
        >>> json.loads(arr,cls=NumpyDecoder)
        [[ 2.468031  0.        0.      ]
        [-1.234015  2.137377  0.      ]
        [ 0.        0.       19.998293]]
    """

    _recursable_types = [str, list, dict]

    def _is_recursive(self, obj) -> bool:
        """ check if the onject is recursiveable

        Returns:
            (bool): the object is recursiveable
        """
        return isinstance(obj, tuple(NumpyDecoder._recursable_types))

    # pylint: disable=R1710, R0912
    def decode(self, obj, *args, **kwargs):
        """ decode the json string """
        if not kwargs.get('recurse', False):
            obj = super().decode(obj, *args, **kwargs)

        if isinstance(obj, list):
            try:
                return np.array(obj)
            except:  # pylint: disable=W0702
                for item in obj:
                    if self._is_recursive(item):
                        obj[item] = self.decode(item, recurse=True)

        elif isinstance(obj, dict):
            for key, value in obj.items():
                if str(key) in "real":
                    return np.complex(obj['real'], obj['imag'])
                elif self._is_recursive(value):
                    obj[key] = self.decode(value, recurse=True)

        elif isinstance(obj, bool):
            return np.bool(obj)

        elif isinstance(obj, float):
            return np.float(obj)

        elif obj is None:
            return np.void

        else:
            return obj


# coding: utf-8

# # IMDB Data EDA
#
# ## Data
#
# Two datasets:
# - tmdb_5000_movies.csv
# - movie_metadata.csv
#
#
# ## Goal
#
# What makes a profitable movie?
#
#
# ## Index
#
# 1- Loading and merging datasets
#
# 2- What makes gross revenue increase? How are these variables related to each other?
#
# 3- What actors and directors have the highest budget to gross ratio?
#
# 4- What movies where the biggest flops? Are there factors that are associated with flops?
#
# 5- What countries make the most profitable and valued films?
#

# ### 1- Loading, cleaning and merging datasets
#
# We'll need to load the datasets, analyze their content and quality, clean de unnecessary columns, create categorical features where needed and finally merge everything into a common and usable Pandas Dataframe.

# Import the needed libraries for the EDA

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
import re

warnings.filterwarnings('ignore')

#Import the three downloaded datasets into DFs and review the first 10 entries

df_movies = pd.read_csv('Datasets/tmdb_5000_movies.csv')
df_metadata = pd.read_csv('Datasets/movie_metadata.csv')

print("Movies DF:\n\n{}\n".format(df_movies.head()))
print("Metadata DF:\n\n{}".format(df_metadata.head()))

# Let's see what kind of data are we working with

print("Movies DF:\n")
print(df_movies.info(),"\n\n")
print(df_movies.describe(),"\n\n")
print(df_movies['original_language'].value_counts())
print("\n{}".format(df_movies['status'].value_counts()))
print("\n\nNulls:\n\n{}\n\n**************************************\n\n".format(df_movies.isnull().sum()))

print("Metadata DF:\n")
print(df_metadata.info(),"\n\n")
print(df_metadata.describe(),"\n\n")
print(df_metadata['color'].value_counts())
print("\n{}".format(df_metadata['content_rating'].value_counts()))
print("\n{}".format(df_metadata['aspect_ratio'].value_counts()))
print("\n\nNulls:\n\n{}\n\n**************************************\n\n".format(df_metadata.isnull().sum()))

# We can see from the list of columns that some of them are repeated, let's get rid of them

df_metadata.drop(['num_critic_for_reviews','duration','gross','genres',
                 'num_voted_users','plot_keywords','num_user_for_reviews',
                 'budget','title_year','imdb_score'],axis=1,inplace=True)

#We can also get rid of the text description of the films, as it won't be necessary for our study
df_movies.drop('overview',axis=1,inplace=True)

#And we can drop the production country and language columns and keep the country and language from metadata
df_movies.drop(['production_countries','spoken_languages'],axis=1,inplace=True)

# We should change categorical data and dictionaries into columns for easier manipulation and visualization

#First let's extract the important info out the string columns that contain of lists of dictionaries

patternname = r"(?:.name.: .)(\w{1,}\s{0,}\w{0,})"
patternlang = r"(?:.iso_639_1.: .)(\w{1,}\s{0,}\w{0,})"

def dict2list(x):
    if type(x) is str:
        templist = x.strip('[]').split(',')
        namelist = []
        lang = False
        for element in templist:
            if re.search(patternlang, element):
                namelist.append(re.search(patternlang, element).group(0)[14:])
                lang = True
            elif re.search(patternname, element) and not lang:
                namelist.append(re.search(patternname, element).group(0)[9:])
        if len(namelist) > 0:
            return {k: 1 for k in namelist}
        else:
            return x
    else:
        return x

df_movies = df_movies.applymap(dict2list)

#Second, lets use the newly created dictionaries to create dummy columns for the categorical columns

def dicttodummy(df,columns):
    columnnames = {}
    for col in columns:
        columnnames[col] = list(df[col].apply(pd.Series).drop([0], axis=1))
        df = pd.concat([df.drop([col], axis=1), df[col].apply(pd.Series).fillna(0).drop([0], axis=1)], axis=1)
    return df,columnnames

dummyfeatures = ['genres']
df_movies,columndictionary = dicttodummy(df_movies,dummyfeatures)

#Now we can merge both the dataframes into one

df_metadata['movie_title'] = df_metadata['movie_title'].map(lambda x: x.strip())
data = pd.merge(df_movies, df_metadata, left_on='original_title'
                , right_on='movie_title', how='left').drop(['movie_title'], axis=1)

print(data.head())

# ### 2- What makes gross revenue increase? How are these variables related to each other?
#
# We need to evaluate genre, content rating (R, PG13, etc.), budget, movie FB likes, IMDB likes, etc. We should also evaluate how the important of those features changed with time.
#
# #### - First of all, we should understand how is the revenue variable. An histogram should be useful.

def plotHistogram(df,col,tit='',xlabel='',ylabel='',log=False):
    bins = np.linspace(df[col].min(),df[col].max(),25)
    plt.xlim([df[col].min(),df[col].max()])
    plt.hist(df[col], bins=bins, alpha=0.5,log=log)
    plt.title(tit)
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.show()

plotHistogram(data,'revenue',ylabel='# of films',xlabel='Revenue',tit='Revenue logarithmic histogram',log=True)

# The log plot tells us that the distribution is close to an exponential

# #### - Let's see how revenue correlates with some other film figures

#Array with the column names for what we want to compare the revenue to
revenue_comparisons = ['budget', 'runtime', 'vote_average', 'popularity','facenumber_in_poster','movie_facebook_likes']

#Iterate through each of the columns made above and plots them separately
for comparison in revenue_comparisons:
    sns.jointplot(x='revenue', y=comparison, data=data, color='navy', size=10, space=0, kind='reg')

# We can see how the strongest relationship to revenue is budget (0.73) and the weakest, obviously, the number of faces in the promotional poster, with a p that would not allow to reject the null hypothesis if we set a confidence over 99%. The correlation of popularity and FB movie likes is also evident, but surprisingly there's also a positive correlation between runtime and revenue (0.25), so this would lead us to advise, if anything, against a short film.

# #### - Let's see now the correlation between all these variables among themselves.

fig, ax = plt.subplots(figsize=(14,12));
new = data[['revenue','budget', 'runtime', 'vote_average', 'popularity','facenumber_in_poster','movie_facebook_likes']].copy()
new = new.corr()
sns.heatmap(new, ax=ax);
ax.set_title('Correlation matrix heatmap');

# #### - Is the type of film content rating important for its profitability?
#
# To know that, ee need to create a new column, revenue_budget_ratio.

##We'll filter by a budget of $100k to avoid independent films
minbudget = 100000
data['revenue_budget_ratio'] = data['revenue'][data['budget']>minbudget]/data['budget'][data['budget']>minbudget]

#We will filter to show only ratings with more than 100 films in the DB
ratinglist = df_metadata.content_rating.value_counts()[df_metadata.content_rating.value_counts() >= 100]

fig, ax = plt.subplots(figsize=(14,12));
ax = sns.violinplot(x='content_rating', y="revenue_budget_ratio"
                    ,data=data[data.content_rating.isin(ratinglist.index.values[:])][data.revenue_budget_ratio<20]
                    , palette="muted", split=True)
ax.set_ylim = (100)

print(data[['revenue_budget_ratio','content_rating']][data.content_rating.isin(ratinglist.index.values[:])]
      .groupby('content_rating').mean())

# We can see how the content rating is in fact affecting the ratio distribution shape, being the G rated films the ones with a higher profitability ratio and the PG-13 those who have a lower one.

# #### - Is budget a factor in profitability of films?

sns.jointplot(x='budget', y='revenue_budget_ratio', data=data, kind='reg', color='navy'
              , size=10, space=0,marginal_kws={'hist_kws': {'log': True}})

# We can see how, although budget strongly correlates with revenue, it does negatively affect the films potential for profitabilty.

# #### - How does revenue and profitability behaves for the main film genres?
#
# We will filter for the 7 main genres by film count.

main_genres = df_movies[columndictionary.get('genres')].sum().sort_values(ascending=False).head(7).index.values[:]

fig, ax = plt.subplots(figsize=(14,12));
new = data[['revenue',main_genres[0],main_genres[1],main_genres[2]
            ,main_genres[3],main_genres[4],main_genres[5],main_genres[6]]].copy()
new = new.corr()
sns.heatmap(new, ax=ax);
ax.set_title('Correlation between fim genre and film revenues');

# It seems that Thrillers and Action films correlates with a higher revenue.
#
# #### - But what about film profitability? Does it favor any specific genre?

fig, ax = plt.subplots(figsize=(14,12));
new = data[['revenue_budget_ratio',main_genres[0],main_genres[1],main_genres[2]
            ,main_genres[3],main_genres[4],main_genres[5],main_genres[6]]].copy()
new = new.corr()
sns.heatmap(new, ax=ax);
ax.set_title('Correlation between fim genre and film profitability');

# It seems that there is no genre in particular that correlates with a higher or lower profitability.

# ### 3- What actors and directors have the highest revenue to budget ratio?
#
# Which directors and actors have the highest average IMBD rating? Which is more loved in facebook? Which ones get the most profitable films? Do any of these have any correlation?

def showbarplot(df,col,tit='',col2=None,nelements=5,tail=False,ylim=None):
    plt.rcParams["figure.figsize"] = [nelements*2,6]
    fig, ax = plt.subplots()
    width = 0.7
    ind = np.arange(nelements)

    if tail:
        bars = df[col].sort_values(ascending=False).tail(nelements)
        plt.xticks((ind), df[col].sort_values(ascending=False).tail(nelements).index.tolist(), rotation=45)
    else:
        bars = df[col].sort_values(ascending=False).head(nelements)
        plt.xticks((ind), df[col].sort_values(ascending=False).head(nelements).index.tolist(), rotation=45)

    rects = ax.bar(range(len(bars)), bars, width = width, color='navy')
    ax.set_title(tit)

    if col2:
        if tail:
            labels = df[[col,col2]].sort_values(by=col,ascending=False).tail(nelements)[col2]
        else:
            labels = df[[col,col2]].sort_values(by=col,ascending=False).head(nelements)[col2]

        plt.legend(['n='+str(col2)], loc=1)
        for i,rect in enumerate(rects):
            height = rect.get_height()
            ax.text(rect.get_x() + rect.get_width()/2., height*1.05,'n='+str(int(labels[i]))
                    , ha='center', va='bottom')

    if ylim:
        ax.set_ylim(ylim[0], ylim[1])

    plt.show()

df_directors = data.groupby('director_name').apply(lambda x:                                                    pd.Series({'Director_likes':x.director_facebook_likes.mean(),
                                                            'Movie_likes_mean':x.movie_facebook_likes.mean(),
                                                            'revenue_budget_ratio':x.revenue_budget_ratio.mean(),
                                                            'budget_mean':x.budget.mean(),
                                                            'revenue_mean':x.revenue.mean(),
                                                            'runtime':x.runtime.mean(),
                                                            'vote_average':x.vote_average.mean(),
                                                            'films':x.director_name.count()}))

# #### - Let's see the directors with a highest average revenue figures

#As we will be considering mean revenues, we'll filter by directors with at least 3 films in the DB

showbarplot(df_directors[df_directors.films>2],'revenue_mean'            ,tit='Directors by average revenue figures',nelements=7)

print("Directors by average revenue figures:\n")
print(df_directors[['revenue_mean','films','budget_mean','vote_average']][df_directors.films>2]      .sort_values(by="revenue_mean",ascending=False).head())

# But that didn't show the profitability of the directors...
#
# #### - Let's see now the directors with at least 3 films with the highest profitable ratios

showbarplot(df_directors[df_directors.films>2],'revenue_budget_ratio',col2='films'            ,tit='Directors by average profitability, with at least 3 films',nelements=7)

print("\n\nDirectors by average profitability, with at least 3 films:\n")
print(df_directors[['films','revenue_budget_ratio','budget_mean','vote_average']]      [df_directors.films>2].sort_values(by="revenue_budget_ratio",ascending=False).head())

# We can see how George Romero, John Carpenter  and GeorgeMiller have the biggest revenue to budget ratio, having all of them more than 5 films in their IMDB database profiles.
#
# #### - Let's see now other stats about directors, like FB fans

showbarplot(df_directors[df_directors.revenue_budget_ratio>0],'Director_likes'            ,tit='Most liked directors',nelements=7)

#We'll filter directors with less than 3 films in the df for the movie means average

showbarplot(df_directors[df_directors.films>2],'Movie_likes_mean',col2='films'            ,tit='Directors with at least 3 movies whose movies are most liked',nelements=7)

# It seems Christopher Nolan has a touch with the audience!
#
# #### - But does having more FB likes relates to a higher profitability for directors?

sns.jointplot(x='revenue_budget_ratio', y='director_facebook_likes', data=data, color='navy', kind='reg'
              , size=10, space=0,marginal_kws={'hist_kws': {'log': True}})

# The number of FB likes of the director and film profitability (revenue/budget ratio) doesn't seem to be important.
#
# #### - What if we chose to compare against the revenue of their films?

sns.jointplot(x='revenue', y='director_facebook_likes', data=data, color='navy', kind='reg'
              , size=10, space=0,marginal_kws={'hist_kws': {'log': True}})

# We can see how the directors of films with great revenues tend to have more FB likes, although that doesn't seem to imply that the profitability of their films are higher.

# #### - Can we relate the FB likes of the cast to the profitability of the film? What about its revenue?

sns.jointplot(x='cast_total_facebook_likes', y='revenue_budget_ratio', data=data
              , kind='reg', color='navy', size=10, space=0,marginal_kws={'hist_kws': {'log': True}})

sns.jointplot(x='cast_total_facebook_likes', y='revenue', data=data
              , kind='reg', color='navy', size=10, space=0,marginal_kws={'hist_kws': {'log': True}})

# As it happened with the directors, the FB likes of the main cast doesn't positively correlate with the profitability of the film, but it clearly does with its revenue. In this case, the profitability of the film is negatively correlated with the number of FB likes of the cast, with a p of 0.046, so its negative correlation would be accepted with a confidence of 95%, and the positive correlation of FB likes and total film revenue is also clearer in this case, with a r of 0.24.
#
# Let's create a specific dataframe for the 3 main actors in each film, to evaluate them separately.

actor1 = data[['actor_1_name','actor_1_facebook_likes','revenue','revenue_budget_ratio','vote_average']]    .rename(index=str, columns={"actor_1_name": "actor", "actor_1_facebook_likes": "actor_FB_likes"})
actor2 = data[['actor_2_name','actor_2_facebook_likes','revenue','revenue_budget_ratio','vote_average']]    .rename(index=str, columns={"actor_2_name": "actor", "actor_2_facebook_likes": "actor_FB_likes"})
actor3 = data[['actor_3_name','actor_3_facebook_likes','revenue','revenue_budget_ratio','vote_average']]    .rename(index=str, columns={"actor_3_name": "actor", "actor_3_facebook_likes": "actor_FB_likes"})

actors = pd.concat([actor1, actor2, actor3]).reset_index(drop=True)

df_actors = actors.groupby('actor').apply(lambda x: pd.Series({'Actor_likes':x.actor_FB_likes.mean(),
                                                               'revenue_budget_ratio':x.revenue_budget_ratio.mean(),
                                                               'revenue_mean':x.revenue.mean(),
                                                               'vote_average':x.vote_average.mean(),
                                                               'films':x.actor.count()}))

# #### - Which are the actors with higher revenue figures and which are the most profitable for their movies?

showbarplot(df_actors[df_actors.films>2],'revenue_mean',tit='Actors by average revenue figures',nelements=7)

print("Actors by average revenue figures:\n")
print(df_actors[['revenue_mean','films','vote_average']][df_actors.films>2]      .sort_values(by="revenue_mean",ascending=False).head())

showbarplot(df_actors[df_actors.films>2],'revenue_budget_ratio'
            ,tit='Actors by average profitability of their films',nelements=7)

print("Actors by average profitability of their films:\n")
print(df_actors[['revenue_budget_ratio','films','vote_average']][df_actors.films>2]      .sort_values(by="revenue_budget_ratio",ascending=False).head())

# ### 4- What movies where the biggest flops? Are there factors that are associated with flops?
#
# What are the number of films that have grossed less than their budgets by year? Is that changing through time?

# #### - It would be interesting to visit first the evolution of films profitability throughout the years

data['release_date'] = pd.to_datetime(data['release_date'])

df_years = data.groupby(data.release_date.dt.year).apply(lambda x: pd.Series({'revenue_ratio_median':x.revenue_budget_ratio.median(),
                    'ratio_top25':x.revenue_budget_ratio.quantile(.75),
                    'ratio_bottom25':x.revenue_budget_ratio.quantile(.25),
                    'flops_ratio':(x[x.revenue_budget_ratio<1].release_date.count())/x.release_date.count(),
                    'vote_average':x.vote_average.mean(),
                    'films':x.release_date.count()}))

# We'll filter for those years with at least 30 films on the database

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(16, 7))

x=df_years.index[df_years.films>29]

axes[0].plot(x, 'revenue_ratio_median', data=df_years[df_years.films>29], marker='o'
             , markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4,label="Median revenue/budget")
axes[1].plot(x, 'flops_ratio', data=df_years[df_years.films>29], marker='', color='olive'
             , linewidth=2)
axes[0].plot(x, 'ratio_top25', data=df_years[df_years.films>29], marker='', color='royalblue'
             , linewidth=2, linestyle='dashed', label="Top 25% profit.")
axes[0].plot(x, 'ratio_bottom25', data=df_years[df_years.films>29], marker='', color='indianred'
             , linewidth=2, linestyle='dashed', label="Bottom 25% profit.")
axes[0].legend()
axes[0].set_title('Profitability evolution over the years')
axes[1].set_title('Evolution of the ratio of films that lost money')

fit = np.polyfit(x, df_years.flops_ratio[df_years.films>29], 1)
axes[1].plot(x, fit[0] * x + fit[1], color='red')

fit = np.polyfit(x, df_years.revenue_ratio_median[df_years.films>29], 1)
axes[0].plot(x, fit[0] * x + fit[1], color='red')

# We can see that the profitability of films has not been getting better through the years. We could even argue that the trend has a downward slope, and that the ratio of films that don't even make it to breaking even is, if anything, increasing.

# As it happened with the directors, the FB likes of the main cast doesn't improve the profitability of the film. If anything, it seems to be a negative correlation, probably because films with lower budgets and less known actors have a much greater potential for big profitability than big blockbusters.

# #### We're going to review now what films in our database are the biggest flops. To do so, we will:
#
# - Filter first by films with budget lower than revenue
# - Filter by those films with a IMDB score under 5 and at least 100 votes
# - Finnaly, sort those films by the amount of money they lost

df_flops = data[data.revenue_budget_ratio<0.5][data.revenue_budget_ratio>0]
df_flops = df_flops[df_flops.vote_average<5][df_flops.vote_count>100]
df_flops['money_lost'] = (1-df_flops['revenue_budget_ratio'])*df_flops['budget']
df_flops.sort_values(by='money_lost',ascending=False,inplace = True)

#print(df_flops[['original_title','money_lost','vote_average']])

fig, ax = plt.subplots(figsize=(14,12));
plt.style.use('ggplot')
explode=[0.1,0.1,0.1,0.1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
df_flops['money_lost'].plot(kind='pie',explode=explode, labels=df_flops.original_title, fontsize=10,shadow=True)
plt.show()

df_flops.set_index('original_title',inplace=True)

print(df_flops[['money_lost','vote_average','revenue_budget_ratio']])

# ### 5- What countries make the most profitable and valued films?
#
# As a last question, to gather the different aspects of profitable movies, we can inspect the relationship of movie characteristics between different countries

df_countries = data.groupby('country').apply(lambda x: pd.Series                                             ({'Movie_likes_mean':x.movie_facebook_likes.mean(),
                                                'revenue_budget_ratio':x.revenue_budget_ratio.mean(),
                                                'budget_mean':x.budget.mean(),
                                                'revenue_mean':x.revenue.mean(),
                                                'runtime':x.runtime.mean(),
                                                'vote_average':x.vote_average.mean(),
                                                'films':x.country.count()}))

# #### - Which are the countries with a higher revenue average?

#We'll filter those countries with less than 3 films in the df

showbarplot(df_countries[df_countries.films>2],'revenue_mean',col2='films'            ,tit='Countries by average film revenue',nelements=10)

# #### - Which country makes the most profitable movies?

showbarplot(df_countries,'revenue_budget_ratio',col2='films',tit='Countries by average profitability',nelements=10)

# It's important to note that these two previous lists are probably biased, as films from most countries don't enter IMDB unless they had enough budget or social impact, so the average revenue and revenue to budget ratios will be higher for films from those countries.

# #### - Which country makes the most acclaimed films?

#Filtering again by countries with more than 2 films

showbarplot(df_countries[df_countries.films>2],'vote_average',tit='Countries with top film reviews'
            ,nelements=7, ylim=[4.5,7.5])

print("\n",df_countries[['vote_average','films','budget_mean','revenue_budget_ratio']][df_countries.films>2]      .sort_values(by="vote_average",ascending=False).head())

# #### - And which countries make the least acclaimed?

#Whith the 3 films or more filter

showbarplot(df_countries[df_countries.films>2],'vote_average',tit='Countries with bottom film reviews'
            , ylim=[4.5,7.5], nelements=7,tail=True)

print("\n",df_countries[['vote_average','films','budget_mean','revenue_budget_ratio']][df_countries.films>2]      .sort_values(by="vote_average",ascending=False).tail())

# #### - Which country makes the longest and shortest flims on average?

showbarplot(df_countries[df_countries.films>2],'runtime'
            ,tit='Countries with at least 3 films by film duration (Top 5)',nelements=5, ylim=[80,140])
showbarplot(df_countries[df_countries.films>2],'runtime'
            ,tit='Countries with at least 3 films by film duration (Bottom 5)',nelements=5,tail=True, ylim=[80,140])

# It's courious to see how Ireland, with the highest revenue to budget ratio, makes the shortest films of all countries.


# This installation script implemented based on the notebook from Jeff Kantor
# see that notebook for more information, examples, and testing
# https://colab.research.google.com/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb#scrollTo=RlmSEN45glrk
# execute this script in your google colab with the following lines
"""
!wget https://raw.githubusercontent.com/carldlaird/colab-utilities/main/utils/06462/install_packages_06462.py
import install_packages_06462
"""

import os
os.system('apt-get install -y -qq glpk-utils')
os.system('apt-get install -y -qq coinor-cbc')
os.system('wget -N -q "https://ampl.com/dl/open/ipopt/ipopt-linux64.zip"')
os.system('unzip -o -q ipopt-linux64')
os.system('pip install pyomo')
import pyomo.common.fileutils as fileutils
fileutils.Executable.rehash()
fileutils.Executable('ipopt').set_path('/content/ipopt')


"""..
    Copyright (c) 2014-2017, Magni developers.
    All rights reserved.
    See LICENSE.rst for further information.

Module for wrapping the Magni IPython Notebook examples.

**This module is based on the "ipnbdoctest.py" script by Benjamin
Ragan-Kelley (MinRK)**, source: https://gist.github.com/minrk/2620735.

This assumes comparison of IPython Notebooks in nbformat.v3

"""

from __future__ import division, print_function
import base64
from datetime import datetime
import os
import platform
import shutil
import subprocess
import sys
import unittest
import types
import warnings
try:
    from Queue import Empty  # Python 2
except ImportError:
    from queue import Empty  # Python 3
try:
    from StringIO import StringIO as BytesIO  # Python 2
except ImportError:
    from io import BytesIO  # Python 3

import numpy as np
from pkg_resources import parse_version
import scipy.misc

import magni

# The great "support IPython 2, 3, 4" strat begins
import IPython
try:
    import jupyter
except ImportError:
    jupyter_era = False
else:
    jupyter_era = True

if jupyter_era:
    # Jupyter / IPython 4.x
    from jupyter_client import KernelManager
    from nbformat import reads, NotebookNode

    def mod_reads(file_):
        return reads(file_, 3)  # Read notebooks as v3

else:
    from IPython.kernel import KernelManager
    with warnings.catch_warnings():
        warnings.simplefilter('error')
        try:
            # IPython 2.x
            from IPython.nbformat.current import reads, NotebookNode

            def mod_reads(file_):
                return reads(file_, 'json')

        except UserWarning:
            # IPython 3.x
            from IPython.nbformat import reads, NotebookNode

            def mod_reads(file_):
                return reads(file_, 3)  # Read notebooks as v3

# End of the great "support IPython 2, 3, 4" strat

# Test for freetype library version
try:
    if parse_version(
            subprocess.check_output(
                ['freetype-config', '--ftversion']).decode().strip()
            ) <= parse_version('2.5.2'):
        _skip_display_data_tests = False
    else:
        _skip_display_data_tests = True
except OSError:
    _skip_display_data_tests = True

if _skip_display_data_tests:
    warnings.warn('Skipping display data ipynb tests.', RuntimeWarning)

class _Meta(type):
    """
    Identification of IPython Notebook examples and construction of test class.

    """

    def __new__(class_, name, bases, attrs):
        path = magni.__path__[0].rsplit(os.sep, 1)[0]
        path = path + os.path.sep + 'examples' + os.path.sep

        for filename in os.listdir(path):
            if (platform.system() == 'Darwin' and
                    sys.version_info.major == 3 and
                    sys.version_info.minor == 3 and
                    filename == 'utils-multiprocessing.ipynb'):
                # Skip the broken multiprocessing on OSX Python 3.3 since case
                warnings.warn(
                    'Skipping multiprocessing ipynb test.', RuntimeWarning)
                continue
            if filename[-6:] == '.ipynb':
                name = 'test_' + filename[:-6].replace('-', '_')
                func = attrs['_run_example']
                func = types.FunctionType(func.__code__, func.__globals__,
                                          name, (path + filename,))
                func.__doc__ = func.__doc__.format(filename)
                attrs[name] = func

        return type.__new__(class_, name, bases, attrs)

# For python 2 and 3 compatibility
class _Hack(_Meta):
    def __new__(class_, name, bases, attrs):
        return _Meta(name, (unittest.TestCase,), attrs)

_TestCase = type.__new__(_Hack, 'temp', (), {})

@unittest.skipIf(
    parse_version(IPython.__version__) <= parse_version('3.0') and
    sys.version_info.major == 2 and
    platform.system() == 'Darwin', 'Due to a problem in IPython, the Magni ' +
    'Notebook example tests stall on Mac OSX with IPython 2 and Python 2.')
class TestIPythonExamples(_TestCase):
    """
    Test of Ipython Notebook examples for equality of output to reference.

    """

    def setUp(self):
        """
        Identify IPython Notebook examples to run.

        """

        path = magni.__path__[0].rsplit(os.sep, 1)[0]
        path = path + os.path.sep + 'examples' + os.path.sep
        files_to_copy = ['example.mi', 'data.hdf5', 'display.py']

        for cfile in files_to_copy:
            shutil.copy(os.path.join(path, cfile), '.')

    def _run_example(self, ipynb):
        """
        Test of {} Magni IPython Notebook example.

        """

        with open(ipynb) as f_ipynb:
            notebook = mod_reads(f_ipynb.read())

        notebook_result = _check_ipynb(notebook)
        passed, successes, failures, errors, report = notebook_result

        self.assertTrue(passed, msg=report)
        error_msg = ('Magni IPython Notebook example status:\n' +
                     'Successes: {}, Failures: {}, Errors: {}').format(
                         successes, failures, errors)
        self.assertEqual(errors + failures, 0, msg=error_msg)

def _check_ipynb(notebook):
    """
    Check an IPython Notebook for matching input and output.

    Each cell input in the `notebook` is executed and the result is compared
    to the cell output saved in the `notebook`.

    Parameters
    ----------
    notebook : IPython.nbformat.current.NotebookNode
        The notebook to check for matching input and output.

    Returns
    -------
    passed : Bool
        The indicator of a successful check (or not).
    sucessess : int
        The number of cell outputs that matched.
    failures : int
        The number of cell outputs that failed to match.
    errors : int
        The number of cell executions that resulted in errors.
    report : str
        The report detailing possible failures and errors.

    """

    kernel_manager = KernelManager()
    kernel_manager.start_kernel()
    kernel_client = kernel_manager.client()
    kernel_client.start_channels()

    try:
        # IPython 3.x
        kernel_client.wait_for_ready()
        iopub = kernel_client
        shell = kernel_client
    except AttributeError:
        # Ipython 2.x
        # Based on https://github.com/paulgb/runipy/pull/49/files
        iopub = kernel_client.iopub_channel
        shell = kernel_client.shell_channel
        shell.get_shell_msg = shell.get_msg
        iopub.get_iopub_msg = iopub.get_msg

    successes = 0
    failures = 0
    errors = 0

    report = ''
    for worksheet in notebook.worksheets:
        for cell in worksheet.cells:
            if cell.cell_type == 'code':
                try:
                    test_results = _execute_cell(cell, shell, iopub)
                except RuntimeError as e:
                    report += ('{!s} in cell number: {}'
                               .format(e, cell.prompt_number))
                    errors += 1
                    break

                identical_output = all(
                    [_compare_cell_output(test_result, reference)
                     for test_result, reference in
                     zip(test_results, cell.outputs)])

                if identical_output:
                    successes += 1
                else:
                    failures += 1

                    try:
                        str_test_results = [
                            '(for out {})\n'.format(k) + '\n'.join(
                                [' : '.join([str(key), str(val)])
                                 for key, val in t.items()
                                 if key not in ('metadata', 'png')]
                            ) for k, t in enumerate(test_results)]
                        str_cell_outputs = [
                            '(for out {})\n'.format(k) + '\n'.join(
                                [' : '.join([str(key), str(val)])
                                 for key, val in t.items()
                                 if key not in ('metadata', 'png')]
                            ) for k, t in enumerate(cell.outputs)]
                    except TypeError as e:
                        report += 'TypeError in ipynb_examples test\n\n'
                        for entry in cell.outputs:
                            if 'traceback' in entry.keys():
                                for item in entry['traceback']:
                                    report += str(item) + '\n'
                    else:
                        report += '\n' * 2 + '~' * 40
                        report += (
                            '\nFailure in {}:{}\nGot: {}\n\n\nExpected: {}'
                        ).format(notebook.metadata.name,
                                 cell.prompt_number,
                                 '\n'.join(str_test_results),
                                 '\n'.join(str_cell_outputs))

    kernel_client.stop_channels()
    kernel_manager.shutdown_kernel()

    passed = not (failures or errors)

    return passed, successes, failures, errors, report

def _compare_cell_output(test_result, reference):
    """
    Compare a cell test output to a reference output.

    Parameters
    ----------
    test_results : IPython.nbformat.current.NotebookNode
        The cell test result that must be compared to the reference.
    reference : IPython.nbformat.current.NotebookNode
        The reference cell output to compare to.

    Returns
    -------
    comparison_result : bool
        The indicator of equality between the test output and the reference.

    """

    skip_compare = ['traceback', 'latex', 'prompt_number']

    if _skip_display_data_tests:
        # Skip graphics comparison
        skip_compare.append('png')

    if test_result['output_type'] == 'display_data':
        # Prevent comparison of matplotlib figure instance memory addresses
        skip_compare.append('text')
        skip_compare.append('metadata')

    for key in reference:

        if key not in test_result:
            raise Exception(str(reference) + '!!!!!' + str(test_result))
            return False
        elif key not in skip_compare:
            if key == 'text':
                if test_result[key].strip() != reference[key].strip():
                    return False
            elif key == 'png':
                reference_img = reference[key]
                test_img = test_result[key]
                if not _compare_images(reference_img, test_img):
                    return False
            else:
                if test_result[key] != reference[key]:
                    return False

    return True

def _compare_images(reference_img, test_img):
    """
    Compare reference and test image to determine if they depict the same.

    Two images are considered to depict the same unless:

    - The image shapes differ
    - The number of differences in non-transparant pixel values which are not
      (likely to be) part of the image border exceeds 2.

    Parameters
    ----------
    reference_img : str
        The base64 encoded reference image.
    test_img : str
        The base64 encoded test image.

    Returns
    -------
    comparison_result : bool
        The idenfifier of a positive match, i.e. True if images are the same.

    """

    ref_png = base64.b64decode(reference_img)
    ref_ndarray = scipy.misc.imread(BytesIO(ref_png))
    cmp_png = base64.b64decode(test_img)
    cmp_ndarray = scipy.misc.imread(BytesIO(cmp_png))

    # check shape of images
    if cmp_ndarray.shape != ref_ndarray.shape:
        print('Image shapes differ')
        return False

    # mask of channels in pixels with different values
    diff = cmp_ndarray != ref_ndarray
    # mask of pixels with different values
    diff = np.any(diff, axis=2)
    # mask of non-transparent pixels with different values
    diff = diff * np.bool_(ref_ndarray[:, :, 3])

    # check if all non-transparent pixels match
    if diff.sum() == 0:
        # Accept difference in tranparent pixels
        return True

    # The rest is all about checking if (it is likely to be) only the image
    # border that has changed. The border may render differently across
    # matplotlib versions.

    # mask of black pixels
    mask = ((ref_ndarray[:, :, 0] == 0) *
            (ref_ndarray[:, :, 1] == 0) *
            (ref_ndarray[:, :, 2] == 0) *
            (ref_ndarray[:, :, 3] == 255))

    # lookup table of the top most connected black pixel of the
    # looked up pixel
    C_N = np.zeros(mask.shape, dtype=np.int16)

    for i in range(0, mask.shape[0] - 1):
        C_N[i + 1, :] = np.logical_not(mask[i, :]) * i + mask[i, :] * C_N[i, :]

    # lookup table of the right most connected black pixel of the
    # looked up pixel
    C_E = np.zeros(mask.shape, dtype=np.int16)

    for i in range(mask.shape[1] - 1, 0, -1):
        C_E[:, i - 1] = np.logical_not(mask[:, i]) * i + mask[:, i] * C_E[:, i]

    # lookup table of the bottom most connected black pixel of the
    # looked up pixel
    C_S = np.zeros(mask.shape, dtype=np.int16)

    for i in range(mask.shape[0] - 1, 0, -1):
        C_S[i - 1, :] = np.logical_not(mask[i, :]) * i + mask[i, :] * C_S[i, :]

    # lookup table of the left most connected black pixel of the
    # looked up pixel
    C_W = np.zeros(mask.shape, dtype=np.int16)

    for i in range(0, mask.shape[1] - 1):
        C_W[:, i + 1] = np.logical_not(mask[:, i]) * i + mask[:, i] * C_W[:, i]

    # coordinates of non-transparent pixels with different values
    points = np.nonzero(diff)
    points = np.int32(points + (np.zeros(points[0].shape),)).T

    # loop over non-transparent pixels with different values
    for i, point in enumerate(points):
        y, x = point[:2]

        # find other non-transparent pixels with different values
        # ... with the same y-coordinate
        matches_y = np.nonzero(points[:, 0] == y)[0]
        # ... with an x-coordinate at least 10 pixels away
        matches_y = matches_y[np.abs(points[matches_y, 1] - x) > 10]
        # ... which is connected by black pixels
        matches_y = matches_y[
            (points[matches_y, 1] >= C_W[y, x]) *
            (points[matches_y, 1] <= C_E[y, x])]

        # find other non-transparent pixels with different values
        # ... with the same x-coordinate
        matches_x = np.nonzero(points[:, 1] == x)[0]
        # ... with a y-coordinate at least 10 pixels away
        matches_x = matches_x[np.abs(points[matches_x, 0] - y) > 10]
        # ... which is connected by black pixels
        matches_x = matches_x[
            (points[matches_x, 0] >= C_N[y, x]) *
            (points[matches_x, 0] <= C_S[y, x])]

        if len(matches_y) + len(matches_x) == 0:
            # this pixel cannot be the corner of a box
            break

        for j in matches_y:
            for k in matches_x:
                # loop over combinations of possible boxes
                y_test = points[k, 0]
                x_test = points[j, 1]

                if not C_W[y_test, x] <= x_test <= C_E[y_test, x]:
                    # one horizontal line of the box isn't black
                    continue

                if not C_N[y, x_test] <= y_test <= C_S[y, x_test]:
                    # one vertical line of the box isn't black
                    continue

                # the box is a box and the corners are flagged
                points[i, 2] = points[j, 2] = points[k, 2] = 1

    if points.shape[0] - np.sum(points[:, 2]) > 2:
        print('The images differ by {} pixels'.format(
            points.shape[0] - np.sum(points[:, 2])))

        # Save images and their difference for visual inspection
        fail_txt = 'Notebook test fail '
        utcnow = datetime.utcnow
        scipy.misc.imsave(fail_txt + str(utcnow()) + 'r' + '.png', ref_ndarray)
        scipy.misc.imsave(fail_txt + str(utcnow()) + 't' + '.png', cmp_ndarray)
        img_diff = cmp_ndarray - ref_ndarray
        scipy.misc.imsave(fail_txt + str(utcnow()) + 'd' + '.png', img_diff)

        return False

    return True

def _execute_cell(cell, shell, iopub, timeout=300):
    """
    Execute an IPython Notebook Cell and return the cell output.

    Parameters
    ----------
    cell : IPython.nbformat.current.NotebookNode
        The IPython Notebook cell to execute.
    shell : IPython.kernel.blocking.channels.BlockingShellChannel
        The shell channel which the cell is submitted to for execution.
    iopub : IPython.kernel.blocking.channels.BlockingIOPubChannel
        The iopub channel used to retrieve the result of the execution.
    timeout : int
        The number of seconds to wait for the execution to finish before giving
        up.

    Returns
    -------
    cell_outputs : list
        The list of NotebookNodes holding the result of the execution.

    """

    # Execute input
    shell.execute(cell.input)
    exe_result = shell.get_shell_msg(timeout=timeout)
    if exe_result['content']['status'] == 'error':
        raise RuntimeError('Failed to execute cell due to error: {!r}'.format(
            str(exe_result['content']['evalue'])))

    cell_outputs = list()

    # Poll for iopub messages until no more messages are available
    while True:
        try:
            msg = iopub.get_iopub_msg(timeout=0.5)
        except Empty:
            break

        msg_type = msg['msg_type']
        if msg_type in ('status', 'pyin', 'execute_input', 'execute_result'):
            continue

        content = msg['content']
        node = NotebookNode(output_type=msg_type)

        if msg_type == 'stream':
            node.stream = content['name']
            if 'text' in content:
                # v4 notebook format
                node.text = content['text']
            else:
                # v3 notebook format
                node.text = content['data']

            bug_text = 'Using Anaconda Cloud api site https://api.anaconda.org'
            if bug_text in node.text:
                # Ignore conda (spam) messages/warnings
                continue
        elif msg_type in ('display_data', 'pyout'):
            node['metadata'] = content['metadata']
            for mime, data in content['data'].items():
                attr = mime.split('/')[-1].lower()
                attr = attr.replace('+xml', '').replace('plain', 'text')
                setattr(node, attr, data)
            if msg_type == 'pyout':
                node.prompt_number = content['execution_count']
        elif msg_type == 'pyerr':
            node.ename = content['ename']
            node.evalue = content['evalue']
            node.traceback = content['traceback']
        else:
            raise RuntimeError('Unhandled iopub message of type: {}'.format(
                msg_type))

        cell_outputs.append(node)

    return cell_outputs