# KICKSTART YOUR PYTHON PROJECT

## Part 1: Coockiecutter, git and virtualenv
### Cookiecutter
1. Install [cookiecutter](https://cookiecutter.readthedocs.io/en/latest/readme.html?highlight=data%20science) in your system/user python profile (not a virtual environment).

    ```bash
    $ pip install --user cookiecutter
    ```

2. Surf the file system until your code folder (e.g. `path/to/repos_folder`). This is the parent folder of your code.
**NB: cookiecutter will create a new folder `project_name` with everything inside. Your actual code will be in a subfolder, i.e. `path/to/repos_folder/project_name/src/`**
Then run `cookiecutter` with the link to the  [data-science-template](https://github.com/drivendata/cookiecutter-data-science) and prompt the question it will ask you:

    ```bash
    $ cd Documents/xxx/xxx/Code/
    $ cookiecutter https://github.com/drivendata/cookiecutter-data-science
    # fill the question using project name: kickstart_python
    # once it is finished, cd the forder kickstart_python
    $ cd kickstart_python
    ```
    **From now on, our current directory will be `path/to/kickstarn_python/`** unless specified

### Virtual Environment pt1
3. set up a virtual environment, named `venv`, specifying the python version.
   `venv` is a typical convention, you could call it `remi` if you want.
   This code will create a folder named `venv` containing lot of things and a **local copy of all the packages** you will pip-install from now on.

    ```bash
    $ virtualenv venv -p python3
    ```

4. edit the `.gitignore`  by adding the virtualenv's folder with you favorite text editor or just run the following command

    ```bash
    $ echo venv >> .gitignore
    ```

### GIT
5. set up git and link it to a new github/gitlab reporitory:
    1. On [github.com](https://github.com/) or [inria's gitlab](https://gitlab.inria.fr)  create an **empty** reporitory online (it means no README and no license. If you do so it will display usefull command).
    2. Start git locally and synch it with the following commands:

    ```bash
    $ git init
    # check we are not 'saving' wried files
    $ git status
    # if so, commit
    $ git add .
    $ git commit -m "first commit"

    # If github
    $ git remote add origin https://github.com/USER/python_kickstart.git
    # If inria gitlab
    $ git config --global user.name "your_name"
    $ git config --global user.email "your_email@inria.fr"
    $ git remote add origin git@gitlab.inria.fr:USER/python_kickstart.git

    $ git push -u origin master
    # avoid writing login and password for the future time
    $ git config credential.helper store
    ```
### Virtual Environment pt2
6. Activate the virtualenv

    ```bash
    [user@localhost] project_name/ $ source venv/bin/activate
    # check that it is activated. You should have (venv) at the beginnig of your command line
    (venv) [user@localhost] project_name/ $
    ```

7. Install the basic dependencies of cookiecutter (if you want). Notice that doing so also you will install the src package by default. Then install your everyday-coding-favorite-life packages: numpy, matplotlib, jupyter

    ```bash
    (venv) $ pip install -r requirements.txt
    (venv) $ pip install numpy matplotlib jupyter
    ```

8. Freeze the requirements ('>' overwrite, '>>' append)

    ```bash
    (venv) $ pip freeze >> requirements.txt
    ```

9. Install the package for a toy example

    ```bash
    (venv) $ pip install sklearn
    ```

10. in `src/` create the `main.py` file and paste the following code:

    ```python
    from numpy.random import permutation
    from sklearn import svm, datasets

    C = 1.0
    gamma = 0.7
    iris = datasets.load_iris()
    perm = permutation(iris.target.size)
    iris.data = iris.data[perm]
    iris.target = iris.target[perm]
    model = svm.SVC(C, 'rbf', gamma=gamma)
    model.fit(iris.data[:90],
            iris.target[:90])
    print(model.score(iris.data[90:],
                    iris.target[90:]))
    ```

11. commit the changes

    ```bash
    $ git add .
    $ git commit -m 'toy svm'
    ```

12. edit the `models/train_model.py` and `models/predict_model.py` files. I
In both of the files (actually python modules) create new function respectively
    In `./src/models/train_model.py`:
    ```python
    from sklearn import svm
    def train(data, target, C, gamma):
        clf = svm.SVC(C, 'rbf', gamma=gamma)
        clf.fit(data[:90],
                target[:90])
        return clf
    ```
    In `./src/models/predict_model.py`:
    ```python
    def predict(clf, data, target):
        return clf.score(data, target)
    ```

13. Update the main file in order to import with the following imports
    In `src/main.py` add:
    ```python
    from models.predict_model import predict
    from models.train_model import train
    ```

    Now The main code should looks like:

    ```python
    # std imports
    from numpy.random import permutation
    from sklearn import datasets
    # my imports
    from models.predict_model import predict
    from models.train_model import train

    C = 1.0
    gamma = 0.7
    iris = datasets.load_iris()
    per = permutation(iris.target.size)
    iris.data = iris.data[per]
    iris.target = iris.target[per]
    model = train(iris.data[:90], iris.target[:90], C, gamma)
    score = predict(model, iris.data[90:], iris.target[90:])
    print(score)
    ```

14. Run and debug

    ```bash
    (venv) $ python src/main.py
    ```

### Sacred
15. PIP-install [Sacred](https://github.com/IDSIA/sacred) for tracking experiments

    ```bash
    (venv) $ pip install sacred pymongo
    ```

16. create a new function for the parameters C and gamma and add the colorators for Sacred

    ```python
    #...add here the new nice imports and add the followings
    from sacred import Experiment
    ex = Experiment('iris_svm') # id of the experiments

    @ex.config
    def cfg():
        C = 1.0
        gamma = 0.7

    @ex.automain
    def run(C, gamma):
        # ...
        # ... paste here the main
        #...
        return score
    ```

17. run it from the project's root directory

    ```bash
    (venv) $ python src/main.py
    ```

### MongoDB and Omniboard

18. install mongodb in your system. In a new terminal

    ```bash
    $ sudo dnf install mongodb mongodb-server mongoose
    # start service
    $ sudo service mongod start
    # verify it is woring
    $ mongo  # it will start the mongo-db-shell
    ```

19. Run and re-run as many time as you want the code with the database flag:

    ```bash
    (venv) $ python src/main.py -m MY_IRIS_EXP
    ```
    notice how the ID value increase at each run

21. In a mongo shell (just run mongo in the command line) check if the MY_IRIS_EXP database exists

    ```bash
    $ mongo
    # after in the mongo shell
    > show dbs
    # look for MY_IRIS_EXP entry
    ```

22. download and install [Ominboard](https://github.com/vivekratnavel/omniboard), the sacred+mongo frontends

    ```bash
    # in a new terminal
    $ sudo npm install -g omniboard
    ```

23. In the same shell run the server listener

    ```bash
    $ omniboard -m localhost:27017:MY_IRIS_EXP
    ```

24. go to [http://localhost:9000](http://localhost:9000) to access omniboard frontends:

25. play with it

### Experiment metrics and omniboard visualization

26. add a metric in the main.py file add

    ```python
    @ex.automain
    def run(C, gamma):
        ... # the code before
        ex.log_scalar("val.score", score)
        return score
    ```

27. And what about a typical loss fuction in a for loop?
    for instance add the following line.
    We need to pass the object `_run` at the `main()`

    ```python
    @ex.automain
    def run(_run, C, gamma):
        ... # the code before
        my_loss = 0
        for i in range(20):
            # Explicit step counter (0, 1, 2, 3, ...)
            # incremented with each call for training.accuracy:
            _run.log_scalar("training.loss", my_loss, i)
            my_loss += 1.5*i + np.random.random(1)
        return score
    ```

1. run some experiments

1. play in omniboard


#!/usr/bin/env python
# -*- coding: utf-8 -*-

def plotData(X, y):
    import matplotlib.pyplot as plt
    import numpy as np
    pos = np.where(y == 1)
    neg = np.where(y == 0)
    plt.scatter(X[pos, 0], X[pos, 1], color='g', marker='+')
    plt.scatter(X[neg, 0], X[neg, 1], color='y', marker='o')

    plt.grid()


#!/bin/env python
# -*- coding: utf-8 -*-
"""test Queue
    by Valentyn Stadnytskyi
     created: August 2, 2019
    This is a test library to evaluate the performance of the code.
    Queue is an abstract data structure, somewhat similar to Stacks.
    Unlike stacks, a queue is open at both its ends.
    One end is always used to insert data (enqueue)
    and the other is used to remove data (dequeue)

    to run unittest: python3 -m unittest test_queue
"""

import unittest
import logging
# from numpy.testing import assert_, assert_almost_equal, assert_equal

from ..queue import Queue

class QueueTest(unittest.TestCase):

    def test_queue_rear(self):
        """
        the freshly created queue has 'rear' value of 0 (the next available pointer in the queue buffer).
        """
        queue = Queue(shape=(100, 2, 2, 2))
        self.assertEqual(queue.rear, 0)

    def test_attributes(self):
        """
        various build-in attributes.
        """
        from numpy import random
        queue = Queue(shape=(100, 2, 3, 4), dtype='int16')
        data = random.randint(1024, size=(5, 2, 3, 4))
        self.assertEqual(queue.isempty, True)
        queue.enqueue(data)
        self.assertEqual(queue.length, 5)
        self.assertEqual(queue.rear, 5)
        self.assertEqual(queue.shape, (100, 2, 3, 4))
        self.assertEqual(queue.size, 100*2*3*4)
        self.assertEqual(queue.dtype, 'int16')
        self.assertEqual(queue.isfull, False)
        self.assertEqual(queue.isempty, False)

    def test_reshape(self):
        """
        reshaping operation
        """
        queue = Queue(shape=(100, 2, 3, 4), dtype='int16')
        queue.reshape(shape=(50, 2, 3, 4), dtype='float64')
        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 0)
        self.assertEqual(queue.shape, (50, 2, 3, 4))
        self.assertEqual(queue.size, 50*2*3*4)
        self.assertEqual(queue.dtype, 'float64')

    def test_loop_around(self):
        queue = Queue(shape=(100, 2, 3, 4), dtype='int16')
        queue.reshape(shape=(50, 2, 3, 4), dtype='float64')
        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 0)
        self.assertEqual(queue.shape, (50, 2, 3, 4))
        self.assertEqual(queue.size, 50*2*3*4)
        self.assertEqual(queue.dtype, 'float64')

    def test_peak_first_N(self):
        from numpy import random, array
        queue = Queue(shape=(10, 2), dtype='int16')

        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 0)
        self.assertEqual(queue.shape, (10, 2))
        self.assertEqual(queue.size, 10*2)
        self.assertEqual(queue.dtype, 'int16')

        queue.buffer[:,0] = array(range(10))
        queue.buffer[:,1] = array(range(10))*10

        for i in range(10):
            for j in range(1,10):
                queue.rear = i
                queue.length = j
                arr = queue.peek_first_N(j)
                arr2 = queue.dequeue(j)
                self.assertEqual((arr==arr2).all(),True)

    def test_peek_last_N(self):
        queue = Queue(shape=(10, 2, 3, 4), dtype='int16')
        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 0)
        self.assertEqual(queue.shape, (10, 2, 3, 4))
        self.assertEqual(queue.size, 10*2*3*4)
        self.assertEqual(queue.dtype, 'int16')

        from numpy import random
        arr_rand = random.randint(4096,size = (25,2,3,4))
        queue.reset()
        j = 0
        for i in range(25):
            j+=1
            queue.enqueue(arr_rand[i].reshape(1,2,3,4))
            if i > queue.shape[0]:
                self.assertEqual(queue.length,queue.shape[0])
            self.assertEqual((queue.peek_last_N(1) == arr_rand[i]).all(), True)

            self.assertEqual(queue.global_rear,j)

        self.assertEqual((queue.peek_last_N(1) == arr_rand[-1]).all(), True)
        self.assertEqual((queue.peek_last_N(2) == arr_rand[-2:]).all(), True)
        self.assertEqual((queue.peek_last_N(5) == arr_rand[-5:]).all(), True)
        print('10',queue.peek_last_N(10),arr_rand[-10:])
        self.assertEqual((queue.peek_last_N(10) == arr_rand[-10:]).all(), True)

        dequeue_data = queue.dequeue(10)
        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 5)
        self.assertEqual(queue.global_rear, 25)
        self.assertEqual((dequeue_data == arr_rand[-10:]).all(), True)

    def test_peek_i_j(self):
        from numpy import random
        queue = Queue(shape=(10, 2, 2), dtype='int16')
        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 0)
        self.assertEqual(queue.shape, (10, 2, 2))
        self.assertEqual(queue.size, 10*2*2)
        self.assertEqual(queue.dtype, 'int16')
        arr_in = random.randint(4096,size = (1,2,2))*0+1
        for i in range(10):
            queue.enqueue(arr_in*i)
        self.assertEqual(queue.dtype, 'int16')
        self.assertEqual(queue.peek_i_j(0,1)[0,0,0],0)
        self.assertEqual(queue.peek_i_j(0,2)[0,0,0],0)

    def test_peek_i_j_2(self):
        from numpy import random
        queue = Queue(shape=(100, 10), dtype='int16')
        queue.length = 48
        queue.rear = 48
        i_pointer = 84
        j_pointer = 0
        self.assertEqual(queue.peek_i_j(i_pointer, j_pointer).shape,  (16, 10))

        i_pointer = 20
        j_pointer = 36
        self.assertEqual(queue.peek_i_j(i_pointer, j_pointer).shape,  (16, 10))

    def test_peek_all(self):
        queue = Queue(shape=(10, 2, 3, 4), dtype='int16')
        self.assertEqual(queue.length, 0)
        self.assertEqual(queue.rear, 0)
        self.assertEqual(queue.shape, (10, 2, 3, 4))
        self.assertEqual(queue.size, 10*2*3*4)
        self.assertEqual(queue.dtype, 'int16')

        from numpy import random
        arr_rand = random.randint(4096,size = (25,2,3,4))
        queue.reset()
        j = 0
        for i in range(25):
            arr_rand[i][0,0,0] = i
            queue.enqueue(arr_rand[i].reshape(1,2,3,4))
            self.assertEqual(queue.peek_last_N(1)[0,0,0,0] ,i)
            j+=1
            if j > queue.shape[0]:
                self.assertEqual(queue.length,queue.shape[0])
            else:
                self.assertEqual(queue.length,j)
        self.assertEqual((queue.peek_all() == arr_rand[15:]).all(), True)

        #the queue.rear pointer has to point at empty space in the queue.
        self.assertEqual(queue.buffer[queue.rear-1][0,0,0],i)
        self.assertEqual(queue.peek_last_N(1)[0,0,0,0] ,i)

    def test_dequeue(self):
        """
        Testing dequeue operaritoin via writing and reading data from a queue multipletimes and keeping track of counters and length.
        """
        from numpy import random
        queue = Queue(shape=(11, 2, 3, 4), dtype='int16')
        for i in range(100):
            arr_in = random.randint(4096,size = (2,2,3,4))
            queue.enqueue(arr_in)
            arr_out = queue.dequeue(2)
            self.assertEqual((arr_in==arr_out).all(), True)
            self.assertEqual(queue.length,0)
            self.assertEqual(queue.global_rear,(i+1)*2)
            self.assertEqual(queue.rear,2*(i+1)-int(2*(i+1)/11)*11)

        from numpy import random
        queue = Queue(shape=(32, 2, 3, 4), dtype='int16')
        for i in range(100):
            arr_in = random.randint(4096,size = (1,2,3,4))
            queue.enqueue(arr_in)
            self.assertEqual(queue.length,1)
            arr_out = queue.dequeue(1)
            self.assertEqual((arr_in==arr_out).all(), True)
            self.assertEqual(queue.length,0)
            self.assertEqual(queue.global_rear,(i+1)*1)
            self.assertEqual(queue.rear,1*(i+1)-int(1*(i+1)/queue.shape[0])*queue.shape[0])

    def test_dequeue_2(self):
        """
        Test to check if queue can perform in case of DI-4108 DATAQ operation mode.
        """
        from numpy import random
        queue = Queue(shape=(100,10), dtype='int16')
        for i in range(5): queue.enqueue( random.randint(0,4096,(16,10)) )
        for i in range(1000):
            self.assertEqual(queue.dequeue(16).shape,(16,10))
            queue.enqueue(random.randint(0,4096,(16,10)) )

    def test_1(self):
        from numpy import std, random
        queue = Queue(shape=(100, 2, 2, 2))
        data = random.randint(0,1024, size=(5, 2, 2, 2))
        queue.enqueue(data)
        self.assertEqual(queue.length, 5)
        self.assertEqual(queue.rear, 5)
        queue.enqueue(data)
        dequeue_data = queue.dequeue(N=3)
        self.assertEqual(queue.length, 7)
        self.assertEqual(dequeue_data.shape, data[:3].shape)
        self.assertEqual(std(dequeue_data), std(data[:3]))

    def test_threaded_1(self):
        from numpy import zeros,arange,copy
        from time import sleep
        queue = Queue(shape=(4, 3000, 4096), dtype='int16')
        for i in range(10):
            arr_in = zeros((1,3000,4096),dtype ='int16')+i
            queue.enqueue(arr_in)
            arr_out = queue.dequeue(1)[-1]
            self.assertEqual(i,arr_out[0,0])

        from ubcs_auxiliary.threading import new_thread

        def run(queue):
            for i in range(100):
                from time import sleep
                arr_in = zeros((1,3000,4096),dtype ='int16')+i
                queue.enqueue(arr_in)
                sleep(0.1)
        new_thread(run, queue)
        j = 0
        arr2 = arange(0,99)
        arr = copy(arr2)*0

        while j < 99:
            if queue.length > 0:
                arr_out = queue.dequeue(1)[-1]
                self.assertEqual(j,arr_out[0,0])
                arr[j] = arr_out[0,0]
                j+=1
                logging.debug(f'dequeue: {j}')
            sleep(0.03)

        self.assertEqual((arr==arr2).all(),True)

    def test_dequeue_async(self):
        #from circular_buffer_numpy.queue import Queue
        from numpy import zeros,arange,copy, random
        from time import sleep
        queue = Queue(shape=(16, 3000, 4096), dtype='int16')
        from ubcs_auxiliary.threading import new_thread
        queue.reset()
        j = 0
        arr2 = arange(0,499)
        arr = copy(arr2)*0
        for i in range(500):
            from time import sleep
            arr_in = random.randint(0,4096,(1,3000,4096),dtype ='int16')
            arr_in[0,0,0] = i
            queue.enqueue(arr_in)
            sleep(0.2)
            if i%5 == 0:
                while queue.length > 0:
                    arr_out = queue.dequeue(1)[-1]
                    #print(j,arr_out[0,0,0])
                    arr[j] = arr_out[0,0]
                    #print(f'dequeue: {arr_out[0,0]}, {j}, {arr_out[0,0]==j}')
                    self.assertEqual(arr_out[0,0],j)
                    j+=1


#!/usr/bin/env python
# coding: utf-8

# # Conhecendo os Dados

# In[103]:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from unidecode import unidecode

pesquisa = pd.read_csv("resultados-pesquisa-anonimizado.csv")
pesquisa.info()

# In[104]:

pesquisa.describe()[:2]

# Inicialmente, podemos ver que todos os campos do resultado da pesquisa consistem de objetos, provavelmente representando informações textuais. Isso também é resultado da forma como os dados foram armazenados na planilha no Google Sheets.
#
# Analisando as colunas, podemos considerar que a informação de data/hora da resposta da pesquisa pode ser descartada, já que não traz um dado que possa ser analisado em conjunto com os demais, representando apenas o momento em que aquela resposta foi preenchida.
#
# A maioria das demais colunas possuem quase que todos os registros com valores únicos. Supondo que isso seja resultado da inserção da informação por um campo de texto, podemos tentar analisar respostas que sejam semelhantes e agrupá-las, criando categorias que possam ser mais úteis para a análise.
#
# A coluna que menos apresenta valores únicos para os seus registros é a pergunta <i>"Com qual frequência você busca por informações fornecidas pela prefeitura de Feira de Santana?"</i>. Supondo que a pergunta tenha disponibilizado opções para resposta, isso faz sentido. Ainda assim, a pergunta supõe uma ordenação entre as respostas, que buscaremos evidenciar.

# In[105]:

pesquisa.drop(["Carimbo de data/hora"], axis=1, inplace=True)

# # Limpeza e Exibição dos Dados

# ## Ocupação

# In[106]:

pesquisa["Qual sua atual ocupação (profissão ou estudante)? "].unique()

# Vendo os resultados únicos e o texto da coluna para a pergunta sobre ocupação, podemos fazer uma breve limpeza nos dados e classificação, criando uma nova coluna.

# In[107]:

pesquisa["OCUPACAO"] = (
    pesquisa["Qual sua atual ocupação (profissão ou estudante)? "]
    .fillna("Não informado")
    .apply(str.strip)
    .apply(str.title)
)

# In[108]:

pesquisa["OCUPACAO"].sort_values().unique()

# Analisando novamente as profissões, podemos considerar algumas categorias.
#
# "Bombeiro Militar", "Policial Militar" e "Polícial Militar" podem ser agrupados em uma categoria "Militar".
#
# "Professor" e "Professora" podem ser agrupados em "Professor/Professora".
#
# "Promotor De Vendas" e "Representante" podem ser agrupados em "Vendas".
#
# "Publicitária" e "Publicitário E Locutor" podem ser agrupados em "Publicitário/Publicitária".
#
# As demais ocupações não parecem se agrupar naturalmente.

# In[109]:

def categorize_profession(profession):
    if "militar" in profession.lower():
        return "Militar"
    if "professor" in profession.lower():
        return "Professor/Professora"
    if profession in ("Promotor De Vendas", "Representante"):
        return "Vendedor/Representante"
    if "publicit" in profession.lower():
        return "Publicitário/Publicitária"
    if "community manager" in profession.lower():
        return "Organizador de Comunidade"
    else:
        return profession

pesquisa["OCUPACAO_CATEGORY"] = pesquisa["OCUPACAO"].apply(categorize_profession)

# In[110]:

pesquisa.groupby("OCUPACAO_CATEGORY").size().sort_values().plot(
    kind="barh", title="Ocupação Profissional"
)

# ## Frequência de Busca de Informações

# In[111]:

# Reescrevendo a resposta em uma escrita mais concisa
pesquisa.loc[
    pesquisa[
        "Com qual frequência você busca por informações fornecidas pela prefeitura de Feira de Santana?"
    ]
    == "Depende da semana, mas geralmente mais de uma vez por semana. ",
    "Com qual frequência você busca por informações fornecidas pela prefeitura de Feira de Santana?",
] = "Mais de 1 vez por semana"

# In[112]:

pesquisa[
    "Com qual frequência você busca por informações fornecidas pela prefeitura de Feira de Santana?"
].value_counts()

# As informações já estão bem descritas, e podemos exibir os dados de forma ordenada.

# In[113]:

order = [
    "Todos os dias",
    "Mais de 1 vez por semana",
    "1 vez por semana",
    "1 vez por mês",
    "Quase nunca",
    "Nunca busquei",
]
pesquisa[
    "Com qual frequência você busca por informações fornecidas pela prefeitura de Feira de Santana?"
].value_counts().reindex(order).plot(
    kind="barh", title="Frequência de busca de informações"
)

# ## Expectativa das Informações

# In[114]:

pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].value_counts()

# Podemos notar que todas as respostas são distintas entre si, provavelmente resultado da coleta através de uma caixa de texto.
#
# Assim como fizemos em outras perguntas, podemos estabelecer categorias através de palavras-chave. Como a quantidade de respostas não é muito extensa, é possível fazer uma análise em cada uma das respostas.
#
# Em uma observação das respostas, podemos estabelecer as seguintes categorias:
#
# 1. Investimentos (infraestrutura, despesas gerais, salário);
# 2. Saúde;
# 3. Educação;
# 4. Segurança;
# 5. Orçamento (verba);
# 6. Transparência;
# 7. Outros;
#
# Uma resposta pode estar representada em mais de uma categoria. Assim, a soma das categorias pode ser maior que a quantidade de respostas obtidas.

# In[115]:

def categorize_expectation_public_investments(answer):
    if any(
        x in unidecode(answer.lower())
        for x in [
            "servidor",
            "investimento",
            "obra",
            "urbanizacao",
            "salario",
            "compra",
            "dinheiro",
            "bairro",
            "recurso",
            "gasto",
            "licitacao",
            "licitacoes",
        ]
    ):
        return 1
    return 0

def categorize_expectation_public_health(answer):
    if any(x in unidecode(answer.lower()) for x in ["saude"]):
        return 1
    return 0

def categorize_expectation_public_education(answer):
    if any(x in unidecode(answer.lower()) for x in ["educacao", "escola", "colegio"]):
        return 1
    return 0

def categorize_expectation_public_safety(answer):
    if any(x in unidecode(answer.lower()) for x in ["seguranca"]):
        return 1
    return 0

def categorize_expectation_public_budget(answer):
    if any(x in unidecode(answer.lower()) for x in ["verba", "dinheiro", "receita"]):
        return 1
    return 0

def categorize_expectation_public_transparency(answer):
    if any(
        x in unidecode(answer.lower())
        for x in [
            "informacoes",
            "informacao",
            "transparencia",
            "gasto",
            "dados",
            "verdade",
            "clareza",
        ]
    ):
        return 1
    return 0

def categorize_expectation_others(answer):
    if any(
        x in unidecode(answer.lower())
        for x in ["geral", "relato", "proposta", "grafico"]
    ):
        return 1
    return 0

def categorize_information(answer):
    if any(
        x in unidecode(answer.lower())
        for x in [
            "retorno",
            "gastos",
            "questionamento",
            "atende",
            "conteudo",
            "informacao",
            "informacoes",
            "dados",
            "confiabilidade",
            "resumo",
            "divulgacao",
        ]
    ):
        return 1
    return 0

def categorize_frequency(answer):
    if any(
        x in unidecode(answer.lower())
        for x in [
            "frequente",
            "atualizacao",
            "tempo real",
            "atualizados",
            "atualizadas",
        ]
    ):
        return 1
    return 0

def categorize_usability(answer):
    if any(
        x in unidecode(answer.lower())
        for x in ["facilidade", "intuitiva", "intuitivo",]
    ):
        return 1
    return 0

pesquisa["ANSWER_INVESTMENT"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_public_investments)

pesquisa["ANSWER_HEALTH"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_public_health)

pesquisa["ANSWER_EDUCATION"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_public_education)

pesquisa["ANSWER_SAFETY"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_public_safety)

pesquisa["ANSWER_BUDGET"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_public_budget)

pesquisa["ANSWER_TRANSPARENCY"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_public_transparency)

pesquisa["ANSWER_OTHERS"] = pesquisa[
    "Que tipo de informação você ESPERA encontrar no Dados abertos de Feira? (Coisas que você acredita que é certeza ter)"
].apply(categorize_expectation_others)

# In[116]:

labels = {
    "ANSWER_INVESTMENT": "INVESTIMENTO",
    "ANSWER_HEALTH": "SAÚDE",
    "ANSWER_EDUCATION": "EDUCAÇÃO",
    "ANSWER_SAFETY": "SEGURANÇA",
    "ANSWER_BUDGET": "ORÇAMENTO",
    "ANSWER_TRANSPARENCY": "TRANSPARÊNCIA",
    "ANSWER_OTHERS": "OUTROS",
}
pesquisa[
    [
        "ANSWER_INVESTMENT",
        "ANSWER_HEALTH",
        "ANSWER_EDUCATION",
        "ANSWER_SAFETY",
        "ANSWER_BUDGET",
        "ANSWER_TRANSPARENCY",
        "ANSWER_OTHERS",
    ]
].apply(np.sum).rename(index=labels).sort_values().plot(
    kind="barh", title="Expectativa do tipo de informação"
)

# ## Informações Adicionais

# In[117]:

pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].value_counts()

# Em relação à pergunta de quais informações seria interessante existirem no portal (indo além do obrigatório), novamente temos respostas únicas para cada respondente. Isso é esperado, decorrente da forma discursiva da coleta dos dados. Faremos a categorização de forma semelhante à pergunta anterior.

# In[118]:

pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].fillna(
    ""
)

# In[119]:

pesquisa["SECOND_ANSWER_INVESTMENT"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_public_investments)

pesquisa["SECOND_ANSWER_HEALTH"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_public_health)

pesquisa["SECOND_ANSWER_EDUCATION"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_public_education)

pesquisa["SECOND_ANSWER_SAFETY"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_public_safety)

pesquisa["SECOND_ANSWER_BUDGET"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_public_budget)

pesquisa["SECOND_ANSWER_TRANSPARENCY"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_public_transparency)

pesquisa["SECOND_ANSWER_OTHERS"] = pesquisa[
    "Que tipo de informação você GOSTARIA que tivesse em um portal de transparência de dados da cidade? (Coisas que vão além do obrigado a ter)"
].apply(categorize_expectation_others)

# In[120]:

labels = {
    "SECOND_ANSWER_INVESTMENT": "INVESTIMENTO",
    "SECOND_ANSWER_HEALTH": "SAÚDE",
    "SECOND_ANSWER_EDUCATION": "EDUCAÇÃO",
    "SECOND_ANSWER_SAFETY": "SEGURANÇA",
    "SECOND_ANSWER_BUDGET": "ORÇAMENTO",
    "SECOND_ANSWER_TRANSPARENCY": "TRANSPARÊNCIA",
    "SECOND_ANSWER_OTHERS": "OUTROS",
}
pesquisa[
    [
        "SECOND_ANSWER_INVESTMENT",
        "SECOND_ANSWER_HEALTH",
        "SECOND_ANSWER_EDUCATION",
        "SECOND_ANSWER_SAFETY",
        "SECOND_ANSWER_BUDGET",
        "SECOND_ANSWER_TRANSPARENCY",
        "SECOND_ANSWER_OTHERS",
    ]
].apply(np.sum).rename(index=labels).sort_values().plot(
    kind="barh", title="Informações além do obrigatório"
)

# ## Incentivo a Retorno

# In[121]:

pesquisa[
    "O que lhe faria voltar a consultar o portal Dados abertos de Feira com frequência?"
].value_counts()

# Para a questão "O que lhe faria voltar a consultar o portal Dados abertos de Feira com frequência?", temos novamente respostas diferentes entre os respondentes. Faremos uma categorização assim como nas perguntas anteriores.

# In[122]:

pesquisa["THIRD_ANSWER_INFO"] = pesquisa[
    "O que lhe faria voltar a consultar o portal Dados abertos de Feira com frequência?"
].apply(categorize_information)

pesquisa["THIRD_ANSWER_FREQ"] = pesquisa[
    "O que lhe faria voltar a consultar o portal Dados abertos de Feira com frequência?"
].apply(categorize_frequency)

pesquisa["THIRD_ANSWER_USABILITY"] = pesquisa[
    "O que lhe faria voltar a consultar o portal Dados abertos de Feira com frequência?"
].apply(categorize_usability)

# In[123]:

labels = {
    "THIRD_ANSWER_INFO": "QUALIDADE DA INFORMAÇÃO",
    "THIRD_ANSWER_FREQ": "FREQUÊNCIA DA INFORMAÇÃO",
    "THIRD_ANSWER_USABILITY": "SISTEMA INTUITIVO",
}
pesquisa[["THIRD_ANSWER_INFO", "THIRD_ANSWER_FREQ", "THIRD_ANSWER_USABILITY"]].apply(
    np.sum
).rename(index=labels).sort_values().plot(
    kind="barh",
    title="O que me faria voltar a utilizar o portal?",
    xticks=np.arange(2, 22, 2),
)

# ## Outras Fontes de Informação

# In[124]:

pesquisa["Quais sites você usa ou já usou para pesquisar esses dados?"] = pesquisa[
    "Quais sites você usa ou já usou para pesquisar esses dados?"
].fillna("Nenhum")
pesquisa["Quais sites você usa ou já usou para pesquisar esses dados?"].value_counts()

# Podemos ver que existem algumas respostas semelhantes, embora a maior parte dos respondentes não tenha informado outra fonte de informação dos dados municipais. De qualquer forma, precisamos categorizar algumas das informações. Apesar do Acorda Cidade ser um website, pela quantidade expressiva de respostas, vamos separá-lo em uma categoria própria.

# In[125]:

def categorize_acorda_cidade(answer):
    if any(x in unidecode(answer.lower()).strip() for x in ["acorda cidade"]):
        return 1
    return 0

def categorize_prefeitura(answer):
    if any(x in unidecode(answer.lower()).strip() for x in ["prefeitura"]):
        return 1
    return 0

def categorize_diario_oficial(answer):
    if any(x in unidecode(answer) for x in ["DO", "oficial"]):
        return 1
    return 0

def categorize_sites(answer):
    if any(
        x in unidecode(answer.lower()).strip() for x in ["sites", "jornal", "jornais"]
    ):
        return 1
    return 0

def categorize_none(answer):
    if any(x in unidecode(answer.lower()).strip() for x in ["nenhum"]):
        return 1
    return 0

# In[126]:

pesquisa["FOURTH_ANSWER_ACORDA_CIDADE"] = pesquisa[
    "Quais sites você usa ou já usou para pesquisar esses dados?"
].apply(categorize_acorda_cidade)

pesquisa["FOURTH_ANSWER_CITY_HALL"] = pesquisa[
    "Quais sites você usa ou já usou para pesquisar esses dados?"
].apply(categorize_prefeitura)

pesquisa["FOURTH_ANSWER_OFFICIAL"] = pesquisa[
    "Quais sites você usa ou já usou para pesquisar esses dados?"
].apply(categorize_diario_oficial)

pesquisa["FOURTH_ANSWER_WEBSITES"] = pesquisa[
    "Quais sites você usa ou já usou para pesquisar esses dados?"
].apply(categorize_sites)

pesquisa["FOURTH_ANSWER_NONE"] = pesquisa[
    "Quais sites você usa ou já usou para pesquisar esses dados?"
].apply(categorize_none)

# In[127]:

labels = {
    "FOURTH_ANSWER_ACORDA_CIDADE": "ACORDA CIDADE",
    "FOURTH_ANSWER_CITY_HALL": "PREFEITURA",
    "FOURTH_ANSWER_OFFICIAL": "DIÁRIO OFICIAL",
    "FOURTH_ANSWER_WEBSITES": "OUTROS SITES",
    "FOURTH_ANSWER_NONE": "NENHUM",
}
pesquisa[
    [
        "FOURTH_ANSWER_ACORDA_CIDADE",
        "FOURTH_ANSWER_CITY_HALL",
        "FOURTH_ANSWER_OFFICIAL",
        "FOURTH_ANSWER_WEBSITES",
        "FOURTH_ANSWER_NONE",
    ]
].apply(np.sum).rename(index=labels).sort_values().plot(
    kind="barh", title="Outros sites utilizados"
)

# In[128]:

labels = {
    "FOURTH_ANSWER_ACORDA_CIDADE": "ACORDA CIDADE",
    "FOURTH_ANSWER_CITY_HALL": "PREFEITURA",
    "FOURTH_ANSWER_OFFICIAL": "DIÁRIO OFICIAL",
    "FOURTH_ANSWER_WEBSITES": "OUTROS SITES",
}
pesquisa[
    [
        "FOURTH_ANSWER_ACORDA_CIDADE",
        "FOURTH_ANSWER_CITY_HALL",
        "FOURTH_ANSWER_OFFICIAL",
        "FOURTH_ANSWER_WEBSITES",
    ]
].apply(np.sum).rename(index=labels).sort_values().plot(
    kind="barh", title="Outros sites utilizados (para aqueles que usam)"
)


#!/usr/bin/env python
# coding: utf-8

# # Project: The Movie Database (TMDb) Data Analysis
#
# ## Table of Contents
#

#
Introduction

#
Data Wrangling

#
Exploratory Data Analysis

#
Conclusions

#

#
# ## Introduction
#
#
# ### Data Overview
#
# > The data is collected from The Movie Databese (TMDb) API. The product uses the TMDb API but is not endorsed or certified by TMDb. The API provides access to data on many movies, actors and actresses, crew members, and TV shows as well as budgets and revenues.
# This dataset contains information about 10,000 movies collected from The Movie Database (TMDb), including user ratings and revenue. We shall analyze this data to dig into some of the important industry questions like what can we say about the success of a movie before it is released?
#
# > Note that budget_adj (budget adjusted) and revenue_adj (revenue adjusted) columns show the budget and revenue of the associated movie in terms of 2010 dollars, accounting for inflation over time.
#
# ### Relevant questions we could ask about the data
#
# > 1. Which genres are most popular from year to year?
# > 2. What kinds of properties are associated with movies that have high revenues?
#

# In[1]:

# Use this cell to set up import statements for all of the packages that you
#   plan to use.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
get_ipython().run_line_magic('matplotlib', 'inline')
sns.set_style ('darkgrid')

# Remember to include a 'magic word' so that your visualizations are plotted
#   inline with the notebook. See this page for more:
#   http://ipython.readthedocs.io/en/stable/interactive/magics.html

#
# ## Data Wrangling
#
#
# ### General Properties
#
# > **Features And Counts**
# > 1. Samples (rows) = 10866
# > 2. Variables(columns) = 21
# > 3. Duplicate rows = 1
# > 4. Rows with missing data = 7930

# In[2]:

# Load your data and print out a few lines. Perform operations to inspect data
df = pd.read_csv('tmdb_movies.csv')
df.head()

#   types and look for instances of missing or possibly errant data.

# In[3]:

df.tail(20)

# In[4]:

# exploring the overall dataset to discover fixes
df.describe()

# In[5]:

# exploring the overall dataset with visuals
df.hist(figsize=(15, 12));

# > release_year and vote_average are good variables to analyze this dataset because of their step-like skewness

# In[6]:

# to find number of samples, number of columns, datatypes of the variables, and missing values
df.info()

#
# ### Data Cleaning
#

# In[7]:

# to find number of duplicate rows
sum(df.duplicated())

# In[8]:

# to find number of unique values
df.nunique()

# #### 1. Cleaning Column Labels
# > **Dropping extraneous columns**
#
# > I dropped features that that are not relevant to the questions for my analysis.
#
# > **imdb_id:** These values are not necessary in analyzing information within the table.
#
# > **homepage, tagline, overview, keywords:** These values are unique to each row and cannot be used to make comparisons across multiple rows.
#
# > **budget, revenue:** Since the **'budget_adj' and 'revenue_adj'** columns adjust the budget and revenue values in terms of 2010 dollars and will therefore allow for more accurate comparisons and analysis, these columns are not needed.
#
# > **release_date:** I will be using release year to analyze this data, so this more specific column is not needed.

# In[9]:

# using pandas' drop function to drop columns from the dataset: 'id', 'imdb_id', 'homepage', 'tagline', 'keywords', 'overview', 'release_date'
df.drop(['imdb_id', 'homepage', 'tagline', 'keywords', 'overview', 'budget', 'revenue', 'release_date'], axis=1, inplace=True)

# In[10]:

df.head()

# In[11]:

df.info()

# #### 2. Fixing Data Types
# > I fixed **revenue_adj** and **budget_adj** datatypes
# >
# >Convert from float to int

# In[12]:

df['revenue_adj'] = df['revenue_adj'].astype(int)
df['budget_adj'] = df['budget_adj'].astype(int)

# In[13]:

df.head()

# #### 3. Rename Data Columns
# > Renaming **budget_adj and revenue_adj** to new **budget and revenue** to make the dataset well labelled

# In[14]:

#rename columns to make variables have good attributes and proper names
df.rename(columns = {'budget_adj':'budget', 'revenue_adj':'revenue'}, inplace=True)
df.head()

# In[15]:

df.info()

# #### 4. Drop Nulls and Dedupe
# > I dropped any rows that contained missing values and dropped any duplicate rows in the dataset
#

# In[16]:

# view missing value count for each feature in the dataset
df.isnull().sum()

# > There are no missing values for all datatypes that are int and float

# In[17]:

df.dropna(inplace=True)
df.info()

# In[18]:

# print numbers of duplicates in the dataset
print(df.duplicated().sum())

# In[19]:

# drop duplicates
df.drop_duplicates(inplace=True)

# In[20]:

# print again number of duplicates to confirm dedupe
print(df.duplicated().sum())

# In[21]:

df.info()

# In[22]:

df.shape

# #### 5. Creating Column hybrids
# > I transform the **genres** column to make it easier to work with

# In[23]:

# turn string into list
df['genres'] = df['genres'].str.split('|')
df.head()

# In[24]:

# create a new dataframe with genres listed out in rows
df_gen = df.genres.apply(pd.Series)
df_gen.head()

# #### 6. Merging Datasets
# > I combined the new dataframe with the main dataframe to have a single dataframe for analysis
#
# > **genres** was originally presented as a string of genre types separated by a **'|'**. This makes it difficult to narrow down movies by one specific genre which is what the first question is all about. First, I turned the strings into a list and then separated the items into their own columns in a new table. Now, I will merge this dataframe with the original dataframe.
# >
# > After that, I will pivot the genre values so that they are presented in their own rows instead of columns. And also, I will remove the original **genres** row and the **variable** row that will be created by the use of the **melt() function** below.
#

# In[25]:

# merge both dataframes
df_combined = df.merge(df_gen, left_index=True, right_index = True)
df_combined.head()

# In[26]:

# pivot genres from columns into rows
df_combined = df_combined.melt(id_vars=['id' ,'popularity','original_title','cast','director','runtime','genres','production_companies','vote_count','vote_average','release_year','budget','revenue'],value_name="genre")
df_combined.head()

# In[27]:

#remove unnecessary columns
df_combined.drop(['genres','variable'],axis=1,inplace=True)
df_combined.head()

# ### 7. Drop Nulls
# > Finally, I am going to drop all rows with null values again. I now have two dataframes, **df** and **df_combined**. The first table is the original table of 9,772 rows and the second is the new table where each movie is presented in a row for every genre it is associated with.

# In[28]:

# remove rows with null values
df_combined.dropna(inplace=True)

# In[29]:

df_combined.info()

# In[30]:

df_combined.shape

#
# ## Exploratory Data Analysis
#
# > **Tip**: Now that you've trimmed and cleaned your data, you're ready to move on to exploration. Compute statistics and create visualizations with the goal of addressing the research questions that you posed in the Introduction section. It is recommended that you be systematic with your approach. Look at one variable at a time, and then follow it up by looking at relationships between variables.
#
# > In this step, I will explore the data to answer the below question and plot different visualizations to identify patterns and dependencies.
#
# ### Research Question 1: Which genres are most popular from year to year?
# > There are two variables that can be used to understand the popularity of genres of a movie, 'popularity' and 'vote_average'. I compare these columns with each other from year to year to identify which one will be more helpful in answering my question.
# >
# > Using histograms, bar charts, box plots and scatterplots to explore the cleaned dataset

# In[31]:

# Use this, and more code cells, to explore your data. Don't forget to add
#   Markdown cells to document your observations and findings.
df_combined.plot(x='vote_average', y='popularity',kind='scatter')
plt.title('Vote Average vs Popularity')
plt.xlabel('Vote Average')
plt.ylabel('Popularity')
plt.show()

# In[32]:

df_combined['vote_average'].plot(kind='box')
plt.show()

# In[33]:

df['popularity'].plot(kind='box')
plt.show()

# > I want to first check to make sure that both **popularity** and **vote_average** represent the same thing. The scatter plot above illustrates that both columns are positively correlated. However, as shown by the box plots, **vote_average** is more evenly distributed and lacks any outliers. Nonetheless, all following analysis will be done with **vote_average** as well as **popularity**.

# In[34]:

df_genre = df_combined.groupby('genre').mean()
df_genre = df_genre.sort_values('vote_average')
df_genre
plt.bar(df_genre.index, df_genre['vote_average'], color = 'blue')
plt.xticks(rotation='90')
plt.title('Vote Average by Genre')
plt.ylabel('Vote Average')
plt.xlabel('Genre')
plt.show()

# > To compare vote averge by genre, I used the new dataframe I made while cleaning the data (**df_combined**) to create another dataframe (**df_genre**) that displays the mean value of all columns aggregated by genre type. Then I plotted the **vote average mean** on a bar chart for each genre.
#
# > The chart shows that the lowest rated movies are **Horror**, **Science Fiction**, and **TV Movie**, while the highest rated are **Documentary**, **History**, and **Music**.

# Now, let's check for the most popular genre and their year

# In[35]:

df_combined['popularity'].describe()

# In[36]:

# use groupby to get popular genres
genre_pop = df_combined.groupby(['release_year', 'genre'], as_index=False)['popularity'].mean()

# In[37]:

# Plot the The Popularity of each genre in the dataset
plt.figure(figsize=[24,12])
sns.barplot(data=genre_pop.sort_values(by='popularity', ascending=False), x='genre', y='popularity')
plt.xlabel('Genre')
plt.ylabel('Popularity')
plt.title('The Popularity of each Genre in TMDB Movies')
plt.show()

# From the combined dataset, **Adventure**, **Animation** and **Fantasy** are the top 3 popular genres

# In[38]:

# use query to determine the most popular genres
most_popular = df_combined.query('popularity > 10')

# In[39]:

# plot the most popular genre (i.e, genres with popularity higher than 10)
most_popular.plot(x='genre', y='popularity', kind='bar', figsize=(8,4))
plt.title('Most Popular Genres in TMDb Movies')
plt.xlabel('Genres')
plt.ylabel('Popularity')
plt.show()

# **Action**, **Adventure**, **Drama**, **Science Fiction** and **Thriller** are the most popular genres year by year

# In[40]:

# print most popular genres and their release year
print(most_popular.groupby(['release_year', 'genre'])['popularity'].mean())

# In[41]:

# check the most popular genre for each year
most_popular.groupby(['release_year', 'genre'])['popularity'].mean().plot(kind='bar')
plt.title('Most Popular Genre by Year')
plt.xlabel('Release Year and Genre')
plt.ylabel('Popularity')
plt.show()

# And again, this shows that **Action**, **Adventure**, **Drama** ,**Science Fiction** and **Thriller** are the most popular genres in their grossing years

# ### Research Question 2: What kinds of properties are associated with movies that have high revenues?

# In[42]:

# use query to determine high revenue movies
high_revenue = df_combined.query('revenue > revenue.mean()')
low_revenue = df_combined.query('revenue <= revenue.mean()')

# In[43]:

# plot to compare high revenue with low revenue movies
low_revenue.vote_average.plot.hist(color='orange', alpha=0.5, label='Low')
high_revenue.vote_average.plot.hist(color='green', alpha=0.5, label='High')
plt.title('Revenue by Rating')
plt.xlabel('Rating')
plt.ylabel('Revenue Counts')
plt.legend()
plt.show()

# It is clear that we have more low revenue movies from the dataset. High revenue skewed more to the right while low revenue skewed more to the left. This shows that high revenue movies gained higher rating than the low revenue movies.
# Therefore, **vote_average** is one of the properties associated with high revenue movies

# In[44]:

# check the association of high revenue movies with their release year
low_revenue.release_year.plot.hist(color='orange', alpha=0.5, label='Low')
high_revenue.release_year.plot.hist(color='green', alpha=0.5, label='High')
plt.title('Revenue by Release Year')
plt.xlabel('Release Year')
plt.ylabel('Revenue Counts')
plt.legend()
plt.show()

# In[45]:

# check the association of high revenue movies with budget
low_revenue.budget.plot.hist(color='orange', alpha=0.5, label='Low')
high_revenue.budget.plot.hist(color='green', alpha=0.5, label='High')
plt.title('Revenue by Budget')
plt.xlabel('Budget')
plt.ylabel('Revenue Counts')
plt.legend()
plt.show()

# High revenue movies are favoured by high budget as well.
# Therefore, **budget** is also associated with high revenue movies

#
# ## Conclusions
#
# > **Tip**: Finally, summarize your findings and the results that have been performed. Make sure that you are clear with regards to the limitations of your exploration. If you haven't done any statistical tests, do not imply any statistical conclusions. And make sure you avoid implying causation from correlation!
#
# > **Tip**: Once you are satisfied with your work here, check over your report to make sure that it is satisfies all the areas of the rubric (found on the project submission page at the end of the lesson). You should also probably remove all of the "Tips" like this one so that the presentation is as polished as possible.
#
# > There are many qualities about movies that make them unique from one another. In this project, I was able to analyze these qualities and identify which properties are associated with movie popularity and high revenue.
#
# > After cleaning and trimming the dataset by removing unnecessary, null, and duplicated values, I created a secondary table that broke each movie down into the separate genre it falls under.
# Then I plotted a few charts to assess what will be used as the dependent variable, popularity and vote average.
#
# ### Findings
#
#
# 1. **Action**, **Adventure**, **Drama**, **SCience Fiction** and **Thriller** are the most popular genres year by year
# 2. **Action**, **Adventure**, **Drama**, **SCience Fiction** and **Thriller** are the most popular genres in their grossing year
# 3. **vote_average**, **release_year** and **budget** are the properties associated with high revenue movies
#
#
# > Since my analysis only illustrates correlation between variables, it does not definitively conclude whether any trait can predict the popularity and revenue status of a movie. That would require deeper statistical analysis that was not performed in this project.
#
# ## Submitting your Project
#
# > Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).
#
# > Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.
#
# > Once you've done this, you can submit your project by clicking on the "Submit Project" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!

# In[46]:

from subprocess import call
call(['python', '-m', 'nbconvert', 'Investigate_a_Dataset.ipynb'])


# coding: utf-8

# # Anomalies in meal prices
#
# In the Chamber of Deputies' CEAP, there is a list of 1,000's of meal expenses made by congresspeople. The law says that the congressperson cannot pay for any other, even being her advisor or SO. We want to work on this analysis to find possibly illegal and immoral expenses. They may have happened when the politician spent more than needed (e.g. the whole menu costs X but the bill was 2X) or too much in an specific period of time. In the end, we also want to alert about too expensive reibursements, even with an explanation behind of it.
#
# Note: remember to correct prices with an inflation index (e.g. IPCA).

# In[1]:

get_ipython().magic('matplotlib inline')
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)

plt.rcParams['figure.figsize'] = (20, 10)

# In[2]:

from serenata_toolbox.datasets import fetch

fetch('2016-11-19-reimbursements.xz', '../data')
fetch('2016-09-03-companies.xz', '../data')
fetch('2016-11-29-yelp-companies.xz', '../data')
fetch('2016-12-02-foursquare-companies.xz', '../data')

# In[3]:

import numpy as np
import pandas as pd

dataset = pd.read_csv('../data/2016-11-19-reimbursements.xz',
                      dtype={'applicant_id': np.str,
                             'cnpj_cpf': np.str,
                             'congressperson_id': np.str,
                             'subquota_number': np.str},
                      low_memory=False)
dataset = dataset[dataset['congressperson_id'].notnull()]
dataset['issue_date'] = pd.to_datetime(dataset['issue_date'], errors='coerce')
dataset['issue_date_day'] = dataset['issue_date'].apply(lambda date: date.day)
dataset['issue_date_month'] = dataset['issue_date'].apply(lambda date: date.month)
dataset['issue_date_year'] = dataset['issue_date'].apply(lambda date: date.year)
dataset['issue_date_weekday'] = dataset['issue_date'].apply(lambda date: date.weekday())
dataset['issue_date_week'] = dataset['issue_date'].apply(lambda date: date.week)

# The `companies.xz` dataset has businesses placed outside Brazilian area. We intentionally disconsider them.

# In[4]:

is_in_brazil = '(-73.992222 < longitude < -34.7916667) & (-33.742222 < latitude < 5.2722222)'
companies = pd.read_csv('../data/2016-09-03-companies.xz',
                        dtype={'cnpj': np.str},
                        low_memory=False)
companies = companies.query(is_in_brazil)
companies['cnpj'] = companies['cnpj'].str.replace(r'\D', '')
dataset = pd.merge(dataset, companies,
                   how='left',
                   left_on='cnpj_cpf',
                   right_on='cnpj',
                   suffixes=('', '_company'))

# In[5]:

dataset =     dataset.query('subquota_description == "Congressperson meal"')
companies =     companies[companies['cnpj'].isin(dataset.loc[dataset['cnpj'].notnull(),
                                                 'cnpj'])]

# In[6]:

dataset['total_net_value'].describe()

# In[7]:

dataset['total_net_value'].median()

# In[8]:

sns.distplot(dataset['total_net_value'],
             bins=30,
             kde=False)

# In[9]:

bottom_99 = dataset['total_net_value'].quantile(0.99)
bottom_99

# In[10]:

dataset[dataset['total_net_value'] < bottom_99].shape

# In[11]:

sns.distplot(dataset.loc[dataset['total_net_value'] < bottom_99, 'total_net_value'],
             bins=30,
             kde=False)

# In[12]:

bottom_99_dataset = dataset.query('total_net_value > {}'.format(bottom_99))
ranking = bottom_99_dataset.groupby('state_company')['total_net_value']     .median().sort_values(ascending=False)

sns.boxplot(x='state_company',
            y='total_net_value',
            data=bottom_99_dataset,
            order=ranking.index)

# In[13]:

bottom_99_dataset.query('state_company == "CE"').shape

# In[14]:

dataset.query('state_company == "CE"').shape

# In[15]:

bottom_99_dataset['state_company'].isnull().sum()

# In[16]:

bottom_99_dataset.query('state_company == "CE"')     .sort_values('total_net_value', ascending=False)

# ## Using Yelp to improve prices information

# In[17]:

yelp = pd.read_csv('../data/2016-11-29-yelp-companies.xz',
                   low_memory=False)
yelp.head()

# We have data for just 8.6% of the companies which received from the "Congressperson meal" subquota.

# In[18]:

yelp['price'].notnull().sum()

# In[19]:

companies.shape

# In[20]:

yelp['price'].isnull().sum()

# In[21]:

yelp['price.int'] = yelp['price'].str.len()
states_with_records =     yelp[yelp['price'].notnull()].groupby('location.state')['location.state'].count() > 10
states_with_records = states_with_records[states_with_records].index

# In[22]:

yelp_just_significant_states =     yelp[yelp['price'].notnull() &
         yelp['location.state'].isin(states_with_records)]
yelp_just_significant_states['location.state'].value_counts()

# Yelp won't be that useful for now, since we don't have a lot of data. Will leave it for another analysis.

# ## Predict prices

# The idea here is to try to predict the "right range for prices" in a specific place. If we can have a good accuracy, everything far from the prediction could be considered an outlier.

# In[23]:

bottom_99_dataset.iloc[0, :57]

# **DummyRegressor with mean strategy as a baseline**

# In[24]:

from sklearn.dummy import DummyRegressor
from sklearn.model_selection import train_test_split

X = bottom_99_dataset[['year']]
y = bottom_99_dataset['total_net_value']
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

model = DummyRegressor(strategy='mean')
model.fit(X_train, y_train)
model.score(X_test, y_test)

# In[25]:

from sklearn.preprocessing import LabelEncoder

le_state = LabelEncoder()
le_city = LabelEncoder()
factor_columns = ['state_company', 'city']
model_dataset = bottom_99_dataset.dropna(subset=factor_columns)
model_dataset['state_company'] = le_state.fit_transform(model_dataset['state_company'])
model_dataset['city'] = le_city.fit_transform(model_dataset['city'])

model_columns = ['cnpj',
                 'issue_date_day',
                 'issue_date_month',
                 'issue_date_year']
X = model_dataset[model_columns + factor_columns]
y = model_dataset['total_net_value']
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# **LinearRegression**
#
# Not very good accuracy.

# In[26]:

from sklearn.linear_model import LinearRegression

model = LinearRegression(n_jobs=-1)
model.fit(X_train, y_train)
model.score(X_test, y_test)

# What if we could improve it using the type of business as a feature? e.g. restaurant, hotel, grill...

# In[27]:

import unicodedata

def normalize_string(string):
    if isinstance(string, str):
        nfkd_form = unicodedata.normalize('NFKD', string.lower())
        return nfkd_form.encode('ASCII', 'ignore').decode('utf-8')

# In[28]:

import nltk
from sklearn.feature_extraction.text import CountVectorizer

stopwords = nltk.corpus.stopwords.words('portuguese')
count_vect = CountVectorizer(stop_words=stopwords)
trade_names = dataset.loc[dataset['supplier'].notnull(),
                          'supplier'].unique()
trade_names = np.vectorize(normalize_string)(trade_names)
dataset_counts = count_vect.fit_transform(trade_names)

# In[29]:

frequent_words = sorted(list(zip(count_vect.get_feature_names(),
    np.asarray(dataset_counts.sum(axis=0)).ravel())), key=lambda x: -x[1])

# In[30]:

frequent_words[:20]

# In[31]:

frequent_words = dict(frequent_words)

excluded_keywords = ['ltda', 'cia', 'epp']
[frequent_words.pop(keyword) for keyword in excluded_keywords]

# In[32]:

def business_type(name):
    fun = np.vectorize(lambda x: normalize_string(x))
    keywords = set(fun(name.split(' '))) - set(stopwords)
    key_freqs = list(map(lambda x: (x, frequent_words.get(x)), list(keywords)))
    key_freqs = [key_freq for key_freq in key_freqs if key_freq[1] is not None]
    if key_freqs:
        key_freq = max(key_freqs, key=lambda x: x[1])
        return key_freq[0]

dataset['supplier_keyword'] = dataset['supplier'].apply(business_type)
bottom_99_dataset['supplier_keyword'] =     bottom_99_dataset['supplier'].apply(business_type)

# In[33]:

le_state = LabelEncoder()
le_city = LabelEncoder()
le_supplier_keyword = LabelEncoder()
factor_columns = ['state_company', 'supplier_keyword']
model_dataset = bottom_99_dataset.dropna(subset=factor_columns)
model_dataset['state_company'] = le_state.fit_transform(model_dataset['state_company'])
model_dataset['city'] = le_city.fit_transform(model_dataset['city'])
model_dataset['supplier_keyword'] = le_city.fit_transform(model_dataset['supplier_keyword'])

model_columns = ['cnpj',
                 'issue_date_day',
                 'issue_date_month',
                 'issue_date_year']
X = model_dataset[model_columns + factor_columns]
y = model_dataset['total_net_value']
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# In[34]:

model = LinearRegression(n_jobs=-1)
model.fit(X_train, y_train)
model.score(X_test, y_test)

# Still not good enough. In fact, there's a serious mistake when considering this linear regression method for outlier detection: not just we're assuming that prices follow a linear regression (IMHO still good assumption, though), but outliers should be removed before running the regression. In other words, to find outliers, we must first remove them, having the regression score as just a accuracy estimator. May still be an interesting approach but we want to engage with something simpler first, to get the easier and most anomalous results.

# ## Common CNPJs
#
# Expenses in the same restaurant are expected to follow a normal distribution. Can we find outliers in companies with enough expenses to analyze?

# In[35]:

from scipy.stats import normaltest

def normaltest_pvalue(values):
    if len(values) >= 20:
        return normaltest(values).pvalue
    else:
        return 1

net_values_by_cnpj = dataset.groupby('cnpj_cpf')['total_net_value']     .agg([len, np.mean, np.std, normaltest_pvalue])     .sort_values('len', ascending=False)     .reset_index()
net_values_by_cnpj['threshold'] = net_values_by_cnpj['mean'] +     3 * net_values_by_cnpj['std']
applicants_per_cnpj = dataset.groupby('cnpj_cpf')['applicant_id']     .aggregate(lambda x: len(set(x))).reset_index()     .rename(columns={'applicant_id': 'congresspeople'})
net_values_by_cnpj = pd.merge(net_values_by_cnpj, applicants_per_cnpj)
net_values_by_cnpj.head()

# In[36]:

len(net_values_by_cnpj.query('normaltest_pvalue < .05')) / len(net_values_by_cnpj)

# In[37]:

data_with_threshold = pd.merge(dataset, net_values_by_cnpj, on='cnpj_cpf')     .sort_values('total_net_value', ascending=False)

# In[38]:

data_with_threshold['main_activity'] =     data_with_threshold['main_activity'].apply(normalize_string)

# Let's discard hotel reibursements. There's no method yet to discover for how long the congressperson stayed in the hotel, so we can expect a high standard deviation in their expenses. Even when detecting outliers, it's too hard to investigate manually given the same reason.
#
# Here, we just consider CNPJs where we have more than 20 reimbursements, from at least 3 congresspeople.

# In[39]:

is_hotel_reimbursement = data_with_threshold['main_activity']     .str.contains('hoteis').astype(np.bool)
outliers = data_with_threshold[~is_hotel_reimbursement]     .query('(congresspeople > 3) & (len >= 20) & (total_net_value > threshold)')
print(len(outliers), outliers['total_net_value'].sum())

# ## Foursquare
#
# Before totally discarding it, let's see how significant is the Foursquare dataset.

# In[40]:

foursquare = pd.read_csv('../data/2016-12-02-foursquare-companies.xz',
                         low_memory=False)
foursquare.head()

# In[41]:

foursquare.iloc[0]

# In[42]:

print(foursquare['price.tier'].notnull().sum(),
      foursquare['price.tier'].notnull().sum() / len(companies),
      foursquare.query('confirmed_match == True')['price.tier'].notnull().sum() / len(companies))

# ### Clustering for find the best group for a new restaurant

# In[43]:

companies.shape

# In[44]:

# is_cnpj = (dataset['cnpj_cpf'].str.len() == 14) & \
#     dataset['cnpj_cpf'].notnull() & \
#     dataset['document_type'] != 2
# cnpjs = dataset.sort_values('issue_date') \
#     .loc[is_cnpj, ['cnpj_cpf', 'supplier']] \
#     .drop_duplicates('cnpj_cpf', keep='last')
# cnpjs.head()

# In[45]:

is_cnpj = dataset['cnpj_cpf'].str.len() == 14
cnpj_list = dataset.loc[is_cnpj].groupby('cnpj')['total_net_value']     .agg([np.mean, np.std]).reset_index()
cnpj_list.shape

# In[46]:

cnpj_list.head()

# In[47]:

cnpj_list = pd.merge(cnpj_list,
                     dataset[['cnpj_cpf', 'supplier']].drop_duplicates('cnpj_cpf'),
                     how='left',
                     left_on='cnpj', right_on='cnpj_cpf')
del cnpj_list['cnpj_cpf']
cnpj_list.head()

# In[48]:

counts = dataset.loc[is_cnpj].groupby('cnpj')['applicant_id']     .agg({'congresspeople': (lambda x: len(np.unique(x))),
          'len': (lambda x: len(x))
         }).reset_index()

cnpj_list = pd.merge(cnpj_list, counts)

# **Calculate threshold for companies using their own receipts**

# In[49]:

threshold_for_cnpjs = cnpj_list.groupby('cnpj')     .apply(lambda x: x['mean'].mean() + 3 * x['std'].mean()).reset_index()     .rename(columns={0: 'threshold'})
threshold_for_cnpjs

cnpj_list = pd.merge(cnpj_list, threshold_for_cnpjs)
cnpj_list.head()

# In[50]:

HOTEL_REGEX = r'hote[l(eis)(ls)]'
is_hotel_reimbursement = (cnpj_list['supplier'].str.lower().str.contains(HOTEL_REGEX))

# Mark companies as having or not significant data.

# In[51]:

rows = (~is_hotel_reimbursement) &     (cnpj_list['congresspeople'] > 3) &     (cnpj_list['len'] > 20)
cnpj_list['has_significant_data'] = False
cnpj_list.loc[rows, 'has_significant_data'] = True

# In[52]:

print(cnpj_list['has_significant_data'].sum(),
      cnpj_list['has_significant_data'].sum() / len(cnpj_list['has_significant_data']))

# In[53]:

sns.lmplot('mean', 'std',
           data=cnpj_list.query('has_significant_data'),
           scatter_kws={'marker': 'D', 's': 100},
           size=10)

# **Predict threshold classifying companies in clusters by their price ranges**

# In[54]:

X = cnpj_list.loc[cnpj_list['has_significant_data'],
                  ['mean', 'std']]

# In[55]:

from sklearn.cluster import KMeans

model = KMeans(n_clusters=3, random_state=0)
model.fit(X)

# In[56]:

cnpj_list.loc[cnpj_list['has_significant_data'], 'y'] = model.predict(X)

# In[57]:

cnpj_list.query('y.notnull()').head()

# In[58]:

rows = (~cnpj_list['has_significant_data']) &     cnpj_list['std'].notnull() &     (~is_hotel_reimbursement)
X = cnpj_list.loc[rows, ['mean', 'std']]
cnpj_list.loc[rows, 'y'] = model.predict(X)

# In[59]:

threshold_for_groups = cnpj_list.groupby('y')     .apply(lambda x: x['mean'].mean() + 4 * x['std'].mean()).reset_index()     .rename(columns={0: 'threshold'})
threshold_for_groups

# In[60]:

group_thresholds = pd.merge(cnpj_list.query('~has_significant_data'),
                            threshold_for_groups,
                            on='y',
                            suffixes=('', '_group'))

cnpj_list = pd.merge(cnpj_list,
                     group_thresholds[['cnpj', 'threshold_group']],
                     how='left')
cnpj_list.loc[~cnpj_list['has_significant_data'], 'threshold'] =     cnpj_list['threshold_group']

# In[61]:

cnpj_list.query('(~has_significant_data) & std.notnull()').head()

# In[62]:

cnpj_list.query('has_significant_data').head()

# In[63]:

cnpj_list.query('threshold.notnull()').sample(5, random_state=10)

# In[64]:

del cnpj_list['threshold_group']

# In[65]:

merged = pd.merge(dataset, cnpj_list,
                  how='left',
                  left_on='cnpj_cpf',
                  right_on='cnpj',
                  suffixes=('', '_company'))

# In[66]:

merged['supplier'] = merged['supplier'].astype(np.str)
is_hotel_reimbursement =     (merged['supplier'].str.lower().str.contains(HOTEL_REGEX))

merged[~is_hotel_reimbursement].query('total_net_value > threshold').shape

# In[67]:

keys = ['year',
        'congressperson_name',
        'document_id',
        'total_net_value',
        'threshold',
        'cnpj_cpf',
        'has_significant_data',
        'name']

merged['diff'] = merged['threshold'] - merged['total_net_value']
merged[~(is_hotel_reimbursement | merged['has_significant_data'])]     .query('(total_net_value > threshold)')     .sort_values('diff', ascending=False).head(10)[keys]

# In[68]:

merged[~is_hotel_reimbursement].shape

# In[69]:

merged[~is_hotel_reimbursement]     .query('(total_net_value > threshold)')['total_net_value'].shape

# In[70]:

merged[~is_hotel_reimbursement]     .query('(total_net_value > threshold)')['total_net_value'].sum()

# In[71]:

merged[~is_hotel_reimbursement]     .query('(total_net_value > threshold) & (has_significant_data == False)')['total_net_value'].shape

# ## Conclusions
#
# For companies with significant data (defined by us as a company which received money at least 20x, from at least 3 distinct congresspeople), we use mean + 3 * std to detect outliers. Does not return all the suspect cases, but all of them, after some sampling investigation, seem to be very suspect.
#
# Since there's "significant data" just for 4% of the companies, we need a way for extrapolating the results for not so known ones. For doing so, we classify companies in 3 clusters using K-Means, considering mean and standard deviation of their prices as features. Once classified, we consider their threshold mean + 4 * stds of their clusters (one extra std compared to places where we have enough reimbursements to know better).
#
# Reimbursements made for expenses in hotels are discarded from this classifier, since they usually contain much more than meals (and we don't know for how long the congressperson was hosted in the place, not yet trustable for legal reports).

# In[ ]:


# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     formats: ipynb,py:light
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.4'
#       jupytext_version: 1.1.1
#   kernel_info:
#     name: python3
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # This notebook create the tests in python code. All this cells must be run to executed the tests

# %load_ext autoreload
# %autoreload 2

# + {"outputHidden": false, "inputHidden": false}
import sys
sys.path.append("../..")
# -

from optimus import Optimus
from optimus.helpers.test import Test

from optimus import Optimus
op = Optimus("dask", n_workers=1, threads_per_worker=8, processes=False, memory_limit="3G", comm=True)

import numpy as np
import pandas as pd

# +
import pandas as pd
from pyspark.sql.types import *
from datetime import date, datetime

cols = [
    ("names", "str"),
    ("height(ft)", ShortType()),
    ("function", "str"),
    ("rank", ByteType()),
    ("age", "int"),
    ("weight(t)", "float"),
    "japanese name",
    "last position seen",
    "date arrival",
    "last date seen",
    ("attributes", ArrayType(FloatType())),
    ("Date Type", DateType()),
    ("timestamp", TimestampType()),
    ("Cybertronian", BooleanType()),
    ("function(binary)", BinaryType()),
    ("NullType", NullType())

]

rows = [
    ("Optim'us", -28, "Leader", 10, 4000000, 4.30, ["Inochi", "Convoy"], "19.442735,-99.201111", "1980/04/10",
     "2016/09/10", [8.5344, 4300.0], date(2016, 9, 10), datetime(2014, 6, 24), True, bytearray("Leader", "utf-8"),
     None),
    ("bumbl#ebéé  ", 17, "Espionage", 7, 5000000, 2.0, ["Bumble", "Goldback"], "10.642707,-71.612534", "1980/04/10",
     "2015/08/10", [5.334, 2000.0], date(2015, 8, 10), datetime(2014, 6, 24), True, bytearray("Espionage", "utf-8"),
     None),
    ("ironhide&", 26, "Security", 7, 7000000, 4.0, ["Roadbuster"], "37.789563,-122.400356", "1980/04/10",
     "2014/07/10", [7.9248, 4000.0], date(2014, 6, 24), datetime(2014, 6, 24), True, bytearray("Security", "utf-8"),
     None),
    ("Jazz", 13, "First Lieutenant", 8, 5000000, 1.80, ["Meister"], "33.670666,-117.841553", "1980/04/10",
     "2013/06/10", [3.9624, 1800.0], date(2013, 6, 24), datetime(2014, 6, 24), True,
     bytearray("First Lieutenant", "utf-8"), None),
    ("Megatron", None, "None", 10, 5000000, 5.70, ["Megatron"], None, "1980/04/10", "2012/05/10", [None, 5700.0],
     date(2012, 5, 10), datetime(2014, 6, 24), True, bytearray("None", "utf-8"), None),
    ("Metroplex_)^$", 300, "Battle Station", 8, 5000000, None, ["Metroflex"], None, "1980/04/10", "2011/04/10",
     [91.44, None], date(2011, 4, 10), datetime(2014, 6, 24), True, bytearray("Battle Station", "utf-8"), None),
    (None, None, None, np.nan, None, None, None, None, None, None, None, None, None, None, None, None),

]

source_df = pd.DataFrame(columns = ['names','height(ft)','function',
                      'rank','age','weight(t)','japanese name','last position seen',
                      'date arrival','last date seen','attributes','Date Type','timestamp',
                      'Cybertronian','function(binary)','NullType'],
             data= rows,
#              dtype = (object,object,object,
#                                 object,object,object,object,object,
#                                 object,object,object,object,object,
#                                 object,object)
            )

# from dask import dataframe as dd

# a=pd.DataFrame(source_df.to_dict())
from dask import dataframe as dd
source_df = dd.from_pandas(source_df, npartitions=1)

# -

source_df.compute()

# ### End Init Section

# # Test

# ## Columns Test

t = Test(op, source_df, "df_cols_dask", imports=["import numpy as np",
                                            "nan = np.nan",
                                            "import datetime",], path="df_cols_dask", final_path="..")

# +
from pyspark.sql import functions as F

def func(col_name, attrs):
    return F.col(col_name) * 2

numeric_col = "height(ft)"
numeric_col_B = "rank"
numeric_col_C = "rank"
string_col = "function"
date_col = "date arrival"
date_col_B = "last date seen"
new_col = "new col"
array_col = "attributes"
# -

source_df.compute()

t.create(None, "cols.clip", "all_columns", "df", None,"*", 3, 5)

t.create(None, "cols.cast", "all_columns", "df", None, "*", "str")

t.create(None, "cols.is_na", "all_columns", "df", None, "*")

for i in source_df.compute().attributes:
    print(type(i[0]))

source_df.cols.unnest("attributes").compute()

source_df.compute()

source_df['height(ft)'].astype(str).str.split(".").compute()

import dask
source_df['height(ft)'].astype(str).str.split(".", expand=True, n=1)
# apply(dask.dataframe.Series, meta=("a")).compute()

source_df['attributes'].compute().apply(pd.Series)

source_df = source_df.ext.repartition(6)

source_df["date arrival"].astype(str).str.split("/", expand=True, n=2).compute()

source_df.compute()


#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# Documentation:
# - http://docs.astropy.org/en/stable/table/index.html#getting-started
# - http://www.astropy.org/astropy-tutorials/FITS-tables.html
# - http://www.astropy.org/astropy-tutorials/FITS-header.html

import argparse
from astropy.io import fits

import numpy as np
import astropy.table

# PARSE OPTIONS ###############################################################

parser = argparse.ArgumentParser(description="An astropy snippet")
parser.add_argument("filearg", nargs=1, metavar="FILE", help="the output FITS file")
args = parser.parse_args()
file_path = args.filearg[0]

# WRITE DATA ##################################################################

name_list = ("column1", "column2", "column3")
dtype_list = ("S128",  # column1 -> 128 bytes string
              "i4",    # column2 -> 4 bytes integer
              "f8")    # column3 -> 8 bytes float

table = astropy.table.Table(names=name_list,
                            dtype=dtype_list)

table.add_row(["A", 1, 1.1])
table.add_row(["B", 2, 2.2])
table.add_row(["C", 3, 3.3])

print(table)

table.write(file_path, overwrite=True)

import numpy,import astropy,from astropy


r"""
Module that creates and modifies parse trees of well formed boolean formulas.

A parse tree of a boolean formula is a nested list, where each branch is either
a single variable, or a formula composed of either two variables and a binary
operator or one variable and a unary operator. The function parse() produces
a parse tree that is simplified for the purposes of more efficient truth value
evaluation. The function polish_parse() produces the full parse tree of a boolean
formula which is used in functions related to proof and inference.  That is,
parse() is meant to be used with functions in the logic module that perform
semantic operations on a boolean formula, and polish_parse() is to be used with
functions that perform syntactic operations on a boolean formula.

AUTHORS:

- Chris Gorecki (2007): initial version

- Paul Scurek (2013-08-01): added polish_parse, cleaned up python code,
  updated docstring formatting

EXAMPLES:

Find the parse tree and variables of a string representation of a boolean formula::

    sage: import sage.logic.logicparser as logicparser
    sage: s = 'a|b&c'
    sage: t = logicparser.parse(s)
    sage: t
    (['|', 'a', ['&', 'b', 'c']], ['a', 'b', 'c'])

Find the full syntax parse tree of a string representation of a boolean formula::

    sage: import sage.logic.logicparser as logicparser
    sage: s = '(a&b)->~~c'
    sage: logicparser.polish_parse(s)
    ['->', ['&', 'a', 'b'], ['~', ['~', 'c']]]

Find the tokens and distinct variables of a boolean formula::

    sage: import sage.logic.logicparser as logicparser
    sage: s = '~(a|~b)<->(c->c)'
    sage: logicparser.tokenize(s)
    (['(', '~', '(', 'a', '|', '~', 'b', ')', '<->', '(', 'c', '->', 'c', ')', ')'], ['a', 'b', 'c'])

Find the parse tree of a boolean formula from a list of the formula's tokens::

    sage: import sage.logic.logicparser as logicparser
    sage: t = ['(', 'a', '->', '~', 'c', ')']
    sage: logicparser.tree_parse(t)
    ['->', 'a', ['~', 'c', None]]
    sage: r = ['(', '~', '~', 'a', '|', 'b', ')']
    sage: logicparser.tree_parse(r)
    ['|', 'a', 'b']

Find the full syntax parse tree of a boolean formula from a list of tokens::

    sage: import sage.logic.logicparser as logicparser
    sage: t = ['(', 'a', '->', '~', 'c', ')']
    sage: logicparser.tree_parse(t, polish = True)
    ['->', 'a', ['~', 'c']]
    sage: r = ['(', '~', '~', 'a', '|', 'b', ')']
    sage: logicparser.tree_parse(r, polish = True)
    ['|', ['~', ['~', 'a']], 'b']

"""
#*****************************************************************************
#       Copyright (C) 2007 Chris Gorecki <chris.k.gorecki@gmail.com>
#       Copyright (C) 2013 Paul Scurek <scurek86@gmail.com>
#
#  Distributed under the terms of the GNU General Public License (GPL)
#  as published by the Free Software Foundation; either version 2 of
#  the License, or (at your option) any later version.
#                  http://www.gnu.org/licenses/
#*****************************************************************************

from types import *
import string

__symbols = '()&|~<->^'
__op_list = ['~', '&', '|', '^', '->', '<->']

def parse(s):
    r"""
    Return a parse tree from a boolean formula s.

    INPUT:

    - ``s`` -- a string containing a boolean formula.

    OUTPUT:

    A list containing the prase tree and a list containing the
    variables in a boolean formula in this order:

    1. the list containing the pase tree
    2. the list containing the variables

    EXAMPLES:

    This example illustrates how to produce the parse tree of a boolean formula s.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: s = 'a|b&c'
        sage: t = logicparser.parse(s)
        sage: t
        (['|', 'a', ['&', 'b', 'c']], ['a', 'b', 'c'])
    """
    toks, vars_order = tokenize(s)
    tree = tree_parse(toks)
    # special case of tree == single variable
    if isinstance(tree, StringType):
        return ['&', tree, tree], vars_order
    return tree, vars_order

def polish_parse(s):
    r"""
    Return the full syntax parse tree from a boolean formula s.

    INPUT:

    - ``s`` -- a string containing a boolean expression

    OUTPUT:

    The full syntax parse tree as a nested list

    EXAMPLES:

    This example illustrates how to find the full syntax parse tree of a boolean formula.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: s = 'a|~~b'
        sage: t = logicparser.polish_parse(s)
        sage: t
        ['|', 'a', ['~', ['~', 'b']]]

    AUTHORS:

    - Paul Scurek (2013-08-03)
    """
    toks, vars_order = tokenize(s)
    tree = tree_parse(toks, polish = True)
    # special case where the formula s is a single variable
    if isinstance(tree, StringType):
        return vars_order
    return tree

def tokenize(s):
    r"""
    Return the tokens and the distinct variables appearing in a boolean formula s.

    INPUT:

    - ``s`` -- a string representation of a boolean formula

    OUTPUT:

    The tokens and variables as an ordered pair of lists in the following order:

    1. A list containing the tokens of s, in the order they appear in s
    2. A list containing the distinct variables in s, in the order they appearn in s

    EXAMPLES:

    This example illustrates how to tokenize a string representation of a boolean formula.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: s = 'a|b&c'
        sage: t = logicparser.tokenize(s)
        sage: t
        (['(', 'a', '|', 'b', '&', 'c', ')'], ['a', 'b', 'c'])
    """
    i = 0
    toks = ['(']
    vars_order = []

    while i < len(s):
        tok = ""
        skip = valid = 1
        if s[i] in '()~&|^':
            tok = s[i]
        elif s[i:i + 2] == '->':
            tok = '->'
            skip = 2
        elif s[i:i + 3] == '<->':
            tok = '<->'
            skip = 3
        # check to see if '-', '<' or '>' are used incorrectly
        elif s[i] in '<->':
            msg = "'%s' can only be used as part of the operators '<->' or '->'." % (s[i])
            raise SyntaxError, msg
        if len(tok) > 0:
            toks.append(tok)
            i += skip
            continue
        else:
            # token is a variable name
            if s[i] == ' ':
                i += 1
                continue

            while i < len(s) and s[i] not in __symbols and s[i] != ' ':
                tok += s[i]
                i += 1

            if len(tok) > 0:
                if tok[0] not in string.letters:
                    valid = 0
                for c in tok:
                    if c not in string.letters and c not in string.digits and c != '_':
                        valid = 0

            if valid == 1:
                toks.append(tok)
                if tok not in vars_order:
                    vars_order.append(tok)
            else:
                msg = 'invalid variable name ' + tok
                msg += ": identifiers must begin with a letter and contain only "
                msg += "alphanumerics and underscores"
                raise NameError, msg

    toks.append(')')
    return toks, vars_order

def tree_parse(toks, polish = False):
    r"""
    Return a parse tree from the tokens in toks.

    INPUT:

    - ``toks`` -- a list of tokens from a boolean formula

    - ``polish`` -- (default: False) a boolean.  When true, tree_parse will return
      the full syntax parse tree.

    OUTPUT:

    A parse tree in the form of a nested list that depends on ``polish`` as follows:

    polish == False -- Return a simplified parse tree.

    polish == True -- Return the full syntax parse tree.

    EXAMPLES:

    This example illustrates the use of tree_parse when polish == False.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: t = ['(', 'a', '|', 'b', '&', 'c', ')']
        sage: logicparser.tree_parse(t)
        ['|', 'a', ['&', 'b', 'c']]

    We now demonstrate the use of tree_parse when polish == True.

    ::

        sage: t = ['(', 'a', '->', '~', '~', 'b', ')']
        sage: logicparser.tree_parse(t)
        ['->', 'a', 'b']
        sage: t = ['(', 'a', '->', '~', '~', 'b', ')']
        sage: logicparser.tree_parse(t, polish = True)
        ['->', 'a', ['~', ['~', 'b']]]
    """
    stack = []
    for tok in toks:
        stack.append(tok)
        if tok == ')':
            lrtoks = []
            while tok != '(':
                tok = stack.pop()
                lrtoks.insert(0, tok)
            branch = parse_ltor(lrtoks[1:-1], polish = polish)
            stack.append(branch)
    return stack[0]

def parse_ltor(toks, n = 0, polish = False):
    r"""
    Return a parse tree from toks, where each token in toks is atomic.

    INPUT:

    - ``toks`` -- a list of tokens. Each token is atomic.

    - ``n`` -- (default: 0) an integer representing which order of
      operations are occurring

    - ``polish`` -- (default: False) a boolean.  When true, double negations
      are not cancelled and negated statements are turned into list of length two.

    OUTPUT:

    The parse tree as a nested list that depends on ``polish`` as follows:

    polish == False - Return a simplified parse tree.

    polish == True - Return the full syntax parse tree.

    EXAMPLES:

    This example illustrates the use of parse_ltor when polish == False.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: t = ['a', '|', 'b', '&', 'c']
        sage: logicparser.parse_ltor(t)
        ['|', 'a', ['&', 'b', 'c']]

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: t = ['a', '->', '~', '~', 'b']
        sage: logicparser.parse_ltor(t)
        ['->', 'a', 'b']

    We now repeat the previous example, but with polish == True.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: t = ['a', '->', '~', '~', 'b']
        sage: logicparser.parse_ltor(t, polish = True)
        ['->', 'a', ['~', ['~', 'b']]]

    """
    i = 0
    for tok in toks:
        if tok == __op_list[n]:
            if tok == '~':
                if not polish:
                    # cancel double negations
                    if toks[i] == '~' and toks[i + 1] == '~':
                        del toks[i]
                        del toks[i]
                        return parse_ltor(toks, n)
                    args = [toks[i], toks[i + 1], None]
                    toks[i] = args
                    del toks[i + 1]
                    return parse_ltor(toks, n)
                # This executes when creating the full syntax parse tree
                else:
                    j = i
                    while toks[j] == '~':
                        j += 1
                    while j > i:
                        args = [toks[j - 1], toks[j]]
                        toks[j - 1] = args
                        del toks[j]
                        j -= 1
                    return parse_ltor(toks, n = n, polish = polish)
            else:
                args = [toks[i - 1], toks[i], toks[i + 1]]
                toks[i - 1] = [args[1], args[0], args[2]]
                del toks[i]
                del toks[i]
                return parse_ltor(toks, n)
        i += 1
    if n + 1 < len(__op_list):
        return parse_ltor(toks, n + 1)
    if len(toks) > 1:
        raise SyntaxError
    return toks[0]

def apply_func(tree, func):
    r"""
    Apply func to each node of tree.  Return a new parse tree.

    INPUT:

    - ``tree`` -- a parse tree of a boolean formula

    - ``func`` -- a function to be applied to each node of tree.  This may
      be a function that comes from elsewhere in the logic module.

    OUTPUT:

    The new parse tree in the form of a nested list

    EXAMPLES:

    This example uses :func:`apply_func` where ``func`` switches two entries of tree.

    ::

        sage: import sage.logic.logicparser as logicparser
        sage: t = ['|', ['&', 'a', 'b'], ['&', 'a', 'c']]
        sage: f = lambda t: [t[0], t[2], t[1]]
        sage: logicparser.apply_func(t, f)
        ['|', ['&', 'c', 'a'], ['&', 'b', 'a']]
    """
    if type(tree[1]) is ListType and type(tree[2]) is ListType:
        lval = apply_func(tree[1], func)
        rval = apply_func(tree[2], func)
    elif type(tree[1]) is ListType:
        lval = apply_func(tree[1], func)
        rval = tree[2]
    elif type(tree[2]) is ListType:
        lval = tree[1]
        rval = apply_func(tree[2], func)
    else:
        lval = tree[1]
        rval = tree[2]
    return func([tree[0], lval, rval])


# coding: utf-8

# # Accessing ERDDAP from Python
#
# ERDDAP rich responses and RESTful API is makes it **THE** most convenient way to serve data.
#
# One can build URLs manually or programmatically like:
#
# <small>`https://erddap-uncabled.oceanobservatories.org/uncabled/erddap/tabledap/CP05MOAS-GL336-02-FLORTM000-flort_m_glider_instrument-telemetered-deployment0005-tabledap.csv?ctdgv_m_glider_instrument_sci_water_temp,time&time>=2017-02-10T00:00:00Z`</small>

# - server: `https://data.ioos.us/gliders/erddap/`
# - protocol: `tabledap`
# - dataset_id: `cp_336-20170817T1159`
# - variables: `time,latitude,longitude,temperature`
# - constraints:
#     - `time>=2017-10-11T00:00:00Z`
#     - `time<=2017-10-18T00:00:00Z`
#     - `latitude>=38.0`
#     - `latitude<=41.0`
#     - `longitude>=-72.0`
#     - `longitude<=-69.0`

# In[1]:

from erddapy import ERDDAP

server = 'https://data.ioos.us/gliders/erddap'

dataset_id = 'cp_336-20170817T1159'

constraints = {
    'time>=': '2017-10-11T00:00:00Z',
    'time<=': '2017-10-18T08:16:57Z',
    'latitude>=': 38.0,
    'latitude<=': 41.0,
    'longitude>=': -72.0,
    'longitude<=': -69.0
}

depth = 'depth'
salinity = 'salinity'
temperature = 'temperature'

variables = [
  depth,
 'latitude',
 'longitude',
  salinity,
  temperature,
 'time',
]

# In[2]:

e = ERDDAP(
    server=server,
    dataset_id=dataset_id,
    constraints=constraints,
    variables=variables,
    protocol='tabledap',
    response='mat',
)

print(e.get_download_url())

# # Obtaining the data
#
# There are a few methods to obtain the data with *to_pandas()* and *to_xarray()*:

# In[3]:

df = e.to_pandas(
    index_col='time',
    parse_dates=True,
    skiprows=(1,)  # units information can be dropped.
).dropna()

# In[4]:

df.head()

# # Let's plot the data

# # Exploring an ERDDAP server

# In[5]:

from erddapy import ERDDAP

e = ERDDAP(server='https://data.ioos.us/gliders/erddap')

# In[6]:

import pandas as pd

df = pd.read_csv(e.get_search_url(response='csv', search_for='all'))

# In[7]:

'We have {} tabledap, {} griddap, and {} wms endpoints.'.format(
    len(set(df['tabledap'].dropna())),
    len(set(df['griddap'].dropna())),
    len(set(df['wms'].dropna()))
)

# # ERDDAP Advanced Search
#
# Let's narrow the search area, time span, and look for *sea_water_temperature* only.

# In[8]:

bbox = [-72.0, -69.0, 38.0, 41.0]

min_time = '2018-02-01T00:00:00Z'
max_time = '2018-02-08T00:00:00Z'

kw = {
    'standard_name': 'sea_water_temperature',
    'search_for': 'glider',
    'min_lon': bbox[0],
    'max_lon': bbox[1],
    'min_lat': bbox[2],
    'max_lat': bbox[3],
    'min_time': min_time,
    'max_time': max_time,
    'cdm_data_type': 'trajectory'
}

# In[9]:

search_url = e.get_search_url(response='csv', **kw)
search = pd.read_csv(search_url)
gliders = search['Dataset ID'].values

msg = 'Found {} Glider Datasets:\n\n{}'.format
print(msg(len(gliders), '\n'.join(gliders)))

# With the Dataset IDs we can explore the metadata with the *get_info_url*

# In[10]:

print(gliders[0])

info_url = e.get_info_url(dataset_id=gliders[0], response='csv')
info = pd.read_csv(info_url)

info.head()

# In[11]:

cdm_profile_variables = info.loc[
    info['Attribute Name'] == 'cdm_profile_variables', 'Value'
]

print(''.join(cdm_profile_variables))

# # Selecting variables by attributes

# In[12]:

e.get_var_by_attr(
    dataset_id='cp_336-20180126T0000',
    standard_name='sea_water_temperature'
)

# # Easy to use CF conventions standards

# In[13]:

t_vars = [
    e.get_var_by_attr(
        dataset_id=glider, standard_name='sea_water_temperature'
    )[0] for glider in gliders
]
t_vars

# In[14]:

s_vars = [
    e.get_var_by_attr(
        dataset_id=glider, standard_name='sea_water_practical_salinity'
    )[0] for glider in gliders
]
s_vars

# In[15]:

d_vars = [
    e.get_var_by_attr(
        dataset_id=glider, standard_name='sea_water_pressure'
    )[0] for glider in gliders
]
d_vars

# In[16]:

# FIX: should not really assume that variables are the same for each dataset
depth = d_vars[0]
salinity = s_vars[0]
temperature = t_vars[0]

# # Putting everything together

# In[17]:

from requests.exceptions import HTTPError

constraints = {
    'time>=': min_time,
    'time<=': max_time,
    'longitude>=': bbox[0],
    'longitude<=': bbox[1],
    'latitude>=': bbox[2],
    'latitude<=': bbox[3]
}

def download_csv(url):
    return pd.read_csv(
        url, index_col='time', parse_dates=True, skiprows=[1]
    )

dfs = {}
for glider in gliders:
    try:
        download_url = e.get_download_url(
            dataset_id=glider,
            protocol='tabledap',
            variables=['time', 'latitude', 'longitude', depth, salinity, temperature],
            response='csv',
            constraints=constraints
        )
    except HTTPError:
        print('Failed to download {}'.format(glider))
        continue
    dfs.update({glider: download_csv(download_url)})

# In[18]:

import numpy as np

for glider in dfs.keys():
    dfs[glider].loc[dfs[glider][salinity] <= .1, salinity] = np.NaN
    dfs[glider].loc[dfs[glider][temperature] <= .1, temperature] = np.NaN

# In[19]:

import folium

zoom_start = 7
lon = (bbox[0] + bbox[1]) / 2
lat = (bbox[2] + bbox[3]) / 2
m = folium.Map(width='100%', height='100%',
               location=[lat, lon], zoom_start=zoom_start)

url = 'https://gis.ngdc.noaa.gov/arcgis/services/gebco08_hillshade/MapServer/WMSServer'
w = folium.WmsTileLayer(
    url,
    name='GEBCO Bathymetry',
    fmt='image/png',
    layers='GEBCO_08 Hillshade',
    attr='GEBCO',
    overlay=True,
    transparent=True)

w.add_to(m)

colors = ['orange','pink','yellow']

k=0
for glider, df in dfs.items():

    line = folium.PolyLine(locations=list(zip(df['latitude'],df['longitude'])),
                           color=colors[k],
                           weight=8,
                           opacity=0.6,
                           popup=glider[:22]).add_to(m)
    k = k+1

m

# In[20]:

import matplotlib.pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')

def glider_scatter(df, ax, glider):
    ax.scatter(df[temperature], df[salinity],
               s=10, alpha=0.5, label=glider)
fig, ax = plt.subplots(figsize=(12, 7))
ax.set_ylabel('salinity')
ax.set_xlabel('temperature')
ax.grid(True)

for glider, df in dfs.items():
    glider_scatter(df, ax, glider)
leg = ax.legend()

# ## Plot one of the glider transects

# In[21]:

df = next(iter(dfs.values()))

# In[22]:

import matplotlib.dates as mdates

fig, ax = plt.subplots(figsize=(17, 2))
cs = ax.scatter(df.index, df[depth], s=15, c=df[temperature], marker='o', edgecolor='none')

ax.invert_yaxis()
ax.set_xlim(df.index[0], df.index[-1])
xfmt = mdates.DateFormatter('%H:%Mh\n%d-%b')
ax.xaxis.set_major_formatter(xfmt)

cbar = fig.colorbar(cs, orientation='vertical', extend='both')
cbar.ax.set_ylabel('Temperature ($^\circ$C)')
ax.set_ylabel('Depth (m)');