#!/usr/bin/env python
# coding: utf-8

#
#
# # Earth Analytics Education - EA  Python Course Spring 2021

# ## Important  - Assignment Guidelines
#
# 1. Before you submit your assignment to GitHub, make sure to run the entire notebook with a fresh kernel. To do this first, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart & Run All)
# 2. Always replace the `raise NotImplementedError()` code with your code that addresses the activity challenge. If you don't replace that code, your notebook will not run.
#
# ```
# # YOUR CODE HERE
# raise NotImplementedError()
# ```
#
# 3. Any open ended questions will have a "YOUR ANSWER HERE" within a markdown cell. Replace that text with your answer also formatted using Markdown.
# 4. **DO NOT RENAME THIS NOTEBOOK File!** If the file name changes, the autograder will not grade your assignment properly.
# 6. When you create a figure, comment out `plt.show()` to ensure the autograder can grade your plots. For figure cells, DO NOT DELETE the code that says `DO NOT REMOVE LINE BELOW`.
#
# ```
# ### DO NOT REMOVE LINE BELOW ###
# student_plot1_ax = nb.convert_axes(plt)
# ```
#
# * Only include the package imports, code, and outputs that are required to run your homework assignment.
# * Be sure that your code can be run on any operating system. This means that:
#    1. the data should be downloaded in the notebook to ensure it's reproducible
#    2. all paths should be created dynamically using the `os.path.join`
#
# ## Follow to PEP 8 Syntax Guidelines & Documentation
#
# * Run the `autopep8` tool on all cells prior to submitting (HINT: hit shift + the tool to run it on all cells at once!
# * Use clear and expressive names for variables.
# * Organize your code to support readability.
# * Check for code line length
# * Use comments and white space sparingly where it is needed
# * Make sure all python imports are at the top of your notebook and follow PEP 8 order conventions
# * Spell check your Notebook before submitting it.
#
# For all of the plots below, be sure to do the following:
#
# * Make sure each plot has a clear TITLE and, where appropriate, label the x and y axes. Be sure to include UNITS in your labels.
#

# ### Add Your Name Below
# **Mike Petrut**

#

# ---

# # Week 04 and 05 Homework - Automate NDVI Workflow
#
# For this assignment, you will write code to generate a plot of the mean normalized difference vegetation index (NDVI) for two different sites in the United States across one year of data:
#
# * San Joaquin Experimental Range (SJER) in Southern California, United States
# * Harvard Forest (HARV) in the Northeastern United States
#
# The data that you will use for this week is available from **earthpy** using the following download:
#
# `et.data.get_data('ndvi-automation')`
#
# ## Assignment Goals
#
# Your goal in this assignment is to create the most efficient and concise workflow that you can that allows for:
#
# 1. The code to scale if you added new sites or more time periods to the analysis.
# 2. Someone else to understand your workflow.
# 3. The LEAST and most efficient (i.e. runs fast, minimize repetition) amount of code that completes the task.
#
# ### HINTS
#
# * Remove values outside of the landsat valid range of values as specified in the metadata, as needed.
# * Keep any output files SEPARATE FROM input files. Outputs should be created in an outputs directory that is created in the code (if needed) and/or tested for.
# * Use the functions that we demonstrated during class to make your workflow more efficient.
# * BONUS - if you  chose - you can export your data as a csv file. You will get bonus points for doing this.
#
#
# ## Assignment Requirements
#
# Your submission to the GitHub repository should include:
# * This Jupyter Notebook file (.ipynb) with:
#     * The code to create a plot of mean NDVI across a year for  2 NEON Field Sites:
#         * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object
#     * The **data should be cleaned to remove the influence of clouds**. See the [earthdatascience website for an example of what your plot might look like with and without removal of clouds](https://www.earthdatascience.org/courses/earth-analytics-python/create-efficient-data-workflows/).
# * BONUS: Create one output `.csv` file that has 3 columns - NDVI, Date and Site Name - with values for SJER and HARV.
#
# Your notebook should:
# * Have *at least* 2 well documented and well named functions with docstrings.
# * Include a Markdown cell at the top of the notebook that outlines the overall workflow using pseudocode (i.e. plain language, not code)
# * Include additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# # Replace this cell with your pseudocode  for this workflow
#
# If you happen to be a diagram person a diagram is ok too
#
#
***Workflow for Mean NDVI Modelling***

1. Get list of all directories and associated site names.

2. Open each site directory, get Landsat scenes, and calculate mean NDVI for each scene for that site.
   # Steps for calculating the mean NDVI for each Landsat Scene
   1. Create a loop that reads the site and date from each directory
   2. Sort the bands by number
   3. Crop to the extent of the focus shapefile
   4. Mask to the range of Landsat Values
   5. Optionally clean the image using the pixel cloud cover image

3. Capture results (including mean NDVI, date, and site name) to a list or dataframe.
   # Steps for sorting the table
   1. append all values from the looped tasks in Step 2
   2. Create pandas data frame, setting date as index

4. Export dataframe with mean NDVI values to csv.
# In[1]:

# Autograding imports - do not modify this cell
import matplotcheck.autograde as ag
import matplotcheck.notebook as nb
import matplotcheck.timeseries as ts
from datetime import datetime

# In[2]:

# Import necessary packages
import os
from glob import glob
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from shapely.geometry import box
import geopandas as gpd
import xarray as xr
import rasterio as rio
import rioxarray as rxr
from rasterio.plot import plotting_extent
from rasterio.mask import mask
import earthpy as et
import earthpy.spatial as es
import earthpy.plot as ep
from datetime import datetime
import matplotlib.dates as mdates

# Get the data
data = et.data.get_data('ndvi-automation')

# Set working directory
os.chdir(os.path.join(et.io.HOME, 'earth-analytics', 'data'))

# In[3]:

# DO NOT MODIFY THIS CELL
# Tests that the working directory is set to earth-analytics/data

path = os.path.normpath(os.getcwd())
student_wd_parts = path.split(os.sep)

if student_wd_parts[-2:] == ['earth-analytics', 'data']:
    print("\u2705 Great - it looks like your working directory is set correctly to ~/earth-analytics/data")
else:
    print("\u274C Oops, the autograder will not run unless your working directory is set to earth-analytics/data")

# # Figure 1: Plot 1 - Mean NDVI For Each Site Across the Year (50 points)
#
# Create a plot of the mean normalized difference vegetation index (NDVI) for the two different sites in the United States across the year:
#
# * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object.
# * Each site should be identified with a different color in the plot and legend.
# * The final plot **data should be cleaned to remove the influence of clouds**.
# * Be sure to include appropriate title and axes labels.
#
# Add additional cells as needed for processing data (e.g. defining functions, etc), but be sure to:
# * follow the instructions in the code cells that have been provided to ensure that you are able to use the sanity check tests that are provided.
# * include only the plot code in the cell identified for the final plot code below

# ## Task 1:
#
# In the cell below, create a single dataframe containing MEAN NDVI, the site name,
# and the date of the data for the HARV site
# scene `HARV/landsat-crop/LC080130302017031701T1-SC20181023151837`.  The column names for the  final
# DataFrame should be`mean_ndvi`, and `site`, and the data should be **indexed on the date**.
#
# Use the functions that we reviewed in class (or create your own versions of them) to implement your code
#
# ### In the Cell below Place  All Functions Needed to Run this Notebook (20 points)

# In[4]:

### DO NOT REMOVE THIS LINE OR EDIT / MOVE THIS CELL ###
start_time = datetime.now()

# In[5]:

# In this cell place all of the functions needed to run your notebook
# You will be graded here on function application, docstrings, efficiency so ensure
# All functions are placed here!

def combine_tifs(tif_list):

    """A function that combines a list of tifs in the same CRS
    and of the same extent into an xarray object

    Parameters
    ----------
    tif_list : list
        A list of paths to the tif files that you wish to combine.

    Returns
    -------
    An xarray object with all of the tif files in the listmerged into
    a single object.

    """
    out_xr = []
    for i, tif_path in enumerate(tif_list):
        out_xr.append(rxr.open_rasterio(tif_path, masked = True).squeeze())
        out_xr[i]["band"] = i + 1

    return xr.concat(out_xr, dim = "band")

def ndvi_mean(main_path, cloud_mask):

    """A function that takes a file path of Landsat imagery (which includes
    separate .tif files for different bands), and combines the files as an
    xarray, then calculates the NDVI.

    The function includes the following tasks:

    1) Read in the path, sort the files in each imagery file by band number
    2) Find the cropping extent file for each site
       (assuming this is a shapefile)
    3) Combine imagery as a xarray and apply the pixel mask
       if cloud mask is entered as True
    4) Mask the file to the Landsat range 0:10000
    5) Calculate the NDVI and Mean NDVI values
    6) Append the loop results to the list
    7) Summarise the appended lists ass a pandas data frame,
       convert the string extracted date as
       date-time and assign to the index
    8) Return panel data frame of mean NDVI values,
       with key = site and index = date

    The function finally calculates the mean for all file paths and combines them as a pandas
    data frame.

    The function extracts the site and date from the Landsat path file and adds
    them as columns to the table, the

    Parameters
    ----------

    Path_list : list
        A list of paths to the paths of tif files that you wish to combine.

    cloud_mask : logical
        True or False based on whether you want the function
        to apply the cloud cover layer to clean the data

    Returns
    -------
    A pandas dataframe of mean NDVI values from the main path provided

    """

    #Create ists to append looped values to
    all_mean = []
    dates = []
    sites = []

    # Specify the valid range of values for landsat
    valid_range = (0, 10000)

    # Cloud no data vals for Landsat 8 -
    vals = [328, 392, 840, 904, 1350, 352, 368, 416,
            432, 480, 864, 880, 928, 944, 992, 480, 992]

    paths = glob(os.path.join(main_path, "*/"))

    for path in paths:

        # Extract site from path
        site = os.path.basename(os.path.normpath(path))

        # Extract list of imigery file paths
        band_files = glob(os.path.join(path, 'landsat-crop', "*/"))

        # Isolate path to shapefile needed for cropping
        crop_path = glob(os.path.join(path, 'vector', "*.shp"))[0]

        # Import the cropping shapefile
        crop_shape = gpd.read_file(crop_path)

        # Create loop to calculate for all imigery files in the site folders
        for files in band_files:

            # Sort files by band
            sorted_files = sorted(glob(os.path.join(files, '*band*')))

            # Use combined_tif fuction to crease an xarray from the sorted tif files
            combined_tif = combine_tifs(sorted_files)

            # Crop the newly combined imagery file to the cropped shape
            ## Each site has one cropping shape file, hence why it is defined in the path loop not the file loop
            combined_crop = combined_tif.rio.clip(crop_shape.geometry, from_disk = True).squeeze()

            # Define cropped xarray as either filtered to the pixel could cover imagery or not

            if(cloud_mask == True):

                pixel_qa_path = glob(os.path.join(files, "*qa*"))
                pixel_qa = rxr.open_rasterio(pixel_qa_path[0],
                                             masked = True).rio.clip(crop_shape.geometry,
                                                                     from_disk = True).squeeze()
                combined_crop = combined_crop.where(~pixel_qa.isin(vals))

            else: combined_crop

            # Mask the cropped range to the Landsat 8 values range
            if valid_range:
                mask = ((combined_crop < valid_range[0]) | (combined_crop > valid_range[1]))
                combined_crop = combined_crop.where(~xr.where(mask, True, False))

            # Calculate the NDVI -
            ## Function will not run properly if this is changed the the earthpy normalized_diff fuction - Do not change
            ndvi = (combined_crop[4] - combined_crop[3]) / (combined_crop[4] + combined_crop[3])

            # Calc the mean NDVI
            ndvi_mean = np.nanmean(ndvi)

            # Isolate filename from the path
            file_name = os.path.basename(os.path.normpath(files))

            # Extract date values from the file name string
            date = file_name[10:18]

            #image - file_name[1:18]

            # Append looped values to the predefined lists
            dates.append(date)
            sites.append(site)
            all_mean.append(ndvi_mean)

            # Generate pandas dataframe from the three lists
            ndvi_mean_df = pd.DataFrame({'date': dates,
                                         'site': sites,
                                         'mean_ndvi': all_mean})

    # Coerce date string to date-time values
    ndvi_mean_df['date'] = pd.to_datetime(ndvi_mean_df['date'], format = '%Y%m%d')

    # Set index as the date values
    ndvi_mean_df.set_index('date', inplace = True)

    return ndvi_mean_df

def ndvi_filter_date(image_directory):

    """A function that extracts the date of image collection from the imagery path name

    Parameters
    ----------
    directory: directory to file including study
    site e.g: "'HARV/landsat-crop/LC080130302017031701T1-SC20181023151837'"

    Returns
    -------
    Date string of the image directory

    """
    filter_path = image_directory
    filter_year = filter_path[28:32]
    filter_month = filter_path[32:34]
    filter_day = filter_path[34:36]

    filter_date = filter_year + "-" + filter_month + "-" + filter_day

    return filter_date

def ndvi_filter_site(image_directory):

    """A function that extracts the site name from the imagery path name

    Parameters
    ----------
    directory: directory to file including study
    site e.g: "'HARV/landsat-crop/LC080130302017031701T1-SC20181023151837'"

    Returns
    -------
    Date string of the image site

    """
    filter_path = image_directory
    filter_site = filter_path[0:4]

    return filter_site

# In[6]:

# Create dataframe of mean NDVI in this cell using the functions created above
# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Call the dataframe at the end of the cell so the tests run on it!
# Be sure that the date column is an index of type date
# HINT: the time series lessons may help you remember how to do this!

sanity_date = ndvi_filter_date('HARV/landsat-crop/LC080130302017031701T1-SC20181023151837')

sanity_site = ndvi_filter_site('HARV/landsat-crop/LC080130302017031701T1-SC20181023151837')

main_path = os.path.join('ndvi-automation', 'sites')

ndvi_mean_results = ndvi_mean(main_path, cloud_mask = False)

ndvi_mean_results[(ndvi_mean_results.index == sanity_date) & (ndvi_mean_results['site'] == sanity_site)]

# In[7]:

# This cell  is testing your data output above

student_ndvi_ts_single_site = _

single_scene_points = 0

# Ensure the data is stored in a dataframe.
if isinstance(student_ndvi_ts_single_site, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    single_scene_points += 1
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Ensure that the date column is the index
if isinstance(student_ndvi_ts_single_site.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    single_scene_points += 2
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_ts_single_site.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    single_scene_points += 2
else:
    print('\u274C The data in your date column is not datetime.')

# Ensure the site name is correct
if student_ndvi_ts_single_site.site.values[0] == 'HARV':
    print('\u2705 You have the correct site name!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct site name.')

if np.allclose(0.281131628228094, student_ndvi_ts_single_site.mean_ndvi.values[0]):
    print('\u2705 You have the correct mean NDVI value!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct mean ndvi value.')

print("\n \u27A1 You received {} out of 15 points for creating a dataframe.".format(
    single_scene_points))
single_scene_points

# ## Task 2:
#
# In the cell below, process all of the landsat scenes. Create a DataFrame that contains the following
# information for each scene
#
#
# |   | index  | site  | mean_ndvi  |
# |---|---|---|---|
# | Date  |   |   |   |
# |  2017-01-07  | 0  | SJER  | .4  |
#
# Be sure to call your dataframe at the end of the cell to ensure autograding works.
# HINT: FOR THIS STEP, leave any rows containing missing values (`NAN`).

# In[8]:

# Create dataframe of NDVI including the cleaning data to deal with clouds
# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Don't forget to set date as the index and make the values of type datetime

###__________________________________________________________

# Call
ndvi_mean_cloud_cover = ndvi_mean(main_path, cloud_mask = True)
ndvi_mean_cloud_cover

# In[9]:

# Last sanity check before creating your plot (10 points)

# Ensure that you call your dataframe at the bottom of the cell above
# and that it has columns called: mean_ndvi and site

# Ensure the data is stored in a dataframe.
student_ndvi_df = _

df_points = 0

if isinstance(student_ndvi_df, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    df_points +=2
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Check that dataframe contains the appropriate number of NAN values
if student_ndvi_df.mean_ndvi.isna().sum() == 15:
    print('\u2705 Correct number of masked data values!')
    df_points +=2
else:
    print('\u274C The amount of null data in your dataframe is incorrect.')

# Ensure that the date column is the index
if isinstance(student_ndvi_df.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    df_points +=3
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_df.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    df_points +=3
else:
    print('\u274C The data in your date column is not datetime.')

# Output for timer, # DO NOT MODIFY
end_time = datetime.now()
total_time = end_time - start_time
print(
    "Your total run time for processing the data was {0}.".format(total_time))

print("\n \u27A1 You received {} out of 10 points for creating a dataframe.".format(
    df_points))

df_points

# In[10]:

# Add only the plot code to this cell

# This is the final figure of mean NDVI
# for both sites across the year
# with data cleaned to deal with clouds

###___________________________AX1______________________________

# Remove NaNs from cleaned dataset
ndvi_mean_cloud_cover_exnan = ndvi_mean_cloud_cover.dropna()

# Set Colors
color_map = {'HARV' : '#7BBDD6',
             'SJER' : '#140034'}

months = mdates.MonthLocator()  # every month

fig, (ax1,ax2) = plt.subplots(2, 1,
                              figsize = (12,15),
                              constrained_layout = False)

for label, grp in ndvi_mean_results.groupby("site"):
    grp.plot(use_index = True,
             y = 'mean_ndvi',
             ax = ax1,
             color = color_map[label],
             label = label,
             style = '.-',
             markersize = 12)

# Set up formatting for graph

ax1.set_xlabel('Date', fontsize=16)
ax1.set_ylabel('Mean NDVI', fontsize=16)
ax1.set_title('Mean Normalized Difference Vegetation Index (NDVI) \n Jan 2017- Dec-2017 \n Landsat 8 Not Cleaned',
              fontsize = 18)

ax1.legend(fontsize = 12)

ax1.xaxis.set_major_formatter(mdates.DateFormatter('%b'))

ax1.tick_params(axis = 'both',
               which = 'major',
               labelsize = 12)

ax1.set_ylim([0, 1])

ax1.grid(True)

###____________________________AX2______________________________

for label, grp in ndvi_mean_cloud_cover_exnan.groupby("site"):
    grp.plot(use_index = True,
             y = 'mean_ndvi',
             ax = ax2,
             color = color_map[label],
             label = label,
             style = '.-',
             markersize = 12)

# Set title and label axes
ax2.set(xlabel = "Date",
        ylabel = "Mean NDVI",
        title = "NDVI")

# Set up formatting for graph

ax2.set_xlabel('Date', fontsize = 16)
ax2.set_ylabel('Mean NDVI', fontsize = 16)
ax2.set_title('Mean Normalized Difference Vegetation Index (NDVI) \n Jan 2017- Dec-2017 \n Landsat With Clouds Removed',
              fontsize = 18)

ax2.legend(fontsize = 12)

ax2.xaxis.set_major_formatter(mdates.DateFormatter('%b %y'))

ax2.tick_params(axis = 'both',
               which = 'major',
               labelsize = 12)

ax2.set_ylim([0, 1])

ax2.grid(True)

fig.tight_layout(pad = 2.5)

### DO NOT REMOVE LINES BELOW ###
final_masked_solution = nb.convert_axes(plt, which_axes="current")

# In[11]:

# Ignore this cell for the autograding tests

# In[12]:

# Ignore this cell for the autograding tests

# # Question 1 (10 points)
#
# Imagine that you are planning NEON’s upcoming flight season to capture remote sensing data in these locations and want to ensure that you fly the area when the vegetation is the most green.
#
# When would you recommend the flights take place for each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# For the HARV study area it would be recommended that to avoid cloud cover and gather high vegetation imagery that the flight taken place late-August to September.
#
# For the SJER study area it would be recommended that to avoid cloud cover and gather high vegetation imagery that the flight taken place late-Feb to March.

# # Question 2 (10 points)
#
# How could you modify your workflow to look at vegetation changes over time in each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# To better understand the vegetation changes over time we could modify the workflow to classify each NDVI model to categorise the xarray into bins for not-vegetation (-1 to 0.19), low vegetation (0. 2 to 0.5) and high vegetation (0.501 to 1.0) and calculate both the mean and total observations of he three bins to see where areas of land have maintained lose or gained vegetation across the categorisations. This would be useful to understand the impact of seasonal variables such as weather and rainfall on the vegetation index over time.

# # Do not edit this cell! (10 points)
#
# The notebook includes:
# * additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# # Do not edit this cell! (20 points)
#
# The notebook will also be checked for overall clean code requirements as specified at the **top** of this notebook. Some of these requirements include (review the top cells for more specifics):
#
# * Notebook begins at cell [1] and runs on any machine in its entirety.
# * PEP 8 format is applied throughout (including lengths of comment and code lines).
# * No additional code or imports in the notebook that is not needed for the workflow.
# * Notebook is fully reproducible. This means:
#    * reproducible paths using the os module.
#    * data downloaded using code in the notebook.
#    * all imports at top of notebook.

# ## BONUS - Export a  .CSV File to Share (10 points possible)
#
# This is optional - if you export a **.csv** file with the columns specified above: Site, Date and NDVI Value you can get an additional 10 points.
#
# * FULL CREDIT: File exists in csv format and contains the columns specified.
# We will check your github repo for this file!
#

# In[13]:

### CSV EXPORT ###

# Set working directory
os.chdir(os.path.join(et.io.HOME,
                      'earth-analytics',
                      'ea-2021-04-ndvi-automation-mike-petrut',
                      'csv_output'))

ndvi_mean_results.to_csv('ndvi_mean_no_clean.csv', index = True)

ndvi_mean_cloud_cover_exnan.to_csv('ndvi_mean_clean.csv', index = True)

# Revert working directory
os.chdir(os.path.join(et.io.HOME, 'earth-analytics', 'data'))


# Databricks notebook source
# MAGIC %md
# MAGIC # HW 5 - Page Rank
# MAGIC __`MIDS w261: Machine Learning at Scale | UC Berkeley School of Information | Fall 2018`__
# MAGIC
# MAGIC In Weeks 8 and 9 you discussed key concepts related to graph based algorithms and implemented SSSP.
# MAGIC In this final homework assignment you'll implement distributed PageRank using some data from Wikipedia.
# MAGIC By the end of this homework you should be able to:
# MAGIC * ... __compare/contrast__ adjacency matrices and lists as representations of graphs for parallel computation.
# MAGIC * ... __explain__ the goal of the PageRank algorithm using the concept of an infinite Random Walk.
# MAGIC * ... __define__ a Markov chain including the conditions underwhich it will converge.
# MAGIC * ... __identify__ what modifications must be made to the web graph inorder to leverage Markov Chains.
# MAGIC * ... __implement__ distributed PageRank in Spark.
# MAGIC
# MAGIC __Please refer to the `README` for homework submission instructions and additional resources.__

# COMMAND ----------

# MAGIC %md
# MAGIC Jeff Li, Sonya Chen, Karthik Srinivasan, Justin Trobec

# COMMAND ----------

# MAGIC %md
# MAGIC # Notebook Set-Up
# MAGIC Before starting your homework run the following cells to confirm your setup.

# COMMAND ----------

# imports
import re
import ast
import time
import numpy as np
import pandas as pd

import seaborn as sns
import networkx as nx
import matplotlib.pyplot as plt

# COMMAND ----------

# MAGIC %md
# MAGIC ### Run the next cell to create your directory in dbfs
# MAGIC You do not need to understand this scala snippet. It simply dynamically fetches your user directory name so that any files you write can be saved in your own directory.

# COMMAND ----------

# RUN THIS CELL AS IS
# This code snippet reads the user directory name, and stores is in a python variable.
# Next, it creates a folder inside your home folder, which you will use for files which you save inside this notebook.
username = dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply('user')
userhome = 'dbfs:/user/' + username
print(userhome)
hw5_path = userhome + "/HW5/"
hw5_path_open = '/dbfs' + hw5_path.split(':')[-1] # for use with python open()
dbutils.fs.mkdirs(hw5_path)

# COMMAND ----------

# RUN THIS CELL AS IS.
tot = 0
DATA_PATH = 'dbfs:/mnt/mids-w261/HW5/'
for item in dbutils.fs.ls(DATA_PATH):
  tot = tot+item.size
tot
# ~4.7GB

# COMMAND ----------

# RUN THIS CELL AS IS. You should see all-pages-indexed-in.txt, all-pages-indexed-out.txt and indices.txt in the results. If you do not see these, please let an Instructor or TA know.
display(dbutils.fs.ls(DATA_PATH))

# COMMAND ----------

# RUN THIS CELL AS IS - A test to make sure your directory is working as expected.
# You should see a result like:
# dbfs:/user/youremail@ischool.berkeley.edu/HW5/test.txt
dbutils.fs.put(hw5_path+'test.txt',"hello world",True)
display(dbutils.fs.ls(hw5_path))

# COMMAND ----------

sc = spark.sparkContext
spark

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 1: Distributed Graph Processing
# MAGIC Chapter 5 from Lin & Dyer gave you a high level introduction to graph algorithms and concernts that come up when trying to perform distributed computations over them. The questions below are designed to make sure you captured the key points from this reading and your async lectures.
# MAGIC
# MAGIC ### Q1 Tasks:
# MAGIC
# MAGIC * __a) short response:__ Give an example of a dataset that would be appropriate to represent as a graph. What are the nodes/edges in this dataset? Is the graph you describe 'directed' or 'undirected'? What would the average "in-degree" of a node mean in the context of your example?
# MAGIC
# MAGIC * __b) short response:__ Other than their size/scale, what makes graphs uniquely challenging to work with in the map-reduce paradigm? *(__HINT__: Do not respond in terms of any specific algorithm. Think in terms of the nature of the graph datastructure itself).*
# MAGIC
# MAGIC * __c) short response:__ Briefly describe Dijskra's algorithm (goal/approach). What specific design component makes this approach hard to parallelize?
# MAGIC
# MAGIC * __d) short response:__ How does parallel breadth-first-search get around the problem that you identified in part `c`? At what expense?

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q1 Student Answers:
# MAGIC
# MAGIC > __1a)__ Give an example of a dataset that would be appropriate to represent as a graph.
# MAGIC > What are the nodes/edges in this dataset? Is the graph you describe 'directed' or 'undirected'?
# MAGIC > What would the average "in-degree" of a node mean in the context of your example?
# MAGIC > * For example, the relationship of twitter users would be appropriate to represent as a graph.
# MAGIC > * In the twitter users example, nodes are users, and their relationship is the edges.
# MAGIC > * In the twitter example, the edge is directed. The reason is that in twitter, userA can follow userB, but userB does not necessarily follow userA.
# MAGIC > * Say UserA follows userA, then the edge starts at UserA and has the arrow pointing toward userB. And userA is a follower of UserB in this case.
# MAGIC > * In-degree represents the number of incoming neighbors, or followers.
# MAGIC > * The average "in-degree" of a node is the average number of followers a user has.
# MAGIC
# MAGIC
# MAGIC > __1b)__ Other than their size/scale, what makes graphs uniquely challenging to work with in the map-reduce paradigm?
# MAGIC > * It is challenging for map-reduce. The reason is a node could have many neighbors, and thus the results of a mapper might need to arrive at different reducers.
# MAGIC > * Say if we have 10 mappers, and each mapper produces 10 results, and each of these 10 results need to arrive at different reducers.
# MAGIC > * With 10 mappers, we might have 100 results that need to do a large amount of shuffling and combing before we can do the next step of computation.
# MAGIC > * And that shuffling and combining is expensive as data traffic is costly.
# MAGIC > * In addition, for graph computation, we have to do matrix multiplication. And if the matrix is very big, and matrix multiplication is expensive, a single machine might not be able to fit the all of that.
# MAGIC > * One thing that can make graphs challenging to work with is that there are not necessarily natural partitionings of them. In other words, paths an algorithm may take through the graph might make it impossible to assign some nodes to an individual partition of the dataset.
# MAGIC
# MAGIC
# MAGIC > __c)__ Briefly describe Dijskra's algorithm (goal/approach). What specific design component makes this approach hard to parallelize?
# MAGIC >> * Dijstka is the algorithm's goal is to find the shortest path from starting position to the destination/all other nodes.
# MAGIC >> * The approach Dijkstra take is:
# MAGIC >> * The approach is we always pick the next node (Frontier) that is unvisited and has the shortest distance from the current node (with the info we have up until this point).
# MAGIC >> * When we are at the next node, we again update the distance of all of its neighboring nodes.
# MAGIC >> * Then again, we pick the next unvisited node with the shortest distance, and go there.
# MAGIC >> * We do so until we reach our destination node.
# MAGIC >> * When we pick which node to go next, we need to know the latest information of all the neighboring node. And the next node will update the information of neighbor nodes. And that update decides which node we go next again.
# MAGIC >> * In Dijkstra's algorithm, we proceed to find the shortest paths by traversing one node at a time. This is done sequentially using a priority queue and works well on a single node computer. However, this does not make it amenable to parallelization.
# MAGIC >> * In Dijkstra, the order of node visits matter! Because that ensures that we use the least cost to find the shortest path.
# MAGIC >> * Thus in this approach, we can ONLY VISIT one node at a time. That means we cannot compute the results for several nodes at the same time.
# MAGIC
# MAGIC
# MAGIC >__d)__ How does parallel breadth-first-search get around the problem that you identified in part `c`? At what expense?
# MAGIC >> * The parallel breadth first search enables us to use a FIFO queue instead of the priority queue as seen in Dijkstra. In parallel BFS, the nodes on the same level can be processed at the same time.
# MAGIC >> * If visiting a node has some cost associated with it, then the parallel breath-first-search will not minimize the complexity of finding the lowest cost. The additional complexity arises when a visited node is put back into the frontier queue. This happens when the aforementioned visited node has a newly computed shorter/smaller cost. This is an expensive operation since it explores all paths.

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 2: Representing Graphs
# MAGIC
# MAGIC In class you saw examples of adjacency matrix and adjacency list representations of graphs. These data structures were probably familiar from HW3, though we hadn't before talked about them in the context of graphs. In this question we'll discuss some of the tradeoffs associated with these representations. __`NOTE:`__ We'll use the graph from Figure 5.1 in Lin & Dyer as a toy example. For convenience in the code below we'll label the nodes `A`, `B`, `C`, `D`, and `E` instead of \\(n_1\\), \\(n_2 \\), etc but otherwise you should be able to follow along & check our answers against those in the text.
# MAGIC
# MAGIC
# MAGIC <img src="https://github.com/kyleiwaniec/w261_assets/blob/master/images/HW5/Lin-Dyer-graph-Q1.png?raw=true" width=50%>
# MAGIC
# MAGIC ### Q2 Tasks:
# MAGIC
# MAGIC * __a) short response:__ Relatively speaking, is the graph you described in Figure 5.1 in Lin & Dyer "sparse" or "dense"?  Explain how sparsity/density impacts the adjacency matrix and adjacency list representations of a graph.
# MAGIC
# MAGIC * __b) short response:__ Run the provided code to create and plot our toy graph. Is this graph directed or undirected? Explain how the adjacency matrices for directed graphs will differ from those of undirected graphs.
# MAGIC
# MAGIC * __c) code:__ Fill in the missing code to complete the function `get_adj_matr()`.
# MAGIC
# MAGIC * __d) code:__ Fill in the missing code to complete the function `get_adj_list()`.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q2 Student Answers:
# MAGIC > __a)__ This is relatively dense, compared to a graph like the world-wide-web. Sparse graphs will have a much smaller adjacency list representation compared to their matrix representation, as the matrix has to have an entry even for non-existent edges.
# MAGIC
# MAGIC > __b)__ The graph is directed. An adjacency matrix for an undirected graph will be symetric, while a directed graph's adjacency matrix will not.

# COMMAND ----------

# part a - a graph is just a list of nodes and edges (RUN THIS CELL AS IS)
TOY_GRAPH = {'nodes':['A', 'B', 'C', 'D', 'E'],
             'edges':[('A', 'B'), ('A', 'D'), ('B', 'C'), ('B', 'E'), ('C', 'D'),
                      ('D', 'E'), ('E', 'A'),('E', 'B'), ('E', 'C')]}

# COMMAND ----------

# part a - simple visualization of our toy graph using nx (RUN THIS CELL AS IS)
G = nx.DiGraph()
G.add_nodes_from(TOY_GRAPH['nodes'])
G.add_edges_from(TOY_GRAPH['edges'])
display(nx.draw(G, pos=nx.circular_layout(G), with_labels=True, alpha = 0.5))

# COMMAND ----------

# part c - adjacency matrix function
def get_adj_matr(graph):
    """
    Function to create an adjacency matrix representation of a graph.
    arg:
        graph - (dict) of 'nodes' : [], 'edges' : []
    returns:
        pd.DataFrame with entry i,j representing an edge from node i to node j
    """
    n = len(graph['nodes'])
    adj_matr = pd.DataFrame(0, columns = graph['nodes'], index = graph['nodes'])
    ############### YOUR CODE HERE ##################
    for node1, node2 in graph['edges']:
      adj_matr[node2][node1] = 1.0
    ############### (END) YOUR CODE #################
    return adj_matr

# COMMAND ----------

# part c - take a look (RUN THIS CELL AS IS)
TOY_ADJ_MATR = get_adj_matr(TOY_GRAPH)
print(TOY_ADJ_MATR)

# COMMAND ----------

# part d - adjacency list function
def get_adj_list(graph):
    """
    Function to create an adjacency list representation of a graph.
    arg:
        graph - (dict) of 'nodes' : [], 'edges' : []
    returns:
        dictionary of the form {node : [list of edges]}
    """
    adj_list = {node: [] for node in graph['nodes']}
    ############### YOUR CODE HERE ##################
    for node1, node2 in graph['edges']:
      adj_list[node1].append(node2)
    ############### (END) YOUR CODE #################
    return adj_list

# COMMAND ----------

# part d - take a look (RUN THIS CELL AS IS)
TOY_ADJ_LIST = get_adj_list(TOY_GRAPH)
print(TOY_ADJ_LIST)

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 3: Markov Chains and Random Walks
# MAGIC
# MAGIC As you know from your readings and in class discussions, the PageRank algorithm takes advantage of the machinery of Markov Chains to compute the relative importance of a webpage using the hyperlink structure of the web (we'll refer to this as the 'web-graph'). A Markov Chain is a discrete-time stochastic process. The stochastic matrix has a principal left eigen vector corresponding to its largest eigen value which is one. A Markov chain's probability distribution over its states may be viewed as a probability vector. This steady state probability for a state is the PageRank of the corresponding webpage. In this question we'll briefly discuss a few concepts that are key to understanding the math behind PageRank.
# MAGIC
# MAGIC ### Q3 Tasks:
# MAGIC
# MAGIC * __a) short response:__ It is common to explain PageRank using the analogy of a web surfer who clicks on links at random ad infinitum. In the context of this hypothetical infinite random walk, what does the PageRank metric measure/represent?
# MAGIC
# MAGIC * __b) short response:__ What is the "Markov Property" and what does it mean in the context of PageRank?
# MAGIC
# MAGIC * __c) short response:__ A Markov chain consists of \\(n\\) states plus an \\(n\times n \\) transition probability matrix. In the context of PageRank & a random walk over the WebGraph what are the $n$ states? what implications does this have about the size of the transition matrix?
# MAGIC
# MAGIC * __d) code + short response:__ What is a "right stochastic matrix"? Fill in the code below to compute the transition matrix for the toy graph from question 2. [__`HINT:`__ _It should be right stochastic. Using numpy this calculation can be done in one line of code._]
# MAGIC
# MAGIC * __e) code + short response:__ To compute the stable state distribution (i.e. PageRank) of a "nice" graph we can apply the power iteration method - repeatedly multiplying the transition matrix by itself, until the values no longer change. Apply this strategy to your transition matrix from `part d` to find the PageRank for each of the pages in your toy graph. Your code should print the results of each iteration. How many iterations does it take to converge? Which node is most 'central' (i.e. highest ranked)? Does this match your intuition?
# MAGIC     * __`NOTE 1:`__ _this is a naive approach, we'll unpack what it means to be "nice" in the next question_.
# MAGIC     * __`NOTE 2:`__ _no need to implement a stopping criteria, visual inspection should suffice_.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q3 Student Answers:
# MAGIC > __a)__ It is common to explain PageRank using the analogy of a web surfer who clicks on links at random ad infinitum.
# MAGIC Q: In the context of this hypothetical infinite random walk, what does the PageRank metric measure/represent?
# MAGIC
# MAGIC > * The PageRank link analysis algorithm "measures" the relative number of visits the "infinite" websurfer will spend on each page in the WebGraph.
# MAGIC
# MAGIC > __b)__ What is the "Markov Property" and what does it mean in the context of PageRank?
# MAGIC > * The Markov property is memorylessness, meaning the evolution of the Markov process in the future depends only on the present state and does not depend on past history. Markov processes provide a principled approach to calculating each page's PageRank.
# MAGIC > * PageRank is the steady-state probability distribution of the Markov process underlying the random-surfer navigation model.
# MAGIC
# MAGIC > __c)__ A Markov chain consists of \\(n\\) states plus an \\( n\times n\\) transition probability matrix. In the context of PageRank & a random walk over the WebGraph what are the \\(n\\) states? what implications does this have about the size of the transition matrix?
# MAGIC > * This implies that the size of the transition matrix will be n x n matrix, where n is the number of nodes, or total number of pages in the whole graph.
# MAGIC
# MAGIC > __d)__ What is a "right stochastic matrix"?
# MAGIC > * Right stochastic matrix is a non-negative real number square matrix, with each row summing to 1. In the context of Pagerank, it represents transition probabilities in a Markov Chain.
# MAGIC
# MAGIC > __e)__ To compute the stable state distribution (i.e. PageRank) of a "nice" graph we can apply the power iteration method - repeatedly multiplying the transition matrix by itself,
# MAGIC until the values no longer change. Apply this strategy to your transition matrix from `part d` to find the PageRank for each of the pages in your toy graph.
# MAGIC
# MAGIC > Q: How many iterations does it take to converge?
# MAGIC > * It takes 50-60 iterations to converge, depending on the stopping criteria.
# MAGIC
# MAGIC > Q: Which node is most 'central' (i.e. highest ranked)? Does this match your intuition?
# MAGIC > * Node E is the most central node (or highest ranking node). It is close to my intuition.

# COMMAND ----------

# part d - recall what the adjacency matrix looked like (RUN THIS CELL AS IS)
TOY_ADJ_MATR

# COMMAND ----------

# part d - use TOY_ADJ_MATR to create a right stochastic transition matrix for this graph
################ YOUR CODE HERE #################
def create_transition_matrix(adj_matrix):
    return adj_matrix.mul((1.0 / adj_matrix.sum(axis=1)), axis=0).fillna(0)

transition_matrix = create_transition_matrix(TOY_ADJ_MATR) # replace with your code
################ (END) YOUR CODE #################
print(transition_matrix)

# COMMAND ----------

# part e - compute the steady state using the transition matrix
def power_iteration(xInit, tMatrix, nIter, verbose = True):
    """
    Function to perform the specified number of power iteration steps to
    compute the steady state probability distribution for the given
    transition matrix.

    Args:
        xInit     - (n x 1 array) representing inial state
        tMatrix  - (n x n array) transition probabilities
        nIter     - (int) number of iterations
    Returns:
        state_vector - (n x 1 array) representing probability
                        distribution over states after nSteps.

    NOTE: if the 'verbose' flag is on, your function should print the step
    number and the current matrix at each iteration.
    """
    state_vector = None
    ################ YOUR CODE HERE #################
    state_vector = xInit/np.sum(xInit)

    for i in range(nIter):
        print(i)
        new_state_vector = np.dot(np.transpose(tMatrix.to_numpy()),state_vector)
        state_vector = new_state_vector
        if verbose:
            for i in range(tMatrix.shape[0]):
                print('Node {}: {}'.format(tMatrix.index[i], new_state_vector[i]))
            print(np.sum(new_state_vector))
    for i in range(tMatrix.shape[0]):
        print('Node {}: {}'.format(tMatrix.index[i], new_state_vector[i]))
    ################ (END) YOUR CODE #################
    return state_vector

# COMMAND ----------

# part e - run 10 steps of the power_iteration (RUN THIS CELL AS IS)
xInit = np.array([1.0, 0.0, 0, 0, 0]) # note that this initial state will not affect the convergence states
states = power_iteration(xInit, transition_matrix, 100, verbose = True)

# COMMAND ----------

# MAGIC %md
# MAGIC __`Expected Output for part e:`__
# MAGIC >Steady State Probabilities:
# MAGIC ```
# MAGIC Node A: 0.10526316
# MAGIC Node B: 0.15789474
# MAGIC Node C: 0.18421053
# MAGIC Node D: 0.23684211
# MAGIC Node E: 0.31578947
# MAGIC ```

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 4: Page Rank Theory
# MAGIC
# MAGIC Seems easy right? Unfortunately applying this power iteration method directly to the web-graph actually runs into a few problems. In this question we'll tease apart what we meant by a 'nice graph' in Question 3 and highlight key modifications we'll have to make to the web-graph when performing PageRank. To start, we'll look at what goes wrong when we try to repeat our strategy from question 3 on a 'not nice' graph.
# MAGIC
# MAGIC __`Additional References:`__ http://pi.math.cornell.edu/~mec/Winter2009/RalucaRemus/Lecture3/lecture3.html
# MAGIC
# MAGIC ### Q4 Tasks:
# MAGIC
# MAGIC * __a) code + short response:__ Run the provided code to create and plot our 'not nice' graph. Fill in the missing code to compute its transition matrix & run the power iteration method from question 3. What is wrong with what you see? [__`HINT:`__ _there is a visible underlying reason that it isn't converging... try adding up the probabilities in the state vector after each iteration._]
# MAGIC
# MAGIC * __b) short response:__  Identify the dangling node in this 'not nice' graph and explain how this node causes the problem you described in 'a'. How could we modify the transition matrix after each iteration to prevent this problem?
# MAGIC
# MAGIC * __c) short response:__ What does it mean for a graph to be irreducible? Is the webgraph naturally irreducible? Explain your reasoning briefly.
# MAGIC
# MAGIC * __d) short response:__ What does it mean for a graph to be aperiodic? Is the webgraph naturally aperiodic? Explain your reasoning briefly.
# MAGIC
# MAGIC * __e) short response:__ What modification to the webgraph does PageRank make in order to guarantee aperiodicity and irreducibility? Interpret this modification in terms of our random surfer analogy.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q4 Student Answers:
# MAGIC
# MAGIC > __a)__ Run the provided code to create and plot our 'not nice' graph. Fill in the missing code to compute its transition matrix & run the power iteration method from question 3. What is wrong with what you see?
# MAGIC > * The states of this graph do not converge to a stable state after power iteration. This is because the transition matrix does not qualify as a stochastic matrix.
# MAGIC
# MAGIC > __b.1)__ Identify the dangling node in this 'not nice' graph
# MAGIC and explain how this node causes the problem you described in 'a'.
# MAGIC > * The dangling node here is Node-E.
# MAGIC > * Node-E has two incoming routes, but NO outward route.
# MAGIC
# MAGIC > __b2)__ How could we modify the transition matrix after each iteration to prevent this problem?
# MAGIC > * We would modify the transition matrix to incorporate the idea of teleporting to solve this problem.
# MAGIC > * $$NewTransitionMatrix=(1-\alpha) \* \text{transition matrix} + \alpha \* \text{teleporting matrix}$$
# MAGIC > * Thus with this new stochastic transition matrix, it will solve both the dangling node and periodic problem.
# MAGIC > * This is important because only a well-behaved graph (irreducible and aperiodic) can converge.
# MAGIC
# MAGIC > __c)__ What does it mean for a graph to be irreducible? Is the webgraph naturally irreducible? Explain your reasoning briefly.
# MAGIC > * A graph is irreducible if there is a route from every node to every other node.
# MAGIC > * The webgraph (Toy2_Graph) is NOT irreducible because you cannot travel to another node from Node E.
# MAGIC
# MAGIC > __d)__ What does it mean for a graph to be aperiodic? Is the webgraph naturally aperiodic? Explain your reasoning briefly.
# MAGIC > * Aperiodic means that the period, or greatest common divisor (GCD), of all cycle lengths is 1.
# MAGIC > * This Toy2 webgraph is not naturally aperiodic because not even one node can arrive at itself. All the nodes in the  Toy2 webgraph has to at least wait more than 1 iteration to arrive at it self.
# MAGIC
# MAGIC > __e)__ What modification to the webgraph does PageRank make in order to guarantee aperiodicity and irreducibility? Interpret this modification in terms of our random surfer analogy.
# MAGIC > * We would modify the transition matrix to incorporate the idea of teleporting to solve this problem. In the context of the random surfer, it means that there is a chance that the surfer could reach any other page in our webgraph from the current page.
# MAGIC > * $$NewTransitionMatrix=(1-\alpha) \* \text{transition matrix} + \alpha \* \text{teleporting matrix}$$
# MAGIC > * Thus with this new stochastic transition matrix, it will solve both the dangling node and periodic problem.
# MAGIC > * This is important because only a well-behaved graph (irreducible and aperiodic) can converge.

# COMMAND ----------

# part a - run this code to create a second toy graph (RUN THIS CELL AS IS)
TOY2_GRAPH = {'nodes':['A', 'B', 'C', 'D', 'E'],
              'edges':[('A', 'B'), ('A', 'C'), ('A', 'D'), ('B', 'D'),
                       ('B', 'E'), ('C', 'A'), ('C', 'E'), ('D', 'B')]}

# COMMAND ----------

# part a - simple visualization of our test graph using nx (RUN THIS CELL AS IS)
G = nx.DiGraph()
G.add_nodes_from(TOY2_GRAPH['nodes'])
G.add_edges_from(TOY2_GRAPH['edges'])
display(nx.draw(G, pos=nx.circular_layout(G), with_labels=True, alpha = 0.5))

# COMMAND ----------

# part a - run 10 steps of the power iteration method here
# HINT: feel free to use the functions get_adj_matr() and power_iteration() you wrote above
################ YOUR CODE HERE #################
TOY_ADJ_MATR2 = get_adj_matr(TOY2_GRAPH)
print(TOY_ADJ_MATR2)
transition_matrix2 = create_transition_matrix(TOY_ADJ_MATR2) # replace with your code
print(transition_matrix2)
xInit = np.array([1.0, 0.0, 0.0, 0.0, 0.0]) # note that this initial state will not affect the convergence states
states = power_iteration(xInit, transition_matrix2, 10, verbose = True)

################ (END) YOUR CODE #################

# COMMAND ----------

# MAGIC %md
# MAGIC # About the Data
# MAGIC The main dataset for this data consists of a subset of a 500GB dataset released by AWS in 2009. The data includes the source and metadata for all of the Wikimedia wikis. You can read more here:
# MAGIC > https://aws.amazon.com/blogs/aws/new-public-data-set-wikipedia-xml-data.
# MAGIC
# MAGIC As in previous homeworks we'll be using a 2GB subset of this data, which is available to you in this dropbox folder:
# MAGIC > https://www.dropbox.com/sh/2c0k5adwz36lkcw/AAAAKsjQfF9uHfv-X9mCqr9wa?dl=0.
# MAGIC
# MAGIC Use the cells below to download the wikipedia data and a test file for use in developing your PageRank implementation(note that we'll use the 'indexed out' version of the graph) and to take a look at the files.

# COMMAND ----------

dbutils.fs.ls(DATA_PATH)

# COMMAND ----------

# open test_graph.txt file to see format (RUN THIS CELL AS IS)
with open('/dbfs/mnt/mids-w261/HW5/test_graph.txt', "r") as f_read:
  for line in f_read:
    print(line)

# COMMAND ----------

# load the data into Spark RDDs for convenience of use later (RUN THIS CELL AS IS)
DATA_PATH = 'dbfs:/mnt/mids-w261/HW5/'
testRDD = sc.textFile(DATA_PATH +'test_graph.txt')
indexRDD = sc.textFile(DATA_PATH + '/indices.txt')
wikiRDD = sc.textFile(DATA_PATH + '/all-pages-indexed-out.txt')

# COMMAND ----------

# display testRDD (RUN THIS CELL AS IS)
testRDD.take(10)

# COMMAND ----------

# display indexRDD (RUN THIS CELL AS IS)
indexRDD.take(10)

# COMMAND ----------

# display wikiRDD (RUN THIS CELL AS IS)
wikiRDD.take(10)

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 5: EDA part 1 (number of nodes)
# MAGIC
# MAGIC As usual, before we dive in to the main analysis, we'll peform some exploratory data anlysis to understand our dataset. Please use the test graph that you downloaded to test all your code before running the full dataset.
# MAGIC
# MAGIC ### Q5 Tasks:
# MAGIC * __a) short response:__ In what format is the raw data? What does the first value represent? What does the second part of each line represent? [__`HINT:`__ _no need to go digging here, just visually inspect the outputs of the head commands that we ran after loading the data above._]
# MAGIC
# MAGIC * __b) code + short response:__ Run the provided bash command to count the number of records in the raw dataset. Explain why this is _not_ the same as the number of total nodes in the graph.
# MAGIC
# MAGIC * __c) code:__ In the space provided below write a Spark job to count the _total number_ of nodes in this graph.
# MAGIC
# MAGIC * __d) short response:__ How many dangling nodes are there in this wikipedia graph? [__`HINT:`__ _you should not need any code to answer this question._]

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q5 Student Answers:
# MAGIC * __a)__ In what format is the raw data? What does the first value represent? What does the second part of each line represent? [__`HINT:`__ _no need to go digging here, just visually inspect the outputs of the head commands that we ran after loading the data above._]
# MAGIC > * Each line looks like a key value pair with a tab separated  value, where the key is the node id, and the value is an adjacency list, where each value is outgoing node and the number of edges to the outgoing node.
# MAGIC > * The raw data is a list of string, where inside each string embedded "node" and its a dictionary, which contains the corresponding outward neighbors from the node (the first value of the string).
# MAGIC > * The first value corresponding to a particular node.
# MAGIC > * The second part of each line represents the outgoing nodes from the corresponding node (which is the key of each line). Simple answer is the the second part of each line is the outgoing neighbors of the node (which is the key of each line).
# MAGIC
# MAGIC * __b)__ Run the provided bash command to count the number of records in the raw dataset. Explain why this is _not_ the same as the number of total nodes in the graph.
# MAGIC > * The number of records is not the same as the number of nodes in the graph. The reason is that each record in the list represents a node and its out-ward neighbors. If say a node has no outgoing edges, then this node will not be represented in a record in the list. Yet, this node might have the incoming edges.
# MAGIC
# MAGIC * __d)__ How many dangling nodes are there in this wikipedia graph? [__`HINT:`__ _you should not need any code to answer this question._]
# MAGIC > * Dangling node (nodes that has not out-going edges)
# MAGIC > * num_node_nodes - num_of_records = 15192277 - 5781290 = 9410987 dangling node

# COMMAND ----------

# part b - count the number of records in the raw data (RUN THIS CELL AS IS)
# 5781290
print(wikiRDD.count())

# COMMAND ----------

# part c - write your Spark job here (compute total number of nodes)
def count_nodes(dataRDD):
    """
    Spark job to count the total number of nodes.
    Returns: integer count
    """
    ############## YOUR CODE HERE ###############

#     temp = dataRDD.flatMap(lambda x: [(k,1) for k,v in ast.literal_eval(x.split('\t')[1]).items()]+ [(x.split('\t')[0],1)]) \
#                   .reduceByKey(lambda x,y : x + y) \
#                   .cache()
    temp = dataRDD.flatMap(lambda x: [k for k,v in ast.literal_eval(x.split('\t')[1]).items()]+ [x.split('\t')[0]]).distinct().cache()
    ############## (END) YOUR CODE ###############
    return temp.count()

# COMMAND ----------

# part c - run your counting job on the test file (RUN THIS CELL AS IS)
start = time.time()
tot = count_nodes(testRDD)
print(f'... completed job in {time.time() - start} seconds.')
print(f'Total Nodes: {tot}')

# COMMAND ----------

# part c - run your counting job on the full file (RUN THIS CELL AS IS)
start = time.time()
tot = count_nodes(wikiRDD)
print(f'... completed job in {time.time() - start} seconds.')
print(f'Total Nodes: {tot}')

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 6 - EDA part 2 (out-degree distribution)
# MAGIC
# MAGIC As you've seen in previous homeworks the computational complexity of an implementation depends not only on the number of records in the original dataset but also on the number of records we create and shuffle in our intermediate representation of the data. The number of intermediate records required to update PageRank is related to the number of edges in the graph. In this question you'll compute the average number of hyperlinks on each page in this data and visualize a distribution for these counts (the out-degree of the nodes).
# MAGIC
# MAGIC ### Q6 Tasks:
# MAGIC * __a) code:__ In the space provided below write a Spark job to stream over the data and compute all of the following information:
# MAGIC  * count the out-degree of each non-dangling node and return the names of the top 10 pages with the most hyperlinks
# MAGIC  * find the average out-degree for all non-dangling nodes in the graph
# MAGIC  * take a 1000 point sample of these out-degree counts and plot a histogram of the result.
# MAGIC
# MAGIC
# MAGIC * __b) short response:__ In the context of the PageRank algorithm, how is information about a node's out degree used?
# MAGIC
# MAGIC * __c) short response:__ What does it mean if a node's out-degree is 0? In PageRank how will we handle these nodes differently than others?
# MAGIC
# MAGIC __`NOTE:`__ Please observe scalability best practices in the design of your code & comment your work clearly. You will be graded on both the clarity and the design.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q6 Student Answers:
# MAGIC
# MAGIC > __b)__ In the context of the PageRank algorithm, how is information about a node's out degree used?
# MAGIC > *  It's used to calculate the probabability of reaching a particular page relative to all of the other outgoing pages.
# MAGIC
# MAGIC > __c1)__ What does it mean if a node's out-degree is 0?
# MAGIC > * If a node's out-degree is 0, it means that this node is a dangling node. You cannot go to any other node from this node.
# MAGIC
# MAGIC > __c2)__ In PageRank how will we handle these nodes differently than others?
# MAGIC > * In PageRank, we redistribute the weight of these dangling nodes evenly across all the nodes.
# MAGIC > * The formula is $$ \left( \frac{m}{|G|} \right) $$ , where m is the weight of the dangling nodes, and $$( |G| )$$ is the number of all of the nodes.
# MAGIC > * Whereas non-dangling nodes (just like what they should do previously), uniformly distribute its weight arcoss all of the other nodes due to the teleportation factor.

# COMMAND ----------

# part a - write your Spark job here (compute average in-degree, etc)
def count_degree(dataRDD, n):
    """
    Function to analyze out-degree of nodes in a a graph.
    Returns:
        top  - (list of 10 tuples) nodes with most edges
        avgDegree - (float) average out-degree for non-dangling nodes
        sampledCounts - (list of integers) out-degree for n randomly sampled non-dangling nodes
    """
    # helper func
    def parse(line):
        node, edges = line.split('\t')
        return (node, ast.literal_eval(edges))

    ############## YOUR CODE HERE ###############

    top, avgDegree, sampledCounts = None, None, None
    tempRDD = dataRDD.map(lambda x: parse(x)).map(lambda x: (x[0], len(x[1].keys()))).cache()
    top = tempRDD.takeOrdered(n, key=lambda x: -x[1])
    tempRDD2 = tempRDD.map(lambda x: x[1]).cache()
    sampledCounts = tempRDD2.takeSample(False, n , 0)
    avgDegree = tempRDD2.mean()

    ############## (END) YOUR CODE ###############

    return top, avgDegree, sampledCounts

# COMMAND ----------

# part a - run your job on the test file (RUN THIS CELL AS IS)
start = time.time()
test_results = count_degree(testRDD,10)
print(f"... completed job in {time.time() - start} seconds")
print("Average out-degree: ", test_results[1])
print("Top 10 nodes (by out-degree:)\n", test_results[0])
print("Top 10 nodes (by out-degree:)\n", test_results[2])

# COMMAND ----------

# part a - plot results from test file (RUN THIS CELL AS IS)
plt.hist(test_results[2], bins=10)
plt.title("Distribution of Out-Degree")
display(plt.show())

# COMMAND ----------

# part a - run your job on the full file (RUN THIS CELL AS IS)
start = time.time()
full_results = count_degree(wikiRDD,1000)

print(f"... completed job in {time.time() - start} seconds")
print("Average out-degree: ", full_results[1])
print("Top 10 nodes (by out-degree:)\n", full_results[0])

# COMMAND ----------

# part a - plot results from full file (RUN THIS CELL AS IS)
plt.hist(full_results[2], bins=50)
plt.title("Distribution of Out-Degree")
display(plt.show())

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 7 - PageRank part 1 (Initialize the Graph)
# MAGIC
# MAGIC One of the challenges of performing distributed graph computation is that you must pass the entire graph structure through each iteration of your algorithm. As usual, we seek to design our computation so that as much work as possible can be done using the contents of a single record. In the case of PageRank, we'll need each record to include a node, its list of neighbors and its (current) rank. In this question you'll initialize the graph by creating a record for each dangling node and by setting the initial rank to 1/N for all nodes.
# MAGIC
# MAGIC __`NOTE:`__ Your solution should _not_ hard code \\(N\\).
# MAGIC
# MAGIC ### Q7 Tasks:
# MAGIC * __a) short response:__ What is \\(N\\)? Use the analogy of the infinite random web-surfer to explain why we'll initialize each node's rank to \\(\frac{1}{N}\\). (i.e. what is the probabilistic interpretation of this choice?)
# MAGIC
# MAGIC * __b) short response:__ Will it be more efficient to compute \\(N\\) before initializing records for each dangling node or after? Explain your reasoning.
# MAGIC
# MAGIC * __c) code:__ Fill in the missing code below to create a Spark job that:
# MAGIC   * parses each input record
# MAGIC   * creates a new record for any dangling nodes and sets it list of neighbors to be an empty set
# MAGIC   * initializes a rank of 1/N for each node
# MAGIC   * returns a pair RDD with records in the format specified by the docstring
# MAGIC
# MAGIC
# MAGIC * __d) code:__ Run the provided code to confirm that your job in `part a` has a record for each node and that your should records match the format specified in the docstring and the count should match what you computed in question 5. [__`TIP:`__ _you might want to take a moment to write out what the expected output should be fore the test graph, this will help you know your code works as expected_]
# MAGIC
# MAGIC __`NOTE:`__ Please observe scalability best practices in the design of your code & comment your work clearly. You will be graded on both the clarity and the design.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q7 Student Answers:
# MAGIC
# MAGIC > __a)__ What is \\(N\\)? Use the analogy of the infinite random web-surfer to explain. why we'll initialize each node's rank to \\(\frac{1}{N}\\). (i.e. what is the probabilistic interpretation of this choice?)
# MAGIC > * In the example of the web-server, N is the number of websites.
# MAGIC > * The probabilistic interpretation of this choice is that we have an equal chance of start surfing at any of the webpage.
# MAGIC > * In fact, no matter which website we start to web-server, say if we do infinite random web-server, eventually the probability that we land at each website will converge to a steady state if the graph is a "well-behaved" graph (an irreducible and aperiodic).
# MAGIC > * Converge to a steady state means that $$ x_{i+1} = x_{i} * P  $$, which means that the result of your i+1 iteration will be the same as your \\( i_{th} \\) iteration.
# MAGIC
# MAGIC > __b)__ Will it be more efficient to compute \\(N\\) before initializing records for each dangling node or after? Explain your reasoning.
# MAGIC > * It will be more efficient to compute \\(N\\) before initializing records for each dangling node.
# MAGIC > * The reason is we will need to know the total number of nodes because we need to redistribute the weights of the dangling nodes uniformly across all the nodes at the 2nd Map Job of each iteration.

# COMMAND ----------

# part c - job to initialize the graph (RUN THIS CELL AS IS)
def initGraph(dataRDD):
    """
    Spark job to read in the raw data and initialize an
    adjacency list representation with a record for each
    node (including dangling nodes).

    Returns:
        graphRDD -  a pair RDD of (node_id , (score, edges))

    NOTE: The score should be a float, but you may want to be
    strategic about how format the edges... there are a few
    options that can work. Make sure that whatever you choose
    is sufficient for Question 8 where you'll run PageRank.
    """
    ############## YOUR CODE HERE ###############

    # write any helper functions here
    def get_adj_list(line):
        node, edges = line.split('\t')
        edge_list = [(k,v) for k,v in ast.literal_eval(edges).items()]
        yield (node, edge_list)
        for edge_node in edge_list:
          yield(edge_node[0], [])

    # write your main Spark code here
    graphRDD = dataRDD.flatMap(lambda x: get_adj_list(x))\
                       .reduceByKey(lambda x,y: x + y).cache()
    N = graphRDD.count()
    graphRDD = graphRDD.map(lambda x: (x[0], (1.0/N, list(set(x[1])))))
    ############## (END) YOUR CODE ##############

    return graphRDD

# COMMAND ----------

# part c - run your Spark job on the test graph (RUN THIS CELL AS IS)
start = time.time()
testGraph = initGraph(testRDD).collect()
print(f'... test graph initialized in {time.time() - start} seconds.')
testGraph

# COMMAND ----------

# part c - run your code on the main graph (RUN THIS CELL AS IS)
start = time.time()
wikiGraphRDD = initGraph(wikiRDD)
print(f'... full graph initialized in {time.time() - start} seconds')

# COMMAND ----------

# part c - confirm record format and count (RUN THIS CELL AS IS)
start = time.time()
print(f'Total number of records: {wikiGraphRDD.count()}')
print(f'First record: {wikiGraphRDD.take(1)}')
print(f'... initialization continued: {time.time() - start} seconds')

# COMMAND ----------

# COMMAND ----------

# MAGIC %md
# MAGIC # Question 8 - PageRank part 2 (Iterate until convergence)
# MAGIC
# MAGIC Finally we're ready to compute the page rank. In this last question you'll write a Spark job that iterates over the initialized graph updating each nodes score until it reaches a convergence threshold. The diagram below gives a visual overview of the process using a 5 node toy graph. Pay particular attention to what happens to the dangling mass at each iteration.
# MAGIC
# MAGIC <img src='https://github.com/kyleiwaniec/w261_assets/blob/master/images/HW5/PR-illustrated.png?raw=true' width=50%>
# MAGIC
# MAGIC
# MAGIC
# MAGIC __`A Note about Notation:`__ The formula above describes how to compute the updated page rank for a node in the graph. The $P$ on the left hand side of the equation is the new score, and the $P$ on the right hand side of the equation represents the accumulated mass that was re-distributed from all of that node's in-links. Finally, $|G|$ is the number of nodes in the graph (which we've elsewhere refered to as $N$).
# MAGIC
# MAGIC ### Q8 Tasks:
# MAGIC * __a) short response:__ In terms of the infinite random walk analogy, interpret the meaning of the first term in the PageRank calculation: \\( \alpha * \frac{1}{|G|} \\)
# MAGIC
# MAGIC * __b) short response:__ In the equation for the PageRank calculation above what does \\(m\\) represent and why do we divide it by \\(|G|\\)?
# MAGIC
# MAGIC * __c) short response:__ Keeping track of the total probability mass after each update is a good way to confirm that your algorithm is on track. How much should the total mass be after each iteration?
# MAGIC
# MAGIC * __d) code:__ Fill in the missing code below to create a Spark job that take the initialized graph as its input then iterates over the graph and for each pass:
# MAGIC   * reads in each record and redistributes the node's current score to each of its neighbors
# MAGIC   * uses an accumulator to add up the dangling node mass and redistribute it among all the nodes. (_Don't forget to reset this accumulator after each iteration!_)
# MAGIC   * uses an accumulator to keep track of the total mass being redistributed.( _This is just for your own check, its not part of the PageRank calculation. Don't forget to reset this accumulator after each iteration._)
# MAGIC   * aggregates these partial scores for each node
# MAGIC   * applies telportation and damping factors as described in the formula above.
# MAGIC   * combine all of the above to compute the PageRank as described by the formula above.
# MAGIC   *
# MAGIC
# MAGIC    __WARNING:__ Some pages contain multiple hyperlinks to the same destination, please take this into account when redistributing the mass.
# MAGIC
# MAGIC
# MAGIC __`NOTE:`__ Please observe scalability best practices in the design of your code & comment your work clearly. You will be graded on both the clarity and the design.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Q8 Student Answers:
# MAGIC
# MAGIC > __a)__ In terms of the infinite random walk analogy, interpret the meaning of the first term in the PageRank calculation: \\(\alpha * \frac{1}{|G|}\\)
# MAGIC > * \\( \alpha\ \\) means the teleportation weight.
# MAGIC > * \\( |G|\\) means the number of webpages in the world.
# MAGIC > * Together \\( \alpha * \frac{1}{|G|} \\) means the probability of randomly going from the current webpage to any random webpage (include the current webpage) in the world when surfer choose to just go randomly instead of clicking a hyperlink of the current webpage.
# MAGIC
# MAGIC > __b)__ In the equation for the PageRank calculation above what does \\( m \\) represent and why do we divide it by \\(|G|\\)?
# MAGIC > * \\(m\\) represent missing PageRank mass of the webpage that do NOT have a hyperlink.
# MAGIC > * We divide \\(m\\) by \\(|G|\\) because we want to evenly distribute that mass of all webpages in the graph.
# MAGIC > * Intuitively it means that when we encounter a dangling webpage, instead of stuck at that webpage, user can type in a random internet address and go to any website (include the current website) from the current place.
# MAGIC
# MAGIC > __c)__ How much should the total mass be after each iteration?
# MAGIC > * The total mass after each iteration should be 1.

# COMMAND ----------

# part d - provided FloatAccumulator class (RUN THIS CELL AS IS)

from pyspark.accumulators import AccumulatorParam

class FloatAccumulatorParam(AccumulatorParam):
    """
    Custom accumulator for use in page rank to keep track of various masses.

    IMPORTANT: accumulators should only be called inside actions to avoid duplication.
    We stringly recommend you use the 'foreach' action in your implementation below.
    """
    def zero(self, value):
        return value
    def addInPlace(self, val1, val2):
        return val1 + val2

# COMMAND ----------

# part d - job to run PageRank (RUN THIS CELL AS IS)
def runPageRank(graphInitRDD, alpha = 0.15, maxIter = 10, verbose = True):
    """
    Spark job to implement page rank
    Args:
        graphInitRDD  - pair RDD of (node_id , (score, edges))
        alpha         - (float) teleportation factor
        maxIter       - (int) stopping criteria (number of iterations)
        verbose       - (bool) option to print logging info after each iteration
    Returns:
        steadyStateRDD - pair RDD of (node_id, pageRank)
    """
    # teleportation:
    a = sc.broadcast(alpha)

    # damping factor:
    d = sc.broadcast(1-a.value)

    # initialize accumulators for dangling mass & total mass
    mmAccum = sc.accumulator(0.0, FloatAccumulatorParam())
    totAccum = sc.accumulator(0.0, FloatAccumulatorParam())

    ############## YOUR CODE HERE ###############

    # write your helper functions here,
    # please document the purpose of each clearly
    # for reference, the master solution has 5 helper functions.

    def distribute_mass(line):
      """
      Function to emit a key-value pair for each record

      ###Input
      line is (node_id, (page_rank, adj_list))
      adj_list is comprised of a list of tuples, where each tuple is (outgoing node id, number of times it is linked in the node id page)

      ###Output
      For each line, emit 2 records
      1) Emit (node, (0.0, adj_list))

      2) Emit based on whether or not it is an dangling node
      If it is a dangling node, emit
        ('Dangling' (page_rank, []))

      Otherwise, emit
        (outgoing node_id, (page_rank * weighted distribution of mass, []))

      """

      # Get node id
      node = line[0]

      #Get page rank
      page_rank = line[1][0]

      # Get adjacency list
      adj_list = line[1][1]

      # Get number of adjacent nodes
      num_adj = len(adj_list)

      # Emit default 0 page rank score and adjacency list for node
      yield(node, (0.0, adj_list))

      total_weight = 0

      #Check if node is a dangling node
      if num_adj > 0:

        # Get total number of outgoing page links
        for item in adj_list:
          total_weight += item[1]

        for item in adj_list:
          yield(item[0], (page_rank*item[1]/total_weight,[]))

      else:

        # Emit mass of dangling node
        yield('##Dangling', (page_rank, []))

    # write your main Spark Job here (including the for loop to iterate)
    # for reference, the master solution is 21 lines including comments & whitespace

    if verbose:
      print('-------- FINISHED INITIALIZATION------')
    N_bc = sc.broadcast(graphInitRDD.count())

    # loop through each iteration
    for i in range(maxIter):
      mmAccum = sc.accumulator(0.0, FloatAccumulatorParam())
      totAccum = sc.accumulator(0.0, FloatAccumulatorParam())

      graphInitRDD.foreach(lambda x: totAccum.add(x[1][0]))

      if verbose:
        print('Initial Dangling Mass for iter {}: {}'.format(i,mmAccum.value))

      # NORMAL MAP REDUCE FOR PAGE_RANK CALCULATION
      steadyStateRDD = graphInitRDD.flatMap(lambda x: distribute_mass(x)) \
                                   .reduceByKey(lambda x,y: (x[0] +  y[0], x[1] + y[1])) \
                                   .cache()
      # Add all dangling masses
      #print(steadyStateRDD.filter(lambda x: x[0]=='##Dangling').collect())
      mmAccum.add(steadyStateRDD.filter(lambda x: x[0]=='##Dangling').collect()[0][1][0])

      # Remove dangling nodes from RDD
      steadyStateRDD = steadyStateRDD.filter(lambda x: x[0]!='##Dangling')

      mmAccum_bc = sc.broadcast(mmAccum.value)
      if verbose:
#         print('{}: {}'.format(i, mmAccum_bc.value))
        print('Dangling Mass for iter {}: {}'.format(i, mmAccum_bc.value))
        print('Total Mass for iter {}: {}'.format(i, totAccum.value))
        data = steadyStateRDD.collect()
        for item in data:
          print(item)

      # SECOND MAP REDUCE JOB
      steadyStateRDD = steadyStateRDD.mapValues(lambda x: (a.value/N_bc.value + d.value * (mmAccum_bc.value/N_bc.value + x[0]), x[1])).cache()

      # Reset graph initialization to result of last iteration
      graphInitRDD = steadyStateRDD

      if verbose:
        data = steadyStateRDD.take(10)
        print("Iter: {}".format(i))
        for item in data:
          print(item)

    # Reformat RDD
    steadyStateRDD = steadyStateRDD.map(lambda x: (x[0], x[1][0]))

    ############## (END) YOUR CODE ###############

    return steadyStateRDD

# COMMAND ----------

# part d - run PageRank on the test graph (RUN THIS CELL AS IS)
# NOTE: while developing your code you may want turn on the verbose option
nIter = 20
testGraphRDD = initGraph(testRDD)
start = time.time()
test_results = runPageRank(testGraphRDD, alpha = 0.15, maxIter = nIter, verbose = True)
print(f'...trained {nIter} iterations in {time.time() - start} seconds.')
print(f'Top 20 ranked nodes:')
test_results.takeOrdered(20, key=lambda x: - x[1])

# COMMAND ----------

# MAGIC %md
# MAGIC __`expected results for the test graph:`__
# MAGIC ```
# MAGIC [(2, 0.3620640495978871),
# MAGIC  (3, 0.333992700474142),
# MAGIC  (5, 0.08506399429624555),
# MAGIC  (4, 0.06030963508473455),
# MAGIC  (1, 0.04255740809817991),
# MAGIC  (6, 0.03138662354831139),
# MAGIC  (8, 0.01692511778009981),
# MAGIC  (10, 0.01692511778009981),
# MAGIC  (7, 0.01692511778009981),
# MAGIC  (9, 0.01692511778009981),
# MAGIC  (11, 0.01692511778009981)]
# MAGIC ```

# COMMAND ----------

# part d - run PageRank on the full graph (RUN THIS CELL AS IS)
# NOTE: wikiGraphRDD should have been computed & cached above!
nIter = 10
start = time.time()
full_results = runPageRank(wikiGraphRDD, alpha = 0.15, maxIter = nIter, verbose = True)
print(f'...trained {nIter} iterations in {time.time() - start} seconds.')
print(f'Top 20 ranked nodes:')
full_results.takeOrdered(20, key=lambda x: - x[1])

# COMMAND ----------

top_20 = full_results.takeOrdered(20, key=lambda x: - x[1])

# COMMAND ----------

# Save the top_20 results to disc for use later. So you don't have to rerun everything if you restart the cluster.

import json
dbutils.fs.put(hw5_path+'top20.json',json.dumps(top_20), True)
display(dbutils.fs.ls(hw5_path))

# COMMAND ----------

import json
with open('/dbfs/user/jeffli930@berkeley.edu/HW5/top20.json') as json_data:
    d = json.load(json_data)
    json_data.close()

df = pd.DataFrame(d)
df.columns = ['Page ID', 'Page Rank']
df

# COMMAND ----------

# view record from indexRDD (RUN THIS CELL AS IS)
# title\t indx\t inDeg\t outDeg
indexRDD.take(1)

# COMMAND ----------

# map indexRDD to new format (index, name) (RUN THIS CELL AS IS)
namesKV_RDD = indexRDD.map(lambda x: (int(x.split('\t')[1]), x.split('\t')[0]))

# COMMAND ----------

# see new format (RUN THIS CELL AS IS)
namesKV_RDD.take(2)

# COMMAND ----------

# MAGIC %md
# MAGIC # OPTIONAL
# MAGIC ### The rest of this notebook is optional and doesn't count toward your grade.
# MAGIC The indexRDD we created earlier from the indices.txt file contains the titles of the pages and thier IDs.
# MAGIC
# MAGIC * __a) code:__ Join this dataset with your top 20 results.
# MAGIC * __b) code:__ Print the results

# COMMAND ----------

# MAGIC %md
# MAGIC ## Join with indexRDD and print pretty

# COMMAND ----------

# part a
joinedWithNames = None
############## YOUR CODE HERE ###############

############## END YOUR CODE ###############

# COMMAND ----------

# part b
# Feel free to modify this cell to suit your implementation, but please keep the formatting and sort order.
print("{:10s}\t| {:10s}\t| {}".format("PageRank","Page id","Title"))
print("="*100)
for r in joinedWithNames:
    print ("{:6f}\t| {:10d}\t| {}".format(r[1][1],r[0],r[1][0]))

# COMMAND ----------

# COMMAND ----------

# MAGIC %md
# MAGIC ## OPTIONAL - GraphFrames
# MAGIC GraphFrames is a graph library which is built on top of the Spark DataFrames API.
# MAGIC
# MAGIC * __a) code:__ Using the same dataset, run the graphframes implementation of pagerank.
# MAGIC * __b) code:__ Join the top 20 results with indices.txt and display in the same format as above.
# MAGIC * __c) short answer:__ Compare your results with the results from graphframes.
# MAGIC
# MAGIC __NOTE:__ Feel free to create as many code cells as you need. Code should be clear and concise - do not include your scratch work. Comment your code if it's not self annotating.

# COMMAND ----------

# imports
import re
import ast
import time
import numpy as np
import pandas as pd
import seaborn as sns
import networkx as nx
import matplotlib.pyplot as plt
from graphframes import *
from pyspark.sql import functions as F

# COMMAND ----------

# load the data into Spark RDDs for convenience of use later (RUN THIS CELL AS IS)
DATA_PATH = 'dbfs:/mnt/mids-w261/HW5/'
testRDD = sc.textFile(DATA_PATH +'test_graph.txt')
indexRDD = sc.textFile(DATA_PATH + '/indices.txt')
wikiRDD = sc.textFile(DATA_PATH + '/all-pages-indexed-out.txt')

# COMMAND ----------

# MAGIC %md
# MAGIC ### You will need to generate vertices (v) and edges (e) to feed into the graph below.
# MAGIC Use as many cells as you need for this task.

# COMMAND ----------

# Create a GraphFrame
from graphframes import *
g = GraphFrame(v, e)

# COMMAND ----------

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.15, maxIter=10)

# COMMAND ----------

start = time.time()
top_20 = results.vertices.orderBy(F.desc("pagerank")).limit(20)
print(f'... completed job in {time.time() - start} seconds.')

# COMMAND ----------

# MAGIC %%time
# MAGIC top_20.show()

# COMMAND ----------

# MAGIC %md
# MAGIC ### Run the cells below to join the results of the graphframes pagerank algorithm with the names of the nodes.

# COMMAND ----------

namesKV_RDD = indexRDD.map(lambda x: (int(x.split('\t')[1]), x.split('\t')[0]))

# COMMAND ----------

namesKV_DF = namesKV_RDD.toDF()

# COMMAND ----------

namesKV_DF = namesKV_DF.withColumnRenamed('_1','id')
namesKV_DF = namesKV_DF.withColumnRenamed('_2','title')
namesKV_DF.take(1)

# COMMAND ----------

resultsWithNames = namesKV_DF.join(top_20, namesKV_DF.id==top_20.id).orderBy(F.desc("pagerank")).collect()

# COMMAND ----------

# TODO: use f' for string formatting
print("{:10s}\t| {:10s}\t| {}".format("PageRank","Page id","Title"))
print("="*100)
for r in resultsWithNames:
    print ("{:6f}\t| {:10s}\t| {}".format(r[3],r[2],r[1]))

# COMMAND ----------

# MAGIC %md
# MAGIC ### Congratulations, you have completed HW5! Please refer to the readme for submission instructions.
# MAGIC
# MAGIC If you would like to provide feedback regarding this homework, please use the survey at: https://docs.google.com/forms/d/e/1FAIpQLSce9feiQeSkdP43A0ZYui1tMGIBfLfzb0rmgToQeZD9bXXX8Q/viewform

# COMMAND ----------


r"""
Random Numbers with Python API

AUTHORS:
    -- Carl Witty (2008-03): new file

This module has the same functions as the Python standard module
\module{random}, but uses the current \sage random number state from
\module{sage.misc.randstate} (so that it can be controlled by the same
global random number seeds).

The functions here are less efficient than the functions in \module{random},
because they look up the current random number state on each call.

If you are going to be creating many random numbers in a row, it is
better to use the functions in \module{sage.misc.randstate} directly.

Here is an example:

(The imports on the next two lines are not necessary, since
\function{randrange} and \function{current_randstate} are both available
by default at the \code{sage:} prompt; but you would need them
to run these examples inside a module.) ::

    sage: from sage.misc.prandom import randrange
    sage: from sage.misc.randstate import current_randstate
    sage: def test1():
    ....:    return sum([randrange(100) for i in range(100)])
    sage: def test2():
    ....:    randrange = current_randstate().python_random().randrange
    ....:    return sum([randrange(100) for i in range(100)])

Test2 will be slightly faster than test1, but they give the same answer::

    sage: with seed(0): test1()
    5169
    sage: with seed(0): test2()
    5169
    sage: with seed(1): test1()
    5097
    sage: with seed(1): test2()
    5097
    sage: timeit('test1()') # random
    625 loops, best of 3: 590 us per loop
    sage: timeit('test2()') # random
    625 loops, best of 3: 460 us per loop

The docstrings for the functions in this file are mostly copied from
Python's \file{random.py}, so those docstrings are "Copyright (c)
2001, 2002, 2003, 2004, 2005, 2006, 2007 Python Software Foundation;
All Rights Reserved" and are available under the terms of the
Python Software Foundation License Version 2.
"""

# We deliberately omit "seed" and several other seed-related functions...
# setting seeds should only be done through sage.misc.randstate .

from sage.misc.randstate import current_randstate

def _pyrand():
    r"""
    A tiny private helper function to return an instance of
    random.Random from the current \sage random number state.
    Only for use in prandom.py; other modules should use
    current_randstate().python_random().

    EXAMPLES::

        sage: from sage.misc.prandom import _pyrand
        sage: _pyrand()
        <...random.Random object at 0x...>
        sage: _pyrand().getrandbits(10)
        114L
    """
    return current_randstate().python_random()

def getrandbits(k):
    r"""
    getrandbits(k) -> x.  Generates a long int with k random bits.

    EXAMPLES::

        sage: getrandbits(10)
        114L
        sage: getrandbits(200)
        1251230322675596703523231194384285105081402591058406420468435L
        sage: getrandbits(10)
        533L
    """
    return _pyrand().getrandbits(k)

def randrange(start, stop=None, step=1):
    r"""
    Choose a random item from range(start, stop[, step]).

    This fixes the problem with randint() which includes the
    endpoint; in Python this is usually not what you want.

    EXAMPLES::

        sage: randrange(0, 100, 11)
        11
        sage: randrange(5000, 5100)
        5051
        sage: [randrange(0, 2) for i in range(15)]
        [0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1]
        sage: randrange(0, 1000000, 1000)
        486000
        sage: randrange(-100, 10)
        -56
    """
    return _pyrand().randrange(start, stop, step)

def randint(a, b):
    r"""
    Return random integer in range [a, b], including both end points.

    EXAMPLES::

        sage: [randint(0, 2) for i in range(15)]
        [0, 1, 0, 0, 1, 0, 2, 0, 2, 1, 2, 2, 0, 2, 2]
        sage: randint(-100, 10)
        -46
    """
    return _pyrand().randint(a, b)

def choice(seq):
    r"""
    Choose a random element from a non-empty sequence.

    EXAMPLES::

        sage: [choice(list(primes(10, 100))) for i in range(5)]
        [17, 47, 11, 31, 47]
    """
    return _pyrand().choice(seq)

def shuffle(x):
    r"""
    x, random=random.random -> shuffle list x in place; return None.

    Optional arg random is a 0-argument function returning a random
    float in [0.0, 1.0); by default, the sage.misc.random.random.

    EXAMPLES::

        sage: shuffle([1 .. 10])
    """
    return _pyrand().shuffle(x)

def sample(population, k):
    r"""
    Choose k unique random elements from a population sequence.

    Return a new list containing elements from the population while
    leaving the original population unchanged.  The resulting list is
    in selection order so that all sub-slices will also be valid random
    samples.  This allows raffle winners (the sample) to be partitioned
    into grand prize and second place winners (the subslices).

    Members of the population need not be hashable or unique.  If the
    population contains repeats, then each occurrence is a possible
    selection in the sample.

    To choose a sample in a range of integers, use xrange as an
    argument (in Python 2) or range (in Python 3).  This is especially
    fast and space efficient for sampling from a large population:
    sample(range(10000000), 60)

    EXAMPLES::

        sage: sample(["Here", "I", "come", "to", "save", "the", "day"], 3)
        ['Here', 'to', 'day']
        sage: sample(range(2^30), 7)
        [357009070, 558990255, 196187132, 752551188, 85926697, 954621491, 624802848]
    """
    return _pyrand().sample(population, k)

def random():
    r"""
    Get the next random number in the range [0.0, 1.0).

    EXAMPLES::

        sage: [random() for i in [1 .. 4]]
        [0.111439293741037, 0.5143475134191677, 0.04468968524815642, 0.332490606442413]
    """
    return _pyrand().random()

def uniform(a, b):
    r"""
    Get a random number in the range [a, b).

    Equivalent to \code{a + (b-a) * random()}.

    EXAMPLES::

        sage: uniform(0, 1)
        0.111439293741037
        sage: uniform(e, pi)
        0.5143475134191677*pi + 0.48565248658083227*e
        sage: RR(_)
        2.93601069876846
    """
    return _pyrand().uniform(a, b)

def betavariate(alpha, beta):
    r"""
    Beta distribution.

    Conditions on the parameters are alpha > 0 and beta > 0.
    Returned values range between 0 and 1.

    EXAMPLES::

        sage: betavariate(0.1, 0.9)
        9.75087916621299e-9
        sage: betavariate(0.9, 0.1)
        0.941890400939253
    """
    return _pyrand().betavariate(alpha, beta)

def expovariate(lambd):
    r"""
    Exponential distribution.

    lambd is 1.0 divided by the desired mean.  (The parameter would be
    called "lambda", but that is a reserved word in Python.)  Returned
    values range from 0 to positive infinity.

    EXAMPLES::

        sage: [expovariate(0.001) for i in range(3)]
        [118.152309288166, 722.261959038118, 45.7190543690470]
        sage: [expovariate(1.0) for i in range(3)]
        [0.404201816061304, 0.735220464997051, 0.201765578600627]
        sage: [expovariate(1000) for i in range(3)]
        [0.0012068700332283973, 8.340929747302108e-05, 0.00219877067980605]
    """
    return _pyrand().expovariate(lambd)

def gammavariate(alpha, beta):
    r"""
    Gamma distribution.  Not the gamma function!

    Conditions on the parameters are alpha > 0 and beta > 0.

    EXAMPLES::

        sage: gammavariate(1.0, 3.0)
        6.58282586130638
        sage: gammavariate(3.0, 1.0)
        3.07801512341612
    """
    return _pyrand().gammavariate(alpha, beta)

def gauss(mu, sigma):
    r"""
    Gaussian distribution.

    mu is the mean, and sigma is the standard deviation.  This is
    slightly faster than the normalvariate() function, but is not
    thread-safe.

    EXAMPLES::

       sage: [gauss(0, 1) for i in range(3)]
       [0.9191011757657915, 0.7744526756246484, 0.8638996866800877]
       sage: [gauss(0, 100) for i in range(3)]
       [24.916051749154448, -62.99272061579273, -8.1993122536718...]
       sage: [gauss(1000, 10) for i in range(3)]
       [998.7590700045661, 996.1087338511692, 1010.1256817458031]
    """
    return _pyrand().gauss(mu, sigma)

def lognormvariate(mu, sigma):
    r"""
    Log normal distribution.

    If you take the natural logarithm of this distribution, you'll get a
    normal distribution with mean mu and standard deviation sigma.
    mu can have any value, and sigma must be greater than zero.

    EXAMPLES::

        sage: [lognormvariate(100, 10) for i in range(3)]
        [2.9410355688290246e+37, 2.2257548162070125e+38, 4.142299451717446e+43]
    """
    return _pyrand().lognormvariate(mu, sigma)

def normalvariate(mu, sigma):
    r"""
    Normal distribution.

    mu is the mean, and sigma is the standard deviation.

    EXAMPLES::

       sage: [normalvariate(0, 1) for i in range(3)]
       [-1.372558980559407, -1.1701670364898928, 0.04324100555110143]
       sage: [normalvariate(0, 100) for i in range(3)]
       [37.45695875041769, 159.6347743233298, 124.1029321124009]
       sage: [normalvariate(1000, 10) for i in range(3)]
       [1008.5303090383741, 989.8624892644895, 985.7728921150242]
    """
    return _pyrand().normalvariate(mu, sigma)

def vonmisesvariate(mu, kappa):
    r"""
    Circular data distribution.

    mu is the mean angle, expressed in radians between 0 and 2*pi, and
    kappa is the concentration parameter, which must be greater than or
    equal to zero.  If kappa is equal to zero, this distribution reduces
    to a uniform random angle over the range 0 to 2*pi.

    EXAMPLES::

        sage: [vonmisesvariate(1.0r, 3.0r) for i in range(1, 5)]  # abs tol 1e-12
        [0.898328639355427, 0.6718030007041281, 2.0308777524813393, 1.714325253725145]
    """
    return _pyrand().vonmisesvariate(mu, kappa)

def paretovariate(alpha):
    r"""
    Pareto distribution.  alpha is the shape parameter.

    EXAMPLES::

        sage: [paretovariate(3) for i in range(1, 5)]
        [1.0401699394233033, 1.2722080162636495, 1.0153564009379579, 1.1442323078983077]
    """
    return _pyrand().paretovariate(alpha)

def weibullvariate(alpha, beta):
    r"""
    Weibull distribution.

    alpha is the scale parameter and beta is the shape parameter.

    EXAMPLES::

        sage: [weibullvariate(1, 3) for i in range(1, 5)]
        [0.49069775546342537, 0.8972185564611213, 0.357573846531942, 0.739377255516847]
    """
    return _pyrand().weibullvariate(alpha, beta)