# Manipulating Time-series

Time-series are a key element when assessing solar resource data. In this section, we present several examples to learn how to deal with different formats in the data and few common tasks to prepare our time-series for later analysis, such as down and up-sampling data when we need different temporal resolution than that initially available or interpolating missing values in the data.

The dataset used in the examples of this section is a customized dataset using solar radiation measurements from the Measurement and Instrumentation Data Center (MIDC) of the U.S. National Renewable Energy Laboratory (NREL). The station selected is located at the University of Nevada - Las Vegas (UNLV) and the data used are 1-minute GHI, DHI and DNI measurements for the year 2020 (Andreas and Stoffel, 2006).

In this section, we cover:

- [1 Time-series handling](#timeseries_handling)

- [2 Down and up-sampling time-series data](#timeseries_downup_sampling)

- [3 Interpolating time-series data](#timeseries_interpolation)

- [4 Visualizating time-series data](#timeseries_visualization)

***

## 1 Time-series handling
Datasets often come in different formats depending on the source. Those formats sometimes cannot be used straightaway to build a time-series and may require additional processing steps before building the time-series. For example:

- **What if date and time are in different columns?**

- **What if the year, month, day and time are in separate columns?**

- **How to the define the timestamp format for a particular dataset?**

- **How to deal with timestamp issues, local vs. universal (UTC) time?**

This subsection presents several examples to deal with different formats in which time-series data could come and shows how to build a time-series or *datetime series*, as known in Python, for later analysis. The processing steps to build time-series are based on [pandas library](https://pandas.pydata.org/).

Let's get started!

# Importing the needed libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import pvlib

### 1.1 Build our customized dataset
In order to build the customized dataset for this section, we make use of the I/O tools of the Python library *pvlib* to retrieve the data from the UNLV station in the MIDC. Data from other stations from the MIDC can be also retrieved using this method by adapting the station ID in the query. The different station IDs are available in the [MIDC raw data page](https://midcdmz.nrel.gov/apps/data_api_doc.pl?_idtextlist).

# Dictionary to rename certain variables from the raw data
var_map = {'Global Horiz [W/m^2]': 'ghi',
           'Direct Normal [W/m^2]':'dni',
           'Diffuse Horiz (calc) [W/m^2]':'dhi',
           'Year':'year'}

# Retrieving the raw data from the station
df_ref = pvlib.iotools.read_midc_raw_data_from_nrel('UNLV',                     # Station id
                                                    pd.Timestamp('20200101'),   # Start date YYYYMMDD
                                                    pd.Timestamp('20201231'),   # End date  YYYYMMDD
                                                    variable_map=var_map)       # Variable Map
# Let's have a look to the first 2 rows of the dataset
df_ref.head(2)

# Let's have a look to the last 2 rows of the dataset
df_ref.tail(2)

The dataset is 1-minute resolution data with 21 variables related to meteorological and other relevant data: ambient temperature, wind speed, wind direction, global horizontal irradiance (GHI), direct normal irradiance (DNI), diffuse horizontal irradiance (DHI), zenith and azimuth angles, airmass, among other.

For the examples in this section we will use GHI, DNI and DHI measurements and time-related data.

# Slice desired variables out of the 21 variables provided in the raw data.
df_ref = df_ref[['ghi', 'dni', 'dhi', 'year']]

# Add multiple temporal data to the dataset
df_ref['month'] = df_ref.index.month
df_ref['day'] = df_ref.index.day
df_ref['hour'] = df_ref.index.hour
df_ref['minute'] = df_ref.index.minute
df_ref['date'] = df_ref.index.strftime('%Y-%m-%d')
df_ref['time'] = df_ref.index.strftime('%H:%M:%S')
df_ref['timestamp'] = df_ref.index.strftime('%Y-%m-%d %H:%M:%S%z')

# Epoch format
df_ref['epoch'] = df_ref.index.astype('int64')//1e9

# Reset the Index of the DataFrame
df_ref = df_ref.reset_index(drop=True)

# Let's have a look to the resulting columns of the dataset
df_ref.columns

Let's visualize the first rows of the **customized reference dataframe:**

# First 3 rows in the dataframe
df_ref.head(3)

Now that we have our customized reference dataset of 1-minute irradiance measurements for 2020 and temporal data, we can start building the timeseries in different ways.

### 1.2 Time-series when timestamps are available:

When timestamps are available, the most straightforward way to build the DataFrame with a datetime index is to convert the column with the timestamp into datetime format and set it as index.

Let's see how!

# A new dataframe copy of the reference dataset
df = df_ref.copy()
# Convert the timestamp string into datetime format
df['timestamp'] = pd.to_datetime(df['timestamp'], format='%Y-%m-%d %H:%M:%S%z')
# Set timestamp column as index
df = df.set_index(df['timestamp'])
# See the first 3 rows of the DataFrame with Datetime Index
df.head(3)

The format of the timestamp is specified in the argument 'format' as a string and can be adapted to any case. The available options in Python can be checked in this [link](https://strftime.org/).

Universal Time Coordinated (UTC) is usually the timestamp provided for many solar radiation data networks and platforms like the BSRN, PVGIS, etc. However, data can be also reported in local time like in our example. Timestamps can be converted to other timezones with the funcion *tz_convert*, which can be useful when dealing with data from different databases and locations worldwide:

# Add UTC timestamp from the local time (Pacific Summer Time)
df['timestamp_utc'] = df.index.tz_convert('UTC')
# See the first 3 rows
df.head(3)

The valid timezone strings for other timezones can be found in this [link](https://pvlib-python.readthedocs.io/en/stable/timetimezones.html). When the timezone is not provided as part of the timestamp, the function *tz_localize* can be used to localize the values in a timezone-naive series. *tz_localize* will be used in the next example.

### 1.3 Time-series when date and time are available:

When date and time are available in separate columns, a timestamp can be created in a new column and the new column can then be set as index and localized. Let's have a look how to do that:

# A new dataframe copy of the reference dataset
df = df_ref.copy()
# New column with the date and time
df['datetime'] = df['date'] + 'T' + df['time']
# Convert the new column into datetime format
df['datetime'] = pd.to_datetime(df['datetime'], format='%Y-%m-%dT%H:%M:%S')
# Set the column 'datetime' as index and localize it to its timezone
df = df.set_index(df['datetime']).tz_localize('Etc/GMT+8')
# See the first 3 rows
df.head(3)

### 1.4 Time-series when the time data is split in multiple columns:

If time-related data are split across multiple columns, a timestamp can be created in a new column similarly than in the previous case. Let's imagine our dataset would have the year, month, day, hour, and minute in separate columns. In that case, we could build our time-series as follows:

# A new dataframe copy of the reference dataset
df = df_ref.copy()
# Let's reduce the code lines and define the new string within the 'to_datetime' function
df['datetime'] = pd.to_datetime(df[['year', 'month', 'day', 'hour', 'minute']],
                                format = '%Y-%m-%d%H:%M')
# Set the column 'datetime' as index
df = df.set_index(df['datetime'])
# Localize the datetime series
df.index = df.index.tz_localize('Etc/GMT+8')
# See the first 3 rows
df.head(3)

### 1.5 Time-series when the timestamp is given as epoch (Unix Time)

If the dataset has epoch timestamps, note that the data will have UTC time. However, it can be converted to any timezone using the function *tz_convert*. If there are epoch timestamps, a datetime series can be formed as follows:

# A new dataframe copy of the reference dataset
df = df_ref.copy()
# Convert epoch timestamps to datetime format and localize
df['datetime'] = pd.to_datetime(df['epoch'], unit='s', utc=True)
# Set datetime as index and convert UTC time to local time
df = df.set_index(df['datetime']).tz_convert('Etc/GMT+8')
# See the results
df.head(3)

We have seen how the same DataFrame with *datetimeindex* can be obtained in multiple ways depending on the format of time data provided.

***

## 2 Down and up-sampling time-series data

When assessing solar resource, you may need a different time-resolution than your data for a particular part of the analysis. In those cases, it is possible to **down-sample and up-sample the data at different temporal resolutions** using two different methods within [pandas library](https://pandas.pydata.org/) called *resample* and *asfreq*. Depending on your needs, you will opt for one or the other. Regardless of the method, both of them require a DataFrame with *datetimeindex* either time-aware (localized) or time-naive (not localized).

###  2.1 Method 'asfreq' vs. 'resample'
Let's first create a new DataFrame with only the columns with solar data and see the differences between both methods with examples.

# New DataFrame with 1-minute data and solar data
df_1min = df[['ghi', 'dhi', 'dni']]
# See our new DataFrame
df_1min.head(3)

Let's try to obtain a DataFrame down-sampled with the maximum monthly data with both methods and see the differences. With *asfreq*, it would be the following:

df_1min.asfreq("1M").max()

With *resample* the result would be:

df_1min.resample("1M").max()

It is obvious that the outputs are not the same and that is because the methods work differently. *asfreq* takes the value at the simultaneous stamps given by the frequency argument. See below:

df_1min.asfreq("1M")

Then *.max()* has returned the maximum of each of the columns.

In contrast, *resample* does return the maximum value within the period of time at the specified frequency. *resample* method requires a mathematical operation to perform in the resampled data (the maximum value in our case). Otherwise, it would return a *DatetimeIndexResampler* object without showing any data. See below:

df_1min.resample("1M")

The *resample* method accepts multiple **mathematical and statistical operations**. For example: maximum (max), minimum (min), arithmetic mean (mean), standard deviation (std), median (median), mode (mode), addition (sum), among others.

Both methods allow for multiple **frequencies options**, the available frequency tags within Python can be found [here](https://stackoverflow.com/questions/35339139/where-is-the-documentation-on-pandas-freq-tags).

### 2.2 Down-sampling the data in a time-series

Down-sampling permits turning more frequent values into less frequent. In the context of solar resource and considering our 1-minute resolution dataset, down-sampling can be used for:
- Producing a timeseries of hourly/daily average irradiance.
- Producing a timeseries of maximum daily irradiance.
- Estimating the hourly/daily/monthly sums of irradiation.
- And many more!

Let's implement some of these listed examples!

#### Producing hourly average irradiance from minutely observations

# Resampling to hourly mean values
df_hourly = df_1min.resample("1H").mean()
# Showing the shape of the new DataFrame
df_hourly.shape # returns Rows, Columns

There are 8760 hours in a year. Yet, we can have a look to the first few rows of the DataFrame:

df_hourly.head(12)

A time-series with the maximum irradiance would be similar replacing *'mean()'* with *'max()'*.

#### Producing time-series of monthly total GHI, DHI, DNI irradiation from minutely observations

# Resampling to monthly aggregated values
monthly_energy = df_1min[['ghi', 'dhi', 'dni']].resample("1M").sum()*(1/60)
# See the results expressed in kWh·sqm
monthly_energy/1000

It could be done in similar way for other resolutions (e.g. daily or annual irradiation).

### 2.3 Up-sampling the data in a time-series

Up-sampling permits obtaining more frequent values from less frequent. For solar data, depending on the application up to sub-minutely data could be required and up-sampling is a technique that provides a manner to increase the temporal resolution to adapt it to our needs. For example, turning an hourly time-series into a half-hourly. Let's see an example using both *resample* and *asfreq*.

#### Producing half-hourly irradiance series from hourly observations
Using the DataFrame *df_hourly* created previously, it can be up-sample as follows:

# Using 'resample' method:
df_hourly.resample("30Min").mean().head(10)

# Using 'asfreq' method:
df_hourly.asfreq("30Min").head(10)

Contrary to the case of down-sampling, both *asfreq* and *resample* provide similar results when up-sampling. However, *asfreq* provides additional functionalities to treat the new timestamps without data, i.e. NaN values.

By passing the argument *'method'* with the string *'backfill'* or *'bfill'* uses the next valid observation to fill the NaN value (back filling). If instead, the string *'pad'* or *'ffill'* is given, the method assigns the last valid observation forward to the next valid (forward filling).

Let's see the same example adding this argument:

# Half-hourly up-sample with back filling function
df_hourly.asfreq("30Min", method='bfill').head(10)

We see that the DataFrame now contains the next valid hourly value in the newly obtained half-hourly timestamps of the previous hour. It would take the previous valid hourly value if we used forward filling. For example:

# Half-hourly up-sample with forward-filling function
df_hourly.asfreq("30Min", method='ffill').head(10)

The forward filling option provides the same value for o'clock and half past timestamps within the same hour. In addition to these two ways to complete the NaN values, the method *asfreq* can replace the NaN values with a constant. See below:

# Half-hourly up-sample filling the new timestamps with a constant
df_hourly.asfreq("30Min", fill_value=0).head(10)

The use of the methods *asfreq* or *resample* will depend on your dataset and the analysis you aim to undertake.

***

## 3 Interpolating time-series data

When up-sampling the data series, it can happen that back-filling, forward-filling and constant replacement does not necessarily work for your analysis/application. An alternative approach is interpolating the replacing the NaN values with an interpolated result. Interpolation in Pandas DataFrames with *DatetimeIndex* is done with the *interpolate* method.

The mathematical interpolation method in *interpolate* is defined with the argument called *'method'*. Pandas permits several interpolation methods, such as 'linear', 'cubic', 'quadratic', 'spline', 'polynomial' and others. All the interpolation options can be found in the [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.interpolate.html) of the *interpolate* method.

Following the previous example, let's implement interpolation in the missing values of the half-hourly timestamps using 'linear', 'cubic' and 'polynomial' methods:

# Up-sample using the 'asfreq' method
df_30min = df_hourly.asfreq("30Min")
# Interpolate missing values (NaN) with linear interpolation
df_linear = df_30min.interpolate(method='linear')
# See the results:
df_linear.head(10)

Similarly, it can be implemented to other methods:

# Interpolate missing values (NaN) with cubic interpolation
df_cubic = df_30min.interpolate(method='cubic')
# See the results:
df_cubic.head(10)

With polynomial interpolation, the degree or order of the polynomial function needs to be defined as an argument:

# Interpolate missing values (NaN) with polynomial interpolation
df_polynomial = df_30min.interpolate(method='polynomial', order=5)
# See the results:
df_polynomial.head(10)

The interpolation of NaN values when up-sampling time-series data can help overcome the issues of using back or forward filling, specially if you aim to up-sample at higher frequencies than the example shown (e.g. 1-hour to 15-minute resolution series). The mathematical methods available for interpolation within Pandas are diverse and cover beyond the most common interpolation functions.

## 4 Visualizing time-series data

It is often useful to visualize the data to grasp insighs and observe trends about the data. This section shows few examples to visualize time-series data.

### 4.1 Plotting a time-series for a day of interest

Below there is an example to visualize a single day of interest. With DataFrames using *DatetimeIndex* it is easy to select a particular day and Pandas interacts with Matplotlib.Pyplot library to plot straight-away.

# Plotting GHI for a given day in the time-series
df_1min['2020-06-01']['ghi'].plot(label='GHI')
plt.ylabel('Irradiance [W/m$^2$]')
plt.xlabel('Local Time [HH:MM]')
plt.legend(loc='best')
plt.show() # Not needed in Jupyter Notebooks but usually required in other IDEs.

We can visualize the effect of using average (*resample*) vs. instantaneous (*asfreq*) measurements when down-sampling our data.

# Plotting GHI for a given day in the time-series
df_1min['2020-06-01']['ghi'].plot(label='1-min data', alpha=0.4) # Reference data
df_1min.asfreq('30Min')['2020-06-01']['ghi'].plot(label='30-min instant.') # Instantaneous 30-min values
df_1min.resample('30Min').mean()['2020-06-01']['ghi'].plot(label='30-min average') # Average 30-min values
plt.title('Average vs. Actual GHI Measurements') # title of the figure
plt.ylabel('Irradiance [W/m$^2$]') # y-axis label
plt.xlabel('Local Time [HH:MM]') # x-axis label
plt.legend(loc='upper left') # insert legend
plt.show() # Not needed in Jupyter Notebook but usually required in other IDEs.

### 4.2 Plotting a time-series for a few consecutive days of interest

Below there is an example to visualize a few consecutive days (e.g. 5 days) of interest. By using ['start date']:['end date'] it is possible to select time ranges easily with a DataFrame having a *DatetimeIndex*.

# Variables to plot
vars = ['ghi', 'dni', 'dhi']
# Create 3 subplots, with shared X and Y axis
fig, axs = plt.subplots(3, sharex=True, sharey=True, figsize=(9,6))
# Add title to the plot
fig.suptitle('Average Hourly Solar Radiation Observations', fontsize=14)

for i in range(3):
    axs[i].plot(df_1min.resample('1H').mean()['2020-06-01':'2020-06-05'][vars[i]], label='Average') # Average hourly
    axs[i].plot(df_1min.resample('1H').max()['2020-06-01':'2020-06-05'][vars[i]], label='Maximum') # Max. hourly
    axs[i].plot(df_1min.resample('1H').min()['2020-06-01':'2020-06-05'][vars[i]], label='Minimum') # Min. hourly
    axs[i].set_title(vars[i].upper()) # Title for each subplot
fig.subplots_adjust(hspace=0.3) # Adjust the white space between the subplots titles
fig.text(0.04, 0.5, 'Irradiance [W/m$^2$]', va='center', rotation='vertical', fontsize=12) # Common Y Axis
fig.text(0.51, 0.04, 'Local Time', ha='center', fontsize=12) # Common X Axis
plt.legend(loc='upper center', ncol=3) # Legend for the last subplot or 'axs[i].legend()' in the loop to a legend to each.
plt.show()

### 4.3 Plotting a time-series for a few non-consecutive days of interest

Below there is an example to visualize a few non-consecutive days of interest, which could be the case when we would like to observe several days scattered throughout the year a single plot. In order to do this, we need to select the day of interest from the DataFrame and then reset its *DatetimeIndex*. For example:

# List of days of interest
days = ['2020-01-01', '2020-03-01', '2020-06-01', '2020-09-01']
# Iterate over the days and plot each of them
for day in days:
    df_day = df_1min.resample('1H').mean()[day]['ghi'].to_frame()  # average hourly of GHI for current day
    df_day = df_day.reset_index(drop=True) # reset its Index to numeric (i.e. 0,1,2,3...)
    plt.plot(df_day, label=day) # plot the current day
plt.title('Average Hourly GHI Measurements for Days of Interest') # title of the figure
plt.xticks(np.arange(0, 25, step=3), np.arange(0, 25, step=3)) # set labels positions and names
plt.ylabel('Irradiance [W/m$^2$]') # y-axis label
plt.xlabel('Local Time') # x-axis label
plt.legend(loc='best') # insert legend
plt.show()

### 4.4 Daily insolation throughout the year

With time-series data, the hourly/daily/monthly insolation (i.e. the sum of accumulated energy) can also be analysed throughout the year with time-series data. For example, below an example to visualize the daily insolation is shown:

# Calculate the daily insolation expressed in kWh·sqm from GHI measurements
daily_energy = (df_1min['ghi'].resample("1D").sum()*(1/60))/1000 # selecting only GHI returns a Pandas Series

# Create time-series plot
daily_energy.plot(figsize=(9,6), legend=False) # plot timeseries
plt.title('Time-series of Daily Insolation')  # add title
plt.ylabel('Energy [kWh/m$^2$]') # add Y-axis label
plt.xlabel('Time') # add X-axis label
plt.show()

Time-series data can also be visualized in other ways, for instance, as a heat map.

# Prepare the data for heat map of hourly insolation
energy_array = pd.DataFrame() # empty DataFrame for the results
for i in range(1,13): # iterate over months
    # select the data in the month and eliminate the datetimeindex
    df_month = daily_energy[daily_energy.index.month==i].reset_index(drop=True)
    # rename the column with the number of the month
    df_month.columns = [str(i)]
    # Append results to the DataFrame
    energy_array = pd.concat([energy_array, df_month], axis=1)
# Transpose to have months in y-axis and days in x-axis
energy_array = energy_array.transpose()
# Rename the columns of the days
energy_array.columns = np.arange(1, 32)

# Plot heat map of daily insolation
months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', # month labels
          'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
plt.figure(figsize=(10, 5))
ax = sns.heatmap(energy_array, cmap='CMRmap', linewidths=0.2, # plot heatmap with Seaborn (sns) library
                xticklabels=2, annot=False,
                cbar_kws={'label': 'Daily Energy [kWh/m$^2$]'})
ax.set_title('Heat Map of Daily Insolation') # add title
ax.set_yticklabels(months,rotation=0) # add the months as tick-labels for the y-axis
ax.set_xticklabels(ax.get_xticklabels(),rotation=0) # add the days as tick-labels for the x-axis
ax.set_xlabel('Day of the Month')
plt.show()

***

## Section summary

This section has shown how to build and work with a time-series in Python with multiple examples:

- We have seen how to prepare a DataFrame with *DatatimeIndex* to be used as a time-series when the timestamps are given in multiple formats in the temporal data and local/UTC time.

- Changes in the temporal resolution of the data can be applied by down and up-sampling the data and the differences between 2 available methods (*asfreq* and *resample*) have been shown with examples and different sampling frequencies.

- The interpolation of missing data in time-series can be used to up-sample the resolution of the data and examples with some methods have been shown.

- Finally, several ideas to visualize data have been presented.

Overall, the possibilities with time-series of solar resource are many. The most useful and suitable analysis and visualizations will be determined by the application and scope of the study.

***

## References

Andreas, A.; Stoffel, T.; (2006). University of Nevada (UNLV):
Las Vegas, Nevada (Data); NREL Report No. DA-5500-56509.
http://dx.doi.org/10.5439/1052548


"""
Training GNN with Neighbor Sampling for Node Classification
===========================================================

This tutorial shows how to train a multi-layer GraphSAGE for node
classification on ``ogbn-arxiv`` provided by `Open Graph
Benchmark (OGB) <https://ogb.stanford.edu/>`__. The dataset contains around
170 thousand nodes and 1 million edges.

By the end of this tutorial, you will be able to

-  Train a GNN model for node classification on a single GPU with DGL's
   neighbor sampling components.

This tutorial assumes that you have read the :doc:`Introduction of Neighbor
Sampling for GNN Training <L0_neighbor_sampling_overview>`.

"""

######################################################################
# Loading Dataset
# ---------------
#
# OGB already prepared the data as DGL graph.
#

import dgl
import torch
import numpy as np
from ogb.nodeproppred import DglNodePropPredDataset

dataset = DglNodePropPredDataset('ogbn-arxiv')
device = 'cpu'      # change to 'cuda' for GPU

######################################################################
# OGB dataset is a collection of graphs and their labels. ``ogbn-arxiv``
# dataset only contains a single graph. So you can
# simply get the graph and its node labels like this:
#

graph, node_labels = dataset[0]
# Add reverse edges since ogbn-arxiv is unidirectional.
graph = dgl.add_reverse_edges(graph)
graph.ndata['label'] = node_labels[:, 0]
print(graph)
print(node_labels)

node_features = graph.ndata['feat']
num_features = node_features.shape[1]
num_classes = (node_labels.max() + 1).item()
print('Number of classes:', num_classes)

######################################################################
# You can get the training-validation-test split of the nodes with
# ``get_split_idx`` method.
#

idx_split = dataset.get_idx_split()
train_nids = idx_split['train']
valid_nids = idx_split['valid']
test_nids = idx_split['test']

######################################################################
# How DGL Handles Computation Dependency
# --------------------------------------
#
# In the :doc:`previous tutorial <L0_neighbor_sampling_overview>`, you
# have seen that the computation dependency for message passing of a
# single node can be described as a series of *message flow graphs* (MFG).
#
# |image1|
#
# .. |image1| image:: https://data.dgl.ai/tutorial/img/bipartite.gif
#

######################################################################
# Defining Neighbor Sampler and Data Loader in DGL
# ------------------------------------------------
#
# DGL provides tools to iterate over the dataset in minibatches
# while generating the computation dependencies to compute their outputs
# with the MFGs above. For node classification, you can use
# ``dgl.dataloading.NodeDataLoader`` for iterating over the dataset.
# It accepts a sampler object to control how to generate the computation
# dependencies in the form of MFGs.  DGL provides
# implementations of common sampling algorithms such as
# ``dgl.dataloading.MultiLayerNeighborSampler`` which randomly picks
# a fixed number of neighbors for each node.
#
# .. note::
#
#    To write your own neighbor sampler, please refer to :ref:`this user
#    guide section <guide-minibatch-customizing-neighborhood-sampler>`.
#
# The syntax of ``dgl.dataloading.NodeDataLoader`` is mostly similar to a
# PyTorch ``DataLoader``, with the addition that it needs a graph to
# generate computation dependency from, a set of node IDs to iterate on,
# and the neighbor sampler you defined.
#
# Let’s say that each node will gather messages from 4 neighbors on each
# layer. The code defining the data loader and neighbor sampler will look
# like the following.
#

sampler = dgl.dataloading.MultiLayerNeighborSampler([4, 4])
train_dataloader = dgl.dataloading.NodeDataLoader(
    # The following arguments are specific to NodeDataLoader.
    graph,              # The graph
    train_nids,         # The node IDs to iterate over in minibatches
    sampler,            # The neighbor sampler
    device=device,      # Put the sampled MFGs on CPU or GPU
    # The following arguments are inherited from PyTorch DataLoader.
    batch_size=1024,    # Batch size
    shuffle=True,       # Whether to shuffle the nodes for every epoch
    drop_last=False,    # Whether to drop the last incomplete batch
    num_workers=0       # Number of sampler processes
)

######################################################################
# .. note::
#
#    Since DGL 0.7 neighborhood sampling on GPU is supported.  Please
#    refer to :ref:`guide-minibatch-gpu-sampling` if you are
#    interested.
#

######################################################################
# You can iterate over the data loader and see what it yields.
#

input_nodes, output_nodes, mfgs = example_minibatch = next(iter(train_dataloader))
print(example_minibatch)
print("To compute {} nodes' outputs, we need {} nodes' input features".format(len(output_nodes), len(input_nodes)))

######################################################################
# ``NodeDataLoader`` gives us three items per iteration.
#
# -  An ID tensor for the input nodes, i.e., nodes whose input features
#    are needed on the first GNN layer for this minibatch.
# -  An ID tensor for the output nodes, i.e. nodes whose representations
#    are to be computed.
# -  A list of MFGs storing the computation dependencies
#    for each GNN layer.
#

######################################################################
# You can get the source and destination node IDs of the MFGs
# and verify that the first few source nodes are always the same as the destination
# nodes.  As we described in the :doc:`overview <L0_neighbor_sampling_overview>`,
# destination nodes' own features from the previous layer may also be necessary in
# the computation of the new features.
#

mfg_0_src = mfgs[0].srcdata[dgl.NID]
mfg_0_dst = mfgs[0].dstdata[dgl.NID]
print(mfg_0_src)
print(mfg_0_dst)
print(torch.equal(mfg_0_src[:mfgs[0].num_dst_nodes()], mfg_0_dst))

######################################################################
# Defining Model
# --------------
#
# Let’s consider training a 2-layer GraphSAGE with neighbor sampling. The
# model can be written as follows:
#

import torch.nn as nn
import torch.nn.functional as F
from dgl.nn import SAGEConv

class Model(nn.Module):
    def __init__(self, in_feats, h_feats, num_classes):
        super(Model, self).__init__()
        self.conv1 = SAGEConv(in_feats, h_feats, aggregator_type='mean')
        self.conv2 = SAGEConv(h_feats, num_classes, aggregator_type='mean')
        self.h_feats = h_feats

    def forward(self, mfgs, x):
        # Lines that are changed are marked with an arrow: "<---"

        h_dst = x[:mfgs[0].num_dst_nodes()]  # <---
        h = self.conv1(mfgs[0], (x, h_dst))  # <---
        h = F.relu(h)
        h_dst = h[:mfgs[1].num_dst_nodes()]  # <---
        h = self.conv2(mfgs[1], (h, h_dst))  # <---
        return h

model = Model(num_features, 128, num_classes).to(device)

######################################################################
# If you compare against the code in the
# :doc:`introduction <../blitz/1_introduction>`, you will notice several
# differences:
#
# -  **DGL GNN layers on MFGs**. Instead of computing on the
#    full graph:
#
#    .. code:: python
#
#       h = self.conv1(g, x)
#
#    you only compute on the sampled MFG:
#
#    .. code:: python
#
#       h = self.conv1(mfgs[0], (x, h_dst))
#
#    All DGL’s GNN modules support message passing on MFGs,
#    where you supply a pair of features, one for source nodes and another
#    for destination nodes.
#
# -  **Feature slicing for self-dependency**. There are statements that
#    perform slicing to obtain the previous-layer representation of the
#     nodes:
#
#    .. code:: python
#
#       h_dst = x[:mfgs[0].num_dst_nodes()]
#
#    ``num_dst_nodes`` method works with MFGs, where it will
#    return the number of destination nodes.
#
#    Since the first few source nodes of the yielded MFG are
#    always the same as the destination nodes, these statements obtain the
#    representations of the destination nodes on the previous layer. They are
#    then combined with neighbor aggregation in ``dgl.nn.SAGEConv`` layer.
#
# .. note::
#
#    See the :doc:`custom message passing
#    tutorial <L4_message_passing>` for more details on how to
#    manipulate MFGs produced in this way, such as the usage
#    of ``num_dst_nodes``.
#

######################################################################
# Defining Training Loop
# ----------------------
#
# The following initializes the model and defines the optimizer.
#

opt = torch.optim.Adam(model.parameters())

######################################################################
# When computing the validation score for model selection, usually you can
# also do neighbor sampling. To do that, you need to define another data
# loader.
#

valid_dataloader = dgl.dataloading.NodeDataLoader(
    graph, valid_nids, sampler,
    batch_size=1024,
    shuffle=False,
    drop_last=False,
    num_workers=0,
    device=device
)

######################################################################
# The following is a training loop that performs validation every epoch.
# It also saves the model with the best validation accuracy into a file.
#

import tqdm
import sklearn.metrics

best_accuracy = 0
best_model_path = 'model.pt'
for epoch in range(10):
    model.train()

    with tqdm.tqdm(train_dataloader) as tq:
        for step, (input_nodes, output_nodes, mfgs) in enumerate(tq):
            # feature copy from CPU to GPU takes place here
            inputs = mfgs[0].srcdata['feat']
            labels = mfgs[-1].dstdata['label']

            predictions = model(mfgs, inputs)

            loss = F.cross_entropy(predictions, labels)
            opt.zero_grad()
            loss.backward()
            opt.step()

            accuracy = sklearn.metrics.accuracy_score(labels.cpu().numpy(), predictions.argmax(1).detach().cpu().numpy())

            tq.set_postfix({'loss': '%.03f' % loss.item(), 'acc': '%.03f' % accuracy}, refresh=False)

    model.eval()

    predictions = []
    labels = []
    with tqdm.tqdm(valid_dataloader) as tq, torch.no_grad():
        for input_nodes, output_nodes, mfgs in tq:
            inputs = mfgs[0].srcdata['feat']
            labels.append(mfgs[-1].dstdata['label'].cpu().numpy())
            predictions.append(model(mfgs, inputs).argmax(1).cpu().numpy())
        predictions = np.concatenate(predictions)
        labels = np.concatenate(labels)
        accuracy = sklearn.metrics.accuracy_score(labels, predictions)
        print('Epoch {} Validation Accuracy {}'.format(epoch, accuracy))
        if best_accuracy < accuracy:
            best_accuracy = accuracy
            torch.save(model.state_dict(), best_model_path)

        # Note that this tutorial do not train the whole model to the end.
        break

######################################################################
# Conclusion
# ----------
#
# In this tutorial, you have learned how to train a multi-layer GraphSAGE
# with neighbor sampling.
#
# What’s next?
# ------------
#
# -  :doc:`Stochastic training of GNN for link
#    prediction <L2_large_link_prediction>`.
# -  :doc:`Adapting your custom GNN module for stochastic
#    training <L4_message_passing>`.
# -  During inference you may wish to disable neighbor sampling. If so,
#    please refer to the :ref:`user guide on exact offline
#    inference <guide-minibatch-inference>`.
#

# Thumbnail Courtesy: Stanford CS224W Notes
# sphinx_gallery_thumbnail_path = '_static/blitz_1_introduction.png'


#!/usr/bin/python
# -*- coding: UTF-8 -*-

import timeit
import numpy as np

import sys
import random as rand

class Queue_array:
    """
    顺序队列
    """

    def __init__(self,capacity):

        self._items = [None]*(capacity+1) #最后一个位置 空置
        self._capacity = capacity
        self._head = 0
        self._tail = 0

    def enqueue(self,item):
        """
        入队
        :param item:
        :return:
        """
        if self._tail== self._capacity: # 队列的最后一个位置为空置，队尾指针指在此处

            if self._head!=0: # 进行数据的搬移 ，在头部腾出空间，插入新的元素

                self._items[0:self._tail-self._head]=self._items[self._head:self._tail]

                self._tail= self._tail-self._head
                self._head=0

            else: # self._head==0 并且 self._tail== self._capacity 表示 队列已满
                print('the Queue is full!')
                return False

        self._items[self._tail]=item
        self._tail+=1

        return True

    def dequeue(self):
        """
        出队
        :return:
        """
        if self._head==self._tail: # 队列为空
            print('the Queue is empty!')
            return None

        res=self._items[self._head]

        self._items[self._head]=None
        self._head += 1

        return res

    def __repr__(self):
        return ','.join(self._items[self._head : self._tail])

class CircularQueue:
    """
    循环队列
    """
    def __init__(self,capacity):

        self._items = [None]*(capacity)
        self._capacity = capacity
        self._head = 0
        self._tail = 0

    def enqueue(self,item):
        """
        入队
        循环队列 省略了 数据搬移的 开销
        :param item:
        :return:
        """

        if (self._tail+1) % self._capacity==self._head: # (tail+1)% n=head  表示 队列已满
            print('the Queue is full!')
            return False

        self._items[self._tail]=item
        self._tail=(self._tail+1)%self._capacity

        return True

    def dequeue(self):
        """
        出队
        :return:
        """
        if self._head==self._tail: # 队列为空
            print('the Queue is empty!')
            return None

        res=self._items[self._head]

        self._items[self._head]=None
        self._head = (self._head+1)%self._capacity

        return res

class BlockingQueue:
    """
    阻塞队列

    """
    def __init__(self, capacity):
        self._items = []
        self._capacity = capacity

    def producer(self,item): #TODO:多线程调用，然后给队列加锁

        if len(self._items)<=self._capacity:
            self._items.append(item)
            return True
        else:
            print('the Queue is full!')
            return False

    def consumer(self):

        if len(self._items)>0:
            res=self._items.pop()
            return res
        else:
            print('the Queue is empty!')
            return None

if __name__ == '__main__':

    # 1. 顺序队列
    # queue=Queue_array(8)
    # string_list=['a','b','c','d','e','f','g','h']
    #
    # for ele in string_list:
    #     queue.enqueue(ele)
    #
    # print(queue._items)
    #
    # queue.enqueue('i')
    #
    # print('pop:',queue.dequeue())
    # print('pop:', queue.dequeue())
    # print('pop:', queue.dequeue())
    # print(queue._items)
    #
    # queue.enqueue('i')
    # print(queue)

    #2. 循环队列
    queue = CircularQueue(8)

    string_list=['e','f','g','h','i','j']

    for ele in string_list:
        queue.enqueue(ele)

    print(queue._items)

    for i in range(3):
        print('pop:',queue.dequeue())

    print(queue._items)

    queue.enqueue('a')
    queue.enqueue('b')
    print(queue._items)

    queue.enqueue('c')
    queue.enqueue('d')
    print(queue._items)

    queue.enqueue('e')


#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time : 2020/11/18 15:50
# @File : pandas_operate
import random

import numpy as np
import pandas as pd
from pandas import datetime

def create_dataframe():
    random.seed(1)
    rnd1 = [random.randrange(1, 20) for x in range(40)]
    rnd2 = [random.randrange(1, 20) for x in range(40)]
    rnd3 = [random.randrange(1, 20) for x in range(40)]
    daytime = pd.date_range('2020-11-09', '2020-12-18')
    data = pd.DataFrame({'daytime': daytime, 'rnd1': rnd1, 'rnd2': rnd2, 'rnd3': rnd3})
    return data

def dataframe_operate():
    """ loc可以按照索引的值来进行行列选择，包含结尾。
        iloc是按照索引的位置来进行选取，不关心索引的具体值是多少，只关心位置是多少，所以使用iloc时方括号中只能使用数值。
        at的使用方法与loc类似，但是比loc有更快的访问数据的速度，只能访问单个元素，不能访问多个元素。
        iat对于iloc的关系就像at对于loc的关系，是一种更快的基于索引位置的选择方法，同at一样只能访问单个元素。
        df[['A列名','B列名']]   #选取列
        df[df['status']=='active']   #选取status列值为active的行数据
        df[df['status'].isnull()]   #选取status列值为空的行数据
        df[df['status'].isin(['waiting', 'running'])]   #选取status列值为waiting 和running的行数据
        df.sort_values(by='active',ascending=False)         # df以active 列降序排序
        gb=df.groupby('active', sort=False)        #df以active 列数据分组
        gy.groups.keys()
        gy.get_group(190)['A']
        g.get_group(k).head(n)

    """
    data = create_dataframe()
    print('DataFrame数据表：\n', data)
    print('DataFrame统计：\n', data.describe())
    print('DataFrame列索引名称：\n', data.columns)
    print('DataFrame行索引名称：\n', data.index)
    print('DataFrame总计行：', len(data), ' 列：', data.columns.size)
    print('DataFrame等于10的数据：\n', data[data == 10])
    print('DataFrame中rnd1大于10的数据：\n', data[data.rnd1 > 10])
    print('DataFrame统计各个值个数：\n', data['rnd2'].value_counts())

    print('DataFrame切取2之5行：\n', data[2:5])  # 前闭后开，包括前不包括后
    print('DataFrame loc取2之5行：\n', data.loc[2:5])
    print('DataFrame提取列[[]]：\n', data[['rnd1', 'rnd3']])  # 返回的是DataFrame属性
    print('DataFrame提取列.：\n', data.rnd1)  # 返回的是Series类型
    print('DataFrame提取列[]：\n', data['rnd1'])  # 返回的是Series类型
    print('DataFrame提取列[[]]：\n', data[['rnd1']])
    print('DataFrame提取前3行：\n', data.head(3))
    print('DataFrame提取后3行：\n', data.tail(3))
    print('DataFrame区块选择：\n', data[3:6][['rnd1', 'rnd3']])
    print('DataFrame loc区块选择：\n', data.loc[3:6, ['rnd1', 'rnd3']])
    print('DataFrame提取第四行二列：\n', data.iat[3, 2])

    index = data.set_index('daytime')  # 将行索引设定为日期
    print('设置日期列为索引：\n', index.head())
    # 生成两个特定日期
    startday = datetime(2020, 11, 12)
    endday = datetime(2020, 11, 16)
    print('DataFrame loc建索引切取：\n', index.loc[startday:endday])
    print('DataFrame iloc行选择最后一行：', type(index.iloc[-1]), '\n', index.iloc[-1])  # 选取DataFrame最后一行，返回的是Series
    print('DataFrame iloc行选择最后一行：', type(index.iloc[-1:]), '\n', index.iloc[-1:])  # 选取DataFrame最后一行，返回的是DataFrame
    print('DataFrame iloc行选择：\n', index.iloc[2:5, :])
    print('DataFrame iloc列选择：\n', index.iloc[:, [1, 2]].head())
    print('DataFrame iloc区块选择：\n', index.iloc[[1, 2, 2], [1, 2]])
    dt = data[data['rnd1'].isin([2, 6, 8, 10])]
    print('DataFrame选取特定行：\n', dt)
    dt = data[~data['rnd1'].isin([2, 6, 8, 10])]
    print('DataFrame排除特定行：\n', dt)

    # 选取某行含有特定数值的列
    cols = [x for i, x in enumerate(data.columns) if len(set([13]) & set(data.iloc[:, i].values.tolist())) > 0]
    # 利用enumerate对row0进行遍历，将含有数字3的列放入cols中
    print('选取列：', cols)
    # df1=data[cols]   选取含有特定数值的列
    df1 = data.drop(cols, axis=1)  # 利用drop方法将含有特定数值的列删除
    print('排除特定数值列', df1)

def dataframe_groupby():
    df = create_dataframe()
    gb = df.groupby(by='rnd1')
    print('DataFrame.groupby()求最大值：\n', gb.max())
    print('DataFrame.groupby()求最小值：\n', gb.min())
    print('DataFrame.groupby()分组keys：\n', gb.keys)
    print('DataFrame.groupby()分组keys值：\n', gb.groups.keys())
    print('DataFrame.groupby()分组rnd1统计：\n', gb['rnd1'].value_counts())
    print('DataFrame.groupby()统计：\n', gb.describe)
    print('DataFrame.groupby()选择最后一行值分组的daytime：\n', gb.get_group(int(df.iloc[-1]['rnd1']))['daytime'])
    print('DataFrame.groupby()选择第一个分组的前三行：\n', gb.get_group(list(gb.groups.keys())[3]).head(3))

def dataframe_query_eval():
    """DataFrame.query(expr, inplace=False, **kwargs)参数说明：
        expr：引用字符串形式的表达式以过滤数据。
        inplace：如果该值为True, 它将在原始DataFrame中进行更改。
        kwargs：引用其他关键字参数。
        Pandas.eval()与DataFrame.eval()类似，都支持表达式操作，注意操作是否符合表达式的数据类型，运算操作只能是数字。
        Pandas.eval()支持多个DataFrame操作，DataFrame.eval()是对自身的数据做操作。
    """
    keys = [10, 12, 16]
    df = create_dataframe()
    sv = df.sort_values(by='rnd2', ascending=True)
    print('DataFrame求个列总和：\n', sv.sum())
    print('DataFrame.sort_values()统计：\n', sv.describe())
    print('DataFrame.query()筛选过滤：\n', sv.query('rnd1 > @keys[0] and rnd2 < @keys[1]'))
    print('DataFrame.query()筛选过滤：\n', sv.query('rnd1 > 10 and rnd2 < 12'))
    print('DataFrame.eval()算术运算：\n', sv.eval('-rnd1 * rnd2 / rnd3 + ( rnd2 + rnd3 )'))
    print('DataFrame.eval()比较运算：\n', sv.eval('rnd1 <= rnd2 !=rnd3'))
    print('DataFrame.eval()位运算：\n', sv.eval('(rnd1< @keys[0]) &(rnd2 < @keys[1]) | (rnd3 < @keys[2])'))
    print('DataFrame.eval()列计算：\n', sv.eval('(rnd1 + rnd2 + rnd3)/3'))
    print('DataFrame.eval()新增列：\n', sv.eval('rowmean = (rnd1 + rnd2 + rnd3)/3', inplace=True))
    print('DataFrame.eval()列操作：\n', sv.eval('rnd1+10'))
    df1 = pd.DataFrame(np.random.randint(0, 20, (100, 3)))
    df2 = pd.DataFrame(np.random.randint(0, 20, (100, 3)))
    df3 = pd.DataFrame(np.random.randint(0, 20, (100, 3)))
    print('Pandas.eval()算术运算：\n', pd.eval('(df1 + df2) / df3'))

if __name__ == '__main__':
    print('......start......')
    # dataframe_operate()
    # dataframe_groupby()
    dataframe_query_eval()
    print('......end......')


"""
.. module:: DBSCAN_unittest

DBSCAN
*************

:Description: Unit tests for DBSCAN algorithm

:Authors: birkholz

:License: BSD 3 clause (due to using API of DBSCAN from scikit-learn)

:Version:

:Created on:

"""
import sys

sys.path.insert(1, '../')
sys.path.insert(1, '../kemlglearn/')

import numpy as np

import unittest
import time
import logging

from sklearn.datasets import make_blobs, make_moons
import matplotlib.pyplot as plt
import pandas as pd

# Our implementation
from kemlglearn.cluster.DBSCAN import DBSCAN, NOISE, NO_CLUSTER

# Reference implementation for comparison
from sklearn.cluster import DBSCAN as sklDBSCAN

from kemlglearn.datasets.gen_dbscan_dataset import gen_dbscan_dataset1, gen_dbscan_dataset2, gen_dbscan_dataset3, \
                                        gen_dbscan_blobs, gen_dbscan_moons, gen_gridbscan_synth10k

# Helper function to plot DBSCAN results based on DBSCAN example in scikit-learn user guide
# If data has more than two dimensions, the first two will be used
def plot_dbscan_results(x, labels, core_sample_indices):
    core_samples_mask=np.zeros_like(labels, dtype=bool)
    core_samples_mask[core_sample_indices] = True
    n_clusters_=len(set(labels))-(1 if -1 in labels else 0)
    n_noise_ = list(labels).count(-1)

    unique_labels = set(labels)
    colors = [plt.cm.Spectral(each) for each in np.linspace(0, 1, len(unique_labels))]

    for k, col in zip(unique_labels, colors):
        if k == -1:
            col = [0, 0, 0, 1] # Black for noise

        class_member_mask = (labels == k)

        xy = x[class_member_mask & core_samples_mask]
        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
            markeredgecolor='k', markersize=14)

        xy = x[class_member_mask & ~core_samples_mask]
        plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
            markeredgecolor='k', markersize=6)

    # Show clusters (and noise)
    plt.title(f'Estimated number of clusters: {n_clusters_}')
    plt.show()

def dbscan_equivalent_results(labels1, corepts1, labels2, corepts2):
    # This is not the absolute most efficient implementation, but it
    # doesn't have to be since it's just a unit test.

    # Confirm results are identical. They should have all the same core points, but
    # it is possible for a border point to be grouped with a different cluster if it's
    # close enough to two different clusters.

    # Noise must be identical
    noise1=set(np.where(labels1==NOISE)[0])
    noise2=set(np.where(labels2==NOISE)[0])

    if noise1 != noise2:
        return False

    # Core samples must be identical (although they might belong to different clusters)
    if set(corepts1) != set(corepts2):
        return False

    # Since the clusters may have different cluster IDs, we'll have to kind of guess which
    # clusters match.
    uniquelabels1 = np.unique(labels1)
    uniquelabels2 = np.unique(labels2)

    clusterdifferences = []

    for lbl1 in uniquelabels1:
        if lbl1 == -1:
            # Already checked noise, so continue with next cluster
            continue
        # Try to find most similar cluster
        set1=set(np.where(labels1==lbl1)[0])
        # Format of tuple: how many different samples there are, the list of different samples
        bestcluster = (None, None)
        for lbl2 in uniquelabels2:
            if lbl2 == -1:
                continue
            set2 = set(np.where(labels2==lbl2)[0])
            cluster_difference = set1 ^ set2
            mag_cluster_difference = len(cluster_difference)
            if bestcluster == (None,None):
                bestcluster = (mag_cluster_difference, cluster_difference)
            elif mag_cluster_difference < bestcluster[0]:
                bestcluster = (mag_cluster_difference, cluster_difference)

            # If they're identical, we can skip to the next iteration directly
            if mag_cluster_difference == 0:
                break

        if bestcluster[0] != None and bestcluster[0] > 0:
            clusterdifferences.append(bestcluster)

    if len(clusterdifferences) != 0:
        for difference in clusterdifferences:
            for pt in difference[1]:
                if pt in set(corepts1):
                    # If any core points are different, we're in trouble. Border
                    # points can be in different clusters though.
                    return False

    return True

def gen_shuffle_unshuffle_idx(data):
    shuffle_idx=np.random.permutation(len(data))
    revidx=[(x,y) for x,y in enumerate(shuffle_idx)]
    revidx=sorted(revidx, key=lambda x: x[1])
    unshuffle_idx=np.array([x[0] for x in revidx])
    return shuffle_idx, unshuffle_idx

class TestDBSCAN(unittest.TestCase):
    def test_db1(self):
        # 200 iterations generally seems to be enough to get an ambiguous border point
        for rs in range(200):
            X=gen_dbscan_dataset1(random_state=rs)

            # Shuffle the data to force (possibly) some border points to be different
            shuffle, unshuffle = gen_shuffle_unshuffle_idx(X)
            shuffleX = X[shuffle]
            mydbscan=DBSCAN(eps=43,min_samples=4).fit(shuffleX)
            # Unshuffle the results so they can be compared with original indices
            unshuffled_labels=mydbscan.labels_[unshuffle]
            unshuffled_corepts=np.array([shuffle[x] for x in mydbscan.core_sample_indices_])

            #plot_dbscan_results(shuffleX, mydbscan.labels_, mydbscan.core_sample_indices_)

            refdbscan=sklDBSCAN(eps=43,min_samples=4).fit(X)
            #plot_dbscan_results(X,refdbscan.labels_, refdbscan.core_sample_indices_)

            self.assertEqual(True,dbscan_equivalent_results(
                             unshuffled_labels, unshuffled_corepts,
                             refdbscan.labels_, refdbscan.core_sample_indices_))

    def test_db2(self):
        # 100 iterations generally seems to be enough to get an ambiguous border point
        for rs in range(100):
            X=gen_dbscan_dataset2(random_state=rs)

            # Shuffle the data to force (possibly) some border points to be different
            shuffle, unshuffle = gen_shuffle_unshuffle_idx(X)
            shuffleX = X[shuffle]
            mydbscan=DBSCAN(eps=36,min_samples=4).fit(shuffleX)
            # Unshuffle the results so they can be compared with original indices
            unshuffled_labels=mydbscan.labels_[unshuffle]
            unshuffled_corepts=np.array([shuffle[x] for x in mydbscan.core_sample_indices_])

            #plot_dbscan_results(shuffleX, mydbscan.labels_, mydbscan.core_sample_indices_)

            refdbscan=sklDBSCAN(eps=36,min_samples=4).fit(X)
            #plot_dbscan_results(X,refdbscan.labels_, refdbscan.core_sample_indices_)

            self.assertEqual(True,dbscan_equivalent_results(
                             unshuffled_labels, unshuffled_corepts,
                             refdbscan.labels_, refdbscan.core_sample_indices_))

    def test_db3(self):
        # 500 iterations generally seems to be enough to get an ambiguous border point
        for rs in range(500):
            X=gen_dbscan_dataset3(random_state=rs)

            # Shuffle the data to force (possibly) some border points to be different
            shuffle, unshuffle = gen_shuffle_unshuffle_idx(X)
            shuffleX = X[shuffle]
            mydbscan=DBSCAN(eps=40,min_samples=4).fit(shuffleX)
            # Unshuffle the results so they can be compared with original indices
            unshuffled_labels=mydbscan.labels_[unshuffle]
            unshuffled_corepts=np.array([shuffle[x] for x in mydbscan.core_sample_indices_])

            #plot_dbscan_results(shuffleX, mydbscan.labels_, mydbscan.core_sample_indices_)

            refdbscan=sklDBSCAN(eps=40,min_samples=4).fit(X)
            #plot_dbscan_results(X,refdbscan.labels_, refdbscan.core_sample_indices_)

            self.assertEqual(True,dbscan_equivalent_results(
                             unshuffled_labels, unshuffled_corepts,
                             refdbscan.labels_, refdbscan.core_sample_indices_))

    def test_4d(self):
        # Dataset from https://archive.ics.uci.edu/ml/datasets/banknote+authentication
        df=pd.read_csv('../data/banknotepreprocessed.csv', header=None)
        X=df.to_numpy()

        # Shuffle the data to force (possibly) some border points to be different
        shuffle, unshuffle = gen_shuffle_unshuffle_idx(X)
        shuffleX = X[shuffle]
        # Used DBSCAN_param_helper.py to pick out good parameters
        mydbscan=DBSCAN(eps=1.66,min_samples=8).fit(shuffleX)
        # Unshuffle the results so they can be compared with original indices
        unshuffled_labels=mydbscan.labels_[unshuffle]
        unshuffled_corepts=np.array([shuffle[x] for x in mydbscan.core_sample_indices_])

        refdbscan=sklDBSCAN(eps=1.66,min_samples=8).fit(X)

        self.assertEqual(True,dbscan_equivalent_results(
                         unshuffled_labels, unshuffled_corepts,
                         refdbscan.labels_, refdbscan.core_sample_indices_))

    def test_timing(self):
        our_timing = 0
        their_timing = 0
        iterations=10

        # For some reason, the first iteration is always much longer
        # than the rest (caching, other???). Let's discard that result
        first_iteration = True

        for rs in range(iterations):
            X=gen_dbscan_dataset1(random_state=rs)
            #X=gen_gridbscan_synth10k(random_state=rs)

            shuffle, unshuffle = gen_shuffle_unshuffle_idx(X)
            shuffleX = X[shuffle]
            start_time=time.time()
            mydbscan=DBSCAN(eps=43,min_samples=4).fit(shuffleX)
            end_time=time.time()
            if not first_iteration:
                our_timing += (end_time-start_time)

            start_time=time.time()
            refdbscan=sklDBSCAN(eps=43,min_samples=4).fit(shuffleX)
            end_time=time.time()
            if not first_iteration:
                their_timing += (end_time-start_time)
            first_iteration = False

        print('Timing results:')
        print(f'Ours:   {our_timing/(iterations-1)}')
        print(f'Theirs: {their_timing/(iterations-1)}')

# These aren't really "tests," but rather demonstrations
class TestDBSCANInteractive(unittest.TestCase):
    def test_one(self):
        n_samples = 4000
        n_blobs = 4
        X=gen_dbscan_blobs(n_samples, n_blobs, std=0.50, random_state=None)
        mydbscan=DBSCAN(eps=0.3,min_samples=4).fit(X)
        plot_dbscan_results(X, mydbscan.labels_, mydbscan.core_sample_indices_)

        X = gen_dbscan_moons(n_samples)
        mydbscan=DBSCAN(eps=0.085,min_samples=4).fit(X)
        plot_dbscan_results(X,mydbscan.labels_, mydbscan.core_sample_indices_)
        self.assertEqual(1,1)

    def test_two(self):
        n_samples = 400
        n_blobs = 4
        X, y_true = make_blobs(n_samples=n_samples,
                               centers=n_blobs,
                               cluster_std=0.60,
                               random_state=0)
        X = X[:, ::-1]

        # Show raw points
        plt.figure(1)
        plt.scatter(X[:,0], X[:,1])
        plt.show()

        mydbscan=DBSCAN(eps=0.5,min_samples=4).fit(X)

        # The following code is based on DBSCAN example in scikit-learn user guide
        plot_dbscan_results(X,mydbscan.labels_, mydbscan.core_sample_indices_)

        self.assertEqual(1,1)

    def test_db1(self):
        X=gen_dbscan_dataset1()
        mydbscan=DBSCAN(eps=43,min_samples=4).fit(X)
        plot_dbscan_results(X, mydbscan.labels_, mydbscan.core_sample_indices_)
        self.assertEqual(1,1)

    def test_db2(self):
        X=gen_dbscan_dataset2()
        mydbscan=DBSCAN(eps=36,min_samples=4).fit(X)
        plot_dbscan_results(X, mydbscan.labels_, mydbscan.core_sample_indices_)
        self.assertEqual(1,1)

    def test_db3(self):
        X=gen_dbscan_dataset3()
        mydbscan=DBSCAN(eps=40,min_samples=4).fit(X)
        plot_dbscan_results(X, mydbscan.labels_, mydbscan.core_sample_indices_)
        self.assertEqual(1,1)

if __name__ == '__main__':
    # Set up logging subsystem. Level should be one of: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET
    logging.basicConfig()
    # DEBUG is a good level for unit tests, but you can change to INFO if you want to shut it up
    # dbscan_logger=logging.get_logger('dbscan')
    # dbscan_logger.setLevel(logging.DEBUG)
    unittest.main()


# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.4'
#       jupytext_version: 1.2.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
#

#

#

#
Computational Seismology

#
Reproducible Papers - Syngine Paper

#

#

#

# + {"deletable": true, "editable": true, "cell_type": "markdown"}
# ---
#
# # Figure 6:  Data Quality - Detecting Wrong Times with Synthetics
#
# This notebook is part of the supplementary materials for the Syngine paper and reproduces figure 6.
#
# This notebook creates the phase relative times figure. Requires matplotlib >= 1.5 and an ObsPy version (>= 1.0) with the syngine client as well as instaseis.
#
# ##### Authors:
# * Lion Krischer ([@krischer](https://github.com/krischer))

# + {"deletable": true, "editable": true}
# %matplotlib inline

# + {"deletable": true, "editable": true}
import obspy
import numpy as np
from obspy.clients.fdsn import Client
from obspy.clients.syngine import Client as SyngineClient

import itertools

import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
plt.style.use("seaborn-paper")

# + {"deletable": true, "editable": true}
c_iris = Client("IRIS")

# + {"deletable": true, "editable": true}
# Get all stations from the Caltech Regional Seismic Network activate
# at the given time.
inv = c_iris.get_stations(level="response", format="xml", channel="BHZ",
                          network="CI", location="  ",
                          startbefore=obspy.UTCDateTime(2012, 1, 1),
                          endafter=obspy.UTCDateTime(2016, 1, 1))
inv.plot(projection="local");
print(inv)

# + {"deletable": true, "editable": true}
# Downloaded from the GCMT page.
# cat = obspy.read_events("./cmtsolutions.txt", format="CMTSOLUTION").filter("latitude > -20", "longitude > 165")

# This is a prototype - we thus are very selective regarding which data we use...
training = ["smi:local/cmtsolution/201510202152A/event",
            "smi:local/cmtsolution/201212212228A/event",
            "smi:local/cmtsolution/201203090709A/event"]
validate = "smi:local/cmtsolution/201401011603A/event"
application = "smi:local/cmtsolution/201501230347A/event"

all_data = []
all_data.extend(training)
all_data.append(validate)
all_data.append(application)

cat.events = [_i for _i in cat.events if _i.resource_id.id in all_data]

print(cat)
cat.plot();

# + {"deletable": true, "editable": true}
# For all of them: download the syngine data 50 seconds before the p-phase
# and 50 seconds after for the 5s ak135f DB.
c_syngine = SyngineClient()

# + {"deletable": true, "editable": true}
# Download all syngine data.
synthetic_data = {}

# Build up bulk request.
bulk = [
    {"networkcode": "CI", "stationcode": _i.code,
     "latitude": _i.latitude, "longitude": _i.longitude} for _i in inv[0]]

# Actually download everything.
for _i, event in enumerate(cat):
    origin = [_i for _i in event.origins if _i.origin_type == "hypocenter"][0]
    mt = event.focal_mechanisms[0].moment_tensor.tensor
    print("Downloading data for event %i of %i ..." % (_i + 1, len(cat)))
    st_syn = c_syngine.get_waveforms_bulk(
        model="ak135f_5s", bulk=bulk, sourcelatitude=origin.latitude,
        sourcelongitude=origin.longitude, sourcedepthinmeters=origin.depth,
        sourcemomenttensor=[mt.m_rr, mt.m_tt, mt.m_pp, mt.m_rt, mt.m_rp, mt.m_tp],
        origintime=origin.time,
        components="Z",
        # Avoid downsampling - just upsample to the data sampling rate.
        dt=1.0 / 40.0,
        kernelwidth=10,
        units="velocity",
        starttime="P-50", endtime="P+50")

    this_event = {"event_object": event, "stream": st_syn}
    synthetic_data[event.resource_id] = this_event
    print("  -> Downloaded %i synthetic traces." % len(st_syn))

# + {"deletable": true, "editable": true}
# Reordering ... I don't want to redownload again...
data = synthetic_data
for value in data.values():
    value["synthetic_stream"] = value["stream"]
    del value["stream"]

# + {"deletable": true, "editable": true}
# Now download all real data.
for _i, value in enumerate(data.values()):
    print("Downloading data for event %i of %i ..." % (_i + 1, len(cat)))
    st_obs = c_iris.get_waveforms_bulk(
        [(tr.stats.network, tr.stats.station, "",
          "BHZ", tr.stats.starttime, tr.stats.endtime) for tr in value["synthetic_stream"]])
    value["observed_stream"] = st_obs
    print("  -> Downloaded %i observed traces." % len(st_obs))

# + {"deletable": true, "editable": true}
# Cleanup - here we make our job easy and just remove all
# stations for which we don't have data for all events.
# Real implementations will of course have to deal with this.

common_stations = set.intersection(*[set((tr.stats.network, tr.stats.station)
                                         for tr in value["observed_stream"])
                                     for value in data.values()])
print(len(common_stations))

for value in data.values():
    value["observed_stream"] = obspy.Stream(traces=[
        tr for tr in value["observed_stream"] if
        (tr.stats.network, tr.stats.station) in common_stations])
    value["synthetic_stream"] = obspy.Stream(traces=[
        tr for tr in value["synthetic_stream"] if
        (tr.stats.network, tr.stats.station) in common_stations])

# + {"deletable": true, "editable": true}
# Create a new inventory object which only contains the common stations.
from copy import deepcopy
filtered_inv = deepcopy(inv)
filtered_inv[0].stations = [_i for _i in filtered_inv[0].stations if ("CI", _i.code) in common_stations]
filtered_inv.plot(projection="local");

# + {"deletable": true, "editable": true}
from obspy.signal.cross_correlation import xcorr_pick_correction

collected_stats = []

good_data = data

# Now calculate the P phase delay for all stations.
for network, station in common_stations:
    print(network, station)
    for event, this_data in good_data.items():
        obs_tr = this_data["observed_stream"].select(network=network, station=station)[0].copy()
        syn_tr = this_data["synthetic_stream"].select(network=network, station=station)[0].copy()

        # Artificially shift the data for two stations for the "application" event.
        if event.id == application:
            if station == "BEL":
                obs_tr.stats.starttime -= 1.0
            if station == "PHL":
                obs_tr.stats.starttime += 1.0

        # p-phase should be centered in the synthetic trace.
        pick = syn_tr.stats.starttime + (syn_tr.stats.endtime - syn_tr.stats.starttime) / 2.0

        # Process data.
        obs_tr.detrend("linear")\
            .taper(max_percentage=0.05, type="hann")\
            .remove_response(inventory=filtered_inv, pre_filt=(0.02, 0.01, 1, 5))\
            .filter("bandpass", freqmin=0.08, freqmax=0.2, corners=4, zerophase=True)\
            ._ltrim(30.0)\
            ._rtrim(30.0)\
            .taper(max_percentage=0.3, type="hann")
        obs_tr.trim(obs_tr.stats.starttime - 50, obs_tr.stats.endtime + 50, pad=True, fill_value=0.0)

        # Also filter the synthetics.
        syn_tr.taper(max_percentage=0.05, type="hann")
        syn_tr.filter("bandpass", freqmin=0.08, freqmax=0.2, corners=4, zerophase=True)
        syn_tr.data = np.require(syn_tr, requirements=["C"])

        # Interpolate both to the same sample points.
        starttime = max(obs_tr.stats.starttime, syn_tr.stats.starttime)
        endtime = min(obs_tr.stats.endtime, syn_tr.stats.endtime)
        npts = int((endtime - starttime) // (1.0 / 40.0) - 1)
        obs_tr.interpolate(sampling_rate=40.0, method="lanczos", a=5,
                           starttime=starttime, npts=npts)
        syn_tr.interpolate(sampling_rate=40.0, method="lanczos", a=5,
                           starttime=starttime, npts=npts)

        print(event.id, network, station)

        # Subsample precision.
        try:
            shift, corr = xcorr_pick_correction(pick, obs_tr, pick, syn_tr, t_before=20,
                                                t_after=20, cc_maxlag=12)
        except:
            shift, corr = None, None
        collected_stats.append({"time_shift": shift, "correlation": corr,
                                "network": network, "station": station, "event": event.id})

        if "processed_observed_stream" not in this_data:
            this_data["processed_observed_stream"] = obspy.Stream()

        this_data["processed_observed_stream"] += obs_tr

        if "processed_synthetic_stream" not in this_data:
            this_data["processed_synthetic_stream"] = obspy.Stream()

        this_data["processed_synthetic_stream"] += syn_tr

# + {"deletable": true, "editable": true}
import pandas
pandas.set_option('display.max_rows', 10)
cs = pandas.DataFrame(collected_stats)
original_cs = pandas.DataFrame(collected_stats)

# + {"deletable": true, "editable": true}
# Now for all events: substract the mean time shift to account for
# unexplained structure and unknown origin time.
cs.loc[:, "time_shift"] = cs.groupby("event").time_shift.transform(lambda df: df - df.mean())
cs = cs.sort_values("station")
cs

# + {"deletable": true, "editable": true}
# Calculate the reference time shift for each station.
reference_shifts = dict(cs.loc[cs.event.isin(training)].groupby("station").time_shift.mean())
reference_shifts

# + {"deletable": true, "editable": true}
import matplotlib.pyplot as plt

def check_event(event, ax=None, ticks=False):
    _ds = cs[cs.event == event]
    mean = _ds.time_shift.mean()

    things = []

    for _i in _ds.itertuples():
        things.append({"station": _i.station,
                       "time_shift": _i.time_shift - mean,
                       "difference_to_mean": (_i.time_shift - mean) - reference_shifts[_i.station]})

    temp = pandas.DataFrame(things).sort_values("station")

    if ax is None:
        plt.figure(figsize=(15, 5))
        ax = plt.gca()
    plt.bar(np.arange(len(_ds)) - 0.4, temp.difference_to_mean, 0.8, color="0.2")
    plt.ylim(-1.4, 1.4)
    if ticks:
        plt.xticks(range(len(_ds)), temp.station, rotation="vertical")
    else:
        plt.xticks(range(len(_ds)), [""] * len(_ds), rotation="vertical")

    plt.xlim(-1, 23)
check_event(application)

# + {"deletable": true, "editable": true}
check_event(validate)

# + {"deletable": true, "editable": true}
check_event(application)

# + {"deletable": true, "editable": true}
def plot_data(event, station, show=True, legend=False):
    val = data[obspy.core.event.ResourceIdentifier(event)]
    obs = val["processed_observed_stream"].select(station=station)[0]
    syn = val["processed_synthetic_stream"].select(station=station)[0]

    plt.plot(obs.times(), obs.data * 1E6, color="0.3", linestyle="--", linewidth=5,
             label="Observed Data")
    plt.plot(syn.times(), syn.data * 1E6, color="0.1", linestyle=":", label="Synthetic Data")

    ts= original_cs[(original_cs.event == validate) &
                    (original_cs.station == "TIN")].time_shift

    plt.plot(syn.times() - float(ts), syn.data * 1E6, color="k", label="Shifted Synthetic Data")
    plt.ylabel("Velocity [$\mu$m/s]")
    plt.xlabel("Relative Time [s]")
    plt.xlim(20, 65)

    plt.xticks([30, 40, 50, 60], ["30", "40", "50", "60"])

    if legend:
        plt.legend(loc="lower left")

    plt.text(0.03, 0.92, "Station: CI.%s" % station, transform=plt.gca().transAxes, va="top")
    if show:
        plt.show()

plot_data(validate, "TIN")
plot_data(validate, "SCI2")
plot_data(validate, "MPP")

# + {"deletable": true, "editable": true}
# Create final complete plot.
import matplotlib.gridspec as gridspec

fig = plt.figure(figsize=(10, 5))

gs1 = gridspec.GridSpec(2, 1, wspace=0, hspace=0.04, left=0, right=0.27, bottom=0.08, top=0.98)
ax1 = fig.add_subplot(gs1[0])
ax2 = fig.add_subplot(gs1[1])

gs2 = gridspec.GridSpec(3, 1, wspace=0, hspace=0, left=0.32, right=0.62, bottom=0.08, top=0.98)
ax3 = fig.add_subplot(gs2[0])
ax4 = fig.add_subplot(gs2[1])
ax5 = fig.add_subplot(gs2[2])

gs3 = gridspec.GridSpec(3, 1, wspace=0, hspace=0, left=0.64, right=0.94, bottom=0.08, top=0.98)
ax6 = fig.add_subplot(gs3[0])
ax7 = fig.add_subplot(gs3[1])
ax8 = fig.add_subplot(gs3[2])

station_lats = [sta.latitude for net in filtered_inv for sta in net]
station_lngs = [sta.longitude for net in filtered_inv for sta in net]

ev_lats = [event.origins[0].latitude for event in cat]
ev_lngs = [event.origins[0].longitude for event in cat]

m_ortho = Basemap(resolution='c', projection='ortho', lat_0=10.,
                  lon_0=-150., ax=ax1)
m_ortho.drawcoastlines(color="0.3", linewidth=1.0)
m_ortho.drawcountries(color="0.8")
m_ortho.fillcontinents(color="0.9")

_xs, _ys = m_ortho(station_lngs, station_lats)
_xe, _ye = m_ortho(ev_lngs, ev_lats)

for sta, ev in itertools.product(zip(station_lngs, station_lats),
                                 zip(ev_lngs, ev_lats)):
    m_ortho.drawgreatcircle(*sta, *ev, linewidth=1.2,
                            color="0.6", alpha=0.5)

m_ortho.scatter(_xs, _ys, marker="v", s=120, zorder=10,
                color="k", edgecolor="w")
m_ortho.scatter(_xe, _ye, marker="o", s=120, zorder=10,
                color="k", edgecolor="w")

deg2m = 2 * np.pi * 6371 * 1000 / 360
m_local = Basemap(projection='aea',
                  resolution="i",
                  area_thresh=1000.0, lat_0=34.9,
                  lon_0=-118,
                  width=deg2m * 8, height=deg2m * 8,
                  ax=ax2)
m_local.drawmapboundary()
m_local.drawrivers(color="0.7")
m_local.drawcoastlines(color="0.3", linewidth=1.4)
m_local.drawcountries(color="0.4", linewidth=1.1)
m_local.fillcontinents(color="0.9")
m_local.drawstates(color="0.5", linewidth=0.9)

_x, _y = m_local(station_lngs, station_lats)
m_local.scatter(_x, _y, marker="v", s=140, zorder=10,
                color="k", edgecolor="w")

# Waveforms.
plt.sca(ax3)
plot_data(validate, "TIN", show=False)
plt.yticks(np.linspace(-1.5, 1.5, 7))

plt.sca(ax4)
plot_data(validate, "SCI2", show=False)
plt.yticks(np.linspace(-2, 2, 9))

plt.sca(ax5)
plot_data(validate, "MPP", show=False, legend=True)
plt.yticks(np.linspace(-2, 2, 9))

plt.sca(ax6)
stations = list(reference_shifts.keys())
delays = [reference_shifts[_i] for _i in stations]
plt.bar(np.arange(len(delays)) - 0.4, delays, 0.8, color="0.2")
plt.xticks(range(len(delays)), [""] * len(delays))
ax6.yaxis.tick_right()
ax6.yaxis.set_label_position("right")
plt.xlim(-1, 23)
plt.ylim(-0.8, 0.7)
plt.yticks([-0.5, 0, 0.5])
plt.ylabel("Average time shift [s]")
plt.text(0.97, 0.90, "Reference Time Shifts", transform=plt.gca().transAxes, va="top", ha="right")

plt.sca(ax7)
check_event(validate, ax=ax7)
ax7.yaxis.tick_right()
ax7.yaxis.set_label_position("right")
plt.yticks(np.linspace(-1.0, 1.0, 5))
plt.ylabel("Time shift deviation [s]")
plt.text(0.97, 0.90, "Validation Event", transform=plt.gca().transAxes, va="top", ha="right")

plt.sca(ax8)
check_event(application, ax=ax8, ticks=True)
ax8.yaxis.tick_right()
ax8.yaxis.set_label_position("right")
plt.yticks(np.linspace(-1.0, 1.0, 5))
plt.ylabel("Time shift deviation [s]")
plt.text(0.97, 0.90, "Test Event", transform=plt.gca().transAxes, va="top", ha="right")

plt.savefig("data_quality.pdf")
plt.show()