#!/usr/bin/env python
# coding: utf-8

# <!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
# doconce format html hw2.do.txt --no_mako -->
# <!-- dom:TITLE: PHY321: Classical Mechanics 1 -->

# # PHY321: Classical Mechanics 1
# **Homework 2, due January 28 (Midnight)**
#
# Date: **Jan 18, 2022**

# ### Practicalities about  homeworks and projects
#
# 1. You can work in groups (optimal groups are often 2-3 people) or by yourself. If you work as a group you can hand in one answer only if you wish. **Remember to write your name(s)**!
#
# 2. Homeworks are available 10 days  before the deadline.
#
# 3. How do I(we)  hand in?  You can hand in the paper and pencil exercises as a  scanned  document. For this homework this applies to exercises 1-5. The scanned document should be uploaded to D2L. Alternatively, you can hand in everyhting (if you are ok with typing mathematical formulae using say Latex) as a jupyter notebook at D2L. The numerical exercise(s) (exercise 6 here) should always be handed in as a jupyter notebook by the deadline at D2L.

# ### Exercise 1 (10 pt), Forces, discussion questions, test your intuition
#
# * 1a (2pt) Single force. Can an object affected only by a single force have zero acceleration?
#
# * 1b (2pt) Zero velocity. If you throw a ball vertically it has zero velocity at its maximum point. Does it also have zero acceleration at this point?
#
# * 1c (3pt) Acceleration of gravity. You measure the acceleration of gravity in an elevator moving at a velocity of 9.8m/s downwards. What will you measure?
#
# * 1d (3pt) Air resistance. You throw a ball straight up and measure the velocity as it passes you on its way down. Will the velocity be larger, the same, or smaller if you did the same experiment in vacuum?

# ### Exercise 2 (10 pt), setting up forces, Newton's second law
#
# Useful material here to read is
# 1. Taylor chapters 1.3 and 1.4 and
#
# 2. Malthe-Sørenssen chapters 5.1, 5.2 and 5.3
#
# A person jumps from an airplane, falling freely for several seconds before the person pulls the cord of her parachute and the parachute unfolds.
# * 2a (3pt)  Identify the forces acting on the parachuter and draw a free-body diagram of the parachuter before the person has pulled the cord.
#
# * 2b (3pt)  Identify the forces acting on the parachuter and draw a free-body diagram of the parachuter after the person has pulled the cord.
#
# * 2c (4pt)  Sketch the net force acting on the parachuter as a function of time, F(t).

# ### Exercise 3 (10 pt), Space shuttle with air resistance
#
# Useful material here to read is
# 1. Malthe-Sørenssen chapters 5.1, 5.2 and 5.3
#
# During lift-off of the space shuttle the engines provide a force of $35\times 10^{6}$ N. The mass of the shuttle is approximately
# $2\times 10^6$ kg.
# * 3a (3pt) Draw a free-body diagram of the space shuttle immediately after lift-off.
#
# * 3b (3pt)  Find an expression for the acceleration of the space shuttle immediately after lift-off.
#
# Let us assume that the force from the engines is constant, and that the mass of the
# space shuttle does not change significantly over the first 20 s.
# * 3c (4pt) Find the velocity and position of the space shuttle after 20 s if you ignore air resistance.

# ### Exercise 4 (15 pt), now hitting a golf ball
#
# Useful material here to read is
# 1. Taylor chapters 1.3-1.6 and
#
# 2. Malthe-Sørenssen chapter 6.3-6.4 and 7.1-7.3
#
# **Taylor exercise 1.35**. The formulae you obtain here will be useful for the numerical exercises below (see exercise 6 below).

# ### Exercise 5 (15 pt), hitting a puck instead
#
# Taylor exercise 1.38.

# ### Exercise 6 (40pt), Numerical elements, moving to more than one dimension
#
# **This exercise should be handed in as a jupyter-notebook** at D2L. Remember to write your name(s).
#
# Last week we:
# 1. Analytically mapped 1D motion over some time
#
# 2. Gained practice with functions
#
# 3. Reviewed vectors and matrices in Python
#
# This week we will:
# 1. Practice using Python syntax and variable manipulation
#
# 2. Utilize analytical solutions to create more refined functions
#
# 3. Work in two, three or even higher dimensions
#
# This material will then serve as background for the numerical part of homework 3. The first part is a simple warm-up, with hints and suggestions you can use for the code to write below.

# In[1]:

get_ipython().run_line_magic('matplotlib', 'inline')

# As usual, here are some useful packages we will be using. Feel free to use more and experiment as you wish.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
get_ipython().run_line_magic('matplotlib', 'inline')

# In class (the falling baseball example) we used an  analytical expression for the height of a falling ball.
# In the first homework we used instead the position from experiment (Usain Bolt's 100m record run) and stored this
# information with one-dimensional arrays in Python.
#
# Let us get some practice with this. The cell below creates two arrays,
# one containing the times to be analyzed and the other containing the $x$
# and $y$ components of the position vector at each point in time.  This is a two-dimensional object. The
# second array is initially empty. Then we define  the initial
# position to be $x=2$ and $y=1$. Take a look at the code and comments
# to get an understanding of what is happening. Feel free to play around with it.

# In[2]:

tf = 4 #length of value to be analyzed
dt = .001 # step sizes
t = np.arange(0.0,tf,dt) # Creates an evenly spaced time array going from 0 to 3.999, with step sizes .001
p = np.zeros((len(t), 2)) # Creates an empty array of [x,y] arrays (our vectors). Array size is same as the one for time.
p[0] = [2.0,1.0] # This sets the inital position to be x = 2 and y = 1

# Below we are printing specific values of our array to see what is being
# stored where. The first number in the array $r[]$ represents which array
# iteration we are looking at, while the number after the  represents
# which listed number in the array iteration we are getting back.

# In[3]:

print(p[0]) # Prints the first array
print(p[0,:]) # Same as above, these commands are interchangeable

# In[4]:

print(p[3999]) # Prints the 4000th array

# In[5]:

print(p[0,0]) # Prints the first value of the first array

# In[6]:

print(p[0,1]) # Prints the second value of first array
print(p[:,0]) # Prints the first value of all the arrays

# Then try running this cell. Notice how it gives an error since we did not implement a third dimension into our arrays

# In[7]:

print(p[:,2])

# In the cell below we want to manipulate the arrays.
# In this example we make each vector's $x$ component valued the same as their respective vector's position in the iteration and the $y$ value will be twice that value, except for  the first vector, which we have already set.
# That is we have $p[0] = [2,1], p[1] = [1,2], p[2] = [2,4], p[3] = [3,6], ...$
#
# Here we set up an array for $x$ and $y$ values.

# In[8]:

for i in range(1,3999):
    p[i] = [i,2*i]
# Checker cell to make sure your code is performing correctly
c = 0
for i in range(0,3999):
    if i == 0:
        if p[i,0] != 2.0:
            c += 1
        if p[i,1] != 1.0:
            c += 1
    else:
        if p[i,0] != 1.0*i:
            c += 1
        if p[i,1] != 2.0*i:
            c += 1

if c == 0:
    print("Success!")
else:
    print("There is an error in your code")

# You could also think of an alternative way of storing the above information. Feel free to explore how to store
# multidimensional objects.
#
# Last week we studied Usain Bolt's 100m run and in class we studied a falling baseball. We made basic plots of the baseball
# moving in one dimension. This week we will be working with a three-dimensional variant. This will be useful for our next homeworks and numerical projects.
#
# Assume we have a soccer ball moving in three dimensions with the following trajectory:
#
# 1. $x(t) = 10t\cos{45^{\circ}} $
#
# 2. $y(t) = 10t\sin{45^{\circ}} $
#
# 3. $z(t) = 10t - \dfrac{9.81}{2}t^2$
#
# Now let us create a three-dimensional (3D) plot using these equations. In the cell below
# we write the equations into their respective labels. We fix a final time in the code below.
#
# Important Concept: Numpy comes with many mathematical packages, some
# of them being the trigonometric functions sine, cosine, tangent. We
# are going to utilize these this week. Additionally, these functions
# work with radians, so we will also be using a function from Numpy that
# converts degrees to radians.

# In[9]:

tf = 2.04  # The final time to be evaluated
dt = 0.1  # The time step size
t = np.arange(0,tf,dt) # The time array
theta_deg = 45 # Degrees
theta_rad = np.radians(theta_deg) # Converts degrees to their radian counterparts
x = 10*t*np.cos(theta_rad) # Equation for our x component, utilizing np.cos() and our calculated radians
y = 10*t*np.sin(theta_rad) # Put the y equation here
z = 10*t-9.81/2*t**2# Put the z equation here

# Then we plot it

# In[10]:

## Once you have entered the proper equations in the cell above, run this cell to plot in 3D
fig = plt.axes(projection='3d')
fig.set_xlabel('x')
fig.set_ylabel('y')
fig.set_zlabel('z')
fig.scatter(x,y,z)

# * 6a (8pt) How would you express $x(t)$, $y(t)$, $z(t)$ for this problem as a single vector, $\boldsymbol{r}(t)$?
#
# Then run the code and plot using the array $r$

# In[11]:

## Run this code to plot using our r array
fig = plt.axes(projection='3d')
fig.set_xlabel('x')
fig.set_ylabel('y')
fig.set_zlabel('z')
fig.scatter(r[0],r[1],r[2])

# * 6b (8pt) What do you think the benefits and/or disadvantages are from expressing our three equations as a single array/vector? This can be both from a computational and physics stand point. Use the **Numpy** package to also print the maximum $x$, $y$ and $z$ components from $\boldsymbol{r}$.
#
# Complete Exercise 4 above (Taylor exercise 1.35) before moving further. (Recall that the golf ball was hit due east at an angle $\theta$ with respect to the horizontal, and the coordinate directions are $x$ measured east, $y$ north, and $z$ vertically up.)
#
# * 6c (8pt) What is the analytical solution for our theoretical golf ball's position $\boldsymbol{r}(t)$ over time from Exercise 4?  Also what is the formula for the time $t_f$ when the golf ball hits the ground? Use this to develop a program with a function called for example Golfball that utilizes our analytical solutions. This program should take in an initial velocity and the angle $\theta$ that the golfball was hit with in degrees. It should also produce  a 3D graph of the motion. You need also to find the maximum values for $x$, $y$ and $z$.
#
# * 6d (8pt) Given initial values of $v_i = 90 m/s$, $\theta = 30^{\circ}$, what would our maximum x, y and z components be?
#
# * 6e (8pt) Given initial values of $v_i = 45 m/s$, $\theta = 45^{\circ}$, what would our maximum x, y and z components be?


"""
==========================
GOES-16: True Color Recipe
==========================
By: [Brian Blaylock](http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/home.html)
with help from Julien Chastang (UCAR-Unidata).

Additional notebooks analyzing GOES-16 and other data can be found in [Brian's
GitHub repository](https://github.com/blaylockbk/pyBKB_v3/).

This notebook shows how to make a true color image from the GOES-16
Advanced Baseline Imager (ABI) level 2 data. We will plot the image with
matplotlib and Cartopy. The methods shown here are stitched together from the
following online resources:

- [**CIMSS True Color RGB Quick Guide**](http://cimss.ssec.wisc.edu/goes/OCLOFactSheetPDFs/ABIQuickGuide_CIMSSRGB_v2.pdf)
- [ABI Bands Quick Information Guides](https://www.goes-r.gov/education/ABI-bands-quick-info.html)
- [Open Commons Consortium](http://edc.occ-data.org/goes16/python/)
- [GeoNetCast Blog](https://geonetcast.wordpress.com/2017/07/25/geonetclass-manipulating-goes-16-data-with-python-part-vi/)
- [Proj documentation](https://proj4.org/operations/projections/geos.html?highlight=geostationary)

True color images are an RGB composite of the following three channels:

|        --| Wavelength   | Channel | Description   |
|----------|--------------|---------|---------------|
| **Red**  | 0.64 µm |    2    | Red Visible   |
| **Green**| 0.86 µm |    3    | Veggie Near-IR|
| **Blue** | 0.47 µm |    1    | Blue Visible  |

For this demo, we use the **Level 2 Multichannel formated data** (ABI-L2-MCMIP)
for the CONUS domain. This file contains all sixteen channels on the ABI fixed
grid (~2 km grid spacing).

GOES-16 data is downloaded from Unidata, but you may also
download GOES-16 or 17 files from NOAA's GOES archive on [Amazon S3](https://aws.amazon.com/public-datasets/goes/).
I created a [web interface](http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/cgi-bin/goes16_download.cgi?source=aws&satellite=noaa-goes16&domain=C&product=ABI-L2-MCMIP)
to easily download files from the Amazon archive. For scripted or bulk
downloads, you should use `rclone` or `AWS CLI`. You may also download files
from the [Environmental Data Commons](http://edc.occ-data.org/goes16/getdata/)
and [NOAA
CLASS](https://www.avl.class.noaa.gov/saa/products/search?sub_id=0&datatype_family=GRABIPRD&submit.x=25&submit.y=9).

File names have the following format...

`OR_ABI-L2-MCMIPC-M3_G16_s20181781922189_e20181781924562_c20181781925075.nc`

`OR`     - Indicates the system is operational

`ABI`    - Instrument type

`L2`     - Level 2 Data

`MCMIP`  - Multichannel Cloud and Moisture Imagery products

`c`      - CONUS file (created every 5 minutes).

`M3`     - Scan mode

`G16`    - GOES-16

`sYYYYJJJHHMMSSZ` - Scan start: 4 digit year, 3 digit day of year (Julian day), hour, minute, second, tenth second

`eYYYYJJJHHMMSSZ` - Scan end

`cYYYYJJJHHMMSSZ` - File Creation
`.nc`    - NetCDF file extension

"""  # noqa: E501

######################################################################
# First, import the libraries we will use
# ---------------------------------------

from datetime import datetime

import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import metpy  # noqa: F401
import numpy as np
import xarray

######################################################################
# Open the GOES-16 NetCDF File
# ----------------------------

# Open the file with xarray.
# The opened file is assigned to "C" for the CONUS domain.

FILE = ('http://ramadda-jetstream.unidata.ucar.edu/repository/opendap'
        '/4ef52e10-a7da-4405-bff4-e48f68bb6ba2/entry.das#fillmismatch')
C = xarray.open_dataset(FILE)
C

######################################################################
# Date and Time Information
# ----------------------------
# Each file represents the data collected during one scan sequence for the
# domain. There are several different time stamps in this file, which are also
# found in the file's name.
# I'm not a fan of numpy datetime, so I convert it to a regular datetime

# Scan's start time, converted to datetime object
scan_start = datetime.strptime(C.time_coverage_start, '%Y-%m-%dT%H:%M:%S.%fZ')

# Scan's end time, converted to datetime object
scan_end = datetime.strptime(C.time_coverage_end, '%Y-%m-%dT%H:%M:%S.%fZ')

# File creation time, convert to datetime object
file_created = datetime.strptime(C.date_created, '%Y-%m-%dT%H:%M:%S.%fZ')

# The 't' variable is the scan's midpoint time
midpoint = str(C['t'].data)[:-8]
print(C.t.data)
scan_mid = datetime.strptime(midpoint, '%Y-%m-%dT%H:%M:%S.%f')

print('Scan Start    : {}'.format(scan_start))
print('Scan midpoint : {}'.format(scan_mid))
print('Scan End      : {}'.format(scan_end))
print('File Created  : {}'.format(file_created))
print('Scan Duration : {:.2f} minutes'.format((scan_end-scan_start).seconds/60))

######################################################################
# True Color RGB Recipe
# ---------------------
#
# Color images are a Red-Green-Blue (RGB) composite of three different
# channels. To make a "Natural True Color" image we assign the following
# channels as our R, G, and B values:
#
# | --                   | RED         | GREEN          | BLUE         |
# |----------------------|-------------|----------------|--------------|
# | **Name**             | Red Visible | Near-IR Veggie | Blue Visible |
# | **Wavelength**       | 0.64 µm     | 0.86 µm        | 0.47 µm      |
# | **Channel**          | 2           | 3              | 1            |
# | **Units**            | Reflectance | Reflectance    | Reflectance  |
# | **Range of Values**  | 0-1         | 0-1            | 0-1          |
# | **Gamma Correction** | 2.2         | 2.2            | 2.2          |
#
#
# Some important details to know about...
#
# **Value Range**: The data units of channel 1, 2, and 3 are in reflectance and
# have a range of values between 0 and 1. RGB values must also be between 0 and
# 1.
#
# **Gamma Correction**: A gamma correction is applied to control the brightness
# and make the image not look too dark.
# `corrected_value = value^(1/gamma)`.
# Most displays have a decoding gamma of 2.2. Read more about gamma correction
# at the following links...
# [source1](https://en.wikipedia.org/wiki/Gamma_correction) and
# [source2](https://www.cambridgeincolour.com/tutorials/gamma-correction.htm)).
#
# **True Green**: The GREEN "veggie" channel on GOES-16 does not measure
# visible green light. Instead, it measures a near-infrared band sensitive to
# chlorophyll. We could use that channel in place of green, but it would make
# the green in our image appear too vibrant. Instead, we will tone-down the
# green channel by interpolating the value to simulate a natural green color.
#
#       `TrueGreen = (0.45*RED) + (0.1*GREEN) + (0.45*BLUE)`
#
# Now we can begin putting the pieces together...

# Confirm that each band is the wavelength we are interested in
for band in [2, 3, 1]:
    print('{} is {:.2f} {}'.format(
        C['band_wavelength_C{:02d}'.format(band)].long_name,
        float(C['band_wavelength_C{:02d}'.format(band)][0]),
        C['band_wavelength_C{:02d}'.format(band)].units))

######################################################################

# Load the three channels into appropriate R, G, and B variables
R = C['CMI_C02'].data
G = C['CMI_C03'].data
B = C['CMI_C01'].data

######################################################################

# Apply range limits for each channel. RGB values must be between 0 and 1
R = np.clip(R, 0, 1)
G = np.clip(G, 0, 1)
B = np.clip(B, 0, 1)

######################################################################

# Apply a gamma correction to the image to correct ABI detector brightness
gamma = 2.2
R = np.power(R, 1/gamma)
G = np.power(G, 1/gamma)
B = np.power(B, 1/gamma)

######################################################################

# Calculate the "True" Green
G_true = 0.45 * R + 0.1 * G + 0.45 * B
G_true = np.clip(G_true, 0, 1)  # apply limits again, just in case.

######################################################################
# Simple Image
# -----------------
#
# Use `plt.imshow` to get a quick look at the channels and RGB composite we
# created.
#
# First, plot each channel individually. The deeper the color means the
# satellite is observing more light in that channel. Clouds appear white because
# they reflect lots of red, green, and blue light. Notice that the land reflects
# a lot of "green" in the veggie channel because this channel is sensitive to
# the chlorophyll.

fig, ([ax1, ax2, ax3, ax4]) = plt.subplots(1, 4, figsize=(16, 3))

ax1.imshow(R, cmap='Reds', vmax=1, vmin=0)
ax1.set_title('Red', fontweight='bold')
ax1.axis('off')

ax2.imshow(G, cmap='Greens', vmax=1, vmin=0)
ax2.set_title('Veggie', fontweight='bold')
ax2.axis('off')

ax3.imshow(G_true, cmap='Greens', vmax=1, vmin=0)
ax3.set_title('"True" Green', fontweight='bold')
ax3.axis('off')

ax4.imshow(B, cmap='Blues', vmax=1, vmin=0)
ax4.set_title('Blue', fontweight='bold')
ax4.axis('off')

plt.subplots_adjust(wspace=.02)

######################################################################
# The addition of the three channels results in a color image. Combine the three
# channels with a stacked array and display the image with `imshow`.

# The RGB array with the raw veggie band
RGB_veggie = np.dstack([R, G, B])

# The RGB array for the true color image
RGB = np.dstack([R, G_true, B])

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))

# The RGB using the raw veggie band
ax1.imshow(RGB_veggie)
ax1.set_title('GOES-16 RGB Raw Veggie', fontweight='bold', loc='left',
              fontsize=12)
ax1.set_title('{}'.format(scan_start.strftime('%d %B %Y %H:%M UTC ')),
              loc='right')
ax1.axis('off')

# The RGB for the true color image
ax2.imshow(RGB)
ax2.set_title('GOES-16 RGB True Color', fontweight='bold', loc='left',
              fontsize=12)
ax2.set_title('{}'.format(scan_start.strftime('%d %B %Y %H:%M UTC ')),
              loc='right')
ax2.axis('off')

######################################################################
# Plot with `Cartopy` Geostationary Projection
# ----------------------------------------------
#
# The image above is not georeferenced. You can see the land and oceans, but we
# do have enough information to draw state and country boundaries. Use the
# `metpy.io` package to obtain the projection information from the file.  Then
# use `Cartopy` to plot the image on a map. The GOES data and image is on a
# [geostationary projection
# ](https://proj4.org/operations/projections/geos.html?highlight=geostationary).

# We'll use the `CMI_C02` variable as a 'hook' to get the CF metadata.
dat = C.metpy.parse_cf('CMI_C02')

geos = dat.metpy.cartopy_crs

# We also need the x (north/south) and y (east/west) axis sweep of the ABI data
x = dat.x
y = dat.y

######################################################################
# The geostationary projection is the easiest way to plot the image on a
# map. Essentially, we are stretching the image across a map with the same
# projection and dimensions as the data.

fig = plt.figure(figsize=(15, 12))

# Create axis with Geostationary projection
ax = fig.add_subplot(1, 1, 1, projection=geos)

# Add the RGB image to the figure. The data is in the same projection as the
# axis we just created.
ax.imshow(RGB, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()), transform=geos)

# Add Coastlines and States
ax.coastlines(resolution='50m', color='black', linewidth=0.25)
ax.add_feature(ccrs.cartopy.feature.STATES, linewidth=0.25)

plt.title('GOES-16 True Color', loc='left', fontweight='bold', fontsize=15)
plt.title('{}'.format(scan_start.strftime('%d %B %Y %H:%M UTC ')), loc='right')

plt.show()

######################################################################
# Using other projections
# ----------------------------------------------
#
# Changing the projections with `Cartopy` is straightforward. Here we display
# the GOES-16 data on a Lambert Conformal projection.

fig = plt.figure(figsize=(15, 12))

# Generate an Cartopy projection
lc = ccrs.LambertConformal(central_longitude=-97.5, standard_parallels=(38.5,
                                                                        38.5))

ax = fig.add_subplot(1, 1, 1, projection=lc)
ax.set_extent([-135, -60, 10, 65], crs=ccrs.PlateCarree())

ax.imshow(RGB, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()),
          transform=geos,
          interpolation='none')
ax.coastlines(resolution='50m', color='black', linewidth=0.5)
ax.add_feature(ccrs.cartopy.feature.STATES, linewidth=0.5)

plt.title('GOES-16 True Color', loc='left', fontweight='bold', fontsize=15)
plt.title('{}'.format(scan_start.strftime('%d %B %Y %H:%M UTC ')), loc='right')

plt.show()

######################################################################
# Plot with `Cartopy`: Plate Carrée  Cylindrical Projection
# ---------------------------------------------------------
#
# It is often useful to zoom on a specific location. This image will zoom in on
# Utah.

fig = plt.figure(figsize=(8, 8))

pc = ccrs.PlateCarree()

ax = fig.add_subplot(1, 1, 1, projection=pc)
ax.set_extent([-114.75, -108.25, 36, 43], crs=pc)

ax.imshow(RGB, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()),
          transform=geos,
          interpolation='none')

ax.coastlines(resolution='50m', color='black', linewidth=1)
ax.add_feature(ccrs.cartopy.feature.STATES)

plt.title('GOES-16 True Color', loc='left', fontweight='bold', fontsize=15)
plt.title('{}'.format(scan_start.strftime('%d %B %Y %H:%M UTC ')), loc='right')

plt.show()

######################################################################
# Overlay Nighttime IR when dark
# ------------------------------
#
# At nighttime, the visible wavelengths do not measure anything and is just
# black. There is information, however, from other channels we can use to see
# clouds at night. To view clouds in portions of the domain experiencing
# nighttime, we will overlay the clean infrared (IR) channel over the true color
# image.
#
# First, open a file where the scan shows partial night area and create the true
# color RGB as before.

# A GOES-16 file with half day and half night

FILE = ('http://ramadda-jetstream.unidata.ucar.edu/repository/opendap'
        '/85da3304-b910-472b-aedf-a6d8c1148131/entry.das#fillmismatch')
C = xarray.open_dataset(FILE)

# Scan's start time, converted to datetime object
scan_start = datetime.strptime(C.time_coverage_start, '%Y-%m-%dT%H:%M:%S.%fZ')

# Create the RGB like we did before

# Load the three channels into appropriate R, G, and B
R = C['CMI_C02'].data
G = C['CMI_C03'].data
B = C['CMI_C01'].data

# Apply range limits for each channel. RGB values must be between 0 and 1
R = np.clip(R, 0, 1)
G = np.clip(G, 0, 1)
B = np.clip(B, 0, 1)

# Apply the gamma correction
gamma = 2.2
R = np.power(R, 1/gamma)
G = np.power(G, 1/gamma)
B = np.power(B, 1/gamma)

# Calculate the "True" Green
G_true = 0.45 * R + 0.1 * G + 0.45 * B
G_true = np.clip(G_true, 0, 1)

# The final RGB array :)
RGB = np.dstack([R, G_true, B])

######################################################################
# Load the Clear IR  10.3 µm channel (Band 13)
# -------------------------------------------------------
#
# When you print the contents of channel 13, notice that the unit of the clean
# IR channel is *brightness temperature*, NOT reflectance. We need to normalize
# the values between 0 and 1 before we can use it in our RGB image.  In this
# case, we normalize the values between 90 Kelvin and 313 Kelvin.

print(C['CMI_C13'])

######################################################################
# Apply the normalization...

cleanIR = C['CMI_C13'].data

# Normalize the channel between a range.
#       cleanIR = (cleanIR-minimumValue)/(maximumValue-minimumValue)
cleanIR = (cleanIR-90)/(313-90)

# Apply range limits to make sure values are between 0 and 1
cleanIR = np.clip(cleanIR, 0, 1)

# Invert colors so that cold clouds are white
cleanIR = 1 - cleanIR

# Lessen the brightness of the coldest clouds so they don't appear so bright
# when we overlay it on the true color image.
cleanIR = cleanIR/1.4

# Yes, we still need 3 channels as RGB values. This will be a grey image.
RGB_cleanIR = np.dstack([cleanIR, cleanIR, cleanIR])

######################################################################
# Show the true color and clean IR images
# ---------------------------------------
#
# We want to overlay these two images, so the clean IR fills in the night sky
# on the True Color image. This way we can still see the clouds at night.

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))

ax1.set_title('True Color', fontweight='bold')
ax1.imshow(RGB)
ax1.axis('off')

ax2.set_title('Clean IR', fontweight='bold')
ax2.imshow(RGB_cleanIR)
ax2.axis('off')

plt.show()

######################################################################
#
# To fill in the dark area on the true color image, we will set each RGB channel
# to equal the maximum value between the visible channels and the IR
# channels. When this is done, where RGB values are black in the true color
# image RGB = (0,0,0), it will be replaced with a higher value of the `cleanIR
# RGB`.
#
# Note that if the clean IR has really bright, cold clouds in the daylight, they
# will replace the color values in the true color image making the clouds appear
# more white. Still, it makes a nice plot and let's you see clouds when it is
# night.

# Maximize the RGB values between the True Color Image and Clean IR image
RGB_ColorIR = np.dstack([np.maximum(R, cleanIR), np.maximum(G_true, cleanIR),
                         np.maximum(B, cleanIR)])

######################################################################

fig = plt.figure(figsize=(15, 12))

ax = fig.add_subplot(1, 1, 1, projection=geos)

ax.imshow(RGB_ColorIR, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()),
          transform=geos)

ax.coastlines(resolution='50m', color='black', linewidth=2)
ax.add_feature(ccrs.cartopy.feature.STATES)

plt.title('GOES-16 True Color and Night IR', loc='left', fontweight='bold',
          fontsize=15)
plt.title('{}'.format(scan_start.strftime('%H:%M UTC %d %B %Y'), loc='right'),
          loc='right')

plt.show()

######################################################################
# Adjust Image Contrast
# ---------------------
#
# I think the color looks a little dull. We could get complicated and make a
# Rayleigh correction to the data to fix the blue light scattering, but that can
# be intense. More simply, we can make the colors pop out by adjusting the image
# contrast. Adjusting image contrast is easy to do in Photoshop, and also easy
# to do in Python.
#
# We are still using the RGB values from the day/night GOES-16 ABI scan.
#
# Note: you should adjust the contrast _before_ you add in the Clean IR channel.

def contrast_correction(color, contrast):
    """
    Modify the contrast of an RGB
    See:
    https://www.dfstudios.co.uk/articles/programming/image-programming-algorithms/image-processing-algorithms-part-5-contrast-adjustment/

    Input:
        color    - an array representing the R, G, and/or B channel
        contrast - contrast correction level
    """
    F = (259*(contrast + 255))/(255.*259-contrast)
    COLOR = F*(color-.5)+.5
    COLOR = np.clip(COLOR, 0, 1)  # Force value limits 0 through 1.
    return COLOR

# Amount of contrast
contrast_amount = 105

# Apply contrast correction
RGB_contrast = contrast_correction(RGB, contrast_amount)

# Add in clean IR to the contrast-corrected True Color image
RGB_contrast_IR = np.dstack([np.maximum(RGB_contrast[:, :, 0], cleanIR),
                             np.maximum(RGB_contrast[:, :, 1], cleanIR),
                             np.maximum(RGB_contrast[:, :, 2], cleanIR)])

######################################################################

# Plot on map with Cartopy

fig = plt.figure(figsize=(15, 12))

ax1 = fig.add_subplot(1, 2, 1, projection=geos)
ax2 = fig.add_subplot(1, 2, 2, projection=geos)

ax1.imshow(RGB_ColorIR, origin='upper',
           extent=(x.min(), x.max(), y.min(), y.max()),
           transform=geos)
ax1.coastlines(resolution='50m', color='black', linewidth=2)
ax1.add_feature(ccrs.cartopy.feature.BORDERS)
ax1.set_title('True Color and Night IR')

ax2.imshow(RGB_contrast_IR, origin='upper',
           extent=(x.min(), x.max(), y.min(), y.max()),
           transform=geos)
ax2.coastlines(resolution='50m', color='black', linewidth=2)
ax2.add_feature(ccrs.cartopy.feature.BORDERS)
ax2.set_title('Contrast Correction = {}'.format(contrast_amount))

plt.subplots_adjust(wspace=.02)

######################################################################
# Can we make plots for a Mesoscale scan?
# ---------------------------------------
#
# Yes. Yes we can.

# M1 is for the Mesoscale1 NetCDF file

FILE = ('http://ramadda-jetstream.unidata.ucar.edu/repository/opendap'
        '/5e02eafa-5cee-4d00-9f58-6e201e69b014/entry.das#fillmismatch')
M1 = xarray.open_dataset(FILE)

# Load the RGB arrays
R = M1['CMI_C02'][:].data
G = M1['CMI_C03'][:].data
B = M1['CMI_C01'][:].data

# Apply range limits for each channel. RGB values must be between 0 and 1
R = np.clip(R, 0, 1)
G = np.clip(G, 0, 1)
B = np.clip(B, 0, 1)

# Apply the gamma correction
gamma = 2.2
R = np.power(R, 1/gamma)
G = np.power(G, 1/gamma)
B = np.power(B, 1/gamma)

# Calculate the "True" Green
G_true = 0.45 * R + 0.1 * G + 0.45 * B
G_true = np.clip(G_true, 0, 1)

# The final RGB array :)
RGB = np.dstack([R, G_true, B])

# Scan's start time, converted to datetime object
scan_start = datetime.strptime(M1.time_coverage_start, '%Y-%m-%dT%H:%M:%S.%fZ')

# We'll use the `CMI_C02` variable as a 'hook' to get the CF metadata.
dat = M1.metpy.parse_cf('CMI_C02')

# Need the satellite sweep x and y values, too.
x = dat.x
y = dat.y

######################################################################

fig = plt.figure(figsize=(10, 8))

ax = fig.add_subplot(1, 1, 1, projection=lc)
ax.set_extent([-125, -70, 25, 50], crs=ccrs.PlateCarree())

ax.imshow(RGB, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()),
          transform=geos)

ax.coastlines(resolution='50m', color='black', linewidth=0.5)
ax.add_feature(ccrs.cartopy.feature.STATES, linewidth=0.5)
ax.add_feature(ccrs.cartopy.feature.BORDERS, linewidth=0.5)

plt.title('GOES-16 True Color', fontweight='bold', fontsize=15, loc='left')
plt.title('Mesoscale Section 1')
plt.title('{}'.format(scan_start.strftime('%H:%M UTC %d %B %Y')), loc='right')

plt.show()

######################################################################

fig = plt.figure(figsize=(15, 12))

ax = fig.add_subplot(1, 1, 1, projection=geos)

ax.imshow(RGB, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()),
          transform=geos)

ax.coastlines(resolution='50m', color='black', linewidth=0.25)
ax.add_feature(ccrs.cartopy.feature.STATES, linewidth=0.25)

plt.title('GOES-16 True Color', fontweight='bold', fontsize=15, loc='left')
plt.title('Mesoscale Section 1')
plt.title('{}'.format(scan_start.strftime('%H:%M UTC %d %B %Y')), loc='right')

plt.show()

######################################################################
# Can we do this for a Full Disk Scan? It's possible...
# -----------------------------------------------------
#
# but data files are so large that plotting is very slow.  Feel free to
# experiment.

FILE = ('http://ramadda-jetstream.unidata.ucar.edu/repository/opendap'
        '/deb91f58-f997-41a3-a077-987529bf02b3/entry.das#fillmismatch')
F = xarray.open_dataset(FILE)

# Load the RGB arrays
R = F['CMI_C02'][:].data
G = F['CMI_C03'][:].data
B = F['CMI_C01'][:].data

# Apply range limits for each channel. RGB values must be between 0 and 1
R = np.clip(R, 0, 1)
G = np.clip(G, 0, 1)
B = np.clip(B, 0, 1)

# Apply the gamma correction
gamma = 2.2
R = np.power(R, 1/gamma)
G = np.power(G, 1/gamma)
B = np.power(B, 1/gamma)

# Calculate the "True" Green
G_true = 0.48358168 * R + 0.45706946 * B + 0.06038137 * G
G_true = np.clip(G_true, 0, 1)

# The final RGB array :)
RGB = np.dstack([R, G_true, B])

# We'll use the `CMI_C02` variable as a 'hook' to get the CF metadata.
dat = F.metpy.parse_cf('CMI_C02')

x = dat.x
y = dat.y

######################################################################
# Geostationary projection is easy...

fig = plt.figure(figsize=(10, 8))

ax = fig.add_subplot(1, 1, 1, projection=geos)

ax.imshow(RGB, origin='upper',
          extent=(x.min(), x.max(), y.min(), y.max()),
          transform=geos)

ax.coastlines(resolution='50m', color='black', linewidth=1)
ax.add_feature(ccrs.cartopy.feature.BORDERS, linewidth=1)

plt.title('GOES-16 True Color', fontweight='bold', fontsize=15, loc='left')
plt.title('Full Disk\n{}'.format(scan_start.strftime('%H:%M UTC %d %B %Y')),
          loc='right')

plt.show()

Class Files/Lab 3/Data Sources/mapping_GOES16_TrueColor.py


# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     formats: ipynb,py
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.6.0
#   kernelspec:
#     display_name: deep_ml_curriculum
#     language: python
#     name: deep_ml_curriculum
# ---

# # Pandas

# +
import pandas as pd
import pandas_profiling as pp
import numpy as np
import warnings
# %matplotlib inline

warnings.simplefilter("ignore")
# -

# ## Read Data

# Pandas has the ability to read data from various formats, including:
# - CSV
# - Excel
# - Html
# - Json
# - Feather
# - Parquet

# Let's start by reading a table from a csv file. Pandas puts the data in an object known as a `DataFrame`.

# The data we are using here is air emissions from industrial facilities in Queensland for the 2005/2006 inventory year taken from [data.gov.au](http://data.gov.au).

# !head -n 3 "../../data/processed/Emission/npi-2006-qld-air-total-emissions.csv"

df = pd.read_csv("../../data/processed/Emission/npi-2006-qld-air-total-emissions2.csv", index_col='index')

# The `DataFrame` is now stored in variable `df`. We can print it:

df

# Similarly, using the respective read method you can read the tables from other file formats supported by Pandas.

# e.g. `pd.read_excel()`

# Pandas can also read the data from SQL database and put them directly into a pandas data frame. To do that, we need to pass in the query and the connection object.

# +
import sqlite3

# create a connection to database
conn = sqlite3.connect("../b01_SQL/Sales.db")

# write a query
query = """
SELECT * from Customers
LIMIT 5
"""

pd.read_sql(query, conn)
# -

# __Tip:__ Sometimes you just want to quickly copy a portion of a dataset from a webpage or excel into a notebook. An easy way to do that is to copy the data from the source and then use `pd.read_clipboard()`. This method will create a pandas data frame from the data you copied. Note that this method only works if the notebook is running on your local machine (not on an external server).

# ## Basic Analysis
# - info
# - describe
# - pandas profiling
# - value_counts()
#

df = pd.read_csv("../../data/processed/Emission/npi-2006-qld-air-total-emissions2.csv", index_col='index')
df

# We can use `.head()` and `.tail()` to view only top or bottom rows of the table.

# top rows
df.head()

# bottom rows
df.tail()

# __Note:__ You can specify how many rows from the top or bottom of the table you want by passing in a number.

# e.g. `df.head(10)` or `df.tail(3)`

# We can use `.columns` to get a list of column names.

df.columns

# Using `.info()` method you can get a list of the columns and the type of data stored in each.

df.info()

# You can also get some basic statistical analysis of the data using `.describe()` method.

df.describe()

# To get a more detailed analysis of the data in the table we can use a package called `pandas-profiling`. This package extends Pandas and adds detailed reports of the data.
#
# If you get an error use this pre-made report
#

# +
# profile = pp.ProfileReport(df, title='Pandas Profiling Report')
# profile.to_file("profile_qld-air-emissions.html")
# profile
# -

# ## Subsetting and indexing

# There are multiple ways to get a subset of the data. `loc` is used when we want to specify the names of columns and `iloc` when we want to use the index of the columns.

#

# We can use `loc` by specifying the rows and columns we want by name. e.g. `df.loc[{row(s)}, {column(s) name}]`

# we can get a subset of a single column:

# +
df.loc[:'Q001BOR001-S52', "jurisdiction"]

# Notice we used :NAME for rows which means give me the rows up to NAME
# -

# __Note:__ `loc` has a unique property. Since it is designed to work with names of columns and rows, when you want to get a subset of rows, the result it returns is inclusive. In other words when we passed in `:10` in almost every other python object that means `0` to `9`, but in `loc` it means `0` to `10`. Likewise, `10:20` in `loc` means rows `10` to `20`.

# We can also get a subset of multiple columns by passing a list of columns we want.

df.loc['Q001BOR001-S14':'Q001BOR001-S52', ["Year", "facility_name", "substance", "quantity_in_kg"]]

# `iloc` works similar to `loc`, but instead of names we pass in index of the rows or the columns we want.

# a single column
df.iloc[:10, 5]

# __Note:__ Notice the number of rows here, and compare it with when we used `loc`.

# You can see the number corresponding to a column name here
list(enumerate(df.columns))

# multiple columns
df.iloc[10:20, [1, 9, -3, -1]]

# Another useful method to get a subset of data is using boolean indexing. Booleans are either True or False. If we pass a list of booleans, pandas will return only the rows with True in the list.

mask = df["substance"] == "Oxides of Nitrogen"
mask

# The list above has the value true only on the rows where the substance is "Oxides of Nitrogen".

# __Note:__ you can only see a small portion of the data so the True values might not be visible.

# Now if we pass this as an index into a data frame we only get the rows where substance is "Oxides of Nitrogen".

df[mask]

# This method can also be used with `loc` and `iloc`.

df.loc[
    df["substance"] == "Oxides of Nitrogen",
    ["facility_name", "substance", "quantity_in_kg"],
]

# ## Sorting
# To sort the data in the table based on a certain column we can use the `.sort_values` method. When sorting we need to specify which column we want to sort and whether we want to sort in ascending order or descending order.

df2 = df.sort_values(by="jurisdiction_facility_id", ascending=False)
df2

# __Note:__ many methods return the result as a data frame as well. This allows us to chain these operations to make the code shorter and easier to read.

#

# Let's sort the table based on the amount of Oxides of Nitrogen only.

df.loc[df["substance"] == "Oxides of Nitrogen"].sort_values(
    by="quantity_in_kg", ascending=False
)

# We can also sort based on multiple columns. To do so we need to pass in the name of the column in a list (in the order we want them to be used for sorting) and also a list to specify whether each column should be ascending or descending.

df.loc[df["substance"] == "Oxides of Nitrogen"].sort_values(
    by=["site_address_postcode", "facility_name"], ascending=[True, False]
)

# ## Data operations
# -merge
# -groupby
# -pivot_table
# -crosstab

# __Groupby:__ It aggregates the data into groups (similar to groupby in SQL). For instance, what if we wanted an average emission of each substance across all the sites? To calculate that we use `.groupby()` method.

# Since we want the average amount of substances, the columns we need will be __substance__ and __quantity_in_kg__.

groups = df[["substance", "quantity_in_kg"]].groupby(by="substance")
groups

name, group = list(groups)[0]
print(name)
group

# But it doesn't show us any tables. The reason is pandas has grouped the data into `DataFrameGroupBy` object and now we need to specify how the values should be aggregated. In this case since we want the average we use `.mean()`.

df[["substance", "quantity_in_kg"]].groupby(by="substance").mean()

# There are other useful aggregation functions such as `.std()` for standard deviation, `.median()` for median, `.count()` for the number of rows in each group, `.sum()` for sum, etc. You can also define your own aggregation function.

agg_func = lambda x: np.sqrt((x ** 2).mean())  # root mean of squares
df[["substance", "quantity_in_kg"]].groupby(by="substance").apply(agg_func)

# Do you know how to use *__lambda__* functions? If not check out this page to learn about them.

# ### Pivot Table
# Another way to represent the data is using pivot tables. You might be familiar with pivot tables in Excel. You can perform the same operations here as well.

# Let's create a pivot table that shows the amount of each substance in every postcode in the dataset.

df.pivot_table(
    index="site_address_postcode",
    columns="substance",
    values="quantity_in_kg",
    aggfunc="mean",
)

# __Note:__ `NaN` stands for Not a Number. In this case it means there was no value available for that cell. This means where you see `NaN` in the table there was no emission recorded for that substance in that specific postcode. This probably means that we can assume the emission was zero. We could let pandas know by passing in `fill_value = 0`. Then, where no value is available pandas put zero instead.
#

df.pivot_table(
    index="site_address_postcode",
    columns="substance",
    values="quantity_in_kg",
    aggfunc="mean",
    fill_value=0,
)

#

#
Exercise

#
# Now to practice what we have learned so far, let's create a table of the total emissions (quantity in kg) of the top 10 substances (most commonly recorded substances in the dataset) for each postcode.

#
# 1. Find how many times each substance occurs `substance_count`
# 2. Sort `substance_count` and find 10 most common substance `top10`
# 3. Create a seperate dataframe that shows the total weight of each substance per postcode `weight_by_postcode`
# 4. Combine `weight_by_postcode` and `top10` to get the weight by postcode of the top 10 substances
#
# ```python
# # 1. Find how many times each substance has been recorded (this has been done for you)
# # substance_count = df[["site_address_postcode", "substance"]].??
#
# # 2. Sort it and find the substances that have been recorded the most (this has been done for you)
# # top10 = substance_count.??
#
# # 3. Create a seperate table that shows the total weight of each substance per postcode (hint: pivot table or groupby)
# # pivot = ??
#
# # 4. Combine the tables, to get a subset which only includes the top 10 substances
# # pivot_top10 = ??
# ```
#
#
#
#   → Hints
#
# 1. use groupby then count
# 2. use .sort_values() then get the first 10 rows
# 3. A pivot table will do this easily
# 4. `weight_by_postcode[top10.index]`
#
#
#
#

#

#
#
#     → Solution
#
#
# ```python
# # you can replace site_address_postcode by any other column. Since we are only counting it doesn't matter which column use.
# substance_count = df[["site_address_postcode", "substance"]].groupby(by="substance").count()
#
# # Now sort it, and take the first 10 results
# substance_count.columns = ["Count"] # rename column
# top10 = substance_count.sort_values(by="Count", ascending=False)[:10]
#
# # Create the pivot table
# pivot = df.pivot_table(
#     index="site_address_postcode",
#     columns="substance",
#     values="quantity_in_kg",
#     aggfunc="sum",
# )
#
# # get only the columns for top 10 substances
# pivot_top10 = pivot[top10.index]
# pivot_top10
# ```
#
#
#
#

# Now it's a good time to discuss dealing with missing values in a table.

# ## Missing Values
# There might be missing data in a table. Having `NaN` in the table can cause trouble in the analysis so we need to decide how we are going to deal with it. A few common scenarios are:
# 1. filling the missing values with a number e.g. zero
# 2. removing rows with missing values
# 3. removeing rows with multiple missing values and filling the remaining with a new value

# +
# you can replace site_address_postcode by any other column. Since we are only counting it doesn't matter which column use.
substance_count = df[["site_address_postcode", "substance"]].groupby(by="substance").count()

# Now sort it, and take the first 10 results
substance_count.columns = ["Count"] # rename column
top10 = substance_count.sort_values(by="Count", ascending=False)[:10]

# Create the pivot table
pivot = df.pivot_table(
    index="site_address_postcode",
    columns="substance",
    values="quantity_in_kg",
    aggfunc="sum",
)

# get only the columns for top 10 substances
pivot_top10 = pivot[top10.index]
pivot_top10
# -

# To replace the missing value with a fixed number we can use `.fillna()` method.

dfnew = pivot_top10.fillna(value=0)
dfnew

# __Note:__ When using `fillna` the changes are not saved in the data frame. The default settings only returns the result and keeps the original data frame intact. If you want to save the changes in the same data frame you can pass in `inplace = True`.

# There are other ways to fill the missing values. In some cases you might want to use different values for each column. A common example is using mean or median of a column for the missing values.

fill_values = pivot_top10.mean()
dfnew = pivot_top10.fillna(value=fill_values)
dfnew

# Pandas has other methods for filling the missing values including forward and backward filling. Forward filling replaces the missing values by the last valid value in the table and backward filling replaces the missing values by next valid value. These techniques are useful for sequential data such as time series and wouldn't make sense to be applied to tabular data.

# To use these methods, when using `fillna` instead of passing in a value, you can pass a method. For forward filling pass in `method = "ffill"` and for backward filling pass in `method = "bfill"`.

# If you simply want to get rid of rows with missing values you can use `.dropna()`

dfnew = pivot_top10.dropna()
dfnew

# __Note:__ Notice the number of rows are much less in the table above compared to the original table.

# If we remove any row that contains missing values we might lose a significant portion of the data. Alternatively, we can only remove rows which have more than a certain number of missing values. To do so, we can set a threshold.

# remove the rows with at least 3 missing values
dfnew = pivot_top10.dropna(thresh=3)
dfnew

# Now we have more rows compared to when we removed all missing values.

# Next step is to replace the missing values using the techniques discussed above.

# ## Saving Data
# After analysis and reshaping the data you might want to save the results in a file. Similar to reading files, pandas supports multiple file formats to save the tables.

pivot_top10.to_csv("final_table.csv")

# ## Pandas Plotting
# Pandas dataframes have plotting methods which help to visualise the data. The following plots are supported in pandas:
# - 'line' : line plot (default)
# - 'bar' : vertical bar plot
# - 'barh' : horizontal bar plot
# - 'hist' : histogram
# - 'box' : boxplot
# - 'kde' : Kernel Density Estimation plot
# - 'density' : same as 'kde'
# - 'area' : area plot
# - 'pie' : pie plot
# - 'scatter' : scatter plot
# - 'hexbin' : hexbin plot.
#
# You can select which plot you want to use by setting `kind` to the string for the plot.

# There are a few other useful options you can set:
# - xlim, ylim: to set limits of axes
# - logx, logy, loglog: to set whether an axis should be displayed in logarithmic scale]
# - title: to set the title of the plot
# - figsize: to set the size of the plot
#

Let's try a few types of charts and graphs.

# Top 10 postcodes with largest carbon monoxide emission
pivot_top10.sort_values(by="Carbon monoxide", ascending=False)[:10].plot(
    kind="barh", y="Carbon monoxide"
)

# histogram of benzene emission
pivot_top10.plot(kind="hist", y="Benzene", bins=50)

# +
# kernel density estimation plot of benzene emission

pivot_top10.plot(kind="kde", y=["Benzene", "Toluene (methylbenzene)"], logx=True)
# -

# histogram of benzene emission in each postcode
pivot_top10.plot(kind="box", logy=True, rot=90)

pivot_top10.plot(kind="scatter", x="Toluene (methylbenzene)", y="Benzene", loglog=True)

# pie chart of emission of the substances in postcode 4008
pivot_top10.loc[4008, :].plot(kind="pie", subplots=True, figsize=(10, 10))

# We will discuss producing more advanced plots in the next notebooks where we learn about various plotting packages in python.

# ## Further reading
# - [Pandas documentation](https://pandas.pydata.org/)
# - [Pandas in 10 minutes](https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html)
# - https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf
# - https://www.kaggle.com/learn/pandas
# - https://www.kaggle.com/kashnitsky/topic-1-exploratory-data-analysis-with-pandas
# - https://www.youtube.com/watch?v=ZyhVh-qRZPA&list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS
#


"""
Xarray : Quick tour
===================

Basically, this example gives a very small introduction to Xarray (very small).
We illustrate how to define a DataArray container, access its components,
perform some of the basic operations and slicing / indexing.
"""
import numpy as np
import xarray as xr
import pandas as pd

###############################################################################
# Simulate data
# -------------
#
# lets start by creating a random spatio-temporal array

n_times = 30
n_roi = 7
times_vec = np.linspace(-1, 1, n_times)
roi_vec = np.array([f"roi_{k}" for k in range(n_roi)])
np_data = np.random.rand(n_times, n_roi)
print(np_data.shape)
print('*' * 79)

###############################################################################
# Xarray conversion and access to the internal components
# -------------------------------------------------------
#
# A DataArray is a container (like a well known numpy array) except that you
# can add a label to each coordinate. To this end, the input `dims` is a tuple
# that describes the dimension names and `coords` describes the value along
# this coordinate

# let's convert it to a DataArray
da_data = xr.DataArray(np_data, dims=('times', 'roi'),
                       coords=(times_vec, roi_vec))
print(da_data.shape)
print(da_data)
print('*' * 79)

# if you want to get the dimension names and values
print(f'Dimension names : {da_data.dims}')
print(f'Dimension values : {da_data.coords}')
print(f"Data of a specific dimension : {da_data.roi.data}")
print('*' * 79)

# if you want to get the original NumPy array enter the following :
da_data.data

# if you want to change the values of a coordinate
da_data['roi'] = np.array([f"roi_{k % 3}" for k in range(n_roi)])
print(f"New ROI names : {da_data.roi.data}")
print('*' * 79)

# if you need to compute or get the min / max / mean across a specific
# dimension
da_data.min('times')  # minimum across time points
da_data.max('times')  # maximum across time points
da_data.mean('roi')   # mean across all ROI

# similarly to Pandas, it's also possible to group along a dimension and then
# take the mean. For example, here's how to group and mean by roi names
da_m = da_data.groupby('roi').mean('roi')
print(da_m)
print('*' * 79)

###############################################################################
# Xarray slicing and indexing
# ---------------------------
#
# Now we show how to slice the container

# select a single specific ROI based on it's name
da_data.sel(roi='roi_0')

# select a time range
da_time_slice = da_data.sel(times=slice(-.5, .5))
print(f"Temporal selection : {da_time_slice.coords}")
print('*' * 79)

# off course, spatio-temporal selection is also supported
da_st = da_data.sel(times=slice(-.5, .5), roi='roi_1')
print(f"Spatio-temporal selection : {da_st.coords}")
print('*' * 79)

# you can also slice according to indices
da_isel = da_data.isel(times=slice(10, 20))
print(f"Integer selection : {da_isel.coords}")
print('*' * 79)

# however, if you want for example select multiple items based on their names,
# you have to use booleans. Here's a small example that's using Pandas
roi = da_data.roi.data
use_roi = ['roi_0', 'roi_2']
is_roi = pd.Series(roi).str.contains('|'.join(use_roi))
da_mi = da_data.isel(roi=is_roi)
print(f"Multi-items selection : {da_mi.coords}")

###############################################################################
# Xarray attributes
# -----------------
#
# One of the nice features of DataArray is that it supporting setting
# attributes. Therefore you can add, for example, the parameters that describe
# your analysis

# adding a few string attributes
da_data.attrs['inference'] = 'ffx'
da_data.attrs['stats'] = 'cluster-based'
da_data.attrs['description'] = """Here's a small description of the analysis
I'm currently running. Trying to find a difference between condition 1. vs 2.
"""

# you can also add vectors (but not arrays) to the attributes
da_data.attrs['vector'] = np.arange(30)

# however, "None" seems to pose a problem when saving the results. Therefore,
# one quick way to solve this is simply to convert it into a string
da_data.attrs['none_problem'] = str(None)

print(da_data)

###############################################################################
# Xarray to an other format
# -------------------------
#
# Finally, we quickly illustrate how to convert a DataArray into, for example,
# a pandas.DataFrame

print(da_data.to_pandas())


#!/usr/bin/env python3
from paint import paint
import sys
from time import sleep
import numpy as np
"""
# Sample format to answer pattern questions
# assuming the pattern would be frag0:
..*
**.
.**
#############################################
# Reread the instructions for assignment 1: make sure
# that you have the version with due date SUNDAY.
# Every student submits their own assignment.
* Delete the names and ccids below, and put
# the names and ccids of all members of your group, including you.
# name                         ccid
Daniel Levy                   danlevy
Sarah Levy                    sarlevy

#############################################
# Your answer to question 1-a:

#############################################
# Your answer to question 1-b:

#############################################
# Your answer to question 2:

#############################################
# Follow the assignment 1 instructions and
# make the changes requested in question 3.
# Then come back and fill in the answer to
# question 3-c:

#############################################
"""
"""
based on life-np.py from course repo
"""

PTS = '.*#'
DEAD, ALIVE, WALL = 0, 1, 2
DCH, ACH, GCH = PTS[DEAD], PTS[ALIVE], PTS[WALL]

def point(r, c, cols): return c + r*cols

"""
board functions
  * represent board as 2-dimensional array
"""

def get_board():
    B = []
    print(sys.argv[1])
    with open(sys.argv[1]) as f:
        for line in f:
            B.append(line.rstrip().replace(' ', ''))
        rows, cols = len(B), len(B[0])
        for j in range(1, rows):
            assert(len(B[j]) == cols)
        return B, rows, cols

def convert_board(B, r, c):  # from string to numpy array
    A = np.zeros((r, c), dtype=np.int8)
    for j in range(r):
        for k in range(c):
            if B[j][k] == ACH:
                A[j, k] = ALIVE
    return A

def expand_grid(A, r, c, t):  # add t empty rows and columns on each side
    N = np.zeros((r+2*t, c+2*t), dtype=np.int8)
    for j in range(r):
        for k in range(c):
            if A[j][k] == ALIVE:
                N[j+t, k+t] = ALIVE
    return N, r+2*t, c+2*t

def print_array(A, r, c):
    print('')
    for j in range(r):
        out = ''
        for k in range(c):
            out += ACH if A[j, k] == ALIVE else DCH
        print(out)

def show_array(A, r, c):
    for j in range(r):
        line = ''
        for k in range(c):
            line += str(A[j, k])
        print(line)
    print('')

"""
Conway's next-state formula
"""

def next_state(A, r, c):
    N = np.zeros((r, c), dtype=np.int8)
    changed = False
    for j in range(r):
        for k in range(c):
            num = 0
            if j > 0 and k > 0 and A[j-1, k-1] == ALIVE:
                num += 1
            if j > 0 and A[j-1, k] == ALIVE:
                num += 1
            if j > 0 and k < c-1 and A[j-1, k+1] == ALIVE:
                num += 1
            if k > 0 and A[j, k-1] == ALIVE:
                num += 1
            if k < c-1 and A[j, k+1] == ALIVE:
                num += 1
            if j < r-1 and k > 0 and A[j+1, k-1] == ALIVE:
                num += 1
            if j < r-1 and A[j+1, k] == ALIVE:
                num += 1
            if j < r-1 and k < c-1 and A[j+1, k+1] == ALIVE:
                num += 1
            if A[j, k] == ALIVE:
                if num > 1 and num < 4:
                    N[j, k] = ALIVE
                else:
                    N[j, k] = DEAD
                    changed = True
            else:
                if num == 3:
                    N[j, k] = ALIVE
                    changed = True
                else:
                    N[j, k] = DEAD
    return N, changed

#############################################
"""
Provide your code for the function
next_state2 that (for the usual bounded
rectangular grid) calls the function num_nbrs2,
and delete the raise error statement:
"""

def next_state2():

    raise NotImplementedError()
#############################################

#############################################
"""
Provide your code for the function
num_nbrs2 here and delete the raise error
statement:
"""

def num_nbrs2():

    raise NotImplementedError()
#############################################

#############################################
"""
Provide your code for the function
next_state_torus here and delete the raise
error statement:
"""

def next_state_torus():

    raise NotImplementedError()
#############################################

#############################################
"""
Provide your code for the function
num_nbrs_torus here and delete the raise
error statement:
"""

def num_nbrs_torus():

    raise NotImplementedError()
#############################################

"""
input, output
"""

pause = 0.2

#############################################
"""
Modify interact as necessary to run the code:
"""
#############################################

def interact(max_itn):
    itn = 0
    B, r, c = get_board()
    print(B)
    X = convert_board(B, r, c)
    A, r, c = expand_grid(X, r, c, 0)
    print_array(A, r, c)
    while itn <= max_itn:
        sleep(pause)
        newA, delta = next_state(A, r, c)
        if not delta:
            break
        itn += 1
        A = newA
        print_array(A, r, c)
    print('\niterations', itn)

def main():
    interact(99)

if __name__ == '__main__':
    main()


import theano
import theano.tensor as T
import numpy as np

"""
theano_utils
Commonly used functions to convert np array to theano.shared variable
theano.shared variable is used to store data into GPUs
"""

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)

def sharedX(X,dtype=theano.config.floatX, name=None):
    return theano.shared(np.asarray(X,dtype=dtype), name=name)

def shared_zeros(shape,dtype=theano.config.floatX,name=None):
    return theano.shared(np.asarray(np.zeros(shape), dtype=dtype), name=name)

def shared_ones(shape, dtype=theano.config.floatX,name=None):
    return theano.shared(np.asarray(np.ones(shape), dtype=dtype), name=name)

def shared_scalar(val=0., dtype=theano.config.floatX,name=None):
    return theano.shared(np.cast[dtype](val))


import numpy as np
#  This program differentiates between the time taken on operations perform on lists and numpy arrays
import time

SIZE = 1000000

L1 = range(SIZE)
L2 = range(SIZE)

A1 = np.arange(SIZE)
A2 = np.arange(SIZE)

start = time.time()

result = [(x, y) for x, y in zip(L1, L2)]

print((time.time() - start) * 1000)

result1 = A1 + A2

print((time.time() - start) * 1000)


#!/usr/bin/env python
# coding: utf-8

# # Open Street Map
#
# **Inhalt:** Daten aus einer Open Source Quelle ziehen
#
# **Nötige Skills**
# - Geopandatenhandling
#
# **Lernziele**
# - Einblick in Funktionsweise von OSM
# - Vor-Aufbereitete OSM-Shapefiles laden
# - Daten herunterladen mit `OSMnx`-Library

# In[1]:

from IPython.display import Image

# In[2]:

import pandas as pd
import geopandas as gpd

# In[3]:

from shapely.geometry import Point, LineString, Polygon

# In[4]:

import networkx as nx
import osmnx as ox

# In[5]:

pd.set_option("display.max_rows", 300)

# ## Über Open Street Map

# «OpenStreetMap is a map of the world, created by people like you and free to use under an open license.»
#
# https://www.openstreetmap.org
#
# => Es ist das Google-Street-Map-Äquivalent der Open-Source-Community.

# In[6]:

Image("dataprojects/osm/osm-luzern.png")

# About:
# - https://de.wikipedia.org/wiki/OpenStreetMap
# - https://wiki.openstreetmap.org/wiki/Main_Page

# Man kann auf verschiedenen Wegen Daten aus OSM laden. Hier eine Übersicht: https://learnosm.org/de/osm-data/getting-data/

# ## 1. via API
#
# Für Softwareentwickler und professionelle Anwender gibt es die OSM API:
# - Overpass API https://wiki.openstreetmap.org/wiki/Overpass_API
#
# Diese ist recht kompliziert, es gibt spezielles Tool, das einem hilft, den Querystring aufzubauen
# - Overpass Turbo: http://overpass-turbo.eu/
#
# Wie man diese Tools in Python benutzen kann, liest man zB hier:
# - https://towardsdatascience.com/loading-data-from-openstreetmap-with-python-and-the-overpass-api-513882a27fd0

# Wir schauen uns stattdessen zwei andere Wege an

# ## 2. via Geofabrik

# Hier werden vor-aufbereitete Shapefiles aus OWM zum Download angeboten: http://download.geofabrik.de/

# In[7]:

Image("dataprojects/osm/geofabrik.png")

# Typischerweise werden dabei alle Informationen zu einem Land in ein grosses zip-Archiv gepackt (das Schweiz-File wiegt zB 628 MB!)

# ### Daten laden

# Um die neueste Version zu erhalten: **folgendes Zip-File runterladen**:

# `http://download.geofabrik.de/europe/switzerland-latest-free.shp.zip`

# im ZIP hat es dann einzelne Shapefiles zu verschiedenen Objektkategorien:
#
# - `gis_osm_buildings_a_free_1.shp`
# - `gis_osm_landuse_a_free_1.shp`
# - `gis_osm_natural_a_free_1.shp`
# - `gis_osm_natural_free_1.shp`
# - `gis_osm_places_a_free_1.shp`
# - `gis_osm_places_free_1.shp`
# - `gis_osm_pofw_a_free_1.shp`
# - `gis_osm_pofw_free_1.shp`
# - `gis_osm_pois_a_free_1.shp`
# - `gis_osm_pois_free_1.shp`
# - `gis_osm_railways_free_1.shp`
# - `gis_osm_roads_free_1.shp`
# - `gis_osm_traffic_a_free_1.shp`
# - `gis_osm_traffic_free_1.shp`
# - `gis_osm_transport_a_free_1.shp`
# - `gis_osm_transport_free_1.shp`
# - `gis_osm_water_a_free_1.shp`
# - `gis_osm_waterways_free_1.shp`

# Hier haben wir das Schweiz-Archiv bereits lokal entpackt.
#
# *Achtung:*
# - lange Ladezeit...
# - `encoding="utf-8"` beachten

# In[63]:

gdf_osm = gpd.read_file("dataprojects/osm/switzerland-latest-free.shp/gis_osm_buildings_a_free_1.shp", encoding="utf-8")

# Kein Wunder dauert es lange, das Datenfile mit allen Gebäuden der Schweiz hat fast 2,5 Millionen Zeilen.

# In[64]:

gdf_osm.shape

# Der Vorteil an den Geofabrik-Daten: Sie sind sehr schön strukturiert:

# In[65]:

gdf_osm.head()

# ### Datenstruktur

# #### ID
#
# Jedes Element in diesem Geodataframe hat eine `osm_id`. Diese ID ist der Unique Identifier einer bestimmten Einheit in OSM. Wir treffen sie auch an, wenn wir auf anderem Wege Daten herunterladen oder direkt auf OSM danach suchen.
#
# Zum Beispiel das KKL Luzern: https://www.openstreetmap.org/relation/1661451

# In[8]:

Image("dataprojects/osm/kkl.png")

# In[67]:

gdf_osm[gdf_osm['osm_id'] == '1661451']

# #### name

# Zu jedem Element, das eine `osm_id` hat, gehört netterweise auch ein `name`.

# #### code
#
# Ein vierstelliger Code, der angibt, um was für eine Art von Feature es sich handelt (Strasse, Fluss, Stadt, Point of Interest, Schule, Spital, ...)
#
# Die ganze Liste der Codes findet sich hier: http://download.geofabrik.de/osm-data-in-gis-formats-free.pdf

# In[68]:

gdf_osm['code'].value_counts()

# #### fclass
#
# Beschreibt den Code in Worten. Weil wir in diesem Shapefile nur Gebäude geladen haben, lautet er immer `building`.

# In[69]:

gdf_osm['fclass'].value_counts()

# #### geometry
#
# Die wichtigste Spalte: Enthält die geografischen Infos zu einem Feature, also zu einem Element der Karte. Das kann sein:
# - ein `Point`
# - ein `Linestring` (z.B. bei einer Strasse)
# - ein `Polygon` (zB bei einem Gebäude)

# Wir können die Geometriespalte nutzen, um zB einen Kartenausschnitt («bbox») für Luzern zu generieren.
#
# Dazu definieren wir zuerst ein Viereck mit den Eckpunkten: https://tools.retorte.ch/map/?swissgrid=2667889,1209686&zoom=14

# In[70]:

north, south, east, west = 47.06954, 47.03474, 8.2715, 8.33184

Luzern = Polygon([[west, north], [east, north], [east, south], [west, south]])

# In[71]:

Luzern.wkt

# Anschliessend können wir das Geodataframe filtern (das Vergleichskennwort heisst hier `within`)

# In[72]:

gdf_luzern = gdf_osm[gdf_osm.within(Luzern)]

# In[73]:

gdf_luzern.head()

# In[74]:

gdf_luzern.shape

# #### type

# Diese Spalte gibt bestimmte Details zum Gebäude an. Ist allerdings nicht sehr zuverlässig, wie zB diese Suche zeigt: In Luzern haben nur drei Gebäude den Typ «university».

# In[75]:

gdf_luzern['type'].value_counts()

# In[76]:

gdf_luzern[gdf_luzern['type'] == 'university']

# Nichtsdestotrotz können wir dieses Attribut zB nutzen, um gewisse Gebäude beim Plotten farblich hervorzuheben.

# In[77]:

# Alle Gebäude
ax = gdf_luzern.plot(figsize=(15,10), color='lightgrey')

# Kirchen
gdf_luzern[gdf_luzern['type'] == "church"].plot(color='green', ax=ax)

ax.set_title("Kirchen in Luzern")

# In[ ]:

# In[ ]:

# In[ ]:

# ## 3. via OSMnx

# OSMnx ist eine alternative Möglichkeit, um Daten aus der Open Street Map zu laden.
#
# Es ist eine Python-Bibliothek, die anhand von selektiven Angaben
# - die ein Suchquery bei der Overpass-API generiert und
# - die Resultate zur praktischen Weiterverarbeitung aufbereitet
#
# OSMnx-Dokumentation: https://osmnx.readthedocs.io/en/stable/
#
# Beispiel-Notebooks zur Anwendung: https://github.com/gboeing/osmnx-examples/tree/main/notebooks
#
# Hier lernen wir einige Möglichkeiten kennen, welche diese Library bietet.

# ## Datenabfrage: Boundaries

# OSMnx kann anhand eines einfachen Suchstrings die OSM durchsuchen und liefert uns so z.B. die Umrisse eines bestimmten geografischen Gebiets.
#
# Die Funktion dafür heisst `geocode_to_gdf()`:

# In[6]:

gdf = ox.geocode_to_gdf("Aargau, Switzerland")

# Wir erhalten ein Geodataframe mit einer einzigen Zeile.

# In[7]:

gdf

# Diese Zeile entspricht einer geografischen Einheit auf Open Street Map.
#
# Wir erkennen das zB wenn wir auf OSM nach der `osm_id` suchen: https://www.openstreetmap.org/relation/1686359

# In[8]:

Image("dataprojects/osm/ag.png")

# Mit unserem Geodataframe können wir nun alles machen, was wir wollen. zB plotten:

# In[9]:

gdf.plot()

# #### Mehrere Gebiete abfragen
#
# Mit einer Liste von Suchstrings können wir ein Geodatenframe erstellen, das mehrere Einträge enthält.

# In[10]:

places = [
    "Kreis 1, Zürich, Switzerland",
    "Kreis 2, Zürich, Switzerland",
    "Kreis 3, Zürich, Switzerland",
    "Kreis 4, Zürich, Switzerland",
    "Kreis 5, Zürich, Switzerland",
    "Kreis 6, Zürich, Switzerland",
    "Kreis 7, Zürich, Switzerland",
    "Kreis 8, Zürich, Switzerland",
    "Kreis 9, Zürich, Switzerland",
    "Kreis 10, Zürich, Switzerland",
    "Kreis 11, Zürich, Switzerland",
    "Kreis 12, Zürich, Switzerland"
]
gdf = ox.geocode_to_gdf(places)

# In[11]:

gdf.plot()

# #### Präzise Suche
#
# Manche Suchstrings können missverständlich sein. ZB «Zürich, Switzerland»: Ist hier die Stadt oder der Kanton gemeint? Um Missverständnisse auszuschliessen, können wir unsere Spezifikation in einem Dictionary präzisieren:

# In[12]:

gdf = ox.geocode_to_gdf(
    {
        "state": "Zürich",
        "country": "Switzerland"
    }
)

# In[13]:

gdf.plot()

# #### Koordinatensystem
#
# Wichtig: Sobald wir mit diesen Daten kartografisch irgendwas sinnvolles machen wollen, sollten wir sie in ein passendes Koordinatensystem umprojizieren.

# In[14]:

gdf.crs

# In[15]:

gdf = gdf.to_crs(epsg=21781)

# In[16]:

gdf.plot()

# ## Datenabfrage: Netzwerke

# Um ein Strassennetzwerk aus OSM zu laden, machen wir zwei Schritte statt nur einen.

# #### 1. Graph erstellen
#
# OSMnx lädt die Daten zuerst in einen sogenannten Graph (des Typs `networkx`). Die Funktion dazu heisst `graph_from_place():`:

# In[17]:

G = ox.graph_from_place("Luzern, Switzerland", network_type="drive")

# Diesen Graph können wir mix OSMnx auch bereits direkt plotten:

# In[18]:

ox.plot_graph(G)

# Um in Geopandas damit zu arbeiten, müssen wir den Graph konvertieren mit `graph_to_gdfs()`:

# In[19]:

gdf_nodes, gdf_edges = ox.graph_to_gdfs(G)

# Wir erhalten zwei Geodataframes, eines mit den Punkten (nodes) und eines mit den Linien (edges):

# In[20]:

gdf_nodes.head()

# In[21]:

gdf_edges.head()

# ### BBox statt Suchstring

# Statt mit einem Suchstring («Luzern, Switzerland») können wir die Datensuche auch mit einer bounding box eingrenzen:

# In[22]:

north, south, east, west = 47.06954, 47.03474, 8.2715, 8.33184

# Die Funktion heisst dann leicht anders: `graph_from_bbox()`

# In[23]:

G = ox.graph_from_bbox(north, south, east, west, network_type="drive")

# In[24]:

ox.plot_graph(G)

# ### Metainformationen

# In der Open Street Map hat es nebst den reinen Geoinformationen – so genannte «nodes» (Punkte), «ways» (Linien) und «relations» (zusammengefasste nodes und ways) – auch eine Menge Metainformationen. Diese kann man mit OSMnx abfragen.

# #### Netzwerktypen

# Statt nach `network="drive"` (fahrbare Strassen) können wir zum Beispiel auch nach anderen Strassennetzen suchen:
#
# - `drive` - get drivable public streets (but not service roads)
# - `drive_service` - get drivable streets, including service roads
# - `walk - get all` streets and paths that pedestrians can use (this network type ignores one-way directionality)
# - `bike - get all` streets and paths that cyclists can use
# - `all - download` all non-private OSM streets and paths
# - `all_private` - download all OSM streets and paths, including private-access ones
#

# #### Filter

# Und wir können die Suche mit so genannten Filtern weiter eingrenzen. ZB nur auf Autobahnen, mit `custom_filter=`:

# In[25]:

G = ox.graph_from_place("Aargau, Switzerland", network_type="drive", custom_filter='["highway"~"motorway"]')

# In[26]:

gdf_nodes, gdf_edges = ox.graph_to_gdfs(G)
gdf_edges = gdf_edges.to_crs(epsg=21781)

# Wir können die aargauer Autobahnen plotten, zB indem wir zusätzlich noch die Umrisse des Kantons holen:

# In[27]:

gdf_ag = ox.geocode_to_gdf("Aargau, Switzerland")
gdf_ag = gdf_ag.to_crs(epsg=21781)

# In[28]:

ax = gdf_ag.plot(color="lightgrey", linewidth=1, edgecolor='black')
ax.set_title("Autobahnen im AG")
gdf_edges.plot(color='red', ax=ax)

# Wie kommt man auf diese Filterwörter? Die Antwort lautet: Es sind so genannte Tags, die in OSM hinterlegt sind.

# #### Tags

# Schauen wir uns zunächst unser Suchergebnis zu den Autobahnen nochmals an.
#
# Unser Filter hiess: '["highway"~"motorway"]'
#
# Im Geodataframe finden wir diese Begriffe wieder:
# - Es gibt eine Spalte namens «highway»
# - Die Einträge darin heissen «motorway» oder ähnlich.

# In[29]:

gdf_edges.head()

# Wo diese Metainformation herkommt, erkennen wir auf der Online-Version von OSM, wenn wir eines der Features aus unserem Geodataframe dort abrufen: https://www.openstreetmap.org/way/827816284

# In[30]:

Image("dataprojects/osm/a3.png")

# Das betreffende Geoelement wurde auf OSM als «highway» getagged und hat, mit der Unterkategorie «motorway».
#
# Das OSM-Wiki verrät uns, was es alles für Typen von highways gibt: https://wiki.openstreetmap.org/wiki/Key:highway?uselang=en
#
# Und auch sämtliche anderen Metainformationen zu diesem Autobahnteilstück («lanes», «oneway», «maxspeed») finden wir in unserem Geodataframe wieder.

# #### Andere Netzwerke

# Welche anderen Tags können wir nutzen, um nach bestimmten Geofeatures zu suchen?
#
# Auf dem OSM-Wiki gibt es dazu eine ellenlange Liste: https://wiki.openstreetmap.org/wiki/Map_features
#
# - `"aerialway"`: Seilbahnen
# - `"amenity"`: Jegliche Art von Objekt mit einer speziellen Funktion: `~"bar"`, `~"library"`, `~"charging_station"`, etc.
# - `"boundary"`: eine Umgrenzung. In Kombination nutzbar mit `"admin_level"`
# - `"building"`: ein Gebäude. zB `~"mosque"`, `~"train_station"`, `~"parking"`, `~"stadium"`, etc.
# - `"landuse"`: Wie ein bestimmtes Landstück genutzt wird, zB `~"farmland"`
# - `"natural"`: Elemente der Natur, zB `~"glacier"`, `~"beach"`, etc.
# - etc.
#
# So können wir zB statt nach Strassen nach Eisenbahnen suchen:

# In[31]:

G = ox.graph_from_place("Aargau, Switzerland", custom_filter='["railway"~"rail"]')

# In[32]:

ax = gdf_ag.plot(color="lightgrey")
ax.set_title("Eisenbahnen im AG")

gdf_nodes, gdf_edges = ox.graph_to_gdfs(G)
gdf_edges.to_crs(21781).plot(ax=ax)

# In[ ]:

# In[ ]:

# ## Datenabfrage: Geometrien

# Bis jetzt haben wir (Strassen-)netzwerke abgefragt. Mithilfe der auf OSM hinterlegten Tags können wir aber auch sämtliche Geo-Elemente jeglicher herunterladen, die in einem bestimmten Gebiet vorhanden sind.

# ### Gebäude

# Dazu gibt es die Funktion `geometries_from_place()`. Sie benötigt:
# - eine geografische Eingrenzung der Suche ("place")
# - eine (Reihe von) Tags, nach denen gesucht werden soll.

# In[33]:

place = "Wettingen, Switzerland"
tags = {"building": True}
gdf_bldg = ox.geometries_from_place(place, tags)

# In Wettingen sind zB über 3000 Gebäude erfasst:

# In[34]:

gdf_bldg.shape

# Da es ziemlich viele Tags gibt, die sonst noch mit diesen Gebäuden verknüpft sind, erhalten wir ein ziemlich breites GDF:

# In[35]:

gdf_bldg.columns

# In[36]:

gdf_bldg.head(5)

# Wir sehen hier auch, dass das Geodatenframe drei übergeordnete Indizes hat.
#
# Die ways (Pfade), relations (Sammlungen) und nodes (Punkte) sind separat aufgeführt.

# In[37]:

gdf_bldg.index.get_level_values('element_type').value_counts()

# Wir können uns aus diesem Multi-Index zB die Pfade heraussuchen.

# In[38]:

gdf_bldg = gdf_bldg[gdf_bldg.index.get_level_values('element_type') == 'way']

# In[39]:

gdf_bldg.head(2)

# Jedes Gebäude hat einen Pfad, das dessen Umriss angibt. Wir können so die Geäbude plotten:

# In[40]:

gdf_bldg = gdf_bldg.to_crs(epsg=21781)
gdf_bldg.plot(figsize=(10,10))

# ## Anwendungen

# Nachfolgend ein paar Beispiele, was wir in Geopandas mit den Datenabfragen anstellen können.

# ### Plots von mehreren Datenabfragen

# Wir haben das schon weiter oben gesehen. Mit `ax=...` können wir mehrere Plots übereinanderlegen:

# In[41]:

# Abfrage und Plot der Gemeindegrenze
gdf_we = ox.geocode_to_gdf("Wettingen, Switzerland").to_crs(epsg=21781)
ax = gdf_we.plot(color='lightgrey', figsize=(10, 10))

# Plot der Gebäude (auf "ax")
gdf_bldg.plot(figsize=(10,10), ax=ax)

# ### Abgefragte Daten filtern

# Wie bereits erwähnt: Wir haben ein riesenbreites Geodataframe mit zig Metadaten.

# In[42]:

gdf_bldg.head(2)

#  Wir können zB all die Gebäude heraussuchen, die gleichzeitig auch eine «amenity» eines bestimmten Typs sind.

# In[43]:

gdf_schools = gdf_bldg[gdf_bldg['amenity'] == 'school']

# Das Ergebnis als Plot:

# In[44]:

# Gemeindegrenze
ax = gdf_we.plot(linewidth=0.5, edgecolor='black', color='lightgrey', figsize=(20, 20))

# Alle Gebäude
gdf_bldg.plot(color='grey', ax=ax)

# Schulen
gdf_schools.plot(color='red', ax=ax)

# Nun haben wir allerdings ein Problem: In Wettingen gibt es noch mehr Schulen als hier eingezeichnet sind.

# ### Daten ergänzen

# Das Problem scheint zu sein: Einige der Schulen wurden zwar als "amenity": "school" kategorisiert, aber nicht als "building" :-/
#
# Wir holen uns deshalb von der OSM nochmals alle Geometrien raus, die als "school" getaggt wurden (aber nicht zwangsläufig auch als Gebäude!

# In[45]:

place = "Wettingen, Switzerland"
tags = {"amenity": "school"}
gdf_schools2 = ox.geometries_from_place(place, tags).to_crs(epsg=21781)

# Ein Beispiel für einen solchen Eintrag ist das Schulhaus Altenburg:

# In[46]:

gdf_schools2[gdf_schools2['name'] == "Altenburg"]

# Es wurden also gewisse Areale als Schule getaggt, aber nicht die dortigen Gebäude.
#
# Was das heisst, sieht man, wenn man die Areale zum obigen Plot hinzufügt.

# In[47]:

# Gemeindegrenze
ax = gdf_we.plot(linewidth=0.5, edgecolor='black', color='lightgrey', figsize=(20, 20))

# Alle Gebäude
gdf_bldg.plot(color='grey', ax=ax)

# Schulen => aus Gebäuden
gdf_schools.plot(color='red', ax=ax)

# Schulen => aus Amenities
gdf_schools2.plot(color='blue', linewidth=0.5, alpha=0.3, edgecolor='black', ax=ax)

# #### Lösungsansatz:
#
# Gebäude, die innerhalb der Schul-Areale liegen, umklassifizieren.
#
# Dazu nutzen wir die Funktion `unary_union`, um alle Schul-Areale zu einer gemeinsamen Shape zusammenzufassen.

# In[48]:

gdf_schools2.unary_union

# Danach filtern wir die Gebäude in unserem `gdf_bldg` danach, ob sie innerhalb dieser Shape liegen.
#
# Die passende Relation dazu heisst `within()`.
#
# Bei allen Treffern schreiben wir in der Spalte «amenity» neu überall "school" rein.

# In[49]:

gdf_bldg.loc[gdf_bldg['geometry'].within(gdf_schools2.unary_union), "amenity"] = "school"

# Letzter Schritt: Wir filtern nochmals das Gebäude-GDF, diesmal inklusive der neu klassifizierten Einträge:

# In[50]:

gdf_schools = gdf_bldg[gdf_bldg['amenity'] == 'school']

# Jetzt haben wir eine vollständige (?) Sammlung aller Schulgebäude.

# In[51]:

# Gemeindegrenze
ax = gdf_we.plot(linewidth=0.5, edgecolor='black', color='lightgrey', figsize=(20, 20))

# Alle Gebäude
gdf_bldg.plot(color='grey', ax=ax)

# Schulen
gdf_schools.plot(color='red', ax=ax)

# ### Räumliche Berechnungen

# Sobald wir Daten als Geodataframe vorliegen haben, können wir allerlei Berechnungen durchführen.
#
# #### Zum Beispiel:
#
# Wie weit liegt ein bestimmtes Gebäude von der nächsten Bushaltestelle entfernt?
#
# Um das zu beantworten, laden wir uns aus der OSM die Busstationen in Wettingen.

# In[52]:

place = "Wettingen, Switzerland"
tags = {"highway": "bus_stop"}
gdf_stops = ox.geometries_from_place(place, tags=tags).to_crs(epsg=21781)

# In[53]:

gdf_stops.shape

# In[54]:

gdf_stops.head()

# Plot auf unserem Gemeindeplan mit den Gebäuden:

# In[55]:

# Gemeindeumriss
ax = gdf_we.plot(linewidth=0.5, edgecolor='black', color='#EEEEEE', figsize=(20, 20))

# Gebäude
gdf_bldg.plot(color='#999999', ax=ax)

# Bushaltestellen
gdf_stops.plot(color='blue', markersize=25, ax=ax)

# Wir können die Funktion `.distance()` nutzen, um die Distanz zwischen zwei Shapes zu berechnen.
#
# Wenn wir diese Funktion auf ein ganzes Dataframe anwenden, erhalten eine Liste von Distanzen: von jedem Element im DF zur angegebenen Shape.
#
# Hier zum Beispiel: die Distanzen sämtlicher Bushaltestellen zum ersten Eintrag im Gebäude-GDF:

# In[56]:

gdf_stops.distance(gdf_bldg.iloc[0]['geometry']).head(10)

# Wenn wir aus dieser Liste das Minimum auswählen, erhalten wir die Distanz von diesem Gebäude zur *nächsten* Bushaltestelle:

# In[57]:

gdf_stops.distance(gdf_bldg.iloc[0]['geometry']).min()

# Diese Berechnung führen wir nun mit sämtlichen Gebäuden durch. Dazu nutzen wir `apply()`.
#
# Das Ergebnis speichern wir in einer neuen Spalte.

# In[58]:

gdf_bldg['distance_next_stop'] = gdf_bldg['geometry'].apply(lambda bldg: gdf_stops.distance(bldg).min())

# Die Distanzen sind folgendermassen verteilt:

# In[59]:

gdf_bldg['distance_next_stop'].hist(bins=50)

# Wir können nun auf unserem Plot zB die Gebäude einfärben entsprechend der Distanz zur nächsten Bushaltestelle.
#
# Das Limit für die Colormap setzen wir auf 500 Meter, damit man die Unterschiede gut erkennt.

# In[60]:

# Gemeindeumrisse
ax = gdf_we.plot(linewidth=0.5, edgecolor='black', color='#000000', figsize=(20, 20))

# Gebäude, eingefärbt entsprechend der Distanz
gdf_bldg.plot(ax=ax, column='distance_next_stop', cmap='autumn_r', vmin=0, vmax=500)

ax.set_title("Wettingen AG: Nähe zur Bushaltestelle")

# In[ ]:

# In[ ]:


## INTRO.   Purpose of the GraphMaker class is to generate random graphs; these graphs can then also be edited, adding
# or deleting edges while visually inspecting the evolving map.  This makes it quick and easy to create custom "road
# networks" to use as search spaces for developing and visualizing the behavior of searching algorithms.
#
#  You ask it to create a graph, you can optionally ask it to
# plot that graph for you for visual verification purposes.  You can then edit the graph as desired, quickly adding
# or deleting edge.   Finall, you can export that graph as a simple text file of Edges
# for use in other programs.
#
# The class has multiple parameters that you can set to fine tune the nature of the graphs produced.  For now, these are
# just set by editing the values in the __init__ function below. For instance,
# you can specify edge distance distortion, percentage of edges to drop (to make sparse graphs), min. distance between
# nodes generated, etc. See comments in the init function below
#
## Usage examples:
##  x=GraphMaker(20)    # create a new random 20 node map.
#
#   x.plot()  # now graphically show it
#   x.edit()  # enter edit mode.  It repeatedly asks you to add/delete edges.
#   x.export('outfile')   # exports the current map to the file "outfile.txt" in the working dir

# TO-DO: implement a load-graph-from-file to *reload* a graph after saving...in case you want to edit and re-save

__author__ = "Eck Doerry"
__copyright__ = "Copyright 2018, Northern Arizona University, Flagstaff AZ"

import numpy as np
import scipy.spatial as graph
import matplotlib.pyplot as plt
from node import Node
from edge import Edge

class GraphMaker:

    def __init__(self,nodeCount=10):
        self.numNodes=nodeCount  # number of nodes to create in graph.
        self.expansion=2  # how wide the graph will be. Basically a factor that increases the x-y extents of plane
        self.mindist = 0.3 * self.expansion  # the minimum separation distance between nodes.  Prevents cluttered graphs
        # distDistortion.  Often you might want to have a graph where the distances are *based on* the actual
        # distances, but not exactly that.  distDistortion is a percentage within which the cartesian distances
        # between points will be randomly distorted while labeling the new edges in the graph.  So at 0.3, 30% of the
        # distance will be randomly distorted between 0 and 100%.  So you could lose/gain 30% distance, randomly.
        self.distDistortion = 0.5  # usually 0.3 is nice
        # edgeLoss: in a roadmap, we don't want edges between EVERY point to EVERY other point.  Hence we randomly drop some edges
        # the edgeLoss specifies the percentage of edges we should randomly drop
        self.edgeLoss= 0.1
        self.nodes=[] # array of node objects in this graph
        self.edges=[] # array of edges in this graph
        self.buildGraph()  # Build the new graph!  Commented out.  Call it separately after creation!

    # if you don't like the graph generated, just try another one!
    def redo(self,numNodes=-1):
        self.edges=[]
        self.nodes=[]
        if numNodes>0: self.numNodes=numNodes
        plt.close()
        self.buildGraph()
        self.plot()

    # Lets you interactively edit a graph to whack out edges
    def edit(self):
        entry=""
        print("Edit Mode:  {k/b}a,b to kill/build an edge between a,b ;  r to replot, q to quit")
        while True:
            plt.pause(0.05)
            entry=input("What now?")
            entry = entry.strip()
            if entry=='r':
                self.replot()
            elif entry=='q': break
            else:
                command=entry[0]
                [start,end]=entry[1:].split(',')
                if command=='b': self.buildEdge(start,end)
                else: self.whackEdge(start,end)
        print("End Edit mode")

    # This method lets one whack out edges.  So you can edit away edges after you generate the graph!
    def whackEdge(self,node1,node2):
        node1=node1.upper()
        node2=node2.upper()
        newEdges=[]
        for edge in self.edges:
            if edge.connects(node1,node2):  # whack the edge
                plt.plot([edge.x1, edge.x2], [edge.y1, edge.y2], 'b-', color='w', linewidth=0.5)  # make the lines
                plt.text(edge.midx, edge.midy, edge.label, size='x-small', color='w')  # label the edge at its midpoint
                plt.pause(0.05)
            else: newEdges.append(edge)
        self.edges=newEdges

    def buildEdge(self,node1,node2):
        node1 = node1.upper()
        node2 = node2.upper()
        for edge in self.edges:  # avoid building redundant new edges!
            if edge.connects(node1,node2):
                print("already an edge  from "+node1+" to "+node2)
                return
        p1=self.nodeLocation(node1)
        p2=self.nodeLocation(node2)
        if p1!=0 and p2!=0:    # both nodes exist
            print("building edge!")
            edge=Edge(p1,p2, self.distDistortion)  # make a new edge
            edge.setLabels(node1,node2)
            self.edges.append(edge)
            plt.plot([edge.x1, edge.x2], [edge.y1, edge.y2], 'b-', color='k', linewidth=0.5)  # make the lines
            plt.text(edge.midx, edge.midy, edge.label, size='x-small',color='k')  # label the edge at its midpoint
        else: print("start or end node doesn't exist")

    # Just kill and redraw the plot.  Useful after you've edited the graph
    def replot(self):
        plt.close()
        self.plot()
        plt.show()

    # MAIN WORKER:  Does the actual work of building a Roadmap.
    # The biggest challenge here is linking the given points with a *non-intersecting* set of edges.  In other words,
    # we would like a "planar graph" imposed on these points. This is what Delauney function gives us:  it basically
    # finds non-overlapping triangles that cover all points; any one point is often a vertex in multiple triagles of
    # the "puzzle" of triangles produced.  We then need to extract unique edges and the nodes from this.
    def buildGraph(self):
        pointsArray = self.genPoints()  # makes the array of (x,y) points to connect
        tri=graph.Delaunay(pointsArray)  # creates a planar graph on the given points. Creates a tri data structure.
        triangles=pointsArray[tri.simplices]  # list of triplets of three points. Each set of three points marks a
        # triangle laid out by Delauney.  Next need to extract the edges (point-to-point connections) from this.
        # Now extract the unique edges from all the triangles to generate all edges in the graph
        self.makeEdges(triangles)  # populates self.edges with Edge objects
        # Finally, turn the points array into actual Node objects.
        Node.reset(Node)  # Starts Node labeling at 'A'
        self.nodes=list(map(Node,pointsArray[:,0], pointsArray[:,1]))  #populates self.nodes with Node objects
        self.labelEdges()  # Just to make edge objects complete, go back and add endlabel (nodelabels) to those objects

    #  Makes all the edges in the new graph, based on the array of tris (basically triagles) produced by Delauney and
    # pass in.  Appends the new edges to the "edges" array.
    def makeEdges(self,triArray):
        edges = []  # temp array holding unique ((x,y),(x1,y1)) edge segments found so far

        # an important helper. When we extract a new edges from a triagle, it might well be the duplicate
        # of an edge from an adjacent triangle created by Delauney.
        def checkAddAdj(adjs):
            for adj in adjs:
                if (adj in edges) or ((adj[1], adj[0]) in edges):  # if edges already in there, skip it
                    continue
                else:
                    edges.append((adj[0], adj[1]))
        # go through the triangle array, extract each edge, check that we don't have it already, and add to edges.
        for triplet in triArray:
            s = list(map(list,triplet))
            adjs = [(s[0], s[1]), (s[1], s[2]), (s[2], s[0])]  # extracting the arcs between the points
            checkAddAdj(adjs)

        # okay, now edges array contains all unique edge segment.  Now make edge objects out of them!
        # use edgeLoss value to randomly drop some pct of Edges
        for edge in edges:
            if (np.random.rand() > self.edgeLoss):
                self.edges.append(Edge(edge[0],edge[1],self.distDistortion))  #make Edge object
            else: print("blip")

    # For each edge, finds the label for its endpoints in the nodes, and adds to edge object.
    def labelEdges(self):
        for edge in self.edges:
            edge.endlabel1=self.getNodeAtXY(edge.p1).label
            edge.endlabel2=self.getNodeAtXY(edge.p2).label

    # Function to ask the roadgraph to plot itself.  If not interactive, need to call plot.show() to see it.
    def plot(self):
        plt.ion # turn on interactive mode.  Not sure if it's critical
        #  first plot the nodes
        plt.plot([x.x for x in self.nodes], [x.y for x in self.nodes], 'ko', color="#BBBCBD")  # plot vertices circles first
        for node in self.nodes:  # now plot the labels
            plt.text(node.x, node.y, node.label, color='b', size='large', weight='normal')
        # Now plot in the edges
        for edge in self.edges:
            plt.plot([edge.x1,edge.x2],[edge.y1,edge.y2], 'b-', color='k', linewidth=0.5) # make the lines
            plt.text(edge.midx,edge.midy,edge.label, size='x-small')  # label the edge at its midpoint

    # Function to return the roadgraph as simple list that can be fed to search program.
    # prints it out, one line per edge.  Format (nodeLabel1, nodeLabel2, edgeLabel, ((x1,y1),(x2,y2),(midx,midy)) )
    # if you pass it a text string, it will print to that, else prints to console
    def export(self,outfile=0):
        if outfile != 0:
            outfilename=outfile+".txt"
            out=open(outfilename,'w')
        for edge in self.edges:
            theEdge=((self.getNodeAtXY(edge.p1).label,self.getNodeAtXY(edge.p2).label,edge.label))
            locInfo=((edge.p1,edge.p2))
            if outfile:
                out.write(str(theEdge+locInfo)+"\n")
            else:
                print(theEdge+locInfo)
        if outfile: out.close()
        print("tada!")

    # Returns a node that is located at a given point
    def getNodeAtXY(self,aPoint):  # gets the Node at location x-y of point
        for node in self.nodes:
            if (node.x==aPoint[0]) and (node.y==aPoint[1]): return node
        return 0

    # Just returns x-y location of a node, given its label
    def nodeLocation(self,label):  # finds and returns the location of node labeled 'label'
        for node in self.nodes:
            if node.label==label: return [node.x,node.y]
        print("nodeLoc: No node with label "+label+" exists")

    # a key function.  Generates a random set of points that are some minimum distance apart.
    def genPoints(self):
        # While you still need points, generate random points, add to set if min dist from all currently found points.
        points = []  # An array to put your points in as you find them.
        found = 0  # Tracks points found so far

        # Simple function that checks that candidate points are some min dist from all other points.
        # keeps it from making cluttered graphs!
        def checkpt(apoint, allpoints):
            for point in allpoints:
                dst = graph.distance.euclidean(point, apoint)
                if (dst < self.mindist): return False
            return True

        while found < self.numNodes:
            p1 = self.expansion * np.random.randn(2)
            if checkpt(p1, points):
                points.append(p1)
                found = found + 1
                # print("found one:"+str(found))
            if len(points) == 0: points.append(p1)

        # Now just turn the points into a numpy array
        pArray = np.array(points)
        # print(points)

        # AWESOME.  Now shift all the points into the positive x-y quadrant
        xmin = min(pArray[:, [0]])[0]
        ymin = min(pArray[:, [1]])[0]
        xshift = abs(xmin) + 1
        yshift = abs(ymin) + 1
        posArray = pArray + [xshift, yshift]
        posArray = posArray.astype(int)  # truncate the entire array of points into integer values. Easier!
        return posArray

    #mainly for debugging: Functions to have the graph display its edges and nodes
    def show(self):
        print("Here is the graph:")
        print("Nodes: ")
        print([x.get() for x in self.nodes])
        print("Edges: ")
        print([x.get() for x in self.edges])

    def die(self):    # just kills any open graph
        plt.close()

### UNIT TESTING
if __name__ == "__main__":
    import sys
    x=GraphMaker(sys.argv[1])
    x.export(sys.argv[2])


from matplotlib import pyplot as plt
import numpy as np
from ch1 import count_data

plt.style.use('ggplot')

colors = ["#348ABD", "#A60628"]

n_count_data = len(count_data)

plt.bar(np.arange(n_count_data), count_data, color = colors[0])

plt.xlabel("Time (days)")
plt.ylabel("Text messages received")
plt.title("Did user's habits change")
plt.show()