#!/usr/bin/env python
# coding: utf-8

# # The Computing Miniproject

# ## Introduction
#
# The computing Miniproject gives you an opportunity to try the "whole nine yards" of *asking and answering* a scientific question in biology (potentially involving multiple sub-questions/hypotheses) in a **fully reproducible way**. It will in essence give you an opportunity to perform a useful "dry run" of executing your actual Dissertation project. *It is an opportunity to do something concrete with all the computational biology techniques you have been learning.*
#
#
# ## Objectives
#
# **The general question you will address is:** *What mathematical models best fit an empirical dataset?*
#
# You may think of this as testing a set of alternative hypotheses — every alternative hypothesis is nothing but a different model to describe an observed phenomenon, as you will have learned in the model fitting lectures.
#
# ## The Project
#
# From the options provided to you (below), you will choose an empirical dataset, and a set of alternative models to fit to the data in that dataset.
#
# The Miniproject must satisfy the following criteria (and follow the accompanying guidelines):
#
# 1. **It employs *as many* of the biological computing tools you have learned so far as necessary**: shell (bash) scripting, Git, LaTeX, R, and Python. Using these tools, you will build a workflow that starts with the data and ends with a written report (in LaTeX). How you choose the different tools (e.g., how much Python vs R) is your choice; that is part of what will be assessed.
#
# 2. **Fits and compares *at least* two alternative mathematical models to the data**. The models should be fitted and selected using an appropriate method. For example you may use a combination of Ordinary Linear and Nonlinear Least Squares (NLLS) methods to fit $\ge 2$ alternative models to data, followed by model selection using AIC and BIC (read the Johnson and Omland 2005 paper in the Readings & Resources section below).*
#
# 3. **It should be fully reproducible.** You will write a script that "glues" the workflow together and runs it, from data processing, to model fitting, to plotting, to compilation of the written report (*More detailed instructions on report below*). Refer back to the TheMulQuaBio Computing chapters to see how you would run the different components. For example, we have covered how to run R and compile $\LaTeX$ using the `subprocess` module in the [second Python Chapter](./06-Python_II.ipynb). The assessor should be able to run just this script to get everything to work without errors.
#
# *You will be given lectures and practicals on model fitting before you start on your Miniproject.*
#
# *Please read the papers in the **Readings and Resources** section below* — these will help orient you in the right direction for tackling your Miniproject.
#
#
# ## The Report
#
# The report should,
#
# * be written in LaTeX using the article document class, in 11pt (any font will do, within reason!).
#
# * be 1.5-spaced, with *continuous* line numbers.
#
# * have a Title, Author name with Affiliation and Word count on a *separate Title page*.
#
# * have an Introduction with objectives of the study, and appropriate additional sections such as Methods, Data, Results, Discussion, etc.
#
# * contain in the *Methods* a sub-section called "Computing tools" which states briefly how each of the scripting languages (bash, R, Python) was used and what packages within them were used and a justification of why.
#
# * have References properly cited in text and formatted in a list using bibtex.
#
# * contain $\leq$3500 words *excluding the contents of the title page, references, and Figure or Table captions+legends*. There should be a word count at the beginning of the document (typically using the `texcount` package).
#
# (Miniproject-report-guidelines)=
# ### Guidelines
#
# Please read the *general* (*not* word count, formatting etc.) dissertation writing guidelines given in the Silwood Masters Student Guidebook.
#
# ```{tip}
# **Start writing early**. Its NEVER too early to start writing! Outline the structure of your report and attempt to write a brief introduction even if you don't have any results, or have not finalized your methods, computational work flow, or analyses. Doing this preliminary writing will force you to think about the logic of what you are planning to do, put your planned work in some context, and (in most cases!) motivate you.
# ```
#
# Here are some additional suggestions/guidelines:
#
# * **In General**:
#     * In scientific writing (papers, reports), a "narrative" or "flow" is important. What this means (more on each of these components below):
#         * Starting with the Title, and through the Introduction, Methods, Results and Discussion, there is a common thread (focal issue or topic).
#         * The *Title* gives a summary of what the article is about, and may even convey what the main finding is.
#         * The *Introduction* clearly and accurately builds and "expectation" for the reader, i.e., what to look for in the subsequent sections.
#         * The *Methods* and *Results*, to thee extent possible, follow the same sequence of topics (and questions/hypotheses) that were laid out in the Introduction
#         * The *Discussion* reminds the reader about what the original goals of the study were, states out key findings succinctly, and then discusses their implications in the wider context and then finished off with some caveats and a conclusion that delivers the final take-away messages.
#     * Avoid sub-sectioning (with headers) the *Introduction* and *Discussion* sections as it breaks the flow of your "narrative". On the other hand, you will almost always sub-section the Methods, and to an extent the Results sections.
#     * Pay attention to detail:
#         * Do a spell check on the final draft
#         * Make sure that all graphics are rendered in good quality (use [vector graphic](https://en.wikipedia.org/wiki/Vector_graphics) formats as much as possible). Remember, $\LaTeX$ allows you to embed vector graphics in pdf.
#         * Make sure that all the display items (Tables + Figures) have a text caption that states what the display item is for, and then a text legend that explains the figure and delivers any take home messages.
#     * The display items alone should be able to tell most of the story. Once you have an outline of the manuscript, first, before doing any more writing, put in the display items (generally, 4-6 should be enough) with Captions and Legends, and see if they are indeed telling the story you would like your paper/report to tell.
#     * Avoid the words "explore" or "look at" to describe your ob objectives
#     * Use direct speech (as it is YOUR work!)
#         * So, for example, avoid phrases such as "This study investigates"; say "Here I investigate" or something like that instead.
#
# * **The Title**
#    * The Title should give a summary of what the article is about, and may even convey the main finding(s). Make it as result-focused as possible, and avoid being vague.
#    * Keep the number of words to a minimum (upto 10-15 words is reasonable).
#        * Some succinct title examples:
#            * "The role of xx in determining yy"
#            * "The relative success of xx models in providing parameter estimates for yy"
#            * *Or better still*: "xx models out-perform yy models for quantifying zz data"
#            * *OR even better still*: "xx [organisms / traits] differ systematically across yy [some grouping variable, such as location or taxonomic categories]"
#        * Some not-so-nice title examples:
#            * "A comparison of models for describing zz using Linear and no-linear model fitting with AIC/BIC"
#            * "An exploration of xx models for describing yy data using Linear and Non-linear model fitting with model selection "
#
# * **Abstract**: *The report must have an Abstract.*
#     * It should be a "mini-paper" in itself: So, 1-2 lines on background, 1-2 lines on the paper's objectives, 1-2 lines on the methods, 1-2 lines on the main results, 1-2 lines on the main conclusions + take home messages. Remember the abstract counts towards the total work limit (at least as far as your Mini-project report is concerned), so you will need to be succinct. About 200 words is the suggested maximum.
#     * Do not be vague about the take home messages at the end of the abstract. For example, do not say something like "Thus this study shows that more work needs to be done...", or "This study shows that model selection is useful...". Try instead to say something like "This study shows that in general, xx models are better suited for yy data..." or "This study provides evidence that in xx [organisms / traits], yy models tend to under-perform because..." (NOTE THAT THESE ARE HYPOTHETICAL/EXAMPLE STATEMENTS!).
#
# * **Introduction**:
#     * The Introduction should open with a general *enough* background (with citations). What is "general enough"? – A context that *justifies* the main focus of the study, and *motivates* the reader. So, for example, if the focus is population growth rates, then provide a context for why growth rates are important to study.
#     * Towards/At the end of the Introduction, provide some specific questions or hypotheses that will be addressed in the study. But do not present hypotheses if they are not backed by logical arguments or mathematical / computational modelling / theory. Asking questions is better than (logically/theoretically) unfounded hypotheses. The *narrative* of the Introduction should funnel the reader's attention *naturally* towards the stated hypotheses/questions; the hypotheses/questions should not come out of the blue.
#     * And if you are going with hypotheses, add statements following the hypotheses that briefly explain the logic behind each hypothesis.
#
# * **Methods**:
#     * This will typically include subsections for key elements of your methods (e.g., Data, Models, Model fitting, etc).
#     * Do not go overboard with describing every detail and every step of your workflow. For example, you do not need to state that "figures were plotted in ggplot and saved to a directory called xx".
#     * Note the additional requirement for the miniproject report to include a section on specific computing languages and tools used and the justification for using them.
#
# * **Results**:
#     * These can also be sub-sectioned by the main questions/hypotheses/issues your are tackling.
#     * Avoid any discussion of the results.
#     * Reference the Figures and Tables clearly and specifically (e.g., refer to key sub-panels of figures when needed)
#
# * **Discussion**:
#     * Stat by reminding the reader about what the original goals of the study were
#     * State key findings succinctly
#     * Then discuss their implications in the wider context (with additional referencing beyond what you had in the intro).
#     * Include a paragraph or two of caveats/shortcomings with clear indication of what future work can do to address them.
#     * End with conclusion that delivers the final take-away messages.
#
# * **Supplementary Information** (SI):
#     * If used, SI should be a separate document and cited in the main text.
#     * Make sure it is a separate document that includes is own references and sections/subsections.
#     * When citing the SI, cite specific sections/subsections.
#     * The SI should be concatenated with the main document in the final submission.
#
# ## Submission
#
# Add, commit and push all your work to your bitbucket repository using a directory called `MiniProject` at the same level as the Week1, Week2 etc. directories, by the Miniproject deadline given to you.
#
# At this stage, you are not going to be told you how to organize your project — that's part of the marking criteria (see next section).
#
# ```{note}
# The single script that runs the whole project should be called *run_MiniProject*, with an appropriate extension (e.g., `run_MiniProject.py` or `run_MiniProject.sh`).
# ```
#
# ## Marking criteria
#
# *Equal weightage will be given to the code+workflow and writeup components — each component will be marked to a max of 100 pts and then rescaled to a single mark / 100 using equal weightage*
#
# The assessor will be looking for the following while assessing your submission:
#
# ### Computing
#
# * A well-organized project where code, results, data, etc., are easy to locate, inspect, and use. In the project's README also include:
#
#     * Version of each language used
#
#     * Any dependencies or special packages the user/marker should be aware of
#
#     * What each package you used is for
#
# * A project that runs smoothly and efficiently, without any errors once a single script is called.
#
#
# ### Report
#
# * A report that contains all the components indicated above in "The Report" subsection, with some original thought and synthesis in the *Introduction* and *Discussion* sections.
#
# * Quality of the presentation of the graphics and tables in your report, as well as any plots showing model fits to the data.
#
# * Don't forget to read the report guidelines [above](Miniproject-report-guidelines).
#
# ### Overall
#
# * The marking criteria you may refer to for both components are the [summative marking criteria](https://github.com/mhasoba/TheMulQuaBio/raw/master/content/readings/MARKING_CRITERIA.pdf).
# * The goal is to fit as many mathematical models as possible, but the minimum being 2 (to allow model comparison). You will get more marks for picking more "difficult" models to fit and compare (basically, one or more non-linear mathematical models).
#     * However, note that you need to pick a problem that is within reach. You will not get extra marks for attempting to fit one or more "difficult" models and then, failing overall to achieve a coherent report and model fitting exercise, because, for example, you ended up spending too much time on the "difficult" models(s).

# ## Suggested Workflow
#
# You will build a workflow that starts with the data and ends with a report written in LaTeX.
#
# The following components and sequence of your workflow are suggested (you may choose to do it differently).
#
# ### Data preparation script
#
# First, a script that imports the data and prepares it for model fitting. This may be in Python or R, and will typically have the following features:
#
# * Creates unique ids so that you can identify unique datasets (e.g., single thermal responses or functional responses). *This may not always be necessary because your data might already contain a field that delineates single curves (e.g., an `ID` field/column)*
# * Deals with missing, and other problematic data values.
# * Saves the modified data to one or more csv file(s).
#
#
# ### Model fitting script
#
# A separate script that does the Model fitting. For example, it may have the following features:
#
# * Opens the (new, modified) dataset from previous step.
#
# * Does model fitting. Ultimately you need to fit at least one mechanistic/nonlinear model along with one or more linear models, but for building your workflow, just go ahead an fit a couple of different linear models (e.g., linear regression bvs quadratic and / or cubic polynomial).
#
# * Calculates AIC, BIC, R$^{2}$, and other statistical measures of model fit (you decide what you want to include).
#
# * Exports the results to a csv that the [final plotting script](#Final-plotting-script) can read.
#
#
# ```{note}
# * Some data series (e.g., a single growth rate or functional response curve) may have insufficient data points for fitting a particular model. That is, the number of unique x-axis values is $\le k$, where $k$ is the number of parameters in the model (e.g., a regression line has two parameters). Your model fitting will fail on such datasets, but you can deal with those failures later (e.g., by using the `try` keyword that you have learned in both Python and R chapters). In particular, the model fitting (or estimation of goodness of fit statistics) will fail for datasets with small sample sizes, and you can then filter these datasets *after* the Model fitting script has finished running and you are in the Analysis phase.
# ```
#
# ### Final plotting and analysis script
#
# * Next, write a script that imports the results from the previous step and plots every curve with the two (or more) models (or none, if nothing converges) overlaid.
#     * Doing this will help you identify poor fits visually and help you decide whether the model fitting (e.g., using NLLS) can be further optimized.
#     * All plots should be saved in a single separate sub-directory.
#
# * This script will also perform any analyses of the results of the Model fitting, for example to summarize which model(s) fit(s) best, and address any biological questions involving co-variates.
#
# ### Report compiling script
#
# * Then comes the $\LaTeX$ source code and a (typically, Bash) script that compiles it.
#
# ### A single script to run them all
#
# * Finally, write a script called `run_MiniProject.py` or `run_MiniProject.sh` respectively, which runs the whole project, right down to compilation of the LaTeX  document.

# ## For NLLS fitting
#
# **FIRST read and work through the materials [here](./20-ModelFitting-NLLS.ipynb) (NLLS in R) and [here](./Appendix-NLLS-Python.ipynb) (NLLS in Python).**
#
# * You will typically need to write a script that calculates starting values (more on this topic [here](Model-Fitting-NLLS-Starting-Values)).
#
# * You will need to use the `try` keyword because not all runs will converge. *The more data curves you are able to fit, the better — that is part of the challenge*
#
# *One thing to note is that you may need to do the NLLS fitting on the logarithm of the function (and therefore, the data) to facilitate convergence (examples are [here](./20-ModelFitting-NLLS.ipynb) and [here](./Appendix-NLLS-Python.ipynb).
#
# ## Getting started
#
# Doing all this may seem a bit scary at the start. However, if you approach the problem systematically and methodically, you will soon be on your way.
#
# ```{tip}
# The Miniproject is also an exercise in learning to pick the right size of (computational) problem given the amount of time you have to solve it. So even if you might be tempted to take on, at the very start, a very ambitious project (basically, picking both linear and non-linear models) and *then* trying to develop your workflow, you will very likely get stuck in "local optima" in terms of the overall workflow design and implementation. It is important that you first pick a "bite sized" problem (e.g., two linear models), and develop the overall computational work flow, from plotting and fitting to model selection. At the same time, *also start at least outlining  the report* based on the first, simple, tractable problem you pick.
# ```
#
# Here are some suggested first steps to get started:
#
# * Explore the data in R or Python (e.g., using Jupyter) (first part of the suggested workflow above).
#
# * Write a preliminary version of the plotting script without the fitted models overlaid. That will also give you a feel for the data and allow you to see (literally) what shapes the curves can take.
#
# * Explore the models you will be fitting. Basically, plot them: Write mathematical functions you want to fit in a Python/R script (you can then re-use these functions in your model fitting script as well), and then evaluate them  numerically to see the shape of the function.
#
# ```{tip}
# Remember to sandbox and/or gitignore any code and output for exploratory plotting of the functions in the final product.
# ```
#
# * **For NLLS fitting**, figure out, using one, "nice-looking" functional response, population growth curve/dataset, or thermal response, test how the NLLS fitting package and its commands work. This is your minimal example that will give you confidence that it works!
#    * Next, write a loop over all unique datasets (data curves) using the `try` to catch errors (and examine them carefully) in case the fitting doesn't converge.

# ## The Dataset and Model Options
#
# You can pick from one of the following three sets of options.
#
# First, let's load some packages to explore the data sets in Python:

# In[1]:

import pandas as pd
import scipy as sc
import matplotlib.pylab as pl
import seaborn as sns # You might need to install this (e.g., pip install seaborn)

# ### Population Growth
#
# #### The Question
#
# *How well do different mathematical models, e.g., based upon population growth (mechanistic) theory  vs. phenomenological ones, fit to functional responses data across species?*
#
# Fluctuations in the abundance (density) of single populations may play a crucial role in ecosystem dynamics and emergent functional characteristics, such as rates of carbon fixation or disease transmission. A population grows exponentially while its abundance is low and resources are not limiting (the Malthusian principle). This growth then slows and eventually stops as resources become limiting. There may also be a time lag before the population growth really takes off at the start. We will focus on microbial (specifically, bacterial) growth rates. Bacterial growth in batch culture follows a distinct set of phases; lag phase, exponential phase and stationary phase. During the lag phase a suite of transcriptional machinery is activated, including genes involved in nutrient uptake and metabolic changes, as bacteria prepare for growth. During the exponential growth phase, bacteria divide at a constant rate, the population doubling with each generation. When the carrying capacity of the media is reached, growth slows and the number of cells in the culture stabilises, beginning the stationary phase. Traditionally, microbial growth rates were measured by plotting cell numbers or culture density against time on a semi-log graph and fitting a straight line through the exponential growth phase – the slope of the line gives the maximum growth rate ($r_{max}$). Models have since been developed which we can use to describe the whole sigmoidal bacterial growth curve.
#
# #### The Data
#
# The dataset is called `LogisticGrowthData.csv`. It contains measurements of change in biomass or number of cells of microbes over time. These data were collected through lab experiments across the world. The field names are defined in a file called  `LogisticGrowthMetaData.csv`, also in the `data` directory. The two main fields of interest are `PopBio` (abundance), and `Time`. Single population growth rate curves can be identified by as unique  temperature-species-medium-citation-replicate combinations (concatenate them to get a new string variable that identifies unique growth curves).
#
# Let's have a look at the data:

# In[20]:

data = pd.read_csv("../data/LogisticGrowthData.csv")
print("Loaded {} columns.".format(len(data.columns.values)))

# In[21]:

print(data.columns.values)

# In[22]:

pd.read_csv("../data/LogisticGrowthMetaData.csv")

# In[23]:

data.head()

# In[24]:

print(data.PopBio_units.unique()) #units of the response variable

# In[25]:

print(data.Time_units.unique()) #units of the independent variable

# Unlike the previous two datasets there are no ID coulmns, so you will have to  infer single growth curves by combining `Species`, `Medium`, `Temp` and `Citation` columns (each species-medium-citation combination is unique):

# In[26]:

data.insert(0, "ID", data.Species + "_" + data.Temp.map(str) + "_" + data.Medium + "_" + data.Citation)

# Note that the `map()` method coverts temperature values to string (`str`) for concatenation.

# In[27]:

print(data.ID.unique()) #units of the independent variable

# These are rather ungainly IDs, so you might want to replace them with numbers!

# In[28]:

data_subset = data[data['ID']=='Chryseobacterium.balustinum_5_TSB_Bae, Y.M., Zheng, L., Hyun, J.E., Jung, K.S., Heu, S. and Lee, S.Y., 2014. Growth characteristics and biofilm formation of various spoilage bacteria isolated from fresh produce. Journal of food science, 79(10), pp.M2072-M2080.']
data_subset.head()

# In[29]:

sns.lmplot("Time", "PopBio", data = data_subset, fit_reg = False) # will give warning - you can ignore it

# #### The Models
#
# Yet again, the simplest mathematical models you can use are the phenomenological quadratic and cubic polynomial models, that is eqns. 1 and 2 above (replace $x$ with Time). A Polynomial model may be able to capture decline in population size after some maximum value (the carrying capacity) has been reached (the "death phase" of population growth). For two mechanistic models of population growth (Logistic and Gompertz), have a look at the [Model Fitting Chapter](./20-ModelFitting-NLLS.ipynb).
#
# ---
#
# ![image](./graphics/Pop_Grow.svg)
# <small> <center> An example population growth curve dataset to which the modified Gompertz model (Zwietering et. al., 1990) has been fitted.
# </center></small>
#
# (See the [Model fitting in NLLS Chapter](./20-ModelFitting-NLLS.ipynb) for more information)
#
# ---
#
# In addtion to the Gompertz model, two growth rate models that also include a lag phase are the Baranyi model (Baranyi, 1993), and the Buchanan model (or three-phase logistic model; Buchanan, 1997). Please see the Readings & Resources section below for the full references of these papers.

# ### Functional Responses
#
# #### The Question
#
# *How well do different mathematical models, e.g., based upon foraging theory (mechanistic) principles  vs. phenomenological ones, fit to functional responses data across species?*
#
# In ecological parlance, a functional response is the relationship between a consumer's (e.g., predator) biomass consumption rate and abundance of the target resource (e.g., prey). Functional responses arise from fundamental biological and physical constraints on consumer-resource interactions (e.g., Holling 1959, Pawar et al, 2012), and determine the rate of biomass flow between species in ecosystems across the full scale of sizes, from the smallest (e.g., microbes) to the largest (e.g., blue whales). Functional responses also play a key sole in determining the stability (responses to perturbations) of the food webs that underpin ecosystems.
#
# #### The Data
#
# The dataset is called `CRat.csv`. It contains measurements of rates of consumption of a single resource (e.g., prey, plants) species' by a consumer species (e.g., predators, grazers). These data were collected through lab and field experiments across the world. The field names are defined in a file called `BiotraitsTemplateDescription.pdf`, also in the `data` directory. The two main fields of interest are `N_TraitValue` (The number of resources consumed per consumer per unit time), and `ResDensity` (the resource abundance). Individual functional response curves can be identified by `ID` values --- each `ID` corresponds to one curve. Or you can reconstruct them as unique combinations of `Citation` (where the functional response dataset came from), `ConTaxa` (consumer species ID), `ResTaxa` (resource species ID).
#
# Let's have a look at the data:

# In[12]:

data = pd.read_csv("../data/CRat.csv")
print("Loaded {} columns.".format(len(data.columns.values)))

# In[13]:

data.head()

# In[14]:

print(data.columns.values)

# In[15]:

print(data.TraitUnit.unique()) #units of the response variable

# In[16]:

print(data.ResDensityUnit.unique()) #units of the independent variable

# In[17]:

print(data.ID.unique()) #units of the independent variable

# In[18]:

data_subset = data[data['ID']==39982]
data_subset.head()

# In[19]:

sns.lmplot("ResDensity", "N_TraitValue", data=data_subset, fit_reg=False)

# (Miniproject-FR-Models)=
# #### The Models
#
# *All the following parameters and variables are in SI units*.
#
# The fundamental measure of interest (the response variable) is consumption rate ($c$). This is expressed in terms of biomass quantity or number of individuals of resource consumed *per unit time per unit consumer* (so units of Mass (or Individuals) / Time).
#
# Again, the simplest mathematical models you can use are the phenomenological quadratic and cubic polynomial models, that is eqns. {eq}`eq:quad` and {eq}`eq:cubic` (replace $x$ with resource abundance).
#
# Then, there is the more mechanistic Holling Type II model (Holling, 1959):
#
# $$
#       c = \frac{a x_R}{1 + h a x_R}
# $$(eq:FR_II)
#
# Here, $x_R$ is resource density (Mass / Area or Volume), $a$ is consumer's search rate (Area or Volume / Time ), and  $h$ is handling time of the consumer for that resource (time taken to overpower and ingest it).
#
# Below is an example FR curve from the dataset you have been given with the Type II model fitted to it.
#
# ---
# ![image](./graphics/3_FR.svg)
# <small>  <center> Example of the a Type II model (eqn. {eq}`eq:FR_II`) fitted to a functional response of a consumer on a resource.
# </center> </small>
#
# ---
#
# There is also the less-mechanistic "generalized" functional response model:
#
# $$
#       c = \frac{a x_R^{q + 1}}{1 + h a x_R^{q + 1}}
# $$(eq:FR_gen)
#
# Where everything is same as eqn. {eq}`eq:FR_II`, but the additional parameter $q$ (dimensionless) is a shape parameter that allows the shape of the response to be more flexible/variable, from "Type I" to "Type III". This model is less mechanistic because it includes a phenomenological parameter $q$ which does not have a formal biological meaning.
#
# ```{note}
# Note that if $q=0$, eqn {eq}`eq:FR_gen` becomes same as the Type II model (eqn. {eq}`eq:FR_II`).
# ```
# ---
# ![image](./graphics/FR.svg)
# <small> <center> The range of functional responses captured by the generalized functional response model (eqn. {eq}`eq:FR_gen`).
# </center>
# </small>
#
# ---
#
# There are not too many other models for functional responses, though you can and should try looking for them in the literature. One more mechanistic model that defines parameters of the Type II functional response in terms of body size of predator and prey can be found in  Pawar et al (2012).

# ### Thermal Performance Curves
#
# #### The Question
#
# *How well do different mathematical models, e.g., based upon biochemical (mechanistic) principles  vs. phenomenological ones, fit to the thermal responses of metabolic traits?*
#
# This is currently a "hot" (no pun intended!) topic in biology. On the *ecological side*, because the temperature-dependence of metabolic rate sets the rate of intrinsic $r_\text{max}$ (papers by Savage et al., Brown et al.) as well as interactions between species, it has a strong effect on population dynamics. In this context, note that 99.9% of life on earth is ectothermic! On the *evolutionary side*, the temperature-dependence of fitness and species interactions also means that warmer environments may have stronger rates of evolution. This may be compounded by the fact that mutation rates may also increase with temperature (papers by Gillooly et al.).
#
# #### The Data
#
# The dataset is called `ThermRespData.csv`. It contains a subset of the full "BioTraits" database. This subset contains hundreds of "thermal responses" for growth, respiration and photosynthesis rates in plants and bacteria (both aquatic and terrestrial). These data were collected through lab experiments across the world, and compiled by various people over the years. The field names are defined in a file called `BiotraitsTemplateDescription.pdf`, also in the `data` directory. The two main fields of interest are `OriginalTraitValue` (the trait values responding to temperature), and `ConTemp` (the temperature). Individual thermal response curves can be identified by `ID` values --- each `ID` corresponds to one thermal performance curve.
#
# Let's have a look at the data:

# In[3]:

data = pd.read_csv("../data/ThermRespData.csv")
print("Loaded {} columns.".format(len(data.columns.values)))

# In[4]:

data.head()

# In[5]:

print(data.columns.values)

# In[6]:

print(data.OriginalTraitUnit.unique()) #units of the response variable

# In[7]:

print(data.ConTempUnit.unique()) #units of the independent variable

# In[8]:

print(data.ID.unique()) #units of the independent variable

# In[9]:

data_subset = data[data['ID']==110]
data_subset.head()

# In[11]:

sns.lmplot("ConTemp", "OriginalTraitValue", data=data_subset, fit_reg=False)

# (Miniproj-TPCs-Models)=
# #### The Models
#
# *All the following parameters and variables are in SI units*.
#
# There are multiple models that might best describe these data. The simplest are the general quadratic and cubic polynomial models:
#
# $$
#     B = B_0 + B_1 x + B_2 x^2
# $$(eq:quad)
#
#
# $$
#     B = B_0 + B_1 x + B_2 x^2 + B_3 x^3
# $$(eq:cubic)
#
# These are phenomenological models, with the parameters $B_0$, $B_1$, $B_2$ and $B_3$ lacking any mechanistic interpretation. $x$ is the independent variable (in this case Temperature, $T$)
#
# Another phenomenological model option is the Briere model:
#
# $$
# B = \left\{
#         \begin{array}{ll}
#             0 & \quad T \leq T_0 \\
#             B_0 T (T-T_0) \sqrt{T_m-T} & \quad T_0 \leq T \leq T_m \\
#             0 & \quad T \geq T_m
#         \end{array}
#     \right.
# $$(eq:Briere1)
#
# Where $T$ is temperature, $T_0$ and $T_m$ are the minimum and maximum feasible temperatures for the trait (below or above which the traits goes to zero), and $B_0$ is a normalization constant.  Example R code for fitting this mdoel can be found [here](Model-Fitting-NLLS-TPCs).
#
# If you look at the original paper, you will find that Briere et al also propose a more general version of this equation (by adding a new parameter $m$ to replace the square root above):
#
# $$
# B = \left\{
#         \begin{array}{ll}
#             0 & \quad T \leq T_0 \\
#             B_0 T (T-T_0) (T_m-T)^\frac{1}{m} & \quad T_0 \leq T \leq T_m \\
#             0 & \quad T \geq T_m
#         \end{array}
#     \right.
# $$
#
#
# In contrast, the Schoolfield model (Schoolfield et al 1981) is a mechanistic option that is based upon thermodynamics and enzyme kinetics:
#
# $$
#     B = \frac{B_0 e^{\frac{-E}{k} (\frac{1}{T} - \frac{1}{283.15})}}
#     { 1 + e^{\frac{E_l}{k} (\frac{1}{T_l} - \frac{1}{T})} +
#     e^{\frac{E_h}{k} (\frac{1}{T_h} - \frac{1}{T})}}
# $$(eq:schoolf)
#
# Here, $k$ is the Boltzmann constant ($8.617 \times 10^{-5}$ eV $\cdot$ K$^{-1}$), $B$ the value of the trait at a given temperature $T$ (K) (K = $^\circ$C + 273.15), while $B_0$ is the trait value at 283.15 K (10$^\circ$C) which stands for the value of the growth rate at low temperature and controls the vertical offset of the curve. $E_l$ is the enzyme's low-temperature de-activation energy (eV) which controls the behavior of the enzyme (and the curve) at very low temperatures, and $T_l$ is the at which the enzyme is 50% low-temperature deactivated. $E_h$ is the
# enzyme's high-temperature de-activation energy (eV) which controls the behavior of the enzyme (and the curve) at very high temperatures, and $T_h$ is the at which the enzyme is 50% high-temperature deactivated. $E$ is the activation energy (eV) which controls the rise of the curve up to the peak in the "normal operating range" for the enzyme (below the peak of the curve and above $T_h$).
#
# *Please also have a look at the Delong et al 2017 paper, which lists this and other mechanistic TPC models* (see [Readings & Resources](Miniproj-Readings)). You may choose additional models listed in that paper for comparison, if you want.
#
# ---
#
# ![image](./graphics/SchoolfEx.png)
# <small> <center>Example of the full Sharpe-Schoolfield model (Eqn. {eq}`eq:schoolf`) fitted to the thermal response curve of a metabolic trait $x$ with resource abundance. </center> </small>
#
# ---
#
# In many cases, a simplified Schoolfield model would be more appropriate for thermal response data, because low temperature inactivation is weak, or is undetectable in the data because low-temperature measurements were not made.
#
# $$
#       B = \frac{B_0 e^{\frac{-E}{k} (\frac{1}{T} - \frac{1}{283.15})}}
#     { 1 +  e^{\frac{E_h}{k} (\frac{1}{T_h} - \frac{1}{T})}}
# $$(eq:schoolfH)
#
# In other cases, a different simplified Schoolfield model would be more appropriate, because high temperature inactivation was not detectable in the data because measurements were not made at sufficiently high temperatures:
#
# $$
#       B = \frac{B_0 e^{\frac{-E}{k} (\frac{1}{T} - \frac{1}{283.15})}}
#     { 1 +  e^{\frac{E_l}{k} (\frac{1}{T_l} - \frac{1}{T})}}
# $$(eq:schoolfL)
#
# Note that the cubic model (Eqn. {eq}`eq:cubic`) has the same number of parameters as the the reduced Schoolfield models (Eqn. {eq}`eq:schoolfH` & {eq}`eq:schoolfL`). Also, the temperature parameter ($T$) of the cubic model (Eqn. {eq}`eq:cubic`) is in $^\circ$C, whereas the Temperature parameter in the Schoolfield model is in K.

# ## Additional models and questions you can tackle
#
# In all three options above, you may try to tackle fitting to additional models you find in the literature. Some Readings have been provided for each of the three data types below.
#
# You may choose to tackle some biological hypotheses or explore patterns by considering additional covariates. For example,
#
# *Do different taxa show different functional responses?*
#
# *Does temperature or taxon identity affect which population growth rate model fits best?*
#
# *Do different models fit different types of thermal performance curves (e.g., Photosynthesis vs Respiration)?*
#
# You may also choose to revisit the results of another paper that has done comparisons of the models you have chosen with your new dataset (but remember, that may well become too ambitious a project given the time you have).

# (Miniproj-Readings)=
# ## Readings & Resources
#
# Many of these papers are in pdf format in the Readings directory on TheMulQuaBio repository.
#
# ### General
#
# * Levins, R. (1966) The strategy of model building in population biology. Am. Sci. 54, 421–431.
#
# * Johnson, J. B. & Omland, K. S. (2004) Model selection in ecology and evolution. Trends Ecol. Evol. 19, 101–108.
#
# * Motulsky, H. & Christopoulos A. (2004) Fitting models to biological data using linear and nonlinear regression: a practical guide to curve fitting. Oxford University Press, USA.
#
# * Bolker, B. M. et al. (2013) Strategies for fitting nonlinear ecological models in R, AD Model Builder, and BUGS. Methods Ecol. Evol. 4, 501–512.
#
#
# ### Functional responses
#
# * Holling, C. S. 1959. Some Characteristics of Simple Types of Predation and Parasitism. The Canadian Entomologist 91 (7): 385–98. https://doi.org/10.4039/Ent91385-7.
#
# * Holling, C S. 1966. The Functional Response of Invertebrate Predators to Prey Density. Mem. Entomol. Soc. Canada 48 (48): 1–86.
#
# * Aljetlawi, A. A., E. Sparrevik, and K. Leonardsson. 2004. Prey-predator size-dependent functional response: derivation and rescaling to the real world. J. Anim. Ecol. 73, 239–252.
#
# * Jeschke, J. M.,  M. Kopp & R. Tollrian. 2002. Predator functional responses: Discriminating between handling and digesting prey. Ecol. Monogr. 72, 95–112.
#
# * Pawar, S., A. I. Dell, and V. M. Savage. 2012. Dimensionality of Consumer Search Space Drives Trophic Interaction Strengths. Nature 486 (7404): 485–89. https://doi.org/10.1038/nature11131.
#
# * Pritchard, D. W., R. A. Paterson, H. C. Bovy, and D. Barrios-O'Neill. 2017. frair: an R package for fitting and comparing consumer functional responses. Methods Ecol. Evol. 8, 1528–1534.
#
# ### Population Growth
#
# * Zwietering, M. H., I. Jongenburger, F. M. Rombouts, and K. Van't Riet. 1990. Modeling of the Bacterial Growth Curve. Applied and Environmental Microbiology 56 (6): 1875–81.
#
# * Buchanan, R. L., R. C. Whiting, and W. C. Damert. 1997. When Is Simple Good Enough: A Comparison of the Gompertz, Baranyi, and Three-Phase Linear Models for Fitting Bacterial Growth Curves. Food Microbiology 14 (4): 313–26. https://doi.org/10.1006/fmic.1997.0125.
#
# * Grijspeerdt, K. and P. Vanrolleghem. 1999. Estimating the parameters of the Baranyi model for bacterial growth. Food Microbiol. 16, 593–605.
#
# * Micha, P., and M. G. Corradini. 2011. Microbial Growth Curves: What the Models Tell Us and What They Cannot. Critical Reviews in Food Science and Nutrition. https://doi.org/10.1080/10408398.2011.570463.
#
# ### Thermal Performance Curves
#
# * Schoolfield, R. M., P. J H Sharpe, and C. E. Magnuson. 1981. Non-Linear Regression of Biological Temperature-Dependent Rate Models Based on Absolute Reaction-Rate Theory. Journal of Theoretical Biology 88 (4): 719–31. https://doi.org/10.1016/0022-5193(81)90246-0.
#
# * Zwietering, M. H.,  J. T de Koos, B. E. Hasenack, J. C. de Witt,  and K. van't Riet. 1991. Modeling of bacterial growth as a function of temperature. Appl. Environ. Microbiol. 57, 1094–101.
#
# * Briere J. F., Pracros P., Le Roux A. Y., Pierre J. S. 1999. A novel rate model of temperature-dependent development for arthropods. Environ Entomol 28: 22–29.
#
# * Dell, A. I., S. Pawar, and V. M. Savage. 2011. Systematic Variation in the Temperature Dependence of Physiological and Ecological Traits. Proceedings of the National Academy of Sciences of the United States of America 108 (26): 10591–10596. https://doi.org/doi: 10.1073/pnas.1015178108.
#
# * DeLong, J. P., J. P. Gibert, T. M. Luhring, G. Bachman, B. Reed, A. Neyer, and K. L. Montooth. 2017. The Combined Effects of Reactant Kinetics and Enzyme Stability Explain the Temperature Dependence of Metabolic Rates. Ecology and Evolution 7 (11): 3940–50. https://doi.org/10.1002/ece3.2955.


def add_numbers(x, y):
    return x + y

add_numbers(1, 2)

def add_numbers(x,y,z=None):
    if (z==None):
        return x+y
    else:
        return x+y+z

print(add_numbers(1, 2))
print(add_numbers(1, 2, 3))

def add_numbers(x, y, z=None, flag=False):
    if (flag):
        print('Flag is true!')
    if (z==None):
        return x + y
    else:
        return x + y + z

print(add_numbers(1, 2, flag=True))

def add_numbers(x,y):
    return x+y

a = add_numbers
a(1,2)

type('This is a string')

type(None)

type(1)

type(1.0)

type(add_numbers)

x = (1, 'a', 2, 'b')
type(x)

x = [1, 'a', 2, 'b']
type(x)

x.append(3.3)
print(x)

for item in x:
    print(item)

i=0
while( i != len(x) ):
    print(x[i])
    i = i + 1

[1,2] + [3,4]

[1]*3

1 in [1, 2, 3]

x = 'This is a string'
print(x[0]) #first character
print(x[0:1]) #first character, but we have explicitly set the end character
print(x[0:2]) #first two characters

x[-1]

x[-4:-2]

x[:3]

x[3:]

firstname = 'Christopher'
lastname = 'Brooks'

print(firstname + ' ' + lastname)
print(firstname*3)
print('Chris' in firstname)

firstname = 'Christopher Arthur Hansen Brooks'.split(' ')[0] # [0] selects the first element of the list
lastname = 'Christopher Arthur Hansen Brooks'.split(' ')[-1] # [-1] selects the last element of the list
print(firstname)
print(lastname)

'Chris' + 2

'Chris' + str(2)

x = {'Christopher Brooks': 'brooksch@umich.edu', 'Bill Gates': 'billg@microsoft.com'}
x['Christopher Brooks'] # Retrieve a value by using the indexing operator

x['Kevyn Collins-Thompson'] = None
x['Kevyn Collins-Thompson']

for name in x:
    print(x[name])

for email in x.values():
    print(email)

for name, email in x.items():
    print(name)
    print(email)

x = ('Christopher', 'Brooks', 'brooksch@umich.edu')
fname, lname, email = x

fname

lname

x = ('Christopher', 'Brooks', 'brooksch@umich.edu', 'Ann Arbor')
fname, lname, email = x

print('Chris' + 2)

print('Chris' + str(2))

sales_record = {
'price': 3.24,
'num_items': 4,
'person': 'Chris'}

sales_statement = '{} bought {} item(s) at a price of {} each for a total of {}'

print(sales_statement.format(sales_record['person'],
                             sales_record['num_items'],
                             sales_record['price'],
                             sales_record['num_items']*sales_record['price']))

import csv
# 设置精度
%precision 2

with open('mpg.csv') as csvfile:
    mpg = list(csv.DictReader(csvfile))

mpg[:3] # The first three dictionaries in our list.

len(mpg)

mpg[0].keys()

sum(float(d['cty']) for d in mpg) / len(mpg)

sum(float(d['hwy']) for d in mpg) / len(mpg)

cylinders = set(d['cyl'] for d in mpg)
cylinders

CtyMpgByCyl = []

for c in cylinders: # iterate over all the cylinder levels
    summpg = 0
    cyltypecount = 0
    for d in mpg: # iterate over all dictionaries
        if d['cyl'] == c: # if the cylinder level type matches,
            summpg += float(d['cty']) # add the cty mpg
            cyltypecount += 1 # increment the count
    CtyMpgByCyl.append((c, summpg / cyltypecount)) # append the tuple ('cylinder', 'avg mpg')

CtyMpgByCyl.sort(key=lambda x: x[0])
CtyMpgByCyl

vehicleclass = set(d['class'] for d in mpg) # what are the class types
vehicleclass

HwyMpgByClass = []

for t in vehicleclass: # iterate over all the vehicle classes
    summpg = 0
    vclasscount = 0
    for d in mpg: # iterate over all dictionaries
        if d['class'] == t: # if the cylinder amount type matches,
            summpg += float(d['hwy']) # add the hwy mpg
            vclasscount += 1 # increment the count
    HwyMpgByClass.append((t, summpg / vclasscount)) # append the tuple ('class', 'avg mpg')

HwyMpgByClass.sort(key=lambda x: x[1])
HwyMpgByClass

import datetime as dt
import time as tm

tm.time()

dtnow = dt.datetime.fromtimestamp(tm.time())
dtnow

dtnow.year, dtnow.month, dtnow.day, dtnow.hour, dtnow.minute, dtnow.second # get year, month, day, etc.from a datetime

delta = dt.timedelta(days = 100) # create a timedelta of 100 days
delta

today = dt.date.today()

today - delta # the date 100 days ago

today > today-delta # compare dates

class Person:
    department = 'School of Information' #a class variable
# you must use self to define in function
    def set_name(self, new_name): #a method
        self.name = new_name
    def set_location(self, new_location):
        self.location = new_location

person = Person()
person.set_name('Christopher Brooks')
person.set_location('Ann Arbor, MI, USA')
print('{} live in {} and works in the department {}'.format(person.name, person.location, person.department))

store1 = [10.00, 11.00, 12.34, 2.34]
store2 = [9.00, 11.10, 12.34, 2.01]
cheapest = map(min, store1, store2)
cheapest

for item in cheapest:
    print(item)

people = ['Dr. Christopher Brooks', 'Dr. Kevyn Collins-Thompson', 'Dr. VG Vinod Vydiswaran', 'Dr. Daniel Romero']

def split_title_and_name(person):
    title = person.split()[0]
    lastname = person.split()[-1]
    return '{} {}'.format(title, lastname)

list(map(split_title_and_name, people))

my_function = lambda a, b, c : a + b

my_function(1, 2, 3)

my_list = []
for number in range(0, 1000):
    if number % 2 == 0:
        my_list.append(number)
my_list

my_list = [number for number in range(0,1000) if number % 2 == 0]
my_list

people = ['Dr. Christopher Brooks', 'Dr. Kevyn Collins-Thompson', 'Dr. VG Vinod Vydiswaran', 'Dr. Daniel Romero']

def split_title_and_name(person):
    return person.split()[0] + ' ' + person.split()[-1]

#option 1
for person in people:
    print(split_title_and_name(person) == (lambda x: x.split()[0] + ' ' + x.split()[-1])(person))

#option 2
list(map(split_title_and_name, people)) == list(map(lambda person: person.split()[0] + ' ' + person.split()[-1], people))

def times_tables():
    lst = []
    for i in range(10):
        for j in range (10):
            lst.append(i*j)
    return lst

times_tables() == [i*j for i in range(10) for j in range(10)]

lowercase = 'abcdefghijklmnopqrstuvwxyz'
digits = '0123456789'

correct_answer = [a+b+c+d for a in lowercase for b in lowercase for c in digits for d in digits]

correct_answer[:50] # Display first 50 ids

import numpy as np

mylist = [1, 2, 3]
x = np.array(mylist)
x

y = np.array([4, 5, 6])
y

m = np.array([[7, 8, 9], [10, 11, 12]])
m

m.shape

n = np.arange(0, 30, 2) # start at 0 count up by 2, stop before 30
n

n = n.reshape(3, 5) # reshape array to be 3x5
n

o = np.linspace(0, 4, 9) # return 9 evenly spaced values from 0 to 4
o

o.resize(3, 3)
o

np.ones((3, 2))

np.zeros((2, 3))

np.eye(3)

np.diag(y)

np.array([1, 2, 3] * 3)

np.repeat([1, 2, 3], 3)

p = np.ones([2, 3], int)
p

np.vstack([p, 2*p])

np.hstack([p, 2*p])

print(x + y) # elementwise addition     [1 2 3] + [4 5 6] = [5  7  9]
print(x - y) # elementwise subtraction  [1 2 3] - [4 5 6] = [-3 -3 -3]

print(x * y) # elementwise multiplication  [1 2 3] * [4 5 6] = [4  10  18]
print(x / y) # elementwise divison         [1 2 3] / [4 5 6] = [0.25  0.4  0.5]

print(x**2) # elementwise power  [1 2 3] ^2 =  [1 4 9]

x.dot(y) # dot product  1*4 + 2*5 + 3*6

z = np.array([y, y**2])
print(len(z)) # number of rows of array

z = np.array([y, y**2])
z

z.shape

z.T

z.T.shape

z.dtype

z = z.astype('f')
z.dtype

a = np.array([-4, -2, 1, 3, 5])

a.sum()

a.max()

a.min()

a.mean()

a.std()

a.argmax()

a.argmin()

s = np.arange(13)**2
s

s[0], s[4], s[-1]

s[1:5]

s[-4:]

s[-5::-2]

r = np.arange(36)
r.resize((6, 6))
r

r[2, 2]

r[3, 3:6]

r[:2, :-1]

r[-1, ::2]

r[r > 30]

r[r > 30] = 30
r

r2 = r[:3,:3]
r2

r2[:] = 0
r2

r

r_copy = r.copy()
r_copy

r_copy[:] = 10
print(r_copy, '\n')
print(r)

test = np.random.randint(0, 10, (4,3))
test

for row in test:
    print(row)

for i in range(len(test)):
    print(test[i])

for i, row in enumerate(test):
    print('row', i, 'is', row)

test2 = test**2
test2

for i, j in zip(test, test2):
    print(i,'+',j,'=',i+j)

Coursera/Introduction to Data Science in Python/Week 1-Python Fundamentals/Week+1.py


"""
@file
@brief Downloads stock prices (from Yahoo website) and other prices.
"""
import os
import urllib.request
import urllib.error
import datetime
import pandas
import numpy
from pyquickhelper.filehelper import is_file_string

class StockPricesException(Exception):
    """
    Raised by StockPrices classes.
    """
    pass

class StockPricesHTTPException(StockPricesException):
    """
    Raised by StockPrices classes.
    """
    pass

class StockPrices:

    """
    Defines a class containing stock prices, provides basic functions,
    the class uses :epkg:`pandas` to load the data.

    .. exref::
        :title: Retrieve stock prices from the Yahoo source

        ::

            from pyensae.finance import StockPrices
            prices = StockPrices(tick="NASDAQ:MSFT")
            print(prices.dataframe.head())

    The class loads a stock price from either a url or a folder
    where the data was cached. If a filename
    ``<folder>/<tick>.<day1>.<day2>.txt`` already exists,
    it takes it from here. Otherwise, it downloads it.

    A couple of providers have been implemented but it is not
    easy to keep them up to date as policies from website
    change on a regular basis.
    If *url* is ``'yahoo'``, the data will be download using
    `CAC 40 <http://finance.yahoo.com/q/cp?s=^FCHI+Components>`_.
    The CAC40 composition is described by
    `Wikipedia CAC 40 <http://fr.wikipedia.org/wiki/CAC_40>`_.
    However `Yahoo Finance <https://fr.finance.yahoo.com/>`_
    introduced the use of cookies in May 2017
    and it is not so easy to automate.
    The default provider could be
    *Google Finance* which has now been integrated into the
    search engine.
    Tick names depends on the data prodiver. More details:
    `European Markets Information <https://www.stockmarketeye.com/users-guide/ticker-symbols-and-data-providers/euro-stocks.html>`_.
    You can also go to `quandl <https://www.quandl.com/data/EURONEXT/BNP-Bnp-Paribas-Act-A-BNP>`_
    and get the tick for the module `quandl <https://www.quandl.com/tools/python>`_.
    As of May 14th, the following error appears when using
    ``url='yahoo'`` which comes from an error in
    :epkg:`pandas_reader`::

        ImmediateDeprecationError(DEP_ERROR_MSG.format('Yahoo Daily'))
        pandas_datareader.exceptions.ImmediateDeprecationError:
        Yahoo Daily has been immediately deprecated due to large breaks in the API without the
        introduction of a stable replacement. Pull Requests to re-enable these data
        connectors are welcome.

        See https://github.com/pydata/pandas-datareader/issues

    ``url='yahoo_new'`` should solve the issue.
    It relies on :epkg:`yahoo_historial`.
    Data can be downloaded for a specific period of time.
    If not specified, it takes the largest available.

    .. exref::
        :title: Compute the average returns and correlation matrix

        ::

            import pyensae, pandas
            from pyensae.finance import StockPrices
            from pyensae.datasource import download_data

            # download the CAC 40 composition from my website (for Yahoo)
            download_data('cac40_2013_11_11.txt', website='xd')

            # download all the prices (if not already done) and store them into files
            actions = pandas.read_csv("cac40_2013_11_11.txt", sep="\\t")

            # we remove stocks with not enough historical data
            stocks = { k:StockPrices(tick = k) for k,v in actions.values }
            dates = StockPrices.available_dates(stocks.values())
            stocks = {k:v for k,v in stocks.items() if len(v.missing(dates)) <= 10}
            print("nb left", len(stocks))

            # we remove dates with missing prices
            dates = StockPrices.available_dates(stocks.values())
            ok = dates[dates["missing"] == 0]
            print("all dates before", len(dates), " after:" , len(ok))
            for k in stocks:
                stocks[k] = stocks[k].keep_dates(ok)

            # we compute correlation matrix and returns
            ret, cor = StockPrices.covariance(stocks.values(), cov = False, ret = True)

    You should also look at
    `pyensae et notebook <http://www.xavierdupre.fr/blog/notebooks/example%20pyensae.html>`_.
    If you use `Google Finance <https://www.google.com/finance>`_
    as a provider, the tick name is usually
    prefixed by the market places (NASDAQ for example). The export
    does not work for all markets places.
    Another provider was added, ``yahoo_new`` which delegates the task
    of getting data from `Yahoo Finance <https://finance.yahoo.com/>`_ to module
    `yahoo-historical <https://github.com/AndrewRPorter/yahoo-historical>`_.
    """

    def __init__(self, tick, url="google", folder="cache",
                 begin=None, end=None, sep=",",
                 intern=False, use_dtime=False):
        """
        @param      tick        tick name, ex ``NASDAQ:MSFT``
        @param      url         if yahoo, downloads the data from there if it was not done before
                                url is possible, ``'google'``, ``'yahoo_new'``,
                                ``'quandl'`` are predefined values
        @param      folder      cache folder (created if it does not exists
        @param      begin       first day (datetime), see below
        @param      end         last day (datetime), see below
        @param      sep         column separator
        @param      intern      do not use unless you know what to do
                                (see :meth:`__getitem__ <pyensae.finance.astock.StockPrices.__getitem__>`)
        @param      use_dtime   if True, use DateTime instead of string
        """
        if isinstance(url, pandas.DataFrame):
            self.datadf = url
            self.tickname = tick
            if "Date" not in url.columns:
                raise StockPricesHTTPException(
                    "the dataframe does not contain any column 'Date': {0}".format(
                        ",".join(
                            _ for _ in url.columns)))
        elif isinstance(tick, str) and is_file_string(tick) and os.path.exists(tick):
            with open(tick, "r") as f:
                for line in f.readlines():
                    if line.startswith('<!DOCTYPE html PUBLIC'):
                        raise StockPricesHTTPException(
                            "pandas cannot parse the file, check your have access to internet: " + str(tick))
                    break
            try:
                self.datadf = pandas.read_csv(tick, sep=sep)
            except Exception as e:
                with open(tick, "r") as t:
                    content = t.read()
                if "Firewall Authentication" in content:
                    raise StockPricesException(
                        "pandas cannot parse the file, check your have access to internet: " + str(tick)) from e
                else:
                    raise
        else:
            if not os.path.exists(folder):
                try:
                    os.mkdir(folder)
                except PermissionError as e:
                    raise StockPricesException(("PermissionError, unable to create directory '{0}', " +
                                                "check you execute the program in a folder you have " +
                                                "permission to modify ({1})").format(folder, os.getcwd())) from e
            self.tickname = tick

            if begin is None:
                begin = datetime.datetime(2000, 1, 3)
            if end is None:
                now = datetime.datetime.now()
                end = now - datetime.timedelta(1)

            sbeg = begin.strftime("%Y-%m-%d")
            send = end.strftime("%Y-%m-%d")
            name = os.path.join(
                folder,
                tick.replace(":", "_").replace("/", "_").replace("\\\\", "_") +
                ".{0}.{1}.txt".format(
                    sbeg,
                    send))

            date_format = None
            if not os.path.exists(name):
                if url == "google":
                    use_url = True
                    url_string = "https://finance.google.com/finance/historical?q={0}".format(
                        self.tickname)
                    url_string += "&startdate={0}&enddate={1}&output=csv".format(
                        begin.strftime('%b %d, %Y'), end.strftime('%b %d, %Y'))
                    url = url_string.replace(" ", "+").replace(",", "%2C")
                    date_format = "%b-%d-%Y"
                elif url == "quandl":
                    import quandl
                    df = quandl.get(
                        "EURONEXT/BNP", start_date=begin.strftime('%Y-%m-%d'), end_date=end.strftime('%Y-%m-%d'))
                    df.reset_index(drop=False).to_csv(
                        name, sep=sep, index=False)
                    use_url = False
                elif url == 'yahoo_new':
                    from yahoo_historical import Fetcher
                    data = Fetcher(tick, [begin.year, begin.month, begin.day],
                                   [end.year, end.month, end.day])
                    df = data.getHistorical()
                    df.to_csv(name, sep=sep, index=False)
                    use_url = False
                elif url in ("yahoo", "google", "fred", "famafrench"):
                    import pandas_datareader.data as web
                    df = web.DataReader(self.tickname, url,
                                        begin, end).reset_index(drop=False)
                    df.to_csv(name, sep=sep, index=False)
                    use_url = False
                else:
                    raise StockPricesHTTPException(
                        "Unable to download data '{0}' from the following website '{1}'".format(tick, url))

                if use_url:
                    self.url_ = url
                    try:
                        u = urllib.request.urlopen(url)
                        text = u.read()
                        u.close()
                    except urllib.error.HTTPError as e:
                        raise StockPricesHTTPException(
                            "HTTPError, unable to load tick '{0}'\nURL: {1}".format(tick, url)) from e

                    if len(text) < 10:
                        raise StockPricesHTTPException(
                            "nothing to download for '{0}' less than 10 downloaded bytes".format(tick))

                    try:
                        f = open(name, "wb")
                        f.write(text)
                        f.close()
                    except PermissionError as e:
                        raise StockPricesException(("PermissionError, unable to create directory '{0}', " +
                                                    "check you execute the program in a folder you have " +
                                                    "permission to modify ({1})").format(folder, os.getcwd())) from e
                else:
                    self.url_ = name

            try:
                self.datadf = pandas.read_csv(name, sep=sep)
            except Exception as e:
                with open(tick, "r") as t:
                    content = t.read()
                if "Firewall Authentication" in content:
                    raise StockPricesException(
                        "pandas cannot parse the file, check your have access to internet '{0}'".format(tick)) from e
                else:
                    raise

            if date_format is not None:
                self.datadf["Date"] = pandas.to_datetime(self.datadf["Date"])
                self.datadf["Date"] = self.datadf["Date"].apply(
                    lambda x: x.strftime('%Y-%m-%d'))
                self.datadf.to_csv(name, sep=sep, index=False)

        if use_dtime:
            self.datadf["Date"] = pandas.to_datetime(self.datadf["Date"])

        if not intern:
            try:
                self.datadf = self.datadf.sort_values("Date")
            except AttributeError:
                self.datadf = self.datadf.sort("Date")
            except KeyError as e:
                raise StockPricesException("schema: {}".format(
                    ",".join(self.datadf.columns))) from e
            self.datadf.reset_index(drop=True, inplace=True)
            self.datadf.set_index("Date", drop=False, inplace=True)

    def __getitem__(self, key):
        """
        Overloads the ``getitem`` operator to get a @see cl StockPrice object.

        @param      key     key
        @return             StockPrice
        """
        return StockPrices(
            self.tick, self.datadf.__getitem__(key), intern=True)

    def __len__(self):
        """
        @return     number of observations
        """
        return len(self.datadf)

    @property
    def shape(self):
        """
        @return     number of observations
        """
        return self.datadf.shape

    @property
    def tick(self):
        """
        Returns the tick name.
        """
        return self.tickname

    @property
    def dataframe(self):
        """
        Returns the dataframe.
        """
        return self.datadf

    def df(self):
        """
        Returns the dataframe.
        """
        return self.datadf

    def FirstDate(self):
        """
        Returns the first date.
        """
        return self.datadf["Date"].min()

    def LastDate(self):
        """
        Returns the first date.
        """
        return self.datadf["Date"].max()

    def missing(self, trading_dates):
        """
        Returnq the list of missing dates from an overset of trading dates.

        @param      trading_dates       trading_dates (DataFrame having the column ``Date`` or in the index)
        @return                         missing dates (or None if issues)
        """
        da = self.dataframe["Date"]
        da2 = {v: 1 for v in da}

        if isinstance(trading_dates, dict):
            se = trading_dates
        else:
            se = trading_dates[
                "Date"] if "Date" in trading_dates.columns else trading_dates.index

        tbl = [{"Date": v} for v in se if v not in da2]
        if len(tbl) > 0:
            df = pandas.DataFrame(tbl)
            try:
                return df.sort_values("Date")
            except AttributeError:
                return df.sort("Date")
        else:
            return None

    @staticmethod
    def available_dates(listStockPrices, missing=True, field="Close"):
        """
        Returns the list of values (Open or High or Low or Close or Volume) from each stock
        for all the available_dates for a list of stock prices.

        A missing date is a date for which there is at least one stock price and one missing stock price.

        if ``missing`` is true a column is added which gives the number of missing stock prices for this dates

        @param      listStockPrices     list of StockPrices
        @param      missing             True or False
        @param      field               which field to use to fill the matrix
        @return                         matrix with the available dates for each stock
        """
        if field == "ohlc":
            field = ["Open", "High", "Low", "Close"]
        dates = []
        if isinstance(field, str):
            for st in listStockPrices:
                lifi = list(st.dataframe.columns)
                index = lifi.index(field)
                for row in st.dataframe.values:
                    date = row[0]
                    dates.append(
                        {"Date": date, "tick": st.tick, field: row[index]})
        elif isinstance(field, (tuple, list)):
            for st in listStockPrices:
                lifi = list(st.dataframe.columns)
                indexes = [lifi.index(f) for f in field]
                for row in st.dataframe.values:
                    date = row[0]
                    r = {"Date": date, "tick": st.tick, }
                    for i, f in zip(indexes, field):
                        r[f] = row[i]
                    dates.append(r)
        else:
            raise TypeError("field must be a string, a tuple or a list")

        df = pandas.DataFrame(dates)
        if isinstance(field, str):
            piv = df.pivot("Date", "tick", field)
        elif isinstance(field, (tuple, list)):
            pivs = [df.pivot("Date", "tick", f) for f in field]
            for fi, piv in zip(field, pivs):
                col = [c + "," + fi for c in piv.columns]
                piv.columns = col
            if len(pivs) == 1:
                piv = pivs[0]
            else:
                piv = pivs[0].merge(pivs[1], how="outer",
                                    left_index=True, right_index=True)
                for p in pivs[2:]:
                    piv = piv.merge(
                        p, how="outer", left_index=True, right_index=True)
        else:
            raise TypeError("field must be a string, a tuple or a list")

        if missing:
            def count_nan(row):
                "count nans"
                n = 0
                for k, v in row.items():
                    if k == "Date":
                        continue
                    if numpy.isnan(v):
                        n += 1
                return n
            piv["missing"] = piv.apply(lambda row: count_nan(row), axis=1)

        try:
            piv = piv.sort_index()
        except AttributeError:
            piv = piv.sort()
        return piv

    def head(self):
        """
        usual
        """
        return self.dataframe.head()

    def tail(self):
        """
        usual
        """
        return self.dataframe.tail()

    def keep_dates(self, trading_dates):
        """
        removes undesired dates

        @param      trading_dates   dates
        @return                     new series
        """
        da = self.dataframe["Date"]
        da2 = {v: 1 for v in da}

        if isinstance(trading_dates, dict):
            se = trading_dates
        else:
            se = trading_dates[
                "Date"] if "Date" in trading_dates.columns else trading_dates.index

        tbl = {v: 1 for v in se if v in da2}
        if len(tbl) > 0:
            ave = self.dataframe.apply(lambda row: row["Date"] in tbl, axis=1)
            return StockPrices(self.tickname, self.dataframe.loc[ave, :])
        else:
            raise StockPricesException("no trading dates left")

    def returns(self):
        """
        Builds the series of returns.

        @param      col     column to use to compute the returns
        @return             StockPrices
        """
        df = self.dataframe
        fd = self.FirstDate()
        ld = self.LastDate()

        plus = df["Date"] > fd    # dates from FirstDate+1 to LastDate
        moins = df["Date"] < ld    # dates from FirstDate to LastDate-1

        res = df.loc[plus, ["Date", "Volume"]]

        for k in df.columns:
            if k in ["Date", "Volume"]:
                continue
            m = numpy.array(df.loc[moins, k])
            p = numpy.array(df.loc[plus, k])
            res[k] = (p - m) / m

        return StockPrices(self.tickname, res)

    @staticmethod
    def covariance(
            listStockPrices, missing=True, field="Close", cov=True, ret=False):
        """
        Computes the covariances matrix (of returns).

        @param      listStockPrices     list of StockPrices
        @param      field               which field to use to fill the matrix
        @param      cov                 if True, returns the covariance, otherwise, the correlations
        @param      ret                 if True, also add the returns
        @return                         square dataframe or 2 dataframe (returns, correlation)
        """
        listStockPrices = [v.returns() for v in listStockPrices]
        mat = StockPrices.available_dates(listStockPrices, False, field)

        npmat = numpy.matrix(mat)
        cov = numpy.cov(
            npmat.transpose()) if cov else numpy.corrcoef(
            npmat.transpose())
        names = [v.tick for v in listStockPrices]
        ret_mat = pandas.DataFrame(cov, columns=names, index=names)

        if ret:
            rows = [{"tick": v.tick, "return": v.dataframe[field].mean()}
                    for v in listStockPrices]
            ret = pandas.DataFrame(rows)
            ret.set_index("tick", drop=True, inplace=True)
            return ret, ret_mat
        else:
            return ret_mat

    def plot(self, begin=None, end=None,
             field="Close", date_format=None,
             existing=None, axis=1, ax=None,
             **args):
        """
        See :meth:`draw <pyensae.finance.astock.StockPrices.draw>`.
        """
        return StockPrices.draw(self, begin=begin, end=end,
                                field=field, date_format=date_format,
                                existing=existing, axis=axis, ax=ax, **args)

    @staticmethod
    def draw(listStockPrices, begin=None, end=None,
             field="Close", date_format=None,
             existing=None, axis=1, ax=None,
             **args):
        """
        Draws a graph showing one or several time series.
        The example was taken
        `date_demo.py <https://matplotlib.org/examples/api/date_demo.html>`_.

        @param      listStockPrices     list of @see cl StockPrices (or one @see cl StockPrices if it is the only one)
        @param      begin               first date (datetime) or None to take the first one
        @param      end                 last included date (datetime) or None to take the last one
        @param      field               Open, High, Low, Close, Adj Close, Volume
        @param      date_format         ``%Y`` or ``%Y-%m`` or ``%Y-%m-%d`` or None if you prefer the function to choose
        @param      args                other arguments to send to ``plt.subplots``
        @param      axis                1 or 2, it only works if existing is not None.
                                        If axis is 2, the function draws the curves on the second axis.
        @param      args                other parameters to give method ``plt.subplots``
        @param      ax                  use existing `axes <http://matplotlib.org/api/axes_api.html>`_
        @return                         `axes <http://matplotlib.org/api/axes_api.html>`_

        The parameter ``figsize`` of the method
        `subplots <https://matplotlib.org/api/pyplot_api.html?highlight=subplots#matplotlib.pyplot.subplots>`_
        can change the graph size (see the example below).

        .. exref::
            :title: graph of a financial series

            ::

                from pyensae.finance import StockPrices
                stocks = [ StockPrices("NASDAQ:MSFT", folder = cache),
                           StockPrices("NASDAQ:GOOGL", folder = cache),
                           StockPrices("NASDAQ:AAPL", folder = cache)]
                fig, ax, plt = StockPrices.draw(stocks)
                fig.savefig("image.png")
                fig, ax, plt = StockPrices.draw(stocks, begin="2010-01-01", figsize=(16,8))
                plt.show()

            You can also chain the graphs and add a series on a second graph:

            ::

                from pyensae.finance import StockPrices
                stock = StockPrices("NASDAQ:MSFT", folder = cache)
                stock2 = StockPrices "NASDAQ:GOOGL", folder = cache)
                fig, ax, plt = stock.plot(figsize=(16,8))
                fig, ax, plt = stock2.plot(existing=(fig,ax), axis=2)
                plt.show()

        .. versionchanged:: 1.1
            Parameter *existing* was removed and parameter *ax* was added.
            If the date overlaps, the method
            `autofmt_xdate <https://matplotlib.org/api/figure_api.html#matplotlib.figure.Figure.autofmt_xdate>`_
            should be called.
        """
        if isinstance(listStockPrices, StockPrices):
            listStockPrices = [listStockPrices]

        data = StockPrices.available_dates(
            listStockPrices, missing=False, field=field)
        if begin is None:
            if end is not None:
                data = data[data.index <= end]
        else:
            if end is not None:
                data = data[(data.index >= begin) & (data.index <= end)]
            else:
                data = data[data.index >= begin]

        dates = [datetime.datetime.strptime(_, '%Y-%m-%d') for _ in data.index]
        begin = dates[0]
        end = dates[-1]

        def price(x):
            "local formatting"
            return '%1.2f' % x

        import matplotlib.pyplot as plt
        import matplotlib.dates as mdates

        if ax is not None:
            ex_h, ex_l = ax.get_legend_handles_labels()
            ex_l = tuple(ex_l)
            ex_h = tuple(ex_h)
            if axis == 2:
                ax = ax.twinx()
            fig = None
        else:
            if 'label' in args:
                args_ = {k: v for k, v in args.items() if k not in ('label', )}
            else:
                args_ = args
            fig, ax = plt.subplots(**args_)
            ex_h, ex_l = tuple(), tuple()

        curve = []
        if field == "ohlc":
            try:
                # since matplotlib 2.2.0
                # see https://github.com/matplotlib/mpl_finance
                from mpl_finance import candlestick_ohlc
            except ImportError:
                from matplotlib.finance import candlestick_ohlc
            ohlc = list(list(data.iloc[i, :4])
                        for i in range(0, data.shape[0]))
            ohlc = [[mdates.date2num(t)] + v for t, v in zip(dates, ohlc)]
            candlestick_ohlc(ax, ohlc, colorup="g")
        else:
            for stock in data.columns:
                if axis == 2:
                    curve.append(
                        ax.plot(dates, data[stock], "r", linestyle='solid', label=str(stock)))
                else:
                    curve.append(
                        ax.plot(dates, data[stock], linestyle='solid', label=str(stock)))

        if existing is None:
            ax.format_xdata = mdates.DateFormatter('%Y-%m-%d')
            if len(dates) < 30:
                days = mdates.DayLocator()
                ax.xaxis.set_major_locator(days)
                ax.xaxis.set_minor_locator(days)
                if date_format is not None:
                    fmt = mdates.DateFormatter(date_format)
                    ax.xaxis.set_major_formatter(fmt)
                else:
                    ax.xaxis.set_major_formatter(
                        mdates.DateFormatter("%Y-%m-%d"))
            elif len(dates) < 500:
                months = mdates.MonthLocator()
                days = mdates.DayLocator()
                ax.xaxis.set_major_locator(months)
                ax.xaxis.set_minor_locator(days)
                ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y-%m"))
                if date_format is not None:
                    fmt = mdates.DateFormatter(date_format)
                    ax.xaxis.set_major_formatter(fmt)
                else:
                    ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y-%m"))
            else:
                years = mdates.YearLocator()
                months = mdates.MonthLocator()
                ax.xaxis.set_major_locator(years)
                ax.xaxis.set_minor_locator(months)
                if date_format is not None:
                    fmt = mdates.DateFormatter(date_format)
                    ax.xaxis.set_major_formatter(fmt)
                else:
                    ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))

        ax.set_xlim(begin, end)
        ax.format_ydata = price
        if fig is not None:
            fig.autofmt_xdate()

        if axis == 2:
            if isinstance(curve, list):
                curve = [_[0] for _ in curve]
            ax.legend(ex_h + tuple(curve), ex_l + tuple(data.columns))
        else:
            ax.grid(True)
            ax.legend(ex_l + tuple(data.columns))

        return ax

    def to_csv(self, filename, sep="\t", index=False, **params):
        """
        Saves the file in text format,
        see `to_csv <https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html>`_

        @param      filename        filename
        @param      sep             separator
        @param      index           to keep or drop the index
        @param      params          other parameters
        """
        self.dataframe.to_csv(filename, sep=sep, index=index, **params)

    def to_excel(self, excel_writer, **params):
        """
        Saves the file in Excel format,
        see `to_excel <https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_excel.html>`_
        """
        self.dataframe.to_excel(excel_writer, **params)