# -*- coding: utf-8 -*-
r"""
Standalone LaTeX Document class and TikzPicture

This module contains two Python classes. Firstly, it contains a class
:class:`Standalone` to represent a LaTeX file using the standalone__
document class.

__ http://www.ctan.org/pkg/standalone

From its documentation:

    *The standalone bundle allows users to easily place picture environments
    or other material in own source files and compile these on their own or as
    part of a main document. A special standalone class is provided for use
    with such files, which by default crops the resulting output file to the
    content. The standalone package enables the user to simply load the
    standalone files using ``\input`` inside a main document.*

Secondly, it contains a class :class:`TikzPicture` which inherits from
:class:`Standalone` that represents a LaTeX file using the standalone
document class and containing a tikzpicture.

A Python Module for PGF/Tikz pictures. A TikzPicture object is created from
a string starting with ``r'\begin{tikzpicture}'`` and ending with
``r'\end{tikzpicture}'``.

The module allows to convert a standalone LaTeX document class file,
including tikzpictures, to an image. It allows conversion to pdf, png and
svg formats. It also show them automatically in Jupyter using rich
representation.

According to wikipedia, `PGF/TikZ`__ is a pair of languages for producing
vector graphics (e.g., technical illustrations and drawings) from a
geometric/algebraic description, with standard features including the
drawing of points, lines, arrows, paths, circles, ellipses and polygons.

__ https://www.ctan.org/pkg/pgf

EXAMPLES:

Standalone LaTeX document class
-------------------------------

First *Hello World* example::

    sage: from sage.misc.latex_standalone import Standalone
    sage: Standalone('Hello World')
    \documentclass{standalone}
    \begin{document}
    Hello World
    \end{document}

Loading a few latex packages::

    sage: Standalone('Hello World', usepackage=['amsmath', 'amsfont'])
    \documentclass{standalone}
    \usepackage{amsmath}
    \usepackage{amsfont}
    \begin{document}
    Hello World
    \end{document}

Setting few standalone options (see documentation of standalone for a
complete list)::

    sage: Standalone('Hello World', standalone_config=["border=4mm", "beamer=true"])
    \documentclass{standalone}
    \standaloneconfig{border=4mm}
    \standaloneconfig{beamer=true}
    \begin{document}
    Hello World
    \end{document}

Adding your own list of macros::

    sage: Standalone('Hello World', macros=[r'\newcommand{\ZZ}{\mathbb{Z}}'])
    \documentclass{standalone}
    \newcommand{\ZZ}{\mathbb{Z}}
    \begin{document}
    Hello World
    \end{document}

It provides conversion to images of different format::

    sage: from sage.misc.latex_standalone import Standalone
    sage: s = Standalone('Hello World')
    sage: _ = s.pdf()    # not tested
    sage: _ = s.png()    # not tested
    sage: _ = s.svg()    # not tested
    sage: s              # not tested, in Jupyter, this shows the image directly below the cell

TikzPicture
-----------

This module also contains a class :class:`TikzPicture` which inherits from
:class:`Standalone` to represent more specifically a tikzpicture which is
within a standalone document class.

First construct a string describing a tikzpicture::

    sage: lines = []
    sage: lines.append(r'\begin{tikzpicture}')
    sage: lines.append(r'\draw[very thick,orange,->] (0,0) -- (1,1);')
    sage: lines.append(r'\end{tikzpicture}')
    sage: s = '\n'.join(lines)
    sage: print(s)
    \begin{tikzpicture}
    \draw[very thick,orange,->] (0,0) -- (1,1);
    \end{tikzpicture}

One may provide it as input to ``TikzPicture``::

    sage: from sage.misc.latex_standalone import TikzPicture
    sage: t = TikzPicture(s)

In the terminal, the following shows the content of the standalone
document class tex file which contains the tikzpicture. In Jupyter, it
shows the picture itself::

    sage: t
    \documentclass[tikz]{standalone}
    \begin{document}
    \begin{tikzpicture}
    \draw[very thick,orange,->] (0,0) -- (1,1);
    \end{tikzpicture}
    \end{document}

As it is the case for :class:`Standalone`, the constructor of
``TikzPicture`` has many arguments allowing for example to add some more
``\usepackage`` lines::

    sage: t = TikzPicture(s, usepackage=['amsmath'])
    sage: t
    \documentclass[tikz]{standalone}
    \usepackage{amsmath}
    \begin{document}
    \begin{tikzpicture}
    \draw[very thick,orange,->] (0,0) -- (1,1);
    \end{tikzpicture}
    \end{document}

Moreover, it allows to load some tikz libraries::

    sage: t = TikzPicture(s, usetikzlibrary=['arrows'])
    sage: t
    \documentclass[tikz]{standalone}
    \usetikzlibrary{arrows}
    \begin{document}
    \begin{tikzpicture}
    \draw[very thick,orange,->] (0,0) -- (1,1);
    \end{tikzpicture}
    \end{document}

The following example illustrates that it works when providing the
tikzpicture code generated by Sage from some polyhedron::

    sage: from sage.misc.latex_standalone import TikzPicture
    sage: V = [[1,0,1],[1,0,0],[1,1,0],[0,0,-1],[0,1,0],[-1,0,0],[0,1,1],[0,0,1],[0,-1,0]]
    sage: P = Polyhedron(vertices=V).polar()
    sage: s = P.projection().tikz([674,108,-731],112)
    sage: t = TikzPicture(s)

Open the image in a viewer (the returned value is a string giving the
absolute path to the file in some temporary directory)::

    sage: path_to_file = t.pdf()                # not tested

Instead, you may save a pdf of the tikzpicture into a file of your choice
(but this does not open the viewer)::

    sage: _ = t.pdf('tikz_polytope.pdf')        # not tested

Opening the image in a viewer can be turned off::

    sage: _ = t.pdf(view=False)      # long time (2s) # optional latex

The same can be done with png format (translated from pdf with convert
command which needs the installation of imagemagick)::

    sage: _ = t.png(view=False)      # long time (2s) # optional latex imagemagick

The string representation gives the header (5 lines) and tail (5 lines) of
the tikzpicture. In Jupyter, it will instead use rich representation and
show the image directly below the cell in png or svg format::

    sage: t
    \documentclass[tikz]{standalone}
    \begin{document}
    \begin{tikzpicture}%
            [x={(0.249656cm, -0.577639cm)},
            y={(0.777700cm, -0.358578cm)},
            z={(-0.576936cm, -0.733318cm)},
    ...
    \node[vertex] at (0.00000, -1.00000, 0.00000)     {};
    \node[vertex] at (-0.50000, -0.50000, -0.50000)     {};
    %%
    %%
    \end{tikzpicture}
    \end{document}

Use ``print(t)`` to see the complete content of the file::

    sage: print(t)               # not tested

Adding a border in the options avoids cropping the vertices of a graph::

    sage: g = graphs.PetersenGraph()       # optional - sage.graphs
    sage: s = latex(g)   # takes 3s but the result is cached # optional latex sage.graphs
    sage: t = TikzPicture(s, standalone_config=["border=4mm"], usepackage=['tkz-graph']) # optional latex sage.graphs
    sage: _ = t.pdf()    # not tested

The current latex representation of a transducer is a tikzpicture using
the tikz library automata. The string can be used as input::

    sage: s = latex(transducers.GrayCode())                # optional sage.combinat
    sage: t = TikzPicture(s, usetikzlibrary=['automata'])  # optional sage.combinat
    sage: _ = t.pdf(view=False)           # long time (2s) # optional sage.combinat latex

AUTHORS:

- Sébastien Labbé, initial version in slabbe-0.2.spkg, nov 2015.
- Sébastien Labbé, inclusion into SageMath from slabbe-0.6.2, July 2021.
"""

# ****************************************************************************
#       Copyright (C) 2015-2022 Sébastien Labbé <slabqc@gmail.com>
#
#  Distributed under the terms of the GNU General Public License (GPL)
#  as published by the Free Software Foundation; either version 2 of
#  the License, or (at your option) any later version.
#                  https://www.gnu.org/licenses/
# ****************************************************************************
from subprocess import run
import os

from sage.structure.sage_object import SageObject
from sage.misc.superseded import experimental

class Standalone(SageObject):
    r"""
    LaTeX standalone document class.

    INPUT:

    - ``content`` -- string, the content to be added in the document
      between lines ``r'\begin{document}'`` and ``r'\end{document}'``
    - ``document_class_options`` -- list of strings (default: ``[]``),
      latex document class standalone options. Such options appear on the
      line ``\documentclass[...]{standalone}`` between the brackets.
    - ``standalone_config`` -- list of strings (default: ``[]``),
      standalone configuration options. Such options are defined with
      ``\standaloneconfig{...}``
    - ``usepackage`` -- list of strings (default: ``[]``), latex packages.
    - ``macros`` -- list of strings (default: ``[]``), stuff you need for the picture.
    - ``use_sage_preamble`` -- bool (default: ``False``), whether to include sage
      latex preamble and sage latex macros, that is, the content of
      :func:`sage.misc.latex.extra_preamble()`,
      :func:`sage.misc.latex.extra_macros()` and
      :func:`sage.misc.latex_macros.sage_latex_macros()`.

    EXAMPLES::

        sage: from sage.misc.latex_standalone import Standalone
        sage: content = "\\section{Intro}\nTest\n"
        sage: t = Standalone(content)
        sage: t
        \documentclass{standalone}
        \begin{document}
        \section{Intro}
        Test
        \end{document}

    ::

        sage: t = Standalone(content, standalone_config=["border=4mm"], usepackage=['amsmath'])
        sage: t
        \documentclass{standalone}
        \standaloneconfig{border=4mm}
        \usepackage{amsmath}
        \begin{document}
        \section{Intro}
        Test
        \end{document}

    """
    def __init__(self, content, document_class_options=None,
            standalone_config=None, usepackage=None, macros=None,
            use_sage_preamble=False):
        r"""
        See :class:`Standalone` for full information.

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: content = "\\section{Intro}\n\nTest\n"
            sage: t = Standalone(content)
        """
        self._content = content
        self._document_class_options = [] if document_class_options is None else list(document_class_options)
        self._standalone_config = [] if standalone_config is None else standalone_config
        self._usepackage = [] if usepackage is None else usepackage
        self._macros = [] if macros is None else macros
        if use_sage_preamble:
            from sage.misc.latex import _Latex_prefs
            for key in ['preamble', 'macros']:
                s = _Latex_prefs._option[key]
                if s:
                    self._macros.append(s)
            from sage.misc.latex_macros import sage_latex_macros
            self._macros.extend(sage_latex_macros())

    def _latex_file_header_lines(self):
        r"""
        EXAMPLES::

            sage: latex.extra_preamble('')
            sage: from sage.misc.latex_standalone import Standalone
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: A = ['tikz']
            sage: B = ["border=4mm"]
            sage: C = ['amsmath']
            sage: t = Standalone(s, document_class_options=A, standalone_config=B, usepackage=C)
            sage: t._latex_file_header_lines()[:6]
            ['\\documentclass[tikz]{standalone}',
             '\\standaloneconfig{border=4mm}',
             '\\usepackage{amsmath}']
        """
        lines = []
        if self._document_class_options:
            options = ','.join(self._document_class_options)
            lines.append(r"\documentclass[{}]{{standalone}}".format(options))
        else:
            lines.append(r"\documentclass{standalone}")
        for config in self._standalone_config:
            lines.append(r"\standaloneconfig{{{}}}".format(config))
        for package in self._usepackage:
            lines.append(r"\usepackage{{{}}}".format(package))
        lines.extend(self._macros)
        return lines

    def _repr_(self):
        r"""
        Return a string representation of the Standalone file.

        It contains the first few and last few lines of the content.

        NOTE::

            Use ``print(t)`` or ``str(t)`` to show or get the full content.

        EXAMPLES:

        When the content has 10 lines or less, it shows it all::

            sage: from sage.misc.latex_standalone import Standalone
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = Standalone(s, document_class_options=['tikz'], standalone_config=["border=4mm"], usepackage=['amsmath'])
            sage: t
            \documentclass[tikz]{standalone}
            \standaloneconfig{border=4mm}
            \usepackage{amsmath}
            \begin{document}
            \begin{tikzpicture}
            \draw (0,0) -- (1,1);
            \end{tikzpicture}
            \end{document}

        When the content more than 10 lines, it shows the head (first 5
        lines) and tail (last 5 lines) of the content together with the
        complete header of the standalone latex document. The number of
        missing lines and the number of characters in the content is
        written allowing to detect a change if needed::

            sage: lines = []
            sage: lines.append(r'\begin{tikzpicture}')
            sage: lines.append(r'\draw[->] (-.5,0) -- (20,0);')
            sage: lines.extend(r'\draw({i},-.5) -- ({i},.5);'.format(i=i) for i in range(20))
            sage: lines.append(r'\end{tikzpicture}')
            sage: t = Standalone('\n'.join(lines), document_class_options=['tikz'])
            sage: t
            \documentclass[tikz]{standalone}
            \begin{document}
            \begin{tikzpicture}
            \draw[->] (-.5,0) -- (20,0);
            \draw(0,-.5) -- (0,.5);
            \draw(1,-.5) -- (1,.5);
            \draw(2,-.5) -- (2,.5);
            ---
            13 lines not printed (566 characters in total).
            Use print to see the full content.
            ---
            \draw(16,-.5) -- (16,.5);
            \draw(17,-.5) -- (17,.5);
            \draw(18,-.5) -- (18,.5);
            \draw(19,-.5) -- (19,.5);
            \end{tikzpicture}
            \end{document}

        """
        lines = self._latex_file_header_lines()
        lines.append(r"\begin{document}")
        L = self._content.splitlines()
        if len(L) <= 10:
            lines.extend(L)
        else:
            lines.extend(L[:5])
            lines.append('---')
            lines.append('{} lines not printed ({} characters in total).'.format(len(L) - 10,
                                                           len(self._content)))
            lines.append('Use print to see the full content.')
            lines.append('---')
            lines.extend(L[-5:])
        lines.append(r"\end{document}")
        return '\n'.join(lines)

    def _rich_repr_(self, display_manager, **kwds):
        r"""
        Rich Output Magic Method

        See :mod:`sage.repl.rich_output` for details.

        EXAMPLES::

            sage: from sage.repl.rich_output import get_display_manager
            sage: dm = get_display_manager()
            sage: dm.is_in_terminal()
            False

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: lines = []
            sage: lines.append(r'\begin{tikzpicture}')
            sage: lines.append(r'\draw[very thick,orange,->] (0,0) -- (1,1);')
            sage: lines.append(r'\end{tikzpicture}')
            sage: s = '\n'.join(lines)
            sage: t = TikzPicture(s)
            sage: t._rich_repr_(dm)      # random result is Text in doctest
            OutputImagePng container

        Using vector svg instead of png::

            sage: dm.preferences.graphics = 'vector'
            sage: t._rich_repr_(dm)      # random result is Text in doctest
            OutputImageSvg container
            sage: dm.preferences.graphics = 'raster'
        """
        # Do not use rich output in the terminal
        if display_manager.is_in_terminal():
            return
        # Do not use rich output if not in IPython notebook (Jupyter)
        from sage.repl.rich_output.backend_ipython import BackendIPythonNotebook
        if not isinstance(display_manager._backend, BackendIPythonNotebook):
            return

        types = display_manager.types
        prefer_raster = (
            ('png', types.OutputImagePng),
        )
        prefer_vector = (
            ('svg', types.OutputImageSvg),
            ('pdf', types.OutputImagePdf),
        )
        graphics = display_manager.preferences.graphics
        if graphics == 'disable':
            return
        elif graphics == 'raster' or graphics is None:
            preferred = prefer_raster + prefer_vector
        elif graphics == 'vector':
            preferred = prefer_vector + prefer_raster
        else:
            raise ValueError('unknown graphics output preference')

        for format, output_container in preferred:
            if output_container in display_manager.supported_output():
                filename = getattr(self, format)(view=False, **kwds)
                from sage.repl.rich_output.buffer import OutputBuffer
                buf = OutputBuffer.from_file(filename)
                return output_container(buf)

    def __str__(self):
        r"""
        Return the complete string of the standalone document class file

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = Standalone(s, document_class_options=['tikz'])
            sage: print(t)
            \RequirePackage{luatex85}
            \documentclass[tikz]{standalone}
            \begin{document}
            \begin{tikzpicture}
            \draw (0,0) -- (1,1);
            \end{tikzpicture}
            \end{document}
        """
        lines = []
        # LuaLaTeX, TeXLive 2016, standalone: undefined control sequence
        # https://tex.stackexchange.com/questions/315025
        # fixed in 2018, meanwhile, we add the fix here
        lines.append(r"\RequirePackage{luatex85}")
        lines.extend(self._latex_file_header_lines())
        lines.append(r"\begin{document}")
        lines.append(self._content)
        lines.append(r"\end{document}")
        return '\n'.join(lines)

    def content(self):
        r"""
        Return the content of the standalone document class file

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: t.content()
            'Hello World'

        ::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: print(t.content())
            \begin{tikzpicture}
            \draw (0,0) -- (1,1);
            \end{tikzpicture}
        """
        return self._content

    def add_document_class_option(self, option):
        r"""
        Add a document class option

        INPUT:

        - ``option`` -- string

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: t.add_document_class_option('beamer')
            sage: t
            \documentclass[beamer]{standalone}
            \begin{document}
            Hello World
            \end{document}
        """
        self._document_class_options.append(option)

    def add_standalone_config(self, config):
        r"""
        Add a standalone config

        INPUT:

        - ``config`` -- string

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: t.add_standalone_config("border=4mm")
            sage: t
            \documentclass{standalone}
            \standaloneconfig{border=4mm}
            \begin{document}
            Hello World
            \end{document}

        """
        self._standalone_config.append(config)

    def add_usepackage(self, package):
        r"""
        Add a ``usepackage`` line

        INPUT:

        - ``package`` -- string, name of package

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: t.add_usepackage('amsmath')
            sage: t
            \documentclass{standalone}
            \usepackage{amsmath}
            \begin{document}
            Hello World
            \end{document}

        """
        self._usepackage.append(package)

    def add_macro(self, macro):
        r"""
        Add a macro

        INPUT:

        - ``macro`` -- string, newcommand line

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: t.add_macro(r'\newcommand{\ZZ}{\mathbb{Z}}')
            sage: t
            \documentclass{standalone}
            \newcommand{\ZZ}{\mathbb{Z}}
            \begin{document}
            Hello World
            \end{document}

        """
        self._macros.append(macro)

    def pdf(self, filename=None, view=True, program=None):
        r"""
        Compiles the latex code with pdflatex and create a pdf file.

        INPUT:

        - ``filename`` -- string (default: ``None``), the output filename.
          If ``None``, it saves the file in a temporary directory.

        - ``view`` -- bool (default:``True``), whether to open the file in a
          pdf viewer. This option is ignored and automatically set to
          ``False`` if ``filename`` is not ``None``.

        - ``program`` -- string (default:``None``) ``'pdflatex'`` or
          ``'lualatex'``. If ``None``, it uses ``'lualatex'`` if it is
          available, otherwise ``'pdflatex'``.

        OUTPUT:

            string, path to pdf file

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: _ = t.pdf(view=False)     # long time (1s)   # optional latex

        Same for instances of :class:`TikzPicture`::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: _ = t.pdf(view=False)     # not tested

        A filename may be provided where to save the file, in which case
        the viewer does not open the file::

            sage: from sage.misc.temporary_file import tmp_filename
            sage: filename = tmp_filename('temp','.pdf')
            sage: path_to_file = t.pdf(filename)   # long time (1s)   # optional latex
            sage: path_to_file[-4:]                # long time (fast) # optional latex
            '.pdf'

        The filename may contain spaces::

            sage: filename = tmp_filename('filename with spaces','.pdf')
            sage: path_to_file = t.pdf(filename)   # long time (1s)   # optional latex

        TESTS:

        We test the behavior when a wrong tex string is provided::

            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: s_missing_last_character = s[:-1]
            sage: t = TikzPicture(s_missing_last_character)
            sage: _ = t.pdf()                 # optional latex
            Traceback (most recent call last):
            ...
            CalledProcessError: Command '['...latex', '-interaction=nonstopmode',
            'tikz_...tex']' returned non-zero exit status 1.

        """
        from sage.features.latex import lualatex, pdflatex

        # Set default program
        if program is None:
            if lualatex().is_present():
                program = 'lualatex'
            else:
                program = 'pdflatex'

        # Check availability of programs
        if program == 'pdflatex':
            pdflatex().require()
        elif program == 'lualatex':
            lualatex().require()
        else:
            raise ValueError("program(={}) should be pdflatex or lualatex".format(program))

        # set up filenames
        from sage.misc.temporary_file import tmp_filename
        temp_filename_tex = tmp_filename('tikz_', '.tex')
        with open(temp_filename_tex, 'w') as f:
            f.write(str(self))
        base, temp_filename_tex = os.path.split(temp_filename_tex)
        temp_filename, ext = os.path.splitext(temp_filename_tex)

        # running pdflatex or lualatex
        cmd = [program, '-interaction=nonstopmode', temp_filename_tex]
        result = run(cmd, cwd=base, capture_output=True, text=True)

        # If a problem with the tex source occurs, provide the log
        if result.returncode != 0:
            print("Command \n"
                  "   '{}'\n"
                  "returned non-zero exit status {}.\n"
                  "Here is the content of the stderr:{}\n"
                  "Here is the content of the stdout:"
                  "{}\n".format(' '.join(result.args),
                                result.returncode,
                                result.stderr.strip(),
                                result.stdout.strip()))
        result.check_returncode()
        temp_filename_pdf = os.path.join(base, temp_filename + '.pdf')

        # move the pdf into the good location
        if filename:
            filename = os.path.abspath(filename)
            os.rename(temp_filename_pdf, filename)
            return filename

        # open the tmp pdf
        elif view:
            from sage.misc.viewer import pdf_viewer
            cmd = pdf_viewer().split()
            cmd.append(temp_filename_pdf)
            # we use check_call as opposed to run, because
            # it gives the sage prompt back to the user
            # see https://stackoverflow.com/a/71342967
            # run(cmd, cwd=base, capture_output=True, check=True)
            from subprocess import check_call, PIPE
            check_call(cmd, cwd=base, stdout=PIPE, stderr=PIPE)

        return temp_filename_pdf

    def png(self, filename=None, density=150, view=True):
        r"""
        Compiles the latex code with pdflatex and converts to a png file.

        INPUT:

        - ``filename`` -- string (default:``None``), the output filename.
          If ``None``, it saves the file in a temporary directory.

        - ``density`` -- integer, (default: ``150``), horizontal and vertical
          density of the image

        - ``view`` -- bool (default:``True``), whether to open the file in a
          png viewer. This option is ignored and automatically set to
          ``False`` if ``filename`` is not ``None``.

        OUTPUT:

            string, path to png file

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: _ = t.png(view=False)     # long time (1s)   # optional latex imagemagick

        Same for instances of :class:`TikzPicture`::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: _ = t.png(view=False)     # not tested

        ::

            sage: from sage.misc.temporary_file import tmp_filename
            sage: filename = tmp_filename('temp','.png')
            sage: path_to_file = t.png(filename) # long time (1s)   # optional latex imagemagick
            sage: path_to_file[-4:]              # long time (fast) # optional latex imagemagick
            '.png'

        """
        from sage.features.imagemagick import ImageMagick
        ImageMagick().require()

        temp_filename_pdf = self.pdf(filename=None, view=False)
        temp_filename, ext = os.path.splitext(temp_filename_pdf)
        temp_filename_png = temp_filename + '.png'

        # convert to png
        cmd = ['convert', '-density',
               '{0}x{0}'.format(density), '-trim', temp_filename_pdf,
               temp_filename_png]
        result = run(cmd, capture_output=True, text=True)

        # If a problem occurs, provide the log
        if result.returncode != 0:
            print("Command \n"
                  "   '{}'\n"
                  "returned non-zero exit status {}.\n"
                  "Here is the content of the stderr:{}\n"
                  "Here is the content of the stdout:"
                  "{}\n".format(' '.join(result.args),
                                result.returncode,
                                result.stderr.strip(),
                                result.stdout.strip()))
        result.check_returncode()

        # move the png into the good location
        if filename:
            filename = os.path.abspath(filename)
            os.rename(temp_filename_png, filename)
            return filename

        # open the tmp png
        elif view:
            from sage.misc.viewer import png_viewer
            cmd = png_viewer().split()
            cmd.append(temp_filename_png)
            # we use check_call as opposed to run, because
            # it gives the sage prompt back to the user
            # see https://stackoverflow.com/a/71342967
            # run(cmd, capture_output=True, check=True)
            from subprocess import check_call, PIPE
            check_call(cmd, stdout=PIPE, stderr=PIPE)

        return temp_filename_png

    def svg(self, filename=None, view=True, program='pdftocairo'):
        r"""
        Compiles the latex code with pdflatex and converts to a svg file.

        INPUT:

        - ``filename`` -- string (default:``None``), the output filename.
          If ``None``, it saves the file in a temporary directory.

        - ``view`` -- bool (default:``True``), whether to open the file in
          a browser. This option is ignored and automatically set to
          ``False`` if ``filename`` is not ``None``.

        - ``program`` -- string (default:``'pdftocairo'``) ``'pdftocairo'`` or
          ``'pdf2svg'``.

        OUTPUT:

            string, path to svg file

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: _ = t.svg(view=False)     # not tested

        Same for instances of :class:`TikzPicture`::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: _ = t.svg(view=False)     # not tested

        ::

            sage: from sage.misc.temporary_file import tmp_filename
            sage: filename = tmp_filename('temp', '.svg')
            sage: path_to_file = t.svg(filename, program='pdf2svg')   # long time (1s)   # optional latex pdf2svg
            sage: path_to_file[-4:]                                   # long time (fast) # optional latex pdf2svg
            '.svg'
            sage: path_to_file = t.svg(filename, program='pdftocairo') # long time (1s)   # optional latex pdftocairo
            sage: path_to_file[-4:]                                    # long time (fast) # optional latex pdftocairo
            '.svg'

        """
        # set the temporary filenames
        temp_filename_pdf = self.pdf(filename=None, view=False)
        temp_filename, ext = os.path.splitext(temp_filename_pdf)
        temp_filename_svg = temp_filename + '.svg'

        # set the command
        if program == 'pdftocairo':
            from sage.features.poppler import pdftocairo
            pdftocairo().require()
            cmd = ['pdftocairo', '-svg', temp_filename_pdf, temp_filename_svg]
        elif program == 'pdf2svg':
            from sage.features.pdf2svg import pdf2svg
            pdf2svg().require()
            cmd = ['pdf2svg', temp_filename_pdf, temp_filename_svg]
        else:
            raise ValueError("program(={}) should be 'pdftocairo' or"
                    " 'pdf2svg'".format(program))

        # convert to svg
        result = run(cmd, capture_output=True, text=True)

        # If a problem occurs, provide the log
        if result.returncode != 0:
            print("Command \n"
                  "   '{}'\n"
                  "returned non-zero exit status {}.\n"
                  "Here is the content of the stderr:{}\n"
                  "Here is the content of the stdout:"
                  "{}\n".format(' '.join(result.args),
                                result.returncode,
                                result.stderr.strip(),
                                result.stdout.strip()))
        result.check_returncode()

        # move the svg into the good location
        if filename:
            filename = os.path.abspath(filename)
            os.rename(temp_filename_svg, filename)
            return filename

        # open the tmp svg
        elif view:
            from sage.misc.viewer import browser
            cmd = browser().split()
            cmd.append(temp_filename_svg)
            # we use check_call as opposed to run, because
            # it gives the sage prompt back to the user
            # see https://stackoverflow.com/a/71342967
            # run(cmd, capture_output=True, check=True)
            from subprocess import check_call, PIPE
            check_call(cmd, stdout=PIPE, stderr=PIPE)

        return temp_filename_svg

    def tex(self, filename=None, content_only=False, include_header=None):
        r"""
        Writes the latex code to a file.

        INPUT:

        - ``filename`` -- string (default:``None``), the output filename.
          If ``None``, it saves the file in a temporary directory.
        - ``content_only`` -- bool (default:``False``) whether to include
          the header latex part. If ``True``, it prints only the
          content to the file.

        OUTPUT:

            string, path to tex file

        EXAMPLES::

            sage: from sage.misc.latex_standalone import Standalone
            sage: t = Standalone('Hello World')
            sage: _ = t.tex()
            sage: _ = t.tex(content_only=True)

        Same for instances of :class:`TikzPicture`::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: _ = t.tex()
            sage: _ = t.tex(content_only=True)

        Write to a given filename::

            sage: from sage.misc.temporary_file import tmp_filename
            sage: filename = tmp_filename('temp','.tex')
            sage: path_to_file = t.tex(filename)
            sage: path_to_file[-4:]
            '.tex'

        """
        if filename is None:
            from sage.misc.temporary_file import tmp_filename
            filename = tmp_filename('tikz_', '.tex')
        else:
            filename = os.path.abspath(filename)

        if include_header is not None:
            content_only = not include_header
            from sage.misc.superseded import deprecation
            deprecation(20343, "When merging this code from slabbe into "
                    "SageMath the argument include_header=False was "
                    "replaced by content_only=True. Please update your code "
                    "before include_header option gets removed from SageMath.")

        if content_only:
            output = self.content()
        else:
            output = str(self)

        with open(filename, 'w') as f:
            f.write(output)

        return filename

class TikzPicture(Standalone):
    r"""
    A TikzPicture embedded in a LaTeX standalone document class.

    INPUT:

    - ``content`` -- string, tikzpicture code starting with ``r'\begin{tikzpicture}'``
      and ending with ``r'\end{tikzpicture}'``
    - ``standalone_config`` -- list of strings (default: ``[]``),
      latex document class standalone configuration options.
    - ``usepackage`` -- list of strings (default: ``[]``), latex
      packages.
    - ``usetikzlibrary`` -- list of strings (default: ``[]``), tikz libraries
      to use.
    - ``macros`` -- list of strings (default: ``[]``), stuff you need for the picture.
    - ``use_sage_preamble`` -- bool (default: ``False``), whether to include sage
      latex preamble and sage latex macros, that is, the content of
      :func:`sage.misc.latex.extra_preamble()`,
      :func:`sage.misc.latex.extra_macros()` and
      :func:`sage.misc.latex_macros.sage_latex_macros()`.

    EXAMPLES:

    Create your own tikz string from scratch and provide it::

        sage: from sage.misc.latex_standalone import TikzPicture
        sage: lines = []
        sage: lines.append(r'\begin{tikzpicture}')
        sage: lines.append(r'\draw[very thick,orange,->] (0,0) -- (1,1);')
        sage: lines.append(r'\end{tikzpicture}')
        sage: s = '\n'.join(lines)
        sage: t = TikzPicture(s)
        sage: t
        \documentclass[tikz]{standalone}
        \begin{document}
        \begin{tikzpicture}
        \draw[very thick,orange,->] (0,0) -- (1,1);
        \end{tikzpicture}
        \end{document}

    Then use it by exporting the tikzpicture to other formats, all of the
    below methods return a string providing the path to the filename, which
    is by default in a temporary folder::

        sage: _ = t.pdf()                     # not tested
        sage: _ = t.png()                     # not tested
        sage: _ = t.svg()                     # not tested
        sage: _ = t.tex()                     # not tested
        sage: _ = t.pdf(filename='abc.pdf')   # not tested

    Here we create a tikzpicture for the latex representation of a graph.
    This is using tkz-graph tex library::

        sage: g = graphs.PetersenGraph()        # optional sage.graphs
        sage: s = latex(g)                      # optional sage.graphs latex
        sage: t = TikzPicture(s, standalone_config=["border=4mm"], usepackage=['tkz-graph']) # optional sage.graphs latex
        sage: _ = t.pdf(view=False)             # long time (2s) # optional - sage.graphs latex latex_package_tkz_graph

    Here are standalone configurations, packages, tikz libraries and macros
    that can be set::

        sage: options = ['preview', 'border=4mm', 'beamer', 'float']
        sage: usepackage = ['nicefrac', 'amsmath', 'pifont', 'tikz-3dplot',
        ....:    'pgfplots']
        sage: tikzlib = ['arrows', 'snakes', 'backgrounds', 'patterns',
        ....:      'matrix', 'shapes', 'fit', 'calc', 'shadows', 'plotmarks',
        ....:      'positioning', 'pgfplots.groupplots', 'mindmap']
        sage: macros = [r'\newcommand{\ZZ}{\mathbb{Z}}']
        sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
        sage: t = TikzPicture(s, standalone_config=options, usepackage=usepackage,
        ....:        usetikzlibrary=tikzlib, macros=macros)
        sage: _ = t.pdf(view=False)   # long time (2s) # optional latex
    """
    def __init__(self, content, standalone_config=None, usepackage=None,
            usetikzlibrary=None, macros=None, use_sage_preamble=False):
        r"""
        See :class:`TikzPicture` for full information.

        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
        """
        Standalone.__init__(self, content, document_class_options=['tikz'],
            standalone_config=standalone_config, usepackage=usepackage,
            macros=macros, use_sage_preamble=use_sage_preamble)

        self._usetikzlibrary = [] if usetikzlibrary is None else usetikzlibrary

    def _latex_file_header_lines(self):
        r"""
        EXAMPLES::

            sage: latex.extra_preamble('')
            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s, standalone_config=["border=4mm"], usepackage=['tkz-graph'])
            sage: t._latex_file_header_lines()[:6]
            ['\\documentclass[tikz]{standalone}',
             '\\standaloneconfig{border=4mm}',
             '\\usepackage{tkz-graph}']
        """
        lines = Standalone._latex_file_header_lines(self)
        for library in self._usetikzlibrary:
            lines.append(r"\usetikzlibrary{{{}}}".format(library))
        return lines

    def add_usetikzlibrary(self, library):
        r"""
        Add a ``usetikzlibrary`` line

        INPUT:

        - ``library`` -- string, name of library

        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: t.add_usetikzlibrary('arrows')
            sage: t
            \documentclass[tikz]{standalone}
            \usetikzlibrary{arrows}
            \begin{document}
            \begin{tikzpicture}
            \draw (0,0) -- (1,1);
            \end{tikzpicture}
            \end{document}

        """
        self._usetikzlibrary.append(library)

    @classmethod
    def from_dot_string(cls, dotdata, prog='dot'):
        r"""
        Convert a graph to a tikzpicture using graphviz and dot2tex.

        .. NOTE::

            Prerequisite: dot2tex optional Sage package and graphviz must be
            installed.

        INPUT:

        - ``dotdata`` -- dot format string
        - ``prog`` -- string (default: ``'dot'``) the program used for the
          layout corresponding to one of the software of the graphviz
          suite: 'dot', 'neato', 'twopi', 'circo' or 'fdp'.

        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: G = graphs.PetersenGraph()                  # optional sage.graphs
            sage: dotdata = G.graphviz_string()               # optional sage.graphs
            sage: tikz = TikzPicture.from_dot_string(dotdata) # optional sage.graphs dot2tex graphviz # long time (3s)
            sage: _ = tikz.pdf()      # not tested

        ::

            sage: dotdata = G.graphviz_string(labels='latex') # optional sage.graphs
            sage: tikz = TikzPicture.from_dot_string(dotdata) # optional sage.graphs dot2tex graphviz # long time (3s)
            sage: _ = tikz.pdf()      # not tested

        ::

            sage: W = CoxeterGroup(["A",2])
            sage: G = W.cayley_graph()                        # optional sage.graphs
            sage: dotdata = G.graphviz_string()               # optional sage.graphs
            sage: tikz = TikzPicture.from_dot_string(dotdata) # optional sage.graphs dot2tex graphviz # long time (3s)
            sage: _ = tikz.pdf()      # not tested

        ::

            sage: dotdata = G.graphviz_string(labels='latex') # optional sage.graphs
            sage: tikz = TikzPicture.from_dot_string(dotdata) # optional sage.graphs dot2tex graphviz # long time (3s)
            sage: _ = tikz.pdf()      # not tested

        """
        from sage.features import PythonModule
        PythonModule("dot2tex").require()
        from sage.features.graphviz import Graphviz
        Graphviz().require()

        import dot2tex
        tikz = dot2tex.dot2tex(dotdata,
                               format='tikz',
                               autosize=True,
                               crop=True,
                               figonly='True',
                               prog=prog).strip()
        return TikzPicture(tikz, standalone_config=["border=4mm"],
                           usetikzlibrary=['shapes'])

    @classmethod
    @experimental(trac_number=20343)
    def from_graph(cls, graph, merge_multiedges=True,
            merge_label_function=tuple, **kwds):
        r"""
        Convert a graph to a tikzpicture using graphviz and dot2tex.

        .. NOTE::

            Prerequisite: dot2tex optional Sage package and graphviz must be
            installed.

        .. WARNING::

            This method might be deleted in the future in favor of a method
            in the graph class returning a tikz picture.

        INPUT:

        - ``graph`` -- graph
        - ``merge_multiedges`` -- bool (default: ``True``), if the graph
          has multiple edges, whether to merge the multiedges into one
          single edge
        - ``merge_label_function`` -- function (default:``tuple``), a
          function to apply to each list of labels to be merged. It is
          ignored if ``merge_multiedges`` is not ``True`` or if the graph
          has no multiple edges.

        Other inputs are used for latex drawing with dot2tex and graphviz:

        - ``prog`` -- string (default: ``'dot'``) the program used for the
          layout corresponding to one of the software of the graphviz
          suite: 'dot', 'neato', 'twopi', 'circo' or 'fdp'.
        - ``edge_labels`` -- bool (default: ``True``)
        - ``color_by_label`` -- bool (default: ``False``)
        - ``rankdir`` -- string (default: ``'down'``)
        - ``subgraph_clusters`` -- (default: []) a list of lists of
          vertices, if supported by the layout engine, nodes belonging to
          the same cluster subgraph are drawn together, with the entire
          drawing of the cluster contained within a bounding rectangle.

        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: g = graphs.PetersenGraph()       # optional sage.graphs
            sage: tikz = TikzPicture.from_graph(g) # optional sage.graphs dot2tex graphviz
            doctest:...: FutureWarning: This class/method/function is marked as experimental.
            It, its functionality or its interface might change without a formal deprecation.
            See http://trac.sagemath.org/20343 for details.
            sage: _ = tikz.pdf()      # not tested

        Using ``prog``::

            sage: tikz = TikzPicture.from_graph(g, prog='neato', color_by_label=True) # optional sage.graphs dot2tex graphviz # long time (3s)
            sage: _ = tikz.pdf()      # not tested

        Using ``rankdir``::

            sage: tikz = TikzPicture.from_graph(g, rankdir='right') # optional sage.graphs dot2tex graphviz # long time (3s)
            sage: _ = tikz.pdf()      # not tested

        Using ``merge_multiedges``::

            sage: alpha = var('alpha')
            sage: m = matrix(2,range(4)); m.set_immutable()
            sage: G = DiGraph([(0,1,alpha), (0,1,0), (0,2,9), (0,2,m)], multiedges=True) # optional sage.graphs
            sage: tikz = TikzPicture.from_graph(G, merge_multiedges=True) # optional sage.graphs dot2tex graphviz
            sage: _ = tikz.pdf()      # not tested

        Using ``merge_multiedges`` with ``merge_label_function``::

            sage: fn = lambda L: LatexExpr(','.join(map(str, L)))
            sage: edges = [(0,1,'a'), (0,1,'b'), (0,2,'c'), (0,2,'d')]
            sage: G = DiGraph(edges, multiedges=True)       # optional sage.graphs
            sage: tikz = TikzPicture.from_graph(G,          # optional sage.graphs dot2tex graphviz
            ....:           merge_multiedges=True, merge_label_function=fn)
            sage: _ = tikz.pdf()      # not tested

        Using subgraphs clusters (broken when using labels, see
        :trac:`22070`)::

            sage: S = FiniteSetMaps(5)
            sage: I = S((0,1,2,3,4))
            sage: a = S((0,1,3,0,0))
            sage: b = S((0,2,4,1,0))
            sage: roots = [I]
            sage: succ = lambda v:[v*a,v*b,a*v,b*v]
            sage: R = RecursivelyEnumeratedSet(roots, succ)
            sage: G = R.to_digraph()                        # optional sage.graphs
            sage: G                                         # optional sage.graphs
            Looped multi-digraph on 27 vertices
            sage: C = G.strongly_connected_components()     # optional sage.graphs
            sage: tikz = TikzPicture.from_graph(G,          # optional sage.graphs dot2tex graphviz
            ....:              merge_multiedges=False, subgraph_clusters=C)
            sage: _ = tikz.pdf()      # not tested

        An example coming from ``graphviz_string`` documentation in SageMath::

            sage: f(x) = -1 / x                                       # optional sage.symbolic
            sage: g(x) = 1 / (x + 1)                                  # optional sage.symbolic
            sage: G = DiGraph()                                       # optional sage.symbolic sage.graphs
            sage: G.add_edges((i, f(i), f) for i in (1, 2, 1/2, 1/4)) # optional sage.symbolic sage.graphs
            sage: G.add_edges((i, g(i), g) for i in (1, 2, 1/2, 1/4)) # optional sage.symbolic sage.graphs
            sage: tikz = TikzPicture.from_graph(G)                    # optional sage.symbolic sage.graphs dot2tex graphviz
            sage: _ = tikz.pdf()      # not tested
            sage: def edge_options(data):
            ....:     u, v, label = data
            ....:     options = {"color": {f: "red", g: "blue"}[label]}
            ....:     if (u,v) == (1/2, -2): options["label"]       = "coucou"; options["label_style"] = "string"
            ....:     if (u,v) == (1/2,2/3): options["dot"]         = "x=1,y=2"
            ....:     if (u,v) == (1,   -1): options["label_style"] = "latex"
            ....:     if (u,v) == (1,  1/2): options["dir"]         = "back"
            ....:     return options
            sage: tikz = TikzPicture.from_graph(G, edge_options=edge_options)  # optional sage.symbolic sage.graphs dot2tex graphviz
            sage: _ = tikz.pdf()      # not tested

        """
        from sage.features.latex import pdflatex
        pdflatex().require()
        from sage.features.graphviz import Graphviz
        Graphviz().require()
        from sage.features import PythonModule
        PythonModule("dot2tex").require()

        if merge_multiedges and graph.has_multiple_edges():
            from collections import defaultdict
            d = defaultdict(list)
            for (u, v, label) in graph.edges():
                d[(u, v)].append(label)
            edges = [(u, v, merge_label_function(label_list)) for (u, v), label_list in d.items()]
            loops = graph.has_loops()
            if graph.is_directed():
                from sage.graphs.digraph import DiGraph
                graph = DiGraph(edges, format='list_of_edges', loops=loops)
            else:
                from sage.graphs.graph import Graph
                graph = Graph(edges, format='list_of_edges', loops=loops)

        options = dict(format='dot2tex', edge_labels=True,
                       color_by_label=False, prog='dot', rankdir='down')
        options.update(kwds)

        graph.latex_options().set_options(**options)
        tikz = graph._latex_()
        return TikzPicture(tikz, standalone_config=["border=4mm"])

    @classmethod
    @experimental(trac_number=20343)
    def from_graph_with_pos(cls, graph, scale=1, merge_multiedges=True,
            merge_label_function=tuple):
        r"""
        Convert a graph with positions defined for vertices to a tikzpicture.

        .. WARNING::

            This method might be deleted in the future in favor of a method
            in the graph class returning a tikz picture.

        INPUT:

        - ``graph`` -- graph (with predefined positions)
        - ``scale`` -- number (default:``1``), tikzpicture scale
        - ``merge_multiedges`` -- bool (default: ``True``), if the graph
          has multiple edges, whether to merge the multiedges into one
          single edge
        - ``merge_label_function`` -- function (default:``tuple``), a
          function to apply to each list of labels to be merged. It is
          ignored if ``merge_multiedges`` is not ``True`` or if the graph
          has no multiple edges.

        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: g = graphs.PetersenGraph()                      # optional sage.graphs
            sage: tikz = TikzPicture.from_graph_with_pos(g)       # optional sage.graphs
            doctest:...: FutureWarning: This class/method/function is marked as experimental.
            It, its functionality or its interface might change without a formal deprecation.
            See http://trac.sagemath.org/20343 for details.

        ::

            sage: edges = [(0,0,'a'),(0,1,'b'),(0,1,'c')]
            sage: kwds = dict(format='list_of_edges', loops=True, multiedges=True)
            sage: G = DiGraph(edges, **kwds)                      # optional sage.graphs
            sage: G.set_pos({0:(0,0), 1:(1,0)})                   # optional sage.graphs
            sage: f = lambda label:','.join(label)                # optional sage.graphs
            sage: TikzPicture.from_graph_with_pos(G, merge_label_function=f) # optional sage.graphs
            \documentclass[tikz]{standalone}
            \standaloneconfig{border=4mm}
            \begin{document}
            \begin{tikzpicture}
            [auto,scale=1]
            % vertices
            \node (node_0) at (0, 0) {0};
            \node (node_1) at (1, 0) {1};
            % edges
            \draw[->] (node_0) -- node {b,c} (node_1);
            % loops
            \draw (node_0) edge [loop above] node {a} ();
            \end{tikzpicture}
            \end{document}

        TESTS::

            sage: edges = [(0,0,'a'),(0,1,'b'),(0,1,'c')]
            sage: kwds = dict(format='list_of_edges', loops=True, multiedges=True)
            sage: G = DiGraph(edges, **kwds)               # optional sage.graphs
            sage: TikzPicture.from_graph_with_pos(G)       # optional sage.graphs
            Traceback (most recent call last):
            ...
            ValueError: vertex positions need to be set first
        """
        pos = graph.get_pos()
        if pos is None:
            raise ValueError('vertex positions need to be set first')

        if merge_multiedges and graph.has_multiple_edges():
            from collections import defaultdict
            d = defaultdict(list)
            for (u, v, label) in graph.edges():
                d[(u, v)].append(label)
            edges = [(u, v, merge_label_function(label_list)) for (u, v), label_list in d.items()]
            loops = graph.has_loops()
            if graph.is_directed():
                from sage.graphs.digraph import DiGraph
                graph = DiGraph(edges, format='list_of_edges', loops=loops)
            else:
                from sage.graphs.graph import Graph
                graph = Graph(edges, format='list_of_edges', loops=loops)

        keys_for_vertices = graph._keys_for_vertices()

        lines = []
        lines.append(r'\begin{tikzpicture}')
        lines.append(r'[auto,scale={}]'.format(scale))

        # vertices
        lines.append(r'% vertices')
        for u in graph.vertices():
            line = r'\node ({}) at {} {{{}}};'.format(keys_for_vertices(u),
                                                      pos[u], u)
            lines.append(line)

        # edges
        lines.append(r'% edges')
        arrow = '->' if graph.is_directed() else ''
        for (u, v, label) in graph.edges():
            if u == v:
                # loops are done below
                continue
            if label:
                line = r'\draw[{}] ({}) -- node {{{}}} ({});'.format(arrow,
                                                    keys_for_vertices(u),
                                                    label,
                                                    keys_for_vertices(v))
            else:
                line = r'\draw[{}] ({}) -- ({});'.format(arrow,
                                                    keys_for_vertices(u),
                                                    keys_for_vertices(v))
            lines.append(line)

        # loops
        lines.append(r'% loops')
        for (u, v, label) in graph.loop_edges():
            line = r'\draw ({}) edge [loop above] node {{{}}} ();'.format(
                keys_for_vertices(u), label)
            lines.append(line)

        lines.append(r'\end{tikzpicture}')
        tikz = '\n'.join(lines)
        return TikzPicture(tikz, standalone_config=["border=4mm"])

    @classmethod
    @experimental(trac_number=20343)
    def from_poset(cls, poset, **kwds):
        r"""
        Convert a poset to a tikzpicture using graphviz and dot2tex.

        .. NOTE::

            Prerequisite: dot2tex optional Sage package and graphviz must be
            installed.

        .. WARNING::

            This method might be deleted in the future in favor of a method
            in the graph class returning a tikz picture.

        INPUT:

        - ``poset`` -- poset
        - ``prog`` -- string (default: ``'dot'``) the program used for the
          layout corresponding to one of the software of the graphviz
          suite: 'dot', 'neato', 'twopi', 'circo' or 'fdp'.
        - ``edge_labels`` -- bool (default: ``True``)
        - ``color_by_label`` -- bool (default: ``False``)
        - ``rankdir`` -- string (default: ``'down'``)

        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: P = posets.PentagonPoset()       # optional sage.combinat
            sage: tikz = TikzPicture.from_poset(P) # optional sage.combinat dot2tex graphviz
            doctest:...: FutureWarning: This class/method/function is marked as experimental.
            It, its functionality or its interface might change without a formal deprecation.
            See http://trac.sagemath.org/20343 for details.

        ::

            sage: tikz = TikzPicture.from_poset(P, prog='neato', color_by_label=True) # optional sage.combinat dot2tex # long time (3s)

        ::

            sage: P = posets.SymmetricGroupWeakOrderPoset(4)     # optional sage.combinat
            sage: tikz = TikzPicture.from_poset(P)               # optional sage.combinat dot2tex graphviz # long time (4s)
            sage: tikz = TikzPicture.from_poset(P, prog='neato') # optional sage.combinat dot2tex graphviz # long time (4s)
        """
        graph = poset.hasse_diagram()
        return cls.from_graph(graph, **kwds)

    def tikz_picture_code(self):
        r"""
        EXAMPLES::

            sage: from sage.misc.latex_standalone import TikzPicture
            sage: s = "\\begin{tikzpicture}\n\\draw (0,0) -- (1,1);\n\\end{tikzpicture}"
            sage: t = TikzPicture(s)
            sage: print(t.tikz_picture_code())
            \begin{tikzpicture}
            \draw (0,0) -- (1,1);
            \end{tikzpicture}
        """
        return self.content()


# %% [markdown]
#
#
# # Earth Analytics Education - EA  Python Course Spring 2021

# %% [markdown]
# # Important  - Assignment Guidelines
#
# 1. Before you submit your assignment to GitHub, make sure to run the entire notebook with a fresh kernel. To do this first, **restart the kernel ** (in the menubar, select Kernel$\rightarrow$Restart & Run All)
# 2. Always replace the `raise NotImplementedError()` code with your code that addresses the activity challenge. If you don't replace that code, your notebook will not run.
#
# ```
# # YOUR CODE HERE
# raise NotImplementedError()
# ```
#
# 3. Any open ended questions will have a "YOUR ANSWER HERE" within a markdown cell. Replace that text with your answer also formatted using Markdown.
# 4. ** DO NOT RENAME THIS NOTEBOOK File!** If the file name changes, the autograder will not grade your assignment properly.
# 6. When you create a figure, comment out `plt.show()` to ensure the autograder can grade your plots. For figure cells, DO NOT DELETE the code that says `DO NOT REMOVE LINE BELOW`.
#
# ```
# ### DO NOT REMOVE LINE BELOW ###
# student_plot1_ax = nb.convert_axes(plt)
# ```
#
# * Only include the package imports, code, and outputs that are required to run your homework assignment.
# * Be sure that your code can be run on any operating system. This means that:
#    1. the data should be downloaded in the notebook to ensure it's reproducible
#    2. all paths should be created dynamically using the `os.path.join`
#
# # Follow to PEP 8 Syntax Guidelines & Documentation
#
# * Run the `autopep8` tool on all cells prior to submitting(HINT: hit shift + the tool to run it on all cells at once!
# * Use clear and expressive names for variables.
# * Organize your code to support readability.
# * Check for code line length
# * Use comments and white space sparingly where it is needed
# * Make sure all python imports are at the top of your notebook and follow PEP 8 order conventions
# * Spell check your Notebook before submitting it.
#
# For all of the plots below, be sure to do the following:
#
# * Make sure each plot has a clear TITLE and , where appropriate, label the x and y axes. Be sure to include UNITS in your labels.
#

# %% [markdown]
# ### Add Your Name Below
# **Your Name: Mitch Thompson**

# %% [markdown]
#

# %% [markdown]
# ---

# %% [markdown]
# # Week 04 and 05 Homework - Automate NDVI Workflow
#
# For this assignment, you will write code to generate a plot of the mean normalized difference vegetation index (NDVI) for two different sites in the United States across one year of data:
#
# * San Joaquin Experimental Range (SJER) in Southern California, United States
# * Harvard Forest (HARV) in the Northeastern United States
#
# The data that you will use for this week is available from **earthpy** using the following download:
#
# `et.data.get_data('ndvi-automation')`
#
# ## Assignment Goals
#
# Your goal in this assignment is to create the most efficient and concise workflow that you can that allows for:
#
# 1. The code to scale if you added new sites or more time periods to the analysis.
# 2. Someone else to understand your workflow.
# 3. The LEAST and most efficient (i.e. runs fast, minimize repetition) amount of code that completes the task.
#
# ### HINTS
#
# * Remove values outside of the landsat valid range of values as specified in the metadata, as needed.
# * Keep any output files SEPARATE FROM input files. Outputs should be created in an outputs directory that is created in the code (if needed) and/or tested for.
# * Use the functions that we demonstrated during class to make your workflow more efficient.
# * BONUS - if you  chose - you can export your data as a csv file. You will get bonus points for doing this.
#
#
# ## Assignment Requirements
#
# Your submission to the GitHub repository should include:
# * This Jupyter Notebook file (.ipynb) with:
#     * The code to create a plot of mean NDVI across a year for  2 NEON Field Sites:
#         * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object
#     * The **data should be cleaned to remove the influence of clouds**. See the [earthdatascience website for an example of what your plot might look like with and without removal of clouds](https://www.earthdatascience.org/courses/earth-analytics-python/create-efficient-data-workflows/).
# * BONUS: Create one output `.csv` file that has 3 columns - NDVI, Date and Site Name - with values for SJER and HARV.
#
# Your notebook should:
# * Have *at least* 2 well documented and well named functions with docstrings.
# * Include a Markdown cell at the top of the notebook that outlines the overall workflow using pseudocode (i.e. plain language, not code)
# * Include additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# %% [markdown]
# **PSEUDOCODE**
#
# * Get sorted list of landsat tif files needed for NDVI for a single scene (bands 4-5)
#
# * Open and crop the bands to sites/site-name/vector/site-name-crop.shp
#
# * Restrict the Landsat 8 values to the 'valid range" of 0 to 10000
#
# * Stack (concat) the bands (optional for NDVI calc)
#
# * Open QA layer & crop
#
# * Generate cloud mask
#
# * Calculate mean NDVI
#
# * Generate DataFrame w/ mean NDVI
#
# * Grab site name and date from filename (e.g. file_name[0:4] for site_name)
#
# * Format date using DateTime
#
# * Add or rename columns
#
# * Index DF on the date
#
# * Output to csv

# %%
# Autograding imports - do not modify this cell
import matplotcheck.autograde as ag
import matplotcheck.notebook as nb
import matplotcheck.timeseries as ts
from datetime import datetime

# %%
# Import needed packages in PEP 8 order
# and no unused imports listed (10 points total)
import os
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import geopandas as gpd
import rioxarray as rxr
import xarray as xr
import earthpy as et
import warnings

from glob import glob
from matplotlib.dates import DateFormatter

sns.set(font_scale=1.5, style='whitegrid', context='notebook')

# %%
et.data.get_data('ndvi-automation')

data_path = os.path.join(et.io.HOME, 'earth-analytics', 'data')

if os.path.exists(data_path):
    os.chdir(data_path)
else:
    os.makedirs(data_path)
    print('The new directory is created!')
    os.chdir(data_path)

print('Current working directory is set to: ', os.getcwd())

# %%
# DO NOT MODIFY THIS CELL
# Tests that the working directory is set to earth-analytics/data

path = os.path.normpath(os.getcwd())
student_wd_parts = path.split(os.sep)

if student_wd_parts[-2:] == ['earth-analytics', 'data']:
    print("\u2705 Great - it looks like your working directory is set correctly to ~/earth-analytics/data")
else:
    print("\u274C Oops, the autograder will not run unless your working directory is set to earth-analytics/data")

# %% [markdown]
# # Figure 1: Plot 1 - Mean NDVI For Each Site Across the Year (50 points)
#
# Create a plot of the mean normalized difference vegetation index (NDVI) for the two different sites in the United States across the year:
#
# * NDVI on the x axis and formatted dates on the y for both NEON sites on one figure/axis object.
# * Each site should be identified with a different color in the plot and legend.
# * The final plot **data should be cleaned to remove the influence of clouds**.
# * Be sure to include appropriate title and axes labels.
#
# Add additional cells as needed for processing data (e.g. defining functions, etc), but be sure to:
# * follow the instructions in the code cells that have been provided to ensure that you are able to use the sanity check tests that are provided.
# * include only the plot code in the cell identified for the final plot code below

# %% [markdown]
# ## Task 1:
#
# In the cell below, create a single dataframe containing MEAN NDVI, the site name,
# and the date of the data for the HARV site
# scene `HARV/landsat-crop/LC080130302017031701T1-SC20181023151837`.  The column names for the  final
# DataFrame should be`mean_ndvi`, and `site`, and the data should be **indexed on the date**.
#
# Use the functions that we reviewed in class (or create your own versions of them) to implement your code
#
# ### In the Cell below Place  All Functions Needed to Run this Notebook (20 points)

# %%
### DO NOT REMOVE THIS LINE OR EDIT / MOVE THIS CELL ###
start_time = datetime.now()

# %%
# In this cell place all of the functions needed to run your notebook
# You will be graded here on function application, docstrings, efficiency so ensure
# All functions are placed here!

def open_clean_bands(band_path, valid_range=None):
    """Open and mask a single landsat band using a pixel_qa layer.

    Parameters
    -----------
    band_path : string
        A path to the array to be opened
    valid_range : tuple (optional)
        A tuple of min and max range of values for the data. Default = None

    Returns
    -----------
    arr : xarray DataArray
        An xarray DataArray with values that should be masked set to 1 for True (Boolean)
    """

    band = rxr.open_rasterio(band_path, masked=True).squeeze()

    if valid_range:
        mask = ((band < valid_range[0]) | (band > valid_range[1]))
        band = band.where(~xr.where(mask, True, False))

    return band

def mask_crop_ndvi(all_bands, crop_bound, pixel_qa, vals):
    """Compute normalized difference vegetation index (NDVI) from given landsat bands. Crop the NDVI layer and the pixel qa layer to the boundary as specified by a given crop_bound file.

    Parameters
    -----------
    all_bands : list
        A list containing the xarray objects for landsat  bands 4 and  5
    crop_bound: geopandas GeoDataFrame
        A geopandas dataframe to be used to crop the raster data using rasterio mask().
    pixel_qa: xarray DataArray
        An xarray DataArray with pixel qa values that have not yet been turned into a mask (0s and 1s)
    vals: list
        A list of values needed to create the cloud mask

    Returns
    -----------
    ndvi_crop : Xarray Dataset
        A cropped and masked xarray object containing NDVI values
    """

    crop_json = crop_bound.geometry

    # Clip pixel qa cloud mask layer
    cl_mask_crop = pixel_qa.rio.clip(crop_json)

    # Calculate NDVI
    ndvi_xr = (all_bands[1]-all_bands[0]) / (all_bands[1]+all_bands[0])

    # Clip NDVI layer
    ndvi_crop = ndvi_xr.rio.clip(crop_json)

    # Apply cloud mask to NDVI
    ndvi_crop = ndvi_crop.where(~cl_mask_crop.isin(vals))

    return ndvi_crop

# %%
# Set base file path for single scene data
harv_data_path = os.path.join('ndvi-automation',
                              'sites',
                              'HARV',
                              'landsat-crop',
                              'LC080130302017031701T1-SC20181023151837')

# Get sorted list of landsat tif files needed for NDVI for a single scene
harv_band_path = sorted(glob(os.path.join(harv_data_path, '*band*[4-5].tif')))

# %%
# Generate list of bands for NDVI calculation
all_bands_harv = []

# Function call in loop
for aband in harv_band_path:
    cleaned_band = open_clean_bands(band_path=aband, valid_range=(0, 10000))
    all_bands_harv.append(cleaned_band)
    date = aband[50:58]

# Set variable for site directories
sites_path = glob(os.path.join('ndvi-automation', 'sites' + '/*/'))

# Get site name from directory path
vector_dir_harv = os.path.join(sites_path[1], 'vector')
site_name = os.path.basename(os.path.normpath(sites_path[1]))

# Format date from filename
site_date = pd.to_datetime(date, format='%Y%m%d')

# %%
# Open crop boundary
site_boundary_path = os.path.join(vector_dir_harv,  site_name + '-crop.shp')
crop_bound = gpd.read_file(site_boundary_path)

# Set path to cloud mask layer
harv_qa_path = glob(os.path.join(harv_data_path, '*qa*'))

# Open the cloud mask layer
qa_layer = rxr.open_rasterio(harv_qa_path[0], masked=True).squeeze()

# List of Landsat 8 cloud no vals
vals = [328, 392, 840, 904, 1350, 352, 368, 416,
        432, 480, 864, 880, 928, 944, 992, 480, 992]

# Function call
ndvi_clean = mask_crop_ndvi(all_bands=all_bands_harv,
                            crop_bound=crop_bound,
                            pixel_qa=qa_layer,
                            vals=vals)

# Compute the arithmetic mean, ignoring NaNs
mean_ndvi = np.nanmean(ndvi_clean)

# %%
# Create dataframe of mean NDVI in this cell using the functions created above
# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Call the dataframe at the end of the cell so the tests run on it!
# Be sure that the date column is an index of type date
# HINT: the time series lessons may help you remember how to do this!

# Dict for df
ndvi_dict = {'site': site_name, 'mean_ndvi': mean_ndvi, 'date': [site_date]}

ndvi_mean_df = pd.DataFrame.from_dict(ndvi_dict)

ndvi_mean_df.set_index('date')

# %%
# This cell  is testing your data output above

student_ndvi_ts_single_site = _

single_scene_points = 0

# Ensure the data is stored in a dataframe.
if isinstance(student_ndvi_ts_single_site, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    single_scene_points += 1
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Ensure that the date column is the index
if isinstance(student_ndvi_ts_single_site.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    single_scene_points += 2
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_ts_single_site.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    single_scene_points += 2
else:
    print('\u274C The data in your date column is not datetime.')

# Ensure the site name is correct
if student_ndvi_ts_single_site.site.values[0] == 'HARV':
    print('\u2705 You have the correct site name!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct site name.')

if np.allclose(0.281131628228094, student_ndvi_ts_single_site.mean_ndvi.values[0]):
    print('\u2705 You have the correct mean NDVI value!')
    single_scene_points += 5
else:
    print('\u274C You do not have the correct mean ndvi value.')

print("\n \u27A1 You received {} out of 15 points for creating a dataframe.".format(
    single_scene_points))
single_scene_points

# %% [markdown]
# ## Task 2:
#
# In the cell below, process all of the landsat scenes. Create a DataFrame that contains the following
# information for each scene
#
#
# |   | index  | site  | mean_ndvi  |
# |---|---|---|---|
# | Date  |   |   |   |
# |  2017-01-07  | 0  | SJER  | .4  |
#
# Be sure to call your dataframe at the end of the cell to ensure autograding works.
# HINT: FOR THIS STEP, leave any rows containing missing values (`NAN`).

# %%
# Create dataframe of NDVI including the cleaning data to deal with clouds

# Important: to use the ungraded tests below as a sanity check,
# name your columns: mean_ndvi and site
# Don't forget to set date as the index and make the values of type datetime

# Supress warnings
warnings.filterwarnings(action='ignore', message='Mean of empty slice')

all_data = []

# Loop through file paths
for site_path in sites_path:
    site_name = os.path.basename(os.path.normpath(site_path))
    vector_dir = os.path.join(site_path, 'vector')
    site_boundary_path = os.path.join(vector_dir, site_name + '-crop.shp')
    crop_bound = gpd.read_file(site_boundary_path)
    landsat_dir = os.path.join(site_path, 'landsat-crop')
    all_scenes = sorted(glob(os.path.join(landsat_dir, 'LC08*')))

    # Loop through files
    for scene in all_scenes:
        band_paths = sorted(glob(os.path.join(scene,
                                              '*band*[4-5].tif')))
        all_bands = []
        # Function call in loop
        for band in band_paths:
            cleaned_band = open_clean_bands(band_path=band,
                                            valid_range=(0, 10000))
            all_bands.append(cleaned_band)

        qa_path = glob(os.path.join(scene, '*pixel_qa*'))
        qa_layer = rxr.open_rasterio(qa_path[0], masked=True).squeeze()

        # Function call
        ndvi_clean = mask_crop_ndvi(all_bands=all_bands,
                                    crop_bound=crop_bound,
                                    pixel_qa=qa_layer,
                                    vals=vals)

        # Compute the arithmetic mean, ignoring NaNs
        ndvi_mean = np.nanmean(ndvi_clean)

        # Grab date from filename convention
        date = os.path.basename(os.path.normpath(band_paths[0]))[17:25]
        site_data = [date, site_name, ndvi_mean]
        all_data.append(site_data)

ndvi_mean_df = pd.DataFrame(data=all_data,
                            columns=['date', 'site', 'mean_ndvi'])
ndvi_mean_df['date'] = pd.to_datetime(ndvi_mean_df['date'])
ndvi_mean_df.set_index('date', inplace=True)

ndvi_mean_df

# %%
# Last sanity check before creating your plot (10 points)

# Ensure that you call your dataframe at the bottom of the cell above
# and that it has columns called: mean_ndvi and site

# Ensure the data is stored in a dataframe.
student_ndvi_df = _

df_points = 0

if isinstance(student_ndvi_df, pd.DataFrame):
    print('\u2705 Your data is stored in a DataFrame!')
    df_points += 2
else:
    print('\u274C It appears your data is not stored in a DataFrame. ',
          'To see what type of object your data is stored in, check its type with type(object)')

# Check that dataframe contains the appropriate number of NAN values
if student_ndvi_df.mean_ndvi.isna().sum() == 15:
    print('\u2705 Correct number of masked data values!')
    df_points += 2
else:
    print('\u274C The amount of null data in your dataframe is incorrect.')

# Ensure that the date column is the index
if isinstance(student_ndvi_df.index, pd.core.indexes.datetimes.DatetimeIndex):
    print('\u2705 You have the index set to the date column!')
    df_points += 3
else:
    print('\u274C You do not have the index set to the date column.')

# Ensure that the date column is datetime
if isinstance(student_ndvi_df.index[0], pd._libs.tslibs.timestamps.Timestamp):
    print('\u2705 The data in your date column is datetime!')
    df_points += 3
else:
    print('\u274C The data in your date column is not datetime.')

# Output for timer, # DO NOT MODIFY
end_time = datetime.now()
total_time = end_time - start_time
print(
    "Your total run time for processing the data was {0}.".format(total_time))

print("\n \u27A1 You received {} out of 10 points for creating a dataframe.".format(
    df_points))

df_points

# %%
# Add only the plot code to this cell

# This is the final figure of mean NDVI
# for both sites across the year
# with data cleaned to deal with clouds

# Plot mean NDVI for both sites across the year.
ndvi_mean_df.dropna(subset=['mean_ndvi'], inplace=True)

colors = {'HARV': 'purple',
          'SJER': 'black'}

fig, ax = plt.subplots(figsize=(12, 8))

for site, group in ndvi_mean_df.groupby('site'):
    ax.plot(group.index,
            group.mean_ndvi,
            marker='o',
            color=colors[site],
            label=site)

date_form = DateFormatter('%b')
ax.xaxis.set_major_formatter(date_form)

fig.suptitle('Mean NDVI, HARV and SJER Field Sites', x=.52, y=.95)
ax.set(title=' Landsat 8, Jan 2017 - Dec 2017',
       xlabel='Month',
       ylabel='Mean NDVI')
ax.legend()

fig.tight_layout()

### DO NOT REMOVE LINES BELOW ###
final_masked_solution = nb.convert_axes(plt, which_axes="current")

# %%
# Ignore this cell for the autograding tests

# %%
# Ignore this cell for the autograding tests

# %% [markdown]
# # Question 1 (10 points)
#
# Imagine that you are planning NEON’s upcoming flight season to capture remote sensing data in these locations and want to ensure that you fly the area when the vegetation is the most green.
#
# When would you recommend the flights take place for each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# %% [markdown]
# The Normalized Difference Vegetation Index measures the levels of chlorophyll in vegetation, ranging from -1 to +1. The higher the measurement, the healthier and denser the vegetation likely is. Arranging flights over the HARV site would be best timed in the months of May through October, according to the 2017 values. Similiarily, the months of March and April would be best for the SJER field site per the 2017 values.

# %% [markdown]
# # Question 2 (10 points)
#
# How could you modify your workflow to look at vegetation changes over time in each site?
#
# Answer the question in 2-3 sentences in the Markdown cell below.

# %% [markdown]
# Monitoring vegetative changes over time in each site would require an increased persistance of data over time instead of the single year plotted above. Initial modifications to the workflow would include these longer time series datasets. Secondary to this longer time series would be the distinct comparison and analyses of the time windows of seasonal changes with the hypothesis as seasonal triggers should not vary by 30-60 days.

# %% [markdown]
# # Do not edit this cell! (10 points)
#
# The notebook includes:
# * additional Markdown cells throughout the notebook to describe:
#     * the data that you used - and where it is from
#     * how data are being processing
#     * how the code is optimized to run fast and be more concise

# %% [markdown]
# # Do not edit this cell! (20 points)
#
# The notebook will also be checked for overall clean code requirements as specified at the **top** of this notebook. Some of these requirements include (review the top cells for more specifics):
#
# * Notebook begins at cell [1] and runs on any machine in its entirety.
# * PEP 8 format is applied throughout (including lengths of comment and code lines).
# * No additional code or imports in the notebook that is not needed for the workflow.
# * Notebook is fully reproducible. This means:
#    * reproducible paths using the os module.
#    * data downloaded using code in the notebook.
#    * all imports at top of notebook.

# %% [markdown]
# ## BONUS - Export a  .CSV File to Share (10 points possible)
#
# This is optional - if you export a **.csv** file with the columns specified above: Site, Date and NDVI Value you can get an additional 10 points.
#
# * FULL CREDIT: File exists in csv format and contains the columns specified.
# We will check your github repo for this file!
#

# %%
csvfile = os.path.join(data_path,
                       'ndvi-automation',
                       'outputs',
                       'harv_sjer_meanNDVI_clean.csv')
ndvi_mean_df.to_csv(csvfile)
print('************Complete*************')


#%% [markdown]
# Lambda School Data Science, Unit 2: Predictive Modeling
#
# # Kaggle Challenge, Module 4
#
# ## Assignment
# - [ ] If you haven't yet, [review requirements for your portfolio project](https://lambdaschool.github.io/ds/unit2), then submit your dataset.
# - [ ] Plot a confusion matrix for your Tanzania Waterpumps model.
# - [ ] Continue to participate in our Kaggle challenge. Every student should have made at least one submission that scores at least 60% accuracy (above the majority class baseline).
# - [ ] Submit your final predictions to our Kaggle competition. Optionally, go to **My Submissions**, and _"you may select up to 1 submission to be used to count towards your final leaderboard score."_
# - [ ] Commit your notebook to your fork of the GitHub repo.
# - [ ] Read [Maximizing Scarce Maintenance Resources with Data: Applying predictive modeling, precision at k, and clustering to optimize impact](https://towardsdatascience.com/maximizing-scarce-maintenance-resources-with-data-8f3491133050), by Lambda DS3 student Michael Brady. His blog post extends the Tanzania Waterpumps scenario, far beyond what's in the lecture notebook.
#
#
# ## Stretch Goals
#
# ### Reading
# - [Attacking discrimination with smarter machine learning](https://research.google.com/bigpicture/attacking-discrimination-in-ml/), by Google Research, with  interactive visualizations. _"A threshold classifier essentially makes a yes/no decision, putting things in one category or another. We look at how these classifiers work, ways they can potentially be unfair, and how you might turn an unfair classifier into a fairer one. As an illustrative example, we focus on loan granting scenarios where a bank may grant or deny a loan based on a single, automatically computed number such as a credit score."_
# - [Notebook about how to calculate expected value from a confusion matrix by treating it as a cost-benefit matrix](https://github.com/podopie/DAT18NYC/blob/master/classes/13-expected_value_cost_benefit_analysis.ipynb)
# - [Simple guide to confusion matrix terminology](https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/) by Kevin Markham, with video
# - [Visualizing Machine Learning Thresholds to Make Better Business Decisions](https://blog.insightdatascience.com/visualizing-machine-learning-thresholds-to-make-better-business-decisions-4ab07f823415)
#
#
# ### Doing
# - [ ] Share visualizations in our Slack channel!
# - [ ] RandomizedSearchCV / GridSearchCV, for model selection. (See module 3 assignment notebook)
# - [ ] More Categorical Encoding. (See module 2 assignment notebook)
# - [ ] Stacking Ensemble. (See below)
#
# ### Stacking Ensemble
#
# Here's some code you can use to "stack" multiple submissions, which is another form of ensembling:
#
# ```python
# import pandas as pd
#
# # Filenames of your submissions you want to ensemble
# files = ['submission-01.csv', 'submission-02.csv', 'submission-03.csv']
#
# target = 'status_group'
# submissions = (pandas.read_csv(file)[[target]] for file in files)
# ensemble = pandas.concat(submissions, axis='columns')
# majority_vote = ensemble.mode(axis='columns')[0]
#
# sample_submission = pandas.read_csv('sample_submission.csv')
# submission = sample_submission.copy()
# submission[target] = majority_vote
# submission.to_csv('my-ultimate-ensemble-submission.csv', index=False)
# ```

#%%
import pandas

DATA_PATH = './data/'

# Merge train_features.csv & train_labels.csv
train = pandas.merge(pandas.read_csv(DATA_PATH+'waterpumps/train_features.csv'),
                 pandas.read_csv(DATA_PATH+'waterpumps/train_labels.csv'))

# Read test_features.csv & sample_submission.csv
test = pandas.read_csv(DATA_PATH+'waterpumps/test_features.csv')
sample_submission = pandas.read_csv(DATA_PATH+'waterpumps/sample_submission.csv')

#%%
from typing import Optional

def keepTopN(	column:pandas.Series,
				n:int,
				default:Optional[object] = None) -> pandas.Series:
	"""
	Keeps the top n most popular values of a Series, while replacing the rest with `default`

	Args:
		column (pandas.Series): Series to operate on
		n (int): How many values to keep
		default (object, optional): Defaults to NaN. Value with which to replace remaining values

	Returns:
		pandas.Series: Series with the most popular n values
	"""
	import numpy

	if default is None: default = numpy.nan

	val_counts = column.value_counts()
	if n > len(val_counts): n = len(val_counts)
	top_n = list(val_counts[:n].index)
	return(column.where(column.isin(top_n), other=default))

def oneHot(	frame:pandas.DataFrame,
			cols:Optional[list] = None,
			exclude_cols:Optional[list] = None,
			max_cardinality:Optional[int] = None,
			required_out_cols:Optional[list] = None) -> pandas.DataFrame:
	"""
	One-hot encodes the dataframe.

	Args:
		frame (pandas.DataFrame): Dataframe to clean
		cols (list, optional): Columns to one-hot encode. Defaults to all string columns.
		exclude_cols (list, optional): Columns to skip one-hot encoding. Defaults to None.
		max_cardinality (int, optional): Maximum cardinality of columns to encode. Defaults to no maximum cardinality.

	Returns:
		pandas.DataFrame: The one_hot_encoded dataframe.
	"""
	import category_encoders
	import numpy

	one_hot_encoded = frame.copy()

	if cols is None: cols = list(one_hot_encoded.columns[one_hot_encoded.dtypes=='object'])

	if exclude_cols is not None:
		for col in exclude_cols:
			cols.remove(col)

	if max_cardinality is not None:
		described = one_hot_encoded[cols].describe(exclude=[numpy.number])
		cols = list(described.columns[described.loc['unique'] <= max_cardinality])

	encoder = category_encoders.OneHotEncoder(return_df=True, use_cat_names=True, cols=cols)
	one_hot_encoded = encoder.fit_transform(one_hot_encoded)

	if required_out_cols is not None:
		for column in set(required_out_cols) - set(one_hot_encoded.columns):
			one_hot_encoded[column] = numpy.zeros(one_hot_encoded.shape[0])

	return(one_hot_encoded)

#%%

def cluster(df, n_clusters=100, kmeans=None):
	from sklearn.cluster import KMeans

	if kmeans is None:
		kmeans=KMeans(n_clusters=n_clusters)
		kmeans.fit(df[['latitude', 'longitude']])
		df['cluster'] = kmeans.labels_
	else:
		df['cluster'] = kmeans.predict(df[['latitude', 'longitude']])
	return(df, kmeans)

#%%

def clean(df, n_clusters=250, kmeans=None, n=5, required_out_cols=None, exclude_cols=None):

	cleaned = df.copy()
	cleaned, kmeans = cluster(cleaned, n_clusters=n_clusters, kmeans=kmeans)

	cleaned['date_recorded_dt'] = pandas.to_datetime(df['date_recorded'])
	cleaned['date_recorded_ts'] = cleaned['date_recorded_dt'].view('int64')
	cleaned['month_recorded'] = cleaned['date_recorded_dt'].dt.month
	cleaned['day_recorded'] = cleaned['date_recorded_dt'].dt.day
	cleaned['year_recorded'] = cleaned['date_recorded_dt'].dt.year
	cleaned['years_in_operation'] = cleaned['year_recorded'] - cleaned['construction_year']
	cleaned = cleaned.drop(columns = ['date_recorded'])

	for column in cleaned.columns[cleaned.dtypes=='object']:
		cleaned[column] = keepTopN(cleaned[column], n=n, default='other')

	encoded = oneHot(cleaned.drop(columns=['date_recorded_dt']), exclude_cols=exclude_cols, max_cardinality=n+1, required_out_cols=required_out_cols)

	return(encoded, kmeans)

#%%

cleaned, kmeans = clean(train, exclude_cols=['status_group'])
train_features = cleaned.drop(columns=['status_group'])
train_target = cleaned['status_group'].values.flatten()
test_features, kmeans = clean(test, kmeans=kmeans, required_out_cols=list(train_features.columns))

#%%
print(list(train_features.columns))

#%%
print(list(test_features.columns))

#%%
train_features.dtypes[train_features.dtypes=='object']

#%%

import category_encoders as ce
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import KFold, RandomizedSearchCV
from scipy.stats import randint, uniform
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

_ss = StandardScaler()
_pca = PCA()
_rfc = RandomForestClassifier(random_state=3)

params = {
	'RandomForestClassifier__n_estimators': [27,270],
	'RandomForestClassifier__min_samples_leaf': [5,10],
	'RandomForestClassifier__oob_score': [False],
	'RandomForestClassifier__criterion': ['gini'],
	'PCA__n_components': [40,80,160]
}

# n_estimators = 1000, min_samples_leaf = 2

pipeline = Pipeline([	('StandardScaler', _ss),
						('PCA', _pca),
						('RandomForestClassifier', _rfc)])

searchCV = RandomizedSearchCV(	pipeline,
								param_distributions=params,
								n_iter=5,
								cv=3,
								scoring='accuracy',
								verbose=10,
								return_train_score=True,
								n_jobs=-1)

target_encoder = ce.OrdinalEncoder()
train_target_encoded = target_encoder.fit_transform(train_target)
train_target_encoded

searchCV.fit(train_features, train_target_encoded)

#%%
print('Cross-validation accuracy', searchCV.best_score_)
print('Best hyperparameters', searchCV.best_params_)

#%%
out = test_features[['id']].copy()

#%%
train_features.shape

#%%
test_features.shape

#%%
test_features_n = test_features.drop(columns=list(set(test_features.columns) - set(train_features.columns)))

#%%

#%%
out['status_group'] = searchCV.predict(test_features_n)

#%%
out['status_group'].value_counts()

#%%
out['status_group'] = target_encoder.inverse_transform(out['status_group'])

out.sort_values(by='id').to_csv('./module4/results.csv', index=False)

#%%