Oct 10, 2017 · Oct 22, 2017 · Oct 23, 2017
diff --git a/doc/devel/MEP/MEP32.rst b/doc/devel/MEP/MEP32.rst
 MEP32: ``matplotlibrc``/``mplstyle`` with Python syntax
 =======================================================

 .. contents:: :local:

 Status
 ------

 Discussion

 Branches and Pull Requests
 --------------------------

 None

 Abstract
 --------

 I propose to replace ``matplotlibrc`` and ``mplstyle`` files (henceforth
 "old-style configs") by files using regular Python syntax (henceforth
 "new-style configs").  This will fix a number of issues that currently exist
 with parsing old-style configs.

 Detailed description
 --------------------

 The problem
 ~~~~~~~~~~~

 Current ("old-style") configuration files use a custom syntax, of the form ::

   key: value  # possible comment
   key: value

 For each allowable key, a specific parser is defined in ``mpl.rcsetup``,
 possibly via some helper functions.  Typical parsers parse booleans, numbers,
 strings (always unquoted), enumerated strings, comma-separated lists of the
 above-mentioned types, etc.

 However, some desirable inputs are difficult to parse, or currently only
 partially parsed:

 - Property cycles (e.g., ``axes.prop_cycle``), of the form ::

     cycler("key", [value1, value2, ...])

  are currently simply parsed via ``eval`` in a restricted environment
  (`#6274`_), which may (or may not) be a security hole (especially combined
  with the ability to load style files from an URL).

 - Path effects (e.g., for implementing an XKCD style sheet, `#6157`_).  They
  would have a form similar to ::

      patheffects.withStroke(linewidth=4, foreground="w")

  and would, like ``cycler``\s, require either a custom parser or being
  ``eval``'d.

 - Long inputs (e.g. property cycles) cannot be split over multiple lines, as
  the parser has no support for line continuations (`#9184`_).

 - LaTeX and PGF preambles (multiple, comma separated strings) cannot contain
  commas, because commas are replaced by newlines by the parser (`#4371`_).
  Note that commas are in fact fairly common in "normal" LaTeX preambles (e.g.,
  ``\usepackage[option1, option2]{package}``.  Actually inputting the preamble
  over multiple lines is not possible due to the lack of multiline support (see
  above).

 - Strings cannot contain the hash (``#``) symbol, as strings are unquoted and
  the hash is unconditionally interpreted as the start of a comment (`#7089`_).
  The hash symbol had been proposed to indicate a "plus" marker.

 - Dash specs of the form ``(offset, (ink-on, ink-off, in-on, ...))`` are
  misparsed: while ::

     plt.plot([1, 2], ls=(0, (5, 5)))

  works just fine, ::

     plt.rcParams["lines.linestyle"] = (0, (5, 5))

  and ::

     plt.rcParams["lines.linestyle"] = "(0, (5, 5))"

  as well as setting this value in ``matplotlibrc`` all raise an exception
  (indirectly a cause of `#7219`_).

 - Custom color palettes (redefining the meaning of ``"r"``, ``"g"``, ``"b"``,
  etc. as seaborn used to do) has been proposed (`#8430`_), but the
  hypothetical rcParam value would have a type of ``Dict[str, Dict[str, str]]``
  (mapping palettes to mappings of color names to color values), which was
  described by @tacaswell as "not too hard to parse" but "would further stress
  our current configuration system".

 Overall, the syntax of the config file is defined as "whatever the parser
 accepts" (`#3670`_).

 An additional feature that has been requested (but shoudl not be particularly
 difficult to implement using the current machinery) is "cascading" style
 sheets, either by adding a ``style`` key to style files (`#4240`_) or by
 loading all ``matplotlibrc``\s in order (`#6320`_).

 Proposed solution
 ~~~~~~~~~~~~~~~~~

 Instead of playing whack-a-mole with parser bugs, I propose to replace the
 syntax of config files to simply use Python config.  Two main possibilities are
 considered below.  In all cases, it is necessary to encode, in a way or
 another, that the config file uses the new syntax, so that Matplotlib can tell
 which file-parser to use.

 Maintain a matplotlibrc-like syntax
 ```````````````````````````````````

 The config files would maintain the format ::

   key: value  # possible comment
   key: value

 but all values would simply be parsed by passing to ``eval`` in the same
 restricted environment as for cyclers.  Further validation of the inputs should
 try to reuse whatever validation code Matplotlib already uses to validate
 the same input when passed to an actual artist's property setter (e.g.,
 validating a linestyle should call the same helper validator function as
 ``Line2D.set_linestyle``).

 - The fact that a config file uses the nex syntax could be indicated by some
  "magic string" (e.g. ``# matplotlibrc-syntax-version: 2``), or a different
  naming convention.

 - Parser handling for line-continuations would still need to be implemented.  A
  relatively simple possibility would be to support backslash continuations
  (lack of support for implicit continuations based on parentheses could be
  somewhat surprising to a user inputting Python syntax, though).

 - From a security point of view, this is exactly as secure as the current
  situation (whatever one can pass to ``eval`` with this syntax, one could
  already do it by passing it as value for the ``axes.prop_cycle`` key).

 - Support for ``patheffects`` would require adding more entries into the
  restricted environment.

 Full Python syntax
 ``````````````````

 The config files would simply be Python source files, of the form ::

   from matplotlib import rcParams
   rcParams["key"] = value  # possible comment
   rcParams["key"] = value

 or ::

   from matplotlib import rcParams
   rcParams.update(
      {"key": value,  # possible comment
       "key": value}
   )

 The files (with a ``.py`` extension, thus immediately distinguishable from
 old-style configs) would be either

 - option 1: ``exec``'d in a completely standard context (empty globals, all
  builtins available).  A few variables (``rcParams``, ``cycler``, etc.) could
  be preloaded into the globals, but I would prefer not (`#8235`_; see also
  `here <explicit-imports_>`_).

 - option 2: Imported (operating by side-effect of the import), and then
  immediately removed from ``sys.modules`` so that reloading works; the config
  loader code would be in charge of locally patching ``sys.path`` to make the
  config files visible to the import system.

 In either case, cascading style sheets can be implemented by having a config
 file ``exec`` or ``import`` (depending on the option chosen) itself another
 config file.

 It would remain possible to disallow (accidental) modification of certain
 rcParams from style files by locally patching ``RcParams.__setitem__`` in
 ``style.use``.  However, the style files would be able to execute arbitrary
 code (this is a *feature* of this proposal).

 As above, validation should share as much code as possible as the actual artist
 property setters.

 No parser would need to be written at all -- it's done for us by Python!

 Direct loading from an URL would be disabled, as it is inherently insecure.
 The documentation would encourage manual downloading (... or could even
 document how to do it using ``urllib`` if we really want to) of style sheets,
 which I believe is a good enough replacement (but I am happy to hear arguments
 that it is not).

 (Lack of) changes for the end-user
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 For the end user, calls to ``matplotlib.style.use`` are unchanged.  If
 maintaining the *same* naming convention, then the lookup mechanism stays the
 same and there is no ambiguity as to what file should be loaded.  If the naming
 convention is changed (e.g. ``.mplstyle-v2`` with the matplotlibrc-like syntax,
 or ``.py``\(!) with the full Python syntax), then a simple priority policy
 (such as "always prefer the newer syntax if available") can be implemented.

 Implementation
 --------------

 The general implementation strategy is outlined in the proposed solutions.
 Neither strategy appears to present large technical difficulties.  Actual work
 will be based on the agreed-upon syntax.

 Backward compatibility
 ----------------------

 New-style configs use a different code path, so old-style config support can
 remain in order to maintain full backward compatibility.  Deprecating support
 for old-style configs can be discussed and decided upon at a later time (or
 never done).

 Alternatives
 ------------

 - Proposal:  Fix the current issues with the parsers and implement custom
  parsers for the additional kinds of values we want to support.

  Issues:  Is it really worth maintaining a large corpus of custom parsers for
  a custom-designed language that is essentially used only by us?

 - Proposal:  Switch to another configuration language (JSON, YAML, etc.).

  Issues:  It remains necessary to be able to encode certain specific Python
  objects (certainly cyclers, possibly path effects), which means that they
  will need to be ``eval``'d (in which case I fail to see the advantage
  over using Python throughout), or that custom syntax (compatible with the
  underlying configuration language!) will need to be invented and custom
  parsers maintained.  Additionally, JSON does not support comments, and YAML
  is an extremely (overly, in my opinion) complex language.  See also the
  discussion that took place over PEP518_ (not that I particularly like the
  final choice of yet another obscure configuration language by that PEP).

 Relevant issues
 ---------------

 - `#3670`_: "Document that rcparam values are unquoted"
 - `#4240`_: "Style RC parameter"
 - `#4371`_: "LaTeX and PGF preambles do not allow commas"
 - `#6157`_: "Add style for xkcd"
 - `#6274`_: "A feeble attempt to plug a theoretical security hole"
 - `#6320`_: "all matplotlibrc files should be loaded in order"
 - `#7089`_: "matplotlibrc reader cannot handle rcparams with a hash"
 - `#7219`_: "Cycler passed as string for rc"
 - `#8235`_: "Investigate why some examples build properly in our gallery
  despite using numpy and not importing it explicitely"
 - `#8430`_: "[feature request] move default colors into styles"
 - `#9184`_: "matplotlibrc parsing is inconsistent with matplotlibrc.template"

 .. _#3670: https://github.com/matplotlib/matplotlib/issues/3670
 .. _#4240: https://github.com/matplotlib/matplotlib/issues/4240
 .. _#4371: https://github.com/matplotlib/matplotlib/issues/4371
 .. _#6157: https://github.com/matplotlib/matplotlib/issues/6157
 .. _#6274: https://github.com/matplotlib/matplotlib/issues/6274
 .. _#6320: https://github.com/matplotlib/matplotlib/issues/6320
 .. _#7089: https://github.com/matplotlib/matplotlib/issues/7089
 .. _#7219: https://github.com/matplotlib/matplotlib/issues/7219
 .. _#8235: https://github.com/matplotlib/matplotlib/issues/8235
 .. _#8430: https://github.com/matplotlib/matplotlib/issues/8430
 .. _#9184: https://github.com/matplotlib/matplotlib/issues/9184
 .. _PEP518: https://www.python.org/dev/peps/pep-0518/#other-file-formats
 .. _explicit-imports: https://www.reddit.com/r/Python/comments/ex54j/seeking_clarification_on_pylonsturbogearspyramid/c1bo1v5/
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,268 @@
		MEP32: ``matplotlibrc``/``mplstyle`` with Python syntax
		=======================================================

		.. contents:: :local:

		Status
		------

		Discussion

		Branches and Pull Requests
		--------------------------

		None

		Abstract
		--------

		I propose to replace ``matplotlibrc`` and ``mplstyle`` files (henceforth
		"old-style configs") by files using regular Python syntax (henceforth
		"new-style configs"). This will fix a number of issues that currently exist
		with parsing old-style configs.

		Detailed description
		--------------------

		The problem
		~~~~~~~~~~~

		Current ("old-style") configuration files use a custom syntax, of the form ::

		key: value # possible comment
		key: value

		For each allowable key, a specific parser is defined in ``mpl.rcsetup``,
		possibly via some helper functions. Typical parsers parse booleans, numbers,
		strings (always unquoted), enumerated strings, comma-separated lists of the
		above-mentioned types, etc.

		However, some desirable inputs are difficult to parse, or currently only
		partially parsed:

		- Property cycles (e.g., ``axes.prop_cycle``), of the form ::

		cycler("key", [value1, value2, ...])

		are currently simply parsed via ``eval`` in a restricted environment
		(`#6274`_), which may (or may not) be a security hole (especially combined
		with the ability to load style files from an URL).

		- Path effects (e.g., for implementing an XKCD style sheet, `#6157`_). They
		would have a form similar to ::

		patheffects.withStroke(linewidth=4, foreground="w")

		and would, like ``cycler``\s, require either a custom parser or being
		``eval``'d.

		- Long inputs (e.g. property cycles) cannot be split over multiple lines, as
		the parser has no support for line continuations (`#9184`_).

		- LaTeX and PGF preambles (multiple, comma separated strings) cannot contain
		commas, because commas are replaced by newlines by the parser (`#4371`_).
		Note that commas are in fact fairly common in "normal" LaTeX preambles (e.g.,
		``\usepackage[option1, option2]{package}``. Actually inputting the preamble
		over multiple lines is not possible due to the lack of multiline support (see
		above).

		- Strings cannot contain the hash (``#``) symbol, as strings are unquoted and
		the hash is unconditionally interpreted as the start of a comment (`#7089`_).
		The hash symbol had been proposed to indicate a "plus" marker.

		- Dash specs of the form ``(offset, (ink-on, ink-off, in-on, ...))`` are
		misparsed: while ::

		plt.plot([1, 2], ls=(0, (5, 5)))

		works just fine, ::

		plt.rcParams["lines.linestyle"] = (0, (5, 5))

		and ::

		plt.rcParams["lines.linestyle"] = "(0, (5, 5))"

		as well as setting this value in ``matplotlibrc`` all raise an exception
		(indirectly a cause of `#7219`_).

		- Custom color palettes (redefining the meaning of ``"r"``, ``"g"``, ``"b"``,
		etc. as seaborn used to do) has been proposed (`#8430`_), but the
		hypothetical rcParam value would have a type of ``Dict[str, Dict[str, str]]``
		(mapping palettes to mappings of color names to color values), which was
		described by @tacaswell as "not too hard to parse" but "would further stress
		our current configuration system".

		Overall, the syntax of the config file is defined as "whatever the parser
		accepts" (`#3670`_).

		An additional feature that has been requested (but shoudl not be particularly
		difficult to implement using the current machinery) is "cascading" style
		sheets, either by adding a ``style`` key to style files (`#4240`_) or by
		loading all ``matplotlibrc``\s in order (`#6320`_).

		Proposed solution
		~~~~~~~~~~~~~~~~~

		Instead of playing whack-a-mole with parser bugs, I propose to replace the
		syntax of config files to simply use Python config. Two main possibilities are
		considered below. In all cases, it is necessary to encode, in a way or
		another, that the config file uses the new syntax, so that Matplotlib can tell
		which file-parser to use.

		Maintain a matplotlibrc-like syntax
		```````````````````````````````````

		The config files would maintain the format ::

		key: value # possible comment
		key: value

		but all values would simply be parsed by passing to ``eval`` in the same
		restricted environment as for cyclers. Further validation of the inputs should
		try to reuse whatever validation code Matplotlib already uses to validate
		the same input when passed to an actual artist's property setter (e.g.,
		validating a linestyle should call the same helper validator function as
		``Line2D.set_linestyle``).

		- The fact that a config file uses the nex syntax could be indicated by some
		"magic string" (e.g. ``# matplotlibrc-syntax-version: 2``), or a different
		naming convention.

		- Parser handling for line-continuations would still need to be implemented. A
		relatively simple possibility would be to support backslash continuations
		(lack of support for implicit continuations based on parentheses could be
		somewhat surprising to a user inputting Python syntax, though).

		- From a security point of view, this is exactly as secure as the current
		situation (whatever one can pass to ``eval`` with this syntax, one could
		already do it by passing it as value for the ``axes.prop_cycle`` key).

		- Support for ``patheffects`` would require adding more entries into the
		restricted environment.

		Full Python syntax
		``````````````````

		The config files would simply be Python source files, of the form ::

		from matplotlib import rcParams
		rcParams["key"] = value # possible comment
		rcParams["key"] = value

		or ::

		from matplotlib import rcParams
		rcParams.update(
		{"key": value, # possible comment
		"key": value}
		)

		The files (with a ``.py`` extension, thus immediately distinguishable from
		old-style configs) would be either

		- option 1: ``exec``'d in a completely standard context (empty globals, all
		builtins available). A few variables (``rcParams``, ``cycler``, etc.) could
		be preloaded into the globals, but I would prefer not (`#8235`_; see also
		`here <explicit-imports_>`_).

		- option 2: Imported (operating by side-effect of the import), and then
		immediately removed from ``sys.modules`` so that reloading works; the config
		loader code would be in charge of locally patching ``sys.path`` to make the
		config files visible to the import system.

		In either case, cascading style sheets can be implemented by having a config
		file ``exec`` or ``import`` (depending on the option chosen) itself another
		config file.

		It would remain possible to disallow (accidental) modification of certain
		rcParams from style files by locally patching ``RcParams.__setitem__`` in
		``style.use``. However, the style files would be able to execute arbitrary
		code (this is a feature of this proposal).

		As above, validation should share as much code as possible as the actual artist
		property setters.

		No parser would need to be written at all -- it's done for us by Python!

		Direct loading from an URL would be disabled, as it is inherently insecure.
		The documentation would encourage manual downloading (... or could even
		document how to do it using ``urllib`` if we really want to) of style sheets,
		which I believe is a good enough replacement (but I am happy to hear arguments
		that it is not).

		(Lack of) changes for the end-user
		~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

		For the end user, calls to ``matplotlib.style.use`` are unchanged. If
		maintaining the same naming convention, then the lookup mechanism stays the
		same and there is no ambiguity as to what file should be loaded. If the naming
		convention is changed (e.g. ``.mplstyle-v2`` with the matplotlibrc-like syntax,
		or ``.py``\(!) with the full Python syntax), then a simple priority policy
		(such as "always prefer the newer syntax if available") can be implemented.

		Implementation
		--------------

		The general implementation strategy is outlined in the proposed solutions.
		Neither strategy appears to present large technical difficulties. Actual work
		will be based on the agreed-upon syntax.

		Backward compatibility
		----------------------

		New-style configs use a different code path, so old-style config support can
		remain in order to maintain full backward compatibility. Deprecating support
		for old-style configs can be discussed and decided upon at a later time (or
		never done).

		Alternatives
		------------

		- Proposal: Fix the current issues with the parsers and implement custom
		parsers for the additional kinds of values we want to support.

		Issues: Is it really worth maintaining a large corpus of custom parsers for
		a custom-designed language that is essentially used only by us?

		- Proposal: Switch to another configuration language (JSON, YAML, etc.).

		Issues: It remains necessary to be able to encode certain specific Python
		objects (certainly cyclers, possibly path effects), which means that they
		will need to be ``eval``'d (in which case I fail to see the advantage
		over using Python throughout), or that custom syntax (compatible with the
		underlying configuration language!) will need to be invented and custom
		parsers maintained. Additionally, JSON does not support comments, and YAML
		is an extremely (overly, in my opinion) complex language. See also the
		discussion that took place over PEP518_ (not that I particularly like the
		final choice of yet another obscure configuration language by that PEP).

		Relevant issues
		---------------

		- `#3670`_: "Document that rcparam values are unquoted"
		- `#4240`_: "Style RC parameter"
		- `#4371`_: "LaTeX and PGF preambles do not allow commas"
		- `#6157`_: "Add style for xkcd"
		- `#6274`_: "A feeble attempt to plug a theoretical security hole"
		- `#6320`_: "all matplotlibrc files should be loaded in order"
		- `#7089`_: "matplotlibrc reader cannot handle rcparams with a hash"
		- `#7219`_: "Cycler passed as string for rc"
		- `#8235`_: "Investigate why some examples build properly in our gallery
		despite using numpy and not importing it explicitely"
		- `#8430`_: "[feature request] move default colors into styles"
		- `#9184`_: "matplotlibrc parsing is inconsistent with matplotlibrc.template"

		.. _#3670: https://github.com/matplotlib/matplotlib/issues/3670
		.. _#4240: https://github.com/matplotlib/matplotlib/issues/4240
		.. _#4371: https://github.com/matplotlib/matplotlib/issues/4371
		.. _#6157: https://github.com/matplotlib/matplotlib/issues/6157
		.. _#6274: https://github.com/matplotlib/matplotlib/issues/6274
		.. _#6320: https://github.com/matplotlib/matplotlib/issues/6320
		.. _#7089: https://github.com/matplotlib/matplotlib/issues/7089
		.. _#7219: https://github.com/matplotlib/matplotlib/issues/7219
		.. _#8235: https://github.com/matplotlib/matplotlib/issues/8235
		.. _#8430: https://github.com/matplotlib/matplotlib/issues/8430
		.. _#9184: https://github.com/matplotlib/matplotlib/issues/9184
		.. _PEP518: https://www.python.org/dev/peps/pep-0518/#other-file-formats
		.. _explicit-imports: https://www.reddit.com/r/Python/comments/ex54j/seeking_clarification_on_pylonsturbogearspyramid/c1bo1v5/