Current state-of-the-art#

Testing

matplotlib currently uses Travis-CI for automated tests. WhileTravis-CI should be praised for how much it does as a free service, ithas a number of shortcomings:

• It often fails due to network timeouts when installing dependencies.

• It often fails for inexplicable reasons.

• build or test products can only be saved from build off of brancheson the main repo, not pull requests, so it is often difficult to"post mortem" analyse what went wrong. This is particularlyfrustrating when the failure cannot be subsequently reproducedlocally.

• It is not extremely fast. matplotlib's cpu and memory requirementsfor testing are much higher than the average Python project.

• It only tests on Ubuntu Linux, and we have only minimal control overthe specifics of the platform. It can be upgraded at any timeoutside of our control, causing unexpected delays at times that maynot be convenient in our release schedule.

On the plus side, Travis-CI's integration with github -- automaticallytesting all pending pull requests -- is exceptional.

Builds

There is no centralized effort for automated binary builds formatplotlib. However, the following disparate things are being done[If the authors mentioned here could fill in detail, that would begreat!]:

• @sandrotosi: builds Debian packages

• @takluyver: Has automated Ubuntu builds on Launchpad

• @cgohlke: Makes Windows builds (don't know how automated that is)

• @r-owen: Makes OS-X builds (don't know how automated that is)

Documentation

Documentation of main is now built by travis and uploaded tohttps://matplotlib.org/devdocs/index.html

@NelleV, I believe, generates the docs automatically and posts them onthe web to chart MEP10 progress.

Peculiarities of matplotlib#

matplotlib has complex requirements that make testing and buildingmore taxing than many other Python projects.

• The CPU time to run the tests is quite high. It puts us beyond thefree accounts of many CI services (e.g. ShiningPanda)

• It has a large number of dependencies, and testing the full matrixof all combinations is impractical. We need to be clever about whatspace we test and guarantee to support.

Requirements#

This section outlines the requirements that we would like to have.

1. Testing all pull requests by hooking into the GitHub API, asTravis-CI does

2. Testing on all major platforms: Linux, Mac OS-X, MS Windows (inthat order of priority, based on user survey)

3. Retain the last n days worth of build and test products, to aid inpost-mortem debugging.

4. Automated nightly binary builds, so that users can test thebleeding edge without installing a complete compilationenvironment.

5. Automated benchmarking. It would be nice to have a standardbenchmark suite (separate from the tests) whose performance couldbe tracked over time, in different backends and platforms. Whilethis is separate from building and testing, ideally it would run onthe same infrastructure.

6. Automated nightly building and publishing of documentation (or aspart of testing, to ensure PRs don't introduce documentation bugs).(This would not replace the static documentation for stablereleases as a default).

7. The test systems should be manageable by multiple developers, sothat no single person becomes a bottleneck. (Travis-CI's designdoes this well -- storing build configuration in the gitrepository, rather than elsewhere, is a very good design.)

8. Make it easy to test a large but sparse matrix of differentversions of matplotlib's dependencies. The matplotlib user surveyprovides some good data as to where to focus our efforts:https://docs.google.com/spreadsheets/d/1jbK0J4cIkyBNncnS-gP7pINSliNy9lI-N4JHwxlNSXE/edit

9. Nice to have: A decentralized design so that those with moreobscure platforms can publish build results to a central dashboard.

CI Infrastructure#

• We like Travis and it will probably remain part of our arsenal inany event. The reliability issues are being looked into.

• Enable Amazon S3 uploads of testing products on Travis. This willhelp with post-mortem of failures (@mdboom is looking into thisnow).

• We want Mac coverage. The best bet is probably to push Travis toenable it for our project by paying them for a Pro account (sincethey don't otherwise allow testing on both Linux and Mac).

• We want Windows coverage. Shining Panda is an option there.

• Investigate finding or building a tool that would collect andsynthesize test results from a number of sources and post it toGitHub using the GitHub API. This may be of general use to theScipy community.

• For both Windows and Mac, we should document (or better yet, script)the process of setting up the machine for a build, and how to buildbinaries and installers. This may require getting information fromRussel Owen and Christoph Gohlke. This is a necessary step fordoing automated builds, but would also be valuable for a number ofother reasons.

The test framework itself#

• We should investigate ways to make it take less time

  • Eliminating redundant tests, if possible

  • General performance improvements to matplotlib will help

• We should be covering more things, particularly more backends

• We should have more unit tests, fewer integration tests, if possible