Seetacaswell/mpl-imageset-demo#1 for what a PR would look like

seehttps://github.com/tacaswell/mpl-imageset-demo#operation for a skeletal user guide but it covers:

• how to run tests
• how to regenerate the baseline images
• how to validate that the baseline set is consistent with the current checkout
• tell the system we have changed a test image
• tell the system about a new test image

Presumablymanage_baseline_images.py will end up in ourtools directory and the the envs will be folded into a pytest plugin / some other way to thread that configuration through.

Of course there are complications 🤣

• 2 of theeps tests actually compare against a pdf output (and there is some special casing in the baseline lookup code for that (has been there from ~2011). Given that we will no longer be storing the files, we can probably drop that long term
• think I have fixed the windows vs posix path issues

Tasks:

• pull apart the "compare" and "generate" functionality
• integrate with pytest test discovery to when in generate mode do not run non-image tests (save time!)
• try to find better internal names for everything in the compare / generate methods
• write docs on how to bootstrap your own test images
• write docs of how to add image
• write docs of how to remove and image
• write docs on how to update an image
• write helper to zip up and systematically name a set of images
• sort out how to leverage caching (on all of the CI services)
• write GHA to generate above zip/tar balls of images (or do git)
• ..and push those someplace (?)
• make a wheel with the images (?)
• write docs on how to use pre-generated images
• look at sqlite instead of json of md cache

The image_lists.txt for Matplotlib are now checked in so people can take a look at what those look like at full scale.

Should the metadata.json files be included in the repo? or are those intended to be local only?

Should the metadata.json files be included in the repo? or are those intended to be local only?

They are intended to go with the generated sets of images, however given that we are not at the point where we can actually pull the images out of the repo, but I still want the tests to run they need to be checked in for now. The final version of this PR will squash them out of existence.

I think this is at a point where (modulo the metadata.json files) where it is starting to be reviewable.

The internal names are...not great, but I think I have pulled it apart enough that it is not complete spaghetti code.

MPLTESTIMAGEPATH='/tmp/test_images2' MPLGENERATEBASELINE=1 pytest

should work and generate you a full tree of test images on this branch!

This now includes a pytest plugin (😱) so that

pytest --image-baseline=/tmp/test_images --generate-images   # generate the baseline imagespytest --image-baseline=/tmp/test_images -n 20               # test against them

works. The$MPLTESTIMAGEPATH and$MPLGENERATEBASELINE also still work (but the flags "win").

All of these names need some feedback.

This seems to be working (successfully generated test images with a new freetype and it passed against them)!

I'm now sure that this is going to work, but lots of details left.

not going to get this done in the next few weeks.