I think some of the 3.10 + numpy-1.22.dev failures are fixed on numpy's default branch (at one point I was seeing issuse with distutils warnings, but they are gone for me locally).

I am in principle 👍🏻 👍🏻 on this idea!

Looks like the test suite is passing with the nightly pandas and numpy wheels now.

https://github.com/greglucas/matplotlib/runs/4290939395?check_suite_focus=true

Some of the tests seem to be a little flaky when they are run:https://github.com/greglucas/matplotlib/actions/runs/1495511858
The new changes seem to be good, buttest_errorbar is failing, even though it passed on a previous run with the same dependency versions installed...

I pushed a local copy of this up to my fork:https://github.com/greglucas/matplotlib/runs/4621007693?check_suite_focus=true
and both 3.9 and 3.10 with latest numpy is failing. However, when I try to run this all locally (installing numpy from nightly wheels) the tests all pass for me. There must be something about the runner environment that makes a few of the tests flaky, but I haven't been able to figure it out yet looking through the logs.

The result images should be obtainable on the Summary page. It doesn't look like the failures are anywhere other than test images.

Yep, I did download them and couldn't find any obvious reason for the failures (they are all quite small). The largest onetest_errorbar has an expanded y-limit, causing the number of ticks to be increased, but the data in the plot appears to be correct. My initial guess is that the failures are due to floating point discrepancies, causing a ylim to be expanded to the next decade, but I'm not sure where that would be coming from since I can't reproduce locally.

Putting some more notes down here:
This test suite does pass CI if I don't upgrade numpy to the nightly wheel:https://github.com/greglucas/matplotlib/actions/runs/1617303489 (I was worried I'd updated some other config to make a standard run fail). I also tried pip installing nightly numpy before everything else and after everything else (including matplotlib), to make sure it wasn't affecting the compilation of any dependencies and it failed in every case there as well.

I can't reproduce these failures on a local macosx or ubuntu machine either, so my debugging capability here is limited unfortunately.

Perhaps this is related to AVX512 instructions now being available in the wheel...
https://numpy.org/devdocs/release/1.22.0-notes.html#vectorize-umath-module-using-avx-512

On the 3.8-3.10 runners:

Supported SIMD extensions in this NumPy install:    baseline = SSE,SSE2,SSE3    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2,AVX512F,AVX512CD,AVX512_SKX    not found = AVX512_KNL,AVX512_KNM,AVX512_CLX,AVX512_CNL,AVX512_ICL

On the 3.7 runner (numpy 1.21):

Supported SIMD extensions in this NumPy install:    baseline = SSE,SSE2,SSE3    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2    not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL

My local Ubuntu is on a chip without AVX512 instructions, which could explain why I'm not able to reproduce this locally. If someone does have a system with a chip that supports AVX512 and could test locally that could verify this guess:python -c "import numpy; numpy.show_config()"
or if someone knows of a way to disable that instruction set from use within numpy at runtime rather than build time, I could push that up to the runner as well.

Perhaps this is related to AVX512 instructions now being available in the wheel...https://numpy.org/devdocs/release/1.22.0-notes.html#vectorize-umath-module-using-avx-512

I suspect this is correct - I'm seeing similar small floating point differences on another project that seems to be related to both numpy version and the specific machine CI is being run on.

I decided to rework this and incorporate it into the current tests.yml file, which seems better than trying to maintain two copies of the CI file. There are new floating-point issues that cropped up again in Normalize with longdouble. It must be a new numpy optimization in.min(), as specific item-access works, so I've updated the tests to account for these changes.