Summary

Currently if a scheduler callbacks queues a new C scheduled callback, that callback will run in the same schedule pass. This means if a scheduler callback re-queues itself, it can "infinite loop" inside the scheduler.

This change means that any call tomp_handle_pending() will only process the callbacks which were queued whenmp_handle_pending() started running. Therefore any callback which re-queues itself should have that callback handled the next timemp_handle_pending() is called.

This is an alternative to#17248 and should remove the need for#17264.Fixes#17246.

This does create potential for some additional callback latency (i.e. if an interrupt fires while a different scheduler callback is running, the interrupt callback is now deferred until the next scheduler run). However the worst-case scheduler latency remains similar (interrupt fires at end of one scheduler pass, has to wait until the next one). I think most MicroPython systems only sparsely call C scheduler callbacks, so this shouldn't be noticeable in most cases.

There is also potential for a misbehaving callback that re-schedules itself continuously to degrade performance of MicroPython (as the callback runs continuously with some allowance for Python code to run), whereas before it would have locked up MicroPython entirely which is more obviously broken.

This work was funded through GitHub Sponsors.

Testing

• I adapted the test case@andrewleech added inpy/scheduler: warning about C callbacks scheduling new tasks. #17248 for the new behaviour and resubmitted here, so coverage test now includes C scheduler callbacks (testing the new logic).
• Manually ran the rp2 unit tests on RP2_PICO board. As this is a tickless port I felt it had the most potential for an interrupt timing bug to surface due to this change. All passed.

Trade-offs and Alternatives

• Possible to manually add anti-recursion checks in callbacks, i.e. approach inall/mpbthciport.c: Don't restart BLE polling if already running. #17264, but it can get fiddly quickly.