Code size report:

   bare-arm:    +0 +0.000% minimal x86:    +0 +0.000%    unix x64:    +0 +0.000% standard      stm32:    +0 +0.000% PYBV10     mimxrt:    +0 +0.000% TEENSY40        rp2:    +0 +0.000% RPI_PICO_W       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS  qemu rv32:    +0 +0.000% VIRT_RV32

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.54%. Comparing base(f5d10c3) to head(dea3035).
Report is 3 commits behind head on master.

@@           Coverage Diff           @@##           master   #17330   +/-   ##=======================================  Coverage   98.54%   98.54%           =======================================  Files         169      169             Lines       21898    21898           =======================================  Hits        21579    21579             Misses        319      319

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report?Share it here.

I reviewed this PR and it looks fine.
The testruns look much better - this is the newest one:https://reports.octoprobe.org/github_selfhosted_testrun_1/octoprobe_summary_report.html

I would propose to merge this PR!

In the summary, you write:

This applies#16513 and#16616 (made for mpremote) to tools/pyboard.py.

It looks like a typo: 16513 should be 16531!

However, there is still a situation where the test time out with aOSError: [Errno 5] Input/output error. See:

• https://reports.octoprobe.org/github_selfhosted_testrun_1/RUN-TESTS_STANDARD_NATIVE@472b-ESP32_S3_DEVKIT/testresults.txt
• https://reports.octoprobe.org/github_selfhosted_testrun_1/RUN-TESTS_STANDARD_NATIVE@1830-LOLIN_C3_MINI/testresults.txt
• https://reports.octoprobe.org/github_selfhosted_testrun_1/RUN-TESTS_STANDARD_NATIVE@0c30-ESP32_C3_DEVKIT/testresults.txt
• https://reports.octoprobe.org/github_selfhosted_testrun_1/RUN-TESTS_STANDARD_NATIVE@2731-RPI_PICO2-RISCV/testresults.txt

The problem was most probably that python multiprocessing terminated the processed and unpowered the tentacle. However, the terminating process still could see the powerdown and this is the ioctl error we see in above logs.

This behaviour was fixed inoctoprobe/testbed_micropython@02ce3a6.
Now we may clearly see that the processes time out:

* https://reports.octoprobe.org/github_selfhosted_testrun_9/RUN-TESTS_STANDARD_NATIVE@2731-RPI_PICO2/testresults.txt* https://reports.octoprobe.org/github_selfhosted_testrun_9/RUN-TESTS_STANDARD_NATIVE@2731-RPI_PICO2-RISCV/testresults.txt

The second link might be related to this PR: The test was hanging without even executing one test.

https://reports.octoprobe.org/github_selfhosted_testrun_13/RUN-TESTS_EXTMOD_HARDWARE_NATIVE@2731-RPI_PICO2-RISCV/testresults.txt

This command still timeouts after 90s as pyboard hangs somehow.
@dpgeorge : Do you have an ideal where pyboard hangs? Or how to get a stacktrace at the moment of timeout..

I was unable to reproduce that error, even with exactly the same hardware (I think?) and using thefirmware.uf2 downloaded from that octoprobe run.

Anyway, I could force a similar error by making the board lock up during a test run. That lead to a lock-up duringserial.write(). So I've now added awrite_timeout=5 parameter to the serial initialisation, and caught such errors duringenter_raw_repl(). My testing shows that this helps to catch such cases of the board locking up, although it's still quite slow because it will try to run every test after the lock-up and each one takes about 10 seconds. Then it takes another 30 seconds or so to even exitrun-tests.py due to the serial port having locked up.

This is not really an issue we can fix, because it's the OS that is blocking, waiting for the USB serial port to respond, and it never does respond.

@hmaerki I tried to trigger a run of this PR, but failed:https://reports.octoprobe.org/github_testbed_micropython_15

Can you please test this PR out (now with the addedwrite_timeout=5 bit)?

@hmaerki I tried to trigger a run of this PR, but failed:https://reports.octoprobe.org/github_testbed_micropython_15

Can you please test this PR out (now with the addedwrite_timeout=5 bit)?

I started this job#16 for you:https://github.com/octoprobe/testbed_micropython/actions/runs/15342563400

Thanks for triggering the new run. It seems to have worked, output is here:https://reports.octoprobe.org/github_selfhosted_testrun_16

It shows that the problem with test timeout is now resolved:

• top-level test log shows it took 64 seconds and didn't timeout:https://reports.octoprobe.org/github_selfhosted_testrun_16/RUN-TESTS_EXTMOD_HARDWARE_NATIVE@2731-RPI_PICO2-RISCV/logger_20_info.log
• test result output shows all 4 tests failed (at least they didn't lock up):https://reports.octoprobe.org/github_selfhosted_testrun_16/RUN-TESTS_EXTMOD_HARDWARE_NATIVE@2731-RPI_PICO2-RISCV/testresults.txt
• the first test failed due to "timeout waiting for first EOF reception", and the remaining three tests all failed due to "could not enter raw repl: Write timeout" (which is a new error made visible by the latest change to this PR)

In summary:

• the "write timeout" introduced in this PR is needed
• even with a write timeout, the failing tests still take a while because the USB serial is broken
• I don't know why the test is failing, locally it works for me

Started another batchhttps://reports.octoprobe.org/github_selfhosted_testrun_18 and got timeouts:https://reports.octoprobe.org/github_selfhosted_testrun_18/RUN-TESTS_STANDARD_NATIVE@2731-RPI_PICO2-RISCV/basics_0prelim.py.out

Intersting observation: Inhttps://reports.octoprobe.org/github_selfhosted_testrun_18/RUN-TESTS_STANDARD_VIA_MPY@2731-RPI_PICO2-RISCV/testresults.txt

pass  micropython/kbd_intr.py FAIL  micropython/native_closure.py (timeout waiting for first EOF reception)

From now on, all tests fail! It is probably the RPI_PICO2-RISCV which locks up!

testresults/RUN-TESTS_STANDARD_NATIVE@2731-RPI_PICO2-RISCV/testresults.txtpass  thread/stress_recurse.py FAIL  thread/stress_schedule.py (timeout waiting for first EOF reception)testresults/RUN-TESTS_STANDARD_NATIVE@2731-RPI_PICO2/testresults.txtpass  thread/stress_recurse.py FAIL  thread/stress_schedule.py (timeout waiting for first EOF reception)

In both cases, onlythread/stress_schedule.py failed.

@dpgeorge: To me, this MR is resolved as pyboard does always recover from timeout! Please merge into master.

From now on, all tests fail! It is probably the RPI_PICO2-RISCV which locks up!

Yes, I have seen this happen with this PR, if the device locks up then all tests from then on fail. That's an improvement though! Previouslyrun-tests.py would just halt forever. At least now it makes progress.