GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)

many jobs are failing :)

Yes, looks harder than what I anticipated ^^'

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

In many cases, a test is correct and should pass – but the feature that's being tested is implemented incorrectly. In those cases, "should_panic" is just wrong. For example,test_bre11 is perfectly fine, just our implementation is buggy.

Arguably,ignore isn'tcorrect, but it's a better way thanshould_panic to deal with the fact that it's a known issue.

Arguably,ignore isn'tcorrect, but it's a better way thanshould_panic to deal with the fact that it's a known issue.

I understand, but I tend to disagree. Ignoring a test will skip it no matter what, so if we were to fix it by accident, we wouldn't notice and could regress it without even knowing it. Using#[should_panic] allows us to detect when a test weexpect to fail (be it because of upstream bugs, or non-implemented features) actually passes.

I agree that bothignore andshould_panic are semantically incorrect. At my work place, we have a testsuite framework that acceptsPASS,FAIL,SKIP, andXFAIL (expected fail) for a test result, and one can specify the reason behind the expected fail. I wanted to replicate that XFAIL within the coreutils¸ but even though, Rust's framework will still reportPASS for a test that successfully panics, when I'd want "XFAIL" or something similar.

In any cases, I don't have a very strong feeling about this. I think it could help, but I may be mistaken.

Regarding the expr'sbre11 testcase, the issue actually comes down to a bug in the oniguruma bindings crate. Dunno if we want to look for an alternative now that it was archived, but that's a matter for another issue.

Yup, you convinced me: Detecting a "fix by accident" is helpful, I'm in favor of it, now. Is there a way to make sure it's easily distinguishable from "really should panic", instead of "currently panics, and if it ever doesn't, then plz change this, because panicking is actually wrong".

Maybe something like this?

#[should_panic] // Should NOT panic! This test documents known issue #XXXX.

Maybe something like this?

#[should_panic] // Should NOT panic! This test documents known issue #XXXX.

Agreed 👍