Thanks for this! It's frustrating that Junit doesn't support the "iterations" concept.

My only question - Is there a discernible difference between relying on the last iteration versusany successful iteration period?

For instance, what is the expected outcome when a fixed number of iterations emits mixed results?

Thanks so much for taking the time to review this 🙌

Thanks for this! It's frustrating that Junit doesn't support the "iterations" concept.

Yes 100%. I was looking for a different popular test report format that could support it, but surprisingly I did not find a good candidate. Most iOS tooling relies on junit. Let me know if you know about any alternative :)

Is there a discernible difference between relying on the last iteration versus any successful iteration period?

I would say so. It depends on the expectation a developer has when repeating the execution of a test:

1. To avoid flakiness: I believe the most common case when running a test more than once, is to avoid flakiness. Example: running UITests, test repetition is a must due to the inherent flakiness of such tests. Under this scenario the most representative outcome of multiple repetitions is the result of the last repetition. Xcodebuild allows this with the flag-retry-tests-on-failure
2. To detect flakiness: other common usecase is to run a test multiple times with the expectation of detecting if it is flaky. In this case each test is run X times, or until it fails. Under this scenario the most representative outcome of multiple repetitions is the result of the last repetition. This is supported by xcodebuild with the flag-run-tests-until-failure.

For instance, what is the expected outcome when a fixed number of iterations emits mixed results?

This scenario, which is running each test X times despite the outcome, seems to me not very practical and I am not clear on what usecase it has:

• If the expectation is to show a failed test result if all iterations fail, the option 1 above is more performant
• If the expectation is to show a failed result if any of the iterations fail, the option 2 above is more performant
• If the expectation is to show a passed test result if all iterations pass, the option 2 above is more performant
• If the expectation is to show a passed test result if any of the iterations pass, the option 1 above is more performant

What do you think, are you able to identify any other usecase I am missing?

The way I see it, if we were to make a decision on what iteration is the one that represents better the result of a test with mixed iterations, based on the usecases above I would say is the last iteration

Thank you for the detailed explanation! Completely agree on the two primary use cases for iterations.

FWIW passing-test-iterations without-retry-tests-on-failure or-run-tests-until-failure is what Xcode calls a "fixed number of iterations" in their UI. I agree that it doesn't really make sense here, and especially in the context of a junit report.