Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.3k
gh-126925: Modify how iOS test results are gathered#127592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
!buildbot iOS |
bedevere-bot commentedDec 4, 2024
🤖 New build scheduled with the buildbot fleet by@freakboy3742 for commit60584e0 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
Moving to draft because of the test failures on macOS buils |
!buildbot iOS |
bedevere-bot commentedDec 5, 2024
🤖 New build scheduled with the buildbot fleet by@freakboy3742 for commit89bb435 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
I've now resolved the testing issue; I've also added some documentation for the new capabilities. |
!buildbot iOS |
bedevere-bot commentedDec 9, 2024
🤖 New build scheduled with the buildbot fleet by@freakboy3742 for commit47870c4 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
There's no command line interface; it's currently only exposed as an option for embedded-mode apps.Briefcase's stub binary, for example, would be able to add stdout redirection as part of the app startup sequence, rather than requiring the @ned-deily I've now made those changes, including surpassing all but the first occurrence of the "Messages dropped..." messages in any block when they occur. Did you want to review this again, or are you OK with me to merge and backport the changes? |
LGTM, thanks |
2041a95 intopython:mainUh oh!
There was an error while loading.Please reload this page.
Thanks@freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. |
Sorry,@freakboy3742, I could not cleanly backport this to |
…honGH-127592)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)Co-authored-by: Russell Keith-Magee <russell@keith-magee.com>
GH-127754 is a backport of this pull request to the3.13 branch. |
#127754)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)
…onGH-127592) (python#127754)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)
…honGH-127592) (python#127754)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)
…honGH-127592) (python#127754)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)
…honGH-127592) (python#127754)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)
)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.
…honGH-127592) (python#127754)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.(cherry picked from commit2041a95)
)Adds a `use_system_log` config item to enable stdout/stderr redirection forApple platforms. This log streaming is then used by a new iOS test runnerscript, allowing the display of test suite output at runtime. The iOS testrunner script can be used by any Python project, not just the CPython testsuite.
Uh oh!
There was an error while loading.Please reload this page.
Xcode 16 deprecated the
xcresultooltool we were using to gather test results after an iOS test run.This tool was less than ideal anyway, as it didn't provide a way to see test results as the test suite was running. As a result, we had intermittent CI test failures that manifested as "test suite didn't finish", with no runtime debugging possible.
This PR adds a new configuration item (
config->use_system_logger) for Apple platforms, providing the option to redirect stdout and stderr to the system log. This mirrors how Android operates - the implementation is strongly influenced by the Android implementation, but is just different enough that refactoring into a common base class is more trouble than it's worth IMHO.This PR also adds a new test runner for iOS projects. This test runner uses the iOS testbed as a template, installing the compiled framework into copy generated from the template; and then building and running the test project in parallel with a log streamer that captures the system log from the iOS simulator as it runs. This allows us to observe test output as the test suite runs.
The test runner also includes options to allows the installation of app code. This means the testbed project can be used to test arbitrary Python code, in addition to the core CPython test suite.
Although this feature is primarily useful for iOS apps, the same codecan be used for macOS apps - something that is useful when embedding Python in to a GUI app (where stdout/stderr isn't visible).
This PR resolves both#126925and#126821; Resolving#126821 was a pre-requisite for fixing#126925 (and is the first commit in this PR), but by itself, there's very little opportunity to evaluate that it's working. You need an updated test runner to be able to observe that stdout/err redirection is working.
Fixes#126925
Fixes#126821
📚 Documentation preview 📚:https://cpython-previews--127592.org.readthedocs.build/