Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork249
Parallel builds support#392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
zanieb commentedNov 16, 2024
Cool thanks for looking into this! |
74a9dc0 to76cdbd9Compareindygreg commentedNov 17, 2024
The 3.13 CI jobs aren't yet fully optimized and already this PR drops Linux builder total time from ~64h to ~43h, or ~67% of baseline. Not bad. |
I'm about to enable support for parallel CPython builds. Thisfunction was currently not safe if called in parallel. This wascausing setuptools / pip downloads to race writing to the same fileand failing builds.
This is less ergonomic (oh Make). But it will make it easier tobuild multiple Python versions at the same time in a future commit.
Previously we generated the same Makefile for every Python version.We can simplify things by only writing 1 Makefile per targetconfiguration.
This refactor allows targets for each major Python version to bedefined simultaneously. This opens the door to a single `make`invocation building multiple versions in parallel.
`build-main.py` now passes in space delimited Python versions tobuild, which the Makefile can happily translate to appropriate maketargets for us.We added support for `--python all` to allow building all supportedPython versions.
Our build system supports this now. So update CI to make use ofthe feature.On Linux, 3.13 freethreaded builds are still their own CI job. Wemay need to tweak `build-main.py` to allow intelligently selectingfreethreaded compatible versions. We could potentially also allowbuilding freethreaded in parallel with non-freethreaded. This can bedone as a followup.
76cdbd9 tod17e9dcCompareindygreg commentedDec 7, 2024
The macOS builds should be fixed. Issue turned out to be a compatibility issue changed in GNU Make 3.82 (macOS ships 3.81). Specifically commit 81f3e4babd128f6740d05b371122762924522fb6 (because I bisected it).
|
indygreg commentedDec 8, 2024
Ok. CI came back clean. macOS CI total time drops from 10:27 to 8:49. 529/627=84% of baseline. Not the percentage savings as Linux. And 3.13 is still on the table for optimization. But still a nice win. |
zanieb commentedDec 18, 2024
My changes in#428 will make this a little more complicated, but it's probably still worth pursuing. I'm happy to investigate it. |
Uh oh!
There was an error while loading.Please reload this page.
This is an attempt to support building multiple Python versions at the same time. The impetus for this is reducing the amount of work that CI does by amortizing the costs for building common dependencies.
The first 3 commits should be relatively safe to merge. 4 and 5 need a bit of testing.
Open issues: