Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Parallel builds support#392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
indygreg wants to merge6 commits intomain
base:main
Choose a base branch
Loading
fromgps/parallel-builds
Draft

Parallel builds support#392

indygreg wants to merge6 commits intomainfromgps/parallel-builds

Conversation

@indygreg
Copy link
Collaborator

@indygregindygreg commentedNov 16, 2024
edited
Loading

This is an attempt to support building multiple Python versions at the same time. The impetus for this is reducing the amount of work that CI does by amortizing the costs for building common dependencies.

The first 3 commits should be relatively safe to merge. 4 and 5 need a bit of testing.

Open issues:

  • Race condition downloading pip/setuptools when building in parallel.
  • Handle 3.13 and freethreaded builds better in CI (still room for job consolidation).
  • macOS builds broken due to old version of GNU Make.

@zanieb
Copy link
Member

Cool thanks for looking into this!

@indygreg
Copy link
CollaboratorAuthor

The 3.13 CI jobs aren't yet fully optimized and already this PR drops Linux builder total time from ~64h to ~43h, or ~67% of baseline. Not bad.

zanieb reacted with rocket emoji

I'm about to enable support for parallel CPython builds. Thisfunction was currently not safe if called in parallel. This wascausing setuptools / pip downloads to race writing to the same fileand failing builds.
This is less ergonomic (oh Make). But it will make it easier tobuild multiple Python versions at the same time in a future commit.
Previously we generated the same Makefile for every Python version.We can simplify things by only writing 1 Makefile per targetconfiguration.
This refactor allows targets for each major Python version to bedefined simultaneously. This opens the door to a single `make`invocation building multiple versions in parallel.
`build-main.py` now passes in space delimited Python versions tobuild, which the Makefile can happily translate to appropriate maketargets for us.We added support for `--python all` to allow building all supportedPython versions.
Our build system supports this now. So update CI to make use ofthe feature.On Linux, 3.13 freethreaded builds are still their own CI job. Wemay need to tweak `build-main.py` to allow intelligently selectingfreethreaded compatible versions. We could potentially also allowbuilding freethreaded in parallel with non-freethreaded. This can bedone as a followup.
@indygreg
Copy link
CollaboratorAuthor

The macOS builds should be fixed. Issue turned out to be a compatibility issue changed in GNU Make 3.82 (macOS ships 3.81). Specifically commit 81f3e4babd128f6740d05b371122762924522fb6 (because I bisected it).

define syntax in GNU Make 3.82 supports an optional=. e.g.define foo =. 3.81 doesn't. I was using the= and it broke 3.81.

@indygreg
Copy link
CollaboratorAuthor

Ok. CI came back clean.

macOS CI total time drops from 10:27 to 8:49. 529/627=84% of baseline. Not the percentage savings as Linux. And 3.13 is still on the table for optimization. But still a nice win.

@zanieb
Copy link
Member

My changes in#428 will make this a little more complicated, but it's probably still worth pursuing. I'm happy to investigate it.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@indygreg@zanieb

[8]ページ先頭

©2009-2025 Movatter.jp