Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-123271: Make builtin zip method safe under free-threading#123272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
colesbury merged 15 commits intopython:mainfromeendebakpt:zip_ft
Aug 27, 2024

Conversation

@eendebakpt
Copy link
Contributor

@eendebakpteendebakpt commentedAug 23, 2024
edited by bedevere-appbot
Loading

In this PR we makezip safe under free-threading.

@eendebakpteendebakpt changed the titlegh-123271: Make builtin zip method safe under free-threadingDraft: gh-123271: Make builtin zip method safe under free-threadingAug 23, 2024
@@ -0,0 +1 @@
Make:meth:`zip` safe under free-threading.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
Make:meth:`zip` safeunder free-threading.
Make:func:`zip`thread-safewithout the:term:`GIL`.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Even with this PR thezip function is not thread-safe: when iterating overzip(range(100), range(100)) one can obtain a tuple(2,3). What does PR (hopefully) does is modifying the code so that the interpreter does not crash. For this reason I would like to be careful with the term "thread-safe". I choose not to makezip fully thread safe because of the performance impact, see the discussion in#120496

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for clarifying. The wordsafe is a bit confusing, I guess a lot of other people would read it asthread-safe. I think we need to clarify the reason whyzip isn't safe infree-threaded build.

eendebakptand others added3 commitsAugust 24, 2024 17:29
Co-authored-by: Kirill Podoprigora <kirill.bast9@mail.ru>
Comment on lines 2974 to 2979
#ifdefPy_GIL_DISABLED
intreuse_tuple=0;
#else
intreuse_tuple=Py_REFCNT(result)==1;
#endif
if (reuse_tuple) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think we want to enable re-use under certain conditions. We should probably put the logic in some internal function, possibly inpycore_object.h. I think the condition for the free-threaded build is:

  1. ob_tid matches_Py_ThreadId()
  2. ob_ref_local is 1
  3. ob_ref_shared is 0

The logic is that no other thread callingzip_next can re-uselz->result because of the thread id condition (condition 1). And no thread outside ofzip_next can incref that object because the combined conditions ensure thatlz->result is the only reference.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@colesbury Thanks for the hint! I implementeded the suggestion

  • I placed the logic in a single method_Py_Reuse_Immutable_Object so we can use the trick code for several other cases (e.g.itertools.pairwise,enumerate). The location and name of the new method can be changed though
  • I re-ordered the three conditions because I suspect thatob_ref_local is most informative and fastest to check
  • I am now checkingob_ref_shared to be zero, but perhaps we should check onPy_ARITHMETIC_RIGHT_SHIFT(Py_ssize_t, ob_ref_shared , _Py_REF_SHARED_SHIFT)?

@eendebakpteendebakpt changed the titleDraft: gh-123271: Make builtin zip method safe under free-threadinggh-123271: Make builtin zip method safe under free-threadingAug 27, 2024
@colesburycolesbury self-requested a reviewAugust 27, 2024 16:45
Copy link
Contributor

@colesburycolesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks@eendebakpt. The check thatob_ref_shared is entirely zero is correct as you wrote it. We don't want non-zero flags in this case as that could allow another thread to concurrently incref the object in some circumstances.

Comment on lines 34 to 35
_= [t.start()fortinworker_threads]
_= [t.join()fortinworker_threads]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The preferred style is to usefor loops instead of using list comprehensions for side effects.

Comment on lines 19 to 21
number_of_threads=8
number_of_iterations=40
n=40_000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We run lots of unit tests in CI, so they individually need to be very fast. For the free-threaded tests, we should aim for <0.1 seconds on two cores (i.e., when run withtaskset -c 0-1 on Linux)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

On my system (windows) the test is 0.07 seconds for the free-threading build. I will reduce the number of threads and iterations a bit so it is even faster.

colesbury reacted with thumbs up emoji
@colesburycolesbury merged commit7e38e67 intopython:mainAug 27, 2024
@colesbury
Copy link
Contributor

Thanks for the fix@eendebakpt!

There's maybe a dozen or so similar patterns in CPython andhas_unique_reference in dictobject.c. Would you be interested in updating them to use_PyObject_IsUniquelyReferenced?

I searched for the regexPy_REFCNT.* == 1, but I probably wouldn't change the matchingassert() statements.

@eendebakpt
Copy link
ContributorAuthor

@colesbury Thanks for the guidance here! I was indeed planning on addressing some of the similar patterns. I'll open an new issue for this.

colesbury reacted with thumbs up emoji

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@Eclips4Eclips4Eclips4 left review comments

@colesburycolesburycolesbury approved these changes

@ericsnowcurrentlyericsnowcurrentlyAwaiting requested review from ericsnowcurrentlyericsnowcurrently is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@eendebakpt@colesbury@Eclips4

[8]ページ先頭

©2009-2025 Movatter.jp