Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-127119: Remove check on accidental deallocation of immortal objects for free-threaded build#127120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
eendebakpt wants to merge17 commits intopython:mainfromeendebakpt:small_int_immortal

Conversation

@eendebakpt
Copy link
Contributor

@eendebakpteendebakpt commentedNov 21, 2024
edited
Loading

(Description was updated) The python small ints are immortal with a fixed reference count (also some other types). However, old extension modules can still modify the reference count and end up deallocating these objects.

This should not be an issue on 64-bit builds (even when accidentically changing the reference count it will take a very long time for an object to reach refcount zero) or the free-threaded build (which used a different reference counts, so has no legacy extensions modules)

In this PR we remove the check on accidental deallocation for the free-threading build. This has performance impact for the long type (where thetp_dealloc is actually called often). For the other types we remove it to be consistent, but it should have no performance impact.

Also see:

@skirpichev
Copy link
Contributor

Apparently, assertion is broken in tests.

nineteendo reacted with thumbs up emoji

@eendebakpt
Copy link
ContributorAuthor

Apparently, assertion is broken in tests.

Yes, my mistake. Should be fixed now

ZeroIntensity
ZeroIntensity previously approved these changesNov 22, 2024
Copy link
Member

@ZeroIntensityZeroIntensity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

@ZeroIntensityZeroIntensity added the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelNov 22, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@ZeroIntensity for commit4dc97c9 🤖

If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelNov 22, 2024
@ZeroIntensityZeroIntensity dismissed theirstale reviewNovember 22, 2024 15:12

CI is failing, because apparently you can build with assertions but not with debug enabled (that's what I get for prematurely approving, ha).

ZeroIntensity
ZeroIntensity previously approved these changesNov 22, 2024
Copy link
Member

@ZeroIntensityZeroIntensity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM (again)!

Copy link
Member

@gpsheadgpshead left a comment
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

this changes from silently re-immortalizing the small ints to actually letting them be deallocated in non-debug builds. in theory this is supposed to be fine, but how confident are we of (buggy) reference counting code not relying in the former behavior?

(I'm not blocking this PR, I think the change is acceptable, just want that pondered...)

@ZeroIntensity
Copy link
Member

in theory this is supposed to be fine, but how confident are we of (buggy) reference counting code not relying in the former behavior?

I thought about that too, but Ithink issues around immortal objects being buggy was fixed byGH-125251 (at least, I thought that was implied by the removal of_Py_IsImmortalLoose). Someone should double check though (I guessmaybe it's possible for stable ABI extensions to break things).

@eendebakpt
Copy link
ContributorAuthor

The possibility of de-allocating immortal objects is described in [PEP 683, sectionAccidental De-Immortalizing](https://peps.python.org/pep-0683/#accidental-de-immortalizing). Also see
[Python-Dev] Re: PEP 683: "Immortal Objects, Using a Fixed Refcount" round 3. Even with the changes from#125251 this is still possible. The check inlong_dealloc is therefore not redundant (although it is unlikely). The condiderations from PEP 683 are only 2 years old, so I do not think enough time has passed to reconsider the tradeoffs. I will update the comments in the PR with a pointer to the PEP.

But if I understand the correctly the accidental-de-immortalizing cannot happen on: 64-bit systems, or with the free-threading build, or whenPy_LIMITED_API is not defined. I removed the check for these cases. I will benchmark this as well to see wether this makes a difference.

And yes, this should be double checked!

@eendebakpt
Copy link
ContributorAuthor

Results on a microbenchmark:

bench_long: Mean +- std dev: [main] 194 ns +- 3 ns -> [pr_small_int_check] 188 ns +- 2 ns: 1.03x fasterbench_alloc: Mean +- std dev: [main] 407 us +- 14 us -> [pr_small_int_check] 393 us +- 21 us: 1.03x fasterBenchmark hidden because not significant (1): bench_collatzGeometric mean: 1.02x faster

I expect no overall improvement on the pyperformance benchmark.

Benchmark script
# Quick benchmark for cpython long objectsimport pyperfdef collatz(a):    while a > 1:        if a % 2 == 0:            a = a // 2        else:            a = 3 * a + 1def bench_collatz(loops):    range_it = range(loops)    t0 = pyperf.perf_counter()    for ii in range_it:        collatz(ii)    return pyperf.perf_counter() - t0def bench_long(loops):    range_it = range(loops)    t0 = pyperf.perf_counter()    x = 10    for ii in range_it:        x = x * x        y = x // 2        x = y + ii + x        if x > 10**10:            x = x % 1000    return pyperf.perf_counter() - t0def bench_alloc(loops):    range_it = range(loops)    t0 = pyperf.perf_counter()    for ii in range_it:        for kk in range(20_000):            del kk    return pyperf.perf_counter() - t0# %timeit bench_long(1000)if __name__ == "__main__":    runner = pyperf.Runner()    runner.bench_time_func("bench_collatz", bench_collatz)    runner.bench_time_func("bench_long", bench_long)    runner.bench_time_func("bench_alloc", bench_alloc)

Copy link
Member

@ZeroIntensityZeroIntensity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is now incorrect :(

Comment on lines 3617 to 3618
#ifdefPy_LIMITED_API
#ifndefPy_GIL_DISABLED

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'm not sure it's possible for us to know when we're using the limited API.Py_LIMITED_API is something that affectsPython.h at the compile-time of an extension module, but this happens during the compilation of CPython, so it will never be true.

@markshannon
Copy link
Member

As an alternative to removing the check altogether we could make it cheaper by using the reserved bithttps://github.com/python/cpython/blob/main/Include/cpython/longintrepr.h#L79 as a marker for immortal/small ints.

@eendebakpt
Copy link
ContributorAuthor

@markshannon Thanks for the idea! I think it works and would not be hard to implement, but I am a bit hesitant to use a reserved bit for a small optimization like this. I created an alternative PR with a faster check based on your idea though

gpshead reacted with thumbs up emoji

@markshannon
Copy link
Member

That bit is reserved for exactly this use case, so it is fine to use it.

@eendebakpteendebakpt changed the titlegh-127119: Remove check on small ints in long_deallocgh-127119: Remove check on accidental deallocation of immortal objects for free-threaded buildDec 7, 2024
@eendebakpt
Copy link
ContributorAuthor

Closing in favor of#127620

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@gpsheadgpsheadgpshead left review comments

@Fidget-SpinnerFidget-SpinnerFidget-Spinner left review comments

@ZeroIntensityZeroIntensityZeroIntensity requested changes

@markshannonmarkshannonAwaiting requested review from markshannon

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@eendebakpt@skirpichev@bedevere-bot@ZeroIntensity@markshannon@gpshead@Fidget-Spinner

[8]ページ先頭

©2009-2025 Movatter.jp