Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-136421: Load_datetime static types during interpreter initialization#136583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
ZeroIntensity wants to merge6 commits intopython:main
base:main
Choose a base branch
Loading
fromZeroIntensity:datetime-interp-init

Conversation

ZeroIntensity
Copy link
Member

@ZeroIntensityZeroIntensity commentedJul 12, 2025
edited by bedevere-appbot
Loading

_datetime is a special module, because it's the only non-builtin C extension that contains static types. As such, it would initialize static types in the module's execution function, which can run concurrently. Since static type initialization is not thread-safe, this caused crashes. This fixes it by moving the initialization of_datetime's static types to interpreter startup (where all other static types are initialized), which is already properly protected through other locks.

I have no idea if this will fix the Windows build, but let's hope.
@ZeroIntensityZeroIntensity requested a review froma team as acode ownerJuly 12, 2025 14:22
@neonene
Copy link
Contributor

My concern is unnecessarily losing portability between the builtin module and the non-builtin one.

This PR avoids a crash not by locking but forcing the main interpreter initialize the_datetime static types, which I think is almost equivalent to the following change in_datetimemodule.c:

PyMODINIT_FUNCPyInit__datetime(void){+   if (PyDateTime_DateType.tp_subclasses == NULL) {+       PyInterpreterState *interp = PyInterpreterState_Get();+       // main interp currently checks a module before subinterp imports+       assert(_Py_IsMainInterpreter(interp));+       init_static_types(interp, 0);+   }    return PyModuleDef_Init(&datetimemodule);}

@ZeroIntensity
Copy link
MemberAuthor

Hmm, that feels really hacky.

I wouldn't call the movement of_datetime to static being totally unnecessary, because_datetime exposes a C API via a capsule. If_datetime isn't included with the build of Python, things using the capsule are very likely to crash, so it makes more sense to me to require it. Are there people actually disabling_datetime in their builds?

@neonene
Copy link
Contributor

Hmm, that feels really hacky.

The crash can be avoided even on pure Python level, and managed staticextsension types are already special only for_datetime. The C API is likely to be redesigned in the future, which will be a standard way to shared extensions including_datetime.

Are there people actually disabling _datetime in their builds?

Not sure about disabling. On Windows,_datetime is a built-in module that can be switched to a DLL. I know someone who want to use such a DLL as a plugin.

@ZeroIntensity
Copy link
MemberAuthor

Right, there are several ways to fix this thatwork, but I'm trying to find the fix that is the mostcorrect. Relying on implementation details liketp_subclasses is a little more maintenance if we ever change type initialization, and it also muddies our general rules about how static types should work. I see several options:

  1. Make_datetime static, as it is in this PR. This makes the most sense, but I guess it could compromise compatibility in some niche cases.
  2. Initialize static types in thePyInit_ function. This works, but will probably lead to some other issues because it mixes single and multi-phase initialization, which isn't supposed to be work.
  3. Import_datetime in_interpreters. I really don't like this fix because it won't fix subinterpreters created via the C API.
  4. Import_datetime innew_interpreter. This slows down initialization and is also a bit of a hack.
  5. Make_PyStaticType_InitForExtension thread-safe. I think this will be needlessly complex.
  6. Serialize calls to_PyStaticType_InitForExtension (gh-136421: Fix crash when _datetime is been initialized in multiple sub-interpreters #136422). This works but again hurts the story about static types, and is technically more of a bottleneck.

We have to find a balance between correctness and simplicity, and I'm not yet sure which that is.

@neonene
Copy link
Contributor

  1. Make_datetime static, as it is in this PR. This makes the most sense, but I guess it could compromise compatibility in some niche cases.

This is at least a feature change more than a bug fix.

@ZeroIntensity
Copy link
MemberAuthor

Yeah, you're right. I'd be fine with a bandaid fix for 3.14 if you want to make a PR. ThePyInit_ approach probably works best, but I don't fully understand why we need thetp_subclasses check.

@neonene
Copy link
Contributor

Still rough example, but checkingtp_subclasses is an alternative toPyType_HasFeature(PyDateTime_DateType, Py_TPFLAGS_READY). I'm not sure which is better on free-threading. Both must be kept in the main interpreter in its life, which meansinterp_count is at least 1 in the runtime state and the subinterp shutdown cannot invokefini_static_type() asfinal.

During the_datetime exec phase, the main interpreter probably needs to skipinit_static_types() to not increaseinterp_count.

@ZeroIntensity
Copy link
MemberAuthor

PyType_HasFeature is definitely the better option.tp_subclasses beingNULL is an implementation detail, and free-threading uses non-atomic loads for the type flags (I think it probably uses stop-the-world when modifying). People look to CPython for inspiration on their own C extensions, we should encourage them to use public APIs where possible.

@aisk
Copy link
Member

A small concern is the startup time. Can you do a micro performance benchmark to measure the time cost of./python -m 'pass' several times before and after this change? My expectation is that this will not add observable time cost.

@neonene
Copy link
Contributor

#136620 is ready for review.

Can you do a micro performance benchmark

I haven't come up with how to measure so far.

Comment on lines +763 to +766
status = _PyDateTime_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
return status;
}
Copy link
Contributor

@neoneneneoneneJul 14, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
status=_PyDateTime_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
returnstatus;
}
if (!_Py_IsMainInterpreter(interp)) {
status=_PyDateTime_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
returnstatus;
}
}

Reply from#136620 (comment)

Can you runtest_concurrent_initialization() with this change? Based on the crashes that come from the change, I said this PR and my example are "almost equivalent." I'm not sure right now what this PR ensures.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hm, what are you trying to achieve here? This will just break the types for the main interpreter.

Copy link
Contributor

@neoneneneoneneJul 14, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This will just break the types for the main interpreter.

Note thattest_concurrent_initialization() does not load the_datetime in the main inter interpreter at all.

Correction: Run thescript of the test without runningtest_datetime.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@neoneneneoneneneonene left review comments

@aiskaiskaisk approved these changes

@pgansslepganssleAwaiting requested review from pgansslepganssle is a code owner

@abalkinabalkinAwaiting requested review from abalkinabalkin is a code owner

@ericsnowcurrentlyericsnowcurrentlyAwaiting requested review from ericsnowcurrentlyericsnowcurrently is a code owner

@erlend-aaslanderlend-aaslandAwaiting requested review from erlend-aaslanderlend-aasland is a code owner

Assignees
No one assigned
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3 participants
@ZeroIntensity@neonene@aisk

[8]ページ先頭

©2009-2025 Movatter.jp