Please work on a single PR to reduce maintainer review burden. Once we are settled that we wish to make this change, we can revisit the other ones.

Will do. For context, there isa proposal to deprecate support for single-phase initialisation in favour of multi-phase. We're working toimprove the documentation here too.

A

Does this impact performance, either import time or runtime?

Local timings usingpython -Ximporttime -c 'import numpy.linalg._umath_linalg' 2>&1| grep _umath_linalg from an editable installation ofnumpy.

No significant impact of multi-phase on import time. I will investigate running the benchmarks (my laptop is ill-suited to it, though), but there should be no real impact -- this only changes the import process, the end result (initialised module) should be identical performance-wise.

A

I don't understand the PyPy-onlynumpy/tests/test_ctypeslib.py::TestAsArray::test_pointer failure.

Local timings usingpython -Ximporttime -c 'import numpy.linalg._umath_linalg' 2>&1| grep _umath_linalg from an editable installation ofnumpy.

Note that you cannot use editable installs for measuring import time. You're mostly measuringimportlib traversable overhead and invokingninja then.

For a regular install, I'm seeing the following timings on a macOS arm64 M1 machine over 15 invocations of your-Ximporttime invocation (cumulative time for_umath_linalg:

• Single-phase init: 680 - 870 us
• Multi-phase init: 700 - 910 us

Total cumulative import time is around 65 ms.

No significant impact of multi-phase on import time.

Yep, that holds up.

Please work on a single PR to reduce maintainer review burden. Once we are settled that we wish to make this change, we can revisit the other ones.

Will do.

Thanks for working on this! It'd be useful to convert the PRs other than the couple that are under active review to Draft status, to make it clear at a glance to other reviewers browsing the PR list what is safe to ignore.

In reply to@rgommers' comment in#29025 (comment), centralising discussion in this PR.

This change looks quite annoying:

-    import_array();+    import_array1(-1);

import_array is public API and used by a ton of extension modules, binding generators, and docs.

At a glance from your PRs, there seems to be a larger issue in the CPython design with the normal meaning of aNULL return used for an error condition no longer being true. Returning0 on success and-1 on error is going to lead to a lot of churn. What is going on there?

This is because the signature changes fromPyObject *PyInit_<NAME>(void) toint module_exec(PyObject *module), soNULL is not a valid return value &-1 is used to signal errors.

I was not around when the API was designed, though in the Python C API we do frequently use-1 as an error sentinel.@encukou /@scoder / etc may be better placed to comment as authors ofPEP 489, but it may just be that it was the most pragmatic choice at the time.

It may be worth considering a shorter alias forimport_array1(-1) assuming no better solution is found, but this is clearly a bigger discussion on NumPy's API and not one I intend to start!

A

I had never looked up the definition ofimport_array() in NumPy, assuming it was a simple function that signaled errors in its return value. It's not:

Luckily, however, itis a function for Cython code:

That mitigates the issue, I guess. A (very) large chunk of the code that interacts with NumPy's C-API probably does it through Cython, not bare C code.

the signature changes fromPyObject *PyInit_<NAME>(void) toint module_exec(PyObject *module)

Right. In the old mechanism, the init function created the module and returned it. In multi-phase init, there is a dedicated creation function that returns it (or an error indicated byNULL), and an exec function (to populate the module namespace) that receives the already created module and returnsonly an error indicator (because it has no objects to return).

It may be worth considering a shorter alias for import_array1(-1) assuming no better solution is found,

We have a better solution already, which is the explicit thing that isn't a macro that includes a return:

    if (PyArray_ImportNumPyAPI() < 0) {        return -1;    }

please just use that (downstream can use it if compiling with NumPy 2, or just vendor it).

Thanks for the pointer, I've switched toPyArray_ImportNumPyAPI() andPyUFunc_ImportUFuncAPI().

A

xrefhttps://discuss.python.org/t/proposal-officially-deprecate-support-for-legacy-single-phase-init-extension-modules/89262/44, which points out that the current docs on multi-phase init say:

All modules created using multi-phase initialization are expected to supportsub-interpreters. Making sure multiple modules are independent is typically enough to achieve this.

Given the opt-out option, I assume it's a matter of those docs needing to be improved, but as it stands this PR conflicts with that requirement so it'd be nice to see that Discourse thread concluding and the docs fixed before we merge this.

Given the opt-out option, I assume it's a matter of those docs needing to be improved, but as it stands this PR conflicts with that requirement so it'd be nice to see that Discourse thread concluding and the docs fixed before we merge this.

For reference, the relevant PR ispython/cpython#134775. Feel free to comment there with any thoughts. The proposed text currently says "All modules created using multi-phase initialization are expected to support sub-interpreters, orotherwise explicitly signal a lack of support." (emph. added).

A

From Petr above:

The PyMutex stuff in the updated HOWTO is 3.13+ non-limited API, but isn't necessary in the medium term (on CPython you're protected by the GIL; even free-threading builds use an import lock which isn't likely to go away). For the long-term (or alternate CAPI implementations) you could add no-op shims.

I chose not to add it as it's only relevant for CPython & the GIL is held during import, so I saw it as needless complexity.

I'm happy to add it you'd prefer to follow the documentation more closely. What would you prefer@seberg?

A

I chose not to add it as it's only relevant for CPython & the GIL is held during import, so I saw it as needless complexity.

Sorry, missed that, happy to not do that then.

Thanks all for the reviews! I'll rebase the others once this has merged, do you have a preferred order?

A

Do we need to wait for clarification around the current requirement that "All modules created using multi-phase initialization are expected to support sub-interpreters" andpython/cpython#134775, which has not been merged yet?

If it provides solace,PEP 630 (2020) and the Python documentationsince 3.11 both contain the text on explicit opt-out that was used as a guide here:

A non-negativePyModuleDef.m_size signals that a module supports multiple interpreters correctly. If this is not yet the case for your module, you can explicitly make your module loadable only once per process.

A

The proposed text currently says "All modules created using multi-phase initialization are expected to support sub-interpreters, orotherwise explicitly signal a lack of support." (emph. added).

Seems perfectly clear now, thanks! Glad to see this PR merged so smoothly.