According to NEWS when these were added, they were unstable. Presumably because they mess with internals that might change. Then shouldn't they be named PyUnstable_...? Or,@markshannon have you changed your mind about their status?

It's too late for 3.12 right?

It's too late for 3.12 right?

That's a question for@Yhg1s, Python 3.12 release manager. This change adds 2 functions to the public C API.

I would prefer to fix the API in Python 3.12, but if it's too late, it's too late :-)

If it's too late for Python 3.12, that's ok. The private functions can be used. But I will update my PR to document the change in What's new in Python 3.13 instead ;-) Just tell what you think about this one@Yhg1s ;-)

No, I think it's too late for 3.12.

@gvanrossum:

According to NEWS when these were added, they were unstable. Presumably because they mess with internals that might change. Then shouldn't they be named PyUnstable_...? Or,@markshannon have you changed your mind about their status?

What do you mean when you say that_PyObject_VisitManagedDict() and_PyObject_ClearManagedDict() areunstable? Are you referring to the risk of bugs? Or the risk that the API may change soon?

Are you referring to the What's New in Python 3.12 doc?

* Extension classes wanting to add a ``__dict__`` or weak reference slot  should use :c:macro:`Py_TPFLAGS_MANAGED_DICT` and  :c:macro:`Py_TPFLAGS_MANAGED_WEAKREF` instead of ``tp_dictoffset`` and  ``tp_weaklistoffset``, respectively.  The use of ``tp_dictoffset`` and ``tp_weaklistoffset`` is still  supported, but does not fully support multiple inheritance  (:gh:`95589`), and performance may be worse.  Classes declaring :c:macro:`Py_TPFLAGS_MANAGED_DICT` should call  :c:func:`!_PyObject_VisitManagedDict` and :c:func:`!_PyObject_ClearManagedDict`  to traverse and clear their instance's dictionaries.  To clear weakrefs, call :c:func:`PyObject_ClearWeakRefs`, as before.

The "does not fully support multiple inheritance" part is scary. ThePyTypeObject.tp_dictoffset documentation strongly suggests moving towards thePy_TPFLAGS_MANAGED_DICT API:

.. c:member:: Py_ssize_t PyTypeObject.tp_dictoffset   While this field is still supported, :c:macro:`Py_TPFLAGS_MANAGED_DICT` should be   used instead, if at all possible.   (...)   **Inheritance:**   This field is inherited by subtypes. A subtype should not override this offset;   doing so could be unsafe, if C code tries to access the dictionary at the   previous offset.   To properly support inheritance, use :c:macro:`Py_TPFLAGS_MANAGED_DICT`.

@Yhg1s:

No, I think it's too late for 3.12.

Ok, the release manager spoke, I re-targeted my PR to Python 3.13 :-) It's ok to use the private API in Python 3.12.

What do you mean when you say that_PyObject_VisitManagedDict() and_PyObject_ClearManagedDict() areunstable? Are you referring to the risk of bugs? Or the risk that the API may change soon?

It's literally what theNEWS entry for 3.12a1 says:

.. date: 2022-07-25-15-54-27.. gh-issue: 92678.. nonce: ziZpxz.. section: C APIAdds unstable C-API functions ``_PyObject_VisitManagedDict`` and``_PyObject_ClearManagedDict`` to allow C extensions to allow the VM tomanage their object's dictionaries.

@gvanrossum

Or,@markshannon have you changed your mind about their status?

So@gvanrossum is worried by the Changelog entry adding these 2 APIs as "unstable".

@markshannon: You added these functions 1 year ago in Python 3.12 as commitde388c0. Do you still consider these APIs as "unstable"? Are you ok to make them public? Well, see the issue and previous comments for the rationale ;-)

For me, it's unfortunate that Python 3.12 broke type inheritance with the new "managed dict" change, but doesn't offer a public API to switch to it, while the documentation strongly advices using this new API. That's why I asked to make this API public in Python 3.12.

But this issue is waiting for 1 month, and Python 3.12 release is following its process towards the final release in one month.

It's unclear to me if inheritance was broken in Python 3.11 or 3.12. I recall that pybind11 got a regression related to that, and as I wrote. It now uses a the private_PyObject_GetDictPtr() API to visit and clear the managed dict:#107073 (comment) So it worked around the lack _PyObject_VisitManagedDict() and _PyObject_ClearManagedDict() in Python 3.11.

Well, even if I dislike the implementation, it's good to know thatthere is a way to implement PyObject_VisitManagedDict() and PyObject_ClearManagedDict() on Python 3.11.

It also means that there is a need for public API for that, since pybind11 already uses it :-)

I recall that pybind11 got a regression related to that, and as I wrote. It now uses a the private _PyObject_GetDictPtr() API to visit and clear the managed dict:#107073 (comment)

If we decide addingPyObject_VisitManagedDict() to the public C API, what I can is to provide a public PyObject_VisitManagedDict() function for Python 3.11 and 3.12 in thehttps://github.com/python/pythoncapi-compat/ project using_PyObject_GetDictPtr(). So projects can use the new public API since Python 3.11 :-)

So I don't have to be pushy with our busy Python 3.12 release manager to get it merged after beta1 (after rc1!!!) ;-)

@dean0x7d@wjakob: Hi, you modified pybin11 to travere and clear the dictionary in pybind11 (commit) with such code:

/// dynamic_attr: Allow the garbage collector to traverse the internal instance `__dict__`.extern "C" inline int pybind11_traverse(PyObject *self, visitproc visit, void *arg) {    PyObject *&dict = *_PyObject_GetDictPtr(self);    Py_VISIT(dict);    return 0;}/// dynamic_attr: Allow the GC to clear the dictionary.extern "C" inline int pybind11_clear(PyObject *self) {    PyObject *&dict = *_PyObject_GetDictPtr(self);    Py_CLEAR(dict);    return 0;}

I see that pybind11 uses Py_TPFLAGS_MANAGED_DICT on Python 3.11. Would it help you to have public function PyObject_VisitManagedDict() and PyObject_ClearManagedDict()? Do you have an opinion on the API? Is it good?

The proposal looks fine to me.

A tangent: a more general solution would be to expose the dictionary in a public (perhaps even stable) API interface, but not like the existing mechanism._PyObject_GetDictPtr is problematic because it exposes CPython implementation details. If the signature wasPyObject_GetDict() instead (returningPyObject* and notPyObject **) then that is enough information to find and break reference cycles. Those two pybind11 functions you highlighted could then be implemented as

/// dynamic_attr: Allow the garbage collector to traverse the internal instance `__dict__`.extern"C"inlineintpybind11_traverse(PyObject*self,visitprocvisit,void*arg) {PyObject*dict=PyObject_GetDict(self);Py_VISIT(dict);return0;}/// dynamic_attr: break a reference cycle involving this objectextern"C"inlineintpybind11_clear(PyObject*self) {PyObject*dict=PyObject_GetDict(self);if (dict) {PyDict_Clear(dict);    }return0;}

In short, you ask to make_PyObject_GetDict() public? Yeah, I am curious if it would make sense to make this function public as well. By the way, this function is documented as: "should be treated as part of the public API"!

/* Helper to get a pointer to an object's __dict__ slot, if any. * Creates the dict from inline attributes if necessary. * Does not set an exception. * * Note that the tp_dictoffset docs used to recommend this function, * so it should be treated as part of the public API. */

@vstinner Ah, interesting—I did not know about this function. The fact that it creates the dictionary if non-existent makes it inappropriate for what I had suggested (for use in garbage collection hooks).

Oops, sorry for the noise :-( I tried a Git tool which messed up my PRs. I fixed my PR.

@vstinner Ah, interesting—I did not know about this function. The fact that it creates the dictionary if non-existent makes it inappropriate for what I had suggested (for use in garbage collection hooks).

Can we discuss exposing _PyObject_GetDict() separetly? Exposing just PyObject_VisitManagedDict() and PyObject_ClearManagedDict() already looks controversial :-)

@markshannon@gvanrossum@encukou: this issue is languishing since July 22. I plan to merge my PR next Friday. If you need more time for review, please speak up.

Once this PR will be merged, I consider adding a compatible layer for Python 3.11 and 3.12 in pythoncapi-compat. Later, I consider making _PyObject_GetDict() public, but apparently this function needs to be adjusted before being made public.

I understand that exposing these functions would be helpful to fix garbage collection of managed dictionaries in extension libraries like pybind11. But unless there is a big rush to push this out for a release (which I don't think is the case here, right?), isn't it better to refine and have a perfect API? In this case, it somehow seems suboptimal to expose two highly specific public functions as public API if they are likely to be subsumed by a more general one later one.

it somehow seems suboptimal to expose two highly specific public functions as public API if they are likely to be subsumed by a more general one later one.

I don't think that anyone has a plan to replace these two convenient functions with something like _PyObject_GetDict().

_PyObject_GetDict() cannot be used directly in traverse and clear functions, IMO it's less convenient to use. See my Doc/c-api/typeobj.rst changes in this PR: PyObject_VisitManagedDict() and PyObject_ClearManagedDict() are convenient to use.

For me, _PyObject_GetDict() fits other use cases, no?

Both Mark and I are on vacation this week. Please wait.

Both Mark and I are on vacation this week. Please wait.

So, any update on these functions? Should they be exposed? Is it the new official way to have a dict in an object?

Or should we burry them deeper in the internal C API?

Pinging@markshannon

I've commented on the issue on my thinking as to why we should add these to the public API.

Thanks for the review@gvanrossum and@markshannon, I really wasn't sure about this one.

Now we should visit the _PyObject_GetDict() case, hum.

⚠️⚠️⚠️ Buildbot failure⚠️⚠️⚠️

Hi! The buildbotARM64 macOS 3.x has failed when building commitfc2cb86.

What do you need to do:

1. Don't panic.
2. Checkthe buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/725/builds/5825) and take a look at the build logs.
4. Check if the failure is related to this commit (fc2cb86) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/725/builds/5825

Failed tests:

• test.test_concurrent_futures.test_deadlock

Summary of the results of the build (if available):

==

remote:Enumerating objects: 26, done.remote:Counting objects:   4% (1/24)remote:Counting objects:   8% (2/24)remote:Counting objects:  12% (3/24)remote:Counting objects:  16% (4/24)remote:Counting objects:  20% (5/24)remote:Counting objects:  25% (6/24)remote:Counting objects:  29% (7/24)remote:Counting objects:  33% (8/24)remote:Counting objects:  37% (9/24)remote:Counting objects:  41% (10/24)remote:Counting objects:  45% (11/24)remote:Counting objects:  50% (12/24)remote:Counting objects:  54% (13/24)remote:Counting objects:  58% (14/24)remote:Counting objects:  62% (15/24)remote:Counting objects:  66% (16/24)remote:Counting objects:  70% (17/24)remote:Counting objects:  75% (18/24)remote:Counting objects:  79% (19/24)remote:Counting objects:  83% (20/24)remote:Counting objects:  87% (21/24)remote:Counting objects:  91% (22/24)remote:Counting objects:  95% (23/24)remote:Counting objects: 100% (24/24)remote:Counting objects: 100% (24/24), done.remote:Compressing objects:   6% (1/16)remote:Compressing objects:  12% (2/16)remote:Compressing objects:  18% (3/16)remote:Compressing objects:  25% (4/16)remote:Compressing objects:  31% (5/16)remote:Compressing objects:  37% (6/16)remote:Compressing objects:  43% (7/16)remote:Compressing objects:  50% (8/16)remote:Compressing objects:  56% (9/16)remote:Compressing objects:  62% (10/16)remote:Compressing objects:  68% (11/16)remote:Compressing objects:  75% (12/16)remote:Compressing objects:  81% (13/16)remote:Compressing objects:  87% (14/16)remote:Compressing objects:  93% (15/16)remote:Compressing objects: 100% (16/16)remote:Compressing objects: 100% (16/16), done.remote:Total 26 (delta 8), reused 8 (delta 8), pack-reused 2From https://github.com/python/cpython * branch                  main       -> FETCH_HEADNote:switching to 'fc2cb86d210555d509debaeefd370d5331cd9d93'.You are in 'detached HEAD' state. You can look around, make experimentalchanges and commit them, and you can discard any commits you make in thisstate without impacting any branches by switching back to a branch.If you want to create a new branch to retain commits you create, you maydo so (now or later) by using -c with the switch command. Example:  git switch -c <new-branch-name>Or undo this operation with:  git switch -Turn off this advice by setting config variable advice.detachedHead to falseHEAD is now at fc2cb86d21 gh-107073: Make PyObject_VisitManagedDict() public (#108763)Switched to and reset branch 'main'In file included from ./Modules/_tkinter.c:52:In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]int (*free_private)();/* called to free private storage*/^                            void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        struct _XImage*(*create_image)();^                                        void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        XID (*resource_alloc)();/* allocator function*/^                              void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]int (*synchandler)();/* Synchronization handler*/^                           void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        Bool (*event_vec[128])();/* vectorfor wire to event*/^                               void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        Status (*wire_vec[128])();/* vectorfor event to wire*/^                                void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        Bool (**error_vec)();/* vectorfor wire to error*/^                           void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]int (*savedsynchandler)();/* user synchandler when Xlib usurps*/^                                void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]typedef void (*XIMProc)();^                        voidIn file included from ./Modules/tkappinit.c:17:In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]int (*free_private)();/* called to free private storage*/^                            void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        struct _XImage*(*create_image)();^                                        void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        XID (*resource_alloc)();/* allocator function*/^                              void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]int (*synchandler)();/* Synchronization handler*/^                           void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        Bool (*event_vec[128])();/* vectorfor wire to event*/^                               void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        Status (*wire_vec[128])();/* vectorfor event to wire*/^                                void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]        Bool (**error_vec)();/* vectorfor wire to error*/^                           void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]int (*savedsynchandler)();/* user synchandler when Xlib usurps*/^                                void/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]typedef void (*XIMProc)();^                        void9 warnings generated.9 warnings generated.make:*** [buildbottest] Error 5

@vstinner:

Now we should visit the _PyObject_GetDict() case, hum.

Oh,@markshannon replied that we should not expose it:#107073 (comment)

@wjakob: If you want, you can now use the pythoncapi-project to use PyObject_VisitManagedDict() and PyObject_ClearManagedDict() on Python 3.11 and Python 3.12, implemented using _PyObject_GetDictPtr(), see:python/pythoncapi-compat@a594354

@vstinner — those two functions would be usable for pybind11 (@rwgk,@henryiii).

For nanobind, I'm still using old-style dictionaries built into each instance :-/. I don't this functionality is sufficiently exposed through limited API / stable ABI interfaces to be able to switch to managed dicts. Is it the plan that stable ABI interfaces can also use them?

Is it the plan that stable ABI interfaces can also use them?

If you want a function to be exposed in the limited C API, please open a new issue for that.