@ZeroIntensity: Here is a simplified change for the 3.12 and 3.13 branch. Would you mind to review it?

Ok, I completed my PR to cover most (almost all?) functions, instead of focusing on alloc/realloc/free and Track/Untrack functions.

@ZeroIntensity: Please review the updated PR.

_PyTraceMalloc_GetTraces

Fixing properly_PyTraceMalloc_GetTraces() requires to redesigntracemalloc! For example, currently, thePyMem_Malloc() hook usesTABLES_LOCK() on a reentrant call. In the main branch, the whole _PyTraceMalloc_GetTraces() function is run withTABLES_LOCK(). If we do the same in 3.13, thePyMem_Malloc() hook must be modified to no longer useTABLES_LOCK() on a reentrant call. It would change the behavior (for correctness!), I'm not sure that it's a good thing in a stable branch.

In the main branch, I redesigned the code to be able to hold the lock for all tables operations. For that, I had to modify the behavior:

tracemalloc_realloc() and tracemalloc_free() no longer remove the trace on reentrant call.

I'm open to consider fixing_PyTraceMalloc_GetTraces() by redesigningtracemalloc in 3.13+3.12, but I'm not sure if it's worth it. Callingget_traces() is a deliberate action from the user, it's different from thetracemalloc hooks which called implicitly by running regular Python code.

It looks like tests somehow were broken.

It looks like tests somehow were broken.

Fixed.

Well... Fixing more functions require to backport more changes. I backported the_PyTraceMalloc_Init() change to fix test_capi and test_gc.

🤖 New build scheduled with the buildbot fleet by@ZeroIntensity for commit54c8f9e 🤖

If you want to schedule another build, you need to add the🔨 test-with-buildbots label again.

buildbot/AMD64 FreeBSD Refleaks PR failed

test_tracemalloc_track_race (test.test_tracemalloc.TestCAPI.test_tracemalloc_track_race) ...Debug memory block at address p=0x82647c840: API 'r'    16 bytes originally requested    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.    The 8 pad bytes at tail=0x82647c850 are FORBIDDENBYTE, as expected.    Data at p: 10 e3 a6 42 08 00 00 00 60 c9 47 26 08 00 00 00Enable tracemalloc to get the memory block allocation tracebackFatal Python error: _PyMem_DebugRawFree: bad ID: Allocated using API 'r', verified using API 'P'Python runtime state: initializedThread 0x0000000831196500 (most recent call first):  <no Python frame>(...)

I mark this PR as a draft because of the FreeBSD crash.

Oh, I can also reproduce thetest_tracemalloc_track_race() crash on the main branch on FreeBSD.

So far, I cannot reproduce the crash on Linux.

I can trigger a crash if I insert a delay inPyMem_SetAllocator():

diff --git a/Objects/obmalloc.c b/Objects/obmalloc.cindex b103deb01ca..c20eb47e8a8 100644--- a/Objects/obmalloc.c+++ b/Objects/obmalloc.c@@ -859,7 +859,14 @@ set_allocator_unlocked(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) {     switch(domain)     {-    case PYMEM_DOMAIN_RAW: _PyMem_Raw = *allocator; break;+    case PYMEM_DOMAIN_RAW:+        _PyMem_Raw.ctx = allocator->ctx;+        (void)sched_yield();+        _PyMem_Raw.malloc = allocator->malloc;+        _PyMem_Raw.calloc = allocator->calloc;+        _PyMem_Raw.realloc = allocator->realloc;+        _PyMem_Raw.free = allocator->free;+        break;     case PYMEM_DOMAIN_MEM: _PyMem = *allocator; break;     case PYMEM_DOMAIN_OBJ: _PyObject = *allocator; break;     /* ignore unknown domain */

The write is protected by a lock (write), butPyMem_RawMalloc() andPyMem_RawFree() don't use the lock (read).

Only debug build are affected, not release build. I modified the test in the main branch to skip it on debug build:#128988

Thanks@vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

Sorry,@vstinner, I could not cleanly backport this to3.12 due to a conflict.
Please backport usingcherry_picker on command line.

cherry_picker 6b47499510e47c0401d1f6cca2660fc12c496e39 3.12

GH-129022 is a backport of this pull request to the3.12 branch.