Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork34.1k
Closed
Description
Bug report
Bug description:
ThreadSanitizer detects a data race between the main thread clearing the interpreter during finalization and daemon threads attempting to acquire the GIL. The race occurs on_PyRuntime.ceval.eval_breaker at address that is accessed by both:
- Main thread (write):
interpreter_clear()→ writes to eval_breaker - Daemon thread T7 (atomic write):
take_gil()→_Py_unset_eval_breaker_bit()→ atomic operation on the same address
This appears to be related to but distinct from#124878, as the race involvesinterpreter_clear rather thanfree_threadstate, and occurs duringtake_gil rather thandrop_gil.
How to Reproduce
Build configuration:
CC=clang CXX=clang++ ./configure --with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes --disable-optimizations --with-lto=full
make -j$(nproc)Run command:
./python -X dev -X showrefcount bug.py
Test script (bug.py):
importthreadingfromfunctoolsimportlru_cache@lru_cache(None)deffib(n):"""Function that JIT will compile"""ifn<2:returnnreturnfib(n-1)+fib(n-2)defhammer_jit():"""Call the function many times to force JIT compilation"""foriinrange(1_000_000):fib(i)# Create a race: all threads simultaneously force JIT compilationthreads= [threading.Thread(target=hammer_jit)for_inrange(16)]fortinthreads:t.start()fortinthreads:t.join()
TSAN Output
Details
➜ main git:(main) ✗ ./python -X dev -X showrefcount bug.py^C^CException ignored in the internal traceback machinery:^CTraceback (most recent call last): File "<frozen importlib._bootstrap>", line 1379, in _find_and_load File "<frozen importlib._bootstrap>", line 1341, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1267, in _find_spec File "<frozen importlib._bootstrap_external>", line 1304, in find_spec File "<frozen importlib._bootstrap_external>", line 1278, in _get_spec File "<frozen importlib._bootstrap_external>", line 1410, in find_spec File "<frozen importlib._bootstrap_external>", line 166, in _path_isfile File "<frozen importlib._bootstrap_external>", line 158, in _path_is_mode_type File "<frozen importlib._bootstrap_external>", line 152, in _path_statKeyboardInterrupt:Traceback (most recent call last): File "/home/shamil/oss/cpython/main/bug.py", line 18, in <module> for t in threads: t.start() File "/home/shamil/oss/cpython/main/Lib/threading.py", line 1126, in join^Cobject address : 0x7ffff5d41c50object refcount : 3object type : 0x555556087868object type name: KeyboardInterruptobject repr : KeyboardInterrupt()lost sys.stderr^CException ignored on threading shutdown:Traceback (most recent call last): File "/home/shamil/oss/cpython/main/Lib/threading.py", line 1577, in _shutdown _thread_shutdown() ~~~~~~~~~~~~~~~~^^KeyboardInterrupt==================WARNING: ThreadSanitizer: data race (pid=22017) Write of size 8 at 0x55555611abc8 by main thread: #0 interpreter_clear /home/shamil/oss/cpython/main/Python/pystate.c:769:26 (python+0x6cd57a) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #1 _PyInterpreterState_Clear /home/shamil/oss/cpython/main/Python/pystate.c:904:5 (python+0x699f80) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #2 finalize_interp_clear /home/shamil/oss/cpython/main/Python/pylifecycle.c:1920:5 (python+0x699f80) #3 _Py_Finalize /home/shamil/oss/cpython/main/Python/pylifecycle.c:2337:5 (python+0x6965a1) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #4 _Py_Finalize /home/shamil/oss/cpython/main/Python/pylifecycle.c:2337:5 (python+0x6965a1) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #5 Py_FinalizeEx /home/shamil/oss/cpython/main/Python/pylifecycle.c:2378:12 (python+0x72569b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #6 Py_RunMain /home/shamil/oss/cpython/main/Modules/main.c:774:9 (python+0x72569b) #7 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #8 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #9 Py_BytesMain /home/shamil/oss/cpython/main/Modules/main.c:826:12 (python+0x180145) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #10 main /home/shamil/oss/cpython/main/./Programs/python.c:15:12 (python+0x180145) #11 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37) #12 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37) Previous atomic write of size 8 at 0x55555611abc8 by thread T7: #0 _Py_atomic_and_uintptr /home/shamil/oss/cpython/main/./Include/cpython/pyatomic_gcc.h:218:10 (python+0x5c6cbe) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #1 _Py_unset_eval_breaker_bit /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:350:5 (python+0x5c6cbe) #2 take_gil /home/shamil/oss/cpython/main/Python/ceval_gil.c:351:21 (python+0x5c6cbe) #3 _PyEval_AcquireLock /home/shamil/oss/cpython/main/Python/ceval_gil.c:588:5 (python+0x6d152e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #4 _PyThreadState_Attach /home/shamil/oss/cpython/main/Python/pystate.c:2100:9 (python+0x6d152e) #5 _Py_HandlePending /home/shamil/oss/cpython/main/Python/ceval_gil.c:1411:9 (python+0x5c8754) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #6 _Py_HandlePending /home/shamil/oss/cpython/main/Python/ceval_gil.c:1411:9 (python+0x5c8754) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #7 check_periodics /home/shamil/oss/cpython/main/Python/ceval_macros.h:412:16 (python+0x53b92b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #8 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:10296:31 (python+0x53b92b) #9 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #10 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7) #11 _PyFunction_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c (python+0x285b04) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #12 _PyObject_VectorcallTstate.3425 /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x28c0f7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #13 method_vectorcall /home/shamil/oss/cpython/main/Objects/classobject.c:73:20 (python+0x28bc07) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #14 _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x586544) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #15 context_run /home/shamil/oss/cpython/main/Python/context.c:728:29 (python+0x586544) #16 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:3710:35 (python+0x5139a5) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #17 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #18 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7) #19 _PyFunction_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c (python+0x285b04) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #20 _PyObject_VectorcallTstate.3425 /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:169:11 (python+0x28c0f7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #21 method_vectorcall /home/shamil/oss/cpython/main/Objects/classobject.c:73:20 (python+0x28bc07) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #22 _PyVectorcall_Call /home/shamil/oss/cpython/main/Objects/call.c:273:16 (python+0x2854fd) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #23 _PyObject_Call /home/shamil/oss/cpython/main/Objects/call.c:348:16 (python+0x285764) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #24 PyObject_Call /home/shamil/oss/cpython/main/Objects/call.c:373:12 (python+0x828572) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #25 thread_run /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:387:21 (python+0x828572) #26 pythread_wrapper /home/shamil/oss/cpython/main/Python/thread_pthread.h:234:5 (python+0x709130) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #27 pythread_wrapper /home/shamil/oss/cpython/main/Python/thread_pthread.h:234:5 (python+0x709130) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) Location is global '_PyRuntime' of size 359232 at 0x5555560c3408 (python+0xbc6bc8) Thread T7 'Thread-7 (hamme' (tid=22032, running) created by main thread at: #0 pthread_create <null> (python+0xfc21e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #1 do_start_joinable_thread /home/shamil/oss/cpython/main/Python/thread_pthread.h:281:14 (python+0x709041) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #2 PyThread_start_joinable_thread /home/shamil/oss/cpython/main/Python/thread_pthread.h:323:9 (python+0x828011) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #3 ThreadHandle_start /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:474:9 (python+0x828011) #4 do_start_new_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:1920:9 (python+0x828011) #5 thread_PyThread_start_joinable_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:2043:14 (python+0x825e58) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #6 thread_PyThread_start_joinable_thread /home/shamil/oss/cpython/main/./Modules/_threadmodule.c:2043:14 (python+0x825e58) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #7 cfunction_call /home/shamil/oss/cpython/main/Objects/methodobject.c:564:18 (python+0x354c1b) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #8 _PyObject_MakeTpCall /home/shamil/oss/cpython/main/Objects/call.c:242:18 (python+0x283b90) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #9 _PyObject_VectorcallTstate /home/shamil/oss/cpython/main/./Include/internal/pycore_call.h:167:16 (python+0x2838b4) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #10 PyObject_Vectorcall /home/shamil/oss/cpython/main/Objects/call.c:327:12 (python+0x51093d) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #11 _PyEval_EvalFrameDefault /home/shamil/oss/cpython/main/Python/generated_cases.c.h:3188:35 (python+0x51093d) #12 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #13 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7) #14 _PyEval_EvalFrame /home/shamil/oss/cpython/main/./Include/internal/pycore_ceval.h:121:16 (python+0x4fa5b7) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #15 _PyEval_Vector /home/shamil/oss/cpython/main/Python/ceval.c:2001:12 (python+0x4fa5b7) #16 PyEval_EvalCode /home/shamil/oss/cpython/main/Python/ceval.c:884:21 (python+0x4f9e99) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #17 run_eval_code_obj /home/shamil/oss/cpython/main/Python/pythonrun.c:1365:12 (python+0x6d7f60) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #18 run_mod /home/shamil/oss/cpython/main/Python/pythonrun.c:1459:19 (python+0x6d7f60) #19 pyrun_file /home/shamil/oss/cpython/main/Python/pythonrun.c:1293:15 (python+0x6d6adb) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #20 _PyRun_SimpleFileObject /home/shamil/oss/cpython/main/Python/pythonrun.c:521:13 (python+0x6d6adb) #21 _PyRun_AnyFileObject /home/shamil/oss/cpython/main/Python/pythonrun.c:81:15 (python+0x6d5736) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #22 pymain_run_file_obj /home/shamil/oss/cpython/main/Modules/main.c:410:15 (python+0x727676) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #23 pymain_run_file /home/shamil/oss/cpython/main/Modules/main.c:429:15 (python+0x727676) #24 pymain_run_python /home/shamil/oss/cpython/main/Modules/main.c:691:21 (python+0x72634e) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #25 Py_RunMain /home/shamil/oss/cpython/main/Modules/main.c:772:5 (python+0x72634e) #26 pymain_main /home/shamil/oss/cpython/main/Modules/main.c:802:12 (python+0x728d18) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #27 Py_BytesMain /home/shamil/oss/cpython/main/Modules/main.c:826:12 (python+0x180145) (BuildId: 9a73f9de8b0c2b3baa8fe75cb55cd39bff0057a1) #28 main /home/shamil/oss/cpython/main/./Programs/python.c:15:12 (python+0x180145) #29 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37) #30 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x2a574) (BuildId: bc071219b774e887d2477b96fa792af74144ff37)SUMMARY: ThreadSanitizer: data race /home/shamil/oss/cpython/main/Python/pystate.c:769:26 in interpreter_clear==================[4156 refs, 1966 blocks]Analysis
The race occurs because:
- Main thread starts finalization and calls
interpreter_clear()which writes to_PyRuntime.ceval.eval_breaker - Meanwhile, daemon threads are still running and attempt to acquire the GIL via
take_gil() take_gil()calls_Py_unset_eval_breaker_bit()which performs atomic operations on the sameeval_breakerfield- These accesses are not properly synchronized during the finalization phase
Environment
- CPython version: main branch (commit: [fbf0843])
- Operating System: Linux | Ubuntu 25.10
- Compiler: Ubuntu clang version 20.1.8 (0ubuntu4)
- Build flags:
--with-thread-sanitizer --with-pydebug --enable-experimental-jit=yes
Related Issues
- race condition in threading when interpreter finalized while daemon thread runs (thread sanitizer identified) #124878 - Similar race during finalization but with
drop_gilandfree_threadstate - test__xxsubinterpreters is Occasionally Crashing #104341 - Background on interpreter deletion during finalization
CPython versions tested on:
3.15, CPython main branch
Operating systems tested on:
Linux