Non-Python Threads Always Hang During Finalization

Many large libraries might need to call Python code in highly asynchronoussituations where the desired interpreter could be finalizing or deleted, butwant to continue running code after invoking the interpreter. This desire has beenbrought up by users.For example, a callback that wants to call Python code might be invoked when:

• A kernel has finished running on a GPU.
• A network packet was received.
• A thread has quit, and a native library is executing static finalizers forthread-local storage.

Generally, this pattern would look something like this:

staticvoidsome_callback(void*closure){/* Do some work *//* ... */PyGILState_STATEgstate=PyGILState_Ensure();/* Invoke the C API to do some computation */PyGILState_Release(gstate);/* ... */}

This means that any non-Python thread may be terminated at any point, whichseverely limits users who want to do more than just execute Pythoncode in their stream of calls.

Locks in Native Extensions Can Be Unusable During Finalization

When acquiring locks in a native API, it’s common to release the GIL (orcritical sections on the free-threaded build) to avoid lock-ordering deadlocks.This can be problematic during finalization, because threads holding locks mightbe hung. For example:

1. A thread goes to acquire a lock, first detaching its thread state to avoiddeadlocks.
2. The main thread begins finalization and tells all thread states to hangupon attachment.
3. The thread acquires the lock it was waiting on, but then hangs while attemptingto reattach its thread state viaPy_END_ALLOW_THREADS.
4. The main thread can no longer acquire the lock, because the thread holding ithas hung.

This affects CPython itself, and there’s not much that can be doneto fix it with the current API. For example,python/cpython#129536remarks that thessl module can emit a fatal error when used atfinalization, because a daemon thread got hung while holding the lockforsys.stderr, and then a finalizer tried to write to it.Ideally, a thread should be able to temporarily prevent the interpreterfrom hanging it while it holds the lock.

Finalization Behavior forPyGILState_Ensure Cannot Change

There will always have to be a point in a Python program wherePyGILState_Ensure() can no longer attach a thread state.If the interpreter is long dead, then Python obviously can’t give athread a way to invoke it.PyGILState_Ensure() doesn’t have anymeaningful way to return a failure, so it has no choice but to terminatethe thread or emit a fatal error, as noted inpython/cpython#124622:

I think a new GIL acquisition and release C API would be needed. The waythe existing ones get used in existing C code is not amenible to suddenlybolting an error state onto; none of the existing C code is written thatway. After the call they always just assume they have the GIL and canproceed. The API was designed as “it’ll block and only return once it hasthe GIL” without any other option.

As a result, CPython can’t make any real changes to howPyGILState_Ensure()works during finalization, because it would break existing code.

The Term “GIL” Is Tricky for Free-threading

A significant issue with the term “GIL” in the C API is that it is semanticallymisleading. This was noted inpython/cpython#127989,created by the author of this PEP:

The biggest issue is that for free-threading, there is no GIL, so userserroneously call the C API insidePy_BEGIN_ALLOW_THREADS blocks oromitPyGILState_Ensure in fresh threads.

Again,PyGILState_Ensure() gets an attached thread state for thethread on both with-GIL and free-threaded builds. An attached thread state isalways needed to call the C API, soPyGILState_Ensure() still needsto be called on free-threaded builds, but with a name like “ensure GIL”, it’snot immediately clear that that’s true.

PyGILState_Ensure Doesn’t Guess the Correct Interpreter

As noted in thedocumentation,thePyGILState functions aren’t officially supported in subinterpreters:

Note that thePyGILState_* functions assume there is only one globalinterpreter (created automatically byPy_Initialize()). Pythonsupports the creation of additional interpreters (usingPy_NewInterpreter()), but mixing multiple interpreters and thePyGILState_* API is unsupported.

This is becausePyGILState_Ensure() doesn’t have any wayto know which interpreter created the thread, and as such, it has to assumethat it was the main interpreter. There isn’t any way to detect this atruntime, so spurious races are bound to come up in threads created bysubinterpreters, because synchronization for the wrong interpreter will beused on objects shared between the threads.

For example, if the thread had access to object A, which belongs to asubinterpreter, but then calledPyGILState_Ensure(), the thread wouldhave an attached thread state pointing to the main interpreter,not the subinterpreter. This means that any GIL assumptions about theobject are wrong, because there is no synchronization between the two GILs.

There’s no great way to solve this, other than introducing a new API thatexplicitly takes an interpreter from the caller.

Subinterpreters Can Concurrently Deallocate

The other way of creating a non-Python thread,PyThreadState_New() andPyThreadState_Swap(), is a lot better for supporting subinterpreters(becausePyThreadState_New() takes an explicit interpreter, rather thanassuming that the main interpreter was requested), but is still limited by thecurrent hanging problems in the C API, and is subject to crashes when thesubinterpreter finalizes before the thread has a chance to start. This is becausein subinterpreters, thePyInterpreterState* structure is allocated on theheap, whereas the main interpreter is statically allocated on the Python runtimestate.

Preventing Interpreter Shutdown

This PEP takes an approach in which an interpreter includes a guarding APIthat prevents it from shutting down. Holding an interpreter guard ensures it issafe to call the C API without worrying about the thread being hung by finalization.

This means that interfacing with Python (for example, in a C++ library) will needa guard to the interpreter in order to safely call the object, which is moreinconvenient than assuming the main interpreter is the right choice, butthere’s not really another option.

This proposal also comes with “views” to an interpreter that can be used tosafely poke at an interpreter that may be dead or alive. Using a view, userscan create an interpreter guard at any point during its lifecycle, and itwill safely fail if the interpreter can no longer support calling Python code.

Compatibility Shim forPyGILState_Ensure

This proposal comes withPyUnstable_InterpreterView_FromDefault() as acompatibility hack for some users ofPyGILState_Ensure(). It is athread-safe way to create a guard for the main (or “default”)interpreter.

The main drawback to porting new code toPyThreadState_Ensure() is thatit isn’t a drop-in replacement forPyGILState_Ensure(), as it needsan interpreter guard argument. In some large applications, refactoring touse aPyInterpreterGuard everywhere might be tricky, so this functionserves as a last resort for users who explicitly want to disallow support forsubinterpreters.

Interpreter Guards

typePyInterpreterGuard

An opaque interpreter guard.

By holding an interpreter guard, the caller can ensure that the interpreterwill not finalize until the guard is destroyed.

This is similar to a “readers-writers” lock; threads may hold aninterpreter’s guard concurrently, and the interpreter will have to waituntil all threads have destroyed their guards before it can enter finalization.

This type is guaranteed to be pointer-sized.

PyInterpreterGuardPyInterpreterGuard_FromCurrent(void)

Create a finalization guard for the current interpreter.

On success, this function guards the interpreter and returns an opaquereference to the guard; on failure, it returns0 with an exception set.

The caller must hold anattached thread state.

PyInterpreterGuardPyInterpreterGuard_FromView(PyInterpreterViewview)

Create a finalization guard for an interpreter through a view.

On success, this function returns a guard to the interpreterrepresented byview. The view is still valid after calling thisfunction.

If the interpreter no longer exists or cannot safely run Python code,this function returns0 without setting an exception.

The caller does not need to hold anattached thread state.

PyInterpreterState*PyInterpreterGuard_GetInterpreter(PyInterpreterGuardguard)

Return thePyInterpreterState pointer protected byguard.

This function cannot fail, and the caller doesn’t need to hold anattached thread state.

PyInterpreterGuardPyInterpreterGuard_Copy(PyInterpreterGuardguard)

Duplicate an interpreter guard.

On success, this function returns a copy ofguard; on failure, it returns0 without an exception set.

The caller does not need to hold anattached thread state.

voidPyInterpreterGuard_Close(PyInterpreterGuardguard)

Destroy an interpreter guard, allowing the interpreter to enterfinalization if no other guards remain. If an interpreter guardis never closed, the interpreter will infinitely wait when tryingto enter finalization.

This function cannot fail, and the caller doesn’t need to hold anattached thread state.

Interpreter Views

typePyInterpreterView

An opaque view of an interpreter.

This is a thread-safe way to access an interpreter that may be finalizedin another thread.

This type is guaranteed to be pointer-sized.

PyInterpreterViewPyInterpreterView_FromCurrent(void)

Create a view to the current interpreter.

This function is generally meant to be used in tandem withPyInterpreterGuard_FromView().

On success, this function returns a view to the current interpreter; onfailure, it returns0 with an exception set.

The caller must hold anattached thread state.

PyInterpreterViewPyInterpreterView_Copy(PyInterpreterViewview)

Duplicate a view to an interpreter.

On success, this function returns a non-zero copy ofview; on failure,it returns0 without an exception set.

The caller does not need to hold anattached thread state.

voidPyInterpreterView_Close(PyInterpreterViewview)

Delete an interpreter view. If an interpreter view is never closed, theview’s memory will never be freed.

This function cannot fail, and the caller doesn’t need to hold anattached thread state.

PyInterpreterViewPyUnstable_InterpreterView_FromDefault()

Create a view for an arbitrary “main” interpreter.

This function only exists for exceptional cases where a specific interpretercan’t be saved.

On success, this function returns a view to the maininterpreter; on failure, it returns0 without an exception set.

The caller does not need to hold anattached thread state.

Ensuring and Releasing Thread States

This proposal includes two new high-level threading APIs that intend toreplacePyGILState_Ensure() andPyGILState_Release().

typePyThreadView

An opaque view of athread state.

In this PEP, a thread view provides no additional properties beyond aPyThreadState* pointer. However, APIs forPyThreadView maybe added in the future.

This type is guaranteed to be pointer-sized.

PyThreadViewPyThreadState_Ensure(PyInterpreterGuardguard)

Ensure that the thread has anattached thread state for theinterpreter protected byguard, and thus can safely invoke thatinterpreter. It is OK to call this function if the thread already has anattached thread state, as long as there is a subsequent call toPyThreadState_Release() that matches this one.

Nested calls to this function will only sometimes create a newthread state. If there is no attached thread state,then this function will check for the most recent attached threadstate used by this thread. If none exists or it doesn’t matchguard,a new thread state is created. If it does matchguard, it is reattached.If there is an attached thread state, then a similar check occurs;if the interpreter matchesguard, it is attached, and otherwise a newthread state is created.

Return a non-zero thread view of the old thread state on success, and0 on failure.

voidPyThreadState_Release(PyThreadViewview)

Release aPyThreadState_Ensure() call. If this function isnot called, the thread state created byPyThreadState_Ensure(),if any, will leak.

Theattached thread state before the correspondingPyThreadState_Ensure() call is guaranteed to be restored uponreturning. The cached thread state as used (the “GIL-state”), byPyThreadState_Ensure() andPyGILState_Ensure(), will alsobe restored.

This function cannot fail.

Deprecation ofPyGILState APIs

This PEP deprecates all of the existingPyGILState APIs in favor of theexisting and newPyThreadState APIs. Namely:

• PyGILState_Ensure(): usePyThreadState_Ensure() instead.
• PyGILState_Release(): usePyThreadState_Release() instead.
• PyGILState_GetThisThreadState(): usePyThreadState_Get() orPyThreadState_GetUnchecked() instead.
• PyGILState_Check(): usePyThreadState_GetUnchecked()!=NULLinstead.

All of thePyGILState APIs are to be removed from the non-limited C API inPython 3.20. They will remain available in the stable ABI forcompatibility.

Examples

These examples are here to help understand the APIs described in this PEP.They could be reused in the documentation.

intLogToPyFile(PyInterpreterViewview,PyObject*file,PyObject*text){PyInterpreterGuardguard=PyInterpreterGuard_FromView(view);if(guard==0){/* Python interpreter has shut down */return-1;}PyThreadViewthread_view=PyThreadState_Ensure(guard);if(thread_view==0){PyInterpreterGuard_Close(guard);fputs("Cannot call Python.\n",stderr);return-1;}constchar*to_write=PyUnicode_AsUTF8(text);if(to_write==NULL){// Since the exception may be destroyed upon calling PyThreadState_Release(),// print out the exception ourselves.PyErr_Print();PyThreadState_Release(thread_view);PyInterpreterGuard_Close(guard);return-1;}intres=PyFile_WriteString(to_write,file);if(res<0){PyErr_Print();}PyThreadState_Release(thread_view);PyInterpreterGuard_Close(guard);returnres<0;}

staticPyObject*my_critical_operation(PyObject*self,PyObject*Py_UNUSED(args)){assert(PyThreadState_GetUnchecked()!=NULL);PyInterpreterGuardguard=PyInterpreterGuard_FromCurrent();if(guard==0){/* Python interpreter has shut down */returnNULL;}Py_BEGIN_ALLOW_THREADS;acquire_some_lock();/* Do something while holding the lock.       The interpreter won't finalize during this period. */// ...release_some_lock();Py_END_ALLOW_THREADS;PyInterpreterGuard_Close(guard);Py_RETURN_NONE;}

staticintthread_func(void*arg){PyGILState_STATEgstate=PyGILState_Ensure();/* It's not an issue in this example, but we just attached       a thread state for the main interpreter. If my_method() was       originally called in a subinterpreter, then we would be unable       to safely interact with any objects from it. */if(PyRun_SimpleString("print(42)")<0){PyErr_Print();}PyGILState_Release(gstate);return0;}staticPyObject*my_method(PyObject*self,PyObject*unused){PyThread_handle_thandle;PyThead_indent_tindent;if(PyThread_start_joinable_thread(thread_func,NULL,&ident,&handle)<0){returnNULL;}Py_BEGIN_ALLOW_THREADS;PyThread_join_thread(handle);Py_END_ALLOW_THREADS;Py_RETURN_NONE;}

staticintthread_func(void*arg){PyInterpreterGuardguard=(PyInterpreterGuard)arg;PyThreadViewthread_view=PyThreadState_Ensure(guard);if(thread_view==0){PyInterpreterGuard_Close(guard);return-1;}if(PyRun_SimpleString("print(42)")<0){PyErr_Print();}PyThreadState_Release(thread_view);PyInterpreterGuard_Close(guard);return0;}staticPyObject*my_method(PyObject*self,PyObject*unused){PyThread_handle_thandle;PyThead_indent_tindent;PyInterpreterGuardguard=PyInterpreterGuard_FromCurrent();if(guard==0){returnNULL;}if(PyThread_start_joinable_thread(thread_func,(void*)guard,&ident,&handle)<0){PyInterpreterGuard_Close(guard);returnNULL;}Py_BEGIN_ALLOW_THREADSPyThread_join_thread(handle);Py_END_ALLOW_THREADSPy_RETURN_NONE;}

staticintthread_func(void*arg){PyInterpreterGuardguard=(PyInterpreterGuard)arg;PyThreadViewthread_view=PyThreadState_Ensure(guard);if(thread_view==0){PyInterpreterGuard_Close(guard);return-1;}/* Close the interpreter guard, allowing it to       finalize. This means that print(42) can hang this thread. */PyInterpreterGuard_Close(guard);if(PyRun_SimpleString("print(42)")<0){PyErr_Print();}PyThreadState_Release(thread_view);return0;}staticPyObject*my_method(PyObject*self,PyObject*unused){PyThread_handle_thandle;PyThead_indent_tindent;PyInterpreterGuardguard=PyInterpreterGuard_FromCurrent();if(guard==0){returnNULL;}if(PyThread_start_joinable_thread(thread_func,(void*)guard,&ident,&handle)<0){PyInterpreterGuard_Close(guard);returnNULL;}Py_RETURN_NONE;}

typedefstruct{PyInterpreterViewview;}ThreadData;staticintasync_callback(void*arg){ThreadData*tdata=(ThreadData*)arg;PyInterpreterViewview=tdata->view;PyInterpreterGuardguard=PyInterpreterGuard_FromView(view);if(guard==0){fputs("Python has shut down!\n",stderr);return-1;}PyThreadViewthread_view=PyThreadState_Ensure(guard);if(thread_view==0){PyInterpreterGuard_Close(guard);return-1;}if(PyRun_SimpleString("print(42)")<0){PyErr_Print();}PyThreadState_Release(thread_view);PyInterpreterGuard_Close(guard);PyInterpreterView_Close(view);PyMem_RawFree(tdata);return0;}staticPyObject*setup_callback(PyObject*self,PyObject*unused){// View to the interpreter. It won't wait on the callback// to finalize.ThreadData*tdata=PyMem_RawMalloc(sizeof(ThreadData));if(tdata==NULL){PyErr_NoMemory();returnNULL;}PyInterpreterViewview=PyInterpreterView_FromCurrent();if(view==0){PyMem_RawFree(tdata);returnNULL;}tdata->view=view;register_callback(async_callback,tdata);Py_RETURN_NONE;}

staticvoidcall_python(void){PyInterpreterViewview=PyUnstable_InterpreterView_FromDefault();if(guard==0){fputs("Python has shut down.",stderr);return;}PyInterpreterGuardguard=PyInterpreterGuard_FromView(view);if(guard==0){fputs("Python has shut down.",stderr);return;}PyThreadViewthread_view=PyThreadState_Ensure(guard);if(thread_view==0){PyInterpreterGuard_Close(guard);PyInterpreterView_Close(view);return-1;}if(PyRun_SimpleString("print(42)")<0){PyErr_Print();}PyThreadState_Release(thread_view);PyInterpreterGuard_Close(guard);PyInterpreterView_Close(view);return0;}

How Should the APIs Fail?

There is some disagreement over how thePyInterpreter[Guard|View] APIsshould indicate a failure to the caller. There are two competing ideas:

1. Return -1 to indicate failure, and 0 to indicate success. On success,functions will assign to aPyInterpreter[Guard|View] pointer passed as anargument.
2. Directly return aPyInterpreter[Guard|View], with a value of 0 beingequivalent toNULL, indicating failure.

Currently, the PEP spells the latter.

Interpreter Reference Counting

There were two iterations of this proposal that both specified that aninterpreter maintain a reference count and would wait for that count to reachzero before shutting down.

The first iteration of this idea did this by adding implicit reference countingtoPyInterpreterState* pointers. A function known asPyInterpreterState_Holdwould increment the reference count (making it a “strong reference”), andPyInterpreterState_Release would decrement it. An interpreter’s ID (astandaloneint64_t) was used as a form of weak reference, which could beused to look up an interpreter state and atomically increment its referencecount. These ideas were ultimately rejected because they seemed to make thingsvery confusing. All existing uses ofPyInterpreterState* would beborrowed, making it difficult for developers to understand whichparts of their code require or use a strong reference.

In response to that pushback, this PEP specifiedPyInterpreterRef APIsthat would also mimic reference counting, but in a more explicit manner thatmade it easier for developers.PyInterpreterRef was analogous toPyInterpreterGuard in this PEP. Similarly, the older revision includedPyInterpreterWeakRef, which was analogous toPyInterpreterView.

Eventually, the notion of reference counting was completely abandoned fromthis proposal for a few reasons:

1. There was contention over overcomplication in the API design; thereference-counting design looked very similar to HPy’s, which had noprecedent in CPython. There was fear that this proposal was beingovercomplicated to look more like HPy.
2. Unlike traditional reference-counting APIs, acquiring a strong reference toan interpreter could fail at any time, and an interpreter would notbe deallocated immediately when its reference count reached zero.
3. There was prior discussion about adding “true” reference counting tointerpreters (which would deallocate upon reaching zero), which would havebeen very confusing if there was an existing API in CPython titledPyInterpreterRef that did something different.

Non-daemon Thread States

In earlier revisions of this PEP, interpreter guards were a property ofa thread state rather than a property of an interpreter. This meant thatPyThreadState_Ensure() kept an interpreter guard held, andit was closed upon callingPyThreadState_Release(). A thread statethat had a guard to an interpreter was known as a “non-daemon threadstate.” At first, this seemed like an improvement because it shifted themanagement of a guard’s lifetime to the thread rather than the user, whicheliminated some boilerplate.

However, this ended up making the proposal significantly more complex andhurt the proposal’s goals:

• Most importantly, non-daemon thread states place too much emphasis on daemonthreads as the problem, which made the PEP confusing. Additionally,the phrase “non-daemon” added extra confusion, because non-daemon Pythonthreads are explicitly joined. In contrast, a non-daemon C thread is onlywaited on until it destroys its guard.
• In many cases, an interpreter guard should outlive a singular threadstate. Stealing the interpreter guard inPyThreadState_Ensure()was particularly troublesome for these cases. IfPyThreadState_Ensure()didn’t steal a guard with non-daemon thread states, it would muddy theownership story of the interpreter guard, leading to a more confusing API.

Exposing anActivate/Deactivate API Instead ofEnsure/Release

In prior discussions of this API, it wassuggested to provide actualPyThreadState pointers in the API in an attempt tomake the ownership and lifetime of the thread state more straightforward:

More importantly though, I think this makes it clearer who owns the threadstate - a manually created one is controlled by the code that created it,and once it’s deleted it can’t be activated again.

This was ultimately rejected for two reasons:

• The proposed API has closer usage toPyGILState_Ensure() &PyGILState_Release(), which helpsease the transition for old codebases.
• It’ssignificantly easierfor code-generators like Cython to use, as there isn’t any additionalcomplexity with trackingPyThreadState pointers around.

UsingPyStatus for the Return Value ofPyThreadState_Ensure

In prior iterations of this API,PyThreadState_Ensure() returned aPyStatus instead of an integer to denote failures, which had thebenefit of providing an error message.

This was rejected because it’snot clearthat an error message would be all that useful; all the conceived use-casesfor this API wouldn’t really care about a message indicating why Pythoncan’t be invoked. As such, the API would only be needlessly more complex touse, which in turn would hurt the transition fromPyGILState_Ensure().

In addition,PyStatus isn’t commonly used in the C API. A fewfunctions related to interpreter initialization use it (simply because theycan’t raise exceptions), andPyThreadState_Ensure() does not fallunder that category.