Abstract

The proposal is to add a new Thread Local Storage (TLS) API to CPython whichwould supersede use of the existing TLS API within the CPython interpreter,while deprecating the existing API. The new API is named the “ThreadSpecific Storage (TSS) API” (seeRationale for Proposed Solution for theorigin of the name).

Because the existing TLS API is only used internally (it is not mentioned inthe documentation, and the header that defines it,pythread.h, is notincluded inPython.h either directly or indirectly), this proposalprobably only affects CPython, but might also affect other interpreterimplementations (PyPy?) that implement parts of the CPython API.

This is motivated primarily by the fact that the old API usesint torepresent TLS keys across all platforms, which is neither POSIX-compliant,nor portable in any practical sense[1].

Specification

The current API for TLS used inside the CPython interpreter consists of 6functions:

PyAPI_FUNC(int)PyThread_create_key(void)PyAPI_FUNC(void)PyThread_delete_key(intkey)PyAPI_FUNC(int)PyThread_set_key_value(intkey,void*value)PyAPI_FUNC(void*)PyThread_get_key_value(intkey)PyAPI_FUNC(void)PyThread_delete_key_value(intkey)PyAPI_FUNC(void)PyThread_ReInitTLS(void)

These would be superseded by a new set of analogous functions:

PyAPI_FUNC(int)PyThread_tss_create(Py_tss_t*key)PyAPI_FUNC(void)PyThread_tss_delete(Py_tss_t*key)PyAPI_FUNC(int)PyThread_tss_set(Py_tss_t*key,void*value)PyAPI_FUNC(void*)PyThread_tss_get(Py_tss_t*key)

The specification also adds a few new features:

• A new typePy_tss_t–an opaque type the definition of which maydepend on the underlying TLS implementation. It is defined:

  typedefstruct{int_is_initialized;NATIVE_TSS_KEY_T_key;}Py_tss_t;

  whereNATIVE_TSS_KEY_T is a macro whose value depends on theunderlying native TLS implementation (e.g.pthread_key_t).

• An initializer forPy_tss_t variables,Py_tss_NEEDS_INIT.
• Three new functions:

  PyAPI_FUNC(Py_tss_t*)PyThread_tss_alloc(void)PyAPI_FUNC(void)PyThread_tss_free(Py_tss_t*key)PyAPI_FUNC(int)PyThread_tss_is_created(Py_tss_t*key)

  The first two are needed for dynamic (de-)allocation of aPy_tss_t,particularly in extension modules built withPy_LIMITED_API, wherestatic allocation of this type is not possible due to its implementationbeing opaque at build time. A value returned byPyThread_tss_alloc isin the same state as a value initialized withPy_tss_NEEDS_INIT, orNULL in the case of dynamic allocation failure. The behavior ofPyThread_tss_free involves callingPyThread_tss_deletepreventively, or is a no-op if the value pointed to by thekeyargument isNULL.PyThread_tss_is_created returns non-zero if thegivenPy_tss_t has been initialized (i.e. byPyThread_tss_create).

The new TSS API does not provide functions which correspond toPyThread_delete_key_value andPyThread_ReInitTLS, because thesefunctions were needed only for CPython’s now defunct built-in TLSimplementation; that is the existing behavior of these functions is treatedas follows:PyThread_delete_key_value(key) is equivalent toPyThread_set_key_value(key,NULL), andPyThread_ReInitTLS() is ano-op[8].

The newPyThread_tss_ functions are almost exactly analogous to theiroriginal counterparts with a few minor differences: WhereasPyThread_create_key takes no arguments and returns a TLS key as anint,PyThread_tss_create takes aPy_tss_t* as an argument andreturns anint status code. The behavior ofPyThread_tss_create isundefined if the value pointed to by thekey argument is not initializedbyPy_tss_NEEDS_INIT. The returned status code is zero on successand non-zero on failure. The meanings of non-zero status codes are nototherwise defined by this specification.

Similarly the otherPyThread_tss_ functions are passed aPy_tss_t*whereas previously the key was passed by value. This change is necessary, asbeing an opaque type, thePy_tss_t type could hypothetically be almostany size. This is especially necessary for extension modules built withPy_LIMITED_API, where the size of the type is not known. Except forPyThread_tss_free, the behaviors ofPyThread_tss_ are undefined if thevalue pointed to by thekey argument isNULL.

Moreover, because of the use ofPy_tss_t instead ofint, there arebehaviors in the new API which differ from the existing API with regard tokey creation and deletion.PyThread_tss_create can be called repeatedlyon the same key–calling it on an already initialized key is a no-op andimmediately returns success. Similarly for callingPyThread_tss_deletewith an uninitialized key.

The behavior ofPyThread_tss_delete is defined to change the key’sinitialization state to “uninitialized”–this allows, for example,statically allocated keys to be reset to a sensible state when restartingthe CPython interpreter without terminating the process (e.g. embeddingPython in an application)[12].

The oldPyThread_*_key* functions will be marked as deprecated in thedocumentation, but will not generate runtime deprecation warnings.

Additionally, on platforms wheresizeof(pthread_key_t)!=sizeof(int),PyThread_create_key will return immediately with a failure status, andthe other TLS functions will all be no-ops on such platforms.

staticPy_tss_ttss_key=Py_tss_NEEDS_INIT;if(PyThread_tss_create(&tss_key)){/*...handlekeycreationfailure...*/}

assert(PyThread_tss_is_created(&tss_key));

int the_value = 1;if (PyThread_tss_get(&tss_key) == NULL) {    PyThread_tss_set(&tss_key, (void *)&the_value);    assert(PyThread_tss_get(&tss_key) != NULL);}/* ... once done with the key ... */PyThread_tss_delete(&tss_key);assert(!PyThread_tss_is_created(&tss_key));

static Py_tss_t *ptr_key = PyThread_tss_alloc();if (ptr_key == NULL) {    /* ... handle key allocation failure ... */}assert(!PyThread_tss_is_created(ptr_key));/* ... once done with the key ... */PyThread_tss_free(ptr_key);ptr_key = NULL;

Platform Support Changes

A new “Native Thread Implementation” section will be added toPEP 11 thatstates:

• As of CPython 3.7, all platforms are required to provide a native threadimplementation (such as pthreads or Windows) to implement the TSSAPI. Any TSS API problems that occur in an implementation without nativethreads will be closed as “won’t fix”.

Motivation

The primary problem at issue here is the type of the keys (int) used forTLS values, as defined by the original PyThread TLS API.

The original TLS API was added to Python by GvR back in 1997, and at thetime the key used to represent a TLS value was anint, and so it hasbeen to the time of writing. This used CPython’s own TLS implementationwhich long remained unused, largely unchanged, in Python/thread.c. Supportfor implementation of the API on top of native thread implementations(pthreads and Windows) was added much later, and the built-in implementationhas been deemed no longer necessary and has since been removed[9].

The problem with the choice ofint to represent a TLS key, is that whileit was fine for CPython’s own TLS implementation, and happens to becompatible with Windows (which usesDWORD for the analogous data), it isnot compatible with the POSIX standard for the pthreads API, which definespthread_key_t as an opaque type not further defined by the standard (aswithPy_tss_t described above)[14]. This leaves it up to the underlyingimplementation how apthread_key_t value is used to look upthread-specific data.

This has not generally been a problem for Python’s API, as it just happensthat on Linuxpthread_key_t is defined as anunsignedint, and so isfully compatible with Python’s TLS API–pthread_key_t’s created bypthread_create_key can be freely cast toint and back (well, notexactly, even this has some limitations as pointed out by issue #22206).

However, as issue #25658 points out, there are at least some platforms(namely Cygwin, CloudABI, but likely others as well) which have otherwisemodern and POSIX-compliant pthreads implementations, but are not compatiblewith Python’s API because theirpthread_key_t is defined in a way thatcannot be safely cast toint. In fact, the possibility of running intothis problem was raised by MvL at the time pthreads TLS was added[2].

It could be argued thatPEP 11 makes specific requirements for supporting anew, not otherwise officially-support platform (such as CloudABI), and thatthe status of Cygwin support is currently dubious. However, this creates avery high barrier to supporting platforms that are otherwise Linux- and/orPOSIX-compatible and where CPython might otherwise “just work” except forthis one hurdle. CPython itself imposes this implementation barrier by wayof an API that is not compatible with POSIX (and in fact makes invalidassumptions about pthreads).

Rationale for Proposed Solution

The use of an opaque type (Py_tss_t) to key TLS values allows the API tobe compatible with all present (POSIX and Windows) and future (C11?) nativeTLS implementations supported by CPython, as it allows the definition ofPy_tss_t to depend on the underlying implementation.

Since the existing TLS API has been available inthe limited API[13] forsome platforms (e.g. Linux), CPython makes an effort to provide the new TSSAPI at that level likewise. Note, however, that thePy_tss_t definitionbecomes to be an opaque struct whenPy_LIMITED_API is defined, becauseexposingNATIVE_TSS_KEY_T as part of the limited API would prevent usfrom switching native thread implementation without rebuilding extensionmodules.

A new API must be introduced, rather than changing the function signatures ofthe current API, in order to maintain backwards compatibility. The new APIalso more clearly groups together these related functions under a single nameprefix,PyThread_tss_. The “tss” in the name stands for “thread-specificstorage”, and was influenced by the naming and design of the “tss” API that ispart of the C11 threads API[15]. However, this is in no way meant to implycompatibility with or support for the C11 threads API, or signal any futureintention of supporting C11–it’s just the influence for the naming and design.

The inclusion of the special initializerPy_tss_NEEDS_INIT is requiredby the fact that not all native TLS implementations define a sentinel valuefor uninitialized TLS keys. For example, on Windows a TLS key isrepresented by aDWORD (unsignedint) and its value must be treatedas opaque[3]. So there is no unsigned integer value that can be safelyused to represent an uninitialized TLS key on Windows. Likewise, POSIXdoes not specify a sentinel for an uninitializedpthread_key_t, insteadrelying on thepthread_once interface to ensure that a given TLS key isinitialized only once per-process. Therefore, thePy_tss_t typecontains an explicit._is_initialized that can indicate the key’sinitialization state independent of the underlying implementation.

ChangingPyThread_create_key to immediately return a failure status onsystems using pthreads wheresizeof(int)!=sizeof(pthread_key_t) isintended as a sanity check: Currently,PyThread_create_key may reportinitial success on such systems, but attempts to use the returned key arelikely to fail. Although in practice this failure occurs earlier in theinterpreter initialization, it’s better to fail immediately at the source ofproblem (PyThread_create_key) rather than sometime later when use of aninvalid key is attempted. In other words, this indicates clearly that theold API is not supported on platforms where it cannot be used reliably, andthat no effort will be made to add such support.

Rejected Ideas

• Do nothing: The status quo is fine because it works on Linux, and platformswishing to be supported by CPython should follow the requirements ofPEP 11. As explained above, while this would be a fair argument ifCPython were being to asked to make changes to support particular quirksor features of a specific platform, in this case it is a quirk of CPythonthat prevents it from being used to its full potential on otherwisePOSIX-compliant platforms. The fact that the current implementationhappens to work on Linux is a happy accident, and there’s no guaranteethat this will never change.
• Affected platforms should just configure Python--without-threads:this is no longer an option as the--without-threads option hasbeen removed for Python 3.7[16].
• Affected platforms should use CPython’s built-in TLS implementationinstead of a native TLS implementation: This is a more acceptablealternative to the previous idea, and in fact there had been a patch to dojust that[4]. However, the built-in implementation being “slower andclunkier” in general than native implementations still needlessly hobblesperformance on affected platforms. At least one other module(tracemalloc) is also broken if Python is built without a native TLSimplementation. This idea also cannot be adopted because the built-inimplementation has since been removed.
• Keep the existing API, but work around the issue by providing a mapping frompthread_key_t values toint values. A couple attempts were made atthis ([5],[6]), but this injects needless complexity and overheadinto performance-critical code on platforms that are not currently affectedby this issue (such as Linux). Even if use of this workaround were madeconditional on platform compatibility, it introduces platform-specific codeto maintain, and still has the problem of the previous rejected ideas ofneedlessly hobbling performance on affected platforms.

Implementation

An initial version of a patch[7] is available on the bug tracker for thisissue. Since the migration to GitHub, its development has continued in thepep539-tss-api feature branch[10] in Masayuki Yamamoto’s fork of theCPython repository on GitHub. A work-in-progress PR is available at[11].

This reference implementation covers not only the new API implementationfeatures, but also the client code updates needed to replace the existingTLS API with the new TSS API.

Copyright

This document has been placed in the public domain.

References and Footnotes