The GIL Makes Many Types of Parallelism Difficult to Express

Neural network-based AI models expose multiple opportunities forparallelism. For example, individual operations may be parallelizedinternally (“intra-operator”), multiple operations may be executedsimultaneously (“inter-operator”), and requests (spanning multipleoperations) may also be parallelized. Efficient execution requiresexploiting multiple types of parallelism[1].

The GIL makes it difficult to express inter-operator parallelism, aswell as some forms of request parallelism, efficiently in Python. Inother programming languages, a system might use threads to rundifferent parts of a neural network on separate CPU cores, but this isinefficient in Python due to the GIL. Similarly, latency-sensitiveinference workloads frequently use threads to parallelize acrossrequests, but face the same scaling bottlenecks in Python.

The challenges the GIL poses to exploiting parallelism in Pythonfrequently come up in reinforcement learning. Heinrich Kuttler,author of the NetHack Learning Environment and Member of TechnicalStaff at Inflection AI, writes:

Recent breakthroughs in reinforcement learning, such as onDota2,StarCraft, andNetHack rely on running multipleenvironments (simulated games) in parallel using asynchronousactor-critic methods. Straightforward multithreaded implementationsin Python don’t scale beyond more than a few parallel environmentsdue to GIL contention. Multiprocessing, with communication viashared memory or UNIX sockets, adds much complexity and in effectrules out interacting with CUDA from different workers, severelyrestricting the design space.

Manuel Kroiss, software engineer at DeepMind on the reinforcementlearning team, describes how the bottlenecks posed by the GIL lead torewriting Python codebases in C++, making the code less accessible:

We frequently battle issues with the Python GIL at DeepMind. In manyof our applications, we would like to run on the order of 50-100threads per process. However, we often see that even with fewerthan 10 threads the GIL becomes the bottleneck. To work around thisproblem, we sometimes use subprocesses, but in many cases theinter-process communication becomes too big of an overhead. Todeal with the GIL, we usually end up translating large parts of ourPython codebase into C++. This is undesirable because it makes thecode less accessible to researchers.

Projects that involve interfacing with multiple hardware devices facesimilar challenges: efficient communication requires use of multipleCPU cores. TheDose-3D project aims to improve cancerradiotherapy with precise dose planning. It uses medical phantoms(stand-ins for human tissue) together with custom hardware and aserver application written in Python. Paweł Jurgielewicz, leadsoftware architect for the data acquisition system on the Dose-3Dproject, describes the scaling challenges posed by the GIL and howusing a fork of Python without the GIL simplified the project:

In the Dose-3D project, the key challenge was to maintain a stable,non-trivial concurrent communication link with hardware units whileutilizing a 1 Gbit/s UDP/IP connection to the maximum. Naturally,we started with the multiprocessing package, but at some point, itbecame clear that most CPU time was consumed by the data transfersbetween the data processing stages, not by data processing itself.The CPython multithreading implementation based on GIL was a deadend too. When we found out about the “nogil” fork of Python it tooka single person less than half a working day to adjust the codebaseto use this fork and the results were astonishing. Now we can focuson data acquisition system development rather than fine-tuning dataexchange algorithms.

Allen Goodman, author ofCellProfiler and staff engineer atPrescient Design and Genentech, describes how the GIL makesbiological methods research more difficult in Python:

Issues with Python’s global interpreter lock are a frequent sourceof frustration throughout biological methods research.

I wanted to better understand the current multithreading situationso I reimplemented parts ofHMMER, a standard method formultiple-sequence alignment. I chose this method because itstresses both single-thread performance (scoring) andmulti-threaded performance (searching a database of sequences). TheGIL became the bottleneck when using only eight threads. This is amethod where the current popular implementations rely on 64 oreven 128 threads per process. I tried moving to subprocesses butwas blocked by the prohibitive IPC costs. HMMER is a relativelyelementary bioinformatics method and newer methods have far biggermulti-threading demands.

Method researchers are begging to use Python (myself included),because of its ease of use, the Python ecosystem, and because “it’swhat people know.” Many biologists only know a little bit ofprogramming (and that’s almost always Python). Until Python’smultithreading situation is addressed, C and C++ will remain thelingua franca of the biological methods research community.

The GIL Affects Python Library Usability

The GIL is a CPython implementation detail that limits multithreadedparallelism, so it might seem unintuitive to think of it as ausability issue. However, library authors frequently care a greatdeal about performance and will design APIs that support workingaround the GIL. These workaround frequently lead to APIs that aremore difficult to use. Consequently, users of these APIs mayexperience the GIL as ausability issue and not just a performanceissue.

For example, PyTorch exposes a multiprocessing-based API calledDataLoader for building data input pipelines. It usesfork()on Linux because it is generally faster and uses less memorythanspawn(), but this leads to additional challenges for users:creating aDataLoader after accessing a GPU can lead to confusingCUDA errors. Accessing GPUs within aDataLoader worker quicklyleads to out-of-memory errors because processes do not share CUDAcontexts (unlike threads within a process).

Olivier Grisel, scikit-learn developer and software engineer at Inria,describes how having to work around the GIL in scikit-learn relatedlibraries leads to a more complex and confusing user experience:

Over the years, scikit-learn developers have maintained ancillarylibraries such asjoblib andloky to try to work around someof the limitations of multiprocessing: extra memory usage partiallymitigated via semi-automated memory mapping of large data buffers,slow worker startup by transparently reusing a pool of longrunning workers, fork-safety problems of third-party native runtimelibraries such as GNU OpenMP by never using the fork-onlystart-method, ability to perform parallel calls of interactivelydefined functions in notebooks and REPLs in cross-platform mannervia cloudpickle. Despite our efforts, this multiprocessing-basedsolution is still brittle, complex to maintain and confusing todatascientists with limited understanding of system-levelconstraints. Furthermore, there are still irreducible limitationssuch as the overhead caused by the pickle-basedserialization/deserialization steps required for inter-processcommunication. A lot of this extra work and complexity would not beneeded anymore if we could use threads without contention onmulticore hosts (sometimes with 64 physical cores or more) to rundata science pipelines that alternate between Python-leveloperations and calls to native libraries.

Ralf Gommers, co-director of Quansight Labs and NumPy and SciPymaintainer, describes how the GIL affects the user experience ofNumPy and numeric Python libraries:

A key problem in NumPy and the stack of packages built around it isthat NumPy is still (mostly) single-threaded — and that has shapedsignificant parts of the user experience and projects built aroundit. NumPy does release the GIL in its inner loops (which do theheavy lifting), but that is not nearly enough. NumPy doesn’t offera solution to utilize all CPU cores of a single machine well, andinstead leaves that to Dask and other multiprocessing solutions.Those aren’t very efficient and are also more clumsy to use. Thatclumsiness comes mainly in the extra abstractions and layers theusers need to concern themselves with when using, e.g.,dask.array which wrapsnumpy.ndarray. It also shows up inoversubscription issues that the user must explicitly be aware ofand manage via either environment variables or a third package,threadpoolctl. The main reason is that NumPy calls into BLASfor linear algebra - and those calls it has no control over, theydo use all cores by default via either pthreads or OpenMP.

Coordinating on APIs and design decisions to control parallelism isstill a major amount of work, and one of the harder challengesacross the PyData ecosystem. It would have looked a lot different(better, easier) without a GIL.

GPU-Heavy Workloads Require Multi-Core Processing

Many high-performance computing (HPC) and AI workloads make heavy useof GPUs. These applications frequently require efficient multi-coreCPU execution even though the bulk of the computation runs on a GPU.

Zachary DeVito, PyTorch core developer and researcher at FAIR(Meta AI), describes how the GIL makes multithreaded scalinginefficient even when the bulk of computation is performed outside ofPython:

In PyTorch, Python is commonly used to orchestrate ~8 GPUs and ~64CPU threads, growing to 4k GPUs and 32k CPU threads for big models.While the heavy lifting is done outside of Python, the speed ofGPUs makes even just the orchestration in Python not scalable. Weoften end up with 72 processes in place of one because of the GIL.Logging, debugging, and performance tuning are orders-of-magnitudemore difficult in this regime, continuously causing lower developerproductivity.

The use of many processes (instead of threads) makes common tasks moredifficult. Zachary DeVito continues:

On three separate occasions in the past couple of months(reducing redundant compute in data loaders, writing modelcheckpoints asynchronously, and parallelizing compileroptimizations), I spent an order-of-magnitude more time figuringout how to work around GIL limitations than actually solving theparticular problem.

Even GPU-heavy workloads frequently have a CPU-intensive component.For example, computer vision tasks typically requiremultiple “pre-processing” steps in the data input pipeline, likeimage decoding, cropping, and resizing. These tasks are commonlyperformed on the CPU and may use Python libraries likePilloworPillow-SIMD. It is necessary to run the data input pipelineon multiple CPU cores in order to keep the GPU “fed” with data.

The increase in GPU performance compared to individual CPU cores makesmulti-core performance more important. It is progressively moredifficult to keep the GPUs fully occupied. To do so requires efficientuse of multiple CPU cores, especially on multi-GPU systems. Forexample, NVIDIA’s DGX-A100 has 8 GPUs and two 64-core CPUs in order tokeep the GPUs “fed” with data.

The GIL Makes Deploying Python AI Models Difficult

Python is widely used to develop neural network-based AI models. InPyTorch, models are frequently deployed as part of multi-threaded,mostly C++, environments. Python is often viewed skepticallybecause the GIL can be a global bottleneck, preventing efficientscaling even though the vast majority of the computationsoccur “outside” of Python with the GIL released. The torchdeploypaper[2] shows experimental evidence for these scalingbottlenecks in multiple model architectures.

PyTorch provides a number of mechanisms for deploying Python AImodels that avoid or work around the GIL, but they all come withsubstantial limitations. For example,TorchScript captures arepresentation of the model that can be executed from C++ without anyPython dependencies, but it only supports a limited subset of Pythonand often requires rewriting some of the model’s code. Thetorch::deploy APIallows multiple Python interpreters, each with its own GIL, in thesame process(similar toPEP 684). However,torch::deploy haslimited support for Python modules that use C-API extensions.

Motivation Summary

Python’s global interpreter lock makes it difficult to use modernmulti-core CPUs efficiently for many scientific and numeric computingapplications. Heinrich Kuttler, Manuel Kroiss, and PawełJurgielewicz found that multi-threaded implementations in Python didnot scale well for their tasks and that using multiple processeswas not a suitable alternative.

The scaling bottlenecks are not solely in core numeric tasks. BothZachary DeVito and Paweł Jurgielewicz described challenges withcoordination and communication in Python.

Olivier Grisel, Ralf Gommers, and Zachary DeVito described how currentworkarounds for the GIL are “complex to maintain” and cause “lowerdeveloper productivity.” The GIL makes it more difficult to developand maintain scientific and numeric computing libraries as wellleading to library designs that are more difficult to use.

Build Configuration Changes

The global interpreter lock will remain the default for CPython buildsand python.org downloads. A new build configuration flag,--disable-gil will be added to the configure script that will buildCPython with support for running without the global interpreter lock.

When built with--disable-gil, CPython will define thePy_GIL_DISABLEDmacro in Python/patchlevel.h. The ABI tag will include the letter “t”(for “threading”).

The--disable-gil builds of CPython will still support optionallyrunning with the GIL enabled at runtime (seePYTHONGIL EnvironmentVariable andPy_mod_gil Slot).

Overview of CPython Changes

Removing the global interpreter lock requires substantial changes toCPython internals, but relatively few changes to the public Pythonand C APIs. This section describes the required changes to theCPython implementation followed by the proposed API changes.

The implementation changes can be grouped into the following fourcategories:

• Reference counting
• Memory management
• Container thread-safety
• Locking and atomic APIs

Reference Counting

Removing the GIL requires changes to CPython’sreference counting implementation to make it thread-safe.Furthermore, it needs to have low execution overhead and allow forefficient scaling with multiple threads. This PEP proposes acombination of three techniques to address these constraints. Thefirst is a switch from plain non-atomic reference counting to biasedreference counting, which is a thread-safe reference countingtechnique with lower execution overhead than plain atomic referencecounting. The other two techniques are immortalization and a limitedform of deferred reference counting; they address some of themulti-threaded scalability issues with reference counting by avoidingsome reference count modifications.

Biased reference counting (BRC) is a technique first described in 2018by Jiho Choi, Thomas Shull, and Josep Torrellas[3]. It is based on theobservation that most objects are only accessed by a single thread,even in multi-threaded programs. Each object is associated with anowning thread (the thread that created it). Reference countingoperations from the owning thread use non-atomic instructions tomodify a “local” reference count. Other threads use atomicinstructions to modify a “shared” reference count. This design avoidsmany atomic read-modify-write operations that are expensive oncontemporary processors.

The implementation of BRC proposed in this PEP largely matches theoriginal description of biased reference counting, but differs indetails like the size of reference counting fields and special bits inthose fields. BRC requires storing three pieces of information in eachobject’s header: the “local” reference count, the “shared” referencecount, and the identifier of the owning thread. The BRC paper packsthese three things into a single 64-bit field. This PEP proposes usingthree separate fields in each object’s header to avoid potential issuesdue to reference count overflow. Additionally, the PEP supports afaster deallocation path that avoids an atomic operation in the commoncase.

The proposedPyObject struct (also calledstruct_object) isbelow:

struct_object{_PyObject_HEAD_EXTRAuintptr_tob_tid;// owning thread id (4-8 bytes)uint16_t__padding;// reserved for future use (2 bytes)PyMutexob_mutex;// per-object mutex (1 byte)uint8_tob_gc_bits;// GC fields (1 byte)uint32_tob_ref_local;// local reference count (4 bytes)Py_ssize_tob_ref_shared;// shared reference count and state bits (4-8 bytes)PyTypeObject*ob_type;};

Theob_tid,ob_ref_local, andob_ref_shared are used bythe biased reference counting implementation. Theob_gc_bits fieldis used store garbage collection flags that were previously stored inPyGC_Head (seeGarbage Collection (Cycle Collection)). Theob_mutex field provides a per-object lock in a single byte.

// low two bits of "ob_ref_shared" are used for flags#define _Py_SHARED_SHIFT 2voidPy_INCREF(PyObject*op){uint32_tnew_local=op->ob_ref_local+1;if(new_local==0)return;// object is immortalif(op->ob_tid==_Py_ThreadId())op->ob_ref_local=new_local;elseatomic_add(&op->ob_ref_shared,1<<_Py_SHARED_SHIFT);}voidPy_DECREF(PyObject*op){if(op->ob_ref_local==_Py_IMMORTAL_REFCNT){return;// object is immortal}if(op->ob_tid==_Py_ThreadId()){op->ob_ref_local-=1;if(op->ob_ref_local==0){_Py_MergeZeroRefcount();// merge refcount}}else{_Py_DecRefShared();// slow path}}void_Py_MergeZeroRefcount(PyObject*op){if(op->ob_ref_shared==0){// quick deallocation code path (common case)op->ob_tid=0;_Py_Dealloc(op);}else{// slower merging path not shown}}

Memory Management

CPython currently uses an internal allocator, pymalloc, which isoptimized for small object allocation. The pymalloc implementation isnot thread-safe without the GIL. This PEP proposes replacing pymallocwith mimalloc, a general-purpose thread-safe allocator with goodperformance, including for small allocations.

Using mimalloc, with some modifications, also addresses two otherissues related to removing the GIL. First, traversing the internalmimalloc structures allows the garbage collector to find all Pythonobjects without maintaining a linked list. This is described in moredetail in the garbage collection section. Second, mimalloc heaps andallocations based on size class enable collections like dict togenerally avoid acquiring locks during read-only operations. This isdescribed in more detail in the collection thread-safety section.

CPython already requires that objects that support garbage collectionuse the GC allocator APIs (typically indirectly by callingPyType_GenericAlloc). This PEP would add additional requirementsto the use of the Python allocator APIs. First, Python objects mustbe allocated through object allocation APIs, such asPyType_GenericAlloc,PyObject_Malloc, or other Python APIsthat wrap those calls. Python objects should not be allocated throughother APIs, such as raw calls to C’s malloc or the C++ new operator.Additionally,PyObject_Malloc should be used only for allocatingPython objects; it should not be used for allocating buffers,storages, or other data structures that are not PyObjects.

This PEP also imposes restrictions on the pluggable allocator API(PyMem_SetAllocator). When compiling without the GIL, allocatorsset using this API must eventually delegate the allocation to thecorresponding underlying allocator, such asPyObject_Malloc, forPython object allocations. This allows for allocators that “wrap”underlying allocators, such as Python’s tracemalloc and debugallocator, but not for wholly replacing the allocator.

Garbage Collection (Cycle Collection)

The CPython garbage collector requires the following changes to workwith this proposal:

• Use of “stop-the-world” to provide thread-safety guarantees thatwere previously provided by the GIL.
• Elimination of generational garbage collection in favor ofnon-generational collector.
• Integration with deferred reference counting and biased referencecounting.

Additionally, the above changes enable removing the_gc_prev and_gc_next fields from GC objects. The GC bitsthat stored the tracked, finalized, and unreachable states are movedto theob_gc_bits field in the PyObject header.

Container Thread-Safety

In CPython, the global interpreter lock protects against corruption ofinternal interpreter states when multiple threads concurrently accessor modify Python objects. For example, if multiple threadsconcurrently modify the same list, the GIL ensures that the length ofthe list (ob_size) accurately matches the number of elements, andthat the reference counts of each element accurately reflect thenumber of references to those elements. Without the GIL — andabsent other changes — concurrent modifications would corrupt thosefields and likely lead to program crashes.

The GIL does not necessarily ensure that operations are atomic orremain correct when multiple operations occur concurrently. Forexample,list.extend(iterable) may not appear atomic if theiterable has an iterator implemented in Python (or releases the GILinternally). Similarly,list.remove(x) can remove the wrongobject if it overlaps with another operation that modifies the list,depending on the implementation of the equality operator. Still, theGIL ensures that some operations are effectively atomic. For example,the constructorlist(set) atomically copies the items of the setto a new list, and some code relies on that copy being atomic(i.e., having a snapshot of the items in the set). This PEP preservesthat property.

This PEP proposes using per-object locks to provide many of the sameprotections that the GIL provides. For example, every list,dictionary, and set will have an associated lightweight lock. Alloperations that modify the object must hold the object’s lock. Mostoperations that read from the object should acquire the object’s lockas well; the few read operations that can proceed without holding alock are described below.

Per-object locks with critical sections provide weaker protectionsthan the GIL. Because the GIL doesn’t necessarily ensure thatconcurrent operations are atomic or correct, the per-object lockingscheme also cannot ensure that concurrent operations are atomic orcorrect. Instead, per-object locking aims for similar protections asthe GIL, but with mutual exclusion limited to individual objects.

Most operations on an instance of a container type require lockingthat object. For example:

• list.append,list.insert,list.repeat,PyList_SetItem
• dict.__setitem__,PyDict_SetItem
• list.clear,dict.clear
• list.__repr__,dict.__repr__, etc.
• list.extend(iterable)
• setiter_iternext

Some operations operate directly on two container objects, withknowledge about both containers’ internal structure. For example,there are internal specializations oflist.extend(iterable) forspecific iterable types, likeset. These operations need to lockboth container objects because they access the internals of bothobjects simultaneously. Note that the generic implementation oflist.extend only needs to lock one object (the list) because theother object is accessed indirectly through the thread-safe iteratorAPI. Operations that lock two containers are:

• list.extend(list),list.extend(set),list.extend(dictitems), and other specializations where the implementationis specialized for argument type.
• list.concat(list)
• list.__eq__(list),dict.__eq__(dict)

Some simple operations can be implemented directly with atomicaccesses and do not need locks because they only access a singlefield. These operations include:

• len(list) i.e.,list_length(PyListObject*a)
• len(dict)
• len(set)

A select few operations optimistically avoid locking to improveperformance. These require special implementations and cooperationfrom the memory allocator:

• list[idx] (list_subscript)
• dict[key] (dict_subscript)
• listiter_next,dictiter_iternextkey/value/item
• list.contains

PyObject*item=PyList_GetItem(list,idx);Py_INCREF(item);

Py_INCREF(a->ob_item[i]);returna->ob_item[i];

PyObject**ob_item=atomic_load(&a->ob_item);PyObject*item=atomic_load(&ob_item[i]);if(!item||!_Py_TRY_INCREF(item))gotoretry;if(item!=atomic_load(&ob_item[i])){Py_DECREF(item);gotoretry;}if(ob_item!=atomic_load(&a->ob_item)){Py_DECREF(item);gotoretry;}returnitem;

retry:PyObject*item;Py_BEGIN_CRITICAL_SECTION(a->ob_mutex);item=a->ob_item[i];Py_INCREF(item);Py_END_CRITICAL_SECTION(a->ob_mutex);returnitem;

Specializing Interpreter

The specializing interpreter requires some changes to be thread-safewhen running without the GIL:

• Concurrent specializations are prevented by using a mutex. Thisprevents multiple threads writing to the same inline cache.
• In multi-threaded programs running without the GIL, each bytecode isonly specialized once. This prevents a thread from reading apartially written inline cache.
• Locking also ensures that cached values oftp_version_tag andkeys_version are consistent with the cached descriptors and othervalues.
• Modifications to inline counters use “relaxed atomics”. In otherwords, some counter decrements may be missed or overwritten, but thatdoes not affect correctness.

Py_mod_gil Slot

In--disable-gil builds, when loading an extension, CPython willcheck for a newPEP 489-stylePy_mod_gil slot. If the slot isset toPy_mod_gil_not_used, then extension loading proceeds asnormal. If the slot is not set, the interpreter pauses all threads andenables the GIL before continuing. Additionally, the interpreter willissue a visible warning naming the extension, that the GIL was enabled(and why) and the steps the user can take to override it.

PYTHONGIL Environment Variable

In--disable-gil builds, the user can also override the behavior atruntime by setting thePYTHONGIL environment variable. SettingPYTHONGIL=0, forces the GIL to be disabled, overriding the moduleslot logic. SettingPYTHONGIL=1, forces the GIL to be enabled.

ThePYTHONGIL=0 override is important because extensions that arenot thread-safe can still be useful in multi-threaded applications. Forexample, one may want to use the extension from only a single thread orguard access by locks. For context, there are already some extensionsthat are not thread-safe even with the GIL, and users already have totake these sorts of steps.

ThePYTHONGIL=1 override is sometimes useful for debugging.

Non-Generational Garbage Collection

This PEP proposes switching from a generational cyclic garbagecollector to a non-generational collector (when CPython is builtwithout the GIL). That is equivalent to only having one generation(the “old” generation). There are two reasons for this proposedchange.

Cyclic garbage collection, even for just the young generation,requires pausing other threads in the program. The author isconcerned that frequent collections of the young generation wouldinhibit efficient scaling in multi-threaded programs. This is aconcern for young generations (but not the old generation) becausethe young generations are collected after a fixed number ofallocations, while the collections for the older generation arescheduled in proportion to the number of live objects in the heap.Additionally, it is difficult to efficiently keep track of objects ineach generation without the GIL. For example, CPython currently usesa linked list of objects in each generation. If CPython were to keepthat design, those lists would need to be made thread-safe, and it’snot clear how to do that efficiently.

Generational garbage collection is used to good effect in many otherlanguage runtimes. For example, many of the Java HotSpot garbagecollector implementations use multiple generations[11]. Inthese runtimes, a young generation is frequently a throughput win:since a large percentage of the young generation is typically “dead,”the GC is able to reclaim a large amount memory relative to theamount of work performed. For example, several Java benchmarks showover 90% of “young” objects are typically collected[12][13]. This is commonly referred to as the “weakgenerational hypothesis;” the observation is that most objects dieyoung. This pattern is reversed in CPython due to the use ofreference counting. Although most objects still die young, they arecollected when their reference counts reach zero. Objects thatsurvive to a garbage collection cycle are most likely to remainalive[14]. This difference means that generationalcollection is much less effective in CPython than in many otherlanguage runtimes[15].

Optimistic Avoiding Locking indict andlist Accesses

This proposal relies on a scheme that mostly avoids acquiring lockswhen accessing individual elements in lists and dictionaries. Notethat this is not “lock free” in the sense of “lock-free”and “wait-free” algorithms that guarantee forward progress. Itsimply avoids acquiring locks (mutexes) in the common case to improveparallelism and reduce overhead.

A much simpler alternative would be to use reader-writer locks toprotect dictionary and list accesses. Reader-writer locks allowconcurrent reads, but not updates, which might seem ideal for listand dictionaries. The problem is that reader-writer locks havesubstantial overhead and poor scalability, particularly when thecritical sections are small, as they are for single-elementdictionary and list accesses[9]. The poor readerscalability stems from the fact that readers must all update the samedata structure, such as the number of readers inpthread_rwlocks.

The technique described in this PEP is related to RCU(“read-copy-update”)[6] and, to a lesser extent, hazardpointers, two well-known schemes for optimizing concurrent,read-mostly data structures. RCU is widely used in the Linux kernelto protect shared data structures in a scalable manner. Both thetechnique in this PEP and RCU work by deferring reclamation whilereaders may be accessing the concurrent data structure. RCU is mostcommonly used to protect individual objects (like hash tables orlinked lists), while this PEP proposes a scheme to protect largerblocks of memory (mimalloc “pages”)[10].

The need for this scheme is largely due to the use of referencecounting in CPython. If CPython only relied on a tracing garbagecollector, then this scheme would probably not be necessary becausetracing garbage collectors already defer reclamation in the requiredmanner. This would not “solve” scaling issues, but would shift manyof the challenges to the garbage collector implementation.

Multiprocessing

The multiprocessing library allows Python programs to start andcommunicate with Python subprocesses. This allows for parallelismbecause each subprocess has its own Python interpreter (i.e., there’sone GIL per process). Multiprocessing has a few substantiallimitations. Communication between processes is limited: objectsgenerally need to be serialized or copied to shared memory. Thisintroduces overhead (due to serialization) and complicates buildingAPIs on top of multiprocessing. Starting a subprocess is also moreexpensive than starting a thread, especially with the “spawn”implementation. Starting a thread takes ~100 µs, while spawning asubprocess takes ~50 ms (50,000 µs) due to Python re-initialization.

Finally, many C and C++ libraries support access from multiplethreads but do not support access or use across multiple processes.

Releasing the GIL in C-API Extensions

C-API extensions can release the GIL around long running functions.This allows for some degree of parallelism, since multiple threadscan run concurrently when the GIL is released, but the overhead ofacquiring and releasing the GIL typically prevents this from scalingefficiently beyond a few threads. Many scientific computinglibraries release the GIL in computational heavy functions, and theCPython standard library releases the GIL around blocking I/O.

Internal Parallelization

Functions implemented in C may use multiple threads internally. Forexample, Intel’s NumPy distribution, PyTorch, and TensorFlow all usethis technique to internally parallelize individual operations. Thisworks well when the basic operations are large enough to beparallelized efficiently, but not when there are many smalloperations or when the operations depend on some Python code. Callinginto Python from C requires acquiring the GIL – even short snippetsof Python code can inhibit scaling.

Per-Interpreter GIL

The recently acceptedPEP 684 proposes a per-interpreter GIL toaddress multi-core parallelism. This would allow parallelism betweeninterpreters in the same process, but places substantial restrictionson sharing Python data between interpreters. Both this PEPandPEP 684 address the multi-core parallelism, but with differenttradeoffs and techniques. It is feasible to implement both PEPs inCPython at the same time.

Gilectomy

Gilectomy[20] was a project by Larry Hastings to remove theGIL in CPython. Like the design proposed by this PEP, the Gilectomysupported multiple threads running in parallel within the sameinterpreter (i.e., “free-threading”) and made use of fine-grainedlocking. The reference implementation in this PEP improves onsingle-threaded performance and scalability compared to theGilectomy.

PyParallel

PyParallel[21] was a proof-of-concept fork of Python 3.3 byTrent Nelson that supported multiple threads running simultaneouslyin a single Python process. The fork introduced the conceptof “parallel threads” – threads that can run simultaneously whilethe main Python thread is suspended. Parallel threads had read-onlyaccess to objects created by the main thread. Objects created withinparallel threads lived for the lifetime of the creating thread. ForHTTP servers, this might correspond to the lifetime of a request.

python-safethread

The python-safethread[22] project was a patch toPython 3.0 by Adam Olsen to remove the GIL. Some aspects of theproject are similar to the design proposed by this PEP. Both usefine-grained locking and optimize reference counting for caseswhere the object is created and accessed by the same thread.

Greg Stein’s Free-Threading Patch

In 1996, Greg Stein published a patch against Python 1.4 that removedthe GIL[23]. The patch used atomic reference counting onWindows and a global reference count lock on Linux. List anddictionary accesses were protected by mutexes. Parts of the patchwere adopted in CPython. In particular, the patch introduced aPyThreadState structure and correct per-thread exception handling.

Dave Beazley revisited the patch in a 2011 blog post[24].

Jython and IronPython

Some alternative Python implementations like Jython[25] andIronPython[26] do not have a global interpreter lock.However, they do not support CPython extensions. (The implementationscan interface with code written in Java or C#).

PyPy-STM

The pypy-stm[27] interpreter is a variant of PyPy that usessoftware transactional memory. The authors report single-threadedperformance overhead in the 20%-50% range compared to PyPy. It isnot compatible with CPython extensions.

Why Not Use a Concurrent Garbage Collector?

Many recent garbage collectors are mostly concurrent – they avoid longstop-the-world pauses by allowing the garbage collector to runconcurrently with the application. So why not use a concurrentcollector?

Concurrent collection requires write barriers (or read barriers). Theauthor is not aware of a way to add write barriers to CPython withoutsubstantially breaking the C-API.

Why Not DeprecatePyDict_GetItem in Favor ofPyDict_FetchItem?

This PEP proposes a new APIPyDict_FetchItem which behaves likePyDict_GetItem, but returns a new reference instead of a borrowedreference. As described inBorrowed References, some uses ofborrowed references that were safe when running with the GIL areunsafe when running without the GIL and need to be replaced byfunctions likePyDict_FetchItem that return new references.

This PEP doesnot propose deprecatingPyDict_GetItem and similarfunctions that return borrowed references for a few reasons:

• Many of the uses of borrowed references are safe, even when runningwithout the GIL. For example, C API functions often usePyDict_GetItem to retrieve items from the keywordargument dictionary. These calls are safe because the keywordargument dictionary is only visible to a single thread.
• I tried this approach early on and found that wholesale replacing ofPyDict_GetItem withPyDict_FetchItem frequently introducednew reference counting bugs. In my opinion, the risk ofintroducing new reference counting bugs generally outweighs therisks of missing aPyDict_GetItem call that is unsafe withoutthe GIL.

Why Not Use PEP 683 Immortalization?

LikePEP 683, this PEP proposes an immortalization scheme forPython objects, but the PEPs use different bit representations tomark immortal objects. The schemes cannot be identical because thisPEP depends on biased reference counting, which has two referencecount fields instead of one.

Improved Specialization

The Python 3.11 release introduced quickening and specialization as partof the faster CPython project, substantially improving performance.Specialization replaces slow bytecode instructions with fastervariants[19]. To maintain thread-safety, applications that usemultiple threads (and run without the GIL) will only specialize eachbytecode once, which can lower performance on some programs. It ispossible to support specializing multiple times, but that requires moreinvestigation and is not part of this PEP.

Python Build Modes

This PEP introduces a new build mode (--disable-gil) that is notABI compatible with the standard build mode. The additional buildmode adds complexity for both Python core developers and extensiondevelopers. The author believes a worthwhile goal is to combinethese build modes and have the global interpreter lock controlled atruntime, possibly disabled by default. The path to this goal remainsan open issue, but a possible path might look like the following:

1. In 2024, CPython 3.13 is released with support for a--disable-gil build time flag. There are two ABIs forCPython, one with the GIL and one without. Extension authorstarget both ABIs.
2. After 2–3 releases, (i.e., in 2026–2027), CPython is releasedwith the GIL controlled by a runtime environment variable orflag. The GIL is enabled by default. There is only a single ABI.
3. After another 2–3 release (i.e., 2028–2030), CPython switches tothe GIL being disabled by default. The GIL can still be enabledat runtime via an environment variable or command line flag.

This PEP covers the first step, with the remaining steps left as openissues. In this scenario, there would be a two to three year periodwhere extension authors would target an extra CPython build persupported CPU architecture and OS.

Integration

The reference implementation changes approximately 15,000 lines of codein CPython and includes mimalloc, which is also approximately 15,000lines of code. Most changes are not performance sensitive and can beincluded in both--disable-gil and the default builds. Somemacros, likePy_BEGIN_CRITICAL_SECTION will be no-ops in thedefault build. Thee author does not expect a huge number of#ifdefstatements to support the--disable-gil builds.

Mitigations for Single-Threaded Performance

The changes proposed in the PEP will increase execution overhead for--disable-gil builds compared to Python builds with the GIL. Inother words, it will have slower single-threaded performance. Thereare some possible optimizations to reduce execution overhead,especially for--disable-gil builds that only use a singlethread. These may be worthwhile if a longer term goal is to have asingle build mode, but the choice of optimizations and theirtrade-offs remain an open issue.