Many Objects Cannot be Shared Between Subinterpreters

In Python 3.14, the newconcurrent.interpreters module can be used tocreate multiple interpreters in a single Python process. This works well forcode without shared state, but since one of the primary applications ofsubinterpreters is to bypass theglobal interpreter lock, it isfairly common for programs to require highly-complex data structures that arenot easily shareable. In turn, this damages the practicality ofsubinterpreters for concurrency.

As of writing, subinterpreters can only sharea handful of types natively, relying on thepickle modulefor other types. This can be very limited, as many types of objects cannot beserialized withpickle (such as file objects returned byopen()).Additionally, serialization can be a very expensive operation, which is notideal for multithreaded applications.

A Fallback for Object Sharing

A shared object proxy is designed to be a fallback for sharing an objectbetween interpreters. A shared object proxy should only be used asa last-resort for highly complex objects that cannot be serialized or sharedin any other way.

This means that even if this PEP is accepted, there is still benefit inimplementing other methods to share objects between interpreters.

Interpreter Switching

When interacting with the wrapped object, the proxy will switch to theinterpreter in which the object was created. This must happen for any accessto the object, such as accessing attributes. To visualize,foo in thefollowing code is only ever called in the main interpreter, despite beingaccessed in subinterpreters through a proxy:

fromconcurrentimportinterpretersdeffoo():assertinterpreters.get_current()==interpreters.get_main()interp=interpreters.create()proxy=interpreters.share(foo)interp.prepare_main(foo=proxy)interp.exec("foo()")

Method Proxying

Methods on a shared object proxy will switch to their owning interpreter whenaccessed. In addition, any arguments passed to the method are implicitly calledwithshare() to ensure they are shareable (onlytypes that are not natively shareable are wrapped in a proxy). The same happensto the return value of the method.

For example, the__add__ method on an object proxy is roughly equivalentto the following code:

def__add__(self,other):withself.switch_interpreter():result=self.value.__add__(share(other))returnshare(result)

Multithreaded Scaling

To switch to a wrapped object’s interpreter, an object proxy must swap theattached thread state of the current thread, which will in turn waiton theGIL of the target interpreter, if it is enabled. This means thata shared object proxy will experience contention when accessed concurrently,but is still useful for multicore threading, since other threads in theinterpreter are free to execute while waiting on the GIL of the targetinterpreter.

As an example, imagine that multiple interpreters want to write to a log througha proxy for the main interpreter, but don’t want to constantly wait on the log.By accessing the proxy in a separate thread for each interpreter, the threadperforming the computation can still execute while accessing the proxy.

fromconcurrentimportinterpretersdefwrite_log(message):print(message)defexecute(n,write_log):fromthreadingimportThreadfromqueueimportQueuelog=Queue()# By performing this in a separate thread, 'execute' can still run# while the log is being accessed by the main interpreter.deflog_queue_loop():whileTrue:write_log(log.get())thread=Thread(target=log_queue_loop)thread.start()foriinrange(100000):n**ilog.put(f"Completed an iteration:{i}")thread.join()proxy=interpreters.share(write_log)forninrange(4):interp=interpreters.create()interp.call_in_thread(execute,n,proxy)

Proxy Copying

Contrary to what one might think, a shared object proxy itself can only be usedin one interpreter, because the proxy’s reference count is not thread-safe(and thus cannot be accessed from multiple interpreters). Instead, when crossingan interpreter boundary, a new proxy is created for the target interpreter thatwraps the same object as the original proxy.

For example, in the following code, there are two proxies created, not just one.

fromconcurrentimportinterpretersinterp=interpreters.create()foo=object()proxy=interpreters.share(foo)# The proxy crosses an interpreter boundary here. 'proxy' is *not* directly# send to 'interp'. Instead, a new proxy is created for 'interp', and the# reference to 'foo' is merely copied. Thus, both interpreters have their# own proxy that are wrapping the same object.interp.prepare_main(proxy=proxy)

Thread-local State

Accessing an object proxy will retain information stored on the currentthread state, such as thread-local variables stored bythreading.local and context variables stored bycontextvars.This allows the following case to work correctly:

fromconcurrentimportinterpretersfromthreadingimportlocalthread_local=local()thread_local.value=1deffoo():assertthread_local.value==1interp=interpreters.create()proxy=interpreters.share(foo)interp.prepare_main(foo=proxy)interp.exec("foo()")

In order to retain thread-local data when accessing an object proxy, eachthread will have to keep track of the last used thread state foreach interpreter. In C, this behavior looks like this:

// Error checking has been omitted for brevityPyThreadState*tstate=PyThreadState_New(interp);// By swapping the current thread state to 'interp', 'tstate' will be// associated with 'interp' for the current thread. That means that accessing// a shared object proxy will use 'tstate' instead of creating its own// thread state.PyThreadState*save=PyThreadState_Swap(tstate);// 'save' is now the most recently used thread state, so shared object// proxies in this thread will use it instead of 'tstate' when accessing// 'interp'.PyThreadState_Swap(save);

In the event that no thread state exists for an interpreter in a given thread,a shared object proxy will create its own thread state that will be owned bythe interpreter (meaning it will not be destroyed until interpreterfinalization), which will persist across all shared object proxy accesses inthe thread. In other words, a shared object proxy ensures that thread localvariables and similar state will not disappear.

Memory Management

All proxy objects hold astrong reference to the object that theywrap. As such, destruction of a shared object proxy may trigger destructionof the wrapped object if the proxy holds the last reference to it, even ifthe proxy belongs to a different interpreter. For example:

fromconcurrentimportinterpretersinterp=interpreters.create()foo=object()proxy=interpreters.share(foo)interp.prepare_main(proxy=proxy)delproxy,foo# 'foo' is still alive at this point, because the proxy in 'interp' still# holds a reference to it. Destruction of 'interp' will then trigger the# destruction of 'proxy', and subsequently the destruction of 'foo'.interp.close()

Shared object proxies support the garbage collector protocol, but will onlytraverse the object that they wrap if the garbage collection is occurringin the wrapped object’s interpreter. To visualize:

fromconcurrentimportinterpretersimportgcproxy=interpreters.share(object())# This prints out [<object object at 0x...>], because the object is owned# by this interpreter.print(gc.get_referents(proxy))interp=interpreters.create()interp.prepare_main(proxy=proxy)# This prints out [], because the wrapepd object must be invisible to this# interpreter.interp.exec("import gc; print(gc.get_referents(proxy))")

fromconcurrentimportinterpretersdeftest():fromconcurrentimportinterpretersclassTest:def__str__(self):return"Alive"returninterpreters.share(Test())interp=interpreters.create()wrapped=interp.call(test)print(wrapped)# Aliveinterp.close()print(wrapped)# None

Directly Sharing Proxy Objects

The initial revision of this proposal took an approach where an instance ofSharedObjectProxy wasimmortal. Thisallowed proxy objects to be directly shared across interpreters, because theirreference count was thread-safe (since it never changed due to immortality).

This proved to make the implementation significantly more complicated, andalso ended up with a lot of edge cases that would have been a burden onCPython maintainers.