Multiple Interpreters and Isolation¶

A Python implementation may support using multiple interpreters in thesame process. CPython has this support. Each interpreter iseffectively isolated from the others (with a limited number ofcarefully managed process-global exceptions to the rule).

That isolation is primarily useful as a strong separation betweendistinct logical components of a program, where you want to havecareful control of how those components interact.

Running in an Interpreter¶

Running in a different interpreter involves switching to it in thecurrent thread and then calling some function. The runtime willexecute the function using the current interpreter’s state. Theconcurrent.interpreters module provides a basic API forcreating and managing interpreters, as well as the switch-and-calloperation.

No other threads are automatically started for the operation.There isa helper for that though.There is another dedicated helper for calling the builtinexec() in an interpreter.

Whenexec() (oreval()) are called in an interpreter,they run using the interpreter’s__main__ module as the“globals” namespace. The same is true for functions that aren’tassociated with any module. This is the same as how scripts invokedfrom the command-line run in the__main__ module.

Concurrency and Parallelism¶

As noted earlier, interpreters do not provide any concurrencyon their own. They strictly represent the isolated executioncontext the runtime will usein the current thread. That isolationmakes them similar to processes, but they still enjoy in-processefficiency, like threads.

All that said, interpreters do naturally support certain flavors ofconcurrency.There’s a powerful side effect of that isolation. It enables adifferent approach to concurrency than you can take with async orthreads. It’s a similar concurrency model to CSP or the actor model,a model which is relatively easy to reason about.

You can take advantage of that concurrency model in a single thread,switching back and forth between interpreters, Stackless-style.However, this model is more useful when you combine interpreterswith multiple threads. This mostly involves starting a new thread,where you switch to another interpreter and run what you want there.

Each actual thread in Python, even if you’re only running in the mainthread, has its owncurrent execution context. Multiple threads canuse the same interpreter or different ones.

At a high level, you can think of the combination of threads andinterpreters as threads with opt-in sharing.

As a significant bonus, interpreters are sufficiently isolated thatthey do not share theGIL, which means combining threads withmultiple interpreters enables full multi-core parallelism.(This has been the case since Python 3.12.)

Communication Between Interpreters¶

In practice, multiple interpreters are useful only if we have a wayto communicate between them. This usually involves some form ofmessage passing, but can even mean sharing data in some carefullymanaged way.

With this in mind, theconcurrent.interpreters module providesaqueue.Queue implementation, available throughcreate_queue().

“Sharing” Objects¶

Any data actually shared between interpreters loses the thread-safetyprovided by theGIL. There are various options for dealing withthis in extension modules. However, from Python code the lack ofthread-safety means objects can’t actually be shared, with a fewexceptions. Instead, a copy must be created, which means mutableobjects won’t stay in sync.

By default, most objects are copied withpickle when they arepassed to another interpreter. Nearly all of the immutable builtinobjects are either directly shared or copied efficiently. For example:

• None

• bool (True andFalse)

• bytes

• str

• int

• float

• tuple (of similarly supported objects)

There is a small number of Python types that actually share mutabledata between interpreters:

• memoryview

• Queue

Interpreter objects¶

classconcurrent.interpreters.Interpreter(id)¶

A single interpreter in the current process.

Generally,Interpreter shouldn’t be called directly.Instead, usecreate() or one of the other module functions.

id¶

(read-only)

The underlying interpreter’s ID.

whence¶

(read-only)

A string describing where the interpreter came from.

is_running()¶

ReturnTrue if the interpreter is currently executing codein its__main__ module andFalse otherwise.

close()¶

Finalize and destroy the interpreter.

prepare_main(ns=None,**kwargs)¶

Bind objects in the interpreter’s__main__ module.

Some objects are actually shared and some are copied efficiently,but most are copied viapickle. See“Sharing” Objects.

exec(code,/,dedent=True)¶

Run the given source code in the interpreter (in the current thread).

call(callable,/,*args,**kwargs)¶

Return the result of calling running the given function in theinterpreter (in the current thread).

call_in_thread(callable,/,*args,**kwargs)¶

Run the given function in the interpreter (in a new thread).

Exceptions¶

exceptionconcurrent.interpreters.InterpreterError¶

This exception, a subclass ofException, is raised whenan interpreter-related error happens.

exceptionconcurrent.interpreters.InterpreterNotFoundError¶

This exception, a subclass ofInterpreterError, is raised whenthe targeted interpreter no longer exists.

exceptionconcurrent.interpreters.ExecutionFailed¶

This exception, a subclass ofInterpreterError, is raised whenthe running code raised an uncaught exception.

excinfo¶

A basic snapshot of the exception raised in the other interpreter.

exceptionconcurrent.interpreters.NotShareableError¶

This exception, a subclass ofTypeError, is raised whenan object cannot be sent to another interpreter.

Communicating Between Interpreters¶

classconcurrent.interpreters.Queue(id)¶

A wrapper around a low-level, cross-interpreter queue, whichimplements thequeue.Queue interface. The underlying queuecan only be created throughcreate_queue().

Some objects are actually shared and some are copied efficiently,but most are copied viapickle. See“Sharing” Objects.

id¶

(read-only)

The queue’s ID.

exceptionconcurrent.interpreters.QueueEmptyError¶

This exception, a subclass ofqueue.Empty, is raised fromQueue.get() andQueue.get_nowait() when the queueis empty.

exceptionconcurrent.interpreters.QueueFullError¶

This exception, a subclass ofqueue.Full, is raised fromQueue.put() andQueue.put_nowait() when the queueis full.