Newlocals() documentation

The heart of this proposal is to revise the documentation for thelocals()builtin to read as follows:

Return a mapping object representing the current local symbol table, withvariable names as the keys, and their currently bound references as thevalues.

At module scope, as well as when usingexec() oreval() with asingle namespace, this function returns the same namespace asglobals().

At class scope, it returns the namespace that will be passed to themetaclass constructor.

When usingexec() oreval() with separate local and globalnamespaces, it returns the local namespace passed in to the function call.

In all of the above cases, each call tolocals() in a given frame ofexecution will return thesame mapping object. Changes made throughthe mapping object returned fromlocals() will be visible as bound,rebound, or deleted local variables, and binding, rebinding, or deletinglocal variables will immediately affect the contents of the returned mappingobject.

At function scope (including for generators and coroutines), each call tolocals() instead returns a fresh dictionary containing the currentbindings of the function’s local variables and any nonlocal cell references.In this case, name binding changes made via the returned dict arenotwritten back to the corresponding local variables or nonlocal cellreferences, and binding, rebinding, or deleting local variables and nonlocalcell references doesnot affect the contents of previously returneddictionaries.

There would also be aversionchanged note for the release making this change:

In prior versions, the semantics of mutating the mapping object returnedfromlocals() were formally undefined. In CPython specifically,the mapping returned at function scope could be implicitly refreshed byother operations, such as callinglocals() again, or the interpreterimplicitly invoking a Python level trace function. Obtaining the legacyCPython behaviour now requires explicit calls to update the initiallyreturned dictionary with the results of subsequent calls tolocals().

For reference, the current documentation of this builtin reads as follows:

Update and return a dictionary representing the current local symbol table.Free variables are returned by locals() when it is called in functionblocks, but not in class blocks.

Note: The contents of this dictionary should not be modified; changes maynot affect the values of local and free variables used by the interpreter.

(In other words: the status quo is that the semantics and behaviour oflocals() are formally implementation defined, whereas the proposedstate after this PEP is that the only implementation defined behaviour will bethat associated with whether or not the implementation emulates the CPythonframe API, with the behaviour in all other cases being defined by the languageand library references)

Module scope

At module scope, as well as when usingexec() oreval() with asingle namespace,locals() must return the same object asglobals(),which must be the actual execution namespace (available asinspect.currentframe().f_locals in implementations that provide accessto frame objects).

Variable assignments during subsequent code execution in the same scope mustdynamically change the contents of the returned mapping, and changes to thereturned mapping must change the values bound to local variable names in theexecution environment.

To capture this expectation as part of the language specification, the followingparagraph will be added to the documentation forlocals():

At module scope, as well as when usingexec() oreval() with asingle namespace, this function returns the same namespace asglobals().

This part of the proposal does not require any changes to the referenceimplementation - it is standardisation of the current behaviour.

Class scope

At class scope, as well as when usingexec() oreval() with separateglobal and local namespaces,locals() must return the specified localnamespace (which may be supplied by the metaclass__prepare__ methodin the case of classes). As for module scope, this must be a direct referenceto the actual execution namespace (available asinspect.currentframe().f_locals in implementations that provide accessto frame objects).

Variable assignments during subsequent code execution in the same scope mustchange the contents of the returned mapping, and changes to the returned mappingmust change the values bound to local variable names in theexecution environment.

The mapping returned bylocals() willnot be used as the actual classnamespace underlying the defined class (the class creation process will copythe contents to a fresh dictionary that is only accessible by going through theclass machinery).

For nested classes defined inside a function, any nonlocal cells referenced fromthe class scope arenot included in thelocals() mapping.

To capture this expectation as part of the language specification, the followingtwo paragraphs will be added to the documentation forlocals():

When usingexec() oreval() with separate local and globalnamespaces, [this function] returns the given local namespace.

At class scope, it returns the namespace that will be passed to the metaclassconstructor.

This part of the proposal does not require any changes to the referenceimplementation - it is standardisation of the current behaviour.

Function scope

At function scope, interpreter implementations are granted significant freedomto optimise local variable access, and hence are NOT required to permitarbitrary modification of local and nonlocal variable bindings through themapping returned fromlocals().

Historically, this leniency has been described in the language specificationwith the words “The contents of this dictionary should not be modified; changesmay not affect the values of local and free variables used by the interpreter.”

This PEP proposes to change that text to instead say:

At function scope (including for generators and coroutines), each call tolocals() instead returns a fresh dictionary containing the currentbindings of the function’s local variables and any nonlocal cell references.In this case, name binding changes made via the returned dict arenotwritten back to the corresponding local variables or nonlocal cellreferences, and binding, rebinding, or deleting local variables and nonlocalcell references doesnot affect the contents of previously returneddictionaries.

This part of the proposaldoes require changes to the CPython referenceimplementation, as CPython currently returns a shared mapping object that maybe implicitly refreshed by additional calls tolocals(), and the“write back” strategy currently used to support namespace changesfrom trace functions also doesn’t comply with it (and causes the quirkybehavioural problems mentioned in the Motivation above).

Summary of proposed implementation-specific changes

• Changes are made as necessary to provide the updated Python level semantics
• Two new functions are added to the stable ABI to replicate the updatedbehaviour of the Pythonlocals() builtin:

PyObject*PyLocals_Get();PyLocals_KindPyLocals_GetKind();

• One new function is added to the stable ABI to efficiently get a snapshot ofthe local namespace in the running frame:

PyObject*PyLocals_GetCopy();

• Corresponding frame accessor functions for these new public APIs are added tothe CPython frame C API
• On optimised frames, the Python levelf_locals API will return dynamicallycreated read/write proxy objects that directly access the frame’s local andclosure variable storage. To provide interoperability with the existingPyEval_GetLocals() API, the proxy objects will continue to use the C levelframe locals data storage field to hold a value cache that also allows forstorage of arbitrary additional keys. Additional details on the expectedbehaviour of these fast locals proxy objects are covered below.
• No C API function is added to get access to a mutable mapping for the localnamespace. Instead,PyObject_GetAttrString(frame,"f_locals") is used, thesame API as is used in Python code.
• PyEval_GetLocals() remains supported and does not emit a programmaticwarning, but will be deprecated in the documentation in favour of the newAPIs that don’t rely on returning a borrowed reference
• PyFrame_FastToLocals() andPyFrame_FastToLocalsWithError() remainsupported and do not emit a programmatic warning, but will be deprecated inthe documentation in favour of the new APIs that don’t require direct accessto the internal data storage layout of frame objects
• PyFrame_LocalsToFast() always raisesRuntimeError(), indicating thatPyObject_GetAttrString(frame,"f_locals") should be used to obtain amutable read/write mapping for the local variables.
• The trace hook implementation will no longer callPyFrame_FastToLocals()implicitly. The version porting guide will recommend migrating toPyFrame_GetLocals() for read-only access andPyObject_GetAttrString(frame,"f_locals") for read/write access.

Providing the updated Python level semantics

The implementation of thelocals() builtin is modified to return a distinctcopy of the local namespace for optimised frames, rather than a direct referenceto the internal frame value cache updated by thePyFrame_FastToLocals() CAPI and returned by thePyEval_GetLocals() C API.

Resolving the issues with tracing mode behaviour

The current cause of CPython’s tracing mode quirks (both the side effects fromsimply installing a tracing function and the fact that writing values back tofunction locals only works for the specific function being traced) is the waythat locals mutation support for trace hooks is currently implemented: thePyFrame_LocalsToFast function.

When a trace function is installed, CPython currently does the following forfunction frames (those where the code object uses “fast locals” semantics):

1. CallsPyFrame_FastToLocals to update the frame value cache
2. Calls the trace hook (with tracing of the hook itself disabled)
3. CallsPyFrame_LocalsToFast to capture any changes made to the framevalue cache

This approach is problematic for a few different reasons:

• Even if the trace function doesn’t mutate the value cache, the final stepresets any cell references back to the state they were in before the tracefunction was called (this is the root cause of the bug report in[1])
• If the trace functiondoes mutate the value cache, but then does somethingthat causes the value cache to be refreshed from the frame, those changes arelost (this is one aspect of the bug report in[3])
• If the trace function attempts to mutate the local variables of a frame otherthan the one being traced (e.g.frame.f_back.f_locals), those changeswill almost certainly be lost (this is another aspect of the bug report in[3])
• If a reference to the frame value cache (e.g. retrieved vialocals()) ispassed to another function, andthat function mutates the value cache, thenthose changesmay be written back to the execution frameif a trace hookis installed

The proposed resolution to this problem is to take advantage of the fact thatwhereas functions typically access theirown namespace using the languagedefinedlocals() builtin, trace functions necessarily use the implementationdependentframe.f_locals interface, as a frame reference is what getspassed to hook implementations.

Instead of being a direct reference to the internal frame value cache historicallyreturned by thelocals() builtin, the Python levelframe.f_locals will beupdated to instead return instances of a dedicated fast locals proxy type thatwrites and reads values directly to and from the fast locals array on theunderlying frame. Each access of the attribute produces a new instance of theproxy (so creating proxy instances is intentionally a cheap operation).

Despite the new proxy type becoming the preferred way to access local variableson optimised frames, the internal value cache stored on the frame is stillretained for two key purposes:

• maintaining backwards compatibility for and interoperability with thePyEval_GetLocals() C API
• providing storage space for additional keys that don’t have slots in thefast locals array (e.g. the__return__ and__exception__ keys set bypdb when tracing code execution for debugging purposes)

With the changes in this PEP, this internal frame value cache is no longerdirectly accessible from Python code (whereas historically it was bothreturned by thelocals() builtin and available as theframe.f_localsattribute). Instead, the value cache is only accessible via thePyEval_GetLocals() C API and by directly accessing the internal storage ofa frame object.

Fast locals proxy objects and the internal frame value cache returned byPyEval_GetLocals() offer the following behavioural guarantees:

• changes made via a fast locals proxy will be immediately visible to the frameitself, to other fast locals proxy objects for the same frame, and in theinternal value cache stored on the frame (it is this last point that providesPyEval_GetLocals() interoperability)
• changes made directly to the internal frame value cache will never be visibleto the frame itself, and will only be reliably visible via fast locals proxiesfor the same frame if the change relates to extra variables that don’t haveslots in the frame’s fast locals array
• changes made by executing code in the frame will be immediately visible to allfast locals proxy objects for that frame (both existing proxies and newlycreated ones). Visibility in the internal frame value cache cache returnedbyPyEval_GetLocals() is subject to the cache update guidelines discussedin the next section

As a result of these points, only code usingPyEval_GetLocals(),PyLocals_Get(), orPyLocals_GetCopy() will need to be concerned aboutthe frame value cache potentially becoming stale. Code using the new frame fastlocals proxy API (whether from Python or from C) will always see the live stateof the frame.

Fast locals proxy implementation details

Each fast locals proxy instance has a single internal attribute that is notexposed as part of the Python runtime API:

• frame: the underlying optimised frame that the proxy provides access to

In addition, proxy instances use and update the following attributes stored on theunderlying frame or code object:

• _name_to_offset_mapping: a hidden mapping from variable names to fast localstorage offsets. This mapping is lazily initialized on the first frame read orwrite access through a fast locals proxy, rather than being eagerly populatedas soon as the first fast locals proxy is created. Since the mapping isidentical for all frames running a given code object, a single copy is storedon the code object, rather than each frame object populating its own mapping
• locals: the internal frame value cache returned by thePyEval_GetLocals()C API and updated by thePyFrame_FastToLocals() C API. This is the mappingthat thelocals() builtin returns in Python 3.10 and earlier.

__getitem__ operations on the proxy will populate the_name_to_offset_mappingon the code object (if it is not already populated), and then either return therelevant value (if the key is found in either the_name_to_offset_mappingmapping or the internal frame value cache), or else raiseKeyError. Variablesthat are defined on the frame but not currently bound also raiseKeyError(just as they’re omitted from the result oflocals()).

As the frame storage is always accessed directly, the proxy will automaticallypick up name binding and unbinding operations that take place as the functionexecutes. The internal value cache is implicitly updated when individualvariables are read from the frame state (including for containment checks,which need to check if the name is currently bound or unbound).

Similarly,__setitem__ and__delitem__ operations on the proxy willdirectly affect the corresponding fast local or cell reference on the underlyingframe, ensuring that changes are immediately visible to the running Python code,rather than needing to be written back to the runtime storage at some later time.Such changes are also immediately written to the internal frame value cache tomake them visible to users of thePyEval_GetLocals() C API.

Keys that are not defined as local or closure variables on the underlying frameare still written to the internal value cache on optimised frames. This allowsutilities likepdb (which writes__return__ and__exception__values into the frame’sf_locals mapping) to continue working as they alwayshave. These additional keys that do not correspond to a local or closurevariable on the frame will be left alone by future cache sync operations.Using the frame value cache to store these extra keys (rather than defining anew mapping that holds only the extra keys) provides full interoperabilitywith the existingPyEval_GetLocals() API (since users of either API willsee extra keys added by users of either API, rather than users of the new fastlocals proxy API only seeing keys added via that API).

An additional benefit of storing only the variable value cache on the frame(rather than storing an instance of the proxy type), is that it avoidscreating a reference cycle from the frame back to itself, so the frame willonly be kept alive if another object retains a reference to a proxy instance.

Note: calling theproxy.clear() method has a similarly broad impact ascallingPyFrame_LocalsToFast() on an empty frame value cache in earlierversions. Not only will the frame local variables be cleared, but also any cellvariables accessible from the frame (whether those cells are owned by theframe itself or by an outer frame). Thiscan clear a class’s__class__cell if called on the frame of a method that uses the zero-argsuper()construct (or otherwise references__class__). This exceeds the scope ofcallingframe.clear(), as that only drop’s the frame’s references to cellvariables, it doesn’t clear the cells themselves. This PEP could be a potentialopportunity to narrow the scope of attempts to clear the frame variablesdirectly by leaving cells belonging to outer frames alone, and only clearinglocal variables and cells belonging directly to the frame underlying the proxy(this issue affectsPEP 667 as well, as the question relates to the handling ofcell variables, and is entirely independent of the internal frame value cache).

Changes to the stable C API/ABI

Unlike Python code, extension module functions that call in to the Python C APIcan be called from any kind of Python scope. This means it isn’t obvious fromthe context whetherlocals() will return a snapshot or not, as it dependson the scope of the calling Python code, not the C code itself.

This means it is desirable to offer C APIs that give predictable, scopeindependent, behaviour. However, it is also desirable to allow C code toexactly mimic the behaviour of Python code at the same scope.

To enable mimicking the behaviour of Python code, the stable C ABI would gainthe following new functions:

PyObject*PyLocals_Get();PyLocals_KindPyLocals_GetKind();

PyLocals_Get() is directly equivalent to the Pythonlocals() builtin.It returns a new reference to the local namespace mapping for the activePython frame at module and class scope, and when usingexec() oreval().It returns a shallow copy of the active namespace atfunction/coroutine/generator scope.

PyLocals_GetKind() returns a value from the newly definedPyLocals_Kindenum, with the following options being available:

• PyLocals_DIRECT_REFERENCE:PyLocals_Get() returns a direct referenceto the local namespace for the running frame.
• PyLocals_SHALLOW_COPY:PyLocals_Get() returns a shallow copy of thelocal namespace for the running frame.
• PyLocals_UNDEFINED: an error occurred (e.g. no active Python threadstate). A Python exception will be set if this value is returned.

Since the enum is used in the stable ABI, an additional 31-bit value is set toensure that it is safe to cast arbitrary signed 32-bit signed integers toPyLocals_Kind values.

This query API allows extension module code to determine the potential impactof mutating the mapping returned byPyLocals_Get() without needing accessto the details of the running frame object. Python code gets equivalentinformation visually through lexical scoping (as covered in the newlocals()builtin documentation).

To allow extension module code to behave consistently regardless of the activePython scope, the stable C ABI would gain the following new function:

PyObject*PyLocals_GetCopy();

PyLocals_GetCopy() returns a new dict instance populated from the currentlocals namespace. Roughly equivalent todict(locals()) in Python code, butavoids the double-copy in the case wherelocals() already returns a shallowcopy. Akin to the following code, but doesn’t assume there will only ever betwo kinds of locals result:

locals=PyLocals_Get();if(PyLocals_GetKind()==PyLocals_DIRECT_REFERENCE){locals=PyDict_Copy(locals);}

The existingPyEval_GetLocals() API will retain its existing behaviour inCPython (mutable locals at class and module scope, shared dynamic snapshototherwise). However, its documentation will be updated to note that theconditions under which the shared dynamic snapshot get updated have changed.

ThePyEval_GetLocals() documentation will also be updated to recommendreplacing usage of this API with whichever of the new APIs is most appropriatefor the use case:

• UsePyLocals_Get() (optionally combined withPyDictProxy_New()) forread-only access to the current locals namespace. This form of usage willneed to be aware that the copy may go stale in optimised frames.
• UsePyLocals_GetCopy() for a regular mutable dict that contains a copy ofthe current locals namespace, but has no ongoing connection to the activeframe.
• UsePyLocals_Get() to exactly match the semantics of the Python levellocals() builtin.
• QueryPyLocals_GetKind() explicitly to implement custom handling(e.g. raising a meaningful exception) for scopes wherePyLocals_Get()would return a shallow copy rather than granting read/write access to thelocals namespace.
• Use implementation specific APIs (e.g.PyObject_GetAttrString(frame,"f_locals"))if read/write access to the frame is required andPyLocals_GetKind()returns something other thanPyLocals_DIRECT_REFERENCE.

Changes to the public CPython C API

The existingPyEval_GetLocals() API returns a borrowed reference, whichmeans it cannot be updated to return the new shallow copies at functionscope. Instead, it will continue to return a borrowed reference to an internaldynamic snapshot stored on the frame object. This shared mapping will behavesimilarly to the existing shared mapping in Python 3.10 and earlier, but the exactconditions under which it gets refreshed will be different. Specifically, itwill be updated only in the following circumstance:

• any call toPyEval_GetLocals(),PyLocals_Get(),PyLocals_GetCopy(),or the Pythonlocals() builtin while the frame is running
• any call toPyFrame_GetLocals(),PyFrame_GetLocalsCopy(),_PyFrame_BorrowLocals(),PyFrame_FastToLocals(), orPyFrame_FastToLocalsWithError() for the frame
• any operation on a fast locals proxy object that updates the sharedmapping as part of its implementation. In the initial referenceimplementation, those operations are those that are intrinsicallyO(n)operations (len(flp), mapping comparison,flp.copy() and rendering asa string), as well as those that refresh the cache entries for individual keys.

Requesting a fast locals proxy willnot implicitly update the shared dynamicsnapshot, and the CPython trace hook handling will no longer implicitly updateit either.

(Note: even thoughPyEval_GetLocals() is part of the stable C API/ABI, thespecifics of when the namespace it returns gets refreshed are still aninterpreter implementation detail)

The additions to the public CPython C API are the frame level enhancementsneeded to support the stable C API/ABI updates:

PyLocals_KindPyFrame_GetLocalsKind(frame);PyObject*PyFrame_GetLocals(frame);PyObject*PyFrame_GetLocalsCopy(frame);PyObject*_PyFrame_BorrowLocals(frame);

PyFrame_GetLocalsKind(frame) is the underlying API forPyLocals_GetKind().

PyFrame_GetLocals(frame) is the underlying API forPyLocals_Get().

PyFrame_GetLocalsCopy(frame) is the underlying API forPyLocals_GetCopy().

_PyFrame_BorrowLocals(frame) is the underlying API forPyEval_GetLocals(). The underscore prefix is intended to discourage use andto indicate that code using it is unlikely to be portable acrossimplementations. However, it is documented and visible to the linker in orderto avoid having to access the internals of the frame struct from thePyEval_GetLocals() implementation.

ThePyFrame_LocalsToFast() function will be changed to always emitRuntimeError, explaining that it is no longer a supported operation, andaffected code should be updated to usePyObject_GetAttrString(frame,"f_locals") to obtain a read/write proxyinstead.

In addition to the above documented interfaces, the draft referenceimplementation also exposes the following undocumented interfaces:

PyTypeObject_PyFastLocalsProxy_Type;#define _PyFastLocalsProxy_CheckExact(self) Py_IS_TYPE(op, &_PyFastLocalsProxy_Type)

This type is what the reference implementation actually returns fromPyObject_GetAttrString(frame,"f_locals") for optimized frames (i.e.whenPyFrame_GetLocalsKind() returnsPyLocals_SHALLOW_COPY).

Reducing the runtime overhead of trace hooks

As noted in[9], the implicit call toPyFrame_FastToLocals() in thePython trace hook support isn’t free, and could be rendered unnecessary ifthe frame proxy read values directly from the frame instead of getting themfrom the mapping.

As the new frame locals proxy type doesn’t require separate data refresh steps,this PEP incorporates Victor Stinner’s proposal to no longer implicitly callPyFrame_FastToLocalsWithError() before calling trace hooks implemented inPython.

Code using the new fast locals proxy objects will have the dynamic locals snapshotimplicitly refreshed when accessing methods that need it, while code using thePyEval_GetLocals() API will implicitly refresh it when making that call.

The PEP necessarily also drops the implicit call toPyFrame_LocalsToFast()when returning from a trace hook, as that API now always raises an exception.

Changinglocals() to return independent snapshots at function scope

Thelocals() builtin is a required part of the language, and in thereference implementation it has historically returned a mutable mapping withthe following characteristics:

• each call tolocals() returns thesame mapping object
• for namespaces wherelocals() returns a reference to something other thanthe actual local execution namespace, each call tolocals() updates themapping object with the current state of the local variables and any referencednonlocal cells
• changes to the returned mappingusually aren’t written back to thelocal variable bindings or the nonlocal cell references, but write backscan be triggered by doing one of the following:
  • installing a Python level trace hook (write backs then happen wheneverthe trace hook is called)
  • running a function level wildcard import (requires bytecode injection in Py3)
  • running anexec statement in the function’s scope (Py2 only, sinceexec became an ordinary builtin in Python 3)

Originally this PEP proposed to retain the first two of these properties,while changing the third in order to address the outright behaviour bugs thatit can cause.

In[7] Nathaniel Smith made a persuasive case that we could make the behaviouroflocals() at function scope substantially less confusing by retaining onlythe second property and having each call tolocals() at function scopereturn anindependent snapshot of the local variables and closure referencesrather than updating an implicitly shared snapshot.

As this revised design also made the implementation markedly easier to follow,the PEP was updated to propose this change in behaviour, rather than retainingthe historical shared snapshot.

Keepinglocals() as a snapshot at function scope

As discussed in[7], it would theoretically be possible to change the semanticsof thelocals() builtin to return the write-through proxy at function scope,rather than switching it to return independent snapshots.

This PEP doesn’t (and won’t) propose this as it’s a backwards incompatiblechange in practice, even though code that relies on the current behaviour istechnically operating in an undefined area of the language specification.

Consider the following code snippet:

defexample():x=1locals()["x"]=2print(x)

Even with a trace hook installed, that function will consistently print1on the current reference interpreter implementation:

>>>example()1>>>importsys>>>defbasic_hook(*args):...returnbasic_hook...>>>sys.settrace(basic_hook)>>>example()1

Similarly,locals() can be passed to theexec() andeval() builtinsat function scope (either explicitly or implicitly) without risking unexpectedrebinding of local variables or closure references.

Provoking the reference interpreter into incorrectly mutating the local variablestate requires a more complex setup where a nested function closes over avariable being rebound in the outer function, and due to the use of eitherthreads, generators, or coroutines, it’s possible for a trace function to startrunning for the nested function before the rebinding operation in the outerfunction, but finish running after the rebinding operation has taken place (inwhich case the rebinding will be reverted, which is the bug reported in[1]).

In addition to preserving the de facto semantics which have been in place sincePEP 227 introduced nested scopes in Python 2.1, the other benefit of restrictingthe write-through proxy support to the implementation-defined frame object APIis that it means that only interpreter implementations which emulate the fullframe API need to offer the write-through capability at all, and thatJIT-compiled implementations only need to enable it when a frame introspectionAPI is invoked, or a trace hook is installed, not wheneverlocals() isaccessed at function scope.

Returning snapshots fromlocals() at function scope also means that staticanalysis for function level code will be more reliable, as only access to theframe machinery will allow rebinding of local and nonlocal variablereferences in a way that is hidden from static analysis.

What happens with the default args foreval() andexec()?

These are formally defined as inheritingglobals() andlocals() fromthe calling scope by default.

There isn’t any need for the PEP to change these defaults, so it doesn’t, andexec() andeval() will start running in a shallow copy of the localnamespace when that is whatlocals() returns.

This behaviour will have potential performance implications, especiallyfor functions with large numbers of local variables (e.g. if these functionsare called in a loop, callingglobals() andlocals() once before theloop and then passing the namespace into the function explicitly will give thesame semantics and performance characteristics as the status quo, whereasrelying on the implicit default would create a new shallow copy of the localnamespace on each iteration).

(Note: the reference implementation draft PR has updated thelocals() andvars(),eval(), andexec() builtins to usePyLocals_Get(). Thedir() builtin still usesPyEval_GetLocals(), since it’s only using itto make a list from the keys).

Additional considerations foreval() andexec() in optimized scopes

Note: while implementingPEP 667, it was noted that neither that PEP nor this oneclearly explained the impact thelocals() changes would have on code execution APIslikeexec() andeval(). This section was added to this PEP’s rationale to betterdescribe the impact and explain the intended benefits of the change.

Whenexec() was converted from a statement to a builtin functionin Python 3.0 (part of the core language changes inPEP 3100), theassociated implicit call toPyFrame_LocalsToFast() was removed, soit typically appears as if attempts to write to local variables withexec() in optimized frames are ignored:

>>>deff():...x=0...exec("x = 1")...print(x)...print(locals()["x"])...>>>f()00

In truth, the writes aren’t being ignored, they just aren’tbeing copied from the dictionary cache back to the optimized localvariable array. The changes to the dictionary are then overwrittenthe next time the dictionary cache is refreshed from the array:

>>>deff():...x=0...locals_cache=locals()...exec("x = 1")...print(x)...print(locals_cache["x"])...print(locals()["x"])...>>>f()010

The behaviour becomes even stranger if a tracing functionor another piece of code invokesPyFrame_LocalsToFast() beforethe cache is next refreshed. In those cases the changeiswritten back to the optimized local variable array:

>>>fromsysimport_getframe>>>fromctypesimportpythonapi,py_object,c_int>>>_locals_to_fast=pythonapi.PyFrame_LocalsToFast>>>_locals_to_fast.argtypes=[py_object,c_int]>>>deff():..._frame=_getframe()..._f_locals=_frame.f_locals...x=0...exec("x = 1")..._locals_to_fast(_frame,0)...print(x)...print(locals()["x"])...print(_f_locals["x"])...>>>f()111

This situation was more common in Python 3.10 and earlierversions, as merely installing a tracing function was enoughto trigger implicit calls toPyFrame_LocalsToFast() afterevery line of Python code. However, it can still happen in Python3.11+ depending on exactly which tracing functions are active(e.g. interactive debuggers intentionally do this so that changesmade at the debugging prompt are visible when code executionresumes).

All of the above comments in relation toexec() apply toany attempt to mutate the result oflocals() in optimizedscopes, and are the main reason that thelocals() builtindocs contain this caveat:

Note: The contents of this dictionary should not be modified;changes may not affect the values of local and free variablesused by the interpreter.

While the exact wording in the library reference is not entirely explicit,bothexec() andeval() have long used the results of callingglobals() andlocals() in the calling Python frame as their defaultexecution namespace.

This was historically also equivalent to using the calling frame’sframe.f_globals andframe.f_locals attributes, but this PEP mapsthe default namespace arguments forexec() andeval() toglobals() andlocals() in the calling frame in order to preservethe property of defaulting to ignoring attempted writes to the localnamespace in optimized scopes.

This poses a potential compatibility issue for some code, as with theprevious implementation that returns the same dict whenlocals() is calledmultiple times in function scope, the following code usually worked due tothe implicitly shared local variable namespace:

deff():exec('a = 0')# equivalent to exec('a = 0', globals(), locals())exec('print(a)')# equivalent to exec('print(a)', globals(), locals())print(locals())# {'a': 0}# However, print(a) will not work heref()

Withlocals() in an optimised scope returning the same shared dict for each call,it was possible to store extra “fake locals” in that dict. While these aren’t reallocals known by the compiler (so they can’t be printed with code likeprint(a)),they can still be accessed vialocals() and shared between multipleexec()calls in the same function scope. Furthermore, because they’renot real locals,they don’t get implicitly updated or removed when the shared cache is refreshedfrom the local variable storage array.

When the code inexec() tries to write to an existing local variable, theruntime behaviour gets harder to predict:

deff():a=Noneexec('a = 0')# equivalent to exec('a = 0', globals(), locals())exec('print(a)')# equivalent to exec('print(a)', globals(), locals())print(locals())# {'a': None}f()

print(a) will printNone because the implicitlocals() call inexec() refreshes the cached dict with the actual values on the frame.This means that, unlike the “fake” locals created by writing back tolocals()(including via previous calls toexec()), the real locals known by thecompiler can’t easily be modified byexec() (it can be done, but it requiresboth retrieving theframe.f_locals attribute to enable writes back to the frame,and then invokingPyFrame_LocalsToFast(), asshownusingctypes above).

As noted in theMotivation section, this confusing side effecthappens even if the local variable is only definedafter theexec() calls:

>>>deff():...exec("a = 0")...exec("print('a' in locals())")# Printing 'a' directly won't work...print(locals())...a=None...print(locals())...>>>f()False{}{'a': None}

Becausea is a real local variable that is not currently bound to a value, itgets explicitly removed from the dictionary returned bylocals() wheneverlocals() is called prior to thea=None line. This removal is intentional,as it allows the contents oflocals() to be updated correctly in optimizedscopes whendel statements are used to delete previously bound local variables.

As noted in thectypesexample, the above behaviouraldescription may be invalidated if the CPythonPyFrame_LocalsToFast() API gets invokedwhile the frame is still running. In that case, the changes toamight become visibleto the running code, depending on exactly when that API is called (and whether the framehas been primed for locals modification by accessing theframe.f_locals attribute).

As described above, two options were considered to replace this confusing behaviour:

• makelocals() return write-through proxy instances (similartoframe.f_locals)
• makelocals() return genuinely independent snapshots so thatattempts to change the values of local variables viaexec()would beconsistently ignored without any of the caveatsnoted above.

The PEP chooses the second option for the following reasons:

• returning independent snapshots in optimized scopes preservesthe Python 3.0 change toexec() that resulted in attemptsto mutate local variables viaexec() being ignored in mostcases
• the distinction between “locals() gives an instantaneoussnapshot of the local variables in optimized scopes, andread/write access in other scopes” and “frame.f_localsgives read/write access to the local variables in all scopes,including optimized scopes” allows the intent of a piece ofcode to be clearer than it would be if both APIs grantedfull read/write access in optimized scopes, even when writeaccess wasn’t needed or desired
• in addition to improving clarity for human readers, ensuringthat name rebinding in optimized scopes remains lexicallyvisible in the code (as long as the frame introspection APIsare not accessed) allows compilers and interpreters to applyrelated performance optimizations more consistently
• only Python implementations that support the optional frameintrospection APIs will need to provide the new write-throughproxy support for optimized frames

With the semantic changes tolocals() in this PEP, it becomes much easier to explainthe behavior ofexec() andeval(): in optimized scopes, they willnever implicitlyaffect local variables; in other scopes, they willalways implicitly affect localvariables. In optimized scopes, any implicit assignment to the local variables will bediscarded when the code execution API returns, since a fresh copy of the local variablesis used on each invocation.

Retaining the internal frame value cache

Retaining the internal frame value cache results in some visible quirks whenframe proxy instances are kept around and re-used after name binding andunbinding operations have been executed on the frame.

The primary reason for retaining the frame value cache is to maintain backwardscompatibility with thePyEval_GetLocals() API. That API returns a borrowedreference, so it must refer to persistent state stored on the frame object.Storing a fast locals proxy object on the frame creates a problematic referencecycle, so the cleanest option is to instead continue to return a frame valuecache, just as this function has done since optimised frames were firstintroduced.

With the frame value cache being kept around anyway, it then further made senseto rely on it to simplify the fast locals proxy mapping implementation.

Note: the factPEP 667doesn’t use the internal frame value cache as part of thewrite-through proxy implementation is the key Python level difference between the two PEPs.

Changing the frame API semantics in regular operation

Note: when this PEP was first written, it predated the Python 3.11 change to drop theimplicit writeback of the frame local variables whenever a tracing function was installed,so making that change was included as part of the proposal.

Earlier versions of this PEP proposed having the semantics of the framef_locals attribute depend on whether or not a tracing hook was currentlyinstalled - only providing the write-through proxy behaviour when a tracing hookwas active, and otherwise behaving the same as the historicallocals()builtin.

That was adopted as the original design proposal for a couple of key reasons,one pragmatic and one more philosophical:

• Object allocations and method wrappers aren’t free, and tracing functionsaren’t the only operations that access frame locals from outside the function.Restricting the changes to tracing mode meant that the additional memory andexecution time overhead of these changes would be as close to zero in regularoperation as we can possibly make them.
• “Don’t change what isn’t broken”: the current tracing mode problems are causedby a requirement that’s specific to tracing mode (support for externalrebinding of function local variable references), so it made sense to alsorestrict any related fixes to tracing mode

However, actually attempting to implement and document that dynamic approachhighlighted the fact that it makes for a really subtle runtime state dependentbehaviour distinction in howframe.f_locals works, and creates severalnew edge cases around howf_locals behaves as trace functions are addedand removed.

Accordingly, the design was switched to the current one, whereframe.f_locals is always a write-through proxy, andlocals() is alwaysa snapshot, which is both simpler to implement and easier to explain.

Regardless of how the CPython reference implementation chooses to handle this,optimising compilers and interpreters also remain free to impose additionalrestrictions on debuggers, such as making local variable mutation through frameobjects an opt-in behaviour that may disable some optimisations (just as theemulation of CPython’s frame API is already an opt-in flag in some Pythonimplementations).

Continuing to support storing additional data on optimised frames

One of the draft iterations of this PEP proposed removing the ability to storeadditional data on optimised frames by writing toframe.f_locals keys thatdidn’t correspond to local or closure variable names on the underlying frame.

While this idea offered some attractive simplification of the fast locals proxyimplementation,pdb stores__return__ and__exception__ values onarbitrary frames, so the standard library test suite fails if that functionalityno longer works.

Accordingly, the ability to store arbitrary keys was retained, at the expenseof certain operations on proxy objects being slower than could otherwise be(since they can’t assume that only names defined on the code object will beaccessible through the proxy).

It is expected that the exact details of the interaction between the fast localsproxy and thef_locals value cache on the underlying frame will evolve overtime as opportunities for improvement are identified.

Historical semantics at function scope

The current semantics of mutatinglocals() andframe.f_locals in CPythonare rather quirky due to historical implementation details:

• actual execution uses the fast locals array for local variable bindings andcell references for nonlocal variables
• there’s aPyFrame_FastToLocals operation that populates the frame’sf_locals attribute based on the current state of the fast locals arrayand any referenced cells. This exists for three reasons:
  • allowing trace functions to read the state of local variables
  • allowing traceback processors to read the state of local variables
  • allowinglocals() to read the state of local variables
• a direct reference toframe.f_locals is returned fromlocals(), so ifyou hand out multiple concurrent references, then all those references will beto the exact same dictionary
• the two common calls to the reverse operation,PyFrame_LocalsToFast, wereremoved in the migration to Python 3:exec is no longer a statement (andhence can no longer affect function local namespaces), and the compiler nowdisallows the use offrommoduleimport* operations at function scope
• however, two obscure calling paths remain:PyFrame_LocalsToFast is calledas part of returning from a trace function (which allows debuggers to makechanges to the local variable state), and you can also still inject theIMPORT_STAR opcode when creating a function directly from a code objectrather than via the compiler

This proposal deliberatelydoesn’t formalise these semantics as is, since theyonly make sense in terms of the historical evolution of the language and thereference implementation, rather than being deliberately designed.

Proposing several additions to the stable C API/ABI

Historically, the CPython C API (and subsequently, the stable ABI) hasexposed only a single API function related to the Pythonlocals builtin:PyEval_GetLocals(). However, as it returns a borrowed reference, it isnot possible to adapt that interface directly to supporting the newlocals()semantics proposed in this PEP.

An earlier iteration of this PEP proposed a minimalist adaptation to the newsemantics: one C API function that behaved like the Pythonlocals() builtin,and another that behaved like theframe.f_locals descriptor (creating andreturning the write-through proxy if necessary).

The feedback[8] on that version of the C API was that it was too heavily basedon how the Python level semantics were implemented, and didn’t account for thebehaviours that authors of C extensions were likely toneed.

The broader API now being proposed came from grouping the potential reasons forwanting to access the Pythonlocals() namespace from an extension moduleinto the following cases:

• needing to exactly replicate the semantics of the Python levellocals()operation. This is thePyLocals_Get() API.
• needing to behave differently depending on whether writes to the result ofPyLocals_Get() will be visible to Python code or not. This is handled bythePyLocals_GetKind() query API.
• always wanting a mutable namespace that has been pre-populated from thecurrent Pythonlocals() namespace, butnot wanting any changes tobe visible to Python code. This is thePyLocals_GetCopy() API.
• always wanting a read-only view of the current locals namespace, withoutincurring the runtime overhead of making a full copy each time. This isn’treadily offered for optimised frames due to the need to check whether namesare currently bound or not, so no specific API is being added to cover it.

Historically, these kinds of checks and operations would only have beenpossible if a Python implementation emulated the full CPython frame API. Withthe proposed API, extension modules can instead ask more clearly for thesemantics that they actually need, giving Python implementations moreflexibility in how they provide those capabilities.

Comparison with PEP 667

NOTE: the comparison below is against PEP 667 as it was in December 2021.It does not reflect the state of PEP 667 as of April 2024 (when this PEP waswithdrawn in favour of proceeding with PEP 667).

PEP 667 offers a partially competing proposal for this PEP that suggests itwould be reasonable to eliminate the internal frame value cache on optimisedframes entirely.

These changes were originally offered as amendments toPEP 558, and the PEPauthor rejected them for three main reasons:

• the initial claim thatPyEval_GetLocals() was unfixable because it returnsa borrowed reference was simply false, as it is still working in thePEP 558reference implementation. All that is required to keep it working is toretain the internal frame value cache and design the fast locals proxy insuch a way that it is reasonably straightforward to keep the cache up to datewith changes in the frame state without incurring significant runtime overheadwhen the cache isn’t needed. Given that this claim is false, the proposal torequire that all code using thePyEval_GetLocals() API be rewritten to usea new API with different refcounting semantics failsPEP 387’s requirementthat API compatibility breaks should have a large benefit to breakage ratio(since there’s no significant benefit gained from dropping the cache, no codebreakage can be justified). The only genuinely unfixable public API isPyFrame_LocalsToFast() (which is why both PEPs propose breaking that).
• without some form of internal value cache, the API performance characteristicsof the fast locals proxy mapping become quite unintuitive.len(proxy), forexample, becomes consistently O(n) in the number of variables defined on theframe, as the proxy has to iterate over the entire fast locals array to seewhich names are currently bound to values before it can determine the answer.By contrast, maintaining an internal frame value cache potentially allowsproxies to largely be treated as normal dictionaries from an algorithmiccomplexity point of view, with allowances only needing to be made for theinitial implicit O(n) cache refresh that runs the first time an operationthat relies on the cache being up to date is executed.
• the claim that a cache-free implementation would be simpler is highly suspect,asPEP 667 includes only a pure Python sketch of a subset of a mutable mappingimplementation, rather than a full-fledged C implementation of a new mappingtype integrated with the underlying data storage for optimised frames.PEP 558’s fast locals proxy implementation delegates heavily to theframe value cache for the operations needed to fully implement the mutablemapping API, allowing it to re-use the existing dict implementations of thefollowing operations:
  • __len__
  • __str__
  • __or__ (dict union)
  • __iter__ (allowing thedict_keyiterator type to be reused)
  • __reversed__ (allowing thedict_reversekeyiterator type to be reused)
  • keys() (allowing thedict_keys type to be reused)
  • values() (allowing thedict_values type to be reused)
  • items() (allowing thedict_items type to be reused)
  • copy()
  • popitem()
  • value comparison operations

Of the three reasons, the first is the most important (since we need compellingreasons to break API backwards compatibility, and we don’t have them).

However, after reviewingPEP 667’s proposed Python level semantics, the authorof this PEP eventually agreed that theywould be simpler for users of thePythonlocals() API, so this distinction between the two PEPs has beeneliminated: regardless of which PEP and implementation is accepted, the fastlocals proxy objectalways provides a consistent view of the current stateof the local variables, even if this results in some operations becoming O(n)that would be O(1) on a regular dictionary (specifically,len(proxy)becomes O(n), since it needs to check which names are currently bound, and proxymapping comparisons avoid relying on the length check optimisation that allowsdifferences in the number of stored keys to be detected quickly for regularmappings).

Due to the adoption of these non-standard performance characteristics in theproxy implementation, thePyLocals_GetView() andPyFrame_GetLocalsView()C APIs were also removed from the proposal in this PEP.

This leaves the only remaining points of distinction between the two PEPs asspecifically related to the C API:

• PEP 667 still proposes completely unnecessary C API breakage (the programmaticdeprecation and eventual removal ofPyEval_GetLocals(),PyFrame_FastToLocalsWithError(), andPyFrame_FastToLocals()) withoutjustification, when it is entirely possible to keep these working indefinitely(and interoperably) given a suitably designed fast locals proxy implementation
• the fast locals proxy handling of additional variables is defined in this PEPin a way that is fully interoperable with the existingPyEval_GetLocals()API. In the proxy implementation proposed inPEP 667, users of the new frameAPI will not see changes made to additional variables by users of the old API,and changes made to additional variables via the old API will be overwrittenon subsequent calls toPyEval_GetLocals().
• thePyLocals_Get() API in this PEP is calledPyEval_Locals() inPEP 667.This function name is a bit strange as it lacks a verb, making it look morelike a type name than a data access API.
• this PEP addsPyLocals_GetCopy() andPyFrame_GetLocalsCopy() APIs toallow extension modules to easily avoid incurring a double copy operation inframes wherePyLocals_Get() already makes a copy
• this PEP addsPyLocals_Kind,PyLocals_GetKind(), andPyFrame_GetLocalsKind() to allow extension modules to identify when codeis running at function scope without having to inspect non-portable frame andcode object APIs (without the proposed query API, the existing equivalent tothe newPyLocals_GetKind()==PyLocals_SHALLOW_COPY check is to includethe CPython internal frame API headers and check if_PyFrame_GetCode(PyEval_GetFrame())->co_flags&CO_OPTIMIZED is set)

The Python pseudo-code below is based on the implementation sketch presentedinPEP 667 as of the time of writing (2021-10-24). The differences thatprovide the improved interoperability between the new fast locals proxy APIand the existingPyEval_GetLocals() API are noted in comments.

As inPEP 667, all attributes that start with an underscore are invisible andcannot be accessed directly. They serve only to illustrate the proposed design.

For simplicity (and as inPEP 667), the handling of module and class levelframes is omitted (they’re much simpler, as_localsis the executionnamespace, so no translation is required).

NULL:Object# NULL is a singleton representing the absence of a value.classCodeType:_name_to_offset_mapping_impl:dict|NULL...def__init__(self,...):self._name_to_offset_mapping_impl=NULLself._variable_names=deduplicate(self.co_varnames+self.co_cellvars+self.co_freevars)...def_is_cell(self,offset):...# How the interpreter identifies cells is an implementation detail@propertydef_name_to_offset_mapping(self):"Mapping of names to offsets in local variable array."ifself._name_to_offset_mapping_implisNULL:self._name_to_offset_mapping_impl={name:indexfor(index,name)inenumerate(self._variable_names)}returnself._name_to_offset_mapping_implclassFrameType:_fast_locals:array[Object]# The values of the local variables, items may be NULL._locals:dict|NULL# Dictionary returned by PyEval_GetLocals()def__init__(self,...):self._locals=NULL...@propertydeff_locals(self):returnFastLocalsProxy(self)classFastLocalsProxy:__slots__"_frame"def__init__(self,frame:FrameType):self._frame=framedef_set_locals_entry(self,name,val):f=self._frameiff._localsisNULL:f._locals={}f._locals[name]=valdef__getitem__(self,name):f=self._frameco=f.f_codeifnameinco._name_to_offset_mapping:index=co._name_to_offset_mapping[name]val=f._fast_locals[index]ifvalisNULL:raiseKeyError(name)ifco._is_cell(offset)val=val.cell_contentsifvalisNULL:raiseKeyError(name)# PyEval_GetLocals() interop: implicit frame cache refreshself._set_locals_entry(name,val)returnval# PyEval_GetLocals() interop: frame cache may contain additional namesiff._localsisNULL:raiseKeyError(name)returnf._locals[name]def__setitem__(self,name,value):f=self._frameco=f.f_codeifnameinco._name_to_offset_mapping:index=co._name_to_offset_mapping[name]kind=co._local_kinds[index]ifco._is_cell(offset)cell=f._locals[index]cell.cell_contents=valelse:f._fast_locals[index]=val# PyEval_GetLocals() interop: implicit frame cache update# even for names that are part of the fast locals arrayself._set_locals_entry(name,val)def__delitem__(self,name):f=self._frameco=f.f_codeifnameinco._name_to_offset_mapping:index=co._name_to_offset_mapping[name]kind=co._local_kinds[index]ifco._is_cell(offset)cell=f._locals[index]cell.cell_contents=NULLelse:f._fast_locals[index]=NULL# PyEval_GetLocals() interop: implicit frame cache update# even for names that are part of the fast locals arrayiff._localsisnotNULL:delf._locals[name]def__iter__(self):f=self._frameco=f.f_codeforindex,nameinenumerate(co._variable_names):val=f._fast_locals[index]ifvalisNULL:continueifco._is_cell(offset):val=val.cell_contentsifvalisNULL:continueyieldnamefornameinf._locals:# Yield any extra names not defined on the frameifnameinco._name_to_offset_mapping:continueyieldnamedefpopitem(self):f=self._frameco=f.f_codefornameinself:val=self[name]# PyEval_GetLocals() interop: implicit frame cache update# even for names that are part of the fast locals arraydelnamereturnname,valdef_sync_frame_cache(self):# This method underpins PyEval_GetLocals, PyFrame_FastToLocals# PyFrame_GetLocals, PyLocals_Get, mapping comparison, etcf=self._frameco=f.f_coderes=0iff._localsisNULL:f._locals={}forindex,nameinenumerate(co._variable_names):val=f._fast_locals[index]ifvalisNULL:f._locals.pop(name,None)continueifco._is_cell(offset):ifval.cell_contentsisNULL:f._locals.pop(name,None)continuef._locals[name]=valdef__len__(self):self._sync_frame_cache()returnlen(self._locals)

Note: the simplest way to convert the earlier iterations of thePEP 558reference implementation into a preliminary implementation of the now proposedsemantics is to remove theframe_cache_updated checks in affected operations,and instead always sync the frame cache in those methods. Adopting that approachchanges the algorithmic complexity of the following operations as shown (wheren is the number of local and cell variables defined on the frame):

• __len__: O(1) -> O(n)
• value comparison operations: no longer benefit from O(1) length check shortcut
• __iter__: O(1) -> O(n)
• __reversed__: O(1) -> O(n)
• keys(): O(1) -> O(n)
• values(): O(1) -> O(n)
• items(): O(1) -> O(n)
• popitem(): O(1) -> O(n)

The length check and value comparison operations have relatively limitedopportunities for improvement: without allowing usage of a potentially stalecache, the only way to know how many variables are currently bound is to iterateover all of them and check, and if the implementation is going to be spendingthat many cycles on an operation anyway, it may as well spend it updating theframe value cache and then consuming the result. These operations are O(n) inboth this PEP and inPEP 667. Customised implementations could be provided thatare faster than updating the frame cache, but it’s far from clear that theextra code complexity needed to speed these operations up would be worthwhilewhen it only offers a linear performance improvement rather than an algorithmiccomplexity improvement.

The O(1) nature of the other operations can be restored by adding implementationcode that doesn’t rely on the value cache being up to date.

Keeping the iterator/iterable retrieval methods as O(1) will involvewriting custom replacements for the corresponding builtin dict helper types,just as proposed inPEP 667. As illustrated above, the implementations wouldbe similar to the pseudo-code presented inPEP 667, but not identical (due tothe improvedPyEval_GetLocals() interoperability offered by this PEPaffecting the way it stores extra variables).

popitem() can be improved from “always O(n)” to “O(n) worst case” bycreating a custom implementation that relies on the improved iteration APIs.

To ensure stale frame information is never presented in the Python fast localsproxy API, these changes in the reference implementation will need to beimplemented before merging.

The current implementation at time of writing (2021-10-24) also still stores acopy of the fast refs mapping on each frame rather than storing a singleinstance on the underlying code object (as it still stores cell referencesdirectly, rather than check for cells on each fast locals array access). Fixingthis would also be required before merging.