Important
This PEP is a historical document. The up-to-date, canonical documentation can now be found atlocals().
×
SeePEP 1 for how to propose changes.
In early versions of Python all namespaces, whether in functions,classes or modules, were all implemented the same way: as a dictionary.
For performance reasons, the implementation of function namespaces waschanged. Unfortunately this meant that accessing these namespaces throughlocals() andframe.f_locals ceased to be consistent and someodd bugs crept in over the years as threads, generators and coroutineswere added.
This PEP proposes making these namespaces consistent once more.Modifications toframe.f_locals will always be visible inthe underlying variables. Modifications to local variables willimmediately be visible inframe.f_locals, and they will beconsistent regardless of threading or coroutines.
Thelocals() function will act the same as it does now for classand modules scopes. For function scopes it will return an instantaneoussnapshot of the underlyingframe.f_locals rather than implicitlyrefreshing a single shared dictionary cached on the frame object.
The implementation oflocals() andframe.f_locals in releases up to andincluding Python 3.12 is slow, inconsistent and buggy.We want to make it faster, consistent, and most importantly fix the bugs.
For example, when attempting to manipulate local variables via frame objects:
classC:x=1sys._getframe().f_locals['x']=2print(x)
prints2, but:
deff():x=1sys._getframe().f_locals['x']=2print(x)f()
prints1.
This is inconsistent, and confusing. Worse than that, the Python 3.12 behavior canresult in strangebugs.
With this PEP both examples would print2 as the function levelchange would be written directly to the optimized local variables inthe frame rather than to a cached dictionary snapshot.
There are no compensating advantages for the Python 3.12 behavior;it is unreliable and slow.
Thelocals() builtin has its own undesirable behaviours. Refer toPEP 558for additional details on those concerns.
frame.f_locals attribute a write-through proxyThe Python 3.12 implementation offrame.f_locals returns a dictionarythat is created on the fly from the array of local variables. ThePyFrame_LocalsToFast() C API is then called by debuggers and tracefunctions that want to write their changes back to the array (untilPython 3.11, this API was called implicitly after every trace functioninvocation rather than being called explicitly by the trace functions).
This can result in the array and dictionary getting out of sync witheach other. Writes to thef_locals frame attribute may not show up asmodifications to local variables ifPyFrame_LocalsToFast() is nevercalled. Writes to local variables can get lost if a dictionary snapshotcreated before the variables were modified is written back to the frame(sinceevery known variable stored in the snapshot is written back tothe frame, even if the value stored on the frame had changed since thesnapshot was taken).
By makingframe.f_locals return a view on theunderlying frame, these problems go away.frame.f_locals is always insync with the frame because it is a view of it, not a copy of it.
locals() builtin return independent snapshotsPEP 558 considered three potential options for standardising the behavior of thelocals() builtin inoptimized scopes:
locals() on a given frameupdate a single shared snapshot of the local variableslocals() return write-through proxy instances (similartoframe.f_locals)locals() return genuinely independent snapshots so thatattempts to change the values of local variables viaexec()would beconsistently ignored rather than being accepted in some circumstancesThe last option was chosen as the one which could most easily be explained in thelanguage reference, and memorised by users:
locals() builtin gives an instantaneous snapshot of the local variables inoptimized scopes, and read/write access in other scopes; andframe.f_locals gives read/write access to the local variables in all scopes,including optimized scopesThis approach allows the intent of a piece of code to be clearer than it would be if bothAPIs granted full read/write access in optimized scopes, even when write access wasn’tneeded or desired. For additional details on this design decision, refer toPEP 558,especially theMotivation section andAdditional considerations for eval() and exec() in optimized scopes.
This approach is not without its drawbacks, which are coveredin the Backwards Compatibility section below.
frame.f_locals attributeFor module and class scopes (includingexec() andeval()invocations),frame.f_locals is a directreference to the local variable namespace used in code execution.
For function scopes (and otheroptimized scopes)it will be an instance of a new write-through proxy type that can directly modifythe optimized local variable storage array in the underlying frame, as well as thecontents of any cell references to non-local variables.
The view objects fully implement thecollections.abc.Mapping interface,and also implement the following mutable mapping operations:
setdefault() methodupdate() methodViews of different frames compare unequal even if they have the same contents.
All writes to thef_locals mapping will be immediately visiblein the underlying variables. All changes to the underlying variableswill be immediately visible in the mapping.
Thef_locals object will be a full mapping, and can have arbitrarykey-value pairs added to it. New names added via the proxieswill be stored in a dedicated shared dictionary stored on theunderlying frame object (so all proxy instances for a given framewill be able to access any names added this way).
Extra keys (which do not correspond to local variables on the underlyingframe) may be removed as usual withdel statements or thepop()method.
Usingdel, or thepop() method, to remove keys that correspond to localvariables on the underlying frame is NOT supported, and attempting to do sowill raiseValueError.Local variables can only be set toNone (or some other value) via the proxy,they cannot be unbound completely.
Theclear() method is NOT implemented on the write-through proxies, as itis unclear how it should handle the inability to delete entries correspondingto local variables.
To maintain backwards compatibility, proxy APIs that need to produce anew mapping (such ascopy()) will produce regular builtindictinstances, rather than write-through proxy instances.
To avoid introducing a circular reference between frame objects and thewrite-through proxies, each access toframe.f_locals returns anewwrite-through proxy instance.
locals() builtinlocals() will be defined as:
deflocals():frame=sys._getframe(1)f_locals=frame.f_localsifframe._is_optimized():# Not an actual frame methodf_locals=dict(f_locals)returnf_locals
For module and class scopes (includingexec() andeval()invocations),locals() continues to return a directreference to the local variable namespace used in code execution(which is also the same value reported byframe.f_locals).
Inoptimized scopes,each call tolocals() will produce anindependentsnapshot of the local variables.
eval() andexec() builtinsBecause this PEP changes the behavior oflocals(), thebehavior ofeval() andexec() also changes.
Assuming a function_eval() which performs the job ofeval() with explicit namespace arguments,eval()can be defined as follows:
FrameProxyType=type((lambda:sys._getframe().f_locals)())defeval(expression,/,globals=None,locals=None):ifglobalsisNone:# No globals -> use calling frame's globals_calling_frame=sys._getframe(1)globals=_calling_frame.f_globalsiflocalsisNone:# No globals or locals -> use calling frame's localslocals=_calling_frame.f_localsifisinstance(locals,FrameProxyType):# Align with locals() builtin in optimized framelocals=dict(locals)eliflocalsisNone:# Globals but no locals -> use same namespace for bothlocals=globalsreturn_eval(expression,globals,locals)
The specified argument handling forexec() is similarly updated.
(In Python 3.12 and earlier, it was not possible to providelocalstoeval() orexec() without also providingglobals as thesewere previously positional-only arguments. Independently of thisPEP, Python 3.13 updated these builtins to accept keyword arguments)
PyEval C APIThree new C-API functions will be added:
PyObject*PyEval_GetFrameLocals(void)PyObject*PyEval_GetFrameGlobals(void)PyObject*PyEval_GetFrameBuiltins(void)
PyEval_GetFrameLocals() is equivalent to:locals().PyEval_GetFrameGlobals() is equivalent to:globals().
All of these functions will return a new reference.
PyFrame_GetLocals C APIThe existingPyFrame_GetLocals(f) C API is equivalent tof.f_locals.Its return value will be as described above for accessingf.f_locals.
This function returns a new reference, so it is able to accommodate thecreation of a new write-through proxy instance on each call in anoptimized scope.
The following C API functions will be deprecated, as they return borrowed references:
PyEval_GetLocals()PyEval_GetGlobals()PyEval_GetBuiltins()
The following functions (which return new references) should be used instead:
PyEval_GetFrameLocals()PyEval_GetFrameGlobals()PyEval_GetFrameBuiltins()
The following C API functions will become no-ops, and will be deprecated withoutreplacement:
PyFrame_FastToLocalsWithError()PyFrame_FastToLocals()PyFrame_LocalsToFast()
All of the deprecated functions will be marked as deprecated in the Python 3.13 documentation.
Of these functions, onlyPyEval_GetLocals() poses any significant maintenance burden.Accordingly, calls toPyEval_GetLocals() will emitDeprecationWarning in Python3.14, with a target removal date of Python 3.16 (two releases after Python 3.14).Alternatives are recommended as described inPyEval_GetLocals compatibility.
This section summarises how the specified behaviour in Python 3.13 and laterdiffers from the historical behaviour in Python 3.12 and earlier versions.
frame.f_locals changesConsider the following example:
defl():"Get the locals of caller"returnsys._getframe(1).f_localsdeftest():if0:y=1# Make 'y' a local variablex=1l()['x']=2l()['y']=4l()['z']=5yprint(locals(),x)
Given the changes in this PEP,test() will print{'x':2,'y':4,'z':5}2.
In Python 3.12, this example will fail with anUnboundLocalError,as the definition ofy byl()['y']=4 is lost.
If the second-to-last line were changed fromy toz, this will stillraiseNameError, as it does in Python 3.12.Keys added toframe.f_locals that are not lexically local variablesremain visible inframe.f_locals,but do not dynamically become local variables.
locals() changesConsider the following example:
deff():exec("x = 1")print(locals().get("x"))f()
Given the changes in this PEP, this willalways printNone(regardless of whetherx is a defined local variable in the function),as the explicit call tolocals() produces a distinct snapshot fromthe one implicitly used in theexec() call.
In Python 3.12, the exact example shown would print1, but seeminglyunrelated changes to the definition of the function involved could makeit printNone instead (Additional considerations for eval() and exec() in optimized scopes in PEP 558goes into more detail on that topic).
eval() andexec() changesThe primary change affectingeval() andexec() is shownin the “locals() changes” example: repeatedlyaccessinglocals() in an optimized scope will no longerimplicitly share a common underlying namespace.
PyFrame_GetLocals changePyFrame_GetLocals can already return arbitrary mappings in Python 3.12,asexec() andeval() accept arbitrary mappings as theirlocals argument,and metaclasses may return arbitrary mappings from their__prepare__ methods.
Returning a frame locals proxy in optimized scopes just adds another case wheresomething other than a builtin dictionary will be returned.
PyEval_GetLocals changeThe semantics ofPyEval_GetLocals() are technically unchanged, but they do change inpractice as the dictionary cached on optimized frames is no longer shared with othermechanisms for accessing the frame locals (locals() builtin,PyFrame_GetLocalsfunction, framef_locals attributes).
The implementation used in versions up to and including Python 3.12 has manycorner cases and oddities. Code that works around those may need to be changed.Code that useslocals() for simple templating, or print debugging,will continue to work correctly. Debuggers and other tools that usef_locals to modify local variables, will now work correctly,even in the presence of threaded code, coroutines and generators.
frame.f_locals compatibilityAlthoughf.f_locals behaves as if it were the namespace of the function,there will be some observable differences.For example,f.f_localsisf.f_locals will beFalse for optimizedframes, as each access to the attribute produces a new write-through proxyinstance.
Howeverf.f_locals==f.f_locals will beTrue, andall changes to the underlying variables, by any means, including theaddition of new variable names as mapping keys, will always be visible.
locals() compatibilitylocals()islocals() will beFalse for optimized frames, socode like the following will raiseKeyError instead of returning1:
deff():locals()["x"]=1returnlocals()["x"]
To continue working, such code will need to explicitly store the namespaceto be modified in a local variable, rather than relying on the previousimplicit caching on the frame object:
deff():ns={}ns["x"]=1returnns["x"]
While this technically isn’t a formal backwards compatibility break(since the behaviour of writing back tolocals() was explicitlydocumented as undefined), there is definitely some code that relieson the existing behaviour. Accordingly, the updated behaviour willbe explicitly noted in the documentation as a change and it will becovered in the Python 3.13 porting guide.
To work with a copy oflocals() in optimized scopes on allversions without making redundant copies on Python 3.13+, userswill need to define a version-dependent helper function that onlymakes an explicit copy on Python versions prior to Python 3.13:
ifsys.version_info>=(3,13):def_ensure_func_snapshot(d):returnd# 3.13+ locals() already returns a snapshotelse:def_ensure_func_snapshot(d):returndict(d)# Create snapshot on older versionsdeff():ns=_ensure_func_snapshot(locals())ns["x"]=1returnns
In other scopes,locals().copy() can continue to be calledunconditionally without introducing any redundant copies.
exec() andeval()Even though this PEP does not modifyexec() oreval() directly,the semantic change tolocals() impacts the behavior ofexec()andeval() as they default to running code in the calling namespace.
This poses a potential compatibility issue for some code, as with theprevious implementation that returns the same dict whenlocals() is calledmultiple times in function scope, the following code usually worked due tothe implicitly shared local variable namespace:
deff():exec('a = 0')# equivalent to exec('a = 0', globals(), locals())exec('print(a)')# equivalent to exec('print(a)', globals(), locals())print(locals())# {'a': 0}# However, print(a) will not work heref()
With the semantic changes tolocals() in this PEP, theexec('print(a)')' callwill fail withNameError, andprint(locals()) will report an empty dictionary, aseach line will be using its own distinct snapshot of the local variables rather thanimplicitly sharing a single cached snapshot stored on the frame object.
A shared namespace acrossexec() calls can still be obtained by using explicitnamespaces rather than relying on the previously implicitly shared frame namespace:
deff():ns={}exec('a = 0',locals=ns)exec('print(a)',locals=ns)# 0f()
You can even reliably change the variables in the local scope by explicitly usingframe.f_locals, which was not possible before (even usingctypes toinvokePyFrame_LocalsToFast was subject to the state inconsistency problemsdiscussed elsewhere in this PEP):
deff():a=Noneexec('a = 0',locals=sys._getframe().f_locals)print(a)# 0f()
The behavior ofexec() andeval() for module and class scopes (includingnested invocations) is not changed, as the behaviour oflocals() in thosescopes is not changing.
pdb andbdb use theframe.f_locals API, and hence will be able toreliably update local variables even in optimized frames. Implementing thisPEP will resolve several longstanding bugs in these modules relating to threads,generators, coroutines, and other mechanisms that allow concurrent code executionwhile the debugger is active.
Other code execution APIs in the standard library (such as thecode module)do not implicitly accesslocals()orframe.f_locals, but the behaviourof explicitly passing these namespaces will change as described in the rest ofthis PEP (passinglocals() in optimized scopes will no longer implicitlyshare the code execution namespace across calls, passingframe.f_localsin optimized scopes will allow reliable modification of local variables andnonlocal cell references).
PyEval_GetLocals compatibilityPyEval_GetLocals() has never historically distinguished between whether it wasemulatinglocals() orsys._getframe().f_locals at the Python level, as they allreturned references to the same shared cache of the local variable bindings.
With this PEP,locals() changes to return independent snapshots on each call foroptimized frames, andframe.f_locals (along withPyFrame_GetLocals) changes toreturn new write-through proxy instances.
BecausePyEval_GetLocals() returns a borrowed reference, it isn’t possible to updateits semantics to align with either of those alternatives, leaving it as the only remainingAPI that requires a shared cache dictionary stored on the frame object.
While this technically leaves the semantics of the function unchanged, it no longer allowsextra dict entries to be made visible to users of the other APIs, as those APIs are no longeraccessing the same underlying cache dictionary.
WhenPyEval_GetLocals() is being used as an equivalent to the Pythonlocals()builtin,PyEval_GetFrameLocals() should be used instead.
This code:
locals=PyEval_GetLocals();if(locals==NULL){gotoerror_handler;}Py_INCREF(locals);
should be replaced with:
//Equivalentto"locals()"inPythoncodelocals=PyEval_GetFrameLocals();if(locals==NULL){gotoerror_handler;}
WhenPyEval_GetLocals() is being used as an equivalent to callingsys._getframe().f_locals in Python, it should be replaced by callingPyFrame_GetLocals() on the result ofPyEval_GetFrame().
In these cases, the original code should be replaced with:
//Equivalentto"sys._getframe()"inPythoncodeframe=PyEval_GetFrame();if(frame==NULL){gotoerror_handler;}//Equivalentto"frame.f_locals"inPythoncodelocals=PyFrame_GetLocals(frame);frame=NULL;//Minimisevisibilityofborrowedreferenceif(locals==NULL){gotoerror_handler;}
For inlined comprehensions within a function,locals() currently behaves thesame inside or outside of the comprehension, and this will not change. Thebehavior oflocals() inside functions will generally change as specified inthe rest of this PEP.
For inlined comprehensions at module or class scope, callinglocals() withinthe inlined comprehension returns a new dictionary for each call. This PEP willmakelocals() within a function also always return a new dictionary for eachcall, improving consistency; class or module scope inlined comprehensions willappear to behave as if the inlined comprehension is still a distinct function.
Each read offrame.f_locals will create a new proxy object that givesthe appearance of being the mapping of local (including cell and free)variable names to the values of those local variables.
A possible implementation is sketched out below.All attributes that start with an underscore are invisible andcannot be accessed directly.They serve only to illustrate the proposed design.
NULL:Object# NULL is a singleton representing the absence of a value.classCodeType:_name_to_offset_mapping_impl:dict|NULL_cells:frozenset# Set of indexes of cell and free variables...def__init__(self,...):self._name_to_offset_mapping_impl=NULLself._variable_names=deduplicate(self.co_varnames+self.co_cellvars+self.co_freevars)...@propertydef_name_to_offset_mapping(self):"Mapping of names to offsets in local variable array."ifself._name_to_offset_mapping_implisNULL:self._name_to_offset_mapping_impl={name:indexfor(index,name)inenumerate(self._variable_names)}returnself._name_to_offset_mapping_implclassFrameType:_locals:array[Object]# The values of the local variables, items may be NULL._extra_locals:dict|NULL# Dictionary for storing extra locals not in _locals._locals_cache:FrameLocalsProxy|NULL# required to support PyEval_GetLocals()def__init__(self,...):self._extra_locals=NULLself._locals_cache=NULL...@propertydeff_locals(self):returnFrameLocalsProxy(self)classFrameLocalsProxy:"Implements collections.MutableMapping."__slots__=("_frame",)def__init__(self,frame:FrameType):self._frame=framedef__getitem__(self,name):f=self._frameco=f.f_codeifnameinco._name_to_offset_mapping:index=co._name_to_offset_mapping[name]val=f._locals[index]ifvalisNULL:raiseKeyError(name)ifindexinco._cellsval=val.cell_contentsifvalisNULL:raiseKeyError(name)returnvalelse:iff._extra_localsisNULL:raiseKeyError(name)returnf._extra_locals[name]def__setitem__(self,name,value):f=self._frameco=f.f_codeifnameinco._name_to_offset_mapping:index=co._name_to_offset_mapping[name]kind=co._local_kinds[index]ifindexinco._cellscell=f._locals[index]cell.cell_contents=valelse:f._locals[index]=valelse:iff._extra_localsisNULL:f._extra_locals={}f._extra_locals[name]=valdef__iter__(self):f=self._frameco=f.f_codeyield fromiter(f._extra_locals)forindex,nameinenumerate(co._variable_names):val=f._locals[index]ifvalisNULL:continueifindexinco._cells:val=val.cell_contentsifvalisNULL:continueyieldnamedef__contains__(self,item):f=self._frameifiteminf._extra_locals:returnTruereturniteminco._variable_namesdef__len__(self):f=self._frameco=f.f_coderes=0forindex,_inenumerate(co._variable_names):val=f._locals[index]ifvalisNULL:continueifindexinco._cells:ifval.cell_contentsisNULL:continueres+=1returnlen(self._extra_locals)+res
PyEval_GetLocals() will be implemented roughly as follows:
PyObject*PyEval_GetLocals(void){PyFrameObject*=...;//Getthecurrentframe.if(frame->_locals_cache==NULL){frame->_locals_cache=PyEval_GetFrameLocals();}else{PyDict_Update(frame->_locals_cache,PyFrame_GetLocals(frame));}returnframe->_locals_cache;}
As with all functions that return a borrowed reference, care must be taken toensure that the reference is not used beyond the lifetime of the object.
When accepted, the PEP text suggested thatPyEval_GetLocals would start returning acached instance of the new write-through proxy, while the implementation sketch indicatedit would continue to return a dictionary snapshot cached on the frame instance. Thisdiscrepancy was identified while implementing the PEP, andresolved by the Steering Councilin favour of retaining the Python 3.12 behaviour of returning a dictionary snapshotcached on the frame instance.The PEP text has been updated accordingly.
During the discussions of the C API clarification, it also became apparent that therationale behindlocals() being updated to return independent snapshots inoptimized scopes wasn’t clear, as it had been inheritedfrom the originalPEP 558 discussions rather than being independently covered in thisPEP. The PEP text has been updated to better cover this change, with additional updatesto the Specification and Backwards Compatibility sections to cover the impact on codeexecution APIs that default to executing code in thelocals() namespace. Additionalmotivation and rationale details have also been added toPEP 558.
In 3.13.0, the write-through proxies did not allow deletion of even extra variableswithdel andpop(). This was subsequently reported as acompatibility regression,andresolved as now describedinThe frame.f_locals attribute.
This PEP andPEP 558 shared a common goal:to make the semantics oflocals() andframe.f_locals()intelligible, and their operation reliable.
The key difference between this PEP and PEP 558 is thatPEP 558 attempted to store extra variables inside a fullinternal dictionary copy of the local variables in an effortto improve backwards compatibility with the legacyPyEval_GetLocals() API, whereas this PEP does not (it storesthe extra local variables in a dedicated dictionary accessedsolely via the new frame proxy objects, and copies them to thePyEval_GetLocals() shared dict only when requested).
PEP 558 did not specify exactly when that internal copy wasupdated, making the behavior of PEP 558 impossible to reasonabout in several cases where this PEP remains well specified.
PEP 558 also proposed the introduction of some additional Pythonscope introspection interfaces to the C API that would allowextension modules to more easily determine whether the currentlyactive Python scope is optimized or not, and hence whetherthe C API’slocals() equivalent returns a direct referenceto the frame’s local execution namespace or a shallow copy ofthe frame’s local variables and nonlocal cell references.Whether or not to add such introspection APIs is independentof the proposed changes tolocals() andframe.f_localsand hence no such proposals have been included in this PEP.
PEP 558 wasultimately withdrawnin favour of this PEP.
The implementation is in development as adraft pull request on GitHub.
This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.
Source:https://github.com/python/peps/blob/main/peps/pep-0667.rst
Last modified:2024-10-27 07:11:46 GMT