Currentlyobject.__getattribute__ andsuper.__getattribute__ peekin the__dict__ of classes on the MRO for a class when looking foran attribute. This PEP adds an optional__getdescriptor__ method toa metaclass that replaces this behavior and gives more control over attributelookup, especially when using asuper object.
That is, the MRO walking loop in_PyType_Lookup andsuper.__getattribute__ gets changed from:
deflookup(mro_list,name):forclsinmro_list:ifnameincls.__dict__:returncls.__dict__returnNotFound
to:
deflookup(mro_list,name):forclsinmro_list:try:returncls.__getdescriptor__(name)exceptAttributeError:passreturnNotFound
The default implementation of__getdescriptor__ looks in the classdictionary:
classtype:def__getdescriptor__(cls,name):try:returncls.__dict__[name]exceptKeyError:raiseAttributeError(name)fromNone
This PEP is deferred until someone has time to update this PEP and push it forward.
It is currently not possible to influence how thesuper class looksup attributes (that is,super.__getattribute__ unconditionallypeeks in the class__dict__), and that can be problematic fordynamic classes that can grow new methods on demand, for example dynamicproxy classes.
The__getdescriptor__ method makes it possible to dynamically addattributes even when looking them up using thesuper class.
The new method affectsobject.__getattribute__ (andPyObject_GenericGetAttr) as well for consistency and to have a singleplace to implement dynamic attribute resolution for classes.
The current behavior ofsuper.__getattribute__ causes problems forclasses that are dynamic proxies for other (non-Python) classes or types,an example of which isPyObjC. PyObjC creates a Python class for everyclass in the Objective-C runtime, and looks up methods in the Objective-Cruntime when they are used. This works fine for normal access, but doesn’twork for access withsuper objects. Because of this PyObjC currentlyincludes a customsuper that must be used with its classes, as well ascompletely reimplementingPyObject_GenericGetAttr for normal attributeaccess.
The API in this PEP makes it possible to remove the customsuper andsimplifies the implementation because the custom lookup behavior can beadded in a central location.
Note
PyObjC cannot precalculate the contents of the class__dict__because Objective-C classes can grow new methods at runtime. Furthermore,Objective-C classes tend to contain a lot of methods while most Pythoncode will only use a small subset of them, this makes precalculatingunnecessarily expensive.
Bothsuper.__getattribute__ andobject.__getattribute__ (orPyObject_GenericGetAttr and in particular_PyType_Lookup in C code)walk an object’s MRO and currently peek in the class’__dict__ to look upattributes.
With this proposal both lookup methods no longer peek in the class__dict__but call the special method__getdescriptor__, which is a slot definedon the metaclass. The default implementation of that method looksup the name the class__dict__, which means that attribute lookup isunchanged unless a metatype actually defines the new special method.
The attribute resolution process as implemented byobject.__getattribute__(orPyObject_GenericGetAttr in CPython’s implementation) is fairlystraightforward, but not entirely so without reading C code.
The current CPython implementation of object.__getattribute__ is basicallyequivalent to the following (pseudo-) Python code (excluding some housekeeping and speed tricks):
def_PyType_Lookup(tp,name):mro=tp.mro()assertisinstance(mro,tuple)forbaseinmro:assertisinstance(base,type)# PEP 447 will change these lines:try:returnbase.__dict__[name]exceptKeyError:passreturnNoneclassobject:def__getattribute__(self,name):assertisinstance(name,str)tp=type(self)descr=_PyType_Lookup(tp,name)f=NoneifdescrisnotNone:f=descr.__get__iffisnotNoneanddescr.__set__isnotNone:# Data descriptorreturnf(descr,self,type(self))dict=self.__dict__ifdictisnotNone:try:returnself.__dict__[name]exceptKeyError:passiffisnotNone:# Non-data descriptorreturnf(descr,self,type(self))ifdescrisnotNone:# Regular class attributereturndescrraiseAttributeError(name)classsuper:def__getattribute__(self,name):assertisinstance(name,unicode)ifname!='__class__':starttype=self.__self_type__mro=startype.mro()try:idx=mro.index(self.__thisclass__)exceptValueError:passelse:forbaseinmro[idx+1:]:# PEP 447 will change these lines:try:descr=base.__dict__[name]exceptKeyError:continuef=descr.__get__iffisnotNone:returnf(descr,Noneif(self.__self__isself.__self_type__)elseself.__self__,starttype)else:returndescrreturnobject.__getattribute__(self,name)
This PEP should change the dict lookup at the lines starting at “# PEP 447” witha method call to perform the actual lookup, making is possible to affect thatlookup both for normal attribute access and access through thesuper proxy.
Note that specific classes can already completely override the defaultbehaviour by implementing their own__getattribute__ slot (with or withoutcalling the super class implementation).
A meta type can define a method__getdescriptor__ that is called duringattribute resolution by bothsuper.__getattribute__andobject.__getattribute:
classMetaType(type):def__getdescriptor__(cls,name):try:returncls.__dict__[name]exceptKeyError:raiseAttributeError(name)fromNone
The__getdescriptor__ method has as its arguments a class (which is aninstance of the meta type) and the name of the attribute that is looked up.It should return the value of the attribute without invoking descriptors,and should raiseAttributeError when the name cannot be found.
Thetype class provides a default implementation for__getdescriptor__,that looks up the name in the class dictionary.
The code below implements a silly metaclass that redirects attribute lookup touppercase versions of names:
classUpperCaseAccess(type):def__getdescriptor__(cls,name):try:returncls.__dict__[name.upper()]exceptKeyError:raiseAttributeError(name)fromNoneclassSillyObject(metaclass=UpperCaseAccess):defm(self):return42defM(self):return"fortytwo"obj=SillyObject()assertobj.m()=="fortytwo"
As mentioned earlier in this PEP a more realistic use case of thisfunctionality is a__getdescriptor__ method that dynamically populates theclass__dict__ based on attribute access, primarily when it is notpossible to reliably keep the class dict in sync with its source, for examplebecause the source used to populate__dict__ is dynamic as well and doesnot have triggers that can be used to detect changes to that source.
An example of that are the class bridges in PyObjC: the class bridge is aPython object (class) that represents an Objective-C class and conceptuallyhas a Python method for every Objective-C method in the Objective-C class.As with Python it is possible to add new methods to an Objective-C class, orreplace existing ones, and there are no callbacks that can be used to detectthis.
A new type flagPy_TPFLAGS_GETDESCRIPTOR with value(1UL<<11) thatindicates that the new slot is present and to be used.
A new slottp_getdescriptor is added to thePyTypeObject struct, thisslot corresponds to the__getdescriptor__ method ontype.
The slot has the following prototype:
PyObject*(*getdescriptorfunc)(PyTypeObject*cls,PyObject*name);
This method should lookupname in the namespace ofcls, without looking atsuperclasses, and should not invoke descriptors. The method returnsNULLwithout setting an exception when thename cannot be found, and returns anew reference otherwise (not a borrowed reference).
Classes with atp_getdescriptor slot must addPy_TPFLAGS_GETDESCRIPTORtotp_flags to indicate that new slot must be used.
The new method is required for metatypes and as such is defined ontype_.Bothsuper.__getattribute__ andobject.__getattribute__/PyObject_GenericGetAttr(through_PyType_Lookup) use the this__getdescriptor__ method whenwalking the MRO.
The change forPyObject_GenericGetAttr will be done by changing the privatefunction_PyType_Lookup. This currently returns a borrowed reference, butmust return a new reference when the__getdescriptor__ method is present.Because of this_PyType_Lookup will be renamed to_PyType_LookupName,this will cause compile-time errors for all out-of-tree users of thisprivate API.
For the same reason_PyType_LookupId is renamed to_PyType_LookupId2.A number of other functions in typeobject.c with the same issue do not getan updated name because they are private to that file.
The attribute lookup cache inObjects/typeobject.c is disabled for classesthat have a metaclass that overrides__getdescriptor__, because using thecache might not be valid for such classes.
Use of the method introduced in this PEP can affect introspection of classeswith a metaclass that uses a custom__getdescriptor__ method. This sectionlists those changes.
The items listed below are only affected by custom__getdescriptor__methods, the default implementation forobject won’t cause problemsbecause that still only uses the class__dict__ and won’t cause visiblechanges to the visible behaviour of theobject.__getattribute__.
dir might not show all attributesAs with a custom__getattribute__ methoddir() might not see all(instance) attributes when using the__getdescriptor__() method todynamically resolve attributes.
The solution for that is quite simple: classes using__getdescriptor__should also implement__dir__() if they want full support for the builtindir() function.
inspect.getattr_static might not show all attributesThe functioninspect.getattr_static intentionally does not invoke__getattribute__ and descriptors to avoid invoking user code duringintrospection with this function. The__getdescriptor__ method will alsobe ignored and is another way in which the result ofinspect.getattr_staticcan be different from that ofbuiltin.getattr.
inspect.getmembers andinspect.classify_class_attrsBoth of these functions directly access the class __dict__ of classes alongthe MRO, and hence can be affected by a custom__getdescriptor__ method.
Code with a custom__getdescriptor__ method that want to play nice withthese methods also needs to ensure that the__dict__ is set up correctlywhen that is accessed directly by Python code.
Note thatinspect.getmembers is used bypydoc and hence this canaffect runtime documentation introspection.
__dict__Any code that directly access the class__dict__ for introspectioncan be affected by a custom__getdescriptor__ method, see the previousitem.
WARNING: The benchmark results in this section are old, and will be updatedwhen I’ve ported the patch to the current trunk. I don’t expect significantchanges to the results in this section.
Issue 18181 has a micro benchmark as one of its attachments(pep447-micro-bench.py) that specifically tests the speed of attributelookup, both directly and through super.
Note that attribute lookup with deep class hierarchies is significantly slowerwhen using a custom__getdescriptor__ method. This is because theattribute lookup cache for CPython cannot be used when having this method.
The pybench output below compares an implementation of this PEP with theregular source tree, both based on changeset a5681f50bae2, run on an idlemachine and Core i7 processor running Centos 6.4.
Even though the machine was idle there were clear differences between runs,I’ve seen difference in “minimum time” vary from -0.1% to +1.5%, with similar(but slightly smaller) differences in the “average time” difference.
-------------------------------------------------------------------------------PYBENCH2.1-------------------------------------------------------------------------------*usingCPython3.4.0a0(default,Jul292013,13:01:34)[GCC4.4.720120313(RedHat4.4.7-3)]*disabledgarbagecollection*systemcheckintervalsettomaximum:2147483647*usingtimer:time.perf_counter*timer:resolution=1e-09,implementation=clock_gettime(CLOCK_MONOTONIC)-------------------------------------------------------------------------------Benchmark:pep447.pybench-------------------------------------------------------------------------------Rounds:10Warp:10Timer:time.perf_counterMachineDetails:PlatformID:Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-FinalProcessor:x86_64Python:Implementation:CPythonExecutable:/tmp/default-pep447/bin/python3Version:3.4.0a0Compiler:GCC4.4.720120313(RedHat4.4.7-3)Bits:64bitBuild:Jul29201314:09:12(#default)Unicode:UCS4-------------------------------------------------------------------------------Comparingwith:default.pybench-------------------------------------------------------------------------------Rounds:10Warp:10Timer:time.perf_counterMachineDetails:PlatformID:Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-FinalProcessor:x86_64Python:Implementation:CPythonExecutable:/tmp/default/bin/python3Version:3.4.0a0Compiler:GCC4.4.720120313(RedHat4.4.7-3)Bits:64bitBuild:Jul29201313:01:34(#default)Unicode:UCS4Testminimumrun-timeaveragerun-timethisotherdiffthisotherdiff-------------------------------------------------------------------------------BuiltinFunctionCalls:45ms44ms+1.3%45ms44ms+1.3%BuiltinMethodLookup:26ms27ms-2.4%27ms27ms-2.2%CompareFloats:33ms34ms-0.7%33ms34ms-1.1%CompareFloatsIntegers:66ms67ms-0.9%66ms67ms-0.8%CompareIntegers:51ms50ms+0.9%51ms50ms+0.8%CompareInternedStrings:34ms33ms+0.4%34ms34ms-0.4%CompareLongs:29ms29ms-0.1%29ms29ms-0.0%CompareStrings:43ms44ms-1.8%44ms44ms-1.8%ComplexPythonFunctionCalls:44ms42ms+3.9%44ms42ms+4.1%ConcatStrings:33ms33ms-0.4%33ms33ms-1.0%CreateInstances:47ms48ms-2.9%47ms49ms-3.4%CreateNewInstances:35ms36ms-2.5%36ms36ms-2.5%CreateStringsWithConcat:69ms70ms-0.7%69ms70ms-0.9%DictCreation:52ms50ms+3.1%52ms50ms+3.0%DictWithFloatKeys:40ms44ms-10.1%43ms45ms-5.8%DictWithIntegerKeys:32ms36ms-11.2%35ms37ms-4.6%DictWithStringKeys:29ms34ms-15.7%35ms40ms-11.0%ForLoops:30ms29ms+2.2%30ms29ms+2.2%IfThenElse:38ms41ms-6.7%38ms41ms-6.9%ListSlicing:36ms36ms-0.7%36ms37ms-1.3%NestedForLoops:43ms45ms-3.1%43ms45ms-3.2%NestedListComprehensions:39ms40ms-1.7%39ms40ms-2.1%NormalClassAttribute:86ms82ms+5.1%86ms82ms+5.0%NormalInstanceAttribute:42ms42ms+0.3%42ms42ms+0.0%PythonFunctionCalls:39ms38ms+3.5%39ms38ms+2.8%PythonMethodCalls:51ms49ms+3.0%51ms50ms+2.8%Recursion:67ms68ms-1.4%67ms68ms-1.4%SecondImport:41ms36ms+12.5%41ms36ms+12.6%SecondPackageImport:45ms40ms+13.1%45ms40ms+13.2%SecondSubmoduleImport:92ms95ms-2.4%95ms98ms-3.6%SimpleComplexArithmetic:28ms28ms-0.1%28ms28ms-0.2%SimpleDictManipulation:57ms57ms-1.0%57ms58ms-1.0%SimpleFloatArithmetic:29ms28ms+4.7%29ms28ms+4.9%SimpleIntFloatArithmetic:37ms41ms-8.5%37ms41ms-8.7%SimpleIntegerArithmetic:37ms41ms-9.4%37ms42ms-10.2%SimpleListComprehensions:33ms33ms-1.9%33ms34ms-2.9%SimpleListManipulation:28ms30ms-4.3%29ms30ms-4.1%SimpleLongArithmetic:26ms26ms+0.5%26ms26ms+0.5%SmallLists:40ms40ms+0.1%40ms40ms+0.1%SmallTuples:46ms47ms-2.4%46ms48ms-3.0%SpecialClassAttribute:126ms120ms+4.7%126ms121ms+4.4%SpecialInstanceAttribute:42ms42ms+0.6%42ms42ms+0.8%StringMappings:94ms91ms+3.9%94ms91ms+3.8%StringPredicates:48ms49ms-1.7%48ms49ms-2.1%StringSlicing:45ms45ms+1.4%46ms45ms+1.5%TryExcept:23ms22ms+4.9%23ms22ms+4.8%TryFinally:32ms32ms-0.1%32ms32ms+0.1%TryRaiseExcept:17ms17ms+0.9%17ms17ms+0.5%TupleSlicing:49ms48ms+1.1%49ms49ms+1.0%WithFinally:48ms47ms+2.3%48ms47ms+2.4%WithRaiseExcept:45ms44ms+0.8%45ms45ms+0.5%-------------------------------------------------------------------------------Totals:2284ms2287ms-0.1%2306ms2308ms-0.1%(this=pep447.pybench,other=default.pybench)
A run of the benchmark suite (with option “-b 2n3”) also seems to indicate thatthe performance impact is minimal:
ReportonLinuxfangorn.local2.6.32-358.114.1.openstack.el6.x86_64#1 SMP Wed Jul 3 02:11:25 EDT 2013 x86_64 x86_64TotalCPUcores:8### call_method_slots ###Min:0.304120->0.282791:1.08xfasterAvg:0.304394->0.282906:1.08xfasterSignificant(t=2329.92)Stddev:0.00016->0.00004:4.1814xsmaller### call_simple ###Min:0.249268->0.221175:1.13xfasterAvg:0.249789->0.221387:1.13xfasterSignificant(t=2770.11)Stddev:0.00012->0.00013:1.1101xlarger### django_v2 ###Min:0.632590->0.601519:1.05xfasterAvg:0.635085->0.602653:1.05xfasterSignificant(t=321.32)Stddev:0.00087->0.00051:1.6933xsmaller### fannkuch ###Min:1.033181->0.999779:1.03xfasterAvg:1.036457->1.001840:1.03xfasterSignificant(t=260.31)Stddev:0.00113->0.00070:1.6112xsmaller### go ###Min:0.526714->0.544428:1.03xslowerAvg:0.529649->0.547626:1.03xslowerSignificant(t=-93.32)Stddev:0.00136->0.00136:1.0028xsmaller### iterative_count ###Min:0.109748->0.116513:1.06xslowerAvg:0.109816->0.117202:1.07xslowerSignificant(t=-357.08)Stddev:0.00008->0.00019:2.3664xlarger### json_dump_v2 ###Min:2.554462->2.609141:1.02xslowerAvg:2.564472->2.620013:1.02xslowerSignificant(t=-76.93)Stddev:0.00538->0.00481:1.1194xsmaller### meteor_contest ###Min:0.196336->0.191925:1.02xfasterAvg:0.196878->0.192698:1.02xfasterSignificant(t=61.86)Stddev:0.00053->0.00041:1.2925xsmaller### nbody ###Min:0.228039->0.235551:1.03xslowerAvg:0.228857->0.236052:1.03xslowerSignificant(t=-54.15)Stddev:0.00130->0.00029:4.4810xsmaller### pathlib ###Min:0.108501->0.105339:1.03xfasterAvg:0.109084->0.105619:1.03xfasterSignificant(t=311.08)Stddev:0.00022->0.00011:1.9314xsmaller### regex_effbot ###Min:0.057905->0.056447:1.03xfasterAvg:0.058055->0.056760:1.02xfasterSignificant(t=79.22)Stddev:0.00006->0.00015:2.7741xlarger### silent_logging ###Min:0.070810->0.072436:1.02xslowerAvg:0.070899->0.072609:1.02xslowerSignificant(t=-191.59)Stddev:0.00004->0.00008:2.2640xlarger### spectral_norm ###Min:0.290255->0.299286:1.03xslowerAvg:0.290335->0.299541:1.03xslowerSignificant(t=-572.10)Stddev:0.00005->0.00015:2.8547xlarger### threaded_count ###Min:0.107215->0.115206:1.07xslowerAvg:0.107488->0.115996:1.08xslowerSignificant(t=-109.39)Stddev:0.00016->0.00076:4.8665xlargerThefollowingnotsignificantresultsarehidden,use-vtoshowthem:call_method,call_method_unknown,chaos,fastpickle,fastunpickle,float,formatted_logging,hexiom2,json_load,normal_startup,nqueens,pidigits,raytrace,regex_compile,regex_v8,richards,simple_logging,startup_nosite,telco,unpack_sequence.
__getattribute_super__An earlier version of this PEP used the following static method on classes:
def__getattribute_super__(cls,name,object,owner):pass
This method performed name lookup as well as invoking descriptors and wasnecessarily limited to working only withsuper.__getattribute__.
tp_getattroIt would be nice to avoid adding a new slot, thus keeping the API simpler andeasier to understand. A comment onIssue 18181 asked about reusing thetp_getattro slot, that is super could call thetp_getattro slot of allmethods along the MRO.
That won’t work becausetp_getattro will look in the instance__dict__ before it tries to resolve attributes using classes in the MRO.This would mean that usingtp_getattro instead of peeking the classdictionaries changes the semantics of thesuper class.
This PEP proposes to add__getdescriptor__ as a method on the metaclass.An alternative would be to add it as a class method on the class itself(similar to how__new__ is astaticmethod of the class and not a methodof the metaclass).
The advantage of using a method on the metaclass is that will give an errorwhen two classes on the MRO have different metaclasses that may have differentbehaviors for__getdescriptor__. With a normal classmethod that problemwould pass undetected while it might cause subtle errors when running the code.
Py_TPFLAGS_GETDESCRIPTOR after talkingwith Guido.The new flag is primarily useful to avoid crashing when loading an extensionfor an older version of CPython and could have positive speed implicationsas well.
__getdescriptor__, the old name didn’tmatch the naming style of other slots and was less descriptive.This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0447.rst
Last modified:2025-02-01 08:55:40 GMT