Movatterモバイル変換


[0]ホーム

URL:


Following system colour schemeSelected dark colour schemeSelected light colour scheme

Python Enhancement Proposals

PEP 575 – Unifying function/method classes

PEP 575 – Unifying function/method classes

Author:
Jeroen Demeyer <J.Demeyer at UGent.be>
Status:
Withdrawn
Type:
Standards Track
Created:
27-Mar-2018
Python-Version:
3.8
Post-History:
31-Mar-2018, 12-Apr-2018, 27-Apr-2018, 05-May-2018

Table of Contents

Withdrawal notice

SeePEP 580 for a better solution to allowing fast calling of custom classes.

SeePEP 579 for a broader discussion of some of the other issues from this PEP.

Abstract

Reorganize the class hierarchy for functions and methodswith the goal of reducing the difference betweenbuilt-in functions (implemented in C) and Python functions.Mainly, make built-in functions behave more like Python functionswithout sacrificing performance.

A new base classbase_function is introduced and the various functionclasses, as well asmethod (renamed tobound_method), inherit from it.

We also allow subclassing the Pythonfunction class.

Motivation

Currently, CPython has two different function classes:the first is Python functions, which is what you getwhen defining a function withdef orlambda.The second is built-in functions such aslen,isinstance ornumpy.dot.These are implemented in C.

These two classes are implemented completely independently and have different functionality.In particular, it is currently not possible to implement a function efficiently in C(only built-in functions can do that)while still allowing introspection likeinspect.signature orinspect.getsourcefile(only Python functions can do that).This is a problem for projects like Cython[1] that want to do exactly that.

In Cython, this was worked around by inventing a new function class calledcyfunction.Unfortunately, a new function class creates problems:theinspect module does not recognize such functions as being functions[2]and the performance is worse(CPython has specific optimizations for calling built-in functions).

A second motivation is more generally making built-in functions and methodsbehave more like Python functions and methods.For example, Python unbound methods are just functions butunbound methods of extension types (e.g.dict.get) are a distinct class.Bound methods of Python classes have a__func__ attribute,bound methods of extension types do not.

Third, this PEP allows great customization of functions.Thefunction class becomes subclassable and custom functionsubclasses are also allowed for functions implemented in C.In the latter case, this can be done with the same performanceas true built-in functions.All functions can access the function object(theself in__call__), paving the way forPEP 573.

New classes

This is the new class hierarchy for functions and methods:

object||base_function/|     \/|      \/|defined_function/|        \cfunction(*)|         \|function|bound_method(*)

The two classes marked with (*) donot allow subclassing;the others do.

There is no difference between functions and unbound methods,while bound methods are instances ofbound_method.

base_function

The classbase_function becomes a new base class for all function types.It is based on the existingbuiltin_function_or_method class,but with the following differences and new features:

  1. It acts as a descriptor implementing__get__ to turn a function into a methodifm_self isNULL.Ifm_self is notNULL,then this is a no-op: the existing function is returned instead.
  2. A new read-only attribute__parent__, represented in the C structure asm_parent.If this attribute exists, it represents the defining object.For methods of extension types, this is the defining class (__class__ in plain Python)and for functions of a module, this is the defining module.In general, it can be any Python object.If__parent__ is a class, it carries special semantics:in that case, the function must be called withself being an instance of that class.Finally,__qualname__ and__reduce__ will use__parent__as namespace (instead of__self__ before).
  3. A new attribute__objclass__ which equals__parent__ if__parent__is a class. Otherwise, accessing__objclass__ raisesAttributeError.This is meant to be backwards compatible withmethod_descriptor.
  4. The fieldml_doc and the attributes__doc__ and__text_signature__ (seeArgument Clinic)are not supported.
  5. A new flagMETH_PASS_FUNCTION forml_flags.If this flag is set, the C function stored inml_meth is called withan additional first argument equal to the function object.
  6. A new flagMETH_BINDING forml_flags which only applies tofunctions of a module (not methods of a class).If this flag is set, thenm_self is set toNULL insteadof the module.This allows the function to behave more like a Python functionas it enables__get__.
  7. A new flagMETH_CALL_UNBOUND to disableself slicing.
  8. A new flagMETH_PYTHON forml_flags.This flag indicates that this function should be treated as Python function.Ideally, use of this flag should be avoided because it goesagainst the duck typing philosophy.It is still needed in a few places though, for exampleprofiling.

The goal ofbase_function is that it supports all different waysof calling functions and methods in just one structure.For example, the new flagMETH_PASS_FUNCTIONwill be used by the implementation of methods.

It is not possible to directly create instances ofbase_function(tp_new isNULL).However, it is legal for C code to manually create instances.

These are the relevant C structures:

PyTypeObjectPyBaseFunction_Type;typedefstruct{PyObject_HEADPyCFunctionDef*m_ml;/*DescriptionoftheCfunctiontocall*/PyObject*m_self;/*__self__:anything,canbeNULL;readonly*/PyObject*m_module;/*__module__:anything(typicallystr)*/PyObject*m_parent;/*__parent__:anything,canbeNULL;readonly*/PyObject*m_weakreflist;/*Listofweakreferences*/}PyBaseFunctionObject;typedefstruct{constchar*ml_name;/*Thenameofthebuilt-infunction/method*/PyCFunctionml_meth;/*TheCfunctionthatimplementsit*/intml_flags;/*CombinationofMETH_xxxflags,whichmostlydescribetheargsexpectedbytheCfunc*/}PyCFunctionDef;

Subclasses may extendPyCFunctionDef with extra fields.

The Python attribute__self__ returnsm_self,except ifMETH_STATIC is set.In that case or ifm_self isNULL,then there is no__self__ attribute at all.For that reason, we write eitherm_self or__self__ in this PEPwith slightly different meanings.

cfunction

This is the new version of the oldbuiltin_function_or_method class.The namecfunction was chosen to avoid confusion with “built-in”in the sense of “something in thebuiltins module”.It also fits better with the C API which use thePyCFunction prefix.

The classcfunction is a copy ofbase_function, with the following differences:

  1. m_ml points to aPyMethodDef structure,extendingPyCFunctionDef with an additionalml_docfield to implement__doc__ and__text_signature__as read-only attributes:
    typedefstruct{constchar*ml_name;PyCFunctionml_meth;intml_flags;constchar*ml_doc;}PyMethodDef;

    Note thatPyMethodDef is part of thePython Stable ABIand it is used by practically all extension modules,so we absolutely cannot change this structure.

  2. Argument Clinic is supported.
  3. __self__ always exists. In the cases wherebase_function.__self__would raiseAttributeError, insteadNone is returned.

The type object isPyTypeObjectPyCFunction_Typeand we definePyCFunctionObject as alias ofPyBaseFunctionObject(except for the type ofm_ml).

defined_function

The classdefined_function is an abstract base class meantto indicate that the function has introspection support.Instances ofdefined_function are required to support all attributesthat Python functions have, namely__code__,__globals__,__doc__,__defaults__,__kwdefaults__,__closure__ and__annotations__.There is also a__dict__ to support attributes added by the user.

None of these is required to be meaningful.In particular,__code__ may not be a working code object,possibly only a few fields may be filled in.This PEP does not dictate how the various attributes are implemented.They may be simple struct members or more complicated descriptors.Only read-only support is required, none of the attributes is required to be writable.

The classdefined_function is mainly meant for auto-generated C code,for example produced by Cython[1].There is no API to create instances of it.

The C structure is the following:

PyTypeObjectPyDefinedFunction_Type;typedefstruct{PyBaseFunctionObjectbase;PyObject*func_dict;/*__dict__:dictorNULL*/}PyDefinedFunctionObject;

TODO: maybe find a better name fordefined_function.Other proposals:inspect_function (anything that satisfiesinspect.isfunction),builtout_function (a function that is better built out; pun on builtin),generic_function (original proposal but conflicts withfunctools.singledispatch generic functions),user_function (defined by the user as opposed to CPython).

function

This is the class meant for functions implemented in Python.Unlike the other function types,instances offunction can be created from Python code.This is not changed, so we do not describe the details in this PEP.

The layout of the C structure is the following:

PyTypeObjectPyFunction_Type;typedefstruct{PyBaseFunctionObjectbase;PyObject*func_dict;/*__dict__:dictorNULL*/PyObject*func_code;/*__code__:code*/PyObject*func_globals;/*__globals__:dict;readonly*/PyObject*func_name;/*__name__:string*/PyObject*func_qualname;/*__qualname__:string*/PyObject*func_doc;/*__doc__:canbeanythingorNULL*/PyObject*func_defaults;/*__defaults__:tupleorNULL*/PyObject*func_kwdefaults;/*__kwdefaults__:dictorNULL*/PyObject*func_closure;/*__closure__:tupleofcellobjectsorNULL;readonly*/PyObject*func_annotations;/*__annotations__:dictorNULL*/PyCFunctionDef_ml;/*Storageforbase.m_ml*/}PyFunctionObject;

The descriptor__name__ returnsfunc_name.When setting__name__, alsobase.m_ml->ml_name is updatedwith the UTF-8 encoded name.

The_ml field reserves space to be used bybase.m_ml.

Abase_function instance must have the flagMETH_PYTHON setif and only if it is an instance offunction.

When constructing an instance offunction fromcode andglobals,an instance is created withbase.m_ml=&_ml,base.m_self=NULL.

To make subclassing easier, we also add a copy constructor:iff is an instance offunction, thentypes.FunctionType(f) copiesf.This conveniently allows using a custom function type as decorator:

>>>fromtypesimportFunctionType>>>classCustomFunction(FunctionType):...pass>>>@CustomFunction...deff(x):...returnx>>>type(f)<class '__main__.CustomFunction'>

This also removes many use cases offunctools.wraps:wrappers can be replaced by subclasses offunction.

bound_method

The classbound_method is used for all bound methods,regardless of the class of the underlying function.It adds one new attribute on top ofbase_function:__func__ points to that function.

bound_method replaces the oldmethod classwhich was used only for Python functions bound as method.

There is a complication because we want to allowconstructing a method from an arbitrary callable.This may be an already-bound method or simply not an instance ofbase_function.Therefore, in practice there are two kinds of methods:

  • For arbitrary callables, we use a single fixedPyCFunctionDefstructure with theMETH_PASS_FUNCTION flag set.
  • For methods which bind instances ofbase_function(more precisely, which have thePy_TPFLAGS_BASEFUNCTION flag set)that haveself slicing,we instead use thePyCFunctionDef from the original function.This way, we don’t lose any performance when calling bound methods.In this case, the__func__ attribute is only used to implementvarious attributes but not for calling the method.

When constructing a new method from abase_function,we check that theself object is an instance of__objclass__(if a class was specified as parent) and raise aTypeError otherwise.

The C structure is:

PyTypeObjectPyMethod_Type;typedefstruct{PyBaseFunctionObjectbase;PyObject*im_func;/*__func__:functionimplementingthemethod;readonly*/}PyMethodObject;

Calling base_function instances

We specify the implementation of__call__ for instances ofbase_function.

Checking __objclass__

First of all, a type check is done if the__parent__ of the functionis a class(recall that__objclass__ then becomes an alias of__parent__):ifm_self isNULL (this is the case for unbound methods of extension types),then the function must be called with at least one positional argumentand the first (typically calledself) must be an instance of__objclass__.If not, aTypeError is raised.

Note that bound methods havem_self!=NULL, so the__objclass__is not checked.Instead, the__objclass__ check is done when constructing the method.

Flags

For convenience, we define a new constant:METH_CALLFLAGS combines all flags fromPyCFunctionDef.ml_flagswhich specify the signature of the C function to be called.It is equal to

METH_VARARGS|METH_FASTCALL|METH_NOARGS|METH_O|METH_KEYWORDS|METH_PASS_FUNCTION

Exactly one of the first four flags above must be setand onlyMETH_VARARGS andMETH_FASTCALL may be combined withMETH_KEYWORDS.Violating these rules is undefined behaviour.

There are one new flags which affects calling functions,namelyMETH_PASS_FUNCTION andMETH_CALL_UNBOUND.Some flags are already documented in[5].We explain the others below.

Self slicing

If the function hasm_self==NULL and the flagMETH_CALL_UNBOUNDis not set, then the first positional argument (if any)is removed from*args and instead passed as first argument to the C function.Effectively, the first positional argument is treated as__self__.This is meant to support unbound methodssuch that the C function does not see the differencebetween bound and unbound method calls.This does not affect keyword arguments in any way.

This process is calledself slicing and a function is said tohave self slicing ifm_self==NULL andMETH_CALL_UNBOUND is not set.

Note that aMETH_NOARGS function which has self slicingeffectively has one argument, namelyself.Analogously, aMETH_O function with self slicing has two arguments.

METH_PASS_FUNCTION

If this flag is set, then the C function is called with anadditional first argument, namely the function itself(thebase_function instance).As special case, if the function is abound_method,then the underlying function of the method is passed(but not recursively: if abound_method wraps abound_method,then__func__ is only applied once).

For example, an ordinaryMETH_VARARGS function has signature(PyObject*self,PyObject*args).WithMETH_VARARGS|METH_PASS_FUNCTION, this becomes(PyObject*func,PyObject*self,PyObject*args).

METH_FASTCALL

This is an existing but undocumented flag.We suggest to officially support and document it.

If the flagMETH_FASTCALL is set withoutMETH_KEYWORDS,then theml_meth field is of typePyCFunctionFastwhich takes the arguments(PyObject*self,PyObject*const*args,Py_ssize_tnargs).Such a function takes only positional arguments and they are passed as plain C arrayargs of lengthnargs.

If the flagsMETH_FASTCALL|METH_KEYWORDS are set,then theml_meth field is of typePyCFunctionFastKeywordswhich takes the arguments(PyObject*self,PyObject*const*args,Py_ssize_tnargs,PyObject*kwnames).The positional arguments are passed as C arrayargs of lengthnargs.Thevalues of the keyword arguments follow in that array,starting at positionnargs.Thekeys (names) of the keyword arguments are passed as atuple inkwnames.As an example, assume that 3 positional and 2 keyword arguments are given.Thenargs is an array of length 3 + 2 = 5,nargs equals 3 andkwnames is a 2-tuple.

Automatic creation of built-in functions

Python automatically generates instances ofcfunctionfor extension types (using thePyTypeObject.tp_methods field) and modules(using thePyModuleDef.m_methods field).The arraysPyTypeObject.tp_methods andPyModuleDef.m_methodsmust be arrays ofPyMethodDef structures.

Unbound methods of extension types

The type of unbound methods changes frommethod_descriptortocfunction.The object which appears as unbound method is the same object whichappears in the class__dict__.Python automatically sets the__parent__ attribute to the defining class.

Built-in functions of a module

For the case of functions of a module,__parent__ will be set to the module.Unless the flagMETH_BINDING is given, also__self__will be set to the module (for backwards compatibility).

An important consequence is that such functions by defaultdo not become methods when used as attribute(base_function.__get__ only does that ifm_self wasNULL).One could consider this a bug, but this was done for backwards compatibility reasons:in an initial post on python-ideas[6] the consensus was to keep thismisfeature of built-in functions.

However, to allow this anyway for specific or newly implementedbuilt-in functions, theMETH_BINDING flag prevents setting__self__.

Further changes

New type flag

A newPyTypeObject flag (fortp_flags) is added:Py_TPFLAGS_BASEFUNCTION to indicate that instances of this type arefunctions which can be called and bound as method like abase_function.

This is different from flags likePy_TPFLAGS_LIST_SUBCLASSbecause it indicates more than just a subclass:it also indicates a default implementation of__call__ and__get__.In particular, such subclasses ofbase_functionmust follow the implementation from the sectionCalling base_function instances.

This flag is automatically set for extension types whichinherit thetp_call andtp_descr_get implementation frombase_function.Extension types can explicitly specify it if theyoverride__call__ or__get__ in a compatible way.The flagPy_TPFLAGS_BASEFUNCTION must never be set for a heap typebecause that would not be safe (heap types can be changed dynamically).

C API functions

We list some relevant Python/C API macros and functions.Some of these are existing (possibly changed) functions, some are new:

  • intPyBaseFunction_CheckFast(PyObject*op): return true ifopis an instance of a class with thePy_TPFLAGS_BASEFUNCTION set.This is the function that you need to use to determinewhether it is meaningful to access thebase_function internals.
  • intPyBaseFunction_Check(PyObject*op): return true ifopis an instance ofbase_function.
  • PyObject*PyBaseFunction_New(PyTypeObject*cls,PyCFunctionDef*ml,PyObject*self,PyObject*module,PyObject*parent):create a new instance ofcls (which must be a subclass ofbase_function)from the given data.
  • intPyCFunction_Check(PyObject*op): return true ifopis an instance ofcfunction.
  • intPyCFunction_NewEx(PyMethodDef*ml,PyObject*self,PyObject*module):create a new instance ofcfunction.As special case, ifself isNULL,then setself=Py_None instead (for backwards compatibility).Ifself is a module, then__parent__ is set toself.Otherwise,__parent__ isNULL.
  • For many existingPyCFunction_... andPyMethod_ functions,we define a new functionPyBaseFunction_...acting onbase_function instances.The old functions are kept as aliases of the new functions.
  • intPyFunction_Check(PyObject*op): return true ifopis an instance ofbase_function with theMETH_PYTHON flag set(this is equivalent to checking whetherop is an instance offunction).
  • intPyFunction_CheckFast(PyObject*op): equivalent toPyFunction_Check(op)&&PyBaseFunction_CheckFast(op).
  • intPyFunction_CheckExact(PyObject*op): return trueif the type ofop isfunction.
  • PyObject*PyFunction_NewPython(PyTypeObject*cls,PyObject*code,PyObject*globals,PyObject*name,PyObject*qualname):create a new instance ofcls (which must be a subclass offunction)from the given data.
  • PyObject*PyFunction_New(PyObject*code,PyObject*globals):create a new instance offunction.
  • PyObject*PyFunction_NewWithQualName(PyObject*code,PyObject*globals,PyObject*qualname):create a new instance offunction.
  • PyObject*PyFunction_Copy(PyTypeObject*cls,PyObject*func):create a new instance ofcls (which must be a subclass offunction)by copying a givenfunction.

Changes to the types module

Two types are added:types.BaseFunctionType corresponding tobase_function andtypes.DefinedFunctionType corresponding todefined_function.

Apart from that, no changes to thetypes module are made.In particular,types.FunctionType refers tofunction.However, the actual types will change:in particular,types.BuiltinFunctionType will no longer be the sameastypes.BuiltinMethodType.

Changes to the inspect module

The new functioninspect.isbasefunction checks for an instance ofbase_function.

inspect.isfunction checks for an instance ofdefined_function.

inspect.isbuiltin checks for an instance ofcfunction.

inspect.isroutine checksisbasefunction orismethoddescriptor.

NOTE: bpo-33261[3] should be fixed first.

Profiling

Currently,sys.setprofile supportsc_call,c_return andc_exceptionevents for built-in functions.These events are generated when calling or returning from a built-in function.By contrast, thecall andreturn events are generated by the function itself.So nothing needs to change for thecall andreturn events.

Since we no longer make a difference between C functions and Python functions,we need to prevent thec_* events for Python functions.This is done by not generating those events if theMETH_PYTHON flag inml_flags is set.

Non-CPython implementations

Most of this PEP is only relevant to CPython.For other implementations of Python,the two changes that are required are thebase_function base classand the fact thatfunction can be subclassed.The classescfunction anddefined_function are not required.

We requirebase_function for consistency but we put no requirements on it:it is acceptable if this is just a copy ofobject.Support for the new__parent__ (and__objclass__) attribute is not required.If there is nodefined_function class,thentypes.DefinedFunctionType should be an alias oftypes.FunctionType.

Rationale

Why not simply change existing classes?

One could try to solve the problem by keeping the existing classeswithout introducing a newbase_function class.

That might look like a simpler solution but it is not:it would require introspection support for 3 distinct classes:function,builtin_function_or_method andmethod_descriptor.For the latter two classes, “introspection support” would meanat a minimum allowing subclassing.But we don’t want to lose performance, so we want fast subclass checks.This would require two new flags intp_flags.And we want subclasses to allow__get__ for built-in functions,so we should implement theLOAD_METHOD opcode for built-in functions too.More generally, a lot of functionality would need to be duplicatedand the end result would be far more complex code.

It is also not clear how the introspection of built-in function subclasseswould interact with__text_signature__.Having two independent kinds ofinspect.signature support on the sameclass sounds like asking for problems.

And this would not fix some of the other differences between built-in functionsand Python functions that were mentioned in themotivation.

Why __text_signature__ is not a solution

Built-in functions have an attribute__text_signature__,which gives the signature of the function as plain text.The default values are evaluated byast.literal_eval.Because of this, it supports only a small number of standard Python classesand not arbitrary Python objects.

And even if__text_signature__ would allow arbitrary signatures somehow,that is only one piece of introspection:it does not help withinspect.getsourcefile for example.

defined_function versus function

In many places, a decision needs to be made whether the oldfunction classshould be replaced bydefined_function or the newfunction class.This is done by thinking of the most likely use case:

  1. types.FunctionType refers tofunction because thattype might be used to construct instances usingtypes.FunctionType(...).
  2. inspect.isfunction() refers todefined_functionbecause this is the class where introspection is supported.
  3. The C API functions must refer tofunction becausewe do not specify how the various attributes ofdefined_functionare implemented.We expect that this is not a problem since there is typically noreason for introspection to be done by C extensions.

Scope of this PEP: which classes are involved?

The main motivation of this PEP is fixing function classes,so we certainly want to unify the existing classesbuiltin_function_or_method andfunction.

Since built-in functions and methods have the same class,it seems natural to include bound methods too.And since there are no “unbound methods” for Python functions,it makes sense to get rid of unbound methods for extension types.

For now, no changes are made to the classesstaticmethod,classmethod andclassmethod_descriptor.It would certainly make sense to put these in thebase_functionclass hierarchy and unifyclassmethod andclassmethod_descriptor.However, this PEP is already big enoughand this is left as a possible future improvement.

Slot wrappers for extension types like__init__ or__eq__are quite different from normal methods.They are also typically not called directly because you would normallywritefoo[i] instead offoo.__getitem__(i).So these are left outside the scope of this PEP.

Python also has aninstancemethod class,which seems to be a relic from Python 2,where it was used for bound and unbound methods.It is not clear whether there is still a use case for it.In any case, there is no reason to deal with it in this PEP.

TODO: shouldinstancemethod be deprecated?It doesn’t seem used at all within CPython 3.7,but maybe external packages use it?

Not treating METH_STATIC and METH_CLASS

Almost nothing in this PEP refers to the flagsMETH_STATIC andMETH_CLASS.These flags are checked only by theautomatic creation of built-in functions.When astaticmethod,classmethod orclassmethod_descriptoris bound (i.e.__get__ is called),abase_function instance is created withm_self!=NULL.For aclassmethod, this is obvious sincem_selfis the class that the method is bound to.For astaticmethod, one can take an arbitrary Python object form_self.For backwards compatibility, we choosem_self=__parent__ for static methodsof extension types.

__self__ in base_function

It may look strange at first sight to add the__self__ slotinbase_function as opposed tobound_method.We took this idea from the existingbuiltin_function_or_method class.It allows us to have a single general implementation of__call__ and__get__for the various function classes discussed in this PEP.

It also makes it easy to support existing built-in functionswhich set__self__ to the module (for example,sys.exit.__self__ issys).

Two implementations of __doc__

base_function does not support function docstrings.Instead, the classescfunction andfunctioneach have their own way of dealing with docstrings(andbound_method just takes the__doc__ from the wrapped function).

Forcfunction, the docstring is stored (together with the text signature)as C string in the read-onlyml_doc field of aPyMethodDef.Forfunction, the docstring is stored as a writable Python objectand it does not actually need to be a string.It looks hard to unify these two very different ways of dealing with__doc__.For backwards compatibility, we keep the existing implementations.

Fordefined_function, we require__doc__ to be implementedbut we do not specify how. A subclass can implement__doc__ thesame way ascfunction or using a struct member or some other way.

Subclassing

We disallow subclassing ofcfunction andbound_methodto enable fast type checks forPyCFunction_Check andPyMethod_Check.

We allow subclassing of the other classes because there is no reason to disallow it.For Python modules, the only relevant class to subclass isfunction because the others cannot be instantiated anyway.

Replacing tp_call: METH_PASS_FUNCTION and METH_CALL_UNBOUND

The new flagsMETH_PASS_FUNCTION andMETH_CALL_UNBOUNDare meant to support cases where formerly a customtp_call was used.It reduces the number of special fast paths inPython/ceval.cfor calling objects:instead of treating Python functions, built-in functions and method descriptorsseparately, there would only be a single check.

The signature oftp_call is essentially the signatureofPyBaseFunctionObject.m_ml.ml_meth with flagsMETH_VARARGS|METH_KEYWORDS|METH_PASS_FUNCTION|METH_CALL_UNBOUND(the only difference is an addedself argument).Therefore, it should be easy to change existingtp_call slotsto use thebase_function implementation instead.

It also makes sense to useMETH_PASS_FUNCTION withoutMETH_CALL_UNBOUNDin cases where the C function simply needs access to additional metadatafrom the function, such as the__parent__.This is for example needed to supportPEP 573.Converting existing methods to useMETH_PASS_FUNCTION is trivial:it only requires adding an extra argument to the C function.

Backwards compatibility

While designing this PEP, great care was taken to not breakbackwards compatibility too much.Most of the potentially incompatible changesare changes to CPython implementation detailswhich are different anyway in other Python interpreters.In particular, Python code which correctly runs on PyPywill very likely continue to work with this PEP.

The standard classes and functions likestaticmethod,functools.partial oroperator.methodcallerdo not need to change at all.

Changes to types and inspect

The proposed changes totypes andinspectare meant to minimize changes in behaviour.However, it is unavoidable that some things changeand this can cause code which usestypes orinspect to break.In the Python standard library for example,changes are needed in thedoctest module because of this.

Also, tools which take various kinds of functions as input will need to dealwith the new function hierarchy and the possibility of customfunction classes.

Python functions

For Python functions, essentially nothing changes.The attributes that existed before still exist and Python functionscan be initialized, called and turned into methods as before.

The namefunction is kept for backwards compatibility.While it might make sense to change the name to something morespecific likepython_function,that would require a lot of annoying changes in documentation and testsuites.

Built-in functions of a module

Also for built-in functions, nothing changes.We keep the old behaviour that such functions do not bind as methods.This is a consequence of the fact that__self__ is set to the module.

Built-in bound and unbound methods

The types of built-in bound and unbound methods will change.However, this does not affect calling such methodsbecause the protocol inbase_function.__call__(in particular the handling of__objclass__ and self slicing)was specifically designed to be backwards compatible.All attributes which existed before (like__objclass__ and__self__)still exist.

New attributes

Some objects get new special double-underscore attributes.For example, the new attribute__parent__ appears onall built-in functions and all methods get a__func__ attribute.The fact that__self__ is now a special read-only attributefor Python functions caused trouble in[4].Generally, we expect that not much will break though.

method_descriptor and PyDescr_NewMethod

The classmethod_descriptor and the constructorPyDescr_NewMethodshould be deprecated.They are no longer used by CPython itself but are still supported.

Two-phase Implementation

TODO: this section is optional.If this PEP is accepted, it shouldbe decided whether to apply this two-phase implementation or not.

As mentioned above, thechanges to types and inspect can break someexisting code.In order to further minimize breakage, this PEP could be implementedin two phases.

Phase one: keep existing classes but add base classes

Initially, implement thebase_function classand use it as common base class but otherwise keep the existing classes(but not their implementation).

In this proposal, the class hierarchy would become:

object||base_function/|     \/|      \/|       \cfunction|defined_function|||         \||bound_method    \||                       \|method_descriptorfunction|builtin_function_or_method

The leaf classesbuiltin_function_or_method,method_descriptor,bound_method andfunction correspond to the existing classes(withmethod renamed tobound_method).

Automatically created functions created in modules become instancesofbuiltin_function_or_method.Unbound methods of extension types become instances ofmethod_descriptor.

The classmethod_descriptor is a copy ofcfunction exceptthat__get__ returns abuiltin_function_or_method instead of abound_method.

The classbuiltin_function_or_method has the same C structure as abound_method, but it inherits fromcfunction.The__func__ attribute is not mandatory:it is only defined when binding amethod_descriptor.

We keep the implementation of theinspect functions as they are.Because of this and because the existing classes are kept,backwards compatibility is ensured for code doing type checks.

Since showing an actualDeprecationWarning would affect a lotof correctly-functioning code,any deprecations would only appear in the documentation.Another reason is that it is hard to show warnings for callingisinstance(x,t)(but it could be done using__instancecheck__ hacking)and impossible fortype(x)ist.

Phase two

Phase two is what is actually described in the rest of this PEP.In terms of implementation,it would be a relatively small change compared to phase one.

Reference Implementation

Most of this PEP has been implemented for CPython athttps://github.com/jdemeyer/cpython/tree/pep575

There are four steps, corresponding to the commits on that branch.After each step, CPython is in a mostly working state.

  1. Add thebase_function class and make it a subclass forcfunction.This is by far the biggest step as the complete__call__ protocolis implemented in this step.
  2. Renamemethod tobound_method and make it a subclass ofbase_function.Change unbound methods of extension types to be instances ofcfunctionsuch that bound methods of extension types are also instances ofbound_method.
  3. Implementdefined_function andfunction.
  4. Changes to other parts of Python, such as the standard library and testsuite.

Appendix: current situation

NOTE:This section is more useful during the draft period of the PEP,so feel free to remove this once the PEP has been accepted.

For reference, we describe in detail the relevant existing classes in CPython 3.7.

Each of the classes involved is an “orphan” class(no non-trivial subclasses nor superclasses).

builtin_function_or_method: built-in functions and bound methods

These are of typePyCFunction_Typewith structurePyCFunctionObject:

typedefstruct{PyObject_HEADPyMethodDef*m_ml;/*DescriptionoftheCfunctiontocall*/PyObject*m_self;/*Passedas'self'argtotheCfunc,canbeNULL*/PyObject*m_module;/*The__module__attribute,canbeanything*/PyObject*m_weakreflist;/*Listofweakreferences*/}PyCFunctionObject;structPyMethodDef{constchar*ml_name;/*Thenameofthebuilt-infunction/method*/PyCFunctionml_meth;/*TheCfunctionthatimplementsit*/intml_flags;/*CombinationofMETH_xxxflags,whichmostlydescribetheargsexpectedbytheCfunc*/constchar*ml_doc;/*The__doc__attribute,orNULL*/};

wherePyCFunction is a C function pointer (there are various forms of this, the most basictakes two arguments forself and*args).

This class is used both for functions and bound methods:for a method, them_self slot points to the object:

>>>dict(foo=42).get<built-in method get of dict object at 0x...>>>>dict(foo=42).get.__self__{'foo': 42}

In some cases, a function is considered a “method” of the module defining it:

>>>importos>>>os.kill<built-in function kill>>>>os.kill.__self__<module 'posix' (built-in)>

method_descriptor: built-in unbound methods

These are of typePyMethodDescr_Typewith structurePyMethodDescrObject:

typedefstruct{PyDescrObjectd_common;PyMethodDef*d_method;}PyMethodDescrObject;typedefstruct{PyObject_HEADPyTypeObject*d_type;PyObject*d_name;PyObject*d_qualname;}PyDescrObject;

function: Python functions

These are of typePyFunction_Typewith structurePyFunctionObject:

typedefstruct{PyObject_HEADPyObject*func_code;/*Acodeobject,the__code__attribute*/PyObject*func_globals;/*Adictionary(othermappingswon't do) */PyObject*func_defaults;/*NULLoratuple*/PyObject*func_kwdefaults;/*NULLoradict*/PyObject*func_closure;/*NULLoratupleofcellobjects*/PyObject*func_doc;/*The__doc__attribute,canbeanything*/PyObject*func_name;/*The__name__attribute,astringobject*/PyObject*func_dict;/*The__dict__attribute,adictorNULL*/PyObject*func_weakreflist;/*Listofweakreferences*/PyObject*func_module;/*The__module__attribute,canbeanything*/PyObject*func_annotations;/*Annotations,adictorNULL*/PyObject*func_qualname;/*Thequalifiedname*//*Invariant:*func_closurecontainsthebindingsforfunc_code->co_freevars,so*PyTuple_Size(func_closure)==PyCode_GetNumFree(func_code)*(func_closuremaybeNULLifPyCode_GetNumFree(func_code)==0).*/}PyFunctionObject;

In Python 3, there is no “unbound method” class:an unbound method is just a plain function.

method: Python bound methods

These are of typePyMethod_Typewith structurePyMethodObject:

typedefstruct{PyObject_HEADPyObject*im_func;/*Thecallableobjectimplementingthemethod*/PyObject*im_self;/*Theinstanceitisboundto*/PyObject*im_weakreflist;/*Listofweakreferences*/}PyMethodObject;

References

[1] (1,2)
Cython (http://cython.org/)
[2]
Python bug 30071, Duck-typing inspect.isfunction() (https://bugs.python.org/issue30071)
[3]
Python bug 33261, inspect.isgeneratorfunction fails on hand-created methods(https://bugs.python.org/issue33261 andhttps://github.com/python/cpython/pull/6448)
[4]
Python bug 33265, contextlib.ExitStack abuses __self__(https://bugs.python.org/issue33265 andhttps://github.com/python/cpython/pull/6456)
[5]
PyMethodDef documentation (https://docs.python.org/3.7/c-api/structures.html#c.PyMethodDef)
[6]
PEP proposal: unifying function/method classes (https://mail.python.org/pipermail/python-ideas/2018-March/049398.html)

Copyright

This document has been placed in the public domain.


Source:https://github.com/python/peps/blob/main/peps/pep-0575.rst

Last modified:2025-02-01 08:55:40 GMT


[8]ページ先頭

©2009-2026 Movatter.jp