Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
Description
Bug
functools.partial is implemented infunctools.py and in_functoolsmodule.c. The former is almost never used, so libraries come to depend on the quirks and corner cases of the C implementation. This is aproblem for PyPy, where the Python implementation is the only one as of the most recent PyPy version. Here's one such difference, which was uncovered by thelxml library. The following code leads to aRecursionError:
importsyssys.modules['_functools']=None# force use of pure python version, if this is commented out it worksfromfunctoolsimportpartialclassBuilder:def__call__(self,tag,*children,**attrib):return (tag,children,attrib)def__getattr__(self,tag):returnpartial(self,tag)B=Builder()m=B.m
this is the traceback:
Traceback (most recent call last): File "/home/cfbolz/projects/cpython/bug.py", line 14, in <module> m = B.m ^^^ File "/home/cfbolz/projects/cpython/bug.py", line 11, in __getattr__ return partial(self, tag) ^^^^^^^^^^^^^^^^^^ File "/home/cfbolz/projects/cpython/Lib/functools.py", line 287, in __new__ if hasattr(func, "func"): ^^^^^^^^^^^^^^^^^^^^^ File "/home/cfbolz/projects/cpython/bug.py", line 11, in __getattr__ return partial(self, tag) ^^^^^^^^^^^^^^^^^^ File "/home/cfbolz/projects/cpython/Lib/functools.py", line 287, in __new__ if hasattr(func, "func"): ^^^^^^^^^^^^^^^^^^^^^... and repeatedThe problem is the following performance shortcut inpartial.__new__:
classpartial: ...def__new__(cls,func,/,*args,**keywords):ifnotcallable(func):raiseTypeError("the first argument must be callable")ifhasattr(func,"func"):# <------------------- problemargs=func.args+argskeywords= {**func.keywords,**keywords}func=func.func
Basically in this casefunc is an object where callinghasattr(func, "func") is not safe. The equivalent C code does this check:
if (Py_TYPE(func)->tp_call== (ternaryfunc)partial_call) {// The type of "func" might not be exactly the same type object// as "type", but if it is called using partial_call, it must have the// same memory layout (fn, args and kw members).// We can use its underlying function directly and merge the arguments.partialobject*part= (partialobject*)func;
In particular, it does not simply callhasattr onfunc.
Real World Version
This is not an artificial problem, we discovered this via theclasslxml.builder.ElementMaker. It has a__call__ method implemented. It also has__getattr__ that looks like this:
def__getattr__(self,tag):returnpartial(self,tag)
Which yields the aboveRecursionError on PyPy.
Solution ideas
One approach would be to file a bug withlxml, but it is likely that more libraries depend on this behaviour. So I would suggest to change the__new__ Python code to add anisinstance check, to bring its behaviour closer to that of the C code:
def__new__(cls,func,/,*args,**keywords):ifnotcallable(func):raiseTypeError("the first argument must be callable")ifisinstance(func,partial)andhasattr(func,"func"):args=func.args+args ...
I'll open a PR with this approach soon. /cc@mgorny