You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
As I already had implementation I though PR might be helpful for others to see and evaluate.
From all the different extensions offunctools.partial I think this one is the best. It is relatively simple and exposes all missing functionality. Otherpartial extensions that I have seen lack functionality and would not provide complete argument ordering capabilities and/or are too complicated in relation to what they offer.
Implementation can be summarised as follows: a) Trailing placeholders are not allowed. (Makes things simpler) b) Throws exception if not all placeholders are filled on call c) retains optimization benefits of application on otherpartial instances.
Performance penalty compared to currentfunctools.partial is minimal for extension class. + 20-30 ns for initialisation and <4 ns when called with or without placeholders.
To put it simply, new functionality extendsfunctools.partial so that it has flexibility oflambda /def approach (in terms of argument ordering), but call overhead is 2x smaller.
The way I see it is that this could only be justified if this extension provided completeness and no new functionality is going to be needed anywhere near in the future. I have thought about it and tried various alternatives and I think there is a good chance that this is the case. Personally, I don't think I would ever need anything more frompartial class.
Current implementation functions reliably.
Benchmark
There is nothing new here in terms of performance. The performance after this PR will be (almost) the same as the performance ofpartial until now.Placeholders only provide flexibility for taking advantage of performance benefits where it is important.
So far I have identified 2 such cases:
More flexible predicate construction for functions inoperator module. This allows for new strategies in making performantiterator recipes.
Partializing input target function. Examples of this are optimizers and similar. I.e. cases where the function will be called over and over within the routine with number of arguments. But the input target function needs partial substitution for positionals and keywords.
Good example of this isscipy.optimize.minimize.
Its signature is:scipy.optimize.minimize(fun, x0, args=(), ...)
Note, it does not havekwds. Why? I don't know. But good reason for it could be:
fun=lambdax:f(x,**kwds)
will need to expand**kwds on every call (even if it is empty), whilepartial will make the most optimal call. (see benchmarks below). So theminimize function can leave outkwds given there is a good way to source callable with already substituted keywords.
This extension allows pre-substituting both positionals and keywords. This allows optimizer signature to leave out bothkwds andargs resulting in simpler interfacescipy.optimize.minimize(fun, x0, ...) and gaining slightly better performance - function calls are at the center of such problems after all.
Usage ofPlaceholders results in very poor performance. However, this has no material implication aslambda is more performant thanpartial in all cases and is an optimal choice.
There is small performance decrease for initialization without placeholders.
Initializing it with placeholders is slower for the same number of arguments (excluding placeholders).
But it is not much slower if placeholders are counted as arguments.
To sum up
This extension:
allows extracting current performance benefits ofpartial to few more important (at least from my POV) cases.
seems to allow for certain simplifications to happen by bringing it more in line withlambda/def behaviour. Thus, allowingpartial to be used forpartialmethod application which allows for some simplifications in handling these in other parts of the library - i.e.inspect.
The reason will be displayed to describe this comment to others.Learn more.
I personally would advocate forPLACEHOLDER instead ofPlaceholder to stress that 1) this is not a global constant, just a module constant, 2) this is not a class, 3) this is named similarly todataclasses.KW_ONLY. (see#119827 (comment))
I just looked at the Placeholder singleton logic and think it is fine because it mirrors the None singleton. It is a little complicated but not in a way that will impair maintenance.
The deallocator which marks the Placeholder as an immortal object is causing memory leaks:#124586. I propose to use a regular singleton by just storing a strong reference in the module state:#124601. My PR fix the leak. I don't think that immortal object is needed here.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
picnixz left a comment•
Uh oh!
There was an error while loading.Please reload this page.
As I already had implementation I though PR might be helpful for others to see and evaluate.
From all the different extensions of
functools.partialI think this one is the best. It is relatively simple and exposes all missing functionality. Otherpartialextensions that I have seen lack functionality and would not provide complete argument ordering capabilities and/or are too complicated in relation to what they offer.Implementation can be summarised as follows:
a) Trailing placeholders are not allowed. (Makes things simpler)
b) Throws exception if not all placeholders are filled on call
c) retains optimization benefits of application on other
partialinstances.Performance penalty compared to current
functools.partialis minimal for extension class. + 20-30 ns for initialisation and <4 ns when called with or without placeholders.To put it simply, new functionality extends
functools.partialso that it has flexibility oflambda/defapproach (in terms of argument ordering), but call overhead is 2x smaller.The way I see it is that this could only be justified if this extension provided completeness and no new functionality is going to be needed anywhere near in the future. I have thought about it and tried various alternatives and I think there is a good chance that this is the case. Personally, I don't think I would ever need anything more from
partialclass.Current implementation functions reliably.
Benchmark
There is nothing new here in terms of performance. The performance after this PR will be (almost) the same as the performance of
partialuntil now.Placeholdersonly provide flexibility for taking advantage of performance benefits where it is important.So far I have identified 2 such cases:
operatormodule. This allows for new strategies in making performantiteratorrecipes.Partializinginput target function. Examples of this are optimizers and similar. I.e. cases where the function will be called over and over within the routine with number of arguments. But the input target function needs partial substitution for positionals and keywords.Good example of this is
scipy.optimize.minimize.Its signature is:
scipy.optimize.minimize(fun, x0, args=(), ...)Note, it does not have
kwds. Why? I don't know. But good reason for it could be:will need to expand
**kwdson every call (even if it is empty), whilepartialwill make the most optimal call. (see benchmarks below). So theminimizefunction can leave outkwdsgiven there is a good way to source callable with already substituted keywords.This extension allows pre-substituting both positionals and keywords. This allows optimizer signature to leave out both
kwdsandargsresulting in simpler interfacescipy.optimize.minimize(fun, x0, ...)and gaining slightly better performance - function calls are at the center of such problems after all.Benchmark Results for
__call__Code for Cases
CPython Results
PyPy Results
Setup:
lambdaotherpartial.CPython:
__call__. Run times are very close of current and new version with Placeholders.pos2kw2_kwe(emptykwds) is much faster ofpartialcall.pos2kw2_kw(non-emptykwds) is currently slower, howevergh-119109: functool.partial vectorcall supports pto->kw & fallback to tp_call removed #120783 will likely to improve its speed so that it outperforms lambda.PyPy:
Placeholdersresults in very poor performance. However, this has no material implication aslambdais more performant thanpartialin all cases and is an optimal choice.Benchmark Results for
__new__To sum up
This extension:
partialto few more important (at least from my POV) cases.lambda/defbehaviour. Thus, allowingpartialto be used forpartialmethodapplication which allows for some simplifications in handling these in other parts of the library - i.e.inspect.📚 Documentation preview 📚:https://cpython-previews--119827.org.readthedocs.build/