Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-85160: Reduce memory usage ofsingledispatchmethod when owner instances cannot be weakref'd#107706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged

Conversation

@AlexWaygood
Copy link
Member

@AlexWaygoodAlexWaygood commentedAug 7, 2023
edited by bedevere-bot
Loading

No news, since this is just a small followup to#107148, which is a new feature in Python 3.13

See#107148 (comment) (cc.@eendebakpt)

@AlexWaygoodAlexWaygood changed the titlegh-85160: Reduce memory usage ofsingledispatchmethod when instances cannot be weakref'dgh-85160: Reduce memory usage ofsingledispatchmethod when owner instances cannot be weakref'dAug 7, 2023
@serhiy-storchaka
Copy link
Member

Alternatively, you can get rid ofself._all_weakrefable_instances:

diff --git a/Lib/functools.py b/Lib/functools.pyindex 2a8a69b3c5..5622eb6467 100644--- a/Lib/functools.py+++ b/Lib/functools.py@@ -935,7 +935,6 @@ def __init__(self, func):         self.dispatcher = singledispatch(func)         self.func = func         self._method_cache = weakref.WeakKeyDictionary()-        self._all_weakrefable_instances = True      def register(self, cls, method=None):         """generic_method.register(cls, func) -> func@@ -945,11 +944,11 @@ def register(self, cls, method=None):         return self.dispatcher.register(cls, func=method)      def __get__(self, obj, cls=None):-        if self._all_weakrefable_instances:+        if self._method_cache:             try:                 _method = self._method_cache[obj]             except TypeError:-                self._all_weakrefable_instances = False+                self._method_cache = None             except KeyError:                 pass             else:@@ -963,7 +962,7 @@ def _method(*args, **kwargs):         _method.register = self.register         update_wrapper(_method, self.func)-        if self._all_weakrefable_instances:+        if self._method_cache is not None:             self._method_cache[obj] = _method          return _method

It reduces memory even more, and in all cases.

@AlexWaygood
Copy link
MemberAuthor

Alternatively, you can get rid ofself._all_weakrefable_instances:

Very nice. And the performance seems to be the same. The first line of__get__ needs to be this, however (rather than justif self._method_cache), or tests start to fail:

def__get__(self,obj,cls=None):ifself._method_cacheisnotNone:

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@serhiy-storchaka
Copy link
Member

Or you need to catch TypeError when save in the cache:

ifself._method_cacheisnotNone:try:self._method_cache[obj]=_methodexceptTypeError:self._method_cache=None

@AlexWaygood
Copy link
MemberAuthor

Or you need to catch TypeError when save in the cache:

try/except is expensive; this would be much slower. With this PR currently:

bench singledispatchmethod: Mean +- std dev: 875 ns +- 32 nsbench singledispatchmethod slots: Mean +- std dev: 2.01 us +- 0.06 us

With this diff applied to my PR branch:

Diff
--- a/Lib/functools.py+++ b/Lib/functools.py@@ -945,7 +945,7 @@ def register(self, cls, method=None):         return self.dispatcher.register(cls, func=method)     def __get__(self, obj, cls=None):-        if self._method_cache is not None:+        if self._method_cache:             try:                 _method = self._method_cache[obj]             except TypeError:@@ -963,8 +963,10 @@ def _method(*args, **kwargs):         _method.register = self.register         update_wrapper(_method, self.func)-        if self._method_cache is not None:+        try:             self._method_cache[obj] = _method+        except TypeError:+            pass         return _method
bench singledispatchmethod: Mean +- std dev: 1.02 us +- 0.04 usbench singledispatchmethod slots: Mean +- std dev: 3.32 us +- 0.27 us
Benchmark script
importpyperfsetup="""from functools import singledispatch, singledispatchmethodclass Test:    @singledispatchmethod    def go(self, item, arg):        pass    @go.register    def _(self, item: int, arg):        return item + argclass Slot:    __slots__ = ('a', 'b')    @singledispatchmethod    def go(self, item, arg):        pass    @go.register    def _(self, item: int, arg):        return item + argt = Test()s= Slot()"""runner=pyperf.Runner()runner.timeit(name="bench singledispatchmethod",stmt="""_ = t.go(1, 1)""",setup=setup )runner.timeit(name="bench singledispatchmethod slots",stmt="""_ = s.go(1, 1)""",setup=setup )

@AlexWaygoodAlexWaygoodenabled auto-merge (squash)August 7, 2023 12:12
@serhiy-storchaka
Copy link
Member

Simplerif self._method_cache: makes the first call faster (this is why I wrote this so at first place). Butif self._method_cache is not None: may make all following calls marginally faster, although I do not know how large the difference. So perhaps the latter variant is better in long run.

@serhiy-storchaka
Copy link
Member

try/except is expensive; this would be much slower.

I mean that you keepif self._method_cache is not None: for setting, but addcatch TypeError. It will not add an overhead.

AlexWaygood reacted with thumbs up emoji

@AlexWaygood
Copy link
MemberAuthor

It will not add an overhead.

I still measure about a 14% overhead from this approach compared to my current PR. (Tested on a PGO-optimised non-debug build on Windows, with a quiet machine.) I'm inclined to stick with how it is currently -- it also feels like slightly cleaner code if we have only onetry/except in__get__

@serhiy-storchaka
Copy link
Member

I still measure about a 14% overhead from this approach compared to my current PR.

With the test forNone preceding thetry block? I am surprised,try/except should not add an overhead if no exception was raised.

Anyway, your current code is better. I just explained why I intentionally did not useis None in the initial proposition.

AlexWaygood reacted with thumbs up emoji

@AlexWaygood
Copy link
MemberAuthor

I still measure about a 14% overhead from this approach compared to my current PR.

With the test forNone preceding thetry block? I am surprised,try/except should not add an overhead if no exception was raised.

Yep. I'm also surprised there was that difference.

@AlexWaygoodAlexWaygood merged commit2ac103c intopython:mainAug 7, 2023
@AlexWaygoodAlexWaygood deleted the singledispatch-memory-usage branchAugust 7, 2023 12:46
@AlexWaygood
Copy link
MemberAuthor

Thanks for the review, Serhiy!

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@serhiy-storchakaserhiy-storchakaserhiy-storchaka approved these changes

@corona10corona10Awaiting requested review from corona10

@rhettingerrhettingerAwaiting requested review from rhettingerrhettinger is a code owner

Assignees

No one assigned

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AlexWaygood@serhiy-storchaka@bedevere-bot

[8]ページ先頭

©2009-2025 Movatter.jp