Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__#131914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
dolfinus wants to merge19 commits intopython:main
base:main
Choose a base branch
Loading
fromdolfinus:improvement/ABCMeta_subclasscheck

Conversation

dolfinus
Copy link

@dolfinusdolfinus commentedMar 30, 2025
edited
Loading

test_performance_abc.py
fromabcimportABCMeta#from _py_abc import ABCMetaimporttimeimportpsutilimportosclassRoot(metaclass=ABCMeta):passclassClass1(Root):passclassClass2(Root):passclassClass3(Root):passclassClass4(Root):subclasses= []@classmethoddef__subclasses__(cls):returncls.subclassesdef_create_subclass1():classNestedClass1(Class1):passreturnNestedClass1def_create_subclass2():classNestedClass2(Class2):passreturnNestedClass2def_create_subclass3():classNestedClass3:passClass3.register(NestedClass3)returnNestedClass3def_create_subclass4():classNestedClass4:passClass4.subclasses.append(NestedClass4)returnNestedClass4deftest_performance():iters=2000print("Creating new classes to check")objects_subclass1= []foriinrange(iters):subclass1=_create_subclass1()objects_subclass1.append(subclass1())objects_subclass2= []foriinrange(iters):subclass2=_create_subclass2()objects_subclass2.append(subclass2())objects_subclass3= []foriinrange(iters):subclass3=_create_subclass3()objects_subclass3.append(subclass3())objects_subclass4= []foriinrange(iters):subclass4=_create_subclass4()objects_subclass4.append(subclass4())# create one more subclass for sibling checkanother_subclass1=_create_subclass1()another_subclass2=_create_subclass2()another_subclass3=_create_subclass3()another_subclass4=_create_subclass4()print("Created subclasses =",len(objects_subclass1)+len(objects_subclass2)+len(objects_subclass3)+len(objects_subclass4))print("Consumed memory, Mb:",psutil.Process(os.getpid()).memory_info().rss/1024**2)print("Started compairing objects")# New subclass against parent, sibling, grantparent and cousinnew_subclass1_vs_class1_ns=0new_subclass1_vs_another_subclass1_ns=0new_subclass1_vs_root_ns=0new_subclass1_vs_class2_ns=0forobj1inobjects_subclass1:start=time.perf_counter_ns()assertisinstance(obj1,Class1)new_subclass1_vs_class1_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj1,another_subclass1)new_subclass1_vs_another_subclass1_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertisinstance(obj1,Root)new_subclass1_vs_root_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj1,Class2)new_subclass1_vs_class2_ns+=time.perf_counter_ns()-start# Same subclass against parent, sibling, grantparent and cousincached_subclass1_vs_root_ns=0cached_subclass1_vs_another_subclass1_ns=0cached_subclass1_vs_class1_ns=0cached_subclass1_vs_class2_ns=0for_inrange(iters):start=time.perf_counter_ns()assertisinstance(obj1,Class1)cached_subclass1_vs_class1_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj1,another_subclass1)cached_subclass1_vs_another_subclass1_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertisinstance(obj1,Root)cached_subclass1_vs_root_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj1,Class2)cached_subclass1_vs_class2_ns+=time.perf_counter_ns()-start# new class (via .register) against parent, sibling, grantparent and cousinnew_subclass3_vs_class3_ns=0new_subclass3_vs_another_subclass3_ns=0new_subclass3_vs_root_ns=0new_subclass3_vs_class1_ns=0forobj3inobjects_subclass3:start=time.perf_counter_ns()assertisinstance(obj3,Class3)new_subclass3_vs_class3_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj3,another_subclass3)new_subclass3_vs_another_subclass3_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertisinstance(obj3,Root)new_subclass3_vs_root_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj3,Class1)new_subclass3_vs_class1_ns+=time.perf_counter_ns()-start# same class (via .register) against parent, sibling, grantparent and cousincached_subclass3_vs_class3_ns=0cached_subclass3_vs_another_subclass3_ns=0cached_subclass3_vs_root_ns=0cached_subclass3_vs_class1_ns=0for_inrange(iters):start=time.perf_counter_ns()assertisinstance(obj3,Class3)cached_subclass3_vs_class3_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj3,another_subclass3)cached_subclass3_vs_another_subclass3_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertisinstance(obj3,Root)cached_subclass3_vs_root_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj3,Class1)cached_subclass3_vs_class1_ns+=time.perf_counter_ns()-start# new class (via __subclasses__) against parent, sibling, grantparent and cousinnew_subclass4_vs_class4_ns=0new_subclass4_vs_another_subclass4_ns=0new_subclass4_vs_root_ns=0new_subclass4_vs_class1_ns=0forobj4inobjects_subclass4:start=time.perf_counter_ns()assertisinstance(obj4,Class4)new_subclass4_vs_class4_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj4,another_subclass4)new_subclass4_vs_another_subclass4_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertisinstance(obj4,Root)new_subclass4_vs_root_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj4,Class1)new_subclass4_vs_class1_ns+=time.perf_counter_ns()-start# same class (via __subclasses__) against parent, sibling, grantparent and cousincached_subclass4_vs_class4_ns=0cached_subclass4_vs_another_subclass4_ns=0cached_subclass4_vs_root_ns=0cached_subclass4_vs_class1_ns=0for_inrange(iters):start=time.perf_counter_ns()assertisinstance(obj4,Class4)cached_subclass4_vs_class4_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj4,another_subclass4)cached_subclass4_vs_another_subclass4_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertisinstance(obj4,Root)cached_subclass4_vs_root_ns+=time.perf_counter_ns()-startstart=time.perf_counter_ns()assertnotisinstance(obj4,Class1)cached_subclass4_vs_class1_ns+=time.perf_counter_ns()-startprint("Completed compairing classes.")print("Consumed memory, Mb:",psutil.Process(os.getpid()).memory_info().rss/1024**2)total_ns=sum([cached_subclass1_vs_root_ns,cached_subclass1_vs_another_subclass1_ns,cached_subclass1_vs_class1_ns,cached_subclass1_vs_class2_ns,cached_subclass3_vs_class3_ns,cached_subclass3_vs_another_subclass3_ns,cached_subclass3_vs_root_ns,cached_subclass3_vs_class1_ns,cached_subclass4_vs_class4_ns,cached_subclass4_vs_another_subclass4_ns,cached_subclass4_vs_root_ns,cached_subclass4_vs_class1_ns,new_subclass1_vs_class1_ns,new_subclass1_vs_another_subclass1_ns,new_subclass1_vs_root_ns,new_subclass1_vs_class2_ns,new_subclass3_vs_class3_ns,new_subclass3_vs_another_subclass3_ns,new_subclass3_vs_root_ns,new_subclass3_vs_class1_ns,new_subclass4_vs_class4_ns,new_subclass4_vs_another_subclass4_ns,new_subclass4_vs_root_ns,new_subclass4_vs_class1_ns,    ])print("Total, sec:",total_ns/10**9)print("isinstance(cached class, parent), us:",cached_subclass1_vs_root_ns/iters/1000)print("isinstance(cached class, sibling), us:",cached_subclass1_vs_another_subclass1_ns/iters/1000)print("isinstance(cached class, grandparent), us:",cached_subclass1_vs_class1_ns/iters/1000)print("isinstance(cached class, cousin), us:",cached_subclass1_vs_class2_ns/iters/1000)print("isinstance(cached class, parent via .register()), us:",cached_subclass3_vs_class3_ns/iters/1000)print("isinstance(cached class, sibling via .register()), us:",cached_subclass3_vs_another_subclass3_ns/iters/1000)print("isinstance(cached class, grandparent via .register()), us:",cached_subclass3_vs_root_ns/iters/1000)print("isinstance(cached class, cousin via .register()), us:",cached_subclass3_vs_class1_ns/iters/1000)print("isinstance(cached class, parent via __subclasses__), us:",cached_subclass4_vs_class4_ns/iters/1000)print("isinstance(cached class, sibling via __subclasses__), us:",cached_subclass4_vs_another_subclass4_ns/iters/1000)print("isinstance(cached class, grandparent via __subclasses__), us:",cached_subclass4_vs_root_ns/iters/1000)print("isinstance(cached class, cousin via __subclasses__), us:",cached_subclass4_vs_class1_ns/iters/1000)print("isinstance(new class, parent), us:",new_subclass1_vs_class1_ns/len(objects_subclass1)/1000)print("isinstance(new class, sibling), us:",new_subclass1_vs_another_subclass1_ns/len(objects_subclass1)/1000)print("isinstance(new class, grandparent), us:",new_subclass1_vs_root_ns/len(objects_subclass1)/1000)print("isinstance(new class, cousin), us:",new_subclass1_vs_class2_ns/len(objects_subclass1)/1000)print("isinstance(new class, parent via .register()), us:",new_subclass3_vs_class3_ns/len(objects_subclass3)/1000)print("isinstance(new class, sibling via .register()), us:",new_subclass3_vs_another_subclass3_ns/len(objects_subclass3)/1000)print("isinstance(new class, grandparent via .register()), us:",new_subclass3_vs_root_ns/len(objects_subclass3)/1000)print("isinstance(new class, cousin via .register()), us:",new_subclass3_vs_class1_ns/len(objects_subclass3)/1000)print("isinstance(new class, parent via __subclasses__), us:",new_subclass4_vs_class4_ns/len(objects_subclass4)/1000)print("isinstance(new class, sibling via __subclasses__), us:",new_subclass4_vs_another_subclass4_ns/len(objects_subclass4)/1000)print("isinstance(new class, grandparent via __subclasses__), us:",new_subclass4_vs_root_ns/len(objects_subclass4)/1000)print("isinstance(new class, cousin via __subclasses__), us:",new_subclass4_vs_class1_ns/len(objects_subclass4)/1000)if__name__=="__main__":test_performance()

For 8k nested subclasses:

ImplMemory before, MBMemory after, MB
_abc5304.32047.453
_py_abc3469.64853.980
ImplTotal time before, secondsTotal time after, seconds
_abc512.12066.352
_py_abc323.217260.137
CheckImplus per check, beforeus per check, afterImplus per check, beforeus per check, after
isinstance(cached class, parent)_abc2.0611.515_py_abc2.4112.185
isinstance(cached class, sibling)_abc2.0271.470_py_abc3.9003.614
isinstance(cached class, grandparent)_abc2.0731.514_py_abc2.3952.173
isinstance(cached class, cousin)_abc2.0581.482_py_abc3.9033.608
isinstance(cached class, parent via .register())_abc1.9012.071_py_abc2.5552.243
isinstance(cached class, sibling via .register())_abc0.4170.467_py_abc0.5220.455
isinstance(cached class, grandparent via .register())_abc1.1331.213_py_abc2.5002.233
isinstance(cached class, cousin via .register())_abc1.1521.251_py_abc4.0873.805
isinstance(cached class, parent via __subclasses__)_abc1.6191.317_py_abc2.0322.349
isinstance(cached class, sibling via __subclasses__)_abc0.6070.430_py_abc0.4220.484
isinstance(cached class, grandparent via __subclasses__)_abc1.6421.188_py_abc2.0392.313
isinstance(cached class, cousin via __subclasses__)_abc1.6781.217_py_abc3.3513.956
isinstance(new class, parent)_abc7.1257.546_py_abc12.89214.153
isinstance(new class, sibling)_abc6.3376.069_py_abc13.83915.672
isinstance(new class, grandparent)_abc3.6064.028_py_abc9.84110.264
isinstance(new class, cousin)_abc15_164.2914_489.091_py_abc20_130.5817_344.474
isinstance(new class, parent via .register())_abc4.0057.557_py_abc707.748739.004
isinstance(new class, sibling via .register())_abc0.9671.144_py_abc1.4751.624
isinstance(new class, grandparent via .register())_abc80_945.5679_185.747_py_abc59_578.9037_271.663
isinstance(new class, cousin via .register())_abc4.0264_630.594_py_abc7.66118_586.478
isinstance(new class, parent via __subclasses__)_abc83.738350.901_py_abc223.813496.861
isinstance(new class, sibling via __subclasses__)_abc1.3231.464_py_abc1.1361.348
isinstance(new class, grandparent via __subclasses__)_abc159_816.3929_749.391_py_abc80_883.0737_469.294
isinstance(new class, cousin via __subclasses__)_abc4.3234_727.533_py_abc7.66318_088.251

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

if (scls == NULL) {
goto end;
}
int r = PyObject_IsSubclass(subclass, scls);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think we have a UAF here.PyObject_IsSubclass can call__subclasscheck__ which can itseslf call arbitrary code so you might mutatesubclasses. The issue already exists with the existing code but can you confirm that we can indeed produce a UAF? (if you don't know how to do it, I'll try to investigate this separately tomorrow)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

can you confirm that we can indeed produce a UAF?

Sorry, my C knowledge is very minimal, I don't know anything about this yet

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

3 similar comments
@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@python-cla-bot
Copy link

All commit authors signed the Contributor License Agreement.

CLA signed

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
Signed-off-by: Martynov Maxim <martinov_m_s_@mail.ru>
@dolfinusdolfinusforce-pushed theimprovement/ABCMeta_subclasscheck branch fromabf4bfe tob7603e0CompareApril 21, 2025 11:03
@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@dolfinusdolfinus changed the titlegh-92810: Avoid O(n^2) complexity in ABCMeta.__subclasscheck__gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__Apr 23, 2025
@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@bedevere-app
Copy link

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

@dolfinusdolfinus marked this pull request as ready for reviewJune 13, 2025 14:15
@dolfinusdolfinus requested a review frompicnixzJune 13, 2025 14:16
@dolfinus
Copy link
Author

dolfinus commentedJun 13, 2025
edited
Loading

I've added a simple recursion check tocls.__subclasses__() clause, both to _py_abc and _abc. This avoids adding the same class to all ABC children class caches, reducing both memory usage and time for very large class trees. Microbenchmark & results are in the description.

Viicos reacted with heart emoji

@dolfinusdolfinus requested a review frompicnixzJune 22, 2025 16:42
@Viicos
Copy link
Contributor

Do you happen to know this will play with#119719?

@dolfinus
Copy link
Author

Do you happen to know this will play with#119719?

No, and I don't like solution of#119719 at all

@dolfinus
Copy link
Author

dolfinus commentedJul 18, 2025
edited
Loading

@picnixz Could you please take a look on this PR?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@ZeroIntensityZeroIntensityZeroIntensity left review comments

@picnixzpicnixzAwaiting requested review from picnixz

Assignees
No one assigned
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@dolfinus@Viicos@picnixz@ZeroIntensity

[8]ページ先頭

©2009-2025 Movatter.jp