Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
gh-104223: Fix issues with inheriting from buffer classes#104227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
bedevere-bot commentedMay 6, 2023
🤖 New build scheduled with the buildbot fleet by@JelleZijlstra for commit88ce24d 🤖 If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again. |
It looks like when you export a python class that subclasses an internal class that implements >>>classB(bytearray):...def__release_buffer__(self,buffer):...B.leak=self.clear()orbytearray()...B.backing=buffer...>>>b=B(bytearray.__basicsize__)>>>m=memoryview(b)>>>m.release()>>>B.leakbytearray(b'')>>>B.backing<memoryat0x109d1f700>>>>B.backing.cast('P').tolist()[1,4454897440,0,0,0,0,0]>>>B.backing.cast('P')[2]=-1>>>len(B.leak)Traceback (mostrecentcalllast):File"<stdin>",line1,in<module>SystemError:<built-infunctionlen>returnedNULLwithoutsettinganexception |
kumaraditya303 left a comment• edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Some high level comments, mostly looks good.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
| ba.clear() | ||
| c.buffer.release() | ||
| ba.clear() | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Can you add a test in which the class has multiple inheritance? Also tests for when there are two or classes in mro which support buffer protocol?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Adding some tests. It's hard to come up with a case where multiple bases support the C buffer protocol, because that will almost inevitably lead to:
Traceback (most recent call last): File "/Users/jelle/py/cpython/Lib/test/test_buffer.py", line 4707, in test_two_buffer_bases class A(bytearray, bytes):TypeError: multiple bases have instance lay-out conflictThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Hmm, maybe try one var length and one pure python type which implements buffer protocol.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Yes, just pushed. Thanks for the review!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Adding some tests. It's hard to come up with a case where multiple bases support the C buffer protocol, because that will almost inevitably lead to:
I see, you would have to write a custom type in C whose layout doesn't conflicts for that. Probably an object which has just object header and supports buffer protocol.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
TBH, I agree this is an edge case but I try to be extra careful when touchingtypeobject, hope you don't mind the extra work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Better to catch this now then right before 3.12 final!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Confirmed the bug I outlined above, I'll have to step out for a few hours and fix it after that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thinking about it more, that crash isn't actually due to PEP 688: it reproduces without any Python__buffer__ method. So I don't think there's any more interesting cases to cover for PEP 688.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Filed#104297 for that case.
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
bedevere-bot commentedMay 7, 2023
🤖 New build scheduled with the buildbot fleet by@JelleZijlstra for commit5536853 🤖 If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again. |
…on#104227)Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* main: (47 commits)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)pythongh-103193: Improve `getattr_static` test coverage (python#104286) Trim trailing whitespace and test on CI (python#104275)pythongh-102500: Remove mention of bytes shorthand (python#104281)pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)pythongh-99108: Replace SHA3 implementation HACL* version (python#103597)pythongh-104273: Remove redundant len() calls in argparse function (python#104274)pythongh-64660: Don't hardcode Argument Clinic return converter result variable name (python#104200)pythongh-104265 Disallow instantiation of `_csv.Reader` and `_csv.Writer` (python#104266)pythonGH-102613: Improve performance of `pathlib.Path.rglob()` (pythonGH-104244)pythongh-103650: Fix perf maps address format (python#103651)pythonGH-89812: Churn `pathlib.Path` methods (pythonGH-104243) ...
* main: (29 commits)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293)pythonGH-104145: Use fully-qualified cross reference types for the bisect module (python#104172)pythongh-103193: Improve `getattr_static` test coverage (python#104286) Trim trailing whitespace and test on CI (python#104275)pythongh-102500: Remove mention of bytes shorthand (python#104281)pythongh-97696: Improve and fix documentation for asyncio eager tasks (python#104256)pythongh-99108: Replace SHA3 implementation HACL* version (python#103597) ...
* main: (156 commits)pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)pythongh-104240: return code unit metadata from codegen (python#104300)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293) ...
* main: (35 commits)pythongh-97696 Add documentation for get_coro() behavior with eager tasks (python#104304)pythongh-97933: (PEP 709) inline list/dict/set comprehensions (python#101441)pythongh-99889: Fix directory traversal security flaw in uu.decode() (python#104096)pythongh-104184: fix building --with-pydebug --enable-pystats (python#104217)pythongh-104139: Add itms-services to uses_netloc urllib.parse. (python#104312)pythongh-104240: return code unit metadata from codegen (python#104300)pythongh-104276: Make `_struct.unpack_iterator` type use type flag instead of custom constructor (python#104277)pythongh-97696: Move around and update the whatsnew entry for asyncio eager task factory (python#104298)pythongh-103193: Fix refleaks in `test_inspect` and `test_typing` (python#104320) require-pr-label.yml: Add missing "permissions:" (python#104309)pythongh-90656: Add platform triplets for 64-bit LoongArch (LA64) (python#30939)pythongh-104180: Read SOCKS proxies from macOS System Configuration (python#104181)pythongh-97696 Remove unnecessary check for eager_start kwarg (python#104188)pythonGH-104308: socket.getnameinfo should release the GIL (python#104307)pythongh-104310: Add importlib.util.allowing_all_extensions() (pythongh-104311)pythongh-99113: A Per-Interpreter GIL! (pythongh-104210)pythonGH-104284: Fix documentation gettext build (python#104296)pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (python#101251)pythongh-104223: Fix issues with inheriting from buffer classes (python#104227)pythongh-99108: fix typo in Modules/Setup (python#104293) ...
Uh oh!
There was an error while loading.Please reload this page.
This fixes various issues that crop up when inheriting from a class that implements the C buffer protocol.
wrap_buffer, we need to make sure to call the exact slot that we're wrapping, or we might recursively call ourselves inside a call tosuper().bufferwrapper_releasebuf, we should only call thebf_releasebufslot if it's a Python function. If it's a C slot inherited from a base class, it will just be called when we release the memoryview.releasebuffer_call_pythonallowed a use-after-free. To guard against that, create a new restricted-mode memoryview that does not allow the user to hold on to the buffer.