Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-125038: Iterator checks are added for some FOR_ITER bytecodes, crash fixed#125051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation

@efimov-mikhail
Copy link
Member

@efimov-mikhailefimov-mikhail commentedOct 7, 2024
edited by bedevere-appbot
Loading

SIGSEGV on generators in case of gi_frame.f_locals is fixed.

This applies to _FOR_ITER bytecode implementation.
Similar checks are added to _FOR_ITER_TIER_TWO and INSTRUMENTED_FOR_ITER bytecode implementations.

PyObject*iter_o=PyStackRef_AsPyObjectBorrow(iter);
PyObject*next_o= (*Py_TYPE(iter_o)->tp_iternext)(iter_o);
PyObject*next_o=NULL;
if (PyIter_Check(iter_o)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

CallingPyIter_Check feels a little wasteful. Instead, I'd assign(*Py_TYPE(iter_o)->tp_iternext) to a variable and check if it's NULL. If it is NULL, we should raise a TypeError. Your code instead makes it immediately end the loop, which doesn't feel like the right behavior.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hmm, I've tried to add a TypeError
Such piece of code is provided:

            PyTypeObject *type = Py_TYPE(iter_o);            iternextfunc iternext = type->tp_iternext;            if (iternext == NULL) {                _PyErr_Format(tstate, PyExc_TypeError,                              "'for' requires an object with "                              "__iter__ method, got %.100s",                              type->tp_name);                DECREF_INPUTS();                ERROR_IF(true, error);            }            PyObject *next_o = (*iternext)(iter_o);

And there is a temporarily result of my test case:

-> % ./python -m unittest -v test.test_generators.GeneratorTest.test_issue125038test_issue125038 (test.test_generators.GeneratorTest.test_issue125038) ... ERROR======================================================================ERROR: test_issue125038 (test.test_generators.GeneratorTest.test_issue125038)----------------------------------------------------------------------Traceback (most recent call last):  File "/home/mikhail.efimov/projects/cpython/Lib/test/test_generators.py", line 274, in test_issue125038    l = list(g)  File "/home/mikhail.efimov/projects/cpython/Lib/test/test_generators.py", line 272, in <genexpr>    g = (x for x in range(10))                    ~~~~~^^^^TypeError: 'for' requires an object with __iter__ method, got range----------------------------------------------------------------------Ran 1 test in 0.001s

It seems like such a message doesn't provide any clarity.
Do you have any better suggestions about error message?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That seems like a pretty good error message, much better than silently stopping the loop.

efimov-mikhail reacted with thumbs up emoji
@efimov-mikhailefimov-mikhail changed the titlegh-125038: PyIter_Checks are added for some FOR_ITER bytecodesgh-125038: Iterator checks are added for some FOR_ITER bytecodes, crash fixedOct 7, 2024
exceptTypeError:
return"TypeError"

# This should not raise
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think you wantwith self.assertRaisesRegex.

efimov-mikhail reacted with thumbs up emoji
@gaogaotiantian
Copy link
Member

gaogaotiantian commentedOct 7, 2024
edited
Loading

We should confirm with@markshannon whether we should fix the interpreter orf_locals. Maybe it's a valid assumption? PEP 667 does not implement everything dict-like forFrameLocalsProxy, but maybe this is necessary? We are adding a small overhead to a very commonly used instruction if I understood it correctly.

efimov-mikhail reacted with thumbs up emoji

@JelleZijlstra
Copy link
Member

I looked at PEP 668 and it seems to assume that mutatingf_locals is supported, which makes me think that we need to add this check. I wouldn't personally mind blocking thef_locals mutation instead if you can find a principled way to do it.

We are adding a small overhead to a very commonly used instruction if I understood it correctly.

Yes, that's right; we're adding a NULL check and a branch. However, FOR_ITER has a couple of specialized versions, so the overhead should be minimal when iterating over the most common types. Still, it may be worth benchmarking how bad the overhead is.

efimov-mikhail reacted with thumbs up emoji

Copy link
Member

@JelleZijlstraJelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This looks correct to me, but will leave to Mark to see if there's a better solution.

efimov-mikhail reacted with thumbs up emoji
@JelleZijlstraJelleZijlstra added the needs backport to 3.13bugs and security fixes labelOct 8, 2024
…e-125038.ffSLCz.rstNews improvementCo-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
@efimov-mikhail
Copy link
MemberAuthor

Thanks for your advice, it definitely looks better now.

But I have a question.
Is it possible to make the piece of code in the original issue correct?
Why we can't assign different value to the iterator object directly through the frame's f_locals?
I can't see something like this in PEP 667.

@efimov-mikhail
Copy link
MemberAuthor

efimov-mikhail commentedOct 8, 2024
edited
Loading

It seems that this code is incorrect now:

g = (x for x in range(10))g.gi_frame.f_locals['.0'] = range(20)

But very similar code is correct:

g = (x for x in range(10))g.gi_frame.f_locals['.0'] = iter(range(20))

Maybe, some change in framelocalsproxy_setitem function can be provided?
Something like this:

if (_PyUnicode_EqualToASCIIString(key, ".0")) {    PyObject *it = PyObject_GetIter(value);    if (it != NULL) {        value = it;    }}

I've provided this change, but it breaks two tests: test_proxy_key_stringlikes_overwrite and test_proxy_key_stringlikes_ftrst_write. Those tests are authored by@encukou and@ncoghlan.

@JelleZijlstra
Copy link
Member

Setting.0 to aiter(range(20)) should work for now, yes. I don't know what you're trying to use this for at a higher level, but I'd recommend trying to find a solution that doesn't involve mutatingf_locals; that's a low-level interface that should probably only be used by tools like debuggers.

I would oppose adding special handling in the FrameLocalsProxy that mutates objects; doing so would complicate the implementation and make it less useful for tools like debuggers.

@efimov-mikhail
Copy link
MemberAuthor

Actually, there is no real world use case when changing underlying iterator for a generator is needed.
And I understand that f_locals is a low-level interface.

Main purpose of my questions is achieving some clarity.
That's totally okay if the current code behavior is desired.
Maybe just some piece of docs about this use case should be provided.
And, of course, raising the appropriate exceptions is much better than crashing.

@efimov-mikhail
Copy link
MemberAuthor

I've moved my test from test_generators.py to test_frame.py, renamed it, and added more test cases to emphasize current code behavior and save it explicitly in tests.
IMHO, there is still a room to improvement.

savannahostrowskiand others added10 commitsOctober 9, 2024 13:08
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>Co-authored-by: Tomas R <tomas.roun8@gmail.com>
Co-authored-by: musvaage <musvaage@users.noreply.github.com>
* Replace unicode_compare_eq() with unicode_eq().* Use unicode_eq() in setobject.c.* Replace _PyUnicode_EQ() with _PyUnicode_Equal().* Remove unicode_compare_eq() and _PyUnicode_EQ().
…008 implementation issues (python#125151)Skip test_fma_zero_result on NetBSD due to IEE 754-2008 implementation issues
…124974)Now it returns a tuple of up to 100 strings (an empty tuple on most locales).Previously it returned the first item of that tuple or an empty string.
…pire far in the future by default (pythonGH-107594)This allows testing Y2038 with system time set to after that,so that actual Y2038 issues can be exposed, and not maskedby expired certificate errors.Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
@efimov-mikhail
Copy link
MemberAuthor

See#125178

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@markshannonmarkshannonAwaiting requested review from markshannonmarkshannon is a code owner

@JelleZijlstraJelleZijlstraAwaiting requested review from JelleZijlstra

@willingcwillingcAwaiting requested review from willingcwillingc is a code owner

@ethanfurmanethanfurmanAwaiting requested review from ethanfurmanethanfurman is a code owner

@rhettingerrhettingerAwaiting requested review from rhettingerrhettinger is a code owner

@encukouencukouAwaiting requested review from encukouencukou is a code owner

@berkerpeksagberkerpeksagAwaiting requested review from berkerpeksagberkerpeksag is a code owner

@erlend-aaslanderlend-aaslandAwaiting requested review from erlend-aaslanderlend-aasland is a code owner

@pgansslepganssleAwaiting requested review from pgansslepganssle is a code owner

@abalkinabalkinAwaiting requested review from abalkinabalkin is a code owner

@vsajipvsajipAwaiting requested review from vsajipvsajip is a code owner

@iritkatrieliritkatrielAwaiting requested review from iritkatrieliritkatriel is a code owner

@ericsnowcurrentlyericsnowcurrentlyAwaiting requested review from ericsnowcurrentlyericsnowcurrently is a code owner

@pablogsalpablogsalAwaiting requested review from pablogsalpablogsal is a code owner

@lysnikolaoulysnikolaouAwaiting requested review from lysnikolaoulysnikolaou is a code owner

@ambvambvAwaiting requested review from ambvambv is a code owner

Assignees

No one assigned

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

20 participants

@efimov-mikhail@gaogaotiantian@JelleZijlstra@vstinner@emilyemorehouse@nedbat@cmaloney@ncoghlan@serhiy-storchaka@picnixz@zuo@Eclips4@monkeyman192@mpage@kumaraditya303@mdboom@spacemanspiff2007@rhettinger@skirpichev@savannahostrowski

[8]ページ先頭

©2009-2025 Movatter.jp