Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

gh-121141: add support forcopy.replace to AST nodes#121162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
JelleZijlstra merged 25 commits intopython:mainfrompicnixz:ast-copy-replace
Jul 4, 2024

Conversation

@picnixz
Copy link
Member

@picnixzpicnixz commentedJun 29, 2024
edited
Loading

cc@JelleZijlstra@serhiy-storchaka

I eventually had enough time for this one, but I had a question on the implementation and efficiency. Assume that I have the following:

node=ast.Name(id='a')node.foo=1node.bar=2

I cannot doast.Name(id='a', foo=...) in 3.15 (it's deprecated) and this forbids me to call the constructor with the keyword arguments passed tocopy.replace. So,copy.replace(node, id='b', foo=0) withkwargs = {'id': '5', 'foo': 0} is implemented as follows:

  1. Createnode2 = ast.Name(id='b', ctx=node.ctx) (this pops 'id' fromkwargs, or more generally, all fields known to the AST class, but not those extra fields).
  2. Donode2.__dict__ |= kwargs (i.e., you make the changes on the extra fields).
  3. Nownode2.id == 'b' andnode2.foo = 0 butnode2 hasno attributebar yet because the constructor did not carry this information.
  4. So, I need again to do something likePyDict_Merge(node2.__dict__, node.__dict__, 0) so that all remaining values fromnode.__dict__ that were neither obtained in 1 or 2 are not set.

Ideally, I'd like to create a function that can add me whatever attributes I want without warning me but I'm not sure of the refactoring. So for now, I left that logic separate. There is a separate routine that checks whether you can replace a field or not (e.g.,copy.replace(node, unknown=1) would fail because the node has no attributes namedunknown for now).


I also fixedast.compare because I maylack attributes or fields, and comparing two objects that lack attributes or fields would still be correct IMO. If you want me to make another PR for that, I can cherry pick the commit (it's the first one).


EDIT: simple wording question but should it be "add support for [...] to AST nodes" or "for AST nodes"? (or whatever else?)


📚 Documentation preview 📚:https://cpython-previews--121162.org.readthedocs.build/

@serhiy-storchaka
Copy link
Member

I think that it should only accept fields known to the AST class.

AlexWaygood reacted with thumbs up emoji

@picnixz
Copy link
MemberAuthor

I think that it should only accept fields known to the AST class.

I can live with that assumption but I need to update the docs then. However, what about constructions that add parent's back references. I would expect such nodes to be able to copy themselves but only change one single attribute (like a sibling) so that's why I only allowed the fields in__dict__ to be mutated.

By the way, is there something I should have taken care of because of singletons such asast.Load? Now that I see my tests, I'm not sure if a special treatement is needed.

@serhiy-storchaka
Copy link
Member

AFAIK, all other__replace__ implementations only allow to change the fixed set of fields, not arbitrary instance attributes.

@picnixz
Copy link
MemberAuthor

Actually, I assumed that AST nodes were somewhat similar to simple namespaces, which allows for arbitrary addition I think:

if (kwargs) {
if (PyDict_Update(((_PyNamespaceObject*)result)->ns_dict,kwargs)<0) {
Py_DECREF(result);
returnNULL;
}
}

But if you think we should restrict to fields known to the class, I can easily remove the code.

@JelleZijlstra
Copy link
Member

I feel AST nodes are more like dataclasses. For dataclasses, unrecognized arguments are rejected bycopy.replace and non-field attributes on the original object are lost.

>>> from dataclasses import dataclass>>> >>> @dataclass... class A:...     a: int...     >>> from copy import replace>>> a = A(1)>>> a.b = 42>>> a2 = replace(a, a=2)>>> a2A(a=2)>>> a2.bTraceback (most recent call last):  File "<python-input-8>", line 1, in <module>    a2.bAttributeError: 'A' object has no attribute 'b'>>> a.b42>>> replace(a, b=5)Traceback (most recent call last):  File "<python-input-10>", line 1, in <module>    replace(a, b=5)    ~~~~~~~^^^^^^^^  File "/Users/jelle/py/cpython/Lib/copy.py", line 294, in replace    return func(obj, **changes)  File "/Users/jelle/py/cpython/Lib/dataclasses.py", line 1630, in _replace    return self.__class__(**changes)           ~~~~~~~~~~~~~~^^^^^^^^^^^TypeError: A.__init__() got an unexpected keyword argument 'b'
AlexWaygood and picnixz reacted with thumbs up emoji

@picnixz
Copy link
MemberAuthor

Just after I sent my comment, I actually thought that maybe I should reject them since we anyway reject arbitrary keyword arguments in the constructor, maybe it's indeed closer to dataclasses.

I convinced myself as well and since we reached a consensus, I'll remove this. I'll extend thevalidate function (good thing I did this).

@serhiy-storchaka
Copy link
Member

SimpleNamespace constructor accepts arbitrary keyword arguments, so__replace__() does the same. All names are valid forSimpleNamespace, but other objects are more picky.

Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think that the implementation can be simpler.

  • Get fields (PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields)).
  • Iterate them, and get corresponding values (PyDict_GetItemRef(dict, name, &value)).
  • Add corresponding name-value pair to the new dictionary.
  • Update that dictionary with the keyword arguments dictionary.
  • Create the object of the same type by passing a new dictionary as keyword arguments.

Validation is performed in the constructor.

Equivalent Python code:

def__replace__(self,/,**kwargs):newkwargs= {}fornameinself._fields:value=getattr(self,name)newkwargs[name]=valuenewkwargs.update(kwargs)returntype(self)(**kwargs)

@picnixz
Copy link
MemberAuthor

picnixz commentedJun 30, 2024
edited
Loading

I think that the implementation can be simpler.

Yes, that's what I decided after our discussion from yesterday. Currently the construction only warns about extra keyword arguments. For now, I'll leave that warning but this also means that the fieldsare replaced, even though the behaviour would disappear. In particular, in 3.15 and later, the tests should be changed, so I added some comments but is there a more rigourous way?

Copy link
Member

@serhiy-storchakaserhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM.

You even made the changes I thought about but didn't write so as not to be too picky. We think alike.

Copy link
MemberAuthor

@picnixzpicnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think with this implementation we raise exceptions whenever it is needed (except at one point where a warning is emitted, but I think it's a separate issue; see my previous comment).

Anyway I hope that I have made the requested changes; please review again.

@picnixz
Copy link
MemberAuthor

Ah the bot did not catch the fact that I said that I was done with the changes.

I have made the requested changes; please review again.

@bedevere-app
Copy link

Thanks for making the requested changes!

@serhiy-storchaka,@JelleZijlstra: please review the changes made to this pull request.

@JelleZijlstraJelleZijlstra added the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelJul 3, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by@JelleZijlstra for commitbddcb97 🤖

If you want to schedule another build, you need to add the🔨 test-with-refleak-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-refleak-buildbotsTest PR w/ refleak buildbots; report in status section labelJul 3, 2024
@picnixz
Copy link
MemberAuthor

(I hope I didn't leak things. The previous implementation didn't but maybe this one does). I'll be leaving for today, my defense is tomorrow so I'll patch if afterwards if needed.

@JelleZijlstra
Copy link
Member

Just running the refleak bots because it's good practice to do that when there's a big pile of new C code.

Good luck with your defense!

picnixz reacted with heart emoji

@JelleZijlstraJelleZijlstra merged commit9728ead intopython:mainJul 4, 2024
@picnixzpicnixz deleted the ast-copy-replace branchJuly 4, 2024 07:00
noahbkim pushed a commit to hudson-trading/cpython that referenced this pull requestJul 11, 2024
estyxx pushed a commit to estyxx/cpython that referenced this pull requestJul 17, 2024
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@JelleZijlstraJelleZijlstraJelleZijlstra approved these changes

@isidenticalisidenticalAwaiting requested review from isidentical

@serhiy-storchakaserhiy-storchakaAwaiting requested review from serhiy-storchaka

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@picnixz@serhiy-storchaka@JelleZijlstra@bedevere-bot

[8]ページ先頭

©2009-2025 Movatter.jp