Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

GH-130798: Add type hints to pathlib.types#131639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
ap-- wants to merge19 commits intopython:main
base:main
Choose a base branch
Loading
fromap--:pathlib-types-add-typehints

Conversation

ap--
Copy link
Contributor

@ap--ap-- commentedMar 23, 2025
edited
Loading

Hello everyone,

This PR addresses#130798

As requested, the type hints are kept compatible with Python 3.9 to simplify release inpathlib-abc.

I also just noticed#131621 which I assumed will be merged soon, because it simplifies typing of_ReadablePath.copy and_ReadablePath.copy_into. Otherwise, forstr targets the_ReadablePath subclass has to be a_WritablePath subclass too to make sense as a target.

Since, I haven't contributed type hinted code to cpython so far, it would be great if you could let me know if this PR should also ship tests for verifying the types somewhere.

Cheers,
Andreas 😃

barneygale reacted with heart emoji
Copy link
Member

@picnixzpicnixz left a comment
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'm worried about the covariant returns that are lacking but I don't remember if it's allowed to have a method withdef f(self: T_co) -> T_co: ...

@ap--
Copy link
ContributorAuthor

Thanks for the review!

I'm worried about the covariant returns that are lacking but I don't remember if it's allowed to have a method withdef f(self: T_co) -> T_co: ...

Could you explain what you mean specifically?_JoinablePath is not generic so I don't see how the covariant typevars would play a role here.

@picnixz
Copy link
Member

Could you explain what you mean specifically

I should avoid reviewing while doing C++. Invariance is sufficient here, my bad.

@barneygale
Copy link
Contributor

barneygale commentedMar 24, 2025
edited
Loading

BTW@ap--, this is awesome! Thanks so much!! And thanks also@picnixz! I appreciate your scrutiny :)

- pathsegments are `str`: joinpath, __truediv__, __rtruediv__, with_segments- symlink_to target is `str`- glob recurse_symlinks is `Literal[True]`- full_match and glob pattern is `str`
@ap--
Copy link
ContributorAuthor

@barneygale@picnixz I made all changes as suggested☺️

I am now wondering if it wouldn't be better to just usetyping.Self and rely on typing_extensions inpathlib-abc. I think it would make this module a lot easier to parse visually.

barneygale reacted with thumbs up emoji

@barneygale
Copy link
Contributor

barneygale commentedMar 24, 2025
edited
Loading

That's a much better plan - please feel free.

@barneygale
Copy link
Contributor

What do you chaps think aboutWritablePath.write_text() andwrite_bytes() returning anint? It might be a pain to implement for users who override those methods. We could make that apathlib.Path-exclusive feature perhaps?

Copy link
Member

@picnixzpicnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Forwrite_* methods, one could return int or int / None if you want children classes to only return None if it's too hard for them to know how many characters were written. WDYT@barneygale?

barneygale reacted with thumbs up emoji
@ap--
Copy link
ContributorAuthor

What do you chaps think aboutWritablePath.write_text() andwrite_bytes() returning anint? It might be a pain to implement for users who override those methods.

Given that__open_wb__ is abstract, and write_text and write_bytes ship default implementations, I don't think they need to be simplified.

barneygale reacted with thumbs up emoji

"""
Return the path to which the symbolic link points.
"""
raise NotImplementedError

def copy(self, target, **kwargs):
def copy(self, target: _WP, **kwargs: Any) -> _WP:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Is it possible to use aParamSpec/similar here to show that the keyword arguments are passed totarget._copy_from()?

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I spent quite a bit of time now trying to get this to work, but I believe the way_ReadablePath.copy and_WritablePath._copy_from are coupled is currently not representable in the Python type system.

Be aware, I am not an expert in this, so all conclusions I draw here should be confirmed with someone who is.

Ideally the typechecker would just be able to infer if keyword arguments are not compatible with the signature of a detected subclass of_WritablePath._copy_from . But all my attempts failed so far. Basically the minimal scenario that needs to be supported is:

classR:defcopy[T:W](self,target:T,**kwargs)->T:target._copy_from(self,**kwargs)returntargetclassW:def_copy_from(self,source:R,/,*,follow_symlinks:Literal[True]=True)->None:return
  1. In this example: "not typing the kwargs" (or typing them as Any) leads to missing incorrect keyword arguments:https://mypy-play.net/?mypy=latest&python=3.13&gist=7608e8dcd5e4b58dc2fc2c355bf6ed89

  2. Trying to use TypedDicts to define the kwargs doesn't work either because the definitions would have to be provided to_ReadablePath.copy for each speficic target class:https://mypy-play.net/?mypy=latest&python=3.13&gist=12f4cbdd725073d8a152a90a941922cf

  3. ParamSpec looks pretty promising but to be able to actually bind the parameters, you need to go via a protocol class, which prevents you from reusing the actual target type as return type of the_ReadablePath.copy method. Moreover ParamSpec can't be used without binding P.args to the function signature:https://mypy-play.net/?mypy=latest&python=3.13&gist=b47ea834a0f57a722b8124f76f3fb6d4

Ideally we'd be able to do something like:

classR:defcopy[T:W,**P](self,target:T[P],**kwargs:P.kwargs)->T[P]: ...classW[**P]:def_copy_from(self,source:R,**kwargs:P.kwargs)->None: ...

But from reading through a bunch of docs and issues, I this seems impossible. I believe the correct related issue here should be this onepython/typing#548 for support of higher-kinded typevars, and we'd needkwargs_only support for ParamSpec:python/typing#1524

My guess is that by redesigning the coupling between_ReadablePath.copy and_WritablePath._copy_from it should be possible to better represent this in the type system. But it's getting late here, and I think I'll revisit this with a fresh mind tomorrow.

barneygale reacted with heart emoji

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Intersection types might also work, since then you could do the following:

defcopy[T:W,**P](self,target:T&SupportsCopyFrom[P],*args:P.args,**kwargs:P.kwargs)->T: ...

This would mean thattarget must be both aT andSupportsCopyFrom simultaneously. Intersections I think are being considered more seriously than HKT, but they're also another quite substantial addition to the type system.

ap-- and barneygale reacted with heart emoji
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for the analysis! Feels like we're pushing the current type system a bit far, so I'm happy withAny. I could log an issue in pathlib-abc and link back to this thread.

I'm also happy to change howcopy() and_copy_from() interact if there's something better we can do.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think I have a solution, though it's awkward. First change is to pass*args too, meaningParamSpec works. Second is to require_copy_to() to redundantly returnself. Then we could do the following:

class_HasCopyTo[**Args,R](Protocol):def_copy_to(self,path:ReadablePath,/,*args:Args.args,**kwargs:Args.kwargs)->R: ...classReadablePath:defcopy[**Args,R](self,path:_HasCopyTo[Args,R],/,*args:Args.args,**kwargs:Args.kwargs,    )->R:res=path._copy_to(self,*args,**kwargs)assertpathisres,"Must return self"returnresclassWritablePath(ReadablePath):@abstractmethoddef_copy_to(self,path:ReadablePath,/,*args:Any,**kwargs:Any)->Self: ...classPath(WritablePath):def_copy_to(self,path:ReadablePath,/,some_option:bool)->Self:do_copy(self,path)returnself

TheAny inWritablePath is required to allow subclasses to require any arguments, but it also (correctly?) means if you callcopy() with something that's justWritablePath it'll accept anything.

ap-- reacted with heart emoji
Copy link
ContributorAuthor

@ap--ap--Mar 28, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It does! I'll make the change:

https://mypy-play.net/?mypy=latest&python=3.11&gist=8c300324c700eea4ffa7b7776a460f6e

code
fromtypingimportLiteralfromtypingimportProtocolfromtypingimportGenericfromtypingimportTypeVarfromtypingimportParamSpecfromtypingimportCallablefromtypingimportSelffromtypingimportClassVarfromtypingimportAnyR=TypeVar("R",bound="_WritablePath",covariant=True)P=ParamSpec("P")class_HasCopyFrom(Protocol[R,P]):def_copy_from(self,source:_ReadablePath,/,*args:P.args,**kwargs:P.kwargs)->R:        ...class_ReadablePath:defcopy(self,path:_HasCopyFrom[R,P],/,*args:P.args,**kwargs:P.kwargs)->R:returnpath._copy_from(self,*args,**kwargs)class_WritablePath:def_copy_from(self,path:_ReadablePath,/,*args:Any,**kwargs:Any)->Self: ...classMyPathR(_ReadablePath):passclassMyPathW0(_WritablePath):def_copy_from(self,path:_ReadablePath,/,*,some_option:bool=True)->Self:returnselfclassMyPathW1(_WritablePath):def_copy_from(self,path:_ReadablePath,/,*,other_option:int=3)->Self:returnselfr=MyPathR()w0=MyPathW0()w1=MyPathW1()# correctreveal_type(r.copy(w0))r.copy(w0,some_option=False)reveal_type(r.copy(w1))r.copy(w1,other_option=4)# errorr.copy(w0,dont_exist=2)r.copy(w1,text="str")r.copy(w0,some_option=1.0)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hmmm, I tried implementing the changes, and they don't fully work. While they solve the copy (*args and)**kwargs issue, they cause type checking issues in for examplecopy_into because mypy can't infer the actual type of path, but only that path has the structural type_HasCopyFrom. This is a limitation of how the protocol is expressed (see here:https://mypy-play.net/?mypy=latest&python=3.12&gist=c744090b69aad82d75c14e6ab7dedb4c ). So it seems intersection types would be required to do this correctly.

Also on a side note: I now wonder iffollow_symlinks should be an explicit keyword argument incopy andcopy_into since it's mandatory in_WritablePath._copy_from's default implementation.

Additionally, forcing_copy_from to returnSelf would interfere with#131636

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Fixed the issues - you need to be returningR from_copy_from, and_HasCopyFrom.__truediv__/joinpath needs to return_HasCopyFrom, not justR. That was causing it to discard the paramspec and specific return value. For#131636, you can still do-> Generator[tuple[...], None, Self].

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yeah, but the problem imo is that bothjoinpath and__truediv__ should actually return the same type. The workaround is nice, but to make it work, one needs to be typed as returningR because it has to return the WritablePath instance and the other as theHasCopyFrom protocol because it's passed again into .copy ...

So in the end you have to adjust the protocol to the exact implementation of copy and copy_into which is unfortunate. If both in copy and copy_into joinpath would be used once as the return type and once to pass as the target arg to copy, the typing workaround would not work.

I'm not sure if the benefit of being able to type check the copy kwargs justifies the introduction of this trick.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Agree with@ap--. We could always revisit this later. Thank you both for your work on this, it's been really valuable.

@barneygale
Copy link
Contributor

Let's keep theint return values forwrite_text() andwrite_bytes() - thanks both.

@ap--
Copy link
ContributorAuthor

Just noticed the discussion in#130798: I'm happy to recreate this PR in typeshed if needed.

@barneygale
Copy link
Contributor

No-one onthe forum has objected yet, so I suspect we're fine to proceed with type hints here. But let's wait a few more days before we merge, just in case someone replies on the forum with an objection.

@ap--
Copy link
ContributorAuthor

ap-- commentedMar 29, 2025
edited
Loading

Sounds good! Three things were brought up on DPO that I think we should address before merging:

  1. "Annotations that are not checked by CI should not be added to any project, [...]"

I fully agree with that one. In universal-pathlib I've been usinghttps://github.com/typeddjango/pytest-mypy-plugins to run tests against the type annotations of the UPath interface(seehere, with an example runhere), to make sure I don't accidentally change the return type of some method to Any (which happened before I had the CI tests in place). The problem is of course that this is enabled through a third party pytest plugin. My guess is for cpython we'd have to roll our own relying on unittest and mypy only.

  1. "Type checkers themselves don’t use stdlib source annotations, they use typeshed"

We need to investigate if there's an issue with actually checking the stdlibpathlib.types annotations with mypy because it might just rely on typeshed? If that's the case, and we'd need to duplicate annotations in typeshed, there's not really a benefit in having the annotations here, because either way we'd have to carry the maintenance burden of keeping the two places in sync.

  1. "I think we should only use simple annotations (some of the mindblowing generic type stuff obfuscates code rather than clarifying it)."

I also agree with this one, and I wonder if we should add a comment to.copy and.copy_into that type annotations for kwargs will be added once it is possible to write these in an intuitive and concise way. (== once intersection types are available and ParamSpec supports binding only kwargs)

barneygale reacted with thumbs up emoji

@ap--
Copy link
ContributorAuthor

ap-- commentedApr 7, 2025

Hi@barneygale

I have time this week to continue with this. Before I start with working on a solution for testing I wanted to run the following idea by you: Currentlypathlib-abc pulls frompathlib.types to sync development. I think if this would be inverted, most of the issues with "type annotations in the stdlib" and testing would go away:

  • pathlib-abc would be fully type annotated and the types could be tested with whatever method is convenient
  • every timepathlib-abc code is synced to the stdlib, the type annotations would be stripped frompathlib.types and synced as stubs totypeshed

But this will probably introduce some friction with feature development and refactoring for pathlib, because things live in two different locations now... Anyways, let me know what you think☺️(Also let me know if this discussion should be moved somewhere else)

For now, I'll ask in typeshed how testing would best be established in the current scenario.

Have a great week,
Andreas 😃

@barneygale
Copy link
Contributor

barneygale commentedApr 10, 2025
edited
Loading

Hi Andreas!

So far I've avoided making pathlib-abc anupstream project of CPython, because I worry I'd be bypassing the usual CPython development process. I'd end up making cpython PRs like "Backport changes from pathlib-abc 0.9.8" where core devs might feel less able to request changes, given things have already landed upstream. Maybe it's not a big deal...

I don't think it's necessary to runmypy in the CPython CI if we run it in the pathlib-abc CI. There will be a delay between things landing in CPython and it flagging in the pathlib-abc backport CI, but I suspect that's OK for the time being.

pathlib.types isn't the only stdlib module with type annotations -wsgiref.types is another example. In that case typeshed appears to copy the type annotations from the stdlib. Would that work forpathlib.types too?

Hope you're having an excellent week also :D

@python-cla-bot
Copy link

python-cla-botbot commentedApr 18, 2025
edited
Loading

All commit authors signed the Contributor License Agreement.

CLA signed

@ap--
Copy link
ContributorAuthor

ap-- commentedApr 23, 2025
edited
Loading

removing runtime dependency on typing

I started to remove the runtime typing imports frompathlib.types but there is a blocking issue, that I ran into. There are two runtime_checkable protocols inpathlib.types:_PathParser andPathInfo. It is possible to remove the dependency ontyping.Protocol andtyping.runtime_checkable forPathInfo (like forio.Reader) as shown here:

classPathInfo(metaclass=ABCMeta):"""Protocol for path info objects, which support querying the file type.    Methods may return cached results.    """@abstractmethoddefexists(self,*,follow_symlinks:bool=True)->bool: ...@abstractmethoddefis_dir(self,*,follow_symlinks:bool=True)->bool: ...@abstractmethoddefis_file(self,*,follow_symlinks:bool=True)->bool: ...@abstractmethoddefis_symlink(self)->bool: ...@classmethoddef__subclasshook__(cls,C):ifclsisPathInfo:return_check_methods(C,"exists","is_dir","is_file","is_symlink")returnNotImplemented

The trick above does not work for_PathParser though. The reason is (1) that_PathParser has an Optionalaltsep attribute and_collections_abc._check_methods interprets attributes that are None as missing. And (2) the_PathParser Protocol should succeed for anisinstance(posixpath, _PathParser) check, which the__subclasshook__ trick does not cover for modules.

Maybe it's ultimately not necessary forisinstance(posixpath, _PathParser) andisinstance(ntpath, _PathParser) to evaluate to True, since users would implement their custom parsers as classes anyways?

type checking inMisc/mypy

So I now added pathlib toMisc/mypy to start type checking the pathlib code in CPython similar to_pyrepl (thanks core.py for the pointer☺️ ) There are a few issues with that though, for which it would be great to get feedback.

(1) I followed the same pattern as_pyrepl did with including a custommypy.ini in pathlib and symlinking, but it seems that theglob imports bleed through to the_pyrepl checks, unless I explicitlytype: ignore[attr-defined]e596130 the import inpathlib/__init__.py. Is there a way to prevent this?

(2) One mypy error showed upLib/pathlib/__init__.py:57: error: Missing type parameters for generic type "Sequence" [type-arg] which I tried fixing with

class_PathParents[T:PurePath](Sequence[T]):    ...

But it crashes mypy. So I added the (wrong) workaround in7cd07f1
I think this is a bug in mypy, and I'll report this there asap. I am wondering if I should wait for this to be fixed instead in mypy instead of the workaround commit I applied here.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@barneygalebarneygalebarneygale left review comments

@TeamSpen210TeamSpen210TeamSpen210 left review comments

@picnixzpicnixzpicnixz left review comments

@ezio-melottiezio-melottiAwaiting requested review from ezio-melottiezio-melotti is a code owner

@hugovkhugovkAwaiting requested review from hugovkhugovk is a code owner

@AA-TurnerAA-TurnerAwaiting requested review from AA-TurnerAA-Turner is a code owner

Assignees
No one assigned
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

4 participants
@ap--@picnixz@barneygale@TeamSpen210

[8]ページ先頭

©2009-2025 Movatter.jp