Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Avoid falseunreachable,redundant-expr, andredundant-casts warnings in loops more robustly and efficiently, and avoid multiplerevealed type notes for the same line.#19118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
JukkaL merged 7 commits intopython:masterfromtyralla:fix/reachability_regression
Jun 6, 2025

Conversation

tyralla
Copy link
Collaborator

@tyrallatyralla commentedMay 20, 2025
edited
Loading

Fixes#18606
Closes#18511
Improves#18991
Fixes#19170

This change is an improvement over9685171. Besides fixing the regressions reported in#18606 and#19170 and removing the duplicates reported in#18511, it should significantly reduce the performance regression reported in#18991. At least runningMeasure-command {python runtests.py self} on my computer (with removed cache) is 10 % faster.

tyrallaand others added2 commitsMay 20, 2025 07:12
… robustly and efficiently, and avoid multiple `revealed type` notes for the same line.This change is an improvement over9685171.  Besides fixing the regression reported inpython#18606 and removing the duplicates reported inpython#18511, it should significantly reduce the performance regression reported inpython#18991.  At least running `Measure-command {python runtests.py self}` on my computer (with removed cache) is 10 % faster.
@github-actionsGitHub Actions

This comment has been minimized.

@tyrallatyralla requested a review fromJukkaLMay 20, 2025 06:58
@tyralla
Copy link
CollaboratorAuthor

@JukkaL: I thought that later responses toreveal_type are always better than earlier ones. However, this is not the case fortestNewRedefineTryStatement test case. Maybe the most general approach would be to collect all the union items of the individual responses and to combine them in the final note. I could try this, if you think it's a good way to go.

Alternatively, I could just remove thereveal_type part of this pull request, as it seems less important than the other two addressed issues.

@github-actionsGitHub Actions

This comment has been minimized.

@tyralla
Copy link
CollaboratorAuthor

I implemented a collection mechanism for the types revealed in different iteration steps. In the case of unions, the items are sorted alphabetically without differentiating upper and lower case letters.

At first glance, the reason for the Mypy primer diff (https://github.com/pydata/xarray) is not clear to me.

I asked for a review of those who opened the mentioned issues. Thanks in advance!

@sterliakov
Copy link
Collaborator

xarray is#19121, we have it on all current PRs

@tyralla
Copy link
CollaboratorAuthor

@sterliakov: Thanks, good to know. (Anybody already working on it?)

@sterliakov
Copy link
Collaborator

@tyralla I'm a bit swamped, there are no related PRs open - so probably not yet? It is likely related to#18976, but I have no idea why it's inconsistent - any MRE extracted from that fails every time, not randomly.https://mypy-play.net/?mypy=master&python=3.12&flags=strict&gist=50f4ae811a650f9e90c6f52269f59d29

Copy link
Collaborator

@A5rocksA5rocks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'm not immediately convinced this is robust!

# We collect errors that indicate unreachability or redundant expressions
# in the first iteration and remove them in subsequent iterations if the
# related statement becomes reachable or non-redundant due to changes in
# partial types:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

What if an unreachable statement is within another unreachable statement?

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I am not exactly sure what you are asking for.

Is this an example that fits your question?

[casetestAvoidUnreachableCodeInUnreachableCodeWithPartialType]# flags: --warn-unreachable --python-version 3.11x=Noney=NonewhilexisNoneandyisNone:reveal_type(x)# N: Revealed type is "None"reveal_type(y)# N: Revealed type is "None"ifxisNoneandyisnotNone:x=1# E: Statement is unreachableifyisNone:y=1[builtinsfixtures/bool.pyi]

In this case, the results are the same for 1.15 and the current state of this PR.

Copy link
Collaborator

@A5rocksA5rocksMay 23, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

something like this:

x=Noney=NonewhileTrue:ifxisnotNone:print("a")ifyisnotNone:print("b")# this is unreachablex=1

(I haven't run the PR on this so I'm not sure the result, but itshould warn about unreachablity. mypy 1.15 does. I'm worried this change won't)

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This example should printb. Is there a typo?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I edited it afterwards, it shouldn't anymore.

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yes, you were totally right. The large temporarily unreachable code section shadowed the persistent unreachable code section. I added your test case and extended the current logic a bit. It's late now, so I'll postpone double-checking until tomorrow.

Can you see any more potential pitfalls?

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I again looked at it. To me, the logic appears a little special, but clear and straightforward. Of course, I cannot exclude that there are more special cases my solution does not cover. So I wait for the next review.

@tyralla
Copy link
CollaboratorAuthor

I'm not immediately convinced this is robust!

What kind of test case could convince you?

@A5rocks
Copy link
Collaborator

Also BTW I notice your "restart CI" commit -- you could usegit commit --allow-empty to avoid having to push a dummy change + a revert. (just a recommendation in case you didn't know)

@tyralla
Copy link
CollaboratorAuthor

Also BTW I notice your "restart CI" commit -- you could usegit commit --allow-empty to avoid having to push a dummy change + a revert. (just a recommendation in case you didn't know)

I didn't know. That's helpful, thanks!

…UnreachableLines` provided by@A5rocks and extend the current logic to fix it.
@tyrallatyralla requested a review fromA5rocksMay 24, 2025 00:41
@github-actionsGitHub Actions

This comment has been minimized.

Copy link
Collaborator

@A5rocksA5rocks left a comment
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

sorry, I think I misunderstood things.

for candidate in set(itertools.chain(*uselessness_errors)):
if all(
(candidate in errors) or (candidate[2] in lines)
for errors, lines in zip(uselessness_errors, unreachable_lines)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why can't this just use the last round of unreachability warnings?

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That is a good question. I initially implemented it so in229e4a6. However, in the last function definition of test casetestNewRedefineTryStatement, the first response toreveal_type does provide complete information, but the second (and final) response does not:

deff4()->None:whileint():try:x=1ifint():x=""breakifint():whileint():ifint():x=Nonebreakfinally:reveal_type(x)# N: Revealed type is "Union[builtins.int, builtins.str, None]" \# N: Revealed type is "Union[builtins.int, None]"

I have no experience with the new--allow-redefinition-new option so far, and have not tried to investigate why this happens. I just extended my approach inc24c950.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Wouldn't that case also allow something not allowed forstr? Eg does that error for:

ifxisnotNone:print(x+5)

Above thereveal_type

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Oh, after re-reading, it seems that I have not hit the point of your question. Another try: There might be some scope for optimisations. However, I am unsure what can possibly happen when running Mypy with different configurations (like in thef4 example discussed above), so I preferred to implement this part of the approach as robustly as possible.

Copy link
Collaborator

@A5rocksA5rocksMay 25, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

On the other hand it's possible to be too robust: mypy only prints the errors for the last round right? So it's possible (through some bug elsewhere, like forgetting to propagate line numbers maybe?) to have unreachable code in the last run that isn't marked -- and so have no signal that the code there isn't checked.

On the other other hand, just using the last iteration's unreachability errors might be wrong in presence of bugs like inf4, but it's simpler (and implementation is trivial to get right) and could never mislead the user about what code could get errors.


Basically I think it's better to match the semantics of how error reporting happens (ie during the last round of checking) then to have only one type of error be correct in face of bugs.


Edit: actually I might be wrong about my assumptions here, let me play with your PR a bit and I'll update this.

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

My bad; the-warn-unreachable flag was missing in the test case. Now I get bothStatement is unreachable andRevealed type is "builtins.str". However, the same happens when I check the modified test with Mypy-Master. Therefore, I don't know if this actually points to a flaw in my current approach. I will investigate tomorrow....

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

...I had no time for a detailed investigation yet, but I tend to think the problem with the currently discussed test case has nothing to do with this PR, because the response toreveal_type and theunreachable error for the same line happen at the same iteration step.

Maybe the first hint on the cause for this discrepancy:expr.node.type = Union[builtins.int, builtins.str, None] butTypeChecker.lookup_type(expr) = Union[builtins.int, None].

Copy link
Collaborator

@A5rocksA5rocksMay 31, 2025
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@JukkaL do you think types getting narrower between iterations (see previous comments) should be allowed?

If not, then could this PR simply suppressall output and only print out the last iteration?

Copy link
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Maybe I misunderstand, but that sounds like reverting the "fix part" of this PR.

This is the current behaviour with the proposed changes to Mypy:

y=NonewhileyisNone:reveal_type(y)# NoneifyisNone:y= []y.append(1)reveal_type(y)# list[int]

The responses toreveal_type are the result of the first iteration step. In the second iteration step, the whole loop's body is unreachable.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think@tyralla is right, and just printing out the last iteration is not the right thing to do. I'll still need to experiment with this however.

test case from @auntsaninja (slightly simplified)
@github-actionsGitHub Actions
Copy link
Contributor

According tomypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@tyrallatyralla changed the titleAvoid falseunreachable andredundant-expr warnings in loops more robustly and efficiently, and avoid multiplerevealed type notes for the same line.Avoid falseunreachable,redundant-expr, andredundant-casts warnings in loops more robustly and efficiently, and avoid multiplerevealed type notes for the same line.May 30, 2025
@tyralla
Copy link
CollaboratorAuthor

@hauntsaninja@hynek

I updated this PR to also fix#19170. (Or, at least the test case provided by Shantanu.) I have edited the PR's title and initial description accordingly.

@JukkaL
Copy link
Collaborator

Thanks for working on the performance regression! It would be great to measure the impact using themisc/perf_compare.py script. I can do this but possibly not until next week.

@tyralla
Copy link
CollaboratorAuthor

I did not work withmisc/perf_compare.py so far, but using it does not seem very complicated. I will try it later.

@tyralla
Copy link
CollaboratorAuthor

@JukkaL

I just executedpython .\misc\perf_compare.py master fix/reachability_regression.

The results:

master                    10.851s (0.0%) | stdev 0.373sfix/reachability_regression 10.044s (-7.4%) | stdev 0.423sTotal time taken by the whole benchmarking program (including any setup): 11 minutes, 32 seconds

So, according to this measurement, this PR achieves a speed increase of 7.4 %, while#18433 resulted in a slowdown of 8.7 %. I am not familiar with the involved tools, but this intuitively makes sense to me and aligns roughly with my initial estimate usingMeasure-command {python runtests.py self} (which is already somewhat outdated because some details of the implementation changed).

@JukkaL
Copy link
Collaborator

So, according to this measurement, this PR achieves a speed increase of 7.4 %, while#18433 resulted in a slowdown of 8.7 %

This looks good. The exact numbers depend on the specifics of hardware, Python version, C compiler, etc.

Copy link
Collaborator

@JukkaLJukkaL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The seems good to me, and at least a significant improvement over the old behavior (plus it's faster!).

@A5rocks Do you still have further concerns?

@A5rocks
Copy link
Collaborator

A5rocks commentedJun 5, 2025
edited
Loading

I still think this is a bug:#19118 (comment)

But it's probably a bug with some other things rather than specifically the logic in this PR (mainly because the logic looks right to me). (though, what's the point of more robust logic if it cannot prevent that)

@tyralla
Copy link
CollaboratorAuthor

@A5rocks

I appreciate your carefulness. Still, I rethought the problem and am now very sure it is beyond the scope of this PR. This PR only tries to correctly aggregate information that Mypy collects stepwise when analysing loops. The bug you discovered, however, can be triggered without loops:

# Mypy 1.16.0# --allow-redefinition-new --local-partial-types --warn-unreachabledeff()->None:try:x=1ifint():x=""returnifint():x=Nonereturnfinally:reveal_type(x)# N: Revealed type is "Union[builtins.int, builtins.str, None]" \# N: Revealed type is "builtins.int"ifisinstance(x,str):# E: Subclass of "int" and "str" cannot exist: would have incompatible method signaturesreveal_type(x)# E: Statement is unreachable \# E: Revealed type is "builtins.str"reveal_type(x)# N: Revealed type is "Union[builtins.int, builtins.str, None]" \# N: Revealed type is "builtins.int"

This suggests thattry-finally statements are also handled iteratively and may require similar treatment to the one we are discussing for loops in this PR. The corresponding code and code comments suggest the same:

mypy/mypy/checker.py

Lines 4954 to 4985 inb025bda

defvisit_try_stmt(self,s:TryStmt)->None:
"""Type check a try statement."""
# Our enclosing frame will get the result if the try/except falls through.
# This one gets all possible states after the try block exited abnormally
# (by exception, return, break, etc.)
withself.binder.frame_context(can_skip=False,fall_through=0):
# Not only might the body of the try statement exit
# abnormally, but so might an exception handler or else
# clause. The finally clause runs in *all* cases, so we
# need an outer try frame to catch all intermediate states
# in case an exception is raised during an except or else
# clause. As an optimization, only create the outer try
# frame when there actually is a finally clause.
self.visit_try_without_finally(s,try_frame=bool(s.finally_body))
ifs.finally_body:
# First we check finally_body is type safe on all abnormal exit paths
self.accept(s.finally_body)
ifs.finally_body:
# Then we try again for the more restricted set of options
# that can fall through. (Why do we need to check the
# finally clause twice? Depending on whether the finally
# clause was reached by the try clause falling off the end
# or exiting abnormally, after completing the finally clause
# either flow will continue to after the entire try statement
# or the exception/return/etc. will be processed and control
# flow will escape. We need to check that the finally clause
# type checks in both contexts, but only the resulting types
# from the latter context affect the type state in the code
# that follows the try statement.)
ifnotself.binder.is_unreachable():
self.accept(s.finally_body)

Would you like to open a separate issue? I think I could work on it next week.

@JukkaLJukkaL merged commit55c4067 intopython:masterJun 6, 2025
19 checks passed
@JukkaLJukkaL mentioned this pull requestJun 6, 2025
tyralla added a commit to tyralla/mypy that referenced this pull requestJun 6, 2025
JukkaL pushed a commit that referenced this pull requestJun 20, 2025
…ally clauses. (#19270)Fixes#19269This PR refactors the logic implemented in#19118 (which only targetedrepeatedly checked loops) and applies it to repeatedly checked finallyclauses.I moved nearly all relevant code to the class `LoopErrorWatcher`, whichnow has the more general name `IterationErrorWatcher`, to avoid codeduplication. However, one duplication is left, which concerns errorreporting. It would be nice and easy to move this functionality to`IterationErrorWatcher`, too, but this would result in import cycles,and I am unsure if working with `TYPE_CHECKING` and postponed importingis acceptable in such cases (both for Mypy and Mypyc).After the refactoring, it should not be much effort to apply the logicto other cases where code sections are analysed iteratively. However,the only thing that comes to my mind is the repeated checking offunctions with arguments that contain constrained type variables. I willcheck it. If anyone finds a similar case and the solution is as simpleas expected, we could add the fix to this PR, of course.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@JukkaLJukkaLJukkaL approved these changes

@A5rocksA5rocksA5rocks approved these changes

@hauntsaninjahauntsaninjaAwaiting requested review from hauntsaninja

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@tyralla@sterliakov@A5rocks@JukkaL

[8]ページ先頭

©2009-2025 Movatter.jp