Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork3k
Avoid falseunreachable
,redundant-expr
, andredundant-casts
warnings in loops more robustly and efficiently, and avoid multiplerevealed type
notes for the same line.#19118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
… robustly and efficiently, and avoid multiple `revealed type` notes for the same line.This change is an improvement over9685171. Besides fixing the regression reported inpython#18606 and removing the duplicates reported inpython#18511, it should significantly reduce the performance regression reported inpython#18991. At least running `Measure-command {python runtests.py self}` on my computer (with removed cache) is 10 % faster.
for more information, seehttps://pre-commit.ci
This comment has been minimized.
This comment has been minimized.
@JukkaL: I thought that later responses to Alternatively, I could just remove the |
This comment has been minimized.
This comment has been minimized.
I implemented a collection mechanism for the types revealed in different iteration steps. In the case of unions, the items are sorted alphabetically without differentiating upper and lower case letters. At first glance, the reason for the Mypy primer diff (https://github.com/pydata/xarray) is not clear to me. I asked for a review of those who opened the mentioned issues. Thanks in advance! |
xarray is#19121, we have it on all current PRs |
@sterliakov: Thanks, good to know. (Anybody already working on it?) |
@tyralla I'm a bit swamped, there are no related PRs open - so probably not yet? It is likely related to#18976, but I have no idea why it's inconsistent - any MRE extracted from that fails every time, not randomly.https://mypy-play.net/?mypy=master&python=3.12&flags=strict&gist=50f4ae811a650f9e90c6f52269f59d29 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I'm not immediately convinced this is robust!
mypy/checker.py Outdated
# We collect errors that indicate unreachability or redundant expressions | ||
# in the first iteration and remove them in subsequent iterations if the | ||
# related statement becomes reachable or non-redundant due to changes in | ||
# partial types: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
What if an unreachable statement is within another unreachable statement?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I am not exactly sure what you are asking for.
Is this an example that fits your question?
[casetestAvoidUnreachableCodeInUnreachableCodeWithPartialType]# flags: --warn-unreachable --python-version 3.11x=Noney=NonewhilexisNoneandyisNone:reveal_type(x)# N: Revealed type is "None"reveal_type(y)# N: Revealed type is "None"ifxisNoneandyisnotNone:x=1# E: Statement is unreachableifyisNone:y=1[builtinsfixtures/bool.pyi]
In this case, the results are the same for 1.15 and the current state of this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
something like this:
x=Noney=NonewhileTrue:ifxisnotNone:print("a")ifyisnotNone:print("b")# this is unreachablex=1
(I haven't run the PR on this so I'm not sure the result, but itshould warn about unreachablity. mypy 1.15 does. I'm worried this change won't)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This example should printb
. Is there a typo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I edited it afterwards, it shouldn't anymore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Yes, you were totally right. The large temporarily unreachable code section shadowed the persistent unreachable code section. I added your test case and extended the current logic a bit. It's late now, so I'll postpone double-checking until tomorrow.
Can you see any more potential pitfalls?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I again looked at it. To me, the logic appears a little special, but clear and straightforward. Of course, I cannot exclude that there are more special cases my solution does not cover. So I wait for the next review.
What kind of test case could convince you? |
Also BTW I notice your "restart CI" commit -- you could use |
I didn't know. That's helpful, thanks! |
…UnreachableLines` provided by@A5rocks and extend the current logic to fix it.
This comment has been minimized.
This comment has been minimized.
A5rocks left a comment• edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
for candidate in set(itertools.chain(*uselessness_errors)): | ||
if all( | ||
(candidate in errors) or (candidate[2] in lines) | ||
for errors, lines in zip(uselessness_errors, unreachable_lines) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Why can't this just use the last round of unreachability warnings?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
That is a good question. I initially implemented it so in229e4a6. However, in the last function definition of test casetestNewRedefineTryStatement
, the first response toreveal_type
does provide complete information, but the second (and final) response does not:
deff4()->None:whileint():try:x=1ifint():x=""breakifint():whileint():ifint():x=Nonebreakfinally:reveal_type(x)# N: Revealed type is "Union[builtins.int, builtins.str, None]" \# N: Revealed type is "Union[builtins.int, None]"
I have no experience with the new--allow-redefinition-new
option so far, and have not tried to investigate why this happens. I just extended my approach inc24c950.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Wouldn't that case also allow something not allowed forstr
? Eg does that error for:
ifxisnotNone:print(x+5)
Above thereveal_type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Oh, after re-reading, it seems that I have not hit the point of your question. Another try: There might be some scope for optimisations. However, I am unsure what can possibly happen when running Mypy with different configurations (like in thef4
example discussed above), so I preferred to implement this part of the approach as robustly as possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
On the other hand it's possible to be too robust: mypy only prints the errors for the last round right? So it's possible (through some bug elsewhere, like forgetting to propagate line numbers maybe?) to have unreachable code in the last run that isn't marked -- and so have no signal that the code there isn't checked.
On the other other hand, just using the last iteration's unreachability errors might be wrong in presence of bugs like inf4
, but it's simpler (and implementation is trivial to get right) and could never mislead the user about what code could get errors.
Basically I think it's better to match the semantics of how error reporting happens (ie during the last round of checking) then to have only one type of error be correct in face of bugs.
Edit: actually I might be wrong about my assumptions here, let me play with your PR a bit and I'll update this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
My bad; the-warn-unreachable
flag was missing in the test case. Now I get bothStatement is unreachable
andRevealed type is "builtins.str"
. However, the same happens when I check the modified test with Mypy-Master. Therefore, I don't know if this actually points to a flaw in my current approach. I will investigate tomorrow....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
...I had no time for a detailed investigation yet, but I tend to think the problem with the currently discussed test case has nothing to do with this PR, because the response toreveal_type
and theunreachable
error for the same line happen at the same iteration step.
Maybe the first hint on the cause for this discrepancy:expr.node.type = Union[builtins.int, builtins.str, None]
butTypeChecker.lookup_type(expr) = Union[builtins.int, None]
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@JukkaL do you think types getting narrower between iterations (see previous comments) should be allowed?
If not, then could this PR simply suppressall output and only print out the last iteration?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Maybe I misunderstand, but that sounds like reverting the "fix part" of this PR.
This is the current behaviour with the proposed changes to Mypy:
y=NonewhileyisNone:reveal_type(y)# NoneifyisNone:y= []y.append(1)reveal_type(y)# list[int]
The responses toreveal_type
are the result of the first iteration step. In the second iteration step, the whole loop's body is unreachable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think@tyralla is right, and just printing out the last iteration is not the right thing to do. I'll still need to experiment with this however.
test case from @auntsaninja (slightly simplified)
According tomypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
unreachable
andredundant-expr
warnings in loops more robustly and efficiently, and avoid multiplerevealed type
notes for the same line.unreachable
,redundant-expr
, andredundant-casts
warnings in loops more robustly and efficiently, and avoid multiplerevealed type
notes for the same line.I updated this PR to also fix#19170. (Or, at least the test case provided by Shantanu.) I have edited the PR's title and initial description accordingly. |
Thanks for working on the performance regression! It would be great to measure the impact using the |
I did not work with |
I just executed The results:
So, according to this measurement, this PR achieves a speed increase of 7.4 %, while#18433 resulted in a slowdown of 8.7 %. I am not familiar with the involved tools, but this intuitively makes sense to me and aligns roughly with my initial estimate using |
This looks good. The exact numbers depend on the specifics of hardware, Python version, C compiler, etc. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The seems good to me, and at least a significant improvement over the old behavior (plus it's faster!).
@A5rocks Do you still have further concerns?
A5rocks commentedJun 5, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
I still think this is a bug:#19118 (comment) But it's probably a bug with some other things rather than specifically the logic in this PR (mainly because the logic looks right to me). (though, what's the point of more robust logic if it cannot prevent that) |
I appreciate your carefulness. Still, I rethought the problem and am now very sure it is beyond the scope of this PR. This PR only tries to correctly aggregate information that Mypy collects stepwise when analysing loops. The bug you discovered, however, can be triggered without loops: # Mypy 1.16.0# --allow-redefinition-new --local-partial-types --warn-unreachabledeff()->None:try:x=1ifint():x=""returnifint():x=Nonereturnfinally:reveal_type(x)# N: Revealed type is "Union[builtins.int, builtins.str, None]" \# N: Revealed type is "builtins.int"ifisinstance(x,str):# E: Subclass of "int" and "str" cannot exist: would have incompatible method signaturesreveal_type(x)# E: Statement is unreachable \# E: Revealed type is "builtins.str"reveal_type(x)# N: Revealed type is "Union[builtins.int, builtins.str, None]" \# N: Revealed type is "builtins.int" This suggests that Lines 4954 to 4985 inb025bda
Would you like to open a separate issue? I think I could work on it next week. |
55c4067
intopython:masterUh oh!
There was an error while loading.Please reload this page.
…ally clauses. (#19270)Fixes#19269This PR refactors the logic implemented in#19118 (which only targetedrepeatedly checked loops) and applies it to repeatedly checked finallyclauses.I moved nearly all relevant code to the class `LoopErrorWatcher`, whichnow has the more general name `IterationErrorWatcher`, to avoid codeduplication. However, one duplication is left, which concerns errorreporting. It would be nice and easy to move this functionality to`IterationErrorWatcher`, too, but this would result in import cycles,and I am unsure if working with `TYPE_CHECKING` and postponed importingis acceptable in such cases (both for Mypy and Mypyc).After the refactoring, it should not be much effort to apply the logicto other cases where code sections are analysed iteratively. However,the only thing that comes to my mind is the repeated checking offunctions with arguments that contain constrained type variables. I willcheck it. If anyone finds a similar case and the solution is as simpleas expected, we could add the fix to this PR, of course.
Uh oh!
There was an error while loading.Please reload this page.
Fixes#18606
Closes#18511
Improves#18991
Fixes#19170
This change is an improvement over9685171. Besides fixing the regressions reported in#18606 and#19170 and removing the duplicates reported in#18511, it should significantly reduce the performance regression reported in#18991. At least running
Measure-command {python runtests.py self}
on my computer (with removed cache) is 10 % faster.