🔗 Helpful Links

🧪 See artifacts and rendered test results athud.pytorch.org/pr/154103

• 📄 PreviewPython docs built from this PR
• 📄 PreviewC++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit thebot commands wiki or ouroffice hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 Cancelled Jobs

As of commit1118893 with merge base975bbc6 ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

I think it's worth to think about what the error/behavior will look like if someone unfreezes their parameters with different dtypes after init time and then tries to run backward.

Also curious if this is an issue on FSDPv1? I think it might be

I think it's worth to think about what the error/behavior will look like if someone unfreezes their parameters with different dtypes after init time and then tries to run backward.

good catch! probably need to check requires_grad again inlazy_init

@weifengpy I was also suggesting to understand what kind of error gets raised. Unless you check every time (which I do not necessarily recommend), there is always a chance a user changes therequires_grad compared to when you last checked it. It might be good to know what kind of error naturally gets raised in that case. If it hangs, then that might imply you want to do something differently.

remove unrelated import

Done, I think it was unexpectedly introduced by my editor linter😂. Also fix existing test failures.

I've tested mixed dtype model backward locally, it will raise error due to:
https://github.com/pytorch/pytorch/blob/ec04186d3ca4ec9baca6dbdfa2acaf5eabf0f33d/torch/distributed/fsdp/_fully_shard/_fsdp_collectives.py#L375-L381

Edit: we may need to invoke the reduce op multiple times for grads with different dtypes.

So in this PR, do we intend to handle the situation where the frozen params being activated again afterfully_shard and running the backward pass?

@xuantengh personally I think it's fine if not in this PR

So in this PR, do we intend to handle the situation where the frozen params being activated again afterfully_shard and running the backward pass?

no need. As you mentioned, we have a clear runtime error instead of nccl hanging. it's already enough

Hi, any further steps to make this PR merged?

@pytorchmergebot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in thewiki.

Questions? Feedback? Please reach out to thePyTorch DevX Team

The mergejob was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information seepytorch-bot wiki.

Seems like one test times out.

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in thewiki.

Questions? Feedback? Please reach out to thePyTorch DevX Team

Merge failed

Reason: 1 jobs have failed, first few of them are:trunk / linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)

@pytorchmergebot -i

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: -iusage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

Try@pytorchbot --help for more info.

@pytorchmergebot -i

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: -iusage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

Try@pytorchbot --help for more info.

@pytorchmergebot merge -f "unrelated failure"

Merge started

Your change will be merged immediately since you used the force (-f) flag,bypassing any CI checks (ETA: 1-5 minutes). Please use-f as last resort and instead consider-i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in thewiki.

Questions? Feedback? Please reach out to thePyTorch DevX Team