Commit8deb4fe

pritamdamania

authored and

facebook-github-bot

committed

Fix flaky NCCL error handling tests. (#42149)

Summary:Pull Requestresolved:#42149Some of these tests were flaky since we could kill the process in someway without cleaning up the ProcessGroup. This resulted in issues where theFileStore didn't clean up appropriately resulting in other processes in thegroup to crash.Fixed this by explicitly deleting the process_group before we bring a processdown forcibly.ghstack-source-id: 108629057Test Plan: waitforbuildbotReviewed By: mrshenliDifferential Revision: D22785042fbshipit-source-id: c31d0f723badbc23b7258e322f75b57e0a1a42cf

1 parentb6a9f42 commit8deb4feCopy full SHA for 8deb4fe

File tree

1 file changed

-1

lines changed

test/distributed
- test_c10d.py

1 file changed

-1

lines changed

`‎test/distributed/test_c10d.py‎`

Lines changed: 3 additions & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -3413,6 +3413,8 @@ def _test_nccl_errors_blocking(self, func):`
`3413`	`3413`	`# aborting nccl communicators before throwing Operation timed out`
`3414`	`3414`	`a=torch.rand(10).cuda(self.rank)`
`3415`	`3415`	`elifself.rank==1:`
	`3416`	`+# Clean up structures (ex: files for FileStore before going down)`
	`3417`	`+delprocess_group`
`3416`	`3418`	`func()`
`3417`	`3419`	`else:`
`3418`	`3420`	`# Wait for timeout`
`@@ -3494,7 +3496,7 @@ def _wait_for_comm_abort(self, process_group):`
`3494`	`3496`	`return`
`3495`	`3497`	`else:`
`3496`	`3498`	`raisee`
`3497`		`-time.sleep(1)`
	`3499`	`+time.sleep(0.1)`
`3498`	`3500`
`3499`	`3501`	`@requires_nccl()`
`3500`	`3502`	`@skip_if_lt_x_gpu(3)`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Commit8deb4fe

File tree

1 file changed

1 file changed

`‎test/distributed/test_c10d.py‎`

0 commit comments