Investigating failures like this:https://github.com/coder/coder/runs/5043435263?check_suite_focus=true#step:7:32

where it looks like the job is completed, but the completion condition is never satisfied

Runs

• Run 1: peer failure
• Run 2: peer failurehttps://github.com/coder/coder/runs/5044293751?check_suite_focus=true#step:7:63
• Run 3: a different data race:https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:83

Issue 1: Data race inp.acquiredJobDone context (provisioners.d)

Failure trace:https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:84

There is a race in thep.acquiredJobDone chan - in particular, there can be a case where we're waiting on the channel to finish (in close) with<-p.acquiredJobDone, but in parallel, anacquireJob could've been started, which would create a new channel forp.acquiredJobDone.

The fix I tried was to also grab theacquiredJobMutex in theClose function. This, at first, caused a deadlock - because there was another case where the mutexes could be grabbed in reverse order (acquiredJobMutex -> thencloseMutex). That other place, though, was storing a bool in an atomic, so actually didn't need the mutex guard.

Attempted fix here:42ce721

Still hit a related race:https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:84

So tried a second fix here:84dd68a

The second fix didn't work, trying to switch from chan -> wait group:a8725cd

• Run 1: ✅
• Run 2: peer failure