This fixes the most pressing bug on#7295 but leaves thehaCoordinator mainly as is, meaning that we're still subject to various distributed races leaving us in a bad state where clients can't connect.

A full fix forhaCoordinator is quite a large change, so I plan to write an RFC for it.

The main idea of this fix is that we establish an output queue of updates on each connection. This allows us to queue up the initial node update while holding the lock, ensuring we don't miss any updates coming from the other side just as we are connecting. The queue is buffered, and actual I/O on the connection is done on a separate goroutine, where it will never contend for the lock.

There was also a lot of complex manipulation of locking and unlocking the mutex in the existing routines, so I refactored thecoordinator to contain a mutex-protected memory "core". All operations on thiscore are via methods that

c.mutex.Lock()defer c.mutex.Unlock()

so it should be much easier to follow what things are holding the lock and what things aren't (core operations vs the rest of thecoordinator methods).