Closescoder/internal#253.

The Tunnel closing mechanism encounters a race if the passedio.ReadWriteCloser is shared between the Tunnel and the Manager, specifically if the closing of the Conn by one causes a read fail on the other.

The sequence of events during the flakey tests is as follows:

1. Tunnel sends stop response
2. Tunnel closes sendch
3. net.Pipe gets closed by Tunnel'ssendLoop ending
4. Manager'sserdes reads the response successfully and passes it to the speaker via therecvCh.
5. The speaker tries to deliver the response via the message'srespCh. TherespCh is buffered so this is non-blocking.
6. TherecvLoopDone channel is closed by therecvCh getting closed (by failing to read from the closed conn)
7. unaryRPC is finally scheduled to run, where both of these conditions in a select block are ready, and so whether or not the response is returned is random.

[...]case <-s.recvLoopDone:logger.Debug(s.ctx, "recvLoopDone while waiting for response")return resp, io.ErrUnexpectedEOFcase resp = <-respCh:logger.Debug(s.ctx, "got response", slog.F("resp", resp))return resp, nil

The solution is to therefore make therespCh unbuffered. That way, it's not possible for therecvLoopDone channel to be closed until therespCh is read from.