Dear all,

Following the bug#414, here is a patch which makes the fork + exec operation non blocking on Unix.
The patch is probably not ready to land, but I believe it's in pretty good shape.

This patch require a few changes in cpython (subprocess.Popen), which are included in this patch for discussion (intmp_subprocess, as a child class ofsubprocess.Popen). Those changes have been tested with cpython unittests on my laptop. The whole patch has been tested with Python 3.5.2 on Linux, so the CI may report undetected errors for other configurations.

The problem:
subprocess.Popen() will fork and synchronously read on a pipe until it is closed as a way to wait and check the result of the exec() call in the new child process. All of this is done in the constructor ofPopen.
This is an issue when thepreexec_fn (user function called before exec) orexec() is slow.

This PR does the following:

• breaks the privatePopen._execute_child() method into several methods so we can isolate the blocking reads (on the fderrpipe_read) in a (temporary) child classtmp_subprocess._Popen.
• introduces_NonBlockingPopen, a version of_Popen which uses the loop and a future to signal the caller when the exec() is done (or failed). It means that an error will usually be reported by the future instead of raised when the_NonBlockingPopen object is created. It also means that to follow the (implicit) contract ofPopen,_NonBlockingPopen is responsible of cleaning resources it opened if an error occurs before exec(). Note that_NonBlockingPopen also uses sockets instead of pipes (following what was done in_UnixSubprocessTransport._start()).
• since it is now required to wait on a future (but only on unix so far),_BaseSubprocessTransport._start() can be a coroutine. The implementation of the transport has been modified to support this change: the fork is now done in a task scheduled by the constructor. This doesn't change much for the caller (_make_subprocess_transport()) which awaited a future reporting errors during the transport initialization or is done once the subprocess transport is in a stable condition (_connect_pipes() finished). Special care has been taken to possible races conditions due to cancellations (usually whenloop.subprocess_exec/shell() are canceled).
• loop._make_subprocess_transport() doesn't change much. The biggest change is that the child_watcher is handled by the transport itself. This is required because we don't want this method to handle the process termination if something failed before the subprocess exec-ed, while now_NonBlockingPopen still have to handle this asynchronously.
• Two tests have been added: one handling a new possible race condition (cancellation before exec), and the other one ensuring that_NonBlockingPopen will eventually raise an exception if something wrong happens inpreexec_fn (this was previously the responsibility ofPopen).
• One test have been modified (test_proc_exited): it assumed that the transport was ready before the waiter was done. I believe this assumption is incorrect, and doesn't hold true now that the _BaseSubprocessTransport._proc attribute is set asynchronously (instead of in the constructor).

This is my biggest PR for asyncio so far, and my first attempt at patching cpython, I'm eager to receive feedback and comments about the patch and about how it can successfully land in the codebase (for instance, should I get in touch with the owner of cpython's subprocess?).

Thanks for your time,
Martin