It's quite amazing thatcommunicate() actually cannot be trusted or is simply not implemented correctly :/.

Well, you are quite right -communicate() CAN be trusted.

I did a small experiment of reading a Gbyte, both in PY27 and PY35, and they do not block:

>>>importsubprocessassb>>>proc=sb.Popen('dd if=/dev/zero bs=1024 count=1000000',bufsize=0,stdin=sb.PIPE,stdout=sb.PIPE,stderr=sb.PIPE)>>>a,b=proc.communicate()

I don't remember when I got this mistrust forcommunicate(), maybe when I had tried PY33, or maybe when epxerimenting with interactive processes (like thegit.cmd.Git.cat-file persistent command); there thecomminicate() is not suitable because it will start consuming streams till the death of the process, but the process is awaiting for more input from python-side before ending, so it's a deadlock.

One or two changes in mywindows fixes were under this wrong assumption thatcommunicate() should not be trusted. But mostly I replaced code accessing directlyproc.stderr/stdout from the main thread - this is definitely freeze-prone code.

A nice experiment ! It doesn't seem to output both on stdout and stderr though, which might be causing the problem we see. Theimplementation of communicate looks asynchronous, but who knows what's hidden in the details.

Even whenstdout/stderr is interplexed,communicate() does not block (at least in PY3):

>>>importsubprocessassb>>>proc=sb.Popen(['python','-c',"import sys\nfor i in range(1000000):\n    print('a'*1024); print('b'*1024, file=sys.stderr)"],stdin=sb.PIPE,stdout=sb.PIPE,stderr=sb.PIPE)>>>%timea,b=proc.communicate()Walltime:48.3s>>>len(a),len(b)(1026000000,1026000000)

For some "unicode" reason cannot run the above experiment in PY2.