Trying to build numpy-1.11.2rc1 wheels for Python 3.6.0b1 on Windows 10 with `python.exe setup.py bdist_wheel`, python36.dll (32 and 64 bit) segfaults in the `find_maxchar_surrogates` function. Some other packages built OK, e.g. Cython, pyzmq and Pillow.The crash is at <https://hg.python.org/cpython/file/v3.6.0b1/Objects/unicodeobject.c#l1607>. The call stack is attached.Locals:+begin0x00454c49464f5250 <Error reading characters of string.>const wchar_t *ch1509860640unsigned int+end0x004550fc90512200 <Error reading characters of string.>const wchar_t *+iter0x00000259a500e7d8 L"\x2"const wchar_t *+maxchar0x0000005a59fea520 {0}unsigned int *+num_surrogates0x0000005a59fea528 {0}__int64 *

end-begin = 0x004550fc90512200-0x00454c49464f5250 = 5168087289776Seems find_maxchar_surrogates() is called with wrong arguments.

> Trying to build numpy-1.11.2rc1 wheels for Python 3.6.0b1 on Windows 10 with `python.exe setup.py bdist_wheel`, python36.dll (32 and 64 bit) segfaults in the `find_maxchar_surrogates` function.I ran "python setup.py bdist_wheel" on numpy-1.11.2rc1 on Linux: no issue.I compiled Python 3.6 (the default branch in fact, so the future 3.7) in debug mode/64 bit with SSL, but the setup.py command failed with:   Don't know how to compile Fortran code on platform 'nt'I don't want to spend too much time to try to reproduce the issue.Can you please try to get the Python traceback? Try:   python -X faulthandler setup.py bdist_wheelIt seems like the crash occurs in parse_envlist(), function to parse environment variables in os.spawnve(). Can you try to dump the env parameter of os.spawnve()?For example, you can put the following lines at the beginning of setup.py:---import osorig_spawnve = os.spawnvedef log_spawnve(*args):    print("spawnve: %r" % (args,))    return orig_spawnve(*args)os.spawnve = log_spawnve---My test:--->>> import os, sys; args=[sys.executable, "-c", "pass"]; os.spawnve(os.P_WAIT, args[0], args, None)spawnve: (0, '/home/haypo/prog/python/default/python', ['/home/haypo/prog/python/default/python', '-c', 'pass'], None)127---

parse_envlist is calling PyUnicode_FromFormat with the format "%U=%U", but passing it bytes:    |Debug Error!    Program: C:\Program Files\Python36\python_d.exe    abort() has been called    (Press Retry to debug the application)    (1e8.374): Break instruction exception - code 80000003 (first chance)    ucrtbased!issue_debug_notification+0x45:    00007ffc`aa1f1985 cc              int     3    1:002> k 0n12    Child-SP          RetAddr           Call Site    00000030`a5fe9bb0 00007ffc`aa1f1b23 ucrtbased!issue_debug_notification+0x45    00000030`a5fe9c00 00007ffc`aa212d7d ucrtbased!__acrt_report_runtime_error+0x13    00000030`a5fe9c60 00007ffc`aa2184cf ucrtbased!abort+0x1d    00000030`a5fe9ca0 00007ffc`aa2164c8 ucrtbased!common_assert_to_stderr<wchar_t>+0xbf    00000030`a5fe9d00 00007ffc`aa218e7f ucrtbased!common_assert<wchar_t>+0x68    00000030`a5fe9d40 00000000`60e59045 ucrtbased!_wassert+0x2f    00000030`a5fe9d70 00000000`60eb9f71 python36_d!_PyUnicode_CheckConsistency+0x45    00000030`a5fe9e00 00000000`60e4cd07 python36_d!unicode_fromformat_arg+0xc81    00000030`a5fe9fa0 00000000`60e4cc31 python36_d!PyUnicode_FromFormatV+0xa7    00000030`a5fea040 00000000`60c69853 python36_d!PyUnicode_FromFormat+0x31    00000030`a5fea080 00000000`60c68286 python36_d!parse_envlist+0x1c3    00000030`a5fea180 00000000`60c5d4ee python36_d!os_spawnve_impl+0x1d6    1:002> .frame 6    06 00000030`a5fe9d70 00000000`60eb9f71 python36_d!_PyUnicode_CheckConsistency+0x45    1:002> ?? op->ob_type->tp_name    char * 0x00000000`60fe1248     "bytes"parse_envlist used to call PyUnicode_FSConverter on key and val; get the underlying buffers via PyBytes_AsString; allocate a new buffer and copy key and val into "%s=%s". Now it instead calls PyUnicode_FromFormat for "%U=%U" and then fsconvert_strdup, which has been modified to call PyUnicode_AsWideCharString on Windows and otherwise PyUnicode_FSConverter. For Windows it could use PyUnicode_FSDecoder on key and val in combination with PyUnicode_FromFormat and the format string "%U=%U". For Unix it could use PyUnicode_FSConverter for key and val in combination with PyBytes_FromFormat and the format string "%s=%s".

> parse_envlist is calling PyUnicode_FromFormat with the format "%U=%U", but passing it bytes:Ah, it's a regression introduced by the issue#27781 with the changee20c7d8a8187: parse_envlist() was modified to first call PyUnicode_FromFormat("%U=%U") and then convert.

Here's a snippet to reproduce this bug:    import os, sys    environb = {os.fsencode(k):os.fsencode(v) for k,v in os.environ.items()}    os.spawnve(os.P_WAIT, sys.executable, ('python', '--version'), environb)(Now that Windows Python provisionally supports bytes via utf-8:surrogatepass, maybe it should also have an os.environb mapping.)

Whoops, yeah that's my fault. I'll get to it before b2, but I need to catch up on my day job the next week or two, so if someone else wants to apply a fix (and add a test - test_crashers, presumably?) then go for it!

Here's a patch for Unix. I will add Windows support when I get my Windows VM.> [...] (and add a test - test_crashers, presumably?) [...]Unfortunately, test_crashers doesn't run since 2011 (skipped in481ad9129a0f.) parse_envlist() is only used by os.execve() and os.spawnve() so I'm not sure what's the best way to test the patch (other than adapting Eryk's snippet inmsg276241.)

The issue is not specific to Windows, the following example also crash on Linux:import os, sysargs = [sys.executable, "-c", "pass"]os.execve(args[0], args, os.environb)

Berker, this is basically what I had in my initial patch on the Unix side. I also addressed the Windows issues in parse_envlist and fsconvert_strdup. I'm uploading that patch for reference. It needs a test. I also need to verify that there are no additional problems with the high-level os._execvpe function regarding bytes support on Windows.

New changeset0ca42273c714 by Victor Stinner in branch '3.6':Issue#28114: Add unit tests on os.spawn*()https://hg.python.org/cpython/rev/0ca42273c714

Berker and/or Eryksun: Please write unit tests for your patch. I just added a new SpawnTests to test_os which tests all os.spawn*() functions.Please add at least one unit test with env contains a bytes key/value entry. Maybe add a use_bytes=False parameter to create_args() to use bytes for the variable added by the test?

Thanks for the spawn test framework, Victor. I've added a use_bytes argument to encode the args and env using os.fsencode. It's encoding args as well because parse_arglist calls fsconvert_strdup, which was assuming Unicode strings on Windows instead of first calling PyUnicode_FSDecoder. test_spawnve_bytes passes for me on Linux and Windows.

Eryk's patch looks good to me, thanks! I will wait for others to review the patch.

New changeset8d80a4291ddc by Berker Peksag in branch '3.6':Issue#28114: Fix a crash in parse_envlist() when env contains byte stringshttps://hg.python.org/cpython/rev/8d80a4291ddcNew changeset8ad99fdc84a3 by Berker Peksag in branch 'default':Issue#28114: Merge from 3.6https://hg.python.org/cpython/rev/8ad99fdc84a3

Thanks for the patch, Eryk!

> #if defined(HAVE_WEXECV) || defined(HAVE_WSPAWNV)Hum, can't we use MS_WINDOWS here? Or maybe pass a parameter inparse_envlist() for Windows?

We already use HAVE_WEXECV and HAVE_WSPAWNV elsewhere to choose the wide-char versions of these functions. You've got a few changes to make if you want to maintain consistency.Personally I prefer feature flags to platform flags, then let pyconfig.h control them, especially when we're not directly using platform-specific APIs.

Ok fine ;-) Let's keep #if defined(HAVE_WEXECV) || defined(HAVE_WSPAWNV).