Python 3.4.3 crashes after some time when running the attached program under Windows 7.The program appends a fixed bytes "string" to two independent bytearray buffers.If slices are removed from the beginnging of the two buffers and the two buffers are print:ed, the program will crash at some random occation.

Reproduced with top of trunk default branch on OS X (so, not Windows-specific).  With debug enabled:$ ./bin/python3.5 /tmp/b/bytearray_bug.pybuf2: 13 5 bytearray(b'1234567890123')buf1: 13 3 bytearray(b'1234567890123')buf2: 21 2 bytearray(b'678901231234567890123')buf1: 23 5 bytearray(b'45678901231234567890123')buf2: 32 28 bytearray(b'89012312345678901231234567890123')buf1: 31 21 bytearray(b'9012312345678901231234567890123')buf2: 17 10 bytearray(b'01231234567890123')buf1: 23 9 bytearray(b'45678901231234567890123')buf2: 20 3 bytearray(b'78901231234567890123')buf1: 27 14 bytearray(b'312345678901231234567890123')buf2: 30 22 bytearray(b'012312345678901231234567890123')buf1: 26 17 bytearray(b'12345678901231234567890123')buf2: 21 1 bytearray(b'678901231234567890123')buf1: 22 12 bytearray(b'5678901231234567890123')buf2: 33 13 bytearray(b'789012312345678901231234567890123')buf1: 23 4 bytearray(b'45678901231234567890123')buf2: 33 23 bytearray(b'789012312345678901231234567890123')buf1: 32 4 bytearray(b'89012312345678901231234567890123')buf2: 23 19 bytearray(b'45678901231234567890123')buf1: 41 36 bytearray(b'23123456789012312345678901231234567890123')buf2: 17 3 bytearray(b'01231234567890123')buf1: 18 3 bytearray(b'901231234567890123')buf2: 27 23 bytearray(b'312345678901231234567890123')buf1: 28 22 bytearray(b'2312345678901231234567890123')buf2: 17 2 bytearray(b'01231234567890123')buf1: 19 5 bytearray(b'8901231234567890123')buf2: 28 10 bytearray(b'2312345678901231234567890123')buf1: 27 2 bytearray(b'312345678901231234567890123')buf2: 31 7 bytearray(b'9012312345678901231234567890123')buf1: 38 2 bytearray(b'23456789012312345678901231234567890123')buf2: 37 23 bytearray(b'3456789012312345678901231234567890123')buf1: 49 2 bytearray(b'4567890123123456789012312345678901231234567890123')buf2: 27 21 bytearray(b'312345678901231234567890123')buf1: 60 1 bytearray(b'678901231234567890123123456789012312345678901231234567890123')Debug memory block at address p=0x10bd238e0: API 'o'    61 bytes originally requested    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.    The 8 pad bytes at tail=0x10bd2391d are not all FORBIDDENBYTE (0xfb):        at tail+0: 0x33 *** OUCH        at tail+1: 0x00 *** OUCH        at tail+2: 0xfb        at tail+3: 0xfb        at tail+4: 0xfb        at tail+5: 0xfb        at tail+6: 0xfb        at tail+7: 0xfb    The block was made by call#44842 to debug malloc/realloc.    Data at p: 34 35 36 37 38 39 30 31 ... 35 36 37 38 39 30 31 32Fatal Python error: bad trailing pad byteCurrent thread 0x00007fff70e18300 (most recent call first):  File "/tmp/b/bytearray_bug.py", line 25 in <module>Abort trap: 6

Above was in 64-bit mode (x86_64).  Same run in 32-bit (i386):$ ./bin/python3.5-32 /tmp/b/bytearray_bug.pybuf2: 13 2 bytearray(b'1234567890123')buf1: 13 11 bytearray(b'1234567890123')buf2: 24 16 bytearray(b'345678901231234567890123')buf1: 15 4 bytearray(b'231234567890123')buf2: 21 16 bytearray(b'678901231234567890123')buf1: 24 3 bytearray(b'345678901231234567890123')buf2: 18 1 bytearray(b'901231234567890123')buf1: 34 28 bytearray(b'6789012312345678901231234567890123')buf2: 30 21 bytearray(b'012312345678901231234567890123')buf1: 19 8 bytearray(b'8901231234567890123')buf2: 22 6 bytearray(b'5678901231234567890123')buf1: 24 11 bytearray(b'345678901231234567890123')buf2: 29 23 bytearray(b'12312345678901231234567890123')buf1: 26 15 bytearray(b'12345678901231234567890123')buf2: 19 15 bytearray(b'8901231234567890123')buf1: 24 4 bytearray(b'345678901231234567890123')buf2: 17 7 bytearray(b'01231234567890123')buf1: 33 21 bytearray(b'789012312345678901231234567890123')buf2: 23 13 bytearray(b'45678901231234567890123')buf1: 25 20 bytearray(b'2345678901231234567890123')buf2: 23 8 bytearray(b'45678901231234567890123')buf1: 18 7 bytearray(b'901231234567890123')buf2: 28 3 bytearray(b'2312345678901231234567890123')buf1: 24 8 bytearray(b'345678901231234567890123')buf2: 38 1 bytearray(b'23456789012312345678901231234567890123')buf1: 29 7 bytearray(b'12312345678901231234567890123')buf2: 50 38 bytearray(b'34567890123123456789012312345678901231234567890123')buf1: 35 7 bytearray(b'56789012312345678901231234567890123')buf2: 25 4 bytearray(b'2345678901231234567890123')buf1: 41 20 bytearray(b'23123456789012312345678901231234567890123')buf2: 34 24 bytearray(b'6789012312345678901231234567890123')buf1: 34 19 bytearray(b'6789012312345678901231234567890123')buf2: 23 1 bytearray(b'45678901231234567890123')buf1: 28 18 bytearray(b'2312345678901231234567890123')buf2: 35 20 bytearray(b'56789012312345678901231234567890123')buf1: 23 13 bytearray(b'45678901231234567890123')buf2: 28 16 bytearray(b'2312345678901231234567890123')buf1: 23 19 bytearray(b'45678901231234567890123')buf2: 25 16 bytearray(b'2345678901231234567890123')buf1: 17 4 bytearray(b'01231234567890123')buf2: 22 18 bytearray(b'5678901231234567890123')buf1: 26 18 bytearray(b'12345678901231234567890123')buf2: 17 14 bytearray(b'01231234567890123')buf1: 21 18 bytearray(b'678901231234567890123')buf2: 16 14 bytearray(b'1231234567890123')buf1: 16 11 bytearray(b'1231234567890123')buf2: 15 10 bytearray(b'231234567890123')buf1: 18 2 bytearray(b'901231234567890123')buf2: 18 2 bytearray(b'901231234567890123')buf1: 29 3 bytearray(b'12312345678901231234567890123')buf2: 29 11 bytearray(b'12312345678901231234567890123')buf1: 39 9 bytearray(b'123456789012312345678901231234567890123')buf2: 31 23 bytearray(b'9012312345678901231234567890123')buf1: 43 31 bytearray(b'0123123456789012312345678901231234567890123')Debug memory block at address p=0x7aec88: API 'o'    49 bytes originally requested    The 3 pad bytes at p-3 are FORBIDDENBYTE, as expected.    The 4 pad bytes at tail=0x7aecb9 are not all FORBIDDENBYTE (0xfb):        at tail+0: 0x31 *** OUCH        at tail+1: 0x32 *** OUCH        at tail+2: 0x33 *** OUCH        at tail+3: 0x00 *** OUCH    The block was made by call#44809 to debug malloc/realloc.    Data at p: 31 32 33 34 35 36 37 38 ... 33 34 35 36 37 38 39 30Fatal Python error: bad trailing pad byteCurrent thread 0xa0ddc1d4 (most recent call first):  File "/tmp/b/bytearray_bug.py", line 25 in <module>Abort trap: 6

Also happening with Python 3.4.0 on Ubuntu 14.04 (after ~ half a minute and A LOT of output):[skipping lots of lines]buf2: 29 13 bytearray(b'12312345678901231234567890123')buf1: 25 9 bytearray(b'2345678901231234567890123')buf2: 29 2 bytearray(b'12312345678901231234567890123')buf1: 29 5 bytearray(b'12312345678901231234567890123')buf2: 40 8 bytearray(b'3123456789012312345678901231234567890123')buf1: 37 10 bytearray(b'3456789012312345678901231234567890123')buf2: 45 31 bytearray(b'890123123456789012312345678901231234567890123')buf1: 40 7 bytearray(b'3123456789012312345678901231234567890123')buf2: 27 3 bytearray(b'312345678901231234567890123')buf1: 46 9 bytearray(b'7890123123456789012312345678901231234567890123')buf2: 37 6 bytearray(b'3456789012312345678901231234567890123')buf1: 50 15 bytearray(b'34567890123123456789012312345678901231234567890123')buf2: 44 5 bytearray(b'90123123456789012312345678901231234567890123')buf1: 48 27 bytearray(b'567890123123456789012312345678901231234567890123')buf2: 52 3 bytearray(b'1234567890123123456789012312345678901231234567890123')buf1: 34 1 bytearray(b'6789012312345678901231234567890123')buf2: 62 16 bytearray(b'45678901231234567890123123456789012312345678901231234567890123')buf1: 46 13 bytearray(b'7890123123456789012312345678901231234567890123')*** Error in `python3': realloc(): invalid pointer: 0x00007f2a45580000 ***

Surprisingly, a much simpler version with just one bytearray seems to run stably (for several minutes at least), but when you wait a while then hit Ctrl-C, you are getting a Segmentation fault:Python 3.4.0 (default, Apr 11 2014, 13:05:11) [GCC 4.8.2] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import random>>> buf1 = bytearray()>>> data = b"1234567890123">>> >>> while True:...     buf1 += data...     l = len(buf1)...     n = random.randrange(1, l-1)...     del buf1[:n]... ^CSegmentation fault (core dumped)The same code crashes spontaneously (without attempting a keyboard interrupt) when run in IDLE.

No problem with python 2.7.6 on Ubuntu 14.04.2 LTS amd64. Attached script was working about 2 hours - no crash.

Win7, 64 bit, 3.5.0a3: crash in about 10 seconds.

Here is a reduced version without using random numbers that reliably crashes for me on 64-bit Linux. Hopefully it also crashes for others, and they can investigate further. Crashes for me with the standard Arch Linux binary:Python 3.4.3 (default, Feb 26 2015, 23:01:07) [GCC 4.9.2 20150204 (prerelease)] on linuxand my recently built v3.5 version:Python 3.5.0a3+ (default:0b3027a2abbc, Apr 11 2015, 23:27:07) [GCC 4.9.1] on linuxIt does not crash on 32 bit computer I tried though.

After cleaning my build and rebuilding with “./configure --with-pymalloc --with-pydebug”, I reduced my script to these four lines:b1 = bytearray(b"abcdefghij")  # 10 bytesdel b1[:1]del b1[:1]b1 += b"klmnopq"  # 7 bytesPatch bytearray-fix.patch fixes the bug by taking account fact that ob_start is offset into the allocated memory buffer when checking if a reallocation is necessary.Explanation with the unpatched code and my four-line script above:1. First line allocates 11 bytes of memory (10 for the byte string + 1 NUL terminator)2. First “del” reduces the bytearray length to 9 bytes, but actually reallocates an expanded memory buffer of 16 bytes! (quirky but not the bug)3. Second “del” reduces the bytearray length to 8 bytes, and increments an internal ob_start offset without any memory copying or reallocation. (Fine.)4. Appending step needs to add 7 bytes to the 8-byte array. 7 + 8 + 1 is 16 bytes total required for bytearray and NUL terminator, but since ob_start is offset from the start of the allocated memory buffer, we overwrite past the end of the buffer.Memory debugging output and extra debug printfs of my own:Resizing to size 10 (current log. offset 0 alloc 0)=> Major upsize, new alloc = 11Assigning to linear slice- Shifting ob_start for negative growth -1Resizing to size 9 (current log. offset 1 alloc 11)=> Moderate upsize, new alloc = 16- Done assigning to linear sliceAssigning to linear slice- Shifting ob_start for negative growth -1Resizing to size 8 (current log. offset 1 alloc 16)=> Minor downsize- Done assigning to linear sliceDebug memory block at address p=0x7f1af630a0d0: API 'o'    16 bytes originally requested    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.    The 8 pad bytes at tail=0x7f1af630a0e0 are not all FORBIDDENBYTE (0xfb):        at tail+0: 0x00 *** OUCH        at tail+1: 0xfb        at tail+2: 0xfb        at tail+3: 0xfb        at tail+4: 0xfb        at tail+5: 0xfb        at tail+6: 0xfb        at tail+7: 0xfb    The block was made by call#32897 to debug malloc/realloc.    Data at p: 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71Fatal Python error: bad trailing pad byte

Posting bytearray-resize.patch which stops “del” from expanding the allocated buffer. This one is not necessary to fix the reported bug, but avoids the separate quirk identified in step 2.

This bug might have been caused by the changes forIssue 19087, so I left a note there. It looks like that issue added the ob_start field to bytearray() objects, so that deleting from the start does not require memory copying.

A test case for this that would trigger when memory debugging is enabled could look something like the following. Would it be appropriate to add it to the test suite?a = bytearray(10)size = sys.getsizeof(a)a.pop()  # Defeat expanding buffer off-by-one quirkself.assertEqual(sys.getsizeof(a), size, "Quirk not defeated")del a[:1]# Or a.pop(0)  # Does not trigger bug# Or a[:1] = ()  # Triggers bugself.assertEqual(sys.getsizeof(a), size, "Test assumes buffer not resized")a += bytes(2)  # Add exactly the number of free bytes in buffer# Or a.extend(bytes(2))  # Unaffected# Or a.append(0); a.append(0)  # Unaffected# Or a[8:] = bytes(2)  # Unaffecteddel a  # Trigger memory buffer to be freed, with verification

Thank you all for working really fast on this issue!I'm happy to see that a fix is already being tried out.

Antoine, would you have a chance to review my patches? I assume you were responsible for adding the ob_start field.It would be nice to see this bug fixed in the next 3.4 and 3.5 releases. As well as the original poster’s problem, I suspect this bug may be the cause of some occasional strange behaviour I have seen in my own bytearray() FIFO type code.

Sorry. I'll take a look!

Posting a new patch which combines both fixes and adds some test cases. However the test needs Python to be built with “./configure --with-pydebug” to detect the buffer overrun; without this the test will probably silently pass.I removed the offending buffer space check, and let it always call PyBufferArray_Resize(). I also looked around the bytearray module for similar errors for other operations but I couldn’t find any. The other cases already tend to always call PyByteArray_Resize().

New changeset98c1201d8eea by Antoine Pitrou in branch '3.4':Issue#23985: Fix a possible buffer overrun when deleting a slice from the front of a bytearray and then appending some other bytes data.https://hg.python.org/cpython/rev/98c1201d8eeaNew changeset06fab9093973 by Antoine Pitrou in branch 'default':Issue#23985: Fix a possible buffer overrun when deleting a slice from the front of a bytearray and then appending some other bytes data.https://hg.python.org/cpython/rev/06fab9093973

I've committed the patch. Thanks, Martin!

New changeset274c1b0a2494 by Serhiy Storchaka in branch '2.7':Issue#23985: Fixed integer overflow in iterator object.  Original patch byhttps://hg.python.org/cpython/rev/274c1b0a2494New changeset5b86a1abc8c3 by Serhiy Storchaka in branch '3.4':Issue#23985: Fixed integer overflow in iterator object.  Patch byhttps://hg.python.org/cpython/rev/5b86a1abc8c3New changeset9f2a1d9d7164 by Serhiy Storchaka in branch 'default':Issue#23985: Fixed integer overflow in iterator object.  Patch byhttps://hg.python.org/cpython/rev/9f2a1d9d7164

Sorry, it was related toissue22939.