NotificationsYou must be signed in to change notification settings
Fork32k
Star67.3k

bpo-40958: Avoid buffer overflow in the parser when indexing the current line#20842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

Closed

pablogsal wants to merge2 commits intopython:masterfrompablogsal:bpo-40958

Closed

bpo-40958: Avoid buffer overflow in the parser when indexing the current line#20842

pablogsal wants to merge2 commits intopython:masterfrompablogsal:bpo-40958

Conversation

Copy link

Member

pablogsal commentedJun 12, 2020•
edited by bedevere-bot
Loading

https://bugs.python.org/issue40958

pablogsal added2 commits

June 12, 2020 22:16

bpo-40958: Avoid buffer overflow in the parser when indexing the curr…

c48f25a

…ent line

Add NEWS entry

b854a0f

pablogsal requested a review fromlysnikolaou

June 12, 2020 21:18

the-knights-who-say-ni added the CLA signed label

Jun 12, 2020

bedevere-bot added the awaiting core review label

Jun 12, 2020

Copy link

MemberAuthor

pablogsal commentedJun 12, 2020•
edited
Loading

Before this PR

venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -m test test_eof0:00:00 load avg: 0.85 Run tests sequentially0:00:00 load avg: 0.85 [1/1] test_eoftest test_eof failed -- Traceback (most recent call last):  File "/home/pablogsal/github/python/master/Lib/test/test_eof.py", line 54, in test_line_continuation_EOF_from_file_bpo2180    self.assertIn(b'unexpected EOF while parsing', err)AssertionError: b'unexpected EOF while parsing' not found in b'Parser/tokenizer.c:978:50: runtime error: pointer index expression with base 0x625000016900 overflowed to 0xbebebebebebee6be\n=================================================================\n==48535==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606000027c51 at pc 0x55ba1fba57d4 bp 0x7fffc21fe430 sp 0x7fffc21fe420\nREAD of size 1 at 0x606000027c51 thread T0\n    #0 0x55ba1fba57d3 in ascii_decode Objects/unicodeobject.c:4941\n    #1 0x55ba1fca4f4a in unicode_decode_utf8 Objects/unicodeobject.c:4999\n    #2 0x55ba203fb1bd in byte_offset_to_character_offset Parser/pegen.c:148\n    #3 0x55ba203fb1bd in _PyPegen_raise_error_known_location Parser/pegen.c:412\n    #4 0x55ba203fbe4d in _PyPegen_raise_error Parser/pegen.c:373\n    #5 0x55ba203ff981 in tokenizer_error Parser/pegen.c:321\n    #6 0x55ba203ff981 in _PyPegen_fill_token Parser/pegen.c:638\n    #7 0x55ba2040277f in _PyPegen_expect_token Parser/pegen.c:753\n    #8 0x55ba2041317a in _tmp_15_rule Parser/parser.c:16184\n    #9 0x55ba204024f9 in _PyPegen_lookahead (/home/pablogsal/github/python/master/python+0x1c344f9)\n    #10 0x55ba20478e2c in compound_stmt_rule Parser/parser.c:1860\n    #11 0x55ba204815c2 in statement_rule Parser/parser.c:1224\n    #12 0x55ba204815c2 in _loop1_11_rule Parser/parser.c:15954\n    #13 0x55ba204815c2 in statements_rule Parser/parser.c:1183\n    #14 0x55ba204854b7 in file_rule Parser/parser.c:716\n    #15 0x55ba204854b7 in _PyPegen_parse Parser/parser.c:24401\n#16 0x55ba20405768 in _PyPegen_run_parser Parser/pegen.c:1077\n    #17 0x55ba204063ef in _PyPegen_run_parser_from_file_pointer Parser/pegen.c:1137\n    #18 0x55ba1feb53c6 in PyRun_FileExFlags Python/pythonrun.c:1057\n    #19 0x55ba1feb572c in PyRun_SimpleFileExFlags Python/pythonrun.c:400\n    #20 0x55ba1f8cbdbb in pymain_run_file Modules/main.c:369\n    #21 0x55ba1f8cbdbb in pymain_run_python Modules/main.c:553\n    #22 0x55ba1f8ce154 in Py_RunMain Modules/main.c:632\n    #23 0x55ba1f8ce154 in pymain_main Modules/main.c:662\n    #24 0x55ba1f8ce154 in Py_BytesMain Modules/main.c:686\n    #25 0x7fa4b6ba7001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)\n    #26 0x55ba1f8c848d in _start (/home/pablogsal/github/python/master/python+0x10fa48d)\n\n0x606000027c51is located 0 bytes to the right of 49-byte region [0x606000027c20,0x606000027c51)\nallocated by thread T0 here:\n    #0 0x7fa4b78db459 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cpp:145\n    #1 0x55ba1fbbaa1d in PyUnicode_New Objects/unicodeobject.c:1437\n    #2 0x55ba1fcee24b in _PyUnicode_Init Objects/unicodeobject.c:15535\n    #3 0x55ba1fe895c3in pycore_init_types Python/pylifecycle.c:599\n    #4 0x55ba1fe895c3 in pycore_interp_init Python/pylifecycle.c:724\n    #5 0x55ba1fe93b4b in pyinit_config Python/pylifecycle.c:765\n    #6 0x55ba1fe93b4b in pyinit_core Python/pylifecycle.c:926\n    #7 0x55ba1fe95b6c in Py_InitializeFromConfig Python/pylifecycle.c:1136\n    #8 0x55ba1f8c8752 in pymain_init Modules/main.c:66\n    #9 0x55ba1f8ce10a in pymain_main Modules/main.c:653\n    #10 0x55ba1f8ce10a in Py_BytesMain Modules/main.c:686\n    #11 0x7fa4b6ba7001 in __libc_start_main (/usr/lib/libc.so.6+0x27001)\n\nSUMMARY: AddressSanitizer: heap-buffer-overflow Objects/unicodeobject.c:4941 in ascii_decode\nShadow bytes around the buggy address:\n  0x0c0c7fffcf30: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00\n0x0c0c7fffcf40: 00 00 00 07 fa fa fa fa 00 00 00 00 00 00 00 00\n  0x0c0c7fffcf50: fa fa fa fa 00 00 00 00 00 00 00 05 fa fa fa fa\n  0x0c0c7fffcf60: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00\n  0x0c0c7fffcf70: 00 00 00 00 fa fa fa fa 00 00 00 00 00 00 00 01\n=>0x0c0c7fffcf80: fa fa fa fa 00 00 00 00 00 00[01]fa fa fa fa fa\n  0x0c0c7fffcf90: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00\n  0x0c0c7fffcfa0: 00 00 05 fa fa fa fa fa 00 00 00 00 00 00 00 fa\n  0x0c0c7fffcfb0: fa fa fa fa 00 00 00 00 00 00 00 00 fa fa fa fa\n  0x0c0c7fffcfc0: fd fd fd fd fd fd fd fd fafa fa fa fd fd fd fd\n  0x0c0c7fffcfd0: fd fd fd fd fa fa fa fa 00 00 00 00 00 00 00 fa\nShadow byte legend (one shadow byte represents 8 application bytes):\n  Addressable:           00\n  Partially addressable: 01 02 03 04 05 06 07 \n  Heap left redzone:       fa\n  Freed heap region:       fd\n  Stack left redzone:      f1\n  Stack mid redzone:       f2\n  Stack right redzone: f3\n  Stack after return:      f5\n  Stack use after scope:   f8\n  Global redzone:          f9\n  Global init order:       f6\n  Poisoned by user:        f7\n  Container overflow:      fc\nArray cookie:            ac\n  Intra object redzone:    bb\n  ASan internal:           fe\n  Left alloca redzone:     ca\n  Right alloca redzone:    cb\n  Shadow gap:              cc\n==48535==ABORTING\n'test_eof failed== Tests result: FAILURE ==1 test failed:    test_eofTotal duration: 355 msTests result: FAILURE~/github/python/master heads/master*venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c '"Ṕýţĥòñ" +'  File "<string>", line 1    "Ṕýţĥòñ" +              ^SyntaxError: invalid syntaxvenv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c 'yield from'  File "<string>", line 1    yield from              ^SyntaxError: invalid syntax

After this PR

venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -m test test_eof0:00:00 load avg: 1.13 Run tests sequentially0:00:00 load avg: 1.13 [1/1] test_eof== Tests result: SUCCESS ==1 test OK.Total duration: 493 msTests result: SUCCESS^[[A~/github/python/master [bpo-40958](https://bugs.python.org/issue40958)*venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c '"Ṕýţĥòñ" +'  File "<string>", line 1    "Ṕýţĥòñ" +              ^SyntaxError: invalid syntax~/github/python/master [bpo-40958](https://bugs.python.org/issue40958)*venv ❯ LSAN_OPTIONS="suppressions=asan-suppression.txt,print_suppressions=0" ./python -c 'yield from'  File "<string>", line 1    yield from              ^SyntaxError: invalid syntax

pablogsal self-assigned this

Jun 12, 2020

tiran requested changes

Jun 12, 2020

View reviewed changes

Copy link

Member

tiran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

A test is failing:

 ======================================================================FAIL: testSyntaxErrorOffset (test.test_exceptions.ExceptionTests) (source=b'Python = "\xcf\xb3\xf2\xee\xed" +', lineno=1, offset=18)----------------------------------------------------------------------Traceback (most recent call last):  File "D:\a\cpython\cpython\lib\test\test_exceptions.py", line 188, in check    self.assertEqual(cm.exception.offset, offset)AssertionError: 13 != 18