Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork34.2k
Description
Description
While working on our C extension, I noticed what I believe is a bug/crash in the list implementation when using_PyList_AsTupleAndClear (which creates a list from a tuple then clears the list), sort of livestd::move-ing a Python list into a tuple.
Long story short, the function setsself->ob_item = NULL andPy_SET_SIZE(self, 0) ("clear the list and disown the memory") but does NOT resetself->allocated to 0.
After this, the list "believes" it has some memory allocated but it actually has none.
If code then tries to append something to the list (throughPyList_Append),
it fails because it directly writes toself->ob_item[len] -- which is null.
Currently, it seems like there's only one caller to_PyList_AsTupleAndClear, and it discards the list right after gettings its tuple, so the bug has been flying under the radar I believe...
Testing
Reproducer
#ifndefPy_BUILD_CORE_MODULE# definePy_BUILD_CORE_MODULE#endif#include"Python.h"#include"pycore_list.h"staticPyObject*test(PyObject*module,PyObject*ignored){PyObject*list=PyList_New(0);if (list==NULL)returnNULL;/* Add items to get allocated > 0 */for (inti=0;i<10;i++) {PyObject*val=PyLong_FromLong(i);if (val==NULL) {Py_DECREF(list);returnNULL; }if (PyList_Append(list,val)<0) {Py_DECREF(val);Py_DECREF(list);returnNULL; }Py_DECREF(val); }PyListObject*lst= (PyListObject* )list;PyObject*tup=_PyList_AsTupleAndClear(lst);if (tup==NULL) {Py_DECREF(list);returnNULL; }if (lst->ob_item==NULL&&lst->allocated!=0) {printf("Attempting append...\n");PyObject*newitem=PyLong_FromLong(123123123);if (newitem==NULL) {Py_DECREF(tup);Py_DECREF(list);returnNULL; }if (PyList_Append((PyObject*)lst,newitem)<0) {Py_DECREF(newitem);Py_DECREF(tup);Py_DECREF(list);returnNULL; }Py_DECREF(newitem); }Py_DECREF(tup);Py_DECREF(list);Py_RETURN_NONE;}staticPyMethodDefmethods[]= { {"test",test,METH_NOARGS,""}, {NULL,NULL,0,NULL}};staticstructPyModuleDefmodule= {PyModuleDef_HEAD_INIT,"ext","ext",-1,methods};PyMODINIT_FUNCPyInit_ext(void){returnPyModule_Create(&module);}
fromsetuptoolsimportsetup,Extensionimportoscpython_root=os.path.dirname(os.path.dirname(os.path.abspath(__file__)))ext=Extension("ext",sources=["ext.c"],include_dirs=[os.path.join(cpython_root,"Include"),os.path.join(cpython_root,"Include","internal"),cpython_root, ],define_macros=[("Py_BUILD_CORE_MODULE","1")],)setup(name="ext",ext_modules=[ext],)
To build the reproducer (from a subdir):
../python.exe setup.py build_ext --inplace
And to trigger it:
../python.exe -c"import ext; ext.test()"gives (I have a TSan build but it does give me what I expect...)
<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'ext', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.Attempting append...ThreadSanitizer:DEADLYSIGNAL==35116==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000101639e50 bp 0x00016f72d570 sp 0x00016f72d530 T25592900)==35116==The signal is caused by a WRITE memory access.==35116==Hint: address points to the zero page.^Zzsh: suspended ../python.exe -c "import ext; ext.test()"Next steps
I have a branch/PR ready with a fix for that:#145680.