Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
Description
Bug report
Bug description:
Support for inlining list/dict/set comprehensions inc3b595e introduced aCO_FAST_HIDDEN, which is applied in combination with a different type code, for exampleCO_FAST_LOCAL. However, when thecode object is copied viacode.replace() function call, this additional flag is lost; consequently, execution of the returned code object results in a bizarre-looking error.
Example:
Consider the following example program
# program.pyimportsysiflen(sys.argv)!=2:print(f"usage:{sys.argv[0]} <dir|locals|globals>")sys.exit(1)mode=sys.argv[1]# The comprehension must use same variable name as the code that attempts `del`._allvalues=''.join([myobjformyobjin ['a','b','c']])myobj=None# for del belowifmode=='dir':print("DIR():",dir())elifmode=='locals':print("LOCALS():",locals())elifmode=='globals':print("GLOBALS():",globals())delmyobj
and the following script that compiles the program to byte-code .pyc:
# compile_script.pyimportsysimportosimportstructimportmarshalimportimportlib.utiliflen(sys.argv)<3:print(f"usage:{sys.argv[0]} <source> <dest> [0|1]")sys.exit(1)filename=sys.argv[1]out_filename=sys.argv[2]strip_co=Falseiflen(sys.argv)<4elsesys.argv[3]!='0'withopen(filename,'rb')asfp:src=fp.read()co=compile(src,filename,'exec')ifstrip_co:co=co.replace()# In real use-case, we would be replacing filename herewithopen(out_filename,'wb')asfp:fp.write(importlib.util.MAGIC_NUMBER)fp.write(struct.pack('<I',0b01))# PEP-552: hash-based pyc, check_source=Falsefp.write(b'\00'*8)# Zero the source hashmarshal.dump(co,fp)
For some context, the above example is a distilled reproduction of what is going inPyInstaller andscipy.stats._distn_infrastructure module inpyinstaller/pyinstaller#7992: the collected module is byte-compiled, and the absolute filename in the code-object is anonymized into environment-relative path viaco.replace() (seehere for details).
But in the above example, no replacement is done, and so one would expect ofco.replace() to return an identical code object.
However, this is not the case (even thoughco == co.replace() in python claims that they are identical):
$ python3.12 compile_script.py program.py compiled-orig.pyc 0# Compile without co.replace()$ python3.12 compile_script.py program.py compiled-copy.pyc 1# Compile with co.replace()
$ sha256sum*.pyc2e03af03bcbb41b3a6cc6f592f5143acf7d82edc089913504c1f8446764795e1 compiled-copy.pyc5034955819efba0dc7ff3ee94101c1f6dfe33b102d547efc77577d77a99f1732 compiled-orig.pycRunning the original version:
$ python3.12 compiled-orig.pyc globalsGLOBALS(): {'__name__':'__main__','__doc__': None,'__package__': None,'__loader__':<_frozen_importlib_external.SourcelessFileLoader object at 0x7fe7fb327830>,'__spec__': None,'__annotations__': {},'__builtins__':<module'builtins' (built-in)>,'__file__':'[...]/compiled-orig.pyc','__cached__': None,'sys':<module'sys' (built-in)>,'mode':'globals','_allvalues':'abc','myobj': None}$ python3.12 compiled-orig.pyc dirDIR(): ['__annotations__','__builtins__','__cached__','__doc__','__file__','__loader__','__name__','__package__','__spec__','_allvalues','mode','myobj','sys']$ python3.12 compiled-orig.pyc localsLOCALS(): {'__name__':'__main__','__doc__': None,'__package__': None,'__loader__':<_frozen_importlib_external.SourcelessFileLoader object at 0x7f2846527830>,'__spec__': None,'__annotations__': {},'__builtins__':<module'builtins' (built-in)>,'__file__':'[...]/compiled-orig.pyc','__cached__': None,'sys':<module'sys' (built-in)>,'mode':'locals','_allvalues':'abc','myobj': None}
Running the version withco.replace():
$ python3.12 compiled-copy.pyc globalsGLOBALS(): {'__name__':'__main__','__doc__': None,'__package__': None,'__loader__':<_frozen_importlib_external.SourcelessFileLoader object at 0x7fd7f1b27830>,'__spec__': None,'__annotations__': {},'__builtins__':<module'builtins' (built-in)>,'__file__':'[...]/compiled-copy.pyc','__cached__': None,'sys':<module'sys' (built-in)>,'mode':'globals','_allvalues':'abc','myobj': None}$ python3.12 compiled-copy.pyc dirDIR(): ['__annotations__','__builtins__','__cached__','__doc__','__file__','__loader__','__name__','__package__','__spec__','_allvalues','mode','sys']Traceback (most recent call last): File"program.py", line 20,in<module> del myobj ^^^^^NameError: name'myobj' is not defined$ python3.12 compiled-copy.pyc localsLOCALS(): {'__name__':'__main__','__doc__': None,'__package__': None,'__loader__':<_frozen_importlib_external.SourcelessFileLoader object at 0x7f8a35d27830>,'__spec__': None,'__annotations__': {},'__builtins__':<module'builtins' (built-in)>,'__file__':'[...]/compiled-copy.pyc','__cached__': None,'sys':<module'sys' (built-in)>,'mode':'locals','_allvalues':'abc'}Traceback (most recent call last): File"program.py", line 20,in<module> del myobj ^^^^^NameError: name'myobj' is not defined
Comparing the compiled-orig.pyc and compiled-copy.pyc in a hex editor, there is one byte of difference; its position corresponds to marshaledco_localspluskinds, and the value is 0x30 (CO_FAST_LOCAL | CO_FAST_HIDDEN) in original and 0x20 (CO_FAST_LOCAL) in copy variant.
CPython versions tested on:
3.12
Operating systems tested on:
Linux, Windows