Bug report

When using Zipapp (and other related projects) to createstandalone Python applications my colleagues and I ran into trouble using any large but valid.zip archive.

Turns outLib/zipimport.py doesn't support ZIP64, and 'gets lost' when reading ZIP64 archives causing it to find 0 files in these archives and report that a__main__.py cannot be found.

Minimal reproduction

Create a Zip64 archive file using a simple__main__.py and random binary data to pad out the size.

"""Use the `zipapp` module to write a Zip64 archive to disk.(Alternatively the `zipfile` module can be used directly.)"""importosimportpathlibimporttempfileimportzipappimportzipfiledefmain()->int:num_dummy_files=10dummy_file_size=int((1.5*zipfile.ZIP64_LIMIT)//num_dummy_files)temp_dir=tempfile.TemporaryDirectory()foriinrange(num_dummy_files):withopen(pathlib.Path(temp_dir.name,f"{i}.bin"),"wb")asdummy_f:dummy_f.write(os.urandom(dummy_file_size))withopen(pathlib.Path(temp_dir.name,"__main__.py"),"w")asmain_f:main_f.write("print('Hello from the zipapp __main__py!')")zipapp.create_archive(temp_dir.name,"zip64_sized.pyz")temp_dir.cleanup()return0if__name__=="__main__":raiseSystemExit(main())

Attempt to execute the largezipapp.

python3.11 zip64_sized.pyz/workspaces/cpython/python: can't find'__main__' module in'/workspaces/cpython/zip64_sized.pyz'# or, using interpreters compiled from latest `main` (698fa8bf)./python zip64_size.pyz/usr/local/bin/python3.11: can't find'__main__' modulein'/workspaces/cpython/zip64_sized.pyz'

The__main__ module is of course present in the archive, which prompts head scratching until you did into the cPython source and ZIP file spec.

How to fix

Thezipapp module will happily produce Zip64 archives because the underlyingzipfile module has defaulted Zip64 support since Python 3.4.

The 'full' fix for this issue would be to refactorLib/zipimport.py to support Zip64 loading.

A first fix I think could be just providing a clearer error message whenLib/zipimport.py is given a Zip64 archive.

I'm happy to provide patches for each of these fixes in turn, if there's support for it. :)

Edit: Began attempting to raise an exception on Zip64 archives, but it seems on raising an exception withinzipimport.py the program doesn't exit and instead continues to start the interpreter:

Traceback (most recent call last):  File "<frozen zipimport>", line 91, in __init__ValueError: ZIP64 archives are unsupportedSyntaxError: Non-UTF-8 code starting with '\xff' in file /workspaces/cpython/zip64_sized.pyz on line 2, but no encoding declared; see https://peps.python.org/pep-0263/ for details

Your environment

- CPython versions tested on:

• Python 3.12.0a0 (heads/main:698fa8bf60, Aug 5 2022, 08:59:06) [GCC 9.4.0] on linux
• Python 3.11.0b5+ (heads/3.11:8570f6d1a0, Aug 2 2022, 07:52:11) [GCC 9.4.0] on linux
• Python 3.10.4 (main, Apr 1 2022, 20:52:12) [GCC 9.4.0] on linux

- Operating system and architecture:

uname -aLinux codespaces-5a1930 5.4.0-1086-azure#91~18.04.1-Ubuntu SMP Thu Jun 23 20:33:05 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Related:

• Python cannot run Subpars larger than 2GiB because they are Zip64 formatted. google/subpar#132