Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
Closed
Description
Bug report
The functions zlib.crc32 and binascii.crc32 share the problematic behavior. When computing the CRC for data >= 2GB macOS arm64 binaries result in different values than all other platforms such as macOS x64, Windows x64, Linux x64. Consequently, problems arise e.g. when using the zipfile module.
A clear and concise description of what the bug is.
Reproduction:
importrandomrandom.seed(0)importzlibimportbinasciidefchunks(list,n):foriinrange(0,len(list),n):yieldlist[i:i+n]random_megabyte=random.randbytes(1024*1024)random_1_gigabyte=random_megabyte*1024*1random_4_gigabyte=random_megabyte*1024*4crc_1_gigabyte_zlib=zlib.crc32(random_1_gigabyte,0)crc_1_gigabyte_binascii=binascii.crc32(random_1_gigabyte,0)crc_4_gigabyte_zlib=zlib.crc32(random_4_gigabyte,0)crc_4_gigabyte_binascii=binascii.crc32(random_4_gigabyte,0)# incremental computation in chunks < 2 GB fixes macOS arm64chunked_crc_4_gigabyte_zlib=0chunked_crc_4_gigabyte_binascii=0forchunkinchunks(random_4_gigabyte,1024*1024*1024*1):chunked_crc_4_gigabyte_zlib=zlib.crc32(chunk,chunked_crc_4_gigabyte_zlib)chunked_crc_4_gigabyte_binascii=binascii.crc32(chunk,chunked_crc_4_gigabyte_binascii)print("crc_1_gigabyte_zlib".ljust(32),"expected: 0xe28bc234 computed:",hex(crc_1_gigabyte_zlib))print("crc_1_gigabyte_binascii".ljust(32),"expected: 0xe28bc234 computed:",hex(crc_1_gigabyte_binascii))print("crc_4_gigabyte_zlib".ljust(32),"expected: 0x278432d6 computed:",hex(crc_4_gigabyte_zlib))print("crc_4_gigabyte_binascii".ljust(32),"expected: 0x278432d6 computed:",hex(crc_4_gigabyte_binascii))print("chunked_crc_4_gigabyte_zlib".ljust(32),"expected: 0x278432d6 computed:",hex(chunked_crc_4_gigabyte_zlib))print("chunked_crc_4_gigabyte_binascii".ljust(32),"expected: 0x278432d6 computed:",hex(chunked_crc_4_gigabyte_binascii))
Output on macOS arm64:
mac-arm64:crc_bug dev_admin$ /opt/homebrew/bin/python3 crc_bug_report.py crc_1_gigabyte_zlib expected: 0xe28bc234 computed: 0xe28bc234crc_1_gigabyte_binascii expected: 0xe28bc234 computed: 0xe28bc234crc_4_gigabyte_zlib expected: 0x278432d6 computed: 0x6b54c6becrc_4_gigabyte_binascii expected: 0x278432d6 computed: 0x6b54c6bechunked_crc_4_gigabyte_zlib expected: 0x278432d6 computed: 0x278432d6chunked_crc_4_gigabyte_binascii expected: 0x278432d6 computed: 0x278432d6Your environment
- CPython versions tested on: Python 3.9.6 Python 3.11.4
- Operating system and architecture: macOS arm64, macOS x64, Windows x64, Linux x64
Linked PRs
- gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig #112615
- [3.12] gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (GH-112615) #112724
- [3.11] gh-105967: Work around a macOS bug, limit zlib C library crc32 API calls to 1gig (GH-112615) #112725
Metadata
Metadata
Assignees
Labels
Projects
Status
Done