Introduction of acompression package

Both thezstd andzstandard import names are claimed by projects onPyPI. To avoid breaking users of one of the existing bindings, this PEPproposes introducing a new namespace for compression libraries,compression. This name is already reserved on PyPI for use in thestandard library. The new Zstandard module will be namedcompression.zstd.Other compression modules will be re-exported in the newcompression package.

Providing a common namespace for compression modules has several advantages.First, it reduces user confusion about where to find compression modules.Second, the top levelcompression module could provide information on whichcompression formats are available, similar tohashlib’salgorithms_available. IfPEP 775 is accepted, acompression.algorithms_guaranteed could be provided as well, listingzlib. Finally, acompression namespace prevents future issues withmerging other compression formats into the standard library. New compressionformats will likely be published to PyPI prior to integration intoCPython. Therefore, any new compression format import name will likely alreadybe claimed by the time a module would be considered for inclusion in CPython.Putting compression modules under a package prefix prevents issues withpotential future name clashes.

Code that would like to remain compatible across Python versions may use thefollowing pattern to ensure compatibility:

try:fromcompression.lzmaimportLZMAFileexceptImportError:fromlzmaimportLZMAFile

This will use the newer import name when available and fall back to the oldname otherwise.

Implementation based onpyzstd

The implementation for this PEP is based on thepyzstd project.This project was chosen as the code wasoriginally written to be upstreamedto CPython by Ma Lin, who also wrote theoutput buffer implementation used inthe standard library today.The project has since been taken over by Rogdham and is published to PyPI. TheAPIs inpyzstd are similar to the APIs for other compression modules in thestandard library such asbz2 andlzma.

Minimum supported Zstandard version

The minimum supported Zstandard was chosen as v1.4.5, released in May of 2020.This version was chosen as a minimum based on reviewing the versions ofZstandard available in a number of Linux distribution package repositories,including LTS releases. This version choice is rather conservative to maximizecompatibility with existing LTS Linux distributions, but a newer Zstandardversion could likely be chosen given that newer Python releases are generallypackaged as part of newer distribution releases.

Thecompression namespace

A new namespace for compression modules will be introduced namedcompression. The top-level module for this package will be empty to beginwith, but a standard API for interacting with compression routines may beadded in the future to the toplevel.

Thecompression.zstd module

A new module,compression.zstd will be introduced with Zstandardcompression APIs that match other compression modules in the standard library,namely

• compress() /decompress() - APIs for one-shot compressionor decompression
• ZstdFile /open() - APIs for interacting with streamsand file-like objects
• ZstdCompressor /ZstdDecompressor - APIs for incrementalcompression or decompression

It will also contain some Zstandard-specific functionality:

• ZstdDict /train_dict() /finalize_dict() - APIs forinteracting with Zstandard dictionaries, which are useful for compressingmany small chunks of similar data

libzstd optional dependency

Thelibzstd library will become an optional dependency of CPython. If thelibrary is not available, thecompression.zstd module will be unavailable.This is handled automatically on Unix platforms as part of the normal buildenvironment detection.

On Windows,libzstd will be added tothe source dependenciesused to build libraries CPython depends on for Windows.

Other compression modules

New import namescompression.lzma,compression.bz2,compression.gzip andcompression.zlib will be introduced in Python 3.14re-exporting the contents of the existinglzma,bz2,gzip andzlib modules respectively. Thecompression sub-modules will becomethe canonical import names going forward. The use of the new compression nameswill be promoted over the original top level module names in the Pythondocumentation when the minimum supported Python version requirements makethat feasible.

The_compression module, given that it is marked private, will beimmediately renamed tocompression._common._streams. The new name wasselected due to the current contents of the module being I/O related helpersfor stream APIs (e.g.LZMAFile) in standard library compression modules.

Name the modulezstdlib and do not make a newcompression namespace

One option instead of making a newcompression namespace would be to finda different name, such aszstdlib, as the import name. Several other names,such aszst,libzstd, andzstdcomp were proposed as well. Indiscussion, the names were found to either be too easy to typo, or unintuitive.Furthermore, the issue of existing import names is likely to persist for futurecompression formats added to the standard library. LZ4, a common high speedcompression format, hasa package on PyPI,lz4, with the import namelz4. Instead of solving this issue for eachcompression format, it is better to solve it once and for all by using thealready-claimedcompression namespace.

Introduce an experimental_zstd package in Python 3.14

Since this PEP was published close to the beta cutoff for new features forPython 3.14, one proposal was to name the package a private module_zstdso that packaging tools could use it sooner, but not deciding on a name. Thiswould allow more time for discussion of the final module name during the 3.15development window. However, introducing a private module was not popular. Theexpectations and contract for external usage of a private module in thestandard library are unclear.

Introduce a standard library namespace instead ofcompression

One alternative to acompression namespace would be to introduce astd namespace for the entire standard library. However, this was seen astoo significant a change for 3.14, with no agreed upon semantics, migrationpath, or name for the package. Furthermore, a future PEP introducing astdnamespace could always define that thecompression sub-modules be flattenedinto thestd namespace.

Includezipfile andtarfile incompression

Compression is often used with archiving tools, so putting bothzipfileandtarfile under thecompression namespace is appealing. However,compression can be used beyond just archiving tools. For example, networkrequests can be gzip compressed. Furthermore, formats like tar do not includecompression themselves, instead relying on external compression. Therefore,this PEP does not propose movingzipfile ortarfile undercompression.

Do not includegzip undercompression

TheGZip format RFC defines a format which can include multipleblocks and metadata about its contents. In this way GZip is rather similar toarchive formats like ZIP and tar. Despite that, in usage GZip is often treatedas a compression format rather than an archive format. Looking at how differentlanguages classify GZip, the prevailing trend is to classify it as acompression format and not an archiving format.

In addition, thegzip module in Python mostly focuses on single blockcontent and has an API similar to other compression modules, making it a goodfit for thecompression namespace.