All commit authors signed the Contributor License Agreement.

Most changes to Pythonrequire a NEWS entry. Add one using theblurb_it web app or theblurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply theskip news label instead.

Could you create some benchmarks for this? Also, please align the code to followPEP-7.

Could you create some benchmarks for this? Also, please align the code to followPEP-7.

@erlend-aasland

Aligned to PEP7.

Regarding the benchmarks, ran benchmark on Ubuntu 16 image (similar to Phoronixhttps://openbenchmarking.org/test/pts/compress-xz).

Attached results running on 12 core i7-10750H (2.60GHz), with 16 GiB RAM

"""./python xz_bench.py"""importjsonimportlzmaimporttime# Taken from:# https://old-releases.ubuntu.com/releases/16.04.3/ubuntu-16.04.3-server-i386.imgwithopen("ubuntu-16.04.3-server-i386.img","rb")asreader:data=reader.read()# First 20 MiBdata=data[: (20* (2<<20))]threads_list= [1,0,2,4]compression_results= {thread: {"times": [],"ratios": []}forthreadinthreads_list}forthreadinthreads_list:forpresetinrange(10):start_time=time.time()compressed_data=lzma.compress(data,preset=preset,threads=thread)end_time=time.time()compression_results[thread]["times"].append(end_time-start_time)ratio=len(data)/len(compressed_data)compression_results[thread]["ratios"].append(ratio)withopen("compression_results.json","w")asresult_file:json.dump(compression_results,result_file)

"""python3 -m pip install matplotlibpython3 plot.py"""importjsonimportmatplotlib.pyplotaspltwithopen("compression_results.json","r")asresult_file:loaded_results=json.load(result_file)presets=list(range(10))# Create line graph for compression timesplt.figure(figsize=(12,5))plt.subplot(1,2,1)forthread,datainloaded_results.items():times=data["times"]plt.plot(presets,times,label=f"Threads={thread}")plt.xlabel("Preset")plt.ylabel("Time (s)")plt.title("XZ Compression Benchmark - Compression Times (20 MiB)")plt.legend()# Create line graph for compression ratiosplt.subplot(1,2,2)forthread,datainloaded_results.items():ratios=data["ratios"]plt.plot(presets,ratios,label=f"Threads={thread}")plt.xlabel("Preset")plt.ylabel("Compression Ratio")plt.title("XZ Compression Benchmark - Compression Ratios (20 MiB)")plt.legend()plt.tight_layout()plt.show()

Are there any other benchmarks you would like to see run?
Or alternatively, is there a way to integrate the benchmarks into the tests themselves?

@thesamesam
Multi-threaded decompression is not implemented in the liblzma lib as of yet.
To quote thexz(1) man page:

Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't even if --block-size=size is used.

(A possible implementation option would be to support multi-threaded decompression in the cpython_lzmamodule.c liblzma wrapper functions, but I don't think that's a wise rabbit-hole to venture down :) )

Your xz is old, I think. It does support it now. What version do you have?

EDIT: Specifically, this was introduced in xz-utils-5.4.0, released on 2022-12-13. The release notes say, among other things:

• Added threaded .xz decompressor lzma_stream_decoder_mt().
  It can use multiple threads with .xz files that have multiple
  Blocks with size information in Block Headers. The threaded
  encoder in xz has always created such files.

Please update the Autoconf script to detect the required functions/symbols, add C pre-processor defines for mt encoding/decoding, and use these to guard the newly added functionality. That way we won't break users with older XZ libs.

Your xz is old, I think. It does support it now. What version do you have?

EDIT: Specifically, this was introduced in xz-utils-5.4.0, released on 2022-12-13.

FTR:

xz.git git:(master)$ git tag --contains 4cce3e27f529af33e0e7749a8cbcec59954946b5 | head -n 1v5.3.3alpha

Yeah, they do odd/even for development (not ABI stable) and stable/prod releases, so 5.4.0 was the first stable version it was available in.

Yeah, they do odd/even for development (not ABI stable) and stable/prod releases, so 5.4.0 was the first stable version it was available in.

Right; I forgot that was a thing. GNU LilyPond used to do that as well (and they probably still do; haven't checked in years.)

@mkomet Are you interested in finishing this off (at least for threaded compression, we can revisit for parallel decompression)? If not, I can put it on my list to try get to, but I can't promise that will be immediately.

Note that XZ Utils < 5.8.1 has a CVE specifically about this function:https://nvd.nist.gov/vuln/detail/CVE-2025-31115. So if this goes in, that will need to be considered for public releases.