Exceptions¶

exceptioncompression.zstd.ZstdError¶

This exception is raised when an error occurs during compression ordecompression, or while initializing the (de)compressor state.

Reading and writing compressed files¶

compression.zstd.open(file,/,mode='rb',*,level=None,options=None,zstd_dict=None,encoding=None,errors=None,newline=None)¶

Open a Zstandard-compressed file in binary or text mode, returning afile object.

Thefile argument can be either a file name (given as astr,bytes orpath-likeobject), in which case the named file is opened, or it can be an existingfile object to read from or write to.

The mode argument can be either'rb' for reading (default),'wb' foroverwriting,'ab' for appending, or'xb' for exclusive creation.These can equivalently be given as'r','w','a', and'x'respectively. You may also open in text mode with'rt','wt','at', and'xt' respectively.

When reading, theoptions argument can be a dictionary providing advanceddecompression parameters; seeDecompressionParameter for detailedinformation about supportedparameters. Thezstd_dict argument is aZstdDict instance to beused during decompression. When reading, if thelevelargument is not None, aTypeError will be raised.

When writing, theoptions argument can be a dictionaryproviding advanced decompression parameters; seeCompressionParameter for detailed information about supportedparameters. Thelevel argument is the compression level to use whenwriting compressed data. Only one oflevel oroptions may be non-None.Thezstd_dict argument is aZstdDict instance to be used duringcompression.

In binary mode, this function is equivalent to theZstdFileconstructor:ZstdFile(file,mode,...). In this case, theencoding,errors, andnewline parameters must not be provided.

In text mode, aZstdFile object is created, and wrapped in anio.TextIOWrapper instance with the specified encoding, errorhandling behavior, and line endings.

classcompression.zstd.ZstdFile(file,/,mode='rb',*,level=None,options=None,zstd_dict=None)¶

Open a Zstandard-compressed file in binary mode.

AZstdFile can wrap an already-openfile object, or operatedirectly on a named file. Thefile argument specifies either the fileobject to wrap, or the name of the file to open (as astr,bytes orpath-like object). Ifwrapping an existing file object, the wrapped file will not be closed whentheZstdFile is closed.

Themode argument can be either'rb' for reading (default),'wb'for overwriting,'xb' for exclusive creation, or'ab' for appending.These can equivalently be given as'r','w','x' and'a'respectively.

Iffile is a file object (rather than an actual file name), a mode of'w' does not truncate the file, and is instead equivalent to'a'.

When reading, theoptions argument can be a dictionaryproviding advanced decompression parameters; seeDecompressionParameter for detailed information about supportedparameters. Thezstd_dict argument is aZstdDict instance to beused during decompression. When reading, if thelevelargument is not None, aTypeError will be raised.

When writing, theoptions argument can be a dictionaryproviding advanced decompression parameters; seeCompressionParameter for detailed information about supportedparameters. Thelevel argument is the compression level to use whenwriting compressed data. Only one oflevel oroptions may be passed. Thezstd_dict argument is aZstdDict instance to be used duringcompression.

ZstdFile supports all the members specified byio.BufferedIOBase, except fordetach()andtruncate().Iteration and thewith statement are supported.

The following method and attributes are also provided:

peek(size=-1)¶

Return buffered data without advancing the file position. At least onebyte of data will be returned, unless EOF has been reached. The exactnumber of bytes returned is unspecified (thesize argument is ignored).

Note

While callingpeek() does not change the file position oftheZstdFile, it may change the position of the underlyingfile object (for example, if theZstdFile was constructed bypassing a file object forfile).

mode¶

'rb' for reading and'wb' for writing.

name¶

The name of the Zstandard file. Equivalent to thenameattribute of the underlyingfile object.

Compressing and decompressing data in memory¶

compression.zstd.compress(data,level=None,options=None,zstd_dict=None)¶

Compressdata (abytes-like object), returning the compresseddata as abytes object.

Thelevel argument is an integer controlling the level ofcompression.level is an alternative to settingCompressionParameter.compression_level inoptions. Usebounds() oncompression_level to get the values that canbe passed forlevel. If advanced compression options are needed, thelevel argument must be omitted and in theoptions dictionary theCompressionParameter.compression_level parameter should be set.

Theoptions argument is a Python dictionary containing advancedcompression parameters. The valid keys and values for compression parametersare documented as part of theCompressionParameter documentation.

Thezstd_dict argument is an instance ofZstdDictcontaining trained data to improve compression efficiency. Thefunctiontrain_dict() can be used to generate a Zstandard dictionary.

compression.zstd.decompress(data,zstd_dict=None,options=None)¶

Decompressdata (abytes-like object), returning the uncompresseddata as abytes object.

Theoptions argument is a Python dictionary containing advanceddecompression parameters. The valid keys and values for compressionparameters are documented as part of theDecompressionParameterdocumentation.

Thezstd_dict argument is an instance ofZstdDictcontaining trained data used during compression. This must bethe same Zstandard dictionary used during compression.

Ifdata is the concatenation of multiple distinct compressed frames,decompress all of these frames, and return the concatenation of the results.

classcompression.zstd.ZstdCompressor(level=None,options=None,zstd_dict=None)¶

Create a compressor object, which can be used to compress dataincrementally.

For a more convenient way of compressing a single chunk of data, see themodule-level functioncompress().

Thelevel argument is an integer controlling the level ofcompression.level is an alternative to settingCompressionParameter.compression_level inoptions. Usebounds() oncompression_level to get the values that canbe passed forlevel. If advanced compression options are needed, thelevel argument must be omitted and in theoptions dictionary theCompressionParameter.compression_level parameter should be set.

Theoptions argument is a Python dictionary containing advancedcompression parameters. The valid keys and values for compression parametersare documented as part of theCompressionParameter documentation.

Thezstd_dict argument is an optional instance ofZstdDictcontaining trained data to improve compression efficiency. Thefunctiontrain_dict() can be used to generate a Zstandard dictionary.

compress(data,mode=ZstdCompressor.CONTINUE)¶

Compressdata (abytes-like object), returning abytesobject with compressed data if possible, or otherwise an emptybytes object. Some ofdata may be buffered internally, foruse in later calls tocompress() andflush(). The returneddata should be concatenated with the output of any previous calls tocompress().

Themode argument is aZstdCompressor attribute, eitherCONTINUE,FLUSH_BLOCK,orFLUSH_FRAME.

When all data has been provided to the compressor, call theflush() method to finish the compression process. Ifcompress() is called withmode set toFLUSH_FRAME,flush() should not be called, as it would write out a new emptyframe.

flush(mode=ZstdCompressor.FLUSH_FRAME)¶

Finish the compression process, returning abytes objectcontaining any data stored in the compressor’s internal buffers.

Themode argument is aZstdCompressor attribute, eitherFLUSH_BLOCK, orFLUSH_FRAME.

set_pledged_input_size(size)¶

Specify the amount of uncompressed datasize that will be provided forthe next frame.size will be written into the frame header of the nextframe unlessCompressionParameter.content_size_flag isFalseor0. A size of0 means that the frame is empty. Ifsize isNone, the frame header will omit the frame size. Frames that includethe uncompressed data size require less memory to decompress, especiallyat higher compression levels.

Iflast_mode is notFLUSH_FRAME, aValueError is raised as the compressor is not at the start ofa frame. If the pledged size does not match the actual size of dataprovided tocompress(), future calls tocompress() orflush() may raiseZstdError and the last chunk of data maybe lost.

Afterflush() orcompress() are called with modeFLUSH_FRAME, the next frame will not include the frame size intothe header unlessset_pledged_input_size() is called again.

CONTINUE¶

Collect more data for compression, which may or may not generate outputimmediately. This mode optimizes the compression ratio by maximizing theamount of data per block and frame.

FLUSH_BLOCK¶

Complete and write a block to the data stream. The data returned so farcan be immediately decompressed. Past data can still be referenced infuture blocks generated by calls tocompress(),improving compression.

FLUSH_FRAME¶

Complete and write out a frame. Future data provided tocompress() will be written into a new frame andcannot reference past data.

last_mode¶

The last mode passed to eithercompress() orflush().The value can be one ofCONTINUE,FLUSH_BLOCK, orFLUSH_FRAME. The initial value isFLUSH_FRAME,signifying that the compressor is at the start of a new frame.

classcompression.zstd.ZstdDecompressor(zstd_dict=None,options=None)¶

Create a decompressor object, which can be used to decompress dataincrementally.

For a more convenient way of decompressing an entire compressed stream atonce, see the module-level functiondecompress().

Theoptions argument is a Python dictionary containing advanceddecompression parameters. The valid keys and values for compressionparameters are documented as part of theDecompressionParameterdocumentation.

Thezstd_dict argument is an instance ofZstdDictcontaining trained data used during compression. This must bethe same Zstandard dictionary used during compression.

Note

This class does not transparently handle inputs containing multiplecompressed frames, unlike thedecompress() function andZstdFile class. To decompress a multi-frame input, you shouldusedecompress(),ZstdFile if working with afile object, or multipleZstdDecompressor instances.

decompress(data,max_length=-1)¶

Decompressdata (abytes-like object), returninguncompressed data as bytes. Some ofdata may be bufferedinternally, for use in later calls todecompress().The returned data should be concatenated with the output of any previouscalls todecompress().

Ifmax_length is non-negative, the method returns at mostmax_lengthbytes of decompressed data. If this limit is reached and furtheroutput can be produced, theneeds_input attribute willbe set toFalse. In this case, the next call todecompress() may providedata asb'' to obtainmore of the output.

If all of the input data was decompressed and returned (eitherbecause this was less thanmax_length bytes, or becausemax_length was negative), theneeds_input attributewill be set toTrue.

Attempting to decompress data after the end of a frame will raise aZstdError. Any data found after the end of the frame is ignoredand saved in theunused_data attribute.

eof¶

True if the end-of-stream marker has been reached.

unused_data¶

Data found after the end of the compressed stream.

Before the end of the stream is reached, this will beb''.

needs_input¶

False if thedecompress() method can provide moredecompressed data before requiring new compressed input.

Zstandard dictionaries¶

compression.zstd.train_dict(samples,dict_size)¶

Train a Zstandard dictionary, returning aZstdDict instance.Zstandard dictionaries enable more efficient compression of smaller sizesof data, which is traditionally difficult to compress due to lessrepetition. If you are compressing multiple similar groups of data (such assimilar files), Zstandard dictionaries can improve compression ratios andspeed significantly.

Thesamples argument (an iterable ofbytes objects), is thepopulation of samples used to train the Zstandard dictionary.

Thedict_size argument, an integer, is the maximum size (in bytes) theZstandard dictionary should be. The Zstandard documentation suggests anabsolute maximum of no more than 100 KB, but the maximum can often be smallerdepending on the data. Larger dictionaries generally slow down compression,but improve compression ratios. Smaller dictionaries lead to fastercompression, but reduce the compression ratio.

compression.zstd.finalize_dict(zstd_dict,/,samples,dict_size,level)¶

An advanced function for converting a “raw content” Zstandard dictionary intoa regular Zstandard dictionary. “Raw content” dictionaries are a sequence ofbytes that do not need to follow the structure of a normal Zstandarddictionary.

Thezstd_dict argument is aZstdDict instance withthedict_content containing the raw dictionary contents.

Thesamples argument (an iterable ofbytes objects), containssample data for generating the Zstandard dictionary.

Thedict_size argument, an integer, is the maximum size (in bytes) theZstandard dictionary should be. Seetrain_dict() forsuggestions on the maximum dictionary size.

Thelevel argument (an integer) is the compression level expected to bepassed to the compressors using this dictionary. The dictionary informationvaries for each compression level, so tuning for the proper compressionlevel can make compression more efficient.

classcompression.zstd.ZstdDict(dict_content,/,*,is_raw=False)¶

A wrapper around Zstandard dictionaries. Dictionaries can be used to improvethe compression of many small chunks of data. Usetrain_dict() if youneed to train a new dictionary from sample data.

Thedict_content argument (abytes-like object), is the alreadytrained dictionary information.

Theis_raw argument, a boolean, is an advanced parameter controlling themeaning ofdict_content.True meansdict_content is a “raw content”dictionary, without any format restrictions.False meansdict_contentis an ordinary Zstandard dictionary, created from Zstandard functions,for example,train_dict() or the externalzstd CLI.

When passing aZstdDict to a function, theas_digested_dict andas_undigested_dict attributes cancontrol how the dictionary is loaded by passing them as thezstd_dictargument, for example,compress(data,zstd_dict=zd.as_digested_dict).Digesting a dictionary is a costly operation that occurs when loading aZstandard dictionary. When making multiple calls to compression ordecompression, passing a digested dictionary will reduce the overhead ofloading the dictionary.

Difference for compression¶

	Digested dictionary	Undigested dictionary
Advanced parameters of the compressor which may be overridden bythe dictionary’s parameters	window_log,hash_log,chain_log,search_log,min_match,target_length,strategy,enable_long_distance_matching,ldm_hash_log,ldm_min_match,ldm_bucket_size_log,ldm_hash_rate_log,and some non-public parameters.	None
ZstdDict internally caches the dictionary	Yes. It’s faster when loading a digested dictionary again with thesame compression level.	No. If you wish to load an undigested dictionary multiple times,consider reusing a compressor object.

If passing aZstdDict without any attribute, an undigesteddictionary is passed by default when compressing and a digested dictionaryis generated if necessary and passed by default when decompressing.

dict_content¶

The content of the Zstandard dictionary, abytes object. It’s thesame as thedict_content argument in the__init__ method. It canbe used with other programs, such as thezstd CLI program.

dict_id¶

Identifier of the Zstandard dictionary, a non-negative int value.

Non-zero means the dictionary is ordinary, created by Zstandardfunctions and following the Zstandard format.

0 means a “raw content” dictionary, free of any format restriction,used for advanced users.

Note

The meaning of0 forZstdDict.dict_id is differentfrom thedictionary_id attribute to theget_frame_info()function.

as_digested_dict¶

Load as a digested dictionary.

as_undigested_dict¶

Load as an undigested dictionary.

Advanced parameter control¶

classcompression.zstd.CompressionParameter¶

AnIntEnum containing the advanced compression parameterkeys that can be used when compressing data.

Thebounds() method can be used on any attribute to get the validvalues for that parameter.

Parameters are optional; any omitted parameter will have it’s value selectedautomatically.

Example getting the lower and upper bound ofcompression_level:

lower,upper=CompressionParameter.compression_level.bounds()

Example setting thewindow_log to the maximum size:

_lower,upper=CompressionParameter.window_log.bounds()options={CompressionParameter.window_log:upper}compress(b'venezuelan beaver cheese',options=options)

bounds()¶

Return the tuple of int bounds,(lower,upper), of a compressionparameter. This method should be called on the attribute you wish toretrieve the bounds of. For example, to get the valid values forcompression_level, one may check the result ofCompressionParameter.compression_level.bounds().

Both the lower and upper bounds are inclusive.

compression_level¶

A high-level means of setting other compression parameters that affectthe speed and ratio of compressing data.

Regular compression levels are greater than0. Values greater than20 are considered “ultra” compression and require more memory thanother levels. Negative values can be used to trade off faster compressionfor worse compression ratios.

Setting the level to zero usesCOMPRESSION_LEVEL_DEFAULT.

window_log¶

Maximum allowed back-reference distance the compressor can use whencompressing data, expressed as power of two,1<<window_log bytes.This parameter greatly influences the memory usage of compression. Highervalues require more memory but gain better compression values.

A value of zero causes the value to be selected automatically.

hash_log¶

Size of the initial probe table, as a power of two. The resulting memoryusage is1<<(hash_log+2) bytes. Larger tables improve compressionratio of strategies <=dfast, and improve compressionspeed of strategies >dfast.

A value of zero causes the value to be selected automatically.

chain_log¶

Size of the multi-probe search table, as a power of two. The resultingmemory usage is1<<(chain_log+2) bytes. Larger tables result inbetter and slower compression. This parameter has no effect for thefast strategy. It’s still useful when usingdfast strategy, in which case it defines a secondaryprobe table.

A value of zero causes the value to be selected automatically.

search_log¶

Number of search attempts, as a power of two. More attempts result inbetter and slower compression. This parameter is useless forfast anddfast strategies.

A value of zero causes the value to be selected automatically.

min_match¶

Minimum size of searched matches. Larger values increase compression anddecompression speed, but decrease ratio. Note that Zstandard can stillfind matches of smaller size, it just tweaks its search algorithm to lookfor this size and larger. For all strategies <btopt,the effective minimum is4; for all strategies>fast, the effective maximum is6.

A value of zero causes the value to be selected automatically.

target_length¶

The impact of this field depends on the selectedStrategy.

For strategiesbtopt,btultra andbtultra2, the value is the length of a matchconsidered “good enough” to stop searching. Larger values makecompression ratios better, but compresses slower.

For strategyfast, it is the distance between matchsampling. Larger values make compression faster, but with a worsecompression ratio.

A value of zero causes the value to be selected automatically.

strategy¶

The higher the value of selected strategy, the more complex thecompression technique used by zstd, resulting in higher compressionratios but slower compression.

See also

Strategy

enable_long_distance_matching¶

Long distance matching can be used to improve compression for largeinputs by finding large matches at greater distances. It increases memoryusage and window size.

True or1 enable long distance matching whileFalse or0disable it.

Enabling this parameter increases defaultwindow_log to 128 MiB except when expresslyset to a different value. This setting is enabled by default ifwindow_log >= 128 MiB and the compressionstrategy >=btopt (compression level 16+).

ldm_hash_log¶

Size of the table for long distance matching, as a power of two. Largervalues increase memory usage and compression ratio, but decreasecompression speed.

A value of zero causes the value to be selected automatically.

ldm_min_match¶

Minimum match size for long distance matcher. Larger or too small valuescan often decrease the compression ratio.

A value of zero causes the value to be selected automatically.

ldm_bucket_size_log¶

Log size of each bucket in the long distance matcher hash table forcollision resolution. Larger values improve collision resolution butdecrease compression speed.

A value of zero causes the value to be selected automatically.

ldm_hash_rate_log¶

Frequency of inserting/looking up entries into the long distance matcherhash table. Larger values improve compression speed. Deviating far fromthe default value will likely result in a compression ratio decrease.

A value of zero causes the value to be selected automatically.

content_size_flag¶

Write the size of the data to be compressed into the Zstandard frameheader when known prior to compressing.

This flag only takes effect under the following scenarios:

• Callingcompress() for one-shot compression

• Providing all of the data to be compressed in the frame in a singleZstdCompressor.compress() call, with theZstdCompressor.FLUSH_FRAME mode.

• CallingZstdCompressor.set_pledged_input_size() with the exactamount of data that will be provided to the compressor prior to anycalls toZstdCompressor.compress() for the current frame.ZstdCompressor.set_pledged_input_size() must be called for eachnew frame.

All other compression calls may not write the size information into theframe header.

True or1 enable the content size flag whileFalse or0disable it.

checksum_flag¶

A four-byte checksum using XXHash64 of the uncompressed content iswritten at the end of each frame. Zstandard’s decompression code verifiesthe checksum. If there is a mismatch aZstdError exception israised.

True or1 enable checksum generation whileFalse or0disable it.

dict_id_flag¶

When compressing with aZstdDict, the dictionary’s ID is writteninto the frame header.

True or1 enable storing the dictionary ID whileFalse or0 disable it.

nb_workers¶

Select how many threads will be spawned to compress in parallel. Whennb_workers > 0, enables multi-threaded compression, a value of1 means “one-thread multi-threaded mode”. More workers improve speed,but also increase memory usage and slightly reduce compression ratio.

A value of zero disables multi-threading.

job_size¶

Size of a compression job, in bytes. This value is enforced only whennb_workers >= 1. Each compression job iscompleted in parallel, so this value can indirectly impact the number ofactive threads.

A value of zero causes the value to be selected automatically.

overlap_log¶

Sets how much data is reloaded from previous jobs (threads) for new jobsto be used by the look behind window during compression. This value isonly used whennb_workers >= 1. Acceptablevalues vary from 0 to 9.

• 0 means dynamically set the overlap amount

• 1 means no overlap

• 9 means use a full window size from the previous job

Each increment halves/doubles the overlap size. “8” means an overlap ofwindow_size/2, “7” means an overlap ofwindow_size/4, etc.

classcompression.zstd.DecompressionParameter¶

AnIntEnum containing the advanced decompression parameterkeys that can be used when decompressing data. Parameters are optional; anyomitted parameter will have it’s value selected automatically.

Thebounds() method can be used on any attribute to get the validvalues for that parameter.

Example setting thewindow_log_max to the maximum size:

data=compress(b'Some very long buffer of bytes...')_lower,upper=DecompressionParameter.window_log_max.bounds()options={DecompressionParameter.window_log_max:upper}decompress(data,options=options)

bounds()¶

Return the tuple of int bounds,(lower,upper), of a decompressionparameter. This method should be called on the attribute you wish toretrieve the bounds of.

Both the lower and upper bounds are inclusive.

window_log_max¶

The base-two logarithm of the maximum size of the window used duringdecompression. This can be useful to limit the amount of memory used whendecompressing data. A larger maximum window size leads to fasterdecompression.

A value of zero causes the value to be selected automatically.

classcompression.zstd.Strategy¶

AnIntEnum containing strategies for compression.Higher-numbered strategies correspond to more complex and slowercompression.

Note

The values of attributes ofStrategy are not necessarily stableacross zstd versions. Only the ordering of the attributes may be reliedupon. The attributes are listed below in order.

The following strategies are available:

fast¶

dfast¶

greedy¶

lazy¶

lazy2¶

btlazy2¶

btopt¶

btultra¶

btultra2¶

Miscellaneous¶

compression.zstd.get_frame_info(frame_buffer)¶

Retrieve aFrameInfo object containing metadata about a Zstandardframe. Frames contain metadata related to the compressed data they hold.

classcompression.zstd.FrameInfo¶

Metadata related to a Zstandard frame.

decompressed_size¶

The size of the decompressed contents of the frame.

dictionary_id¶

An integer representing the Zstandard dictionary ID needed fordecompressing the frame.0 means the dictionary ID was notrecorded in the frame header. This may mean that a Zstandard dictionaryis not needed, or that the ID of a required dictionary was not recorded.

compression.zstd.COMPRESSION_LEVEL_DEFAULT¶

The default compression level for Zstandard:3.

compression.zstd.zstd_version_info¶

Version number of the runtime zstd library as a tuple of integers(major, minor, release).

Examples¶

Reading in a compressed file:

fromcompressionimportzstdwithzstd.open("file.zst")asf:file_content=f.read()

Creating a compressed file:

fromcompressionimportzstddata=b"Insert Data Here"withzstd.open("file.zst","w")asf:f.write(data)

Compressing data in memory:

fromcompressionimportzstddata_in=b"Insert Data Here"data_out=zstd.compress(data_in)

Incremental compression:

fromcompressionimportzstdcomp=zstd.ZstdCompressor()out1=comp.compress(b"Some data\n")out2=comp.compress(b"Another piece of data\n")out3=comp.compress(b"Even more data\n")out4=comp.flush()# Concatenate all the partial results:result=b"".join([out1,out2,out3,out4])

Writing compressed data to an already-open file:

fromcompressionimportzstdwithopen("myfile","wb")asf:f.write(b"This data will not be compressed\n")withzstd.open(f,"w")aszstf:zstf.write(b"This *will* be compressed\n")f.write(b"Not compressed\n")

Creating a compressed file using compression parameters:

fromcompressionimportzstdoptions={zstd.CompressionParameter.checksum_flag:1}withzstd.open("file.zst","w",options=options)asf:f.write(b"Mind if I squeeze in?")