| Zstandard | |
|---|---|
 | |
| Original author | Yann Collet |
| Developers | Yann Collet, Nick Terrell, Przemysław Skibiński[1] |
| Initial release | 23 January 2015 (2015-01-23) |
| Stable release | |
| Repository | |
| Written in | C |
| Operating system | Cross-platform |
| Platform | Portable |
| Type | Data compression |
| License | BSD-3-Clause orGPL-2.0-or-later (dual-licensed) |
| Website | facebook |
Zstandard is alosslessdata compression algorithm developed by Yann Collet atFacebook.Zstd is the correspondingreference implementation inC, released asopen-source software on 31 August 2016.[3][4]
The algorithm was published in 2018 asRFC 8478, which also defines an associatedmedia type "application/zstd",filename extension "zst", andHTTP content encoding "zstd".[5]
Zstandard was designed to give acompression ratio comparable to that of theDEFLATE algorithm (developed in 1991 and used in the originalZIP andgzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)[6] to 22 (slowest in compression speed, but best compression ratio).
Starting from version 1.3.2 (October 2017), zstd optionally implements very-long-range search and deduplication (--long, 128 MiB window) similar torzip orlrzip.[7]
Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.[8] The Zstandard command-line has an "adaptive" (--adapt) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.
Zstd at its maximum compression level gives a compression ratio close tolzma,lzham, andppmx, andperforms better[vague] thanlza orbzip2.[improper synthesis?][9][10] Zstandard reaches the currentPareto frontier, as it decompresses faster than any other currently available algorithm with similar or better compression ratio.[as of?][11][12]
Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.[13][14] In particular, one dictionary can be loaded to process large sets of files with redundancy between files, but not necessarily within each file, such as forlog files.
Zstandard combines a dictionary-matching stage (LZ77) with a large search window and a fastentropy-coding stage. It uses bothHuffman coding (used for entries in the Literals section)[15] and finite-state entropy (FSE) –a fast tabled version of ANS,tANS, used for entries in the Sequences section. Because of the manner in which FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).
| Zstandard | |
|---|---|
| Filename extension | .zst[16] |
| Internet media type | application/zstd[16] |
| Magic number | 28 b5 2f fd[16] |
| Type of format | Data compression |
| Standard | RFC 8878 |
| Website | github |
| Zstandard Dictionary | |
|---|---|
| Magic number | 37 a4 30 ec[16] |
| Standard | RFC 8878 |
| Website | github |
TheLinux kernel has included Zstandard since November 2017 (version 4.14) as a compression method for thebtrfs andsquashfs filesystems.[17][18][19]
In 2017, Allan Jude integrated Zstandard into theFreeBSD kernel,[20] and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-conceptOpenZFS compression method[8] which was integrated in 2020.[21]
TheAWS Redshift andRocksDB databases include support for field compression using Zstandard.[22]
In March 2018,Canonical tested[23] the use of zstd as adeb package compression method by default for theUbuntu Linux distribution. Compared withxz compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Support was added to Debian (and subsequently, Ubuntu) in April 2018 (in version 1.6~rc1).[24][23][25]
Fedora added ZStandard support toRPM in May 2018 (Fedora release 28) and used it for packaging the release in October 2019 (Fedora 31).[26] In Fedora 33, the filesystem is compressed by default with zstd.[27][28]
Arch Linux added support for zstd as a package compression method in October 2019 with the release of thepacman 5.2 package manager[29] and in January 2020 switched from xz to zstd for the packages in the official repository. Arch useszstd -c -T0 --ultra -20 -; the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, and compression memory increased but scales with the number of threads used.[30][31][32] Arch Linux later also switched to zstd as the default compression algorithm for mkinitcpioinitial ramdisk generator.[33]
A full implementation of the algorithm with an option to choose the compression level is used in the .NSZ/.XCZ[34] file formats developed by thehomebrew community for theNintendo Switch hybrid game console.[35] It is also one of many supported compression algorithms in the .RVZWii andGameCubedisc image file format.
On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37]
In March 2024,Google Chrome version 123 (andChromium-based browsers such asBrave orMicrosoft Edge) added zstd support in theHTTP headerContent-Encoding.[38] In May 2024,Firefox release 126.0 added zstd support in theHTTP headerContent-Encoding.[39]
The reference implementation is licensed under theBSD license, published atGitHub.[40] Since version 1.0, published 31 August 2016,[41] it had an additional Grant of Patent Rights.[42]
From version 1.3.1, released 20 August 2017,[43] this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.[44]
The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.
| Archiving only | |
|---|---|
| Compressing only | |
| Archiving and compressing | |
| Software packaging and distributing | |
| Document packaging and distributing | |
