Repository files navigation

libdeflate is a library for fast, whole-buffer DEFLATE-based compression anddecompression.

The supported formats are:

• DEFLATE (raw)
• zlib (a.k.a. DEFLATE with a zlib wrapper)
• gzip (a.k.a. DEFLATE with a gzip wrapper)

libdeflate is heavily optimized. It is significantly faster than the zliblibrary, both for compression and decompression, and especially on x86 and ARMprocessors. In addition, libdeflate provides optional high compression modesthat provide a better compression ratio than the zlib's "level 9".

libdeflate itself is a library. The following command-line programs which usethis library are also included:

• libdeflate-gzip, a program which can be a drop-in replacement for standardgzip under some circumstances. Note thatlibdeflate-gzip has somelimitations; it is provided for convenience and isnot meant to be themain use case of libdeflate. It needs a lot of memory to process large files,and it omits support for some infrequently-used options of GNU gzip.

• benchmark, a test program that does round-trip compression and decompressionof the provided data, and measures the compression and decompression speed.It can use libdeflate, zlib, or a combination of the two.

• checksum, a test program that checksums the provided data with Adler-32 orCRC-32, and optionally measures the speed. It can use libdeflate or zlib.

For the release notes, see theNEWS file.

• Building
  • Using CMake
  • Directly integrating the library sources
  • Supported compilers
• API
• Bindings for other programming languages
• DEFLATE vs. zlib vs. gzip
• Compression levels
• Motivation
• License

libdeflate usesCMake. It can be built just like anyother CMake project, e.g. with:

cmake -B build && cmake --build build

By default the following targets are built:

• The static library (normally calledlibdeflate.a)
• The shared library (normally calledlibdeflate.so)
• Thelibdeflate-gzip program, including its aliaslibdeflate-gunzip

Besides the standard CMake build and installation options, there are somelibdeflate-specific build options. SeeCMakeLists.txt for the list of theseoptions. To set an option, add-DOPTION=VALUE to thecmake command.

Prebuilt Windows binaries can be downloaded fromhttps://github.com/ebiggers/libdeflate/releases.

Although the official build system is CMake, care has been taken to keep thelibrary source files compilable directly, without a prerequisite configurationstep. Therefore, it is also fine to just add the library source files directlyto your application, without using CMake.

You should compile bothlib/*.c andlib/*/*.c. You don't need to worryabout excluding irrelevant architecture-specific code, as this is alreadyhandled in the source files themselves using#ifdefs.

If you are doing a freestanding build with-ffreestanding, you must add-DFREESTANDING as well (matching what theCMakeLists.txt does).

• gcc: v4.9 and later
• clang: v3.9 and later (upstream), Xcode 8 and later (Apple)
• MSVC: Visual Studio 2015 and later
• Other compilers: any other C99-compatible compiler should work, though if yourcompiler pretends to be gcc, clang, or MSVC, it needs to be sufficientlycompatible with the compiler it pretends to be.

The above are the minimums, but using a newer compiler allows more of thearchitecture-optimized code to be built. libdeflate is most heavily optimizedfor gcc and clang, but MSVC is supported fairly well now too.

The recommended optimization flag is-O2, and theCMakeLists.txt sets thisfor release builds.-O3 is fine too, but often-O2 actually gives betterresults. It's unnecessary to add flags such as-mavx2 or/arch:AVX2, thoughyou can do so if you want to. Most of the relevant optimized functions arebuilt regardless of such flags, and appropriate ones are selected at runtime.For the same reason, flags like-mno-avx2 donot cause all code using thecorresponding instruction set extension to be omitted from the binary; this isworking as intended due to the use of runtime CPU feature detection.

If using gcc, your gcc should always be paired with a binutils version that isnot much older than itself, to avoid problems where the compiler generatesinstructions the assembler cannot assemble. Usually systems have their gcc andbinutils paired properly, but rarely a mismatch can arise in cases such as theuser installing a newer gcc version without a proper binutils alongside it.Since libdeflate v1.22, the CMake-based build system will detect incompatiblebinutils versions and disable some optimized code accordingly. In olderversions of libdeflate, or if CMake is not being used, a too-old binutils cancause build errors like "no such instruction" from the assembler.

libdeflate has a simple API that is not zlib-compatible. You can createcompressors and decompressors and use them to compress or decompress buffers.See libdeflate.h for details.

There is currently no support for streaming. This has been considered, but italways significantly increases complexity and slows down fast paths.Unfortunately, at this point it remains a future TODO. So: if your applicationcompresses data in "chunks", say, less than 1 MB in size, then libdeflate is agreat choice for you; that's what it's designed to do. This is perfect forcertain use cases such as transparent filesystem compression. But if yourapplication compresses large files as a single compressed stream, similarly tothegzip program, then libdeflate isn't for you.

Note that with chunk-based compression, you generally should have theuncompressed size of each chunk stored outside of the compressed data itself.This enables you to allocate an output buffer of the correct size withoutguessing. However, libdeflate's decompression routines do optionally providethe actual number of output bytes in case you need it.

Windows developers: note that the calling convention of libdeflate.dll is"cdecl". (libdeflate v1.4 through v1.12 used "stdcall" instead.)

The libdeflate project itself only provides a C library. If you need to uselibdeflate from a programming language other than C or C++, consider using thefollowing bindings:

• C#:LibDeflate.NET
• Delphi:libdeflate-pas
• Go:go-libdeflate
• Java:libdeflate-java
• Julia:LibDeflate.jl
• Nim:libdeflate-nim
• Perl:Gzip::Libdeflate
• PHP:ext-libdeflate
• Python:deflate
• Ruby:libdeflate-ruby
• Rust:libdeflater

Note: these are third-party projects which haven't necessarily been vetted bythe authors of libdeflate. Please direct all questions, bugs, and improvementsfor these bindings to their authors.

Also, unfortunately many of these bindings bundle or pin an old version oflibdeflate. To avoid known issues in old versions and to improve performance,before using any of these bindings please ensure that the bundled or pinnedversion of libdeflate has been upgraded to the latest release.

The DEFLATE format (rfc1951), the zlibformat (rfc1950), and the gzip format(rfc1952) are commonly confused witheach other as well as with thezlib software library, whichactually supports all three formats. libdeflate (this library) also supportsall three formats.

Briefly, DEFLATE is a raw compressed stream, whereas zlib and gzip are differentwrappers for this stream. Both zlib and gzip include checksums, but gzip caninclude extra information such as the original filename. Generally, you shouldchoose a format as follows:

• If you are compressing whole files with no subdivisions, similar to thegzipprogram, you probably should use the gzip format.
• Otherwise, if you don't need the features of the gzip header and footer but dostill want a checksum for corruption detection, you probably should use thezlib format.
• Otherwise, you probably should use raw DEFLATE. This is ideal if you don'tneed checksums, e.g. because they're simply not needed for your use case orbecause you already compute your own checksums that are stored separately fromthe compressed stream.

Note that gzip and zlib streams can be distinguished from each other based ontheir starting bytes, but this is not necessarily true of raw DEFLATE streams.

An often-underappreciated fact of compression formats such as DEFLATE is thatthere are an enormous number of different ways that a given input could becompressed. Different algorithms and different amounts of computation time willresult in different compression ratios, while remaining equally compatible withthe decompressor.

For this reason, the commonly used zlib library provides nine compressionlevels. Level 1 is the fastest but provides the worst compression; level 9provides the best compression but is the slowest. It defaults to level 6.libdeflate uses this same design but is designed to improve on both zlib'sperformanceand compression ratio at every compression level. In addition,libdeflate's levels goup to 12 to make room for aminimum-cost-path based algorithm (sometimes called "optimal parsing") that cansignificantly improve on zlib's compression ratio.

If you are using DEFLATE (or zlib, or gzip) in your application, you should testdifferent levels to see which works best for your application.

Despite DEFLATE's widespread use mainly through the zlib library, in thecompression community this format from the early 1990s is often consideredobsolete. And in a few significant ways, it is.

So why implement DEFLATE at all, instead of focusing entirely onbzip2/LZMA/xz/LZ4/LZX/ZSTD/Brotli/LZHAM/LZFSE/[insert cool new format here]?

To do something better, you need to understand what came before. And it turnsout that most ideas from DEFLATE are still relevant. Many of the newer formatsshare a similar structure as DEFLATE, with different tweaks. The effects oftrivial but very useful tweaks, such as increasing the sliding window size, areoften confused with the effects of nontrivial but less useful tweaks. Andactually, many of these formats are similar enough that common algorithms andoptimizations (e.g. those dealing with LZ77 matchfinding) can be reused.

In addition, comparing compressors fairly is difficult because the performanceof a compressor depends heavily on optimizations which are not intrinsic to thecompression format itself. In this respect, the zlib library sometimes comparespoorly to certain newer code because zlib is not well optimized for modernprocessors. libdeflate addresses this by providing an optimized DEFLATEimplementation which can be used for benchmarking purposes. And, of course,real applications can use it as well.

libdeflate isMIT-licensed.

I am not aware of any patents or patent applications relevant to libdeflate.