Introduction¶

XZ is a general purpose data compression format with high compressionratio and relatively fast decompression. The primary compressionalgorithm (filter) is LZMA2. Additional filters can be used to improvecompression ratio even further. E.g. Branch/Call/Jump (BCJ) filtersimprove compression ratio of executable data.

The XZ decompressor in Linux is called XZ Embedded. It supportsthe LZMA2 filter and optionally also BCJ filters. CRC32 is supportedfor integrity checking. The home page of XZ Embedded is at<https://tukaani.org/xz/embedded.html>, where you can find thelatest version and also information about using the code outsidethe Linux kernel.

For userspace, XZ Utils provide a zlib-like compression libraryand a gzip-like command line tool. XZ Utils can be downloaded from<https://tukaani.org/xz/>.

XZ related components in the kernel¶

The xz_dec module provides XZ decompressor with single-call (bufferto buffer) and multi-call (stateful) APIs. The usage of the xz_decmodule is documented in include/linux/xz.h.

The xz_dec_test module is for testing xz_dec. xz_dec_test is notuseful unless you are hacking the XZ decompressor. xz_dec_testallocates a char device major dynamically to which one can write.xz files from userspace. The decompressed output is thrown away.Keep an eye on dmesg to see diagnostics printed by xz_dec_test.See the xz_dec_test source code for the details.

For decompressing the kernel image, initramfs, and initrd, thereis a wrapper function in lib/decompress_unxz.c. Its API is thesame as in other decompress_*.c files, which is defined ininclude/linux/decompress/generic.h.

scripts/xz_wrap.sh is a wrapper for the xz command line tool foundfrom XZ Utils. The wrapper sets compression options to values suitablefor compressing the kernel image.

For kernel makefiles, two commands are provided for use with$(call if_needed). The kernel image should be compressed with$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2dictionary. It will also append a four-byte trailer containing theuncompressed size of the file, which is needed by the boot code.Other things should be compressed with $(call if_needed,xzmisc)which will use no BCJ filter and 1 MiB LZMA2 dictionary.

Notes on compression options¶

Since the XZ Embedded supports only streams with no integrity check orCRC32, make sure that you don’t use some other integrity check typewhen encoding files that are supposed to be decoded by the kernel. Withliblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32when encoding. With the xz command line tool, use –check=none or–check=crc32.

Using CRC32 is strongly recommended unless there is some other layerwhich will verify the integrity of the uncompressed data anyway.Double checking the integrity would probably be waste of CPU cycles.Note that the headers will always have a CRC32 which will be validatedby the decoder; you can only change the integrity check type (ordisable it) for the actual uncompressed data.

In userspace, LZMA2 is typically used with dictionary sizes of severalmegabytes. The decoder needs to have the dictionary in RAM, thus bigdictionaries cannot be used for files that are intended to be decodedby the kernel. 1 MiB is probably the maximum reasonable dictionarysize for in-kernel use (maybe more is OK for initramfs). The presetsin XZ Utils may not be optimal when creating files for the kernel,so don’t hesitate to use custom settings. Example:

xz --check=crc32 --lzma2=dict=512KiB inputfile

An exception to above dictionary size limitation is when the decoderis used in single-call mode. Decompressing the kernel itself is anexample of this situation. In single-call mode, the memory usagedoesn’t depend on the dictionary size, and it is perfectly fine touse a big dictionary: for maximum compression, the dictionary shouldbe at least as big as the uncompressed data itself.

Future plans¶

Creating a limited XZ encoder may be considered if people think it isuseful. LZMA2 is slower to compress than e.g. Deflate or LZO even atthe fastest settings, so it isn’t clear if LZMA2 encoder is wantedinto the kernel.

Support for limited random-access reading is planned for thedecompression code. I don’t know if it could have any use in thekernel, but I know that it would be useful in some embedded projectsoutside the Linux kernel.

Conformance to the .xz file format specification¶

There are a couple of corner cases where things have been simplifiedat expense of detecting errors as early as possible. These should notmatter in practice all, since they don’t cause security issues. Butit is good to know this if testing the code e.g. with the test filesfrom XZ Utils.

Reporting bugs¶

Before reporting a bug, please check that it’s not fixed alreadyat upstream. See <https://tukaani.org/xz/embedded.html> to get thelatest code.

Report bugs to <lasse.collin@tukaani.org> or visit #tukaani onFreenode and talk to Larhzu. I don’t actively read LKML or otherkernel-related mailing lists, so if there’s something I should know,you should email to me personally or use IRC.

Don’t bother Igor Pavlov with questions about the XZ implementationin the kernel or about XZ Utils. While these two implementationsinclude essential code that is directly based on Igor Pavlov’s code,these implementations aren’t maintained nor supported by him.