Repository files navigation

lib842 provides efficient, accelerated implementation of the 842 compression algorithm available from userspace. 842 is a compression algorithm by IBM, similar to LZ77, designed for very fast compression and decompression. Thus, it is suitable for cases such as compressed RAM (zram) or I/O link compression.

You can find more information about the algorithm, including a description of its implementation, in our publications:

• Plauth, Max, and Andreas Polze. "Towards Improving Data Transfer Efficiency for Accelerators Using Hardware Compression." 2018 Sixth International Symposium on Computing and Networking Workshops (CANDARW). IEEE, 2018.

• Plauth, Max, and Andreas Polze. "GPU-Based Decompression for the 842 Algorithm." 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW). IEEE, 2019.

lib842 provides multiple implementations designed for different accelerators:

• Asimple serial implementation for the CPU intended to serve as a reference. This implementation is a port of the reference implementation available in themainline Linux kernel.

• Anoptimized serial implementation for the CPU designed to achieve high compression/decompression speeds.

• AOpenCL implementation for OpenCL-capable GPUs designed to achieve high decompression speeds.

• ACUDA implementation for CUDA-capable GPUs designed to achieve high decompression speeds.

• Acryptodev implementation that allows using the dedicated hardware compressor on IBM POWER7+ hardware.

Currently, thesimple serial,optimized serial andcryptodev implementations support both compression and decompression, while theOpenCL andCUDA implementations support decompression only.

CMake is required in order to execute the build process.

Thesimple serial andoptimized serial implementations can be built and used on any machine with a modern C/C++ compiler. However,OpenMP is needed in order to enable parallel (multithreaded) compression on the CPU.

Thecryptodev implementation requires building, installing and loading ourmodified cryptodev kernel module with compression support. On IBM POWER7+ hardware, the dedicated hardware compressor will be automatically used. Otherwise, the Linux kernel will fall back to a non-accelerated implementation.

TheOpenCL andCUDA implementations require a working OpenCL or CUDA environment respectively.

First off, make sure to clone the repository including its submodules (git clone --recurse-submodules) in order to pull all required dependencies.

To build lib842, type the following in a shell on the root of the cloned repository:

mkdir build && cd build && cmake -DCMAKE_BUILD_TYPE=Release .. && make -j$(nproc)

After a successful compilation, at least thesimple serial andoptimized serial implementations will be compiled. You can test compression and decompression by executing one of the samples:

./test_serial /path/to/my/file./test_serial_optimized /path/to/my/file

Other implementations will be automatically built if the required dependencies are supported. The build process should provide suitable output to determine which implementations are built or not and why.

You can also verify the implementations are working correctly on your hardware by running the unit tests using thectest command (included with CMake) after a successful build.