Repository files navigation

A comparison of 20+ lossless image compression formats on several datasets.

• Compress Formats:
  • PNG, JPEG-LS, JPEG-XL, WEBP, etc. SeeFormat List for details.
  • All formats are instrictly lossless compression mode.
• Comparison Metrics:
  • Compressed Size (the smaller the better)
  • Compress Time (the smaller the better)
  • Decompress Time (the smaller the better)
• Benchmark datasets:
  • CLIC2021: RGB 24-bit, 585 images, 4.1 GB
  • LPCB: RGB 24-bit, 107 images, 3.5 GB
  • GDCC2020: RGB 24-bit, 100 images, 1.0 GB
  • UCID: RGB 24-bit, 1338 images, 789 MB
  • ImgInfoRGB: RGB 24-bit, 14 images, 470 MB
  • ImgInfoGray: Gray 8-bit, 15 images, 163 MB
  • Kodak: RGB 24-bit, 24 images, 28 MB
  • All benchmark images are in.pnm format (a simple image format which stores raw pixels), seePNM file spec for detail.
• Environment:
  • Intel Core i7 12700H, 16GB DDR4 in 3200MHz, Windows 10 system.
  • All codecs run in single threaded, but SIMD (AVX2,AVX,SSE4,SSSE3,SSE3,SSE2,SSE) is permitted. This rule is as same as theGDCC competition.
• Benchmark Steps:
  • Step1: compress all.pnm images one-by-one in a specified folder. The compressed files will be stored in another folder. Measure the total compress time and the total compressed size.
  • Step2: decompress them back to.pnm one-by-one. Measure the total decompress time.
  • The above steps are done by a Python script, SeeHow to run benchmark .
  • For datasets that have small images (UCID, Kodak, etc.), program startup time accounts for a significant portion of compression/decompression time. On the contrary, time consumption on large images can better reflect the true performance of the codecs.

(1) KVICK failed to compress 006.pnm

Figure: result on CLIC2021. The bottom left corner is optimal.

Note: My LPCB isslightly different from theLPCB by Alex Rhatushnyak. This is because the astronomical photo website mentioned by Alex has changed, and the images I found are not exactly the same as the ones Alex downloaded.

(1) KVICK failed to compress PIA12811.pnm

(1) QLIC2 failed to compress 534.pnm(2) KVICK failed to compress 1.pnm

(1) HALIC fast failed to decompress zone_plate.pnm(2) To be tested, extremely slow

Figure: result on ImgInfoGray. The bottom left corner is optimal.

There are 3 types of codecs (coder/decoder):

• Most of them are Windows executable files (.EXE). I put them in thecodec folder.
• Some of them are Linux binary files which runs under Windows Subsystem of Linux (WSL). I put them in thecodec folder.
• Some of them arePillow 10.1.0 library , e.g., for PNG:
  • To compress, run this in Python:Image.open(r"in.pnm").save(r"comp.png")
  • To decompress, run this in Python:Image.open(r"comp.png").save(r"out.pnm")

Note: If you want to add any other formats, please submit an issue.

(1): The well-knownFLIF (Free Lossless Image Format) was superseded byJPEG-XL.*(2): MRP can only compress gray 8-bit images with size multiple of 8.*

The benchmarking script is written in Python3. If you had not install Python3 yet, it is recommended to installAnaconda (It seems that you should fill your email to download now). Just select the newest version for Windows (the version was Python 3.12 when I wrote this document).

In this benchmark, some of the formats are compressed using Windows .EXE files (in./codec/ folder), while others are compressed use Python 3.9 with Pillow 10.1.0 library. Therefore, after installing Anaconda, you need to use CMD (command line) to run following commands to install the required packages:

python -m pip install Pillow==10.1.0       # for PNG, WEBP, and JPEG2000 compress/decompresspython -m pip install pillow_jpls==1.3.2   # for JPEG-LS compress/decompress

The benchmarking script isRunBenchmark.py. Use following command to run:

python RunBenchmark.py <image_folder>

There must be some.pnm image files in <image_folder>, either RGB 24bit-images or Gray 8-bit images. The script will benchmark these images.

For example, I provide two images inexample_image folder, so you can run a simple test on it:

python RunBenchmark.py example_image

To specify which compression formats to participate in comparison, simply modify theCODEC_LIST list inRunBenchmark.py (just comment/uncomment corresponding lines).

Just prepare more.pnm images in a folder, and run theRunBenchmark.py.

CLIC2021 dataset has 585 RGB 24-bit images totaling 4.1 GB, which is the training set ofCLIC2021 competition. To get it:

• Downloaddata.vision.ee.ethz.ch/cvl/clic/professional_train_2020.zip
• Extract the zip, getting 585 .png image files.
• Download and installNConvert 7.163. We will use nconvert.exe (You can add it to your Windows environment variable).
• Convert all the downloaded .png images toRGB 24-bit .pnm format using nconvert.exe.The method is to run the following command in the directory where the images are located:

  nconvert.exe -out pnm *.png

• Put these 585 .pnm files into a folder.
• Run benchmark.

GDCC2020 dataset has 100 RGB 24-bit images totaling 1 GB, which is the test set ofGDCC2020 competition. To get it:

• Download itfrom this link.
• Extract it, getting 107 .pnm files.
• Run benchmark.

UCID dataset has 1338 RGB 24-bit images totaling 789 MB. To get it:

• Download itfrom this link. Please click the linkUCID Database (Zipped)
• Extract it, getting 1338 .tif image files.
• Download and installNConvert 7.163. We will use nconvert.exe (You can add it to your Windows environment variable).
• Convert all the downloaded .tif images toRGB 24-bit .pnm format using nconvert.exe.The method is to run the following command in the directory where the images are located:

  nconvert.exe -out pnm *.tif

• Put these 1338 .pnm files into a folder.
• Run benchmark.

ImgInfoRGB dataset has 14 RGB 24-bit images totaling 470 MB. To get it:

• Download itfrom this link, please click theRGB 8 bit.
• Extract it, getting 14 .ppm files.
• Rename their file suffix from .ppm to .pnm. (actually ppm and pnm are same file formats)
• Run benchmark.

ImgInfoGray dataset has 15 Gray 8-bit images totaling 163 MB. To get it:

• Download itfrom this link, please click theGray 8 bit.
• Extract it, getting 15 .pgm files.
• Rename their file suffix from .pgm to .pnm. (actually pgm and pnm are same file formats)
• Run benchmark.

Kodak dataset has 24 RGB 24-bit images totaling 28 MB. To get it:

• VisitKodak True Color Image Suite, you will see 24 .png images. Download them one-by-one.
• Download and installNConvert 7.163. We will use nconvert.exe (You can add it to your Windows environment variable).
• Convert all the downloaded .png images toRGB 24-bit .pnm format using nconvert.exe.The method is to run the following command in the directory where the images are located:

  nconvert.exe -out pnm *.png

• Put these 24 .pnm files into a folder.
• Run benchmark.

LPCB dataset has 107 RGB 24-bit images totaling 3.4 GB, which is similar to the dataset used inthe benchmark by Alex Rhatushnyak, but not exactly the same. This is because the astronomical photo website mentioned by Alex has changed, and the images I found are not exactly the same as the ones Alex downloaded. The method that Alex described for obtaining LPCB can be foundhere.

My LPCB consists of 62 photographs and 45 astronomical images. To get the 62 photographs:

• First, download all the "Sample RAW Images" from the following four sites:Olympus,Sony,Fujifilm, andCanon.Note: Please find the "Sample RAW Images" section on the webpage to download, rather than the "Sample Images" section.
• After downloading, remove some similar images among them.
  • InSony set, remove images 12, 13, 14.
  • InCanon set, remove images 7, 8, 9, 10.
  • InFujifilm set, remove images 7, 8, 9.After removing, there are 62 images left.
• Convert these 62 images to24-bit RGB .pnm format usingNConvert 7.163. The method is to run the following command in the directory where the image is located:

  nconvert.exe -out pnm -high_res -raw_camerabalance *.orf *.arw *.raf *.cr2

• List of the 62 test images in LPCB

    name                         width x height  canon_eos_1100d_01            4290 x 2856  canon_eos_1100d_02            2856 x 4290  canon_eos_1100d_03            4290 x 2856  canon_eos_1100d_04            4290 x 2856  canon_eos_1100d_05            4290 x 2856  canon_eos_1100d_06            2856 x 4290  canon_eos_1100d_11            4290 x 2856  canon_eos_1100d_12            4290 x 2856  canon_eos_1100d_13            2856 x 4290  canon_eos_1100d_14            4290 x 2856  canon_eos_1100d_15            2856 x 4290  fujifilm_finepix_x100_01      4310 x 2870  fujifilm_finepix_x100_02      4310 x 2870  fujifilm_finepix_x100_03      2870 x 4310  fujifilm_finepix_x100_04      2870 x 4310  fujifilm_finepix_x100_05      2870 x 4310  fujifilm_finepix_x100_06      2870 x 4310  fujifilm_finepix_x100_10      4310 x 2870  fujifilm_finepix_x100_11      4310 x 2870  fujifilm_finepix_x100_12      4310 x 2870  fujifilm_finepix_x100_13      2870 x 4310  fujifilm_finepix_x100_14      4310 x 2870  fujifilm_finepix_x100_15      4310 x 2870  olympus_xz1_01                2760 x 3680  olympus_xz1_02                2760 x 3680  olympus_xz1_03                2760 x 3680  olympus_xz1_04                2760 x 3680  olympus_xz1_05                2760 x 3680  olympus_xz1_06                3680 x 2760  olympus_xz1_07                2760 x 3680  olympus_xz1_08                2760 x 3680  olympus_xz1_09                2760 x 3680  olympus_xz1_10                2760 x 3680  olympus_xz1_11                3680 x 2760  olympus_xz1_12                3680 x 2760  olympus_xz1_13                3680 x 2760  olympus_xz1_14                3680 x 2760  olympus_xz1_15                3680 x 2760  olympus_xz1_16                3680 x 2760  olympus_xz1_17                3680 x 2760  olympus_xz1_18                2760 x 3680  olympus_xz1_19                3680 x 2760  olympus_xz1_20                2760 x 3680  olympus_xz1_21                2760 x 3680  olympus_xz1_22                2760 x 3680  olympus_xz1_23                2760 x 3680  olympus_xz1_24                2760 x 3680  olympus_xz1_25                3680 x 2760  olympus_xz1_26                3680 x 2760  olympus_xz1_27                2760 x 3680  sony_a55_01                   4928 x 3280  sony_a55_02                   4928 x 3280  sony_a55_03                   4928 x 3280  sony_a55_04                   4928 x 3280  sony_a55_05                   4928 x 3280  sony_a55_06                   3280 x 4928  sony_a55_07                   4928 x 3280  sony_a55_08                   3280 x 4928  sony_a55_09                   4928 x 3280  sony_a55_10                   3280 x 4928  sony_a55_11                   4928 x 3280  sony_a55_15                   3280 x 4928

To get the 45 astronomical images:

• Download 45 images (.tif) from the following links:

  https://photojournal.jpl.nasa.gov/tiff/PIA12811.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA12813.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13452.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13453.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13454.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13455.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13456.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13457.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13458.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13459.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13757.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13771.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13779.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13781.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13782.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13785.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13789.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13799.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13803.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13810.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13812.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13815.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13833.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13843.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13844.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13845.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13855.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13856.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13857.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13859.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13862.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13872.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13882.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13891.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13894.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13900.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13901.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13904.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13908.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13912.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13913.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13914.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13915.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13942.tifhttps://photojournal.jpl.nasa.gov/tiff/PIA13943.tif

• Convert these 45 .tif images to24-bit RGB .pnm format usingNConvert 7.163. The method is to run the following command in the directory where the image is located:

  nconvert.exe -out pnm *.tif

Finally, Put these 62+45 .pnm files to a folder, then run benchmark.

There are not many Gray 8-bit images available online for you to test. To obtain more gray8 images, you can convert the images fromRGB 24-bit toGray 8-bit usingNConvert 7.163. The method is to run the following command in the directory where the.pnm images are located:

nconvert.exe -grey 256 -out pnm *.pnm

This command may cause the original RGB image file to be replaced. Please make a backup before running it.

• ThanksAlex Rhatushnyak, whoseLPCB benchmark inspires this project.
• ThanksHakan Abbas, who provides good advice to this project.
• Thanks to every who develops these image codecs.