README

Decompress - Pure OCaml implementation of decompression algorithms

decompress is a library which implements:

• RFC1951

• Zlib

• Gzip

• LZO

The library

The library is available with:

$ opam install decompress

It provides three sub-packages:

• decompress.de to handle RFC1951 stream

• decompress.zl to handle Zlib stream

• decompress.gz to handle Gzip stream

• decompress.lzo to handle LZO contents

Each sub-package provide 3 sub-modules:

• Inf to inflate/decompress a stream

• Def to deflate/compress a stream

• Higher as a easy entry point to use the stream

How to use it

The binary

The distribution provides a simple binary which is able to compress/uncompress anything:

$ decompress -fgzip --deflate < my_document.txt > my_document.gzip$ decompress -fgzip < my_document.gzip > my_document.out$ diff my_document.txt my_document.out

It does the GZip compression, the Zlib one and the DEFLATE one. It can do an LZO compression too.

Link issue

decompress usescheckseum to compute CRC of streams.checkseum provides 2 implementations:

• a C implementation to be fast

• an OCaml implementation to be usable withjs_of_ocaml (or, at least, require only thecaml runtime)

When the user wants to make an OCaml executable, it must choose which implementation ofcheckseum he wants. A compilation of an executable withdecompress.zl is:

$ ocamlfind opt -linkpkg -package checkseum.c,decompress.zl main.ml

Otherwise, the end-user should have a linking error (see#47).

Withdune

checkseum uses a mechanism integrated intodune which solves the link issue. It provides a way to silently choose the default implementation ofcheckseum:checkseum.c.

By this way (and only withdune), an executable withdecompress.zl is:

(executable (name main) (libraries decompress.zl))

Of course, the user still is able to choose which implementation he wants:

(executable (name main) (libraries checkseum.ocaml decompress.zl))

The API

decompress proposes to the user a full control of:

• the input/output loop

• the allocation

Input / Output

The process of the inflation/deflation is non-blocking and it does not require anysyscalls (as an usual MirageOS project). The user can decide how to get the input and how to store the output.

An usualloop (which can fit intolwt orasync) ofdecompress.zl is:

let rec go decoder = match Zl.Inf.decode decoder with  | `Await decoder ->    let len = input itmp 0 (Bigstringaf.length tmp) in    go (Zl.Inf.src decoder itmp 0 len)  | `Flush decoder ->    let len = Bigstringaf.length otmp - Zl.Inf.dst_rem decoder in    output stdout otmp 0 len ;    go (Zl.Inf.flush decoder)  | `Malformed err -> invalid_arg err  | `End decoder ->    let len = Bigstringaf.length otmp - Zl.Inf.dst_rem decoder in    output stdout otmp 0 len ingo decoder

Allocation

Then, the process does not allocate large objects but it requires at the initialisation these objects. Such objects can be re-used by another inflation/deflation process - of course, these processes can not use same objects at the same time.

val decompress : window:De.window -> in_channel -> out_channel -> unitlet w0 = De.make_windows ~bits:15(* Safe use of decompress *)let () =  decompress ~window:w0 stdin stdout ;  decompress ~window:w0 (open_in "file.z") (open_out "file")(* Unsafe use of decompress,   the second process must use an other pre-allocated window. *)let () =  Lwt_main.run @@    Lwt.join [ (decompress ~window:w0 stdin stdout |> Lwt.return)             ; (decompress ~window:w0 (open_in "file.z") (open_out "file")       |> Lwt.return) ]

This ability can be used on:

• the input buffer given to the encoder/decoder withsrc

• the output buffer given to the encoder/decoder

• the window given to the encoder/decoder

• the shared-queue used by the compression algorithm and the encoder

Example

An example exists intobin/decompress.ml where you can see how to usedecompress.zl anddecompress.de.

Higher interface

However,decompress provides ahigher interface close to whatcamlzip provides to help newcomers to usedecompress:

val compress :     refill:(bigstring -> int)  -> flush:(bigstring -> int -> unit)  -> unitval uncompress :     refill:(bigstring -> int)  -> flush:(bigstring -> int -> unit)  -> unit

Benchmark

decompress has a benchmark aboutinflation to see if any update has a performance implication. The process try toinflate a stream and stop at N second(s) (default is 30), The benchmark requireslibzlib-dev,cmdliner andbos to be able to compilezpipe and the executable to produce the CSV file. To build the benchmark:

$ dune build --profile benchmark bench/output.csv

On linux machines,/dev/urandom will generate the random input for piping to zpipe. To run the benchmark:

$ cat /dev/urandom | ./_build/default/bench/zpipe \  | ./_build/default/bench/bench.exe 2> /dev/null

The output file is a CSV file which can be processed by aplot software. It records input bytes, output bytes and memory usage at each second. You can show results withgnuplot:

$ gnuplot -p -e \  'set datafile separator ",";   set key autotitle columnhead;   plot "_build/default/bench/output.csv" using 1:2 with lines,        "" using 1:3 with lines'$ gnuplot -p -e \  'set datafile separator ",";   set key autotitle columnhead;   plot "_build/default/bench/output.csv" using 1:4 with lines'

The second graph ensure that the inflation does not allocate while it processes. It ensure that, at another layer,decompress does not leak memory.

Build Requirements

• OCaml >= 4.07.0

• dune to build the project

• base-bytes meta-package

• checkseum

• optint

1. checkseum>= "0.2.0"
2. optint>= "0.1.0"
3. cmdliner>= "1.1.0"
4. dune>= "2.8.0"
5. ocaml>= "4.07.0"

1. astringwith-test & >= "0.8.5"
2. boswith-test & >= "0.2.1"
3. rresultwith-test
4. crowbarwith-test & >= "0.2"
5. base64>= "3.0.0" & with-test
6. camlzip>= "1.10" & with-test
7. fmtwith-test & >= "0.8.7"
8. alcotestwith-test & >= "1.7.0"
9. bigstringafwith-test & >= "0.9.0"

1. albatross>= "1.1.1"
2. builder-web>= "0.2.0"
3. carton>= "0.4.4"
4. carton-git>= "0.3.0"
5. carton-lwt>= "0.3.0" & < "1.0.0"
6. clz
7. doi2bib>= "0.4.0" & < "0.5.2"
8. dream-encoding
9. esperanto-cosmopolitan>= "0.0.5"
10. git>= "3.0.0" & < "3.1.1" | >= "3.9.1"
11. git-unix= "3.1.0" | >= "3.3.1"
12. imagelib>= "20210402"
13. nomad
14. rfc1951>= "1.5.3"
15. SZXX>= "2.0.0"
16. sherlodoc
17. tar>= "2.1.0"

None