Repository files navigation

fpc is a Go implementation of Burtscher and Ratanaworabhan's'FPC' algorithm for compressing a stream of floating point data.

The FPC algorithm can losslessly encode and decode huge amounts offloating-point data very quickly. It scales well togigabyte-per-second streams. Compression ratios are better than justabout any generic compressor like gzip or bzip, and compression anddecompression throughput are much better (like, 8x to 300x faster)than other algorithms. For more on this,the paper introducing FPCis really readable - I highly recommend it!

fpc provides aWriter and aReader, following the pattern set bythe Go standard library's compression packages. The Writer wraps anio.Writer that you want to write compressed data into, and the Readerwraps an io.Reader that you want to read compressed data out of.

Since FPC encodes streams of float64s, they impose some additionalexpectations on callers: when callingReader.Read(p []byte) orWriter.Write(p []byte), the length ofp must be a multiple of 8,to match the expectation that the bytes represent a stream of 8-bytefloat64s.

In addition, utility methods are provided:Reader has aReadFloats(fs []float64) (int, error) method which will read bytesfrom its underlying source, parse them as float64s, put them infs,and return the number of float64s it placed in fs. When it reaches theend of the compressed stream, it will return0, io.EOF.

Similarly,Writer has aWriteFloat(f float64) error method whichwrites a single float64 to the compressed stream.

In benchmarks on a fairly vanilla laptop, reading or writing from anin-memory stream,fpc is able to encode at about 1.2 gigabytes persecond, and it can decode at about 0.9 gigabytes persecond. Benchmarks can be run on your own hardware withgo test -bench "Read|Write" ..