RabbitMQ series

4.2.x

Operating system (distribution) used

macOS

How is RabbitMQ deployed?

Community Docker image

What would you like to suggest for a future version of RabbitMQ?

Streaming incurs considerable cross-AZ replication cost. Stream replication in RabbitMQ (and Apache Kafka) make identical copies on replica members for the sake of durability and availability. In the Kafka world, blog posts and "disk-less Kafka" KIPs cite cross-AZ replication as a major cost. (Aiven blog,KIP-1150 "Disk-less Topics",KIP-1176 "Slack's KIP",Confluent blog.)

We could add an opt-in feature to compress replication traffic to make it possible to reduce this cost. This option would trade lower traffic costs for lower max-throughput and higher latencies. Osiris replica-readers can compress each chunk before sending it to the Osiris replica. This necessary changes torabbitmq/osiris are small, we we'd need even smaller accompanying changes in the server. Here's a proof-of-concept change:rabbitmq/osiris@ab18236. This is based onZstandard compression at level 3 (Erlang/OTP default, a reasonable tradeoff). Zstd is considered the currentPareto front for compression algorithms: it balances the least costly tradeoffs (CPU time) with the most reward (compression factor). Zstd is included in the Erlang standard library since OTP 28.

Initial testing

For realistic-ish bodies we use the--json-body feature ofperf-test. (Sidenote: this would be a nice addition tostream-perf-test as well - could be contributed separately.)

% perf-test -sq -u sq -x 1 -y 0 --json-body --time 60# compressionid: test-150059-817, sending rate avg: 54157 msg/s# baselineid: test-145319-916, sending rate avg: 75478 msg/s

Max potential throughput drops in this scenario by 28.24%. Keep in mind this is AMQP 0-9-1 without publisher confirms.

In another scenario we keep the ingress consistent and, by measuring bytes sent for replication, we see an overall reduction for replication of 32.5%.

% perf-test -sq -u sq -x 1 -y 0 --json-body --size 1000 --rate 30000 --time 60# compression1_545_106_008 bytes# baseline2_288_388_489 bytes

I measured this by adding a counter toosiris_log for bytes replicated.

While not shown here, commit and end-to-end latency seems to increase by some small but significant factor, say 30% (eyeballing it). Further testing is needed.

Configuration

# rabbitmq.conf example:stream.compression.algorithm = zstd# prefer lower CPU cost for larger size:stream.compression.level = -7

Compression algorithm and maybe also compression arguments could be configured in Cuttlefish config. Users serious about throughput would not enable compression but lower-throughput users aiming for cost efficiency could consider it. The configuration for this applies per-writer-node and can be changed - a cluster could even run a mixed set of compression configurations (during the rollout of a config change for example).

Algorithms

zstd is a nice choice for a new feature like this since it's now included in the standard library and can be tuned. By default we could also include the algorithms available fromzlib. And we could consider making the interface pluggable so that other algorithms could be introduced via NIFs. LZ4 and Snappy are popular algorithms that trade faster compression times for lower compression yield - these are not provided as BIFs in Erlang but could be considered as plugins backed by NIFs.