Repository files navigation

MemQ is a new PubSub system that augments Kafka at Pinterest. It uses a decoupled storage and serving architecture similar to Apache Pulsar and Facebook Logdevice; however, it relies on a pluggable replicated storage layer i.e. Object Store / DFS / NFS for storing data. The net result is a PubSub system that:

• Handles GB/s traffic
• Independently scales, writes, and reads
• Doesn’t require expensive rebalancing to handle traffic growth
• Is 90% more cost effective than our Kafka footprint

Checkout ourGetting Started Guide

MemQ can be a valuable PubSub choice for the following use case families:

• Large scale data ingestion
• Bulk uploads
• Near Real Time Analytics

Detailed architecture docs can be foundhere

Please checkout theoriginal MemQ blog for details and background.

Ambud Sharma

Ping Min-Lin

Seecontributors

MemQ distributed underApache License, Version 2.0