Druid is acolumn-oriented,open-source,distributeddata store written inJava. Druid is designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.[3] The name Druid comes from theshapeshiftingDruid class in manyrole-playing games, to reflect that the architecture of the system can shift to solve different types of data problems.
Druid is commonly used inbusiness intelligence-OLAP applications to analyze high volumes ofreal-time and historical data.[4] Druid is used in production by technology companies such asAlibaba,[4]Airbnb,[4]Nielsen,[4]Cisco,[5][4]eBay,[6]Lyft,[7]Netflix,[8]PayPal,[4]Pinterest,[9]Reddit,[10]Twitter,[11]Walmart,[12]Wikimedia Foundation[13] andYahoo.[14]
Druid was started in 2011 by Eric Tschetter, Fangjin Yang, Gian Merlino and Vadim Ogievetsky[15] to power the analytics product of Metamarkets. The project was open-sourced under the GPL license in October 2012,[16][17][18] and moved to an Apache License in February 2015.[19][20]
Fully deployed, Druid runs as a cluster of specialized processes (called nodes in Druid) to support afault-tolerant architecture[21] where data is stored redundantly, and there is no single point of failure.[22] The cluster includes external dependencies for coordination (Apache ZooKeeper), metadata storage (e.g.MySQL,PostgreSQL, orDerby), and a deep storage facility (e.g.HDFS, orAmazon S3) for permanent data backup.
Client queries first hit broker nodes, which forward them to the appropriate data nodes (either historical or real-time). Since Druid segments may be partitioned, an incoming query can require data from multiple segments and partitions (orshards) stored on different nodes in the cluster. Brokers are able to learn which nodes have the required data, and also merge partial results before returning the aggregated result.
Operations relating to data management in historical nodes are overseen by coordinator nodes. Apache ZooKeeper is used to register all nodes, manage certain aspects of internode communications, and provide for leader elections.
In 2019, researchers compared the performance ofHive,Presto, and Druid using a denormalizedStar Schema Benchmark based on theTPC-H standard. Druid was tested using both a “Druid Best” configuration using tables with hashed partitions and a “Druid Suboptimal” configuration which does not use hashed partitions.[23]
Tests were conducted by running the 13 TPC-H queries using TPC-H Scale Factor 30 (a 30GB database), Scale Factor 100 (a 100GB database), and Scale Factor 300 (a 300GB database).
| Scale Factor | Hive | Presto | Druid Best | Druid Suboptimal |
|---|---|---|---|---|
| 30 | 256s | 33s | 2.09s | 3.21s |
| 100 | 424s | 90s | 6.12s | 8.08s |
| 300 | 982s | 452s | 7.60s | 20.02s |
Druid performance was measured as at least 98% faster than Hive and at least 90% faster than Presto in each scenario, even when using the Druid Suboptimized configuration.
