Apache HBase at Airbnb
15 likes5,945 views
The document discusses Airbnb's data infrastructure and the use of Apache HBase for efficient data management, event logging, and real-time data processing. It highlights the architecture for batch and streaming data workflows, including the integration of various technologies like Spark, Kafka, and Druid. Key features of HBase, its case studies, and the advantages of its stateful computation and scalability within the Hadoop ecosystem are also covered.
1 of 35
Ad
Recommended
More Related Content
What's hot(20)
Similar to Apache HBase at Airbnb(20)
PDF
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...HostedbyConfluentPDF
Paradigmas de procesamiento en Big Data: estado actual, tendencias y oportu...Facultad de Informática UCMParadigmas de procesamiento en Big Data: estado actual, tendencias y oportu...Facultad de Informática UCM
Ad
More from HBaseCon(20)
Ad
Recently uploaded(20)
Apache HBase at Airbnb
- 1.Apache HBase at AirbnbJINGWEI LU, LIYIN TANG, AND JASON ZHANG1
- 2.Data Infrastructure at Airbnb
- 3.EventLogsMySQLDumpsGold ClusterHDFSHiveKafkaSqoopSilver Cluster Spark ClusterSparkReAirAirflow SchedulingS3Presto ClusterAirPalCaravelTableauBatch InfrastructureYarn HDFSHiveYarnJingwei Lu, Liyin Tang, Jason Zhang3
- 4.Streaming at AirbnbEventLoggingMySQLBINLOGClusterHDFSHiveSpinal tapPresto ClusterYarnKafkaHBaseSpark StreamingDatadogDruidKafkaJingwei Lu, Liyin Tang, Jason Zhang4
- 6.StatelessJingwei Lu, Liyin Tang, Jason ZhangComputation SinkSourceDStream DF DF
- 7.StatefulJingwei Lu, Liyin Tang, Jason ZhangComputationSourceDStream DF DFSink1Sink2Sink NState StorageRDD
- 8.Multiple StreamsJingwei Lu, Liyin Tang, Jason ZhangDataFrameSink1ProcessASink2Sink3SinkN…DataFrameSink1ProcessNSink2Sink3SinkN…SourceDStreamAlign by TimeDataFrameDataFrameStateStorageSourceDStream…
- 9.Streaming + BatchJingwei Lu, Liyin Tang, Jason ZhangDataFrameSink1ProcessASink2Sink3SinkN…DataFrameStateStorageSourceDStreamSource…Align by Time…DataFrameSink1ProcessASink2Sink3SinkN…
- 11.AirStream ArchitectureJingwei Lu, Liyin Tang, Jason ZhangSourcesStream #1 Stream #NHive TablesHBaseTablesVirtual Table Views for ComputationSinks…Customized ComputationSpark SQLSimple ConfigHBase ServicesStreamingSourcesDruid
- 12.AirStream ArchitectureJingwei Lu, Liyin Tang, Jason ZhangSourcesStream #1 Stream #NHive TablesHBaseTablesVirtual Table Views for ComputationSinks…Customized ComputationSpark SQLHBase ServicesStreamingSourcesDruidSame Computationfor Batchprocessing
- 13.Stateful
- 14.Jingwei Lu, Liyin Tang, Jason ZhangState Store• Merge changes• Provide fast lookup• Fast persistent storage acrossstreaming and batch jobs14
- 15.Why HBaseJingwei Lu, Liyin Tang, Jason ZhangRich FunctionalitiesRich Integration with Hadoop EcoSystemEasy ManagementStrong CommunityReliable and Scalable
- 16.HBase State StoreOperators in AirstreamJingwei Lu, Liyin Tang, Jason Zhang16Full Table ScanSimple AggregationBulk UploadKey/Prefix LookupUpdate
- 17.Jingwei Lu, Liyin Tang, Jason ZhangComputation DAG17Input DataLeft Outer Join ResultKey Lookup
- 18.Jingwei Lu, Liyin Tang, Jason ZhangKey Space Design• Hash partition key spacefor load balance• Composite key for K -> V• Support full key lookup• Prefix lookup supported forall keys used in hashfunctionHash key1 key2 key3Hash based on key prefixHash key1 key2Lookup based on key prefixkey1 = ‘value1’ and key2 = ‘value2’18
- 19.• Partition based on key before write• Use bulk upload for large volume updateWrite PerformanceJingwei Lu, Liyin Tang, Jason Zhang19
- 20.Case StudyJingwei Lu, Liyin Tang, Jason ZhangExperiment realtime feedbackUpdateExperiment AssignmentEventLookupHBasewith TTLBooking EventDruidDatadog20one airstream job
- 22.Realtime Ingestion on HBaseData InfrastructureMySQLAnalytical EventsKafkaSparkStreamingHBaseHDFSPresto/Hive/SparkSourceIngestRealtimeQuerySnapshotBatchQueryJingwei Lu, Liyin Tang, Jason Zhang22
- 23.Access Data in HBaseJingwei Lu, Liyin Tang, Jason ZhangHBaseHive PrestoSparkSQLSparkStreamingBatch Jobs Interactive Query StreamingHDFSSnapshotTable Mapping/Unifed View on realtime data23
- 24.Snapshot & ReseedJingwei Lu, Liyin Tang, Jason ZhangHBase HDFSSnapshot（HFile Links)Bulk Upload24
- 25.Case Study 1: Events IngestionJingwei Lu, Liyin Tang, Jason ZhangKafkatopic…topictopicSparkExecutor1…Executor2ExecutorKHBaseDeDupHDFSRegion1…Region2Region MDailySnapshotRealtimeQueryHivePrestoEventsPartition25
- 26.Case Study 2: Streaming DB ExportKafkaRDSTable1…Spinaltap.Table1…Table2TableNSpinaltap.Table2Spinaltap.TableNSparkExecutor1…Executor2Executor KHBaseRegion1…Region2Region MHDFSRegion1…Region2Region MDaily SnapshotRealtime QueryJingwei Lu, Liyin Tang, Jason Zhang26
- 27.Case Study: Streaming DB ExportRows CF: Colums Version Value<ShardKey><DB_TABLE_#1><PK_a=A> id Fri May 19 00:33:19 2016 101<ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 19 00:33:19 2016 San Francisco<ShardKey><DB_TABLE_#1><PK_a=A> city Fri May 10 00:34:19 2016 New York<ShardKey><DB_TABLE_#2><PK_a=A’> id Fri May 19 00:33:19 2016 1Jingwei Lu, Liyin Tang, Jason Zhang27
- 28.Case Study: Streaming DB ExportTXN 1Commit_TS:101…TXN 2Commit_TS:102TXN 3Commit_TS:103TXN NCommit_TS:N’Binlog OrderJingwei Lu, Liyin Tang, Jason Zhang28
- 29.Case Study: Streaming DB ExportTXN 1Commit_TS:101…TXN 2Commit_TS:103TXN 3Commit_TS:102TXN NCommit_TS:N’NTPBinlog OrderJingwei Lu, Liyin Tang, Jason Zhang29
- 30.Case Study: Streaming DB ExportTXN 1Commit_TS:101…Binlog OrderTXN 2Commit_TS:103TXN 3Commit_TS:102TXN NCommit_TS:N’Point-in-Time Restore on TS 102Jingwei Lu, Liyin Tang, Jason Zhang30
- 31.Case Study: Streaming DB ExportRows CF: Colums Version Value<ShardKey><DB_TABLE_#1><PK_a=A> id bin100 101<ShardKey><DB_TABLE_#1><PK_a=A> city bin101 San Francisco<ShardKey><DB_TABLE_#1><PK_a=A> city bin102 New York<ShardKey><DB_TABLE_#2><PK_a=A’> id bin100 1Jingwei Lu, Liyin Tang, Jason Zhang31
- 32.Case Study: Streaming DB ExportRows Version (Logical Offset) Value<ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100<ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101<ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103<ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102Jingwei Lu, Liyin Tang, Jason Zhang32
- 33.Case Study: Streaming DB ExportRows Version (Logical Offset) Value<ShardKey><DB_TABLE_#1><2016-05-23 23><100> 100 mysql-bin.00000:100<ShardKey><DB_TABLE_#1><2016-05-23 23><101> 101 mysql-bin.00000:101<ShardKey><DB_TABLE_#1><2016-05-23 23><103> 103 mysql-bin.00000:103<ShardKey><DB_TABLE_#1><2016-05-24 00><102> 102 mysql-bin.00000:102Jingwei Lu, Liyin Tang, Jason Zhang33
- 34.SummaryJingwei Lu, Liyin Tang, Jason ZhangScalable and ReliableRich Stateful ComputationRich Integration with Hadoop EcoSystemEasy Operation
- 35.35
Editor's Notes
- #4: *Disaster recovery*High Slow SLA job isolation
- #15: Slide why Stateful process vs stateless
- #17: Use diagram to show operators
- #22: Realtime ingestion provides fast feedback loop.Advanced monitoring infrastructure Tracking changes instead of full snapshot for RDS dump
- #23: What is the goal of realtime ingestion:*fast feedback loop for experiment to reduce testing cycle*provide realtime view of production database for many offline workload(for example, machine learning)
- #24: Table mapping provide a unified view to access realtime ingested data.
- #25: For snapshot using scan it takes 10-30 minutes per table. This does not scale.Take 10 minutes to do the link and restore. All tables can be accessed afterward.
- #27: Backup based db export restore takes 9 - 12 hours and it is subject to AWS network situation. Long latency and fragile. We just need to track changes and apply to snapshot.Provide near realtime snapshot of db. Unify across mysql and dynamodb