- Notifications
You must be signed in to change notification settings - Fork0
yuhexiong/kafka-data-pipeline-structured-flink-java
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
(also provided Traditional Chinese version documentREADME-CH.md.)
Data pipeline written by flink to transfer Kafka to Kafka, Doris and also merge the two data sources.
- Platform: JDK 11
- Build Tool: Apache Maven v3.9.6
- Data Processing Framework: Flink v1.18.1
Use IntelliJ IDEA
Transfer all Product messages fromtopic-source in Kafka (localhost:9092) totopic-sink in Kafka (localhost:9092).
- Kafka Topic
topic-sourceMessage
{"id":"12345","name":"Wireless Mouse","category":"Electronics","manufacturer":"TechCorp","description":"A sleek, ergonomic wireless mouse with advanced optical tracking.","cost":29.99}Convert all message intopic-product in Kafka (localhost:9092) and insert it into the Doris (localhost:9030) databasedatabase.product.
- Kafka Topic
topic-productMessage
{"id":"12345","name":"Wireless Mouse","category":"Electronics","manufacturer":"TechCorp","description":"A sleek, ergonomic wireless mouse with advanced optical tracking.","cost":29.99}- Doris Table
database.product
| id | name | category | manufacturer | description | cost ||---------|-----------------|-------------|---------------|-----------------------------------|-------|| 12345 | Wireless Mouse | Electronics | TechCorp | A sleek, ergonomic wireless mouse | 29.99 |Combine messagetopic-product in Kafka (localhost:9092) with message fromtopic-sale. Then, transfer the resulting data into the Doris (localhost:9030) databasedatabase.sale_report.
- Kafka Topic
topic-productMessage
{"id":"12345","name":"Wireless Mouse","category":"Electronics","manufacturer":"TechCorp","description":"A sleek, ergonomic wireless mouse with advanced optical tracking.","cost":29.99}- Kafka Topic
topic-saleMessage
{"id":"A98765","productId":"12345","unit":3,"unitPrice":49.99,"totalPrice":149.97,"saleDate":"2024-11-28"}- Doris Table
database.sale_report
| sale_id | product_id | unit | unit_price | total_price | sale_date | product_name | product_unit_cost | profit ||-----------|------------|------|------------|-------------|----------------------|---------------|-------------------|--------|| A98765 | 12345 | 3 | 49.99 | 149.97 | 2024-11-28T08:00:00 | Wireless Mouse| 29.99 | 60.00 |About
Data pipeline from Kafka to Kafka, Doris and Doris to Kafka using Flink Java.
Topics
Resources
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
No releases published
Packages0
No packages published