An end to end anime recommendation system based on data scrapped from myanimelist.net

• UpdatedMar 27, 2022
• Python

ecommerce GCP Streaming pipeline ― Cloud Storage, Compute Engine, Pub/Sub, Dataflow, Apache Beam, BigQuery and Tableau; GCP Batch pipeline ― Cloud Storage, Dataproc, PySpark, Cloud Spanner and Tableau

• UpdatedMar 9, 2022
• Python

Playground for Apache Beam and Scio experiments, driven by real-world use cases.

• UpdatedOct 1, 2020
• Scala

ETL pipeline on GCP

• UpdatedAug 14, 2018
• Jupyter Notebook

Export Dialogflow conversation logs to BigQuery with masking PII using DLP API

• UpdatedApr 9, 2019
• JavaScript

Trigger a Dataflow job when a file is uploaded to Cloud Storage using a Cloud Function

• UpdatedDec 13, 2019
• Python

A data pipeline to ingest, process, store storm events datasets so we can access them through different means.

• UpdatedApr 7, 2021
• Jupyter Notebook

GCP Dataflow pipeline with BigQuery as source and side input

• UpdatedAug 9, 2018
• Python

This repo is dedicated for GCP data engineering concepts: BigTable, BigQuery, DataFlow, PubSub, DataProc Spark on GCP. Apache Beam, Apache AirFlow

• UpdatedOct 13, 2020
• Java

GCP Dataflow pipeline with mapreduce in python

• UpdatedAug 11, 2018
• Python

Boilerplate for batch-processing scenarios' orchestration. Apache Airflow w/ realistic product analytics use case

• UpdatedMar 20, 2020
• Python

Apache beam sandbox w/ Dataflow for 10+ use cases

• UpdatedMar 20, 2020
• Python

Big Data ETL Pipeline for ASL-to-Text (Computer Vision), using Apache Beam on GCP Dataflow

• UpdatedFeb 23, 2021
• Jupyter Notebook

• UpdatedOct 15, 2020
• Java

Black Friday, the biggest shopping day of the year, presents a unique opportunity for retailers like Walmart to boost sales, attract new customers, and clear inventory. Managing the surge in transaction volumes, understanding customer preferences, and optimizing inventory in real time are critical challenges that require sophisticated data solution

• UpdatedFeb 4, 2024
• Python

This repo is to demonstrate rag data processing pipeline using dataflow flex templates

• UpdatedJan 2, 2025
• Python

GCP Dataflow pipeline in python

• UpdatedAug 6, 2018
• Java

• UpdatedJan 29, 2019
• Scala

GCP Space Shepherd - service for monitoring Google DataFlow executions

• UpdatedAug 31, 2024
• Java

Sample projects to explore various Google Cloud service-offerings and architecture approaches

• UpdatedJul 2, 2020