- Notifications
You must be signed in to change notification settings - Fork0
salimt/Spotify-API-Pipeline
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A data pipeline to extract Spotify data from a playlist that is created by students.
Output is a Google Data Studio report, providing insight into the track features and preferences.
It provided a good opportunity to develop skills and experience in a range of tools. As such, project is more complex than required, utilising dbt, airflow, docker and cloud based storage, and usage of localstack for testing.
- Extract data usingSpotify API
- Simulate AWS S3 locally for testing withlocalstack
- Load intoAWS S3
- Copy intoSnowflake
- Transform usingdbt
- CreateGoogle Looker Studio Dashboard
- Orchestrate withAirflow inDocker
- Final output from Google Looker Studio. Linkhere. Note that Dashboard is reading from a static CSV output from Snowflake.
NOTE: This was developed using Windows 10. If you're on Mac or Linux, you may need to amend certain components if issues are encountered.
git clone https://github.com/salimt/Spotify-API-Pipeline.gitcd Spotify-API-Pipeline