Repository files navigation

A simple Spotify scrobbler. Gets your listening history from Spotify, saves it to a database and creates a weekly backup in Google Drive.

• Free of charge* - uses the AWS free tier and the Google Drive API, which is free to use
• Utilities to get refresh tokens from both Google and Spotify
• Easily customizable
• Listening history export to Google Drive

Spotify's API only exposes thelast 50 songs you've listened to.

This project seeks to provide an easy and free solution to saving your Spotify listening history in an accessible place (Google Drive) where you can retrieve and analyze it quickly.

Other than that, you can of course use everything here as a starting point/guideline to create something else with Spotify, AWS, Google Drive and Serverless.

• This project makes use of two AWS Lambda functions, one for getting your history from Spotify and one for creating backups in Google Drive.

• Unlike Last.FM, Spotify apparently counts as song aslistened to when you listen to it for"over 30 seconds". The exact behaviour of how Spotify counts a song aslistened to is not clear to me, but it seems like 30 seconds are the minimum.

• By default, the history Lambda (scrobbler) isscheduled to get the history from Spotify at an hourly interval. With this interval, most "regular" users who listen through a song will have their full listening history captured. Assuming a very low average song duration of ~2 minutes would mean that one could listen to max. 30 songs per hour. As Spotify keeps track of the last 50 songs you've listened to, this interval would cover the entire hour. However, you may change the schedule.

• By default, the backup Lambda isscheduled to run weekly at the start of the week (Monday at 12:30 a.m.). A week is defined according to theISO 8610 standard and thus starts on Monday.

• By default,items in the database expire after 1 month since they have already been backed up and are not needed anymore.

• You might want to adjust the region inserverless.yml - provider.region if you don't live near Frankfurt (default iseu-central-1).Available regions.

You can customize the backup, schedules, item expiration and much more.Customization guide.

• An AWS account
• A Spotify account
• serverless >= 3
• node >= v14.17.4
• Docker (optional)

1. Fork and/or clone this repository and install dependencies:

git clone git@github.com:eegli/spotify-history.gitcd spotify-historyyarn

2. Spotify setup -Create a Spotify application - app status "development" is fine - and set the redirect URL tohttp://localhost:3000.
3. In the root directory, create a folder named.secrets (notice the dot!)
4. Create a file namedcredentials_spotify.json and copy the template below. Insert your client id and client secret. Your Spotify secrets file should look like this:

{"clientId":"<your-client-id>","clientSecret":"<your-client-secret>"}

5. Google Drive setup -Follow the quickstart guide to create a Google Cloud project and enable the Drive API. When asked to configure the consent screen, your publishing status should betesting. You will need to manually add the Google account who's drive you want to use under "Test users". In the end, you should be prompted to download your OAuth client credentials for your newly created desktop client as a JSON file.
6. Download the credentials file, rename it tocredentials_google.json and put it in the.secrets folder. It should look like this:

{"installed": {"client_id":"blablabla","project_id":"spotify-history-32as4","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"blablabla","redirect_uris": ["urn:ietf:wg:oauth:2.0:oob","http://localhost"]  }}

Almost done!

7. Run the following command and follow the steps. This will create atoken_spotify.json file in the.secrets folder containing your long-lived Spotify refresh token.KEEP THIS FILE SECURE!

npm run token:spotify

8. Run the following command and follow the steps. This will create atoken_google.json file in the.secrets folder containing your long-lived Google Drive refresh token.KEEP THIS FILE SECURE!

npm run token:google

9. Done!

This project includes both a staging and production environment. By default,the schedules are only enabled in production in order to save quota. Thestaging version is meant to be deployed butinvoked manually only.If you wish to enable the schedule on staging as well, changeserverless.yml:

custom:scheduleEnabled:prod:truestg:true# Schedule enabled on staging

Keep in mind that this will double the calls made to Lambda and DynamoDB!

In order to deploy the production version, run:

npm run prod:deploy

You can deploy the staging version as well:

# Deploy everythingnpm run stg:deploy# Deploy functions onlynpm run stg:deploy:historynpm run stg:deploy:backup

Again, the staging functions areNOT scheduled by default as they are meant to be invoked manually:

# Get history from Spotify and save to DynamoDBnpm run stg:invoke:history# Create backup in Google Drivenpm run stg:invoke:backup

To check the logs of your Lambda functions, either go to the AWS CloudWatch dashboard or retrieve them in your console.

Example: Getting the logs for production in the last 24h

sls logs -f spotify-history -s prod --startTime 1dsls logs -f spotify-history-backup -s prod --startTime 1d

More info about logging with Serverless.

By default, these are the song properties that are saved to the database (and backup):

interfaceDynamoHistoryElement{name:string;id:string;playedAt:string;}

If you want to save other song properties, simply change this interface insrc/config/types.ts and TypeScript will show you where you'll need to make adjustments. Obviously, it makes sense to at least store the timestamp of when the song was played (playedAt) and its id (id).

By default, items in DynamoDB are set to expire after 1 month. If you wish to disable this, set the TTL specification inserverless.yml tofalse (or remove the implementation altogether for a cleaner codebase).

TimeToLiveSpecification:AttributeName:'expire_at'Enabled:false

If you want to specify a different TTL, change thedynamoExpireAfter default insrc/config/defaults.ts

If you want to change the backup schedule, e.g. running it daily or monthly, you'll need to adjust the cron expression inserverles.yml. Here are some resources regarding cron jobs.

• AWS EventBridge schedule expressions (cron)
• Cron expression tester

⚠️ Keep in mind that the backup schedule, item expiration and time range to retrieve the items for the backup are logically connected!⚠️

If you change the backup schedule, you'll also need to change the time range of the backup and, most likely, the item TTL as shown above.

// Example: Include history from last monthconstdefaults:Readonly<Defaults>={dynamoExpireAfter:[2,'months'],// Extend the expiration datebackupRange:[1,'month'],// Extend the backup range  ...};

Update the stage and production folder names insrc/config/defaults.ts.

Note that, for security reasons, the backup handler only has access to folders and files it has created itself (seeOAuth 2.0 scopes andscripts/google.ts). For simplicity, the backup folder is created at the root of your Google Drive.

For local development and testing the db integration, AWS's official DynamoDB Docker image can be run along with another image that provides a nice GUI for inspecting the tables and items.

1. Start the containers (DynamoDB and GUI):

npm run dynamo:start

2. Migrate the table and seed:

npm run dynamo:migrate

3. If you want to check if everything has been setup correctly, visithttp://localhost:8001/

4. Invoke locally:

   Note thatnpm run local:backup will, despite its naming,still hit the Google Drive API but save the content in a folder separate fromstg andprod (local).

# Gets the history and saves it to local DynamoDBnpm run local:history# Backs up the history to Google Drivenpm run local:backup

The core of this project usesAWS DynamoDB Data Mapper. Unfortunately, this package does not seem to be actively maintaned and is only compatible with the AWS SDK v2. By default,the AWS SDK v2 is included in the Lambda runtime environment, but not the modular version 3. Those are the reasons why it is currently not possible to upgrade this project to use the modular AWS SDK.

• Mocking TS method overloads with Jest
• Amazon DynamoDB DataMapper For JavaScript
• Amazon DynamoDB DataMapper Annotations
• Using the DynamoDB Document Client
• Serverless DynamoDB Local
• TypeScript: adjusting types in reduce function with an async callback

* Serverless uses S3 to store the code of the deployed functions. Technically, S3 is not free. It costsa fraction of a $ per GB, but a deployment takes up so little space, you most likely won't be billed. A full month of testing "cost" me 0.01$ and I was not billed. Be aware that, if you change the schedules, this project may not be "free" anymore!