Repository files navigation

This project demonstrates a partitioned signup flow based on a primer from"Designing Data-Intensive Applications" book by Martin Kleppmann@martinkl (Thank you!): users sign up at Account service which requires a username. There are so many people willing to register,that a single PostgreSQL database can't hold all account records, but three servers are enough for this hypothetical service load.Therefore we should split (partition) user accounts on three databases and make surea username is unique across all of them.

The idea is to write signup requests intoaccount.signup_request Kafka topic which is partitioned by username.Hence all attempts to claim username Bob will be stored in the same Kafka partition based onconsistent hashing algorithm.For example,{username: Bob, request_id: 13rUw7cUfrGO9Go9xbZearzuuAu} message is written tohash('Bob') % partitions_count partition.Since we have three PostgreSQL instances, we need to splitaccount.signup_request topic into three partitions (0, 1, 2).

Each signup-server process sequentially reads Kafka messages from its own partition andstores user accounts in its PostgreSQL database.If Bob username exists in Postgres, the program emits a failure message toaccount.signup_response topic.Otherwise, Bob's account is created and success message is written to the topic. For example:

• {username: Bob, success: false, request_id: 13rVCgpmD0UgKH6zNHdfcPG63Df}
• {username: Bob, success: true, request_id: 13rUw7cUfrGO9Go9xbZearzuuAu}

Note,request_id is generated by a client who sends signup requests.Request IDs are needed to deduplicate messages. IDs are kept for a certain duration(until a message ages out) or limited by storage size. I have not tried deduplication in this project,although I was curious what storage will be the way to go.For instance, Segment shared how they leverageRocksDB inDelivering Billions of Messages Exactly Once,while CockroachDB uses RocksDB as aStorage Layer.Now I know what to try next!

A word about artificial keys in PostgreSQL.UUID v4 is a common choice to generate a random unique ID for an entity, e.g., invoice ID.Indexing of highly randomized values cause write amplification, so INSERTs become slow.InSQL Keys in Depth the authorshows the superior performance of UUID v1 algorithm which producesnode MAC address + timestampmonotonically increasing values.In this demo I usedK-Sortable Unique IDentifier (timestamp + randomly generated payload)to assign user IDs in PostgreSQL.Segment goes into KSUID details inA Brief History of the UUID.

Let's run three PostgreSQL docker containers on 5433, 5434, 5435 ports withaccount dbs created.We also need Kafka which will haveaccount.signup_request andaccount.signup_response topicswith 3 partitions and 1 replica. Docker Compose will take care of that. The only caveat is thatyou should setKAFKA_ADVERTISED_HOST_NAME.

$cd ./docker/$ KAFKA_ADVERTISED_HOST_NAME=$(ipconfig getifaddr en0) docker-compose up

Install dependencies usingdep package manager and build all commands.

$ dep ensure$ make build

Create PostgreSQL schema in every db with schema command.

$ ./schema -pgport=5433&& ./schema -pgport=5434&& ./schema -pgport=5435

Run three signup-server for eachaccount.signup_request partition to process signup requests.

$ ./signup-server -partition=0 -pgport=5433$ ./signup-server -partition=1 -pgport=5434$ ./signup-server -partition=2 -pgport=5435

Finally, run signup-ctl and type usernames to send signup requests.Note, both programs have a debug mode to show more logs.

$ ./signup-ctlbob2:0 13rUw7cUfrGO9Go9xbZearzuuAu bob ✅alice2:1 13rUwm0PI5tMT3FEx4OwW905yWw alice ✅john2:2 13rUyyeODTy1GDdvRhtgLjC5sbG john ✅lloyd0:0 13rV46Yp6Ng0uEPuUmsF51S5pi2 lloyd ✅aaron0:1 13rV4lXEoSQrcSpaFYamWMBaDWt aaron ✅peter1:0 13rVCAFeRJxK671227gtFSq069F peter ✅bob2:3 13rVCgpmD0UgKH6zNHdfcPG63Df bob ❌lloyd0:2 13rVEyTTAh7P76aKQ3ZEomDEzqX lloyd ❌sam2:4 13rVFmwyaw2u5UXXNKIKMplycqb sam ✅

Signup responses are printed inpartition_id:offset request_id username format.As you can see, bob successfully registered and the attempt to sign up as bob again failed.

2:0 13rUw7cUfrGO9Go9xbZearzuuAu bob ✅...2:3 13rVCgpmD0UgKH6zNHdfcPG63Df bob ❌

To run tests you will need Postgres and test env variables set up.

$ make docker_run_postgres$ maketest