postgres-ai/database-lab-enginePublic

NotificationsYou must be signed in to change notification settings
Fork68
Star2.4k

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.

License

Apache-2.0 license

2.4k stars 68 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
.gitlab/merge_request_templates		.gitlab/merge_request_templates
assets		assets
config		config
src		src
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
Dockerfile		Dockerfile
LICENSE.txt		LICENSE.txt
Makefile		Makefile
README.md		README.md

Repository files navigation

Database Lab - Thin clones for faster development

Boost your development process

![Joe Demo](./assets/db-lab.png)

Install

Currently, there are no ready-to-use binaries or a Docker image. The setupis to be done using the source code.

Get the source code:git clone https://gitlab.com/postgres-ai/db-lab.git.
Golang is required.
- Ubuntu: In some cases, standard Ubuntu package might not work. Seehttps://gitlab.com/postgres-ai/db-lab/snippets/1918769.
- On macOS:brew install golang
Install ZFS (Ubuntu:https://gitlab.com/postgres-ai/db-lab/snippets/1918768).

ZFS Store

Create a ZFS store with a clone ofthe production Postgres database (e.g. using WAL-E, WAL-G or Barman archive).
Shutdown Postgres, create a new ZFS snapshot(sudo zfs snapshot -r zpool@db_state_1) and remember its name. It willbe needed for further configuration (initialSnapshot option inconfig/config.yml).
Start Postgres.

Run

Deploy DB Lab instance in your infrastructure. You would need to:

Createconfig/config.yml (see sample inconfig/).
Buildmake all and run DB Lab with some token for REST API authorization./bin/dblab -v some-token (or, with log:./bin/dblab -v some-token 2>&1 | tee -a dblab.log).

REST API

Instance statuses:

OK - instance functions well.
WARNING - still functional, but have some problems, e.g. disk space shortage, insecure connection (upcoming MRs).

Clone statuses:

OK - clone is ready to accept postgres connections.
CREATING - clone is being created.
DELETING - clone is being deleted.
FATAL - fatal error happened (details in status message).

Basic models:

Clone{  id: "xxx",  status: {    code: "OK",    message: "Database is ready"  },  project: "proj1"  snapshot: "" (optional, ID or timestamp)  db: {    username: "postgres"    password: "" (never set on DB Lab side)    host: "xxx",    port: "xxx",    connStr: "xxx"  },  protected: true  deleteAt: "" (timestamp),  name: "",  username: "",  createdAt: ""}Error{  code: "NOT_ENOUGH_DISK_SPACE",  name: "Not enough disk space",  hint: "Stop idle clones, change snapshot policy or increase disk size"}

REST API:

Get DB Lab instance status and list clonesGET /statusResponse:{  status: {    code: "OK",    message: "DB Lab is ready"  },  disk: {    size: 1000.0, (bytes)    free: 1200.0  },  expectedCloningTime: 8.0, (secondss)  numClones: 10,  clones: [{    id: "id"    status: {      code: "OK",      message: "Database is ready"    },    ...  }, ... ], snapshots: [{   id: "xxx",   timestamp: "123" }, ... ]}Create a clonePOST /clone/Request:{  project: "proj1",  snapshot: (optional): "",  db: {    username: "xxx",    password: "xxx"  }  username: "xxx",  name: "xxx"}Response:{  id: "xxx"}Update a clonePATCH /clone/:idReset a clonePOST /clone/:id/resetDelete a cloneDELETE /clone/:idGet status of a cloneGET /clone/:idResponse:{  id: "xxx",  status: {    code: "OK",    message: "Database clone is ready"  },  cloneSize: 1000.0,  cloningTime: 5,  project: "xxx",  snapshot: "xxx",  db: {    username: "xxx",    host: "xxx",    port: "xxx",    connStr: "xxx"  },  protected: true,  deleteAt: "",  name: "xxx",  username: "xxx",  createdAt: "123"}