- Notifications
You must be signed in to change notification settings - Fork2
Change Data Capture with Debezium & PostgreSQL. This repository contains the demo Docker Compose file for setting up and running the whole CDC infrastructure (Kafka Connect, Debezium, PostgreSQL etc.)
License
rbiedrawa/cdc-postgres
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This repository contains the demo Docker Compose (docker-compose.yml) file for setting up andrunning the whole CDC infrastructure (Kafka Connect, Debezium, PostgreSQL etc.)
Change Data Capture, or CDC, is an older term for a system that monitors and captures the changes in data so that othersoftware can respond to those changes. Data warehouses often had built-in CDC support, since data warehouses need tostay up-to-date as the data changed in the upstream OLTP databases.
--https://debezium.io/documentation/faq/#what_is_change_data_capture
The following components are available when you run the whole infrastructure:
- Docker
cd dockerdocker compose up -ddocker compose ps# NAME SERVICE STATUS PORTS# adminer adminer running 0.0.0.0:7775->8080/tcp, :::7775->8080/tcp# kafka kafka running 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:9101->9101/tcp, :::9101->9101/tcp# kafka-connect kafka-connect running (starting) 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp# kafka-connect-ui kafka-connect-ui running 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp# kowl kowl running 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp# postgres postgres running 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp, 0.0.0.0:6532->6532/tcp, :::6532->6532/tcp# schema-registry schema-registry running 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp# zookeeper zookeeper running 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 2888/tcp, 3888/tcp
Run below command or open your web browser and go to Kafka Connect UIpage toverify cdc connector status.
curl http://localhost:8083/connectors/postgres-cdc-demo| jq# {# "name": "postgres-cdc-demo",# "config": {# "connector.class": "io.debezium.connector.postgresql.PostgresConnector",# "database.user": "postgres",# "database.dbname": "demo_db",# "database.hostname": "postgres",# "tasks.max": "1",# "database.password": "postgres",# "name": "postgres-cdc-demo",# "database.server.name": "DockerServer",# "database.port": "5432"# },# "tasks": [# {# "connector": "postgres-cdc-demo",# "task": 0# }# ],# "type": "source"# }
***Docker Compose file contains a setup that will automatically install a Kafka Connectplugin (debezium/debezium-connector-postgresql) and deploy Source Connector(postgres-cdc-demo).
Open your web browser and go toKowl UI then check the contentofDockerServer.public.demo_table topic.
Run the following command to insert more data.
docker-composeexec postgres bash -c"export PGPASSWORD='postgres'; psql -h 'postgres' -U 'postgres' -d 'demo_db'"INSERT INTO demo_table(id, message) VALUES (3,'Hello World 3');INSERT INTO demo_table(id, message) VALUES (4,'Hello World 4');
When you're done, stop Docker containers by running.
docker compose down -v
| Name | Endpoint |
|---|---|
Kafka Connect | http://localhost:8088/ |
Kafka Connect UI | http://localhost:8000/ |
Kowl UI | http://localhost:8080/ |
Adminer - Demo Table view (username: cdcdemo, password: cdcdemo) | http://localhost:7775/?pgsql=postgres&username=cdcdemo&db=demo_db&ns=public&select=demo_table |
Schema-registry | http://localhost:8081/ |
- Kafka Docker Images
- Debezium connector for PostgreSQL
- Debezium docker images - PostgreSQL
- Confluent Hub
- PostgreSQL
- Adminer
- Kafka Connect UI
- cloudhut/kowl
Distributed under the MIT License. SeeLICENSE for more information.
About
Change Data Capture with Debezium & PostgreSQL. This repository contains the demo Docker Compose file for setting up and running the whole CDC infrastructure (Kafka Connect, Debezium, PostgreSQL etc.)
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.