- Notifications
You must be signed in to change notification settings - Fork43
A PostgreSQL logical decoder output plugin to deliver data as Protocol Buffers, used by Debezium (http://debezium.io). Please log issues in our JIRA athttps://issues.jboss.org/projects/DBZ/issues
License
debezium/postgres-decoderbufs
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A PostgreSQL logical decoder output plugin to deliver data asProtocol Buffers, adapted for Debezium
- The originalDecoderbufs Project on which this is based
- The PostgreSQL Team for addinglogical decoding support
This code depends on the following libraries and requires them for compilation:
- PostgreSQL 9.6+
- Protobuf-c 1.2+ - used for data serialization
- PostGIS 2.1+ - used for Postgres geometric types support
postgres-decoderbufs has to be built from source after installing required dependencies. The required dependencies are first PostgreSQL(for pg_config), PostgreSQL server development packages, protobuf-c for the Protocol Buffer support and some PostGIS development packages.
# Core build utilitiesapt-get update&& apt-get install -f -y software-properties-common build-essential pkg-config git postgresql-server-dev-9.6# PostGIS dependencyapt-get install -f -y libproj-dev liblwgeom-dev# Protobuf-c dependency (requires a non-stable Debian repo)add-apt-repository"deb http://ftp.debian.org/debian testing main contrib"&& apt-get updateapt-get install -y libprotobuf-c-dev=1.2.1-1+b1
When updating the ProtoBuf definition, also install the ProtoBuf C compiler:
apt-get install -y protobuf-c-compiler=1.2.*The above are taken from the Debeziumcontainer images.
You just need to make sure the above software packages (or some flavour thereof) are installed for your distro.Note that the last step from the above sequence is only required for Debian to be able to installlibprotobuf-c-dev:1.2.1
If you have all of the above prerequisites installed, clone this git repo to build from source:
git clone https://github.com/debezium/postgres-decoderbufs.gitcd postgres-decoderbufsThis is only needed after changes to the ProtoBuf definition (_proto/pg_logicaldec.proto):
cd protoprotoc-c --c_out=../src/proto pg_logicaldec.protocd ..
Commit the generated files to git then.
If you have multiple Postgres versions installed, you can select which version to install decoderbufs into by altering your$PATH to point to the right version.Thenmake andmake install for each version. Here is an example:
# Install for Postgres 9.6 if I have multiple local versionsexport PATH=/usr/lib/postgresql/9.6/bin:$PATHmakemake install
Once the extension has been installed you just need to enable it and logical replication in postgresql.conf:
# MODULESshared_preload_libraries ='decoderbufs'# REPLICATIONwal_level = logical# minimal, archive, hot_standby, or logical (change requires restart)max_wal_senders = 8# max number of walsender processes (change requires restart)wal_keep_segments = 4# in logfile segments, 16MB each; 0 disables#wal_sender_timeout = 60s # in milliseconds; 0 disablesmax_replication_slots = 4# max number of replication slots (change requires restart)
In addition, permissions will have to be added for the user that connects to the DB to be able to replicate. This can be modified inpg_hba.conf like so:
local replication <youruser> trusthost replication <youruser> 127.0.0.1/32 trusthostreplication<youruser> ::1/128 trust
And restart PostgreSQL.
-- can use SQL for demo purposesselect*from pg_create_logical_replication_slot('decoderbufs_demo','decoderbufs');-- DO SOME TABLE MODIFICATIONS (see below about UPDATE/DELETE)-- peek at WAL changes using decoderbufs debug mode for SQL consoleselect datafrom pg_logical_slot_peek_changes('decoderbufs_demo',NULL,NULL,'debug-mode','1');-- get WAL changes using decoderbufs to update the WAL positionselect datafrom pg_logical_slot_get_changes('decoderbufs_demo',NULL,NULL,'debug-mode','1');-- check the WAL position of logical replicatorsselect*from pg_replication_slotswhere slot_type='logical';
If you're performing an UPDATE/DELETE on your table and you don't see results for those operations from logical decoding, make sure you have setREPLICA IDENTITY appropriately for your use case.
The binary format will be consumed by the Debezium Postgres Connector.
The following table shows how current PostgreSQL type OIDs are mapped to which decoderbuf fields:
| PostgreSQL Type OID | Decoderbuf Field |
|---|---|
| BOOLOID | datum_boolean |
| INT2OID | datum_int32 |
| INT4OID | datum_int32 |
| INT8OID | datum_int64 |
| OIDOID | datum_int64 |
| FLOAT4OID | datum_float |
| FLOAT8OID | datum_double |
| NUMERICOID | datum_double |
| CHAROID | datum_string |
| VARCHAROID | datum_string |
| BPCHAROID | datum_string |
| TEXTOID | datum_string |
| JSONOID | datum_string |
| XMLOID | datum_string |
| UUIDOID | datum_string |
| TIMESTAMPOID | datum_string |
| TIMESTAMPTZOID | datum_string |
| BYTEAOID | datum_bytes |
| POINTOID | datum_point |
| PostGIS geometry | datum_point |
| PostGIS geography | datum_point |
File bug reports and feature requests usingDebezium's JIRA and thepostgresql-connector component
About
A PostgreSQL logical decoder output plugin to deliver data as Protocol Buffers, used by Debezium (http://debezium.io). Please log issues in our JIRA athttps://issues.jboss.org/projects/DBZ/issues
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Languages
- C99.0%
- Makefile1.0%