50.6. Logical Decoding Output Plugins
An example output plugin can be found in thecontrib/test_decoding subdirectory of the Postgres Pro source tree.
50.6.1. Initialization Function
An output plugin is loaded by dynamically loading a shared library with the output plugin's name as the library base name. The normal library search path is used to locate the library. To provide the required output plugin callbacks and to indicate that the library is actually an output plugin it needs to provide a function named_PG_output_plugin_init. This function is passed a struct that needs to be filled with the callback function pointers for individual actions.
typedef struct OutputPluginCallbacks{ LogicalDecodeStartupCB startup_cb; LogicalDecodeBeginCB begin_cb; LogicalDecodeChangeCB change_cb; LogicalDecodeTruncateCB truncate_cb; LogicalDecodeCommitCB commit_cb; LogicalDecodeAbortCB abort_cb; LogicalDecodeMessageCB message_cb; LogicalDecodeFilterPrepareCB filter_prepare_cb; LogicalDecodePrepareCB prepare_cb; LogicalDecodeCommitPreparedCB commit_prepared_cb; LogicalDecodeAbortPreparedCB abort_prepared_cb; LogicalDecodeFilterByOriginCB filter_by_origin_cb; LogicalDecodeFilterDecodeTxnCB filter_decode_txn_cb; LogicalDecodeShutdownCB shutdown_cb;} OutputPluginCallbacks;typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb); Thebegin_cb,change_cb andcommit_cb callbacks are required, whilestartup_cb,filter_by_origin_cb,truncate_cb, andshutdown_cb are optional. Iftruncate_cb is not set but aTRUNCATE is to be decoded, the action will be ignored.
50.6.2. Capabilities
To decode, format and output changes, output plugins can use most of the backend's normal infrastructure, including calling output functions. Read only access to relations is permitted as long as only relations are accessed that either have been created byinitdb in thepg_catalog schema, or have been marked as user provided catalog tables using
ALTER TABLE user_catalog_table SET (user_catalog_table = true);CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
Any actions leading to transaction ID assignment are prohibited. That, among others, includes writing to tables, performing DDL changes, and callingtxid_current().
50.6.3. Output Modes
Output plugin callbacks can pass data to the consumer in nearly arbitrary formats. For some use cases, like viewing the changes via SQL, returning data in a data type that can contain arbitrary data (e.g.,bytea) is cumbersome. If the output plugin only outputs textual data in the server's encoding, it can declare that by settingOutputPluginOptions.output_type toOUTPUT_PLUGIN_TEXTUAL_OUTPUT instead ofOUTPUT_PLUGIN_BINARY_OUTPUT in thestartup callback. In that case, all the data has to be in the server's encoding so that atext datum can contain it. This is checked in assertion-enabled builds.
50.6.4. Output Plugin Callbacks
An output plugin gets notified about changes that are happening via various callbacks it needs to provide.
Concurrent transactions are decoded in commit order, and only changes belonging to a specific transaction are decoded between thebegin andcommit callbacks. Transactions that were rolled back explicitly or implicitly never get decoded. Successful savepoints are folded into the transaction containing them in the order they were executed within that transaction. A transaction that is prepared for a two-phase commit usingPREPARE TRANSACTION will also be decoded if the output plugin callbacks needed for decoding them are provided. It is possible that the current transaction which is being decoded is aborted concurrently via aROLLBACK PREPARED command. In that case, the logical decoding will be aborted midways.
Note
Only transactions that have already safely been flushed to disk will be decoded. That can lead to aCOMMIT not immediately being decoded in a directly followingpg_logical_slot_get_changes() whensynchronous_commit is set tooff.
50.6.4.1. Startup Callback
The optionalstartup_cb callback is called whenever a replication slot is created or asked to stream changes, independent of the number of changes that are ready to be put out.
typedef void (*LogicalDecodeStartupCB) (struct LogicalDecodingContext *ctx, OutputPluginOptions *options, bool is_init);
Theis_init parameter will be true when the replication slot is being created and false otherwise.options points to a struct of options that output plugins can set:
typedef struct OutputPluginOptions{ OutputPluginOutputType output_type; bool receive_rewrites;} OutputPluginOptions;output_type has to either be set toOUTPUT_PLUGIN_TEXTUAL_OUTPUT orOUTPUT_PLUGIN_BINARY_OUTPUT. See alsoSection 50.6.3. Ifreceive_rewrites is true, the output plugin will also be called for changes made by heap rewrites during certain DDL operations. These are of interest to plugins that handle DDL replication, but they require special handling.
The startup callback should validate the options present inctx->output_plugin_options. If the output plugin needs to have a state, it can usectx->output_plugin_private to store it.
50.6.4.2. Shutdown Callback
The optionalshutdown_cb callback is called whenever a formerly active replication slot is not used anymore and can be used to deallocate resources private to the output plugin. The slot isn't necessarily being dropped, streaming is just being stopped.
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
50.6.4.3. Transaction Begin Callback
The requiredbegin_cb callback is called whenever a start of a committed transaction has been decoded. Aborted transactions and their contents never get decoded.
typedef void (*LogicalDecodeBeginCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn);
Thetxn parameter contains meta information about the transaction, like the time stamp at which it has been committed and its XID.
50.6.4.4. Transaction End Callback
The requiredcommit_cb callback is called whenever a transaction commit has been decoded. Thechange_cb callbacks for all modified rows will have been called before this, if there have been any modified rows.
typedef void (*LogicalDecodeCommitCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
50.6.4.5. Transaction Prepare Callback
The optionalprepare_cb callback is called whenever a transaction which is prepared for two-phase commit has been decoded. Thechange_cb callbacks for all modified rows will have been called before this, if there have been any modified rows.
typedef void (*LogicalDecodePrepareCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr prepare_lsn);
50.6.4.6. Commit Prepared Transaction Callback
The optionalcommit_prepared_cb callback is called whenever a commit prepared transaction has been decoded. Thegid field, which is part of thetxn parameter can be used in this callback.
typedef void (*LogicalDecodeCommitPreparedCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr commit_lsn);
50.6.4.7. Rollback Prepared Transaction Callback
The optionalabort_prepared_cb callback is called whenever a rollback prepared transaction has been decoded. Thegid field, which is part of thetxn parameter can be used in this callback.
typedef void (*LogicalDecodeAbortPreparedCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr abort_lsn);
50.6.4.8. Transaction Abort Callback
The requiredabort_cb callback is called whenever a transaction abort has to be initiated. This can happen if we are decoding a transaction that has been prepared for two-phase commit and a concurrent rollback happens while we are decoding it. It might make sense, even before we commence decoding, in such cases to check if the rollback happened even before we start looking at the changes to completely avoid the decoding of such transactions.
typedef void (*LogicalDecodeAbortCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr abort_lsn);
50.6.4.9. Change Callback
The requiredchange_cb callback is called for every individual row modification inside a transaction, may it be anINSERT,UPDATE, orDELETE. Even if the original command modified several rows at once the callback will be called individually for each row.
typedef void (*LogicalDecodeChangeCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, Relation relation, ReorderBufferChange *change);
Thectx andtxn parameters have the same contents as for thebegin_cb andcommit_cb callbacks, but additionally the relation descriptorrelation points to the relation the row belongs to and a structchange describing the row modification are passed in.
Note
Only changes in user defined tables that are not unlogged (seeUNLOGGED) and not temporary (seeTEMPORARY orTEMP) can be extracted using logical decoding.
50.6.4.10. Truncate Callback
Thetruncate_cb callback is called for aTRUNCATE command.
typedef void (*LogicalDecodeTruncateCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change);
The parameters are analogous to thechange_cb callback. However, becauseTRUNCATE actions on tables connected by foreign keys need to be executed together, this callback receives an array of relations instead of just a single one. See the description of theTRUNCATE statement for details.
50.6.4.11. Origin Filter Callback
The optionalfilter_by_origin_cb callback is called to determine whether data that has been replayed fromorigin_id is of interest to the output plugin.
typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ctx, RepOriginId origin_id);
Thectx parameter has the same contents as for the other callbacks. No information but the origin is available. To signal that changes originating on the passed in node are irrelevant, return true, causing them to be filtered away; false otherwise. The other callbacks will not be called for transactions and changes that have been filtered away.
This is useful when implementing cascading or multidirectional replication solutions. Filtering by the origin allows to prevent replicating the same changes back and forth in such setups. While transactions and changes also carry information about the origin, filtering via this callback is noticeably more efficient.
50.6.4.12. Decode Filter Callback
The optionalfilter_decode_txn_cb callback is called to determine whether data that is part of the current transaction should be continued to be decoded.
typedef bool (*LogicalDecodeFilterDecodeTxnCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn);
Thectx parameter has the same contents as for the other callbacks. Thetxn parameter contains meta information about the transaction, like its XID. Note however that it can be NULL in some cases. To signal that decoding process should terminate, return true; false otherwise.
50.6.4.13. Prepare Filter Callback
The optionalfilter_prepare_cb callback is called to determine whether data that is part of the current two-phase commit transaction should be considered for decode at this prepare stage or as a regular one-phase transaction atCOMMIT PREPARED time later.
typedef bool (*LogicalDecodeFilterPrepareCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, TransactionId xid, const char *gid);
Thectx parameter has the same contents as for the other callbacks. Thetxn parameter contains meta information about the transaction. Thexid contains the XID becausetxn can be NULL in some cases. Thegid is the identifier that later identifies this transaction forCOMMIT PREPARED orROLLBACK PREPARED. The callback has to provide the same static answer for a given combination ofxid andgid every time it is called. To signal that decoding should be skipped, return true; false otherwise.
50.6.4.14. Generic Message Callback
The optionalmessage_cb callback is called whenever a logical decoding message has been decoded.
typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message);
Thetxn parameter contains meta information about the transaction, like the time stamp at which it has been committed and its XID. Note however that it can be NULL when the message is non-transactional and the XID was not assigned yet in the transaction which logged the message. Thelsn has WAL location of the message. Thetransactional says if the message was sent as transactional or not. Theprefix is arbitrary null-terminated prefix which can be used for identifying interesting messages for the current plugin. And finally themessage parameter holds the actual message ofmessage_size size.
Extra care should be taken to ensure that the prefix the output plugin considers interesting is unique. Using name of the extension or the output plugin itself is often a good choice.
50.6.5. Functions for Producing Output
To actually produce output, output plugins can write data to theStringInfo output buffer inctx->out when inside thebegin_cb,commit_cb, orchange_cb callbacks. Before writing to the output buffer,OutputPluginPrepareWrite(ctx, last_write) has to be called, and after finishing writing to the buffer,OutputPluginWrite(ctx, last_write) has to be called to perform the write. Thelast_write indicates whether a particular write was the callback's last write.
The following example shows how to output data to the consumer of an output plugin:
OutputPluginPrepareWrite(ctx, true);appendStringInfo(ctx->out, "BEGIN %u", txn->xid);OutputPluginWrite(ctx, true);