Datastream uses the PostgreSQL WAL (Write Ahead Log) transaction log toread PostgreSQL streams. The log is stored in WAL files on the database server.Each record in the WAL log represents a single change to the actual data in oneof the tables in the database.

Set configuration parameters for PostgreSQL WAL files

It is recommended that you apply the following configuration settings to yourPostgreSQL database:

• max_slot_wal_keep_size: set this parameter (available only for PostgreSQL13 and above) to limit the amount of storage used by the replication slot.This is particularly important for long-running transactions, which, inextreme cases, can lead to the WAL file size taking up the entire storageand crashing the database.

  Note: Some managed databases, such as Cloud SQL and AlloyDB for PostgreSQL,don't supportmax_slot_wal_keep_size.

• statement_timeout: set this parameter to a selected value to reducelatency caused by long-running transactions. You can also usestatement_timeout as an alternative precaution measure for databases thatdon't supportmax_slot_wal_keep_size.

• wal_sender_timeout: set this parameter to0 (to disable thetimeout) or to a value greater than or equal to 10 minutes.

If you plan to create more than 10 streams, or the number of logical replicationslots that is used by other resources in addition to the number of plannedstreams exceeds 10, make sure to modify the following parameters:

• max_replication_slots: increase the value of this parameter, depending onthe number of replication slots set for your database (you need 1replication slot per stream). You can only setmax_replication_slotsat server start.

• max_wal_senders: increase the value of this parameter, so that it'sgreater than the value of themax_replication_slots parameter.You can only setmax_wal_senders when you start the server.

Optimize WAL log files

To avoid high latency of your streams and rapid growth in the size of WAL logfiles when replicating data from a PostgreSQL source, consider applying thefollowing precautions:

• Avoid large long-running operations because they can significantly increasethe size of your WAL file.
• UseUNLOGGED orTEMPORARY tables during batch operations.
• Check your WAL configuration and consider reducing the checkpoint frequency.For more information, seeWAL configuration
• Check for largeDELETE operations and consider replacing them withTRUNCATE operations. Doing this can significantly reduce WAL file data,however you need to be cautious, because Datastream doesn't replicateTRUNCATE operations.

What's next

• Learn more aboutPostgreSQL as asource.
• Learn more aboutconfiguring a source PostgreSQLdatabase.