AlloyDB

Destination AlloyDB#

This page describes how to load data in real time intoGoogle’s AlloyDB, a fully managed PostgreSQL-compatible database service.

The following steps refer to the extractedArcion self-hosted CLI download as the$REPLICANT_HOME directory.

Required permissions#

Privilege to create tables#

Make sure that the user being used for replication has the privilege to create tables on the target catalogs or schemas where you want to replicate the tables to. Use the following command to grant the privileges:

GRANTCREATEONDATABASE<catalog_name>TO<replication_user>;

Replace the following:

• catalog_name: the catalog name in the target
• replication_user: the user being used for replication

Replicant metadata#

In order to store Arcion’s replication metadata, you must ensure one of the following:

• Point to an external metadata database. For more information, seeMetadata configuration.

• Grant theCREATEDB privilege to the user being used for replication. This allows the user to create theio database. The user must also possess the privilege to create tables in theio database. Replicant uses thisio database to maintain internal checkpoint and metadata.

  The following command assigns theCREATEDB privilege to a useralex:

  ALTERUSER alexCREATEDB;

  If the user does not haveCREATEDB privilege, then follow these two steps:

  1. Create a database manually with the nameio:

     CREATEDATABASE io;

  2. Grant all privileges for theio database to that user:

     GRANTALLONDATABASE ioTO alex;

I. Set up connection configuration#

Specify your AlloyDB connection details to Replicant with a connection configuration file. You can find a sample connection configuration filealloydb.yaml in the$REPLICANT_HOME/conf/conn directory.

Specify the connection details in the following manner:

type:ALLOYDBhost:ALLOYDB_IPport:PORT_NUMBERdatabase:'DATABASE_NAME'username:'USERNAME'password:'PASSWORD'max-connections:30max-retries:10retry-wait-duration-ms:1000socket-timeout-s:60

Replace the following:

• ALLOYDB_IP: the IP address of the AlloyDB instance
• PORT_NUMBER: the port number
• DATABASE_NAME: the AlloyDB database name
• USERNAME: the username of theDATABASE_NAME user
• PASSWORD: the password associated withUSERNAME

Feel free to change the following parameter values as you need:

• max-connections: the maximum number of connections Replicant opens in AlloyDB.
• max-retries: number of times Replicant retries a failed operation.
• retry-wait-duration-ms: duration in milliseconds Replicant waits between each retry of a failed operation.
• socket-timeout-s: the timeout value in seconds specifying socket read operations. A value of0 disables socket reads.

The following demonstrates a sample connnection configuration:

type:ALLOYDBhost:12.34.456.78port:5444database:'tpch'username:'replicate'password:'Replicate#123'max-connections:30max-retries:10retry-wait-duration-ms:1000socket-timeout-s:60

II. Configure mapper file (optional)#

If you want to define data mapping from your source to AlloyDB, specify the mapping rules in the mapper file. For more information on how to define the mapping rules and run Replicant CLI with the mapper file, seeMapper configuration.

For example, the following sample applies to a PostgreSQL-to-AlloyDB pipeline:

rules:[tpch, public]:source:    - [tpch, public]

III. Configure metadata (optional)#

To ensure fault tolerance and resilience in replication, Arcion Replicant needs to maintain a number of metadata tables. Replicant uses a metadata configuration file to handle metadata-related operations. For more information, seeMetadata configuration.

The following shows a sample metadata configuration:

type:ALLOYDBconnection:host:localhostport:5435database:'tpch'username:'replicant'password:'Replicant#123'max-connections:30catalog:'io'schema:'replicate'

IV. Set up Applier configuration#

To configure replication mode according to your requirements, specify your configuration in the Applier configuration file. You can find a sample Applier configuration filealloydb.yaml in the$REPLICANT_HOME/conf/dst directory.

Configuresnapshot mode#

For operating insnapshot mode, specify your configuration under thesnapshot section of the configuration file. For example:

snapshot:threads:16batch-size-rows:5_000txn-size-rows:1_000_000skip-tables-on-failures:falsebulk-load:enable:truetype:FILE_traceDBTasks:trueuse-quoted-identifiers:trueuse-upsert-based-recovery:false

For more information about the Applier parameters forsnapshot mode, seeSnapshot mode.

Configurerealtime mode#

For operating inrealtime mode, specify your configuration under therealtime section of the conifiguration file. For example:

realtime:threads:8txn-size-rows:10000batch-size-rows:1000skip-tables-on-failures:falsereplay-replace-as-upsert:falseuse-quoted-identifiers:true# Transactional mode config# realtime:#   threads: 1#   batch-size-rows: 1000#   replay-consistency: GLOBAL#   txn-group-count: 100#   _oper-queue-size-rows: 20000#   skip-upto-cursors: #last failed cursor

For more information about the configuration parameters forrealtime mode, seeRealtime mode.