Google Cloud Storage

Destination Google Cloud Storage (GCS)#

From version 23.04.30.4, Arcion supports replicating data intoGoogle Cloud Storage using the S3 compatibility API. Follow the instructions in this page to set up Cloud Storage as data target in Arcion Replicant.

The following steps referthe extracted Arcion self-hosted CLI download as the$REPLICANT_HOME directory.

S3 file format#

When Replicant loads data into S3, Replicant first converts the data to either a CSV or a JSON file. To better understand the data format for the CSV and JSON converted files, seeArcion Internal CDC Format for Amazon S3. We highly recommended that you read theArcion Internal CDC Format for Amazon S3 page when using S3 as the target system.

I. Set up connection configuration#

Specify your Cloud Storage connection details to Replicant with a connection configuration file. You can find a sample connection configuration filegcs.yaml in the$REPLICANT_HOME/conf/conn directory.

type:S3endpoint:service-endpoint:SERVICE_ENDPOINTsigningRegion:"SIGNING_REGION"access-key:"YOUR_HMAC_ACCESS_ID"secret-key:"YOUR_HMAC_SECRET"force-path-style: {true|false}bucket:"BUCKET_NAME"root:"ROOT_PATH_UNDER_BUCKET"stage:type:SHARED_FSroot-dir:PATH_TO_STAGE_DIRECTORYmax-connections:50

Replace the following:

• SERVICE_ENDPOINT: The service endpoint with or without the protocol—for example,https://storage.googleapis.com orstorage.googleapis.com.
• SIGNING_REGION: The region to use forAWS Signature Version 4 requests—for example,us-west-1.
• YOUR_HMAC_ACCESS_ID: TheHMAC access ID associated with the service account or the user account.
• YOUR_HMAC_SECRET: TheHMAC secret linked to the HMAC access ID you specify inYOUR_HMAC_ACCESS_ID.
• BUCKET_NAME:The Cloud Storage bucket name—for example,arcion-gcp-as-target.
• ROOT_PATH_UNDER_BUCKET: The root path under the Cloud Storage bucket. Replicant creates all data files underBUCKET_NAME/ROOT_PATH_UNDER_BUCKET. For example, if you setbucket toarcion-gcp-as-target androot toreplicant/s3dst, Replicant creates the data files underarcion-gcp-as-target/replicant/s3dst.
• PATH_TO_STAGE_DIRECTORY: Directory where Replicant stages CSV files before uploading them to Cloud Storage—for example,/home/user/stage.

Setforce-path-style totrue if you preferpath-style URL access.

II. Set up Applier configuration#

Follow the instructions inApplier configuration for Amazon S3.