Destination Amazon S3#
The following steps referthe extracted Arcion self-hosted CLI download as the$REPLICANT_HOME directory.
S3 file format#
When Replicant loads data into S3, Replicant first converts the data to either a CSV or a JSON file. To better understand the data format for the CSV and JSON converted files, seeArcion Internal CDC Format for Amazon S3. We highly recommended that you read theArcion Internal CDC Format for Amazon S3 page when using S3 as the target system.
I. Set up connection configuration#
Specify your Amazon S3 connection details to Replicant with a connection configuration file. You can find a sample connection configuration files3.yaml in the$REPLICANT_HOME/conf/conn directory.
type:S3access-key:"ACCESS_KEY_ID"secret-key:"SECRET_ACCESS_KEY"bucket:"BUCKET_NAME"root:"ROOT_PATH_UNDER_BUCKET"stage:type:SHARED_FSroot-dir:PATH_TO_STAGE_DIRECTORYfile-format: {CSV|JSON}max-connections:50Replace the following:
ACCESS_KEY_ID: Theaccess key ID of the user access key—for example,AKIAIOSFODNN7EXAMPLE.SECRET_ACCESS_KEY: Thesecret access key of the user access key—for example,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY. Make sure that the user possesses theAmazonS3FullAccess managed policy.BUCKET_NAME:The S3 bucket name.ROOT_PATH_UNDER_BUCKET: The root path under S3 bucket. Replicant creates all data files underBUCKET_NAME/ROOT_PATH_UNDER_BUCKET. For example, if you setbuckettoarcionandroottoreplicant/s3dst, Replicant creates the data files underarcion/replicant/s3dst.PATH_TO_STAGE_DIRECTORY: Directory where Replicant stages CSV files before uploading them to S3—for example,/home/user/stage.
II. Set up Applier configuration#
To configure replication mode according to your requirements, specify your configuration in the Applier configuration file. You can find a sample Applier configuration files3.yaml in the$REPLICANT_HOME/conf/dst directory. For example:
snapshot:threads:16max-file-size:33_554_432#32MBdelimiter:`,`#CSV files created will have provided delimiterquote:`”`escape:`\`include-header:false#enable or disable toggle column names as header in CSV filerealtime:threads:16