Extractor Configuration

Extractor Configuration#

The Extractor configuration file contains all the parameters that Replicant uses while extracting data from the source database. While it is not necessary to modify most of the parameters, you can adjust them to optimize the extraction as you need. You can find sample extractor configuration files for different databases inside theconf/src directory ofyour Replicant self-hosted CLI download folder.

The configuration file can have three sections. Each corresponds to a type of replication.

• snapshot. Configurations for running insnapshot mode. You must specify your configurations for the initial data capture under this section.
• delta-snapshot. Configurations for running indelta-snapshot mode.
• realtime. Configurations for running inrealtime andfull mode. For carrying out realtime replications, specify your requirements under this section.

Snapshot Mode#

Replicant can run in snapshot mode using the default Extractor configurations. Changing the snapshot configurations is therefore not necessary. But depending on the replication job and your requirements, you can adjust or specify the these parameters. This can aid in optimizing Replicant.

The following configuration parameters are available for snapshot mode. You can specify them under thesnapshot section of the configuration file.

threads#

Maximum number of threads Replicant should use for writing to target.

fetch-size-rows#

Maximum number of records or documents Replicant can fetch at once from the source system.

lock#

Note: This parameter is not relevant for sources such as MongoDB and Cassandra which don’t support locking.

Option to do object locking on source. No locking is done by default. The following parameters are available to configure object locking:

min-job-size-rows#

Replicant chunks tables/collections into multiple jobs for replication. This configuration specifies the minimum size for each such job. The minimum size specified has a positive correlation with the memory footprint of Replicant.

max-jobs-per-chunk#

The maximum number of jobs in a chunk.

split-key#

Replicant uses this configuration to split the table/collection into multiple jobs in order to perform parallel extraction. The specified split key column must be of numeric or timestamp type.

Note: We don’t supportsplit-keys over macro/procedure queries. Curerntly, we only support splitting jobs oversplit-keys fortables andviews.

• Thesplit-key has uniform data distribution in the table (and there is minimal data skew in the values ofsplit-key) .
• An index is present onsplit-key on the source system.

split-method[v20.05.12.3]#

Replicant supports two split methods:

RANGE

Replicant splits the work in a range-based manner by uniformly distributing the split-key value ranges between the MIN and MAX values.

MODULO

The split key must be of numeric type for this split method. Each extraction job is assigned aJobID of0 toJOB-CNT -1. Each job then pulls data from the source whereMOD(split-key, JOB-CNT) = JobID.

extraction-method[v20.07.02.1]#

Replicant supports the following extraction methods:

QUERY

The default extraction method.

TPT

Stands for Teradata Parallel Transporter Utility (TPT). Supported only for Teradata as a source.

CSVLOAD

Extraction from the CSV files containing the data already exported from tables.

DSBULK[v21.05.04.1]

Extraction using the DataStax Bulk Loader (DSBulk) tool. Supported only for Cassandra.

enable-extraction-governor[v23.03.31.1]#

true orfalse

Whether to throttle the Extractor when memory consumption of Replicant hits threshold limit. Throttling prevents out of memory (OOM) errors and allows the Applier to catch up and free up memory.

If you enable this feature, it monitors the memory usage of Replicant. If Replicant’s memory usage goes over 80%, the Extractor slows down. This allows the Applier threads to finish applying data to the target database and free up memory.

Default:false.

extraction-governor-wait-ms[v23.03.31.1]#

Duration in milliseconds you want Extractor threads to sleep for. Use this parameter in conjunction withenable-extraction-governor.

Default:100.

tpt-num-files-per-job[v20.07.02.1]#

This parameter is only applicable when theextraction-method isTPT. It indicates how many CSV files should be exported by each TPT job (default value set to 16).

fetch-PK#

Option to fetch (and replicate) primary key constraints for tables. Default value istrue.

fetch-UK#

Option to fetch ( and replicate) unique key constraints for tables. Default value istrue..

fetch-FK#

Option to fetch (and replicate) foreign key constraints for tables. Default value istrue..

fetch-indexes#

Option to fetch (and replicate) indexes for table. Default value istrue.

fetch-user-roles#

Option to fetch (and replicate) user/roles. The default istrue for homogeneous pipelines, butfalse otherwise.

normalize[v20.09.14.10]#

This parameter is only supported for MongoDB as a source. The configuration is used to configure the normalization of data.

fetch-schemas-from-system-tables[v20.10.07.05]#

Option to use system tables to fetch schema information. By default, the value istrue, and the option is enabled. If disabled, schemas need to be provided using--src-schemas.

allow-table-extraction-overlap[v23.03.31.1]#

true orfalse.

Controls whether or not multiple tables can overlap during extraction. If you set this parameter tofalse, Extractor extracts only one table at a time.

Default:true.

fetch-create-sql#

true orfalse.

Whether to apply exactCREATE SQL on source to target. This parameter is only supported for Oracle-to-Oracle pipeline.

fetch-identity-info[v23.05.31.9]#

true orfalse.

Controls whether or not to replicate identity information of columns (auto-increment columns). Iftrue, replication captures and replicates identity information of columns. Iffalse, replication omitsAUTO_INCREMENT (and all equivalent attributes depending on the storage) information while creating tables on the target database. You can also specify this parameter in aper-table configuration.

Default:true.

Example#

The following example captures identity column information for all tables exceptexample_table:

snapshot:threads:16fetch-identity-info:trueper-table-config:  -catalog:testdbtables:example_table:fetch-identity-info:false

per-table-config#

You can use this section to override certain configurations in specific tables if necessary.

catalog

The catlog name.

schema

The schema name.

tables

Multiple tables can be specified under this section using the following parameters:

TABLE_NAME

Specify your table name here. Under the table name, specify your table-specific requirements:

• max-jobs-per-chunk. Use this to control intra-table parallelism. Set it to1 if there is nosplit-key candidate in a given collection or table. For example, in MongoDB, nosplit-key candidate has a uniform different data type across all rows or documents in a given collection.

• split-key. Use it to specifysplit-key for this table.

• split-method[v20.05.12.3]. Use it to override the globalsplit-method for this table.

• extraction-method[v20.07.02.1]. Use it to override the globalextraction-method for this table.

• tpt-num-files-per-job[v20.07.02.1]. Use it to override the globaltpt-num-files-per-job for this table.

• row-identifier-key. To specify a list of columns that uniquely identify a row in this table. We strongly recommend to specify a subset of columns as arow-identifider-key in the following conditions:

  • If a table does not have a PK or UK defined.
  • If the table has a subset of columns that can uniquely identify rows in the table.

  Specifying an identifier can significantly improve the performance of incremental replication of this table.

• extraction-priority[v20.09.14.1]. Priority for scheduling extraction of this table. Higher value is higher priority. Both positive and negative values are allowed.Default:0.

• normalize[v20.09.14.10]. Use it to override the globalnormallize parameter for this table.

• fetch-create-sql. Use it to override the globalfetch-create-sql parameter for this table.

• fetch-identity-info. Use it to override the globalfetch-identity-info parameter for this table.

• fetch-PK. Use it to override the globalfetch-PK parameter for this table.

• fetch-UK. Use it to override the globalfetch-UK parameter for this table.

• fetch-FK. Use it to override the globalfetch-FK parameter for this table.

• fetch-indexes. Use it to override the globalfetch-indexes parameter for this table.

You can configure as many tables as necessary and specify them under each other using the preceding format we discussed. For example, configuring two tables would look like the following:

first_table:max-jobs-per-chunk:split-key:split-method:extraction-method:tpt-num-files-per-job:row-identifier-key:extraction-priority:normalize:#Only for Mongo Database as sourcede-duplicate:extract-upto-depth:second_table:max-jobs-per-chunk:split-key:split-method:extraction-method:tpt-num-files-per-job:row-identifier-key:extraction-priority:normalize:#Only for Mongo Database as sourcede-duplicate:extract-upto-depth:

Heartbeat table#

For real-time replication, you must create a heartbeat table. Replicant periodically updates this table at a configurable frequency. This table helps forcefully flush the CDC logs for all committed transactions so that Replicant can Replicate them. Notice the following:

• The table must be created in the catalog or schema that Replicant is going to replicate.
• The user configured for Replicant must have INSERT, DELETE, and UPDATE privileges to the heartbeat table.
• For simplicity, we recommend that you use the exact DDL in the Extractor configuration setup of your source database’s documentation to create the heartbeat table.

You must specify the configurations for the heartbeat table undertherealtime section of the Extractor configuration file.

Realtime mode#

Unless you have given your heartbeat table different table and column names than the ones used in the provided command to create the table, the extractor configurations under therealtime section do not need to be changed. However, changing certain parameters may improve replication performance depending on the use case.

The following configuration parameters are available for realtime mode:

threads#

Maximum number of threads Replicant uses for realtime extraction.

fetch-size-rows#

Maximum number of records or documents Replicant can fetch at once from the source system.

fetch-duration-per-extractor-slot-s#

Number of seconds a thread should continue extraction from a given replication channel or slot. For example, in MongoDB source, a MongoDB shard is one replication channel.

After a thread finishes extracting from a particular replication channel, it gets scheduled to process another replication channel. This option is relevant and important to avoid starvation from any replication channel when the number of threads provided is less than the number of replication channels.

heartbeat#

This parameter providesa heartbeat table configuration to Replicant on the source system.

To create the heartbeat table, you can use this DDL:

CREATETABLEcatalog_name.schema_name.table_name(timestamp_column_name numeric_long_data_type, replicant_ID_column string_data_typeprimarykey)

Replace the following:

• catalog_name: the catalog name
• schema_name: the schema name
• table_name: name of the heartbeat table
• timestamp_column: name of the timestamp column
• numeric_long_data_type: a numeric long data type for the timestamp column
• replicant_ID_column: the column name containing Replicant’s ID for the particular replication job
• string_data_type: a string data type for thereplicant_ID_column column

Make sure that the user you provide to Replicant has the INSERT, UPDATE, and DELETE privileges for this heartbeat table.

Use the following parameters to configure a heartbeat table:

enable

true orfalse.

Whether to enable heartbeat table mechanism. You must set this totrue for real-time replication.

catalog

Catalog of the heartbeat table.

schema

Schema of the heartbeat table.

table-name[v20.09.14.3]

Name of the heartbeat table.

timestamp-column-name

Name of the timestamp column in the heartbeat table.

replicantId-column-name

Column name containing Replicant’s ID. This parameter allows you to specify the column name for Replicant’s ID in the heartbeat table. To know how to specify Replicant IDs, see the samples inRunning Replicant.

interval-ms

Interval at which Replicant updates heartbeat table with the latest timestamp (milliseconds since epoch).

The following shows a sample heartbeat table configuration:

heartbeat:enable:truecatalog:tpchinterval-ms:100replicantId-column-name:'arcion_replicant_id'timestamp-column-name:'arcion_replicant_timestamp'

fetch-interval-s[v20.07.16.1]#

Interval in seconds after which Replicant tries to fetch the CDC log.

Not all realtime sources support this parameter.

start-position[v20.09.14.1]#

Specifies the starting log position for realtime replication. The structure for providing start positions varies for different databases.

SELECTCURRENTTIMESTAMP-CURRENT TIMEZONEAS UTC_TIMESTAMPFROM SYSIBM.SYSDUMMY1

idempotent-replay[v20.09.14.1]#

This parameter can have the following possible values:

• ALWAYS. Means always do INSERT as REPLACE.
• NONE. Means publish operation as is.
• NEVER. Default value.

normalizev[20.09.14.12]#

This parameter is only supported for MongoDB as a source. It configures the normalization of data.

fetch-identity-info[v23.05.31.9]#

true orfalse.

Controls whether or not to replicate identity information of columns (auto-increment columns). Iftrue, replication captures and replicates identity information of columns. Iffalse, replication omitsAUTO_INCREMENT (and all equivalent attributes depending on the storage) information while creating tables on the target database.

Default:true.

Important: Inrealtime mode,fetch-identity-info only applies withDDL replication. For example, whenALTER TABLE ADD COLUMN... occurs and the new column is an identity column.

Example#

The followigng example ignores identity information only during CDC replication with DDLs:

snapshot:threads:16...realtime:fetch-identity-info:false

per-table-config#

You can use this section to override certain configurations in specific tables if necessary. It followsthe same structure as described in theper-table-config of snapshot mode.

The following is a sample table-specific configuration for realtime MongoDB source:

per-table-config:-schema:tpchtables:lineitem:normalize:extract-upto-depth:3

Delta-snapshot mode#

Arcion supports a third mode of replication calleddelta-snapshot. Delta-snapshot is required when the source database does not provide access to CDC logs but the customer is interested in realtime replication—for example, Teradata. The delta-snapshot is a recurring snapshot which replicates thedelta of the records which have been inserted or updated since the previous delta-snapshot iteration. The following describes parameters of thedelta-snapshot section of the Extractor configuration file.

threads#

Maximum number of threads Replicant should use for writing to target.

fetch-size-rows#

Maximum number of records or documents Replicant can fetch at once from the source system.

lock#

Note: This configuration is not relevant for sources such as MongoDB, Cassandra which do not support locking.

Option to do object locking on source. No locking is done by default. The following parameters are available to configure object locking:

min-job-size-rows#

Replicant chunks tables/collections into multiple jobs for replication. This configuration specifies the minimum size for each such job. The minimum size specified has a positive correlation with the memory footprint of Replicant.

max-jobs-per-chunk#

The maximum number of jobs in a chunk.

split-key#

Replicant uses this configuration to split the table/collection into multiple jobs in order to perform parallel extraction. The specified split key column must be of numeric or timestamp type.

Note: We don’t supportsplit-keys over macro/procedure queries. Curerntly, we only support splitting jobs oversplit-keys fortables andviews.

• Thesplit-key has uniform data distribution in the table (and there is minimal data skew in the values ofsplit-key) .
• An index is present onsplit-key on the source system.

split-method[v20.05.12.3]#

Replicant supports two split methods:

RANGE

Replicant splits the work in a range-based manner by uniformly distributing the split-key value ranges between the MIN and MAX values.

MODULO

The split key must be of numeric type for this split method. Each extraction job is assigned aJobID of0 toJOB-CNT -1. Each job then pulls data from the source whereMOD(split-key, JOB-CNT) = JobID.

extraction-method[v20.07.02.1]#

Replicant supports the following extraction methods:

QUERY

The default extraction method.

TPT

Stands for Teradata Parallel Transporter Utility (TPT). Supported only for Teradata as a source.

CSVLOAD

Extraction from the CSV files containing the data already exported from tables.

DSBULK[v21.05.04.1]

Extraction using the DataStax Bulk Loader (DSBulk) tool. Supported only for Cassandra.

tpt-num-files-per-job[v20.07.02.1]#

This parameter is only applicable when theextraction-method isTPT. It indicates how many CSV files should be exported by each TPT job (default value set to 16).

delta-snapshot-key#

Tables requiring incremental replication must have a NON-NULL numeric or timestamp column. This column gets updated on each insert or update on each row of that table. The value of this column must be monotonically increasing and non-repeatable. We call such a column adelta-snapshot-key. This parameter lets you specify that key. Replicant uses this column to perform its incremental replication for each table being replicated that has this column (unless you override this parameter in theper-table-config for this table).

delta-snapshot-key is deprecated. Please usedelta-snapshot-keys instead.

delta-snapshot-keys[v21.12.02.1]#

This parameter lets you specify one or moredelta-snapshot keys.

Each column in this list acts as an individualdelta-snapshot-key. Updating on any of the columns in this list will trigger replication. For example:

delta-snapshot-keys: [col1, col2, col3]`

row-identifier-key#

This parameter lets you specify a globalrow-identifier-key. You can override this configuration usingper-table-config for this table).

If a table does not have a PK or UK defined on it, then we strongly recommend that you specify arow-identifier-key. This is a single column or a group of columns that are guaranteed to be unique in the table by your applications. Arcion leverages this key to achieve a much better overall performance for incremental replication (in the absence of PK or UK).

update-key#

This lets users specify a key that Replicant should use to perform deletes or updates on the target system under the following scenarios:

• PK or UK does not exist.
• There’s no uniquerow-identifier-key present in the table.

This specifies a single column or a group of columns for Replicant to use. We strongly recommended that you create an index onupdate-key on the target table explicitly to get better replication performance.

delta-snapshot-interval#

The interval between two incremental replication cycles in seconds.

This parameter allows you to specify how frequently Replicant should query the source database and pull incremental changes.

replicate-deletes#

This parameter allows you to disable delete replication.

delta-snapshot-delete-interval#

This parameter allows you to specify a different interval for delete replication that the one specified for insert or update incremental replication.

native-extract-options#

This parameter allows you to specify the following options for native extraction:

column-size-map[v20.08.13.9]#

This parameter lets users specify the column size or length to use while extracting data.

per-table-config#

You can use this section to override certain configurations in specific tables (or views, queries) if necessary. It follows the same structure as described in theper-table-config ofsnapshot mode andrealtime mode.

The following is a sample table-specific configuration for Teradata delta-snapshot source:

per-table-config:-schema:tpchtables:testTable:split-key:split-key-columnextraction-method:TPTtpt-num-files-per-job:16extraction-priority:1native-extract-options:charset:"UTF8"column-size-map:"COL1":2"COL2":4"COL3":3compression-type:"GZIP"delta-snapshot-keys: [col1, col2, col3]row-identifier-key: [col1, col2]update-key: [col1, col2]