Type-2 CDC

Use Type-2 CDC#

From version 22.07.19.3 onwards, Arcion supports Type-2 CDC. Type-2 CDC enables a target to possess a history of all transactions performed in the source.

Supported platforms#

Arcion supports Type-2 CDC for the following targets:

• Databricks
• Snowflake

Overview#

Type-2 CDC enables a target to possess a history of all transactions performed in the source. For example:

• An INSERT in the source appears as an INSERT in the target.
• An UPDATE in the source appears as an INSERT in the target with additional metadata like operation performed, time of operation, and so on.
• A DELETE in the source appears as an INSERT in the target. For example, INSERT with OPER_TYPE as DELETE.

Arcion supports the following metadata related to source-specific fields:

• USER_QUERY_TIMESTAMP: Time when the user on source fires a query.
• EXTRACTION_TIMESTAMP: Time when Replicant detects the DML from logs.
• OPERATION_TYPE: Type of the operation (INSERT, UPDATE, or DELETE).

Youmust enable full row logging in the source to use Type-2 CDC.

Limitations#

Support for Type-2 CDC possesses the following limitations:

• Type-2 CDC works on sources that support CDC.
• Type-2 CDC works inrealtime andfull modes.

Enable Type-2 CDC#

1. Setcdc-metadata-type andreplay-strategy to the following values under therealtime section of theApplier configuration file:

   realtime:cdc-metadata-type:TYPE2_CDCreplay-strategy:NONE

2. In theExtractor configuration file of source, set the following parameter under thesnapshot section:

   snapshot:csv-publish-method:READ