Data mapping

Enterprise data might reside in various sources and formats, making it difficult to integrate them into a unified data model or data pipeline. Data mapping is the process of extracting and standardizing data from multiple sources in order to establish a relationship between them and the related target data fields in the destination. Some examples of using data mapping in an integration include the following:

• Extracting fields from a complex data structure such as a JSON.
• Mapping data source to the target schema.
• Transforming data by appling transform functions.
• Generating output values and storing/using them as integration variables.

Application Integration lets you perform data mapping using the followingtasks:

• Data Transformer task (Preview)
• Data Mapping task

Data Transformer task

Preview — Data Transformer task

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

TheData Transformer task is a template engine-based data mapping feature available in Application Integration. It uses Google'sJsonnet configuration language to create and edit Jsonnet templates that define the mapping relationships for specified source and target integration variables in your integration. TheData Transformer task also provides a visual mapping canvas (Diagram mode) to perform data assignments and mappings in your integrations.

Diagram mode

TheDiagram mode provides a visual canvas containing the following integration elements:

• Input. Displays the input variables of the data transformation. The source can be variables or constants. To assign an input variable, you can either select an existing variable or create a new variable. These variables are mapped with the related output variables by clicking the input element and dragging the line to map with the related output variable.
• Output. Displays the output variables of the data transformation. Target variables can be used for mapping in subsequent input rows. To assign an output variable, you can either select an existing variable or create a new variable.
• Canvas. The canvas is used to visually map the input and output variables.

For more information about variables in Application Integration, seeUsing variables in Application Integration.

The following image shows the sample layout of theData Transformer diagram mode:

Transformation operations

You can use the predefinedtransformation operations to transform and standardize mapping data in your integration. Transformation operations can have one or more input parameters, where each parameter can hold a literal value or a variable. You can use multiple mapping functions for a single input source, forming a mapping transform expression.

The end data type of an input source is based on the return type of the transform expression defined in the respective data mapping input row.

Script mode

Using theData Transformer Script editor and the supportedData Transformer functions you can write custom data mapping logic, perform variable assignments, and add or modify integration variables.

The following image shows the sample layout of theData Transformer Script editor:

For information about how to add and configure theData Transformer task, seeData Transformer task.

Data Mapping task

TheData Mapping task is a no-code low-code feature in Application Integration that provides a visual mapping canvas–Data Mapping editor–to perform data assignments and mappings in your integrations. In addition, you can also use the supportedmapping functions to further transform your data into meaningful variables/formats to make them accessible to the other tasks or triggers in your integration.

With theData Mapping task, you can:

• Use theData Mapping editor to visualize and define variable mapping for single or nested variables.
• Transform variables from one data type to another data type. TheData Mapping task lets you apply multiple mapping functions (including nested functions) to transform the variable data.

For information about how to add and configure theData Mapping task, seeData Mapping task.

Data Mapping editor and layout

TheData Mapping editor provides a visual canvas containing the following integration elements:

• Variables pane: Displays the different types of variables that are available to the integration:
  • Inputs. Input variables of the integration.
  • Outputs. Output variables of the integration.
  • Local Variables. Variables that exist within the scope of the integration.

  If no variables are listed, clickAdd + to configure a new variable.

  Clickchevron_right(Expand) to expand each variable and view the available subfields of that variable. To search for any variable or its subfield from the available variable list, clicksearch(Search variables).

  For more information about variables in Application Integration, seeVariables.

• Input column: Displays input mapping rows containing the source of the data mapping input. Source can be a literal value, a base function, or an input variable, withmapping functions. ClickVariable or Value in an input mapping row to add a source.Tip: To add an input variable,you can directly drag and drop a variable from theVariables column to theInput column.
• Output column: Displays the output mapping rows containing the related target variables for the respective input mapping row. Target variables can be used for mapping in subsequent input rows. To assign an output variable, you can either create a new variable or directly drag and drop an existing output variable from theVariables column.

The following image shows the sample layout of theData Mapping editor:

Mapping functions

TheData Mapping task provides various predefinedmapping functions to transform and standardize the mapping data in your integration. A mapping function can have one or more input parameters, wherein each parameter can further hold a literal value, a variable, or a base function with mapping functions applied. You can use multiple mapping functions for a single input source, forming a mappingtransform expression.

The end data type of an input source is based on the return type of the transform expression defined in the respective data mapping input row. TheData Mapping editor displays a validation error error under the respective data mapping input row if the return type of the input source doesn't match the return type of the corresponding output mapping target variable.

Transform expression

A transform expression is a combination of severalmapping functions that are either chained together in-series or in a nested structure. Using theData Mapping editor, you can easily insert, modify, or remove a function or a function parameter in a defined transform expression. If the defined transform expression is invalid, theData Mapping editor displays a validation error error next to the respective function or function parameter that is causing the error in the expression. To view the complete error message, hold the pointer over the validation error error icon.

The following image shows a sample mapping with validation errors in theData Mapping editor:

For more information about how to configure a mapping in aData Mapping task, seeAdd a mapping.

For information about the supported pre-defined mapping functions in Application Integration, seeSupported data types and mapping functions.

Mapping order

Mappings specified in theData Mapping editor are run in sequence from top tobottom. For example, in the preceding image,Num1 is mapped toNum1ToInt in the first row, makingNum1ToInt available for mapping in the subsequent rows.

Quotas and limits

For information about quotas and limits, seeQuotas and limits.

What's next

• Add and configure aData Transformer task (Preview)
• Learn aboutData Transformer functions (Preview)
• Add and configure aData Mapping task