This document shows you how to define the relationship between objects in yourworkflow in Dataform by declaring dependencies.

You can define a dependency relationship between objects of a workflow.In a dependency relationship, the execution of the dependent object depends onthe execution of the dependency object. This means that Dataformruns the dependent after the dependency. You define the relationship bydeclaring dependencies inside the SQLX definition file of the dependent object.

The dependency declarations make up a dependency tree of your workflowthat determines the order in which Dataform runs your workflow actions.

You can define the dependency relationship between the followingworkflow actions:

Data source declarations

Declarations of BigQuery data sources that let you reference thesedata sources in Dataform table definitions and SQL operations.You can set a data source declaration as a dependency, but not as a dependent.

Tables

Tables that you create in Dataform based on the declared datasources or other tables in your workflow. Dataform supports thefollowing table types: table, incremental table, view, and materialized view.You can set a table as a dependency and as a dependent.

Custom SQL operations

SQL statements that Dataform runs in BigQuery as they are,without modification. You can set a custom SQL operation defined in atype: operations file as a dependency and as a dependent. To declare a customSQL operation as a dependency in theref function, you need toset thehasOutput property totruein the custom SQL operation SQLX definition file.

Assertions

Data quality test queries that you can use to test table data.Dataform runs assertions every time it updates your workflow andit alerts you if any assertions fail. You can set an assertion defined in atype: assertion file as a dependency and as a dependent by declaringdependencies in theconfig block.

You can define the dependency relationship in the following ways:

• Declare a dependency by using the Dataform coreref function to reference the dependency in aSELECT statement.
• Declare a list of dependencies in theconfig block of a SQLX definition file.

Before you begin

1. Create and initialize a development workspace in your repository.
2. Optional:Declare a data source.
3. Create at least two workflow actions:tables,assertions,data source declarations,oroperations.

Required roles

To get the permissions that you need to declare dependencies for tables, assertions, data sourcedeclarations, and custom SQL operations, ask your administrator to grant you theDataform Editor (roles/dataform.editor) IAM role on workspaces. For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Declare a dependency as an argument of theref function

To reference and automatically declare a dependency in aSELECT statement,add the dependency as an argument of theref function.

Theref function is a Dataform core built-in function that lets youreference and automatically depend on any table, data source declaration, orcustom SQL operation with thehasOutput property set totruein your workflow.

For more information about theref function, seeDataform core context methods reference.

For more information about using theref function in a table definition, seeAbout table definitions.

The following code sample shows thesource_data data source declaration addedas an argument of theref function in theincremental_table.sqlx SQLXdefinition file of an incremental table:

// filename is incremental_table.sqlxconfig { type: "incremental" }SELECT * FROM ${ref("source_data")}

In the preceding code sample,source_data is automatically declared adependency ofincremental_table.

The following code sample showssome_table table definition SQLX file addedas an argument of theref function in thecustom_assertion.sqlxSQLX definition file of an assertion:

// filename is custom_assertion.sqlxconfig { type: "assertion" }SELECT  *FROM  ${ref("some_table")}WHERE  a is null  or b is null  or c is null

In the preceding code sample,some_table is automatically declared adependency ofcustom_assertion. During execution, Dataform runssome_table first, and then runscustom_assertion oncesome_table is created.

Declare dependencies in theconfig block

To declare dependencies that are not referenced in the SQL statement definitionof the dependent, but need to be run before the table, assertion, orcustom SQL operation, follow these steps:

1. In your development workspace, in theFiles pane, expandthedefinitions/ directory.
2. Select the table, assertion, or custom SQL operation SQLX file thatyou want to edit.

3. In theconfig block of the file, enter the following code snippet:

   dependencies: [ "DEPENDENCY", ]

   ReplaceDEPENDENCY with the string target—for example, the filenameof the action that you want to add as a dependency. You can enter multipletargets, separated by commas.

4. Optional: ClickFormat.

The following code sample shows thesome_table table andsome_assertionassertion added as dependencies to theconfig block of a table definition file:

config { dependencies: [ "some_table", "some_assertion" ] }

Set assertions as dependencies

When workflow actionB depends on workflow actionA, which has assertions,the failure of the assertions of actionA does not block Dataformfrom executing actionB. To run actionB only if the assertions ofactionA pass, you need to set the assertions of actionA asdependencies of actionB.

You can set assertions as dependencies of a selected action in the followingways:

Set selected assertions as dependencies

You can manually set selected assertions as dependencies by adding themto thedependencies: [ "" ] line in theconfig block of the edited action.

For example, if actionB depends on actionA, and you want actionB to depend only on the selected assertions of actionA, you can addthose selected assertions to theconfig block of actionB.

You can manually set selected assertions as dependencies for all actiontypes except data source declarations.

Set the assertions of a selected dependency action as dependencies

You can set theincludeDependentAssertions parameter to automaticallyset all the direct assertions of a selected dependency workflow action asdependencies of the edited action. Dataform adds these assertions asdependencies during each compilation of the action to verify that thedependencies are up to date if the assertions of the dependency action change.

For example, if actionC depends on actionsA andB,but you only want actionC to depend on the assertions of actionA,you can edit actionC and set theincludeDependentAssertions parameterto automatically set all assertions of actionA as dependenciesof actionC.

You can set theincludeDependentAssertions parameter for actionsof the following types:

• table
• view
• operations

Set the assertions of all the dependency actions as dependencies

You can set thedependOnDependencyAssertions parameter to automatically setall the direct assertions from all the dependency actions of the edited actionas additional dependencies of the edited action. Dataform adds theseassertions as dependencies during each compilation of the action to verify that the dependencies are up to date if the assertions of the dependency action change.

For example, if actionC depends on actionsA andB, you can editactionC and set thedependOnDependencyAssertions parameter toautomatically set all the assertions of actionsA andB asdependencies of actionC.

You can set thedependOnDependencyAssertions parameter for actionsof the following types:

• table
• view
• operations

When you set thedependOnDependencyAssertions parameter and theincludeDependentAssertions parameters in a single file, theincludeDependentAssertions parameter takes priority. For example, if you setdependOnDependencyAssertions totrue, but you also setincludeDependentAssertions tofalse for a selected dependency action,Dataform won't add the assertions of that action to the dependencies.

The following code sample shows thedependOnDependencyAssertions andincludeDependentAssertions parameters set in the same table definition file:

// filename is tableName.sqlxconfig {type: "table",dependOnDependencyAssertions: true,dependencies: [ "actionA", {name: "actionB", includeDependentAssertions: false} ]}SELECT * FROM ${ref("actionC")}

In the preceding code sample, Dataform adds all the direct assertionsofactionA andactionC to the dependencies oftableName duringcompilation.

Set selected assertions as dependencies

To run a workflow action only when selected assertions pass, you can add theselected assertion to thedependencies: [ "" ] line in theconfig block ofthe edited action.

To set a selected assertion as a dependency of a selected workflow action,follow these steps:

1. In your development workspace, in theFiles pane, expanddefinitions/.
2. Select a workflow action SQLX file.
3. In theconfig block of the action file, enterdependencies: [ "" ].

4. Insidedependencies: [ "" ], enter the name of the action assertion orthe filename of the manual assertion that you want to set as a dependencyin one of the following formats:

   nonNull

   config {  type: "ACTION_TYPE",  dependencies: [ "ACTION_DATASET_NAME_ACTION_NAME_assertions_nonNull"]}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • ACTION_DATASET_NAME: the name of the dataset in which theaction is defined.The default dataset isdefined in the workflow settings file.
   • ACTION_NAME: the name of the action in which the assertionis defined.

   rowConditions

   config {  type: "ACTION_TYPE",  dependencies: [ "ACTION_DATASET_NAME_ACTION_NAME_assertions_rowConditions"]}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • DATASET_NAME: the name of the dataset in which the action is defined.The default dataset isdefined in the workflow settings file.
   • ACTION_NAME: the name of the action in which the assertion is defined.

   uniqueKey

   config {  type: "ACTION_TYPE",  dependencies: [ "ACTION_DATASET_NAME_ACTION_NAME_assertions_uniqueKey_INDEX"]}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • DATASET_NAME: the name of the dataset in which the tableis defined.The default dataset isdefined in the workflow settings file.
   • ACTION_NAME: the name of the table in which the assertionis defined.
   • INDEX: the index of the array of keys defined in theuniqueKey assertion that you want to add as a dependency—for example,0 or1. If only one array of keys is defined in the assertion,the index is0.

   uniqueKeys

   config {  type: "ACTION_TYPE",  dependencies: [ "ACTION_DATASET_NAME_ACTION_NAME_assertions_uniqueKeys_INDEX"]}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • DATASET_NAME: the name of the dataset in which the tableis defined.The default dataset isdefined in the workflow settings file.
   • ACTION_NAME: the name of the table in which the assertionis defined.
   • INDEX: the index of the array of keys defined in theuniqueKeys assertion that you want to add as a dependency—for example,0 or1. If only one array of keys is defined in the assertion,the index is0.

   manual assertion

   config {  type: "ACTION_TYPE",  dependencies: [ "MANUAL_ASSERTION_NAME"]}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • MANUAL_ASSERTION_NAME the name of the manual assertion.

5. To add another assertion as a dependency to the edited table, repeatStep 4.

6. Optional: ClickFormat.

The following code sample shows assertions added to tableA, which isdefined in thedataform dataset:

config {  type: "table",  assertions: {    uniqueKey: ["user_id"],    nonNull: ["user_id", "customer_id"],  }}

The following code sample shows tableA assertions added as dependenciesto tableB:

config {  type: "table",  dependencies: [ "dataform_A_assertions_uniqueKey_0",  "dataform_A_assertions_nonNull"]}

The following code sample shows a manual assertion that's defined in themanualAssertion.sqlx file and added as a dependency to a view:

config {  type: "view",  dependencies: [ "manualAssertion"]}

The following code sample shows themanual_assertion file and theassertions of thesometable table added as dependencies to a table:

config {  type: "table",  dependencies: [ "manual_assertion",  "dataform_sometable_assertions_nonNull" ,  "dataform_sometable_assertions_rowConditions"]}SELECT * FROM ${ref("referenced_table")} LEFT JOIN ...

Set the assertions of a selected action as dependencies

To run a workflow action only when all the direct assertions of a selecteddependency action pass, set theincludeDependentAssertions parameter totrue in the edited action. Dataform automatically adds the directassertions of the selected dependency action to the dependencies duringcompilation. The default value isfalse.

To set all the assertions of a selected dependency action as dependencies,follow these steps:

1. In your development workspace, in theFiles pane, expanddefinitions/.
2. Select a workflow action SQLX file.

3. In the file, set theincludeDependentAssertions parameter totruein one of the following ways:

   In theconfig block

   config {type: "ACTION_TYPE",dependencies: [{name: "dEPENDENCY_ACTION_NAME", includeDependentAssertions: true}]}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • DEPENDENCY_ACTION_NAME: the name of the dependency actionwith the assertions that you want to set as dependencies of the editedaction.

   In theSELECT statement

     config { type: "ACTION_TYPE" }  SELECT * FROM ${ref({name: "DEPENDENCY_ACTION_NAME", includeDependentAssertions: true})}

   Replace the following:

   • ACTION_TYPE: the type of workflow action:table,view, oroperations.
   • DEPENDENCY_ACTION_NAME: the name of the dependency actionwith the assertions that you want to set as dependencies of the editedaction.

4. Optional: ClickFormat.

The following code sample showstableC, which depends onviewA,tableB,and all the assertions oftableB:

// filename is tableC.sqlxconfig {type: "table",dependencies: ["viewA", {name: "tableB", includeDependentAssertions: true}]}SELECT * FROM ...

In the preceding code sample, Dataform automatically adds all thedirect assertions oftableB as dependencies totableC during compilation.

Set the assertions of all the dependency actions as dependencies

To run a workflow action only when all the direct assertions of allthe dependency actions pass, set thedependOnDependencyAssertions parametertotrue in the edited action. Dataform automatically adds the directassertions of the dependency actions as dependencies during compilation. Thedefault value isfalse.

When you set thedependOnDependencyAssertions parameter and theincludeDependentAssertions parameters in a single file, theincludeDependentAssertions parameter takes priority for the dependencyaction for which it is set.

To set all the assertions of a selected dependency action as dependencies,follow these steps:

1. In your development workspace, in theFiles pane, expanddefinitions/.
2. Select a workflow action SQLX file.

3. In the file, set thedependOnDependencyAssertions parameter totruein the following format:

   config {type: "ACTION_TYPE",dependOnDependencyAssertions: true,dependencies: [ "dependency1", "dependency2" ]}

   ReplaceACTION_TYPE: the type of workflow action.Supported values includetable,view, andoperations.

4. Optional: ClickFormat.

The following code sample showssometableE, which depends onsometableA,sometabletableB,sometableC,sometableD, and all the direct assertions ofthe dependency tables:

// filename is sometableE.sqlxconfig {type: "table",dependOnDependencyAssertions: true,dependencies: [ "sometableA", "sometableB" ]}SELECT * FROM ${ref("sometableC")}SELECT * FROM ${ref("sometableD")}

In the preceding code sample, Dataform automatically adds all thedirect assertions ofsometableA,sometableB,sometableC, andsometableDas dependencies tosometableE during compilation.

Reference a table with an overridden table name

• To reference a table with an overridden table name, in theref function,enter the overridden table name that's set inname: "".

The following code sample references a table with a name overridden tooverridden_name:

  SELECT * FROM ${ref("overridden_name")}

For more information about overriding table names,seeOverride table settings.

What's next

• To learn how to set assertions as dependencies, seeTest data quality.
• To learn how to declare a data source, seeDeclare a data source.
• To learn how to define custom SQL operations, seeCreate operations.
• To learn how to reuse code across your workflow with includes, seeReuse code across a single repository with includes.