Manage Dataflow resources using custom constraints

Google Cloud Organization Policy gives you centralized, programmaticcontrol over your organization's resources. As theorganization policy administrator, you can define an organization policy,which is a set of restrictions calledconstraints that apply toGoogle Cloud resources and descendants of those resources in theGoogle Cloud Platform resource hierarchy. You can enforce organization policies atthe organization, folder, or project level.

Organization Policy providespredefined constraints for variousGoogle Cloud services. However, if you want more granular, customizablecontrol over the specific fields that are restricted in your organizationpolicies, you can also create custom organization policies.

Benefits

You can use a custom organization policy to allow or deny creation of Dataflow jobs with conditions based onsupported resource attributes, such as job name, type, and service options.

Policy inheritance

By default, organization policies are inherited by the descendants of theresources on which you enforce the policy. For example, if you enforce a policyon a folder, Google Cloud enforces the policy on all projects in thefolder. To learn more about this behavior and how to change it, seeHierarchy evaluation rules.

Pricing

The Organization Policy Service, including predefined and custom organization policies, isoffered at no charge.

Limitations

  • Custom constraints for DataflowJob resources can only be set up by using the Google Cloud console or Google Cloud CLI.

  • Custom constraints can only be enforced on theCREATE method for DataflowJob resources.

  • Newly enforced custom constraints don't apply to existing resources.

Before you begin

For more information about what organization policies and constraints are andhow they work, see theIntroduction to the Organization Policy Service.

Required roles

To get the permissions that you need to manage organization policies, ask your administrator to grant you theOrganization policy administrator (roles/orgpolicy.policyAdmin) IAM role on the organization. For more information about granting roles, seeManage access to projects, folders, and organizations.

This predefined role contains the permissions required to manage organization policies. To see the exact permissions that are required, expand theRequired permissions section:

Required permissions

The following permissions are required to manage organization policies:

  • orgpolicy.constraints.list
  • orgpolicy.policies.create
  • orgpolicy.policies.delete
  • orgpolicy.policies.list
  • orgpolicy.policies.update
  • orgpolicy.policy.get
  • orgpolicy.policy.set

You might also be able to get these permissions withcustom roles or otherpredefined roles.

Create a custom constraint

A custom constraint is defined in a YAML file by the resources, methods,conditions, and actions that are supported by the service on which you areenforcing the organization policy. Conditions for your custom constraints aredefined usingCommon Expression Language (CEL). For more information about how to buildconditions in custom constraints using CEL, see the CEL section ofCreating and managing custom constraints.

To create a YAML file for a custom constraint:

name:organizations/ORGANIZATION_ID/customConstraints/CONSTRAINT_NAMEresourceTypes:-dataflow.googleapis.com/RESOURCE_NAMEmethodTypes:-CREATEcondition:"CONDITION"actionType:ACTIONdisplayName:DISPLAY_NAMEdescription:DESCRIPTION

Replace the following:

  • ORGANIZATION_ID: your organization ID, such as123456789.

  • CONSTRAINT_NAME: the name you want for your newcustom constraint. A custom constraint must start withcustom., and canonly include uppercase letters, lowercase letters, or numbers—forexample, custom.denyPrimeJobs. The maximum length of this field is 70characters, not counting the prefix—for example,organizations/123456789/customConstraints/custom.

  • RESOURCE_NAME: the name (not the URI) of theDataflow API REST resource containing the object and fieldyou want to restrict. For example,Job.

  • CONDITION: aCEL condition that is written againsta representation of a supported service resource. Thisfield has a maximum length of 1000 characters. SeeSupported resources for more information about theresources available to write conditions against. For example,"resource.environment.serviceOptions.exists(value, value=='enable_prime')".

  • ACTION: the action to take if thecondition ismet. Supported values areALLOW andDENY.

  • DISPLAY_NAME: a human-friendly name for theconstraint. This field has a maximum length of 200 characters.

  • DESCRIPTION: a human-friendly description of theconstraint to display as an error message when the policy is violated. Thisfield has a maximum length of 2000 characters.

For more information about how to create a custom constraint, seeDefining custom constraints.

Set up a custom constraint

After you have created the YAML file for a new custom constraint, you must set it up to makeit available for organization policies in your organization. To set up a custom constraint, usethegcloud org-policies set-custom-constraint command:
gcloudorg-policiesset-custom-constraintCONSTRAINT_PATH
ReplaceCONSTRAINT_PATH with the full path to yourcustom constraint file. For example,/home/user/customconstraint.yaml.Once completed, your custom constraints are available as organization policiesin your list of Google Cloud Platform organization policies.To verify that the custom constraint exists, use thegcloud org-policies list-custom-constraints command:
gcloudorg-policieslist-custom-constraints--organization=ORGANIZATION_ID
ReplaceORGANIZATION_ID with the ID of your organization resource.For more information, seeViewing organization policies.

Enforce a custom organization policy

You can enforce a constraint by creating an organization policy that references it, and thenapplying that organization policy to a Google Cloud Platform resource.

Console

  1. In the Google Cloud console, go to theOrganization policies page.

    Go to Organization policies

  2. From the project picker, select the project for which you want to set the organization policy.
  3. From the list on theOrganization policies page, select your constraint to view thePolicy details page for that constraint.
  4. To configure the organization policy for this resource, clickManage policy.
  5. On theEdit policy page, selectOverride parent's policy.
  6. ClickAdd a rule.
  7. In theEnforcement section, select whether enforcement of this organization policy is on or off.
  8. Optional: To make the organization policy conditional on a tag, clickAdd condition. Note that if you add a conditional rule to an organization policy, you must add at least one unconditional rule or the policy cannot be saved. For more information, seeSetting an organization policy with tags.
  9. ClickTest changes to simulate the effect of the organization policy. Policy simulation isn't available for legacy managed constraints. For more information, see Test organization policy changes with Policy Simulator.
  10. To finish and apply the organization policy, clickSet policy. The policy requires up to 15 minutes to take effect.

gcloud

To create an organization policy with boolean rules, create a policy YAML file that references the constraint:

name:projects/PROJECT_ID/policies/CONSTRAINT_NAMEspec:rules:-enforce:true

Replace the following:

  • PROJECT_ID: the project on which you want to enforce your constraint.
  • CONSTRAINT_NAME: the name you defined for your custom constraint. For example,custom.denyPrimeJobs.

To enforce the organization policy containing the constraint, run the following command:

gcloudorg-policiesset-policyPOLICY_PATH

ReplacePOLICY_PATH with the full path to your organization policy YAML file. The policy requires up to 15 minutes to take effect.

Example: Create a constraint to deny creation of a job with prime enabled

gcloud

  1. Create adenyPrimeJobs.yaml constraint file with the followinginformation. ReplaceORGANIZATION_ID with yourorganization ID.

    name:organizations/ORGANIZATION_ID/customConstraints/custom.denyPrimeJobsresource_types:dataflow.googleapis.com/Jobcondition:"resource.environment.serviceOptions.exists(value,value=='enable_prime')"action_type:DENYmethod_types:CREATEdisplay_name:Restrict creation of job with prime enableddescription:Deny creation of jobs with prime enabled.

  2. Set the custom constraint.

    gcloud org-policies set-custom-constraint denyPrimeJobs.yaml

  3. Create anenforce-policy-denyPrimeJobs.yaml policy file with the followinginformation. In this example, the constraint is enforced at the projectlevel. You might also set this constraint at the organization or folder level.ReplacePROJECT_ID with your project ID.

    name:projects/PROJECT_ID/policies/custom.denyPrimeJobsspec:rules:        –enforce:true

  4. Enforce the policy by running following command.

    gcloud org-policies set-policy enforce-policy-denyPrimeJobs.yaml

  5. To test the constraint, try to create a Dataflow job with theenable_prime option. Follow theCreate a Dataflow pipeline using Javaquickstart to create a WordCount job.

    mvn -Pdataflow-runner compile \exec:java \-Dexec.mainClass=org.apache.beam.examples.WordCount \-Dexec.args="--project=PROJECT_ID \--gcpTempLocation=gs://BUCKET_NAME/temp/ \--output=gs://BUCKET_NAME/output \--runner=DataflowRunner \--region=us-central1 \--dataflowServiceOptions=enable_prime" \-Pdataflow-runner

    The output is similar to the following example:

    "details":[{"@type":"type.googleapis.com/google.rpc.ErrorInfo","reason":"CUSTOM_ORG_POLICY_VIOLATION"}]

    The audit log should show violation details like following:

    policyViolationInfo:{orgPolicyViolationInfo:{violationInfo:[0:{constraint:"customConstraints/custom.denyPrimeJobs"errorMessage:"Restrict creation of job with prime enabled"policyType:"CUSTOM_CONSTRAINT"}]}}

Expression fields for conditions

The following table contains the expression fields that you can use to createconditions. Conditions are written inCommon Expression Language (CEL).The value of the expression fields is case-sensitive.

For descriptions of the following expression fields and which values you can specify, see theDataflowJob JSON representation.

Expression fieldValue type
namestring
typestring
transformNameMappingmap
locationstring
environmentmessage
environment.serviceOptionslist of string
environment.serviceKmsKeyNamestring
environment.serviceAccountEmailstring
environment.workerRegionstring
environment.workerZonestring
environment.streamingModestring
environment.debugOptionsmessage
environment.debugOptions.enableHotKeyLoggingbool

Example use cases

Some example use cases are listed in the following table.

Use CaseActionCustom Constraint
Disallow use of prime jobDENYresource.environment.serviceOptions.exists(value, value=='enable_prime')
Prevents VMs from accepting SSH keys that are stored in project metadata.DENY!resource.environment.serviceOptions.exists(value, value=='block_project_ssh_keys')
Disallow jobs without setting the maximum number of seconds the job can runDENY!resource.environment.serviceOptions.exists(value, value.contains('max_workflow_runtime_walltime_seconds=')

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-11-24 UTC.