Manage Dataproc resources using custom constraints

Google Cloud Organization Policy gives you centralized, programmaticcontrol over your organization's resources. As theorganization policy administrator, you can define an organization policy,which is a set of restrictions called constraints that apply toGoogle Cloud resources and descendants of those resources in theGoogle Cloud resource hierarchy. You can enforce organization policies atthe organization, folder, or project level.

Organization Policy providespredefined constraints for variousGoogle Cloud services. However, if you want more granular, customizablecontrol over the specific fields that are restricted in your organizationpolicies, you can also createcustom constraints and use those customconstraints in a custom organization policy.

Benefits

You can use a custom organization policy to allow or deny specificoperations on Dataproc clusters. For example, ifa request to create or update a cluster fails to satisfycustom constraint validation as set by your organization policy,the request will fail, and an error will be returned to the caller.

Policy inheritance

By default, organization policies are inherited by the descendants of theresources on which you enforce the policy. For example, if you enforce a policyon a folder, Google Cloud enforces the policy on all projects in thefolder. To learn more about this behavior and how to change it, refer toHierarchy evaluation rules.

Pricing

The Organization Policy Service, including predefined and custom organization policies, isoffered at no charge.

Before you begin

  1. Set up your project
    1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

    2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Roles required to select or create a project

      • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
      • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.
      Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

      Go to project selector

    3. Verify that billing is enabled for your Google Cloud project.

    4. Enable the Dataproc API.

      Roles required to enable APIs

      To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

      Enable the API

    5. Install the Google Cloud CLI.

    6. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    7. Toinitialize the gcloud CLI, run the following command:

      gcloudinit

    8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Roles required to select or create a project

      • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
      • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.create permission.Learn how to grant roles.
      Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

      Go to project selector

    9. Verify that billing is enabled for your Google Cloud project.

    10. Enable the Dataproc API.

      Roles required to enable APIs

      To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

      Enable the API

    11. Install the Google Cloud CLI.

    12. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    13. Toinitialize the gcloud CLI, run the following command:

      gcloudinit
    14. Ensure that you know yourorganization ID.

Required roles

To get the permissions that you need to manage organization policies, ask your administrator to grant you the following IAM roles:

For more information about granting roles, seeManage access to projects, folders, and organizations.

These predefined roles contain the permissions required to manage organization policies. To see the exact permissions that are required, expand theRequired permissions section:

Required permissions

The following permissions are required to manage organization policies:

  • orgpolicy.constraints.list
  • orgpolicy.policies.create
  • orgpolicy.policies.delete
  • orgpolicy.policies.list
  • orgpolicy.policies.update
  • orgpolicy.policy.get
  • orgpolicy.policy.set

You might also be able to get these permissions withcustom roles or otherpredefined roles.

Create a custom constraint

A custom constraint is defined in a YAML file by the resources, methods,conditions, and actions it is applied to. Dataproc supportscustom constraints that are applied to theCREATE andUPDATE methods of theCLUSTER resource (seeDataproc constraints on resources and operations).

To create a YAML file for a Dataproc custom constraint:

name:organizations/ORGANIZATION_ID/customConstraints/CONSTRAINT_NAMEresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-METHODcondition:"CONDITION"actionType:ACTIONdisplayName:DISPLAY_NAMEdescription:DESCRIPTION

Replace the following:

  • ORGANIZATION_ID: your organization ID, such as123456789.

  • CONSTRAINT_NAME: the name you want for your newcustom constraint. A custom constraint must start withcustom., and canonly include uppercase letters, lowercase letters, or numbers, forexample, custom.dataprocEnableComponentGateway. The maximum length of this field is 70characters, not counting the prefix, for example,organizations/123456789/customConstraints/custom.

  • METHOD: When creating a cluster creation constraint,specifyCREATE. When creating a clusterUPDATE constraint, specify both asfollows:
    methodTypes:- CREATE- UPDATE

  • CONDITION: aCEL condition that is written againsta representation of a supported service resource. This field has a maximumlength of 1000 characters. SeeSupported resources formore information about the resources available to write conditions against.For example,"resource.config.endpointConfig.enableHttpPortAccess==true".

  • ACTION: the action to take if thecondition ismet. This can be eitherALLOW orDENY.

  • DISPLAY_NAME: a human-friendly name for theconstraint, for example, "Enforce enabling Dataproc Component Gateway".This field has a maximum length of 200 characters.

  • DESCRIPTION: a human-friendly description of theconstraint to display as an error message when the policy is violated, forexample, "Only allow Dataproc cluster creation if the Component Gateway isenabled". This field has a maximum length of 2000 characters.

For more information about how to create a custom constraint, seeDefining custom constraints.

Set up a custom constraint

Console

To create a custom constraint, do the following:

  1. In the Google Cloud console, go to theOrganization policies page.

    Go to Organization policies

  2. From the project picker, select the project that you want to set the organization policy for.
  3. ClickCustom constraint.
  4. In theDisplay name box, enter a human-readable name for the constraint. This name is used in error messages and can be used for identification and debugging. Don't use PII or sensitive data in display names because this name could be exposed in error messages. This field can contain up to 200 characters.
  5. In theConstraint ID box, enter the name that you want for your new custom constraint. A custom constraint can only contain letters (including upper and lowercase) or numbers, for examplecustom.disableGkeAutoUpgrade. This field can contain up to 70 characters, not counting the prefix (custom.), for example,organizations/123456789/customConstraints/custom. Don't include PII or sensitive data in your constraint ID, because it could be exposed in error messages.
  6. In theDescription box, enter a human-readable description of the constraint. This description is used as an error message when the policy is violated. Include details about why the policy violation occurred and how to resolve the policy violation. Don't include PII or sensitive data in your description, because it could be exposed in error messages. This field can contain up to 2000 characters.
  7. In theResource type box, select the name of the Google Cloud REST resource containing the object and field that you want to restrict—for example,container.googleapis.com/NodePool. Most resource types support up to 20 custom constraints. If you attempt to create more custom constraints, the operation fails.
  8. UnderEnforcement method, select whether to enforce the constraint on a RESTCREATE method or on bothCREATE andUPDATE methods. If you enforce the constraint with theUPDATE method on a resource that violates the constraint, changes to that resource are blocked by the organization policy unless the change resolves the violation.

    9. Not all Google Cloud services support both methods. To see supported methods for each service, find the service in Supported services.

  9. To define a condition, clickEdit condition.
    1. In theAdd condition panel, create a CEL condition that refers to a supported service resource, for example,resource.management.autoUpgrade == false. This field can contain up to 1000 characters. For details about CEL usage, see Common Expression Language. For more information about the service resources you can use in your custom constraints, see Custom constraint supported services.
    2. ClickSave.
  10. UnderAction, select whether to allow or deny the evaluated method if the condition is met.

    11. The deny action means that the operation to create or update the resource is blocked if the condition evaluates to true.

    The allow action means that the operation to create or update the resource is permitted only if the condition evaluates to true. Every other case except ones explicitly listed in the condition is blocked.

  11. ClickCreate constraint.

    12. When you have entered a value into each field, the equivalent YAML configuration for this custom constraint appears on the right.

gcloud

  1. To create a custom constraint, create a YAML file using the following format:
    2. name:organizations/ORGANIZATION_ID/customConstraints/CONSTRAINT_NAMEresourceTypes:-RESOURCE_NAMEmethodTypes:-CREATEcondition:"CONDITION"actionType:ACTIONdisplayName:DISPLAY_NAMEdescription:DESCRIPTION

    Replace the following:

    • ORGANIZATION_ID: your organization ID, such as123456789.
    • CONSTRAINT_NAME: the name that you want for your new custom constraint. A custom constraint can only contain letters (including upper and lowercase) or numbers, for example,custom.dataprocEnableComponentGateway. This field can contain up to 70 characters.
    • RESOURCE_NAME: the fully qualified name of the Google Cloud resource containing the object and field that you want to restrict. For example,cluster.
    • CONDITION: a CEL condition that is written against a representation of a supported service resource. This field can contain up to 1000 characters. For example,"resource.config.endpointConfig.enableHttpPortAccess==true".

      • For more information about the resources available to write conditions against, seeSupported resources.

    • ACTION: the action to take if thecondition is met. Can only beALLOW.

      • The allow action means that if the condition evaluates to true, the operation to create or update the resource is permitted. This also means that every other case except the one explicitly listed in the condition is blocked.

    • DISPLAY_NAME: a human-friendly name for the constraint. This field can contain up to 200 characters.
    • DESCRIPTION: a human-friendly description of the constraint to display as an error message when the policy is violated. This field can contain up to 2000 characters.
  2. After you have created the YAML file for a new custom constraint, you must set it up to make it available for organization policies in your organization. To set up a custom constraint, use thegcloud org-policies set-custom-constraint command:
    3. gcloudorg-policiesset-custom-constraintCONSTRAINT_PATH

    ReplaceCONSTRAINT_PATH with the full path to your custom constraint file. For example,/home/user/customconstraint.yaml.

    After this operation is complete, your custom constraints are available as organization policies in your list of Google Cloud organization policies.

  3. To verify that the custom constraint exists, use thegcloud org-policies list-custom-constraints command:
    4. gcloudorg-policieslist-custom-constraints--organization=ORGANIZATION_ID

    ReplaceORGANIZATION_ID with the ID of your organization resource.

    For more information, see Viewing organization policies.

Enforce a custom constraint

You can enforce a constraint by creating an organization policy that references it, and thenapplying that organization policy to a Google Cloud resource.

Console

  1. In the Google Cloud console, go to theOrganization policies page.

    Go to Organization policies

  2. From the project picker, select the project that you want to set the organization policy for.
  3. From the list on theOrganization policies page, select your constraint to view thePolicy details page for that constraint.
  4. To configure the organization policy for this resource, clickManage policy.
  5. On theEdit policy page, selectOverride parent's policy.
  6. ClickAdd a rule.
  7. In theEnforcement section, select whether this organization policy is enforced or not.
  8. Optional: To make the organization policy conditional on a tag, clickAdd condition. Note that if you add a conditional rule to an organization policy, you must add at least one unconditional rule or the policy cannot be saved. For more information, see Setting an organization policy with tags.
  9. ClickTest changes to simulate the effect of the organization policy. For more information, see Test organization policy changes with Policy Simulator.
  10. To enforce the organization policy in dry-run mode, clickSet dry run policy. For more information, see Create an organization policy in dry-run mode.
  11. After you verify that the organization policy in dry-run mode works as intended, set the live policy by clickingSet policy.

gcloud

  1. To create an organization policy with boolean rules, create a policy YAML file that references the constraint:
    2. name:projects/PROJECT_ID/policies/CONSTRAINT_NAMEspec:rules:-enforce:truedryRunSpec:rules:-enforce:true

    Replace the following:

    • PROJECT_ID: the project that you want to enforce your constraint on.
    • CONSTRAINT_NAME: the name you defined for your custom constraint. For example,custom.dataprocEnableComponentGateway.
  2. To enforce the organization policy indry-run mode, run the following command with thedryRunSpec flag:
    3. gcloudorg-policiesset-policyPOLICY_PATH\--update-mask=dryRunSpec

    ReplacePOLICY_PATH with the full path to your organization policy YAML file. The policy requires up to 15 minutes to take effect.

  3. After you verify that the organization policy in dry-run mode works as intended, set the live policy with theorg-policies set-policy command and thespec flag:
    4. gcloudorg-policiesset-policyPOLICY_PATH\--update-mask=spec

    ReplacePOLICY_PATH with the full path to your organization policy YAML file. The policy requires up to 15 minutes to take effect.

Test the custom constraint

The following cluster creation example assumes a custom organization policy hasbeen created and enforced on cluster creation to require enabling theComponent Gateway(resource.config.endpointConfig.enableHttpPortAccess==true).

gclouddataprocclusterscreateexample-cluster\--project=PROJECT_ID\--zone=COMPUTE_ZONE

Sample output (by default, the Component Gateway is not enabled when aDataproc cluster is created):

Operation denied by custom org policies: ["customConstraints/custom.dataprocEnableComponentGateway": "Only allow Dataproc cluster creation if the Component Gateway is enabled"]

Dataproc constraints on resources and operations

The following Dataproc custom constraint fields areavailable to use when youcreate or update a Dataproc cluster.Note that when updating a cluster, only the constraints related to editable clusterparameters are supported (seeUpdating a cluster).

  • Compute Engine network configuration (networkUri, internalIpOnly, serviceAccount, and metadata)
    • resource.config.gceClusterConfig.networkUri
    • resource.config.gceClusterConfig.internalIpOnly
    • resource.config.gceClusterConfig.serviceAccount
    • resource.config.gceClusterConfig.metadata
  • Compute Engine instance group configuration (imageUri and machineTypeUri)
    • resource.config.masterConfig.imageUri
    • resource.config.masterConfig.machineTypeUri
    • resource.config.workerConfig.imageUri
    • resource.config.workerConfig.machineTypeUri
    • resource.config.secondaryWorkerConfig.imageUri
    • resource.config.secondaryWorkerConfig.machineTypeUri
  • Compute Engine instance group disk configuration (bootDiskType, bootDiskSizeGb, numLocalSsds, and localSsdInterface)
    • resource.config.masterConfig.diskConfig.bootDiskType
    • resource.config.workerConfig.diskConfig.bootDiskType
    • resource.config.secondaryWorkerConfig.diskConfig.bootDiskType
    • resource.config.masterConfig.diskConfig.bootDiskSizeGb
    • resource.config.workerConfig.diskConfig.bootDiskSizeGb
    • resource.config.secondaryWorkerConfig.diskConfig.bootDiskSizeGb
    • resource.config.masterConfig.diskConfig.numLocalSsds
    • resource.config.workerConfig.diskConfig.numLocalSsds
    • resource.config.secondaryWorkerConfig.diskConfig.numLocalSsds
    • resource.config.masterConfig.diskConfig.localSsdInterface
    • resource.config.workerConfig.diskConfig.localSsdInterface
    • resource.config.secondaryWorkerConfig.diskConfig.localSsdInterface
  • Initialization actions (executableFile)
    • resource.config.initializationActions.executableFile
  • Software config (imageVersion, properties, and optionalComponents)
    • resource.config.softwareConfig.imageVersion
    • resource.config.softwareConfig.properties
    • resource.config.softwareConfig.optionalComponents
  • Kerberos config (enableKerberos and crossRealmTrustKdc)
    • resource.config.securityConfig.kerberosConfig.enableKerberos
    • resource.config.securityConfig.kerberosConfig.crossRealmTrustKdc
  • Component gateway (enableHttpPortAccess)
    • resource.config.endpointConfig.enableHttpPortAccess
  • Metastore config (dataprocMetastoreService)
    • resource.config.metastoreConfig.dataprocMetastoreService
  • Persistent Disk CMEK (gcePdKmsKeyName)
    • resource.config.encryptionConfig.gcePdKmsKeyName
  • Cluster labels
    • resource.labels
  • Cluster size
    • resource.config.masterConfig.numInstances
    • resource.config.workerConfig.numInstances
    • resource.config.secondaryWorkerConfig.numInstances
  • Autoscaling
    • resource.config.autoscalingConfig.policyUri
  • Shielded instance config (enableSecureBoot, enableVtpm and enableIntegrityMonitoring)
    • resource.config.gceClusterConfig.shieldedInstanceConfig.enableSecureBoot
    • resource.config.gceClusterConfig.shieldedInstanceConfig.enableVtpm
    • resource.config.gceClusterConfig.shieldedInstanceConfig.enableIntegrityMonitoring

Example custom constraints for common use cases

The following table provides examples of custom constraints:

DescriptionConstraint syntax
Restrict the number of Dataproc worker instances to 10 or fewer when a cluster is created or updated.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocNoMoreThan10WorkersresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATE-UPDATEcondition:"resource.config.workerConfig.numInstances+resource.config.secondaryWorkerConfig.numInstances >10"actionType:DENYdisplayName:Total number of worker instances cannot be larger than 10description:Cluster cannot have more than 10 workers, including primary andsecondary workers.
Prevent application master from running on Dataproc cluster preemptible workers.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocAmPrimaryOnlyEnforcedresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATEcondition:"('dataproc:am.primary_only'inresource.config.softwareConfig.properties) &&(resource.config.softwareConfig.properties['dataproc:am.primary_only']==true)"actionType:ALLOWdisplayName:Application master cannot run on preemptible workersdescription:Property "dataproc:am.primary_only" must be "true".
Disallow custom Hive properties on Dataproc clusters.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocNoCustomHivePropertiesresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATEcondition:"resource.config.softwareConfig.properties.all(p,!p.startsWith('hive:'))"actionType:ALLOWdisplayName:Cluster cannot have custom Hive propertiesdescription:Only allow Dataproc cluster creation if no propertystarts with Hive prefix "hive:".
Disallow the use of then1-standard-2 machine type for Dataproc master instances.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocMasterMachineTyperesourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATEcondition:"resource.config.masterConfig.machineTypeUri.contains('n1-standard-2')"actionType:DENYdisplayName:Master cannot use the n1-standard-2 machine typedescription:Prevent Dataproc cluster creation if the master machine type is n1-standard-2.
Enforce the use of a specified initialization action script.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocInitActionScriptresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATEcondition:"resource.config.initializationActions.exists(action,action.executableFile=='gs://some/init-action.sh')"actionType:ALLOWdisplayName:Initialization action script "gs://some/init-action.sh" must be useddescription:Only allow Dataproc cluster creation if the "gs://some/init-action.sh"initialization action script is used.
Enforce the use of a specified persistent disk encryption key.
name:organizations/ORGANIZATION_ID/custom.dataprocPdCmekresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATEcondition:"resource.config.encryptionConfig.gcePdKmsKeyName=='projects/project-id/locations/global/keyRings/key-ring-name/cryptoKeys/key-name'"actionType:ALLOWdisplayName:Cluster PD must be encrypted with "key-name" from "key-ring-name" key-ringdescription:Only allow Dataproc cluster creation if the PD is encrypted with "key-name" from "key-ring-name" key-ring.
Enforce cluster label restrictions.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocEnvLabelresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATE-UPDATEcondition:"('env'inresource.labels) &&(resource.labels.env=='test')"actionType:DENYdisplayName:Cluster cannot have the "env=test" labeldescription:Deny Dataproc cluster creation or update if the cluster will be labeled "env=test".
Enforce the use of a non-default network.
name:organizations/ORGANIZATION_ID/customConstraints/custom.dataprocNoDefaultNetworkresourceTypes:-dataproc.googleapis.com/ClustermethodTypes:-CREATEcondition:"resource.config.gceClusterConfig.networkUri.contains('networks/default')"actionType:DENYdisplayName:Cluster cannot be created in the default networkdescription:Deny Dataproc cluster creation if the cluster will be created in the default network.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-15 UTC.